Matt Wells
a07c840a6a
join with threads when exiting -- to no avail
...
exit status is still foobar.
2015-12-17 10:15:39 -08:00
Matt
f2be319dcd
try to fix exiting w/ pthreads some more (part 2)
2015-12-17 08:38:12 -07:00
Matt Wells
a3a7635dcf
Merge branch 'diffbot' into diffbot-testing
2015-12-04 09:03:16 -08:00
Matt Wells
b049554531
added some more quickpolls.
...
improved heartbeat log msg.
timed pthread_join.
brought back max heart beat delay parm.
2015-12-04 09:02:03 -08:00
Matt Wells
34b33f478a
added gb rwtest and exposed seektest and thrutest in gb -h.
...
use -o sync when mounting ssds to avoid really slow and spiky
linux file/page cache. allow launching of more than 1 non-disk
thread again. should help with unlinking, intersects, etc.
2015-11-30 21:29:17 -07:00
Matt
68f41bd22a
debug why we don't dump core sometimes.
2015-11-18 16:11:27 -07:00
Matt
37cc4f2ba8
Merge branch 'diffbot-testing' into testing
2015-11-09 11:13:42 -07:00
Matt Wells
08b6fa67d7
improve spider performance when we have lots of collections.
...
fix core from corrupt titledb rec of some sort.
automatically turn off profiler when you get data back for
simplicity.
2015-11-01 20:23:18 -08:00
Matt
08877b6334
Merge branch 'diffbot-testing' into testing
...
Conflicts:
Parms.cpp
Threads.cpp
2015-10-12 10:39:35 -06:00
Matt
8763fd6d78
make gb shutdown easier (./gb stop)
2015-10-10 11:12:02 -07:00
Matt
4600ce0816
fix threads from freezing up just because pthread_create()
...
had an error. need to return the thread stack.
2015-10-07 07:50:44 -06:00
Matt
836aa1756d
fix threads from freezing up just because
...
pthread_create_thread() had errors. need
to return the thread stack to the linked list
of thread stacks.
2015-10-07 07:49:34 -06:00
Matt
6d90ea2e5f
try to launch threads even if none need cleanup.
...
hopefully fixes thread freeze.
2015-10-07 07:29:37 -06:00
Matt
f9c4f8fc9a
test with 2 first
2015-09-14 19:16:07 -06:00
Matt
16d90e60d1
all up to 10 filter threads now
2015-09-14 19:03:52 -06:00
Matt
a738fe993f
fix core
2015-09-12 12:28:47 -06:00
Matt
fa2ecf8681
allow for niceness 1 'merge' threads to have
...
their own queue. so merge operations take precedence
in that cpu-based thread queue over regular spider
operation rdblist merges.
2015-09-05 08:44:52 -06:00
Matt Wells
3a522f52d3
no longer use read size based thread queues.
...
re-added merge disk read thread queue.
fixed attemptMergeAll().
2015-09-04 16:14:18 -07:00
Matt Wells
c653b0989c
undo some possible averse changes
2015-09-04 11:31:43 -07:00
Matt Wells
adac0033d9
remove a couple unused parms
2015-09-03 20:41:40 -07:00
Matt Wells
611eaeca8e
added 'separate disk reads' parm so we can allow
...
spider disk threads to compete with query disk threads.
that way ppl constantly doing queries won't slow the spiders
but their queries might be slower.
2015-09-03 20:27:54 -07:00
Matt
b50f36cd73
a whole new threads stack
2015-09-03 11:59:43 -06:00
Matt
d9422d8b0e
get rid of limits on file sizes. dynamically allocate
...
file names and fixed-size File array in BigFile class. should
save gigabytes of memory in many-collection systems with
1+ million files or so.
2015-08-14 20:14:50 -06:00
Matt
a1ed368d82
bring back max mem control into master controls.
...
it's useful to limit per process mem usage to prevent
oom killer because we can't save if we get killed.
overhaul diskpagecache to just use rdbcache. much simpler
and faster, but disabled for now until debugged more.
reduce min files to merge for crawlbot collections so
they stay more tightly merged to conserve fds and mem.
improved logDebugDisk msgs.
overhauled File.cpp fd pool. now it is way faster and
doesn't use any extra mem. much simpler too. although
could be sped up a little by using a linked list, but
probably is not significant enough to warrant doing right now.
increase mem ptr table from 3M to 8M slots. should really make
dynamic though. fix core from null msg20s[0]->m_r.
only call attemptMergeAll once every 60 seconds really.
do not attempt merge if already merging.
2015-08-14 12:58:54 -06:00
Matt Wells
e1dbaf81e9
count threads whose callback has not been called
...
as 'outstanding' for purposes of shutting down so
we can update the rdbmap after having written the rdblist
to disk in rdbdump.cpp.
2015-05-03 10:58:23 -07:00
Matt Wells
f2a87358e6
try to speed up threads more
2015-02-05 15:00:18 -08:00
Matt
93fce690d6
more speedups. do not calls sigprocmask in main thread
...
before pthread_create(). instead call pthread_sigmask()
from thread like we were doing already for SIGINT.
2015-02-03 13:39:23 -08:00
Matt
cbf913522d
make pthread_create() and pthread_join()
...
faster by supplying our own guarded stack.
2015-02-03 11:58:44 -08:00
Matt
e43365fc70
more pthread_t pid_t fixes
2014-12-10 14:06:17 -08:00
Matt
619e980a97
try to fix pthread_t pid_t issues on Threads.cpp
2014-12-10 12:13:25 -08:00
Matt
4e8a42e024
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Matt
931a1c4bc6
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Matt
c6605d7b33
64 bit somewhat working at runtime. need to test all functionality
...
to make sure. fixes are pretty trivial.
2014-11-12 19:18:25 -08:00
Matt
4c19453ea9
working with -m32 for basic testing.
...
compiles for 64-bit.
2014-11-12 11:38:37 -08:00
Matt
96b8197ad3
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matt Wells
e7dd8f7956
replace long long with int64_t
2014-10-30 13:36:39 -06:00
Matt Wells
b13f3d24d7
replaced unsigned long long with uint64_t
2014-10-30 13:30:39 -06:00
Matt Wells
2256c1fdba
disable threads for disk and intersect/merge by default
...
for better performance since pthreads suck.
2014-10-02 15:24:40 -07:00
mwells
082b39e027
turn off images for qa tests.
...
fix loop stuff some more. seewms to be slower
2014-09-10 14:13:39 -07:00
mwells
7e3215d176
verified SIGCHLD being sent when thread completes
2014-09-03 11:05:15 -07:00
mwells
7bd3b42477
comment out unused code. make thread cleanups
...
basically untimed.
2014-09-03 09:48:43 -07:00
mwells
f73195870b
hacked up to debug why we're not getting
...
signals on redhat etc.
2014-08-27 10:37:03 -07:00
mwells
628fe2336f
make code compile cleaner.
2014-06-07 14:11:12 -07:00
Matt Wells
36e26436b5
logging fixes
2014-04-10 23:04:00 -07:00
mwells
5ee79a4c2f
daemonize on ./gb 0 etc.
2014-04-06 15:57:38 -07:00
Matt Wells
edbd61b0c5
thread fixes. if pthread_create fails then
...
keep thread queue and just return. will try to
relaunch later. do not count delete keys towards
shard rebalance count.
2014-03-15 20:07:02 -07:00
Matt Wells
5ea852dac3
fix core when thread fails to spawn.
2014-02-03 07:27:32 -07:00
Matt Wells
77ca55f712
fix send email notification bug.
...
increase unlink threads from 1 to 30. seemed to
be going to slow after doing a ddump with like
3000 collections. it was unlink like 1 file per sec.
2014-01-23 16:59:55 -08:00
Matt Wells
dba382f7f7
added max cpu merge threads parm and defaulted to 10
...
up from 2 for better disk reading latencies.
2014-01-21 13:11:53 -08:00
Matt Wells
4e803210ee
tons of changes from live github on neo.
...
lots of core fixes.
took out ppthtml powerpoint convert, it hangs.
dynamic rdbmap to save memory per coll.
fixed disk page cache logic and brought it
back.
2014-01-17 21:01:43 -08:00
