Commit Graph

32 Commits

Author SHA1 Message Date
mwells
ee070d9378 try to fix inner loop logic some more 2014-09-11 05:26:14 -07:00
mwells
2870c1b0be fixed system call hanging. i guess system calls fork
and it did not like or SIGALRMs because when i set the iterm
off for those it was ok.
2014-09-11 04:57:27 -07:00
mwells
e5aa7a0247 fix loop stuff some more. now seems fast again. 2014-09-10 14:56:56 -07:00
mwells
082b39e027 turn off images for qa tests.
fix loop stuff some more. seewms to be slower
2014-09-10 14:13:39 -07:00
Matt Wells
ed66bf57b7 git ride of select on writefds. pretty pointless unless
we tried to write to a socket before and the buffer was full
so the write failed. then we'd want to know if it was ready for writing
again i guess.... anyway i'm not so sure that happens a lot so i took it
out and i guess we'll see what happens. also added more udp/loop
debugging statements.
2014-09-03 21:24:51 -07:00
mwells
7e3215d176 verified SIGCHLD being sent when thread completes 2014-09-03 11:05:15 -07:00
mwells
c1a8053b52 compiler warning abatement 2014-09-03 10:55:29 -07:00
mwells
7bd3b42477 comment out unused code. make thread cleanups
basically untimed.
2014-09-03 09:48:43 -07:00
mwells
55a1a4626f more signal count stats 2014-09-03 09:18:30 -07:00
mwells
bb76ffa8b2 do not bother doing fcntl(F_SETSIG,fd) since
we use select() now. however we still do get
thread-cleanup signals from a thread when it
is done.
2014-09-03 09:02:57 -07:00
mwells
cd97944d1e fix cpu usage reporting 2014-09-03 08:38:43 -07:00
mwells
d9462e740b handle threads on EINTR from select() in Loop.cpp. 2014-09-03 06:39:19 -07:00
mwells
048aa60fd9 try polling fds now with select(). real-time signals
were somewhat of an artifact from bygone days and the
select() call when done right seems to not use any more
cpu. plus it should work well with apple os x and cygwin etc.
2014-09-02 22:05:15 -07:00
mwells
caee238c46 fixes to make easier to compile on max os x. 2014-08-28 12:55:02 -07:00
mwells
d5ef8a36e7 fix crawldelay bug. we were ignoring it. 2014-08-27 17:19:13 -07:00
Matt Wells
8772e7fffe overhauled the main loop. (BIGLOOP) in Loop.cpp.
sigtimedwait() was cutting it, it was queueing up too
many DUPLICATE signals and overflowing the rt signal queue.
now gb has its own real-time signal queueing logic that
just sets the bit of the FDs that need attention.
i think threaded reads/writes are better now too but
the performance graph is broken so i need to fix that first.
the threads page looks good though. overhaul this hopefully
is a massive and stable performance improvement.
2014-08-27 14:07:13 -07:00
mwells
317af88770 take out debug logs. 2014-08-27 10:52:44 -07:00
mwells
f73195870b hacked up to debug why we're not getting
signals on redhat etc.
2014-08-27 10:37:03 -07:00
mwells
4276700c8a fix qa tests 2014-07-29 12:21:22 -07:00
mwells
778430a543 cygwin updates 2014-06-07 14:37:21 -07:00
Matt Wells
4e803210ee tons of changes from live github on neo.
lots of core fixes.
took out ppthtml powerpoint convert, it hangs.
dynamic rdbmap to save memory per coll.
fixed disk page cache logic and brought it
back.
2014-01-17 21:01:43 -08:00
Matt Wells
9b080ff89c more parmdb bug fixes 2013-12-16 13:36:31 -08:00
Matt Wells
9be1ab6323 more parmdb fixes 2013-12-16 12:20:13 -08:00
Matt Wells
660f43cec7 fix bugs of pthreads junk not being async safe.
we were calling fprintf from a signal handler
(interrupt) while fprintf was currently in progress
and the pthread junk did not like that.
2013-12-15 11:41:41 -07:00
Matt Wells
e1712fc94f fix uninitialized diffbot titlerec
header parms. ignore them when not
a custom crawl.
2013-12-07 22:11:26 -07:00
Matt Wells
06edfddf31 a bunch of bug fixes, mostly spider related.
also some for pagereindex.
2013-12-07 21:56:37 -07:00
Matt Wells
6b330da240 cleanup warnings in log. 2013-09-13 14:37:35 -07:00
mwells
54c9353dbd try to fix core from g_inSigHandler being set.
it should never be set since we do not use
real time signals any more
2013-09-08 12:34:37 -06:00
mwells
ca2a024d04 fixed up thread/spider log msgs.
fixed core from calling fprintf in
alarm signal missed quickpoll handler.
2013-08-29 21:15:42 -06:00
mwells
80179525c1 when using pthreads block SIGIO
so it does not silently kill the gb
process because we no longer have a handler
for it because it was bogging down the cpu
because it went off every time a udp datagram
was sent/received and it seemed to have a ton
of overhead with it. SIGIO used to be sent
when the signal queue was full so we'd resort
to polling the file descriptors, so i'm not
sure how this will affect us. also updated
Threads.cpp to use getpidtid() instead of
getpid() to get the thread id when using
pthreads, not the process id. using pthreads
is now default behaviour even though they suck.
we used to use clone() but the newer stuff doesn't
allow us to override errno_location anymore.
2013-08-21 15:01:26 -06:00
mwells
a2a57addd9 try fixing the cpu being slammed in the
sigiohandler. seems like signals meaning
might have changed in the kernel, etc.
over the years. fixed Loop.cpp.
2013-08-20 14:12:44 -06:00
Matt Wells
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00