Commit Graph

61 Commits

Author SHA1 Message Date
Matt Wells
b80a70a6fd fix for https urls through proxies
using newly updated tcp/loop code.
2015-02-21 09:25:54 -08:00
Matt
d14cb2d5b0 fix debug log msgs. 2015-02-17 19:15:43 -07:00
Matt
2488c1a338 added proper write callback registration into
TcpServer.cpp so we only register write callbacks
when a non-blocking write does not write all the
bytes requested of it, or when a connection does not
complete. also fixed up the sslHandshake() function
which calls SSL_connect().
2015-02-16 14:48:39 -07:00
Matt
cd9c158199 loop.cpp cleanups.
make it so non-linux os will break out
of the select() loop eventually even if select()
only gets EINTRs all the time. so we can process
shutdown cmd.
save ips.txt again for qatest123 qa collection.
do not use winnerlist cache when we have 'sitepages'
url filter expression. it messes it up.
2015-02-13 12:07:10 -08:00
Matt Wells
acbf4c582f show sigpipes and sigios for help debugging 2015-02-10 12:20:32 -08:00
Matt Wells
f2a87358e6 try to speed up threads more 2015-02-05 15:00:18 -08:00
Matt
93fce690d6 more speedups. do not calls sigprocmask in main thread
before pthread_create(). instead call pthread_sigmask()
from thread like we were doing already for SIGINT.
2015-02-03 13:39:23 -08:00
Matt
1eb9fdc658 fix some cores. fix debug log linkdb stuff. 2015-01-29 19:42:29 -07:00
Matt Wells
c9702d768c handle sigquit abrt sys to see if those are why
gb exits abruptly
2015-01-29 09:39:10 -08:00
Matt
e7a12fc2e5 fix printing stack trace on core for 64bit gb 2015-01-25 10:52:13 -07:00
Matt
c3290393f9 show just backtrace in hex on core 2015-01-25 08:46:27 -07:00
Matt
eb2a449379 Merge branch 'diffbot-testing' into testing 2015-01-20 19:13:33 -07:00
Matt
db26c7ed76 more fixes for profiler. 2015-01-20 17:02:10 -07:00
Matt
6e7b329cef speed up gb by fixing excessive calling to
gettimeofday() system call.
2015-01-20 16:06:01 -07:00
Matt
9d7a1a5868 ./gb inject <warcfile> <ip:port> now works somewhat. 2015-01-17 11:17:58 -07:00
mwells
4969aa728e Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
Conflicts:
	Address.cpp
	PagePerf.cpp
	Parms.cpp
	Xml.cpp
2015-01-13 12:29:49 -07:00
mwells
87285ba3cd use gbmemcpy not memcpy so we can get profiler working again
since memcpy can't be interrupted and backtrace() called.
2015-01-13 12:25:42 -07:00
Matt
19e493437a only print stack trace on core for 32-bit
arches. won't work for 64bit right now.
2015-01-10 13:03:12 -08:00
Matt
ef7b0c54fd log stack trace on core/segfault. 2015-01-10 12:05:39 -08:00
Matt
730b131bbf added new indicators so we can make gb more stable.
now hosts table reports # ooms, disk read corruptions,
closed sockets from overloads, and we # of outstanding
spiders. made ping request a class so we can easily add
new indicators.
2014-12-16 16:22:50 -08:00
Matt
41c8817bdb fixed summary initialization error
of the flags buffer.
fixed term freq algo. use exact term freq
for qatest123. made Summary.o -O3 again.
fix gbsystem() to disable both timers.
2014-12-06 10:14:48 -07:00
Matt
931a1c4bc6 good checkpoint. quite a few fixes. 2014-11-17 18:13:36 -08:00
Matt
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
Matt Wells
e7dd8f7956 replace long long with int64_t 2014-10-30 13:36:39 -06:00
Matt Wells
b13f3d24d7 replaced unsigned long long with uint64_t 2014-10-30 13:30:39 -06:00
mwells
bca24fb0e6 fix collection swap logic a bunch. seems to work now. 2014-09-29 13:05:20 -07:00
mwells
5df26b8bd4 sigalrm fixes 2014-09-12 02:42:00 -07:00
mwells
b50d3ee016 more select polling fixes 2014-09-11 07:16:39 -07:00
mwells
0b230995ee more fixes for inner loop code 2014-09-11 05:56:47 -07:00
mwells
ee070d9378 try to fix inner loop logic some more 2014-09-11 05:26:14 -07:00
mwells
2870c1b0be fixed system call hanging. i guess system calls fork
and it did not like or SIGALRMs because when i set the iterm
off for those it was ok.
2014-09-11 04:57:27 -07:00
mwells
e5aa7a0247 fix loop stuff some more. now seems fast again. 2014-09-10 14:56:56 -07:00
mwells
082b39e027 turn off images for qa tests.
fix loop stuff some more. seewms to be slower
2014-09-10 14:13:39 -07:00
Matt Wells
ed66bf57b7 git ride of select on writefds. pretty pointless unless
we tried to write to a socket before and the buffer was full
so the write failed. then we'd want to know if it was ready for writing
again i guess.... anyway i'm not so sure that happens a lot so i took it
out and i guess we'll see what happens. also added more udp/loop
debugging statements.
2014-09-03 21:24:51 -07:00
mwells
7e3215d176 verified SIGCHLD being sent when thread completes 2014-09-03 11:05:15 -07:00
mwells
c1a8053b52 compiler warning abatement 2014-09-03 10:55:29 -07:00
mwells
7bd3b42477 comment out unused code. make thread cleanups
basically untimed.
2014-09-03 09:48:43 -07:00
mwells
55a1a4626f more signal count stats 2014-09-03 09:18:30 -07:00
mwells
bb76ffa8b2 do not bother doing fcntl(F_SETSIG,fd) since
we use select() now. however we still do get
thread-cleanup signals from a thread when it
is done.
2014-09-03 09:02:57 -07:00
mwells
cd97944d1e fix cpu usage reporting 2014-09-03 08:38:43 -07:00
mwells
d9462e740b handle threads on EINTR from select() in Loop.cpp. 2014-09-03 06:39:19 -07:00
mwells
048aa60fd9 try polling fds now with select(). real-time signals
were somewhat of an artifact from bygone days and the
select() call when done right seems to not use any more
cpu. plus it should work well with apple os x and cygwin etc.
2014-09-02 22:05:15 -07:00
mwells
caee238c46 fixes to make easier to compile on max os x. 2014-08-28 12:55:02 -07:00
mwells
d5ef8a36e7 fix crawldelay bug. we were ignoring it. 2014-08-27 17:19:13 -07:00
Matt Wells
8772e7fffe overhauled the main loop. (BIGLOOP) in Loop.cpp.
sigtimedwait() was cutting it, it was queueing up too
many DUPLICATE signals and overflowing the rt signal queue.
now gb has its own real-time signal queueing logic that
just sets the bit of the FDs that need attention.
i think threaded reads/writes are better now too but
the performance graph is broken so i need to fix that first.
the threads page looks good though. overhaul this hopefully
is a massive and stable performance improvement.
2014-08-27 14:07:13 -07:00
mwells
317af88770 take out debug logs. 2014-08-27 10:52:44 -07:00
mwells
f73195870b hacked up to debug why we're not getting
signals on redhat etc.
2014-08-27 10:37:03 -07:00
mwells
4276700c8a fix qa tests 2014-07-29 12:21:22 -07:00
mwells
778430a543 cygwin updates 2014-06-07 14:37:21 -07:00
Matt Wells
4e803210ee tons of changes from live github on neo.
lots of core fixes.
took out ppthtml powerpoint convert, it hangs.
dynamic rdbmap to save memory per coll.
fixed disk page cache logic and brought it
back.
2014-01-17 21:01:43 -08:00