14 Commits

Author	SHA1	Message	Date
mwells	4e485b6649	increase dolebuf cache time from 2 to 5 mins ... for better performance. cache empty dolebufs if winner tree list was not from cache, so in case we have a huge spiderdb scan list of urls we aren't spidering we can cache it, like twitter.com e.g. do not call strstr in getUrlFilterNum2() for .css? or /print/ since it was taking way too much cpu time.	2015-02-21 15:17:28 -07:00
Matt	4c19453ea9	working with -m32 for basic testing. ... compiles for 64-bit.	2014-11-12 11:38:37 -08:00
Matt	96b8197ad3	now it compiles with -m32	2014-11-10 14:45:11 -08:00
Matt Wells	e7dd8f7956	replace long long with int64_t	2014-10-30 13:36:39 -06:00
mwells	74385939c6	put dropped requests in bold red	2014-09-04 11:01:49 -07:00
Matt Wells	b265e8d027	change bad master link to admin link	2014-07-22 10:42:30 -07:00
Matt Wells	acd05aa740	fix a few minor bugs. ... /master/->/admin/ and crawl type mismatch.	2014-03-16 10:34:58 -07:00
Matt Wells	7cd746f567	fix msge0 msg0 overload in sockets table ... when all diffbot replies timed out at once at released thousands of spiders.	2014-01-22 20:34:55 -08:00
Matt Wells	8a9b1f7a19	added diffbot retry rules. ... added maxTotalSpiders parm for all colls to follow. tried to fix msg 0x00 socket jam up.	2014-01-22 19:57:38 -08:00
Matt Wells	33c5d9c07f	a lot of times rdb tree has invalid collection ... numbers in it so fix our counting algo in case the collection rec no longer exists!	2014-01-21 19:01:44 -08:00
Matt Wells	089d7f34a0	more spiderdb spider request fixes	2014-01-19 18:00:56 -08:00
Matt Wells	b755b4d581	formatting fixes	2014-01-19 00:57:20 -08:00
Matt Wells	fe3a879758	formatting changes	2014-01-19 00:38:02 -08:00
Matt Wells	f6e560c1f4	Initial file population.	2013-08-02 13:12:24 -07:00