gigablast/open-source-search-engine
1
0
Fork 0
mirror of https://github.com/gigablast/open-source-search-engine.git synced 2024-10-05 12:47:37 +03:00
Code Issues Projects Releases Wiki Activity
1,460 Commits 3 Branches 0 Tags 222 MiB
a811462d5f
Commit Graph

4 Commits

Author SHA1 Message Date
mwells
b6e5424e32 do not download bulkjob urls in crawlbot. 
just return a fake http reply.
however, do use crawl-delay throttling
logic. deduping is already turned off for
bulk jobs so it should be ok.
 2014-03-21 12:40:38 -07:00
Matt Wells
0f3374e3f3 measure crawl delay by default from 
start of each download now. it is
a parm in msg13request.
 2013-11-26 14:07:28 -08:00
Matt Wells
e8065a0f0a enforce crawl delay perfectly. 2013-11-22 18:26:34 -08:00
Matt Wells
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00