Commit Graph

5 Commits

Author SHA1 Message Date
mwells
40192249f9 spider speedups and fixes. 2013-09-25 11:58:03 -06:00
mwells
b90ef3de0d more spider fixes. right after getting lock,
use msg12 to remove rec from doledb/doleiptable
and add 0 entry to waiting table so doledb is
again immediately repopulated with that firstIp
so we can spider multiple urls from the same ip
at the same time.
2013-09-23 20:25:28 -06:00
mwells
7c31ecff4a fixed fakedb key support. 2013-09-23 15:16:23 -06:00
mwells
4d33737ac1 fakedb fixes 2013-09-23 08:19:54 -07:00
Matt Wells
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00