gigablast/open-source-search-engine
1
0
Fork 0
mirror of https://github.com/gigablast/open-source-search-engine.git synced 2024-10-04 20:27:43 +03:00
Code Issues Projects Releases Wiki Activity

ignore maxtocrawl for bulk jobs too

Browse Source
This commit is contained in:
Matt Wells 2014-02-16 22:24:17 -08:00
parent a4deb7ff08
commit f942183104
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Spider.cpp
View File

@ -5437,7 +5437,7 @@ void SpiderLoop::spiderDoledUrls ( ) {
// hit pages to crawl max?
if ( cr->m_maxToCrawl > 0 &&
cr->m_isCustomCrawl &&
cr->m_isCustomCrawl == 1 && // bulk jobs exempt!
cr->m_globalCrawlInfo.m_pageDownloadSuccessesThisRound >=
cr->m_maxToCrawl ) {
// now once all hosts have no urls ready to spider