mwells
87285ba3cd
use gbmemcpy not memcpy so we can get profiler working again
...
since memcpy can't be interrupted and backtrace() called.
2015-01-13 12:25:42 -07:00
Matt Wells
b693fe1530
fix bugs related to restarting a cored shard
...
during repair mode. need to be able to resume
repair/rebuild scan.
2015-01-06 11:28:55 -08:00
Matt Wells
f52e163fb0
fix a couple bugs.
...
added out of sync indicator.
2014-12-17 14:28:32 -08:00
Matt Wells
d4179634a1
crc fixes
2014-12-16 16:38:54 -08:00
Matt
adcef39376
Merge branch 'diffbot-testing' into diffbot-matt
...
Conflicts:
Collectiondb.cpp
Collectiondb.h
Conf.cpp
Conf.h
Msg39.cpp
PageEvents.cpp
PageResults.cpp
PageTurk.cpp
Pages.cpp
Parms.cpp
Posdb.cpp
Proxy.cpp
Query.cpp
Query.h
RdbBase.cpp
RdbMap.cpp
Repair.cpp
Repair.h
SafeBuf.cpp
Spider.cpp
Tagdb.cpp
TopTree.cpp
XmlDoc.cpp
main.cpp
2014-11-20 16:53:07 -08:00
Matt
4e8a42e024
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Matt
931a1c4bc6
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Matt
4c19453ea9
working with -m32 for basic testing.
...
compiles for 64-bit.
2014-11-12 11:38:37 -08:00
Matt
96b8197ad3
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matt Wells
23c565afc8
fix a couple of cores.
...
reduce memory usage significantly by not
pre-allocating some per-collection hashtables.
2014-11-05 09:36:42 -08:00
Matt Wells
e7dd8f7956
replace long long with int64_t
2014-10-30 13:36:39 -06:00
mwells
538f6103d5
get qa tests working again.
...
fixed facet links.
made data import function actually work so we can
import data from one collection (files) into another.
made url filters profile compatible with UFP_ stuff.
2014-09-23 17:48:40 -07:00
mwells
1148be91b1
log msgs useful for debug
2014-09-19 17:06:35 -07:00
mwells
060e887f08
misc/various bug fixes.
...
fix canonical redir url bug with iframes.
2014-08-28 18:07:22 -07:00
Matt Wells
3ad667765a
fix retrying error forever pointlessly
...
when msg4 request is corrupt
2014-07-16 07:01:33 -07:00
Matt Wells
d6434191d1
nomenclature changes to reduce collissions.
...
name collection 'qatest123' for doing smoke tests,
not 'test'.
2014-03-31 15:02:17 -07:00
Matt Wells
8aa0662a27
Merge branch 'diffbot' into testing
...
Conflicts:
Make.depend
PageResults.cpp
Parms.cpp
Spider.cpp
Spider.h
gb.conf
2014-03-08 09:38:44 -07:00
Matt Wells
734ce1fc55
fix core from a high priority
...
injection insert records at the same
time as a lower priority spider.
2014-02-14 10:51:02 -08:00
Matt Wells
6c9a44367f
code checkpoint
2014-02-09 12:38:40 -07:00
Matt Wells
26b98a591a
fixed bug of not saving waiting trees!
...
took out misleading Collectiondb::getNumRecs() func.! bad
2014-01-23 01:02:11 -08:00
Matt Wells
33c5d9c07f
a lot of times rdb tree has invalid collection
...
numbers in it so fix our counting algo in case
the collection rec no longer exists!
2014-01-21 19:01:44 -08:00
Matt Wells
d091c7e959
fix hostsinagreement bug
2014-01-14 11:24:32 -08:00
Matt Wells
8a49e87a61
got code with shard rebalancing compiling.
...
now we store a "sharded by termid" bit in posdb
key for checksums, etc keys that are not sharded
by docid. save having to do disk seeks on every
host in the cluster to do a dup check, etc.
2014-01-11 16:08:42 -08:00
Matt Wells
f64b53bfb3
almost done with rebalancing code
2014-01-10 14:12:58 -08:00
Matt Wells
60dddfc669
final fixes for parms
2013-12-18 15:22:54 -08:00
mwells
22271c0bb2
do not accept msg4 add requests until in sync with host 0
2013-12-10 13:20:23 -08:00
mwells
f2d5661965
parmdb overhaul. support collection add/del
...
sync when host comes back online. use udp not tcp.
host #0 can now handle a new incoming request while
a parm change is currently outstanding.
all missed "command" parms will be received when a dead host
comes back online, too, like a tight merge for instance.
does not use msg4, uses msg3e and msg3f for syncing and
sending parms.
2013-12-10 13:09:55 -08:00
Matt Wells
2a503095d4
nothing
2013-11-04 14:41:36 -08:00
Matt Wells
4892e9eee1
fix issue of losing data destined for
...
a valid collection when one rec caused
an error because it was for a deleted collection
2013-10-30 15:48:31 -07:00
Matt Wells
615e459986
fix double round increment bug.
...
make msg4 send out adds every 500ms
not 5000ms so spider is zippier.
2013-10-24 14:05:39 -07:00
Matt Wells
ecab57ff0f
change collnum of reset collection
...
so any adds in progress will fail.
2013-10-18 15:46:00 -07:00
Matt Wells
fe97e08281
move from groups to shards. got rid of annoying
...
groupid bit mask thing.
2013-10-04 16:18:56 -07:00
mwells
9730e5f3ef
fix lost spiders from updating crawl info.
...
fix maxspidersperip limitation not being obeyed.
removed fakedb.
only add "0" time waiting tree keys to waiting tree.
only scanSpiderdb() will change their times to
a future time or add them to doledb directly.
confirmLockAcquisition() will not add to waitingtree
if max spiders per ip limit would be exceeded.
an incoming spider reply will trigger the add to
waiting tree with a time of "0".
2013-09-28 13:12:33 -06:00
mwells
40192249f9
spider speedups and fixes.
2013-09-25 11:58:03 -06:00
mwells
b90ef3de0d
more spider fixes. right after getting lock,
...
use msg12 to remove rec from doledb/doleiptable
and add 0 entry to waiting table so doledb is
again immediately repopulated with that firstIp
so we can spider multiple urls from the same ip
at the same time.
2013-09-23 20:25:28 -06:00
mwells
7c31ecff4a
fixed fakedb key support.
2013-09-23 15:16:23 -06:00
mwells
4d33737ac1
fakedb fixes
2013-09-23 08:19:54 -07:00
Matt Wells
f6e560c1f4
Initial file population.
2013-08-02 13:12:24 -07:00
