mwells
d9ae010371
shard gbfacetstr:gbxpathsitehash123456 terms by termid for speed.
...
got them working again multicasting a msg 0x39 to the appropriate shard.
set special msg39request flag for better performance for those guys.
2014-07-07 12:32:27 -07:00
Matt Wells
98b317b421
Merge branch 'diffbot-testing' into diffbot-matt
...
Conflicts:
Parms.cpp
Query.cpp
2014-06-27 17:23:03 -07:00
Matt Wells
e9ff8c48d8
try to remove the sluggishness from
...
all hosts... should really reduce load.
2014-06-25 17:46:28 -07:00
mwells
a811462d5f
spider proxy stuff compiles now
2014-05-30 15:05:00 -07:00
mwells
c3a1c674c3
now we run gb without a hostid.
...
we use its path and the local ip to identify its
hostid # in the hosts.conf.
2014-05-11 19:36:24 -07:00
Matt Wells
9edd5c8264
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
mwells
1b5c6a6278
create hosts.conf into cwd if not there.
...
pretty up logging system.
update admin.html
2014-04-06 21:12:52 -07:00
Matt Wells
ca4aafa8a6
added host disk usage redbox and stats.
2014-02-12 09:47:44 -07:00
Matt Wells
4346fcee29
added recovery mode display in hosts table
2014-02-01 10:16:46 -08:00
Matt Wells
e8a6d8f345
fix another core from freening wrong byte sized
...
crawl info reply.
2014-01-30 20:16:41 -08:00
Matt Wells
e3f769dffe
fixes for sudden revitilization of dead crawls.
2014-01-25 11:03:15 -08:00
Matt Wells
d091c7e959
fix hostsinagreement bug
2014-01-14 11:24:32 -08:00
Matt Wells
eed606601e
added emergency msg box on all admin pages
2014-01-11 20:14:44 -08:00
Matt Wells
8a49e87a61
got code with shard rebalancing compiling.
...
now we store a "sharded by termid" bit in posdb
key for checksums, etc keys that are not sharded
by docid. save having to do disk seeks on every
host in the cluster to do a dup check, etc.
2014-01-11 16:08:42 -08:00
Matt Wells
f64b53bfb3
almost done with rebalancing code
2014-01-10 14:12:58 -08:00
Matt Wells
2cd53386ad
parm updates
2013-12-17 09:51:08 -08:00
mwells
3f8c6378b3
parmdb fixes
2013-12-10 17:45:34 -08:00
mwells
82494baa89
move CollectionRec stuff into Collectiondb files
...
for simplicity.
2013-12-10 15:28:04 -08:00
mwells
f2d5661965
parmdb overhaul. support collection add/del
...
sync when host comes back online. use udp not tcp.
host #0 can now handle a new incoming request while
a parm change is currently outstanding.
all missed "command" parms will be received when a dead host
comes back online, too, like a tight merge for instance.
does not use msg4, uses msg3e and msg3f for syncing and
sending parms.
2013-12-10 13:09:55 -08:00
Matt Wells
f69b1f46c6
fix another bug from shard change.
2013-10-04 16:49:50 -07:00
Matt Wells
0fea60eaae
fix core
2013-10-04 16:35:19 -07:00
Matt Wells
fe97e08281
move from groups to shards. got rid of annoying
...
groupid bit mask thing.
2013-10-04 16:18:56 -07:00
Matt Wells
5dc7bd2ab4
integrate diffbot from svn back into git.
2013-09-13 09:23:18 -07:00
Matt Wells
f6e560c1f4
Initial file population.
2013-08-02 13:12:24 -07:00
