Matt
efa93aad18
prevent double ./gb start calls from messing
...
things up.
2015-08-31 11:13:33 -06:00
Matt
744cd54131
Merge branch 'ia' into ia-zak
2015-08-31 09:14:27 -06:00
Matt
01d77ee220
make umsg00 electric fence code SPECIAL
2015-08-24 18:37:30 -06:00
Matt
f12d3ffd01
use useElectricFence var for clarity
2015-08-24 14:35:23 -06:00
Matt
9c686a40d3
fix realloc core from special umsg00 electric fence code
2015-08-24 14:25:04 -06:00
Matt
76bfe4a8ba
elec free is delayed.
2015-08-24 14:08:56 -06:00
Matt
49e9f5a827
fixes for umsg00 electric fence.
...
take out catdb/statsdb merging attempts.
2015-08-24 11:35:33 -06:00
Matt Wells
65f61351ee
efence only on umsg00 buffers.
...
fix BigFile so we dont realloc file buf which could change
file ptrs that are engaged in outstanding reads, and go back
to using file ptrs again.
2015-08-24 09:23:40 -07:00
Matt Wells
74ec812959
try to fix core from adding a file that already exists.
...
just return an error now. hopefully merge will try again later.
also core if you try to write recs to an rdbmap that
has already had its memory footprint reduced so we can find
that overrun bug.
2015-08-21 14:00:40 -07:00
Matt Wells
9642947136
fix so host #0 will delete then re-add collections
...
that use the same collnum but have a different name.
fixed some unlabelled safebufs.
fix core when deleting collnum from tree/buckets that
is higher than Collectiondb.m_numRecs.
fix File::m_filename safebufs that were not freed on exit.
2015-08-18 14:09:16 -07:00
Matt Wells
be1ebfbcd0
do not execute backtrace function if core
...
was in Mem.cpp basically otherwise we don't save state.
2015-08-16 20:29:14 -07:00
Matt
a1ed368d82
bring back max mem control into master controls.
...
it's useful to limit per process mem usage to prevent
oom killer because we can't save if we get killed.
overhaul diskpagecache to just use rdbcache. much simpler
and faster, but disabled for now until debugged more.
reduce min files to merge for crawlbot collections so
they stay more tightly merged to conserve fds and mem.
improved logDebugDisk msgs.
overhauled File.cpp fd pool. now it is way faster and
doesn't use any extra mem. much simpler too. although
could be sped up a little by using a linked list, but
probably is not significant enough to warrant doing right now.
increase mem ptr table from 3M to 8M slots. should really make
dynamic though. fix core from null msg20s[0]->m_r.
only call attemptMergeAll once every 60 seconds really.
do not attempt merge if already merging.
2015-08-14 12:58:54 -06:00
Zak Betz
36b8d384bd
Fixes to injector script.
...
New colors and metrics on performance graph.
2015-08-13 23:29:20 -06:00
Matt Wells
1ba57f9278
fix pesky memory leak finally
2015-07-13 17:47:34 -06:00
Matt Wells
5f84ad2c5d
raise mem table ptrs from 1.2M to 3M. shard 22 was suffering
...
really slow mem ops because of it.
2015-06-14 11:34:13 -07:00
Matt Wells
69e7b1165d
do not truncate termlist reads.
...
at least allow up to 2GB until we change
minrecsizes from 32bit to 64bit.
we were truncating at 90MB before.
2015-05-07 11:36:56 -07:00
Matt
95e3a760e9
proxy fixes
2015-03-05 11:10:40 -08:00
Matt
0eafc68a13
debug msg helper
2015-03-04 12:45:06 -08:00
Matt
e886f1bbac
replace memcpy_ass with bcopy
2015-01-14 14:12:55 -08:00
mwells
87285ba3cd
use gbmemcpy not memcpy so we can get profiler working again
...
since memcpy can't be interrupted and backtrace() called.
2015-01-13 12:25:42 -07:00
Matt
730b131bbf
added new indicators so we can make gb more stable.
...
now hosts table reports # ooms, disk read corruptions,
closed sockets from overloads, and we # of outstanding
spiders. made ping request a class so we can easily add
new indicators.
2014-12-16 16:22:50 -08:00
Matt Wells
251f7d2f22
fix core when removing row from url filters table.
...
tail safebuf was not detaching buf.
clear all of memtable on startup, use sizeof(char) not 4.
fix m_memtablesize since it can't be based on m_maxMem
because g_hostdb inits before g_conf.m_maxMem and calls
Mem::addMem()
2014-12-02 16:17:06 -08:00
Matt Wells
abfa9a500e
mem fix
2014-12-02 07:09:57 -08:00
Matt Wells
a1d673936f
fix some final issues with 64bit stuff
2014-12-02 06:48:56 -08:00
Matt
ea67c688b9
fixed a couple really nasty mem leak bugs from new facet code
2014-11-25 11:00:27 -07:00
Matt
adcef39376
Merge branch 'diffbot-testing' into diffbot-matt
...
Conflicts:
Collectiondb.cpp
Collectiondb.h
Conf.cpp
Conf.h
Msg39.cpp
PageEvents.cpp
PageResults.cpp
PageTurk.cpp
Pages.cpp
Parms.cpp
Posdb.cpp
Proxy.cpp
Query.cpp
Query.h
RdbBase.cpp
RdbMap.cpp
Repair.cpp
Repair.h
SafeBuf.cpp
Spider.cpp
Tagdb.cpp
TopTree.cpp
XmlDoc.cpp
main.cpp
2014-11-20 16:53:07 -08:00
Matt
dbd8af0eaa
-O4 put backs in makefile. efence off.
2014-11-17 18:14:13 -08:00
Matt
931a1c4bc6
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Matt
4a0554c76f
more 64bit fixes
2014-11-14 17:30:32 -08:00
Matt
4c19453ea9
working with -m32 for basic testing.
...
compiles for 64-bit.
2014-11-12 11:38:37 -08:00
Matt
96b8197ad3
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matt Wells
e7dd8f7956
replace long long with int64_t
2014-10-30 13:36:39 -06:00
Matt Wells
b13f3d24d7
replaced unsigned long long with uint64_t
2014-10-30 13:30:39 -06:00
mwells
3457245893
fix printf compiler warnings
2014-08-28 13:23:46 -07:00
mwells
caee238c46
fixes to make easier to compile on max os x.
2014-08-28 12:55:02 -07:00
Matt Wells
2137e150e7
Merge branch 'testing' into diffbot-matt
...
Conflicts:
Collectiondb.cpp
Make.depend
Parms.cpp
2014-06-27 17:17:14 -07:00
Matt Wells
3b8741a7cb
trying to prevent some cores.
2014-06-16 07:03:51 -07:00
mwells
4a2717a88f
Merge branch 'diffbot-testing' into diffbot-matt
2014-06-09 12:42:54 -07:00
mwells
628fe2336f
make code compile cleaner.
2014-06-07 14:11:12 -07:00
mwells
ee5af6b30e
more spider proxy fixes
2014-06-02 14:59:15 -07:00
mwells
7c30c6b970
make install fixes. getting ready for pkg build.
2014-05-11 14:20:24 -07:00
mwells
1b5c6a6278
create hosts.conf into cwd if not there.
...
pretty up logging system.
update admin.html
2014-04-06 21:12:52 -07:00
mwells
5ee79a4c2f
daemonize on ./gb 0 etc.
2014-04-06 15:57:38 -07:00
Matt Wells
8aa0662a27
Merge branch 'diffbot' into testing
...
Conflicts:
Make.depend
PageResults.cpp
Parms.cpp
Spider.cpp
Spider.h
gb.conf
2014-03-08 09:38:44 -07:00
Matt Wells
1b62f1582b
print memtable when almost full so we can see
...
where the leak is. more spiders for ethan.
do not try to get diffbot reply if page is already json.
likely it is an injected diffbot json reply.
2014-03-04 18:19:50 -08:00
Matt Wells
94a55bf9a6
fixes for new link info code so it doesn't
...
bottleneck. got EFENCE_SIZE working so we
can use efence on large allocs only so we don't
go oom using it. might help finding some of
the out of bounds writing going on.
2014-02-25 10:55:05 -08:00
Matt Wells
ecdd167d9b
code checkpoint
2014-02-09 16:41:43 -07:00
Matt Wells
7b424a6236
always use kstart.
...
fixed restrictDomain bug of not saving parm.
sped up csv download around 2x.
2014-01-28 14:37:21 -08:00
Matt Wells
33c5d9c07f
a lot of times rdb tree has invalid collection
...
numbers in it so fix our counting algo in case
the collection rec no longer exists!
2014-01-21 19:01:44 -08:00
Matt Wells
fe3a879758
formatting changes
2014-01-19 00:38:02 -08:00
