Matt
a9410738ae
fix permissions bug when creating directories,
...
need to put in user/group execute bit.
2015-10-07 08:26:27 -06:00
Matt
2fde3ac5bc
call umask() to fix gb process umask so
...
files created are group writable
2015-09-22 12:23:33 -06:00
Matt
100888d691
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
Matt
74cde33a3a
just use the user's umask val for all file/dir creation
2015-09-21 11:33:38 -06:00
Matt
ce7b06fc4d
all files made are now group writable.
...
if you don't like that then you can make
a special group and set the directory just
group writable for that group using chmod g+s <dir>.
2015-09-21 11:19:34 -06:00
Matt Wells
a29cf8c787
fix resuming a killed merge
2015-09-12 08:01:40 -07:00
Matt Wells
0f7910125b
make it so we can still save coll.conf on
...
malloc/free cores. do not call RdbMap::reduceMemFootprint()
on maps that are from files being merged into and we're
resuming the killed merge at startup.
2015-08-21 18:07:07 -07:00
Matt
a1ed368d82
bring back max mem control into master controls.
...
it's useful to limit per process mem usage to prevent
oom killer because we can't save if we get killed.
overhaul diskpagecache to just use rdbcache. much simpler
and faster, but disabled for now until debugged more.
reduce min files to merge for crawlbot collections so
they stay more tightly merged to conserve fds and mem.
improved logDebugDisk msgs.
overhauled File.cpp fd pool. now it is way faster and
doesn't use any extra mem. much simpler too. although
could be sped up a little by using a linked list, but
probably is not significant enough to warrant doing right now.
increase mem ptr table from 3M to 8M slots. should really make
dynamic though. fix core from null msg20s[0]->m_r.
only call attemptMergeAll once every 60 seconds really.
do not attempt merge if already merging.
2015-08-14 12:58:54 -06:00
Matt Wells
eccb969e5b
put in some fixes to deal with doledb tree
...
that seems to have m_data[i] and m_data[j]
pointing to the same thing. wtf? anyway,
deal with that. it should fix the tree or
something automatically at startup?
2015-03-08 20:36:13 -07:00
Matt Wells
93b505e7bb
fix isCollAdmin() function to return false
...
if not using coll passwords. they'll have to
be master admin.
2015-03-02 07:47:05 -08:00
Matt Wells
596a674c61
fixes for rebuilding the active list
...
in SpiderLoop class.
2015-02-12 17:00:38 -08:00
Matt
04cc8adbdd
fix &admin=0 so it works again
2015-02-12 11:16:34 -08:00
Matt
adcef39376
Merge branch 'diffbot-testing' into diffbot-matt
...
Conflicts:
Collectiondb.cpp
Collectiondb.h
Conf.cpp
Conf.h
Msg39.cpp
PageEvents.cpp
PageResults.cpp
PageTurk.cpp
Pages.cpp
Parms.cpp
Posdb.cpp
Proxy.cpp
Query.cpp
Query.h
RdbBase.cpp
RdbMap.cpp
Repair.cpp
Repair.h
SafeBuf.cpp
Spider.cpp
Tagdb.cpp
TopTree.cpp
XmlDoc.cpp
main.cpp
2014-11-20 16:53:07 -08:00
Matt
931a1c4bc6
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Matt
69ef3c14ef
fixes for repair/rebuild functionality.
...
more to come.
2014-11-13 13:04:28 -08:00
Matt
96b8197ad3
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matt Wells
3bba6881fa
fix gblocal.conf bug
2014-10-06 21:35:54 -07:00
mwells
ce56fb93ab
fix qa test so we can roll out proxy code.
2014-09-30 15:40:02 -07:00
mwells
e3dbeafa5f
more updates to cloud code
2014-09-29 18:28:36 -07:00
mwells
46290fa52f
new password systems. individual collection passwords/accessIps.
2014-09-28 18:59:49 -07:00
mwells
58f5a2dd57
save conf files safely to disk so we don't
...
lose them because the disk is full.
2014-07-29 10:02:43 -07:00
mwells
2a094accff
add qa page
2014-07-25 17:39:29 -07:00
mwells
2f8207ccf7
qa fixes
2014-07-11 19:07:49 -07:00
mwells
a154f679d1
some setup for qaspider()
2014-07-08 20:33:13 -07:00
Matt Wells
aaec46f612
added gbdocspiderdate and gbdocindexdate terms
...
just for docs and not spider reply "documents".
do not index plain terms for CT_STATUS spider reply
docs. create gb.conf if does not exist, take out of
repo.
2014-06-19 15:27:46 -07:00
Matt Wells
1bef36c03c
emergency bug fixes
2014-06-18 05:04:45 -07:00
mwells
108c281c33
fix annoying bug when adding new parms.
2014-06-10 12:29:50 -07:00
Matt Wells
1d766826ae
retry if too man docids deduped when &stream=1
2014-05-01 17:07:31 -07:00
Matt Wells
75032da5b9
fix pagination for &stream=1
2014-04-22 11:18:21 -07:00
Matt Wells
8aa0662a27
Merge branch 'diffbot' into testing
...
Conflicts:
Make.depend
PageResults.cpp
Parms.cpp
Spider.cpp
Spider.h
gb.conf
2014-03-08 09:38:44 -07:00
Matt Wells
c143ee1fba
fix core when creating a new collection because
...
we incremented m_numRecs but did not grow the ptr buffer.
also added support for localgb.conf so we can use that
instead of gb.conf to avoid git push/pull conflicts.
2014-03-07 09:05:14 -08:00
Matt Wells
3b0a571cea
fix security system to actually work now
2014-02-12 00:06:00 -07:00
Matt Wells
156b50240a
code checkpoint
2014-02-08 16:24:33 -07:00
Matt Wells
645360b730
parm simplifcations
2014-01-09 19:00:21 -08:00
Matt Wells
501f49c81b
gui and parm updates. simplifcations.
2014-01-09 17:29:18 -08:00
Matt Wells
161a5c5d6b
logging cleanups
2014-01-09 12:38:38 -08:00
mwells
76bb3d05e1
clean up logging so i can see what's going on
2013-12-10 16:41:30 -08:00
mwells
82494baa89
move CollectionRec stuff into Collectiondb files
...
for simplicity.
2013-12-10 15:28:04 -08:00
Matt Wells
dd3b49faa9
collection name hell
2013-12-08 16:44:37 -07:00
Matt Wells
25dd764dac
Merge branch 'master' into diffbot
...
Conflicts:
Makefile
PageResults.cpp
2013-11-18 16:59:33 -08:00
Matt Wells
e27646c088
cleanup fixes.
2013-11-15 15:01:56 -07:00
Matt Wells
5e30728a3a
new graphic icons. minor clean ups.
2013-11-15 14:47:05 -07:00
Matt Wells
afb5a2be64
Merge branch 'master' into diffbot
2013-11-06 10:18:04 -08:00
Matt Wells
fbc743ad5f
fixed core dump when host does not have /etc/hostname
...
file present.
2013-11-05 10:13:25 -07:00
Matt Wells
c0f1330d70
Merge branch 'master' into diffbot
...
Conflicts:
HttpServer.cpp
Makefile
PageGet.cpp
Pages.h
SafeBuf.h
2013-09-28 13:13:12 -07:00
mwells
5884951190
only do certain things if running
...
on a machine in matt wells datacenter.
like fan switching based on temps,
or printing seo links. made seo functions
weak overridable placeholder stubs so if
seo.o is linked in it will override.
include seo.o object if seo.cpp file exists
for automatic seo module building and linking.
2013-09-28 13:43:56 -06:00
mwells
8fde0c5343
added support for serialize/deserialize
...
of TYPE_SAFEBUF parms over distributed network.
2013-09-26 08:56:14 -06:00
Matt Wells
a034604cef
clean up to remove g_conf.m_useDiffbot
2013-09-16 15:00:43 -07:00
Matt Wells
19056fc3f2
show "processed" instead of "matched".
...
other fixes for spider stats. add
new crawl stats. attempts and successes.
2013-09-13 11:51:55 -07:00
Matt Wells
e3e6551e23
fix diffbot bugs.
2013-09-13 11:34:40 -07:00
