gigablast/open-source-search-engine
1
0
Fork 0
mirror of https://github.com/gigablast/open-source-search-engine.git synced 2024-10-04 04:07:13 +03:00
Code Issues Projects Releases Wiki Activity
3,801 Commits 3 Branches 0 Tags 222 MiB
c124eda914
Commit Graph

83 Commits

Author SHA1 Message Date
Matt
296651d416 fix getLeastLoadedInShard() to only return 
the appropriate nospider/noquery hosts when using
nospider/noquery in hosts.conf.
 2015-11-16 09:53:40 -07:00
Zak Betz
9ff387a898 More fixes to prevent spider traffic from hitting hosts with nospider 
directive.
Bug fix for msg20 lookups always being directed away from noquery hosts.
 2015-11-13 15:03:02 -07:00
Matt
fe448173d5 Merge branch 'ia' into testing 2015-11-09 11:14:00 -07:00
Zak Betz
baa817b51d Fix load balance of msg22s to use the udp slots in pinginfo. 
Fix sigchild interrupting popen, when pdftohtml segfaults
popen was hanging forever.
Fix another bug when content length in http header was one off.
 2015-11-03 11:51:19 -07:00
Zak Betz
ea139a65e6 Warc stream busy loop fixes. 
Load balance msg22 to the one with the least outstanding requests.
 2015-10-15 22:30:07 -06:00
Matt
e2fad81227 Merge branch 'testing' of github.com:gigablast/open-source-search-engine into testing 2015-09-25 08:24:54 -06:00
Matt
3ce6c7d941 Merge branch 'ia-zak' into testing 
Conflicts:
	main.cpp
 2015-09-25 08:24:12 -06:00
Matt
100888d691 fix file/dir creation permissions bugs 2015-09-21 12:44:41 -06:00
Matt
74cde33a3a just use the user's umask val for all file/dir creation 2015-09-21 11:33:38 -06:00
Matt
ce7b06fc4d all files made are now group writable. 
if you don't like that then you can make
a special group and set the directory just
group writable for that group using chmod g+s <dir>.
 2015-09-21 11:19:34 -06:00
Zak Betz
cb6ca24c26 Allow nospider and noquery on the same host. 
Fix punycoding of non-ascii domains.
 2015-09-13 17:15:31 -06:00
Zak Betz
911b2837ca Merge branch 'testing' of https://github.com/gigablast/open-source-search-engine into testing 
Conflicts:
	Makefile
	Spider.cpp
 2015-09-12 15:51:59 -06:00
Matt Wells
129c9d65db fix default hosts.conf generation 2015-09-02 09:26:03 -07:00
Zak Betz
ddf4ae2240 More testing on nospider, noquery. 
Add flags to make the nospider and noquery visible on hosts page.
 2015-08-31 10:47:19 -06:00
Matt
e373f28728 update hosts.conf generation. removed old stuff. 2015-08-31 09:29:28 -06:00
Zak Betz
60c4c5c437 Add nospider and noquery options. 2015-08-25 13:48:20 -06:00
Matt
f5a7423336 fix bug of never calling callback 2015-04-13 09:56:21 -06:00
Matt Wells
97d3b185c1 just use INCOMING udp slots/sockets for jam detection. 
this will highlight the slow nodes better.
 2015-04-08 15:52:43 -06:00
mwells
f8db6288ae ignore dead shards when doing queries so they remain fast. 2015-02-27 08:02:19 -07:00
Matt
f5383d98db if a shard is dead skip it when searching. 2015-02-27 07:28:41 -07:00
mwells
aff7e49db2 fix case bug 2015-02-06 19:55:45 -07:00
Matt Wells
85b244337c fix parm out of band core. fix hostdb conf symlink bug. 2015-02-06 15:35:00 -08:00
Matt
430a96b2c8 followup fix 2015-01-24 16:11:06 -07:00
Matt
fe14079ffe show shards with excessive udp slots to 
detect jam up.
 2015-01-22 14:47:30 -07:00
mwells
87285ba3cd use gbmemcpy not memcpy so we can get profiler working again 
since memcpy can't be interrupted and backtrace() called.
 2015-01-13 12:25:42 -07:00
Matt Wells
24fd6a1a26 fix log rotation logic. 2015-01-06 12:50:41 -08:00
Matt Wells
e5b81cfb04 fix ping age being negative in hosts table bug. 2015-01-05 15:19:46 -08:00
Matt Wells
d57f2264c4 more indicator fixes 2014-12-17 15:11:49 -08:00
Matt Wells
2fd511f002 updates 2014-12-16 17:09:25 -08:00
Matt
730b131bbf added new indicators so we can make gb more stable. 
now hosts table reports # ooms, disk read corruptions,
closed sockets from overloads, and we # of outstanding
spiders. made ping request a class so we can easily add
new indicators.
 2014-12-16 16:22:50 -08:00
Matt
4e8a42e024 text replacements for bad int32_t substitutions 2014-11-17 18:24:38 -08:00
Matt
c6605d7b33 64 bit somewhat working at runtime. need to test all functionality 
to make sure. fixes are pretty trivial.
 2014-11-12 19:18:25 -08:00
Matt
4c19453ea9 working with -m32 for basic testing. 
compiles for 64-bit.
 2014-11-12 11:38:37 -08:00
Matt
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
Matt Wells
e7dd8f7956 replace long long with int64_t 2014-10-30 13:36:39 -06:00
Matt Wells
b13f3d24d7 replaced unsigned long long with uint64_t 2014-10-30 13:30:39 -06:00
mwells
10f897e5be use gbsystem() not system() so it can turn off alarms 
since it forks.
 2014-09-11 05:01:55 -07:00
mwells
d9ae010371 shard gbfacetstr:gbxpathsitehash123456 terms by termid for speed. 
got them working again multicasting a msg 0x39 to the appropriate shard.
set special msg39request flag for better performance for those guys.
 2014-07-07 12:32:27 -07:00
Matt Wells
98b317b421 Merge branch 'diffbot-testing' into diffbot-matt 
Conflicts:
	Parms.cpp
	Query.cpp
 2014-06-27 17:23:03 -07:00
Matt Wells
e9ff8c48d8 try to remove the sluggishness from 
all hosts... should really reduce load.
 2014-06-25 17:46:28 -07:00
mwells
a09d4cd723 Merge branch 'master' into diffbot-matt 
Conflicts:
	Collectiondb.cpp
	Pages.cpp
	XmlDoc.cpp
	gb.conf
 2014-06-20 09:35:39 -07:00
mwells
494c43d5dd fix gb execution in main.cpp::getcwd2() function. 2014-06-19 06:03:11 -07:00
mwells
584af942d4 Merge branch 'testing' into diffbot-matt 
Conflicts:
	Collectiondb.cpp
	Make.depend
	Parms.cpp
 2014-06-16 20:42:28 -07:00
Matt Wells
549f8eb5bc fix bug in hosts.conf when expanding working dir. 2014-06-16 11:32:10 -07:00
mwells
4a2717a88f Merge branch 'diffbot-testing' into diffbot-matt 2014-06-09 12:42:54 -07:00
mwells
d57ce8a2df simplify compilation more. remove clones() 2014-06-07 14:26:11 -07:00
mwells
a1f1daad16 Merge branch 'master' into diffbot-matt 
Conflicts:
	Spider.cpp
 2014-06-03 11:41:46 -07:00
mwells
a811462d5f spider proxy stuff compiles now 2014-05-30 15:05:00 -07:00
Matt Wells
b0f9227bbc path fixes for gb startup 2014-05-25 10:28:13 -04:00
Matt Wells
037067170c fix for symlinks in host paths in hosts.conf 2014-05-12 20:50:11 -07:00