Ivan Skytte Jørgensen
9b02e87913
Removed global #pragma pack(4) from <types.h>
...
The global setting was causing fragility other places requiring include order
to be specific. Remoed the global "#pragma pack(4)" and replaced it with the
equally ugly "__attribute__((packed, aligned(4)))", but at least it is explicit
and doesn't interfere with system and 3rd-party header files and libraries.
Origin: e4323f397f
2021-06-18 13:07:08 +10:00
Dmitry Smirnov
b1ace63607
codespell: spelling corrections
2021-05-06 01:52:55 +10:00
Zak Betz
f10fdada73
Fix infinite loop on malformed proxy.
2017-06-02 11:28:58 -06:00
Matt Wells
9147d6bb02
fix some diffbot crawls.
...
do not spider pages at the hopcount limit
when 'only spider urls if new' is enabled.
meaning only spider each url once. (unless there is
a temporary error)
fix malformed url bug some more.
added some commented out code for indexing spider replies
(gbss docs) for certain fatal/critical errors, in which
case they are not being indexed.
2015-12-23 13:49:21 -08:00
Matt Wells
d55932d0b6
fix spider proxy table bug that seemed to be the
...
reason for the table getting so full. but in case
it does get full again added a call the hashtablex::empty()
so we don't freeze up any more.
2015-11-21 10:43:23 -08:00
Matt Wells
b3729ed214
tune spider proxy table flushing logic a bit
2015-11-21 10:29:02 -08:00
Matt Wells
3c766451d1
try to fix the proxy load balancing table logic some more.
...
seems to not cleanup after itself very well.
2015-11-21 10:20:20 -08:00
Matt Wells
6cf6abf3d9
fix spider proxy clean up algo a little
...
so it won't freeze up
2015-11-11 08:27:09 -08:00
Matt Wells
44e3b0ca19
try to fix spider proxy load table pruning bug.
2015-11-06 13:30:42 -08:00
Matt
51d68c4b3d
pass proxy info back to diffbot
2015-10-20 15:53:16 -06:00
Matt Wells
cc1fcdd8a1
fix spider proxy load table clean out again
2015-09-12 13:58:08 -07:00
Matt Wells
583974093e
clean out proxy load table more often to keep things fast.
2015-09-12 13:33:42 -07:00
Matt Wells
fb332a23c4
another fix for infinite loop in spider proxy code
2015-09-12 11:58:46 -07:00
Matt Wells
782bc9cdee
inf loop fix
2015-09-12 07:11:39 -07:00
Matt Wells
a4bfbb31f8
fix save prevention when coring in malloc/free.
2015-08-23 11:51:46 -07:00
Matt
a81dcb6442
fix slow spider proxy loop
2015-04-23 21:17:55 -06:00
Matt
95e3a760e9
proxy fixes
2015-03-05 11:10:40 -08:00
Matt Wells
480072274d
emergency proxy fixes
2015-02-19 12:49:42 -08:00
Matt
739b296cf2
fix proxy bugs
2015-02-02 13:29:52 -08:00
Matt
c15bd53e52
added support for supplying basic proxy authorization
...
to spider proxies. username:password@1.2.3.4:80
2015-02-02 13:23:38 -08:00
Matt Wells
4894bf51ce
fix core
2014-12-03 14:08:18 -08:00
Matt
4e8a42e024
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Matt
931a1c4bc6
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Matt
96b8197ad3
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matt Wells
e7dd8f7956
replace long long with int64_t
2014-10-30 13:36:39 -06:00
Matt Wells
b13f3d24d7
replaced unsigned long long with uint64_t
2014-10-30 13:30:39 -06:00
mwells
4e7152b487
fix more bugs in squid proxy implementation.
...
force squid proxy stack to use floaters.
2014-10-02 11:54:50 -07:00
mwells
42b891219d
several fixes for floater proxy through squid proxy.
...
gb needs to act like squid for the rendering machines so
it can do crawl delay backoff and load balancing over the
floaters.
2014-10-02 02:08:38 -07:00
mwells
65840d969e
update to spider proxy choose set logic
2014-10-01 10:00:24 -07:00
mwells
7d4c4e8db1
update spider proxy logic.
2014-10-01 09:26:41 -07:00
mwells
b4ca812ef8
added parm to reset proxy stats in table. erases
...
all our knowledge/stats for each proxy.
2014-09-30 17:38:59 -07:00
mwells
2af806993b
update proxy algo so not all proxies get cutoff
...
at once.
2014-09-30 13:08:35 -07:00
mwells
afd41676d2
bring back meta tag display in results again.
...
added qa tests for advanced search and api parms.
various api parm fixes and hides.
do not do test url on proxies if test url empty.
2014-09-27 15:54:55 -07:00
mwells
f81bb6d072
added floater coll override switch.
2014-09-26 21:28:04 -07:00
Matt Wells
6b797f5023
more core stability fixes. prevent core dumps
2014-07-16 12:07:39 -07:00
mwells
9249564191
now floaters are working pretty well
2014-06-30 16:26:10 -06:00
Matt Wells
859c5ee12f
fix spider proxy core
2014-06-30 12:09:51 -07:00
mwells
3f2dcda4e1
got new floater/proxy logic compiling.
2014-06-06 15:11:51 -07:00
mwells
a772e21db6
only show proxy stuff in logs when debugging is on for it
2014-06-02 17:37:43 -07:00
mwells
abbc116442
show more spider proxy stats in table
2014-06-02 17:12:25 -07:00
mwells
2582a487a5
more spider proxy fixes
2014-06-02 16:53:06 -07:00
mwells
38854e44f3
added load points in table display of spider proxies
2014-06-02 16:25:56 -07:00
mwells
1ba445ae41
update times used
2014-06-02 15:20:46 -07:00
mwells
ee5af6b30e
more spider proxy fixes
2014-06-02 14:59:15 -07:00
mwells
806cf79b73
spider proxy updates
2014-06-02 13:18:18 -07:00
mwells
a811462d5f
spider proxy stuff compiles now
2014-05-30 15:05:00 -07:00
mwells
8fb8669da1
more spider proxy updates.
2014-05-29 21:17:51 -06:00
Matt Wells
f341dba0c8
got the general framework for load-balanced/reliabled
...
floaters in place for the distributed spider network.
need to fill in the blanks now.
2014-05-27 15:21:12 -07:00
