11 Commits

Author	SHA1	Message	Date
Matt Wells	25cf0efdbf	first compiled stab at multi collection searching.	2014-03-06 10:45:13 -08:00
Matt Wells	2ab9aaeeaa	streaming csv fixes	2014-03-04 11:04:26 -08:00
Matt Wells	b1381cc610	make csv streamable, faster and take almost no memory.	2014-03-04 10:45:57 -08:00
Matt Wells	b429f12346	add logic to save memory when streaming ... over 200 results back. should fix oom when streaming back hundreds of thousands of results.	2014-02-26 20:33:35 -08:00
Matt Wells	cd6069e5a6	send single space to socket if not streaming ... and search results still not ready after 10 seconds. send it every 10 seconds to prevent client from closing socket. sped up all downloads, json and csv, but not doing "fuzzy" deduping of search results, but just deduping on page content hash. added TcpSocket::m_numDestroys to ensure we do not send heartbeat on a socket that was closed and re-opened for another client.	2014-02-13 08:45:13 -08:00
Matt Wells	8d534b8ed8	many more fixes for streaming mode	2014-02-06 18:21:22 -08:00
Matt Wells	f9dbd64056	get streaming time sliced results working	2014-02-06 14:25:44 -08:00
Matt Wells	189999509b	code checkpoint. time slicing, faster spider code ... compiling. now needs debug.	2014-02-04 17:34:43 -08:00
Matt Wells	7874c8d832	added ifdef NEEDSLICENSE	2013-12-01 14:47:08 -07:00
mwells	6c2c9f7774	trying to bring back dmoz integration.	2013-10-02 22:34:21 -06:00
Matt Wells	f6e560c1f4	Initial file population.	2013-08-02 13:12:24 -07:00