gigablast/open-source-search-engine
1
0
Fork 0
mirror of https://github.com/gigablast/open-source-search-engine.git synced 2024-10-04 12:17:35 +03:00
Code Issues Projects Releases Wiki Activity
251d72b8b1
open-source-search-engine/Rebalance.h

58 lines
983 B
C
Raw Normal View History

got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
// Matt Wells, Jan 2014
#ifndef REBALANCE_H
#define REBALANCE_H
#include "gb-include.h"
#include "types.h"
#include "RdbList.h"
#include "Msg4.h"
#include "Msg5.h"
#include "SafeBuf.h"
class Rebalance {
public:
Rebalance();
char *getNeedsRebalance ( ) ;
void rebalanceLoop ( ) ;
void scanLoop ( ) ;
bool scanRdb ( ) ;
rebalancer is running now
2014-01-16 04:09:38 +04:00
bool gotList ( ) ;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
bool saveRebalanceFile ( ) ;
tight merge during rebalance to save disk space, so neg recs annihilate pos recs.
2014-03-15 10:37:30 +04:00
//bool m_inRebalanceLoop;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
long m_numForeignRecs;
git rebalancing working right
2014-01-16 05:40:17 +04:00
long long m_rebalanceCount;
fix problem scanning spiderdb. move dedup spiderdb code to RdbMerge.cpp where it really should be.
2014-01-17 05:04:08 +04:00
long long m_scannedCount;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
long m_rdbNum;
collnum_t m_collnum;
git rebalancing working right
2014-01-16 05:40:17 +04:00
collnum_t m_lastCollnum;
class Rdb *m_lastRdb;
rebalancer working pretty well now
2014-01-16 07:08:47 +04:00
long m_lastPercent;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
char m_nextKey[MAX_KEY_BYTES];
char m_endKey[MAX_KEY_BYTES];
bool m_needsRebalanceValid;
char m_needsRebalance;
bool m_warnedUser;
bool m_userApproved;
bool m_isScanning;
long m_blocked;
rebalancer working pretty well now
2014-01-16 07:08:47 +04:00
bool m_allowSave;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
keep files tightly merged when doing rebalanced to avoid running out of disk space
2014-03-15 06:19:41 +04:00
bool m_registered;
got code with shard rebalancing compiling. now we store a "sharded by termid" bit in posdb key for checksums, etc keys that are not sharded by docid. save having to do disk seeks on every host in the cluster to do a dup check, etc.
2014-01-12 04:08:42 +04:00
RdbList m_list;
SafeBuf m_posMetaList;
SafeBuf m_negMetaList;
Msg4 m_msg4a;
Msg4 m_msg4b;
Msg5 m_msg5;
};
extern Rebalance g_rebalance;
#endif
Reference in New Issue Copy Permalink