google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-12-29 11:11:58 +03:00
Code Issues Packages Projects Releases Wiki Activity

Fixed critical memory leak bug

Browse Source
This commit is contained in:
Taku Kudo 2017-03-07 21:33:20 +09:00
parent 032ae48af3
commit c7cf151f2a
2 changed files with 4 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -158,7 +158,7 @@ We have evaluated SentencePiece segmentation with the following configurations.
* Segmentation algorithms:
* **BPE** (Byte Pair
Encoding) [[Sennrich et al.](http://www.aclweb.org/anthology/P16-1162)]] (`--model_type=bpe`)
Encoding) [[Sennrich et al.](http://www.aclweb.org/anthology/P16-1162)] (`--model_type=bpe`)
* **Unigram**. Language-model based segmentation. (`--model_type=unigram`)
* Pre-tokenization methods:

7
src/unigram_model.cc
View File

@ -68,8 +68,8 @@ void Lattice::Clear() {
end_nodes_.clear();
sentence_.clear();
surface_.clear();
all_nodes_.clear();
port::STLDeleteElements(&all_nodes_);
all_nodes_.clear();
}
void Lattice::SetSentence(StringPiece sentence) {
@ -392,9 +392,8 @@ void ModelBase::BuildTrie(std::vector<std::pair<std::string, int>> *pieces) {
}
trie_ = port::MakeUnique<Darts::DoubleArray>();
CHECK_EQ(0,
trie_->build(key.size(), const_cast<char **>(&key[0]), nullptr,
&value[0]))
CHECK_EQ(0, trie_->build(key.size(), const_cast<char **>(&key[0]), nullptr,
&value[0]))
<< "cannot build double-array";
// Computes the maximum number of shared prefixes in the trie.