google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-09-11 10:55:42 +03:00
Code Issues Packages Projects Releases Wiki Activity
904 Commits 6 Branches 28 Tags 31 MiB
revert-957-dependabot/github_actions/github-actions-bcafe21e81
Commit Graph

904 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Taku Kudo
27103a9e02
Merge pull request #701 from monologg/docs/typo 
Fix typo in doc
 2022-02-16 10:16:10 +09:00
Taku Kudo
4c8a377627
Merge pull request #721 from matteobaccan/patch-1 
Fixed typo error
 2022-02-16 10:14:50 +09:00
Matteo Baccan
dbc83417bb
Fixed typo error 
Fixed unknow -> unknown
 2022-01-25 19:49:26 +01:00
xiefangqi
f144921644 fix address sanitizers on clang problem 2021-12-21 19:52:47 +08:00
zy566
d972932e33
Fix typo in cmakelist 2021-10-28 14:28:26 -07:00
monologg
38278a0098 Fix typo in doc 2021-10-03 07:09:36 +00:00
Alexis Praga
142662e585 Add FreeBSD support 
<endian.h> should be <sys/endian.h> instead.
 2021-09-10 22:06:15 +02:00
Taku Kudo
8420f21790
Merge pull request #669 from reiyw/patch-1 
Fix link to nfkc.tsv in normalization.md
 2021-07-03 00:44:09 +09:00
Ryo Takahashi
cefb97b59d
Fix link to nfkc.tsv in normalization.md 2021-07-02 16:06:52 +09:00
Taku Kudo
995460307e
Merge pull request #666 from felixdae/patch-1 
Update options.md
 2021-07-01 11:33:29 +09:00
Zhangqiu Yu
31505e065a
Update options.md 
fix doc
 2021-06-30 23:30:19 +08:00
Taku Kudo
d8711f55d9
Merge pull request #664 from danieldk/remove-build-path 
Strip build directory prefix in the __FILE__ macro
 2021-06-18 01:10:09 +09:00
Taku Kudo
cbfc6b3c2c updated *.tsv file. 2021-06-18 01:06:29 +09:00
Daniël de Kok
cab2e3c486 Strip build directory prefix in the __FILE__ macro 
Various macros use __FILE__ to embed the source file name in
errors/logs. However, the full path is encoder, making reproducible
builds harder.

This change uses the -fmacro-prefix-map of GCC >= 8.0 and clang >= 10.0
to remove the source directory prefix. Since only the prefix is removed,
the relevant source file names are still included.

Background:
https://reproducible-builds.org/docs/build-path/
 2021-06-17 17:23:10 +02:00
Taku Kudo
62eafec17a updated the comment. 2021-06-17 19:32:05 +09:00
Taku Kudo
a61f985cde fixed build error. 2021-06-17 13:08:46 +09:00
Taku Kudo
f4f8309cfe fixed build error. 2021-06-17 12:53:39 +09:00
Taku Kudo
705cdc1194 added endian.h 2021-06-17 11:11:02 +09:00
Taku Kudo
cc2d2c8d31 fixed build error 2021-06-17 02:31:48 +09:00
Taku Kudo
d1e33330a7 fixed build error 2021-06-17 02:07:59 +09:00
Taku Kudo
5c194ecc34 fixed link error 2021-06-17 01:56:17 +09:00
Taku Kudo
fab966ad21 sync from internal 2021-06-16 19:04:14 +09:00
Taku Kudo
3a5bc5815b Revert "sync from internal" 
This reverts commit 05db0894d8.
 2021-06-16 14:51:52 +09:00
Taku Kudo
05db0894d8 sync from internal 2021-06-16 01:12:46 +09:00
Taku Kudo
897fe9d50c fixed build error on mac 2021-06-15 18:56:02 +09:00
Taku Kudo
faa76a03e7 fixed build error on mac 2021-06-15 14:27:56 +09:00
Taku Kudo
687885bd98 update the mac pywhell script to support py3.5 2021-06-15 13:21:20 +09:00
Taku Kudo
8a176d8ca6 use latest ubuntu image 2021-06-14 17:17:44 +09:00
Taku Kudo
6a9706c9d7
Merge pull request #658 from George-Ogden/patch-1 
Fixed mistake in README
 2021-05-26 14:45:33 +09:00
George Ogden
7316ef15d2
Fixed mistake in README 2021-05-25 12:41:39 +01:00
Taku Kudo
6256ef2438
Merge pull request #651 from wanchichen/master 
Fix typo in readme
 2021-05-08 10:31:08 +09:00
William Chen
c9ea0f3e08
fix typo 2021-04-22 19:46:42 -04:00
Taku Kudo
351600c297
Update README.md 2021-04-20 11:48:37 +09:00
Taku Kudo
2f4da31b56
Merge pull request #642 from AdolfVonKleist/restore_boundaries 
Restore the sentence boundary marker insertion for the unigram model trainer
 2021-04-20 11:39:29 +09:00
Taku Kudo
35cc60b39b
Merge pull request #632 from brettfazio/patch-1 
Fix typo in readme
 2021-04-20 11:38:19 +09:00
joe
9f544a66ff Restore the sentence boundary marker insertion for the unigram 
trainer.  Dramatically speeds up training time.
 2021-03-22 17:26:20 +00:00
Brett Fazio
72be462dcf
Fix typo in readme 2021-02-25 22:20:25 -05:00
Taku Kudo
bc53923a91
Merge pull request #630 from sarubi/zwj_fix 
Prevent Zero Width Joiner replaced with whitespace
 2021-02-26 01:25:37 +09:00
Taku Kudo
d429804123
Merge pull request #623 from erasaur/master 
add spm proto headers to install
 2021-02-24 01:34:46 +09:00
Sarubi
c970dedd8f Removed codes where Zero Width Joiner replaced with whitespace. 2021-02-23 20:47:25 +05:30
mark
f78087f696 only install proto headers if not using builtin proto 2021-02-10 10:59:56 -08:00
mark
e03761f36d add spm proto headers to install 2021-02-10 00:14:40 -08:00
Taku Kudo
ba7e11a17f
Merge pull request #617 from mhsmith/master 
Add missing #include for BYTE_ORDER
 2021-01-13 02:23:24 +09:00
Malcolm Smith
4bc9ae0df2 Add missing #include for BYTE_ORDER 2021-01-12 13:43:28 +00:00
Taku Kudo
d9a0892980 add ubuntu focal image to travis target 2021-01-10 22:23:07 +09:00
Taku Kudo
0e6dfbf86e fixed python module to check the id range. 2021-01-10 14:49:13 +09:00
Taku Kudo
3589bfbebd fixed build break. 2021-01-10 00:26:09 +09:00
Taku Kudo
8083d4f6e7 checks the range of id in Decode method 2021-01-09 16:58:41 +09:00
Taku Kudo
0e03b57a33 change the type of input_sentence_size from int32 to uint64 2021-01-09 01:44:49 +09:00
Taku Kudo
fe046e1c97 change the type of input_sentence_size from int32 to uint64 2021-01-08 16:40:03 +09:00