google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-09-11 10:55:42 +03:00
Code Issues Packages Projects Releases Wiki Activity
935 Commits 6 Branches 28 Tags 31 MiB
0ba506938c
Commit Graph

935 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Taku Kudo
c1e40b7278 update setup.py 2022-06-02 00:33:19 +09:00
Taku Kudo
a61584b770 update setup.py 2022-06-02 00:04:45 +09:00
Taku Kudo
db0339cc3b
Update cmake.yml 2022-06-01 20:49:47 +09:00
Taku Kudo
3bdfd88525
Update cmake.yml 2022-06-01 20:41:55 +09:00
Taku Kudo
e90e90c41c
Update cmake.yml 2022-06-01 19:56:21 +09:00
Taku Kudo
7b326ebd88 updated the python setup script for github actions 2022-06-01 19:50:11 +09:00
Taku Kudo
4ba845d1a7
add python build 2022-06-01 18:37:51 +09:00
Taku Kudo
82b380497d
Update cmake.yml 2022-06-01 14:03:51 +09:00
Taku Kudo
892bbbed34
Update cmake.yml 2022-06-01 14:01:26 +09:00
Taku Kudo
e9507a3b6a
Update cmake.yml 2022-06-01 02:30:10 +09:00
Taku Kudo
1a425bd952
Create cmake.yml 2022-06-01 01:06:52 +09:00
Taku Kudo
b108472a70 fixed CI errors 2022-05-31 02:50:10 +09:00
Taku Kudo
60bb2062d3 updated test case 2022-05-31 01:17:18 +09:00
Taku Kudo
c86a8a62de addd nbest|sample encoding method to python wrapper 2022-05-30 19:39:57 +09:00
Taku Kudo
7d8fabefcb 1) override logging stream in training, 2) Makes 1-best and viterbi decoding identical 2022-05-30 01:50:59 +09:00
Taku Kudo
9ca65fa9b6 add optional NFKD support. 2022-05-29 11:43:42 +09:00
Taku Kudo
9f3ed99f5c Sync internal to github. DP related features are added. 2022-05-25 14:03:45 +09:00
Taku Kudo
9fe52f1aa6 Sync internal to github. DP related features are added. 2022-05-25 14:02:51 +09:00
Taku Kudo
d06cd7960b
Merge pull request #750 from sidpagariya/update-setup-cfg 
Update setup.cfg to use underscore for 'description_file' attribute
 2022-05-17 21:24:23 +09:00
Taku Kudo
8b44d9d798
Merge pull request #753 from jaepil/master 
Fix compilation error in Clang 13 on macOS
 2022-05-17 21:23:49 +09:00
Sid Pagariya
24e925bf16 Update setup.cfg to use underscore for 'description_file' instead of dash 'description-file' since it loses support in future versions 2022-05-13 10:33:38 -04:00
Jaepil Jeong
b56a1d66d1
Fix compile error in clang 13 2022-05-13 06:42:38 +09:00
Jaepil Jeong
7a9cb24a21
Fix compile error in clang 13 2022-05-13 06:42:11 +09:00
Taku Kudo
b47b10c831
Merge pull request #745 from ddaspit/inherited-script 
Fix incorrect handling of "Inherited" script characters
 2022-04-12 14:44:27 +09:00
Taku Kudo
bb6d4424b9
Merge pull request #743 from akashhansda/patch-1 
Update README.md
 2022-04-12 14:41:58 +09:00
Damien Daspit
a25db30c76 Fix incorrect handling of "Inherited" script characters 2022-04-11 15:33:12 +07:00
Taku Kudo
3a6021f6c6 add python 3.10 support for windows 2022-04-11 13:43:14 +09:00
akashhansda
ea1c0ae63f
Update README.md 
Use https
 2022-03-31 07:59:18 -07:00
Taku Kudo
0c09e31605
Merge pull request #715 from xiefangqi/fix_address_sanitizers_on_clang 
fix address sanitizers on clang problem
 2022-02-16 13:30:27 +09:00
Taku Kudo
43cd284683
Merge pull request #705 from zy566/minor 
Fix typo in cmakelist
 2022-02-16 10:17:12 +09:00
Taku Kudo
9ecdc91512
Merge pull request #693 from apraga/freebsd 
Add FreeBSD support
 2022-02-16 10:16:45 +09:00
Taku Kudo
27103a9e02
Merge pull request #701 from monologg/docs/typo 
Fix typo in doc
 2022-02-16 10:16:10 +09:00
Taku Kudo
4c8a377627
Merge pull request #721 from matteobaccan/patch-1 
Fixed typo error
 2022-02-16 10:14:50 +09:00
Matteo Baccan
dbc83417bb
Fixed typo error 
Fixed unknow -> unknown
 2022-01-25 19:49:26 +01:00
xiefangqi
f144921644 fix address sanitizers on clang problem 2021-12-21 19:52:47 +08:00
zy566
d972932e33
Fix typo in cmakelist 2021-10-28 14:28:26 -07:00
monologg
38278a0098 Fix typo in doc 2021-10-03 07:09:36 +00:00
Alexis Praga
142662e585 Add FreeBSD support 
<endian.h> should be <sys/endian.h> instead.
 2021-09-10 22:06:15 +02:00
Taku Kudo
8420f21790
Merge pull request #669 from reiyw/patch-1 
Fix link to nfkc.tsv in normalization.md
 2021-07-03 00:44:09 +09:00
Ryo Takahashi
cefb97b59d
Fix link to nfkc.tsv in normalization.md 2021-07-02 16:06:52 +09:00
Taku Kudo
995460307e
Merge pull request #666 from felixdae/patch-1 
Update options.md
 2021-07-01 11:33:29 +09:00
Zhangqiu Yu
31505e065a
Update options.md 
fix doc
 2021-06-30 23:30:19 +08:00
Taku Kudo
d8711f55d9
Merge pull request #664 from danieldk/remove-build-path 
Strip build directory prefix in the __FILE__ macro
 2021-06-18 01:10:09 +09:00
Taku Kudo
cbfc6b3c2c updated *.tsv file. 2021-06-18 01:06:29 +09:00
Daniël de Kok
cab2e3c486 Strip build directory prefix in the __FILE__ macro 
Various macros use __FILE__ to embed the source file name in
errors/logs. However, the full path is encoder, making reproducible
builds harder.

This change uses the -fmacro-prefix-map of GCC >= 8.0 and clang >= 10.0
to remove the source directory prefix. Since only the prefix is removed,
the relevant source file names are still included.

Background:
https://reproducible-builds.org/docs/build-path/
 2021-06-17 17:23:10 +02:00
Taku Kudo
62eafec17a updated the comment. 2021-06-17 19:32:05 +09:00
Taku Kudo
a61f985cde fixed build error. 2021-06-17 13:08:46 +09:00
Taku Kudo
f4f8309cfe fixed build error. 2021-06-17 12:53:39 +09:00
Taku Kudo
705cdc1194 added endian.h 2021-06-17 11:11:02 +09:00
Taku Kudo
cc2d2c8d31 fixed build error 2021-06-17 02:31:48 +09:00