google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-09-11 10:55:42 +03:00
Code Issues Packages Projects Releases Wiki Activity
963 Commits 6 Branches 28 Tags 31 MiB
6225e08edb
Commit Graph

963 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Taku Kudo
6225e08edb
Merge pull request #1020 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-7788e0fc24 
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
 2024-06-04 17:59:23 +09:00
dependabot[bot]
7455a93073
Bump the build-time-deps group 
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [twine](https://github.com/pypa/twine), [setuptools](https://github.com/pypa/setuptools) and [cibuildwheel](https://github.com/pypa/cibuildwheel).


Updates `twine` from 5.0.0 to 5.1.0
- [Release notes](https://github.com/pypa/twine/releases)
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pypa/twine/compare/5.0.0...5.1.0)

Updates `setuptools` from 69.5.1 to 70.0.0
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v69.5.1...v70.0.0)

Updates `cibuildwheel` from 2.17.0 to 2.18.1
- [Release notes](https://github.com/pypa/cibuildwheel/releases)
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md)
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.17...v2.18.1)

---
updated-dependencies:
- dependency-name: twine
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
- dependency-name: setuptools
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: build-time-deps
- dependency-name: cibuildwheel
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-06-01 15:28:31 +00:00
Taku Kudo
b9ecf1ceed
Merge pull request #1013 from alexlnkp/master 
bump CMake minimum required version to avoid warnings
 2024-05-25 16:22:15 +09:00
Taku Kudo
b35273478e
Merge pull request #1015 from Cassini-chris/patch-3 
fixing minor typos in the API.md
 2024-05-24 16:09:18 +09:00
Dr. Christoph Mittendorf
d36fa98e96
fixing minor typos in the API.md 2024-05-23 21:46:38 +02:00
Alex Murkoff
e2127b9b93 bump CMake minimum required version to avoid warnings 2024-05-22 16:24:04 +07:00
Taku Kudo
58b550871d
Merge pull request #1012 from Cassini-chris/patch-2 
adding vocab_size consistency
 2024-05-22 15:38:53 +09:00
Taku Kudo
33b01c8ee8
Merge pull request #1011 from google/dependabot/pip/dot-github/workflows/requirements/pip-aed9f6bbcd 
Bump requests from 2.31.0 to 2.32.0 in /.github/workflows/requirements in the pip group
 2024-05-22 15:37:51 +09:00
Dr. Christoph Mittendorf
d7a25aa968
vocab_size consistency 
Consistency of --vocab_size=2000 for sentence piece trainer.
Fixing various spelling mistakes
 2024-05-21 21:41:22 +02:00
dependabot[bot]
ca2259edb8
--- 
updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-21 07:16:03 +00:00
Taku Kudo
5253649c81
Merge pull request #1007 from xu-song/patch-1 
Fix typo
 2024-05-10 14:50:45 +09:00
Xu Song
1b7d45c88d
Fix typo 2024-05-08 20:14:52 +08:00
Taku Kudo
238fd2cc43
Merge pull request #1005 from Cassini-chris/patch-1 
Fixing issues with the normalizer.cc (typo, type safety, cast fucn)
 2024-05-06 17:02:27 +09:00
Taku Kudo
0da168ca45
Merge pull request #1004 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-0dd82dcf5c 
Bump setuptools from 69.2.0 to 69.5.1 in /.github/workflows/requirements in the build-time-deps group
 2024-05-05 12:30:39 +09:00
Taku Kudo
0575107765
Merge pull request #1002 from mcognetta/small_changes 
Add missing output formats to spm_encode flag documentation
 2024-05-05 12:29:54 +09:00
Dr. Christoph Mittendorf
7425587c6a
Fixing issues with the normalizer.cc 
1x fixed typo
1x Changed uint32 to uint32_t for consistency and type safety.
1x The original code uses const_cast<char*> before reinterpreting it as uint32*. Without const_cast, the compiler will treat the blob.data() as a pointer to constant data (since std::string is typically constant).
 2024-05-04 10:42:03 +02:00
dependabot[bot]
34042dc854
Bump setuptools 
Bumps the build-time-deps group in /.github/workflows/requirements with 1 update: [setuptools](https://github.com/pypa/setuptools).


Updates `setuptools` from 69.2.0 to 69.5.1
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v69.2.0...v69.5.1)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-01 15:51:12 +00:00
Marco Cognetta
530f556b82 update spm_encode docs for bpe dropout 2024-05-01 14:40:27 +09:00
Taku Kudo
7dcb541451
Merge pull request #998 from google/dependabot/pip/dot-github/workflows/requirements/idna-3.7 
Bump idna from 3.6 to 3.7 in /.github/workflows/requirements
 2024-04-15 01:26:05 +09:00
dependabot[bot]
3fcf4d36cb
Bump idna from 3.6 to 3.7 in /.github/workflows/requirements 
Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](https://github.com/kjd/idna/compare/v3.6...v3.7)

---
updated-dependencies:
- dependency-name: idna
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-12 04:24:28 +00:00
Taku Kudo
7dc9a76ec7
Merge pull request #995 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-6d60a12ad5 
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
 2024-04-05 22:12:34 +09:00
dependabot[bot]
404882df3f
Bump the build-time-deps group 
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [cibuildwheel](https://github.com/pypa/cibuildwheel), [wheel](https://github.com/pypa/wheel) and [setuptools](https://github.com/pypa/setuptools).


Updates `cibuildwheel` from 2.16.5 to 2.17.0
- [Release notes](https://github.com/pypa/cibuildwheel/releases)
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md)
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.16.5...v2.17)

Updates `wheel` from 0.42.0 to 0.43.0
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.42.0...0.43.0)

Updates `setuptools` from 69.1.1 to 69.2.0
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v69.1.1...v69.2.0)

---
updated-dependencies:
- dependency-name: cibuildwheel
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
- dependency-name: wheel
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
- dependency-name: setuptools
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-01 15:29:14 +00:00
Taku Kudo
4d6a1f4106
Merge pull request #985 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-0b1593cb10 
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
 2024-03-03 09:53:10 +09:00
dependabot[bot]
1d7ce29586
Bump the build-time-deps group 
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [twine](https://github.com/pypa/twine), [pip](https://github.com/pypa/pip) and [setuptools](https://github.com/pypa/setuptools).


Updates `twine` from 4.0.2 to 5.0.0
- [Release notes](https://github.com/pypa/twine/releases)
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pypa/twine/compare/4.0.2...5.0.0)

Updates `pip` from 23.3.2 to 24.0
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/pip/compare/23.3.2...24.0)

Updates `setuptools` from 69.0.3 to 69.1.1
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v69.0.3...v69.1.1)

---
updated-dependencies:
- dependency-name: twine
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: build-time-deps
- dependency-name: pip
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: build-time-deps
- dependency-name: setuptools
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-03-01 15:09:24 +00:00
Taku Kudo
725952d8b8 makes the return value of --help same as official abseil library 2024-02-26 13:01:58 +00:00
Taku Kudo
52a7f156a4 increment version v0.2.1 2024-02-26 05:56:28 +00:00
Taku Kudo
9082653296 use ::testing::TempDir/SrcDir 2024-02-26 05:30:23 +00:00
Taku Kudo
3b2ea62d20 fix build error 2024-02-25 16:19:08 +00:00
Taku Kudo
0ba506938c add nfc, nfd normalization tsv files 2024-02-25 15:47:08 +00:00
Taku Kudo
a216bd01d1
Merge pull request #981 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.4 
Bump cryptography from 42.0.2 to 42.0.4 in /.github/workflows/requirements
 2024-02-22 16:56:11 +09:00
dependabot[bot]
c7b4cd5019
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/42.0.2...42.0.4)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-21 20:45:34 +00:00
Taku Kudo
1d91514435
Merge pull request #979 from h-vetinari/libdir 
move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use
 2024-02-21 14:26:56 +09:00
H. Vetinari
b2863fc8d3 also change CMAKE_INSTALL_INCDIR->CMAKE_INSTALL_INCLUDEDIR in src/CMakeLists.txt 2024-02-21 12:08:41 +11:00
H. Vetinari
270120812e unify spelling of CMAKE_INSTALL_INCLUDEDIR 
Following GNUInstallDirs defaults, see also CMake docs:
https://cmake.org/cmake/help/latest/command/install.html
 2024-02-20 21:20:08 +11:00
H. Vetinari
26f9f58806 move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use 2024-02-20 21:13:20 +11:00
Taku Kudo
17d7580d64 suppress warnings in testharnress 2024-02-19 08:06:52 +00:00
Taku Kudo
4a3cd1cbaf
Merge pull request #975 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.2 
Bump cryptography from 42.0.0 to 42.0.2 in /.github/workflows/requirements
 2024-02-19 13:52:25 +09:00
dependabot[bot]
670d2e7ee0
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.2.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/42.0.0...42.0.2)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-17 00:55:36 +00:00
Taku Kudo
2b8772ae8f
Merge pull request #974 from xunkai55/patch-1 
Fix a typo in api.md
 2024-02-14 21:50:54 +09:00
Xunkai
ffd8e9efb8
Fix a typo in api.md 2024-02-12 18:57:30 +08:00
Taku Kudo
03243af616
Merge pull request #970 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-bd99d7bc59 
Bump the build-time-deps group in /.github/workflows/requirements with 1 update
 2024-02-06 18:12:26 +09:00
Taku Kudo
d0fe40512e
Merge pull request #972 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.0 
Bump cryptography from 41.0.7 to 42.0.0 in /.github/workflows/requirements
 2024-02-06 18:12:06 +09:00
dependabot[bot]
a8a618fb66
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.7 to 42.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/41.0.7...42.0.0)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-06 03:24:50 +00:00
dependabot[bot]
bbbe548a20
Bump the build-time-deps group 
Bumps the build-time-deps group in /.github/workflows/requirements with 1 update: [cibuildwheel](https://github.com/pypa/cibuildwheel).


Updates `cibuildwheel` from 2.16.2 to 2.16.5
- [Release notes](https://github.com/pypa/cibuildwheel/releases)
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md)
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.16.2...v2.16.5)

---
updated-dependencies:
- dependency-name: cibuildwheel
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-01 16:00:54 +00:00
Taku Kudo
53de76561c allows to load precomputed seed sentencepieces for unigram from a file. 2024-01-28 16:17:08 +00:00
Taku Kudo
0fe7add363 fixed crash bug in unigram model training 2024-01-28 05:19:00 +00:00
Taku Kudo
41c4b7f080 returns unicode characetr offsets in normalize method 2024-01-22 07:19:04 +00:00
Taku Kudo
6b468a0e01 support bytes output in decode method 2024-01-20 08:16:17 +00:00
Taku Kudo
7b9ee4c93e
Merge pull request #962 from Halmoni100/external-absl-2 
Additional external absl fixes
 2024-01-18 15:31:43 +09:00
Christopher Hong
0ea22c03e2 Fix build for external absl 2024-01-17 19:33:51 -05:00