Taku Kudo
6225e08edb
Merge pull request #1020 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-7788e0fc24
...
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
2024-06-04 17:59:23 +09:00
dependabot[bot]
7455a93073
Bump the build-time-deps group
...
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [twine](https://github.com/pypa/twine ), [setuptools](https://github.com/pypa/setuptools ) and [cibuildwheel](https://github.com/pypa/cibuildwheel ).
Updates `twine` from 5.0.0 to 5.1.0
- [Release notes](https://github.com/pypa/twine/releases )
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst )
- [Commits](https://github.com/pypa/twine/compare/5.0.0...5.1.0 )
Updates `setuptools` from 69.5.1 to 70.0.0
- [Release notes](https://github.com/pypa/setuptools/releases )
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/setuptools/compare/v69.5.1...v70.0.0 )
Updates `cibuildwheel` from 2.17.0 to 2.18.1
- [Release notes](https://github.com/pypa/cibuildwheel/releases )
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md )
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.17...v2.18.1 )
---
updated-dependencies:
- dependency-name: twine
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
- dependency-name: setuptools
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: build-time-deps
- dependency-name: cibuildwheel
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-01 15:28:31 +00:00
Taku Kudo
b9ecf1ceed
Merge pull request #1013 from alexlnkp/master
...
bump CMake minimum required version to avoid warnings
2024-05-25 16:22:15 +09:00
Taku Kudo
b35273478e
Merge pull request #1015 from Cassini-chris/patch-3
...
fixing minor typos in the API.md
2024-05-24 16:09:18 +09:00
Dr. Christoph Mittendorf
d36fa98e96
fixing minor typos in the API.md
2024-05-23 21:46:38 +02:00
Alex Murkoff
e2127b9b93
bump CMake minimum required version to avoid warnings
2024-05-22 16:24:04 +07:00
Taku Kudo
58b550871d
Merge pull request #1012 from Cassini-chris/patch-2
...
adding vocab_size consistency
2024-05-22 15:38:53 +09:00
Taku Kudo
33b01c8ee8
Merge pull request #1011 from google/dependabot/pip/dot-github/workflows/requirements/pip-aed9f6bbcd
...
Bump requests from 2.31.0 to 2.32.0 in /.github/workflows/requirements in the pip group
2024-05-22 15:37:51 +09:00
Dr. Christoph Mittendorf
d7a25aa968
vocab_size consistency
...
Consistency of --vocab_size=2000 for sentence piece trainer.
Fixing various spelling mistakes
2024-05-21 21:41:22 +02:00
dependabot[bot]
ca2259edb8
---
...
updated-dependencies:
- dependency-name: requests
dependency-type: indirect
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-21 07:16:03 +00:00
Taku Kudo
5253649c81
Merge pull request #1007 from xu-song/patch-1
...
Fix typo
2024-05-10 14:50:45 +09:00
Xu Song
1b7d45c88d
Fix typo
2024-05-08 20:14:52 +08:00
Taku Kudo
238fd2cc43
Merge pull request #1005 from Cassini-chris/patch-1
...
Fixing issues with the normalizer.cc (typo, type safety, cast fucn)
2024-05-06 17:02:27 +09:00
Taku Kudo
0da168ca45
Merge pull request #1004 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-0dd82dcf5c
...
Bump setuptools from 69.2.0 to 69.5.1 in /.github/workflows/requirements in the build-time-deps group
2024-05-05 12:30:39 +09:00
Taku Kudo
0575107765
Merge pull request #1002 from mcognetta/small_changes
...
Add missing output formats to spm_encode flag documentation
2024-05-05 12:29:54 +09:00
Dr. Christoph Mittendorf
7425587c6a
Fixing issues with the normalizer.cc
...
1x fixed typo
1x Changed uint32 to uint32_t for consistency and type safety.
1x The original code uses const_cast<char*> before reinterpreting it as uint32*. Without const_cast, the compiler will treat the blob.data() as a pointer to constant data (since std::string is typically constant).
2024-05-04 10:42:03 +02:00
dependabot[bot]
34042dc854
Bump setuptools
...
Bumps the build-time-deps group in /.github/workflows/requirements with 1 update: [setuptools](https://github.com/pypa/setuptools ).
Updates `setuptools` from 69.2.0 to 69.5.1
- [Release notes](https://github.com/pypa/setuptools/releases )
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/setuptools/compare/v69.2.0...v69.5.1 )
---
updated-dependencies:
- dependency-name: setuptools
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-01 15:51:12 +00:00
Marco Cognetta
530f556b82
update spm_encode docs for bpe dropout
2024-05-01 14:40:27 +09:00
Taku Kudo
7dcb541451
Merge pull request #998 from google/dependabot/pip/dot-github/workflows/requirements/idna-3.7
...
Bump idna from 3.6 to 3.7 in /.github/workflows/requirements
2024-04-15 01:26:05 +09:00
dependabot[bot]
3fcf4d36cb
Bump idna from 3.6 to 3.7 in /.github/workflows/requirements
...
Bumps [idna](https://github.com/kjd/idna ) from 3.6 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v3.6...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-04-12 04:24:28 +00:00
Taku Kudo
7dc9a76ec7
Merge pull request #995 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-6d60a12ad5
...
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
2024-04-05 22:12:34 +09:00
dependabot[bot]
404882df3f
Bump the build-time-deps group
...
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [cibuildwheel](https://github.com/pypa/cibuildwheel ), [wheel](https://github.com/pypa/wheel ) and [setuptools](https://github.com/pypa/setuptools ).
Updates `cibuildwheel` from 2.16.5 to 2.17.0
- [Release notes](https://github.com/pypa/cibuildwheel/releases )
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md )
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.16.5...v2.17 )
Updates `wheel` from 0.42.0 to 0.43.0
- [Release notes](https://github.com/pypa/wheel/releases )
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst )
- [Commits](https://github.com/pypa/wheel/compare/0.42.0...0.43.0 )
Updates `setuptools` from 69.1.1 to 69.2.0
- [Release notes](https://github.com/pypa/setuptools/releases )
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/setuptools/compare/v69.1.1...v69.2.0 )
---
updated-dependencies:
- dependency-name: cibuildwheel
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
- dependency-name: wheel
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
- dependency-name: setuptools
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-04-01 15:29:14 +00:00
Taku Kudo
4d6a1f4106
Merge pull request #985 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-0b1593cb10
...
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
2024-03-03 09:53:10 +09:00
dependabot[bot]
1d7ce29586
Bump the build-time-deps group
...
Bumps the build-time-deps group in /.github/workflows/requirements with 3 updates: [twine](https://github.com/pypa/twine ), [pip](https://github.com/pypa/pip ) and [setuptools](https://github.com/pypa/setuptools ).
Updates `twine` from 4.0.2 to 5.0.0
- [Release notes](https://github.com/pypa/twine/releases )
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst )
- [Commits](https://github.com/pypa/twine/compare/4.0.2...5.0.0 )
Updates `pip` from 23.3.2 to 24.0
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/pip/compare/23.3.2...24.0 )
Updates `setuptools` from 69.0.3 to 69.1.1
- [Release notes](https://github.com/pypa/setuptools/releases )
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/setuptools/compare/v69.0.3...v69.1.1 )
---
updated-dependencies:
- dependency-name: twine
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: build-time-deps
- dependency-name: pip
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: build-time-deps
- dependency-name: setuptools
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: build-time-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-03-01 15:09:24 +00:00
Taku Kudo
725952d8b8
makes the return value of --help same as official abseil library
2024-02-26 13:01:58 +00:00
Taku Kudo
52a7f156a4
increment version v0.2.1
2024-02-26 05:56:28 +00:00
Taku Kudo
9082653296
use ::testing::TempDir/SrcDir
2024-02-26 05:30:23 +00:00
Taku Kudo
3b2ea62d20
fix build error
2024-02-25 16:19:08 +00:00
Taku Kudo
0ba506938c
add nfc, nfd normalization tsv files
2024-02-25 15:47:08 +00:00
Taku Kudo
a216bd01d1
Merge pull request #981 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.4
...
Bump cryptography from 42.0.2 to 42.0.4 in /.github/workflows/requirements
2024-02-22 16:56:11 +09:00
dependabot[bot]
c7b4cd5019
Bump cryptography in /.github/workflows/requirements
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 42.0.2 to 42.0.4.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/42.0.2...42.0.4 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-21 20:45:34 +00:00
Taku Kudo
1d91514435
Merge pull request #979 from h-vetinari/libdir
...
move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use
2024-02-21 14:26:56 +09:00
H. Vetinari
b2863fc8d3
also change CMAKE_INSTALL_INCDIR->CMAKE_INSTALL_INCLUDEDIR in src/CMakeLists.txt
2024-02-21 12:08:41 +11:00
H. Vetinari
270120812e
unify spelling of CMAKE_INSTALL_INCLUDEDIR
...
Following GNUInstallDirs defaults, see also CMake docs:
https://cmake.org/cmake/help/latest/command/install.html
2024-02-20 21:20:08 +11:00
H. Vetinari
26f9f58806
move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use
2024-02-20 21:13:20 +11:00
Taku Kudo
17d7580d64
suppress warnings in testharnress
2024-02-19 08:06:52 +00:00
Taku Kudo
4a3cd1cbaf
Merge pull request #975 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.2
...
Bump cryptography from 42.0.0 to 42.0.2 in /.github/workflows/requirements
2024-02-19 13:52:25 +09:00
dependabot[bot]
670d2e7ee0
Bump cryptography in /.github/workflows/requirements
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 42.0.0 to 42.0.2.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/42.0.0...42.0.2 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-17 00:55:36 +00:00
Taku Kudo
2b8772ae8f
Merge pull request #974 from xunkai55/patch-1
...
Fix a typo in api.md
2024-02-14 21:50:54 +09:00
Xunkai
ffd8e9efb8
Fix a typo in api.md
2024-02-12 18:57:30 +08:00
Taku Kudo
03243af616
Merge pull request #970 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-bd99d7bc59
...
Bump the build-time-deps group in /.github/workflows/requirements with 1 update
2024-02-06 18:12:26 +09:00
Taku Kudo
d0fe40512e
Merge pull request #972 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.0
...
Bump cryptography from 41.0.7 to 42.0.0 in /.github/workflows/requirements
2024-02-06 18:12:06 +09:00
dependabot[bot]
a8a618fb66
Bump cryptography in /.github/workflows/requirements
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 41.0.7 to 42.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/41.0.7...42.0.0 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-06 03:24:50 +00:00
dependabot[bot]
bbbe548a20
Bump the build-time-deps group
...
Bumps the build-time-deps group in /.github/workflows/requirements with 1 update: [cibuildwheel](https://github.com/pypa/cibuildwheel ).
Updates `cibuildwheel` from 2.16.2 to 2.16.5
- [Release notes](https://github.com/pypa/cibuildwheel/releases )
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md )
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.16.2...v2.16.5 )
---
updated-dependencies:
- dependency-name: cibuildwheel
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: build-time-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 16:00:54 +00:00
Taku Kudo
53de76561c
allows to load precomputed seed sentencepieces for unigram from a file.
2024-01-28 16:17:08 +00:00
Taku Kudo
0fe7add363
fixed crash bug in unigram model training
2024-01-28 05:19:00 +00:00
Taku Kudo
41c4b7f080
returns unicode characetr offsets in normalize method
2024-01-22 07:19:04 +00:00
Taku Kudo
6b468a0e01
support bytes output in decode method
2024-01-20 08:16:17 +00:00
Taku Kudo
7b9ee4c93e
Merge pull request #962 from Halmoni100/external-absl-2
...
Additional external absl fixes
2024-01-18 15:31:43 +09:00
Christopher Hong
0ea22c03e2
Fix build for external absl
2024-01-17 19:33:51 -05:00