google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2024-08-15 22:00:43 +03:00
Code Issues Packages Projects Releases Wiki Activity
935 Commits 6 Branches 28 Tags 31 MiB
0ba506938c
Commit Graph

935 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Taku Kudo
0ba506938c add nfc, nfd normalization tsv files 2024-02-25 15:47:08 +00:00
Taku Kudo
a216bd01d1
Merge pull request #981 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.4 
Bump cryptography from 42.0.2 to 42.0.4 in /.github/workflows/requirements
 2024-02-22 16:56:11 +09:00
dependabot[bot]
c7b4cd5019
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/42.0.2...42.0.4)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-21 20:45:34 +00:00
Taku Kudo
1d91514435
Merge pull request #979 from h-vetinari/libdir 
move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use
 2024-02-21 14:26:56 +09:00
H. Vetinari
b2863fc8d3 also change CMAKE_INSTALL_INCDIR->CMAKE_INSTALL_INCLUDEDIR in src/CMakeLists.txt 2024-02-21 12:08:41 +11:00
H. Vetinari
270120812e unify spelling of CMAKE_INSTALL_INCLUDEDIR 
Following GNUInstallDirs defaults, see also CMake docs:
https://cmake.org/cmake/help/latest/command/install.html
 2024-02-20 21:20:08 +11:00
H. Vetinari
26f9f58806 move setting of default CMAKE_INSTALL_{BIN,INCLUDE,LIB}DIR before first use 2024-02-20 21:13:20 +11:00
Taku Kudo
17d7580d64 suppress warnings in testharnress 2024-02-19 08:06:52 +00:00
Taku Kudo
4a3cd1cbaf
Merge pull request #975 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.2 
Bump cryptography from 42.0.0 to 42.0.2 in /.github/workflows/requirements
 2024-02-19 13:52:25 +09:00
dependabot[bot]
670d2e7ee0
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.2.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/42.0.0...42.0.2)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-17 00:55:36 +00:00
Taku Kudo
2b8772ae8f
Merge pull request #974 from xunkai55/patch-1 
Fix a typo in api.md
 2024-02-14 21:50:54 +09:00
Xunkai
ffd8e9efb8
Fix a typo in api.md 2024-02-12 18:57:30 +08:00
Taku Kudo
03243af616
Merge pull request #970 from google/dependabot/pip/dot-github/workflows/requirements/build-time-deps-bd99d7bc59 
Bump the build-time-deps group in /.github/workflows/requirements with 1 update
 2024-02-06 18:12:26 +09:00
Taku Kudo
d0fe40512e
Merge pull request #972 from google/dependabot/pip/dot-github/workflows/requirements/cryptography-42.0.0 
Bump cryptography from 41.0.7 to 42.0.0 in /.github/workflows/requirements
 2024-02-06 18:12:06 +09:00
dependabot[bot]
a8a618fb66
Bump cryptography in /.github/workflows/requirements 
Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.7 to 42.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/41.0.7...42.0.0)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-06 03:24:50 +00:00
dependabot[bot]
bbbe548a20
Bump the build-time-deps group 
Bumps the build-time-deps group in /.github/workflows/requirements with 1 update: [cibuildwheel](https://github.com/pypa/cibuildwheel).


Updates `cibuildwheel` from 2.16.2 to 2.16.5
- [Release notes](https://github.com/pypa/cibuildwheel/releases)
- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md)
- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.16.2...v2.16.5)

---
updated-dependencies:
- dependency-name: cibuildwheel
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: build-time-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-01 16:00:54 +00:00
Taku Kudo
53de76561c allows to load precomputed seed sentencepieces for unigram from a file. 2024-01-28 16:17:08 +00:00
Taku Kudo
0fe7add363 fixed crash bug in unigram model training 2024-01-28 05:19:00 +00:00
Taku Kudo
41c4b7f080 returns unicode characetr offsets in normalize method 2024-01-22 07:19:04 +00:00
Taku Kudo
6b468a0e01 support bytes output in decode method 2024-01-20 08:16:17 +00:00
Taku Kudo
7b9ee4c93e
Merge pull request #962 from Halmoni100/external-absl-2 
Additional external absl fixes
 2024-01-18 15:31:43 +09:00
Christopher Hong
0ea22c03e2 Fix build for external absl 2024-01-17 19:33:51 -05:00
Christopher Hong
a34fb4018c Add idempotency to external absl mod 2024-01-17 19:32:48 -05:00
Taku Kudo
4ce471c002
Update cross_build.yml 
runs apt-get update to update the local index
 2024-01-17 01:02:45 +09:00
Taku Kudo
de1747bbd4 added functionality to override normalizer spec 2024-01-16 04:06:05 +00:00
Taku Kudo
0018af1f31 better exteranl abseil and protobuf support 2024-01-16 02:15:46 +00:00
Taku Kudo
acf8ebe61f build universal osx binary 2024-01-14 01:51:01 +00:00
Taku Kudo
ed76ecc478 add more advanced SentencePieceNormalizer class 2024-01-13 17:19:50 +00:00
Taku Kudo
f5c736302c remove absl/random and absl/memory, add absl::btree_map 2024-01-07 10:48:48 +00:00
Taku Kudo
adf9e81b63 move SharedBitGen to random namespace 2024-01-06 15:56:51 +00:00
Taku Kudo
49afc4c6cc
Merge pull request #959 from google/revert-957-dependabot/github_actions/github-actions-bcafe21e81 
Revert "Bump the github-actions group with 2 updates"
 2024-01-07 00:24:40 +09:00
Taku Kudo
fb490c58c2
Revert "Bump the github-actions group with 2 updates" 2024-01-06 22:55:18 +09:00
Taku Kudo
06eee09847 Added Normalization API 2024-01-04 09:04:20 +00:00
Taku Kudo
e7b5260e4a
Merge pull request #955 from pnacht/pinned-pip 
Hash-pin Python dependencies in CI/CD release workflows
 2024-01-03 12:29:39 +09:00
Taku Kudo
2909148446
Merge pull request #957 from google/dependabot/github_actions/github-actions-bcafe21e81 
Bump the github-actions group with 2 updates
 2024-01-03 12:29:14 +09:00
dependabot[bot]
b888bd7295
Bump the github-actions group with 2 updates 
Bumps the github-actions group with 2 updates: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/setup-python](https://github.com/actions/setup-python).


Updates `actions/upload-artifact` from 3.1.3 to 4.0.0
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](a8a3f3ad30...c7d193f32e)

Updates `actions/setup-python` from 4.7.1 to 5.0.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](65d7f2d534...0a5c615913)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-01-01 15:40:30 +00:00
Pedro Kaj Kjellerup Nacht
eecae396a0
Set up dependabot to keep them updated 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-12-27 23:33:26 +00:00
Pedro Kaj Kjellerup Nacht
e31c147bb9
Hash-pin CI/CD pip dependencies 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-12-27 23:33:26 +00:00
Taku Kudo
3c2fc666dd
Update common.h 2023-12-24 02:00:56 +09:00
Taku Kudo
96aabaef96 add set_min_log_level function to python to change the loglevel from python wrapper. 2023-12-23 09:28:40 +00:00
Taku Kudo
bd3925a12e
Merge pull request #936 from google/dependabot/github_actions/github-actions-a69c9d1746 
Bump the github-actions group with 1 update
 2023-12-23 17:17:09 +09:00
Taku Kudo
b937146233
Merge pull request #938 from pnacht/token-permissions 
Set minimal permissions for GitHub workflows
 2023-12-23 17:16:39 +09:00
Taku Kudo
a5262b57eb
Merge pull request #947 from chenqy4933/master 
fix(cmake): fix android build error
 2023-12-23 17:16:25 +09:00
Taku Kudo
6b32c01286 merges internal changes to github exteranl repos 2023-12-23 07:20:11 +00:00
ChenQiyou
fc1584ea36 fix(cmake): fix android build error 2023-12-05 10:17:59 +08:00
Pedro Kaj Kjellerup Nacht
3ced0ec995
wheel.yml: Update cibuildwheel to pass in MacOS 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-28 13:46:20 +00:00
Pedro Kaj Kjellerup Nacht
ad56e74c40
Set minimal permissions for workflows 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-17 21:38:13 +00:00
dependabot[bot]
14f5e57dda
Bump the github-actions group with 1 update 
Bumps the github-actions group with 1 update: [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator).

- [Release notes](https://github.com/slsa-framework/slsa-github-generator/releases)
- [Changelog](https://github.com/slsa-framework/slsa-github-generator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/slsa-framework/slsa-github-generator/compare/v1.2.0...v1.9.0)

---
updated-dependencies:
- dependency-name: slsa-framework/slsa-github-generator
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-11-17 03:49:37 +00:00
Taku Kudo
022f8c3fed
Merge pull request #934 from pnacht/pinned-gha 
Hash-pin GitHub Actions, add dependabot
 2023-11-17 12:49:04 +09:00
Pedro Kaj Kjellerup Nacht
02ea8ed099
Add dependabot to monitor GHA 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-13 20:50:39 +00:00