google/sentencepiece
1
1
Fork 0
mirror of https://github.com/google/sentencepiece.git synced 2025-01-01 05:04:14 +03:00
Code Issues Packages Projects Releases Wiki Activity
951 Commits 6 Branches 28 Tags 34 MiB
238fd2cc43
Commit Graph

951 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Taku Kudo
2909148446
Merge pull request #957 from google/dependabot/github_actions/github-actions-bcafe21e81 
Bump the github-actions group with 2 updates
 2024-01-03 12:29:14 +09:00
dependabot[bot]
b888bd7295
Bump the github-actions group with 2 updates 
Bumps the github-actions group with 2 updates: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/setup-python](https://github.com/actions/setup-python).


Updates `actions/upload-artifact` from 3.1.3 to 4.0.0
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](a8a3f3ad30...c7d193f32e)

Updates `actions/setup-python` from 4.7.1 to 5.0.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](65d7f2d534...0a5c615913)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-01-01 15:40:30 +00:00
Pedro Kaj Kjellerup Nacht
eecae396a0
Set up dependabot to keep them updated 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-12-27 23:33:26 +00:00
Pedro Kaj Kjellerup Nacht
e31c147bb9
Hash-pin CI/CD pip dependencies 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-12-27 23:33:26 +00:00
Taku Kudo
3c2fc666dd
Update common.h 2023-12-24 02:00:56 +09:00
Taku Kudo
96aabaef96 add set_min_log_level function to python to change the loglevel from python wrapper. 2023-12-23 09:28:40 +00:00
Taku Kudo
bd3925a12e
Merge pull request #936 from google/dependabot/github_actions/github-actions-a69c9d1746 
Bump the github-actions group with 1 update
 2023-12-23 17:17:09 +09:00
Taku Kudo
b937146233
Merge pull request #938 from pnacht/token-permissions 
Set minimal permissions for GitHub workflows
 2023-12-23 17:16:39 +09:00
Taku Kudo
a5262b57eb
Merge pull request #947 from chenqy4933/master 
fix(cmake): fix android build error
 2023-12-23 17:16:25 +09:00
Taku Kudo
6b32c01286 merges internal changes to github exteranl repos 2023-12-23 07:20:11 +00:00
ChenQiyou
fc1584ea36 fix(cmake): fix android build error 2023-12-05 10:17:59 +08:00
Pedro Kaj Kjellerup Nacht
3ced0ec995
wheel.yml: Update cibuildwheel to pass in MacOS 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-28 13:46:20 +00:00
Pedro Kaj Kjellerup Nacht
ad56e74c40
Set minimal permissions for workflows 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-17 21:38:13 +00:00
dependabot[bot]
14f5e57dda
Bump the github-actions group with 1 update 
Bumps the github-actions group with 1 update: [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator).

- [Release notes](https://github.com/slsa-framework/slsa-github-generator/releases)
- [Changelog](https://github.com/slsa-framework/slsa-github-generator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/slsa-framework/slsa-github-generator/compare/v1.2.0...v1.9.0)

---
updated-dependencies:
- dependency-name: slsa-framework/slsa-github-generator
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-11-17 03:49:37 +00:00
Taku Kudo
022f8c3fed
Merge pull request #934 from pnacht/pinned-gha 
Hash-pin GitHub Actions, add dependabot
 2023-11-17 12:49:04 +09:00
Pedro Kaj Kjellerup Nacht
02ea8ed099
Add dependabot to monitor GHA 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-13 20:50:39 +00:00
Pedro Kaj Kjellerup Nacht
eaf71c2d85
Hash-pin GHA 
Signed-off-by: Pedro Kaj Kjellerup Nacht <pnacht@google.com>
 2023-11-13 20:49:59 +00:00
Taku Kudo
8cbdf13794 Improves the thread utilization in batch encoding/decoding 2023-08-05 09:01:02 +00:00
Taku Kudo
635fe8423a Upgrade the sentencepiece_model_pb2.py and sentencepiece.py 2023-07-01 02:38:33 +00:00
Taku Kudo
7b694e4bdb
Merge pull request #867 from vmarkovtsev/patch-1 
Fix nasty bug in BPE position encoding
 2023-05-25 16:31:30 +09:00
Taku Kudo
cb22883439
Merge pull request #870 from ryandesign/ryandesign-protobuf-lite 
Fix overlinking with protobuf
 2023-05-25 01:27:11 +09:00
Ryan Schmidt
41835971b7 Fix pkg-config file to avoid overlinking 
When configured to use external protobuf, remove -lprotobuf-lite from
pkg-config file's Libs section. The sentencepiece libraries link with
libprotobuf-lite; it is not necessary for users of the sentencepiece
libraries to link with protobuf unless they are using static linking so
this should have been in Libs.private. Use Requires.private so that we
use protobuf's pkg-config file to get the flags.
 2023-05-21 13:38:08 -05:00
Ryan Schmidt
e081c671b2 Remove empty placeholders in pkg-config file 
Remove @pkgconfiglibs@ and @pkgconfigcflags@ which were being replaced
with the empty string because these variables were never set anywhere.
 2023-05-21 13:31:47 -05:00
Vadim Markovtsev
3805cbb616
Fix nasty bug in BPE position encoding 
Example: sid=1313136 left=37596 right=37597 produce 18446744071878513373, which is incorrect.
 2023-05-18 19:39:30 +02:00
Taku Kudo
f2219b53e2 prepare for 0.2.00 2023-05-14 14:35:14 +00:00
Taku Kudo
0b344d0b61 Added arm architecture 2023-05-14 11:21:38 +00:00
Taku Kudo
2f66fbff33 Added arm architecture 2023-05-14 11:17:25 +00:00
Taku Kudo
6693e7eb68 Fixes test workpath 2023-05-14 10:57:55 +00:00
Taku Kudo
b857ba94e9 Split build and test 2023-05-14 10:51:31 +00:00
Taku Kudo
fad8ae6def Added fail first flag 2023-05-14 10:36:13 +00:00
Taku Kudo
f2fcd859b3 Fixes cross build yaml 2023-05-14 10:29:06 +00:00
Taku Kudo
6c901b0fb5 Fixes build test errors in big-endian machines 2023-05-14 09:54:52 +00:00
Taku Kudo
17f9c6bd2c Fixes build test errors in big-endian machines 2023-05-14 09:53:35 +00:00
Taku Kudo
827591a0c5 Fixes build test errors in big-endian machines 2023-05-14 09:08:39 +00:00
Taku Kudo
3863f7648e increases the max number of threads 2023-04-30 17:37:15 +00:00
Taku Kudo
25b64fc630 Fix the test error on windows 2023-04-28 06:20:50 +00:00
Taku Kudo
bb0b610fae Fix the ULM training bugs 2023-04-27 17:32:57 +00:00
Taku Kudo
ba44ab1ca0 Fix bugs in the handling of duplicated bigrams 2023-04-24 07:25:10 +00:00
Taku Kudo
69d34c7171 prepare for v0.1.99 2023-04-15 06:33:01 +00:00
Taku Kudo
d9a2b216b1 Fix bugs the seed score computation. 2023-04-15 05:59:52 +00:00
Taku Kudo
518c57c335 build wheel from sdist for testing 2023-04-12 07:41:58 +00:00
Taku Kudo
fabfe3095b build wheel from sdist for testing 2023-04-12 07:35:45 +00:00
Taku Kudo
d6e597b391 build wheel from sdist for testing 2023-04-12 07:24:31 +00:00
Taku Kudo
f2884a17e9 test loacl sdist build on github actions 2023-04-12 04:42:27 +00:00
Taku Kudo
609a2b7d88 test loacl sdist build on github actions 2023-04-12 02:17:27 +00:00
Taku Kudo
8fd5c6b587 test loacl sdist build on github actions 2023-04-12 01:43:39 +00:00
Taku Kudo
e07ebf74d7 support pretokenization in BPE mode. 2023-04-11 06:48:08 +00:00
Taku Kudo
119e58d97a Fixes include path when using external protobuf 2023-04-10 10:15:46 +00:00
Taku Kudo
2b0713791a fixes IS_BIGENDIAN macro places 2023-04-10 02:28:20 +00:00
Taku Kudo
e58bb684d0 add pretokenization_delimiter options. Initialize seed pieces more accurately. 2023-04-10 02:11:37 +00:00