871 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Taku Kudo	f2fcd859b3	Fixes cross build yaml	2023-05-14 10:29:06 +00:00
Taku Kudo	6c901b0fb5	Fixes build test errors in big-endian machines	2023-05-14 09:54:52 +00:00
Taku Kudo	17f9c6bd2c	Fixes build test errors in big-endian machines	2023-05-14 09:53:35 +00:00
Taku Kudo	827591a0c5	Fixes build test errors in big-endian machines	2023-05-14 09:08:39 +00:00
Taku Kudo	3863f7648e	increases the max number of threads	2023-04-30 17:37:15 +00:00
Taku Kudo	25b64fc630	Fix the test error on windows	2023-04-28 06:20:50 +00:00
Taku Kudo	bb0b610fae	Fix the ULM training bugs	2023-04-27 17:32:57 +00:00
Taku Kudo	ba44ab1ca0	Fix bugs in the handling of duplicated bigrams	2023-04-24 07:25:10 +00:00
Taku Kudo	69d34c7171	prepare for v0.1.99	2023-04-15 06:33:01 +00:00
Taku Kudo	d9a2b216b1	Fix bugs the seed score computation.	2023-04-15 05:59:52 +00:00
Taku Kudo	518c57c335	build wheel from sdist for testing	2023-04-12 07:41:58 +00:00
Taku Kudo	fabfe3095b	build wheel from sdist for testing	2023-04-12 07:35:45 +00:00
Taku Kudo	d6e597b391	build wheel from sdist for testing	2023-04-12 07:24:31 +00:00
Taku Kudo	f2884a17e9	test loacl sdist build on github actions	2023-04-12 04:42:27 +00:00
Taku Kudo	609a2b7d88	test loacl sdist build on github actions	2023-04-12 02:17:27 +00:00
Taku Kudo	8fd5c6b587	test loacl sdist build on github actions	2023-04-12 01:43:39 +00:00
Taku Kudo	e07ebf74d7	support pretokenization in BPE mode.	2023-04-11 06:48:08 +00:00
Taku Kudo	119e58d97a	Fixes include path when using external protobuf	2023-04-10 10:15:46 +00:00
Taku Kudo	2b0713791a	fixes IS_BIGENDIAN macro places	2023-04-10 02:28:20 +00:00
Taku Kudo	e58bb684d0	add pretokenization_delimiter options. Initialize seed pieces more accurately.	2023-04-10 02:11:37 +00:00
Taku Kudo	6c9fd791cf	Merge pull request #845 from chris-ha458/patch-1 ... Update sentencepiece_python_module_example.ipynb	2023-04-09 17:13:58 +09:00
Chris Ha	9b53e211e8	Update sentencepiece_python_module_example.ipynb ... fix typo	2023-04-08 23:26:13 +09:00
Taku Kudo	c032c261c2	automatically detect -latomic linker option	2023-04-05 00:01:15 +00:00
Taku Kudo	5489c0a56a	add -latomic in static linking	2023-04-04 17:26:29 +00:00
Taku Kudo	c945229958	updated set-output commands	2023-04-04 15:51:06 +00:00
Taku Kudo	799c025aea	creates sdist with build_sdist.sh	2023-04-04 05:05:44 +00:00
Taku Kudo	59d84babc9	Ubuntu 18.04 to 20.04 migration	2023-04-04 03:53:27 +00:00
Taku Kudo	f54d8ba070	includes the sentencepiece source files in python source package	2023-04-04 03:15:11 +00:00
Taku Kudo	d0d1066dbf	use /MD to build wheel package on windows	2023-04-03 18:18:29 +00:00
Taku Kudo	573cc39aab	make the error message more descriptive. null termnate string in Utf8ToWide	2023-04-03 02:24:52 +00:00
Taku Kudo	359c04397c	handle the exception of std::random_device	2023-04-02 18:56:19 +00:00
Taku Kudo	d4c58fc779	handle the exception of std::random_device	2023-04-02 18:23:42 +00:00
Taku Kudo	ba466a6bae	prepare for 0.1.98	2023-04-02 18:06:40 +00:00
Taku Kudo	c0766c9870	added option to /MT flag	2023-04-02 16:56:20 +00:00
Taku Kudo	7e0137c1ba	added option to /MT flag	2023-04-02 16:17:36 +00:00
Taku Kudo	8772159684	Merge pull request #837 from ilya-lavrenov/msvc-remove-static-runtime ... Removed replacing of /MD with /MT for MSVC	2023-03-29 15:52:06 +09:00
Ilya Lavrenov	1983663e62	Removed replacing of /MD with /MT for MSVC	2023-03-26 22:10:01 +04:00
Taku Kudo	9ffb33a14c	Merge pull request #820 from juliusfrost/patch-2 ... Fix setup-python version not detected	2023-02-21 10:50:26 +09:00
Julius Frost	f2dacdff52	setup-python@v4 parity	2023-02-17 10:56:51 -05:00
Julius Frost	4de04ccd6a	Fix setup-python version not detected	2023-02-17 10:48:29 -05:00
Taku Kudo	9c211b6c2d	Merge pull request #819 from juliusfrost/patch-1 ... Add Python 3.11 builds	2023-02-18 00:00:20 +09:00
Julius Frost	4c2a713fbe	Use latest setup-python==4.5	2023-02-15 15:28:24 -05:00
Julius Frost	de2fabe5df	Update wheel.yml	2023-02-15 15:28:01 -05:00
Taku Kudo	c5a49eb5de	Merge pull request #808 from jacek-michalak/patch-1 ... Update README.md	2023-01-24 13:57:30 +09:00
jacek-michalak	14b67a4e05	Update README.md ... Two unnecessary characters	2023-01-17 21:31:20 +01:00
Taku Kudo	31656da0c9	Merge pull request #799 from kyoto7250/fix_tutorial ... fix the path in add_new_vocab.ipynb	2022-12-12 23:55:52 +09:00
kyoto7250	2ba0a5aae3	fix the path in add_new_vocab.ipynb ... Because the location of the path is different from when it was committed	2022-12-12 15:39:18 +09:00
Taku Kudo	77a65e0230	Merge pull request #795 from A2va/master ... CMake need endif	2022-11-29 11:57:18 +09:00
A2va	225fb19811	CMake need endif	2022-11-28 21:19:09 +01:00
Taku Kudo	f9dac76274	Merge pull request #793 from A2va/master ... Disable shared build on windows	2022-11-28 12:30:38 +09:00