Summary:
Numerals that require intersection with negative numbers don't work, since the negative sign gets parsed
before the intersect rules happen. This fixes it by adding a guard in for positive in the intersection rule.
Reviewed By: patapizza
Differential Revision: D7592225
fbshipit-source-id: 2bc9c708cadeea4012c1f3ef487c61a144325f2a
Summary:
generalize chinese digit specifier (十百千万亿) parsing, and add hanzi tests
These digits specifiers can be parsed as (<num><speci>)<num>,
by using the multiplicater value <num><speci>, and a connect function that adds them together
(two cases, skipping digits [which requires a 零 in between], and digits in consecutive locations).
Note that 个 is technically a digit specifier,
but in Chinese it is never used directly as a numeral specifier, and always as a counter.
Reviewed By: zliu41
Differential Revision: D7424249
fbshipit-source-id: 20a85a7df1f908ee9879e92b904178fa26a9a5e5
Summary: Add an option to return latent time entities. This can be used when one is pretty certain that the input contains a datetime.
Reviewed By: patapizza
Differential Revision: D7254245
fbshipit-source-id: e9e0503cace2691804056fcebdc18fd9090fb181
Summary:
Before, duckling would parse `"1 2 2"` as the three digit number 122 through
`ruleSkipHundreds`. This, however, allowed the string `"Pay Kiran1 10eur"` to be
parsed as "110 EUR", which was reported in https://github.com/facebook/duckling/issues/159.
This change only accepts alphabetic numerals to pass through `ruleSkipHundreds`
(for example `"one twenty two"`), which presumably was the original intention of
this rule. This fixes the above issue, without any change in Corpus.
Reviewed By: patapizza
Differential Revision: D7151934
fbshipit-source-id: 024a7a0b6a53beb3a0d42d4bb7f542ce3b05726b
Summary:
* added `isMultipliable` helper and used that in patterns along with `isPositive`
* fixed double negatives in most languages
Reviewed By: niteria
Differential Revision: D7034982
fbshipit-source-id: a0bb67056d3107167830ece0c34d761c5563c5a7
Summary:
Hi, in this pr:
* Added time dimension to Arabic language, thanks to Hussein-Dahir & Yazeed-Obaid for writing time corpus.
* Fixed some bugs in numeral & ordinals and added more test cases for them.
Also, I don't really understand why do we use classifiers in time dimension?
Closes https://github.com/facebook/duckling/pull/123
Reviewed By: blandinw
Differential Revision: D6583313
Pulled By: patapizza
fbshipit-source-id: f7acdef0c032d7b7fd7d224832fdaf484d2df825
Summary: Create hashmaps for the pattern matching rules, and make the regex matching read the value from the hashmap.
Reviewed By: patapizza
Differential Revision: D6592508
fbshipit-source-id: 0f4dd9bc548f71d0e6e52c938bd2e39bfd909c01
Summary: Create hashmaps for the pattern matching rules, and make the regex matching read the value from the hashmap.
Reviewed By: panagosg7
Differential Revision: D6536790
fbshipit-source-id: ecb0bf28cf22b82ef10c3c2408eda15409189431
Summary:
* Closes PRs #115, #116.
* `Numeral`: Moves `ruleFractions` to common
* `Numeral/PT`: fixes dot in regex
* `Time`: Moves `isNumeralSafeToUse` to `isIntegerBetween` (more robust and cleaner)
* `Time/EN`: refactored `ruleCycleLastNextN` to `ruleDurationLastNext` (so it also accounts for non-safe Numerals, like "in a few days")
Reviewed By: blandinw
Differential Revision: D6502958
fbshipit-source-id: 6d9c08a23dab88f441aadade08d663c8e0da415d
Summary:
* `ruleIntegerNumeric` was used in all languages but Burmese.
* it seems like the hindu-arabic numerals are slowly getting in Burmese (e.g. recent car plates)
* Moving the rule in `Duckling/Numeral/Common.hs`
Reviewed By: blandinw
Differential Revision: D6498349
fbshipit-source-id: e868dc9960f18f0781e4aa98a0dfcd14969537c9
Summary:
When I write "dois mil e duzentos" the result should be 2200, but duckling recognize the numbers separated and give the result:
`[{"dim":"number","body":"dois","value":{"value":2,"type":"value"},"start":0,"end":4},{"dim":"number","body":"mil","value":{"value":1000,"type":"value"},"start":5,"end":8},{"dim":"time","body":"mil","value":{"values":[],"value":"1000-01-01T00:00:00.000-07:53","grain":"year","type":"value"},"start":5,"end":8},{"dim":"number","body":"duzentos","value":{"value":200,"type":"value"},"start":11,"end":19}]`
Now with this commit, duckling gives the correct result:
`[{"dim":"number","body":"dois mil e duzentos","value":{"value":2200,"type":"value"},"start":0,"end":19}]`
Closes https://github.com/facebook/duckling/pull/117
Reviewed By: blandinw
Differential Revision: D6477925
Pulled By: patapizza
fbshipit-source-id: 26ab503cc8def739c51ceb5bae7546016ba65ad6
Summary:
This adds support for greek times.
There are still some issues with expressions of the form:
```
9:30 - 11:00 την πέμπτη
```
Where `11:00 την πέμπτη` is parsed first (as 11:30 on Thu), instead of prioritizing `9:30 - 11:00` as the training data suggests. These tests are for the moment excluded from the corpus.
Reviewed By: patapizza
Differential Revision: D6376271
fbshipit-source-id: 2f31e058fb88386429070e3b51cd33f93b9c5936
Summary:
- Added Duration dimension to Russian language
- Added TimeGrain dimension to Russian language
- Refactored isNatural and isNaturalWith out of Duration helpers into Numeral helpers
- Implemented <integer> and a half rule for Russian Numeral
- Changed the type of inSeconds to polymorphic one
Closes https://github.com/facebook/duckling/pull/105
Reviewed By: blandinw
Differential Revision: D6312604
Pulled By: patapizza
fbshipit-source-id: 9ae237b4beb6915ff8da013230457937d8e56733
Summary:
* Locales support for the library, following `<Lang>_<Region>` with ISO 639-1 code for `<Lang>` and ISO 3166-1 alpha-2 code for `<Region>` (#33)
* `Locale` opaque type (composite of `Lang` and `Region`) with `makeLocale` smart constructor to only allow valid `(Lang, Region)` combinations
* API: `Context`'s `lang` parameter has been replaced by `locale`, with optional `Region` and backward compatibility.
* `Rules/<Lang>.hs` exposes
- `langRules`: cross-locale rules for `<Lang>`, from `<Dimension>/<Lang>/Rules.hs`
- `localeRules`: locale-specific rules, from `<Dimension>/<Lang>/<Region>/Rules.hs`
- `defaultRules`: `langRules` + specific rules from select locales to ensure backward-compatibility
* Corpus, tests & classifiers
- 1 classifier per locale, with default classifier (`<Lang>_XX`) when no locale provided (backward-compatible)
- Default classifiers are built on existing corpus
- Locale classifiers are built on
- `<Dimension>/<Lang>/Corpus.hs` exposes a common `corpus` to all locales of `<Lang>`
- `<Dimension>/<Lang>/<Region>/Corpus.hs` exposes `allExamples`: a list of examples specific to the locale (following `<Dimension>/<Lang>/<Region>/Rules.hs`).
- Locale classifiers use the language corpus extended with the locale examples as training set.
- Locale examples need to use the same `Context` (i.e. reference time) as the language corpus.
- For backward compatibility, `<Dimension>/<Lang>/Corpus.hs` can expose also `defaultCorpus`, which is `corpus` augmented with specific examples. This is controlled by `getDefaultCorpusForLang` in `Duckling.Ranking.Generate`.
- Tests run against each classifier to make sure runtime works as expected.
* MM/DD (en_US) vs DD/MM (en_GB) example to illustrate
Reviewed By: JonCoens, blandinw
Differential Revision: D6038096
fbshipit-source-id: f29c28d
Summary: Replaced pattern matching with Hashmap. Also, removed ruleInteger17 and moved its regex to ruleInteger.
Reviewed By: patapizza
Differential Revision: D5812629
fbshipit-source-id: f0c1a06
Summary:
We noticed that using UTF-8 characters directly in regexes work.
Hence converting back the escaped characters for readability and maintenance.
Reviewed By: blandinw
Differential Revision: D5787146
fbshipit-source-id: e5a4b9a
Summary: Introducing Georgian (KA), and the very beginnings of numeral support
Reviewed By: patapizza
Differential Revision: D5757952
fbshipit-source-id: 89d05f8
Summary:
Today things like `at single`, `at a few`, `at a couple of` would return a `Time`.
Discussed with blandinw to do this very explicit hack right now until other use cases show up.
Reviewed By: niteria
Differential Revision: D5325369
fbshipit-source-id: aec0402
Summary:
The one restriction on using DuplicateRecordFields is that record
selectors have to be imported under their constructor, instead of as
top-level functions. Do this for si_sigma so D5242707 passes the compat
check.
Reviewed By: watashi
Differential Revision: D5326634
fbshipit-source-id: 74ec0dd
Summary:
I fixed some bugs I found in Portuguese. This is my first attempt to contribute so let me know if there's any thing I could do better next time! thanks! awesome project!
Closes https://github.com/facebookincubator/duckling/pull/56
Differential Revision: D5318968
Pulled By: patapizza
fbshipit-source-id: 94ff30f
Summary:
Transform large case matches into HashMap lookups.
Add an extra example for a rule set that wasn't tested before.
Reviewed By: patapizza
Differential Revision: D5253349
fbshipit-source-id: 303dbca
Summary:
"1/0" was returning "null" -- this is not a valid fraction.
Now "1/0" returns 2 Numeral.
Reviewed By: niteria
Differential Revision: D5037579
fbshipit-source-id: 70fa4c9
Summary:
Also added real-world test to English `Quantity` corpus ("3/4 cup", as a culinary example)
Closes https://github.com/facebookincubator/duckling/pull/14
Reviewed By: patapizza
Differential Revision: D5035990
Pulled By: niteria
fbshipit-source-id: c1b8f65
