Summary: Add implementation of "a grand" and "<num> grand" for AU, BZ, CA, GB, IE, IN, JM, NZ, PH, TT, ZA locales. Some resolve to the local currency (AU, IN), others resolve to Dollar (NZ, PH).
Reviewed By: patapizza
Differential Revision: D7943186
fbshipit-source-id: c71ab462fa9df0ee65223ee82dc2c98457a4e13b
Summary:
Hello!
I have added __Hindi (HI)__ support for the __Duration__ dimension in ```Duckling/Duration/Rules.hs```. Written tests as well in ```Duckling/Duration/Corpus.hs``` and everything is passing and verified (`253 tests passed`).
Cheers!
Closes https://github.com/facebook/duckling/pull/194
Reviewed By: patapizza
Differential Revision: D7999571
Pulled By: chinmay87
fbshipit-source-id: deb5d60ba7f7ecbc2aa4d97ce5fb96e9bbe63b3d
Summary: Add implementation of "a grand" and "<num> grand" for EN_CA and EN_GB locales
Reviewed By: patapizza
Differential Revision: D7916643
fbshipit-source-id: 0cd55f17ec522c0334f48436a8a8cc19e0560b0b
Summary: Added support for ",number> grand" and "a grand" to result in thousands of currency amount
Reviewed By: haoxuany
Differential Revision: D7688809
fbshipit-source-id: 72a81c7a1c48329f85c9f525b72b00c479a9edb0
Summary:
Modified `Entity` to use the new `ResolvedVal` data type. Other changes follow naturally. Related issues: https://github.com/facebook/duckling/issues/121 and https://github.com/facebook/duckling/issues/172
Now one can pattern match on the output value, for instance:
```
{-# LANGUAGE GADTs #-}
import Data.Text
import Duckling.Core
import Duckling.Testing.Types
import qualified Duckling.PhoneNumber.Types as PN
parsePhoneNumber :: Text -> Text
parsePhoneNumber input =
case value entity of
(RVal PhoneNumber (PN.PhoneNumberValue v)) -> v
where
(entity:_) = parse input testContext testOptions [This PhoneNumber]
```
Reviewed By: patapizza
Differential Revision: D7502020
fbshipit-source-id: 76ba7b315cfd0d2c61ff95c855b7c95efc0a401c
Summary:
generalize chinese digit specifier (十百千万亿) parsing, and add hanzi tests
These digits specifiers can be parsed as (<num><speci>)<num>,
by using the multiplicater value <num><speci>, and a connect function that adds them together
(two cases, skipping digits [which requires a 零 in between], and digits in consecutive locations).
Note that 个 is technically a digit specifier,
but in Chinese it is never used directly as a numeral specifier, and always as a counter.
Reviewed By: zliu41
Differential Revision: D7424249
fbshipit-source-id: 20a85a7df1f908ee9879e92b904178fa26a9a5e5
Summary:
Hello,
I have added new directory for AmountOfMoney NL Support
Added two files
- Duckling/AmountOfMoney/NL/Corpus.hs
- Duckling/AmountOfMoney/NL/Rules.hs
Updated File
- Duckling/Rules/NL.hs
Added test cases
- tests/Duckling/AmountOfMoney/NL/Tests.hs
Updated Test file
- tests/Duckling/AmountOfMoney/Tests.hs
Updated
duckling.cabal
Thanks for the review and the latest merge. Looking forward
Regards
Closes https://github.com/facebook/duckling/pull/173
Reviewed By: JonCoens
Differential Revision: D7592192
Pulled By: patapizza
fbshipit-source-id: 5895c29bf7f1033e4ffd791d5915a16d230e9375
Summary: Now that years are latent, let's absorb the "in" and make them not latent.
Reviewed By: zliu41
Differential Revision: D7587599
fbshipit-source-id: 61a19ac389244df491591d78c28f0301f9124439
Summary: show the input that an ambiguous parse is failing at
Reviewed By: patapizza
Differential Revision: D7502191
fbshipit-source-id: 9f0fbf8301413d9007236ba5b6af1f4b41c20269
Summary:
During ranking, due to how candidates are ordered, it is completely possible to have multiple correct candidates
have the exact same rank (equal range and exact equal score). In this case `analyze` returns all of them, which gets
misinterpreted as having multiple tokens in output rather than multiple solutions. Checks this case and gives the
correct prompt for ambiguous parses.
Reviewed By: patapizza
Differential Revision: D7489391
fbshipit-source-id: b66947e37bddb3ac6273843dd79b559aff9d0083
Summary: Add an option to return latent time entities. This can be used when one is pretty certain that the input contains a datetime.
Reviewed By: patapizza
Differential Revision: D7254245
fbshipit-source-id: e9e0503cace2691804056fcebdc18fd9090fb181
Summary:
- add `Node` field to `Entity`
- ignore `Node` field of `Entity` for toJSON for now (will be fixed later)
- change `Debug.hs` so that we print the respective Entity's Node
- add wildcard for the new Node field in `parseTest` of Api/Tests.hs`
Reviewed By: patapizza
Differential Revision: D7174696
fbshipit-source-id: 240e4c53b72323b500ac58a74f873ce247bb3387
Summary:
Before, duckling would parse `"1 2 2"` as the three digit number 122 through
`ruleSkipHundreds`. This, however, allowed the string `"Pay Kiran1 10eur"` to be
parsed as "110 EUR", which was reported in https://github.com/facebook/duckling/issues/159.
This change only accepts alphabetic numerals to pass through `ruleSkipHundreds`
(for example `"one twenty two"`), which presumably was the original intention of
this rule. This fixes the above issue, without any change in Corpus.
Reviewed By: patapizza
Differential Revision: D7151934
fbshipit-source-id: 024a7a0b6a53beb3a0d42d4bb7f542ce3b05726b
Summary:
* added `isMultipliable` helper and used that in patterns along with `isPositive`
* fixed double negatives in most languages
Reviewed By: niteria
Differential Revision: D7034982
fbshipit-source-id: a0bb67056d3107167830ece0c34d761c5563c5a7
Summary: Added ordinal support for Hindi
Reviewed By: patapizza
Differential Revision: D6944156
fbshipit-source-id: eb5da698e5cccde9a1cc31adf7bc433b89e07454
Summary: "yesterday" would resolve to an entity without any `values`.
Reviewed By: JonCoens
Differential Revision: D6697432
fbshipit-source-id: 7b15727f92703842a2995210fdeb99c00be74bc3
Summary:
Hi, in this pr:
* Added time dimension to Arabic language, thanks to Hussein-Dahir & Yazeed-Obaid for writing time corpus.
* Fixed some bugs in numeral & ordinals and added more test cases for them.
Also, I don't really understand why do we use classifiers in time dimension?
Closes https://github.com/facebook/duckling/pull/123
Reviewed By: blandinw
Differential Revision: D6583313
Pulled By: patapizza
fbshipit-source-id: f7acdef0c032d7b7fd7d224832fdaf484d2df825
Summary:
* `ruleIntegerNumeric` was used in all languages but Burmese.
* it seems like the hindu-arabic numerals are slowly getting in Burmese (e.g. recent car plates)
* Moving the rule in `Duckling/Numeral/Common.hs`
Reviewed By: blandinw
Differential Revision: D6498349
fbshipit-source-id: e868dc9960f18f0781e4aa98a0dfcd14969537c9
Summary:
This adds support for greek times.
There are still some issues with expressions of the form:
```
9:30 - 11:00 την πέμπτη
```
Where `11:00 την πέμπτη` is parsed first (as 11:30 on Thu), instead of prioritizing `9:30 - 11:00` as the training data suggests. These tests are for the moment excluded from the corpus.
Reviewed By: patapizza
Differential Revision: D6376271
fbshipit-source-id: 2f31e058fb88386429070e3b51cd33f93b9c5936
Summary:
- Added Duration dimension to Russian language
- Added TimeGrain dimension to Russian language
- Refactored isNatural and isNaturalWith out of Duration helpers into Numeral helpers
- Implemented <integer> and a half rule for Russian Numeral
- Changed the type of inSeconds to polymorphic one
Closes https://github.com/facebook/duckling/pull/105
Reviewed By: blandinw
Differential Revision: D6312604
Pulled By: patapizza
fbshipit-source-id: 9ae237b4beb6915ff8da013230457937d8e56733
Summary:
When using speech recognition, we might see things like "six thirty six a m" or
"ten thirty p m".
Also fixed the argument order of `timeOfDayAMPM` to be more idiomatic.
Reviewed By: blandinw
Differential Revision: D6316542
fbshipit-source-id: 0008c049040219b3a1dd80d9e4661ba8a246fa7f
Summary:
* add flag for this/next/last time
* fix thanskgiving in EN locales
* `analyzedRangeTest` helper with `rangeTests` for `Time/EN`
Reviewed By: blandinw
Differential Revision: D6191209
fbshipit-source-id: 6eaa117
Summary: The date format changes between EN locales (https://en.wikipedia.org/wiki/Date_format_by_country). This diff fixes how dates are handled in each locale.
Reviewed By: patapizza
Differential Revision: D6156147
fbshipit-source-id: 22f296c
Summary:
* `en_CA` locale
* In Canada, Thanksgiving Day is the second Monday of October.
* Black Friday is the same as the US.
* However Canada observes both DDMM and MMDD formats. Defer to later, falling back to US.
Reviewed By: blandinw
Differential Revision: D6058909
fbshipit-source-id: 3d4e05e
Summary:
* Moving `ruleNationalDay` from `ZH` rules to specific locales: `zh_CN`, `zh_HK`, `zh_MO`
* Fixed National Day for `zh_TW`.
Reviewed By: blandinw
Differential Revision: D6057565
fbshipit-source-id: 8f9f2ab
Summary:
* Locales support for the library, following `<Lang>_<Region>` with ISO 639-1 code for `<Lang>` and ISO 3166-1 alpha-2 code for `<Region>` (#33)
* `Locale` opaque type (composite of `Lang` and `Region`) with `makeLocale` smart constructor to only allow valid `(Lang, Region)` combinations
* API: `Context`'s `lang` parameter has been replaced by `locale`, with optional `Region` and backward compatibility.
* `Rules/<Lang>.hs` exposes
- `langRules`: cross-locale rules for `<Lang>`, from `<Dimension>/<Lang>/Rules.hs`
- `localeRules`: locale-specific rules, from `<Dimension>/<Lang>/<Region>/Rules.hs`
- `defaultRules`: `langRules` + specific rules from select locales to ensure backward-compatibility
* Corpus, tests & classifiers
- 1 classifier per locale, with default classifier (`<Lang>_XX`) when no locale provided (backward-compatible)
- Default classifiers are built on existing corpus
- Locale classifiers are built on
- `<Dimension>/<Lang>/Corpus.hs` exposes a common `corpus` to all locales of `<Lang>`
- `<Dimension>/<Lang>/<Region>/Corpus.hs` exposes `allExamples`: a list of examples specific to the locale (following `<Dimension>/<Lang>/<Region>/Rules.hs`).
- Locale classifiers use the language corpus extended with the locale examples as training set.
- Locale examples need to use the same `Context` (i.e. reference time) as the language corpus.
- For backward compatibility, `<Dimension>/<Lang>/Corpus.hs` can expose also `defaultCorpus`, which is `corpus` augmented with specific examples. This is controlled by `getDefaultCorpusForLang` in `Duckling.Ranking.Generate`.
- Tests run against each classifier to make sure runtime works as expected.
* MM/DD (en_US) vs DD/MM (en_GB) example to illustrate
Reviewed By: JonCoens, blandinw
Differential Revision: D6038096
fbshipit-source-id: f29c28d
Summary: In Romanian, `sa` is fairly common: hai sa ne vedem (let's see), hai sa mergem (let's go).
Reviewed By: blandinw
Differential Revision: D5801345
fbshipit-source-id: db677e4
Summary: Introducing Georgian (KA), and the very beginnings of numeral support
Reviewed By: patapizza
Differential Revision: D5757952
fbshipit-source-id: 89d05f8
Summary: 'nie' means 'no' in Polish, and isn't a common abbreviation for 'niedziela' (Sunday).
Reviewed By: blandinw
Differential Revision: D5587036
fbshipit-source-id: bfda7fc
Summary:
In French, the form "at hh" is not valid (it requires an hour indicator).
This fixes false positives such as in "John a un rendez-vous."
Fixes https://github.com/wit-ai/wit/issues/666.
Reviewed By: JonCoens
Differential Revision: D5530713
fbshipit-source-id: ecee1e5
Summary:
not sure about this. Maybe I need some guidance.
Closes https://github.com/facebookincubator/duckling/pull/42
Reviewed By: blandinw
Differential Revision: D5228520
Pulled By: patapizza
fbshipit-source-id: 4f99cc5
Summary:
The numerical ordinal matching rule in DE is too broad. An ordinal like "1." may not be proceeded or followed by numbers.
* Added negative lookbehind - avoids matching the first "1." in "1.1" as an ordinal.
* Added negative lookahead - avoids matching the second "1." in "1.1. as an ordinal
Closes https://github.com/facebookincubator/duckling/pull/18
Reviewed By: patapizza
Differential Revision: D5069200
Pulled By: niteria
fbshipit-source-id: 0583076
Summary:
I notice that there are several missing dimensions for the IT language: this patch is for the Volume dimension
Regards
Matteo
Closes https://github.com/facebookincubator/duckling/pull/4
Reviewed By: JonCoens
Differential Revision: D4986389
Pulled By: patapizza
fbshipit-source-id: 314d33e
Summary:
Time dimension for Hebrew.
Commented out the failing tests that actually also fail in Clojure.
Reviewed By: JonCoens
Differential Revision: D4970308
fbshipit-source-id: b455142
Summary:
This change refactors the Engine to use a different
code path for when we're calling `lookupItem` to find
a first token `Node` matching the rule and a different
one for subsequent ones.
This division lets us get better invariants and more importantly
do full text regexp matches only when necessary.
This should be particularly useful for longer texts.
Reviewed By: patapizza
Differential Revision: D4953918
fbshipit-source-id: e3a69ad
Summary:
My change had a couple of problems:
* utf8 character width logic was completely wrong for characters that need 3 or 4 bytes
* `Array.listArray (start, end)` produces an array where `end` is a valid index
* because of ^ the `arraySize` logic also has to change
Reviewed By: watashi, darshankapashi
Differential Revision: D4894355
fbshipit-source-id: 8d07dfd
Summary:
This converts the code to monadic style, so that
we can in the future:
* stop threading the `Document` parameter everywhere
* keep some state, like regexp match cache (I've already checked that it makes a substantial difference)
There should be no difference in performance or behavior
at this point.
Reviewed By: patapizza
Differential Revision: D4778808
fbshipit-source-id: a167ed8
Summary:
This works around https://github.com/haskell/cabal/issues/4350
If we don't do this files get compiled multiple times
and cabal is unhappy.
Reviewed By: patapizza
Differential Revision: D4782749
fbshipit-source-id: 5bbe425
