Summary: There are different implementations of isRangeValid that work well for different languages, thus it makes sense to facilitate having different implementations based on the language.
Reviewed By: patapizza
Differential Revision: D28362777
fbshipit-source-id: 5f2991d54af3095c8e95cf534e2dd3b4a34dee3a
Summary:
Make the code reflect the call graph, which looks roughly like this:
```
parseAndResolve
runDuckling
resolveNode
parseString
saturateParseString
parseString1
matchFirst
... low level stuff
matchFirstAnywhere
... low level stuff
```
I found the existing order pretty hard to untangle when I was writing some architecture notes on this module, I think the new ordering will help
Reviewed By: chessai
Differential Revision: D28441933
fbshipit-source-id: 07c722aa6d4038baa7f14fec84660ecc2736ed2e
Summary:
I spent a surprising amount of time trying to figure out what
this comment was referring to because it wasn't at all clear to me
that it meant a comment in another file. Making it more specific
Reviewed By: chessai
Differential Revision: D28411103
fbshipit-source-id: 26cd29b47367a7e0d865f616f289fef570544c39
Summary:
Easy style fixes for ExampleMain.hs, Debug.hs, Api.hs, Core.hs
Most of these are just lint fixes, but I also made a few not-just-lint changes
to conform to some elements of our style guide that I agree with:
- if the type signature doesn't fit on one line, then put one type per line
with nothing on the first line, so that all types are vertically aligned - makes
for a quick skim
- try to avoid mixing same-line function args with hanging function args: hang
all arguments or none at all to get a more outline-like feel, again better for
skimming
I was actually able to eliminate all errors for most of these modules - the name
collisions I usually give up on were manageable by hiding + easy variable renames
Reviewed By: chessai
Differential Revision: D28213246
fbshipit-source-id: 1f77d56f2ff8dccfd5f3b534f087c07047b92885
Summary:
Modified `Entity` to use the new `ResolvedVal` data type. Other changes follow naturally. Related issues: https://github.com/facebook/duckling/issues/121 and https://github.com/facebook/duckling/issues/172
Now one can pattern match on the output value, for instance:
```
{-# LANGUAGE GADTs #-}
import Data.Text
import Duckling.Core
import Duckling.Testing.Types
import qualified Duckling.PhoneNumber.Types as PN
parsePhoneNumber :: Text -> Text
parsePhoneNumber input =
case value entity of
(RVal PhoneNumber (PN.PhoneNumberValue v)) -> v
where
(entity:_) = parse input testContext testOptions [This PhoneNumber]
```
Reviewed By: patapizza
Differential Revision: D7502020
fbshipit-source-id: 76ba7b315cfd0d2c61ff95c855b7c95efc0a401c
Summary: Add an option to return latent time entities. This can be used when one is pretty certain that the input contains a datetime.
Reviewed By: patapizza
Differential Revision: D7254245
fbshipit-source-id: e9e0503cace2691804056fcebdc18fd9090fb181
Summary:
* Locales support for the library, following `<Lang>_<Region>` with ISO 639-1 code for `<Lang>` and ISO 3166-1 alpha-2 code for `<Region>` (#33)
* `Locale` opaque type (composite of `Lang` and `Region`) with `makeLocale` smart constructor to only allow valid `(Lang, Region)` combinations
* API: `Context`'s `lang` parameter has been replaced by `locale`, with optional `Region` and backward compatibility.
* `Rules/<Lang>.hs` exposes
- `langRules`: cross-locale rules for `<Lang>`, from `<Dimension>/<Lang>/Rules.hs`
- `localeRules`: locale-specific rules, from `<Dimension>/<Lang>/<Region>/Rules.hs`
- `defaultRules`: `langRules` + specific rules from select locales to ensure backward-compatibility
* Corpus, tests & classifiers
- 1 classifier per locale, with default classifier (`<Lang>_XX`) when no locale provided (backward-compatible)
- Default classifiers are built on existing corpus
- Locale classifiers are built on
- `<Dimension>/<Lang>/Corpus.hs` exposes a common `corpus` to all locales of `<Lang>`
- `<Dimension>/<Lang>/<Region>/Corpus.hs` exposes `allExamples`: a list of examples specific to the locale (following `<Dimension>/<Lang>/<Region>/Rules.hs`).
- Locale classifiers use the language corpus extended with the locale examples as training set.
- Locale examples need to use the same `Context` (i.e. reference time) as the language corpus.
- For backward compatibility, `<Dimension>/<Lang>/Corpus.hs` can expose also `defaultCorpus`, which is `corpus` augmented with specific examples. This is controlled by `getDefaultCorpusForLang` in `Duckling.Ranking.Generate`.
- Tests run against each classifier to make sure runtime works as expected.
* MM/DD (en_US) vs DD/MM (en_GB) example to illustrate
Reviewed By: JonCoens, blandinw
Differential Revision: D6038096
fbshipit-source-id: f29c28d
Summary:
This change refactors the Engine to use a different
code path for when we're calling `lookupItem` to find
a first token `Node` matching the rule and a different
one for subsequent ones.
This division lets us get better invariants and more importantly
do full text regexp matches only when necessary.
This should be particularly useful for longer texts.
Reviewed By: patapizza
Differential Revision: D4953918
fbshipit-source-id: e3a69ad
Summary:
Document had its internal details leaked over 2 files.
This consolidates it.
It took a long time to make this perf neutral (now it's even a tiny
win), for reasons I don't completely understand.
The INLINE pragma on byteStringFromPos I semi-understand,
but I also had to move isRangeValid to Document and that's
a bit of a mystery.
Reviewed By: patapizza
Differential Revision: D4948449
fbshipit-source-id: ffb251a
Summary:
This diff was generated by running `hsclimps`
PLEASE TAKE ONE OF THE FOLLOWING ACTIONS AS SOON AS POSSIBLE:
1) Select Accept and Ship to land this change
2) If you have issues with this diff, request changes
3) If you are no longer the owner, add reviewers and update the `.context` file with the appropriate owner
NOTE: If the diff is unable to land because of a merge conflict I will automatically update it for you.
#accept2ship
Reviewed By: niteria
Differential Revision: D4937839
fbshipit-source-id: bb3d330
Summary:
Separating out the first pass lets us avoid repeated filtering
and makes the structure of the algorithm a bit more clear.
Previously `Stash.null` was used as a test for being part of
the first pass or not, but that is a bit indirect. Encoding
the algorithm structure (the state automaton) as function calls
lets us make additional assumptions.
It also has a nice side effect of costs being attributed to
first/subsequent passes in the profile.
I also prepend to `matches` because it's likely to be bigger.
Reviewed By: patapizza
Differential Revision: D4922195
fbshipit-source-id: 0aec79f
Summary:
This converts the code to monadic style, so that
we can in the future:
* stop threading the `Document` parameter everywhere
* keep some state, like regexp match cache (I've already checked that it makes a substantial difference)
There should be no difference in performance or behavior
at this point.
Reviewed By: patapizza
Differential Revision: D4778808
fbshipit-source-id: a167ed8
Summary:
Contrary to my intuitions this part is the lion share
of allocations in `lookupRegexp`. I'd have expected `Text`
operations to dwarf it.
It's a bit doubious that we build such big lists that it
matters, perhaps in the future we can explore limiting the
number of matches considered.
Reviewed By: patapizza
Differential Revision: D4745711
fbshipit-source-id: ebdc1aa
Summary:
`isRangeValid` was doing lots of random indexing inside a Text.
Since we already have a convenient O(1), indexable `Vector Char`
we can just use it instead.
Reviewed By: patapizza
Differential Revision: D4744297
fbshipit-source-id: b23011b
Summary:
`isAdjacent` was doing a ton of useless copies and
redundant work. But pre-computing a `firstNonAdjacent` table
we can answer every `isAdjacent` query in `O(1)` time and
(almost?) no allocations.
It may be a symptom of algorithmic problems, but we shouldn't
make it more expensive than it needs to be.
Reviewed By: patapizza
Differential Revision: D4744172
fbshipit-source-id: dd70be2
Summary:
This will let me do smarter things on document construction,
like precomputing where all the whitespace is so that
I can answer `isAdjacent` in O(1) time.
If I'm measuring things right my next diff will cut down
allocations 4x on problematic inputs.
Reviewed By: patapizza
Differential Revision: D4742664
fbshipit-source-id: 7e14e25
