Summary:
This is a complete reworking of the way that Haxl schedules I/O. The
main benefits are:
* Data fetches are no longer organised into rounds, but can be
arbitrarily overlapped with each other and with computation. The
scheduler supports an arbitrary queue of work items which it can
evaluate while data-fetching is taking place in the background. To
take advantage of this, data sources must implement a new form of
`PerformFetch`, namely `BackgroundFetch`. The old forms of
`PerformFetch` are still supported, but won't benefit from any
additional concurrency.
* It is now possible to specify on a per-data-source basis whether
fetching should be optimised for batching or for latency. A request
to a data source that doesn't benefit from batching can be submitted
immediately. This is done with the new `schedulerHint` method of
`DataSource`.
Reviewed By: niteria
Differential Revision: D4938005
fbshipit-source-id: 96f12ad05ee62d62474ee4cc1215f19d0a6fcdf3
Summary: This provides a way to override test file locations for other build systems.
Reviewed By: yfeldblum
Differential Revision: D5218507
fbshipit-source-id: 7087ca13afb105b727ecf3f7dfdaecd26d27ea81
Summary:
These two operators are subtly non-deterministic, but can potentially
improve performance significantly in cases where
* We don't want to use .|| because it's too sequential
* We don't want to use || because it forces us to do wasted work
(and equivalently for &&).
The implementation is a bit subtle, see Note [tricky pOr/pAnd]
Reviewed By: xich
Differential Revision: D4611809
fbshipit-source-id: 832ace29dfc44e48c14cc5d4f52a0114ee326c92
Summary:
I didn't look into this too deeply but I'm guessing it was because the
constant expression had been lifted out, so I made it not a constant
expression.
Reviewed By: JonCoens
Differential Revision: D4521430
fbshipit-source-id: 687075d8486b38743b3bd8b9a9f26aa198b2d258
Summary:
Labels which throw Haxl exceptions are recorded, but pure exceptions bubble up
and labels are lost. This test demonstrates this.
Reviewed By: simonmar
Differential Revision: D3672479
fbshipit-source-id: fab10878e7eb067e0c65bcf401d75604c333007f
Summary:
This revision generalizes the existing memoization framework to 1-ary and 2-ary
functions (namely functions of type (a -> GenHaxl u b) and (a -> b _> GenHaxl u c)).
For every support arity (currently 0, 1, and 2), a family of functions {
newMemoWithX, prepareMemoX and runMemoX } are provided. newMemo itself is
generic across all arities.
Reviewed By: simonmar
Differential Revision: D3555791
fbshipit-source-id: 010a9889d42327607c8b03a5f7a609ee0c70de49
Summary:
This revision refactors cachedComputation to only contain logic relevant to
where the request-scope memo lives; memo creation and running logic is delegated
to newMemo(with) and runMemo.
Comments in cachedComputation have been moved over to newMemo/runMemo, and a
benchmark for cachedComputation has been added to monadbench. Surprisingly,
performance might have improved, albeit very slightly.
Reviewed By: simonmar
Differential Revision: D3514791
fbshipit-source-id: b2f0627824adc79b766e4f4e28c4af957ff00a00
Summary:
This diff adds the createMemo and updateMemo helper functions, which abstract
the memoization reference management logic of cachedComputation. This separates
the work of *how* a memoized computation is created/updated, from *where* the
memo reference lives, allowing the same code to be used to manage request-scope
and feature-scope memos simultaneously.
A refactor of cachedComputation to use this abstraction is forthcoming.
Reviewed By: simonmar
Differential Revision: D3492803
fbshipit-source-id: 9dadd3860d5bec3bf776eef7c1bd610c25283729
Summary:
Memoized operations were not represented in monadbench, this diff fixes that. Three tests are included:
1. Unmemoized computation, repeated N times.
2. Memoized computation, repeated N times.
3. Memoized computation, repeated N times **under different memo keys**
Reviewed By: simonmar
Differential Revision: D3444238
fbshipit-source-id: b2df534232acd5c02f9f6aea030c55d5cc846eb0
Summary: Some tests were failing, but we ignored the test failures by not checking the return value from the test runner. This patch fixes both the test runner and the tests.
Reviewed By: watashi
Differential Revision: D3379609
fbshipit-source-id: 0a1278879faa5beb0f9779ddfaa622cdbf05a73f
Summary:
This collects the highest round in which a label adds a fetch, as well as
number of fetches per label per datasource. It reports these, along with
aggregated values with scuba sample of profiling data.
Aggregation for number of rounds is the maximum round of label or any of
label's children. Aggregation for number of fetches is sum.
Reviewed By: simonmar
Differential Revision: D3316018
fbshipit-source-id: 152690c7b8811d22f566437675c943f755029528
Summary: We use a FB-specific test runner in fbcode. As a result currently tests/Main.hs is different on github to allow 'cabal test' to pass. This diff resolves the difference by creating a common list of tests and two separate entry points for running the tests: tests/Main.hs for internal use, and tests/TestMain.hs for github. tests/Main.hs will (eventually) be excluded from the public sources.
Reviewed By: simonmar
Differential Revision: D3371609
fbshipit-source-id: 46a7382df814687230db43136acd496d0c5ebca9
Summary: ignore this diff. just trying to kick a sandcastle build
Reviewed By: niteria
Differential Revision: D3327658
fbshipit-source-id: a774f53e54db5e7a29a3a6152dcf7fb2d62db4b0
Summary:
Some people want to use Haxl for large batch jobs or long-running
computations. For these use-cases, caching everything is not
practical, but we still want the batching behaviour that Haxl provides.
The new Flag field, caching, controls whether caching is enabled or
not. If not, we discard the cache after each round.
Reviewed By: niteria
Differential Revision: D3310266
fbshipit-source-id: 3fb707f77075dd42410bd50fc7465b18b216804e
Summary:
The value was too sensitive, it was different in the dbg way, and even
when running individual tests vs. all tests.
Reviewed By: codemiller
Differential Revision: D3310273
fbshipit-source-id: b15f81350f389888189409e7195486dc218f2573
Summary:
Does three things:
1. Fix order of arguments to `updEntry` to update the existing entry, instead of replacing it with 2*new.
2. Subtract nested allocations from parent.
3. Call setAllocationCounter after recording profiling data, so profiling overhead doesn't count towards the parent or the allocation limit.
Reviewed By: simonmar
Differential Revision: D3235413
fbshipit-source-id: a9f287399516fc90600b15a1524592f9c3b0674b
Summary: Add serialization and deserialization functions to non-risk datasources that don't require special treatment (like CacheClient, etc), for use with D2628780
Reviewed By: simonmar
Differential Revision: D2645436
fbshipit-source-id: 2777dbfbb11528bc079e2e82e88ebbdc880a8914
Summary:
I'm using this to test variants of the monad. In particular, the
current monad displays O(n^2) performance with the seql version of
this benchmark. This is a well-studied problem, see for example
"Reflection without remorse: revealing a hidden sequence to speed up
monadic reflection" (van der Ploeg / Kiselyov, Haskell '14)
Test Plan:
Built it and ran it a few times
unit tests still work
Reviewed By: bnitka@fb.com
Subscribers: ldbrandy, kjm, jlengyel, memo, watashi, smarlow, akr, bnitka, jcoens
FB internal diff: D2419419
Signature: t1:2419419:1441640727:9f5f82212c829fcbf2e8c063d4dbd0db495b0ba2
Summary:
Test that the ApplicativeDo extension batches things correctly in the
Haxl monad.
Test Plan: beholdunittests
Reviewed By: bnitka@fb.com
Subscribers: ldbrandy, memo, watashi, smarlow, akr, bnitka, jcoens
FB internal diff: D2039149
Tasks: 5504687
Signature: t1:2039149:1430501733:98fd1cf0f69663d6db3b07c3aed6e261ae9884d6
Summary:
Turning IO monad exceptions into Haxl monad exceptions can result in the
IO monad exception being lost, which is what was happening to our
AllocationLimitExceeded exceptions. See the comment with
rethrowAsyncExceptions for more details.
See also D1870627
Test Plan:
P19741543 is a request that blows the alloc limit but wasn't being
caught before.
Reviewed By: akr@fb.com
Subscribers: ldbrandy, memo, watashi, smarlow, akr, bnitka, jcoens
FB internal diff: D1870631
Tasks: 6240444
Signature: t1:1870631:1424867073:d03bd0368ee968cecbcc5a0f654772b6f0eaf147
Summary: Collect time in each data sources in each round
Test Plan:
test in haxlsh
test the overhead in replay
Reviewed By: smarlow@fb.com
Subscribers: anfarmer, ldbrandy, watashi, smarlow, akr, bnitka, jcoens
FB internal diff: D1521346
Tasks: 4589842
Summary:
Haxl.Core.Env and Haxl.Core.Fetch were collapsed into Haxl.Core.Monad in
D1427283. Update haxl.cabal to reflect this and use PatternGuards to
suppress warnings.
Test Plan: ~/local/haxl-github $ cabal test
Reviewed By: jon.coens@fb.com
Subscribers: ldbrandy, smarlow, akr, bnitka, jcoens
FB internal diff: D1441939
Summary:
We weren't forcing the DataCache after inserting a new element, which
meant that an error would appear when next using the cache, rather than
when we insert the element.
Test Plan: new unit test
Reviewed By: bnitka@fb.com
Subscribers: ldbrandy, smarlow, akr, bnitka, jcoens
FB internal diff: D1420686
Tasks: 4637866
Summary: Useful for catching multiple kinds of exception with a single handler.
Test Plan: haxl/core unit tests
Reviewed By: jon.coens@fb.com
Subscribers: ldbrandy, jonp, smarlow, akr, bnitka, jcoens
FB internal diff: D1387298
Tasks: 4526990, 3727129
