Summary:
On my last diff I forgot to generalize `mapWrites` in Haxl core, leaving it working only with `WriteTree`s. Mapping functionality is needed if we want to use alternative data structures.
Luckily, the function can be trivially generalized by mapping the `Monoid` instead and exposing `mapWriteTree` to lift a function mapping elements to one mapping `WriteTree`s.
Reviewed By: josefs
Differential Revision: D39500863
fbshipit-source-id: c0c140544e135814575bc1500fec649fabc1fdec
Summary:
Currently, the `Haxl` monad stores writes using a hardcoded sequential data structure, a `WriteTree`. This makes it efficient to append new writes and merge two collections of writes, operations needed for `Haxl` to perform memoization efficiently.
However, in some cases storing and reading logs this way is inefficient in terms of space and time. A very simple example is a counter that can be incremented during `Haxl` computations. A `WriteTree Int` would store one entry per increment, while this case could be trivially represented using a single `Int`.
On this diff I'm changing the core monad implementation to accept any monoid as write structure. Rather than dealing with individual write objects inserted into a collection, writes are provided as monoids which get merged on each write.
**This is a breaking change**, as Haxl's `w` now represents the whole write log rather than a single write. However, `WriteTree` was kept and made a monoid, so `WriteTree w` can be used by clients in place of `w` to keep the same behavior.
Reviewed By: josefs
Differential Revision: D39305208
fbshipit-source-id: 7b488682280cda170e8b156765f69c3377bcadfc
Summary:
Only needed changes to the tests, the main library is untouched.
Also updates the stack.yaml to use GHC 9.0.2
Pull Request resolved: https://github.com/facebook/Haxl/pull/148
Reviewed By: simonmar
Differential Revision: D35500924
Pulled By: pepeiborra
fbshipit-source-id: 16619084c4f2d76c9495fd778401f8bab66e26d2
Summary:
Fix profling build for tests and update TARGETS to include profling build for
tests in contbuild.
Closes#146
Reviewed By: simonmar
Differential Revision: D34542444
fbshipit-source-id: 0e188f5b31bf30e684dbcb1447cbfeac779add02
Summary:
Add a new `ReportFlag` `ReportExceptionStack`. When it is enabled, we will
include the label stack in `HaxlException`. This will allow us provide stack
in `HaxlException` in non-prof rts. The label stack can also provide details
at different level compared to the `CostCentreStack`:
* with `-fprof-auto` or `-fprof-auto-top`, we can get reasonably detailed ccs.
* label stack only works well when enough labels are added either manually or
by preprocessing.
* label stack works better for cases where the different `Haxl` computations
are constructed dynamically in some common framework, where we also create
dynamic labels.
Reviewed By: simonmar
Differential Revision: D32936619
fbshipit-source-id: 4aacd576dff062706a848a93f60d93982e4b8717
Summary:
Make report flags a bitmask instead of sequential numbers. The reporting
behavior may not be strictly ordered, some reporting behavior can be enabled
independently. By not using magic numbers, it will also be easier to introduce
new reporting behavior.
We plan to add a new `ReportExceptionStack` flag include label stack in
`HaxlException` in the future, which can be enabled independently with
`ReportFetchStack`.
Reviewed By: simonmar
Differential Revision: D32933603
fbshipit-source-id: 4028aa91b68caa738684a77392c916970b4605c9
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/143
Rather enqueue fallback results onto the completion queue, to better mimic what happens in a real fetch.
This allows `F <$> a <*> b` to fetch both a and b, even if a throws an exception.
Reviewed By: josefs
Differential Revision: D28964005
fbshipit-source-id: e50814498919a0d8f1ccf5113c419243684d81de
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/141
There are two ways into the DataFetch cache, through cacheResult, and dataFetch. The fallback only worked for dataFetch, and this diff adds cacheResult. This allows tests to inject state such as time through the fallback, and that should allow consistent tests
Differential Revision: D28534164
fbshipit-source-id: 64811913742c98708eb7f0cf3d651d3a12816323
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/140
Stats can still be important with the fallback cache, as this provides a way to tie fetches to labels
Reviewed By: josefs
Differential Revision: D27265549
fbshipit-source-id: f264c39de259e65cff7f510c2f79b7d8b5c31f52
Summary: Add `Ord` to `SimpleWrite` and `assertEqualIgnoreOrder` that sorts lists before comparison
Reviewed By: ahaym
Differential Revision: D26985618
fbshipit-source-id: 2c5f34775815a2d1af4a304fd0ce2719dc9417a6
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/139
This diff allows the user to insert a fallback datacache lookup, which will allow test caches to be injected into the running of a Haxl computation
Reviewed By: simonmar
Differential Revision: D26691327
fbshipit-source-id: ad901697a6bff08adddd6cc98ab9c4e64227098f
Summary: Adds `mapWrites`, which runs a computation and returns the result, and also maps over the performed writes with the provided function
Reviewed By: DylanZA
Differential Revision: D26615745
fbshipit-source-id: 1478b0658a2054f87e9f21352e6a18832398ad78
Summary: `writeLogsRef`/`writeLogsRefNoMemo` are not cleared after the `runHaxlWithWrites` execution inspite of returning the write logs. This causes the writes to keep piling up across computations with the same env.
Reviewed By: DylanZA
Differential Revision: D26049434
fbshipit-source-id: f03742dc0cf1d7969b9fa2529a68371ab6280fd7
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/136
Adds unit test for schedulerHint in DataSource. Currently tests the TryToBatch and SubmitImmediately modes. Tests fail if
. Batch size is not equal to the no of requests in `TryToBatch` mode
. More than one request is submitted in `SubmitImmediately` mode.
Reviewed By: DylanZA
Differential Revision: D24922114
fbshipit-source-id: f786b93e9e0b7c7c731ede5fc54780d9e1b87432
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/135
Do not allow lazy evaluation to leak out of the profiling scope. This only makes calculations strict to WHNF, but that is enough to let user code add a `force` if it desires.
Reviewed By: josefs
Differential Revision: D24859691
fbshipit-source-id: 3c0e795abd07574858149298541f0d2418ed1cbf
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/134
Local memos should be profiled too.
If users do not want this, it would be trivial to add a field to MemoCall to show what type of memo it is, but I do not think it is necesary.
Reviewed By: anubhav94N
Differential Revision: D24189842
fbshipit-source-id: 379c80fe9b670e1c4b04fa22337ca8714903aad3
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/130
Label hits were incorrectly counted when a Blocked computation was hit
Reviewed By: anubhav94N
Differential Revision: D23573141
fbshipit-source-id: 79f4e17fe5fa0002a9e334e401ca1620ce07ff5b
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/129
Some datasources will throw exceptions but this does not indicate a problem with the datasource itself. This can make the statistics difficult to use to track actual problems versus problems with the way the datasource is used.
Here we allow the datasource to classify some failures to be ignored by the stats collection. They are not simply ignored however - but stored in a new field `fetchIgnoredFailures`
Reviewed By: josefs
Differential Revision: D23475953
fbshipit-source-id: a35ee0fc44ae98db86ae56573f5e7462e0355709
Summary:
This unit test demonstrates 2 things:
1) It is possible for Haxl computation to be interrupted in ways that block the scheduler indefinitely
2) Calling `sanitizeEnv` on the env from such a computation allows us to reuse it for future computations. If we don't do that, future computations can still block even without any exception thrown during the 2nd run.
Reviewed By: DylanZA
Differential Revision: D22397981
fbshipit-source-id: 48dfca49ab3485693bc772ff346945779809d9e8
Summary:
Add profile information as to what datasources the scheduler was blocked on when waiting for completions that were scheduled with BackgroundFetch.
This will not give any information on SyncFetch/AsyncFetch fetches, but it is easier to reason about those using the datasource stats.
Reviewed By: watashi
Differential Revision: D21903376
fbshipit-source-id: 42df4c567619b7e2dd6ac6acc36bcdafa85dcbe7
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/120
This adds tracking of memo/fetches per label by a unique id for each. Using this we can track exactly where time was spent, and where it was shared
Reviewed By: simonmar
Differential Revision: D20792435
fbshipit-source-id: 55c1e778d313d103a910c6dd5be512f95125acce
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/119
Currently profiling with labels does not track full stacks.
This is problematic in the case where all top level methods end up calling a common method with a label, as we have no way of attributing to the top level methods.
Eg if A/B/C all call X when running work then we cannot tell which work is expensive, as we have no connection from the work through X
Reviewed By: simonmar
Differential Revision: D20384255
fbshipit-source-id: f9aa0462904c17dee32d37a659b491e8d252d6db
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/118
* add ability to set reporting flag for benchmarking
* add argument parser
* add a test for using lots of labels
Reviewed By: simonmar
Differential Revision: D20792436
fbshipit-source-id: c87bd8e996397cb2ef229cf59927530a4dac20df
Summary:
The standard semantics test only tests the easy case when there is no blocking involved.
Also, change `sync_test` so that the data fetches are not cached. That confused me quite a lot during a debugging session.
Reviewed By: simonmar
Differential Revision: D19142099
fbshipit-source-id: 89697dbb896a1696aa916e3fcf659bf6a031f076
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/116
Add backgroundFetch methods that run a batch in the background. The Seq method will run the batch sequentially, the par method will run each request in the batch in parallel
Reviewed By: simonmar
Differential Revision: D20029453
fbshipit-source-id: d66a7959dbe09468ff67981fc3adf51704925165
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/114
When processing batches it would be quite easy to double count the allocations from child threads. This diff fixes it by setting the counter to zero after taking the current allocation count
Reviewed By: malmerey
Differential Revision: D19580472
fbshipit-source-id: 4b9a97f75e82052f4c5d94e1a6762a862a907ffb
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/115
Add a new API that makes it convenient to call backgroundFetchAcquireRelease from Haskell without going through the C APIs
Reviewed By: simonmar
Differential Revision: D19580473
fbshipit-source-id: 408f2a8c50381ddf98b35f946fbace2cd1194e55
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/112
Include what datasource and what type of fetches are included in the fetch round for tracing.
Reviewed By: simonmar
Differential Revision: D19554548
fbshipit-source-id: 747ea86ab355c9208bad1dcd938a0eec5b08dd72
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/111
Right now BackgroundFetches produce multiple FetchStats for the same batch, but it is not possible to link these together to get an idea of how big the batch was.
This introduces a field to FetchStats that can be used to link batches together as well as a utility method to do this in a default manner
Reviewed By: watashi
Differential Revision: D19469048
fbshipit-source-id: fce687c49ac4cbdc7cbd6804f37b6f120d7efad3
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/110
The test would fail under large concurrency as sometimes the result got processed before the injected exception.
Reviewed By: ahaym
Differential Revision: D19454121
fbshipit-source-id: eb2953c14c75c0233248a152aea027266f3e0e69
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/109
FutureFetch is unused (except for one test) and overall has not proven itself to be a useful fetch type. It adds a new waiting point (the others being BackgroundFetch and Async/Sync fetches) which can add latency. For example if all three are dispatched in one round how would the scheduler know ahead of time which one to wait on in order to make forward progress.
Reviewed By: simonmar
Differential Revision: D19410093
fbshipit-source-id: 40c900fbff9e06098acb2a21fc59b49adefadc5b
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/107
Add a simple mechanism (similar to asyncFetchAcquireRelease) to allow simple converting of data sources from AsyncFetch to BackgroundFetch
Reviewed By: simonmar
Differential Revision: D19272624
fbshipit-source-id: 3aec107de26fb59a4be3b2818b4f769f3404b15f
Summary: Switch to use `BasicHashTable` from `hashtables` package instead of `Data.HashMap.Strict` for `DataCache`
Reviewed By: simonmar
Differential Revision: D19219048
fbshipit-source-id: a2a8df0e715c202ca79a1f3a1c0dad133788e1eb
Summary: Bump the minimum Haxl version to GHC 8.2, which at this point is 2.5 years old but more importantly has many features that are really helpful in Haxl (such as the hs_try_put_mvar API function, which is really useful for BackgroundFetch)
Reviewed By: josefs
Differential Revision: D19327952
fbshipit-source-id: f635068fe9fb8f1d1f0d83ccbf9c3c04947183a0
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/105
`GenHaxl` doesn't implement the `MonadFail` class. This causes the ParallelTests to fail on ghc 8.6 and above.
This diff shuffles the code so we don't need the `MonadFail` class.
Reviewed By: awalterschulze
Differential Revision: D18749419
fbshipit-source-id: 98398dae9cb687076c7aaec62260dd21ca83ef3e
Summary:
Make `pAnd`/`pOr` symmetric in their shortcutting behavior. Before this diff they only returned faster if the first argument returned `False`/`True` respectively. They can now also return faster on the second argument.
The combinator `biselect` from selective applicative functors is also introduced and has a symmetric shortcutting behavior just like `pAnd` and `pOr`.
Reviewed By: simonmar
Differential Revision: D17313992
fbshipit-source-id: edf4f0000d2e6146195107f362486197f78cc4df
Summary:
This diff provides a way to do logs from a haxl computation, but not
have them memoized. This is a better alternative than doing arbitrary IO from
haxl.
Reviewed By: xich
Differential Revision: D17809398
fbshipit-source-id: 1cfe6fe24df09f12d352a18a1a78486b0a9976f8
Summary:
This test is flaky and failed in recent travis ci run:
https://travis-ci.org/facebook/Haxl/jobs/560169278
2ms is too small and a context switch or gc can affect the order of events.
Let's enlarge this to 100ms, which is similar to what we have in
FullyAsyncTest.
Reviewed By: anubhav94N
Differential Revision: D16522892
fbshipit-source-id: d27dd0b185115fd2ab0df9496b0811066e731305
Summary:
We have define this in tests and other libraries. Let's simply expose this
functionality from `Haxl.Prelude` as it can be useful in a few cases.
Reviewed By: anubhav94N
Differential Revision: D16522569
fbshipit-source-id: b35726e9ad172a36d76b755498fbb53d9a9db188
Summary:
Pull Request resolved: https://github.com/facebook/Haxl/pull/99
This adds unit tests to haxl, to make sure we are tracking the outgone
fetches correctly..
Reviewed By: simonmar
Differential Revision: D14683672
fbshipit-source-id: 49a318f0b8aa38c2af154fcbe0946122e70b9565
Summary:
Add benchmarks for sequential and parallel writes.
Ran the benchmark for 10^6 writes
parallel - 0.19s
seq - 0.07s
Running in parallel probably has more contention for the IORef
Reviewed By: simonmar
Differential Revision: D14386951
fbshipit-source-id: 164972e714eac14406fc106df073474d141e9ca0
Summary:
Most important thing I want to test is that when a memoized
computation happens again - the writes are indeed duplicated in the
environment. This diff adds tests for different ways memoized computation can
happen in conjunction with non-memoized computation..
Reviewed By: simonmar
Differential Revision: D14386667
fbshipit-source-id: a03a9a41697def968bf6e11ad66b9dd9f3a9a7f1
Summary:
Expose a conveniece wrapper `runHaxlWithWrites` which returns the writes along with
the result of the `Haxl` computation.
Reviewed By: simonmar
Differential Revision: D14386668
fbshipit-source-id: 95757916691f7b9b1291c7dceae7eafe8738cfca
Summary:
Here I try to populate the writes done as part of a Haxl computation
in an IORef inside the Environment.
`IVar` which is the synchornisation point,
also acts as the point where we store intermediate writes for Haxl
computations, so they can be memoized and reused whenever a memoized
computation is done again. This is done inside `getIVarWithWrites` function.
This works, because we create new IVars when running a memo computation
or a data fetch, and it is only at these places where we need to create a new
environment with empty writes to run the computation in. So I run every memoized
computation in a new environment (with empty writes) and populate the writes in this new
environment. At the end of the memoized computation, I look up these writes from the `IVar`
and also add them to the original environment. This way ultimately all writes are correctly
propagated upwards to the top level environment user passes to `runHaxl`.
This logic lives inside `execMemoNow`.
Reviewed By: simonmar
Differential Revision: D14342181
fbshipit-source-id: a410dae1a477f27b480804b67b2212e7500997ab
Summary:
This diff removes the scuba field as described in the task, as well as removing numRounds from Stats. This involved removing the numRounds assertion from expectRounds* functions, which I chose to rename to expectResult* (lmk if you prefer something different there).
Within Stats, I merely deleted the numRounds function. I didn't go looking for anything deeper to clean up because it looked like `rs` was used in other functions.
Reviewed By: zilberstein
Differential Revision: D8963298
fbshipit-source-id: d367b53007be03bd290222c676539680acd9f929
Summary: I noticed this test was broken in `cabal test` recently.
Reviewed By: mic47
Differential Revision: D6857296
fbshipit-source-id: ca7d15ba841f1dc79acccf1cd4999e8fcea994c8
