Summary: I was seeing mysql timeouts (where client doesn't consume query results within 10s) in walker runs using fetch_id(). Spawning the query means the results are collected and ready when we do end up polling.
Differential Revision: D25804025
fbshipit-source-id: 443dd87028fe68de16c174deb7b017d7ce5439d1
Summary: Simplify the code by allowing us to remove the old windowing logic.
Differential Revision: D25804022
fbshipit-source-id: 1f2837c2f83adcb3afdb453a9220ac68509a36ec
Summary:
When querying for changesets in a repo, often there are very few changesets for a repo inside a 65536 wide range of ids, which means multiple round trips to the database.
This change adds a LIMIT based mysql query that can always return up to the specified limit rows if the repo has them, and then using bounded_traversal_stream to unfold from the highest id loaded from that query to the next chunk to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D25804023
fbshipit-source-id: 46df2ea48d01bc4143d96642e45066f520faa4d6
Summary: Add a method to get just the ids. This is a new best case for the fetching, so updated the benchmark as well.
Reviewed By: StanislavGlebik
Differential Revision: D25804027
fbshipit-source-id: ccc4573c8a4ebc07db854a0ffa737f572087019e
Summary: Simplify the tests for bulkops, no need to map to hg to check the expected changesets. Can compare to expected bonsai directly instead as its simpler
Reviewed By: StanislavGlebik
Differential Revision: D25804020
fbshipit-source-id: eb4381c37ed6a4fb1e213f0397ffb2136ddee473
Summary: When scrubbing repos it is preferable to scrub newest data first. This diff adds Direction::NewestFirst to bulkops for use in scrubbing and updates existing call sites to Direction::OldestFirst so as not to change behaviour
Reviewed By: StanislavGlebik
Differential Revision: D25742279
fbshipit-source-id: 363a4854b14e9aa970b2c1ec491dcaccac7a6ec9
Summary: Preparation for adding OldestFirst vs NewestFirst direction of fetching in next diff.
Reviewed By: StanislavGlebik
Differential Revision: D25742281
fbshipit-source-id: 655f297efc2094d4325717d97cce53e697c35597
Summary: Add a benchmark tool for bulkops that uses the criterion benchmarking framework. This is so we can measure effect of optimizations later in stack.
Differential Revision: D25804026
fbshipit-source-id: 71b8addf1145c0ecb69d6392b4602172f2b52080
Summary:
We had a mix of callsites using fetch_all_public_changesets directly vs those using the PublicChangesetBulkFetch::fetch.
Update callsites to use PublicChangesetBulkFetch::fetch for consistency. This also has the nice side effect to removing some explicit config_store usage.
Reviewed By: StanislavGlebik
Differential Revision: D25804019
fbshipit-source-id: 5a88888dd915d1d693fb26ffe3bb359c9e918d5c
Summary: There is only one implementation of the trait so remove it and use that impl directly. Removing the trait makes it simpler to work on bulkops in the rest of this stack.
Reviewed By: farnz
Differential Revision: D25804021
fbshipit-source-id: 22fe797cf87656932d383ae236f2f867e788a832
Summary: Convert `Changsets` trait and all its uses to new type futures
Reviewed By: krallin
Differential Revision: D25638875
fbshipit-source-id: 947423e2ee47a463861678b146641bcc6b899a4a
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary:
This allows for more flexibility in structuring the code that wants to read all
the public changesets.
The usecase I have in mind is the SegmentedChangelog Seeder. The logic is
defined in the segmented_changelog crate. Constructing the Seeder is more
straight forward if it doesn't have to take direct dependency on SqlPhases and
SqlChangesets.
Reviewed By: quark-zju
Differential Revision: D24096966
fbshipit-source-id: dffa909cd27d6c05d745fd0fe0609114a50f1892
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary: Move all mercurial changeset generation logic to `blobrepo_hg`. This is preliminary step is required to decouples BlobRepo from mercurial, and in later stages it will be moved to derived data infra once blobrepo is free of mercurial.
Reviewed By: StanislavGlebik
Differential Revision: D22089677
fbshipit-source-id: bca28dedda499f80899e729e4142e373d8bec0b8
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
At first glance people will assume that changesets are returned in the same
order that they were added in the database or that at least commits are
returned in a deterministic fashion. That didn't happen because the both
changeset ids and changeset entries were received without any order.
This diff updates the function to returns results in order they were added
to the database.
Reviewed By: krallin
Differential Revision: D21676663
fbshipit-source-id: 912e6bea0532796b1d8e44e47d832c0420d97bc1
Summary:
I want to reuse the functionality provided by `fetch_all_public_changesets`
in building Segmented Changelog. To share the code I am adding a new crate
intended to store utilities in dealing with bulk fetches.
Reviewed By: krallin
Differential Revision: D21471477
fbshipit-source-id: 609907c95b438504d3a0dee64ab5a8b8b3ab3f24
