Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/38
The tool is used in some integration tests, make it public so that the tests might pass
Reviewed By: ikostia
Differential Revision: D22815283
fbshipit-source-id: 76da92afb8f26f61ea4f3fb949044620a57cf5ed
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/37
mononoke_hg_sync_job is used in integration tests, make it public
Reviewed By: krallin
Differential Revision: D22795881
fbshipit-source-id: 7a32c8e8adf723a49922dbb9e7723ab01c011e60
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/36
This command is used in some integration tests, make it public.
Reviewed By: krallin
Differential Revision: D22792846
fbshipit-source-id: 39ac89b1a674ea63dc924cafa07107dbf8e5a098
Summary:
Once we start moving the bookmark across the imported commits (D22598159 (c5e880c239)), we need to check dependent systems to avoid overloading them when parsing the commits. In this diff we added the functionality to check Phabricator. We use an external service (jf graphql - find discussion here: https://fburl.com/nr1f19gs) to fetch commits from Phabricator. Each commit id starts with "r", followed by a call sign (e.g FBS for fbsource) and the commit hash (https://fburl.com/qa/9pf0vtkk). If we try to fetch an invalid commit id (e.g not having a call sign), we should receive an error. Otherwise, we should receive a JSON.
An imported commit should have the following query result: https://fburl.com/graphiql/20txxvsn - nodes has one result with the imported field true.
If the commit hasn't been recognised by Phabricator yet, the nodes array will be empty.
If the commit has been recognised, but not yet parsed, the imported field will be false.
If we haven't parsed the batch, we will try to check Phabricator again after sleeping for a couple of seconds.
If it has parsed the batch of commits, we move the bookmark to the next batch.
Reviewed By: krallin
Differential Revision: D22762800
fbshipit-source-id: 5c02262923524793f364743e3e1b3f46c921db8d
Summary: On MacOS if you kill a process without waiting on it to be killed you will receive a warning on the terminal saying that the process was killed. To suppress that output, which is messing with the integratino tests, use a combination of kill and wait (the custom "killandwait" bash function). It will wait for the process to stop which is probably what most integration tests would prefer to do
Reviewed By: krallin
Differential Revision: D22790485
fbshipit-source-id: d2a08a5e617e692967f8bd566e48f5f9b50cb94d
Summary: Using "/usr/bin/date" rather than just "date" is very limiting, not all systems have common command line tools installed in the same place, just use "date".
Reviewed By: krallin
Differential Revision: D22762186
fbshipit-source-id: 747da5a388932fb5b9f4c068014c01ee90a91f9b
Summary: On MacOS the default localisation configuration (UTF-8) won't allow operations on arbitrary bytes of data via some commands, because not all sequences of bytes are valid utf-8 characters. That is why when handling arbitrary bytes it is better to use the "C" locale, which can be achieved by setting the LC_ALL env variable to "C".
Reviewed By: krallin
Differential Revision: D22762189
fbshipit-source-id: aa917886c79fba5ea61ff7168767fc4b052a35a1
Summary: Use brew on MacOS GitHub CI runs to update bash from 3.* to 5.*.
Reviewed By: krallin
Differential Revision: D22762195
fbshipit-source-id: b3a4c9df7f8ed667e88b28aacf7d87c6881eb775
Summary: MacOS uses FreeBSD version of command line tools. This diff uses brew to install the GNU tooling on GitHub CI and uses it to run the integration tests.
Reviewed By: krallin
Differential Revision: D22762198
fbshipit-source-id: 1f67674392bf6eceea9d2de02e929bb3f9f7cadd
Summary:
On unexpected errors like missing blobstore keys the walker will now log the preceding node (source) and an interesting step to this node (not necessarily the immediately preceding, e.g. the affected changeset).
Validate mode produces route information with interesting tracking enabled, scrub currently does not to save time+memory. Blobstore errors in scrub mode can be reproduced in validate mode when the extra context from the graph route is needed.
Reviewed By: farnz
Differential Revision: D22600962
fbshipit-source-id: 27d46303a2f2c07219950c20cc7f1f78773163e5
Summary:
Now that we can configure ACL checking on a per-repo basis, use the
`enforce_acl_check` config option as a killswitch to quickly disable ACL
enforcement, if required.
Further, remove the `acl_check` config flag that was always set to True.
As part of this change I've refactored the integration test a little and
replaced the phrase "ACL check" with "ACL enforcement", as we always check the
ACL inside of the LFS server.
Reviewed By: krallin
Differential Revision: D22764510
fbshipit-source-id: 8e09c743a9cd78d54b1423fd2a5cfc9bf7383d7a
Summary: Some versions of sqlite don't allow using LIKE operation on BLOB data, so first cast it to TEXT. This test was failing on Linux runs on GitHub.
Reviewed By: krallin
Differential Revision: D22761041
fbshipit-source-id: 567d68050297c3a2ac781b252d3e9b21ea5b2201
Summary: Have a comprehensive list of OSS tests that do not pass yet.
Reviewed By: krallin
Differential Revision: D22762196
fbshipit-source-id: 19ab920c4c143179db65a6d8ee32974db16c5e3d
Summary:
Update the LFS server to use the `enforce_lfs_acl_check` to enforce
ACL checks for specific repos and also reject clients with missing idents.
In the next diff, I will use the existing LFS server config's
`enforce_acl_check` flag as a killswitch.
Reviewed By: krallin
Differential Revision: D22762451
fbshipit-source-id: 61d26944127711f3503e04154e8c079ae75dc815
Summary:
Add log sequence numbers to the scuba sample builder. This provides an ordering
over the logs made by an individual instance of Mononoke, allowing them to be
sorted.
Reviewed By: krallin
Differential Revision: D22728880
fbshipit-source-id: 854bde51c7bfc469677ad08bb738e5097cb05ad5
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/32
This parsing uses the standard "subject name" field of a x509 certificate to create MononokeIdentity.
Reviewed By: farnz
Differential Revision: D22627150
fbshipit-source-id: 7f4bfc87dc2088bed44f95dd224ea8cdecc61886
Summary: If fsnodes point to non-existent content we should be able to detect that.
Reviewed By: farnz
Differential Revision: D22723866
fbshipit-source-id: 31510aada5e21109b498a26e28e0f6f3b7358ec4
Summary: It's nice to have this flag available
Reviewed By: krallin
Differential Revision: D22693732
fbshipit-source-id: 9d0d8f44cb0f5f7263a33e86e9c5b8a9927c0c85
Summary: Fix flaky test-walker-validate.t. There can be more than one route to the bad filenode, so wildcard the src and via fields when matching the output.
Reviewed By: StanislavGlebik
Differential Revision: D22664371
fbshipit-source-id: f4d880187ec2b557fb5f69ad546c2486d150b337
Summary:
For large lists it's much more convenient to specify them in a file - we are
not limited by cmd line size limit.
Reviewed By: krallin
Differential Revision: D22595023
fbshipit-source-id: 93035208700f981453eaf98f84341a86f2f1c04d
Summary: This is to be able to inspect `LiveCommitSyncConfig` from our admin tooling.
Reviewed By: StanislavGlebik
Differential Revision: D22497065
fbshipit-source-id: 3070890b7dc2a4075a5c15aca703494e33ee6530
Summary: `commit_validator` cannot just use latest `CommitSyncConfig` to validate how a commit was synced between large and small repos. Instead, when querying `synced_commit_mapping`, it needs to pay attention to `version_name` used to create the mapping. Then it needs to query `LiveCommitSyncConfig` for a config of this version and use it as a validation basis.
Reviewed By: StanislavGlebik
Differential Revision: D22525606
fbshipit-source-id: 6c32063b18461d592d931316aec7fd041bcc1ae4
Summary:
When we change `CommitSyncConfig`, we want to not have to restart `scs` servers, and instead have them pick up the new config by using `LiveCommitSyncConfig`.
This diff turned out larger than I expected, mainly due to the need to introduce various things around `TestLiveCommitSyncConfig`:
- `TestLiveCommitSyncConfig`: the trait implementer to use in `mononoke_api::Repo`
- `TestLiveCommitSyncConfigSource`: the helper struct to keep around for new values injection (very similar to how our `ConfigStore` has an inner `ConfigSource`, which can also be `TestSource`, but here injected values can be `CommitSyncConfig` instead of JSON).
- all the places in integration tests, where `setup_configerator_configs` is now needed (I plan to start setting up configerator configs always in a separate diff, as it is cheap)
Here are the explanations for a few things I think may be not immediately obvious:
- I removed the `Clone` bound from `LiveCommitSyncConfig` trait, as `Clone` traits [cannot be made into trait objects](https://doc.rust-lang.org/book/ch17-02-trait-objects.html#object-safety-is-required-for-trait-objects)
- `TestLiveCommitSyncConfigSource` vs `TestLiveCommitSyncConfigSourceInner` discrepancy is to ensure nobody should instantiate `TestLiveCommitSyncConfigSourceInner` outside of `live_commit_sync_config/src`
- I am aware of the ugly discrepancy between the main `--mononoke-config-path`, which is used to load initial configuration and can be both a file-based and a configerator-based config; and `--local-configerator-path`, used to override config sources for `Tunables` and `LiveCommitSyncConfig`. Ideally these two should just be unified somehow, but that is a little out of scope of this diff (I've already added it to the dirt doc though).
- in `mononoke_api::Repo` there are methods `new_test_xrepo` and `new_test_common`, which changed from maybe accepting just a `CommitSyncConfig` to now maybe accepting both `CommitSyncConfig` and `LiveCommitSyncConfig`. It can be made a bit cleaner: I can just query `CommitSyncConfig` from `LiveCommitSyncConfig` in `new_test_common` and avoid having two args. I was too lazy to do this, lmk if you feel strongly about it.
Reviewed By: StanislavGlebik
Differential Revision: D22443623
fbshipit-source-id: 0d6bbda2223e77b89cc59863b703db5081bcd601
Summary: After we derived the bonsaichangesets (D22455297 (c315c56c05)), we want to move a bookmark in small increments to reveal the commits to the users (https://fburl.com/wiki/zp9hgd7z step 8). This diff adds this functionality to repo_import and automate this step.
Reviewed By: StanislavGlebik
Differential Revision: D22598159
fbshipit-source-id: 01db898f07a0b7be50c3f595e78931204f33bb46
Summary: This makes tests depend less on revision numbers.
Reviewed By: DurhamG
Differential Revision: D22468669
fbshipit-source-id: 74a06930faa3e6ee9d246ecc718c2a3740f57a54
Summary:
I dropped the special case of wdir handling. With the hope that we will handle
the virtual commits differently eventually (ex. drop special cases, insert real
commits to Rust DAG but do not flush them to disk, support multiple wdir
virtual commits, null is no longer an ancestor of every commit).
`test-listkeyspatterns.t` is changed because `0` no longer resolves to `null`.
Reviewed By: DurhamG
Differential Revision: D22368836
fbshipit-source-id: 14b9914506ef59bb69363b602d646ec89ce0d89a
Summary:
This diff adds a minimal workflow for running integrations tests for Mononoke. Currently only one test is run and it fails.
This also splits the regular Mononoke CI into separate files for Linux and Mac to match the current style in Eden repo.
There are the "scopeguard::defer" fixes here that somehow escaped the CI tests.
Some tweaks have been made to "integration_runner_real.py" to make it runnable outside FB context.
Lastly the change from using "[[ -v ... ]" to "[[ -n "${...:-}" ]]; in "library.sh" was made because the former is not supported by the default Bash version preinstalled on modern MacOS.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/26
Reviewed By: krallin
Differential Revision: D22541344
Pulled By: lukaspiatkowski
fbshipit-source-id: 5023d147823166a8754be852c29b1e7b0e6d9f5f
Summary:
If a particular is blob is too popular, we can saturate a LFS host through
consistent routing, and possibly OOM the host as well.
Historically, we haven't had enough traffic to LFS to make this a problem, but
we're getting there now.
This diffs adds support for reporting the popularity of a blob through SCS (not
Mononoke SCS — the couting one), and for using this popularity to identify when
we should stop consistently-routing a given blob.
The idea is that if e.g. something was requested 300 times in the last 20
seconds, it'll take a second for all the hosts to have it in cache, so we might
as well distribute this load.
There are plenty of things we could do slightly better here, such as making the
interval configurable, or having something in-between "consistently route to a
single host" and "don't consistently route at all". That said, I don't think
those are necessary right now, so let's start simple and find out.
Reviewed By: HarveyHunt
Differential Revision: D22503748
fbshipit-source-id: 48827bcfb7658ad22c88a8433359e29b0d56ad5a
Summary: After we shifted the bonsaichangesets (D22307039 (f5db2fdcdc)), we want to derive all the data types available in the target repo. Previously, we used a script to specify what we want to derive (https://fburl.com/wiki/ivk76muf step 4). This diff adds this functionality to repo_import and automate this step.
Reviewed By: StanislavGlebik
Differential Revision: D22455297
fbshipit-source-id: b38ac68606687350ace57d68464e68ca8229f7a5
Summary:
Remove the `repo_id` parameter from the `Bookmarks` trait methods.
The `repo_id` parameters was intended to allow a single `Bookmarks` implementation
to serve multiple repos. In practise, however, each repo has its own config, which
results in a separate `Bookmarks` instance for each repo. The `repo_id` parameter
complicates the API and provides no benefit.
To make this work, we switch to the `Builder` pattern for `SqlBookmarks`, which
allows us to inject the `repo_id` at construction time. In fact nothing here
prevents us from adding back-end sharing later on, as these `SqlBookmarks` objects
are free to share data in their implementation.
Reviewed By: StanislavGlebik
Differential Revision: D22437089
fbshipit-source-id: d20e08ce6313108b74912683c620d25d6bf7ca01
Summary:
Separate out the `BundleReplayData` from the `BookmarkUpdateReason` enum. There's
no real need for this to be part of the reason, and removing it means we can
abstract away the remaining dependency on Mercurial changeset IDs from
the main bookmarks traits.
Reviewed By: mitrandir77, ikostia
Differential Revision: D22417659
fbshipit-source-id: c8e5af7ba57d10a90c86437b59c0d48e587e730e
Summary: For populating the XDB blobstore, we'd like to copy data from Manifold - the easiest way to do that is to exploit MultiplexedBlobstore's scrub mode to copy data directly.
Reviewed By: krallin
Differential Revision: D22373838
fbshipit-source-id: 550a9c73e79059380337fa35ac94fe1134378196
Summary:
Bypass truncation-based transaction if narrow-heads is on.
The transaction abort still works logically because commit references stay
unchanged on abort.
Related EdenFS and Mononoke tests are updated. Mononoke tests probably
shouldn't rely on revlog / fncache implementation details in hg.
Reviewed By: DurhamG
Differential Revision: D22240186
fbshipit-source-id: f97efd60855467b52c9fb83e7c794ded269e9617
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary:
Sometimes you want to fetch a file. Using curl and the LFS server works, but
this really should be part of Mononoke admin.
Reviewed By: ikostia
Differential Revision: D22397472
fbshipit-source-id: 17decf4aa2017a2c1be52605a254692f293d1bcd
Summary:
This got broken when we moved to Tokio 0.2. Let's fix it and add a test to make
sure it does not regress.
Reviewed By: ikostia
Differential Revision: D22396261
fbshipit-source-id: a8359aee33b4d6d840581f57f91af6c03125fd6a
Summary:
This diff adds two new bits of functionality to `LiveCommitSyncConfig`:
- getting all possible versions of `CommitSyncConfig` for a given repo
- getting `CommitSyncConfig` for a repo by version name
These bits are meant to be used in:
- `commit_validator` and `bookmarks_validator`, which would
need to run validation against a specific config version
- `mononoke_admin`, which would need to be able to query all versions,
display the version used to sync two commits and so on
Reviewed By: StanislavGlebik
Differential Revision: D22235381
fbshipit-source-id: 42326fe853b588849bce0185b456a5365f3d8dff
Summary:
Add the `scsc list-bookmarks` command, which lists bookmarks in a repository.
If a commit id is also provided, `list-bookmark` will be limited to bookmarks that
point to that commit of one of its descendants.
Reviewed By: mitrandir77
Differential Revision: D22361240
fbshipit-source-id: 17067ba47f9285b8137a567a70a87fadcaabec80
Summary:
This is a mirror image of a diff, which made backsyncer use `LiveCommitSyncConfig`: we want to use configerator-based live configs, when we run in the continuous tailing mode.
As no-op iteration time used to be 10s and that's a bit wasteful for tests, this diff changes it to be configurable.
Finally, because of instantiating various additional `CommitSyncerArgs` structs, this diff globs out some of the `using repo` logs (which aren't very useful as test signals anyway, IMO).
Reviewed By: StanislavGlebik
Differential Revision: D22209205
fbshipit-source-id: fa46802418a431781593c41ee36f468dee9eefba
Summary:
Let's use new option in CLI. Unfortunately we can't easily accept commit ids in
named params so it has to be a postional one.
Differential Revision: D22234412
fbshipit-source-id: a9c27422fa65ae1c42cb1c243c7694507a957437
Summary: This allowed me to compare two alternative approaches to queue draining, and generally seems like a useful thing to do.
Reviewed By: krallin
Differential Revision: D22364733
fbshipit-source-id: b6c76295c85b4dec6f0bfd7107c30bb4e4a28942
Summary: It's useful to derive all enabled derived data at once
Reviewed By: krallin
Differential Revision: D22336338
fbshipit-source-id: 54bc27ab2c23c175913fc02e6bf05d18a54c249c
Summary:
We've recently added an option to perform a stack move in megarepolib. A "stack
move" it's a stack of commits that move a files according to a mover. Now let's
expose it in the megarepotool
Reviewed By: ikostia
Differential Revision: D22312486
fbshipit-source-id: 878d4b2575ed2930bbbf0b9b35e51bb41393e622
Summary:
The LIKE pattern used by bookmark prefixes needs to be escaped, otherwise
users looking for bookmarks containing `\`, `_` or `%` will get the
wrong results.
Reviewed By: krallin
Differential Revision: D22336716
fbshipit-source-id: 99b0ad6097f096358e66042752e4d153359935be
Summary:
EdenAPI's `make_req` tools allows developers to create ad-hoc CBOR request payloads for debugging purposes (e.g., for use with `curl`). The tool generates requests from human-created JSON, which are particularly useful in Mercurial and Mononoke's integration tests.
Later in this stack, the use of this JSON format will be extended beyond just this one tool. As such, it is important that the representation be sufficiently extensible so accommodate future changes to the request structs. In the case of the JSON representation of `DataRequest`, this means changing from an array to a single-attribute object, so that additional fields can potentially be added in the future.
Reviewed By: quark-zju
Differential Revision: D22319314
fbshipit-source-id: 5931bc7ab01ca48ceab5ffd1c9177dd3035b643c
Summary:
ReplicaLagMonitor is aimed to generalize over different stategies of fetching
the replication lag in a SQL database. Querying a set of connections is one
such strategy.
Reviewed By: ikostia
Differential Revision: D22104348
fbshipit-source-id: bbbeccb55a664e60b3c14ee17f404982d09f2b25
Summary:
Blobstore healer has a logic, which prevents it from doing busy work, when the
queue is empty. This is implemented by means of checking whether the DB query
fetched the whole `LIMIT` of values. Or that is the idea, at least. In
practice, here's what happens:
1. DB query is a nested one: first it gets at most `LIMIT` distinct
`operation_key` entries, then it gets all rows with such entries. In practice
this almost always means `# of blobstores * LIMIT` rows, as we almost always
succeed writing to every blobstore
2. Once this query is done, the rows are grouped by the `blobstore_key`, and a
future is created for each such row (for simplicity, ignore that future may not
be created).
3. We then compare the number of created futures with `LIMIT` and report an
incomplete batch if the numbers are different.
This logic has a flaw: same `blobstore_key` may be written multiple times with
different `operation_key` values. One example of this: `GitSha1` keys for
identical contents. When this happens, grouping from step 2 above will produce
fewer than `LIMIT` groups, and we'll end up sleeping for nothing.
This is not a huge deal, but let's fix it anyway.
My fix also adds some strictly speaking unnecessary logging, but I found it
helpful during this investigation, so let's keep it.
The price of this change is collecting two `unique_by` calls, both of which
allocates a temporary hash set [1] of the size `LIMIT * len(blobstore_key) * #
blobstores` (and another one with `operation_key`). For `LIMIT=100_000`
`len(blobstore_key)=255`, `# blobstores = 3` we have roughly 70 mb for the
larger one, which should be ok.
[1] https://docs.rs/itertools/0.9.0/itertools/trait.Itertools.html#method.unique
Reviewed By: ahornby
Differential Revision: D22293204
fbshipit-source-id: bafb7817359e2c867cf33c319a886653b974d43f
Summary:
Previous commit: D22233127 (fa1caa8c4e)
In this diff, I added rewrite commit path functionality using Mover https://fburl.com/diffusion/6rnf9q2f to repo_import.
Given a prefix (e.g. new_repo), we prepend the paths of the files extracted from the bonsaichangesets given by gitimport (e.g. folder1/file1 => new_repo/folder1/file1). Previously, we did this manually when importing a git repo (https://www.internalfb.com/intern/wiki/Mercurial/Admin/ImportingRepos/) using convert extension.
Reviewed By: StanislavGlebik
Differential Revision: D22307039
fbshipit-source-id: 322533e5d6cbaf5d7eec589c8cba0c1b9c79d7af
Summary: Convert the bookmarks traits to use new-style `BoxFuture<'static>` and `BoxStream<'static>`. This is a step along the path to full `async`/`await`.
Reviewed By: farnz
Differential Revision: D22244489
fbshipit-source-id: b1bcb65a6d9e63bc963d9faf106db61cd507e452
Summary:
Enable narrow-heads.
Changed log revset from `:` to `all()` to make the test compatible.
Reviewed By: krallin
Differential Revision: D22200495
fbshipit-source-id: 148a82e77c953b9e7dbed055ef464c318e56cafa
Summary:
Enable narrow-heads, and mutation. Disable obsmarker related features.
Change phase manipulation to `debugmakepublic` which works with narrow-heads.
Reviewed By: krallin
Differential Revision: D22200511
fbshipit-source-id: 8dec050f137e6cc055015fe084eb4cc67faa1216
Summary:
Enable narrow-heads.
The test output seems a bit unstable - sometimes I got 28 there. So I globbed
it out.
Reviewed By: krallin
Differential Revision: D22200497
fbshipit-source-id: f005381a341d88c0bcbb09150e7d1878df7a38f3
Summary:
Enable narrow-heads.
Change the revset `:` to `all()`. With narrow-heads, `:` selects all commits
including those that are not referred by visible heads. The `all()` revset
only selects commits reachable from visible heads.
Reviewed By: krallin
Differential Revision: D22200498
fbshipit-source-id: beb863d42069ae898e419a4a75b3a707c72ae1f9
Summary:
Enable remotenames, selectivepull, and narrow-heads. Use the new stream clone
code path.
Selectivepull makes a difference. `hg pull -r HASH` also pulls the selected
bookmarks so an extra `pull` was unnecessary. Change the clone command to use
`-U` to trigger the new clone code path.
Reviewed By: krallin
Differential Revision: D22200499
fbshipit-source-id: 764202098c7e8afdbb5e2ee83679da7570c08c90
Summary:
Enable remotenames and narrow-heads.
Local bookmarks are replaced by remote bookmarks, causing the test change.
Reviewed By: krallin
Differential Revision: D22200500
fbshipit-source-id: aeee528d1766e0642c12e78a6c1a50cadc8a579a
Summary:
Enable remotenames and narrow-heads.
The commits become 'draft' because there are no remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200514
fbshipit-source-id: 04d0befa7c22756e936a28ffdcdf1305057cf062
Summary:
Enable remotenames and narrow-heads.
The test was migrated cleanly. The only change is that local bookmarks are
replaced by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200510
fbshipit-source-id: f5b8cd2ed125e9fc4e5daac897851d91fef5693f
Summary:
Enable remotenames and narrow-heads.
Local bookmarks are replaced by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200503
fbshipit-source-id: 41ac4f4f606011dcaf6d0d9867b01fb77b9a79d8
Summary:
Enable remotenames and narrow-heads.
Phase exchange is gone because of narrow-heads.
The remtoenames extension was written suboptimally so it issued a second
bookmarks request (which, hopefully can be removed by having selective
pull everywhere and migrate pull to use the new API).
Reviewed By: krallin
Differential Revision: D22200506
fbshipit-source-id: c522bb9fc1396d813e0f1f380c4290445bab3db3
Summary:
Enable remotenames and narrow-heads. The `master_bookmark` is no longer a local
bookmark in the client repo.
Reviewed By: krallin
Differential Revision: D22200513
fbshipit-source-id: bc3c1715ce21f45a35bc67148eb00e44944bea6e
Summary:
Enable remotenames and narrow-heads. The server gets one more request from
remotenames.
Reviewed By: krallin
Differential Revision: D22200502
fbshipit-source-id: 26bc28b19438c7be4a19eae6be728c83b113f822
Summary:
Enable remotenames and narrow-heads. The client gets remote bookmarks instead
of local bookmarks during clone and phases are decided by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200515
fbshipit-source-id: 12a9e892855b3a8f62f01758565de5f224c4942b
Summary:
Change the template to show remote bookmarks, which will be more relevant once
we migrate to modern configs. Namely, phases will be decided by remote bookmarks.
The named branches logic was mostly removed from the code base. Therefore
drop the `{branches}` template.
Reviewed By: StanislavGlebik
Differential Revision: D22200512
fbshipit-source-id: 8eca3a71ff88b8614023f4920a448156fcd712d5
Summary:
Most tests pass without changes. Some incompatible tests are added to the
special list.
Reviewed By: krallin
Differential Revision: D22200505
fbshipit-source-id: 091464bbc7c9c532fed9ef91f2c955d6e4f2df0b
Summary: This is the final diff of the stack - it starts logging pushed commits to scribe
Reviewed By: farnz
Differential Revision: D22212755
fbshipit-source-id: ec09728408468acaeb1c214d43f930faac30899b
Summary:
At the moment we can't test logging to scribe easily - we don't have a way to
mock it. Scribe are supposed to help with that.
They will let us to configure all scribe logs to go to a directory on a
filesystem similar to the way we configure scuba. The Scribe itself will
be stored in CoreContext
Reviewed By: farnz
Differential Revision: D22237730
fbshipit-source-id: 144340bcfb1babc3577026191428df48e30a0bb6
Summary: Many tests are incompatible. But many are passing.
Reviewed By: kulshrax
Differential Revision: D22052475
fbshipit-source-id: 1f30ac2b0fe034175d5ae818ec2be098dbd5283d
Summary:
For Lua hooks, we needed to know whether to run the hook per file, or per changeset. Rust hooks know this implicitly, as they're built-in to the server.
Stop having the tests set an unnecessary config
Reviewed By: krallin
Differential Revision: D22282799
fbshipit-source-id: c9f6f6325823d06d03341f04ecf7152999fcdbe7
Summary:
D21642461 (46d2b44c0e) converted Mononoke server to use the
`--mononoke-config-path` common argument style to select a config path.
Now that this change has been running for a while, remove the extra logic in
the server that allowed it to accept both the deprecated `--config_path / -P`
and the new arg.
Reviewed By: ikostia
Differential Revision: D22257386
fbshipit-source-id: 7da4ed4e0039d3659f8872693fa4940c58bae844
Summary:
`get_entry_with_small_repo_mapings` is a function that turns a `CommitEntry`
struct into `CommitEntryWithSmallReposMapped` struct - the idea being that this
function looks up hashes of commits into which the original commit from the
large repo got rewritten (in practice rewriting may have happened in the small
-> large direction, but it is not important for the purpose of this job). So it
establishes a mapping. Before this
diff, it actually established `Large<ChangesetId> ->
Option<(Small<ChangesetId>, Option<BookmarkName>)>` mapping, meaning that it
recorded into which bookmark large bookmark was rewritten. This was a useless
information (as evidenced by the fact that it was ignored by the
`prepare_entry` function, which turns `CommitEntryWithSmallReposMapped` into
`EntryPreparedForValidation`. It is useless because bookmarks are mutable and
it is impossible to do historic validation of the correctness of bookmark
renaming: bookmarks may have been correctly renamed when commits where pushes,
but they may be incorrectly renamed now and vice-versa. To deal with bookmarks,
we have a separate job, `bookmarks_validator`.
So this diff stops recording this useless information. As a bonus, this will
make migration onto `LiveCommitSyncConfig` easier.
Reviewed By: StanislavGlebik
Differential Revision: D22235389
fbshipit-source-id: c02b3f104a8cbd1aaf76100aa0930efeac475d42
Summary:
This diff migrates `backsyncer_cmd` (the thing that runs in the separate backsyncer job, as opposed to bakcsyncer, triggered from push-redirector) onto `LiveCommitSyncConfig`. Specifically, this means that on every iteration of the loop, which calls `backsync_latest` we reload `CommitSyncConfig` from configerator, build a new `CommitSyncer` from it, and then pass that `CommitSyncer` to `backsync_latest`.
One choice made here is to *not* create `CommitSyncer` on every iteration of the inner loop of `backsync_latest` and handle live configs outside. The reason for this is twofold:
- `backsync_latest` is called form `PushRedirector` methods, and `PushRedirector` is recreated on each `unbundle` using `LiveCommitSyncConfig`. That call provides an instance of `CommitSyncer` used to push-redirect a commit we want to backsync. It seems strictly incorrect to try and maybe use a different instance.
- because of some other consistency concerns (different jobs getting `CommitSyncConfig` updates at different times), any sync config change needs to go through the following loop:
- lock the repo
- land the change
- wait some time, until all the possible queues (x-repo sync and backsync) are drained
- unlock the repo
- this means that it's ok to have the config refreshed outside of `backsync_latest`
Reviewed By: farnz
Differential Revision: D22206992
fbshipit-source-id: 83206c3ebdcb2effad7b689597a4522f9fd8148a
Summary: I have previously moved the gitimport functionality (D22159880 (2cf5388835)) into a separate library, since repo_import shares similar behaviours. In this diff, I setup repo_import to be able to call gitimport to get the commits and changes. (Next steps include using Mover to set the paths of the files in the commits given by gitimport)
Reviewed By: StanislavGlebik
Differential Revision: D22233127
fbshipit-source-id: 4680c518943936f3e29d21c91a2bad60108e49dd
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch some of the blobstore interfaces to new-style `BoxFuture` with a `'static` lifetime.
This does not enable any fixes at this point, but does mean that `.compat()` moves to the places that need old-style futures instead of new. It also means that the work needed to make the transition fully complete is changed from a full conversion to new futures, to simply changing the lifetimes involved and fixing the resulting compile failures.
Reviewed By: krallin
Differential Revision: D22164315
fbshipit-source-id: dc655c36db4711d84d42d1e81b76e5dddd16f59d
Summary:
Push supported multiple bookmarks in theory, but in practice we never used it.
Since we want to start logging pushed commits in the next diffs we need to decide what to do with
bookmarks, since at the moment we can log only a single bookmark to scribe
let's just allow a single bookmark push
Reviewed By: farnz
Differential Revision: D22212674
fbshipit-source-id: 8191ee26337445ce2ef43adf1a6ded3e3832cc97
Summary:
This diff enables `unbundle` flow to start creating `push_redirector` structs from hot-reloaded `CommitSyncConfig` (by using the `LiveCommitSyncConfig` struct).
Using `LiveCommitSyncConfig` unfortunately means that we need to make sure those tests, which don't use standard fixtures, need to have both the `.toml` and the `.json` commit sync configs present, which is a little verbose. But it's not too horrible.
Reviewed By: StanislavGlebik
Differential Revision: D21962960
fbshipit-source-id: d355210b5dac50d1b3ad277f99af5bab56c9b62e
Summary: When running integration tests we should make the paths absolute, but kept it relative so far. This results it breaking the tests.
Reviewed By: krallin
Differential Revision: D22209498
fbshipit-source-id: 54ca3def84abf313db32aecfac503c7a42ed6576
Summary:
This diff is probably going to sound weird ... but xavierd and I both think
this is the best approach for where we are right now. Here is why this is
necessary.
Consider the following scenario
- A client creates a LFS object. They upload it to Mononoke LFS, but not
upstream.
- The client shares this (e.g. with Sandcastle), and includes a LFS pointer.
- The client tries to push this commit
When this happens, the client might not actually have the object locally.
Indeed, the only pieces of data the client is guaranteed to have is
locally-authored data.
Even if the client does have the blob, that's going to be in the hgcache, and
uploading from the hgcache is a bit sketchy (because, well, it's a cache, so
it's not like it's normally guaranteed to just hold data there for us to push
it to the server).
The problem boils down to a mismatch of assumptions between client and server:
- The client assumes that if the data wasn't locally authored, then the server
must have it, and will never request this piece of data again.
- The server assumes that if the client offers a blob for upload, it can
request this blob from the client (and the client will send it).
Those assumptions are obviously not compatible, since we can serve
not-locally-authored data from LFS and yet want the client to upload it, either
because it is missing in upstream or locally.
This leaves us with a few options:
- Upload from the hg cache. As noted above, this isn't desirable, because the
data might not be there to begin with! Populating the cache on demand (from
the server) just to push data back to the server would be quite messy.
- Skip the upload entirely, either by having the server not request the upload
if the data is missing, by having the server report that the upload is
optional, or by having the client not offer LFS blobs it doens't have to the
server, or finally by having the client simply disobey the server if it
doesn't have the data the server is asking for.
So, why can we not just skip the upload? The answer is: for the same reason we
upload to upstream to begin with. Consider the following scenario:
- Misconfigured client produces a commit, and upload it to upstream.
- Misconfigured client shares the commit with Sandcastle, and includes a LFS
pointer.
- Sandcastle wants to push to master, so it goes to check if the blob is
present in LFS. It isn't (Mononoke LFS checks both upstream and internal, and
only finds the blob in upstream, so it requests that the client submit the
blob), but it's also not not locally authored, so we skip the push.
- The client tries to push to Mononoke
This push will fail, because it'll reference LFS data that is not present in
Mononoke (it's only in upstream).
As for how we fix this: the key guarantee made by our proxying mechanism is
that if you write to either LFS server, your data is readable in both (the way
we do this is that if you write to Mononoke LFS, we write it to upstream too,
and if you write to upstream, we can read it from Mononoke LFS too).
What does not matter there is where the data came from. So, when the client
uploads, we simply let it submit a zero-length blob, and if so, we take that to
mean that the client doesn't think it authored data (and thinks we have it), so
we try to figure out where the blob is on the server side.
Reviewed By: xavierd
Differential Revision: D22192005
fbshipit-source-id: bf67e33e2b7114dfa26d356f373b407f2d00dc70
Summary:
I landed D22118926 (e288354caf) yesterday expecting those messages at about the same time
xavierd landed D21987918 (4d13ce1bcc), which removed them. This removes them from the
tests.
Reviewed By: StanislavGlebik
Differential Revision: D22204980
fbshipit-source-id: 6b1d696c93a07e942f86cd8df9a8e43037688728
Summary:
The Rust store code has been enabled everywhere for a few weeks now, let's
enable it by default in the code. Future changes will remove the config as well
as all the code associated with the non Rust store code.
The various tests changes are due to small difference between the Rust code and
the Python one, the biggest one being it's handling of corrupted packfiles. The
old code silently ignored them, while the new one errors out for local
packfiles. The test-lfs-bundle.t difference is just due to an ordering
difference between Python and Rust.
Reviewed By: kulshrax
Differential Revision: D21985744
fbshipit-source-id: 10410560193476bc303a72e7583f84924a6de820
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: This will be used in the following diffs. It just adds commitsync fixtures in a single place, so that we can later play with them in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21952665
fbshipit-source-id: 2933a9f7ea8343d5d52e6c3207e7d78a3ef0be25
Summary: This diff introduces `BlobRepoHg` extension trait for `BlobRepo` object. Which contains mercurial specific methods that were previously part of `BlobRepo`. This diff also stars moving some of the methods from BlobRepo to BlobRepoHg.
Reviewed By: ikostia
Differential Revision: D21659867
fbshipit-source-id: 1af992915a776f6f6e49b03e4156151741b2fca2
Summary: Not all facebook-specific code was moved out of integration_runner_real.py, but removing part of the code that is left would made the code less readable, the rest of it will be removed while the integration_runner_real.py is made usable for OSS
Reviewed By: farnz
Differential Revision: D22114948
fbshipit-source-id: d9c532a6a9ea653de2b12cffc92fbf45826dad37
Summary:
We support unicode file paths, and in python 3 those get passed to
python libraries as unicode strings. The tests set LANG=C which mean the python
library tries to convert the path to ascii, but fails for any non-ascii
characters. Let's switch to LANG="en_US.UTF-8" to match our production
behavior and make tests about unicode paths work.
Reviewed By: xavierd
Differential Revision: D22098359
fbshipit-source-id: c3057edc66e6e32f7b8b49374e622d02bd05711f
Summary: Megarepo is simplified if we can avoid copying hooks everywhere - run megarepo hooks as well as small repo hooks during pushredirection.
Reviewed By: StanislavGlebik
Differential Revision: D20652331
fbshipit-source-id: f42216797b9061db10b50c1440253de1f56d6b85
Summary:
Add optional compress on put controlled by a command line option.
Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.
Reviewed By: farnz
Differential Revision: D22037756
fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
Summary:
Rename the `subtree` endpoint on the EdenAPI server to `complete_trees` to better express what it does (namely, fetching complete trees, in contrast to the lighter weight `/trees` endpoint that serves individual tree nodes). This endpoint is not used by anything yet, so there isn't much risk in renaming it at this stage.
In addition to renaming the endpoint, the relevant request struct has been renamed to `CompleteTreeRequest` to better evoke its purpose, and the relevant client and test code has been updated accordingly. Notably, now that the API server is gone, we can remove the usage of this type from Mononoke's `hgproto` crate, thereby cleaning up our dependency graph a bit.
Reviewed By: krallin
Differential Revision: D22033356
fbshipit-source-id: 87bf6afbeb5e0054896a39577bf701f67a3edfec
Summary:
This is consistent with what is being done for now in hg for tests that haven't been migrated
to modern configurations yet, and ensures we get stable commit hashes in our tests: D21899139. It's already explicitly turned on on tests that want it.
In the future, this should probably be updated to use "modern configs" like the
Mercurial tests do.
Reviewed By: ikostia
Differential Revision: D22016705
fbshipit-source-id: b27f6423bf4ec5244ef3ce2e7676306165a331a8
Summary: This message is gone we shouldn't expect it anymore: D21913608
Reviewed By: ikostia
Differential Revision: D22016684
fbshipit-source-id: 97d86e9750e775c1bb3a1e75939f506cd35851c0
Summary:
This test broke when this got turned on for all tests (D21899139). It's not
enabled for other commit cloud tests there, so let's be consistent.
Reviewed By: ikostia
Differential Revision: D22016686
fbshipit-source-id: 5f4385b60fd31c89e335e971f262da1226f32254
Summary:
APIServer is deprecated, tw jobs stopped and deleted.
This diffs removes the source code.
Reviewed By: krallin
Differential Revision: D21839442
fbshipit-source-id: 5a4089d9205d8b0061c8aa01dcd74674fe9baca8
Summary: D21880220 renamed the `depth` parameter in Mononoke's history fetching functions to be `length`. This diff makes the same change for EdenAPI's `HistoryRequest` struct.
Reviewed By: StanislavGlebik
Differential Revision: D21948599
fbshipit-source-id: fe8649a5789f07d8262ad3d5e2f477a8b50f2c6f
Summary:
Add a new `subtree` endpoint to the EdenAPI server, which fetches trees using the underlying server-side logic for the `gettreepack` wire protocol command. This is intended for use in situations where compatibility with `gettreepack` is desired when using HTTP for tree fetching.
The name of the endpoint is up for bikeshedding. It seemed weird to name the endpoint `gettreepack` since that name is a verb and refers to a "pack" which is not relevant in this context (there are no wirepacks or packfiles involved). I chose the name `subtree` since the endpoint logically returns all of the nodes underneath a given node in the tree (though in the most frequent case, the node will be the root node and therefore the subtree will be the entire tree).
In practice, this initial implementation is not ideal because it buffers the trees in memory, which is problematic because `gettreepack` requests are likely to produce a very large number of trees. Later in this stack, the endpoint will be updated to produce a streaming response instead.
Reviewed By: StanislavGlebik
Differential Revision: D21782764
fbshipit-source-id: 726925858352c33c923da1979da9d20fbcf930f6
Summary:
There are people that are hurt by usage of these terms, this should be more
then enough reason to replace these. Newly chosen terms are more
self-explanatory as well.
This doesn't yet touch the actualy config files, as that requires a bit more
effort than 1 diff and will require more coordination.
Reviewed By: krallin
Differential Revision: D21924440
fbshipit-source-id: e24fc638dc8c9d6d20b6f3fa5f0d0bbc91bbf77b
Summary:
This test checks that we can start Mononoke and serve pull/push/update with
filenodes
Reviewed By: ahornby
Differential Revision: D21904753
fbshipit-source-id: 86690c5ed5ce7d022844809b09beb25c7961cac8
Summary: This will allow me to test it easily.
Reviewed By: StanislavGlebik
Differential Revision: D21840079
fbshipit-source-id: 1b3743da9c7908eb0dedd665aa24a4bf7aabd94f
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary:
See D21765065 for more context. TL;DR is that we want to control
lfs rollout from client side to make sure we don't put lfs pointers in the
shared memcache
Reviewed By: xavierd
Differential Revision: D21822159
fbshipit-source-id: daea6078d95eb4e9c040d353a20bcdf1b6ae07b1
Summary:
Out `CommitSyncConfig` struct now contains a `version_name` field, which is intended to be used as an identifier of an individual version of the `commitsyncmap` in use. We want to record this value in the `synced_commit_mapping` table, so that later it is possible to attribute a commit sync map to a given commit sync.
This is part of a broader goal of adding support for sync map versioning. The broader goal is important, as it allows us to move faster (and troubleshoot better) when sync maps need changing.
Note that when commit is preserved across repos, we set `version_name` to `NULL`, as it makes no sense to attribute commit sync maps to those case.
Reviewed By: farnz
Differential Revision: D21765408
fbshipit-source-id: 11a77cc4d926e4a4d72322b51675cb78eabcedee
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary:
Change the signature of `CreateCommitContext::as_file` and its associated
functions so that content is `impl Into<String>`, rather than
`impl AsRef<str>`. The content will immediately be converted to a `String`
anyway, so we can avoid a string copy if the caller already has a string that
can be moved.
Reviewed By: krallin
Differential Revision: D21743429
fbshipit-source-id: d54914386439489fe4e47e37ff9a75c52b1a0443
Summary:
Add support for drawdag in Mononoke unit tests. Tests can use ASCII DAGs to construct
commit graphs, and can optionally customize the content of each commit.
Reviewed By: krallin
Differential Revision: D21743431
fbshipit-source-id: 9e6a52d1efe67ef4a5519ed7783f953fef7358f1
Summary:
It was used only once for testing push redirection. We no longer need it, so
I'd like to delete it to remove this old code and also to make it easier to
support ManualMove bookmarks.
Differential Revision: D21745630
fbshipit-source-id: 362952d95edb923cc4b60359321b563c1e4961de
Summary:
This diff extends the integration test for the forward filler to execute queue operations, as well as the core business logic.
It also adds a test for the reverse filler, which does the same, but in a different difection.
Reviewed By: krallin
Differential Revision: D21628705
fbshipit-source-id: fb4ee0ecacc990d073425f3f37f794f74c057ea2
Summary:
This diff finally introduce the continuous reverse filler. Specifically, this adds a cli (and underlying wiring) to operate the filler logic in the `reversefillerqueue` table.
To achieve this:
- the filler class is turned into a base class with two subclasses for the forward and reverse fillers
- the main file is renamed from `forwardfiller.py` into `filler.py`, to better reflect the independence of direction.
Reviewed By: krallin
Differential Revision: D21628259
fbshipit-source-id: 5676a162a62f0dc6fe80e6300b72d30370fc80b4
Summary: Add devdb support to integration test runner so that one can more easily inspect mysql test data, also makes it easier to run tests with locally modified schema.
Reviewed By: HarveyHunt
Differential Revision: D21645234
fbshipit-source-id: ec75d70ef59f04548c7346a122298567dd09c264
Summary:
At first glance people will assume that changesets are returned in the same
order that they were added in the database or that at least commits are
returned in a deterministic fashion. That didn't happen because the both
changeset ids and changeset entries were received without any order.
This diff updates the function to returns results in order they were added
to the database.
Reviewed By: krallin
Differential Revision: D21676663
fbshipit-source-id: 912e6bea0532796b1d8e44e47d832c0420d97bc1
Summary:
Currently we record them only during pushrebase. Let's record during push as
well.
To simplify things a little bit let's allow only a very simple push case:
1) Single bookmark.
2) All pushed commits should be reachable by this bookmark.
Reviewed By: krallin
Differential Revision: D21451337
fbshipit-source-id: bf2f1e6025ac116fb8096824b7c4c6440d073874
Summary:
Add a --sample-path-regex option for use in the corpus dumper so we can dump out just a subset of directories from a repo.
This is most useful on large repos.
Reviewed By: farnz
Differential Revision: D21325548
fbshipit-source-id: bfda87aa76fbd325e4e01c2df90b5dcfc906a8f6
Summary:
Update the corpus walker to dump the sampled bytes as early as possible to the Inflight area of the output dir, then move them to final location once path is known.
When walking large files and manifests this uses a lot less memory that holding the bytes in a map!
Layout is changed is to make comparison by file type easier. we get a top level dir per extension, e.g. all .json files are under FileContent/byext/json
This also reduces the number of bytes taken from the sampling fingerprint used to make directories, 8 was overkill. 3 is enough to limit directory size.
Reviewed By: farnz
Differential Revision: D21168633
fbshipit-source-id: e0e108736611d552302e085d91707cca48436a01
Summary:
Add corpus dumper for space analysis
This reuses the path based tracking from compression-benefit and the size sampling from scrub.
The core new functionality is the dump to disk from inside corpus stream.
Reviewed By: StanislavGlebik
Differential Revision: D20815125
fbshipit-source-id: 01fdc9dd69050baa8488177782cbed9e445aa3f7
Summary:
We now have auto pull logic that covers most unknown rev use-cases. The hint
message is no longer necessary. It's also unclear how to use `hg pull`
correctly. For example, should it be `-r`, `-B remote/foo` or `-B foo`?
Reviewed By: DurhamG
Differential Revision: D21526667
fbshipit-source-id: 40583bfb094e52939130250dd71b96db4d725ad5
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary:
Right now, we debug-print the root cause and pretty-print everything else. This
is pretty bad because the root cause is usually the one thing we would want to
pretty print so we can add instructions there (such as "your hooks failed, fix
it").
This fixes this so we stop pretty-printing the root cause, but also debug print
the whole error, which gives us more developer-friendly context and is easier
for automation to match on.
This is actually in common/rust ... but we're the only people using it AFAICT.
Reviewed By: StanislavGlebik
Differential Revision: D21522518
fbshipit-source-id: 10158811574b56024e14852229e4541da19d5609
Summary:
At least let's tell the use what to do about the problem and, where we can,
what the conflicting file was (see the attached task).
Reviewed By: farnz
Differential Revision: D21459412
fbshipit-source-id: 52b90cf7d41ebe6550083c6673b4e93b10edf5e2
Summary:
Not all node types can have a path associated
Reset the tracked path to None if the route is taking us through a node type that can't have a repo path.
Reviewed By: krallin
Differential Revision: D21228372
fbshipit-source-id: 2b1e291f09232500adce79c630d428f09cd2d2cc
Summary:
Add new --sample-offset argument so that in combination with the existing --sample-rate the whole repo can be sampled in slices
For --sample-rate=N, this allows us to scrub or corpus dump 1/Nth of the repo a time, which is particularly useful for corpus dumping on machines with limited disk.
Also factored out the sampling args construction as 3 of the 4 walk variants use them (only validate does not)
Reviewed By: krallin
Differential Revision: D21158486
fbshipit-source-id: 94f98ceb71c22e0e9d368a563cdb04225b6fc459
Summary:
Add a simple `/repos` endpoint that returns the list of repos available in a JSON response.
While the handler itself is quite simple, this diff establishes the general pattern by which new handlers will be added to the server.
Reviewed By: krallin
Differential Revision: D21330778
fbshipit-source-id: 77f57c969c34c8c1f7c94979fac383ec442a1e14
Summary:
We want to be able to record all the bundles Mononoke processes to be later
replayed by Mercurail.
Reviewed By: krallin
Differential Revision: D21427622
fbshipit-source-id: b88e10e03d07dae35369286fe31022f36a1ee5cf
Summary: Cover as much as remining code with `Cargo.toml`s, for the rest create an exlusion list in the autocargo config.
Reviewed By: krallin
Differential Revision: D21383620
fbshipit-source-id: 64cc78a38ce0ec482966f32a2963ab4939e20eba
Summary:
- Change get return value for `Blobstore` from `BlobstoreBytes` to `BlobstoreGetData` which include `ctime` metadata
- Update the call sites and tests broken due to this change
- Change `ScrubHandler::on_repair` to accept metadata and log ctime
- `Fileblob` and `Manifoldblob` attach the ctime metadata
- Tests for fileblob in `mononoke:blobstore-test` and integration test `test-walker-scrub-blobstore.t`
- Make cachelib based caching use `BlobstoreGetData`
Reviewed By: ahornby
Differential Revision: D21094023
fbshipit-source-id: dc597e888eac2098c0e50d06e80ee180b4f3e069
Summary:
This test is flaky right now, but it's not clear why. I'm also unable to repro.
Let's add more logging.
Reviewed By: StanislavGlebik
Differential Revision: D21405284
fbshipit-source-id: 3ce5768066091de61e62339286410a6223d251d5
