Summary:
Add a --sample-path-regex option for use in the corpus dumper so we can dump out just a subset of directories from a repo.
This is most useful on large repos.
Reviewed By: farnz
Differential Revision: D21325548
fbshipit-source-id: bfda87aa76fbd325e4e01c2df90b5dcfc906a8f6
Summary:
Update the corpus walker to dump the sampled bytes as early as possible to the Inflight area of the output dir, then move them to final location once path is known.
When walking large files and manifests this uses a lot less memory that holding the bytes in a map!
Layout is changed is to make comparison by file type easier. we get a top level dir per extension, e.g. all .json files are under FileContent/byext/json
This also reduces the number of bytes taken from the sampling fingerprint used to make directories, 8 was overkill. 3 is enough to limit directory size.
Reviewed By: farnz
Differential Revision: D21168633
fbshipit-source-id: e0e108736611d552302e085d91707cca48436a01
Summary:
Add corpus dumper for space analysis
This reuses the path based tracking from compression-benefit and the size sampling from scrub.
The core new functionality is the dump to disk from inside corpus stream.
Reviewed By: StanislavGlebik
Differential Revision: D20815125
fbshipit-source-id: 01fdc9dd69050baa8488177782cbed9e445aa3f7
Summary:
We now have auto pull logic that covers most unknown rev use-cases. The hint
message is no longer necessary. It's also unclear how to use `hg pull`
correctly. For example, should it be `-r`, `-B remote/foo` or `-B foo`?
Reviewed By: DurhamG
Differential Revision: D21526667
fbshipit-source-id: 40583bfb094e52939130250dd71b96db4d725ad5
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary:
Right now, we debug-print the root cause and pretty-print everything else. This
is pretty bad because the root cause is usually the one thing we would want to
pretty print so we can add instructions there (such as "your hooks failed, fix
it").
This fixes this so we stop pretty-printing the root cause, but also debug print
the whole error, which gives us more developer-friendly context and is easier
for automation to match on.
This is actually in common/rust ... but we're the only people using it AFAICT.
Reviewed By: StanislavGlebik
Differential Revision: D21522518
fbshipit-source-id: 10158811574b56024e14852229e4541da19d5609
Summary:
At least let's tell the use what to do about the problem and, where we can,
what the conflicting file was (see the attached task).
Reviewed By: farnz
Differential Revision: D21459412
fbshipit-source-id: 52b90cf7d41ebe6550083c6673b4e93b10edf5e2
Summary:
Not all node types can have a path associated
Reset the tracked path to None if the route is taking us through a node type that can't have a repo path.
Reviewed By: krallin
Differential Revision: D21228372
fbshipit-source-id: 2b1e291f09232500adce79c630d428f09cd2d2cc
Summary:
Add new --sample-offset argument so that in combination with the existing --sample-rate the whole repo can be sampled in slices
For --sample-rate=N, this allows us to scrub or corpus dump 1/Nth of the repo a time, which is particularly useful for corpus dumping on machines with limited disk.
Also factored out the sampling args construction as 3 of the 4 walk variants use them (only validate does not)
Reviewed By: krallin
Differential Revision: D21158486
fbshipit-source-id: 94f98ceb71c22e0e9d368a563cdb04225b6fc459
Summary:
Add a simple `/repos` endpoint that returns the list of repos available in a JSON response.
While the handler itself is quite simple, this diff establishes the general pattern by which new handlers will be added to the server.
Reviewed By: krallin
Differential Revision: D21330778
fbshipit-source-id: 77f57c969c34c8c1f7c94979fac383ec442a1e14
Summary:
We want to be able to record all the bundles Mononoke processes to be later
replayed by Mercurail.
Reviewed By: krallin
Differential Revision: D21427622
fbshipit-source-id: b88e10e03d07dae35369286fe31022f36a1ee5cf
Summary: Cover as much as remining code with `Cargo.toml`s, for the rest create an exlusion list in the autocargo config.
Reviewed By: krallin
Differential Revision: D21383620
fbshipit-source-id: 64cc78a38ce0ec482966f32a2963ab4939e20eba
Summary:
- Change get return value for `Blobstore` from `BlobstoreBytes` to `BlobstoreGetData` which include `ctime` metadata
- Update the call sites and tests broken due to this change
- Change `ScrubHandler::on_repair` to accept metadata and log ctime
- `Fileblob` and `Manifoldblob` attach the ctime metadata
- Tests for fileblob in `mononoke:blobstore-test` and integration test `test-walker-scrub-blobstore.t`
- Make cachelib based caching use `BlobstoreGetData`
Reviewed By: ahornby
Differential Revision: D21094023
fbshipit-source-id: dc597e888eac2098c0e50d06e80ee180b4f3e069
Summary:
This test is flaky right now, but it's not clear why. I'm also unable to repro.
Let's add more logging.
Reviewed By: StanislavGlebik
Differential Revision: D21405284
fbshipit-source-id: 3ce5768066091de61e62339286410a6223d251d5
Summary:
I'm going to send a diff to get rid of failure chains, and the LFS Server
actually uses that quite a bit. Let's make sure we don't affect the error
rendering there.
Reviewed By: StanislavGlebik
Differential Revision: D21383032
fbshipit-source-id: e0ec9c88760e7fd48d39fa1570efd1870a9ef532
Summary:
Looks like this broke yesterday. There was a Reindeer update yesterday IIRC, so
I'm guessing that's the cause. In any case, this is easy to fix forward.
Reviewed By: farnz
Differential Revision: D21399830
fbshipit-source-id: 5cf33411e089a8c675a8b3fdf7b6ae5ae267058d
Summary:
This adds support for running Gitimport with `--readonly-storage`. The way we
do this is by masking the various storages we use (blobstore, changesets,
bonsai).
Reviewed By: markbt
Differential Revision: D21347939
fbshipit-source-id: 68084ba0d812dc200776c761afdfe41bab9a6d82
Summary:
The original gitimport wasn't really designed for concurrency, since it did
commits one by one. With this update, we can now derive Bonsais from multiple
commits in parallel, and use multiple threads to communicate with the Git
repository (which is actually somewhat expensive when that's all we do).
We also store Bonsais iteratively. There is a bit of extra work that could be
done also here by saving Bonsais asynchronously to the Blobstore, and inserting
a single batch in Changesets once we're finished.
Reviewed By: farnz
Differential Revision: D21347941
fbshipit-source-id: e0ea86bf4d164599df1370844d3f0301d1031801
Summary:
This adds support for deriving commits within a range in gitimport, which gets
us one step closer to resumable gitimport. The primary goal of this is to
evaluate whether using Gitimport for Configerator might be suitable.
Differential Revision: D21347942
fbshipit-source-id: aa3177466e389ceb675328999ccf836f29912698
Summary:
This adds some basic functionality for deriving hg manifests in gitimport. I'd
like to add this to do some correctness testing on importing Git manifests from
Configerator.
Differential Revision: D21347940
fbshipit-source-id: 6f819fa8a62b3088fb163138fc23910b8f2ff3ce
Summary:
Currently, Mononoke's configs are loaded at startup and only refreshed
during restart. There are some exceptions to this, including throttling limits.
Other Mononoke services (such as the LFS server) have their own implementations
of hot reloadable configs, however there isn't a universally agreed upon method.
Static configs makes it hard to roll out features gradually and safely. If a
bad config option is enabled, it can't be rectified until the entire tier is
restarted. However, Mononoke's code is structured with static configs in mind
and doesn't support hot reloading. Changing this would require a lot of work
(imagine trying to swap out blobstore configs during run time) and wouldn't
necessarily provide much benefit.
Instead, add a subset of hot reloadable configs called tunables. Tunables are
accessible from anywhere in the code and are cheap to read as they only require
reading an atomic value. This means that they can be used even in hot code
paths.
Currently tunables support reloading boolean values and i64s. In the future,
I'd like to expand tunables to include more functionality, such as a rollout
percentage.
The `--tunables-config` flag points to a configerator spec that exports a
Tunables thrift struct. This allows differents tiers and Mononoke services to
have their own tunables. If this isn't provided, `MononokeTunables::default()`
will be used.
This diff adds a proc_macro that will generate the relevant `get` and `update`
methods for the fields added to a struct which derives `Tunables`. This struct is
then stored in a `once_cell` and can be accessed using `tunables::tunables()`.
To add a new tunable, add a field to the `MononokeTunables` struct that is of
type `AtomicBool` or `AtomicI64`. Update the relevant tunables configerator
config to include your new field, with the exact same name.
Removing a tunable from `MononokeTunables` is fine, as is removing a tunable
from configerator.
If the `--tunables-config` path isn't passed, then a default tunables config
located at `scm/mononoke/tunables/default` will be loaded. There is also the
`--disable-tunables` flag that won't load anything from configerator, it
will instead use the `Tunable` struct's `default()` method to initialise it.
This is useful in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21177252
fbshipit-source-id: 02a93c1ceee99066019b23d81ea308e4c565d371
Summary:
Building on the previous two commits, this adds a test which performs the following steps:
- does an infintiepush push to a Mercurial server
- looks into the `forwardfillerqueue`
- runs commitcloud forwardfiller's `fill-one` to replay the bundle to Mononoke
- verifies that this action causes the commit to appear in Mononoke
As this test uses `getdb.sh` from Mercurial test suite, it needs to be whitelisted from network blackholing (Note: we whitelist mononoke tests, which run with `--mysql` automatically, but this one is different, so we need to add it manually. See bottom diff of the stack for why we don't use `--mysql` and ephemeral shards here).
Reviewed By: krallin
Differential Revision: D21325071
fbshipit-source-id: d4d6cbdb10a2bcf955ee371278bf2bbbd5f5122c
Summary:
Microwave doesn't normally allow writes, which can cause cache warmup to fail
if master has underived commits. So, let's go back in bookmarks history to
whatever is most recent and derived. We can do so using the existing logic we
use in the warm bookmarks cache.
Reviewed By: farnz
Differential Revision: D21325485
fbshipit-source-id: 11e758cd512a22e02704ac34458fead18c284c20
Summary: The changes to server/context, gotham_ext and the code that depends on them are the only reminding places where aclchecker is used directly and it is not easy to split this diff to convert them separately.
Reviewed By: krallin
Differential Revision: D21067809
fbshipit-source-id: a041ab141caa6fe6871e1fda6013e33f1f09bc56
Summary:
It can be quite convenient when you have a bundle ID to be able to quickly
translate it to a Bonsai and get information about said Bonsai in order to e.g.
identify whether it was particularly large. This adds that functionality.
Reviewed By: StanislavGlebik
Differential Revision: D21228409
fbshipit-source-id: fc2ff938ff16e99c88b3e522b7ac39c4f39d60f2
Summary:
This code had grown into a pretty big monster of a future. This makes it a bit
easier to modify and work with.
Reviewed By: StanislavGlebik
Differential Revision: D21227210
fbshipit-source-id: 5982daac4d77d60428e80dc6a028cb838e6fade0
Summary:
Add --derived-data-type=fsnodes to blobimport to a couple of walker tests so we have test data present to load.
Includes a small change to library.sh to add default_setup_pre_blobimport entry point used by these tests
Reviewed By: StanislavGlebik
Differential Revision: D21202480
fbshipit-source-id: d7eb3e5736531a11da87d92d0d03a528ff2c91a7
Summary:
This adds the proper hooks in the right place to upload the LFS blobs and write
to the bundle as LFS pointers. That last part is a bit hacky as we're writing
the pointer manually, but until that code is fully Rust, I don't really see a
good way of doing it.
Reviewed By: DurhamG
Differential Revision: D20843139
fbshipit-source-id: f2ef7b045c6604398b89580b468c354d14de1660
Summary:
Add the Mononoke Mercurial mutation store. This stores mutation information
for draft commits so that it can be shared between clients. The mutation
entries themselves are stored in a database, and the mutation store provides
abstractions for adding and querying them.
This commit adds the `all_predecessors` method to the mutation store, which
allows Mononoke to fetch all predecessors for a given set of commits. This
will be used to serve mutation information to clients who are pulling draft
commits.
Reviewed By: krallin
Differential Revision: D20287381
fbshipit-source-id: b4455514cb8a22bef2b9bf0229db87c2a0404448
Summary:
Make the default output for `scsc log` shorter by only including the first line of the commit message, and omitting less interesting fields like commit extras.
The full details are hidden behind a `--verbose` flag, similar to `hg log`.
Reviewed By: mitrandir77
Differential Revision: D21202318
fbshipit-source-id: f15a0f8737f17e3189ea1bbe282d78a9c7199dd9
Summary: Make it possible to traverse fsnodes in walker.
Reviewed By: ahornby
Differential Revision: D21153883
fbshipit-source-id: 047ab73466f48048a34cb52e7e0f6d04cda3143b
Summary:
For some nodes like FileContent from a BonsaiChangset, the file path is not part of node identity, but it is important for tracking which nodes are related to which paths.
This change adds an optional path field to the OutGoingEdge so that it can be used in route generation and as part of edge identity for sampling.
Its optional as some walks don't need the paths, for example scrub.
Reviewed By: farnz
Differential Revision: D20835653
fbshipit-source-id: f609c953da8bfa0cdfdfb26328149d567c73dbc9
Summary: Report stats by the node type (e.g. FileContent, HgManifest etc) for blobstore usage when scrubbing so we can see how large each type is.
Reviewed By: StanislavGlebik
Differential Revision: D20564327
fbshipit-source-id: 55efd7671f893916d8f85fa9a93f95c97a098af4
Summary:
Correctly identify infinitepush without bookmarks as infinitepush instead of plain push.
Current behavior would sometimes pass `infinitepush` bundles through the `push` pipeline. Interestingly, this does not result in any user-visible effects at the moment. However, in the future we may want to diverge these pipelines:
- maybe we want to disable `push`, but enable `infinitepush`
- maybe there will be performance optimizations, applicable only to infinitepush
In any case, the fact that things worked so far is a consequence of a historical accident, and we may not want to keep it this way. Let's have correct identification.
Reviewed By: StanislavGlebik
Differential Revision: D18934696
fbshipit-source-id: 69650ca2a83a83e2e491f60398a4e03fe8d6b5fe
Summary:
In next diffs with permission_checker the panic is changed to anyhow::Error.
The previous behavior of this code was that when AclChecker updated failed
after 10s this fact was ignored and the hooks were simply not using ACLs. This
diff fixes it so that the server exits when AclChecheker update is timing out.
Reviewed By: johansglock
Differential Revision: D21155944
fbshipit-source-id: ab4a5071acbe6a1282a7bc5fdbf301b4bd53a347
Summary:
Allow us to see the sizes for each node type (e.g. manifests, bonsais etc), and extends the default reporting to all types.
The progress.rs changes update its summary by type reporting to be reusable, and then it is reused by the changes to sizing.rs.
Reviewed By: krallin
Differential Revision: D20560962
fbshipit-source-id: f09b45b34f42c5178ba107dd155abf950cd090a7
Summary:
This was how this was supposed to work all along, but there was a bug in the
sense that if the counter is where want to set it, then the update affects 0
rows. This is a bit of a MySQL idiosyncrasy — ideally we would set
CLIENT_FOUND_ROWS on our connections in order to be consistent with SQLite.
That said, for now, considering we are the only ones touching this counter, and
considering this code isn't intended to be long-lived, it seems reasonable to
just check the counter after we fail to set it.
(see https://dev.mysql.com/doc/refman/8.0/en/mysql-affected-rows.html for
context)
Reviewed By: HarveyHunt
Differential Revision: D21153966
fbshipit-source-id: 663881c29a11a619ec9ab20c4291734ff13d798a
Summary:
These failed when I did a local run against fbcode warm
count-objects.t and enabled-derive.t were flaky depending on the exact path taken through the graph.
Reviewed By: ikostia
Differential Revision: D21092866
fbshipit-source-id: ac4371cf81128b4d38cd764d86fc45d44d639ecc
Summary:
This is a POC attempt to increase the type safety of the megarepo codebase by introducing the `Small`/`Large` [newtype](https://doc.rust-lang.org/rust-by-example/generics/new_types.html) wrappers for some of the function arguments.
As an example, compare these two function signatures:
```
pub async fn verify_filenode_mapping_equivalence<'a>(
ctx: CoreContext,
source_hash: ChangesetId,
source_repo: &'a BlobRepo,
target_repo: &'a BlobRepo,
moved_source_repo_entries: &'a PathToFileNodeIdMapping,
target_repo_entries: &'a PathToFileNodeIdMapping,
reverse_mover: &'a Mover,
) -> Result<(), Error>
```
and
```
async fn verify_filenode_mapping_equivalence<'a>(
ctx: CoreContext,
source_hash: Large<ChangesetId>,
large_repo: &'a Large<BlobRepo>,
small_repo: &'a Small<BlobRepo>,
moved_large_repo_entries: &'a Large<PathToFileNodeIdMapping>,
small_repo_entries: &'a Small<PathToFileNodeIdMapping>,
reverse_mover: &'a Mover,
) -> Result<(), Error>
```
In the first case, it is possible to call function with the source and target repo inverted accidentally, whereas in the second one it is not.
Reviewed By: StanislavGlebik
Differential Revision: D20463053
fbshipit-source-id: 5f4f9ac918834dbdd75ed78623406aa777950ace
Summary:
The ID in Hgsql is supposed to the next globalrev to assign, not the last one
that was assigned. We would have otherwise noticed during the rollout since
we'd have seen that the counter in Mercurial wasn't `globalrev(master) + 1`
(and we could have fixed it up manually before it had any impact), but let's
fix it now.
Reviewed By: StanislavGlebik
Differential Revision: D21089653
fbshipit-source-id: 0a37e1b7299a0606788bd87f788799db6e3d55f4
