Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary:
This diff adds support for the `version_name` field, coming from the
`commitsyncmap` config, stored in the configerator.
Note: ATM, this field is optional in the thrift config, but once we get past
the initial deployment stage, I expect it to be present always. This is why
in `CommmitSyncConfig` I make it `String` (with a default value of `""`) rather
than `Option<String>`. The code, which will be writing this value into
`synced_commit_mapping` should not ever care whether it's present or not, since
every mapping should always have a `version_name` present.
Reviewed By: StanislavGlebik
Differential Revision: D21764641
fbshipit-source-id: 35a7f487acf0562b309fd6d1b6473e3b8024722d
Summary: Reverting as D20763778 is a suspect in causing thrift_server_overload T67609407
Reviewed By: StanislavGlebik
Differential Revision: D21762641
fbshipit-source-id: 545b448afc0954271a2e29d1d3b48fdb959e3d3d
Summary:
Change the signature of `CreateCommitContext::as_file` and its associated
functions so that content is `impl Into<String>`, rather than
`impl AsRef<str>`. The content will immediately be converted to a `String`
anyway, so we can avoid a string copy if the caller already has a string that
can be moved.
Reviewed By: krallin
Differential Revision: D21743429
fbshipit-source-id: d54914386439489fe4e47e37ff9a75c52b1a0443
Summary:
Add support for drawdag in Mononoke unit tests. Tests can use ASCII DAGs to construct
commit graphs, and can optionally customize the content of each commit.
Reviewed By: krallin
Differential Revision: D21743431
fbshipit-source-id: 9e6a52d1efe67ef4a5519ed7783f953fef7358f1
Summary:
The parser currently uses pattern destructuring for `RawInfinitepushParams`. This will break
if new fields are added to this structure. Instead, use field access like the other raw
params parsers.
Reviewed By: mitrandir77
Differential Revision: D21742558
fbshipit-source-id: 6bfbb080a5e5cdbb02519855472f4df80f9d7453
Summary:
It was used only once for testing push redirection. We no longer need it, so
I'd like to delete it to remove this old code and also to make it easier to
support ManualMove bookmarks.
Differential Revision: D21745630
fbshipit-source-id: 362952d95edb923cc4b60359321b563c1e4961de
Summary: Useful for determining where an incremental building step left off.
Reviewed By: StanislavGlebik
Differential Revision: D21634698
fbshipit-source-id: e9b0473003c529d5c934754f1ece23df69c4be66
Summary:
This diff extends the integration test for the forward filler to execute queue operations, as well as the core business logic.
It also adds a test for the reverse filler, which does the same, but in a different difection.
Reviewed By: krallin
Differential Revision: D21628705
fbshipit-source-id: fb4ee0ecacc990d073425f3f37f794f74c057ea2
Summary:
This diff finally introduce the continuous reverse filler. Specifically, this adds a cli (and underlying wiring) to operate the filler logic in the `reversefillerqueue` table.
To achieve this:
- the filler class is turned into a base class with two subclasses for the forward and reverse fillers
- the main file is renamed from `forwardfiller.py` into `filler.py`, to better reflect the independence of direction.
Reviewed By: krallin
Differential Revision: D21628259
fbshipit-source-id: 5676a162a62f0dc6fe80e6300b72d30370fc80b4
Summary: Add devdb support to integration test runner so that one can more easily inspect mysql test data, also makes it easier to run tests with locally modified schema.
Reviewed By: HarveyHunt
Differential Revision: D21645234
fbshipit-source-id: ec75d70ef59f04548c7346a122298567dd09c264
Summary:
At first glance people will assume that changesets are returned in the same
order that they were added in the database or that at least commits are
returned in a deterministic fashion. That didn't happen because the both
changeset ids and changeset entries were received without any order.
This diff updates the function to returns results in order they were added
to the database.
Reviewed By: krallin
Differential Revision: D21676663
fbshipit-source-id: 912e6bea0532796b1d8e44e47d832c0420d97bc1
Summary:
This structure has similar functionality to the IdMap that is backed by SQL.
It is probably going to be useful for caching in the case of batch operations.
Reviewed By: quark-zju
Differential Revision: D21601820
fbshipit-source-id: 9c3ebc3e9dc92a59ce0908fc241bd2b97da88dca
Summary:
`Dag::build_all_graph` will load the whole graph for a given repository and
construct the segmented changelog from it.
Reviewed By: StanislavGlebik
Differential Revision: D21538029
fbshipit-source-id: b4ba846bb2870ba73257bed6128b8e198a0aab3e
Summary:
Change from CHashMap to DashMap for the walk state tracking.
DashMap is using slightly less memory in testing, and is slightly quicker in walk rate (number of graph nodes processed per second ).
Reviewed By: HarveyHunt
Differential Revision: D21662210
fbshipit-source-id: ea0601df17e0e596fd59b67d9d01d0dc4e90799b
Summary:
CacheBlob logs hit and miss stastics to Scuba. Let's add the same for
ODS.
Reviewed By: krallin
Differential Revision: D21640922
fbshipit-source-id: 8f7d17f048bf53bdc6cd8bda0384a51cae7b6a30
Summary:
CountedBlobstore is a Blobstore that wraps another blobstore in order
to report metrics about it, such as the number of gets or errors. It's commonly
used to wrap a CacheBlob blobstore, which itself is a caching wrapper over an
inner blobstore.
CountedBlobstore exposes metrics that are supposed to track the number of hits
or misses when fetching blobs. However, these metrics don't make sense as the
CountedBlobstore has no view into cache activity. These metrics actually report
the number of requests and the number of missing blobs rather than hits and
misses.
Remove these misleading counters.
Reviewed By: krallin
Differential Revision: D21640923
fbshipit-source-id: 07b9fc9864c70991415c2b84f35d631b702c17d1
Summary:
now fastlog will be able to show a history of file like that:
```
Added
|
Deleted
|
Added
```
We do it by using deleted file manifest to find where a file was deleted. We do
it in case we find a unode which doesn't have a parent.
Differential Revision: D21664248
fbshipit-source-id: 8e80fccc87361e66e4e00ae2da3671ed8327f1f8
Summary:
... or rather make this special case a bit smaller.
In fastlog we have a `prefetch` which was already returned to the caller,
however its parents hasn't been processed yet. Previously we had two separate
places where these parents were processed, now we have only one.
This will make it easier to add support for history across deletions in the
next diff.
Differential Revision: D21664246
fbshipit-source-id: a5a109fafaeec28a0e7a36ffa44545aebc032b00
Summary:
Just top-level mononoke and cache_warmup as a more or less random
selection.
Reviewed By: StanislavGlebik
Differential Revision: D21648852
fbshipit-source-id: 0e5137da9a7742377112e085e9d29182abdfbf41
Summary:
Previously we created `visited` and immediately drop it.
Let's instead create it one level up. I don't think it'd matter in practice,
but it should be cleaner
Reviewed By: HarveyHunt
Differential Revision: D21662456
fbshipit-source-id: 36823309ea09448c5be10c2655ea0f317649f290
Summary:
Follow up from D21596758 - current logic of pushrebasing a merge is very
complicated. To prevent repo corruptions let's do a very simple validation -
generate hg changeset from a rebased bonsai changeset.
Given that generating hg changeset is an expensive operation let's do it only
after the first rebase attempt - otherwise we might risk constantly losing the
pushrebase race.
Differential Revision: D21659187
fbshipit-source-id: f43a855cf0fbdbd11a40d3ec38283af344cde5e6
Summary: This is to bring it into sync with the `forwardfillerqueue` types.
Reviewed By: markbt
Differential Revision: D21660012
fbshipit-source-id: 5148023478c175cd49707d88251701a08fcbe0ce
Summary:
Mononoke server is the only binary using plain `clap::App`, whilst all
other services use `MononokeApp`. The `MononokeApp` variant configures some
default arguments and provides consistent argument naming between binaries.
Update the server to use `MononokeApp` and add some logic to allow passing
either `--config_path` or `--mononoke-config-path` until we remove all uses of
`--config-path`.
Reviewed By: StanislavGlebik
Differential Revision: D21642461
fbshipit-source-id: f42e0b6625f3979ced0920db269bdb4528f99e49
Summary:
When starting up with many repos, it can take a while before the server is ready
to accept connections, and there is no log to indicate when that is the case.
Add one.
Reviewed By: farnz
Differential Revision: D21641189
fbshipit-source-id: 057ca06c1b1725799a8445841b6eea89bba783b0
Summary:
Rather than logging `Refreshing tunables` all the time, log only when they
change, and include the values that they are changing from and to.
Reviewed By: farnz
Differential Revision: D21641191
fbshipit-source-id: b0b5edd687e2e41074ce1c8be13fd6918dff924d
Summary:
There are a few paths through `resolve` that return early with a failure, and thus never record what happened.
Make a record the moment we enter `resolve` - then, we can use `count` type ODS charts to determine the failure rate deterministically, and alarm if the failure rate is too high
Reviewed By: ahornby
Differential Revision: D21647575
fbshipit-source-id: 667787ec000a8cd8e715563df10dbb84832fefa1
Summary:
D21573455 accidentally removed the scuba sampling for scrub_get
operations. Add this back in.
Reviewed By: StanislavGlebik
Differential Revision: D21638972
fbshipit-source-id: eee66dbce161de69246f4da0a15dc2cf00e1ba01
Summary:
See bottom diff in the stack for the motivation. Though you can probably guess
the motivation :)
Reviewed By: farnz
Differential Revision: D21623154
fbshipit-source-id: a0940d766a67080ddcb346c2e3313eb08699edad
Summary: First diff in the stack that removes getfiles since it's no longer needed.
Reviewed By: farnz
Differential Revision: D21623156
fbshipit-source-id: 44f310ec4e4f34845cc5bf1738f1a8ece14e6694
Summary:
This updates a placeholder in the authentication process for requests
in the rust thrift services to allow authentication with CATs. This change
allows for passing in a configurable server identity to be used as the
verifier identity here.
Additionally this sets the verifer identity to be "scm_service_identity"
for the mononoke scs server (the service identity for this service).
This is needed to use the mononoke scs server from corp.
Reviewed By: markbt
Differential Revision: D21560204
fbshipit-source-id: 02e41718307fddab6f0ac2101306bb7f4a59cf7e
Summary:
Currently, we have a gap between updating and subsequently reading the
value of `write_order`. If another blobstore's put operation completes before
we have called `record_put_stats` then we may have an incorrect `write_order`
value.
Whilst this is only a minor issue, let's fix it anyway. :-)
Reviewed By: krallin
Differential Revision: D21619669
fbshipit-source-id: 1b8bacbcb4c195e6765ebdfaa68425f286f88c06
Summary:
After other optimizations the CHashMap version of visit_count was showing as a hot spot.
Given the number of possible NodeTypes is small we can store the visit_count in array indexed by ordinal instead.
Reviewed By: farnz
Differential Revision: D21618518
fbshipit-source-id: 84978778034df11a9a48452adf9269db2dc17145
Summary:
Currently we record them only during pushrebase. Let's record during push as
well.
To simplify things a little bit let's allow only a very simple push case:
1) Single bookmark.
2) All pushed commits should be reachable by this bookmark.
Reviewed By: krallin
Differential Revision: D21451337
fbshipit-source-id: bf2f1e6025ac116fb8096824b7c4c6440d073874
Summary:
This is the fix that might be hard to explain.
First, why we need `generate_additional_bonsai_file_changes()` function at all.
TL;DR is to fix pushrebasing of a diamond merge commit. Merge bonsai changeset
must have a file change entry if p1 and p2 has the same file with different
content, and when merge commit is pushrebased we need to preserve this property.
more details are in D18065013 and in the comments.
Secondly, what was wrong with generate_additional_bonsai_file_changes. While it
was technically correct (i.e. it was producing valid bonsai changeset) it had
an unexpected side-effect for non-diamond merges (e.g. repo imports). It was
adding all changed files between root and onto commit (i.e. all changed files
from commits over which it was rebased). This in turn leads to unnecessary
push failures with "conflicts" errors.
Thirdly, why it was wrong. generate_additional_bonsai_file_changes
a) finds all files changed between onto and root (see comments for more details about onto
and root),
b)then finds those that needs to be added in the pushrebase merge bonsai changeset.
The problem was in step b). It needed to check which files exists in merge
parent but instead it checked which files exists in merge commit itself.
This is because merge commit itself cannot change any of the files that were changed between root
and onto - in that case we get a pushrebase conflict. So all the files come
either from root commit or from another parents. And the entry can become
"stale" only if it existed in one of the parents and later was changed between
root and onto.
Reviewed By: krallin
Differential Revision: D21596758
fbshipit-source-id: f2bea36fcc29fc736caaa7494696a1f6dc848b9e
Summary: Add MPathHash memoization to WrappedPath as the paths are hashed multiple times
Reviewed By: farnz
Differential Revision: D21613720
fbshipit-source-id: b850cf5ea1668b5ff75b07a0489b54f86078677a
Summary:
Currently, there isn't a way to override the value of a tunable during
a unit test. This is unfortunate and may push people to creating integration
tests when unit tests would be best.
Add `with_tunables`, which allows callers to override the tunables values that
a closure accesses. Further, add an example unit test that shows the behaviour.
Thanks to krallin for suggesting this and helping with the implementation
details.
Reviewed By: krallin
Differential Revision: D21595220
fbshipit-source-id: c31da88fb4fcfd38f761960ffe8025132025e6c9
Summary:
Let's add an option to log how many files and trees were fetched in a
particular repo that start with a prefix.
Reviewed By: farnz
Differential Revision: D21617347
fbshipit-source-id: a57f74eadf32781e6c024e18da252c98af21996d
Summary:
Just as for ints and bools, let's add support for strings.
A few notes:
1) we don't have AtomicString, so I decided to use ArcSwap<String>.
However to make code generation simpler (see tunables-derive) I've added a type
alias for that
2) Return type is Arc<String> to avoid unnecessary copies.Another option that we might have here is to put the whole Tunables structure inside ArcSwap, and changing `tunables()` to return Arc<MononookeTunables>
instead of &'static MononokeTunables. However that's a bigger change to make.
Reviewed By: markbt
Differential Revision: D21592390
fbshipit-source-id: 6d3cf340b13f7aef9adb2b1b99ed2bf260033285
