Summary:
This is nice when running locally to know when we're done with setting up
cachelib and such.
Reviewed By: farnz
Differential Revision: D17569829
fbshipit-source-id: adfe5944991c8842a459df8606d9d81c2dcd02de
Summary:
After this diff new sync job (the one that generates bundles instead of reusing
existing) should be feature complete i.e. it should support the same pushes
as normal sync job.
One noteable refactoring is adding BookmarkChange enum. Previously we were passing and converting `from_` and `to_` parameters down the stack and also doing conversion in the middle, and it was easy to mix them up and use `from_hg_cs_id` instead of `to_hg_cs_id` and vice versa. My hope is that with this enum isolates most of the `from_` and `to_` shenanigans in one places so that's it's easier to keep track of it.
Reviewed By: krallin
Differential Revision: D17546808
fbshipit-source-id: 1b5e2553f80b33996ec361e89b01cb55ec811ec7
Summary:
After this diff sync job [1] starts to keep track of bookmark on hg server.
There are two reasons for that:
1) Some pushes just move a bookmark i.e. they don't push any new commits.
However without knowing hg server state it's impossible to tell apart normal push vs
bookmark-only push and generate correct bundles for it (note that the actual
support for bookmark-only pushes will be added in the next diffs, this diffs
just tracks hg sever bookmarks).
2) There are force pushes that move a bookmark to a completely new place AND
push new commits. Without knowing the state of hg server the sync job might try
to push too many commits.
To track the bookmarks they are fetched from hg server on sync job start up,
and later updated and the sync job progresses. To fetch bookmark from hg server
a separate listserverbookmarks extension is used.
If bundles are prepared together (see --bundle-prefetch option) then we also need to have bookmark overlay
i.e. bookmarks that were modified by the previous bundles in the same batch.
Note that combine-bundles is removed since it hasn't been used at all, include tw specs.
[1] It only keeps track if the job needs to regenerate bundles
Reviewed By: krallin
Differential Revision: D17525983
fbshipit-source-id: 528949ad4ad57ae51ad68fced9caf7256a057ba3
Summary: This is the bare minimum to ask the remapping table for any values it has. Combined with other admin tool commands, you can use this to track down a synced commit.
Reviewed By: StanislavGlebik
Differential Revision: D17554261
fbshipit-source-id: f5f967a55182b614b68b5a8c1421401921c9a268
Summary: Modified statistics_collector tool that now it saves statistics to Scuba
Reviewed By: HarveyHunt
Differential Revision: D17570918
fbshipit-source-id: b22094dcc915bc9ba9cb76c09f73e9590c65e641
Summary: Implement the `file_info` method, which returns information about a file.
Reviewed By: krallin
Differential Revision: D17547874
fbshipit-source-id: 89c360d043601beb9e6e55b8bc28cd88eb528ea7
Summary:
All source control service methods will need to get contexts and could create
errors if the requested objects don't exist. Refactor this code out so that it
can be re-used in each method.
Reviewed By: krallin
Differential Revision: D17547879
fbshipit-source-id: 16c6206ad0fae2adc555f8de8b840ced33dd849f
Summary: Implement the `file_exists` method, which returns true if the file exists.
Reviewed By: krallin
Differential Revision: D17547876
fbshipit-source-id: 55dbc3472a7f06a1878ebd409ed8b5d56fda3033
Summary: Implement the `tree_list` method, which lists all the subentries in a tree.
Reviewed By: krallin
Differential Revision: D17547878
fbshipit-source-id: 12d296cad69fdb90be7bcaf4cedc9ba82cda3f2f
Summary:
Implement the `commit_path_info` method, which returns entry information
describing the file or tree at a particular path in a commit.
Reviewed By: krallin
Differential Revision: D17547877
fbshipit-source-id: 9f7f01ae601ab5b618916caa268e038d7240bf21
Summary:
We will need to convert between the thrift request and response types and the
Mononoke API types in all of the functions. Add traits to support this.
We can't use `std::convert::From` for this, as we need to
convert between types that are both defined in other crates. Instead, define
`FromRequest` and `IntoResponse` which translate from request and to response
types accordingly.
Extend `thrift::CommitId` to provide useful methods on `thrift::CommitId`.
Reviewed By: krallin
Differential Revision: D17547875
fbshipit-source-id: 2f89802f5c4aa2df280bac699934abe23e94e4a6
Summary:
Add representations of files to the Mononoke API.
Like trees, files can exist either at paths within commits (which are
represented by a `ChangesetPathContext`, and may also refer to trees), or on
their own (which are represented by a `FileContext` and uniquely identified by
their content id, or one of their content hashes). We keep these separate, as
history information is only relevant for `ChangesetPathContext`, and
`FileContext` may be more useful for clients that don't care about history.
Reviewed By: krallin
Differential Revision: D17547880
fbshipit-source-id: 4d2fdb27f4e9ebca98a9f7e902af064468a27e71
Summary:
Add representations of trees to the Mononoke API.
Trees can exist either at paths within commits (which are represented by a
`ChangesetPathContext`, and may also refer to files), or on their own (which
are represented by a `TreeContext` and uniquely identified using their fsnode
id). We keep these separate, as history information is only relevant for
`ChangesetPathContext`, and `TreeContext` may be more useful for clients that
don't care about history.
Reviewed By: krallin
Differential Revision: D17547881
fbshipit-source-id: be1b742303be84b816f632f1f1185f6fd1969d9e
Summary:
Use the filestore to upload file contents in fixtures, so that content aliases
are correctly generated.
Reviewed By: farnz
Differential Revision: D17552582
fbshipit-source-id: e1b3030fb33c8646314bb1777ec1fb49af8d3b78
Summary:
This updates the lfs_server to publish client_hostname to Scuba. It also
unifies the various mechanisms we had to get data about the client into a
single piece of middleware, which ensures we get a consistent view of e.g.
client IPs in Scuba and stderr lgoging.
Reviewed By: HarveyHunt
Differential Revision: D17570749
fbshipit-source-id: e5b62abf440d5d09e78c1f51632444200768126c
Summary: Add `fsnodes` as a kind of derived data that can be backfilled.
Reviewed By: krallin
Differential Revision: D17549835
fbshipit-source-id: 14d7540567c0bd0eae4f53944a5c4c19e5902105
Summary:
Speed up derivation of fsnodes by prefetching all the metadata we will need
before deriving the fsnodes for a changeset.
Reviewed By: krallin
Differential Revision: D17549836
fbshipit-source-id: 9c66621b75160db80ea706a200b33703916d7d15
Summary: I added get_repo_name method, which will be useful while logging statistics to scuba in statistics_collector tool
Reviewed By: HarveyHunt
Differential Revision: D17547629
fbshipit-source-id: 481ce2fc498dbc8834da495c43b530928ff367e1
Summary:
This is currently a bit hacky, as bonsai->hg conversion does not work like other derived data, but it does the job.
We want to be able to dump Bonsai changesets from synced commits onto a bookmark, and have a tailer fill in Mercurial changesets behind us. This code does that via the existing mechanism for creating changesets
Reviewed By: StanislavGlebik
Differential Revision: D17552077
fbshipit-source-id: d5b13f884332e57e7d8c55256ccb4ea918fe4921
Summary: I modified tool that now it calculates statistics starting from current master (or other provided by us bookmark), then it waits for new commits and updates statistics.
Reviewed By: StanislavGlebik
Differential Revision: D17502571
fbshipit-source-id: c98b06529cc7fc6ab9ee358b87dc9fa161546da5
Summary: Remove use of try_from crate now that it's stable.
Reviewed By: farnz
Differential Revision: D17550530
fbshipit-source-id: 1537af929fb294a725a44003fa5c34620a4cba3b
Summary:
When I made the LFS server not wait for upstream if it can reply immediately, I
also made the tests for its proxy functionality a bit racy, since it no longer
always connects to upstream immediately. This fixes that.
While I'm in there, I also:
- Added an integration test for the "skip upstream" functionality.
- Made our stderr logging log empty responses.
- Made our invalid URI error reporting a little more comprehensive.
Reviewed By: HarveyHunt
Differential Revision: D17547889
fbshipit-source-id: 47f150136ef91a7f6334bb09f95782357b72f01a
Summary:
Previously it was possible to sync just a single commit - this diff adds
support for generating bundle with stacks of commits.
It uses DifferenceOfUnionsOfAncestorsNodeStream which is exactly the same
stream we use in Mononoke to serve `hg pull` requests.
Reviewed By: krallin
Differential Revision: D17498873
fbshipit-source-id: d94066ad0f0cbdd44fd958c57739de1e070eec9f
Summary:
This is the very first and very limited implementation of a new mode of the
sync job. This is the mode where bundles are generated by the sync job instead
of reusing the bundles pushed by the client.
At the moment it syncs only a single commit, new functionality has to be added
later.
A few notes:
1) We need to manually generate replycaps part . Mononoke essentially ignores it
(with a few small exceptions), however Mercurial actually pays attention to it, and
generates different response.
At the moment reply capabilities are hard-coded in the sync job, and that might be
problematic if replycaps change by Mercurial client but sync job hasn't been
updated.
There are a few protections from it - we can make replycaps dynamic (for
example, specify it in the tw spec) and we can also rely on integration tests
to catch regressions.
2) create_filenode_entries_stream has a complicated logic for renames - please
have a look at it to double check my understanding!
Reviewed By: krallin
Differential Revision: D17477983
fbshipit-source-id: fdb4584e768bdc5637868468a035c81f1584f7fe
Summary:
Previously, the RequestContext stored a string representing the method
type. Replace this with an enum, so that matching on it is easier - allowing
for different histogram configs for each method type.
For the batch endpoint, use a histogram of durations from 0 -> 500ms.
Reviewed By: farnz
Differential Revision: D17530495
fbshipit-source-id: 23ff424fc0aca5c1022f9d6521bbd70d0af3ccf6
Summary: We've improved the MySQL version in review - make the SQLite version match, to avoid nasty test/prod issues
Reviewed By: StanislavGlebik
Differential Revision: D17528682
fbshipit-source-id: 498467210608cce0ec27d014d64a22115355cda2
Summary: Let's have all concerned parties use the same type under the same name.
Reviewed By: farnz
Differential Revision: D17481135
fbshipit-source-id: d0a99cce19fa0941d351f5754ed4517aea6a6105
Summary:
If we have all the objects a client is asking for, we might as well answer
immediately with them, rather than wait for upstream.
This effectively means that this LFS server can be faster than its upstream on
batch downloads (for uploads, we still have to check in with upstream), as it's
no longer limited by upstream's performance (or errors!) when responding to
batch requests.
Reviewed By: HarveyHunt
Differential Revision: D17526491
fbshipit-source-id: 5d008011fd5ac444c84569f8235ecc9bde6cef9c
Summary: This adds the request_id to our request slog logger. This is nicer because it lets us tie logs for a given request together.
Reviewed By: HarveyHunt
Differential Revision: D17526490
fbshipit-source-id: c6a57828a097f96b0e752373b98a1ea834009f95
Summary: We no longer have any middleware that requires to be able to capture variables from the "inbound" phase (i.e. prior to handling the request) to the "outbound" phase (i.e. once the response is ready). Instead, we're passing everything through the State. So, let's get rid of the dynamism we don't need.
Reviewed By: HarveyHunt
Differential Revision: D17503373
fbshipit-source-id: 569d180250821aa3707245133a223b1f4efba3b6
Summary:
Middleware executes when a response is ready, but since responses can contain a stream, that might not be the full story if we're e.g. downloading a big file.
This diff updates our middleware to introduce a post-send context that lets us conditionally dispatch those actions after we've finished sending the response to the client.
Reviewed By: HarveyHunt
Differential Revision: D17503374
fbshipit-source-id: 4dc97c0057d6e1705d116cbc1d283fc73de213ef
Summary:
This reworks our middleware to not use the Gotham router for middleware, and
instead creates our own service to wrap the one provided by the Gotham Router.
The upshot of this is that we can then run our own middleware even if the
Gotham Router finds a 404 or 405 (in which case it doesn't normally run any
middleware).
This also simplifies our middleware a little bit, since our middleware isn't
modifying responses at all, but Gotham middleware allows that (we can still run
response transformation in Gotham).
As part of this change, I've also cleaned up our LoggingContext (which didn't
really belong in `lfs_server_context`), and moved our router implementation in
its own module.
Reviewed By: HarveyHunt
Differential Revision: D17500363
fbshipit-source-id: 9b0d7449f707f158d9f5433e2953d270b3446c8f
Summary:
This adds a little more headers to make it easier for clients to identify which
server they are talking to.
Reviewed By: HarveyHunt
Differential Revision: D17498981
fbshipit-source-id: 758562f0ed631dc6f54eb567200e3bf1af04b078
Summary:
This function finds entries that were "introduced" by this manifest i.e.
entries that are present in this manifest but not present in any of
`diff_against` manifests.
This function was used in fastlog, and I'm planning to use it in the new sync
job, so move it to ManifestOps crate (and rename along the way).
One small change in behaviour is that we removed a single traced...() call. I
talked with aslpavel who introduced it and he's fine with removing it -
unfortunately our traces are not in a great shape yet, so this trace call is
not used at all.
Reviewed By: aslpavel
Differential Revision: D17477055
fbshipit-source-id: cba1a55a64299857efe4f4be7a67b6faf31e7019
Summary: It will be used in the next diffs
Reviewed By: krallin
Differential Revision: D17498798
fbshipit-source-id: c71319a21aa586208871555f2055c81afb021b52
Summary: This is convenient to get the content ID for a file at a given path.
Reviewed By: HarveyHunt
Differential Revision: D17498393
fbshipit-source-id: f40706b9289ca77e99cb3d7070b396ee134f79c3
Summary:
tl;dr Got things working again by ignoring the `--config-path` from rls.
Despite my efforts in D17089540, `rustfmt` appeared to have stopped
working again in VS Code. I thought maybe something changed in our move
to 1.38.0-beta.3. I saw something about a `VersionMismatch` in the
console (which was the issue last time), but unfortunately the error
did not include what the expected and observed versions were, so I
built `rustfmt` from source locally and modified it to include this
information (which I should try to upstream).
Unfortunately, our VS Code/rls setup is not great when it comes to
logging errors to a file, so once again I ran:
```
sudo execsnoop.py -n rust
```
and observed the external `rustfmt` command that `rls` ultimately
constructed and executed so I would have a simpler repro case.
Incidentally, after rls had a `rustfmt` error of some sort, it seemed
to end up in a broken state, so debugging the issue outside of rls
was a lot simpler. Anyway, here was an example of such a command I saw
from `execsnoop`:
```
/data/users/mbolin/eden-fbsource/fbcode/common/rust/tools/common/rustfmt_wrapper.sh --unstable-features --skip-children --emit stdout --quiet --config-path /dev/shm/tmp/i0gVWxrcpp
```
As explained in D17089540, `/dev/shm/tmp/i0gVWxrcpp` ends up being this
fully serialized `rustfmt` configuration. I took this command and repro'd
by making a local formatting error and piping the file in:
```
/data/users/mbolin/eden-fbsource/fbcode/common/rust/tools/common/rustfmt_wrapper.sh --unstable-features --skip-children --emit stdout --quiet --config-path /dev/shm/tmp/i0gVWxrcpp < scm/dotslash/src/main.rs | head
```
and got:
```
Warning for license template file "": No such file or directory (os error 2)
1 282 282
// Copyright 2004-present Facebook. All Rights Reserved.
mod backend;
mod config;
#[cfg(not(build = "instrumentation"))]
mod curl;
mod curl_args;
mod decompress;
mod dotslash_cache;
```
What was that garbage at the top about a license template file? I ended up
looking at `/dev/shm/tmp/i0gVWxrcpp` and found this line:
```
license_template_path = ""
```
Apparently the default value for this configuration option is `""`
and `rustfmt` tries to open the path without bothering to check whether
it is the empty string:
4449250539/src/config/license.rs (L215-L221)
Great. (I filed an issue for this at https://github.com/rust-lang/rustfmt/issues/3802.)
It finally dawned on me that between `required_version` and `license_template_path`,
this generated config file from rls was not doing us any favors. Since we are already
using `rustfmt_wrapper.sh` to rewrite some options that we care about, why not rewrite
`--config-path` to use the `.rustfmt.toml` config in fbcode?
So that's what I did and now everything seems to work again.
Also, I appeared to have figured out why we need a wrapper script in the first place!
rls tries to set the `emit_mode` to `ModifiedLines`, which would generate the output
from rustfmt in the way that rls wants. It looks like someone then made reading from
stdin incompatible with `emit_mode=ModifiedLines`. I filed this as:
https://github.com/rust-lang/rls/issues/1555
Reviewed By: dtolnay
Differential Revision: D17492254
fbshipit-source-id: 3415bdab3c1030d3082ae2b8fab0c2e6b312534a
Summary:
Add a string representing the method to LoggingContext,
which can then be used by the ODS and Scuba middleware.
NOTE: This changes the batch endpoint key from:
mononoke.lfs.request.<repo>.objects
to:
mononoke.lfs.request.<repo>.batch
Reviewed By: krallin
Differential Revision: D17475551
fbshipit-source-id: 8692f165719c9f69bf0d783845ed9d87b8baf86f
Summary:
Rather than using hardcoded path maps and ad-hoc moveres, lets use the
logic which creates moves from config.
Reviewed By: farnz
Differential Revision: D17424678
fbshipit-source-id: 64a0a0b1c7332661408444a6d81f5931ed680c3c
Summary:
This diff achieves two things:
- configs are verified from this point of view when read, not just when
compiled
- we have a way to get config by repoid if we have `RepoConfigs`
Reviewed By: farnz
Differential Revision: D17460214
fbshipit-source-id: cedce280d6f8209c2f4e1a4ff0d53780242bac30
Summary:
I want to use TreepackPartInput in the next diffs, and it's easier to use if we
are passing fullpath here
Reviewed By: krallin
Differential Revision: D17445971
fbshipit-source-id: 936fb544df3d57b5565f900402fc61dceb422bc6
