Summary:
Right now, we have zero visibility HTTP level errors (such as broken pipes),
because this is all set up on Gotham with no logging. This diff moves the
bootstrapping from Gotham to us to fix this.
There's a bit of code I'd like to deduplicate here, but it's tricky to do given
that the code is a little different in the HTTP vs. HTTPS branches. For now,
this will give us some logging we need without too much effort. We can make it
more robust (and route it to Scuba or give it session IDs) if this proves
useful.
Reviewed By: StanislavGlebik
Differential Revision: D25851426
fbshipit-source-id: 4ca5d1ecb3931715f04af735aa1b7cfdac87846d
Summary:
This updates Gotham. Under the hood I rebased our fork, you can see the diff
here: P161171514.
The stuff that is relevant is that Gotham got rid of its dependency on
`failure` and now uses `anyhow` instead, and I also added a little bit to our
existing socket data patch by making a few things public so that we can get
access to a few more internals.
Reviewed By: StanislavGlebik
Differential Revision: D25850262
fbshipit-source-id: 25ebf5d63be39e3e93208705d91abc5c61c90453
Summary:
commit cloud commands shouldn't fail if the set is empty
the bug was introduced by using the function bookmarks.selectivepullbookmarknames that has a check at the end that the set is not empty, commit cloud doesn't need the check
the issue introduced D25802193 (713faa71cd) via code refactoring
Differential Revision: D25853480
fbshipit-source-id: 7d3f057dead097b86269e7b03d78f5523e8f1ec7
Summary:
Depending on the thrift defition, `thrift_library` targets may also depend on `ref-cast`.
Add this to the `Cargo.toml`.
Reviewed By: lukaspiatkowski
Differential Revision: D25636872
fbshipit-source-id: 8263395db2bb31127528f5c66c4cc5dd9180d89f
Summary:
This diff adds a debug command that allows inserting different kinds of mapping
entries:
1) Rewritten, meaning that source repo commit rewrites into a target repo
commit
2) Equivalent working copy, meaning that source repo commit doesn't rewrite
into a target repo commit, but one of its ancestors does
3) NotSyncCandidate, meaning that large repo commit shouldn't be remapped into
a small repo
Reviewed By: ahornby
Differential Revision: D25844996
fbshipit-source-id: 1ba64540cf511da8cc50c80a5bee822a950707be
Summary: Let's be a bit more consistent and use ARG_VERSION_NAME
Reviewed By: krallin
Differential Revision: D25844995
fbshipit-source-id: c09be5a38ef97bc491b324f49a2c7d0b6a47212e
Summary:
Rough progress reporting. The progress bars are straight coming from the
`indicatif` crate. Integrating with the `IO` object is not trivial because
we only have a reference. It gets tricky. I think that it makes sense for
us to expand the IO object to something that is more than a `Box<dyn Write>`.
We have about 3 scenarios:
1. Write object that we need to implement interior mutability for to give out
clones.
2. Stdin/Stdout which have their own imlementation for interior mutability.
3. PyObject which has locking already implemented.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D25840469
fbshipit-source-id: 87f466f06f2c5d4c63ccb3bbc5c009fae41ed002
Summary:
Right now we don't log handler errors to Scuba. This can make debugging a
little tricky: if a client is sending an invalid request, we'll see that we
sent them a 400, but we won't know what was invalid about the request.
This diff updates our logging to actually log that.
Reviewed By: sfilipco
Differential Revision: D25826522
fbshipit-source-id: 89486014e0eeaac5c9b149224601db54a26080d9
Summary:
There is a little bug here. We produce a stream of futures of futures, then we
buffer it, which gives us a stream of futures, and then we await the futures
one by one, here:
```
while let Some(next) = stream.next().await {
next.await?
}
```
This is not really correct, because it means we don't actually do fetches
concurrently at all (we just instantiate futures concurrently, but that's not
really async work).
This fixes that by removing one layer of future-ing.
Reviewed By: singhsrb
Differential Revision: D25825895
fbshipit-source-id: 3ad3367f1eb802ce5b9b5288f04fd3705e172537
Summary:
Cross repo sync has two tables to store mapping between commits (this is
something we should probably change, but this is what we have right now).
"map" subcommand was a bit useless because it searched only in a single table,
so for lots of commits it would just return empty response.
Let's instead return CommitSyncOutcome which gives more useful information
Reviewed By: krallin
Differential Revision: D25823629
fbshipit-source-id: afc14f48b6c30bec3714dc9b79cfc4a7d67e38a9
Summary:
This allows adding progress bars tracking downloads from the server.
We could be smarter in this instance if we were to deserialize on the fly.
The first part of the payload contains the number of idmap entries that we need
but it needs more work to make it clear. The progress object right now is
designed for general bytes.
Reviewed By: quark-zju
Differential Revision: D25840470
fbshipit-source-id: c466c8d606b44981fe63c95352db2d8f14d6071b
Summary:
Unbundlereplay command was not implemented in the mononoke but it is used by sync job. So let's add this command here
together with additional integration test for sync between 2 mononoke repos. In addition I'm adding non fast forward bookmark movements by specifying key to sync job.
Reviewed By: StanislavGlebik
Differential Revision: D25803375
fbshipit-source-id: 6be9e8bfed8976d47045bc425c8c796fb0dff064
Summary:
The segmented changelog tailer is going to run with multiple instances which
may race to update the database. This change adds a test that checks that
concurrent updates keep the IdMap correct.
Reviewed By: ahornby
Differential Revision: D25684783
fbshipit-source-id: a09f6e6c915bde38158d9737dcfdc7adc3f15cb7
Summary:
The most common scenario where we see matcher errors is when we iterate through
a manifest and the user sends SIGTERM to the process. The matcher may be both
Rust and Python code. The Python code handles the interrupt and prevents future
function calls. The iterating Rust code will continue to call matcher functions
through this time so we get matcher errors from the terminated Python stack.
As long as we have Python matcher code, errors are valid.
It is unclear to me whether the matcher trait should have `Result` return
values when all implementations are Rust. It is easy to imagine implementations
that can fail in different circumstances but the ones that we use if we just
port the Python code wouldn't fail.
All in all, I think that this is a reasonable step forward.
Reviewed By: quark-zju
Differential Revision: D25697099
fbshipit-source-id: f61c80bd0a8caa58040a447ed02d48a1ae84ad60
Summary: These globs were lost as part of D25315954 (ec0b533381).
Reviewed By: quark-zju
Differential Revision: D25814934
fbshipit-source-id: b1896893e37e355a73eb136758f8966666e0ec05
Summary:
On Windows, it's possible that not all files could be removed from the
repository due to some other process holding a reference to it. When that
happens the `edenfsctl rm` operation will fail. Sometimes, instead of failing
with the actual reason for why the removal failed, it throws a cryptic "list
index out of range" error.
The reason was that when the file that can't be removed is actually a
directory, the `errors` list would be empty. Since filtering the folders is a
bit silly, let's not do it.
Reviewed By: fanzeyi
Differential Revision: D25836412
fbshipit-source-id: 36f936ff9d7697dfd2f4c68d4e56bdb18b66b06a
Summary:
Fixed `README.md` so commands in it work now.
Fixed integration_runner.
Reviewed By: lukaspiatkowski
Differential Revision: D25823461
fbshipit-source-id: 0d6784758c9f86bca38beafe014af4766169bee3
Summary:
This unbreaks the test. The reversefiller need access to SMC to talk to
scmquery (we could set up our own scmquery instance but I don't think it's worh
it).
Reviewed By: krallin
Differential Revision: D25824395
fbshipit-source-id: 676b3ac1e3af95e8e02bd272f7cb25250e047eed
Summary:
Sometimes we want to rechunk just a few file contents, this diff makes it
possible to do so.
Reviewed By: ahornby
Differential Revision: D25804144
fbshipit-source-id: 6ce69f7cee8616a872531bdf5a48746dd401442d
Summary: There is only one implementation of the trait so remove it and use that impl directly. Removing the trait makes it simpler to work on bulkops in the rest of this stack.
Reviewed By: farnz
Differential Revision: D25804021
fbshipit-source-id: 22fe797cf87656932d383ae236f2f867e788a832
Summary:
Unless we can't update to a public root, there is nothing wrong with having local changes and switching workspaces feature.
Those are not related. Uncommited changes shouldn't impact switching workspaces.
Reviewed By: mitrandir77
Differential Revision: D25802406
fbshipit-source-id: 3fcb70864002bed11ad32621947294f643ca1fc3
Summary:
Right now we get zero logs from the blobstore healer, which is pretty annoying
because it makes it impossible to really tell what it's doing.
This fixes that.
Reviewed By: HarveyHunt
Differential Revision: D25823800
fbshipit-source-id: ded420753ba809626d6e4291eb3d900dcfbff3d1