Summary:
Originally, we wanted to log EdenAPI batch sizes to Scuba in order to A/B test the effect of batch size on fetching speed. It turns out that the original arbitrary choice of 1000 works pretty well, so there isn't really a need to log this to Scuba anymore.
In addition to removing these fields, this diff moves the "http_enabled" column to be logged from the config by the wrapper instead of being manually logged by hg.
Reviewed By: quark-zju
Differential Revision: D20568104
fbshipit-source-id: 1e113489d005e93e6283f7e4a439b1c78b6f0fb9
Summary:
Fix code in `memcommit/serialization.py` that was assigning a `str` to a
`bytes` variable.
Reviewed By: singhsrb
Differential Revision: D20564418
fbshipit-source-id: 578ebe9d2b8823353d5a4727f6ea888678ca881f
Summary: the exception object is unused in each of these catch blocks.
Reviewed By: chadaustin
Differential Revision: D20562251
fbshipit-source-id: 12502429e47f5603b73cfc88b10dda0db5daeb93
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary:
the type of `mode_t` that we were using for this on Windows
recently changed in the folly portability layer, resulting in a padding/alignment
problem that breaks our assumption at the size of this struct.
I don't believe that we were using bitfields correctly here; in my experience
all of the members must have the same type in order to get packed in the
struct correctly, because the standard doesn't define much in the way
of bitfield behavior beyond the syntax.
This commit uses an unsigned integer type for all of the bitfields
and takes a stab at using a cast when returning the actual mode
member for macOS. This may not be right; I'm just throwing up this
diff first in order to be able to get it on a mac machine.
Reviewed By: simpkins
Differential Revision: D20529006
fbshipit-source-id: e9362bf263fab03d51cc8aa97f94f00806650618
Summary:
There isn't really much to annotate here, but this lets us eliminate a couple
`pyre-fixme` comments about not being able to find these modules during type
checkign.
Reviewed By: singhsrb
Differential Revision: D20550267
fbshipit-source-id: 271f8406890787c0613294a9047365fdebcdeda1
Summary:
These were copied into the `edenscmnative` directory in D15798642, but never
removed from their original location.
Reviewed By: singhsrb
Differential Revision: D20550268
fbshipit-source-id: dadce279f1717a66ece86376bd51179b8669cc8a
Summary: The signatures were used by the linter to warn if the files require regenerating, since the linter now regenerates the files regardless of the signature it is no longer needed to sign the files.
Reviewed By: krallin
Differential Revision: D20467745
fbshipit-source-id: aff2643f80939d5693e7a30abf07484c9060796f
Summary:
OpenNSA has prebuilt SDK but kernel modules have to be built from sources. Add
a utility script to build and extend packaging script.
In future, we can consider folding this build into fbcode_builder itself.
Differential Revision: D20549883
fbshipit-source-id: f9475b7e0223e9f357117d7d7d27df8904fa1d73
Summary:
This fixes handling of Ctrl-C when using `edenfsctl start --foreground` or
`edenfsctl start --gdb`.
The changes in D19861810 to make `exec_daemon()` fork and exec instead of
simply calling exec accidentally broke this. While it attempted to forward
SIGINT to the child process, in practice this would normally fail in
development environments. In development environments EdenFS is normally
started using `sudo`, and the CLI does not have permissions to send signals to
this child process since it is running with elevated privileges. As a result
on Ctrl-C the foreground `edenfsctl` process would die since it did not handle
the error attempting to forward the signal, and the actual EdenFS process
would continue running.
The main goal of D19861810 was to allow telemetry logging of success or
failure on restart. This changes the `edenfsctl restart` code to use the
existing `start_daemon()` function, and reverts the old behavior of
`exec_daemon()` to use `execve()` to replace the current process.
Reviewed By: genevievehelsel
Differential Revision: D20355252
fbshipit-source-id: 5e61f70e8b411ee023eee6fe06e0cd641f732631
Summary:
Fix `EdenFS.shutdown()` to call `edenfsctl stop` with a timeout of 0 seconds,
telling it not to wait for EdenFS to exit. This code then performs its own
wait with a timeout.
Previously the code called `edenfsctl stop` asking it to wait for EdenFS to
exit with a 30 second timeout. However, since the integration test could be
the immediate parent process of EdenFS the `edenfs` process may not actually
go away until the test called `wait()` on this process, which wouldn't happen
until `edenfsctl stop` returned. This only caused problems for cases where
the test could run `edenfs` directly without needing to run it through `sudo`:
when run through `sudo` the edenfs process would get cleaned up since `sudo`
was the immediate parent and it would wait on the process.
Reviewed By: genevievehelsel
Differential Revision: D20434081
fbshipit-source-id: 513fd2ebb5fc24a54c546a76e94827c81a4ab754
Summary:
The rust-crypto library appears to be unmaintained, switching
- `crypto::digest::Digest` to `digest::Digest`
- `crypto::sha1::Sha1` to `sha1::Sha1`
- `crypto::sha2::Sha256` to `sha2::Sha256`
Reviewed By: jsgf
Differential Revision: D20456962
fbshipit-source-id: 2e3406dedba05245265d96b480c35ba2421aa3fd
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
The rust-crypto library appears to be unmaintained, switching
- `crypto::digest::Digest` to `digest::Digest`
- `crypto::sha1::Sha1` to `sha1::Sha1`
- `crypto::sha2::Sha256` to `sha2::Sha256`
Reviewed By: jsgf
Differential Revision: D20456840
fbshipit-source-id: 90cc031ec5402b60b6eb06a301a3733bd92bbc69
Summary: For LFS blobs, these can be obtained very easily by querying the ContentStore.
Reviewed By: DurhamG
Differential Revision: D20504235
fbshipit-source-id: 937ef20184d6524b1355565f9ab81e40b56d7ab0
Summary:
This diff asyncifies the `loop_fn` call in `run_statistics`.
I was unable to find an existing example of asyncifying an infinite
loop - my solution requires allowing the `Ok` around my `loop`
to be unreachable via the `#[allow(unreachable_code)]` annotation. There may be
a better solution.
We also swap out the `tokio-timer` dependency, which uses old-style
futures, for current-version `tokio` so we can use the new-style future
`tokio::time::delay_for`.
Reviewed By: farnz
Differential Revision: D20527530
fbshipit-source-id: 90d30ec9465402d06d3b4b30c1bbd5e340ac94b6
Summary:
This is only intended for Mercurial .t tests and not in any production
environment.
Reviewed By: DurhamG
Differential Revision: D20504236
fbshipit-source-id: 618e17631b73afa650875cb7217ba7c55fb9f737
Summary:
This will enables the fast-path for comparing LFS blobs without reading the
entire blob.
Reviewed By: DurhamG
Differential Revision: D20504233
fbshipit-source-id: 446cec57fba77e02cc7070203bd759d341fc01ab
Summary:
For now, this is only used for LFS, as this is the only store that can
correctly answer both.
This API will be exposed to Python to be able to have cheap filectx comparison,
and other use cases.
Reviewed By: DurhamG
Differential Revision: D20504234
fbshipit-source-id: 0edb912ce479eb469d679b7df39ba80fceef05f2
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
Summary: Most of our keys are prefixed, let's prefix this as well
Differential Revision: D20536231
fbshipit-source-id: 9a2ffecfc7de46d109a9fba2444212735cacbebf
Summary:
Changeset info is less expensive to load than Bonsai, so we would like to use it in SCS as a source of commit info if possible.
This diff adds a method into the Repo object that checks `changeset_info` derivation is enabled for the repo in the `DerivedDataConfig`. If derivation is enabled, then SCS derives this info otherwise it awaits for bonsai and converts it into the changeset info. The bonsai fields aren't copied but moved to the `ChangesetInfo`.
Reviewed By: StanislavGlebik
Differential Revision: D20282403
fbshipit-source-id: b8ddad50dcd5c6de109728b2081ca5a13f440988
Summary:
This diff asyncifies the outermost layer of `statistics_collector`,
so that `main` doesn't need a `compat`, by extracting the
futures portion of `main` into an async function `run_statistics`
and using async futures for the outermost layers of `run_statistics`
logic (everything outside the `loop_fn` call)
Reviewed By: farnz
Differential Revision: D20527529
fbshipit-source-id: 00ad9033584360f45715719f2636dcfac1926004
Summary:
This is the start of migrating blackbox events to tracing events. The
motivation is to have a single data source for log processing (for simplicity)
and the tracing data seems a better fit, since it can represent a tree of
spans, instead of just a flat list. Eventually blackbox might be mostly
a wrapper for tracing data, with some minimal support for logging some indexed
events.
Reviewed By: DurhamG
Differential Revision: D19797710
fbshipit-source-id: 034f17fb5552242b60e759559a202fd26061f1f1
Summary:
The `all()` revset is much slower with narrow-heads for correctness. Use an
alternative that is fast.
Reviewed By: markbt
Differential Revision: D20528063
fbshipit-source-id: c8ae35e67e60407406ca81d67878278392626e9a
Summary: A little step towards asyncifying the filestore. This is just mechanical, without removing clones. TBD: add a diff, which starts to actually use the benefits of new futures.
Reviewed By: farnz
Differential Revision: D20534272
fbshipit-source-id: a038e6f22b666f3f2c9782ee25c0c2582ddced6c
Summary:
Last diff. Fully migrates all of populate_healer.rs to async/await
futures. This makes `put_resume_state()` async, with one `.compat()` call needed
for dealing with `manifold.put()`. Also changes `populate_healer_queue()` to
use the new async `put_resume_state()`. At this point, the only `.compat()`
calls remaining are for interop with ThrifManifoldBlob's interface, and can be
removed once ThriftManifoldBlob is updated or provides async replacement
functions. All explicit old-style future creation sites have been removed in
favor of 0.3 futures.
Reviewed By: krallin
Differential Revision: D20479264
fbshipit-source-id: baad535da3fc8b621d72de567454bcd64862977a
Summary: Moves to using 0.3 futures inside of the populate_healer_queue() function. This leaves only one remaining source of `.compat()` calls inside of populate_healer.rs, which will be removed in the following diff.
Reviewed By: krallin
Differential Revision: D20473834
fbshipit-source-id: 6d76e0673b875fba15611a495d86b9ca0b1695db
Summary:
Run buck build -c rust.clippy=true eden/mononoke/:mononoke#check and fix some
of them manually. I wasn't able to make rustfix to work - will try to see
what's wrong and run it.
The suggestions looks non-controversial
Reviewed By: krallin
Differential Revision: D20520123
fbshipit-source-id: 25d4eb493f2363c5aa77bdb3876da4378483f6cb
Summary: This makes thigs a little more readable.
Reviewed By: krallin
Differential Revision: D20515645
fbshipit-source-id: ae04e18b0f415353431a995ae22844f6e301780c
Summary:
This is going to be used in D20469131, but in a nutshell the idea is to
perform as many checks as possible before actually doing the rechunking.
This way we can avoid churning through the entire blobstore.
Reviewed By: krallin
Differential Revision: D20491189
fbshipit-source-id: 4f7c2a8e02c890db789d25aa819b5c91d08ea7be
Summary:
The way decoders work in Tokio is that they get repeatedly presented whatever
is on the wire right now, and they have to report whether the data being
presented is valid and they'd like to consume it (and otherwise expect Tokio to
provide more data).
It follows that decoders have to be pretty fast, because they will be presented
a bunch of data a bunch of times. Unfortunately, it turns out our SSH Protocol
decoder is everything but.
This hadn't really been a problem until now, because we had ad-hoc decoding for
things like Getpack that might have a large number of parameters, but for now
the designated nodes implementation is decoded in one go through the existing
Gettreepack decoder, so it is important ot make the parsing fast (not to
mention, right now, we buffer the entire request for Getpack as well ... so
maybe we could actually update it to this too!).
Unfortunately, as I mentioned, right now the parsing wasn't fast. The reason is
because it copies parameters to a `Vec<u8>` while it decodes them. So, if
you start decoding and copying, say, 50MB of arguments, before you find out
you're missing a few more bytes, then you just copied 50MB that you need to
throw away.
Unfortunately, the buffer size is 8KiB, so if we say "I need more data", we get
8KiB. That means that if we want to decode a 70MiB request, we're going to make
8960 ( = 70 * 1024 / 8) copies of the data (the first 8KiB, then the first 16,
and so on), which effectively means we are going to copy and throw away ~612GiB
of data (8960 * 70 / 2). That's a lot of work, and indeed it is slow.
Fortunately, our implementation is really close to doing the right thing. Since
everything is length delimited, we can parse pretty quick if we don't make
copies: all we need to do is read the first length, skip ahead, read the second
length, and so on.
This is what this patch does: it extracts the parsing into something that
operates over slices. Then, **assuming the parsing is successful** (and that is
the operative part here), it does the conversion to an owned Vec<u8>.
In O(X) terms .. this means the old parsing is O(N^2) and the new one is O(N).
I actually think we could take this one step further and do the conversion even
later (once we want to start decoding), but for now this is more than fast
enough.
After this patch, it takes < 1 second to parse a 70MiB Gettreepack request.
Before this patch, it took over 2 minutes (which is 3 times longer than it
takes to actually service it).
PS: While in there, I also moved the `gettreepack_directories` function to a
place that makes more sense, which I had introduced earlier in the wrong place
(`parse_star`, `parse_kv` and `params` are a group of things that go together,
it's a bit clowny to have `gettreepack_directories` in the middle of them!).
Reviewed By: kulshrax
Differential Revision: D20517072
fbshipit-source-id: 85b10e82768bf14530a1ddadff8f61a28fdcbcbe
Summary: The `Node` type in Mercurial's Rust code was renamed to `HgId`, with an alias to `Node` to keep older code building. Let's rename the usages in Mononoke to `HgId` to reduce ambiguity and keep the terminology consistent with Mercurial.
Reviewed By: StanislavGlebik
Differential Revision: D20460543
fbshipit-source-id: f6d8e3aef42743370323cde79ec10b21de956313
Summary:
It was (or rather, might have been) useful during debugging of S197766.
Let's now count both "count" (i.e. how often the method was called)
and count how many filenodes were inserted
Reviewed By: krallin
Differential Revision: D20519701
fbshipit-source-id: f19f413171fcbcc300deffbe29baa946ebbe8dce
