Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary:
I'm about to add another reference to our service name, so instead of
hardcoding "Eden", add a kServiceName constant. While I'm at it, I
renamed it to EdenFS.
Reviewed By: genevievehelsel
Differential Revision: D20500668
fbshipit-source-id: 4974363156ba2934a1a5cd86d9e8fd7e93d89181
Summary:
This diff - which asyncifies `prefetch_content` - isn't a big win on readabiliy,
but by switching to async (taking references) and then changing upstream functions
to references we're able to save two layers of cloning in the `warmup` functions,
which ought to reduce heap allocations
Reviewed By: farnz
Differential Revision: D20612696
fbshipit-source-id: a52d6246789e964d3b02d0bdc1bfafbded8f25dd
Summary:
Straightforward asyncification, which lets us pass down refs and save
a couple of clones.
Reviewed By: farnz
Differential Revision: D20612725
fbshipit-source-id: 83aa30e2954101faa40d9427ad267e8e42f57d46
Summary:
The api is now asyncified, and we no longer have to clone when calling
`subcommand_tail`.
The inner stream logic is still using the old streams api - untangling that code was difficult for me, so I'll try to tackle it in a later, isolated diff
Reviewed By: farnz
Differential Revision: D20611556
fbshipit-source-id: acd79f8048b5e611c316fc1986160b6395ee6fb9
Summary:
Asyncified `subcommand_backfill`. Some of the inner code is still using
old futures because I was not able to find a new-futures version of
`future_stats::Timed`, but most of the logic is using new futures /
new streams apis.
Reviewed By: farnz
Differential Revision: D20612724
fbshipit-source-id: 00e524f606fc02f4706c02347c0bd6feba46c3eb
Summary:
While failures in the Rust updater aren't expected, at least one valid case
requires requires retrying the operation in Python: old-style LFS pointers.
When these are stored in packfiles/indexedlog, only the Python code knows how
to deal with them, and thus the operation needs to be retried there.
Reviewed By: DurhamG
Differential Revision: D20603709
fbshipit-source-id: 7d24ba573f0ff540906d909f1b4440fd4d3469a6
Summary:
The ContentStore cannot deserialize LFS pointers stored in packfiles, to avoid
potential damage, let's refuse to update LFS blobs. A proper solution will be
built in a separate diff.
Reviewed By: DurhamG
Differential Revision: D20576575
fbshipit-source-id: 4e4ce6a9432157e2ce69881c0079e943ea3f3acd
Summary:
Instead of having the magic number 0x2000 all over the place, let's move the
logic to this method.
Reviewed By: DurhamG
Differential Revision: D20637749
fbshipit-source-id: bf666f8787e37e6d6c58ad8982a5679b7e3e717b
Summary:
On Unix, pretend that NTFS doesn't support symlinks. While this isn't
technically true, NTFS on Linux is only used to alleviate performance issues
with `hg update` on Windows. With the pyworker code, I'm expecting these
performance issues to disappear allowing this code to be removed.
Reviewed By: ikostia
Differential Revision: D20527976
fbshipit-source-id: 4194f4b5af065de2e293b41b9d03e9d4ab6ea006
Summary:
Move them into a VFS struct, a future step will move the VFS code into its own
crate.
Reviewed By: DurhamG
Differential Revision: D20527977
fbshipit-source-id: 3250b05840688db72e1c43c72ec6defbc7f20851
Summary:
In this diff I create a function to handle the logic for choosing a subcommand
to run, returning a Future that we execute. By using `helpers::block_execute`
we're able to extract the spinning up of the fb303 server, which is the key
to making this an async function.
Within each of the subcommands in the new function `choose_subcmd`, this change
also asyncifies the logic inside the `async` block.
Reviewed By: krallin
Differential Revision: D20590742
fbshipit-source-id: a6298e6da3827638cc23369f72ebdd66b5051768
Summary:
This diff moves blame integration tests out from the main `test-scs.t`.
The change makes `test-scs.t` be able to complete and not time out anymore.
Reviewed By: ikostia
Differential Revision: D20629281
fbshipit-source-id: 67ba047442e7216a8addd0945c94d2f932eca08a
Summary: This diff moves lookup integration tests out from the `test-scs.t`.
Reviewed By: krallin
Differential Revision: D20620537
fbshipit-source-id: a8e1020901271b0e66dd4caa43ad3eddbf887a41
Summary:
We'd like to have health checks for scs server, at the very least to block bad
pushes before they reach production.
This diff adds a few counters that are generally useful, but the two we are going to
use for health checks is internal_failure_permille/invalid_request_permille[1]. This is the simplest
healtch check, and and it has a few drawbacks. In particular, it won't "notice" any
failures if a request doesn't get called often. It's not great, but I suggest to start with
this health check and expand it later.
Note that we use internal_failure_permille instead of doing (internal_failure /
total_requests), because canaries do not allow tracking for formulas (see
D20418610). And we might use failure rate in our canaries.
Also the diff explicitly logs "0" i.e. it logs "0" failures instead of not
logging anything. That makes it easier to set up detectors/alarms because we
don't have to deal with dead detectors.
[1] health check with internal_failure_permille will have a lower threshold
invalid_request_permille because this counter should be more reliable.
However it's probably worth alarming on invalid_request_permille as well to catch cases
e.g. where there's a bug in error classification itself.
Reviewed By: markbt
Differential Revision: D20644662
fbshipit-source-id: 694015c9add0702d47faaac8d36120d10c7003e2
Summary:
Plumb the read-only mount flag into FUSE on macOS. This fixes a build
warning and enables the --read-only mount flag.
Reviewed By: wez
Differential Revision: D20634119
fbshipit-source-id: a5e68cd163e36ceb6d86fd753844718ee9a5727f
Summary:
When we deserialize commit extras they come out as a str->str dict, so
let's make sure clients add values that are also strings. We do this by type
checking a serialization time. This fixes an issue where D19942522 assumed
extras were strings and hit a type error for globalrevs, which were ints.
Apparently the code that reads globalrevs from the extras never treats it as an
int anyway, so none of the reading code needed to be updated to convert the
string back to an int.
Reviewed By: singhsrb
Differential Revision: D20631889
fbshipit-source-id: 8c8b3c9a9f3369376e08146d670f2d6321df141f
Summary:
It's going to be useful to share certain structures between client and server.
Looking ahead, the plan is to share the segment graph along with all the
algorithms implemented for it.
Reviewed By: StanislavGlebik
Differential Revision: D20550951
fbshipit-source-id: f498a6b0cba1bcdd35fc9720125b223d7e891a44
Summary:
`iter_segments_with_parent` has a few more conditions attached to it than the
name would imply. We are renaming it to give a better sense of its true
behavior.
Reviewed By: quark-zju
Differential Revision: D20547631
fbshipit-source-id: 406f46b9de5efc9e8e6a8c4bc22ab18fa5bc54bb
Summary:
The main question I had while writing the tests was whether we expect a
specific order for Segments for `iter_segments_with_parent`. `InProcessStore`
will return the segments in the order that they were inserted.
Reviewed By: quark-zju
Differential Revision: D20501401
fbshipit-source-id: 48ceb78f3191c7425c1488a3392cf3167f7e7268
Summary:
First 6 methods implemented from the IdDagStore trait for the InProcessStore.
Any suggestions welcome.
Reviewed By: quark-zju
Differential Revision: D20499228
fbshipit-source-id: cb536a3a0136077ada78934d82a25d079a5bc809
Summary:
If `HGPLAIN` is set, and the non-legacy graph renderer is in use, use the ASCII
renderer to ensure compatible output on all platforms and configurations.
Reviewed By: farnz
Differential Revision: D20622425
fbshipit-source-id: b25a8d0526652bab07059492f7adbb684b5cbee7
Summary:
At this point all future-returning `fn`s in `benchmark_filestore`
are async.
Reviewed By: krallin
Differential Revision: D20615922
fbshipit-source-id: 8b9998e4d8933e46f0c2612270faefcb061eae37
Summary:
This diff is a combination of an asyncification and a refactor:
- isolatedthe logic for setting `blob` (which gets overwritten
5 times, often blocking on futures in the process) into it's own
function `get_blob`, which I think makes it a lot easier to see
which variables depend on one another.
- push all the rest of the logic into an async function
`run_benchmark_filestore`
Reviewed By: krallin
Differential Revision: D20615743
fbshipit-source-id: 9d0918e4078b8193c59758701c5dc5e744298d1d
Summary: Fix test that was broken by D20557896
Reviewed By: krallin
Differential Revision: D20619387
fbshipit-source-id: 3fd7ee501144528b18b7162f73dcf3d251fd5f2f
Summary:
From time to time we're experiencing the blobstore healer to crash
because its SQL queries timing out. The rootcause of the problem
is that the same blob_key may show up on the queue many times repeatedly
and the query is trying to select all occurences.
But, the original intention of blobstore healer is to act on a single
put operation across all blobstores. To be able to identify which
puts in the healer queue are part of the same operation we need
some unique id that we'll use per such operation, let's call it OperationKey.
corresponding configerator change to create db column: D20557659
NOTE: This diff has to be landed and rolled out first, before D20557700 is rolled out. I'm assuming that after some time since rolling out this diff all the rows in the production db will have proper `operation_key` value set.
Reviewed By: krallin
Differential Revision: D20557702
fbshipit-source-id: 404d9fdea6796b38193292d1bbd4b8cd4b5b3eb8
Summary:
In the previous diffs, I asyncified the api of all `fn`s in
`aliasverify`, but I still had old stream logic, which was hard to
read and required many clones of parameters to handle static lifetime
requirements.
This diff switches all the internal code over to new-style streams.
In the process of doing this, I was able to remove almost all of the
clones, and also switch `get_bounded` from needing to explicitly return
a `Future` via and `async` block to being a standard `async fn` returning
a `Result`.
Reviewed By: krallin
Differential Revision: D20582402
fbshipit-source-id: a98bb31075a41e8d6897b6893cd51ea0c5a94767
Summary:
This diff asyncifies the APIs in `aliasverify` so that all
functions are either `async fns` or (in one case where there were
challenges dealing with lifetimes) functions returning new-style
`Future`.
This diff does not asyncify the old stream API uses inside of functions,
and the remaining issues (complicated lifetime handling and extra
`clone` calls) are all related to using the old stream api, but the
function signatures should all be forward-compatible with new futures.
Reviewed By: krallin
Differential Revision: D20577861
fbshipit-source-id: 79ca2ef8f91ee9a1be9e27d8bf6f083dbc0bb7cd
Summary:
This diff pushes `compat` down one layer in `aliasverify`, by:
- Moving the future-based portion of main into a new acync fn
`run_aliasverify`
- Using new-style futures for `run_aliasverify`
Reviewed By: krallin
Differential Revision: D20572805
fbshipit-source-id: f0363793715944963a02380e861b851905975653
Summary:
This diff moves all of the Stream operations in `statistics_collector` over
from the old futures api to the new futures-preview api, and in the process
removes all `clone` calls (except where we interface with lower-level
old-futures-based apis).
For reasons that aren't entirely clear to me, rust seems to be unable to
infer the lifetimes of some parameters in `update_statistics`, so I had to
anotate it explicitly.
Reviewed By: krallin
Differential Revision: D20583198
fbshipit-source-id: 1ea247f905649b05ebaabcf27d504d903d5e6365
Summary:
Asyncify all remaining helper functions in `statistics_collector`, in the
sense that they all are now either `async fn` or return a new `Future`
explicitly.
The remaining work is to convert the guts of a couple of functions (particularly
`update_statistics` and `get_statistics_from_changeset` are still using
old-style futures. I haven't figured out how to convert from the old stream
api to the new one, so I didn't touch those spots.
Because of a lifetime issue resolving `ctx` and `repo` - which need
to be cloned before the async scope when we know the references are
still good - it was necessary to explicitly return a `NewFuture`
in `update_statistics`. The lifteime issues also forced me to do
two levels of clones of the parameters in this function, but at least the
signature is now forward-compatible.
Reviewed By: krallin
Differential Revision: D20569773
fbshipit-source-id: c2162a1fbde6d226028b3a66d2f6526c0a268232
