Summary: Update more tests to avoid relying on pack files and legacy LFS, and override configs in `test-inconsistent-hash.t` to continue using pack files even after the scmstore rollout to test the Mononoke's response to corruption, which is not currently as easy with indexedlog.
Reviewed By: quark-zju
Differential Revision: D29229650
fbshipit-source-id: 11fe677fcecbb19acbefc9182b17062b8e1644d8
Summary:
Pull in a patch which fixes writing out an incorrect entsize for the
`SHT_GNU_versym` section:
ddbae72082
Reviewed By: igorsugak
Differential Revision: D29248208
fbshipit-source-id: 90bbaa179df79e817e3eaa846ecfef5c1236073a
Summary:
For context and high level goal, see: https://fb.quip.com/8zOkAQRiXGQ3
On RedactedBlobs, let's return an `Arc<HashMap>` instead of `&Hashmap`.
This is not needed now, but when reloading information from configerator, we won't be able to return a reference, only a pointer.
Reviewed By: StanislavGlebik
Differential Revision: D28962040
fbshipit-source-id: 0848acc1a81a87c0b51d968efe31f61dacd57c47
Summary:
For context and high level goal, see: https://fb.quip.com/8zOkAQRiXGQ3
Instead of using `HashMap<String, RedactedMetadata>` everywhere, let's use a `Arc<RedactedBlobs>` object from which we can instead borrow a map. The borrow function is async because it will need to be when we're fetching from configerator, as it may need to rebuild the redaction data.
Wrapping it in `Arc` will also makes it re-use the same across repos, I believe right now it's cloned everywhere.
In later diffs I'll use this enum to add a new way to fetch configs.
Reviewed By: markbt
Differential Revision: D28935506
fbshipit-source-id: befa96810ee7ebb9487f99f9e769a945981b58ed
Summary:
We're doing imports for AOSP megarepo work, and want a tool to quickly check that our imports are what we expect.
Use libgit2 and a simple LFS parser to read git SHA-256 entries, and FSNodes to get the Mononoke entries to match
Reviewed By: StanislavGlebik
Differential Revision: D29169743
fbshipit-source-id: 1ef1e2c780b8742c7fa5f15f9ee01bc0481a6543
Summary: This is a minimal fix so that it builds, not enough to test the new bit, but enough to unbreak contbuild
Reviewed By: yancouto, HarveyHunt
Differential Revision: D29263246
fbshipit-source-id: c5430ff4bc885103664c33caca90af5819d97ddd
Summary:
Looks like D29145340 (d0e16f1a25) introduced regression - "hg pull" fails with
"TypeError: Population must be a sequence or set. For dicts, use list(d).", stack trace - P424458181.
This diff fixing it by converted a dag.nameset to a list first
Reviewed By: mzr
Differential Revision: D29258771
fbshipit-source-id: 9ffcc756f9931d6d24b69dadf1cd2d08faccb443
Summary: Spotted this in passing. Save a DashMap lookup in the OldestFirst case by checking the enum first
Reviewed By: farnz
Differential Revision: D29232280
fbshipit-source-id: 72e93ee704767a42c36ffeec505fd79a22c4d88e
Summary:
At the moment we have a few ways of deriving data:
1) "normal", which is used by most of the mononoke code. In this case we insert
derived data mapping after all the data for a given derived data type was
safely saved.
2) "backfill", which is used when we backfilling a lot of commits. In this case
we write all the data to in-memory blobstore first, and only later we save data
to real blobstore, and then write derived data mapping
3) "batch", when we derive data for a few commits at once. It can be combined
with "backfill" mode.
We also have a special scuba table for derived data derivation, however there
are a few problems with it.
Only "normal" mode has good and predictable logging i.e. it logs once before we
attempt to derive a commit, and once after commit was derived or failed.
"backfill" logs right after data for a given commit was "derived", however this is an in-memory
derivation, and at this point no data was saved to the blobstore.
So if backfill process crashes a bit later then commit might not be derived
after all, and it's impossible to tell it just by looking at the scuba table.
With "batch" mode it's even worse - we don't get any logs at all.
A bigger refactoring is needed here, because currently the process of
derivation is very hard to grok. But for now I suggest to slightly improve
scuba logging by logging and even when a derived data mapping was actually written (or failed to be
written). After this diff we'll get the following:
1) "normal" mode will get three entries in scuba table in this order: derivation start,
mapping written, derivation end,
2) "backfill" mode will also get three entries in scuba table by in a different
order: derivation start, derivation end, mapping written
3) "batch" mode will get one entry for writing the mapping. Not great, but
better than nothing!
Reviewed By: farnz
Differential Revision: D29231404
fbshipit-source-id: 2c601e7dc58c00e22fda1ddd542833a818d1d023
Summary: Just moving a code around a bit to make derive_impl file a bit smaller
Reviewed By: farnz
Differential Revision: D29231405
fbshipit-source-id: c923f42710f4be98147bc58d5b828d5d6c7bf1a6
Summary:
Invalidation on Windows is tricky, and I got it wrong in subtle ways
previously. The main obvious issue is that when the on-disk invalidation fails,
the refcount shouldn't be decremented as the placeholder/file is still present
on disk. This could cause weird issues in later checkout. The second one is how
invalidating a directory doesn't remove a placeholder (it actually adds one),
and thus we shouldn't decrement the FS refcount. And lastly, the refcount
should be decremented regardless of whether the inode is loaded or unloaded. As
long as it is known by the InodeMap it needs invalidation.
Reviewed By: fanzeyi
Differential Revision: D28970899
fbshipit-source-id: 0d64cadae01fcd4e028c53de9357ece7d648cdd4
Summary:
A user may have some undesirable environment variables when calling `edenfsctl start`,
which we do not want to propagate to edenfs as this may affect EdenFS's
ability to run properly. Having PYTHONPATH set to inside a repository may for
instance lead to a deadlock when EdenFS is trying to setup redirections.
To avoid this, we need to sanitize the environment before calling edenfs. This
functionality already exist but was bypassed on Windows.
Reviewed By: chadaustin
Differential Revision: D29244358
fbshipit-source-id: bc96698732e71412296ed5e28842b59b2c758699
Summary:
Introduce `LegacyStore` trait, which contains ContentStore methods not covered by other datastore traits.
Implement this trait for both contentstore and scmstore, and modify rust code which consumes `contentstore` directly to use `PyObject` and `LegacyStore` to abstract over both contentstore and scmstore instead.
Reviewed By: DurhamG
Differential Revision: D29043162
fbshipit-source-id: 26e10b23efc423265d47a8a13b25f223dbaef25c
Summary: Introduce a new config option, `scmstore.enableshim`, which replaces instances of contentstore in Python with scmstore objects instead. Currently, this config is not safe to enable. Addition fixes are incoming.
Reviewed By: DurhamG
Differential Revision: D29213190
fbshipit-source-id: 7fd4db77d55cd25cc08c40bee28798d6a6d2555c
Summary: Previously, we just fetched "best effort", and logged any encountered errors using `tracing`, leaving it up to the client to inspect errors if necessary. Python relies on catching these fetch errors as exceptions, though, so this change introduces some utility methods to help propagate them correctly.
Reviewed By: DurhamG
Differential Revision: D29211683
fbshipit-source-id: 5e9dee942c2b60e0f77a051624d7f393a811fc4e
Summary:
Remove packfile-specific parts of tests and modify them to test without depending on packfiles where possible.
Currently debugindexedlogdatastore and debugindexedloghistorystore appear to be broken, and debugdumpindexedlog just dumps the raw indexedlog contents, without any semantic information, so for the time being I've simply removed most packfile inspection.
Reviewed By: DurhamG
Differential Revision: D29099241
fbshipit-source-id: 86c4f9c83520374560587b8bec5c569d9c5c6510
Summary: My previous fix was actually incorrect, we now log actual remote requests, but join that with the logs from the contentstore fallback.
Reviewed By: DurhamG
Differential Revision: D29206878
fbshipit-source-id: d22e58792bf380c274e8086ce08aebe20dd9b848
Summary: The original commit broke globbing more than it fixed it. D29175333 will fully fix it, but in the meantime, let's revert the change to get a release out.
Reviewed By: singhsrb
Differential Revision: D29231954
fbshipit-source-id: 7a42e980c6fc4de09bee713a3a4141d52272b6d1
Summary:
Previously, when fetching data using several concurrent requests, the EdenAPI client would wait for the headers for every request to finish coming in before starting to deserialize and yield entries from the bodies of any of the requests.
Normally, this isn't a huge deal since the response headers on all of the requests are usually roughly the same size, so they all finish downloading at roughly the same time when the requests are run concurrently. However, this does become an issue when `edenapi.maxrequests` is set. This option makes EdenAPI configure libcurl to queue outgoing connections once the configured limit is hit.
This means that although from EdenAPI's perspective all of the requests are running concurrently, they are not actually running in parallel. The result is that the EdenAPI client ends up waiting for all of the queued requests to be sent before yielding any data to the caller, which forces it to buffer all of the received data, resulting in massive memory consumption.
This diff fixes the problem by rearranging the structure of the Futures/Streams involved such that the client immediately begins yielding entries when they are received from any of the underlying transfers.
Reviewed By: quark-zju
Differential Revision: D29204196
fbshipit-source-id: b6b56bb7d60457de3c4046a07a5965749e9dd371
Summary:
When the `send_async` method is used to dispatch multiple concurrent requests, the method needs to return an `AsyncResponse` for each request. Since `AsyncResponse`'s constructor is itself `async` (it waits for all of the headers to be received), internally the method ends up with a collection of `AsyncResponse` futures.
Previously, in an attempt to simplify the API, the method would insert all of these futures into a `FuturesUnordered`, thereby conceptually returning a `Stream` of `AsyncResponses`. Unfortunately, this API ends up making it harder to consume the resulting `AsyncResponses` concurrently, as one might want to do when streaming lots of data over several concurrent requests.
This diff changes the API to just insert the `AsyncResponse` futures into a `Vec` to allow the caller to use them as desired. To maintain compatibility with the old behavior for the sake of this diff, the one current callsite has been updated to just dump the returned `Vec` into a `FuturesUnordered`. This will be changed later in the stack.
Reviewed By: quark-zju
Differential Revision: D29204195
fbshipit-source-id: ecee8cff430badd8213c2efef62fc68fbd91fde9
Summary: Nothing was using this metadata, and removing it simplifies the subsequent diffs in this stack.
Reviewed By: quark-zju
Differential Revision: D29147228
fbshipit-source-id: aa4828b710c3ef719f4d66adec5f66cd5b7d05d1
Summary:
I'm seeing significant Zippy load when I do a check scrub of our big repo to make sure that it's all in SQL Blobstore as well as our main blob stores.
Teach scrub to not bother talking to the main blobstores unless the write-mostly blobstore is either missing the data or unable to retrieve it.
Reviewed By: ahornby
Differential Revision: D29233349
fbshipit-source-id: 1127129ff283477558cddb03686c3c13aee47fb5
Summary:
Now that the InodeMap is pre-populated with the Overlay, we no longer need the
`folly::kIsWindows` test during checkout as inodes will enter this condition
when unloaded (IsInodeRemembered).
Reviewed By: fanzeyi
Differential Revision: D25106153
fbshipit-source-id: 511d795ae947651e1eaf3c54b8f1ab83c77f5cc4
Summary:
This dep got updated in D29165283 (b82c5672fc) across a major version but the code depending
on it wasn't so now it's broken.
Reviewed By: mitrandir77
Differential Revision: D29229087
fbshipit-source-id: 5f2a14dd9f0447dd4578e8321991dfb3df32dcc2
Summary:
This assumes the Eden binary is next to edenfsctl. Like in the previous diffs,
it isn't.
Reviewed By: xavierd
Differential Revision: D29200149
fbshipit-source-id: a7d335c5f3748c795b20480e26b24b9287708c50
Summary:
Like it says in the title. This fails because it assumes it's next to the
EdenFS binary but that's not true in Buck v2.
Reviewed By: xavierd
Differential Revision: D29200150
fbshipit-source-id: 1be5b6c647047a50dd553031691e9bbec5f9caaf
Summary: Pretty much the same as D29061439 (8f263485d9), but in another place that needs it.
Reviewed By: xavierd
Differential Revision: D29197979
fbshipit-source-id: eafccefee54a64131eb7547ad19ab1da708852e6
Summary:
On a non-EdenFS mount, a case change of a file or directory isn't reported as
the file can still be accessed with the same path, and it has the same content
as before. EdenFS had a different behavior whereas renaming a file to a
different case would report a missing file and an untracked one, with different
casing.
This can be surprising to users as the behavior is different from Mercurial,
and it's also hard to actually fix as 2 renames would need to happen, a single
one would not work due to the case insensitivity nature of the filesystem.
While I do believe that reporting the case change might be more desirable to
allow users to actually commit changes the case of files, Mercurial might be
broken in subtle ways today if we allow this, thus it's best to make EdenFS
behave similarly to Mercurial.
Reviewed By: genevievehelsel
Differential Revision: D29150552
fbshipit-source-id: 6cceaa4c9fa61c03f35fcd91a9c01554da252222
Summary:
On case insensitive mounts, updating between commits that have a different
casing for a file/directory would lead to the update failing due to EdenFS
believing that an untracked file is present in the mount. That conflict is
however bogus and EdenFS simply gets confused in
TreeInode::processCheckoutEntry about the entry with the different casing.
To fix this, we should avoid comparing paths in a case sensitive manner and
instead compare then in a case insensitive fashion. This allows the rest of the
checkout code to update the directory/file in place.
On Windows, there is one more subtlety: we can't change the casing of a
file/directory that is already a placeholder and thus we need to force the
entire hierarchy to be removed, this will also make the checkout fail in case
of untracked files in the hierarchy, which is also the behavior on case
sensitive systems.
Reviewed By: genevievehelsel
Differential Revision: D29121741
fbshipit-source-id: 3d2cdacf296a3d061fc828cd6d04d249542cb63f
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary:
## This diff
* We are migrating `edenfsctl strace` to cpp due to segfaulting from Thrift-py3 (similar to {D25515377 (a152fa4585)}).
* This diff implements new subcmd `edenfsctl trace strace` for this cpp migration.
Reviewed By: xavierd
Differential Revision: D29164534
fbshipit-source-id: 4d8ed23393004f394159c36f71e0c78c077c7c73
Summary: It was rolled out weeks ago and there seem to be no complaints so far.
Reviewed By: DurhamG
Differential Revision: D29172396
fbshipit-source-id: 4f3310597ef1bb1abd92ee7207700a3d0039c598
Summary:
add an option to pass some metadata in the token
This will be used for content tokens, for example. We would like to guarantee that the specific content has been uploaded and it had the specific length. This will be used for hg filenodes upload.
Reviewed By: markbt
Differential Revision: D29136295
fbshipit-source-id: 2fbd3917ee0a55f43216351fdbc1a6686eb80176
Summary:
use `commitknown` edenapi api for checking the existing commits
it uses the same `lookup_commits` under the hood but a bit shorter to use
we won't need the tokens for existing changesets, so can use a simpler api
also, make `lookupfilenodes` function a bit shorter
Reviewed By: markbt
Differential Revision: D29134677
fbshipit-source-id: 257624d64480102c34761560b2bd768049cbfa83
Summary:
upload file content into blobstore
the existing Mononoke API already validates the provided hashes and calculates the missing one
we would probably need to write to all multiplexed blobstores, but multiplexing will be addressed separately
Reviewed By: markbt
Differential Revision: D29103111
fbshipit-source-id: 0cac837efc238f618a35420523279fb7aa91668a
Summary: Allow puts to sqlblob with mysql backing to use the InlineBase64 hash type.
Reviewed By: farnz
Differential Revision: D28829452
fbshipit-source-id: 265cf45e55284d34d3002a9db205e14eaee4fa39
Summary: Obsolescence markers have been deprecated in favour of mutation and visibility for some time. Remove syncing of obsmarkers via the commit cloud service.
Reviewed By: liubov-dmitrieva
Differential Revision: D29159443
fbshipit-source-id: 33e1d526a9df5c02059931c880529d695036c616
Summary:
It's useful to have it configurable.
While here, also use slog instead of println to attach timestamp as well
Reviewed By: Croohand
Differential Revision: D29165693
fbshipit-source-id: d844926560b15042445d5861a281870ac102d12e
Summary:
Non-buck build cannot succeed if Python.h is missing.
Let's check it explicitly so we don't pick a bad Python.
Reviewed By: DurhamG
Differential Revision: D29179295
fbshipit-source-id: 421c824053d066914a6611f05815527768f257ee
Summary:
It turns out that `python3` might be a symlink to a fbcode Python that cannot
perform the build in some (not all) environment. Let's try more names like
`python3.8`, `python3.7`, etc.
Reviewed By: DurhamG
Differential Revision: D29178933
fbshipit-source-id: da6cae351f25a90ab8a9da85282d09f79505c5e7
Summary:
IPython 7 has compatibility issues when bundling into zip:
- The dependency parso does not work when embedded in zip. It expects
`python/grammar38.txt` to exist on the filesystem (not in a zip).
- It stacktraces after exiting the IPython shell.
- TAB completion does not really work.
Downgrading to IPython 5 and avoid the bundling the parso dep to resolve the
issues.
This should fix `test-autofix.t` and `test-argspans.py` failures.
Reviewed By: DurhamG
Differential Revision: D29170653
fbshipit-source-id: 14a3d16deaca72fbfb7b3acc0a4246a771c4d0aa
