Summary: This diff addresses the unreleased mapped pack files in the old datapack code.
Reviewed By: chadaustin
Differential Revision: D19399737
fbshipit-source-id: 86a6254a2939fd69e1ce2b25b8bfcb36b0deb16b
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
There was a typo in the name of the parameter; this commit
makes this code consistent with the upstream pywatchman code.
Reviewed By: fanzeyi
Differential Revision: D19453811
fbshipit-source-id: 033cf30c47ff327913a07e177457a4ac23505bbd
Summary: this is largely just markdownified comments from `eden/fs/utils/PathFuncs.h`. I've been a bit confused on the differences lately and how they interact, so I read through that file and thought a markdown file would be helpful for future reference, since the file is reaching 2000 lines.
Reviewed By: fanzeyi
Differential Revision: D19434132
fbshipit-source-id: 0ec5551e9da0a3202fcc3efff52f9f6f48f99db4
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.
Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.
git clones to OpenNSA started failing yesterday with:
Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.
Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.
Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).
Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.
A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.
While github does not support that, it seems bitbucket does.
Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.
Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.
Reviewed By: wez
Differential Revision: D19437386
fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
Summary:
This is inteneded to solve multiple problems:
- Mitigate issues where our CI system does not enable remotenames in some
random code paths, which leads to missing or stale `remote/master`.
This further mitigates bad cases with narrow-heads' phase calculation
where a stale `remote/master` can cause many commits to become draft
incorrectly.
- Mitigate issues in `remotenames` where race condition can happen for
pushrebase (remotenames does a "listnames" after the push operation).
See D18601035 for a hacky workaround.
Reviewed By: DurhamG
Differential Revision: D19380940
fbshipit-source-id: 4481ff114d35be37c331c72ac561c2a0894206cb
Summary:
We recently saw a high rate of build failures on MacOS.
They all fail with compiler errors like:
In file included from edenscm/mercurial/cext/osutil.c:1326:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/ApplicationServices.framework/Headers/ApplicationServices.h:23:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:39:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/Security.framework/Headers/Security.h:38:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/Security.framework/Headers/SecProtocolOptions.h:28:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/Security.framework/Headers/SecProtocolTypes.h:158:19: error: missing ',' between enumerators
kTLSProtocol12 CF_ENUM_DEPRECATED(10_2, 10_15, 5_0, 13_0) = 8,
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks/Security.framework/Headers/SecProtocolTypes.h:158:20: error: redefinition of enumerator '__AVAILABILITY_INTERNAL__MAC_10_2_DEP__MAC_10_15'
kTLSProtocol12 CF_ENUM_DEPRECATED(10_2, 10_15, 5_0, 13_0) = 8,
^
It's unclear whether this is a compiler configuration issue, or the Apple SDK
being buggy.
The only user of ApplicationServices is `isgui()` when `$SSH_CONNECTION` is not
set on MacOS.
It seems to me that if we just replace `isgui()` to `True`, very few if nobody
will notice the difference. Therefore let's remove the problematic
ApplicationServices as an attempt to stabilize the build.
FWIW Git does not seem to have any sort of GUI detection. It seems to solely
rely on (static) user configuration.
This reverts part of https://www.mercurial-scm.org/repo/hg/rev/16118b4859a1.
Reviewed By: singhsrb
Differential Revision: D19441351
fbshipit-source-id: f1bfbaf0d015e8d577a83112b5ea0a4be6367156
Summary:
stderr: eden/fs/inodes/EdenMount.cpp:1122:37:
error: local variable 'unmountError' will be copied despite being thrown by name [-Werror,-Wreturn-std-move]
eden/fs/inodes/EdenMount.cpp:1122:37: note: call 'std::move' explicitly to avoid copying
Reviewed By: simpkins
Differential Revision: D19408659
fbshipit-source-id: 1575104414daaeb06892586492cdb74e2df604c5
Summary:
This makes it safer to do revlog-fallback -> server-fallback switch in the
future, because the server might not know all the draft commits.
Reviewed By: DurhamG
Differential Revision: D19416287
fbshipit-source-id: 8d69c1c5465b50710110370d84d5fb7c8ba6a6c4
Summary:
In both cases (clone with or without dirty content), the external key buffers
used by indexes should be re-created, since mem_buf location has changed.
Reviewed By: DurhamG
Differential Revision: D19432657
fbshipit-source-id: fe6f76e7ccfd16ccd2f5c1d89866687a3503603e
Summary: This allows us to build the multi-log structure.
Reviewed By: DurhamG
Differential Revision: D19431786
fbshipit-source-id: e0e09b0d5a73d293a80626924b74ddf2ce6d6fa3
Summary: Migrate to the new API. This is more compatible with the next change.
Reviewed By: DurhamG
Differential Revision: D19431788
fbshipit-source-id: dba1a88fd003d17fc1774b0cec9157d32c5acdb0
Summary:
This allows us to incrementally abstract away parts related to filesystem from
Log. For example, instead of using std::fs to access filesystem directly, use
methods on the GenericPath instead.
Right now the main motivation is to add support for "multi-logs" sharing a single meta
file. To do that, methods reading and writing metas are moved to the
GenericPath type.
This also unifies the open functions. Now OpenOptions::create_in_memory can be
private. Empty in-memory Logs can be opened via `open(())`.
Reviewed By: DurhamG
Differential Revision: D19431784
fbshipit-source-id: dbdf94f60261e09f131c6fdd9fe3b99242a28af5
Summary: This makes an upcoming change easier.
Reviewed By: DurhamG
Differential Revision: D19432656
fbshipit-source-id: 65adc883c3c3937aa7022196b83c75715b820721
Summary: The goal is to split log.rs to smaller modules so it's easier to reason about.
Reviewed By: DurhamG
Differential Revision: D19431787
fbshipit-source-id: 23b8346c461da322e7a2240b8224e6194867aaee
Summary:
The goal is to split log.rs to smaller modules so it's easier to reason about.
OpenOptions has many dependencies. Move the most complicated one (repair) to
another standalone module.
Reviewed By: DurhamG
Differential Revision: D19431783
fbshipit-source-id: 28c3af7cd6e19588bfe9c395e2648244faf3179d
Summary: The goal is to split log.rs to smaller modules so it's easier to reason about.
Reviewed By: DurhamG
Differential Revision: D19431785
fbshipit-source-id: b9a598b3018267ff6e43ef57dc807cb4e880467c
Summary: based on comment on D18647089, changes the behavior when a directory is replaced with a file, and then the directory is ignored. The behavior should be that the file is reported as `ADDED`, previously the file would have been reported as `IGNORED`
Reviewed By: simpkins
Differential Revision: D19318796
fbshipit-source-id: 8da1b7c6df182ca591e4aea21f6573ca81790cb4
Summary:
We want to have a shared pool of `HgImporter` across different threads, but that would require `HgImporter` is safe to be *passed* between threads. (This is not making `HgImporter` entirely thread safe.)
However, `HgImporter` currently holds a pointer to the thread local fb303 counter. This diff pushes down `EdenStats` so we only access the thread local stats variable when we need to add counters. This might be a little slower but it does make it safer.
Reviewed By: chadaustin
Differential Revision: D19053250
fbshipit-source-id: 44a897acc90c6042ae22a0417eece39e099ee13f
Summary: The only reason `HgImporter` still holds a pointer to LocalStore is for looking up proxy hashes. We can directly pass in the Mercurial hash and path instead.
Reviewed By: chadaustin
Differential Revision: D19049039
fbshipit-source-id: 45d1e1f363ed73dca447b22e5891495cf9ad202b
Summary: scuba logging for mismatched parent commits, it would be nice to have a count on how often this is encountered so we can pinpoint spikes of this case. I am not sure how helpful having the parent commits would be in the event, my thought there was that it could help see how far away the commits are and if one of the commtis doesn't exist (like in the case of stripping commtis with a cancelled hg split), but I am open to having an empty event as well if storing these strings is too expensive.
Reviewed By: fanzeyi
Differential Revision: D19432378
fbshipit-source-id: fa3e7cd6aef832c4f918f339b781590b7484cfdc
Summary:
Right now, we check if extras contain `convert_revision` (and tolerate it not
being there), but if we don't have extras at all, we crash. Let's instead treat
"no extras" and "convert_revision not in extras" as the same thing.
Reviewed By: DurhamG, farnz
Differential Revision: D19428698
fbshipit-source-id: 74268dd3bde0f9cdb1b8164d2c2d322837803eaa
Summary:
There are few cases where we need to prefetch data that is already present in
the local store, but the server holds more up-to-date data. For instance, after
pushing a change to a pushrebase server, the linkrev for the just pushed commit
is no longer valid, and the history data is re-fetched from the server to fix
it.
For now, let's keep the fix for this contained in Python as we will move away
from linkrev altogether in the future, by then, this change will be unecessary
and can be reverted.
I'm overall not a big fan of the hacks needed here, suggestions are welcomed for
a cleaner way to achieve this :)
Reviewed By: DurhamG
Differential Revision: D19412620
fbshipit-source-id: 331f3e4dbb51bcd687149370a62c60c325534f60
Summary:
In order to avoid pathological linkrevfixup after pushing a change to a
pushrebase repo, we need to be able to fetch history data that is already
present locally from the server. Since the Rust stores will always check
whether the data is present locally before fetching it, we would not be
fetching anything, causing the pathological linkrevfixup to kick in.
By allowing the stores to be built without a local component, the prefetch
function will not find the local history, and will thus be able to fetch it
properly.
Reviewed By: DurhamG
Differential Revision: D19412619
fbshipit-source-id: 421c59c63634ead7f98e6ba89da0532067f7b412
Summary:
In the cpython bindings, the Rust String can take both PyBytes and PyUnicode
strings, which is perfect for Python3 compatibility as string literals are
PyBytes in Python2 and PyUnicode in Python3. The return values are kept as
Bytes for now as changing this is a much larger change in itself.
Other approaches tried:
- Using PyUnicode as input/output: an extremely large codemod had to be done,
with very little benefits
- Using String as output: since we do have some configs that are unicode, on
the Python side, the output might either be bytestrings or unicode, leading
to weird bugs.
Reviewed By: DurhamG
Differential Revision: D18650466
fbshipit-source-id: aebdf30590dcae40b7df2787e5ece88e2ec9395c
Summary:
Update all amend tests to use mutation and visibility. For these tests, `mutation.record`
is disabled so that hashes do not change.
Most tests pass as-is. A few tests have changes which are the results of slight behavioural
changes or bug-fixes compared with obsmarkers.
Reviewed By: quark-zju
Differential Revision: D19413580
fbshipit-source-id: 18696a08db69130b08f829366659783df4c9bcc1
Summary:
Previously the mutation commit extras were the source of truth for mutation
information, and the mutation store served as a kind of cache. This turned out
to be less useful than expected, as oftentimes commits are missing, and the
store is better indexed, so in practice using the store as the source of truth
is better.
This change makes the mutationstore the (sole) source of truth for mutation
data. The extras are kept, but they are now only useful as human-readable
debug information, and to ensure the commit hash is unique.
Collecting the mutation information during commit creation is now done through
a new `mutinfo` object. This is a dict with the same keys as the mutation
extras, for simplicity, but it is now passed through the `committablectx` and
used to generate the mutation store entry directly.
The `mutation.enabled` config option is now used to control all aspects of
enabling mutation.
The `mutation.record` config option is now only used to indicate whether the
mutation extras should also added to the commit. Generally this should be set
to `true`, however the option is retained so that mutation extras can be
stripped by running `hg amend --config mutation.record=false`, which no longer
has the side-effect of not recording mutation information to the store.
The "remote commit" mutation record origin is now obsolete, and won't be
generated anymore.
Pushrebase now relies on the obsmarker information coming back from the server
in order to correctly generate mutation information. We will need to change
this so that the server returns mutation records before we can fully deprecate
obsmarkers.
Reviewed By: DurhamG
Differential Revision: D19410650
fbshipit-source-id: 8d7094e4bfd8d8e97916898d899a8debd339485f
Summary: Begin logging the common directory path for the commit's updated files.
Reviewed By: DurhamG
Differential Revision: D19399717
fbshipit-source-id: d47994f573eac6e7cd1596b0b102cb9577e1eea1
Summary: When I was first prototyping the converter, I used a prefix to make it clear which commits represented merged commit history. This commit removes that prefix so the commit messages are an exact match.
Reviewed By: tchebb
Differential Revision: D19404075
fbshipit-source-id: 73b4aab39b30ce1dcff917e40d459f9615896fca
Summary:
The linknode is part of the history, not the data, no need to force prefetch
the blobs.
Reviewed By: quark-zju
Differential Revision: D19394351
fbshipit-source-id: 2e63d82928ebd5cf8e3a0d1b87d115b45d1428a7
Summary:
Mercurial assumes some specific ordering, let's comment about this to make sure
we don't re-order and introduce subtle bugs.
Reviewed By: quark-zju
Differential Revision: D19394352
fbshipit-source-id: 0f9e02d2c6addf040311a54b8161b06bbeaa6be9
Summary:
The error message and the exception type are slightly different between the
Rust and Python ContentStore. For now, let's just fix this up manually.
Reviewed By: quark-zju
Differential Revision: D19394350
fbshipit-source-id: e432094a9dfcf605568a1890c0303b733e98d203
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md
With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.
Reviewed By: wez
Differential Revision: D19413708
fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
Summary: The rust telemetry wrapper and the python code were not in sync in how they presented errors to the user on failure. This makes them identical in message and error code (however the "abort" from python is red in color).
Reviewed By: quark-zju
Differential Revision: D19313848
fbshipit-source-id: d5b95e8d5130f20fc85c6664933ee18702f7cc5e
Summary: This eliminates the broken pipes that have been more prevalent recently.
Reviewed By: DurhamG
Differential Revision: D19412153
fbshipit-source-id: 0062eb57ec07dfcbfee3ed9b77cf82cdcda09fb3
Summary:
Previously, NameDag only supports writing directly to disk. That is not easy
for hg to use, since hg knows commits before bookmark names during pull, while
the API requires both commits and bookmarks (to decide which commits belong to
the "master" group) at the same time.
The old API also does not match other storage layers like indexedlog and
metalog, where writes are buffered until explicitly flushed.
This diff adds the "write-in-memory" (named "add_heads"), and "flush" APIs to
make NameDag easier to use. Under the hood, it just copies the "added" portion
of the DAG, and re-use the old API ("build", renamed to "add_heads_and_flush")
to do the job.
Note: The buffered changes pattern cannot be used in IdDag, since Ids might
have to be re-assigned on flush. But NameDag does not expose the raw Ids in its
"normal" interface, so it's fine to reassign Ids on flush.
Reviewed By: markbt
Differential Revision: D19405474
fbshipit-source-id: 75e5e5815c78c3577a0138f48185f6c4b5a80891
Summary:
This is more consistent with the name "IdDag", because both "Name" and "Id" are
noun.
Reviewed By: singhsrb
Differential Revision: D19405473
fbshipit-source-id: a3b7546dd0ddcae5d9ed562838bd6be10a47063c
Summary: This is more consistent with the name "NamedDag".
Reviewed By: singhsrb
Differential Revision: D19405472
fbshipit-source-id: f7023307acaf96bf77c9fa9704dcaf6fc59b56f2
