Summary:
We'll be adding a bunch of Facebook specific configuration and values
here. Let's move it to someplace not open source.
Reviewed By: quark-zju
Differential Revision: D21241038
fbshipit-source-id: 2ac9cdce40b1b46f15f171d9d1f6b6692dcd29bf
Summary:
Implements an ensure_location_supersets function who's goal is to
verify that a given config location specifies the exact same configs as a given
set of other locations. Any inconsistencies are removed from the config and
reported to the caller.
This will be used to ensure our dynamic configs match our existing rc file
configs exactly, before we delete the file configs.
Reviewed By: quark-zju
Differential Revision: D21240837
fbshipit-source-id: e2c8ec054a3696d2cf02e65c212ad886c5117253
Summary: `cargo autocargo` should normally produce no changes on `master`. The features of the `log` crate was updated in D21303891 without re-running autocargo. This fixes it.
Reviewed By: dtolnay
Differential Revision: D21349799
fbshipit-source-id: ce487bc5989e179673297350249593103b4d34dd
Summary: Fix permission issues we are seeing with the latest Mercurial release.
Reviewed By: xavierd
Differential Revision: D21294499
fbshipit-source-id: bcfb13dd005258b2e3b74fa281dbd8df36133ef6
Summary:
I wanted to figure out "who added this visible head", "what is the difference
between this metalog root and that root". Those are actually source control
operations (blame, diff). Add a git export feature so we can export metalog
to git to run those queries.
Choosing git here as we don't have native Rust utilities to create a more
efficient hg repo yet.
Ideally we can also make hg operate on a metalog directory as a "metalogrepo"
directly. However that seems to be quite difficult right now due to poor
abstractions.
Reviewed By: DurhamG
Differential Revision: D21213073
fbshipit-source-id: 4cc0331fbad6e1586907c0a66c18bcc25608ea49
Summary: This allows the Python world to obtain the root ID for logging purpose.
Reviewed By: DurhamG
Differential Revision: D21179513
fbshipit-source-id: 3f289c06d3d470ff492de39fa985203b3facbf00
Summary:
We removed the feature in D20704618 and it does not cause complaints.
Let's remove the code supporting the chown feature.
Reviewed By: DurhamG
Differential Revision: D21170307
fbshipit-source-id: c845016219e8c681930bb1780b94e6d31ca99730
Summary:
While the change looks fairly mechanical and simple, the why is a bit tricky.
If we follow the calls of `ContentStore::get`, we can see that it first goes
through every on-disk stores, and then switches to the remote ones, thanks to
that, when we reach the remote stores there is no reason to believe that the
local store attached to them contains the data we're fetching. Thus the
code used to always prefetch the data, before reading from the store what was
just written.
While this is true for regular stores (packstore, indexedlog, etc), it starts
to break down for the LFS store. The reason being that the LFS store is
internally represented as 2 halves: a pointer store, and a blob store. It is
entirely possible that the LFS store contains a pointer, but not the actual
blob. In that case, the `get` executed on the LFS store will simply return
`Ok(None)` as the blob just isn't present, which will cause us to fallback to
the remote stores. Since we do have the pointer locally, we shouldn't try to
refetch it from the remote store, and thus why a `get_missing` needs to be run
before fetching from the remote store.
As I was writing this, I realized that all of this subtle behavior is basically
the same between all the stores, but unfortunately, doing a:
impl<T: RemoteDataStore + ?Sized> HgIdDataStore for T
Conflicts with the one for `Deref<Target=HgIdDataStore>`. Macros could be used
to avoid code duplication, but for now let's not stray into them.
Reviewed By: DurhamG
Differential Revision: D21132667
fbshipit-source-id: 67a2544c36c2979dbac70dac5c1d055845509746
Summary: implement the get() functions on the various LocalDataStore interface implementations
Reviewed By: quark-zju
Differential Revision: D21220723
fbshipit-source-id: d69e805c40fb47db6970934e53a7cc8ac057b62b
Summary:
Memcache isn't available for Mac, but we can build the revisionstore with Buck
on macOS when building EdenFS. Let's only use Memcache for fbcode builds on
Linux for now.
Reviewed By: chadaustin
Differential Revision: D21235247
fbshipit-source-id: 5943ad84f6442e4dabbd2a44ae105457f5bb9d21
Summary:
When creates directories sometime we want to make sure other users within the same group have the write access to it to enable data sharing. Previously we rely on setting umask for the entire process to make sure the newly created directories have the correct permission bit. This is kind fragile and error-prone when running in a multi-thread environment.
This diff introduces an internal function `create_dir_with_mode` to create directory with specified permission mode. It first creates a temporary directory within the parent of the directory being created, setting up the correct permission bit, then attempts to rename the temporary directory to the desired name. This ensures that we never leave a directory without the correct permission in the place we need and without changing umask for the process.
Reviewed By: xavierd
Differential Revision: D21188903
fbshipit-source-id: 381bff7d3aaca097b9d50150e86cbbf70a90a0a5
Summary:
The second phase of pending changes is to iterate over the treestate
and figure out what files were not seen in the filesystem walk. This diff
implements that.
Reviewed By: xavierd
Differential Revision: D20546899
fbshipit-source-id: 3523fbc7e31ef0ed09c4937c72264b64e2a3db5b
Summary:
The first phase of pending changes is inspecting the filesystem for
changes. This diff adds that logic.
Reviewed By: xavierd
Differential Revision: D20546909
fbshipit-source-id: 1c2c0fa7f700dbff4acfce4d5271b4472a13571f
Summary:
On repack, when the Rust stores are in use, the repack code relies on
ContentStore::commit_pending to return the path of a newly created packfile, so
it won't delete it when going over the repacked ones. When LFS is enabled, both
the shared and the local stores are behind the LfsMultiplexer store that
unfortunately would always return `Ok(None)`. In this situation, the repack
code would delete all the repacked packfiles, which usually is the expected
behvior, unless only one packfile is being repacked, in which case the repack
code effectively re-creates the same packfile, and is then subsequently
deleted.
The solution is for the multiplex stores to properly return a path if one was
returned from the underlying stores.
Reviewed By: DurhamG
Differential Revision: D21211981
fbshipit-source-id: 74e4b9e5d2f5d9409ce732935552a02bdde85b93
Summary:
Add two utility programs for ad-hoc debugging of EdenAPI. EdenAPI requests and responses are encoded as CBOR, which is not easy to work with manually on the command line. In order to allow debugging the HTTP API using tools like `curl`, we need tools that can generate raw request payloads and interpret CBOR responses.
The utility programs included in this diff are:
- `make_req` - Can construct EdenAPI request payloads from a human-editable JSON representation.
- `data_util` - Can list, validate, and extract the contents of an EdenAPI data response.
These tools can be used by themselves or as part of a pipeline. See test plan for examples.
Reviewed By: xavierd
Differential Revision: D21136575
fbshipit-source-id: d1ac8d92964614005078a6ac76dd0835c29a80a5
Summary: Move the MutationEntry type to the Mercurial types crate. This will allow us to use it from Mononoke.
Reviewed By: quark-zju
Differential Revision: D20871338
fbshipit-source-id: 8de3bb8a2673673bc4c8a6cc7578a0a76358c14a
Summary:
The part of status that lists what files have changed is called
PendingChanges. This diff introduces the initial stub for PendingChanges. The
pending changes algorithm involves three parts:
1. Looking at files on the filesystem for changes.
2. Looking at files in the dirstate map for changes.
3. Looking at the content for any files that we were unsure of during steps 1
and 2.
This diff puts the basic state machine in place, and accepts the basic
information about the working copy (the root and what type of filesystem it is).
In the future we might have it detect what type of filesystem it is, but for now
this makes it easy.
Reviewed By: xavierd
Differential Revision: D20546898
fbshipit-source-id: a3030b7c846b3cb2fcba805b7fe4744df7c5764e
Summary:
treestate.get_filtered_keys passes directory paths to the filter
function and returns directory matches with a trailing '/' on the end. This
makes it difficult to act as a path normalization function when the caller
doesn't know if the path is a file or directory.
It seems like we can just strip the trailing '/' before exposing the strings to
the caller (both as filter inputs and as get_filtered_keys outputs).
This is useful in the following diff that adds a case normalization crate.
Reviewed By: xavierd
Differential Revision: D20880881
fbshipit-source-id: 6e9f419178b4e278844244bd6aff2fc10e09d2cd
Summary:
This logic will be used in a variety of places (update workers, status,
etc). Let's move it somewhere common.
Reviewed By: xavierd
Differential Revision: D20771623
fbshipit-source-id: b4de7c1d20055a10bbc1143d44c55ea1045ec62a
Summary:
PathAuditor will be needed for native status soon. Let's move it into
the workingcopy crate.
Reviewed By: xavierd
Differential Revision: D20546906
fbshipit-source-id: ef69f88ee828a72e82b5e944cc7913f391bd8a2f
Summary: This will help us debug slow commands
Reviewed By: xavierd
Differential Revision: D21075895
fbshipit-source-id: 3e7667bb0e4426d743841d8fda00fa4a315f0120
Summary:
The Memcache store is voluntarily added to the ContentStore read store, first
as a regular store, and then as a remote one. The regular store is added to
enable the slower remote store to write to it so that blobs are uploaded to
Memcache as we read them from the network. The subtle part of this is that the
HgIdDataStore methods should not do anything, or the data fetched won't be
written to any on-disk store, forcing a refetch next time the blob is needed.
Reviewed By: DurhamG
Differential Revision: D21132669
fbshipit-source-id: 96e963c7bb4209add5a51a5fc48bc38f6bcd2cd9
Summary:
When comparing empty file with file with content our xdiff wrongly included
warning about missing newline, which also made the line counter in the hunk
header off-by-one.
Empty files are quite rare in our repos, that's why I discovered this bug only
now (it broke phabricator parsing of this single commit).
Reviewed By: markedson1024
Differential Revision: D21141341
fbshipit-source-id: 9d3e0d8a61ac4ee2cf27978b99b3a092259ee186
Summary:
Ideally, either the ContentStore, or the upper layer should verify that we
haven't missed uploading a blob, which could lead to weird behavior down the
line. For now, all the stores will return the keys of the blobs that weren't
uploaded, which allows us to return these keys to Python.
Reviewed By: DurhamG
Differential Revision: D21103998
fbshipit-source-id: 5bab0bbec32244291c65a07aa2a13aec344e715e
Summary:
We'll be adding more data to the filesystem layer, so let's move this
out of lib.rs.
Also made a slight tweak to expose File metadata in the walk results, which will be used by the future pending changes logic to avoid re-stating the file.
Reviewed By: xavierd
Differential Revision: D20546903
fbshipit-source-id: 70456055b0da601990e6d6ff535678d2df6c50ba
Summary:
This allows the streampager to be configured via hgrc files.
Default are picked so the behavior is closer to the current default pager
(`less -FRX`).
Reviewed By: DurhamG
Differential Revision: D20902034
fbshipit-source-id: 994ab963ceace02eeb1d18cfa5768e411ca3610b
Summary: This makes it work with chg, since `/dev/tty` is not available for chg.
Reviewed By: DurhamG
Differential Revision: D20936967
fbshipit-source-id: f3ded1aa5552f321ff7043a039f4e35a88160a51
Summary:
We want Mercurial to become more responsible for it's own
configuration, instead of relying on chef and other means. To do so, let's
introduce a new `hg debugdynamicconfig` that can generate dynamic configs for
a given repository based on various states, like what tier it's in or what shard
that machine is in. By default it generates to '.hg/hgrc.dynamic' for the given
repository.
Currently it just sets the hostgroup config.
Future diffs will make Mercurial consume this config, and possibly have Mercurial
call this command asynchronously when it notices the file is out-of-date.
Reviewed By: quark-zju
Differential Revision: D20828132
fbshipit-source-id: 6f5bf749f5b04e0a5989d6dc19ee788c2e47f88f
Summary:
A future diff will want to generate configs programmatically and write
them to a file. Let's add write support to ConfigSet.
Reviewed By: quark-zju
Differential Revision: D20828133
fbshipit-source-id: 702f6f9bdfdf99ef25c6e1c0ab33373a4b6508fe
Summary:
The revisionstore is a large crate with many dependencies, split out the types part which is most likely to be shared between different pieces of eden/mononoke infrastructure.
With this split it was easy to get eden/mononoke/mercurial/bundles
Reviewed By: farnz
Differential Revision: D20869220
fbshipit-source-id: e9ee4144e7f6250af44802e43221a5b6521d965d
Summary: Since the old Edenfs warning is usually for simply picking up new eden releases, we can suggest the user runs a graceful restart instead of a normal restart to avoid them running into `Transport not connected` errors. This path is only hit in unix environments, so windows users will not see this (since graceful restart isn't supported there yet). Since this is a manual step as well, it will be easier for a user to see if they run into an issue here. This can also enable us to get more telemetry from users running graceful restarts.
Reviewed By: wez
Differential Revision: D20901597
fbshipit-source-id: 9e5c9a90313901be159f66afcbbadc5d7af4fe28
Summary:
Sometimes the Rust io::Error is generated without an errno (ex. pipe
0.2 would generate BrokenPipe error without an errno). The Python
land uses errno to check error type (Python does not have io::ErrorKind).
Therefore attempt to translate ErrorKind to Python errno. Without this
exiting the rust pager early would crash like:
StdioError: [Errno None] pipe reader has been dropped
abort: pipe reader has been dropped
Reviewed By: markbt
Differential Revision: D20898559
fbshipit-source-id: ef863617e0e500d878ea0f9aeac06b4d87ffbcf2
Summary: This makes the tracing features easier to use.
Reviewed By: DurhamG
Differential Revision: D19797703
fbshipit-source-id: fb5cb17cd389575cf0134a708bcd9df3b90e9ab4
Summary:
On upload, read all the local blobs and upload them to the LFS server. This is
necessary to support `hg push` or `hg cloud sync` for local LFS blobs.
One of the change made here is to switch from having the batch method return an
Iterator to having them take a callback. This made it easier to write the gut
of the batch implementation in a more generic way.
A future change will also take care of moving local blobs to the shared store
after upload.
Reviewed By: DurhamG
Differential Revision: D20843136
fbshipit-source-id: 92d34a0971263829ff58e137e9905b527e18358d
Summary: This method will be used to upload local LFS blobs to the LFS server.
Reviewed By: DurhamG
Differential Revision: D20843137
fbshipit-source-id: 33a331c42687c47442189ee329da33cb5ce4d376
Summary:
Loose files makes it easier to interact with a Mercurial server for tests, use
it instead of an IndexedLog.
Reviewed By: DurhamG
Differential Revision: D20786432
fbshipit-source-id: 61c1fc601d9a6ed157c5add9748e40840b081870
Summary:
This exposes the Rust's pager to Python. Right now it's using the system
terminal.
Reviewed By: DurhamG
Differential Revision: D20887174
fbshipit-source-id: c72f31a58475e76f8097c515dd29f911d2ac4df1
Summary:
Do not convert the entire output to a string. This makes `debugindexedlog dump`
a good test case for native pager support - it takes a while to write the full
output for a large input.
Reviewed By: DurhamG
Differential Revision: D20885567
fbshipit-source-id: 35ed8f68dff1916f0833577c3cf2a52cbf2a658c
Summary:
Implement the core API to start pager in native Rust. For now it is only
enabled for the entire command if `--pager=always` is set.
Reviewed By: DurhamG
Differential Revision: D20849644
fbshipit-source-id: 860b4e18d841da607864c3447d78dbac126f5f18
Summary:
This diff turns off the support_old_nightly feature of async-trait (https://github.com/dtolnay/async-trait/blob/0.1.24/Cargo.toml#L28-L32) everywhere in fbcode. I am getting ready to remove the feature upstream. It was an alternative implementation of async-trait that produces worse error messages but supports some older toolchains dating back to before stabilization of async/await that the default implementation does not support.
This diff includes updating async-trait from 0.1.24 to 0.1.29 to pull in fixes for some patterns that used to work in the support_old_nightly implementation but not the default implementation.
Differential Revision: D20805832
fbshipit-source-id: cd34ce55b419b5408f4f7efb4377c777209e4a6d
Summary:
Instead of using Sha256::from_slice, just use Sha256::from with a correctly
sized array.
Reviewed By: quark-zju
Differential Revision: D20756181
fbshipit-source-id: 17c869325025078e4c91a564fc57ac1d9345dd15
Summary:
Update `hg status` to print errors that were returned by EdenFS's
getScmStatusV2() call, and to exit unsuccessfully if there were any errors.
Previously errors were silently ignored.
Reviewed By: quark-zju
Differential Revision: D19958503
fbshipit-source-id: cb3109df40eb86a5bf7e3818ddfb8da74d670405
Summary:
Fix the `test_status()` function to properly canonicalize the input paths, so
that it works successfully on Windows. Previously this function would fail on
Windows as some paths would end up as UNC-style paths and some would be plain
paths, causing the equality comparison to fail even though the paths were
equivalent. This canonicalizes the repo path so that they both use the same
format.
Reviewed By: quark-zju
Differential Revision: D20662921
fbshipit-source-id: fdd36bac755f9694b4a482615d3dca43ff21e05e
Summary:
All of the callers are already using an Arc, so instead of forcing the remote
store to be cloneable, and thus wrap an inner self with an Arc, let's just pass
self as an Arc.
Reviewed By: DurhamG
Differential Revision: D20715580
fbshipit-source-id: 1bef23ae7da7b314d99cb3436a94d04134f1c0e4
Summary:
When LFS will be enabled on fbsource, the enablement will rolled out server,
with the server serving pointers (or not). In the catastrophic scenario where
Mononoke has to be rolled out, the Mononoke LFS server will be unable to serve
blobs, but some clients may still have LFS pointers locally but not the
corresponding blob. For this, we need to be able to fallback to fetching the
blob via the getpackv2 protocol.
Reviewed By: DurhamG
Differential Revision: D20662667
fbshipit-source-id: 4ac45558f6d205cbd1db33c21c6fb137a81bdbd5
Summary:
The LFS server might be temporarily having issues, let's retry a bit before
giving up.
Reviewed By: DurhamG
Differential Revision: D20686659
fbshipit-source-id: 90dabd19e45a681d6eae5cd50c72b635d44c0517
