Summary: Similar to previous diff in this stack, make this type serializable so we can send it as part of an HTTP request.
Reviewed By: singhsrb
Differential Revision: D13858440
fbshipit-source-id: 9173a3e76bcfa6a6600d30ada39d65475f95bc5e
Summary: Make this type serializable so it can be sent as part of an HTTP request. By using Serde, we can easily support a variety of serialization formats without code changes.
Reviewed By: singhsrb
Differential Revision: D13858443
fbshipit-source-id: b6c83f38eaadbb2a28be6d66faf6a3610ede970f
Summary:
The conditional if statement did not prevent the logic inside the
condition from being compiled, which in this case fails on windows. Instead of
using an if, let's just define two functions and conditionally compile the
functions.
Reviewed By: ikostia
Differential Revision: D13855560
fbshipit-source-id: ac417e6bd8fb272106fe8f3b9a8b7db57214ad88
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
As a last step towards getting rid of loosefiles, memcache will soon be changed
to produce packfiles. One of the missing piece to achieve is the ability to
read and write packfiles asynchronously, as memcache is purely async.
As a first step, we can wrap the packfile into a blocking context.
Reviewed By: DurhamG
Differential Revision: D13806738
fbshipit-source-id: 2211c2a984a453edbb1647830f7f5fb399a03023
Summary:
As a last step towards getting rid of loosefiles, memcache will soon be changed
to produce packfiles. One of the missing piece to achieve is the ability to
read and write packfiles asynchronously, as memcache is purely async.
As a first step, we can wrap the packfile into a blocking context.
Reviewed By: DurhamG
Differential Revision: D13804184
fbshipit-source-id: 01fcb57af1558feca662b1070969f553c479871a
Summary:
The tempfile rust crates opens the file with RW permissions for the user only,
but once written out to disk, the permissions needs to be readable by everyone.
Unfortunately, rust doesn't have a portable way of doing this, so we have to
resort to using `if cfg!(unix)` conditions for doing this.
Reviewed By: DurhamG
Differential Revision: D13703406
fbshipit-source-id: 688bc679b5c1a7943ceab723c1f649d555b61a7a
Summary:
This allows de-duplicating the logic for setting proper permissions on the
files. Most of the changes is code movement and rustfmt formatting.
Reviewed By: DurhamG
Differential Revision: D13703392
fbshipit-source-id: 28be85ef2d4b440202cf4885e50e62ac3c41f774
Summary:
reconnect logic was not working properly (at least with commitcloud.internal.tfbnw.net:443)
users of hg cloud sync --full could hit it when too many work (more than 5
minutes) happens between get_references and update_references requests.
close()
connect()
seems to work
Reviewed By: suitingtseng
Differential Revision: D13818570
fbshipit-source-id: 6829da60e5b6071a9020c5f7773cd0402df3b294
Summary: This allows us to run pushbackup and cloud sync commands for Read Only Mononoke repos.
Reviewed By: ikostia
Differential Revision: D13804545
fbshipit-source-id: 8026fc4668afc8bb5c2c0a9587ca024e3c6920da
Summary:
Loosefiles are still being fetched from memcache, and thus until we switch
memcache to provide pack files, repack needs to be able to handle loosefiles.
We could either implement the loosefile repack logic in rust, or we could just
let the python code do it. Since we plan on having memcache serve pack files
soon, the second approach is prefered as it result in the least amount of work.
Reviewed By: DurhamG
Differential Revision: D13707886
fbshipit-source-id: 5373b61f9749c158e6dfb026f6b12a7dd2a67802
Summary: Allow the credentials for TLS mutual authentication (namely, the client certificate and private key) to come from separate PEM files. At Facebook, these are usually stored in the same file, but Mercurial's standard TLS configuration options allow these to be configured separately. As such, in order to support the standard options (which will happen in a later diff), provide the ability to handle separate files, but for now just pass the same path for both from Python to Rust.
Reviewed By: markbt
Differential Revision: D13791525
fbshipit-source-id: 556d99d77a4273b9b0bd91cac8940da136088e45
Summary:
Add Python doctests for `util.ring` and fix the bugs in the `pop` and `items`
methods.
Reviewed By: ikostia
Differential Revision: D13803352
fbshipit-source-id: f14acebf4f86da25b9ef5a8238df66cb1a7ebd06
Summary:
The bottom revset can be made faster by switching the order of the
operands to the `&` operation, as this triggers the `_phaseandancestors`
optimization.
Reviewed By: liubov-dmitrieva
Differential Revision: D13800853
fbshipit-source-id: 51d9b65bea1f3a5c8d1ba456d46fe3d3fc03314b
Summary: Use a builder struct rather than a constructor function to configure and initialize new `MononokeClient` instances. Doing it this way is helpful because later in this stack, we'll need to pass a lot of additional configuration to `MononokeClient`; adding all of these items as parameters to the constructor quickly becomes unwieldily. Using a builder keeps the number of parameters in check.
Differential Revision: D13780408
fbshipit-source-id: bfc43ecbe474d5285ae87d4df9cce244a7ff391d
Summary:
Split up the functionality in `MononokeClient` by moving all of the Mononoke API methods to their own separate trait. This maintains a distinction between functionality that is part of the API vs methods for setting up and configuring the client.
Originally, I had tried to avoid using a trait here because of limitations on trait methods (for example, we can't use `impl Trait` for return types). In practice, I don't think this limitation will be an issue since the API exposed by the client needs to be synchronous (since it will be called by FFI bindings to Python), and as such, there shouldn't be any complex Future return types in the API. (The client will still use async code internally, but the external API will be synchronous.)
Differential Revision: D13780089
fbshipit-source-id: 17e80f549d6ac7c41c60b2b8389eb1760531883e
Summary: Boxed slices are difficult to use in practice, so use `Vec<u8>` instead. (No need for `Bytes` here since there is no reference counting required.)
Reviewed By: DurhamG
Differential Revision: D13770055
fbshipit-source-id: 78f48ac32a4da9c105bf05eb44889c1f492721a8
Summary: Use `Bytes` instead of `Rc<Box<[u8]>>` since the former is a nicer type to represent a reference counted heap allocated byte buffer. (Note that `Rc<Box<[u8]>>` should have originally been `Rc<[u8]>` -- the former introduces an unnecessary allocation and layer of indirection.)
Differential Revision: D13769306
fbshipit-source-id: 5f3e788426e28c7e9ccc478f993c717b23663f56
Summary: Boxed bytes slices (e.g., `Box<[u8]>`, `Rc<[u8]>`) are not very ergonomic to use and are somewhat unusual in Rust code. Use the more common and easier to use `Bytes` type instead. Since this type supports shallow, referenced-counted copies, there shouldn't be any new O(n) copying behavior compared to `Rc<[u8]>`.
Reviewed By: markbt
Differential Revision: D13754730
fbshipit-source-id: d5fbc8e39c84c56d30174f4bb194ee21a14bf944
Summary:
Since the gc command is specified with norepo=True, the dispatch code will
never try to open a repo, and thus will never try to close it. One important
thing that repo.close() does is to commit all the pending pack files that may
have been created and written to.
In most cases, gc doesn't attempt to create a pack file, as it merely look at
the cache files to keep to not remove used ones. However, this operation may
involve fetching pack files, for instance if the caches were nuked. In this
situation, gc will have to download data from the server, and thus create
temporary pack file to store that data. Since the gc code is never attempting
to close the repo files, these temporary files would never be commited (or
removed), and thus would just lie around.
Reviewed By: DurhamG
Differential Revision: D13738476
fbshipit-source-id: c9823bfbdfd8b97a58e7c8693dffa471b276d677
Summary: Use `failure::Fallible<T>` in place of `Result<T, failure::Error>`.
Reviewed By: singhsrb
Differential Revision: D13754688
fbshipit-source-id: cfbe418f5213884816d4837d1077cd90a17359b6
Summary:
Previously, `use` statements were inconsistently and arbitrarily grouped. This diff groups them in the following order:
- 3rd party crates from crates.io
- local crates
- std library imports (collapsed into a single multiline `use` statement)
- modules within current crate
This new ordering ensures that upon migration to Rust 2018, all imports from within the current crate will be grouped together with the `crate::` prefix.
Reviewed By: singhsrb
Differential Revision: D13754393
fbshipit-source-id: e774c09e0547066afa5f797c1a9c2e5ec4190834
Summary: Run the latest version of rustfmt over the code to ensure consistent style.
Reviewed By: singhsrb
Differential Revision: D13754394
fbshipit-source-id: 6cf5937bcb642530bdf41aaf83399366a9ba3c9a
Summary: There were some warnings about unused private fields in various structs in this crate. Add `#[allow(dead_code)]` as needed to suppress these warnings.
Reviewed By: singhsrb
Differential Revision: D13754234
fbshipit-source-id: ca95a2afbfc67ddb66e7c7436c81cde0fa59f06c
Summary:
Before this diff, if `sys.path` contains a path with `..`, we might fail to
remove `filedir` from it, which leads to `mercurial` dir being present in
`sys.path` and for things like `import json` importing the wrong module.
Reviewed By: markbt
Differential Revision: D13597432
fbshipit-source-id: 403ac13a96c2cf85dec3a9e77894000505f81807
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13732298
fbshipit-source-id: 2577bc4c34da5b7a88ae2703f9b898bc2a83b816
Summary:
Replace the `-w` option of the commit cloud commands that process workspace
names with general options of the form:
* `-u` or `--user` - to select the workspaces of a particular user.
* `-w` or `--workspace` - to select a particular workspace for that user.
A (hidden) `--raw-workspace` option is provided to allow use a workspace
by its exact name, should that be needed.
The `-w` option still accepts full user workspaces, but shows a warning.
These options apply uniformly to any command that takes a workspace.
Reviewed By: liubov-dmitrieva
Differential Revision: D13730839
fbshipit-source-id: d1b884b100caf909619511db4861eb1b880a0d3e
Summary: The canonical URL type in Rust, `http::Uri`, does not support manipulating URLs easily. (e.g., concatenating path components, etc.) As such, switch to using the `Url` type from the `url` crate, which does support URL manipulation, and convert to `http::Uri` before passing the resulting URL to Hyper.
Reviewed By: phillco
Differential Revision: D13738139
fbshipit-source-id: c7de67f1596ebc1bdde89d3fe87086f49c32b5db
Summary:
Commit cloud maintains the infinitepush backup state to match what it has
synced to commit cloud. It should only include locally backed-up heads.
When syncing with the server and omitting some heads, don't include the omitted
ones.
Reviewed By: liubov-dmitrieva
Differential Revision: D13719477
fbshipit-source-id: 817c1a73a34af6b5550f6254e2b08d01283760a5
Summary:
If one of the cache is somehow corrupted, we won't be garbage collecting the
other caches, potentially leading to an increase in cache sizes and disk usage.
For now, let's just log the error and continue.
Reviewed By: singhsrb
Differential Revision: D13718419
fbshipit-source-id: 25a26e43fc193a0439f73f0145ca3c5638d4325b
Summary:
When commit cloud sync is running, record the progress of the sync operation in
a progress file in the `.hg` directory.
When displaying in smartlog that a background sync is in progress, describe
what the background sync is doing.
Reviewed By: liubov-dmitrieva
Differential Revision: D13716457
fbshipit-source-id: b6db5bf7cf12aa0e5defd9d8aab6fda32be42a50
Summary:
We should not be attaching copy headers to lfs metadata blobs since
they store the data separately. All this logic was a little overly complicated.
The file text we receive should be passed through unmodified in all cases except
when storing in the loose file format, which needs to strip the copy metadata
from the blob.
Reviewed By: ikostia
Differential Revision: D13703708
fbshipit-source-id: d7e4eb703599d2355009efe070833013d1fcd4fc
Summary:
Enabling packlocaldata is causing local move data contents to be
corrupted by the copy header being tacked on the front. Let's add a test
demonstrating the failure.
Reviewed By: ikostia
Differential Revision: D13703709
fbshipit-source-id: d891042a5cab639a4533a89a508865115c482ba1
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
Directory listing is different in every OS, and due to the current repack
implementation, this directly affect the order in which the packfiles are added
to the new one. Since the resulting packfile name depends on the hash of its
content, the name was influenced by the directory order.
By sorting the files in list_packs, the packfile name will be independent of
the directory listing and thus be the same for all the OSes.
Reviewed By: singhsrb
Differential Revision: D13700935
fbshipit-source-id: 01e055a0c1bcf7fb2dc4faf614dfb20cd4499017
Summary:
This ports the logic from `eden/hg/eden/` to `scm/hg/mercurial/`.
Note this does not delete the logic from `eden/hg/eden` as part of this
change because we may continue to do Eden releases before we roll out a
version of Hg with this code. Only once Hg has been rolled out everywhere
[that is using Eden] can we consider removing
`/usr/local/fb-mercurial/eden/hgext3rd/eden`.
Reviewed By: quark-zju
Differential Revision: D10316761
fbshipit-source-id: cae1dfad831ad6505590628cf969897167e84b30
Summary:
The user, description, and bookmarks must be converted from the local encoding
before they can be included in the JSON object, as JSON can only contain UTF-8
strings.
Note that `crdump` will not work with filenames that are not UTF-8 encoded.
This doesn't fix that.
Reviewed By: HarveyHunt
Differential Revision: D13635999
fbshipit-source-id: 5379656e39b73d7d7f905d06de50851f17c54e8b
Summary: For now, combine all files smaller than 100MB that accumulate to less than 4GB.
Reviewed By: DurhamG
Differential Revision: D13603760
fbshipit-source-id: 3fa74f1ced3d3ccd463af8f187ef5e0254e1820b
Summary: Use the newly introduced PackWriter to write the {data,history}packs.
Reviewed By: markbt
Differential Revision: D13603759
fbshipit-source-id: 528a6af7c4ac3321aeec0559805de12114224cfd
Summary:
The packfiles are currently being written via an unbuffered file. This is
inefficient as every write to the file results results in a write(2) syscall.
By buffering these writes we can reduce the number of syscalls and thus
increase the throughput of pack writing operations.
Reviewed By: markbt
Differential Revision: D13603758
fbshipit-source-id: 649186a852d427a1473695b1d32cc9cd87a74a75
Summary:
Update pest to 2.1.0.
This version has a new behaviour for parser error messages: the line feed at
the end of the line is shown in the error output.
Reviewed By: wez
Differential Revision: D13671099
fbshipit-source-id: b8d1142a44a56a0b21b3b72cf027f3f8a30f421e
