Summary:
Split up the functionality in `MononokeClient` by moving all of the Mononoke API methods to their own separate trait. This maintains a distinction between functionality that is part of the API vs methods for setting up and configuring the client.
Originally, I had tried to avoid using a trait here because of limitations on trait methods (for example, we can't use `impl Trait` for return types). In practice, I don't think this limitation will be an issue since the API exposed by the client needs to be synchronous (since it will be called by FFI bindings to Python), and as such, there shouldn't be any complex Future return types in the API. (The client will still use async code internally, but the external API will be synchronous.)
Differential Revision: D13780089
fbshipit-source-id: 17e80f549d6ac7c41c60b2b8389eb1760531883e
Summary: Boxed slices are difficult to use in practice, so use `Vec<u8>` instead. (No need for `Bytes` here since there is no reference counting required.)
Reviewed By: DurhamG
Differential Revision: D13770055
fbshipit-source-id: 78f48ac32a4da9c105bf05eb44889c1f492721a8
Summary: Use `Bytes` instead of `Rc<Box<[u8]>>` since the former is a nicer type to represent a reference counted heap allocated byte buffer. (Note that `Rc<Box<[u8]>>` should have originally been `Rc<[u8]>` -- the former introduces an unnecessary allocation and layer of indirection.)
Differential Revision: D13769306
fbshipit-source-id: 5f3e788426e28c7e9ccc478f993c717b23663f56
Summary: Boxed bytes slices (e.g., `Box<[u8]>`, `Rc<[u8]>`) are not very ergonomic to use and are somewhat unusual in Rust code. Use the more common and easier to use `Bytes` type instead. Since this type supports shallow, referenced-counted copies, there shouldn't be any new O(n) copying behavior compared to `Rc<[u8]>`.
Reviewed By: markbt
Differential Revision: D13754730
fbshipit-source-id: d5fbc8e39c84c56d30174f4bb194ee21a14bf944
Summary:
Since the gc command is specified with norepo=True, the dispatch code will
never try to open a repo, and thus will never try to close it. One important
thing that repo.close() does is to commit all the pending pack files that may
have been created and written to.
In most cases, gc doesn't attempt to create a pack file, as it merely look at
the cache files to keep to not remove used ones. However, this operation may
involve fetching pack files, for instance if the caches were nuked. In this
situation, gc will have to download data from the server, and thus create
temporary pack file to store that data. Since the gc code is never attempting
to close the repo files, these temporary files would never be commited (or
removed), and thus would just lie around.
Reviewed By: DurhamG
Differential Revision: D13738476
fbshipit-source-id: c9823bfbdfd8b97a58e7c8693dffa471b276d677
Summary: Use `failure::Fallible<T>` in place of `Result<T, failure::Error>`.
Reviewed By: singhsrb
Differential Revision: D13754688
fbshipit-source-id: cfbe418f5213884816d4837d1077cd90a17359b6
Summary:
Previously, `use` statements were inconsistently and arbitrarily grouped. This diff groups them in the following order:
- 3rd party crates from crates.io
- local crates
- std library imports (collapsed into a single multiline `use` statement)
- modules within current crate
This new ordering ensures that upon migration to Rust 2018, all imports from within the current crate will be grouped together with the `crate::` prefix.
Reviewed By: singhsrb
Differential Revision: D13754393
fbshipit-source-id: e774c09e0547066afa5f797c1a9c2e5ec4190834
Summary: Run the latest version of rustfmt over the code to ensure consistent style.
Reviewed By: singhsrb
Differential Revision: D13754394
fbshipit-source-id: 6cf5937bcb642530bdf41aaf83399366a9ba3c9a
Summary: There were some warnings about unused private fields in various structs in this crate. Add `#[allow(dead_code)]` as needed to suppress these warnings.
Reviewed By: singhsrb
Differential Revision: D13754234
fbshipit-source-id: ca95a2afbfc67ddb66e7c7436c81cde0fa59f06c
Summary:
Before this diff, if `sys.path` contains a path with `..`, we might fail to
remove `filedir` from it, which leads to `mercurial` dir being present in
`sys.path` and for things like `import json` importing the wrong module.
Reviewed By: markbt
Differential Revision: D13597432
fbshipit-source-id: 403ac13a96c2cf85dec3a9e77894000505f81807
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13732298
fbshipit-source-id: 2577bc4c34da5b7a88ae2703f9b898bc2a83b816
Summary:
Replace the `-w` option of the commit cloud commands that process workspace
names with general options of the form:
* `-u` or `--user` - to select the workspaces of a particular user.
* `-w` or `--workspace` - to select a particular workspace for that user.
A (hidden) `--raw-workspace` option is provided to allow use a workspace
by its exact name, should that be needed.
The `-w` option still accepts full user workspaces, but shows a warning.
These options apply uniformly to any command that takes a workspace.
Reviewed By: liubov-dmitrieva
Differential Revision: D13730839
fbshipit-source-id: d1b884b100caf909619511db4861eb1b880a0d3e
Summary: The canonical URL type in Rust, `http::Uri`, does not support manipulating URLs easily. (e.g., concatenating path components, etc.) As such, switch to using the `Url` type from the `url` crate, which does support URL manipulation, and convert to `http::Uri` before passing the resulting URL to Hyper.
Reviewed By: phillco
Differential Revision: D13738139
fbshipit-source-id: c7de67f1596ebc1bdde89d3fe87086f49c32b5db
Summary:
Commit cloud maintains the infinitepush backup state to match what it has
synced to commit cloud. It should only include locally backed-up heads.
When syncing with the server and omitting some heads, don't include the omitted
ones.
Reviewed By: liubov-dmitrieva
Differential Revision: D13719477
fbshipit-source-id: 817c1a73a34af6b5550f6254e2b08d01283760a5
Summary:
If one of the cache is somehow corrupted, we won't be garbage collecting the
other caches, potentially leading to an increase in cache sizes and disk usage.
For now, let's just log the error and continue.
Reviewed By: singhsrb
Differential Revision: D13718419
fbshipit-source-id: 25a26e43fc193a0439f73f0145ca3c5638d4325b
Summary:
When commit cloud sync is running, record the progress of the sync operation in
a progress file in the `.hg` directory.
When displaying in smartlog that a background sync is in progress, describe
what the background sync is doing.
Reviewed By: liubov-dmitrieva
Differential Revision: D13716457
fbshipit-source-id: b6db5bf7cf12aa0e5defd9d8aab6fda32be42a50
Summary:
We should not be attaching copy headers to lfs metadata blobs since
they store the data separately. All this logic was a little overly complicated.
The file text we receive should be passed through unmodified in all cases except
when storing in the loose file format, which needs to strip the copy metadata
from the blob.
Reviewed By: ikostia
Differential Revision: D13703708
fbshipit-source-id: d7e4eb703599d2355009efe070833013d1fcd4fc
Summary:
Enabling packlocaldata is causing local move data contents to be
corrupted by the copy header being tacked on the front. Let's add a test
demonstrating the failure.
Reviewed By: ikostia
Differential Revision: D13703709
fbshipit-source-id: d891042a5cab639a4533a89a508865115c482ba1
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
Directory listing is different in every OS, and due to the current repack
implementation, this directly affect the order in which the packfiles are added
to the new one. Since the resulting packfile name depends on the hash of its
content, the name was influenced by the directory order.
By sorting the files in list_packs, the packfile name will be independent of
the directory listing and thus be the same for all the OSes.
Reviewed By: singhsrb
Differential Revision: D13700935
fbshipit-source-id: 01e055a0c1bcf7fb2dc4faf614dfb20cd4499017
Summary:
This ports the logic from `eden/hg/eden/` to `scm/hg/mercurial/`.
Note this does not delete the logic from `eden/hg/eden` as part of this
change because we may continue to do Eden releases before we roll out a
version of Hg with this code. Only once Hg has been rolled out everywhere
[that is using Eden] can we consider removing
`/usr/local/fb-mercurial/eden/hgext3rd/eden`.
Reviewed By: quark-zju
Differential Revision: D10316761
fbshipit-source-id: cae1dfad831ad6505590628cf969897167e84b30
Summary:
The user, description, and bookmarks must be converted from the local encoding
before they can be included in the JSON object, as JSON can only contain UTF-8
strings.
Note that `crdump` will not work with filenames that are not UTF-8 encoded.
This doesn't fix that.
Reviewed By: HarveyHunt
Differential Revision: D13635999
fbshipit-source-id: 5379656e39b73d7d7f905d06de50851f17c54e8b
Summary: For now, combine all files smaller than 100MB that accumulate to less than 4GB.
Reviewed By: DurhamG
Differential Revision: D13603760
fbshipit-source-id: 3fa74f1ced3d3ccd463af8f187ef5e0254e1820b
Summary: Use the newly introduced PackWriter to write the {data,history}packs.
Reviewed By: markbt
Differential Revision: D13603759
fbshipit-source-id: 528a6af7c4ac3321aeec0559805de12114224cfd
Summary:
The packfiles are currently being written via an unbuffered file. This is
inefficient as every write to the file results results in a write(2) syscall.
By buffering these writes we can reduce the number of syscalls and thus
increase the throughput of pack writing operations.
Reviewed By: markbt
Differential Revision: D13603758
fbshipit-source-id: 649186a852d427a1473695b1d32cc9cd87a74a75
Summary:
Update pest to 2.1.0.
This version has a new behaviour for parser error messages: the line feed at
the end of the line is shown in the error output.
Reviewed By: wez
Differential Revision: D13671099
fbshipit-source-id: b8d1142a44a56a0b21b3b72cf027f3f8a30f421e
Summary:
The revisionstore crate currently consists of several public submodules,
each exposing several public types. The APIs exposed by each of the modules
require using types from the other modules. As such, users of this crate are
forced to have complex nested imports to use any of its functionality.
This diff helps ease this problem by reexporting the public types exposed from
each of the public submodules at the top level, thereby allowing crate users to
`use` all of the required types without needing nested imports.
Reviewed By: singhsrb
Differential Revision: D13686913
fbshipit-source-id: 9fb3cce8783787aa5f3f974c7168afada5952712
Summary:
The later tries to read from the disk, while the former is purely in memory and
thus more efficient.
Reviewed By: DurhamG, markbt
Differential Revision: D13603757
fbshipit-source-id: 5fd120ba4065d6a65cb2982db9ab81db3ea26524
Summary: In D13363853, a basic rust based repack was implemented, but couldn't be called from Python. For now, this version will be called when the repack.userust config is set, and when requesting a pack-only repack, as the rust version doesn't support loose objects (?).
Reviewed By: DurhamG
Differential Revision: D13555855
fbshipit-source-id: 83d53b33f2d3b6ae3167801d0594cd1490a0a09d
Summary:
D13646642 used the `--date` flag for the `touch` command which
unfortunately does not work on our prod OSX hosts. Therefore, lets switch to
the more compatible `-t` flag.
Reviewed By: DurhamG
Differential Revision: D13673021
fbshipit-source-id: 6f9e0234e7fd3d4c345bedecdd2197949d71c420
Summary:
The progress bar for LFS doesn't specify a unit. By specifying bytes, it will
show the progress in MB or GB as appropriate.
Reviewed By: HarveyHunt
Differential Revision: D13635194
fbshipit-source-id: 8b9a17cc0ca793d2731077b05c7027525e7b215e
Summary:
Generalise the `migrateonpull` mechanism of `treestate` into a generic
`automigrate` step that is invoked at the start of pulling. This will be used
for other migrations in the future.
Reviewed By: liubov-dmitrieva
Differential Revision: D13608718
fbshipit-source-id: d558dc21176a6b8d786836d06414e3fc88a20d47
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13657313
fbshipit-source-id: ae249bc15037cc2be019ce7ce8a440c153aa31cc
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13657312
fbshipit-source-id: 55134ee93f1f3aaaeefe5644a4a1f2285603bc1c
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13657314
fbshipit-source-id: f1a379089972f7f0066c49ddedf606d36b7ac260
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13657310
fbshipit-source-id: cae73fc239a6ad30bb6ef56a664d1ef5a2a19b5f
Summary:
On some platforms, removing a file can fail if it's still mapped or opened. In
mercurial, this can happen during repack as the datapacks are removed while
still being mapped.
Reviewed By: DurhamG
Differential Revision: D13615938
fbshipit-source-id: fdc1ff9370e2767e52ee1828552f4598105f784f
Summary:
The cpython crate forces all the method to take a &self, which forbids
modification of the embedded pack datastructure. Its documentation recommends
using internal mutability for this purpose. Most of the code is wrapped to
avoid lots of boiler plate code.
Reviewed By: DurhamG
Differential Revision: D13640638
fbshipit-source-id: 3b7513b6117d429322efe32868e683239c68806e
Summary:
In D13433558 we moved lastpulled into the sqlite database for sqlite
backed revmaps. It turns out the update/rebuildmeta code circumvents the revmap
abstraction and attempts to read the lastpulled file directly from disk. In a
sql backed world, this attempt fails and it then rebuilds the entire revmap
which is very slow.
There was a comment stating the code was intentionally not reading from the
revmap, but I don't believe it applies anymore, and the tests pass fine with
this new change.
Reviewed By: singhsrb
Differential Revision: D13662697
fbshipit-source-id: 2db8f346d89053604d34fbda8f531f688cf71210
Summary:
We've observed some users with very large hgcache directories that were filled
with temporary pack files that for some reasons were not removed/renamed on a
previous repack.
These files can appear due to a variety of reasons, such as forcibly killing
hg, or a host power-off, or simply due to a bug in mercurial. It is likely that
the later case is what causes some of the hgcache directories to grow and this
patch doesn't attempt on finding the underlying mercurial issue. Rather, let's
alleviate the issue by simply removing the temporary files older than 24h.
Reviewed By: ikostia
Differential Revision: D13646642
fbshipit-source-id: faa0605e322d440a75187e2517cbbcb13031dae0
Summary: If the there is no changeset left after sparse skip in `--command` mode, show the result and return, as it's done in manual testing mode.
Reviewed By: markbt
Differential Revision: D13650568
fbshipit-source-id: 8e867a38858d84d9a10078b74e2087318c81b01e
Summary: Adding new test to show that sparse skip with `--command` doesn't show the correct answer
Reviewed By: markbt
Differential Revision: D13650567
fbshipit-source-id: f4b6670fe67d6ef2543efedd91d9760e1e6bc74c
Summary:
After repacking the data/history packs, we need to cleanup the
repacked files. This was an omission from D13363853.
Reviewed By: markbt
Differential Revision: D13577592
fbshipit-source-id: 36e7d5b8e86affe47cdd10d33a769969f02b8a62
Summary:
The python version of the mutable packs set the permission to read-only after
writing them, while the rust version keeps them writeable. Let's make the rust
one more consistent.
Reviewed By: markbt
Differential Revision: D13573572
fbshipit-source-id: 61256994562aa09058a88a7935c16dfd7ddf9d18
