Summary:
The Rc is required by the c_api, but there is no longer a reason for
UnionDataStore and UnionHistoryStore to use an Rc, so let's move the Rc into
c_api.
Reviewed By: DurhamG
Differential Revision: D13928332
fbshipit-source-id: a93b54e022d539dc4df9144a8c59e9ffbe3453e0
Summary:
By specifying the IntoIterator differently, we can avoid the clone requirement.
Since Clone isn't implemented on either DataPack or HistoryPack, this will
simplify the callers a bit
Reviewed By: DurhamG
Differential Revision: D13928274
fbshipit-source-id: f0261c50d73868689ebb3ae226f84d41c4c40925
Summary: This way, HistoryStore type constraint will work with these types.
Reviewed By: DurhamG
Differential Revision: D13928128
fbshipit-source-id: aaa9f2633166c137dca5fc2b1f44caab92b57a80
Summary: This way, DataStore type constraint will work with these types.
Reviewed By: DurhamG
Differential Revision: D13928090
fbshipit-source-id: 1567556e3ffea2901acbc754b3bd67491e23056b
Summary: The UnionStore doesn't need internal mutability, so let's simplify it.
Reviewed By: DurhamG
Differential Revision: D13928058
fbshipit-source-id: f0ba085ff8401dcc99fc69c3eb6f5e20c071d650
Summary: This just reuses the AsyncHistoryStore methods.
Reviewed By: DurhamG
Differential Revision: D13891142
fbshipit-source-id: 9553e9824eebc5eacf6a82f9d0f212a62ec8955f
Summary:
Similarly to AsyncDataStore, this is just a blocking wrapper around a
HistoryStore.
Reviewed By: DurhamG
Differential Revision: D13891140
fbshipit-source-id: 76acadfc1849770b47e2400ce8c70f7e32bba4df
Summary: This will be used to wrap an HistoryStore into a AsyncHistoryStore.
Reviewed By: DurhamG
Differential Revision: D13891139
fbshipit-source-id: 41a0ec740f05268259a654e769ff0909617102ff
Summary: Add metadata to each delta entry written to the datapack. Since the HTTP API never serves LFS files, and the only flag currently used simple indicates whether a file should use LFS, the flag field is intentionally set to `None`, leaving only the size in the metadata (which, since we're storing full file content, is the same as the content length).
Differential Revision: D13894292
fbshipit-source-id: 36db25adb0c46cd1c7fde841a69d3e6d48d08d06
Summary: Give MononokeClient the ability to fetch multiple files concurrently. Right now this functionality is not exposed via the Python bindings, so as far as the Mercurial Python code is concerned, nothing has changed. The multi-get functionality will be used later in the stack.
Differential Revision: D13893575
fbshipit-source-id: c9e514fbeb41bbb37f52f6df3920eb01a66df293
Summary: As `MononokeClient` grows, we're going to add more inherent methods on the struct. To avoid cluttering the `client` module, split out all the builder-related things into a separate module.
Reviewed By: singhsrb
Differential Revision: D13892198
fbshipit-source-id: 42918d8a775d8328cfad8a6ac0365cb336893d8f
Summary: Add a new `get_file()` method to `MononokeClient` that fetches Mercurial file content from the API server and writes it to a datapack in the cache. This functionality is exposed via the new `hg debuggetfile` debug command, which takes a filenode and file path and fetches the corresponding file.
Differential Revision: D13889829
fbshipit-source-id: 2b68bf114ee72d641de7a1043cca1975e34cf4e6
Summary:
Crate adding easy conversions between `http::Uri` and `url::Url`.
Rust has two main types for working with URLs: `http::Uri` and `url::Url`. `http::Uri` comes from the `http` crate, which is supposed to be a set of common types to be used throughout the Rust HTTP ecosystem, to ensure mutual compatibility between different HTTP crates and web frameworks. This is the type that HTTP clients like Hyper expect when specifying URLs.
Unfortunately, `http::Uri` is a very simple type that does not expose any means of mutating or otherwise manipulating the URL. It can only parse URLs from strings, forcing the users to construct URLs via error-prone string concatenation.
In contrast, the `url::Url` comes from the `rust-url` crate from the Servo project. This type does support easily constructing and manipulating URLs, making it very useful for assembling a URL from components.
The only way to convert between the two types is to first convert back to a string, and then re-parse as the desired type. Several issues [have](https://github.com/hyperium/hyper/issues/1219) [been](https://github.com/hyperium/hyper/issues/1102) [raised](https://github.com/hyperium/hyper/issues/1219) about this upstream, but there has been no consensus or action as of yet. To get around the problem for now, this crate adds convenience methods to perform the conversions.
Reviewed By: DurhamG
Differential Revision: D13887403
fbshipit-source-id: ecfaf3ea9d884621493b0fe44a6b5658d10108b4
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary: Some of the revisionstore imports were unused.
Reviewed By: kulshrax
Differential Revision: D13865074
fbshipit-source-id: 79c7c2ba869f2e1d72fa06aac70a4b027367c831
Summary: Similar to previous diff in this stack, make this type serializable so we can send it as part of an HTTP request.
Reviewed By: singhsrb
Differential Revision: D13858440
fbshipit-source-id: 9173a3e76bcfa6a6600d30ada39d65475f95bc5e
Summary: Make this type serializable so it can be sent as part of an HTTP request. By using Serde, we can easily support a variety of serialization formats without code changes.
Reviewed By: singhsrb
Differential Revision: D13858443
fbshipit-source-id: b6c83f38eaadbb2a28be6d66faf6a3610ede970f
Summary:
The conditional if statement did not prevent the logic inside the
condition from being compiled, which in this case fails on windows. Instead of
using an if, let's just define two functions and conditionally compile the
functions.
Reviewed By: ikostia
Differential Revision: D13855560
fbshipit-source-id: ac417e6bd8fb272106fe8f3b9a8b7db57214ad88
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
As a last step towards getting rid of loosefiles, memcache will soon be changed
to produce packfiles. One of the missing piece to achieve is the ability to
read and write packfiles asynchronously, as memcache is purely async.
As a first step, we can wrap the packfile into a blocking context.
Reviewed By: DurhamG
Differential Revision: D13806738
fbshipit-source-id: 2211c2a984a453edbb1647830f7f5fb399a03023
Summary:
As a last step towards getting rid of loosefiles, memcache will soon be changed
to produce packfiles. One of the missing piece to achieve is the ability to
read and write packfiles asynchronously, as memcache is purely async.
As a first step, we can wrap the packfile into a blocking context.
Reviewed By: DurhamG
Differential Revision: D13804184
fbshipit-source-id: 01fcb57af1558feca662b1070969f553c479871a
Summary:
The tempfile rust crates opens the file with RW permissions for the user only,
but once written out to disk, the permissions needs to be readable by everyone.
Unfortunately, rust doesn't have a portable way of doing this, so we have to
resort to using `if cfg!(unix)` conditions for doing this.
Reviewed By: DurhamG
Differential Revision: D13703406
fbshipit-source-id: 688bc679b5c1a7943ceab723c1f649d555b61a7a
Summary:
This allows de-duplicating the logic for setting proper permissions on the
files. Most of the changes is code movement and rustfmt formatting.
Reviewed By: DurhamG
Differential Revision: D13703392
fbshipit-source-id: 28be85ef2d4b440202cf4885e50e62ac3c41f774
Summary: Allow the credentials for TLS mutual authentication (namely, the client certificate and private key) to come from separate PEM files. At Facebook, these are usually stored in the same file, but Mercurial's standard TLS configuration options allow these to be configured separately. As such, in order to support the standard options (which will happen in a later diff), provide the ability to handle separate files, but for now just pass the same path for both from Python to Rust.
Reviewed By: markbt
Differential Revision: D13791525
fbshipit-source-id: 556d99d77a4273b9b0bd91cac8940da136088e45
Summary: Use a builder struct rather than a constructor function to configure and initialize new `MononokeClient` instances. Doing it this way is helpful because later in this stack, we'll need to pass a lot of additional configuration to `MononokeClient`; adding all of these items as parameters to the constructor quickly becomes unwieldily. Using a builder keeps the number of parameters in check.
Differential Revision: D13780408
fbshipit-source-id: bfc43ecbe474d5285ae87d4df9cce244a7ff391d
Summary:
Split up the functionality in `MononokeClient` by moving all of the Mononoke API methods to their own separate trait. This maintains a distinction between functionality that is part of the API vs methods for setting up and configuring the client.
Originally, I had tried to avoid using a trait here because of limitations on trait methods (for example, we can't use `impl Trait` for return types). In practice, I don't think this limitation will be an issue since the API exposed by the client needs to be synchronous (since it will be called by FFI bindings to Python), and as such, there shouldn't be any complex Future return types in the API. (The client will still use async code internally, but the external API will be synchronous.)
Differential Revision: D13780089
fbshipit-source-id: 17e80f549d6ac7c41c60b2b8389eb1760531883e
Summary: Boxed slices are difficult to use in practice, so use `Vec<u8>` instead. (No need for `Bytes` here since there is no reference counting required.)
Reviewed By: DurhamG
Differential Revision: D13770055
fbshipit-source-id: 78f48ac32a4da9c105bf05eb44889c1f492721a8
Summary: Use `Bytes` instead of `Rc<Box<[u8]>>` since the former is a nicer type to represent a reference counted heap allocated byte buffer. (Note that `Rc<Box<[u8]>>` should have originally been `Rc<[u8]>` -- the former introduces an unnecessary allocation and layer of indirection.)
Differential Revision: D13769306
fbshipit-source-id: 5f3e788426e28c7e9ccc478f993c717b23663f56
Summary: Boxed bytes slices (e.g., `Box<[u8]>`, `Rc<[u8]>`) are not very ergonomic to use and are somewhat unusual in Rust code. Use the more common and easier to use `Bytes` type instead. Since this type supports shallow, referenced-counted copies, there shouldn't be any new O(n) copying behavior compared to `Rc<[u8]>`.
Reviewed By: markbt
Differential Revision: D13754730
fbshipit-source-id: d5fbc8e39c84c56d30174f4bb194ee21a14bf944
Summary: Use `failure::Fallible<T>` in place of `Result<T, failure::Error>`.
Reviewed By: singhsrb
Differential Revision: D13754688
fbshipit-source-id: cfbe418f5213884816d4837d1077cd90a17359b6
Summary:
Previously, `use` statements were inconsistently and arbitrarily grouped. This diff groups them in the following order:
- 3rd party crates from crates.io
- local crates
- std library imports (collapsed into a single multiline `use` statement)
- modules within current crate
This new ordering ensures that upon migration to Rust 2018, all imports from within the current crate will be grouped together with the `crate::` prefix.
Reviewed By: singhsrb
Differential Revision: D13754393
fbshipit-source-id: e774c09e0547066afa5f797c1a9c2e5ec4190834
Summary: Run the latest version of rustfmt over the code to ensure consistent style.
Reviewed By: singhsrb
Differential Revision: D13754394
fbshipit-source-id: 6cf5937bcb642530bdf41aaf83399366a9ba3c9a
Summary: There were some warnings about unused private fields in various structs in this crate. Add `#[allow(dead_code)]` as needed to suppress these warnings.
Reviewed By: singhsrb
Differential Revision: D13754234
fbshipit-source-id: ca95a2afbfc67ddb66e7c7436c81cde0fa59f06c
Summary:
Use the `Fallible` type alias provided by `failure` rather than defining our
own.
Differential Revision: D13732298
fbshipit-source-id: 2577bc4c34da5b7a88ae2703f9b898bc2a83b816
Summary: The canonical URL type in Rust, `http::Uri`, does not support manipulating URLs easily. (e.g., concatenating path components, etc.) As such, switch to using the `Url` type from the `url` crate, which does support URL manipulation, and convert to `http::Uri` before passing the resulting URL to Hyper.
Reviewed By: phillco
Differential Revision: D13738139
fbshipit-source-id: c7de67f1596ebc1bdde89d3fe87086f49c32b5db
Summary:
Directory listing is different in every OS, and due to the current repack
implementation, this directly affect the order in which the packfiles are added
to the new one. Since the resulting packfile name depends on the hash of its
content, the name was influenced by the directory order.
By sorting the files in list_packs, the packfile name will be independent of
the directory listing and thus be the same for all the OSes.
Reviewed By: singhsrb
Differential Revision: D13700935
fbshipit-source-id: 01e055a0c1bcf7fb2dc4faf614dfb20cd4499017
Summary: For now, combine all files smaller than 100MB that accumulate to less than 4GB.
Reviewed By: DurhamG
Differential Revision: D13603760
fbshipit-source-id: 3fa74f1ced3d3ccd463af8f187ef5e0254e1820b
Summary: Use the newly introduced PackWriter to write the {data,history}packs.
Reviewed By: markbt
Differential Revision: D13603759
fbshipit-source-id: 528a6af7c4ac3321aeec0559805de12114224cfd
Summary:
The packfiles are currently being written via an unbuffered file. This is
inefficient as every write to the file results results in a write(2) syscall.
By buffering these writes we can reduce the number of syscalls and thus
increase the throughput of pack writing operations.
Reviewed By: markbt
Differential Revision: D13603758
fbshipit-source-id: 649186a852d427a1473695b1d32cc9cd87a74a75
Summary:
Update pest to 2.1.0.
This version has a new behaviour for parser error messages: the line feed at
the end of the line is shown in the error output.
Reviewed By: wez
Differential Revision: D13671099
fbshipit-source-id: b8d1142a44a56a0b21b3b72cf027f3f8a30f421e
Summary:
The revisionstore crate currently consists of several public submodules,
each exposing several public types. The APIs exposed by each of the modules
require using types from the other modules. As such, users of this crate are
forced to have complex nested imports to use any of its functionality.
This diff helps ease this problem by reexporting the public types exposed from
each of the public submodules at the top level, thereby allowing crate users to
`use` all of the required types without needing nested imports.
Reviewed By: singhsrb
Differential Revision: D13686913
fbshipit-source-id: 9fb3cce8783787aa5f3f974c7168afada5952712
Summary:
The later tries to read from the disk, while the former is purely in memory and
thus more efficient.
Reviewed By: DurhamG, markbt
Differential Revision: D13603757
fbshipit-source-id: 5fd120ba4065d6a65cb2982db9ab81db3ea26524
