Summary: We are copying `MPath` a lot, this change should reduce amount of allocations and memory reuse for `MPath`
Reviewed By: farnz
Differential Revision: D15939495
fbshipit-source-id: 8da8f2c38f7b46f27d0df661210c9964aed52101
Summary: `HgEntryId` is much more useful in a typed from (enum of `(FileType, HgFileNodeId)` and `HgManifestId`), most of the time we now which type entry should contain and it makes it harder to make and error, all other use cases which require just hash should use `HgNodeHash` instead. This diff includes minimal changes which are necessary to make it work. Some of the cases which do sequence of `Entry::get_hash().into_nondehash() -> HgManifestId::new() | HgFileNodeId::new()` are left for future diffs.
Reviewed By: farnz
Differential Revision: D15866081
fbshipit-source-id: 5be9ecc30dbfd0a49ae6c5d084cdfe2dac351dac
Summary:
Side-effect changes:
- Doc comments on macro invocations generate a warning that they don't document what the macro expands to => make them non-doc comments
- New warnings from some bad borrowing patterns which were not previously diagnosed (T45010740)
Reviewed By: Imxset21
Differential Revision: D15515133
fbshipit-source-id: a71db51433598e338b660cbf9d2079f4bd51cfba
Summary:
Mononoke and hg both have their own implementation wrappers for lz4
compression, unify these to avoid duplication.
Reviewed By: StanislavGlebik
Differential Revision: D14131430
fbshipit-source-id: 3301b755442f9bea00c650c22ea696912a4a24fd
Summary: HgFileNodeId is a stronger typed id, so it is prefered to use it instead of HgNodeHash whenever it is identifying a filenode
Reviewed By: aslpavel
Differential Revision: D13986172
fbshipit-source-id: c0334652345acb868e86c38b8c0045e9c023c176
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
Mercurial has a hack to determine if a file was renamed. If p1 is None then
copy metadata is checked. Note that this hack is purely to make finding renames
faster and we don't need it in Mononoke. So let's just read copy metadata.
This diff also removes `maybe_copied()` method and unused code like `Symlink`
Reviewed By: farnz
Differential Revision: D12826409
fbshipit-source-id: 53792218cb61fcba96144765790278d17eecdbb1
Summary:
Removed:
cmd-line cmd tool for filenodes and bookmarks. These should be a part of
mononoke_admin script
Outdates docs folder
Commitsim crate, because it's replaced by real pushrebase
unused hooks_old crate
storage crate which wasn't used
Reviewed By: aslpavel
Differential Revision: D13301035
fbshipit-source-id: 3ae398752218915dc4eb85c11be84e48168677cc
Summary:
Mononoke blobimports filenodes from Mercurial Host Machines to its blobstore.
It parses revlogs to retrieve flags used for parsing.
Revlogs Contains the flags for Marking Ext Stored FIles. This flag means that the file is not stored at the Mercurial Host machine, but somewhere remotely.
This diff provides the api to get the extstored flag from revlog.
Reviewed By: farnz
Differential Revision: D13081950
fbshipit-source-id: ba5bc04ad3659d4880960995d1cc46594d89e220
Summary:
According to [blobimport logic](diffusion/FBS/browse/master/fbcode/scm/mononoke/cmdlib/src/blobimport_lib/changeset.rs;fd16808edd6e51c1d0b82f4812fe843e797025e0$163-164) blobimport requests
for parents and node content
Previous implementation was reconstructing filenode to
current revidx for both cases: for getting raw_content and for getting parents.
New implementation avoid reconstruction of the file content to retrieve parents.
Reviewed By: quark-zju
Differential Revision: D12857440
fbshipit-source-id: e1118affe85647931dd551b9ca7be5297afe56ce
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary:
According to [Mercurial Lfs Plan](https://www.mercurial-scm.org/wiki/LfsPlan), on push, for files which size is above the threshold (lfs.threshold config) hg client is sending LFS metadata instead of actual files contents. The main part of LFS metadata is SHA-256 of the file content (oid).
The format requires the following mandatory fields: version, oid, size.
When lfs metadata is sent instead of a real file content then lfs_ext_stored flag is in the request's revflags.
If this flag is set, We are ignoring sha-1 hash verification inconsistency.
Later check that the content is actually loaded to the blobstore and create filenode envelope from it, load the envelope to the blobstore.
Filenode envelope requires the following info:
- size - retrieved on fetching the actual data from blobstore.
- copy_from - retrieved from the file, sent by hg client.
Mononoke still does the same checks for LFS push as for non-lfs push (i.e. checks that all the necessary manifests/filelogs were uploaded by a client)
Reviewed By: StanislavGlebik
Differential Revision: D10255314
fbshipit-source-id: efc8dac4c9f6d6f9eb3275d21b7b0cbfd354a736
Summary:
Add libnfs, libffi and starlark.
Also nom now has "verbose-errors" feature (via bindgen -> cexpr -> nom), so make some tweaks to cope.
Reviewed By: farnz
Differential Revision: D10371391
fbshipit-source-id: ba889ad16a7b662c5eddafcb1e705b068ccc9af7
Summary:
We used generate copy metadata which mercurial understand but works a bit differently
Relevant mercurial code:
```
def packmeta(meta, text):
keys = sorted(meta)
metatext = "".join("%s: %s\n" % (k, meta[k]) for k in keys)
return "\1\n%s\1\n%s" % (metatext, text)
```
Reviewed By: StanislavGlebik
Differential Revision: D10200805
fbshipit-source-id: f17a51a6aac2e1d3671fbbf3e969ed747e2fce18
Summary:
Add support for the storerequirements feature of Mercurial repositories, which
requires the reader to additionally check the store/requires file for store
requirements.
Reviewed By: StanislavGlebik
Differential Revision: D9850335
fbshipit-source-id: 557ea0f90f3d138d1df56edd94ee23760b9fd849
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary: Cleans things up a bit, esp when matching Context/Chain.
Reviewed By: lukaspiatkowski
Differential Revision: D9439062
fbshipit-source-id: cde8727437f58b288bed9dfacb864bdcd7dea45c
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary:
This causes `buck test` to return an error code, as it skipped this
test. It's trivial to change `extras_roundtrip` to get a longer test run if
developing this code, and I'd like to be in a place where `buck test && jf
submit` is a good idiom to use for only submitting tested code
Reviewed By: StanislavGlebik
Differential Revision: D9334941
fbshipit-source-id: 98edb18346f75b27175b00096232471839caed71
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary: These files import the asyncmemo crate but do not use it. Remove it.
Reviewed By: StanislavGlebik
Differential Revision: D8951887
fbshipit-source-id: e34f8c34a3f652156b63d795023d67260242a58e
Summary:
I don't like this because particularly the empty string for regular
files looks weird.
Reviewed By: StanislavGlebik
Differential Revision: D8888553
fbshipit-source-id: 20a9048a19b3fdfe681160a637bc2dfc8932c113
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
Shouldn't return Symlink for MPath, and a blob with stripped metadata
is not an `HgBlob`.
Reviewed By: farnz
Differential Revision: D8508834
fbshipit-source-id: 387f2bd4f9439ed6b7593b057be035ed9f5452ad
Summary: Added logic to save `FileChange` as a Mercurial format `HgBlobEntry`
Reviewed By: sunshowers
Differential Revision: D8187792
fbshipit-source-id: 4714c81ab23ebac528cfec15c4a9e66083d4fb6c
Summary:
Unfortunately `HgParents` can't represent all valid parents, because
it can't represent the semantically important case where `p1` is `None` and
`p2` is not. (For incoming changesets we'd like to keep full fidelity with
Mercurial.)
All the Thrift definitions store `p1` and `p2` separately anyway, so just make
that change throughout `RevlogChangeset` and `BlobChangeset`.
Reviewed By: StanislavGlebik
Differential Revision: D8374125
fbshipit-source-id: 63674daaad05d4d4cae3778744dbf1c14b3c2e3b
Summary:
The old blobimport tool will not be able to import commits with the new Thrift serialization they'll be switching to.
`blobrepo::utils::RawNodeBlob` is also used by the admin tool, and it will go away once we start using Thrift serialization.
Reviewed By: farnz
Differential Revision: D8372455
fbshipit-source-id: d02a37e33e1ccd4dd1f695e38dbb40851dd51cd6
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: extras use python's escaping functionality in mercurial, the '\r' case is happening in production data
Reviewed By: StanislavGlebik
Differential Revision: D7925149
fbshipit-source-id: 23c544b8a757c40a120b69d1ec1ad942c672de17
Summary: This will replace `RawCSBlob` and the current bincode serialization.
Reviewed By: StanislavGlebik
Differential Revision: D7869524
fbshipit-source-id: 1a2f5d159a20889b10bb6b235f48769da4a187c1
Summary:
Very similar to file envelopes.
Also add some missing `#[inline]` annotations for file envelopes.
Reviewed By: StanislavGlebik
Differential Revision: D7868445
fbshipit-source-id: 3fb0d87e087612f37c8d5d0a90065359c671ceb8
Summary: Apparently p1 == p2 is a thing that happens in Mercurial, let's keep this logic in Mercurial-facing objects
Reviewed By: StanislavGlebik
Differential Revision: D7830183
fbshipit-source-id: eb473d57cb05f6553327bee2d5aeff9cf7d50eba
Summary:
If baserev == Some(idx), we changed it to None. Also we can have None if
baserev == -1 in mercurial. However these two cases are different. In the first
case it means that we have a literal chunk, not delta. In the second case it
means that we have a delta against empty string! So this is technically almost
the same, except that delta against empty string also have a 12 bytes prefix.
Previously this prefix was used as part of a revision data.
This diff fixes it.
Reviewed By: quark-zju
Differential Revision: D7815713
fbshipit-source-id: def2e54b2cc7379ba8f931ecf3f3c0c38d716058
Summary: The current behavior of delta::apply will panic Mononke server when the client sends malformed delta request. This change will instead propagate an Error explaining why the delta was malformed.
Reviewed By: jsgf
Differential Revision: D7775544
fbshipit-source-id: a64c27c7b9f13323b8be70eb8c2cdf315ce8f08d
Summary:
We'd like to move away from `RawNodeBlob` and `RawCSBlob` to data structures
serialized by Thrift. This is the first step to doing that.
The most important thing here is that it reuses file content IDs from native
Mononoke storage.
Reviewed By: jsgf
Differential Revision: D7771990
fbshipit-source-id: de4ee0b56aa6610caeff84b2235e19855df086cb
Summary:
We're going to keep this around for now as part of double-writing.
All the hashes here are definitely Mercurial hashes, so use them that way.
Reviewed By: lukaspiatkowski
Differential Revision: D7683890
fbshipit-source-id: 270091cd11f3cec7ef4cf565de5ef913fcf7adea
Summary:
file::File works entirely in the Mercurial domain, so these
conversions are good.
Reviewed By: StanislavGlebik
Differential Revision: D7665973
fbshipit-source-id: 8a192c5d1886492ad21593693b080c8e5ddf8f7e
Summary:
I was trying to debug something with the new blobimport, and this was
getting in the way.
Reviewed By: StanislavGlebik
Differential Revision: D7664660
fbshipit-source-id: 2ec4ee79fbe13584f35e7dcd9e8df2b8bdf181c0
Summary:
The base type is better because it can represent dates from before
1970 as well.
Reviewed By: StanislavGlebik
Differential Revision: D7652095
fbshipit-source-id: 6d66a06e18ba28e13e70b9f0e921acbd3d55baaf
