Summary: This update our UploadHgFileContents::ContentUploaded implementation to not require buffering file contents in order to produce a Mercurial Filenode ID.
Reviewed By: farnz
Differential Revision: D16457833
fbshipit-source-id: ce2c5577ffbe91dfd0de1cac7d85b8d90ded140e
Summary:
NOTE: This isn't 100% complete yet. I have a little more work to do around the aliasverify binary, but I think it'll make sense to rework this a little bit with the Filestore anyway.
This patch incorporates the Filestore throughout Mononoke. At this time, what this means is:
- Blobrepo methods return streams of `FileBytes`.
- Various callsites that need access to `FileBytes` call `concat2` on those streams.
This also eliminates the Sha256 aliasing code that we had written for LFS and replaces it with a Filestore-based implementation.
However, note that this does _not_ change how files submitted through `unbundle` are written to blobstores right now. Indeed, those contents are passed into the Filestore through `store_bytes`, which doesn't do chunking. This is intentional since it lets us use LFS uploads as a testbed for chunked storage before turning it on for everything else (also, chunking those requires further refactoring of content uploads, since right now they don't expect the `ContentId` to come back through a Future).
The goal of doing it this way is to make the transition simpler. In other words, this diff doesn't change anything functionally — it just updates the underlying API we use to access files. This is also important to get a smooth release: it we had new servers that started chunking things while old servers tried to read them, things would be bad. Doing it this way ensures that doesn't happen.
This means that streaming is there, but it's not being leveraged just yet. I'm planning to do so in a separate diff, starting with the LFS read and write endpoints in
Reviewed By: farnz
Differential Revision: D16440671
fbshipit-source-id: 02ae23783f38da895ee3052252fa6023b4a51979
Summary: The size of entries isn't related to our local pointer size. This updates that.
Reviewed By: StanislavGlebik
Differential Revision: D16336407
fbshipit-source-id: 2f500c54d218d81f06ba0827bd72ad8be1192b9d
Summary:
It's used only in a very few places, and most likely that's by accident. We
pass logger via CoreContext now
Reviewed By: krallin
Differential Revision: D16336953
fbshipit-source-id: 36ea4678b3c3df448591c606628b93ff834fae45
Summary:
This diff sets two Rust lints to warn in fbcode:
```
[rust]
warn_lints = bare_trait_objects, ellipsis_inclusive_range_patterns
```
and fixes occurrences of those warnings within common/rust, hg, and mononoke.
Both of these lints are set to warn by default starting with rustc 1.37. Enabling them early avoids writing even more new code that needs to be fixed when we pull in 1.37 in six weeks.
Upstream tracking issue: https://github.com/rust-lang/rust/issues/54910
Reviewed By: Imxset21
Differential Revision: D16200291
fbshipit-source-id: aca11a7a944e9fa95f94e226b52f6f053b97ec74
Summary: It's a footgun, as its memory use is bounded by the size of the repo, so it's almost never what you really want, but instead a band-aid over a deeper problem.
Reviewed By: StanislavGlebik
Differential Revision: D16132084
fbshipit-source-id: 12683801652509ebb09881595768d640df0b7490
Summary: We are copying `MPath` a lot, this change should reduce amount of allocations and memory reuse for `MPath`
Reviewed By: farnz
Differential Revision: D15939495
fbshipit-source-id: 8da8f2c38f7b46f27d0df661210c9964aed52101
Summary:
Hex encoding and decoding is in the hot path for gettreepacks, particularly
when we have large fetches (for example with clones), we can easily do this in
the order of tens or hundreds of thousands.
Switch to using a SSE/AVX2 optimized hex encoding/decoding which results in
a 10x performance improvement for decoding. Encoding was already using a
relatively optimized version, so gain is expected to be much lower for encoding.
Reviewed By: farnz
Differential Revision: D15940209
fbshipit-source-id: 28734f45f7508a94b110e25c01e1baa955ebd4e4
Summary: `HgEntryId` is much more useful in a typed from (enum of `(FileType, HgFileNodeId)` and `HgManifestId`), most of the time we now which type entry should contain and it makes it harder to make and error, all other use cases which require just hash should use `HgNodeHash` instead. This diff includes minimal changes which are necessary to make it work. Some of the cases which do sequence of `Entry::get_hash().into_nondehash() -> HgManifestId::new() | HgFileNodeId::new()` are left for future diffs.
Reviewed By: farnz
Differential Revision: D15866081
fbshipit-source-id: 5be9ecc30dbfd0a49ae6c5d084cdfe2dac351dac
Summary:
Convert to `2018` version and remove `#[macro_use]` and `extern crate` from following folders:
- blobrepo
- blobstore
- mercurial_types
- mononoke_types
Reviewed By: StanislavGlebik
Differential Revision: D14827294
fbshipit-source-id: 4b8a2edb5d9ad1fe75083e172cd0da359fa6dc2e
Summary:
It's cheap so let's return it. It will be used in the next diffs
Context:
There are some commit cloud commits that take ages to compute. For some reason
commit cloud sends too many manifests, which mononoke doesn't filter out and
does a lot of unnecessary checks, which take ages. This stack fixes it.
Reviewed By: farnz
Differential Revision: D14666306
fbshipit-source-id: 75e39f7d88e139f3558825faec5d4e94f6987050
Summary: Add new API server endpoints for performing batch fetches of data and history. The requests and responses are currently serialized as JSON, though the format can be trivially changed later to something like CBOR (or any other format supported by serde).
Reviewed By: xavierd
Differential Revision: D14573735
fbshipit-source-id: ac46aa8a55db44b695c53f9e32cc4ea2e17504c8
Summary:
getpackv1 needs to return copy metadata together with the content. This diff
fixes it
Reviewed By: kulshrax
Differential Revision: D14668319
fbshipit-source-id: 56a6ea2eb3a116433446773fa4fbe1f5a66c5746
Summary:
Removed references to HgNodeHash in repo_client in the specified functions. In
addition, updated other files due to type related dependencies.
Reviewed By: StanislavGlebik
Differential Revision: D14543934
fbshipit-source-id: b0d860fe7085ed4b91a62ab1e27fb2907a642a1d
Summary: `LooseHistoryEntry` and `PackHistoryEntry` aren't the best names for these types, since the latter is what most users should use, whereas the former should only typically used for data transmission. As such, we should rename these to clarify the intent.
Differential Revision: D14512749
fbshipit-source-id: 5293df89766825077b2ba07224297b958bf46002
Summary:
The diff adds support for getpackv1 request. Supporting this request is
necessary because hg client will stop using getfiles soon.
There are two main differences between getfiles and getpackv1. getfiles returns
loose files formats, while getpackv1 returns wirepack pack file (same format as
used by gettreepack). getpackv1 supports fetching more than one filenode per
file.
Differential Revision: D14404234
fbshipit-source-id: cfaef6a2ebb76da2df68c05e838d89690f56a9f0
Summary:
The problem was in using `file_changes()` of a bonsai object. If a file
replaces a directory, then it just returns an added file, but not a removed
directory.
However `changed_entry_stream` didn't return an entry if just it's mode was changed (i.e. file became executable, or file became a symlink). This diff fixes it as well
Let's use the same computing changing files method instead of `file_changes()`.
Differential Revision: D14279470
fbshipit-source-id: 976b0abd93646f7d68137c83cb07a8564922ce17
