Summary:
There was a bug when if the entry with the same key is present in the hash and
the hash is full, then some entry will be evicted even though it's not
necessary.
It adds one hash map lookup, but it shouldn't be a big problem
Reviewed By: jsgf
Differential Revision: D6702001
fbshipit-source-id: 1c8cadc5e4bad6f6d95279b21edf00ed99f62c49
Summary: The Framed and FramedParts duet fits following diffs a bit better than FramedStream and ReadLeadingBuffer.
Reviewed By: jsgf
Differential Revision: D6567554
fbshipit-source-id: 88d117ad9e8227f9de278037b333da7ffc4fdf1f
Summary:
- Added test for checking if decompressing isn't overreading it's input
- As a result removed ZSTD decompression which is overreading input
- Changes in decompression code propagated usage of BufRead in bundle2 parsing code
- As a result a bug in OuterDecoder was found where the buf was not consumed if the header value is "0"
Reviewed By: jsgf
Differential Revision: D6557440
fbshipit-source-id: 89a9f4c8790017c5b86d28d467e45f687d7323f6
Summary:
The no-compression decompressor cann't provide framing as other decompressors, so a safe approach would be not to have it at all.
I replaced occurances of no-compression with `Either` reader that seem to be a pattern present in the community (f.e. `futures::future::Either` or `itertools::Either`)
Reviewed By: jsgf
Differential Revision: D6555922
fbshipit-source-id: 998dafab8d9b2f00d058ce2f9e0aced76cf15b4e
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary:
One config option is now obsolete, and the other one got renamed.
(Note: this ignores all push blocking failures!)
Reviewed By: farnz
Differential Revision: D6668021
fbshipit-source-id: aff949662a9584b7aead3f571512787aa1e9a260
Summary: It removes a bit of copy-pastes code
Reviewed By: sid0
Differential Revision: D6627250
fbshipit-source-id: b110dcad998a240b6da60bc3347c7c5c0370aae0
Summary:
It won't be used in prod anyway, and for local use rocksdb blob state is still
preferable
Reviewed By: sid0
Differential Revision: D6627197
fbshipit-source-id: d9ca55b0221c050e8e8e35914aff22906198874a
Summary:
The test wasn't waiting for the Mononoke server to be available. Also
improve the error message slightly.
Reviewed By: jsgf
Differential Revision: D6606102
fbshipit-source-id: d0fde39aef8b3423fd1a8996a01f12e8a9661597
Summary:
This is very similar to the support for encoding history entries in the
previous patch. The NULL_HASH wart makes it a little annoying to deal with.
We need to be able to compute how many bytes a delta has efficiently -- add a
function and tests for that as well.
Reviewed By: jsgf
Differential Revision: D6589117
fbshipit-source-id: e3a05672e1e8f455a2e95b30418468d1e3254407
Summary:
This counter is generally quite useful, so extract it into a location
where Mononoke can also use it.
Also add ShipIt and corresponding license linter rules for this.
Reviewed By: jsgf
Differential Revision: D6600409
fbshipit-source-id: ce63d718ca259ac30d2ed77e72004fb83158f818
Summary:
In the next diff we'll `task::current()` inside MemoFuture. It fails if no
task exists. Let's use spawn to create a Task for each future.
Reviewed By: jsgf
Differential Revision: D6611510
fbshipit-source-id: 95a825dff9714579e9b16e74f355c54d8fa83a24
Summary: Previously inserting the same key twice will add size of this key twice. This diff fixes it.
Reviewed By: jsgf
Differential Revision: D6602501
fbshipit-source-id: eb8f785296945bca1df4b6cd33bd3050a48e2174
Summary:
This is the opposite of decode. Encode -> decode trips are also easily
testable, which is nice.
Reviewed By: jsgf
Differential Revision: D6586165
fbshipit-source-id: 9300844da5c3210cfbe4b6a3c98c6429d33ccbcb
Summary:
This is what we get or send over the wire, so while encoding we don't have to
check whether the value is in bounds.
Reviewed By: jsgf
Differential Revision: D6596025
fbshipit-source-id: 57fcefbe3545a1af4c4c4ea24350e636860d2392
Summary:
This just adds a basic verification test for now. Roundtrip tests are coming up
in future diffs.
Reviewed By: jsgf
Differential Revision: D6585421
fbshipit-source-id: 1f38366f658324447dbe5634afb04080aad91cd3
Summary:
The only verification that needs to be done is for if the delta base
is null.
Also move a rustfmt_skip attribute to the smallest piece of code that it needs to
apply to.
Reviewed By: jsgf
Differential Revision: D6589118
fbshipit-source-id: 287c94871ed30d196c9eaa673cfd7c7f0dd6ed93
Summary:
The encoding code is going to use this to ensure that it isn't getting invalid
data.
Reviewed By: jsgf
Differential Revision: D6585201
fbshipit-source-id: f7ca749d9d0a8862621816e362add7546ca413c5
Summary:
This allows (actually forces) us to drop a bunch of `as_ref` calls. It also
makes upcoming diffs look cleaner.
Reviewed By: jsgf
Differential Revision: D6585202
fbshipit-source-id: a53902696cf12050a851c30b557aeefe09c41903
Summary:
Add an intrinsic method that acts as an `Arbitrary::arbitrary` implementation
with parameters. The one parameter we have for now is `empty_allowed`, which stands
for whether empty components can be generated.
Reviewed By: jsgf
Differential Revision: D6584040
fbshipit-source-id: 2005e1a77b1bd53294c4386259847a3a113f8f8f
Summary:
While encoding a wirepack, the entry count for a single filename needs to be
encoded upfront. To avoid buffering in the packer (upcoming diffs), turn this into a stream
entry. The packer will verify that the actual items on the stream match the
entry count provided upfront.
Also remove the path since the caller can now keep track of the current path,
and in that way avoid a path clone for each entry.
Reviewed By: jsgf
Differential Revision: D6577181
fbshipit-source-id: 577bdb9b5fc1a3fadb9b5f7ab4b312afce823ed8
Summary:
This will let us write roundtrip tests once encoding is done in an upcoming
diff.
Reviewed By: StanislavGlebik
Differential Revision: D6576990
fbshipit-source-id: 46cbaf4654bc1fd6d1f90351bbbdfd41234a00c7
Summary:
Fetching file sizes is very expensive - it's one more round-trip to the
blobstore.
Add method that returns tree content, but doesn't fetch blob sizes.
Reviewed By: jsgf
Differential Revision: D6545108
fbshipit-source-id: 3d335eafa9dc367fb9e607f5c147b7e5c37cf133
Summary:
Adds an option that sets the number of filelogs and revlogs that will be loaded
in memory. That let's us use blobimporting in memory constrained
enviroments.
Reviewed By: jsgf
Differential Revision: D6532734
fbshipit-source-id: b748478ec80e75f56a8e07ae1532b0d69c4a5e16
Summary:
When using forward/send_all on a Sink it is being flushed and closed, this means that if we call forward multiple times on a Sink it might fail if the underlying IO is already closed.
For ZSTD compression it didn't mattered, because it doesn't use the close/finish, but for other compressions it does.
Reviewed By: jsgf
Differential Revision: D6545343
fbshipit-source-id: 079d950f34716d621dd1fabda4f97d58d0e1d02d
Summary:
Like the other BlobState components, Linknode was too generic -
reduce down to a practical set for live implementations.
Error handling is not great here or in Bookmarks, but I'm going to await the
decision on moving to Failure before I improve it.
Reviewed By: jsgf
Differential Revision: D6459012
fbshipit-source-id: 00314309f62ba070b5908a28f5174a31b6dd0d84
Summary:
Remove the last associated types from BlobStore - this means that
BlobStore now has an associated trait object type.
Reviewed By: jsgf
Differential Revision: D6425414
fbshipit-source-id: 7186dab9b56593dd1d70be732d4ad56d1e7b3c63
Summary:
Don't use failure's bail!() and ensure!() macros.
Instead, failure_ext provides:
- bail_err!(err) - Converts its single parameter to the expected error and returns; ie `return Err(From::from(err));`
- bail_msg!(fmt, ...) - takes format string parameters and returns a `failure::err_msg()` error
- ensure_err!(), ensure_msg!() - corresponding changes
Also:
- remove all stray references to error-chain
- remove direct references to failure_derive (it's reexported via failure and failure_ext)
- replace uses of `Err(foo)?;` with `bail_err!()` (since `bail_err` unconditionally returns, but `Err(x)?` does not in principle, which can affect type inference)
Reviewed By: kulshrax
Differential Revision: D6507717
fbshipit-source-id: 635fb6f8c96d185b195dff171ea9c8db9e83af10
Summary:
We don't read the dirstate, but treedirstate has a new repo requires
as it changes dirstate format. Learn to ignore it.
Reviewed By: jsgf
Differential Revision: D6509774
fbshipit-source-id: 46faedcece308e2ebc34d87a62d2391a68eeea38
Summary: This diff also moves parsing of Batch subcommands inside sshproto where it belongs
Reviewed By: farnz
Differential Revision: D6492001
fbshipit-source-id: a31864f50e9468f360f9a30fc75fc2cfdee56169
Summary:
Two tests can't be run simultaneously because they use the same port. This diff
fixes it.
Reviewed By: farnz
Differential Revision: D6457855
fbshipit-source-id: 3e5f82211bcd87abc608bdeab239fe680976316d
Summary:
Make it possible to skip a number of commits.
Also change type from usize to u64, to make sure it works the same on 32-bit
platforms (although that shouldn't matter much).
Reviewed By: jsgf
Differential Revision: D6395743
fbshipit-source-id: 88a12583de2b23d4f55115d696c5398f6814c2da
Summary:
This diff upgrades the version of rustc in fbcode to version 1.22.1,
including the requisite third-party2 symlink updates for the `rust` and
`rust-crates-io` projects, config.py changes in the third-party2 and
third-party-buck directories, and code fixes for Rust targets in common/rust
and scm/mononoke.
Reviewed By: jsgf
Differential Revision: D6489808
fbshipit-source-id: 48377ea937916beb7dae7a3806de1fce1e29dd24
Summary:
Convert scm/mononoke to use failure, and update common/rust crates it depends on as well.
What it looks like is a lot of deleted code...
General strategy:
- common/rust/failure_ext adds some things that are in git failure that aren't yet in crates.io (`bail!` and `ensure!`, `Result<T, Error>`)
- everything returns `Result<T, failure::Error>`
- crates with real error get an error type, with a derived Fail implementation
- replicate error-chain by defining an `enum ErrorKind` where the fields match the declared errors in the error! macro
- crates with dummy error-chain (no local errors) lose it
- `.chain_err()` -> `.context()` or `.with_context()`
So far the only place I've needed to extract an error is in a unit test.
Having a single unified error type has simplified a lot of things, and removed a lot of error type parameters, error conversion, etc, etc.
Reviewed By: sid0
Differential Revision: D6446584
fbshipit-source-id: 744640ca2997d4a85513c4519017f2e2e78a73f5
Summary: `changeset_exists` only checks that the changeset is in the blob store. Once we start accepting pushes, we'll also want to check that the changeset is reachable from a head. Use revsets to check this. Once we have a notion of "completeness" for changesets, we'll switch `known` to use that instead - this code is still useful as a way to go from `changeset_exists` to `changeset_complete` in bulk.
Reviewed By: jsgf
Differential Revision: D6205695
fbshipit-source-id: 7e3b5c30bc5e459feb95a20913d8a04f3fda7469
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
