Summary:
I'm going to factor out some of this code into a new crate, and I want
to get these changes out of the way first.
Reviewed By: lukaspiatkowski
Differential Revision: D5924471
fbshipit-source-id: 7007638ce688e62397637ff6f9a2e51247ec9312
Summary: Removes usage of BoxFuture and BoxStream from futures crate in blobstore/fileblob, following instructions in https://our.intern.facebook.com/intern/wiki/Rust/Tokio_cleanup/
Reviewed By: jsgf
Differential Revision: D5884656
fbshipit-source-id: dcf979daf939cd193f4d6361eb36d03977380a22
Summary: `Effect` is a more descriptive name.
Reviewed By: farnz
Differential Revision: D5910892
fbshipit-source-id: 86341e18ea0fcc770c637e7adafcb220864ca1b7
Summary:
These get pretty annoying to construct in tests.
`mercurial-types-mocks` seems like a great place to have them.
I ended up using the `const_fn` feature for convenience.
Reviewed By: StanislavGlebik
Differential Revision: D5898773
fbshipit-source-id: f9ead4e7720ed8ffb0535826603656ff7115b6e4
Summary:
Mercurial filelog entries may have metadata fields in the beginning, usually used to track copies/renames. Previously mononoke eden server returned this metadata as part of the file blob.
This diff changes it. Now `get_content()` method returns file content without metadata, and to make it consistent, both `get_content()` of the blobrepo and revlog repo do the same.
This decision certainly has it's tradeoffs, because now it's more difficult to get metadata (`get_raw_content` needs to be used).
But we'll probably change how metadata is stored anyway, that's why I think changing `get_content` method is fine.
This diff also cleans up server/src/main.rs file, because previously it had to strip metadata itself.
Also diff fixes problem in metadata parsing - it previously failed if file is less than 2 bytes
Reviewed By: farnz
Differential Revision: D5901476
fbshipit-source-id: f3ade0179710352590068c238e6a733aab68a512
Summary:
Mercurial test suite have #testcases features. It allows to use the same test
file for two different tests while tweaking just small parts of it.
I used testcases in test-eden-server.t, but I used it incorrectly - #files test
case wasn't actually used at all. This is because hghave is not present in the
repo.
This diff adds hghave. It's a copy-paste from upstream mercurial
Reviewed By: farnz
Differential Revision: D5901439
fbshipit-source-id: 2131733d8c523b7f7ce9e6a087c77fed6a427e6d
Summary:
Set difference is the last of the three core set operations - with
this in place, it's now possible to implement any set combinator you need to
get just the revisions you want.
Reviewed By: jsgf
Differential Revision: D5901957
fbshipit-source-id: af403b43781bbb9686d36443fe9d0eac6499cf94
Summary: Bottleneck while blob importing of a huge repo is in the lock contention. Replacing it with RwLock speeds up fbsource import from 20 hours down to 6 hours.
Reviewed By: farnz
Differential Revision: D5891097
fbshipit-source-id: bbac2e113896958d6f2da270837c9787e701b5cb
Summary:
`Path` has the potential to be confused with `std::path::Path`.
`MPath` is nice, concise, and clearly different from `Path`.
Reviewed By: jsgf
Differential Revision: D5895665
fbshipit-source-id: dc5ed5c3866b227d753c6d904d3c6d213c882cd7
Summary: Going to make a few changes here.
Reviewed By: jsgf
Differential Revision: D5895642
fbshipit-source-id: 79483e15087d4c552b6bc9801ad3fe0aaba071d6
Summary: Compaction can slow down blobimporting a lot. Let's add an option to postpone it till the end
Reviewed By: farnz
Differential Revision: D5882003
fbshipit-source-id: 0611a8e94b3d7331bdacf909d820526f547414a0
Summary: Removed use of deprecated Tokio APIs. Firstly, removed usage of BoxFuture and BoxStream from futures crate and instead used it from futures_ext crate. Also, replaced stream::iter() with stream::iter_ok().
Reviewed By: sid0
Differential Revision: D5882902
fbshipit-source-id: 329231cca798371701e8966251a43a78db6e4ee1
Summary: Generation has nothing to do with machine words or pointer sizes, so it shouldn't be using usize. Use u64 instead.
Reviewed By: wez
Differential Revision: D5885943
fbshipit-source-id: 49b748cf592e270f78d85fe3c39b8d1a2a3e1671
Summary:
lukaspiatkowski pointed out that I don't test that an input that's NotReady doesn't hit
the panic cases. Add tests to check that this works.
Reviewed By: jsgf
Differential Revision: D5881789
fbshipit-source-id: 6357262c070c8c6a524d7bedeb1966843bd7612b
Summary: Teach the revsets code to do set intersection.
Reviewed By: jsgf
Differential Revision: D5881788
fbshipit-source-id: 972fd85a47ee29a1037f5e4072c03e25189b8300
Summary:
Switch from hand-rolled errors that eat the underlying cause to
`error_chain!` errors that allow you to extract what happened.
Reviewed By: StanislavGlebik, lukaspiatkowski
Differential Revision: D5862059
fbshipit-source-id: 4f7e849249b124cc6f48498045de1d01a717619b
Summary:
Manually adding all the dependencies in the TARGETS file is quite annoying. This simple tool does it for you!
It queries buck to get list of files for the target, and also queries all the rust_library mononoke dependencies.
Then it parses source files and generates TARGETS list.
Tool is super-simple and that's why it's not always correct (for example, it fails to find a deps if hasn't been used in mononoke before), but it works good enough for many use-cases (see test plan).
Reviewed By: lukaspiatkowski
Differential Revision: D5861828
fbshipit-source-id: ab8cc96d8bc394af172ee09da3aceb0ffb7493e8
Summary: This diffs reverts D5842977 and adds fetching mononoke-config inside tupperware, so that it can be served by mononoke server
Reviewed By: StanislavGlebik
Differential Revision: D5866239
fbshipit-source-id: 6f3fd794e5274c8bab0c50efc9c95e31f88be688
Summary: memheads can't fail, so use `!` for its `Error` type
Reviewed By: sid0
Differential Revision: D5849874
fbshipit-source-id: fbefb3cdaa11d9c10cbfc988e43978751c807472
Summary: membookmarks can't fail, so use `!` for its `Error` type
Reviewed By: sid0
Differential Revision: D5849875
fbshipit-source-id: 5dbb4edf9bb57dde55877d2c90936b51dcc5d7ff
Summary: memblob can't fail, so use `!` for its error type
Reviewed By: sid0
Differential Revision: D5849876
fbshipit-source-id: 48ca1c345d788a5f6e632d73f55bc485c036a0e3
Summary:
We can use the "never_type" feature to enable more general use
of the `!` type, which indicates that the type state is impossible (ie,
`!` has no values). Not only is this more expressive than using - say -
`()`, it also allows irrefutable matching against a result (`let Ok(x)
= ...`), since the compiler knows that `Err(!)` can never happen.
Reviewed By: sid0
Differential Revision: D5849877
fbshipit-source-id: 48c14f9c2042fb8168eb1dd308fd3a79c9ed0fc2
Summary:
We want a mechanism to quickly and easily find sets of revisions in
Mononoke. As a first step, introduce a mechanism to find single nodes, and a
mechanism to union two streams of nodes.
Later diffs will add the other 2 set operations, plus range (`a::b` in
Mercurial terminology).
Reviewed By: jsgf
Differential Revision: D5778925
fbshipit-source-id: 9e320ccab21c9b7b9c0fab8fbbc5c7389d2b7da0
Summary: file blobstore was read data from incorrect folders. This diff fixes it.
Reviewed By: farnz
Differential Revision: D5852736
fbshipit-source-id: 1bc08788023ff2cc9d631985d70329475afc589f
Summary:
`indicatif` is a command-line progress bar crate for Rust. It seems
pretty generally useful, and I plan to use it in the Dewey client replacement.
As a side note, it was written by Armin Ronacher (of Flask fame), so it should
hopefully be solid. See https://github.com/mitsuhiko/indicatif.
As part of this update, the `bitflags` crate went from `0.9.1` to `1.0.0`, which adds
a breaking change wherein bitflag constants are now namespaced by their
containing struct. This diff includes the appropriate fixes for revlog and zeus,
which seem to be the only things affected.
Reviewed By: jsgf
Differential Revision: D5855753
fbshipit-source-id: 38d88335126e787f3acd92189a1ed17f55138799
Summary:
Without this, any lines that are too long would cause `rustfmt` to
error out. That seems not very useful.
Reviewed By: kulshrax
Differential Revision: D5853676
fbshipit-source-id: b7335a14079ebf6245da0c028039d88745b32785
Summary:
We're going to add more stores like obsmarkers and linknodes very
soon. Stuffing all of them into type parameters is going to get ugly very soon.
Instead, add a trait which represents all the state, and make `BlobRepo`
generic on that trait. Add a few implementations for common use cases like
files and RocksDB-based stores.
This allows the dependency lists for the Mononoke and Eden servers to be more
straightforward.
Reviewed By: jsgf
Differential Revision: D5850372
fbshipit-source-id: 37a0687611687b9616ebbddce70f53e1e5d3267b
Summary: I'm going to move this to a common location very soon.
Reviewed By: jsgf
Differential Revision: D5850319
fbshipit-source-id: 09937659330de93cf74bf827f225418dd413865d
Summary:
Just like the previous diff with bookmarks, every user needs this and it makes
sense anyway.
Reviewed By: jsgf
Differential Revision: D5847877
fbshipit-source-id: b91a49e94da2d7e207061b6c52b78c55a2229dec
Summary: All its users need this, and it makes sense.
Reviewed By: jsgf
Differential Revision: D5847753
fbshipit-source-id: 042c7b5637e5f83918fa9bc5d131c8745d7bbc3e
Summary: Also ensure that `blobimport` doesn't use its own copy.
Reviewed By: jsgf
Differential Revision: D5847604
fbshipit-source-id: 5390848cd5fab8abd967ef9701720491d703c0f1
Summary:
Finally use hashencode to hash long path.
This is basically a translation of core mercurial python code to rust.
Tests are also copied from core mercurial
Reviewed By: jsgf
Differential Revision: D5719573
fbshipit-source-id: a53bdeafe22cc9f492b8c08d4ae302a393f9b8fb
Summary: Encoding only index path is not enough. It works fine for now because we don't use hash encoding. Next diff adds hash encoding, so we need to encode datapath too.
Reviewed By: jsgf
Differential Revision: D5719574
fbshipit-source-id: 4f2a4a75baad73313e80ffb81031166d4bab3e29
Summary:
Let's use separate function fsencode instead of two methods fsencode_dir and fsencode_file.
There are a few reasons for that:
1) This is similar to upsteram mercurial code - it also uses separate function, not a method of the class.
2) Path is supposed to represent a file in the mercurial vfs. Previously we joined this file with "00manifest.i" - it creates a file that doesn't exist in mercurial vfs. This point is debatable though, so I'm fine with making it a method of the class. But probably it doesn't matter that much.
2) We never actually need to encode directory - even in tree manifest case we use fsencode to find location of`00manifest.*` files. That means, that we don't really need to have separate fsencode_dir function, so I was wrong when I added them in the first place.
3) Special hash encoding is used to encode paths that are longer than 120 chars (will be added in the next diff). `00manifest.i` and `00manifest.d` are used in the hash digest, and that means that one fsencode_dir() method is not enough - we'd need to add separate methods to fsencode idx file and separate method to fsencode data file.
Reviewed By: jsgf
Differential Revision: D5719576
fbshipit-source-id: ca6b38dd7d0c6c0c5a345d8fcbe1b0d6fa10a062
Summary:
We want to be able to test against repo fixtures. Turn repo fixtures
that have been created by blobimport into in-memory BlobRepos for testing.
Reviewed By: jsgf
Differential Revision: D5834109
fbshipit-source-id: d6fc01f19986b5a6951f25c7c8bda8aa7c9dee56
Summary:
Removes usage of BoxFuture and BoxStream from futures crate and instead use
it from internal futures_ext lib.
Reviewed By: sid0
Differential Revision: D5847771
fbshipit-source-id: 01823513176add37caa73c0ea2810e80d7fae955
Summary:
Allow Arc<R> where R: Repo to be used more generally. Also Box for
completeness.
Reviewed By: farnz
Differential Revision: D5846659
fbshipit-source-id: e101647badfb18a4a59d595ee9b21814b332fe34
Summary: Use `impl Future` rather than a boxed future.
Reviewed By: sid0
Differential Revision: D5829773
fbshipit-source-id: 40c4339e96f7194544f416534952b78a23d93fa6
Summary: The idea is to deploy a no-op mononoke server to tupperware and slowly add more parts like fb303 page, proper args for startup etc.
Reviewed By: farnz
Differential Revision: D5842977
fbshipit-source-id: 227d144b71132e73b76ce7067ac998b1945cb5e1
