Summary: In addition to the dirs and files this diff adds a VfsWalker that enables the client to traverse the Vfs
Reviewed By: jsgf
Differential Revision: D5755004
fbshipit-source-id: 4d6fd8fee4d7b6d36d4bac866c45409310337522
Summary:
The url::percent_encoding module was split into its own crate. Use
that instead.
Reviewed By: jsgf
Differential Revision: D5769525
fbshipit-source-id: 4d95c5d791e486f56a16091ffa5114b3cfbfd47e
Summary:
There doesn't seem to be any need for the generic, and it makes things harder
for consumers.
Reviewed By: farnz
Differential Revision: D5778597
fbshipit-source-id: bc1018437b53130f95420e6e54767326e03d089a
Summary: a really basic prototype of how to integrate hooks into Mononoke.
Reviewed By: jsgf
Differential Revision: D5743853
fbshipit-source-id: 9ee62d9c4eb068defc58a684e47895e24f9ee5d3
Summary:
These prevent most errors while running the new rustfmt against
Mononoke.
Reviewed By: jsgf
Differential Revision: D5760091
fbshipit-source-id: 42d73fd282bf3c9cf56db1138e4f63c4c4416c41
Summary: Add the `--blobstore manifold` option to blobimport to make it write blobs to Manifold.
Reviewed By: jsgf
Differential Revision: D5758930
fbshipit-source-id: a14a3c155b5d8d7b171ed7a4e53f8569539cb2e9
Summary: Loosen type constraint on the error slightly, and rustfmt.
Reviewed By: sid0
Differential Revision: D5764616
fbshipit-source-id: 265071f775fac5ea40e8ba45a7da961904b3b397
Summary: A Key which is a Repo/Nodehash tuple is more generally useful.
Reviewed By: lukaspiatkowski
Differential Revision: D5717652
fbshipit-source-id: c00f4257434e958dfdbf712bb50ddbe081566e66
Summary:
Initially, compute the generation number for a changeset. This recursively
depends on the generation number of its parents (if any).
Reviewed By: lukaspiatkowski
Differential Revision: D5652702
fbshipit-source-id: 7e34395bd08e2bdcc91777f16c94f17f02402776
Summary:
`:` is a reserved character for Windows paths, so Mercurial rejects
them from being committed. Use `-` instead, so that we can commit file blob
repo test fixtures.
Reviewed By: kulshrax
Differential Revision: D5731525
fbshipit-source-id: 8d14fc03f1b135cbc4d42aeaf2f3a0ae6d13f956
Summary: This gets us `Display` support as well.
Reviewed By: lukaspiatkowski
Differential Revision: D5734383
fbshipit-source-id: 1485cf80bb310cdd282b4546bed56c60082be8ec
Summary: Just a few minor changes that make our lives easier overall.
Reviewed By: lukaspiatkowski
Differential Revision: D5737854
fbshipit-source-id: da951d7872433bffa8fc64d15cd0e917f77144b5
Summary: Add a unit test for failing to compute a value.
Reviewed By: Imxset21
Differential Revision: D5717910
fbshipit-source-id: cb5b1560a539f4b99d87dc2f8fc0afd37f4b5b97
Summary:
Pass a reference to the cache to the fill function. This allows the
function to be recursive based on memoized values.
This also required quite a bit of restructuring to make sure that locks
and ownership are handled properly during recursive calls. Specifically,
a new `Slot` state - `Polling` - is used to indicate when a thread/task
is currently calling `.poll()` on the future. This contains a list of
futures Tasks which are interested in the state of the slot which can be
notified when it changes state.
Also removed unused Entry API code.
Reviewed By: sid0
Differential Revision: D5652704
fbshipit-source-id: 29cd3fe37d4eb9316235872b7e2e228bf10a016f
Summary: auxencode panicked on path with like ' ' or '.'.
Reviewed By: jsgf
Differential Revision: D5706325
fbshipit-source-id: d2ecca2bc983e8adc581cac4e39c314002c171b6
Summary:
661974c25e00979eedffb30b432518135f0dc631 broke the integration test.
This diff fixes it.
We don't use `fsencode_*()` functions to convert manifest Path to
mercurial-types Path, because it was incorrect. Let's convert to OsString and
then to Path instead.
Reviewed By: lukaspiatkowski
Differential Revision: D5697885
fbshipit-source-id: a958a923973ca368bed495a19afdfdbb2e0a40d2
Summary:
PathElement is a blackbox now and it is hard to read inside it.
as_bytes() should at least enable reading the content of it and allow an attempt to create a string out of it. Alternative would be to create a Path using Path::join(&[PathElement]), but that is a bit too much.
Reviewed By: jsgf
Differential Revision: D5688868
fbshipit-source-id: f30879251e3da7e6b90c70bfbadb901a3d818dd7
Summary: Would be nice to be able to return future that references a boxed Entry
Reviewed By: jsgf
Differential Revision: D5688859
fbshipit-source-id: acb7bcc461f886bca08b84f31dbebe4f692187f1
Summary:
Core mercurial takes "data/" and "meta/" prefixes into account when does
fsencode.
It doesn't make a difference now, but it will make a difference when we'll add
hashencode to the fsencode() function.
Reviewed By: jsgf
Differential Revision: D5670748
fbshipit-source-id: 661974c25e00979eedffb30b432518135f0dc631
Summary:
We want to avoid putting the same entries twice in the blobstore. And even more - we want to avoid generating list of these entries at all in the first place.
The first approach was to add a `Mutex<HashSet>` that worker threads will use to filter out entries that were already imported. Turned out that this Mutex kills almost all the speedup from concurrency.
But since we have linkrevs then for each entry we know in which commit this entry was created [1]. That means that all of the entries are already nicely split between the threads. So no synchronization is needed.
It gives a good speedup - from ~7min to 2min of importing of hg upstream treemanifest repo using file blobstore.
Note: there is still a lock contention - tree revlogs and file revlogs maps are protected by mutex. We can optimize it later if needed.
[1] There is a well-known linkrev issue in mercurial. It shouldn't affect our case at all.
Reviewed By: jsgf
Differential Revision: D5650074
fbshipit-source-id: c4f9e2763127ffe4402417dd3963f1f450d7b325
Summary: While doing blob importing from smallish revlog repo to mononoke blobstore I noticed that most of the time is spent in waiting for mutexes. This diff is about removing the lock completely and prefilling all mutable datastructures in the `::new()` method.
Reviewed By: jsgf
Differential Revision: D5631055
fbshipit-source-id: 5776143014c92372b316dd8dba31785430f1f222
Summary: We'll need to implement blob importing from revlogrepo to blobrepo
Reviewed By: jsgf
Differential Revision: D5622462
fbshipit-source-id: c57f016711bcec7c0bd432d40881588ebdce6f7f
Summary: Main part is `get_stream_of_manifest_entries` that creates a stream of all tree manifest entries by recursively going through all of them.
Reviewed By: jsgf
Differential Revision: D5622490
fbshipit-source-id: 4a8b2707df0300a37931c465bafb1ed54d6d4d25
Summary:
Will be tested in the next diffs, but the fix is quite obvious - apply_deltas
doesn't modify it's parameters, it returns new versions.
Reviewed By: ryanmce
Differential Revision: D5649250
fbshipit-source-id: d2f1badb474911e7a5ac79dd59150444a52b5904
Summary: Repos with flat manifest and with tree manifest use different path. Let's open tree manifest file if it's present and flat manifest otherwise
Reviewed By: sid0
Differential Revision: D5583566
fbshipit-source-id: e35eef4b1f8067c2a91ebfc62718fec100f19e2e
Summary: Revlog Entry can be both for file and for tree manifest. Previously get_parents() worked correctly only for files. This diff fixes it.
Reviewed By: sid0
Differential Revision: D5583433
fbshipit-source-id: 13642f4de8310e67c1ef416fc83176bf4975f3b6
Summary: Implements get_content() for tree manifest entry. Note that we need to prepend current entry path to the child entries, because child entries paths are just basenames.
Reviewed By: sid0
Differential Revision: D5563970
fbshipit-source-id: 52e896a7a8892f4ffb5fc14b64dd7ebc5692471d
Summary:
Add function similar to get_file_revlog to get tree manifest revlog.
Will be used in the next diffs.
Reviewed By: jsgf
Differential Revision: D5563895
fbshipit-source-id: 0ff84c458eb071763cbdc6d98bcf92e9b8ccc1b8
Summary:
Previously `fsencode()` worked incorrectly if Path was a directory. We didn't notice it before because we've never used Path to store directories, but we will use it for TreeManifest.
I considered two options when implementing it.
1) Put some kind of flag `isDir` inside Path struct. But that would create complications with `join()` method. For example, you can't join anything to the file - what should we do in this case? panic? return result?
2) and another `fsencode_dir()` method. Clients need to know what kind of Path they have. I choose this option because it requires less changes and brings less complications compared to the option 1
Reviewed By: sid0
Differential Revision: D5574847
fbshipit-source-id: c4c476a7fc3b884de847c431a56ff5f313c1389f
Summary:
Adds method similar to https://doc.rust-lang.org/std/path/struct.Path.html#method.join.
Will be used in the next diff to prepend path prefix to the tree manifest entries
Reviewed By: jsgf
Differential Revision: D5563808
fbshipit-source-id: 3637275093c301bf159083cb9bfedaa0e490a75e
Summary:
Both changelog and manifest are revlogs and they are both protected by their
own mutexes. There is no need to protect by RevlogRepo mutex.
It requires making `Revlog::get_heads()` method accept immutable `&self`. This
is fine since all `Revlog` methods are protected by mutex.
Reviewed By: farnz
Differential Revision: D5618949
fbshipit-source-id: 0511d547360e9785cb6e2cefadf8c10626a433c4
Summary:
A preparation step before blob importing of tree manifest repos to blobrepo.
`get_parents()` method of BlobEntry reads parents from the blobstore. It works fine for file entries because file entries can stores its parents in the blobstore. With tree manifests BlobEntry can contain also tree manifest entries, and that means that tree manifest entries parents should also be stored somewhere in the blobstore.
I suggest to use the same logic for the tree manifest entries as for the file entries. File and manifest entries have two blobstore entries - one stores hash of the content and parents, another stores the actual content.
To do this I moved `RawNodeBlob` and `get_node()` to the separate module and made fields public.
Reviewed By: jsgf
Differential Revision: D5622342
fbshipit-source-id: c9f0c446107d4697b042544ff8b37a159064f061
Summary: The `boxfnonce` crate is a workaround for the lack of `FnBox` in stable Rust. Without it, it isn't possible to call a `Box<FnOnce>`.
Reviewed By: jsgf
Differential Revision: D5626687
fbshipit-source-id: 01ab37b98add5dae10f3c4834c2a330593639ab6
Summary: As per title. This includes fixes by kulshrax to make things not break.
Reviewed By: jsgf
Differential Revision: D5569907
fbshipit-source-id: f6a4ad88b33caf8c8de3ef8db7932d36993e4e95
Summary:
Instead of storing `Vec<u8>`, let's store `Vec<PathComponent>`, where PathComponent is Vec<u8> without b'\'.
To make sure len() is still `O(1)` let's store it too.
Reviewed By: sid0
Differential Revision: D5573721
fbshipit-source-id: 91967809284d79bf0fcdcabcae9fd787a37c318b
Summary:
Let's create a separate function `fetch_file_blob_from_blobstore` that will be
used to get content of the blob without creating file::BlobEntry.
file::BlobEntry requires Path and type parameters that won't be passed to eden
server.
Reviewed By: jsgf
Differential Revision: D5537647
fbshipit-source-id: 13fb2fbba7d3068b88aac2d842cc92b4da6ca80c
Summary: Method that returns content of the manifest entry in json format.
Reviewed By: sid0
Differential Revision: D5527659
fbshipit-source-id: 1832b645f69da40cbd620a6bff318e25594c5148
Summary: Will be used by eden server in the next diff
Reviewed By: jsgf
Differential Revision: D5553955
fbshipit-source-id: 196e2da597d0456b4337a6d9ada8fe188075e87e
Summary: We're going to use this code in another context very soon.
Reviewed By: StanislavGlebik
Differential Revision: D5581807
fbshipit-source-id: 6d43eb114b3d82fd97a9b9cce775f1f398853e84
Summary: I'm going to make significant changes to this crate in upcoming diffs.
Reviewed By: StanislavGlebik
Differential Revision: D5581790
fbshipit-source-id: 2a3a4b183b554ca46c3bc611452e55eb17238f20
Summary:
I don't think it's possible for linkrevs to be `-1` (not sure why it was
written as `!0` -- jsgf, any idea why?)
Reviewed By: StanislavGlebik
Differential Revision: D5581780
fbshipit-source-id: 69d685d863ee51adc846e889eb766f3fecc47681
Summary: We're going to reuse this code in an upcoming diff.
Reviewed By: StanislavGlebik
Differential Revision: D5581683
fbshipit-source-id: 08724333c2f637bbf2c2e534855c9802c15328ee
