Summary: Just a few minor changes that make our lives easier overall.
Reviewed By: lukaspiatkowski
Differential Revision: D5737854
fbshipit-source-id: da951d7872433bffa8fc64d15cd0e917f77144b5
Summary:
We want to avoid putting the same entries twice in the blobstore. And even more - we want to avoid generating list of these entries at all in the first place.
The first approach was to add a `Mutex<HashSet>` that worker threads will use to filter out entries that were already imported. Turned out that this Mutex kills almost all the speedup from concurrency.
But since we have linkrevs then for each entry we know in which commit this entry was created [1]. That means that all of the entries are already nicely split between the threads. So no synchronization is needed.
It gives a good speedup - from ~7min to 2min of importing of hg upstream treemanifest repo using file blobstore.
Note: there is still a lock contention - tree revlogs and file revlogs maps are protected by mutex. We can optimize it later if needed.
[1] There is a well-known linkrev issue in mercurial. It shouldn't affect our case at all.
Reviewed By: jsgf
Differential Revision: D5650074
fbshipit-source-id: c4f9e2763127ffe4402417dd3963f1f450d7b325
Summary: Main part is `get_stream_of_manifest_entries` that creates a stream of all tree manifest entries by recursively going through all of them.
Reviewed By: jsgf
Differential Revision: D5622490
fbshipit-source-id: 4a8b2707df0300a37931c465bafb1ed54d6d4d25
Summary:
A preparation step before blob importing of tree manifest repos to blobrepo.
`get_parents()` method of BlobEntry reads parents from the blobstore. It works fine for file entries because file entries can stores its parents in the blobstore. With tree manifests BlobEntry can contain also tree manifest entries, and that means that tree manifest entries parents should also be stored somewhere in the blobstore.
I suggest to use the same logic for the tree manifest entries as for the file entries. File and manifest entries have two blobstore entries - one stores hash of the content and parents, another stores the actual content.
To do this I moved `RawNodeBlob` and `get_node()` to the separate module and made fields public.
Reviewed By: jsgf
Differential Revision: D5622342
fbshipit-source-id: c9f0c446107d4697b042544ff8b37a159064f061
Summary:
Instead of storing `Vec<u8>`, let's store `Vec<PathComponent>`, where PathComponent is Vec<u8> without b'\'.
To make sure len() is still `O(1)` let's store it too.
Reviewed By: sid0
Differential Revision: D5573721
fbshipit-source-id: 91967809284d79bf0fcdcabcae9fd787a37c318b
