facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-10 00:45:18 +03:00
Code Issues Projects Releases Wiki Activity

mononoke: filter the same manifest entries in blobimporting

Browse Source 
Summary:
There is no need to insert the same entries twice. Let's filter them.
Note that while it's possible to have the same manifest entries (for example,
file or dirs with the same content), all changeset entries should be unique,
because each changeset in the repo is unique and is processed exactly once.

Reviewed By: farnz

Differential Revision: D6076667

fbshipit-source-id: 64bdf25a21884eb2faf43f32590f7cbb8f8dd300
This commit is contained in:
Stanislau Hlebik 2017-10-18 01:34:45 -07:00 committed by Facebook Github Bot
parent 58777d54ca
commit 553343bc0d
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
cmds/blobimport.rs
View File

@ -417,12 +417,20 @@ where
let receiverstream = stream::iter_ok::<_, ()>(recv);
let mut core = Core::new().expect("cannot create core in iothread");
let blobstore = open_blobstore(output, blobtype, &core.remote(), postpone_compaction)?;
// Filter only manifest entries, because changeset entries should be unique
let mut inserted_manifest_entries = std::collections::HashSet::new();
let stream = receiverstream
.map(move |sender_helper| match sender_helper {
BlobstoreEntry::Changeset(bcs) => {
bcs.save(blobstore.clone()).from_err().boxify()
}
BlobstoreEntry::ManifestEntry((key, value)) => blobstore.put(key, value),
BlobstoreEntry::ManifestEntry((key, value)) => {
if inserted_manifest_entries.insert(key.clone()) {
blobstore.put(key, value)
} else {
Ok(()).into_future().boxify()
}
},
})
.map_err(|_| Error::from("error happened"))
.buffer_unordered(channel_size);