Summary:
We currently call "p4 where" on all files in the new client spec, which is roughly O(all files in ovrsource), and it's only going to get worse as ovrsource grows :(
This diff makes the number of calls O(files added & removed between two client specs) by filtering them in the following way:
- get a full list of files from old client (L1)
- get a full list of files from new client (L2)
- files_added = L2 - L1
- files_removed = L1 - L2
- `p4 where` on (files_added + files_removed)
How much speedup do we get?
from
[2018-04-25 16:07:27,725]INFO:root: Start sync import in ovrsource-master
[2018-04-25 17:35:35,873]INFO:root: Finish sync import in ovrsource-master after 5288.147963762283 seconds
to
[2018-04-26 13:43:28,819]INFO:root: Start sync import in ovrsource-master
[2018-04-26 13:51:35,575]INFO:root: Finish sync import in ovrsource-master after 486.7560772895813 seconds
on this change D7722798
Differential Revision: D7772403
fbshipit-source-id: 05a16343264007ee3ee466621da9da888c2368d7
Summary: Curernt sync importer modifies mercurial storage data structures directly, which is error prone and hard to debug. This reimplements the sync importer with a higher level mercurial API (what `p4seqmport` uses).
Differential Revision: D7714322
fbshipit-source-id: 0269839b5ee3a4b45f166dce74dfd29c8ec5135a
Summary: D7676184 changes the sync importer tests to use `p4seqimport` instead of `p4fastimport`, but there's one left. This fixes the leftover one.
Differential Revision: D7687451
fbshipit-source-id: a7f4c4fa93f5a2a52a99cd1d8263718fc656f683
Summary: Since `p4seqimport` reaplces `p4fastimport`, the test for `p4syncimport` (which currently uses `p4fastimport`) should be updated to use `p4seqimport` as well.
Differential Revision: D7676184
fbshipit-source-id: 4273975159190933a71a1fe538dfddfbe4eb8a91
Summary:
running p4 syncimport when a client spec changes is very slow, and this blocks us from using it in prod.
This diff speeds up p4 syncimport by using multiple workers when calling `p4 where` on a big list of files.
It reduces the runtime from ~14mins to ~3mins when I add a new mapping `u'//depot/Tools/...': u'Tools/...',` to ovrsource config and run p4 syncimport.
Reviewed By: dsp
Differential Revision: D6838406
fbshipit-source-id: e7260be5dea2b41e176fdc3508a78134cb1e9c35
Summary:
Previously we allowed a sync commit to happen at any time, but imported the
full repository to bring it into sync. This is slow for very large repositories
and might be accidentally triggered. We change this behavior to only do a sync
commit if only the imported client changed but nothing else.
Reviewed By: dsp
Differential Revision: D6838370
fbshipit-source-id: eecdfb0ea295585058784f8d1f70de5f8c733645
Summary:
This commit moves most of the stuff in hgext3rd and related tests to
hg-crew/hgext and hg-crew/test respectively.
The things that are not moved are the ones which require some more complex
imports.
Depends on D6675309
Test Plan: - tests are failing at this commit, fixes are in the following commits
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D6675329
