Summary:
Read file from perforce, write into the hg repo.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7050157
fbshipit-source-id: 4389ba11f62c8ed825d6a6ef3c001095339eb551
Summary:
Creates ChangelistImporter, which will be responsible for translating a p4 CL to
a hg commit
For now it only goes through files touched by the CL and lists what was added or
removed. Next diffs will evolve it to the point where it effectively performs the
translation.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7049961
fbshipit-source-id: 6a9f3bd57cadc2b9ea8a81373cc10dfda76311e7
Summary:
Pulls the logic to define changelists from p4fastimport into separate function
and re-uses that in p4seqimport
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7035674
fbshipit-source-id: 699e9148d35e437f306062f290c8ec2a857df480
Summary:
This change:
Moves some opts sanitizing logic into function `sanitizeopts`
Adds checks for `limit` being a positive integer
Uses `sanitizeopts` new function in p4seqimport
Adds a test covering `sanitizeopts`
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7035217
fbshipit-source-id: cd677fb254ff83d123673d51a1c682639de08a30
Summary:
p4seqimport will be the new command to import from p4 to hg changelist by
changelist. This should provide us with a more robust importer that doesn't rely
on fiddling with hg's data structures directly. p4fastimport was important to
create ovrsource from scratch and import thousands of changelists, but moving
forward it is probably safer and easier to understand/maintain something that is
based on higher level Mercurial APIs
All that said, this is the first change, this change:
1. Creates p4seqimport command as part of the p4fastimport extension
2. Refactors the p4 client checking logic into `enforce_p4_client_exists`
3. Adds a test that checks the new function works through using `p4seqimport`.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7015941
fbshipit-source-id: cb5c59b2f104f336a078025544a44028bf01fa85
Summary: progress.bar() is incorrectly called without passing ui.
Differential Revision: D7415250
fbshipit-source-id: 22c7419561879ed9293e2c79cc9d4271e805be76
Summary:
The breakage we had on branch importer was related to filetransaction trying to
close a file that didn't exist. We're still not sure why this happens yet, but
the workaround was to use mercurial's transaction and to force not using workers,
which is this change.
Differential Revision: D7108127
fbshipit-source-id: 71fa63824984bfb91de3b732166f7bae496187ad
Summary:
Running `hg log --traceback --template '{file_copies}' -r XXXX` on a file with long history is slow for 2 reasons
- p4 fast importer preserves full history for deleted and re-added files
- p4 fast importer records the wrong parent of a file
This diff tries to fix these two issues.
In mercurial, if a file is added, deleted, and then added back, it should start a new file history when the file is added again.
For example,
commits commit1 commit2 commit3
actions add a.txt delete a.txt add a.txt
timeline ------------X------------X------------------------X------------
`hg debugindex a.txt` at commit3 shows a.txt as a new file without previous history
rev offset length delta linkrev nodeid p1 p2
0 0 3 -1 0 b789fdd96dc2 000000000000 000000000000
However, this is different in p4. `p4 filelog test.txt` gives you
//depot/Software/Apps/Main/Native/.castle/test.txt
... #3 change 523261 add on 2018/01/23 by zhihuih@devbig415 (text) 'test:add-again-same-file'
... #2 change 523254 delete on 2018/01/23 by zhihuih@devbig415 (text) 'testfile:delete'
... #1 change 523253 add on 2018/01/23 by zhihuih@devbig415 (text) 'testfile:add'
Currently, p4 fast importer preserves history the same way as p4, and this causes slowness (even timeout) in hg when it runs `hg log --traceback --template '{file_copies}' -r XXXX` on a revision that contains files with long history in p4 (mostly contributed by automation). To mitigate this, we want the p4 fast importer to behave the same way as hg, and starts a new history for a file that's added again.
Currently, p4 fast importer takes the tip of a filelog and treats that as the parent of the newly written entry diffusion/FBS/browse/master/fbcode/scm/hg/hgext/p4fastimport/importer.py;19ad9b05f50e3ff0265cdc7b4b45174dcf820343$468-469. This can be wrong when there are revisions from branches.
For example, if I edit file a in master in CL1, 2, 4, and I branch at CL3, and edit the file in branch in CL5, the current importer implementation will take filenode at CL4 as the parent of CL3
(CL1,2,3,4,5 corresponds to rev0,1,3,2,4)
{F120393661}
However, the correct behavior is to take filenode at CL2 as the parent of CL3
(CL1,2,3,4,5 corresponds to rev0,1,3,2,4)
{F120393662}
(This is also the example I use in `test-fb-hgext-p4fastimport-import-branch-filelogorder.t`, so if the description here looks confusing, please refer to the test)
Reviewed By: dsp
Differential Revision: D6962019
fbshipit-source-id: 24de76ae009e0d6f976d247087fe4702c99e0f82
Summary:
running p4 syncimport when a client spec changes is very slow, and this blocks us from using it in prod.
This diff speeds up p4 syncimport by using multiple workers when calling `p4 where` on a big list of files.
It reduces the runtime from ~14mins to ~3mins when I add a new mapping `u'//depot/Tools/...': u'Tools/...',` to ovrsource config and run p4 syncimport.
Reviewed By: dsp
Differential Revision: D6838406
fbshipit-source-id: e7260be5dea2b41e176fdc3508a78134cb1e9c35
Summary:
Previously we allowed a sync commit to happen at any time, but imported the
full repository to bring it into sync. This is slow for very large repositories
and might be accidentally triggered. We change this behavior to only do a sync
commit if only the imported client changed but nothing else.
Reviewed By: dsp
Differential Revision: D6838370
fbshipit-source-id: eecdfb0ea295585058784f8d1f70de5f8c733645
Summary:
This commit moves most of the stuff in hgext3rd and related tests to
hg-crew/hgext and hg-crew/test respectively.
The things that are not moved are the ones which require some more complex
imports.
Depends on D6675309
Test Plan: - tests are failing at this commit, fixes are in the following commits
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D6675329
