facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-07 15:27:13 +03:00
Code Issues Projects Releases Wiki Activity
a72a47d515
sapling/tests/test-p4fastimport-import.t

109 lines
3.1 KiB
Perl
Raw Normal View History

p4fastimport : introducing fast Perforce to Mercurial convert extension Summary: `p4fastimport` is a fast convert extensions for Perforce to Mercurial. It is designed to generate filelogs in parallel from Perforce. It tries to minimize the use of Perforce commands and reads from the the Perforce store on a Perforce server directly. The core of p4fastimport is the idea to generate a Mercurial filelog directly from the underlying Perforce data, as a Perforce file in most cases matches a filelog directly (per-file branches is an exception). To generate a filelog we are reading each file for an imported revision. A file in Perforce is locally either stored in RCS, as a compressed GZIP or as an flat file (binaries). If we do not find a version locally on disk we fallback to downloading it from Perforce. We are generating manifests after all filelogs are imported. A manifest is constructed by adding and removing files from an initial state. We are generating the correct offset from a manifest into the filelog by keeping track of how often a file was touched. We then generate the changelog. Linkrev generation is a bit tricky. For every file in Perforce know to which changelist it belongs, as it's stored revisions contains the changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this refers to the "original" changelist, before a potential renumbering, which is why we use the -O switch). We use the CL number obtained from the revision to reverse lookup the offset in the sorted list of changelists, which corresponds to it's place in the changelog later, and therefore it's correct linkrev. Parallel imports: In order to run parallel imports we MUST keep one lock at a time, even if we import multiple file logs at the same time. However filelogs use a singular `fncache`, which will be corrupted if we generate filelogs in parallel. To avoid this, repositories must be generated with *fncache* disabled! This restricts `p4fastimport` with workers to run only on case sensitive file systems. Test Plan: The included tests as well as multiple imports from a small testing Perforce client. Afterwards successfully run `hg verify` make tests Reviewers: #idi, quark, durham Reviewed By: durham Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4776651 Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
$ echo "[extensions]" >> $HGRCPATH
$ echo "p4fastimport= " >> $HGRCPATH
create p4 depot
$ p4wd=`pwd`/p4
$ hgwd=`pwd`/hg
$ P4ROOT=`pwd`/depot; export P4ROOT
$ P4AUDIT=$P4ROOT/audit; export P4AUDIT
$ P4JOURNAL=$P4ROOT/journal; export P4JOURNAL
$ P4LOG=$P4ROOT/log; export P4LOG
$ P4PORT=localhost:$HGPORT; export P4PORT
$ P4DEBUG=1; export P4DEBUG
$ mkdir $hgwd
$ mkdir $p4wd
$ cd $p4wd
start the p4 server
$ [ ! -d $P4ROOT ] && mkdir $P4ROOT
$ p4d -f -J off >$P4ROOT/stdout 2>$P4ROOT/stderr &
$ echo $! >> $DAEMON_PIDS
$ trap "echo stopping the p4 server ; p4 admin stop" EXIT
$ # wait for the server to initialize
$ while ! p4 ; do
> sleep 1
> done >/dev/null 2>/dev/null
create a client spec
$ cd $p4wd
$ P4CLIENT=hg-p4-import; export P4CLIENT
$ DEPOTPATH=//depot/...
$ p4 client -o | sed '/^View:/,$ d' >p4client
$ echo View: >>p4client
$ echo " $DEPOTPATH //$P4CLIENT/..." >>p4client
$ p4 client -i <p4client
Client hg-p4-import saved.
populate the depot
$ mkdir Main
$ mkdir Main/b
$ echo a > Main/a
$ echo c > Main/b/c
$ echo d > Main/d
$ p4 add Main/a Main/b/c Main/d
//depot/Main/a#1 - opened for add
//depot/Main/b/c#1 - opened for add
//depot/Main/d#1 - opened for add
$ p4 submit -d initial
Submitting change 1.
Locking 3 files ...
add //depot/Main/a#1
add //depot/Main/b/c#1
add //depot/Main/d#1
Change 1 submitted.
$ p4 edit Main/a Main/b/c Main/d
//depot/Main/a#1 - opened for edit
//depot/Main/b/c#1 - opened for edit
//depot/Main/d#1 - opened for edit
$ echo a >> Main/a
$ echo c >> Main/b/c
$ echo d >> Main/d
$ p4 submit -d second
Submitting change 2.
Locking 3 files ...
edit //depot/Main/a#2
edit //depot/Main/b/c#2
edit //depot/Main/d#2
Change 2 submitted.
Simple import
$ cd $hgwd
$ hg init --config 'format.usefncache=False'
$ hg p4fastimport --debug -P $P4ROOT hg-p4-import
loading changelist numbers.
2 changelists to import.
loading list of files.
3 files to import.
importing repository.
writing filelog: b789fdd96dc2, p1 000000000000, linkrev 0, 2 bytes, src: *, path: depot/Main/a (glob)
writing filelog: a80d06849b33, p1 b789fdd96dc2, linkrev 1, 4 bytes, src: *, path: depot/Main/a (glob)
writing filelog: 149da44f2a4e, p1 000000000000, linkrev 0, 2 bytes, src: *, path: depot/Main/b/c (glob)
writing filelog: b11e10a88bfa, p1 149da44f2a4e, linkrev 1, 4 bytes, src: *, path: depot/Main/b/c (glob)
writing filelog: a9092a3d84a3, p1 000000000000, linkrev 0, 2 bytes, src: *, path: depot/Main/d (glob)
writing filelog: f83f0637e55e, p1 a9092a3d84a3, linkrev 1, 4 bytes, src: *, path: depot/Main/d (glob)
changelist 1: Writing manifest.
changelist 1: Writing changelog: initial
changelist 2: Writing manifest.
changelist 2: Writing changelog: second
2 revision(s), 3 file(s) imported.
Verify
$ hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
3 files, 2 changesets, 6 total revisions
$ hg update tip
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
End Test
stopping the p4 server
Reference in New Issue Copy Permalink