2017-04-18 04:08:07 +03:00
|
|
|
#require p4
|
|
|
|
|
|
2017-05-12 03:13:31 +03:00
|
|
|
$ . $TESTDIR/p4setup.sh
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
|
|
|
|
|
populate the depot
|
|
|
|
|
$ mkdir Main
|
|
|
|
|
$ mkdir Main/b
|
|
|
|
|
$ echo a > Main/a
|
2017-08-02 10:00:47 +03:00
|
|
|
$ ln -s ../a Main/b/c
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
$ echo d > Main/d
|
|
|
|
|
$ chmod +x Main/d
|
|
|
|
|
$ p4 add Main/a Main/b/c Main/d
|
|
|
|
|
//depot/Main/a#1 - opened for add
|
|
|
|
|
//depot/Main/b/c#1 - opened for add
|
|
|
|
|
//depot/Main/d#1 - opened for add
|
|
|
|
|
$ p4 submit -d initial
|
|
|
|
|
Submitting change 1.
|
|
|
|
|
Locking 3 files ...
|
|
|
|
|
add //depot/Main/a#1
|
|
|
|
|
add //depot/Main/b/c#1
|
|
|
|
|
add //depot/Main/d#1
|
|
|
|
|
Change 1 submitted.
|
|
|
|
|
|
|
|
|
|
$ p4 edit Main/a Main/b/c Main/d
|
|
|
|
|
//depot/Main/a#1 - opened for edit
|
|
|
|
|
//depot/Main/b/c#1 - opened for edit
|
|
|
|
|
//depot/Main/d#1 - opened for edit
|
|
|
|
|
$ echo a >> Main/a
|
|
|
|
|
$ echo d >> Main/d
|
|
|
|
|
$ p4 submit -d second
|
|
|
|
|
Submitting change 2.
|
|
|
|
|
Locking 3 files ...
|
|
|
|
|
edit //depot/Main/a#2
|
|
|
|
|
edit //depot/Main/b/c#2
|
|
|
|
|
edit //depot/Main/d#2
|
|
|
|
|
Change 2 submitted.
|
|
|
|
|
|
|
|
|
|
Simple import
|
|
|
|
|
|
|
|
|
|
$ cd $hgwd
|
|
|
|
|
$ hg init --config 'format.usefncache=False'
|
|
|
|
|
$ hg p4fastimport --debug -P $P4ROOT hg-p4-import
|
|
|
|
|
loading changelist numbers.
|
|
|
|
|
2 changelists to import.
|
|
|
|
|
loading list of files.
|
|
|
|
|
3 files to import.
|
2017-05-02 06:43:11 +03:00
|
|
|
reading filelog * (glob)
|
|
|
|
|
reading filelog * (glob)
|
|
|
|
|
reading filelog * (glob)
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
importing repository.
|
2017-04-25 17:29:39 +03:00
|
|
|
writing filelog: b789fdd96dc2, p1 000000000000, linkrev 0, 2 bytes, src: *, path: Main/a (glob)
|
|
|
|
|
writing filelog: a80d06849b33, p1 b789fdd96dc2, linkrev 1, 4 bytes, src: *, path: Main/a (glob)
|
2017-08-02 10:00:47 +03:00
|
|
|
updating the branch cache (?)
|
|
|
|
|
writing filelog: 1f6b5bb93f1d, p1 000000000000, linkrev 0, 4 bytes, src: *, path: Main/b/c (glob)
|
|
|
|
|
writing filelog: 3b479db02621, p1 1f6b5bb93f1d, linkrev 1, 4 bytes, src: *, path: Main/b/c (glob)
|
2017-04-25 17:29:39 +03:00
|
|
|
writing filelog: a9092a3d84a3, p1 000000000000, linkrev 0, 2 bytes, src: *, path: Main/d (glob)
|
|
|
|
|
writing filelog: f83f0637e55e, p1 a9092a3d84a3, linkrev 1, 4 bytes, src: *, path: Main/d (glob)
|
2017-08-02 10:00:47 +03:00
|
|
|
changelist 1: writing manifest. node: 05414d16d473 p1: 000000000000 p2: 000000000000 linkrev: 0
|
2017-04-20 09:33:06 +03:00
|
|
|
changelist 1: writing changelog: initial
|
2017-08-02 10:00:47 +03:00
|
|
|
changelist 2: writing manifest. node: fb65d73ad7d5 p1: 05414d16d473 p2: 000000000000 linkrev: 1
|
2017-04-20 09:33:06 +03:00
|
|
|
changelist 2: writing changelog: second
|
2017-05-19 01:19:14 +03:00
|
|
|
updating the branch cache (?)
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
2 revision(s), 3 file(s) imported.
|
|
|
|
|
|
|
|
|
|
Verify
|
|
|
|
|
|
|
|
|
|
$ hg verify
|
|
|
|
|
checking changesets
|
|
|
|
|
checking manifests
|
|
|
|
|
crosschecking files in changesets and manifests
|
|
|
|
|
checking files
|
|
|
|
|
3 files, 2 changesets, 6 total revisions
|
|
|
|
|
|
|
|
|
|
$ hg update tip
|
|
|
|
|
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
2017-08-02 10:00:47 +03:00
|
|
|
|
|
|
|
|
# Ensure that the file is a symlink (-L) and is valid (-f) this ensures
|
|
|
|
|
# we correctly handle symlinks.
|
|
|
|
|
|
|
|
|
|
$ test -L Main/b/c
|
|
|
|
|
$ test -f Main/b/c
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
$ hg --debug manifest
|
2017-04-25 17:29:39 +03:00
|
|
|
a80d06849b333b8a3d5c445f8ba3142010dcdc9e 644 Main/a
|
2017-08-02 10:00:47 +03:00
|
|
|
3b479db02621d5ff591921d4946681bebd4b2e2e 644 @ Main/b/c
|
2017-04-25 17:29:39 +03:00
|
|
|
f83f0637e55e3c48e9922f14a016761626d79d3d 755 * Main/d
|
p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.
The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.
We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.
We then generate the changelog.
Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g. 1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch). We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.
Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.
Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`
make tests
Reviewers: #idi, quark, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776651
Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 21:11:09 +03:00
|
|
|
|
|
|
|
|
End Test
|
|
|
|
|
|
|
|
|
|
stopping the p4 server
|
