Summary:
seqimport currently has a gap: if a changelist touches files outside of
clientspec, it will blow up when trying to get the move info for it. Even if that
was not the case, it could blow up if the file was moved into the clientspec.
This change makes it resilient to that by providing the same list created by
`parse_fstat` to prefilter files we check for move info, and then making
`parse_where` take an optional parameter saying it is fine if file is not on
client (i.e. moved into clientspec)
Differential Revision: D7574415
fbshipit-source-id: 63f6a32436d3d53d6f9402575a9a13bb4187b76c
Summary:
Turns out I incorrectly assessed this situation before. We do use content from
perforce servers a lot. This change makes p4seqimport read from local disk
directly if possibel rather than resorting solely on `p4 print` to obtain file content.
```name=Checking file content src on master-importer task 0 (running for 15h+)
[15:40:23 twsvcscm@priv_global/independent_devinfra/ovrsource-master-importer/0 ~]$ egrep -o 'src: (gzip|rcs|p4)' /logs/stdout | sort | uniq -c
2567 src: gzip
24 src: p4
```
Differential Revision: D7388797
fbshipit-source-id: 5fe1a525bc211d64a75954d529edc152d22970a7
Summary:
dsp had a look at the whole stack and suggested some changes:
* Only write bookmark once at the end of the import - we are doing a single transaction anyways so updating the bookmark after every changelist import is moot
* Remove unused function seqimporter.ChangelistImporter._safe_open
* Require fncache to preserve behavior from p4fastimport
Differential Revision: D7375481
fbshipit-source-id: f4407d5d0276f96d72bf67544091640fe1c46044
Summary:
When testing p4seqimport with remotefilelog it would barf on call to `.tip()`,
because remotefilelog doesn't have that.
This change makes use of the change context from the repo instead to get the
tip node.
Differential Revision: D7294979
fbshipit-source-id: 18b4a5107f4cbf676016d44d5134bf0d252eeff3
Summary:
Perforce supports RCS keyworded files, more info here:
http://answers.perforce.com/articles/KB/3482
We replace things back in p4fastimport, this replicates the behavior in
p4seqimport (unit test should clarify what this means)
Differential Revision: D7188163
fbshipit-source-id: 594f71d6114c73001753ae36c4973c2db3310e62
Summary:
Respect the executable bit on files based on perforce type.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7185388
fbshipit-source-id: 59afec7bd857572b8347ebe546d131017a79928c
Summary:
p4seqimport has used very high level mercurial abstractions so far (almost
equivalent to running hg add / mv / rm / commit on command line). This is very
easy to grasp as we use it day to day. It is not performant enough for our
importer:
- It does the work twice (write to working copy, then commit changing hg metadata)
- It requires the working copy (this would force us to update between revs,
materializing a prohibitively large number of files)
This change makes use of memctx, which is basically an in-memory commit. This way
we don't need a working copy and we save time + a lot of space.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7176903
fbshipit-source-id: 2773d7c001b615837496ea9db3229d9afc020124
Summary:
p4seqimport has a bookmark option, it was completely ignored before this change.
This makes use of the opt, moving the bookmark as we import changes.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7172867
fbshipit-source-id: be63765088b0583df2e1c9e0ccec869c5278d782
Summary:
Properly create files as symlinks if they are symlinks in P4
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Reviewed By: wlis
Differential Revision: D7157772
fbshipit-source-id: ac3e5010f3d15460592a449c817824c0b28a8435
Summary:
Similar to #10 (D7113181), we need to track large files.
This change adds the bits to do so, reusing the logic from p4fastimport which was
moved to lfs.py
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7115654
fbshipit-source-id: 56ccfadf6fa14dcfb8005cc5ef03fb175835bcda
Summary:
This change makes seqimport write revision info (i.e. (CL, hghash) pairs) to a
sqlite file. This is used by the importer TW job wrapper to write the info
into `xdb.p4sync` table `revmap`
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7113181
fbshipit-source-id: e55a8cf0b794216a4855ae7486885c3d956cd7fb
Summary:
Adds p4changelist to commit extra info
With p4changelist info, make p4seqimport incremental
Add debug message to have more accurate info on what is actually being imported
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7090090
fbshipit-source-id: 17529aa57452453cfe29c3c3dc9d9e7daa8cffb2
Summary:
Adds copy tracing to `p4seqimport` by:
- Leveraging `fromFile` from `p4 -ztag describe` to introduce source for moved
files into P4Changelist.load's
- Utilizing that info from P4 CL when creating hg commit
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7074892
fbshipit-source-id: e105a608bb953a8137ec6c9afc7e0571a902c868
Summary:
Consolidates manipulation of p4 CL info into p4 module, pulling the relevant code
out of ChangeManifestImporter creategen so it can be easily shared by
p4fastimport and p4seqimport
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7064179
fbshipit-source-id: 72c5bcad209eebf40ec8152a07f98f7f7fa544fb
Summary:
Adds logic to create the commit, using info from p4 CL + the list of added and
removed files.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7063983
fbshipit-source-id: c64e44c19d06e54fe35121a8d6128de050f93823
Summary:
Read file from perforce, write into the hg repo.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7050157
fbshipit-source-id: 4389ba11f62c8ed825d6a6ef3c001095339eb551
Summary:
Creates ChangelistImporter, which will be responsible for translating a p4 CL to
a hg commit
For now it only goes through files touched by the CL and lists what was added or
removed. Next diffs will evolve it to the point where it effectively performs the
translation.
For a high-level overview of p4seqimport, please check https://our.intern.facebook.com/intern/wiki/IDI/p4seqimport/
Differential Revision: D7049961
fbshipit-source-id: 6a9f3bd57cadc2b9ea8a81373cc10dfda76311e7
