Summary:
This diff contains proposal for basic rebase model for new checkout.
This diff introduces two types of file changes - basic no conflict Actions(that will replace CheckoutPlan actions) and ConflictAction for actions that contain conflicts
No conflict actions are produced when comparing two manifests
Conflict actions can be produced when rebasing manifests.
Eventually we will adjust checkout code to take either list of Actions or Conflict actions and perform corresponding operations on the file.
Rebase::rebase is used to generate list of conflict actions when rebasing.
Currently `Rebase::rebase` produces list of actions on top of 'base' commit.
Before doing actual checkout we will need to take into account what current commit is and adjust this list of actions accordingly (however, this adjustment will be trivial, it won't produce more conflicts)
After getting adjusted list of actions, checkout code can roll it out to fs and call merge drivers if needed.
In the future, if we will want to allow check in conflicts, we will need to adjust `Rebase::resolve_conflict` to take `ConflictAction` as input, instead of an Action.
Reviewed By: DurhamG
Differential Revision: D27028001
fbshipit-source-id: b02b7ad8030fcf146b53639d69e66fdc8f38a62d
Summary:
Since we're rolling out native checkout and resumable checkout around
the same time, let's make resumable checkout optional so we can turn it off it
causes issues, without turning off native checkout.
Reviewed By: quark-zju
Differential Revision: D27481986
fbshipit-source-id: a0a3e68567ca2a468e852dce95c03c4b606aaf22
Summary:
We already checked size and mtime, which should allow us to identify if
a file has changed since the last update, but we need to check hgid as well, to
make sure the previously-written content is supposed to be the same as the
about-to-be-written content
Reviewed By: andll
Differential Revision: D26955401
fbshipit-source-id: 859ad35b008e68d699601217f2a4398c6789913e
Summary:
Updates native checkout to store which files have already been written
in .hg/upgradeprogress. Then enables it to load that file and skip writing those
files if they're already on disk and their mtime and size match the previously
written values. In theory we could record and check file hashes as well, but
that'd likely slow things down quite a bit.
Future diffs will add:
- Recording and checking the hgid that was written before vs what is about to be
written. Just an hgid comparison, not a full hash computation.
- Some UX to inform the user when hg checkout can be continued, and possibly to
implement 'hg checkout --continue'.
Reviewed By: andll
Differential Revision: D26830249
fbshipit-source-id: 88a75080966dae5241550ed7eedbc057c65966dd
Summary:
During sparse profile change, there might be a case when file has changed between checkouts, and also is added due to sparse profile change
Currently this would lead to creating duplicated update action that would waste resources. This diff deduplicates such action
Reviewed By: quark-zju
Differential Revision: D26953657
fbshipit-source-id: ccd9084cc3984ac0692ee6178fce018a94cd56e7
Summary:
Current store implementation returns value that may contain matadata.
We need to strip it before writing out file to disk
This is not going to be a problem with new Eden API, but it is not erady yet and rolling it out may take time
So in order to start testing native checkout now, we are incorporating metadata removal in the checkout code
Reviewed By: quark-zju
Differential Revision: D26953658
fbshipit-source-id: 05d8be55bb96e7c3162594498ef0b9944b422c90
Summary: These were generally useful when debugging native checkout.
Reviewed By: quark-zju
Differential Revision: D26955400
fbshipit-source-id: 967a4fa2d831b8b060409a5f7be31a1416c1ba63
Summary: The goal is to reduce load on tokio scheduler by using threads & channels instead of spawning new task every time
Reviewed By: DurhamG
Differential Revision: D26801249
fbshipit-source-id: a8d9accc721c7ffc981fd538c06ab8cbd908f715
Summary: This treats update_meta collection same way as update_content
Reviewed By: DurhamG
Differential Revision: D26694077
fbshipit-source-id: f2a7afb988eb99afa6e8f62ae6cda57a108a987d
Summary: This diff adds unit test for sparse checkout plan
Reviewed By: DurhamG
Differential Revision: D26673701
fbshipit-source-id: 8f810df317c3447f8445765e3a510559d4f1b378
Summary:
This method updates checkout plan to account for sparse profile change
The logic is somewhat similar to sparse.py, except it is smaller because it does not account for merges
Reviewed By: DurhamG
Differential Revision: D26670413
fbshipit-source-id: 54294ad7f1c61e39400450ff139f40ac8fffdd62
Summary:
Currently running native checkout shows much higher sys time then regular checkout
One theory that this is due to high scheduler overhead due to large number small tasks spawned with spawn_blocking
This diff batches 100 operations before spawning blocking proc
Reviewed By: quark-zju
Differential Revision: D26620395
fbshipit-source-id: f5be84c3ee14ffc55e4332e714a48f55701d5f8f
Summary:
This diff makes CheckoutPlan::apply take &self instead of self, so that CheckoutPlan can be preserved.
This result in cloning of some paths(but realistically, they probably get moved anyway in this code path, so this should not have perf impact)
This will simplify pycheckout as it will allow to keep plan and later use it to update dirstate
This diff also exposes accessors for dirstate update (e.g. removed_files / updated_content_files / updated_meta_files)
Reviewed By: quark-zju
Differential Revision: D26551118
fbshipit-source-id: 4e2d02da0a692345eae7d5e7124663c577e9029c
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: This helps with pipelining (once chunk is prefetched it will start rolling out to fs) and also helps with limiting number of keys prefetched at once
Reviewed By: quark-zju
Differential Revision: D26496789
fbshipit-source-id: 9aa6b08c1605ab2a06a02347897f5dcfa22297fd
Summary: Also removing todo: test from checkout plan as there are tests already
Reviewed By: singhsrb
Differential Revision: D26496790
fbshipit-source-id: eb99771cb0cda7a90477190875997027b25e2482
Summary: This is most universal/flexible interface
Reviewed By: quark-zju
Differential Revision: D26410641
fbshipit-source-id: cae74675f89c0c1c05c84331c3ce0e78fd0b0b2f
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary: This diff adds auto-generated test cases to checkout code. It generates partially overlapping trees and tests transitions between them
Reviewed By: quark-zju
Differential Revision: D26384962
fbshipit-source-id: 6140bbb7ff8b87843a2235f8325f57829cdd8cae
Summary:
Before this diff VFS::write_regular did not handle correctly use case when file already existed as as symlink - it would write into symlink location, instead of replacing symlink with a regular file (see updated test_symlink_overwrite that is failing on old implementation)
This diff adds O_NOFOLLOW option on unix when overwriting the file. When destination is a symlink, attempt to write fails with E_LOOP and triggers clear_conflict that removes symlink and allows retry write to succeed.
This also allows one of test cases in checkout that previously did not work
Reviewed By: quark-zju
Differential Revision: D26378893
fbshipit-source-id: 28bcdaba78db283ac7a25bb232c198d3d8f73e5d
Summary: This diff contains basic test setup for checkout tests - we compare transition between two trees without dirty changes
Reviewed By: quark-zju
Differential Revision: D26359502
fbshipit-source-id: ef670c944200bae1652863c91ada92c6fecce4ac
Summary: Allow different type of storages for CheckoutPlan::apply
Reviewed By: quark-zju
Differential Revision: D26291577
fbshipit-source-id: a1c9ba4dbef09e844727ae32eac25c37dd01358e
Summary:
Previously, `write` can fail because the destination file exists as a
directory, or the parent directory is missing. pyworker handles those cases
by calling `clear_conflicts` to remove conflicted directories and create
missing parent directories and retry `write`. Practically, for all `write`
usecases (including checkout) we always want the `clear_conflicts` behavior.
Therefore, move `clear_conflicts` to vfs `write` and make it private.
Reviewed By: quark-zju
Differential Revision: D26257829
fbshipit-source-id: 03d1da0767202edba61c47ae5654847c0ea3b33e
Summary: Currently we pick simple approach and just delete existing file before re-creating it if there is some non-trivial flag change
Reviewed By: quark-zju
Differential Revision: D26239977
fbshipit-source-id: 167efa1bf6018e7f967ef3a9e3c8c62781486ec9
Summary:
Basic implementation only process file updates for now.
See todos on CheckoutPlan::apply for more detail
Reviewed By: quark-zju
Differential Revision: D26209984
fbshipit-source-id: fcfbf568359d553a51ea02ea194634048d093d0e
Summary: This is needed becase later during checkout list of updates that needs fetch will go into storage stream
Reviewed By: quark-zju
Differential Revision: D26209983
fbshipit-source-id: 9fb54d48c6f0afc4fb67320aafc2e981c96ab5a9
