Summary:
Now both IdDag stores have remove_flat_segment implemented, let's test the
resize feature based on remove_flat_segment.
Reviewed By: DurhamG
Differential Revision: D33821387
fbshipit-source-id: 5b92adadf8cc6989298b2637e3fb7141898c4503
Summary:
The `index_parent_key` function can derive `group` from its `child_id`
parameter. So let's drop the redundant parameter.
Reviewed By: DurhamG
Differential Revision: D33821382
fbshipit-source-id: 3a31c3fa5d4049ec36974421c21347cbdc6c473c
Summary:
The `StoreId` is an implementation detail of `InProcessStore`. It does
not apply to other kind of stores. Move it to a better place.
Reviewed By: yancouto
Differential Revision: D33821388
fbshipit-source-id: d7d20d9a61ebeea58e22f3f779fc9651097857df
Summary: Previously the assertions run per entry. Make them only run once.
Reviewed By: yancouto
Differential Revision: D33821389
fbshipit-source-id: 6a7380071a3f014d2034dfbc5760f94bd6d99dc2
Summary:
See previous diff for context. Implements segment deletion for
`InProcessStore`. Extra care needs to be taken for serialization.
Reviewed By: DurhamG
Differential Revision: D33821381
fbshipit-source-id: ace59345425988c2dae30b9bd55388679c36ae63
Summary:
See D33779403 for context. This is a step forward to implement "strip".
This diff defines the API that can be used to remove or "shrink"
a flat segment and adds some sanity checks.
The actual implementation is in upcoming diffs.
Testing will focus on "remove" first, "resize" tests will be added later.
Differential Revision: D33821386
fbshipit-source-id: 61848593e698fe7f9358b25a4cb72788637c50a5
Summary:
See D31146158 (39f61996c7) for reasons why SegmentWithWrongHead exists.
The tl;dr is the parent->child index in the indexedlog implementation (key:
(parent, child), value: segments) can point to old segments that were replaced.
Let's ignore the "segments" from the index and just do an extra lookup
using "child" to find the up-to-date (replaced) segment. This simplifies
upcoming changes and provides better error messages in some cases.
Differential Revision: D33821383
fbshipit-source-id: 3f63ac0e3a3953d0ddcccebe2be4de2a3a3e0589
Summary:
Upcoming changes need the ability to query ranged parent->child for the entire
graph, not just the master group. Update the range query API to do that.
The only user of `iter_master_flat_segments_with_parent_span` is
`IdDag::to_first_ancestor_nth_known_universally_with_errors` and it is fine with
the change. Even better, by returning non-master segments we can answer the
`to_first_ancestor_nth` query with non-master ids. That makes the server more
flexible, as shown by the added `test_lazy_hash_on_non_master_group` test.
Differential Revision: D33821379
fbshipit-source-id: 330bd84f8535f857f6306f9ce791066688a39d48
Summary: See the previous diff for context. Add `IdMapWrite::remove_range` to remove range of entries in an IdMap.
Differential Revision: D33779404
fbshipit-source-id: 24ede5949ad3c1cb0b93c3c64ca692c04b354457
Summary:
By changing the index key it allows us to delete specific parent->child
entries in the index without affecting parent->other_child indexes.
Long story:
This is the first step to implement in-place "strip".
Having a proper way to "strip" unblocks:
- Repalce costly "rebuild_non_master" with more precisely removals without
rebuilding anything.
- Remove problematic commits in the master group to work better with stripped
heads server-side.
- Remove broken commits in the non-master group which lost corrosponding
data (ex. commit message, or trees).
Previously there were a few blockers to "strip":
- IndexedLog is append-only.
- Segments must be continuous.
Now the blockers are "mostly" clear:
- IndexedLog's Index supports removal by key.
- Discontinuous segments supports punching "hole"s.
Once problem about the index removal is the parent index. Previously it
is a single key => multiple entries index. For example, parent 6 might
have children [7, 8, 9]. This makes it harder to just remove child 8
without affecting children 7 or 9. By making the index key contain
child. The index will look like (6, 7) => [7], (6, 8 ) => [8], (6. 9)
=> [9] and it's easier to remove just (6, 8 ) without affecting others.
Note the values in indexedlog is a linked list so it's in theory possible
to make indexedlog support removing just part of the values. But at this
point I'd like to implement it without changing indexedlog. We can always
go back and revisit the indexedlog fancier index removal later.
Differential Revision: D33779403
fbshipit-source-id: 4d73e7bdc66d3a8ec1482fd25f8ed5b9cc301aec
Summary:
This makes use of the new Rust repo initialization added in previous commits into the `init` command in Python by replacing the initialization that was done mostly by the `localrepo` object. The codepaths for `git` and `bundle` remain active.
Another smaller addition was the option of removing `narrowheads` from the list of requirements if `experimental.narrow-heads` was explicitly disabled in the configurations.
Reviewed By: quark-zju
Differential Revision: D33901725
fbshipit-source-id: d9d976a1b844e1dd6a5e065f8ec8bf1d04fc4c03
Summary:
Prefetch code paths execute the prefetch as one huge batch, and it
turns out the lfs fetcher only fetches one file at a time. Let's increase the
parallelism
Note, checkout is less affected by this, since nativecheckout issues prefetches in smaller batches and in parallel, so we get some parallelism at a higher layer, but I do see a 15% speed up in a large file heavy repository.
Reviewed By: quark-zju
Differential Revision: D34020837
fbshipit-source-id: fb11c1aebf175b49563f48cb20f8b7c527fa3019
Summary: It moved from facebookexperimental to facebook sometime ago
Reviewed By: fanzeyi
Differential Revision: D34009265
fbshipit-source-id: c11ca0e92d9574aa8eccd31d578b9f61a411833c
Summary: This diff is to update the rust thrift file. Generated by cargo build.
Reviewed By: fanzeyi
Differential Revision: D33289588
fbshipit-source-id: f26a79ae60eadf2671ca0e06bcdb062fe583de17
Summary: Respect our "can_color()" method to determine if tracing output should be ansified or not.
Reviewed By: DurhamG
Differential Revision: D33985125
fbshipit-source-id: c025c48e40945f9dc0c51d928f663936f582e4d2
Summary: Add public autocargo mapping. Actual change is to eden_scm_opensource.toml and manifests/eden_scm, rest is from autocargo run.
Reviewed By: DurhamG
Differential Revision: D33896971
fbshipit-source-id: 19d75be205d9cb0179c99bd61f1be5500ba61129
Summary: scm/lib/version is needed by scm/lib/configparser. Move some targets so that they are visible to autocargo of scm/lib for the next diff. Moving the target saves needing to autocargo all of scm.
Reviewed By: quark-zju
Differential Revision: D33896972
fbshipit-source-id: f055b714675e0a85aac64defacc4ca54e16fa5ce
Summary:
Currently a single LFS 503 during clone/checkout aborts the operation and leaves the user in a situation that is difficult to recover from. Now we treat a 503 as any other 5XX server error and retry.
The 5XX retry count/backoff can be configured via the lfs.backofftimes config knob. It defaults in the code to [1, 4, 8], which is 3 retries with rand(1), rand(4) and rand(8) sleep in between.
503 can be served from proxygen indicating there are no live lfs servers, or from x2pagentd in cases of network failure.
Reviewed By: quark-zju
Differential Revision: D33804601
fbshipit-source-id: cbb3514051843d3517bb676cf7c83dce057e42b9
Summary:
Adds a method that will be used in the future for replacing the repo initialization done in Python.
More specifically, this method creates and populates the files required for a new repo in the same way as we currently do via the constructor of `localrepo` with the extensions and configs we have enabled by default.
Reviewed By: DurhamG
Differential Revision: D33901712
fbshipit-source-id: 0d12732f5c9d36fb115a97e53b356a88fd609b6c
Summary: In production we do currently see some cases of lfs_read_hash_mismatch. I realized it may be due to missing chunks rather than bad chunk integrity. The new "skip_hash_on_read" path is still able to detect size mismatch, so tweak things to only log mismatch message if the sizes match.
Reviewed By: quark-zju
Differential Revision: D33928640
fbshipit-source-id: 5b63db2b05bc7b53779cdb5ae0041a7d22bb3962
Summary: It is a left-over from D33518822 (b897a19073).
Reviewed By: DurhamG
Differential Revision: D33837780
fbshipit-source-id: f42fb7f9099ab4cb388c7bd8f038839fd6976cbf
Summary: Since `Repo` will soon contain more logic on its own not related to `clidispatcher`, we are moving its own crate.
Reviewed By: DurhamG
Differential Revision: D33883867
fbshipit-source-id: 94414e1e2692962bf491cc2abb4e1d3799cbcc03
Summary: Due to a bug in the x2pagent with websocket connection polling in h2 we had to force h1. Now it's fixed and the new version of the agent with the fix should be rolled out to (almost - it's corp) 100% of users.
Reviewed By: quark-zju
Differential Revision: D33842574
fbshipit-source-id: aaf5c66143806a900253120a9f750e37ebe55b80
Summary:
Point the thrift-types at the rust target, no need for the checked in generated code anymore
Saves generation, gives quicker signal on thrift changes (no codegen step), and stops thrift lib vs generated thrift code mismatches
Reviewed By: yancouto
Differential Revision: D33843588
fbshipit-source-id: 856faeb16dc0a2a42f2f07fd3005079a01b6e7bd
Summary: I moved some stuff around and am having trouble getting everything to be happy. One thing I forgot was to actually pipe the "fb" feature down from the entry point crates (hgcommands and backingstore).
Reviewed By: xavierd
Differential Revision: D33852732
fbshipit-source-id: aabdd145452e66435e130fcbab0ba8a80f9d943a
Summary:
This gets rid of some config boilerplate and allows for centralized cert checks in the http client object.
I left the subtle cert validation checks as-is. We will revisit these later.
Reviewed By: DurhamG
Differential Revision: D33717226
fbshipit-source-id: e9787f7068e5988b83d7c18b60d15ac6eae3c72c
Summary: This gets rid of some config boilerplate and allows for centralized cert checks in the http client object.
Reviewed By: DurhamG
Differential Revision: D33717225
fbshipit-source-id: c3baf271ac0bc56b1c21eab3fd363bccd7dcff37
Summary: This allows TLS config to flow through the http client object rather than being manually configured by each http user.
Reviewed By: DurhamG
Differential Revision: D33717227
fbshipit-source-id: e8f2a49b81f5d30aaedce3517bad8630fb770dfc
Summary:
Move the remaining hg_http::HgHttpConfig fields into http_client::Config. hg_http::http_config() now can derive these config fields from the hg config.
This change cause HTTP requests via EdenFS to:
1. Support x2pagentd unix socket proxy
2. Set X-Client-Info header
3. Respect the http.verbose hg config item to dump request details
Previously the above config items were only set by hg.
Reviewed By: quark-zju
Differential Revision: D33148772
fbshipit-source-id: 4359399aa2d915a2e833fa9ebf8c8860372f0810
Summary: Don't set the "fb" cargo feature via cmake unless we are in fb build.
Reviewed By: quark-zju
Differential Revision: D33781554
fbshipit-source-id: 39c61dc1631ebfd8a2c858b5e214c6a09ce0b005
Summary: fb303_core is used. fb303 is not, so remove it
Reviewed By: HarveyHunt
Differential Revision: D33815465
fbshipit-source-id: 149ef2cc48e2eed17fc859707300db555ea5b6b4
Summary:
On Windows we're reconverting certs for every request. Let's cache
them.
I thought about trying to store the cache on the client or something, but it
seemed more straight forward and fairly safe to just have a global cache.
Reviewed By: quark-zju
Differential Revision: D33690775
fbshipit-source-id: d3216f5403ea2912f3e49a37aad154d22922487c
Summary: The test was made pass for D33352777 (cf612fcd26) but didn't update for D33518822 (b897a19073).
Reviewed By: DurhamG
Differential Revision: D33746543
fbshipit-source-id: 6e3655a1c1806f6a177b1422b7800f56a5ddac17
Summary: The git repos using git protocols cannot use commitcloud and infinitepush.
Reviewed By: DurhamG
Differential Revision: D33688111
fbshipit-source-id: bc8f8d7b8b5c84f567546b486bef15b5c92c1f32
Summary:
Tags are treated as remote bookmarks prefixed with `tags/`. With previous
changes, pulling (via auto pull) or pushing tags just work. Add tests about it.
Reviewed By: DurhamG
Differential Revision: D33518819
fbshipit-source-id: 4e59d23d27cb92d884b66a24bf61bf2e36161fa7
Summary:
Previously the git remote url (ex. url of origin) is stored in git (by
`git remote add`). It has the benefit that git recognizes the names like
`"origin"`, and does the reference mapping work (ex. `refs/heads/foo` ->
`refs/remotes/origin/foo`) during pull.
But it turns out the approach has more downsides, namely:
- `hg paths --add` does not work.
- auto pulling like `hg checkout remote/tags/v1` does not work.
- `repo.pull` API is broken for git repos.
This diff changes the remote urls in git repos to be stored in `hgrc`, just
like a regular hg repo to address above issues.
Regarding on git, now we can maintain the reference explicitly so there is
no need for git to do the mapping on fetch.
Reviewed By: DurhamG
Differential Revision: D33518822
fbshipit-source-id: 3955d76bf76d73be60cb1363cbc3684f5336f605
Summary:
Some git repos (ex. pytorch) has thousands of branches that won't scale well.
Change no-argument pull to respect selective pull to mitigate it.
Reviewed By: DurhamG
Differential Revision: D33485186
fbshipit-source-id: b735704850847c49b0bfeb94972e4bb778e726a4
Summary:
Reading a null OID causes an error in libgit2:
odb: cannot read object: null OID cannot exist; class=Odb (9); code=NotFound (-3)
Since libgit2 special-cased the null id and hg wants to read null sometimes.
Let's just handle it in the gitstore.
Reviewed By: markbt
Differential Revision: D33485189
fbshipit-source-id: ed96921353ef592750fe361eee06ebcc0a25f07b
Summary:
Support git submodule in the trees (serialzation + deserialization).
This enables reading git trees with submodules without errors, and
modifying the trees without losing the submodules entries.
The added "submodule" type forces downstream users to update.
Namely, `checkout` is updated to silently ignore them.
Reviewed By: DurhamG
Differential Revision: D33369607
fbshipit-source-id: c2bce1882df3958b272dd8a6dcc3e4052f2704f5
Summary:
Use an enum variant for regular files so `Option` becomes unnecessary.
This makes the type simpler and makes upcoming changes cleaner.
Note: The type now matches `manifest::FileType` but `manifest::FileType`
will be changed in upcoming changes.
Reviewed By: danielocfb
Differential Revision: D33369606
fbshipit-source-id: 2b01d74875f97675d79082fe39bf4bdea26fdb07
Summary: Found it when I was reading the code.
Reviewed By: DurhamG
Differential Revision: D33458497
fbshipit-source-id: 06cc1e3d2afbcd345ba9b0471546aa5719106e77
Summary:
The `%include builtin:git.rc` will be written to repo `hgrc` when creating
repos backed by git. The main benefit is that we can update the builtin
git.rc without migrating existing hgrc files in the future.
Reviewed By: DurhamG
Differential Revision: D33352777
fbshipit-source-id: 9b6904e4bc0eff53bf623ce9ae382d723da1213e
