Summary:
Dechunker has a feature of saving entire dechunked bundle contents in memory
this is used to save the raw bundles to manifold. Unitl now this feature worked
properly when accesed via `Read` trait methods. When accessed via `BufRead`
trait the logic that was collecting the read contents was skipped.
This manifested in the saved infinitepush bundles being always trimmed to 4kb.
Reviewed By: markbt
Differential Revision: D25020371
fbshipit-source-id: c606c9fb116a1cd00ae7f4558a7249364faa9c13
Summary:
Add Blame to walker so that we can scrub it.
For public commits, if blame is requested we wait for blame to be derived before traversing unodes. This saves traversing the unode tree twice.
For non-public commits they are not expected to have blame, so for those we don't wait for blame to be derived before traversing unodes.
Reviewed By: farnz
Differential Revision: D24896243
fbshipit-source-id: 66226a8e47f115bcda62269ade63874e0fff4ba0
Summary:
This change allows the walk steps to check the underlying WalkState's phase information to see if a changeset is public. Its exposed to the steps via Checker::is_public().
This saves repeat checks from the changeset phase from the bonsai_to_hg_mapping_step if the walk state already knows its public, and makes sure we are passing the get_phase ephemeral_derive flag from just one place (it should always be the opposite of enable_derive).
Reviewed By: farnz
Differential Revision: D24954837
fbshipit-source-id: b911d69837db8ef34fbe2c27f642d6819ea46908
Summary:
delegate FromStr, AsRef and sampling_fingerprint from BlameId to the
inner FileUnodeId
Reviewed By: markbt
Differential Revision: D24917140
fbshipit-source-id: 1f39b15c91c1f90f371baf03d97f03d00d8798ea
Summary:
The NodeTypes variants and the EdgeType variants generated from them were getting rather long which makes them hard to work with.
This change shortens them by
- changing BonsaiChangeset* to Changeset (this also has better consistency with changeset its blobstore keys)
- updating Bonsai*Mapping to *Mapping )except for BonsaiHgMapping where it disambiguates with HgBonsaiMapping)
Due to the way walker output is sorted it made more sense to do these in one diff rather than two.
Reviewed By: markbt
Differential Revision: D24864061
fbshipit-source-id: dbd395a89be828ac97cf056f03787097d8f1491d
Summary: The test wasn't compiling due to these 3 missing headers, add them.
Reviewed By: genevievehelsel
Differential Revision: D24997597
fbshipit-source-id: 6e3be0763362e41be138c670dc88a63bd9e88024
Summary:
We've seen some hangs with http 2 in lfs. Switching to http 1.1 seems
to fix it. Let's make this configurable so we can tweak this if we see it in
edenapi. For now we continue to default to http 2.
Reviewed By: krallin
Differential Revision: D24901201
fbshipit-source-id: 9806e2c37fa299e4bd381ebdcb17d00800408de3
Summary:
To support the ability for Mercurial to talk to Mononoke directly while going
through an HTTP proxy, we need to be able to accept HTTP prefixed requests.
This adds the support for Mononoke side. Mercurial client will be added in
subsequent diff.
Reviewed By: krallin
Differential Revision: D24989335
fbshipit-source-id: 597eaa974cea661332967e34abc80b2b609b94ff
Summary:
This is a preparatory phase to make the refcount usuable on Windows. For more
details, see D24716801 (e50725d2cb)
Reviewed By: chadaustin
Differential Revision: D24764568
fbshipit-source-id: 1e8c6ab00d4c1ec79c347fd5ae7167b2ce1dff68
Summary:
The skip_on_mode_mac argument to cpp_unittest doesn't exist on Windows, and to
be consistent with the rest of the code, we can simply ifdef the code that
either doesn't compile, or doesn't run.
Note: For some reason the accessIncrementsAccessCount test fails when running
on Windows, but only if running after accessAddsProcessToProcessNameCache...
Reviewed By: chadaustin
Differential Revision: D24496450
fbshipit-source-id: fe18fe1d791a27fbe4bd03bd3e8c811feeb23f5f
Summary:
This is what `eden debug` shows:
modified (m, a, t, e, r, i, a, l, i, z, e, d)
Enumerate all potentially-modified inode paths
Reviewed By: chadaustin
Differential Revision: D24908570
fbshipit-source-id: 62e91b6f0212c70de4bb1705539d3674e6e000d9
Summary:
On Windows, `eden debug file_stats` would crash due to the fileSize not being
filled. Since computing the file size is just a matter of calling stat(2) on
the file, let's simply do that.
Reviewed By: chadaustin
Differential Revision: D24908430
fbshipit-source-id: 07ffb97ada15a07565bea397b436fd21d09b5565
Summary:
We had a bug: if two files were reverted and then we try to amend one of them
mercurial will actually amend both of them.
Looks like the problem was caused by "Prune files reverted by amend" block.
Previously this block was considering every file that was changed in a commit we
are about to amend and comparing with working copy. If a file is the same in a
commit we are about to amend and in the working copy then it will amended as
well.
This diff attempts to fix it by considering only files that were selected for
amending.
Reviewed By: DurhamG
Differential Revision: D24949727
fbshipit-source-id: cf6cb95af3f67ec769e8a58db3b829945133b830
Summary:
We have an edge case - if we reverted two files and then we try to amend only a
single one then both of them will be amended.
This diff adds a test for this and other edge-cases. The next diff will fix it
Reviewed By: DurhamG
Differential Revision: D24949726
fbshipit-source-id: c5c53de1d67f161efa8564f89127e61ac2f28ac9
Summary:
`BonsaiChangesets` are immutable. Rather than recomputing their changeset ids
every time (which involves cloning, serializing, and hashing the changeset),
instead store it alongside the changeset data, computing it once at the point
the changeset is frozen.
When loading the changeset, we already know the id, so there's no need to
compute it at all.
Reviewed By: farnz
Differential Revision: D24951230
fbshipit-source-id: 5350e94eb6ea799a89ced2a211baa657a06b83d0
Summary:
When we streamclone, we snapshot the revlogs under a lock, then we start
sending. That works fine, unless we have a file whose size changes during
the sending phase. This can happen if it's promoted from a single `.i` to a
`.i` and a `.d`.
When that happens, the clone fails (the client reports it received a bunch of
corrupted data because it starts interpreting parts of files as inputs). Since
the breakage is also confusing client side, I updated the server side to also
assert that it's sending what it thinks it's sending.
Reviewed By: DurhamG
Differential Revision: D24958885
fbshipit-source-id: a0349c651b7cb63ab27546adf9944e7fba63a95d
Summary: This is unused, except in tests, so let's just remove it.
Reviewed By: chadaustin
Differential Revision: D24930011
fbshipit-source-id: cb132962e1dff9d12ce12e7eb75bd34a026c58b7
Summary:
As of right now, opendir is the expensive callbacks due to fetching the sizes
for all the files in a directory. This strategy however breaks down when
timeouts are added as very large directories would trigger the timeout, not
because EdenFS is having a hard time reaching Mononoke, but because of
bandwidth limitation.
To avoid this issue, we need to have a per-file timeout and thus makes opendir
just trigger the futures, but not wait on them. The waiting bit will happen
at readdir time which will also enable having a timeout per file.
The one drawback to this is that the ObjectFetchContext that is passed in by
opendir cannot live long enough to be used in the size future. For now, we can
use a null context, but a proper context will need to be passed in, in the
future.
Reviewed By: wez
Differential Revision: D24895089
fbshipit-source-id: e10ceae2f7c49b4a006b15a34f85d06a2863ae3a
Summary:
The rest of the non debug commands use - instead of _.
Lets flip the prefretch profiles cli to be consistant.
Reviewed By: genevievehelsel
Differential Revision: D24910172
fbshipit-source-id: a5f18a9c9d5fb4ef9417f14ef9d053cdc1599d76
Summary:
The second line of the summary got cut off due to a bad multiline comment
signals_oops
Reviewed By: wez
Differential Revision: D24909886
fbshipit-source-id: 844778e7d47a2b7b413fdd0c4fa0ef71dec9dadb
Summary:
`eden prefetch` can trigger a bunch of wasted network requests to fetch metadata
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
So lets skip metadata prefetching during eden prefetches.
Reviewed By: genevievehelsel
Differential Revision: D24658066
fbshipit-source-id: f8a32807a4e238222158f100cdd5ffa1b92fd833
Summary: Fix the walker build. Had a land race on non-conflicting lines.
Reviewed By: farnz
Differential Revision: D24953152
fbshipit-source-id: b3d43d745242542ba1a6df9ba730460112bee6dc
Summary:
This diff makes it possible to rate limit public or infinitepush pushes for individual repos based on the total number of file changes across all commits accepted in the given sliding window.
## Why support infintiepush rate limiting?
We want to hit the same servers with infinitepush and public traffic, yet we want to avoid the situation when a spike in infinitepush pushes (like after a rebase of a huge codemod stack) makes our servers incapable of serving public writes.
## Why support public rate limiting?
I don't think there's any reason to do it, but it's essentially free, so why not.
## Why per repo rather than cumulative?
If we only rate-limit cumulative traffic, a single misbehaving repo can fill in the quota and prevent other repos from accepting any traffic even with small limits.
Reviewed By: mitrandir77
Differential Revision: D24871837
fbshipit-source-id: 6da36794b2fdddd70f1a54e0afeaa4c49065a98d
Summary: Step into history via UnodeFile parents to check they are valid
Reviewed By: markbt
Differential Revision: D24838456
fbshipit-source-id: 3b19973f9f7da6e502595b4ffa33a031e1df03ab
Summary: Check that the UnodeFile's link is good by stepping to it.
Reviewed By: markbt
Differential Revision: D24838458
fbshipit-source-id: c6c8ef5dc580bb591f484df659a76f0ba43a68e6
Summary: add UnodeFile to walker graph to completment UnodeManifest, along with edges from UnodeFile to FileContent
Reviewed By: ikostia
Differential Revision: D24838460
fbshipit-source-id: 4bc860a65c5489ba9b2bff4582967d9db6181f25
Summary: When doing a deep walk step into history via the unode's linked changeset.
Reviewed By: ikostia
Differential Revision: D24838459
fbshipit-source-id: cf2196f95e7b6bf3c3ba2cd1c44fef748a11597f
Summary: add UnodeManifest support to walker so they can be scrubbed
Reviewed By: mitrandir77
Differential Revision: D24604174
fbshipit-source-id: c60a020802fae109005263393ea5618f22f21888
Summary: Add bonsai to unode manifest mapping to the walker so we can scrub from bonsais
Reviewed By: mitrandir77
Differential Revision: D24604188
fbshipit-source-id: d98938c5ec640c479007b59163899c16d0e3fb9d
Summary:
Track ChangesetInfo derivation in the same way as Fsnode. This fixes repeated visits to ChangesetInfo
As part of this the matching in WalkState::needs_visit is made exhaustive, as it the default match had previously been returning true for ChangesetInfo nodes.
Reviewed By: mitrandir77
Differential Revision: D24838455
fbshipit-source-id: 33b8201984b294a1560da104b2cc9c65849b9297
