Summary:
After D27144492 (48cd15ab14) we disabled revision number resolution. There is no need to
consider it when calculating shortest prefix.
Reviewed By: DurhamG
Differential Revision: D28072997
fbshipit-source-id: 832017c7b626265eb8cd2dd78946a03c4e7228f6
Summary:
This diff defines symlink type in `DirType`.
Even though it is not directly used in the FSCK diff. This will allow us to support symlink in EdenFS Windows in the future.
Reviewed By: genevievehelsel
Differential Revision: D28016305
fbshipit-source-id: 67c1aa22e39198f9c91845129695f27b8303a5f1
Summary: Add strum derivations to bulkops so we can use them in command line parsing later in stack.
Differential Revision: D28069912
fbshipit-source-id: 4d997e20e18f2011b51933ed4322c85bb7468980
Summary:
We were ignoring the return value of runWhileMaterialized, and thus we were
returning to FUSE before fallocate returned.
Reviewed By: fanzeyi
Differential Revision: D28081991
fbshipit-source-id: f398942ddb2432e48e80c148abc8edb7e5ada71d
Summary: Start logging mtime as relatedness key in the walker scrub pack info output
Differential Revision: D28055637
fbshipit-source-id: 4c24c5f2af0414ae7df17ade69bba9ff18861264
Summary:
We used to carry patches for Tokio 0.2 to add support for disabling Tokio coop
(which was necessary to make Mononoke work with it), but this was upstreamed
in Tokio 1.x (as a different implementation), so that's no longer needed. Nobody
else besides Mononoke was using this.
For Hyper we used to carry a patch with a bugfix. This was also fixed in Tokio
1.x-compatible versions of Hyper. There are still users of hyper-02 in fbcode.
However, this is only used for servers and only when accepting websocket
connections, and those users are just using Hyper as a HTTP client.
Reviewed By: farnz
Differential Revision: D28091331
fbshipit-source-id: de13b2452b654be6f3fa829404385e80a85c4420
Summary:
This used to be used by Mononoke, but we're now on Tokio 1.x and on
corresponding versions of Gotham so it's not needed anymore.
Reviewed By: farnz
Differential Revision: D28091091
fbshipit-source-id: a58bcb4ba52f3f5d2eeb77b68ee4055d80fbfce2
Summary:
Connect up the scrub stream types so they will be uniform for scrubs that log pack info and those that do not.
This is in preprepation for the next diff which connects up the pack info logging of path hashes to scrub. CI for this diff verifies its not broken the non-path tracking case.
Differential Revision: D28031868
fbshipit-source-id: 7bf91eb1778f57487f6a2847f215cf7f5cd2dff7
Summary: This moves evolve_path up to WrappedPathLike so that we can use sample route evolution logic for routes that track paths (e.g. corpus sampling) and path hashes (e.g. scrub, where path hashes take less memory than full paths).
Differential Revision: D28031867
fbshipit-source-id: cdabdc466158a8db1c770536747c996dddb27e71
Summary: Name the fields rather than leave it as a tuple struct. This makes it a bit easier to work with in the rest of the stack
Differential Revision: D28062254
fbshipit-source-id: 9e5202b4d6f1f29d44d98b86aa9b6ddb97d821eb
Summary: Makes more sense for this to be a method on NodeType
Differential Revision: D28031869
fbshipit-source-id: 1ddbafa0d7634ac67fd8d5112e6f57759ed91638
Summary: Name the fields rather than leave it as a tuple struct
Differential Revision: D28031866
fbshipit-source-id: 039f004e0b81294aa6d6b13e79cb45ee2b84567c
Summary: This new trait abstracts across WrappedPath and WrapperPathHash. Later in the stack I make path tracking use this to track either full paths (for corpus sampling) or path hashes (for logging from scrub).
Differential Revision: D28031870
fbshipit-source-id: d1c57230f68fffff179929a3cb92c82d92e0588c
Summary:
Like it says in the title. This isn't giving us the same error consistently
causing flaky failures.
Reviewed By: StanislavGlebik
Differential Revision: D28091747
fbshipit-source-id: dfc7a28b443c6577823c71cee7b006ed30fec18e
Summary: This is no longer needed, as all construction is performed by facet factories.
Reviewed By: StanislavGlebik
Differential Revision: D28001390
fbshipit-source-id: 237dd4f7b8b08bec5b85360edc3be7018d9161de
Summary:
Keeping the `Changesets` trait as well as its implementations in the same crate means that users of `Changesets` also transitively depend on everything that is needed to implement it.
Flatten the dependency graph a little by splitting it into two crates: most users of `Changesets` will only depend on the trait definition. Only the factories need depend on the implementations.
Reviewed By: krallin
Differential Revision: D27430612
fbshipit-source-id: 6b45fe4ae6b0fa1b95439be5ab491b1675c4b177
Summary:
The changesets object is only valid to access the changesets of a single repo
(other repos may have different metadata database config), so it is pointless
for all methods to require the caller to provide the correct one. Instead,
make the changesets object remember the repo id.
Reviewed By: krallin
Differential Revision: D27430611
fbshipit-source-id: bf2c398af2e5eb77c1c7c55a89752753020939ab
Summary:
The `get_sql_changesets` method on `Changesets` is an abstraction violation,
and prevents extraction of `SqlChangesets` to a separate crate as it would
introduce a circular dependency.
It is used to allow bulk queries to enumerate changesets by integer unique ID,
so promote this to a full feature of `changesets`, and remove the
`get_sql_changesets` method.
Reviewed By: krallin
Differential Revision: D27426921
fbshipit-source-id: 2839503029b262dd5e6a8be09bb35bb143b4c5ac
Summary:
folly::via is a Future API, and thus it creates one, which requires allocating
it and then attaching it to the Executore. Since the code to dispatch a request
isn't Future based, we don't need to use folly::via, and we can simply add the
lambda to the Executor directly. This removes expensive memory allocations from
the EventBase.
Reviewed By: kmancini
Differential Revision: D27976674
fbshipit-source-id: 8fa9724a94ba69b071ab894cdbbad0d33733c098
Summary:
Neither macOS, nor Linux are sending multi-fragment requests to the NFS server.
Since supporting these means calling into memmove, which can be expensive for
large requests, let's just remove support for them for now. If somehow macOS
and/or Linux start sending these, the XCHECK(isLast) will catch this and we can
fix the code by then.
Reviewed By: kmancini
Differential Revision: D27976671
fbshipit-source-id: 77c758b2bb36517d22d5b637e6f0ebf84cc19e5b
Summary:
The EventBase is single threaded, and for heavily concurrent client workflows,
it could see a lot activity, thus every cycle saved can be used to drive more
client requests. The construction of the IOBuf doesn't need to be done while in
the EventBase, thus let's build it outside.
Reviewed By: kmancini
Differential Revision: D27976670
fbshipit-source-id: c6c015ef26df1dcb3fc0c5f179e474bafbd71fac
Summary:
Passing 64 to preallocate means that the AsyncSocket code will issue reads of
64 bytes, even though the IOBufQueue has significantly more space available. We
can thus pass a bigger size to preallocate to reduce both the cost of
allocation, and the syscall cost. For heavily concurrent client code, this will
allow us to read more than one request per syscall.
The careful reader may have noticed that for very small requests the code may
reallocate more often that it should as it will always reallocate when falling
under 4KB. This is likely to not be an issue in practice.
Reviewed By: kmancini
Differential Revision: D27976672
fbshipit-source-id: 4c7e3aecc4763ab20854f3c466ce0872332f9b77
Summary:
These are various cleanups that should make the code easier to read, there is
no behavior changes.
Reviewed By: kmancini
Differential Revision: D27976673
fbshipit-source-id: 470eb628ca75bf1712a93c6e9aa3a27c3f314d01
Summary:
Running `rg foobar` in a loop and profiling EdenFS shows that we're spending a
significant amount of time collecting the size of non-materialized files. Since
this will never change, we can easily cache it for much faster access.
Reviewed By: chadaustin
Differential Revision: D27924804
fbshipit-source-id: 8b8af63dcb82664db2ecd81b3fcdc006a3a52d72
Summary: Create a proxy that stored RECAS-> Eden Hash, similar to SCS and HG proxy.
Reviewed By: chadaustin
Differential Revision: D27873498
fbshipit-source-id: 0b3e50e3a74b8f0914547178789cb6684b780866
Summary: Create a RE-CAS backing store with all APIs unimplemented, and Linux only.
Reviewed By: chadaustin
Differential Revision: D27771047
fbshipit-source-id: de00c6e290f924872eae7290b1945e6b3f40d610
Summary:
The "known" API is expected to accept a list with ordering guarantees.
Avoid passing `set` to it. This also stabilizes debug outputs in (upcoming)
tests.
Update `setdiscovery` to pass a list with stable order too.
Reviewed By: kulshrax
Differential Revision: D27951627
fbshipit-source-id: a2718e4380f80584fc8afc35d14e524ab428f9bd
Summary: They will be useful in the pull exchange path.
Reviewed By: kulshrax
Differential Revision: D27951625
fbshipit-source-id: 31dfd54cda7a0d0d0b2565e0ecddc2e82eebcc05
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way.
Actual implementation is TBD.
Differential Revision: D27951634
fbshipit-source-id: 66210a833d8fc87452a19d1935e9d208a1d31b14
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way. Actual implementation is TBD.
I dropped the progress callback parameter since the native Rust progress bars
make it unnecessary. I also avoided the blocking API interface, since we would
like pure Rust code to not call blocking APIs which is likely a mistake, and
only use `async_runtime::block_on` in Python bindings - in that case blocking
API is not needed.
Reviewed By: andll
Differential Revision: D27951624
fbshipit-source-id: 3844dd96df265cc6e61d7cf5e79f39c891e8117d
Summary: This makes use of plan calculation logic in merge.py
Reviewed By: quark-zju
Differential Revision: D28045174
fbshipit-source-id: f2bfd1104ba2d010c2b911aa6c643b057e4cb91d
Summary: This factors out native checkout plan calculation (including handling sparse profiles) into separate fn
Reviewed By: quark-zju
Differential Revision: D28045175
fbshipit-source-id: cd034b2c335496e5e2c6994795dd5c40d3388238
Summary:
Add a way to construct the EagerRepo from Python, and use it to serve EdenApi.
Example in `hg dbsh`:
Prepare EagerRepo:
In [2]: r=b.eagerepo.EagerRepo.open('/tmp/r1')
In [3]: r.addsha1blob(b'1')
Out[3]: '5j\x19+y\x13\xb0LTWM\x18\xc2\x8dF\xe69T(\xab'
In [4]: r.getsha1blob(Out[3])
Out[4]: '1'
In [5]: r.addcommit?
Docstring:
Insert a commit. Return the commit hash.
(parents: [node], rawtext: bytes) -> node
Type: builtin_function_or_method
In [6]: c=r.addcommit([], 'commit 1')
In [7]: c
Out[7]: '\x8a\x8a\x0bq\x1e\xa4\x08\xd4\xa5\x14a\xc8\x95Y\xed\x8d$\xba1\x9a'
In [8]: r.flush()
Use it to serve EdenApi:
In [9]: e=r.edenapiclient()
In [10]: e.commitdata?
Docstring:
commitdata(repo: str, nodes: [bytes], progress=None) -> [(node: bytes, data: bytes)], stats
Fetch commit data in raw HG format (sorted([p1, p2]) + text).
Type: builtin_function_or_method
In [11]: e.commitdata('', [c])
Out[11]: (<stream at 0x7f96194bcc70>, <future at 0x7f96194bf810>)
In [12]: list(e.commitdata('', [c])[0])
Out[12]:
[{'hgid': '\x8a\x8a\x0bq\x1e\xa4\x08\xd4\xa5\x14a\xc8\x95Y\xed\x8d$\xba1\x9a',
'revlog_data': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00commit 1'}]
Reviewed By: andll
Differential Revision: D27951623
fbshipit-source-id: c5423d55b6359146e5a416c440d85f817e43396d
Summary:
Add a way to construct the Python EdenApi wrapper from a native Rust EdenApi instance,
bypassing the config, etc.
Reviewed By: kulshrax
Differential Revision: D27951622
fbshipit-source-id: 2e3d16a91e20c6aab6baced3dea60db9834fa103
Summary: Implement the endpoint. Most of the complexity is type conversion.
Reviewed By: andll
Differential Revision: D27929203
fbshipit-source-id: ff865ae727e383cd2b465f8bcd1e29f0c9316ff6
Summary: Implement the endpoint. Most of the complexity is type conversion.
Reviewed By: andll
Differential Revision: D27929201
fbshipit-source-id: b96f65a0173a31f716272c8e0dd47ce8a90759cd
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary: Implement the files API. It's just reading content from the zstore.
Reviewed By: kulshrax
Differential Revision: D27926251
fbshipit-source-id: 54d04caa63e01b6ce5b9c785990c14043f7f22ad
Summary: The will be useful for "push" logic.
Reviewed By: kulshrax
Differential Revision: D27951633
fbshipit-source-id: 38bbdc554f017d5776df0577b82fbb0c78d18a83
Summary:
This will be useful for "push" related logic.
The name "eager" is to make it explicit that the repo is not lazy.
Reviewed By: kulshrax
Differential Revision: D27951618
fbshipit-source-id: 8039059beba68d269c752bc8ed3e72bde0c55790
Summary:
Currently it's hard to test EdenApi related features in hg tests. The Mononoke
test suite can do it but it's too heavyweight. Looking at the API surface of
EdenApi it's actually quite small. So let's add a minimal Rust struct that can
serve as an EdenApi server.
This diff just adds a few minimal features. EdenApi related features and
push/pull support will be added later.
The name "eager" is to make it explicit that the repo is not lazy. I thought
about names like "testrepo" or "serverrepo", but the implementation is
somewhat "sound" to be used as a client, non-test repo. It can potentially
be used as starting point for a real "repo" in pure Rust. So I didn't choose
those names.
(I'm not entirely happy with the name but it's more like a placeholder
that makes it look different from other names for now).
Reviewed By: kulshrax
Differential Revision: D27926255
fbshipit-source-id: ad7a023de5e77605a553509de82ff13ae8112439
Summary:
This allows an external crate C that implements `EdenApi` to depend on a more
lightweight library just providing `EdenApi` without things like `hg_http`.
Then the `edenapi` crate can depend on C too.
Didn't move it to `edenapi_types` because it would add extra dependencies
(http, http_client, auth, etc.)
Reviewed By: kulshrax
Differential Revision: D28006548
fbshipit-source-id: 6e828974fd3f78fec70d4a04ae7be85abc459b36
Summary:
The `Builder` API is the main API used by external users to obtain an `EdenApi`
client. In the future we want to support different kinds of `EdenApi`, like a
local repo serving it, if `paths.default` is set to something like
`myrepotype:path`. Make `Builder` more flexible to support non-HTTP `EdenApi`s,
by returning `EdenApi` trait objects.
The old builder that is coupled with HTTP is renamed to HttpClientBuilder.
Reviewed By: kulshrax
Differential Revision: D28018586
fbshipit-source-id: 1eff7bbb8f0e5521a9bcf5a225ac361ddf7c310f
Summary:
This ensures the User-Agent is always set. It also makes the `header` less
unnecessary.
Reviewed By: DurhamG
Differential Revision: D28018587
fbshipit-source-id: 1125d2122431579f127e81c4713de45135b1f972
Summary:
Make it easier to use.
This makes it easier for other crates to use `edenapi::Result<T>`, which is
a bit shorter than `Result<T, EdenApiError>`. Also re-export `Metadata`
from revisionstore-types so callsite does not need to depend on
revisionstore-types explicitly.
Reviewed By: kulshrax
Differential Revision: D27926250
fbshipit-source-id: c85198b5c151e10a2d4d2567e23e32605a3e7c36
Summary: Ensure these settings are available in mononokepeer.
Reviewed By: mitrandir77
Differential Revision: D28061520
fbshipit-source-id: 68cbe9f427d4a1528a4c9968b3f1f9dcd2541004
Summary:
New endpoint. This endpoint can be used for prefix lookup and the contains
check.
Reviewed By: quark-zju
Differential Revision: D28034533
fbshipit-source-id: d724b85c3816414475b142215e3052d0b555cf59
Summary:
These structures are going to be used to implement the `commit/hash_lookup`
endpoint in EdenApi.
Reviewed By: quark-zju
Differential Revision: D28034532
fbshipit-source-id: 7b00d0d97dd0593dfa43834cda9fc9e9ab9021c5
Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary:
I want to add prefix lookup functionality to EdenApi.
I considered adding some sort of prefix abstraction to Hg or our wire types.
so that I could use the method that was already implemented. I found the
serialization and conversion logic complicated to reason about. It is easier
to transform the prefix query in a range query and add a range query endpoint.
No need to invent new fancy types and deal conversions.
This diff updates the prefix query implementation to rely on the range query.
The range query is functionality that is more general than prefix lookup.
Reviewed By: quark-zju
Differential Revision: D28034531
fbshipit-source-id: 491db2354e3804c4cea6db16fe7d056a962515f7
Summary:
NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing.
There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs).
Here's a detailed list of what is going on:
- I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace.
- Lots of little Tokio changes:
- Stream abstractions moving to `tokio-stream`
- `tokio::time::delay_for` became `tokio::time::sleep`
- `tokio::sync:⌚:Sender::send` became `tokio::sync:⌚:Sender::broadcast`
- `tokio::sync::Semaphore::acquire` returns a `Result` now.
- `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`).
- `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case.
- Runtime builders have changed quite a bit:
- My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`.
- Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out.
- Tokio 1.x now uses Bytes 1.x:
- At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore).
- In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine.
- In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x.
- In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy.
- tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to:
- How we listen in Mononoke Server & LFS
- How we connect in hgcli.
- Note: all this stuff has test coverage.
- The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this.
- Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it.
- I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x.
Reviewed By: mitrandir77
Differential Revision: D26669620
fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
Summary:
MyRouter is no longer used by Mononoke services, it is deprecated and will stop working when we upgrade the tokio.
This diff removes MyRouter support from Mononoke and simplifies the Mysql connection type struct.
Before we had `MysqlOptions` and `MysqlConnectionType` enum to represent what kind of a client we want to use. Now we use only MySQL FFI so I removed `MysqlConnectionType` completely and put everything into the options struct.
As setting up the connections (aka conn pool) is not an async operation, some of the methods don't need to be async anymore. Because this diff is already enormous, I'm refactoring this in the next one.
Reviewed By: StanislavGlebik
Differential Revision: D28007850
fbshipit-source-id: 32c3740f4bb132f06e1e256b0530ace755446cdd
Summary:
This command executes native checkout dry run code for updating between specificed revisions
Command tests network and storage perf, without actually affecting working copy
Examples of usages:
1) User reports that when updating between revisions multiple times, they see unexpected fetches, even though they expected that all data should already be in cache. Running this command shows that update between those revs fetches more data then a cache size, so repating fetches are not a surprise
2) People working on storage/network layer optimizations can utilize this command to test performance without messing with their working copy
This command understands nativecheckout.usescmstore config, same way it is used in actual native checkout
Differential Revision: D28040641
fbshipit-source-id: e5454f3110ade11f3227d6adc804a22176f530b9
Summary: This fn can be use to format time using human readable units.
Differential Revision: D28042138
fbshipit-source-id: 8b1eb6fa892754ee1008b529477fd555bd41c692
Summary: Those methods only access store/network to fetch content but does not write to disk
Differential Revision: D28040640
fbshipit-source-id: e45dd08e12d128d54b3446e1137465981cde8f13
Summary:
HgManifestFileNode is one of the last remaining types we don't walk ( other known one is the git derived data).
Its added as a separate NodeType from HgFileNode as HgManifestFileNode is used much less and users may want to see only the HgFileNodes. Server side the manifest file node is only used to build the bundles returned to the client.
Differential Revision: D28010248
fbshipit-source-id: ce4c773b0f1996df308f1b271890f29947c2c304
Summary:
This uses code that depends on Tokio 0.x so let's get this removed. I think
I could have updated this to just use manifold-thrift, but while I'm at it,
might as well update to the new shiny client.
Reviewed By: farnz
Differential Revision: D28025165
fbshipit-source-id: 4b79c8a4bd0b8789e6827d2135d36245db4447d5
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
We collect some cache statistics already, lets expose them through thrift with
the rest of the stats.
Reviewed By: chadaustin
Differential Revision: D27052265
fbshipit-source-id: d7fdf70260599b8df43824e2442471e332c1b0cf
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
Here we actually start to use the cache!
Reviewed By: chadaustin
Differential Revision: D27050310
fbshipit-source-id: e35db193fea0af7f387b6f44c49b5bcc2a902858
Summary:
This introduces some basic unit tests to ensure correctness of the cache.
We are adding tests to cover the simple methods of the object cache since we
are using that code path here. And adding a few sanity check tests to make sure
the cache works with trees.
Reviewed By: chadaustin
Differential Revision: D27050296
fbshipit-source-id: b5f0577c1662483f732bb962c5b40bca8e1dcb40
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
This introduces the class for the in memory cache and is based on the existing
BlobCache. note that we keep the minimum number of entries functionality from
the blob cache. This is unlikely to be needed as trees are much less likely
than blobs to exceed a reasonable cache size limit, but kept since we already
have it.
Reviewed By: chadaustin
Differential Revision: D27050285
fbshipit-source-id: 9dd46419761d32387b6f55ff508b60105edae3af
Summary:
On all the code paths that matter we always acquire the write lock. Since we are
basically just using a simple lock, distributed mutex is a more efficient implementation
that does fancy tricks with cachelines. This improves performance from testing
globs which cause many concurrent reads from the cache.
Reviewed By: chadaustin
Differential Revision: D27810990
fbshipit-source-id: d22470f3f39e2cd3895f5ea772955b62030d154a
Summary:
Now that Object Cache actually does most the work, I am moving the
BlobCache tests to be ObjectCache tests. I am leaving a few blob cache
tests to sanity check that the cache works with blobs.
Reviewed By: chadaustin
Differential Revision: D27776113
fbshipit-source-id: ef58279d93035588beb162ee19173a42e3ca4e5b
Summary:
We would like to use a limited size LRU cache fore trees as well as blobs,
so I am templatizing this to allow us to use this cache for trees.
Trees will not need to use Interest handles, but in the future we could use
this cache for blob metadata, which might want to use interest handles.
Additionally if we at somepoint change the inode tree scheme that would remove
the tree content from the inodes itself, interest handle might be useful for
trees. We could also use this cache proxy hashes which may or may not use
interest handles. Since some caches may want interest handles and others will
not I am creating get/insert functions that work with and without interest
handles.
Reviewed By: chadaustin
Differential Revision: D27797025
fbshipit-source-id: 6db3e6ade56a9f65f851c01eeea5de734371d8f0
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: xavierd
Differential Revision: D27986185
fbshipit-source-id: d90aaf27f25f2ecfcbbbe7886e0c0d784f607a87
Summary:
This and following diff will refactor CheckoutPlan creation.
Right now we create CheckoutPlan from manifest diff and then manipulate it with methods like `with_sparse_profile` to adjust plan for different cases.
Those 'adjustment' do not work great with the structure of CheckoutPlan, for example `with_sparse_profile` has to create temporary HashSet just to index files in CheckoutPlan
We are going to add more adjustments in the future (for example, checkout --clean), and will run into same issues with current structure of CheckoutPlan
To avoid those issues, we are going to refactor this code, so that instead of Diff -> CheckoutPlan -> adjustments, we are going to have Diff -> ActionMap -> adjustments -> CheckoutPlan
The structure of CheckoutPlan is still good for it's direct purpose (execution), but all the 'changes' to the plan will be done in ActionMap instead.
Reviewed By: DurhamG
Differential Revision: D27980390
fbshipit-source-id: 403f371fd2fe7760984925a38429e1bfb88d8e3f
Summary: For now this does not work with --clean flag(fallback to regular checkout in that case)
Reviewed By: quark-zju
Differential Revision: D27953967
fbshipit-source-id: 71c097cf1e395ff2cba2f4ee528145d3b2c83c23
Summary: This crate does not calculate status, but allows to convert python status into rust status, that can later be used by python code
Reviewed By: quark-zju
Differential Revision: D27953962
fbshipit-source-id: ab91d9d035140e43d8b17988b24bd030af77c96d
Summary: When checking out on dirty copy without --clean this function can be used to check if checkout operation conflicts with currently modified files
Reviewed By: quark-zju
Differential Revision: D27953965
fbshipit-source-id: 4096506e4cbf8b102e0afa1a929c066dfa474825
Summary:
This crate introduces consumer API for status in rust
Currently the implementation will just take status from Python and convert it into this struct
But in the future we can get pure Rust implementation to get status
Reviewed By: quark-zju
Differential Revision: D27953963
fbshipit-source-id: 29c876400c82056eaf81fffa4adc814473853c1e
Summary: This method can be used to 'normalize' path for case insentive use cases
Reviewed By: quark-zju
Differential Revision: D27953964
fbshipit-source-id: 421832af22af9a3b56eec0d045b9f983592ed192
Summary:
Fix missing stats for the bookmarks endpoint because we have the wrong name in
code.
Reviewed By: quark-zju
Differential Revision: D28008091
fbshipit-source-id: 128fe00e00a06ebe9b65fb11512cd03a042d55fe
Summary:
This doesn't link the errors into Scuba, which makes it not very useful for
debugging, since we're not routinely looking at stderr on our tasks, and makes
it impossible to e.g. look anywhere and find a count of failed requests.
Instead, update this to use `capture_first_err`, which will report both the
first error and the total error count to Scuba.
Reviewed By: sfilipco
Differential Revision: D27998037
fbshipit-source-id: b941d44a2ac21bbf640b9bc977de749207f12d9a
Summary:
In EdenAPI, most endpoints don't raise errors when they fail, and instead just take items out of the response stream (though there are some exceptions: D24315399 (0ccae1cef9).
Right now, this just gets logged to stderr, which isn't great. This diff introduces a CaptureFirstError wrapper we can use to capture those errors and expose them in post-response callbacks (i.e. Scuba).
Reviewed By: sfilipco
Differential Revision: D27998036
fbshipit-source-id: 960d0e09a82ca79adfafe22e2eeef2e0294d27dc
Summary:
`lld` is faster than `ld.gold`.
To compare, `make local3`, then add a blank line in `hgmain`, then run `make
local3` again.
Using `lld`:
building 'hgmain' binary
Compiling hgmain v0.1.0 (/home/quark/fbsource-lazy/fbcode/eden/scm/exec/hgmain)
Finished release [optimized + debuginfo] target(s) in 3.73s
building 'mkscratch' binary
Finished release [optimized] target(s) in 0.37s
building 'scm_daemon' binary
Finished release [optimized] target(s) in 0.42s
Using `ld.gold`:
building 'hgmain' binary
Compiling hgmain v0.1.0 (/home/quark/fbsource-lazy/fbcode/eden/scm/exec/hgmain)
Finished release [optimized + debuginfo] target(s) in 10.15s
building 'mkscratch' binary
Finished release [optimized] target(s) in 0.39s
building 'scm_daemon' binary
Finished release [optimized] target(s) in 0.46s
Reviewed By: ikostia
Differential Revision: D28006551
fbshipit-source-id: 55b19be93d8152634d79ed92c9cd53237f91b820
Summary:
This raw XDB connections API is not used in Mononoke although there are projects depending on it.
The API creates connection objects based on shed/sql crate.
This diff moves `SqlConnections` into `shed/sql:sql_common`, closer to the `Connection` definition, and moves `create_raw_xdb_connections` API into the `shed/sql:sql_facebook::raw`.
Reviewed By: krallin
Differential Revision: D28003638
fbshipit-source-id: ea4a29b6e239a89c97237877e2dfde4c7c7ff89b
Summary:
It turns out during the initial clone, we're not using selectivepull for some
tiers that do not have the non-repo selectivepull config.
We've been using selectivepull for devservers and corp (and it's effective
during clone) for a long time. Let's just enable it by default so even if
dynamicconfig does not set it properly, we can still use selectivepull clone
to avoid pulling 10k+ remote bookmarks (which triggers auto bookmark cleanups
as logged in hgfeatures).
There are too many incompatible tests so I'm not migrating them for now.
Reviewed By: DurhamG
Differential Revision: D28006488
fbshipit-source-id: f0dc000156abde530fd8881bd26b4642a36167be
Summary:
As I split the pushrebase crate in D27968029 (93b8cf116b) it makes sense to stop re-exporting hook definitions from it.
This change will also make dependencies more accurate.
Reviewed By: krallin
Differential Revision: D28028045
fbshipit-source-id: 622ef5d7db737e19153109f4d2dcefe54fba2bb4
Summary:
Finally! This is basically the end goal of this stack (though we still need to
do the same thing with the "ForwardErr" combinator used by EdenAPI next).
With this, we can now capture errors that occur while sending the response
stream, instead of just errors that occur while producing the response (which
is usually a little piece of the work we have to do).
Reviewed By: mitrandir77
Differential Revision: D27967991
fbshipit-source-id: a5483c58f0550a19e711e712cf860d9328a0eb9e
Summary:
`ContentMeta` sounds a lot like a struct containing content metadata, so let's
rename this to something less ambiguous (`ContentMetaProvider`).
The reaosn I'm doing this now is because I'd like to introduce a similar
trait for errors that occur on our response streams and there I'll need
both a struct and a trait.
Reviewed By: mitrandir77
Differential Revision: D27998039
fbshipit-source-id: f0372c62d13167ef4bd07cb9e9e9fb75ea105b9a
Summary:
Like it says in the title. The goal here is to make it not matter where the
error came from. In this diff, we capture the same errors as before, but we
do it via PostResponseInfo instead of via ad-hoc things in our `State`.
Reviewed By: mitrandir77
Differential Revision: D27967994
fbshipit-source-id: dbbb1a8f5ea1a439089c41b4a08cd6088476ae33
Summary: Like it says in the title.
Reviewed By: mitrandir77
Differential Revision: D27967992
fbshipit-source-id: 0deb4d90538a6889bee6b41de4c5d1533b29519b
Summary:
Very small refactor. I want this stuff to all be in the same module instead
of spread across `response` and `error`.
Reviewed By: mitrandir77
Differential Revision: D27967993
fbshipit-source-id: aca22f952d756d298e5e342f0c4f8ebd31f108bf
Summary:
This is a bit of an abstract change, but the goal of this change is to make
post-response handlers oblivious to the distinction between sending a response
(or failing to send one) and returning a response that actually contains a
(fallible) stream.
The underlying / ultimate goal here is to move our error reporting out of
ad-hoc router wrappers where we call `set_error_message` on some context
entity, and to instead move it into post-response callbacks.
The upshot of that is that if we fail to send a response even though we sent a
200 from the handler, we'll be able to log it! Indeed, remember that when
sending a streaming response, we have to send a 200 to start streaming but we
might hit an error producing later parts of the response!
Reviewed By: mitrandir77
Differential Revision: D27966422
fbshipit-source-id: ab49639bfcc4af23ddc2e84181278f105ebb28b9
Summary:
This stuff runs after we sent the response, so PostResponse is a more
appropriate name than PostRequest.
Reviewed By: ikostia
Differential Revision: D27966420
fbshipit-source-id: 1f7b7a55490f731eb512793024bcfafb0ea4ef79
Summary:
Those two have historically used different (but largely copy pasted) code to
produce their responses. Let's unify them by
While in there, let's also modernize the formatting a little bit by letting
anyhow do the formatting for us (we used to use `failure` when this code was
written, and it didn't do it).
There's a bit of ugliness here in the sense that out formatting is injecting
the error into the state so it can be logger later. This is equivalent to what
we had, but it's kinda clonwy. That said, I'm working on refactoring our error
handling in this stack, so the plan is to clean this up (i.e. it won't stay
very long).
Finally, note that this diff removes the `ResponseCreationFailure` error kind
in LFS. This is replaced by a `.context()` in `gotham_ext`. That said, we never
really use this stuff: creations are fallible in Hyper but you only run into
an error if you e.g. forget to add a status code, so we don't expect them to
actually occur outside of development.
Reviewed By: mitrandir77
Differential Revision: D27966421
fbshipit-source-id: 097f3b69f25fe39f8fbef925a272e88199896b39
Summary:
Like it says in the title, I'd like to use names that reflect that this isn't
just *any* content stream: it's specifically for responses.
Reviewed By: ahornby
Differential Revision: D27964045
fbshipit-source-id: 50530cf85ba7840a2baa14151351d0b288d9ae70
Summary:
We have a set of things that are meant to be used together that are spread
across 3-4 different modules. Let's move them together. This also allows us to
make some things (e.g. the `ContentMeta` trait) private as they're no longer
needed.
Note: this diff only move stuff around & renames things a bit. I'll also split
some of those modules in the next diff.
Reviewed By: HarveyHunt
Differential Revision: D27964047
fbshipit-source-id: 02528d84adfd70ec346c32163cb185d89266a53e
Summary:
We have a module called "content" that contains two completely unrelated
things: some enums for content encodings (+ associated parsing), and our output
streams.
I'd like to add more of said output streams, so let's clean this up.
Reviewed By: HarveyHunt
Differential Revision: D27964046
fbshipit-source-id: b42e56aa3fadaf9b93a44216977da19d950a16b9
Summary:
We used to have to do this because of overly strict trait bounds in Hyper, but
we no longer do, so let's get rid of it. Note that we have one Tokio task per
request, and polling the response stream is basically the only thing that task
does, so this should make little difference as far as task scheduling is
concerned besides avoiding unnecessary context switches.
Reviewed By: ahornby
Differential Revision: D27963458
fbshipit-source-id: 24e762eb173156dab909fefe11dcf2d58272048a
Summary: We don't use this anymore. Might as well remove it.
Reviewed By: ahornby
Differential Revision: D27963411
fbshipit-source-id: a6ac337936e8b2bd788dd79a835eef11b19dde70
Summary: It has been fixed and we now set auth config with higher priority anyway.
Reviewed By: johansglock
Differential Revision: D28026081
fbshipit-source-id: 7086b48139bb05ffadd782898a1758ae06236aca
Summary:
This change makes it so that our binaries do not instantiate a real configo
client in integration test setup.
Reviewed By: ahornby
Differential Revision: D28026790
fbshipit-source-id: 0fb9ce66a1324e845f4b8a80d4479263ec6e4ee1
Summary:
First, some background on the existing WrappedPath type: In Mononoke the MPath type is such that None==Root and Some(MPath)==NonRoot. This means that where a path may be present one needs to use double-Option with Option<Option<MPath>>, so that Root is Some(None).
To reduce the need for double Option, and subsequently to allow for newtype features like memoization, the walker has WrappedPath, so we can use Option<WrappedPath> instead.
This change introduces a similar type WrappedPathHash for MPathHash, which means that the sample_fingerprint for WrappedPath can be now be non-optional as even root paths/manifests can now have a sample_fingerprint.
Reviewed By: mitrandir77
Differential Revision: D27995143
fbshipit-source-id: b674abd4ec94749f4f5797c697ae7381e1a08d02
Summary:
This adds the first part of new logging from the walker that can be used to gather details on what keys might make sense to pack together.
Unlike the corpus command that dumps file content by path (which was useful for analysis on compression approaches), this adds logging to the scrub command that includes the path hash rather than the full path. This should keep memory usage down during the run, hopefully mean we log from existing scrub jobs and and mean the logs are more compact.
Reviewed By: mitrandir77
Differential Revision: D27974448
fbshipit-source-id: 47b55112b47e9b022f16fbb473cf233a7d46bcf3
Summary:
Knowing the prepushrebase changeset id is required for retroactive_review.
retroactive_review checks landed commits, but verify_integrity hook runs on a commit before landing. This way the landed commit has no straightforward connection with the original one and retroactive_review can't acknowledge if verify_integrity have seen it.
Reviewed By: krallin
Differential Revision: D27911317
fbshipit-source-id: f7bb0cfbd54fa6ad2ed27fb9d4d67b9f087879f1
Summary:
Split pushrebase crate into pushrebase hook definition and pushrebase implementation.
Before this change it was impossible to store an attribute in BlobRepo that would depend on PushrebaseHook as it would create a circular dependency `pushrebase -> blobrepo -> pushrebase`.
Reviewed By: krallin
Differential Revision: D27968029
fbshipit-source-id: 030ef1b02216546cd3658de1f417bc52b8dd9843
Summary:
It's rare (only seem to be used by chistedit) but if revrange got an int, the
expected behavior is to treat it as a revision number.
Reviewed By: kulshrax
Differential Revision: D27983989
fbshipit-source-id: f9f8d9cb39af4ec1de7ed8ca69f7f1879b4a4614
Summary:
Adds a message for users to use 'hg checkout --continue' if there's a
.hg/updatestate file (indicating an aborted checkout) and if they're on the null
rev (indicating they likely just cloned).
Also adds support for 'hg checkout --continue' to work with non-merge commits.
Note, it really only currently works when checking out from null, since
otherwise there will be a lot of modified files in the way. Once native checkout
is more mature, we can teach it to ignore modified files that match the desired
checkout destination.
Reviewed By: quark-zju
Differential Revision: D26967976
fbshipit-source-id: 7397c5fe82027e22bf1b4db0f14bb180239fae25
Summary:
The check unknown logic would block checkout for any unknown files that
were to be overwritten. We want to allow checkouts where the unknown file has
the same content as the desired checkout value. Ideally we'd check it against
the SHA1 hash of the file we're about to checkout, but since content hashes
aren't available yet we can limit our check to resumed checkouts for now.
Reviewed By: andll
Differential Revision: D27804719
fbshipit-source-id: e129ca694080051420e2cb685c7eeb5f1adee005
Summary:
Every function on CheckoutPlan required the VFS already, and the
CheckoutProgress is storing the VFS and living on the CheckoutPlan, so it makes
sense to just store the VFS on the CheckoutPlan.
Reviewed By: andll
Differential Revision: D27825088
fbshipit-source-id: 3d063fdfd1a50983b60d00a3992a893e71732f94
Summary:
Now that CheckoutPlan can look for untracked files, it breaks the
ability to continue a checkout since those untracked files are considered dirty.
In a later diff we'll use the CheckoutProgress to inspect the dirty files and
determine which are actually dirty and which can be overwritten. To do so
though, we need access to the CheckoutProgress earlier. So let's just store it
on the CheckoutPlan.
This is a little awkward because we're passing the root VFS to the constructor
so CheckoutProgress can be instantiated, but then also passing it to every
CheckoutPlan function as well. We should probably just store the vfs on the
CheckoutPlan. If others agree, I can make a diff to do that.
Reviewed By: andll
Differential Revision: D27804720
fbshipit-source-id: e819c27fa8580c82a8cf8f0baf22ac1ea707ee54
Summary:
Python on Windows has a couple bugs around passing stdin/out/err to
subprocesses. In Python 2 we patched our Python to fix one of the bugs. With
Python 3 we're using the vanilla Python distribution, so we'd rather not patch
it.
Instead, let's fix it by passing stdin/out/err explicitly.
See D15764537 (b8c747b6d5) for the details of the bug.
Reviewed By: kulshrax
Differential Revision: D28010523
fbshipit-source-id: d88f789a8100b04da996271de7dfe566c0f715df
Summary: Allows us to background a prefetch (similar to how prefetch-profile fetches are backgrounded). A thing to note here is that we do not deduplicate fetches for prefetches, but if there is enough busy work between bulk filesystem accesses and the prefetch finishing, this should not be an issue.
Reviewed By: chadaustin
Differential Revision: D27028428
fbshipit-source-id: 5c528fff76719f42151542eaa3499271f7ab6fa3
Summary:
This diff wires up actual `scs_server` methods `megarepo_add_sync_target_config`, `megarepo_add_sync_target`, `megarepo_add_sync_target_poll` to the underlying logic in the `CfgrMononokeMegarepoConfigs`.
One of these is a synchronous method (`megarepo_add_sync_target_config`), so it is implemented properly. This method only allows adding new configs to an existing target.
The other two are a pair of async methods (create reqest + poll request) for target creation with an initial config. On the one hand, we don't yet have any infrastructure for async methods, so properly implementing this pair is not possible. What's more, target creation is a two-part operation: save a config + create an initial repo commit. Second part is not yet implemented at all (and is what requires async implementation, as it is going to be expensive). On the other hand, I would like to expose the concept of target creation for the client to test, that's why I add `FAKE_ADD_TARGET_TOKEN` to mask a so-far synchronous impl of this method as asynchronous.
Once I implement async methods, I will come back and work on a proper `megarepo_add_sync_target` impl (this is the first method to be implemented).
Important: any use of these methods now should be considered experimental, and we'll have to delete all of these configs later (because all of the targets won't have any corresponding bookmarks in the real repos, which makes them invalid).
Reviewed By: StanislavGlebik
Differential Revision: D27885979
fbshipit-source-id: 9e2a914af1a7db2ec00ffa11a832ddd71fd19d0f
Summary:
This will help people to introduce new blobstore objects in their code (for
instance I intend to use it in the following diff).
The `private` module exists to allow the use of the exported macro without the
need to write a bunch of `use` statements, and without pollution the re-export
namespace. The idea is that everything needed by the exported macro exists in
the `private` module of the crate, and this module is public.
So long as some other crate imports the macro, it expands to
`$crate::private::NEEDED_THING` in the right places and no further `use`
statements of dependencies are needed. At the same time, the name `private`
should discourage people from using whatever is in this module directly. The
idea is taken from `anyhow`.
Reviewed By: StanislavGlebik
Differential Revision: D27997228
fbshipit-source-id: fd2c421d0daf0fe88e2b9001bb4088fe7b4d59b7
Summary:
Collecting into SortedVecMap an unsorted iterator is inefficient, because of
how [try_collect()](https://docs.rs/futures-util/0.3.0/src/futures_util/stream/try_stream/try_collect.rs.html#57) works. It calls `extend()` method every time a new element was fetched from the
stream. So if stream was unsorted, then we ran into a quadratic behaviour with
lots of reallocations.
Let's instead collect into BTreeMap, and then convert it into SortedVecMap.
Reviewed By: markbt
Differential Revision: D27997469
fbshipit-source-id: 58f837e6cc946ccc8809cce3d7a5a2e6ca24df40
Summary:
I'd like to have a quick way of documenting how this is supposed to be used,
so let's add it.
Reviewed By: HarveyHunt
Differential Revision: D27996500
fbshipit-source-id: 0d138ac3335a3ecb7f0e15aebbdc89d01941cbed
Summary: In python 3 these strings are already unicode, so let's just .upper() them. Otherwise it crashes with 'no decode() on str'. This only impacts eden checkouts, since non-eden uses treestate which doesn't use this codepath.
Reviewed By: quark-zju
Differential Revision: D27978369
fbshipit-source-id: a298c1b455fdb8aa09db0ac667bd97b8e419bbe8
Summary:
During pull, commitcloud may try to auto join a cloud workspace. If
there are no certs, the join will fail and will cause the overall pull to exit
non-zero. Let's just print a warning instead and allow the pull to succeed.
Reviewed By: sfilipco
Differential Revision: D27928397
fbshipit-source-id: 432ee589438bb5af9f47f7aaa735bbbb5a17ad6b
Summary:
These methods will be used in my later Windows fsck diff as it will need to scan disk state to find changes.
It is a bit unfortunate that we'll need to stick with boost for now. However this should be a fairly easy migration to `std::filesystem` once that is available.
Reviewed By: kmancini
Differential Revision: D27872828
fbshipit-source-id: f6b27a171026aeaaea3db9f17b8f43cfa25004e4
Summary:
Currently we have limited test coverage of the NFS code. Let's start running
our integration tests on NFS mounts. We already duplicate tests to run them on
both Git and Hg repos using a python decorator. We can update this decorator to
run a copy of tests on an nfs mount.
This covers most of the tests, but a few tests do not use this decorator. See next
change.
Note some tests are currently broken, so I am using the same skip list functionality
we use for windows so we use a uniform framework.
Reviewed By: xavierd
Differential Revision: D27874662
fbshipit-source-id: c7d425830b691e395b5228d0e0f797f67987b4ec
Summary:
Copy from Watchman.
This allows us to show stack trace when EdenFS terminates on Windows.
Reviewed By: chadaustin
Differential Revision: D27896966
fbshipit-source-id: f3238a37a1176f879d5e6bc051ec97031c9a7096
Summary:
Add a way to extend the graph with concrete commit hashes, without specifying
exact commit messages.
Reviewed By: sfilipco
Differential Revision: D27897894
fbshipit-source-id: fccd64b2fef1386d79cddd841208da6a938a5217
Summary:
scrub blobstore logging was missing the common server logging fields that LogBlob and MultiplexedBlobstore add.
Also moved the LogBlob scuba construction closer to use site for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D27966453
fbshipit-source-id: 77fe70606602753301a2503691a490c0b11c755a
Summary:
Currently when we call with_mutated_scuba() we create a new LoggingContainer.
That means that all the data from the previous LoggingContainer like PerfCounters
is lost. I suspect this is the reason we don't log any BlobGets/Puts for
repo_create_commit methods (see
[scuba](https://fburl.com/scuba/mononoke_scs_server/fautos3s)) - we call
[with_mutated_scuba method](https://fburl.com/code/srd1c4xu) right before
calling repo_create_commit(), and I suspect this loses the counters.
Let's instead copy all the Logging fields when calling `with_mutated_scuba`.
Reviewed By: krallin
Differential Revision: D27964719
fbshipit-source-id: 881c11bb5fb1927dbf55d0d625ea8bfbf11be131
Summary:
In test-cross-repo-commit-validator.t and test-cross-repo-commit-sync., we
modify bookmarks outside of Mononoke, so we need to flush them before pulling.
In test-megarepo-invisible-merge.t, things are actually a little more subtle
and I wonder if there might be another issue laying around there. If we don't
flush bookmarks, then we attempt to upload one more hg commit, and that
blows up: P410235472. However, if we flush bookmarks, then we don't attempt to
upload, and all is fine. Here, flushing is just a workaround, but for no
that'll do. There was also another bug here, where we change configs but
don't force them to take effect.
Reviewed By: StanislavGlebik
Differential Revision: D27964959
fbshipit-source-id: 9c4304b38513177e402ee64f309e019e227ed2a7
Summary:
When we're packing, we pay an overhead price for keeping the key in the pack. As we're only bothered about reducing size, let's limit that price to when the savings from packing are worth it.
There are two cases where it's not worth it:
1. When the compressed pack is larger than the sum of single compressed files sizes.
2. When compressing a single file on its own.
Reviewed By: ahornby
Differential Revision: D27913258
fbshipit-source-id: 36cdc2a2b30aa508281ac3bbd70da41322533edb
Summary:
Like it says in the title. This test wants to see consistent bookmarks so it
should flush them.
Note that this used to just opt out of bookmarks caching entirely, but I'd like
us to try and avoid having so many snowflakes in our tests (because it makes
their maintenance harder), so instead of changing the environment, let's change
the test to do what it needs to do.
Reviewed By: mzr, HarveyHunt
Differential Revision: D27964099
fbshipit-source-id: 72e00bad07dec15f18faaf4aa2e32e78cb333ab0
Summary: Was getting orphan autocargo lints on these so add a config for them.
Reviewed By: krallin
Differential Revision: D27947231
fbshipit-source-id: 925fb78889d8f80f51145536a157fa0e63cc68d7
Summary:
Current implementation had a bug(demonstrated in test case) in handling unknown files on case insensitive fs.
When file is replaced with another file, whose name only differs in case, we get two distinct update operations - rm file, and create file.
Create operation checks against unknown files, and see that file "exists". In this case operation is aborted.
However, we should proceed in this case, and this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27926953
fbshipit-source-id: 48c8824322d6e5dd9ae57fee1f849b57dc11a4df
Summary: Will be useful on case insensitive fs
Reviewed By: quark-zju
Differential Revision: D27946982
fbshipit-source-id: e7a2fd0ee503c4a580531e6f52225fe2316e5b76
Summary: This diff adds flag to VFS to detect whether FS is case sensitive. The logic in this code losely follows similar logic in Python
Reviewed By: quark-zju
Differential Revision: D27926952
fbshipit-source-id: 36fdf4187ae513b25346f704050c64f9a1a4ec74
Summary: This way the fallback server know which traffic is coming from mononoke
Reviewed By: krallin
Differential Revision: D27946019
fbshipit-source-id: 8c13ae641ba340ba55322871ca30fb6accb3f007
Summary:
Update the zstd crates.
This also patches async-compression crate to point at my fork until upstream PR https://github.com/Nemo157/async-compression/pull/117 to update to zstd 1.4.9 can land.
Reviewed By: jsgf, dtolnay
Differential Revision: D27942174
fbshipit-source-id: 26e604d71417e6910a02ec27142c3a16ea516c2b
Summary:
When EdenFS is killed, either due to `eden stop` timing out, or when simply
rebooting the host, the edenfs.log becomes filled with fsck errors, which also
slows down the fsck process.
Since we already print the number of errors per mount, limiting ourself to the
first 50 errors is probably good enough.
Reviewed By: fanzeyi
Differential Revision: D27943618
fbshipit-source-id: 2b3e6e3ae4df648d4b1dccf73c71f8dbbded3892
Summary:
We get pretty frequent query errors from MySQL on this, but it's hard to debug
without knowing what is being queried.
Reviewed By: StanislavGlebik
Differential Revision: D27941603
fbshipit-source-id: 62e0f0fe9c3af36ed829c401e957ecf7683a4000
