Summary:
Make it possible to use async functions in MetaSet functions.
It will be used when DagAlgorithm becomes async.
Reviewed By: sfilipco
Differential Revision: D25345229
fbshipit-source-id: 0469d572b56df21fbdbdfae4178377e572adbcda
Summary: This will make it easier if the `Set` passed in requires async.
Reviewed By: sfilipco
Differential Revision: D25345230
fbshipit-source-id: a327d4e5d425b7eb5296b2fbe25c446492aa9ea7
Summary: This makes it easier to make DagAlgorithm async.
Reviewed By: sfilipco
Differential Revision: D25345234
fbshipit-source-id: 5ca4bac38f5aac4c6611146a87f423a244f1f5a2
Summary: This will make it easier to use `.await` if part of `dag` becomes async.
Reviewed By: sfilipco
Differential Revision: D25345237
fbshipit-source-id: 7f07cdaa9c2e0468667638066611fabe3a3f7f28
Summary: `impl Trait` does not work with `async_trait`.
Reviewed By: sfilipco
Differential Revision: D25345238
fbshipit-source-id: e7890dbaeb162d44e072ea4428d045004608719b
Summary: This makes it easier to migrate to async.
Reviewed By: sfilipco
Differential Revision: D25345228
fbshipit-source-id: e819f0de5f805377a6977325216ef11b14d68c1d
Summary:
Marking IdConvert Sync makes it possible to be used as a trait object with async-trait.
See https://docs.rs/async-trait/0.1.41/async_trait/#dyn-traits
`dag` uses a lot `dyn DagAlgorithm`. In the future when async is used more, the
trait object will be required to be Send or Sync. Just require it on the trait
to make our life easier.
Marking `IdDagStore` as Send + Sync makes async migration easier.
Reviewed By: sfilipco
Differential Revision: D25345231
fbshipit-source-id: 45b96057907cbe2a1d38fd424e7d4c963dd1b245
Summary: Use async function for the PrefixLookup trait.
Reviewed By: sfilipco
Differential Revision: D24840820
fbshipit-source-id: d22cac9f11b06e3127fa956e3f116cf232214125
Summary: This makes the trait objects slightly easier to use.
Reviewed By: sfilipco
Differential Revision: D24840821
fbshipit-source-id: 22fcdf13b62420302b562c309874e08360d02372
Summary: This makes `dyn IdConvert` include `PrefixLookup`.
Reviewed By: sfilipco
Differential Revision: D24840819
fbshipit-source-id: 8d4e25c534f6e4397ec6f643eb3aa116bff12a2c
Summary:
In the future, when async APIs are used, Python bindings will have lifetime
issues. Make it possible to clone the IdMap so the Python bindings can be made
to work.
Reviewed By: sfilipco
Differential Revision: D24840822
fbshipit-source-id: 6aa4e369c877c428ed39d2cbea79e6943836afa8
Summary: This makes NameSet more friendly for async use-cases, interface-wise.
Reviewed By: sfilipco
Differential Revision: D24806695
fbshipit-source-id: 6e640ba2666872a9128d6460e8b53d6a0e595e56
Summary:
Change the main API of NameSet to async. Use the `nonblocking` crate to bridge
the sync and async world for compatibility. Future changes will migrate
Iterator to async Stream.
Reviewed By: sfilipco
Differential Revision: D24806696
fbshipit-source-id: f72571407a5747a4eabe096dada288656c9d426e
Summary:
This will make it easier to incrementally migrate APIs to async, and make sync
programs able to use sync interface if they'd like to. For example, the dag
crate is likely going to be async-ize. Its async APIs are only useful for
places where the dag is lazy. It's still possible to have a non-lazy dag
that a sync interface is suitable.
Reviewed By: ikostia
Differential Revision: D24777747
fbshipit-source-id: 6d56149fbd1f9b29f5fc62387cbf194800755d12
Summary:
We already have a config to write shared history to an indexedlog. Let's
add a similar config for local history.
Reviewed By: quark-zju
Differential Revision: D23910457
fbshipit-source-id: eca21db0350895f573a60e4e1a6b05514e70f1c7
Summary:
In a future diff we'll use the indexedlog stores for local history. We
want those to exist forever, so let's move IndexedLogHgIdHistoryStore to use a
Store under the hood, and add an enum for distinguishing between the two types
at creation time.
Reviewed By: quark-zju
Differential Revision: D25429675
fbshipit-source-id: 5f2dc494e1175d4c1dc74992d3311d2e55d784ca
Summary: Previously, the EdenAPI client (and Mercurial's `http-client`) required both a client certificate and private key to be specified individually in order to use TLS mutual authentication. However, libcurl supports having both the certificate and key concatenated together into a single PEM file. This diff makes it possible to simply specify the combined file as the `cert`, leaving the `key` unset.
Reviewed By: quark-zju
Differential Revision: D25323786
fbshipit-source-id: 1800be3ef82ec4dfa89d725f5860190172994c89
Summary: Treat empty paths as `None`, which allows certificate and key paths to be unset via a `--config` flag (e.g. `hg debughttp --config auth.edenapi.key=`). This would normally require adding an `%unset` line to the appropriate hgrc, which adds friction to ad-hoc command line usage.
Reviewed By: quark-zju
Differential Revision: D25416531
fbshipit-source-id: 15aae78d9b3a82227278f33def45fa960aa65a98
Summary:
We already have a config to write shared data to an indexedlog. Let's
add a similar config for local data.
Reviewed By: xavierd
Differential Revision: D23909569
fbshipit-source-id: 87317554beb6bef8237e6a900403701662c3c0d0
Summary:
We temporarily dropped repair support when transitioning to using Store instead
of a raw RotateLog. Let's add that back now.
Reviewed By: xavierd
Differential Revision: D25371622
fbshipit-source-id: e28fc425a6ffb50c93540672b0df75a172ebbe9c
Summary:
In a future diff we'll use the indexedlog stores for local data. We
want those to exist forever, so let's move IndexedLogHgIdDataStore to use a
Store under the hood, and add an enum for distinguishing between the two types
at creation time.
Reviewed By: xavierd
Differential Revision: D23915622
fbshipit-source-id: 296cf6dfcd53e5cf1ae7624fdccedf0a60a77f22
Summary:
In a future diff we'll be moving the IndexedLogHgIdDataStore to use the Store
type (that hides the differences between IndexedLog and RotateLog). To do so, it
needs to support repairs. Let's do some minor refactoring to enable this.
Reviewed By: xavierd
Differential Revision: D25371623
fbshipit-source-id: 846cb5f8c21f1e6b550a45259cc8da24cc65b13b
Summary:
The end goal is to have clients using a sparse IdMap. There is still some work
to get there though. In the mean time we can test repositories that don't use
any revlogs. The current expections for those repositories are that they have
a full idmap locally.
Reviewed By: quark-zju
Differential Revision: D25075341
fbshipit-source-id: 52ab881fc9c64d0d13944e9619c087e0d4fb547c
Summary:
This commit adds a new eden configuration option that
controls whether we try to load our edenfs.kext in preference to
an alternative fuse implementation on macOS.
The majority of this diff is plumbing to convey the configuration
value through to the privhelper, which is relatively restrictive
due to its root-ness.
I've also updated watchman and mercurial to be aware of the new
filesystem type that shows up in the mount table.
Reviewed By: genevievehelsel
Differential Revision: D25065462
fbshipit-source-id: 4f35b9440654298e2706a0d0613d97eb63451999
Summary:
Introduce `edenapithin` crate, which offers C bindings to EdenAPI Client.
There are two top-level `owned` types, `EdenApiClient` and `TreeEntryFetch`, which represent `Box`ed results from calls to EdenAPI's blocking client. The C / C++ code is responsible for calling the associated `free` functions for these types, as well as `OwnedString`, which is used to represent error variants of a `Result` as a string.
Most functionality is provided through functions which operate on and return references into these top-level owned types, providing access into Rust `Result`s and `Vec`s (manually-monomorphized), and EdenApi's `TreeEntry` and `TreeChildEntry`.
Additional non-pointer types are defined in the `types` module, which do not require manual memory management.
C++ bindings are not included currently, but will be introduced soon.
Reviewed By: fanzeyi
Differential Revision: D24866065
fbshipit-source-id: bb15127b84cdbc6487b2d0e1798f37ef62e5b32d
Summary:
Introduce a new API type, `TreeAttributes`, corresponding to the existing type `WireTreeAttributesRequest`, which exposes which optional attributes are available for fetching. An `Option<TreeAttributes>` parameter is added to the `trees` API, and if set to `None`, the client will make a request with TreeAttributes::default().
The Python bindings accept a dictionary for the attributes parameter, and any fields present will overwrite the default settings from TreeAttributes::default(). Unrecognized attributes will be silently ignored.
Reviewed By: kulshrax
Differential Revision: D25041255
fbshipit-source-id: 5c581c20aac06eeb0428fff42bfd93f6aecbb629
Summary: Adding `full_idmap_clone` to edenapi and usign that in `debugsegmentclone`.
Reviewed By: quark-zju
Differential Revision: D25139730
fbshipit-source-id: 682055f7c30a94a941acd16e2b8e61b9ea1d0aef
Summary:
This method reconstructs a dag from clone data.
At the moment we only have a clone data construction method in Mononoke. It's
the Dags job to construct and import the clone_data. We'll consolidate that at
a later time.
Reviewed By: quark-zju
Differential Revision: D24954823
fbshipit-source-id: fe92179ec80f71234fc8f1cf7709f5104aabb4fb
Summary:
The server expects post requests. At this time we don't want to cache this data
so POST.
Reviewed By: singhsrb
Differential Revision: D24954824
fbshipit-source-id: 433672189ad97100eee7679894a894ab1d8cff7b
Summary:
The config is something that makes sense for all commands to have access to.
Commands that don't use a repo don't have access to the config that is prepared
by the dispatches. This change is a stop-gap to allow new commands that don't
require a repository to receive the config as an argument.
The construction of the config is something that we should iterate on. I see
the current implementaiton as a workaround.
Reviewed By: quark-zju
Differential Revision: D24954822
fbshipit-source-id: 42254bb201ba8838e7cc107394e8fab53a1a95c7
Summary:
While trying to repro a user report (https://fburl.com/jqvm320o), I ran into a
new hg error: P151186623, i.e.:
```
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("fbcode/thrift/facebook/test/py/TARGETS"), hgid: HgId("55713728544d5955703d604299d77bb1ed50c62d") })'
```
After some investigation (and adding a lot of prints), I noticed that this
was trying to query the EdenAPI server for this filenode. That request should
succeed, given Mononoke knows about this filenode:
```
[torozco@devbig051]~/fbcode % mononoke_exec mononoke_admin fbsource --use-mysql-client filenodes by-id fbcode/thrift/facebook/test/py/TARGETS 55713728544d5955703d604299d77bb1ed50c62d
mononoke_exec: Using config stage prod (set MONONOKE_EXEC_STAGE= to override)
I1126 08:10:02.089614 3697890 [main] eden/mononoke/cmdlib/src/args/mod.rs:1097] using repo "fbsource" repoid RepositoryId(2100)
I1126 08:10:02.995172 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:137] Filenode: HgFileNodeId(HgNodeHash(Sha1(55713728544d5955703d604299d77bb1ed50c62d)))
I1126 08:10:02.995282 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:138] -- path: FilePath(MPath("fbcode/thrift/facebook/test/py/TARGETS"))
I1126 08:10:02.995341 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:139] -- p1: Some(HgFileNodeId(HgNodeHash(Sha1(ccb76adc7db0fc4a395be066fe5464873cdf57e7))))
I1126 08:10:02.995405 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:140] -- p2: None
I1126 08:10:02.995449 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:141] -- copyfrom: None
I1126 08:10:02.995486 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:142] -- linknode: HgChangesetId(HgNodeHash(Sha1(dfe46f7d6cd8bc9b03af8870aca521b1801126f0)))
```
Turns out, the success rate for this method is actually 0% — out of thousands of requests,
not a single one succeeded :(
https://fburl.com/scuba/edenapi_server/cma3c3j0
The root cause here is that the server side is not properly deserializing
requests (actually finding that was a problem of its own, I filed T80406893 for this).
If you manage to get it to print the errors, it says:
```
{"message":"Deserialization failed: missing field `path`","request_id":"f97e2c7c-a432-4696-9a4e-538ed0db0418"}
```
The reason for this is that the server side tries to deserialize the request
as if it was a `WireHistoryRequest`, but actually it's a `HistoryRequest`, so
all the fields have different names (we use numbers in `WireHistoryRequest`).
This diff fixes that. I also introduced a helper method to make this a little
less footgun-y and double-checked the other callsites. There is one callsite
right now that looks like it might be broken (the commit one), but I couldn't
find the client side interface for this (I'm guessing it's not implemented
yet), so for now I left it as-is.
Reviewed By: StanislavGlebik
Differential Revision: D25187639
fbshipit-source-id: fa993579666dda762c0d71ccb56a646c20aee606
Summary:
Right now we just get a "deadline exceeded" error, which isn't very amenable to
helping us understand why we timed out. Let's add more logging. Notably,
I'd like to understnad what we've actually received at this point, if anything,
and how long we waited, as I'm starting to suspect this issue doesn't have much
to do with HTTP.
See https://fb.workplace.com/groups/scm/permalink/3361972140519049/ for
more context.
Reviewed By: quark-zju
Differential Revision: D25128159
fbshipit-source-id: b45d3415526fdf21aa80b7eeed98ee9206fbfd12
Summary:
We've seen some hangs with http 2 in lfs. Switching to http 1.1 seems
to fix it. Let's make this configurable so we can tweak this if we see it in
edenapi. For now we continue to default to http 2.
Reviewed By: krallin
Differential Revision: D24901201
fbshipit-source-id: 9806e2c37fa299e4bd381ebdcb17d00800408de3
Summary:
osxfuse is rebranding as macfuse in 4.x.
That has ripple effects through how the filesystem is mounted and shows up in
the system.
This commit adjusts for the new opaque and undocumented mount procedure and
speculatively updates a couple of other code locations that were sensitive to
looking for "osxfuse" when examining filesystems.
Reviewed By: genevievehelsel
Differential Revision: D24769826
fbshipit-source-id: dab81256a31702587b0683079806558e891bd1d2
Summary:
We've seen http 2 potentially causing hangs for users. Let's make this
configurable for lfs, so we can disable it and see if things get fixed.
Reviewed By: krallin
Differential Revision: D24898322
fbshipit-source-id: dc7842c0247dc6b9590a1f160076b17788aab1b9
Summary:
As discussed in a group thread (see link below), HTTP 2 may be causing
hangs for users. Let's start by making the http-client configurable. In
subsequent diffs we'll make edenapi and lfs configurable as well.
Reviewed By: krallin
Differential Revision: D24898323
fbshipit-source-id: f0035a1b8df3cee626ebe519e9e99358c1b3f043
Summary:
This isn't code that compiles, but the convention in Rust is that code actually
is doctests unless annotated otherwise, so if tested with Cargo, those fail.
This fixes that.
Reviewed By: farnz
Differential Revision: D24917364
fbshipit-source-id: 62fe11700ce561c13dc5498e01d15894b17b5b22
Summary:
Transfers iddag flat segments along with the head_id that should be use to
rebuild a full fledged IdDag. It also transfers idmap details. In the current
version it only transfers universal commit mappings.
Reviewed By: krallin
Differential Revision: D24808329
fbshipit-source-id: 4de9edcab56b54b901df1ca4be5985af2539ae05
Summary:
BE: remove old subscription to save resources in IceBreaker. The client code will recreate it anyway if missing but cleaning up will help us to reduce number of unused subscriptions.
Classic example: repo opsfiles or configerator maybe needed once and then a user don't use
Another example: switching workspaces failed and it could be result in subscriptions are not cleaned up properly
Reviewed By: markbt
Differential Revision: D24859931
fbshipit-source-id: 6df6c7e5f95859946726e04bce8bc8f3ac2d03df
Summary:
This function is useful in the mononoke to compute the universal commit idmap
that is required for clone.
Reviewed By: quark-zju
Differential Revision: D24808327
fbshipit-source-id: 0cccd59bd7982dd0bc024d5fc85fb5aa5eafb831
Summary:
`flat_segments` are going to be used to generate CloneData. These segments will
be sent to a client repository and are going to bootstrap the iddag.
Reviewed By: quark-zju
Differential Revision: D24808331
fbshipit-source-id: 00bf9723a43bb159cd98304c2c4c6583988d75aa
Summary: This is the object that will be used to bootstrap a Dag after a clone.
Reviewed By: quark-zju
Differential Revision: D24808328
fbshipit-source-id: 2c7e97c027c84a11e8716f2e288500474990169b
Summary:
The goal is to reused the functionality provided by AssignHeadOutcome for clone
purposes.
Reviewed By: quark-zju
Differential Revision: D24717924
fbshipit-source-id: e88f21ee0d8210e805e9d6896bc8992009bd7975
Summary:
The goal of this code was to divide the cache limit by the number of
logs. Instead it divided the cache limit by the default per-log size (2GB). That
results in a very small max-bytes-per-log so data was being thrown out
constantly. This fixes it and updates tests to actually demonstrate the issue.
Reviewed By: kulshrax
Differential Revision: D24712842
fbshipit-source-id: 8062758b5bfa40493e2003d5a9028d601b1522b1
