Summary:
With the internal streampager, progress bars must be sent on a separate stream so that
streampager can render them correctly.
Reviewed By: quark-zju
Differential Revision: D21906173
fbshipit-source-id: eb41b0bf22807d9cae518b3f676996ab1c642c6e
Summary: Upgrade the internal streampager to the latest version, which includes line wrapping.
Reviewed By: quark-zju
Differential Revision: D21906172
fbshipit-source-id: 1bc63723f55ac115090e7ae0a2541158863056b9
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff encapsulates the newly added batch blob import API in backingstore crate, and expose it to EdenFS. It converts incoming complex data structures down to primitive data types that can go through FFI boundary.
Reviewed By: xavierd
Differential Revision: D21697578
fbshipit-source-id: 8f5505ef4cab342dfa710798a04df4e0e94055d9
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff implements getBlobBatch in `backingstore` crate that is able to process multiple blob import request at once. This will help EdenFS to process blob import request efficiently. See following diffs in the stack for usage.
To import a list of import requests, the function first check if the file exists locally. When it is already in local store, the function will call the provided `resolve` function to send the blob back. Then it proceeds to send **ONE** request to remote store (in our case, EdenAPI) for all these missing files if this import permits remote importing.
We use the callback style `resolve` parameter because we want to awake waiting threads as soon as the blob is ready, so we don't waste time on unnecessary waiting.
Reviewed By: xavierd
Differential Revision: D21697580
fbshipit-source-id: b550accf6f6163cf6f2e9be6b628e46f44076c37
Summary: This makes the API more flexible.
Reviewed By: sfilipco
Differential Revision: D21626196
fbshipit-source-id: d0d28ba075ee3321a1a757f848fb72592827d75d
Summary: This allows DAG algorithms to be executed on revlog.
Reviewed By: sfilipco
Differential Revision: D21626217
fbshipit-source-id: e9fd61ad62f95be7b055a0ef8879c59cbeeb60b9
Summary:
Mostly copy-paste from code added in D19503373 and D19511574. Adjusted to match
the revlog index interface.
Reviewed By: sfilipco
Differential Revision: D21626201
fbshipit-source-id: 05d160e4c03d7e2482b6a4f2d68c3688ad78f568
Summary: The trait converts NameSet to IdSet. It'll be used by the revlog index.
Reviewed By: sfilipco
Differential Revision: D21795869
fbshipit-source-id: 55f7a238158442db9d8bdfe84e64438be504f618
Summary: Add a way to inverse the DAG (swap parent / children relations).
Reviewed By: sfilipco
Differential Revision: D21795870
fbshipit-source-id: 2d076f4ae491141aa758faa5f5f303c97f7e56dc
Summary:
Similar to LazySet, but the iterator is using Ids. This will be useful for
lazy calculations that are cheaper with Ids.
Reviewed By: sfilipco
Differential Revision: D21626208
fbshipit-source-id: 9a34fbf18f0039caeb4f6e698294c4d335354093
Summary:
The NameSet is not really about Dag. It is about using Id and is static.
Rename it to clarify. In an upcoming change we'll have IdLazySet.
Reviewed By: sfilipco
Differential Revision: D21626204
fbshipit-source-id: 84f25008f7032f6e26a26fc656ccbcd2a5880ecf
Summary:
This makes it possible to use NameIter without manually specifying out iterator
types, which might be quite long.
Reviewed By: sfilipco
Differential Revision: D21626202
fbshipit-source-id: 67b338765c09629645794cf73a9b496271524f9d
Summary: Take advantage of Hints and add fast paths.
Reviewed By: sfilipco
Differential Revision: D21626216
fbshipit-source-id: 6d43666bd6cdec7ff4b93032c1064cafd8de85cf
Summary: Update hints if they are easy to obtain or calculate.
Reviewed By: sfilipco
Differential Revision: D21626206
fbshipit-source-id: 453b7db2444406ce51d574c688fe536316fb9b0f
Summary:
In some rare cases, we would have hgcache that contains broader permission than we are expecting. We shouldn't be fixing it if that's the case.
We also might be in situations where hgcache directory isn't entirely created by Mercurial, and the owner of the directory will be different than the process. This will cause the `chmod` call to fail with permission error. In that case, this will cause EdenFS to panic. This is undesirable. We should be handling this case more gracefully and let the original error populate.
Reviewed By: xavierd
Differential Revision: D21854542
fbshipit-source-id: e9d11399aeb40b375725b49f4bcd54050afdcbad
Summary:
Previously, the NameSet has properties like "is_all", "is_topo_sorted", etc.
To make lazy sets efficient, it's important to have hints about min / max Ids
and maybe some other information.
Add a dedicated Hints structure for that.
Reviewed By: sfilipco
Differential Revision: D21626219
fbshipit-source-id: 845e88d3333f0f48f60f2739adae3dccc4a2dfc4
Summary:
Implement a small subset of DagAlgorithm by default. This makes
other implementations of DagAlgorithm slightly easier.
Reviewed By: sfilipco
Differential Revision: D21626199
fbshipit-source-id: ac6dfb5c22bf1da44f521fc9e76d59bfb95063c7
Summary:
D21479023 broke it. It should convert to Id, and check Id against the SpanSet,
instead of just checking the IdMap ignoring the SpanSet.
Reviewed By: sfilipco
Differential Revision: D21626193
fbshipit-source-id: 6daf86f292a7acfd3688893a55e2a794cfe068fe
Summary: This makes the next change easier to implement.
Reviewed By: sfilipco
Differential Revision: D21626198
fbshipit-source-id: 57ab69cba7f43350767e5d0d52ebfe66764895ca
Summary:
Implements part of the dag IdMap related traits.
It does not get used yet, but eventually I'd like `pydag` to be able to work
with an abstracted dag including RevlogIndex.
Reviewed By: sfilipco
Differential Revision: D21626210
fbshipit-source-id: 53f19622f03fd71b76073dccf8dcc9b4778b40ca
Summary:
This will allow RevLogIndex to answer node -> rev and hex lookup queries.
Also change RevlogIndex::new to take file names so it can write back the
nodemap index when the index is lagging. That part of logic currently exists in
pyindexes + clindex.pyx, which are going to be replaced by revlogindex.
Practically, this will generate a `00changelog.nodemap` file in svfs, which is
temporarily unused, but will be used once clindex.pyx gets replaced.
Reviewed By: sfilipco
Differential Revision: D21626209
fbshipit-source-id: 297d9eff26a73c26558708f7a2290d4d8ba1e777
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary: In line with other crates that contain utility binaries alongside the crate, rename the `edenapi/utils` directory to `edenapi/tools`.
Reviewed By: quark-zju
Differential Revision: D21796899
fbshipit-source-id: 058319e2756b1d596f06d6e57d17a6c07a7f1c9c
Summary:
Now that the shared directories are created with the right permissions, we
could still have some in the wild with the incorrect permissions. Let's make
sure that we fix these up.
Reviewed By: wez
Differential Revision: D21832436
fbshipit-source-id: d8ee40f61b16857d29e1360ec6df50b2a95ea7aa
Summary:
This is required for the shared cache to avoid permissions issues when multiple
users are trying to use it.
Reviewed By: fanzeyi
Differential Revision: D21830490
fbshipit-source-id: 3db0dbd674b6d2e10b9361ff3c3a668d046f5d78
Summary:
Adds a remotefilelog.write-hgcache-to-indexedlog config which directs
all hgcache writes to the indexedlog.
This change also removes remotefilelog.indexedlogdatastore as it's on by default
now.
Reviewed By: xavierd
Differential Revision: D21772132
fbshipit-source-id: a71e188df2958fb4f8c4776da23ba87deccb3b79
Summary:
Change `NodeRevMap`'s changelog type from `[u8]` to `[RevlogEntry]`.
This makes it consistent with `RevlogIndex`.
Reviewed By: sfilipco
Differential Revision: D21626203
fbshipit-source-id: 7457f48ccd7b3489264684a5db21d21e9eb7a937
Summary:
NodeRevMap helps converting from a commit hash to a rev number. It's similar to
IdMap in the dag crate, but was designed for the revlog.
Move NodeRevMap to revlogindex so it becomes easier to implement the IdConvert
trait required by the dag crate.
Reviewed By: sfilipco
Differential Revision: D21626211
fbshipit-source-id: 14996f1234231b507efb5186ec30f84df5aaad10
Summary:
The idea is that the pure Rust revlogindex crate can implement the DagAlgorithm
interface so we will have a consistent interface in the code base that works
for both the existing storage (revlog) and the new segmented changelog.
The other way to do this is to implement the `bindings.dag.namedag` interface
in pure Python for the revlog-based DAG, or supporting quite different
interfaces (ex. revlog DAG and the Rust segmented changelog DAG) in the code
base. At present, I think implementing the Rust DAG traits for revlog is the
most appealing, partially because we already have some key algorithms
implemented (ex. prefix lookup, common ancestors, etc).
Reviewed By: sfilipco
Differential Revision: D21626197
fbshipit-source-id: 733b1af1bcd5fc0784764fc7103412988894d43b
Summary:
We can skip the round trip to the server if we have nothing to ask or tell the
server.
Reviewed By: xavierd
Differential Revision: D21763025
fbshipit-source-id: 7f199c12ccaa0f66ce765dd976723e4002c5dd4e
Summary: Added comments showing the expected JSON input format for each kind of EdenAPI request.
Reviewed By: xavierd
Differential Revision: D21782765
fbshipit-source-id: bf08dd4b36a33aca506eb0fa0341e40d0150d7cb
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary: This diff replaces the use of the oxidized typechecker types in `to_oxidized`, replacing them with oxidized_by_ref definitions. This unblocks deleting typechecker types and decls from the oxidized crate, since they are used only in this debugging-only path. It also reduces the amount of conversion necessary for `to_oxidized` (since we need not convert the oxidized_by_ref structures the typechecker already uses, like `Ty`).
Differential Revision: D21587518
fbshipit-source-id: 5e7ec2237d0059070653b45d15eb7c1259588ccb
Summary:
Let's hide the repo list in the fb/ module since we don't really want
to expose the whole repo list in our public repo.
Reviewed By: sfilipco
Differential Revision: D21731419
fbshipit-source-id: 22ca20a3a80637c852e313f1390849aac1fecbf4
Summary:
This adds flexibility. Now every type that implements DagAddHeads, including
NameDag, can import ASCII graphs.
Reviewed By: sfilipco
Differential Revision: D21626213
fbshipit-source-id: e258d88f97cbcc9aaf98d353a929803325185df7
Summary: The trait was about converting between Id and VertexName. Rename to clarify.
Reviewed By: sfilipco
Differential Revision: D21626195
fbshipit-source-id: 874ca4ca3a1467084a08c6d9aa321201974e1978
Summary: This allows other kinds of DAG to implement the operations.
Reviewed By: sfilipco
Differential Revision: D21626220
fbshipit-source-id: 896c5ccebb1672324d346dfca6bcac9b4d3b4929
Summary: This makes things a bit more flexible.
Reviewed By: sfilipco
Differential Revision: D21626194
fbshipit-source-id: f3ad486bcd5a6478d9e00f674d48f99504cded8c
Summary: This makes it possible for other types to implement the hex prefix lookup.
Reviewed By: sfilipco
Differential Revision: D21626218
fbshipit-source-id: 96e8b8c37e5aae2bd60658a238333b61902936d1
Summary: It will be used in the next change.
Reviewed By: sfilipco
Differential Revision: D21626207
fbshipit-source-id: bbef70ef9d4f9aaa2039a6bc15d296e88db7f8dc
Summary:
Types like IdDag are not really used. The use of the word "name" is sometimes
confusing in other context. Therefore export shorter names like Dag, MemDag,
Vertex, avoid "name" in NameDag, MemNameDag and NameSet. This makes external
code shorter and less ambiguous.
Reviewed By: sfilipco
Differential Revision: D21626212
fbshipit-source-id: 5bcf3cecfd38277149b41bf3ba9e6d4ef2a07b2b
Summary:
This decouples DagAlgorithm from the IdMap + IdDag backend, making it possible
to support other kinds of backends of DagAlgorithm (ex. a revlog backend).
Reviewed By: sfilipco
Differential Revision: D21626200
fbshipit-source-id: f53cc271a200062e9c02f739b6453e1d7de84e6d
Summary:
xavierd and I realized that memcache is not actually enabled in EdenFS as `ContentStoreBuilder` only sets it when there is are remote store. We believe turning on memcache store can help us improve some performance for importing blobs (80% hit rate).
This diff adds a noop remote store that does nothing to trick `ContentStoreBuilder` to include the memcache store. We decided to go this way instead of modifying `ContentStoreBuilder` is because EdenFS is going to have a real remotestore very soon, and changes to `ContentStoreBuilder` may have some impact on how Mercurial uses it.
Reviewed By: xavierd
Differential Revision: D21621234
fbshipit-source-id: 5502e001ab498134b6a927d83be823261e70e9e1
Summary:
There was a bug where if the dynamicconfig chose a value that matched
one of the ported rc files, but it did not match the final value of that config
(from a not-yet-ported rc file), it would chose the dynamicconfig value instead
of the original final value. Let's drop the dynamicconfig value in this case as
well.
Reviewed By: quark-zju
Differential Revision: D21674469
fbshipit-source-id: efcd36e9602e16210999ec8c88e86b1d7ee355e4
Summary:
If either the config, or the environment variable is empty, the URL parsing
would raise an error, while the intent is to not have proxy. Let's handle it
properly.
Reviewed By: DurhamG
Differential Revision: D21669922
fbshipit-source-id: 84c8d09242ac8d5571f7592d874f4954692110f5
Summary:
At clone time we apply dynamic configs in memory, instead of loading
them from disk. The validation logic operated on the config value's location
field, which isn't set for data that comes from in memory. So we need to update
the validation logic to also consider the value source if location is not set.
Reviewed By: quark-zju
Differential Revision: D21664205
fbshipit-source-id: 8460c58c6d654780048de51ada8178c70ff0a9e6
Summary:
We kick off a background process every 15 minutes or so to compute the
new dynamicconfig. If the config hasn't changed, we should just touch the file
instead of rewriting the whole thing.
Reviewed By: quark-zju
Differential Revision: D21653098
fbshipit-source-id: 9d53d2a636cff082cd048994255cc809ce1b0221
