Summary:
In the next diffs we'll make it possible to disable filenodes in Mononoke. See
D21787848 and attached task for more details, but TL;DR is that if xdb is down
we still want to serve "hg update" traffic.
If filenodes are disabled we obviously can't generate filenodes for new
commits. So one option would be to just return an error from
FilenodesOnlyPublic::derive(...) call. But that would mean that any attempt to
call derivation would fail, and e.g. Mononoke servers won't be able to start up
- (see https://fburl.com/diffusion/roau028d). We could change callers to always
process errors from FilenodesOnlyPublic, but I think it would be harder to
enforce and easier to forget.
So this diff changes FilenodesOnlyPublic to be an enum, and
FilenodesOnlyPublic::Disabled is returned immediately if filenodes are
disabled. For callers it means that they can't rely on filenodes being present
in db even after FilenodesOnlyPublic were derived. That's the whole of the
stack, and the next diffs will update the callers to properly deal with missing
filenodes.
One caveat is that when we re-enable filenodes back we might need to derive
them for a lot of commits.
I don't expect it to happen often (i.e. if xdb is down then we probably can't
commit anyway), but if somehow it happened, then we should be a bit more
careful with re-enabling them after the problem was fixed. For example, we can
first derive all the filenodes locally by e.g. running backfill_derived_data,
and only after that has finished successfully we can re-enable them.
Reviewed By: krallin
Differential Revision: D21840328
fbshipit-source-id: ce9594d4a21110a5cb392c3049ccaede064c1e66
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
Instead of always building from scratch, continue assiging Vertexes and
Segments from the last commit that was processed.
Reviewed By: StanislavGlebik
Differential Revision: D21634699
fbshipit-source-id: 9f8b890dcf65c59a66651343f0ccc1487efc2394
Summary:
Somehow, on Windows, the socket file that was removed during `prepare` may be
back, but not bound, preventing Thrift from binding to it, let's remove it
again since it's not supposed to be there.
Note that this happens 100% of the time when starting EdenFS via the Task
Scheduler at log on, I couldn't reproduce when triggering the task manually,
or when starting edenfs by hand.
Reviewed By: simpkins
Differential Revision: D21755498
fbshipit-source-id: 32a343d9de531b90417dac0aa382a07c85cdee5a
Summary:
The CREATE_NO_WINDOW is documented as:
The process is a console application that is being run without a console
window. Therefore, the console handle for the application is not set.
The last part is what matters here, by default stdin, stdout and stderr are
inherited from the parent process, with this flag, they won't as these 3 will
simply not be set. Removing it allows Mercurial's output to be sent directly to
the edenfs log which will greatly improve our debuggability.
Reviewed By: akrieger
Differential Revision: D21820195
fbshipit-source-id: 63496a1ad28ecf5440c0846d80e1fbda5756b971
Summary:
Previously, the NameSet has properties like "is_all", "is_topo_sorted", etc.
To make lazy sets efficient, it's important to have hints about min / max Ids
and maybe some other information.
Add a dedicated Hints structure for that.
Reviewed By: sfilipco
Differential Revision: D21626219
fbshipit-source-id: 845e88d3333f0f48f60f2739adae3dccc4a2dfc4
Summary:
Implement a small subset of DagAlgorithm by default. This makes
other implementations of DagAlgorithm slightly easier.
Reviewed By: sfilipco
Differential Revision: D21626199
fbshipit-source-id: ac6dfb5c22bf1da44f521fc9e76d59bfb95063c7
Summary:
D21479023 broke it. It should convert to Id, and check Id against the SpanSet,
instead of just checking the IdMap ignoring the SpanSet.
Reviewed By: sfilipco
Differential Revision: D21626193
fbshipit-source-id: 6daf86f292a7acfd3688893a55e2a794cfe068fe
Summary: This makes the next change easier to implement.
Reviewed By: sfilipco
Differential Revision: D21626198
fbshipit-source-id: 57ab69cba7f43350767e5d0d52ebfe66764895ca
Summary:
Implements part of the dag IdMap related traits.
It does not get used yet, but eventually I'd like `pydag` to be able to work
with an abstracted dag including RevlogIndex.
Reviewed By: sfilipco
Differential Revision: D21626210
fbshipit-source-id: 53f19622f03fd71b76073dccf8dcc9b4778b40ca
Summary:
This will allow RevLogIndex to answer node -> rev and hex lookup queries.
Also change RevlogIndex::new to take file names so it can write back the
nodemap index when the index is lagging. That part of logic currently exists in
pyindexes + clindex.pyx, which are going to be replaced by revlogindex.
Practically, this will generate a `00changelog.nodemap` file in svfs, which is
temporarily unused, but will be used once clindex.pyx gets replaced.
Reviewed By: sfilipco
Differential Revision: D21626209
fbshipit-source-id: 297d9eff26a73c26558708f7a2290d4d8ba1e777
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary: In line with other crates that contain utility binaries alongside the crate, rename the `edenapi/utils` directory to `edenapi/tools`.
Reviewed By: quark-zju
Differential Revision: D21796899
fbshipit-source-id: 058319e2756b1d596f06d6e57d17a6c07a7f1c9c
Summary:
Clean up some of the conversion functions by renaming variables that are
keywords in other languages, and simplifying error handling code.
Differential Revision: D21839019
fbshipit-source-id: d8945a14a230caa744040e134203a908ad9cef20
Summary: `ErrorKind` is not meaningful, and is an artifact of older-style error handling crates. A better name is `ConfigurationError`.
Reviewed By: krallin
Differential Revision: D21837271
fbshipit-source-id: 709d9e2ab7f18dd2f7cb2489f24e91612bc378db
Summary:
Replace the use of `RepoConfigs::read*` associated functions with free
functions. These didn't really need to be associated functions (and in the
case of the common and storage configs, really didn't belong there either).
Reviewed By: krallin
Differential Revision: D21837270
fbshipit-source-id: 2dc73a880ed66e11ea484b88b749582ebdf8a73f
Summary:
Refactor parsing of repo config using a new `Convert` trait to allow
definition of each part of parsing separately.
The wireproto logging args require access to the storage definitions, so need
to be parsed by their own special function for now.
Differential Revision: D21837269
fbshipit-source-id: 7ab0e3f4b3b8549aaefb45201388c3dfc7633ef7
Summary:
Refactor parsing of storage config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766761
fbshipit-source-id: 7e224e9d322a3a16a64f5ebba2243bbe6341c8f0
Summary:
Refactor parsing of commit sync config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766760
fbshipit-source-id: 3c95d70788753316d3c1f36280e7d6dbb52a9710
Summary:
We'd like to serve read traffic even if filenodes are disabled. Let's add a
tunable that can control it.
Reviewed By: HarveyHunt
Differential Revision: D21839672
fbshipit-source-id: 4ec4dd16b9e6e3ffb1ada0d812e1153e1a33a268
Summary: It was replaced with a parameter
Reviewed By: HarveyHunt
Differential Revision: D21839397
fbshipit-source-id: e75900b3da80985cd762659993b8b285411fe928
Summary:
Now that the shared directories are created with the right permissions, we
could still have some in the wild with the incorrect permissions. Let's make
sure that we fix these up.
Reviewed By: wez
Differential Revision: D21832436
fbshipit-source-id: d8ee40f61b16857d29e1360ec6df50b2a95ea7aa
Summary:
This is required for the shared cache to avoid permissions issues when multiple
users are trying to use it.
Reviewed By: fanzeyi
Differential Revision: D21830490
fbshipit-source-id: 3db0dbd674b6d2e10b9361ff3c3a668d046f5d78
Summary: Add more information for bisect states.
Reviewed By: quark-zju
Differential Revision: D21758828
fbshipit-source-id: 4bc617e50f70277428dc4c7c1be68b652d78b8f8
Summary:
In D20286499 I added chunking of remotefilelog requests, however apparently the
place where I've added it is no longer used (xavierd mentioned it might be
because of migration to rust stores).
let's put chunking logic into getpack() function
Reviewed By: xavierd
Differential Revision: D21765414
fbshipit-source-id: b481ddf070f7bd86d0071cea7be2f9cc6ef1e5d9
Summary:
DefferedDerivedMapping was added so that we can make deriving stack of commits faster - it does it by postponing updating
derived data mapping (e.g. writing to a blobstore) until the whole stack is derived.
While it probably makes derivation a bit faster, we now think it's better to remove it. A few reasons:
1) It's confusing to understand and it already caused us ubns before
2) It's increases write amplification - because we release the lease before we wrote to a blobstore, writers will try to rederive the same commit a few times. That has caused us a ubn today
Reviewed By: farnz
Differential Revision: D20113854
fbshipit-source-id: 169e05febcd382334bf4da209a20aace0b7c2333
Summary:
See D21765065 for more context. TL;DR is that we want to control
lfs rollout from client side to make sure we don't put lfs pointers in the
shared memcache
Reviewed By: xavierd
Differential Revision: D21822159
fbshipit-source-id: daea6078d95eb4e9c040d353a20bcdf1b6ae07b1
Summary: [Folly] Cut the `FOR_EACH_ENUMERATE` macro, which may be replaced by a combination of range-for, `ranges::view::enumerate`, and structured bindings.
Reviewed By: markisaa
Differential Revision: D21813019
fbshipit-source-id: fc9ac09a4e2f72f1433d0a518f03d5cd69a59c55
Summary:
Adds a remotefilelog.write-hgcache-to-indexedlog config which directs
all hgcache writes to the indexedlog.
This change also removes remotefilelog.indexedlogdatastore as it's on by default
now.
Reviewed By: xavierd
Differential Revision: D21772132
fbshipit-source-id: a71e188df2958fb4f8c4776da23ba87deccb3b79
Summary:
`fixcorrupt` accesses changelog in a way that is going to cause trouble with
the next change. Since `hg doctor` can do what `fixcorrupt` does in production
setup. Let's sunset fixcorrupt.
Reviewed By: singhsrb
Differential Revision: D21780575
fbshipit-source-id: 5efec3f066a2929018ccc68b1f52d10a76e59637
Summary:
Change `NodeRevMap`'s changelog type from `[u8]` to `[RevlogEntry]`.
This makes it consistent with `RevlogIndex`.
Reviewed By: sfilipco
Differential Revision: D21626203
fbshipit-source-id: 7457f48ccd7b3489264684a5db21d21e9eb7a937
Summary:
NodeRevMap helps converting from a commit hash to a rev number. It's similar to
IdMap in the dag crate, but was designed for the revlog.
Move NodeRevMap to revlogindex so it becomes easier to implement the IdConvert
trait required by the dag crate.
Reviewed By: sfilipco
Differential Revision: D21626211
fbshipit-source-id: 14996f1234231b507efb5186ec30f84df5aaad10
Summary:
The idea is that the pure Rust revlogindex crate can implement the DagAlgorithm
interface so we will have a consistent interface in the code base that works
for both the existing storage (revlog) and the new segmented changelog.
The other way to do this is to implement the `bindings.dag.namedag` interface
in pure Python for the revlog-based DAG, or supporting quite different
interfaces (ex. revlog DAG and the Rust segmented changelog DAG) in the code
base. At present, I think implementing the Rust DAG traits for revlog is the
most appealing, partially because we already have some key algorithms
implemented (ex. prefix lookup, common ancestors, etc).
Reviewed By: sfilipco
Differential Revision: D21626197
fbshipit-source-id: 733b1af1bcd5fc0784764fc7103412988894d43b
Summary: This cache didn't improve performance, and is a reasonable amount of extra code. Just rip it out, and improve higher caching layers if needed.
Reviewed By: StanislavGlebik
Differential Revision: D21818412
fbshipit-source-id: 3ca58bff180c6da91d451754b67b23e61b736059
Summary:
We can kill a single shard's replication with weight of puts; by tracking lag once a second, we can slow down to a point where no shard overloads.
This is insufficient by itself, as it slows all shards down equally (whereas we only want to slow down the laggy shard), but this now avoids high replication lag
Reviewed By: StanislavGlebik
Differential Revision: D21720833
fbshipit-source-id: a819f641206641e80f8edde92006fb08cdcf36a9
Summary:
Previously the return type was String which, in Python 2, could turn into bytes or
unicode depending on the contents of the string. We always want bytes in Python
2, so let's use the Str type instead.
Reviewed By: quark-zju
Differential Revision: D21794189
fbshipit-source-id: 6493fbacab354a78476f522fc3c41b7336dbbdb1
Summary:
Out `CommitSyncConfig` struct now contains a `version_name` field, which is intended to be used as an identifier of an individual version of the `commitsyncmap` in use. We want to record this value in the `synced_commit_mapping` table, so that later it is possible to attribute a commit sync map to a given commit sync.
This is part of a broader goal of adding support for sync map versioning. The broader goal is important, as it allows us to move faster (and troubleshoot better) when sync maps need changing.
Note that when commit is preserved across repos, we set `version_name` to `NULL`, as it makes no sense to attribute commit sync maps to those case.
Reviewed By: farnz
Differential Revision: D21765408
fbshipit-source-id: 11a77cc4d926e4a4d72322b51675cb78eabcedee
Summary:
The checkout step for EdenFS can take a few seconds if the distance is large or
if EdenFS has to fetch data. Reassure the user that something is happening
with a progress spinner.
Reviewed By: chadaustin
Differential Revision: D21684077
fbshipit-source-id: 43b8793f1a55ef6fb3bf78cabe3afcf2faed1d6c
Summary:
The motivation for the whole stack:
At the moment if mysql is down then Mononoke is down as well, both for writes
and for reads. However we can relatively easily improve the situation.
During hg update client sends getpack() requests to fetch files, and currently
for each file fetch we also fetch file's linknode. However hg client knows how
to deal with null linknodes [1], so when mysql is unavailable we can disable
filenode fetching completely and just return null linknodes. So the goal of this stack is to
add a knob (i.e. a tunable) that can turn things filenode fetches on and off, and make
sure the rest of the code deals nicely with this situation.
Now, about this diff. In order to force callers to deal with the fact that
filenodes might unavailable I suggest to add a special type of result, which (in
later diffs) will be returned by every filenodes methods.
This diff just introduces the FilenodeResult and convert BlobRepo filenode
methods to return it. The reason why I converted BlobRepo methods first
is to limit the scope of changes but at the same time show how the callers' code will look
like after FilenodeResult is introduced, and get people's thoughts of whether
it's reasonable or not.
Another important change I'd like to introduce in the next diffs is modifying FilenodesOnlyPublic
derived data to return success if filenodes knob is off. If we don't do that
then any attempt to derive filenodes might fail which in turn would lead to the
same problem we have right now - people won't be able to do hg update/hg
pull/etc if mysql is down.
[1] null linknodes might make some client side operation slower (e.g. hg rebase/log/blame),
so we should use it only in sev-like situations
Reviewed By: krallin
Differential Revision: D21787848
fbshipit-source-id: ad48d5556e995af09295fa43118ff8e3c2b0e53e
Summary:
We can skip the round trip to the server if we have nothing to ask or tell the
server.
Reviewed By: xavierd
Differential Revision: D21763025
fbshipit-source-id: 7f199c12ccaa0f66ce765dd976723e4002c5dd4e
Summary:
`flush()` takes a timeout, which is in milliseconds. However, that's not super
obvious, so that means whoever is using this has to re-discover it. Let's make
it explicit, and use a `Duration` argument instead.
(In fact, some places even got it wrong, notably common/rust/cli_usage)
Reviewed By: farnz
Differential Revision: D21783991
fbshipit-source-id: 6e3ac7c22b5c3297b41d5d61373ee077f12d5dd4
Summary: Added comments showing the expected JSON input format for each kind of EdenAPI request.
Reviewed By: xavierd
Differential Revision: D21782765
fbshipit-source-id: bf08dd4b36a33aca506eb0fa0341e40d0150d7cb
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary:
We encountered an issue where an infinitepush bundle did not contain
trees for some of the commits it contained. This happened when multiple users
were working on the same branch. Our previous heuristic decided to not send
trees if they weren't in the local store, but with us transitioning to Mononoke
(which enables draft commit data to be fetched ondemand) this invariant doesn't
always hold true.
Let's always upload trees for draft commits. Hopefully Mononoke can take over on
the server completely soon and we can get rid of all this special casing and do
normal discovery.
This is a revert of D7992502, but that was added because it was possible to pull
directly from svn which would result in public commits that didn't exist on the
server yet and therefore needed to send their trees. Since svn is gone, this is
probably safe to change back.
Reviewed By: quark-zju
Differential Revision: D21697921
fbshipit-source-id: c1abaa061207222490b146ee52f7949be6fe4b2a
