Summary:
Previously, the NameSet has properties like "is_all", "is_topo_sorted", etc.
To make lazy sets efficient, it's important to have hints about min / max Ids
and maybe some other information.
Add a dedicated Hints structure for that.
Reviewed By: sfilipco
Differential Revision: D21626219
fbshipit-source-id: 845e88d3333f0f48f60f2739adae3dccc4a2dfc4
Summary:
Implement a small subset of DagAlgorithm by default. This makes
other implementations of DagAlgorithm slightly easier.
Reviewed By: sfilipco
Differential Revision: D21626199
fbshipit-source-id: ac6dfb5c22bf1da44f521fc9e76d59bfb95063c7
Summary:
D21479023 broke it. It should convert to Id, and check Id against the SpanSet,
instead of just checking the IdMap ignoring the SpanSet.
Reviewed By: sfilipco
Differential Revision: D21626193
fbshipit-source-id: 6daf86f292a7acfd3688893a55e2a794cfe068fe
Summary: This makes the next change easier to implement.
Reviewed By: sfilipco
Differential Revision: D21626198
fbshipit-source-id: 57ab69cba7f43350767e5d0d52ebfe66764895ca
Summary:
Implements part of the dag IdMap related traits.
It does not get used yet, but eventually I'd like `pydag` to be able to work
with an abstracted dag including RevlogIndex.
Reviewed By: sfilipco
Differential Revision: D21626210
fbshipit-source-id: 53f19622f03fd71b76073dccf8dcc9b4778b40ca
Summary:
This will allow RevLogIndex to answer node -> rev and hex lookup queries.
Also change RevlogIndex::new to take file names so it can write back the
nodemap index when the index is lagging. That part of logic currently exists in
pyindexes + clindex.pyx, which are going to be replaced by revlogindex.
Practically, this will generate a `00changelog.nodemap` file in svfs, which is
temporarily unused, but will be used once clindex.pyx gets replaced.
Reviewed By: sfilipco
Differential Revision: D21626209
fbshipit-source-id: 297d9eff26a73c26558708f7a2290d4d8ba1e777
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary: In line with other crates that contain utility binaries alongside the crate, rename the `edenapi/utils` directory to `edenapi/tools`.
Reviewed By: quark-zju
Differential Revision: D21796899
fbshipit-source-id: 058319e2756b1d596f06d6e57d17a6c07a7f1c9c
Summary:
Clean up some of the conversion functions by renaming variables that are
keywords in other languages, and simplifying error handling code.
Differential Revision: D21839019
fbshipit-source-id: d8945a14a230caa744040e134203a908ad9cef20
Summary: `ErrorKind` is not meaningful, and is an artifact of older-style error handling crates. A better name is `ConfigurationError`.
Reviewed By: krallin
Differential Revision: D21837271
fbshipit-source-id: 709d9e2ab7f18dd2f7cb2489f24e91612bc378db
Summary:
Replace the use of `RepoConfigs::read*` associated functions with free
functions. These didn't really need to be associated functions (and in the
case of the common and storage configs, really didn't belong there either).
Reviewed By: krallin
Differential Revision: D21837270
fbshipit-source-id: 2dc73a880ed66e11ea484b88b749582ebdf8a73f
Summary:
Refactor parsing of repo config using a new `Convert` trait to allow
definition of each part of parsing separately.
The wireproto logging args require access to the storage definitions, so need
to be parsed by their own special function for now.
Differential Revision: D21837269
fbshipit-source-id: 7ab0e3f4b3b8549aaefb45201388c3dfc7633ef7
Summary:
Refactor parsing of storage config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766761
fbshipit-source-id: 7e224e9d322a3a16a64f5ebba2243bbe6341c8f0
Summary:
Refactor parsing of commit sync config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766760
fbshipit-source-id: 3c95d70788753316d3c1f36280e7d6dbb52a9710
Summary:
We'd like to serve read traffic even if filenodes are disabled. Let's add a
tunable that can control it.
Reviewed By: HarveyHunt
Differential Revision: D21839672
fbshipit-source-id: 4ec4dd16b9e6e3ffb1ada0d812e1153e1a33a268
Summary: It was replaced with a parameter
Reviewed By: HarveyHunt
Differential Revision: D21839397
fbshipit-source-id: e75900b3da80985cd762659993b8b285411fe928
Summary:
Now that the shared directories are created with the right permissions, we
could still have some in the wild with the incorrect permissions. Let's make
sure that we fix these up.
Reviewed By: wez
Differential Revision: D21832436
fbshipit-source-id: d8ee40f61b16857d29e1360ec6df50b2a95ea7aa
Summary:
This is required for the shared cache to avoid permissions issues when multiple
users are trying to use it.
Reviewed By: fanzeyi
Differential Revision: D21830490
fbshipit-source-id: 3db0dbd674b6d2e10b9361ff3c3a668d046f5d78
Summary: Add more information for bisect states.
Reviewed By: quark-zju
Differential Revision: D21758828
fbshipit-source-id: 4bc617e50f70277428dc4c7c1be68b652d78b8f8
Summary:
In D20286499 I added chunking of remotefilelog requests, however apparently the
place where I've added it is no longer used (xavierd mentioned it might be
because of migration to rust stores).
let's put chunking logic into getpack() function
Reviewed By: xavierd
Differential Revision: D21765414
fbshipit-source-id: b481ddf070f7bd86d0071cea7be2f9cc6ef1e5d9
Summary:
DefferedDerivedMapping was added so that we can make deriving stack of commits faster - it does it by postponing updating
derived data mapping (e.g. writing to a blobstore) until the whole stack is derived.
While it probably makes derivation a bit faster, we now think it's better to remove it. A few reasons:
1) It's confusing to understand and it already caused us ubns before
2) It's increases write amplification - because we release the lease before we wrote to a blobstore, writers will try to rederive the same commit a few times. That has caused us a ubn today
Reviewed By: farnz
Differential Revision: D20113854
fbshipit-source-id: 169e05febcd382334bf4da209a20aace0b7c2333
Summary:
See D21765065 for more context. TL;DR is that we want to control
lfs rollout from client side to make sure we don't put lfs pointers in the
shared memcache
Reviewed By: xavierd
Differential Revision: D21822159
fbshipit-source-id: daea6078d95eb4e9c040d353a20bcdf1b6ae07b1
Summary: [Folly] Cut the `FOR_EACH_ENUMERATE` macro, which may be replaced by a combination of range-for, `ranges::view::enumerate`, and structured bindings.
Reviewed By: markisaa
Differential Revision: D21813019
fbshipit-source-id: fc9ac09a4e2f72f1433d0a518f03d5cd69a59c55
Summary:
Adds a remotefilelog.write-hgcache-to-indexedlog config which directs
all hgcache writes to the indexedlog.
This change also removes remotefilelog.indexedlogdatastore as it's on by default
now.
Reviewed By: xavierd
Differential Revision: D21772132
fbshipit-source-id: a71e188df2958fb4f8c4776da23ba87deccb3b79
Summary:
`fixcorrupt` accesses changelog in a way that is going to cause trouble with
the next change. Since `hg doctor` can do what `fixcorrupt` does in production
setup. Let's sunset fixcorrupt.
Reviewed By: singhsrb
Differential Revision: D21780575
fbshipit-source-id: 5efec3f066a2929018ccc68b1f52d10a76e59637
Summary:
Change `NodeRevMap`'s changelog type from `[u8]` to `[RevlogEntry]`.
This makes it consistent with `RevlogIndex`.
Reviewed By: sfilipco
Differential Revision: D21626203
fbshipit-source-id: 7457f48ccd7b3489264684a5db21d21e9eb7a937
Summary:
NodeRevMap helps converting from a commit hash to a rev number. It's similar to
IdMap in the dag crate, but was designed for the revlog.
Move NodeRevMap to revlogindex so it becomes easier to implement the IdConvert
trait required by the dag crate.
Reviewed By: sfilipco
Differential Revision: D21626211
fbshipit-source-id: 14996f1234231b507efb5186ec30f84df5aaad10
Summary:
The idea is that the pure Rust revlogindex crate can implement the DagAlgorithm
interface so we will have a consistent interface in the code base that works
for both the existing storage (revlog) and the new segmented changelog.
The other way to do this is to implement the `bindings.dag.namedag` interface
in pure Python for the revlog-based DAG, or supporting quite different
interfaces (ex. revlog DAG and the Rust segmented changelog DAG) in the code
base. At present, I think implementing the Rust DAG traits for revlog is the
most appealing, partially because we already have some key algorithms
implemented (ex. prefix lookup, common ancestors, etc).
Reviewed By: sfilipco
Differential Revision: D21626197
fbshipit-source-id: 733b1af1bcd5fc0784764fc7103412988894d43b
Summary: This cache didn't improve performance, and is a reasonable amount of extra code. Just rip it out, and improve higher caching layers if needed.
Reviewed By: StanislavGlebik
Differential Revision: D21818412
fbshipit-source-id: 3ca58bff180c6da91d451754b67b23e61b736059
Summary:
We can kill a single shard's replication with weight of puts; by tracking lag once a second, we can slow down to a point where no shard overloads.
This is insufficient by itself, as it slows all shards down equally (whereas we only want to slow down the laggy shard), but this now avoids high replication lag
Reviewed By: StanislavGlebik
Differential Revision: D21720833
fbshipit-source-id: a819f641206641e80f8edde92006fb08cdcf36a9
Summary:
Previously the return type was String which, in Python 2, could turn into bytes or
unicode depending on the contents of the string. We always want bytes in Python
2, so let's use the Str type instead.
Reviewed By: quark-zju
Differential Revision: D21794189
fbshipit-source-id: 6493fbacab354a78476f522fc3c41b7336dbbdb1
Summary:
Out `CommitSyncConfig` struct now contains a `version_name` field, which is intended to be used as an identifier of an individual version of the `commitsyncmap` in use. We want to record this value in the `synced_commit_mapping` table, so that later it is possible to attribute a commit sync map to a given commit sync.
This is part of a broader goal of adding support for sync map versioning. The broader goal is important, as it allows us to move faster (and troubleshoot better) when sync maps need changing.
Note that when commit is preserved across repos, we set `version_name` to `NULL`, as it makes no sense to attribute commit sync maps to those case.
Reviewed By: farnz
Differential Revision: D21765408
fbshipit-source-id: 11a77cc4d926e4a4d72322b51675cb78eabcedee
Summary:
The checkout step for EdenFS can take a few seconds if the distance is large or
if EdenFS has to fetch data. Reassure the user that something is happening
with a progress spinner.
Reviewed By: chadaustin
Differential Revision: D21684077
fbshipit-source-id: 43b8793f1a55ef6fb3bf78cabe3afcf2faed1d6c
Summary:
The motivation for the whole stack:
At the moment if mysql is down then Mononoke is down as well, both for writes
and for reads. However we can relatively easily improve the situation.
During hg update client sends getpack() requests to fetch files, and currently
for each file fetch we also fetch file's linknode. However hg client knows how
to deal with null linknodes [1], so when mysql is unavailable we can disable
filenode fetching completely and just return null linknodes. So the goal of this stack is to
add a knob (i.e. a tunable) that can turn things filenode fetches on and off, and make
sure the rest of the code deals nicely with this situation.
Now, about this diff. In order to force callers to deal with the fact that
filenodes might unavailable I suggest to add a special type of result, which (in
later diffs) will be returned by every filenodes methods.
This diff just introduces the FilenodeResult and convert BlobRepo filenode
methods to return it. The reason why I converted BlobRepo methods first
is to limit the scope of changes but at the same time show how the callers' code will look
like after FilenodeResult is introduced, and get people's thoughts of whether
it's reasonable or not.
Another important change I'd like to introduce in the next diffs is modifying FilenodesOnlyPublic
derived data to return success if filenodes knob is off. If we don't do that
then any attempt to derive filenodes might fail which in turn would lead to the
same problem we have right now - people won't be able to do hg update/hg
pull/etc if mysql is down.
[1] null linknodes might make some client side operation slower (e.g. hg rebase/log/blame),
so we should use it only in sev-like situations
Reviewed By: krallin
Differential Revision: D21787848
fbshipit-source-id: ad48d5556e995af09295fa43118ff8e3c2b0e53e
Summary:
We can skip the round trip to the server if we have nothing to ask or tell the
server.
Reviewed By: xavierd
Differential Revision: D21763025
fbshipit-source-id: 7f199c12ccaa0f66ce765dd976723e4002c5dd4e
Summary:
`flush()` takes a timeout, which is in milliseconds. However, that's not super
obvious, so that means whoever is using this has to re-discover it. Let's make
it explicit, and use a `Duration` argument instead.
(In fact, some places even got it wrong, notably common/rust/cli_usage)
Reviewed By: farnz
Differential Revision: D21783991
fbshipit-source-id: 6e3ac7c22b5c3297b41d5d61373ee077f12d5dd4
Summary: Added comments showing the expected JSON input format for each kind of EdenAPI request.
Reviewed By: xavierd
Differential Revision: D21782765
fbshipit-source-id: bf08dd4b36a33aca506eb0fa0341e40d0150d7cb
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary:
We encountered an issue where an infinitepush bundle did not contain
trees for some of the commits it contained. This happened when multiple users
were working on the same branch. Our previous heuristic decided to not send
trees if they weren't in the local store, but with us transitioning to Mononoke
(which enables draft commit data to be fetched ondemand) this invariant doesn't
always hold true.
Let's always upload trees for draft commits. Hopefully Mononoke can take over on
the server completely soon and we can get rid of all this special casing and do
normal discovery.
This is a revert of D7992502, but that was added because it was possible to pull
directly from svn which would result in public commits that didn't exist on the
server yet and therefore needed to send their trees. Since svn is gone, this is
probably safe to change back.
Reviewed By: quark-zju
Differential Revision: D21697921
fbshipit-source-id: c1abaa061207222490b146ee52f7949be6fe4b2a
Summary:
Under heavy parallelism or system load, our tests could trigger
short-ish timeouts and cause tests to flake. The stats test in
particular often failed in continuous integration. It looks like
opening a unix domain Thrift socket early and holding onto it can
cause it to sometimes hit ThriftServer's default idle timeout of 60
seconds, which results in the test failing with BrokenPipeError
(EPIPE).
Reviewed By: simpkins
Differential Revision: D21780023
fbshipit-source-id: 7e8838429475c2a322d836b9a497411199948cce
Summary:
This diff adds support for the `version_name` field, coming from the
`commitsyncmap` config, stored in the configerator.
Note: ATM, this field is optional in the thrift config, but once we get past
the initial deployment stage, I expect it to be present always. This is why
in `CommmitSyncConfig` I make it `String` (with a default value of `""`) rather
than `Option<String>`. The code, which will be writing this value into
`synced_commit_mapping` should not ever care whether it's present or not, since
every mapping should always have a `version_name` present.
Reviewed By: StanislavGlebik
Differential Revision: D21764641
fbshipit-source-id: 35a7f487acf0562b309fd6d1b6473e3b8024722d
Summary: This diff replaces the use of the oxidized typechecker types in `to_oxidized`, replacing them with oxidized_by_ref definitions. This unblocks deleting typechecker types and decls from the oxidized crate, since they are used only in this debugging-only path. It also reduces the amount of conversion necessary for `to_oxidized` (since we need not convert the oxidized_by_ref structures the typechecker already uses, like `Ty`).
Differential Revision: D21587518
fbshipit-source-id: 5e7ec2237d0059070653b45d15eb7c1259588ccb
Summary:
Rolling out LFSv2 on fbsource is a bit complex. Initially, the intent was to
roll it out via a server-only config and send pointers to say 5% of the
clients. The big snag in this is that LFS pointers are stored in Memcache, and
thus, a client who wasn't supposed to use LFS may end up reading a pointer from
Memcache, and issuing a request to the LFS server. Thanks to the way Memcache
works, this may lead to an avalance effect where everyone is fetching LFS
blobs, even if the server rollout is at a small percentage.
There are several solutions to this, the first obvious one would be to simply
not use Memcache for pointers, but that also means a forced connection to the
server, and a higher latency, ie: not a very desireable situation. An
alternative would be to have a proper capability exchange at connection time,
but that's unfortunately not feasible today due to the need to support the old
Mercurial server. Long term this is definitively the approach we want to go to,
and depending on the exchanged capabilities, we can even imagine using
different memcache keys automatically.
For now, we can hack this up by re-using the only free-form channel that the
client has to the server: clienttelemetry. Mononoke can then use the passed in
information to decide on whether to send LFS pointers (or not). This
unfortunately means that the rollout will be entirely client-side driven. To alleviate
the issue of Memcache keys being shared between clients wanting LFS pointers
and the ones not wanting them, a different Memcache key space will be used.
Reviewed By: StanislavGlebik
Differential Revision: D21765065
fbshipit-source-id: aebda3c567a827342b2fa96d374a06a23ea0ca34
Summary:
We're going to be changing the configs.* configs via chef as this rolls
out. Let's prevent them from being logged as mismatches.
Also updates the tree bfs config to match the current production value.
Reviewed By: quark-zju
Differential Revision: D21748693
fbshipit-source-id: b1d96f911d7ba7391546515179ac517fc62abe8b
Summary:
This list is going to grow to list all our rc files, which we don't
need in our public facing repo. Let's move this to a fb internal module.
Reviewed By: quark-zju
Differential Revision: D21747604
fbshipit-source-id: 8d72d8bf981c49b1e10260f5b0858729c1895da1
Summary:
Let's hide the repo list in the fb/ module since we don't really want
to expose the whole repo list in our public repo.
Reviewed By: sfilipco
Differential Revision: D21731419
fbshipit-source-id: 22ca20a3a80637c852e313f1390849aac1fecbf4
Summary:
Adds support for posix, windows, and windows_sandcastle rc files.
This should be low risk, as the dynamic config validator will remove any
incorrect configs before they are used.
Reviewed By: sfilipco
Differential Revision: D21731056
fbshipit-source-id: 0ab8c60906abc2e8d552509462a70a499a52cf86
Summary:
Adds support for most files from fb/staticfiles/etc/mercurial/tier-specific
It's missing the windows, posix, and tupperwar files, which I'll do in a later
diff.
This should be low risk, as the dynamic config validator will remove any
incorrect configs before they are used.
Reviewed By: quark-zju
Differential Revision: D21692664
fbshipit-source-id: 9b056640edc880bc683eb331921d08ac24212d59
Summary: When calling `eden start` from the CLI layer, make sure to redirect stdin in order to daemonize.
Reviewed By: chadaustin
Differential Revision: D21675707
fbshipit-source-id: 26085cc2ff7774e86f03872030c8885bc3c3b949
Summary:
Currently when you run commands on eden on mac that both fetch data for commit
and update to that commit there is a multithreading bug. This could be a new bug
or an old bug that was not causing problems before, but is showing up a lot more
recently.
See http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
for a nice explanation of this.
The root cause of this is still unknown, but this flag should bring back the
old behavior where we were not seeing this bug.
Reviewed By: fanzeyi
Differential Revision: D21744987
fbshipit-source-id: 85092e32212e83b16bc00dc0188f03b643c48eea
Summary:
Added an optional feature to scm-prompt.sh to show the dirty state of the working directory.
Users need to opt in by set the environmental variable `SHOW_DIRTY_STATE` to yes (or whatever nonempty string) in their shell rc file. The idea is that users who do not need/want this feature should see minimal performance hit of their prompt.
Unfortunately, in some (large) repos, calling `hg status` or `git diff` to figure out the dirty state can slow down showing the prompt quite a bit. So, in this case, user can opt out individual repo by setting `shell.showDirtyState` in .hg/hgrc or .git/config.
Reviewed By: ahornby
Differential Revision: D21493543
fbshipit-source-id: 13f7cf2dd2f8d00b58cf2a63dc2cbbf770028411
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
Wanting to have EdenFS run as a service on Windows cause a couple of issues
on Windows:
- Needing to log out after installing
- Forcing Sandcastle to use --foreground, which means the edenfs.log is
empty, making it impossible to debug anything on Sandcastle,
- Services can only be started by an elevated user, while the rest of EdenFS
doesn't have this requirement,
- Uses undocumented template services
The next diffs in the stack will attempt to solve all of these by starting
EdenFS as a scheduled task instead, which should solve all the above.
Reviewed By: fanzeyi
Differential Revision: D21732280
fbshipit-source-id: b959344da1e77819220d26695ff6634f13ac4e0d
Summary: Reverting as D20763778 is a suspect in causing thrift_server_overload T67609407
Reviewed By: StanislavGlebik
Differential Revision: D21762641
fbshipit-source-id: 545b448afc0954271a2e29d1d3b48fdb959e3d3d
Summary:
Change the signature of `CreateCommitContext::as_file` and its associated
functions so that content is `impl Into<String>`, rather than
`impl AsRef<str>`. The content will immediately be converted to a `String`
anyway, so we can avoid a string copy if the caller already has a string that
can be moved.
Reviewed By: krallin
Differential Revision: D21743429
fbshipit-source-id: d54914386439489fe4e47e37ff9a75c52b1a0443
Summary:
Add support for drawdag in Mononoke unit tests. Tests can use ASCII DAGs to construct
commit graphs, and can optionally customize the content of each commit.
Reviewed By: krallin
Differential Revision: D21743431
fbshipit-source-id: 9e6a52d1efe67ef4a5519ed7783f953fef7358f1
Summary:
The parser currently uses pattern destructuring for `RawInfinitepushParams`. This will break
if new fields are added to this structure. Instead, use field access like the other raw
params parsers.
Reviewed By: mitrandir77
Differential Revision: D21742558
fbshipit-source-id: 6bfbb080a5e5cdbb02519855472f4df80f9d7453
Summary:
It was used only once for testing push redirection. We no longer need it, so
I'd like to delete it to remove this old code and also to make it easier to
support ManualMove bookmarks.
Differential Revision: D21745630
fbshipit-source-id: 362952d95edb923cc4b60359321b563c1e4961de
Summary:
This adds flexibility. Now every type that implements DagAddHeads, including
NameDag, can import ASCII graphs.
Reviewed By: sfilipco
Differential Revision: D21626213
fbshipit-source-id: e258d88f97cbcc9aaf98d353a929803325185df7
Summary: The trait was about converting between Id and VertexName. Rename to clarify.
Reviewed By: sfilipco
Differential Revision: D21626195
fbshipit-source-id: 874ca4ca3a1467084a08c6d9aa321201974e1978
Summary: This allows other kinds of DAG to implement the operations.
Reviewed By: sfilipco
Differential Revision: D21626220
fbshipit-source-id: 896c5ccebb1672324d346dfca6bcac9b4d3b4929
Summary: This makes things a bit more flexible.
Reviewed By: sfilipco
Differential Revision: D21626194
fbshipit-source-id: f3ad486bcd5a6478d9e00f674d48f99504cded8c
Summary: This makes it possible for other types to implement the hex prefix lookup.
Reviewed By: sfilipco
Differential Revision: D21626218
fbshipit-source-id: 96e8b8c37e5aae2bd60658a238333b61902936d1
Summary: It will be used in the next change.
Reviewed By: sfilipco
Differential Revision: D21626207
fbshipit-source-id: bbef70ef9d4f9aaa2039a6bc15d296e88db7f8dc
Summary:
Types like IdDag are not really used. The use of the word "name" is sometimes
confusing in other context. Therefore export shorter names like Dag, MemDag,
Vertex, avoid "name" in NameDag, MemNameDag and NameSet. This makes external
code shorter and less ambiguous.
Reviewed By: sfilipco
Differential Revision: D21626212
fbshipit-source-id: 5bcf3cecfd38277149b41bf3ba9e6d4ef2a07b2b
Summary:
This decouples DagAlgorithm from the IdMap + IdDag backend, making it possible
to support other kinds of backends of DagAlgorithm (ex. a revlog backend).
Reviewed By: sfilipco
Differential Revision: D21626200
fbshipit-source-id: f53cc271a200062e9c02f739b6453e1d7de84e6d
Summary:
On Windows, if edenfsctl is started by pythonw, sys.stdout will be None,
check if it is before calling isatty on it.
Reviewed By: fanzeyi
Differential Revision: D21732282
fbshipit-source-id: f0a63094f2c53b026c388f4e5b58433be4397ba9
Summary:
The FormatMessage API isn't very well documented in adding \r\n at the end of
the generated message, and the bit of code that needs to be used to remove
these is not very clear either.
We won't get gratuitous empty lines in the log with this.
Reviewed By: chadaustin
Differential Revision: D21663502
fbshipit-source-id: 203ff5015da6f3cebf7eaee9a8f989db342d096a
Summary:
When we got rid of the delta logic, we also needed to get rid of some
unused functions.
Reviewed By: singhsrb
Differential Revision: D21725043
fbshipit-source-id: ac069e6b0468e2275f353a9970b8971b5a2cfa23
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: chadaustin
Differential Revision: D21712029
fbshipit-source-id: 3a332b4bf6fac6b3cf396d34e6d5ca4849181a6d
Summary: Useful for determining where an incremental building step left off.
Reviewed By: StanislavGlebik
Differential Revision: D21634698
fbshipit-source-id: e9b0473003c529d5c934754f1ece23df69c4be66
Summary:
This diff extends the integration test for the forward filler to execute queue operations, as well as the core business logic.
It also adds a test for the reverse filler, which does the same, but in a different difection.
Reviewed By: krallin
Differential Revision: D21628705
fbshipit-source-id: fb4ee0ecacc990d073425f3f37f794f74c057ea2
Summary:
This diff finally introduce the continuous reverse filler. Specifically, this adds a cli (and underlying wiring) to operate the filler logic in the `reversefillerqueue` table.
To achieve this:
- the filler class is turned into a base class with two subclasses for the forward and reverse fillers
- the main file is renamed from `forwardfiller.py` into `filler.py`, to better reflect the independence of direction.
Reviewed By: krallin
Differential Revision: D21628259
fbshipit-source-id: 5676a162a62f0dc6fe80e6300b72d30370fc80b4
Summary: Add devdb support to integration test runner so that one can more easily inspect mysql test data, also makes it easier to run tests with locally modified schema.
Reviewed By: HarveyHunt
Differential Revision: D21645234
fbshipit-source-id: ec75d70ef59f04548c7346a122298567dd09c264
Summary:
1. Add directory existence checking logic before checking "legacy_bind_mounts_dir"
2. Still print the disk consumption returned by the du command when it fails with an error invoking du -skxc
Reviewed By: chadaustin
Differential Revision: D21683421
fbshipit-source-id: 531b6b289e9ffa445ed691611c9cbf22f393e742
Summary:
At first glance people will assume that changesets are returned in the same
order that they were added in the database or that at least commits are
returned in a deterministic fashion. That didn't happen because the both
changeset ids and changeset entries were received without any order.
This diff updates the function to returns results in order they were added
to the database.
Reviewed By: krallin
Differential Revision: D21676663
fbshipit-source-id: 912e6bea0532796b1d8e44e47d832c0420d97bc1
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary:
This structure has similar functionality to the IdMap that is backed by SQL.
It is probably going to be useful for caching in the case of batch operations.
Reviewed By: quark-zju
Differential Revision: D21601820
fbshipit-source-id: 9c3ebc3e9dc92a59ce0908fc241bd2b97da88dca
Summary:
`Dag::build_all_graph` will load the whole graph for a given repository and
construct the segmented changelog from it.
Reviewed By: StanislavGlebik
Differential Revision: D21538029
fbshipit-source-id: b4ba846bb2870ba73257bed6128b8e198a0aab3e
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary:
xavierd and I realized that memcache is not actually enabled in EdenFS as `ContentStoreBuilder` only sets it when there is are remote store. We believe turning on memcache store can help us improve some performance for importing blobs (80% hit rate).
This diff adds a noop remote store that does nothing to trick `ContentStoreBuilder` to include the memcache store. We decided to go this way instead of modifying `ContentStoreBuilder` is because EdenFS is going to have a real remotestore very soon, and changes to `ContentStoreBuilder` may have some impact on how Mercurial uses it.
Reviewed By: xavierd
Differential Revision: D21621234
fbshipit-source-id: 5502e001ab498134b6a927d83be823261e70e9e1
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary:
There was a bug where if the dynamicconfig chose a value that matched
one of the ported rc files, but it did not match the final value of that config
(from a not-yet-ported rc file), it would chose the dynamicconfig value instead
of the original final value. Let's drop the dynamicconfig value in this case as
well.
Reviewed By: quark-zju
Differential Revision: D21674469
fbshipit-source-id: efcd36e9602e16210999ec8c88e86b1d7ee355e4
Summary: I've updated most messages to suggest a graceful restart, but I overlooked this instance and we got a user question about this. So lets update this message here as well.
Reviewed By: chadaustin
Differential Revision: D21681035
fbshipit-source-id: 32bf5e75b9ad1d98bcd70427101aabeee0be5046
Summary: I didn't update it on my last diff.
Reviewed By: xavierd
Differential Revision: D21687344
fbshipit-source-id: 01c9662b9fd8c4670c2af34a999c69e6b93ed7f7
Summary:
When markbt converted histedit to be compatible with new style mutation
he missed a few spots.
This diffs adds a large test checking the that all the scenarios that were
supported with obsmarkers are supported with mutation now.
Reviewed By: markbt
Differential Revision: D21665484
fbshipit-source-id: a8a289910adb068a382e32f9d106dedc1573d413
Summary:
The histedit goes down to the latest branching point by default which is not
always the right choice for `amend --to`
Reviewed By: markbt
Differential Revision: D21665485
fbshipit-source-id: d981d2216b5384f61c0082684eafa1e84eaeada8
Summary:
This test addtion showcases a bug in `amend --to` when
a side branch is present
Reviewed By: markbt
Differential Revision: D21665486
fbshipit-source-id: 8c21e7a8cccfc6c313d3b7aa09be8309a662b9cc
Summary:
Change from CHashMap to DashMap for the walk state tracking.
DashMap is using slightly less memory in testing, and is slightly quicker in walk rate (number of graph nodes processed per second ).
Reviewed By: HarveyHunt
Differential Revision: D21662210
fbshipit-source-id: ea0601df17e0e596fd59b67d9d01d0dc4e90799b
Summary:
CacheBlob logs hit and miss stastics to Scuba. Let's add the same for
ODS.
Reviewed By: krallin
Differential Revision: D21640922
fbshipit-source-id: 8f7d17f048bf53bdc6cd8bda0384a51cae7b6a30
Summary:
CountedBlobstore is a Blobstore that wraps another blobstore in order
to report metrics about it, such as the number of gets or errors. It's commonly
used to wrap a CacheBlob blobstore, which itself is a caching wrapper over an
inner blobstore.
CountedBlobstore exposes metrics that are supposed to track the number of hits
or misses when fetching blobs. However, these metrics don't make sense as the
CountedBlobstore has no view into cache activity. These metrics actually report
the number of requests and the number of missing blobs rather than hits and
misses.
Remove these misleading counters.
Reviewed By: krallin
Differential Revision: D21640923
fbshipit-source-id: 07b9fc9864c70991415c2b84f35d631b702c17d1
Summary:
now fastlog will be able to show a history of file like that:
```
Added
|
Deleted
|
Added
```
We do it by using deleted file manifest to find where a file was deleted. We do
it in case we find a unode which doesn't have a parent.
Differential Revision: D21664248
fbshipit-source-id: 8e80fccc87361e66e4e00ae2da3671ed8327f1f8
Summary:
... or rather make this special case a bit smaller.
In fastlog we have a `prefetch` which was already returned to the caller,
however its parents hasn't been processed yet. Previously we had two separate
places where these parents were processed, now we have only one.
This will make it easier to add support for history across deletions in the
next diff.
Differential Revision: D21664246
fbshipit-source-id: a5a109fafaeec28a0e7a36ffa44545aebc032b00
Summary: I had hardcoded the path separator, which is wrong on Windows.
Reviewed By: singhsrb
Differential Revision: D21679423
fbshipit-source-id: 309c84b0698355654baee1dae7f201310e695c49
Summary:
Just top-level mononoke and cache_warmup as a more or less random
selection.
Reviewed By: StanislavGlebik
Differential Revision: D21648852
fbshipit-source-id: 0e5137da9a7742377112e085e9d29182abdfbf41
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary:
If either the config, or the environment variable is empty, the URL parsing
would raise an error, while the intent is to not have proxy. Let's handle it
properly.
Reviewed By: DurhamG
Differential Revision: D21669922
fbshipit-source-id: 84c8d09242ac8d5571f7592d874f4954692110f5
Summary:
Looks like echo prints a \r\n which messes up the output. Let's use
printf to force it to use \n
Reviewed By: xavierd
Differential Revision: D21670125
fbshipit-source-id: f991c9c9fb55720bb68a3f71e8ddd3b4f16b5375
Summary:
At clone time we apply dynamic configs in memory, instead of loading
them from disk. The validation logic operated on the config value's location
field, which isn't set for data that comes from in memory. So we need to update
the validation logic to also consider the value source if location is not set.
Reviewed By: quark-zju
Differential Revision: D21664205
fbshipit-source-id: 8460c58c6d654780048de51ada8178c70ff0a9e6
Summary:
We kick off a background process every 15 minutes or so to compute the
new dynamicconfig. If the config hasn't changed, we should just touch the file
instead of rewriting the whole thing.
Reviewed By: quark-zju
Differential Revision: D21653098
fbshipit-source-id: 9d53d2a636cff082cd048994255cc809ce1b0221
Summary:
If we release a new version of Mercurial, we want to ensure that it's
builtin configs are used immediately. To do so, let's write a version number
into the generated config file, and if the version number doesn't match, we
force a synchronous regeneration of the config file.
For now, if regeneration fails, we just log it. In the future we'll probably
throw an exception and block the user since we want to ensure people are running
with modern configuration.
Reviewed By: quark-zju
Differential Revision: D21651317
fbshipit-source-id: 3edbaf6777f4ca2363d8617fad03c21204b468a2
Summary:
Now that regular users can create symlinks, a handful of tests were failing
due to the symlink call no longer raising an exception. For now, let's always
consider that symlinks aren't supported on Windows.
Reviewed By: fanzeyi
Differential Revision: D21664213
fbshipit-source-id: c55a99d1cb92e68b9861701b5517b1d5db2d40c6
Summary:
Previously we created `visited` and immediately drop it.
Let's instead create it one level up. I don't think it'd matter in practice,
but it should be cleaner
Reviewed By: HarveyHunt
Differential Revision: D21662456
fbshipit-source-id: 36823309ea09448c5be10c2655ea0f317649f290
Summary:
Follow up from D21596758 - current logic of pushrebasing a merge is very
complicated. To prevent repo corruptions let's do a very simple validation -
generate hg changeset from a rebased bonsai changeset.
Given that generating hg changeset is an expensive operation let's do it only
after the first rebase attempt - otherwise we might risk constantly losing the
pushrebase race.
Differential Revision: D21659187
fbshipit-source-id: f43a855cf0fbdbd11a40d3ec38283af344cde5e6
Summary: This hint has served its purpose. Remove it.
Reviewed By: farnz
Differential Revision: D21658927
fbshipit-source-id: 0395fa2cb898732b2c64154104c703a428f62864
Summary: This is to bring it into sync with the `forwardfillerqueue` types.
Reviewed By: markbt
Differential Revision: D21660012
fbshipit-source-id: 5148023478c175cd49707d88251701a08fcbe0ce
Summary:
Mononoke server is the only binary using plain `clap::App`, whilst all
other services use `MononokeApp`. The `MononokeApp` variant configures some
default arguments and provides consistent argument naming between binaries.
Update the server to use `MononokeApp` and add some logic to allow passing
either `--config_path` or `--mononoke-config-path` until we remove all uses of
`--config-path`.
Reviewed By: StanislavGlebik
Differential Revision: D21642461
fbshipit-source-id: f42e0b6625f3979ced0920db269bdb4528f99e49
Summary:
When starting up with many repos, it can take a while before the server is ready
to accept connections, and there is no log to indicate when that is the case.
Add one.
Reviewed By: farnz
Differential Revision: D21641189
fbshipit-source-id: 057ca06c1b1725799a8445841b6eea89bba783b0
Summary:
Rather than logging `Refreshing tunables` all the time, log only when they
change, and include the values that they are changing from and to.
Reviewed By: farnz
Differential Revision: D21641191
fbshipit-source-id: b0b5edd687e2e41074ce1c8be13fd6918dff924d
Summary:
There are a few paths through `resolve` that return early with a failure, and thus never record what happened.
Make a record the moment we enter `resolve` - then, we can use `count` type ODS charts to determine the failure rate deterministically, and alarm if the failure rate is too high
Reviewed By: ahornby
Differential Revision: D21647575
fbshipit-source-id: 667787ec000a8cd8e715563df10dbb84832fefa1
Summary:
During clone the hgrc.dynamic file doesn't exist and doesn't even have
a place for us to generate it to. Let's instead generate and apply the config in
memory.
In the future, if generate fetches data from the network, this will mean clone
would depend on the network, since if generate fails the clone would fail. In
some situations this is desirable, since users shouldn't be cloning without our
approved configs, but if it causes problems we could probably tweak generate to
support an offline mode.
Reviewed By: quark-zju
Differential Revision: D21643086
fbshipit-source-id: d9a758207738d5983213d95725061517e0aa17db
Summary:
The Rust version command does not support `-T` and does not read
`__version__.py`. Remove tests about them. As we're here, also remove tests
about 3rd-party extension version checking as we bundle all extensions.
Reviewed By: xavierd
Differential Revision: D21648591
fbshipit-source-id: 97c9e2ff3d224ff12b34801a1af07532d9e9bff5
Summary:
Implements a "debugdetectissues" command, which runs a series of checks on the repository meant to detect potential issues, logging the data to Scuba so that we can determine how common certain issues are, and under what conditions they appear.
Future extensions of this change may involve merging the functionality into hg doctor, and setting the command to run automatically in the background on some interval.
Reviewed By: DurhamG
Differential Revision: D21558170
fbshipit-source-id: a878ae1804d5f11c83a574e0dc3c802b564d2ced
Summary:
This script deals with obtaining the right privilege to start the EdenFS
service. This effectively enables `edenfsctl start`.
Reviewed By: wez
Differential Revision: D21585739
fbshipit-source-id: 7b835434d865fa4c4c8473e13665ae669fd86108
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary:
There is no need to upload content referred by a public commit.
This affects cases like `debugstrip` (ex. for testing `pull` performance with a
lagged commit graph). Without this change, the uploadrevs code path scans
stripped public commits without efficient tree prefetching, which results in
1-by-1 tree fetches and is unusably slow.
Reviewed By: xavierd
Differential Revision: D21630096
fbshipit-source-id: 385edf76cb4eb913b2d64422910cdb46b603e6c0
Summary:
With `HGPLAINEXCEPT=color`, colors should still be enabled if the terminal is
capable of supporting them and output is to the terminal.
Currently this doesn't work if `--color=auto` (the default), as
`color._modesetup` uses `ui.formatted` to check if the output is a terminal,
and thus has colors available, but this is `False` for all `HGPLAIN` modes.
Instead, add a new method to `ui` that checks whether `fout` is a terminal, and
use that for color autodetection.
This function also allows us to add `HGPLAINEXCEPT=pager` as we can use
that in the same way.
Reviewed By: farnz
Differential Revision: D21617170
fbshipit-source-id: 7ee4eaa8963f3d6eb7ed8044a678a4804b9a98f0
Summary:
Setting the fake pager in the pager test doesn't work on Windows.
For some reason, the `pager.pager=C:/path/to/python.exe C:/path/to/test/fakepager.py` triggers
some kind of PATH translation code that results in `setconfig.py` receiving
`pager.pager=C;C:\\path\\to\\python.exe C;C:\\path\\to\\test\\fakepager.py` as its
argument. This is clearly invalid, so the test fails with messages like
`'C' is not recognized as an internal or external command`.
Workaround this for now by setting the pager by appending directly to HGRCPATH.
Reviewed By: ikostia
Differential Revision: D21638507
fbshipit-source-id: e8a6b0c281030b23302116a79ef3ba754d37f601
Summary:
D21573455 accidentally removed the scuba sampling for scrub_get
operations. Add this back in.
Reviewed By: StanislavGlebik
Differential Revision: D21638972
fbshipit-source-id: eee66dbce161de69246f4da0a15dc2cf00e1ba01
Summary:
See bottom diff in the stack for the motivation. Though you can probably guess
the motivation :)
Reviewed By: farnz
Differential Revision: D21623154
fbshipit-source-id: a0940d766a67080ddcb346c2e3313eb08699edad
Summary: First diff in the stack that removes getfiles since it's no longer needed.
Reviewed By: farnz
Differential Revision: D21623156
fbshipit-source-id: 44f310ec4e4f34845cc5bf1738f1a8ece14e6694
Summary:
Before this change, pull_shallow only calls setuprepo if the remotefilelog
requirement is not in repo. With D21011401, the remotefilelog requirement
will be added by clone.py and pull_shallow can skip calling setuprepo, causing
the pull code paths to write file logs (and fail).
Change the pull_shallow to always call `setuprepo` to solve the issue.
The final fix is probably moving more remotefilelog related clone logic to core.
Right now I just did the minimal change to fix things.
Reviewed By: sfilipco
Differential Revision: D21632429
fbshipit-source-id: 17775ac0df18cda10247419b40f9c27436b22606
Summary:
We didn't have a high level overview of the types and traits and their
purposes, add that.
Reviewed By: DurhamG
Differential Revision: D21599759
fbshipit-source-id: 18e8d23ed00134f9d662778eddaee4a9451f7c2c
Summary: EdenFS doesnt daemonize correctly due to the privhelper not closing fd 0 (see http://www.faqs.org/faqs/unix-faq/programmer/faq/). This redirects stdin to /dev/null/ in order to do so.
Reviewed By: xavierd
Differential Revision: D21602545
fbshipit-source-id: 0aeb589efbf214ef22c0db039fbb6a436a71e360
Summary:
This updates a placeholder in the authentication process for requests
in the rust thrift services to allow authentication with CATs. This change
allows for passing in a configurable server identity to be used as the
verifier identity here.
Additionally this sets the verifer identity to be "scm_service_identity"
for the mononoke scs server (the service identity for this service).
This is needed to use the mononoke scs server from corp.
Reviewed By: markbt
Differential Revision: D21560204
fbshipit-source-id: 02e41718307fddab6f0ac2101306bb7f4a59cf7e
Summary: Otherwise the version information is only available in Python.
Reviewed By: DurhamG
Differential Revision: D19803762
fbshipit-source-id: 044c5da86efc8c657d0c422a2b1947086444895e
Summary:
Currently, we have a gap between updating and subsequently reading the
value of `write_order`. If another blobstore's put operation completes before
we have called `record_put_stats` then we may have an incorrect `write_order`
value.
Whilst this is only a minor issue, let's fix it anyway. :-)
Reviewed By: krallin
Differential Revision: D21619669
fbshipit-source-id: 1b8bacbcb4c195e6765ebdfaa68425f286f88c06
Summary:
After other optimizations the CHashMap version of visit_count was showing as a hot spot.
Given the number of possible NodeTypes is small we can store the visit_count in array indexed by ordinal instead.
Reviewed By: farnz
Differential Revision: D21618518
fbshipit-source-id: 84978778034df11a9a48452adf9269db2dc17145
Summary:
Currently we record them only during pushrebase. Let's record during push as
well.
To simplify things a little bit let's allow only a very simple push case:
1) Single bookmark.
2) All pushed commits should be reachable by this bookmark.
Reviewed By: krallin
Differential Revision: D21451337
fbshipit-source-id: bf2f1e6025ac116fb8096824b7c4c6440d073874
