Summary:
DefferedDerivedMapping was added so that we can make deriving stack of commits faster - it does it by postponing updating
derived data mapping (e.g. writing to a blobstore) until the whole stack is derived.
While it probably makes derivation a bit faster, we now think it's better to remove it. A few reasons:
1) It's confusing to understand and it already caused us ubns before
2) It's increases write amplification - because we release the lease before we wrote to a blobstore, writers will try to rederive the same commit a few times. That has caused us a ubn today
Reviewed By: farnz
Differential Revision: D20113854
fbshipit-source-id: 169e05febcd382334bf4da209a20aace0b7c2333
Summary:
See D21765065 for more context. TL;DR is that we want to control
lfs rollout from client side to make sure we don't put lfs pointers in the
shared memcache
Reviewed By: xavierd
Differential Revision: D21822159
fbshipit-source-id: daea6078d95eb4e9c040d353a20bcdf1b6ae07b1
Summary: [Folly] Cut the `FOR_EACH_ENUMERATE` macro, which may be replaced by a combination of range-for, `ranges::view::enumerate`, and structured bindings.
Reviewed By: markisaa
Differential Revision: D21813019
fbshipit-source-id: fc9ac09a4e2f72f1433d0a518f03d5cd69a59c55
Summary:
Adds a remotefilelog.write-hgcache-to-indexedlog config which directs
all hgcache writes to the indexedlog.
This change also removes remotefilelog.indexedlogdatastore as it's on by default
now.
Reviewed By: xavierd
Differential Revision: D21772132
fbshipit-source-id: a71e188df2958fb4f8c4776da23ba87deccb3b79
Summary:
`fixcorrupt` accesses changelog in a way that is going to cause trouble with
the next change. Since `hg doctor` can do what `fixcorrupt` does in production
setup. Let's sunset fixcorrupt.
Reviewed By: singhsrb
Differential Revision: D21780575
fbshipit-source-id: 5efec3f066a2929018ccc68b1f52d10a76e59637
Summary:
Change `NodeRevMap`'s changelog type from `[u8]` to `[RevlogEntry]`.
This makes it consistent with `RevlogIndex`.
Reviewed By: sfilipco
Differential Revision: D21626203
fbshipit-source-id: 7457f48ccd7b3489264684a5db21d21e9eb7a937
Summary:
NodeRevMap helps converting from a commit hash to a rev number. It's similar to
IdMap in the dag crate, but was designed for the revlog.
Move NodeRevMap to revlogindex so it becomes easier to implement the IdConvert
trait required by the dag crate.
Reviewed By: sfilipco
Differential Revision: D21626211
fbshipit-source-id: 14996f1234231b507efb5186ec30f84df5aaad10
Summary:
The idea is that the pure Rust revlogindex crate can implement the DagAlgorithm
interface so we will have a consistent interface in the code base that works
for both the existing storage (revlog) and the new segmented changelog.
The other way to do this is to implement the `bindings.dag.namedag` interface
in pure Python for the revlog-based DAG, or supporting quite different
interfaces (ex. revlog DAG and the Rust segmented changelog DAG) in the code
base. At present, I think implementing the Rust DAG traits for revlog is the
most appealing, partially because we already have some key algorithms
implemented (ex. prefix lookup, common ancestors, etc).
Reviewed By: sfilipco
Differential Revision: D21626197
fbshipit-source-id: 733b1af1bcd5fc0784764fc7103412988894d43b
Summary: This cache didn't improve performance, and is a reasonable amount of extra code. Just rip it out, and improve higher caching layers if needed.
Reviewed By: StanislavGlebik
Differential Revision: D21818412
fbshipit-source-id: 3ca58bff180c6da91d451754b67b23e61b736059
Summary:
We can kill a single shard's replication with weight of puts; by tracking lag once a second, we can slow down to a point where no shard overloads.
This is insufficient by itself, as it slows all shards down equally (whereas we only want to slow down the laggy shard), but this now avoids high replication lag
Reviewed By: StanislavGlebik
Differential Revision: D21720833
fbshipit-source-id: a819f641206641e80f8edde92006fb08cdcf36a9
Summary:
Previously the return type was String which, in Python 2, could turn into bytes or
unicode depending on the contents of the string. We always want bytes in Python
2, so let's use the Str type instead.
Reviewed By: quark-zju
Differential Revision: D21794189
fbshipit-source-id: 6493fbacab354a78476f522fc3c41b7336dbbdb1
Summary:
Out `CommitSyncConfig` struct now contains a `version_name` field, which is intended to be used as an identifier of an individual version of the `commitsyncmap` in use. We want to record this value in the `synced_commit_mapping` table, so that later it is possible to attribute a commit sync map to a given commit sync.
This is part of a broader goal of adding support for sync map versioning. The broader goal is important, as it allows us to move faster (and troubleshoot better) when sync maps need changing.
Note that when commit is preserved across repos, we set `version_name` to `NULL`, as it makes no sense to attribute commit sync maps to those case.
Reviewed By: farnz
Differential Revision: D21765408
fbshipit-source-id: 11a77cc4d926e4a4d72322b51675cb78eabcedee
Summary:
The checkout step for EdenFS can take a few seconds if the distance is large or
if EdenFS has to fetch data. Reassure the user that something is happening
with a progress spinner.
Reviewed By: chadaustin
Differential Revision: D21684077
fbshipit-source-id: 43b8793f1a55ef6fb3bf78cabe3afcf2faed1d6c
Summary:
The motivation for the whole stack:
At the moment if mysql is down then Mononoke is down as well, both for writes
and for reads. However we can relatively easily improve the situation.
During hg update client sends getpack() requests to fetch files, and currently
for each file fetch we also fetch file's linknode. However hg client knows how
to deal with null linknodes [1], so when mysql is unavailable we can disable
filenode fetching completely and just return null linknodes. So the goal of this stack is to
add a knob (i.e. a tunable) that can turn things filenode fetches on and off, and make
sure the rest of the code deals nicely with this situation.
Now, about this diff. In order to force callers to deal with the fact that
filenodes might unavailable I suggest to add a special type of result, which (in
later diffs) will be returned by every filenodes methods.
This diff just introduces the FilenodeResult and convert BlobRepo filenode
methods to return it. The reason why I converted BlobRepo methods first
is to limit the scope of changes but at the same time show how the callers' code will look
like after FilenodeResult is introduced, and get people's thoughts of whether
it's reasonable or not.
Another important change I'd like to introduce in the next diffs is modifying FilenodesOnlyPublic
derived data to return success if filenodes knob is off. If we don't do that
then any attempt to derive filenodes might fail which in turn would lead to the
same problem we have right now - people won't be able to do hg update/hg
pull/etc if mysql is down.
[1] null linknodes might make some client side operation slower (e.g. hg rebase/log/blame),
so we should use it only in sev-like situations
Reviewed By: krallin
Differential Revision: D21787848
fbshipit-source-id: ad48d5556e995af09295fa43118ff8e3c2b0e53e
Summary:
We can skip the round trip to the server if we have nothing to ask or tell the
server.
Reviewed By: xavierd
Differential Revision: D21763025
fbshipit-source-id: 7f199c12ccaa0f66ce765dd976723e4002c5dd4e
Summary:
`flush()` takes a timeout, which is in milliseconds. However, that's not super
obvious, so that means whoever is using this has to re-discover it. Let's make
it explicit, and use a `Duration` argument instead.
(In fact, some places even got it wrong, notably common/rust/cli_usage)
Reviewed By: farnz
Differential Revision: D21783991
fbshipit-source-id: 6e3ac7c22b5c3297b41d5d61373ee077f12d5dd4
Summary: Added comments showing the expected JSON input format for each kind of EdenAPI request.
Reviewed By: xavierd
Differential Revision: D21782765
fbshipit-source-id: bf08dd4b36a33aca506eb0fa0341e40d0150d7cb
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary:
We encountered an issue where an infinitepush bundle did not contain
trees for some of the commits it contained. This happened when multiple users
were working on the same branch. Our previous heuristic decided to not send
trees if they weren't in the local store, but with us transitioning to Mononoke
(which enables draft commit data to be fetched ondemand) this invariant doesn't
always hold true.
Let's always upload trees for draft commits. Hopefully Mononoke can take over on
the server completely soon and we can get rid of all this special casing and do
normal discovery.
This is a revert of D7992502, but that was added because it was possible to pull
directly from svn which would result in public commits that didn't exist on the
server yet and therefore needed to send their trees. Since svn is gone, this is
probably safe to change back.
Reviewed By: quark-zju
Differential Revision: D21697921
fbshipit-source-id: c1abaa061207222490b146ee52f7949be6fe4b2a
Summary:
Under heavy parallelism or system load, our tests could trigger
short-ish timeouts and cause tests to flake. The stats test in
particular often failed in continuous integration. It looks like
opening a unix domain Thrift socket early and holding onto it can
cause it to sometimes hit ThriftServer's default idle timeout of 60
seconds, which results in the test failing with BrokenPipeError
(EPIPE).
Reviewed By: simpkins
Differential Revision: D21780023
fbshipit-source-id: 7e8838429475c2a322d836b9a497411199948cce
Summary:
This diff adds support for the `version_name` field, coming from the
`commitsyncmap` config, stored in the configerator.
Note: ATM, this field is optional in the thrift config, but once we get past
the initial deployment stage, I expect it to be present always. This is why
in `CommmitSyncConfig` I make it `String` (with a default value of `""`) rather
than `Option<String>`. The code, which will be writing this value into
`synced_commit_mapping` should not ever care whether it's present or not, since
every mapping should always have a `version_name` present.
Reviewed By: StanislavGlebik
Differential Revision: D21764641
fbshipit-source-id: 35a7f487acf0562b309fd6d1b6473e3b8024722d
Summary: This diff replaces the use of the oxidized typechecker types in `to_oxidized`, replacing them with oxidized_by_ref definitions. This unblocks deleting typechecker types and decls from the oxidized crate, since they are used only in this debugging-only path. It also reduces the amount of conversion necessary for `to_oxidized` (since we need not convert the oxidized_by_ref structures the typechecker already uses, like `Ty`).
Differential Revision: D21587518
fbshipit-source-id: 5e7ec2237d0059070653b45d15eb7c1259588ccb
Summary:
Rolling out LFSv2 on fbsource is a bit complex. Initially, the intent was to
roll it out via a server-only config and send pointers to say 5% of the
clients. The big snag in this is that LFS pointers are stored in Memcache, and
thus, a client who wasn't supposed to use LFS may end up reading a pointer from
Memcache, and issuing a request to the LFS server. Thanks to the way Memcache
works, this may lead to an avalance effect where everyone is fetching LFS
blobs, even if the server rollout is at a small percentage.
There are several solutions to this, the first obvious one would be to simply
not use Memcache for pointers, but that also means a forced connection to the
server, and a higher latency, ie: not a very desireable situation. An
alternative would be to have a proper capability exchange at connection time,
but that's unfortunately not feasible today due to the need to support the old
Mercurial server. Long term this is definitively the approach we want to go to,
and depending on the exchanged capabilities, we can even imagine using
different memcache keys automatically.
For now, we can hack this up by re-using the only free-form channel that the
client has to the server: clienttelemetry. Mononoke can then use the passed in
information to decide on whether to send LFS pointers (or not). This
unfortunately means that the rollout will be entirely client-side driven. To alleviate
the issue of Memcache keys being shared between clients wanting LFS pointers
and the ones not wanting them, a different Memcache key space will be used.
Reviewed By: StanislavGlebik
Differential Revision: D21765065
fbshipit-source-id: aebda3c567a827342b2fa96d374a06a23ea0ca34
Summary:
We're going to be changing the configs.* configs via chef as this rolls
out. Let's prevent them from being logged as mismatches.
Also updates the tree bfs config to match the current production value.
Reviewed By: quark-zju
Differential Revision: D21748693
fbshipit-source-id: b1d96f911d7ba7391546515179ac517fc62abe8b
Summary:
This list is going to grow to list all our rc files, which we don't
need in our public facing repo. Let's move this to a fb internal module.
Reviewed By: quark-zju
Differential Revision: D21747604
fbshipit-source-id: 8d72d8bf981c49b1e10260f5b0858729c1895da1
Summary:
Let's hide the repo list in the fb/ module since we don't really want
to expose the whole repo list in our public repo.
Reviewed By: sfilipco
Differential Revision: D21731419
fbshipit-source-id: 22ca20a3a80637c852e313f1390849aac1fecbf4
Summary:
Adds support for posix, windows, and windows_sandcastle rc files.
This should be low risk, as the dynamic config validator will remove any
incorrect configs before they are used.
Reviewed By: sfilipco
Differential Revision: D21731056
fbshipit-source-id: 0ab8c60906abc2e8d552509462a70a499a52cf86
Summary:
Adds support for most files from fb/staticfiles/etc/mercurial/tier-specific
It's missing the windows, posix, and tupperwar files, which I'll do in a later
diff.
This should be low risk, as the dynamic config validator will remove any
incorrect configs before they are used.
Reviewed By: quark-zju
Differential Revision: D21692664
fbshipit-source-id: 9b056640edc880bc683eb331921d08ac24212d59
Summary: When calling `eden start` from the CLI layer, make sure to redirect stdin in order to daemonize.
Reviewed By: chadaustin
Differential Revision: D21675707
fbshipit-source-id: 26085cc2ff7774e86f03872030c8885bc3c3b949
Summary:
Currently when you run commands on eden on mac that both fetch data for commit
and update to that commit there is a multithreading bug. This could be a new bug
or an old bug that was not causing problems before, but is showing up a lot more
recently.
See http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
for a nice explanation of this.
The root cause of this is still unknown, but this flag should bring back the
old behavior where we were not seeing this bug.
Reviewed By: fanzeyi
Differential Revision: D21744987
fbshipit-source-id: 85092e32212e83b16bc00dc0188f03b643c48eea
Summary:
Added an optional feature to scm-prompt.sh to show the dirty state of the working directory.
Users need to opt in by set the environmental variable `SHOW_DIRTY_STATE` to yes (or whatever nonempty string) in their shell rc file. The idea is that users who do not need/want this feature should see minimal performance hit of their prompt.
Unfortunately, in some (large) repos, calling `hg status` or `git diff` to figure out the dirty state can slow down showing the prompt quite a bit. So, in this case, user can opt out individual repo by setting `shell.showDirtyState` in .hg/hgrc or .git/config.
Reviewed By: ahornby
Differential Revision: D21493543
fbshipit-source-id: 13f7cf2dd2f8d00b58cf2a63dc2cbbf770028411
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
Wanting to have EdenFS run as a service on Windows cause a couple of issues
on Windows:
- Needing to log out after installing
- Forcing Sandcastle to use --foreground, which means the edenfs.log is
empty, making it impossible to debug anything on Sandcastle,
- Services can only be started by an elevated user, while the rest of EdenFS
doesn't have this requirement,
- Uses undocumented template services
The next diffs in the stack will attempt to solve all of these by starting
EdenFS as a scheduled task instead, which should solve all the above.
Reviewed By: fanzeyi
Differential Revision: D21732280
fbshipit-source-id: b959344da1e77819220d26695ff6634f13ac4e0d
