Summary: Those conflicts can be resolved in Python using textual 3-way merge
Reviewed By: DurhamG
Differential Revision: D27752770
fbshipit-source-id: 816a601112ee2e747d780f8b17473049df46b469
Summary:
This diff modifies rebase flow(based on config) and attempts to create commit wihthout using merge.py:update
This currently passes some test cases, but not all.
This implementaiton currently does not attempt to resolve conflicts and fallbacks to on-disk merge if they are encountered.
This fails some test cases, because they expect some trivial conflicts to be resolved by in-memory merge.
There are also certain rebase flags that currently are not handled
Reviewed By: DurhamG
Differential Revision: D27639394
fbshipit-source-id: d8f71e955930e3a8a64d7d95a0cf184d9b4ccadc
Summary: This diff exposes manifestbuilder that can be used to construct memctx from Python
Reviewed By: DurhamG
Differential Revision: D27639395
fbshipit-source-id: ed047d3d7533f9d2bc592a5d948dc01e429692a7
Summary:
This diff makes MySQL FFI client the default option for a MySQL connection. It means that if no arguments provided, the MySQL FFI client is used. `--use-mysql-client` option is still accepted, as it is used in the configs, and will be removed a bit later.
I also removed raw connections as a way to connect to MySQL from Mononoke, as it is no longer used. Although I had to keep some `sql_ext` API for now because other projects rely on it.
(I talked to the teams and they are willing to switch to the new client as well. I'm helping where it's possible to replace these raw xdb conns.)
Reviewed By: krallin
Differential Revision: D27925435
fbshipit-source-id: 4f08eef07df676a4e6be58b6e351be3e3d3e8ab7
Summary:
Right now, we can't have defaults in our tunables, because some tests clobber
them. Let's start updating tunables instead of replacing them.
NOTE: I was planning to use this in my next diff, but I ended up not needing
it, that said, it feels like a generally positive improvements, so I figured
I'd keep it.
Reviewed By: StanislavGlebik
Differential Revision: D27915402
fbshipit-source-id: feeb868d99565a375e4e9352520f05493be94a63
Summary:
This updates the bookmarks cache TTL to be something we configure using
tunables instead of repo configs. There's a few goals here:
- Less us tune the pressure we put on SQL via a tunable
- Letting us experiment more easily with disabling this cache and tuning the
WBC poll interval.
Right now, this defaults to what we had set up in all our repo configs, which
is 2000ms.
Reviewed By: farnz
Differential Revision: D27915403
fbshipit-source-id: 4361d38939e5b2a0fe37442ed8c1dd0611af5098
Summary:
One of our plans for this half is to replace the warm bookmarks cache with a
service, and we suspect this will effectively eliminate bookmarks queries from
our hosts, because we think they all come from the WBC.
But, before we invest our time into this, let's make sure that this assumption
is actually correct, by tracking who's querying bookmarks.
Reviewed By: StanislavGlebik
Differential Revision: D27938407
fbshipit-source-id: d9a9298e7409c9518a4b9bf8ac0a6cef53750473
Summary:
I'd like to be able to track the proportion of traffic coming to bookmarks from
warm bookmarks cache vs. from elsewhere. We don't have a great abstraction to
pass this via the CoreContext at this time, but the SessionClass seems like a
pretty good fit.
Indeed, since it's always available in the CoreContext, and can be freely
mutated without having to rebuild the whole session. Besides, it aligns pretty
well with the existing use cases we have for SessionClass, which is to give you
different level of service depending on who you are.
Reviewed By: StanislavGlebik
Differential Revision: D27938413
fbshipit-source-id: a9dcc5a10c8d1459ee9586324a727c668e2e4e40
Summary:
phases calculation could be expensive on the server and it should be a perf win to disable it if not needed
It shouldn't be needed if narrow heads are enabled
Reviewed By: quark-zju
Differential Revision: D27908691
fbshipit-source-id: 7000fb23f9332d58c2c488ffbef14d73af4ac532
Summary:
`MononokeMegarepoConfig` is going to be a single point of access to
config storage system - provide both writes and reads. It is also a trait, to
allow for unit-test implementations later.
This diff introduces a trait, as well as implements the write side of the
configerator-based implementor. The read side/oss impl/test impl
is left `unimplemented`. Read side and test impl will be implemented in the future.
Things I had to consider while implementing this:
- I wanted to store each version of `SyncTargetConfig` in an individual
`.cconf` in configerator
- at the same time, I did not want all of them to live in the same dir, to
avoid having dirs with thousands of files in it
- dir sharding uses sha1 of the target repo + target bookmark + version name,
then separates it into a dir name and a file name, like git does
- this means that these `.cconf` files are not "human-addressable" in the
configerator repo
- to help this, each new config creation also creates an entry in one of the
"index" files: human-readable maps from target + version name to a
corresponding `.cconf`
- using a single index file is also impractical, so these are separated by
ascification of the repo_id + bookmark name
Note: this design means that there's no automatic way to fetch the list of all
targets in use. This can be bypassed by maintaining an extra index layer, whihc
will list all the targets. I don't think this is very important atm.
Reviewed By: StanislavGlebik
Differential Revision: D27795663
fbshipit-source-id: 4d824ee4320c8be5187915b23e9c9d261c198fe1
Summary:
We started getting the message
```stderr: eden/fs/utils/SpawnedProcess.cpp:798:21: error: 'getIOExecutor' is deprecated: getIOExecutor is deprecated. To use the global mutable executor use getUnsafeMutableGlobalIOExecutor. For a better solution use getGlobalIOExecutor. [-Werror,-Wdeprecated-declarations]
```
I don't see why we would need a mutable executor here so I chose `getGlobalIOExecutor` over `getUnsafeMutableGlobalIOExecutor`.
Reviewed By: kmancini
Differential Revision: D27912276
fbshipit-source-id: 95b1053f72c2b4eb2746e3c40c0cf76b69d90d6e
Summary:
In case the Mononoke server cannot provide the commit graph, and we need to
checkout and push changes. Let's add an emergency mode where the commit graph
only contains a single commit: master.
This can be used using `--config unsafe.emergency-clone=1`:
~/hg % lhg clone --shallow -U mononoke://mononoke.internal.tfbnw.net/fbsource ~/tmp/c1 --config unsafe.emergency-clone=1 --configfile /data/users/quark/.eden-backing-repos/fbs-lazy/.hg/hgrc.dynamic
connected to <remote host> session yyvXqQlHnMYQMEfw
warning: cloning as emergency commit+push use-case only! accessing older commits is broken!
resolving master
connected to <remote host> session ODc4PPiJ21L6r4Sn
added master: 248bd246f4467a2d4d0cacc09c5e55131ada9919
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
Smartlog:
~/hg % cd ~/tmp/c1
~/tmp/c1 % lhg sl
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
o 248bd246f 25 seconds ago remote/master
Pull:
~/tmp/c1 % lhg pull
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
pulling from ssh://hg.vip.facebook.com//data/scm/fbsource?stage1_read
connected to twshared1103.03.prn6.facebook.com session L4sDKzLm093aLUbo
searching for changes
adding commits
adding manifests
adding file changes
added 8 commits with 0 changes to 0 files
Checkout:
~/tmp/c1 % lhg sparse include .gitignore
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
~/tmp/c1 % lhg up master
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
19 files updated, 0 files merged, 0 files removed, 0 files unresolved
Commit:
~/tmp/c1 % vim .gitignore
~/tmp/c1 % lhg c -m gitignorewarning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
Smartlog:
~/tmp/c1 % lhg sl
warning: this repo was cloned for emergency commit+push use-case only! accessing older commits is broken!
@ cc43f0e5b (Backup pending) 4 seconds ago quark
╭─╯ gitignore
│
o 10ef2879e 5 minutes ago remote/master
│
~
Reviewed By: andll
Differential Revision: D27897892
fbshipit-source-id: f1770482455968dac217c9c6ee34ec0a20e5f432
Summary:
I found that there are still lots of (automation) users use the legacy clone
code path but it's unclear why (not having selectivepull?). Let's log the
reasons why the legacy path is used.
Reviewed By: sfilipco
Differential Revision: D27913616
fbshipit-source-id: b83f15e42a4afa94164b68bc9a91b4f0c022260c
Summary: The config is True everywhere for a long time.
Reviewed By: sfilipco
Differential Revision: D27913615
fbshipit-source-id: f2af34323c38f11db6bf3137652adea0bf38e858
Summary:
Non-remotefilelog logic might want to use edenapi too. So let's move it to
core.
It might make sense to make the `None` case an explicit error saying
`edenapi.url` is not set. For now I just keep it for compatibility.
The edenapi module is added so one can construct the edenapi client
without using a repo.
Reviewed By: kulshrax
Differential Revision: D27897889
fbshipit-source-id: 2a1fdf4c68464873f294ac1423d2348c1e526d5f
Summary:
Expose the correlator to core. This also reduces the lifetime of correlator
from global (process lifetime) to ui (dispatch.request/command), which
makes more sense and is more compatible with a multi-command per
process world (not using it by default yet).
This is needed to move edenapi to core.
Reviewed By: kulshrax
Differential Revision: D27897891
fbshipit-source-id: 7bd7e422c15e09a82e726436f92d4315ae876d94
Summary:
The zstore for commit messages are now handled by Rust entirely. There is no
need to keep the Python zstore around, except for migration and doctor
use-cases.
Reviewed By: andll
Differential Revision: D27897893
fbshipit-source-id: 21b10596af28c45425f6f102fd13f0421d1e8373
Summary: Add edenapi client python bindings for bookmarks so we use the bookmark api in python mercurial. At this layer there's a mixture of methods that iterate through the response stream and await the stats future and methods that directly return the response stream and stat future. I wasn't certain what to do in this case. The former may be cleaner later on, but the latter may be desirable if we paginate bookmark --list-remote output and therefore don't end up consuming the entire stream.
Reviewed By: kulshrax
Differential Revision: D27331446
fbshipit-source-id: 4dcbcb4bbd69117e02f2028e527f3d6f91c9bf99
Summary: Add bookmark method to Edenapi client for use in Mercurial client.
Reviewed By: kulshrax
Differential Revision: D27174441
fbshipit-source-id: cdc324e9115e87eab2e078209bbbc266e4e1dcdc
Summary:
The `shallowclone` method uses (legacy) revlog. We'll add more APIs that clone
in different ways but all of them use "shallow" (or remotefilelog). Name it to
clarify.
Reviewed By: andll
Differential Revision: D27897890
fbshipit-source-id: 2397a9621d3b207c394c995dff54deda4016e6fa
Summary:
It is considered an anti-pattern (https://rust-unofficial.github.io/patterns/anti_patterns/deny-warnings.html)
and is causing Github CI breakage unnecessarily (https://github.com/facebookexperimental/eden/runs/2402094456):
error: function is never used: `example_blob`
--> src/lfs.rs:1860:8
|
1860 | fn example_blob() -> (Sha256, usize, Bytes) {
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(dead_code)]` implied by `#[deny(warnings)]`
Reviewed By: andll
Differential Revision: D27911477
fbshipit-source-id: df8d642fe74fe311eb0f329d977b9b8270c27bf4
Summary:
Like it says in the title. It would be a good idea for us to have support for
tweaking this so that we can control the load it forces onto our backend
storage.
Reviewed By: StanislavGlebik
Differential Revision: D27909638
fbshipit-source-id: 1783e1fddbbd609877ff3a9b3084f12e005fca4b
Summary:
Right now, this function has:
- Sleeping
- Spawning
- Doing the work
Let's split this up into a few separate parts to make it easier to refactor.
Concretely I want to make the sleep interval tunable and it's much easier to do
this if that doesn't affect the tests.
Reviewed By: StanislavGlebik
Differential Revision: D27908957
fbshipit-source-id: d28388b8eb8c8f2e8693e1c765d1417bfc4b4825
Summary:
I'd like to make this a tunable, and having some tests turn it off makes that
more complicated. We can now actually flush this, so it also seems a bit
unnecessary. Let's remove.
Reviewed By: StanislavGlebik
Differential Revision: D27911049
fbshipit-source-id: 4634f9f3243132c53c228bd7476002412bace873
Summary:
In case the stderr is redirected but stdout is not:
EDENSCM_LOG=edenscm::mercurial=trace lhg log archival.py 2>/tmp/1
cat /tmp/1
The expected behavior is to still use the pager for stdout output, and the
stderr should be redirected to the specified file.
Reviewed By: andll
Differential Revision: D27867884
fbshipit-source-id: c369bc6be40fc200c4c0e2c9bb38b5faeb1208f2
Summary: This will be used in smartset layer to avoid 1-by-1 fetches.
Reviewed By: andll
Differential Revision: D27867668
fbshipit-source-id: 9d0d30ea214a1176e8c62e25171252c70df707b3
Summary: Added an API to record Python tracebacks to figure out 1-by-1 fetches.
Reviewed By: andll
Differential Revision: D27867670
fbshipit-source-id: d6941dd385db6eb2f810d97815056b660b970032
Summary:
`shorttraceback` returns a shorter version of the "traceback". It will be used
to figure out (frequent) callsite that triggers suboptimal 1-by-1 fetches.
Reviewed By: andll
Differential Revision: D27867673
fbshipit-source-id: 8f1f823007c2c94e1f62e72d816f03cbb4c8bc59
Summary: Split `smarttraceback` so the frame extraction logic can be reused.
Reviewed By: andll
Differential Revision: D27867669
fbshipit-source-id: 0e198f5400df5c1841926f9fac30f70ae74e8108
Summary: Expose Rust APIs to test if a callsite is enabled or not.
Reviewed By: andll
Differential Revision: D27867674
fbshipit-source-id: 0734b5ad6a65040f41a6f8b1bfc1e9a9109b9a8d
Summary:
This will be useful to test if a (frequently called) callsite is enabled or not
without having the logging side effect.
Reviewed By: andll
Differential Revision: D27867672
fbshipit-source-id: 361cb18a7a4680932dcfc9d5496d2906e1dc1f9f
Summary:
The previous choice of the upload / download arrows do not render in Windows
`cmd.exe` terminal using common fonts:
- Cascadia Code
- Cascadia Mono
- Consolas
- Courier New
- JetBrains Mono
- Lucida Console
Replace them with tri-angles that can be rendered using the above fonts.
Note: newer terminals like Microsoft Terminal and WezTerm can render the characters
just fine.
Reviewed By: andll, ikostia
Differential Revision: D27894085
fbshipit-source-id: c53d995355e66ba793d80afc1b36fc83853d07f8
Summary:
Cargo builds run multiple tests in same process, which was causing a failure since MononokeMatches started doing the one time log::set_boxed_logger() call.
I made MononokeMatches:new() error rather than panic in this case as it was already returning a Result anyway.
The failing test was only testing a very small section of code in block_execute, so this change removes it (its covered by integration test). It also gave some MononokeMatches coverage as a side effect (also covered by integration test).
Reviewed By: krallin
Differential Revision: D27891187
fbshipit-source-id: 9787029b610040cbf0125ab79748d6a3e540d2ae
Summary: Remove gotham and hyper patches as referenced PRs have been released
Reviewed By: krallin
Differential Revision: D27905248
fbshipit-source-id: a2b25562614b71b25536b29bb1657a3f3a5de83c
Summary:
We want to distinguish user vs system errors in `configo_client` and its users
(`mononoke_config` for instance). The reason is to allow `scs_server`
distinguish the two types of errors. Normally user errors would only ever be
instantiated fairly "shallowly" in the `scs_server` itself, but `configo_client`
is a transactional client (by this I mean that it calls user-provided transformation functions on fetched data), so we need to allow for a user error to originate from these updater functions.
Reviewed By: StanislavGlebik
Differential Revision: D27880928
fbshipit-source-id: e00a5580a339fdabc4b858235da9cd7e9fc7a376
Summary:
hg.py has a hard coded list of what configs it will copy from the source repo to the remote repo object (hg.py remoteui())
we use one of lfs and one of infinitepush option in edenscm/hgext/clienttelemetry.py where the remote repo object is used
Reviewed By: ahornby
Differential Revision: D27906275
fbshipit-source-id: d551934437126fdd0b920354bf4c51a6e09bafb2
Summary:
Integration test showing that MismatchedHeads errors are sent over the wire to
the client.
Reviewed By: quark-zju
Differential Revision: D27798555
fbshipit-source-id: b14a213e9055486bf07ecbb4b5453385df701b48
Summary:
This error occurs when the client sends us a set of heads that we cannot match.
For example when the client forces a commit in the master group; this was
possible with revlogs but should be a bug with Segmented Changelog. This can
also happen when master moves backwards, clients and server have multiple heads
then the server reseeds.
Clients that get this error should reclone.
Reviewed By: quark-zju
Differential Revision: D27786602
fbshipit-source-id: 9854ccee929ae0a845236ebd83ed68158c93fc7a
Summary:
We want to distinguish between no location and failure to compute location.
It is useful to know on the client side if we failed to compute the location of
some hash or we are not sending it because we don't have a location for it.
We have different options for hash-to-location in particular but the problem is
a general one. Does it make sense for us to skip sending error at the EdenApi
handler layer? I don't think so, I think that's an old hask. We should be
evolving the EdenApi handler layer to handle application errors.
This is still at the exploration phase. I haven't fixed the client side to
handle the errors.
Reviewed By: quark-zju, kulshrax
Differential Revision: D27549926
fbshipit-source-id: 7094160fe997af0e69c8d006b9731ea7dfb3e3f8
Summary:
The goal here is to add an easy way of propagating application errors from
server to the client.
The most convenient form for an error message is a message string so we just
start with that. I think the most important thing to add is some way of
communicating whether the error is retryable or not.
With conversions for anyhow::Error to WireError and from WireError to
ServerError it should be trivial for application code to pass application
errors down to the client.
The format here is driven by the fact that we have streams in the response
object. A batch oriented format for responses has more options. For example
with batches it is common to have a response object that has 3 categories:
1. found: HashMap, 2. not_found: Vec, 3. errors: Vec/HashMap
Reviewed By: kulshrax
Differential Revision: D27549923
fbshipit-source-id: 33b790253adc4761ea9ac7caced4148f4026e620
Summary:
On Mac this introduces 150ms delays in every indexedlog flush. During
an amend of a stack with 32 commits, this fsync accounted for 2/3rds of the time
spent.
Since commitcloud is pretty reliable these days, I think this is no longer
necessary.
Reviewed By: andll
Differential Revision: D27896589
fbshipit-source-id: a13a494c54ffea5a670ed942b366620619af2bd0
Summary: This is needed for rendering multiple progress bars in `cmd.exe`.
Reviewed By: andll
Differential Revision: D27887601
fbshipit-source-id: 9aa987cef327de91408f2e38b4d2e551fe10e39b
Summary:
`distutils_rust` tries to use `mt` to merge the "long path aware" Windows
manifest. It tries to deal with 2 cases:
- The exe already has an embedded manifest. Use `-inputresource` to merge them.
- The exe does not have an embedded manifest. Do not use `-inputresource` since
it will cause an error.
It does so by first passing `-inputresource`, then drop it if the error message
contains "The specified resource type cannot be found in the image file".
However, the error message is translated. Depending on system locale setting,
`mt` might output:
mt : general error c101008c: Failed to read the manifest from the resource of file "hg.exe". The specified resource type cannot be found in the image file.
mt : general error c101008c: Failed to read the manifest from the resource of file "hg.exe". 指定的映像文件不包含资源区域。
So let's check the error code `c101008c` instead.
Reviewed By: andll
Differential Revision: D27885096
fbshipit-source-id: 3d42a3b88ae593434e7104c7899bf6b4f3366180
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary:
This is a baseline packer, to let us experiment with packing.
It chooses the dictionary that gives us the smallest size, but does nothing intelligent around key handling.
Note that at this point, packing is not always a win - e.g. the alias blobs are 432 bytes in total, but the resulting pack is 1528 bytes.
Reviewed By: ahornby
Differential Revision: D27795486
fbshipit-source-id: 05620aadbf7b680cbfcb0932778f5849eaab8a48
Summary:
Let's make sure filenodes are warmed at the same time as hg changesets.
We do it implicitly by deriving them in getbundle, however it results in more
attempts of deriving filenodes, because we'd try to derive on every pull.
This is not a huge problem, but that would mean that we could send more
requests to the root filenode to check whether filenodes for a commit were derived or not.
And I remember looking at the logs a few days ago and seeing a few filenodes errors in our
scuba table, which I suspected might be related to the fact that filenodes are
not in derived data cache.
There's a high chance i'm wrong about the root cause of these problems this change should be
reasonable to do anyway.
Reviewed By: krallin
Differential Revision: D27822519
fbshipit-source-id: c878206a09bd19e4a68327dddf27aae10490740f
Summary:
Instead of using get_public method which queries bookmarks let's call
get_public_raw instead, which just does a phases fetch from a local db.
See previous diff for more motivation
Reviewed By: krallin
Differential Revision: D27821547
fbshipit-source-id: a71c8c9ad283259e9be98e63c9c72428e35c6142
Summary: `%s` with a rev number is not accepted in the current codebase. Use `%d` instead.
Reviewed By: ikostia
Differential Revision: D27874102
fbshipit-source-id: b83c40b7182da8639e82bea7bd00a036be4120c4
Summary: `%s` with a rev number is not accepted in the current codebase. Use `%d` instead.
Reviewed By: ikostia
Differential Revision: D27873899
fbshipit-source-id: b34eb0b80f0789c9e06af366bfdaa884c5c69357
Summary: `%s` with `revid` is not accepted in the current codebase.
Reviewed By: ikostia
Differential Revision: D27873898
fbshipit-source-id: e3790855892d3b07e1e5ea6bd92a14738bf6c100
Summary:
We didn't log it to perf counters log, and that makes it hard to aggregate,
show distributions etc
Let's start doing that
Reviewed By: krallin
Differential Revision: D27856968
fbshipit-source-id: 82fbba70154ee011073f3122256bd296bbb938ae
Summary: Its more efficient to bulk load the mappings for a chunk than do the queries one by one
Differential Revision: D27801830
fbshipit-source-id: 9c38ddfb1c1d827fc3028cd09f9ad51e3cbee5dc
Summary: Add an accessor so that we keep a reverse mapping of the WalkState::bcs_to_hg member as a cache of bonsai to hg mappings and also populate it on derivations.
Differential Revision: D27800533
fbshipit-source-id: f9b1c279a78ce3791013c3c83a32251fdc3ad77f
Summary: Add an accessor so that we can use the WalkState::hg_to_bcs member as a cache of hg to bonsai mappings
Reviewed By: farnz
Differential Revision: D27797638
fbshipit-source-id: 44322e93849ea78b255b2e3cb05feb8db6b4c7a7
Summary: This diff makes treeoverlay the default overlay type for Windows users.
Reviewed By: kmancini
Differential Revision: D27247658
fbshipit-source-id: 866eafc794eff1c262eab3061f14eb597bea0a66
Summary: This diff allows EdenFS to create tree overlay based on checkout configuration.
Reviewed By: kmancini
Differential Revision: D27242580
fbshipit-source-id: d0ebe33017c16517c117c1886f62bc9c6fe9f466
Summary:
`debugresetheads` is expected to remove all non-essential heads. That
includes bookmarks.
Reviewed By: kulshrax
Differential Revision: D27861548
fbshipit-source-id: 045976a5a9e27e7eee7ee48448c44552da439983
Summary:
Now that lastCheckoutTime is a single uint64_t, we no longer need a lock to
protect it, simple atomics are sufficient. Since reading an atomic usually
doesn't require any atomic operation, this will save a handful of atomics when
loading an inode where the last checkout time is read.
Reviewed By: chadaustin
Differential Revision: D27860653
fbshipit-source-id: 464e950c949ca243664d213da99d96ff5d0fcbb8
Summary:
The lastCheckoutTime is mostly used to initialize the timestamps of newly
loaded inodes and since these store an EdenTimestamp, we incur a conversion for
every inode loads. Instead of doing this conversion, we can simply make it an
EdenTimestamp directly.
Similarly, the getNow function was always converted to an EdenTimestamp
(sometimes more than once), we can also make it return the EdenTimestamp
directly.
Reviewed By: chadaustin
Differential Revision: D27860652
fbshipit-source-id: 9ea8fe9a312e6c3d8667b93130bb32a46ab92961
Summary:
Some test runner don't properly redirect stdout/stderr of nested processes, or
even direct writes to filedescriptors. On these, debugging a test failure is
almost impossible for EdenFS as we rely on the test output to be interleaved
with the EdenFS logs to understand what the daemon is doing.
To solve this, we can simply create a thread that redirects the output of
EdenFS to sys.std{out,err}.
Reviewed By: kmancini
Differential Revision: D27570966
fbshipit-source-id: 6a8d5229d8d5d141e6ab423f7658744b42af46e3
Summary: The Python `[auth]` matching code does not take cert validity into account when performing certificate matching, whereas the Rust version of the code does. In practice, the existing call sites for the Rust code disable match-time validation, and instead validate the certificate at time-of-use. This diff makes the Rust code's behavior match Python so we can remove the latter entirely.
Reviewed By: DurhamG
Differential Revision: D27837343
fbshipit-source-id: 0bfb5ebc3a36c8fa748cb289e2d8d1495ba8b296
Summary:
The svfs might have a different permission setup (ex. g+s, on ext4) that cannot
be applied to other vfs (ex. on edenfs). Do not inherit it. Instead, calculate
proper mode from the vfs root (ex. `.hg`).
Practically, the `createemode` is `None` in most of our repos. However,
`debugsegmentclone` might create svfs with `g+s` mode due to indexedlog's
`mkdir -p` recursively chmods created parents.
The original logic was added in 6590bef21 (FBS), 80e51429cb9a (HG) in 2008 with
little review comments: https://www.mercurial-scm.org/pipermail/mercurial-devel/2008-July/007269.html
Reviewed By: DurhamG
Differential Revision: D27860581
fbshipit-source-id: 43f93080621aaef168d2cecae46fd6dfb879fa1c
Summary: Enables the `Serialize` and `Deserialize` impls on the `Uuid` type.
Reviewed By: dtolnay
Differential Revision: D27799952
fbshipit-source-id: 4b0e2f8ab4ede20a2113fc1dda42c2ba8b3d0b35
Summary:
Previously we were always caching bookmarks, but D27323369 (f902acfcd1) accidentally remove
that. Let's add it back
Reviewed By: krallin
Differential Revision: D27859523
fbshipit-source-id: 8137c838fc56ecbbc64ba139d4a590dccd011bbc
Summary:
I am debugging why some people get vim to pop up during a merge conflict and some do not.
also fixes a few lint issues
Reviewed By: DurhamG
Differential Revision: D27684419
fbshipit-source-id: f636d71c18353a3816d1e442c05790cf4fd7b90b
Summary: I am removing this change because we've decided to store prepushrebase changeset id server-side.
Reviewed By: ikostia
Differential Revision: D27853518
fbshipit-source-id: 888897bc48c67477309b09af5f8c1825ce20cbca
Summary:
To prevent bonsai changeset divergence between prod and backup repo by copying
bonsais from prod repo directly during hg sync job push.
See more details about motivation in D27824210
Reviewed By: ikostia
Differential Revision: D27852341
fbshipit-source-id: 93e0b1891008858eb99d5e692e4dd60c2e23f446
Summary:
In the next diff it's going to be used to copy bonsais from the prod repo
during hg sync job, and in this diff I move this code to the common place so
that we can use it in the next diff
Differential Revision: D27852340
fbshipit-source-id: 9744571430e15a9d7f1e569d9b6690bc45787bd2
Summary:
This is not used on its on, but in subsequent diffs I will add a use-case, by
the megarepo configs crate.
When built in non-fbcode mode, this crate does not export anything. I chose this approach as opposed to the approach of exporting no-op stubs to force the clients to pay attention and implement gating their side too. This seems reasonable for a rather generic configo client.
Reviewed By: StanislavGlebik
Differential Revision: D27790753
fbshipit-source-id: d6dcec884ed7aa88abe5796ef0e58be8525893e2
Summary:
This diff does the following:
1. makes `megarepo_add_sync_target` into an async call
2. wraps all the async call responses into an additional struct to convey the
idea of pending requests
3. fixes code which imports/implements these interfaces
1 is needed because adding a new target will need to create an initial state of
the megarepo (so not just write some configs), and that is an expensive
operation.
2 is needed because we don't want to express the idea of "this request is not
yet processed" through a thrift exception. Instead, let make `_poll` calls
return a struct with a single optional field. When present, that field will
contain the payload of response to the underlying request. When absent, it will
indicate the fact that the request is still pending.
Of course, this is a compatibility-breaking change, that's why I want to get it in as early as possible (while there are no real clients calling changed methods).
Reviewed By: StanislavGlebik
Differential Revision: D27823377
fbshipit-source-id: dc2a5ed327b38d1cacd575af9d7edf5768f9c377
Summary: Now we're on rustc 1.51 the fork is no longer needed.
Reviewed By: dtolnay
Differential Revision: D27827632
fbshipit-source-id: 131841590d3987d53f5f8afb5ebc205cd36937fb
Summary:
If you ask for a range starting at byte 3, and the chunks are of size 3, then
you don't actually need need the first chunk, but right now we'll fetch it
then extract zero bytes from it.
This is quite wasteful, and for LFS range fetches will be problematic since it
basically doubles the volume of stuff we need to keep in cache (we need both
chunks).
Reviewed By: farnz
Differential Revision: D27824411
fbshipit-source-id: 7103f5b4d5bb78f023245f3e8a1bcb0c2f28faab
Summary:
Like it says in the title. We'd like to ask for the exact size that was
configured, because this way we can set the chunk size to the LFS threshold and
it avoids overlapping any file chunks server side.
Reviewed By: DurhamG
Differential Revision: D27824418
fbshipit-source-id: 43f40eb87080ec58e813ba1f1dda5b6a5e9f98ee
Summary:
While soft mount are nice as they allow the server (edenfs) to die and the
client applications to not end up in D state, this also force a maximum
(non-configuerable) 60s timeout for all IOs, after which application receive a
ETIMEDOUT. Thus, we need to not make the mount hard, thankfully, since the
mount is INTR, applications should not stay in D state if EdenFS dies.
Reviewed By: genevievehelsel
Differential Revision: D27808311
fbshipit-source-id: 17c30e88e5b236418064d8c309d85fdc6f1ca3e9
Summary:
Like it says in the title, this includes rendezvous into changesets. This is
our busiest connection by far, and it is often hitting the limits of our
connection pool: https://fburl.com/ods/kuq5x1vw
Reviewed By: markbt
Differential Revision: D27794574
fbshipit-source-id: e2574ce003f12f6c9ecafd0079fe5194cc63c24b
Summary:
I'd like to add RendezVous here (because this is our busiest connection:
https://fburl.com/ods/6d4a9qb5), and it'll be easier to do so if I just have
one code path to change instead of two.
Reviewed By: farnz
Differential Revision: D27794575
fbshipit-source-id: 350e3f8e3f3a74cb7c675cef1264c8083c516480
Summary:
After doing some local benchmarking (using MononokeApi instantiation as the
benchmark), one thing that's apparent is that we have quite a few parameters
here and that tuning them is likely to be a challenge.
One parameter in particular is the batch "objective", which controls how many
requests we want to see in the last batching interval before we choose to
batch (this is `rendezvous_dispatch_min_threshold`).
The problem with this is this is that there is no good, real-world, metric to
set it based on. This in contrast to the other parameters we have, which do
have some reasonable metric to compare to:
- rendezvous_dispatch_delay_ms: this is overhead we add to queries, so it
should be small & on the order of query execution latency (i.e. a few ms).
- rendezvous_dispatch_max_threshold: this controls how big our batches get, so
it should be on the order of what makes a SQL query too big (i.e. less than
a hundred records).
In contrast, we want to set `rendezvous_dispatch_min_threshold` such that
batching kicks in before we start using too many concurrent connections (which
is what query batching seeks to reduce), but the problem is that those two
numbers aren't directly connected. One clear problem, for example, is that if
our DB is in-region vs. out of-region, then for a given query execution time,
and a desired concurrency level before batching kicks in, we'd need different
values of `rendezvous_dispatch_min_threshold` (it would have to kick in faster
for the out-of-region workload).
So, this diff updates rendez vou to actually track concurrent connection count
before we force batching. This is the actual metric we care about here, and it
has a pretty natural "real world" values we can look at to decide where to set
it (our connection pool — which is limited at 100 concurrent connections —, and
our open connection baseline).
Note: I set this at 5 because that's more or less what servers look like
outside of spikes for Bonsai hg mapping, and of Changesets where I'm planning to
introduce this in the future:
- bonsai: https://fburl.com/ods/6d4a9qb5
- changesets: https://fburl.com/ods/kuq5x1vw (note: to make sense of this,
focus on just one server, otherwise the constnat spikes we get sort of hide
the big picture).
Reviewed By: farnz
Differential Revision: D27792603
fbshipit-source-id: 1a9189f6b50d48444b3373bd1cb14dc51b85a6d2
Summary:
Like it says in the title. There's no reason for this to be ad ad-hoc "throw in
an arg" when everything else is done by adding arg types.
Reviewed By: HarveyHunt
Differential Revision: D27791333
fbshipit-source-id: 38e5a479800179b249ace5cc599340cb84eb53e2
Summary:
Like it says in the title. Let's remove ad-hoc "add an arg then look the arg"
mechanisms like this one.
Reviewed By: HarveyHunt
Differential Revision: D27791334
fbshipit-source-id: 257cea7763ab5130525ad739fe4ebdda4e8bfeb6
Summary:
This module is way too big and bundles many different functions:
- Our app builder
- Our matches object and environment initialization
- A bunch of utility functions
Let's split it up
Reviewed By: HarveyHunt
Differential Revision: D27790730
fbshipit-source-id: 8353b18a28fde5267d03ba0342c8cb98ad855e37
Summary:
This isn't useful anymore. Let's ask our MononokeMatches what is set up for
caching instead of parsing the args one more time.
Reviewed By: HarveyHunt
Differential Revision: D27767697
fbshipit-source-id: 9da83769284a4aed4a96cd0eb212f42dd01ade87
Summary:
There is a very frustrating operation that happens often when working on the
Mononoke code base:
- You want to add a flag
- You want to consume it in the repo somewhere
Unfortunately, when we need to do this, we end up having to thread this from a
million places and parse it out in every single main() we have.
This is a mess, and it results in every single Mononoke binary starting with
heaps of useless boilerplate:
```
let matches = app.get_matches();
let (caching, logger, mut runtime) = matches.init_mononoke(fb)?;
let config_store = args::init_config_store(fb, &logger, &matches)?;
let mysql_options = args::parse_mysql_options(&matches);
let blobstore_options = args::parse_blobstore_options(&matches)?;
let readonly_storage = args::parse_readonly_storage(&matches);
```
So, this diff updates us to just use MononokeEnvironment directly in
RepoFactory, which means none of that has to happen: we can now add a flag,
parse it into MononokeEnvironment, and get going.
While we're at it, we can also remove blobstore options and all that jazz from
MononokeApiEnvironment since now it's there in the underlying RepoFactory.
Reviewed By: HarveyHunt
Differential Revision: D27767700
fbshipit-source-id: e1e359bf403b4d3d7b36e5f670aa1a7dd4f1d209
Summary:
ScrubOptions normally represents options we parsed from the CLI, but right now
we abuse this a little bit to throw a ScrubHandler into them, which we
sometimes mutate before using this config.
In this stack, I'm unifying how we pass configs to RepoFactory, and this little
exception doesn't really fit. So, let's change this up, and make ScrubHandler
something you may give the RepoFactory if you're so inclined.
Reviewed By: HarveyHunt
Differential Revision: D27767699
fbshipit-source-id: fd38bf47eeb723ec7d62f8d34e706d8581a38c43
Summary:
Basically every single Mononoke binary starts with the same preamble:
- Init mononoke
- Init caching
- Init logging
- Init tunables
Some of them forget to do it, some don't, etc. This is a mess.
To make things messier, our initialization consists of a bunch of lazy statics
interacting with each other (init logging & init configerator are kinda
intertwined due to the fact that configerator wants a logger but dynamic
observability wants a logger), and methods you must only call once.
This diff attempts to clean this up by moving all this initialization into the
construction of MononokeMatches. I didn't change all the accessor methods
(though I did update those that would otherwise return things instantiated at
startup).
I'm planning to do a bit more on top of this, as my actual goal here is to make
it easier to thread arguments from MononokeMatches to RepoFactory, and to do so
I'd like to just pass my MononokeEnvironment as an input to RepoFactory.
Reviewed By: HarveyHunt
Differential Revision: D27767698
fbshipit-source-id: 00d66b07b8c69f072b92d3d3919393300dd7a392
Summary:
We actually require tunables in our binaries, but some of our tests have
historically not initialized them, because the underlying binaries don't
load tunables (so they get defaults).
I'd like to remove the footgun of binaries not initializing tunables, but to do
this I need tunables to be everywhere, which is what this does.
Reviewed By: StanislavGlebik
Differential Revision: D27791723
fbshipit-source-id: 13551a999ecebb8e35aef55c0e2c0df0dac20d43
Summary:
I want to call something else MononokeEnvironment (the environment the whole
binary is running in), so let's rename this one.
Reviewed By: StanislavGlebik
Differential Revision: D27767696
fbshipit-source-id: bd6f2f282a7fc1bc09926a0286ecb8a5777a0a24
Summary: This test failed on CI for unknown reasons, log the sizes in the failure as a clue.
Reviewed By: farnz
Differential Revision: D27822287
fbshipit-source-id: d15c8165c1d5a5a588b48d7b8469e5cd9cba1a35
Summary: Changing generic anyhow::Error to ErrorKind so there is no need to downcast when we want to match on errors.
Reviewed By: krallin
Differential Revision: D27742374
fbshipit-source-id: ba4c1779d5919eb989dadf5f457d893a3618fffc
Summary:
In the next diff, the packer will need to create PackBlobs with access to link and unlink operations on the underlying data store.
Rearrange blobstore factory so that this is guaranteed by design, noting that we will want to manually create just a PackBlob later.
Reviewed By: ahornby
Differential Revision: D27795485
fbshipit-source-id: e16c7baea4f2402a4f8f95d722adb5c422c5b8e3
Summary:
This replicates behaviour of Python code - if unknown file content matches content of the file to be check out, do not abort checkout
This is useful for resuming interrupted checkouts / clones
Reviewed By: DurhamG
Differential Revision: D27799147
fbshipit-source-id: 7d2582583525da18fba08dfcd8bced2b619304de
Summary:
activate recently got broken when we added the prefetch-metadata flag,
this needs to be on activate as well as fetch
Reviewed By: xavierd
Differential Revision: D27778771
fbshipit-source-id: 052710c2f206e66d8042314773b6b408cff4915c
Summary: Currently native checkout aborts on unknown files even with --clean flag. It should not abort with --clean
Reviewed By: DurhamG
Differential Revision: D27779554
fbshipit-source-id: 2badc84c10eab28d2b1fc8840142ef883ac48c26
Summary: It's been showing up while building mononoke. Let's fix it
Reviewed By: sfilipco
Differential Revision: D27789928
fbshipit-source-id: a15912f66b9ad3370545aed88405dbeb800e63de
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary: This will show proper checkout progress when using native checkout
Reviewed By: quark-zju
Differential Revision: D27775423
fbshipit-source-id: 79f2afa02bd1fab7d5f747da1c714d4d1126ce7c
Summary:
EdenAPI makes heavy use of streaming HTTP responses consisting of a series of serialized CBOR values. In order to process the data in a streaming manner, we use the `CborStream` combinator, which attempts to deserialize the CBOR values as they are received.
`CborStream` hits a pathological case when it receives a very large CBOR value. Previously, it would always buffer the input stream into 1 MB chunks, and attempt to deserialize whenever a new chunk was received. In the case of downloading values that are >1GB in size, this meant potentially thousands of wasted deserialization attempts. In practice, this meant that EdenAPI would hang when receiving the content of large files.
To address this problem, this diff adds a simple heuristic: If a partial CBOR value exceeds the current buffer size, double the size threshold before attempting to deserialize again. This reduces the pathological case from `O(n^2)` to `O(log(n))` (with some caveats, described in the comment in the code).
Reviewed By: krallin
Differential Revision: D27759698
fbshipit-source-id: 67882c31ce95a934b96c61f1c72bd97cad942d2e
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Add a new `edenapi.maxrequests` config option to allow controlling the number of parallel in-flight requests. This can be used to bound resource usage of the client when requesting large amounts of data.
Reviewed By: sfilipco
Differential Revision: D27724817
fbshipit-source-id: 8d607efa83d8b0b94074d1a6e06f6c536cc0c791
Summary: Add a method to allow setting `CURLMOPT_MAX_TOTAL_CONNECTIONS`, which limits the number of concurrent requests within a curl multi session. If the number of requests in the session exceeds this number, they will be queued and sent once earlier requests have completed.
Reviewed By: sfilipco
Differential Revision: D27724818
fbshipit-source-id: 436384aed9d6d29f426e5e45aebb7a72c24ba667
Summary:
Without this, `make local` will build `hostcaps` without fb-specific logic and
cause wrong configs being used. `hg up master` will error out like:
File "treemanifest/__init__.py", line 690, in _httpgetdesignatednodes
self.edenapi.trees(dpack, self.name, keys)
RustError: Server reported an error (403 Forbidden)
Reviewed By: quark-zju
Differential Revision: D27759821
fbshipit-source-id: d42895f44bc53003f2578b65640ebe4ee05d52e6
Summary:
Right now, if prefetch fails, we just give the client back an error saying
"content not found".
This isn't super helpful, because usually the reason the content is not found
is because we cannot talk to the server that has the content, so showing the
user why we cannot talk to said server is more useful.
I'd like to ship this gradually, so I also added a config flag to turn it off.
Initially I'll have the flag set, but I did default it to not-set in the code
so that our tests run with the desired configuration.
Note: I initially contemplated adding logging for this here, but after
discussion with xavierd it looks like just failing instead of eating the error
is probably a better approach (and it's much simpler). This is also consistent
with what EdenAPI does.
Reviewed By: mzr
Differential Revision: D27761572
fbshipit-source-id: 3506d9c97a00e3f076bd346883e07f49194b0b06
Summary:
Right now, if the server ever tells us a file is missing, we fail the entire
batch download. This is a bit unfortunate because other objects could still be
downloaded, but also because we lose the distinction between "server crashed"
and "server told us the data we want does not exist".
Besides, it's actually a bit unnecessary. Indeed, this fails, we just ignore
the error anyway up the stack, so it never actually gets accessed.
Reviewed By: mzr
Differential Revision: D27761574
fbshipit-source-id: cb4fb0526a3bf19c04ecb81c05d44d4d8afb81ad
Summary: We can just return the actual error here now.
Reviewed By: sfilipco
Differential Revision: D27761573
fbshipit-source-id: 0866f976b4ed434deffd96be6820ad05d27b7b93
Summary:
If a operation can't proceed because we are in an interrupted update state,
indicate in the hint that `hg update` needs a destination.
Reviewed By: sfilipco
Differential Revision: D27764182
fbshipit-source-id: f0734a4929e34833c4bf84e148db04d57b779246
Summary:
This will allow us to have greater visibility into what's going on when there are production issues.
Note: for getpack, the params data model is `[MPath, [Node]]`. In practice there seems to always just be 1 node per mpath. However, to preserve the mapping, I log every mpath in a separate sample.
Reviewed By: ahornby
Differential Revision: D26690685
fbshipit-source-id: 36616256747b61390b0435467892daeff2b4dd07
Summary:
NOTE: The revisionstore LFS tests talk to prod Mononoke LFS, so the test here
will fail until that is deployed.
Like it says in the title, this adds support for downloading content in chunks
instead of all in one go. The underlying goal here is to let us do better load
balancing server side. This earlier diff in this stack explains that in more
detail: D27188610 (820b538001).
Reviewed By: quark-zju
Differential Revision: D27191594
fbshipit-source-id: 19b1be9bddc6b32b1fabfe4ab5738dfcf71b1a85
Summary:
Historically, we haven't really cared about the sizes provided by clients in
LFS, and we went as far as just echoing them back to the client.
However, with the support I'm adding for range requests, this will start to
matter, because clients will ask for content in this range, so if the client
has the wrong size, we should correct it for them rather than just let them
proceed, lest they fail to download the file properly.
That being said, I am pretty sure there are places relying on us not caring
about the size, so I'm not throwing errors there.
Reviewed By: mitrandir77
Differential Revision: D27710438
fbshipit-source-id: ab670b44364604c07c449e500e379ca40b8c5ec1
Summary:
Like it says in the title. This is helpful for the next diff here, and it's
generally convenient to have nice conversions between our frontend and backend
types.
Reviewed By: HarveyHunt
Differential Revision: D27710439
fbshipit-source-id: f7d4279d750715866844ee0b32418825fd325499
Summary: Since HeaderClientChannel now accepts a transport unique_ptr there's no need to have this deleter exposed outside of HeaderClientChannel.
Reviewed By: iahs
Differential Revision: D27729209
fbshipit-source-id: 064b03afdfe567b6df6437348596f0f6f97f6aaf
Summary: Introduce `FetchKey` and `FetchValue` traits to simplify repeated trait bounds in many `ReadStore` implementations. We also newly require `Clone` for both keys and values, which was already required by the fallback combinator.
Reviewed By: DurhamG
Differential Revision: D27652499
fbshipit-source-id: 6a3d5eb18a904b982fdb9946b80fcc9025d391ea
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary: Like `FileScmStoreBuilder`, but for trees. As LFS is not used for trees, `TreeScmStoreBuilder` defaults to `ExtStoredPolicy::Use` (pass along anything you find without LFS-specific checks).
Reviewed By: DurhamG
Differential Revision: D27641290
fbshipit-source-id: 637340a23cef058e7e37a41ae7f5b4fcc9481190
Summary: Introduce a new `FileScmStoreBuilder` structured much like `ContentStoreBuilder`, but supporting the features needed for the intermingling of `contentstore` and `filescmstore` construction (shared indexedlog, scmstore fallback to contentstore).
Reviewed By: DurhamG
Differential Revision: D27640702
fbshipit-source-id: e9771e6f61d80698a9dd761a0db66407b565c010
Summary: The previous change here wasn't sufficient. We need to wrap the handle fd in a Handle now as well.
Reviewed By: quark-zju, sfilipco
Differential Revision: D27753458
fbshipit-source-id: bd8ebbdcdc9acb411362795263b1419360f15e1a
Summary: This test fails without other diffs in stack because previously native checkout was overwriting untracked files
Reviewed By: DurhamG
Differential Revision: D27667151
fbshipit-source-id: 9b3aea37ba5c2d07fe4fc975dd40b4d7bea9d810
Summary: These tests were broken in D27710099 (876f812e4b), but they show as passing unless run in a particular environment, so it went unnoticed. This change reverts the tests to use the pre- D27710099 (876f812e4b) behavior, which should unbreak them until they can be updated correctly.
Reviewed By: quark-zju
Differential Revision: D27756348
fbshipit-source-id: cfa6c12871b6ac0d22b8c70400e72b3ec5dd83a3
Summary:
The `add_heads_and_flush` method might add new nodes in the master group,
and it should update `overlay_map_next_id` accordingly. Without it, it
might error out like:
RustError: ProgrammingError: Server returned x~n (x = 9ebc9ebc49f1819767b40f9ceb22c37547a10c37 8459584, n = 1411).
But x exceeds the head in the local master group {}. This is not expected and indicates some logic error on the server side.
Full error: P387088806
Reviewed By: sfilipco
Differential Revision: D27637278
fbshipit-source-id: b45370db0561dec52cd513a12e2fd0110f18e0e5
Summary:
The filternodes is an API that can batch hasnode checks. It is more efficient
if the commit hashes are lazy.
Reviewed By: sfilipco
Differential Revision: D27636338
fbshipit-source-id: 4eb2dcd20b939faa38611b82e32ed97cf0c8f175
Summary:
The filternodes is an API that can batch hasnode checks. It is more efficient
if the commit hashes are lazy.
Reviewed By: sfilipco
Differential Revision: D27636341
fbshipit-source-id: 69cd708a46c719624d654c53de3d92968392d572
Summary:
If a vertex was resolved via remote protocol, respect it and
avoid requesting the same thing twice.
Reviewed By: sfilipco
Differential Revision: D27636340
fbshipit-source-id: 43cf86010745a85cf622c67be4261ea47a33c802
Summary:
Many places use the `[n for n in nodes if hasnode(n)]` pattern, which
can be slow with a lazy graph due to N+1 problem. Add an API so the
Python world can use a more efficient way for the same query.
Reviewed By: sfilipco
Differential Revision: D27636339
fbshipit-source-id: 99ccb85b2266aed22f1cf87a5e2528106edb3783
Summary:
That could cause a slow loop testing node.__contains__ remotely.
This changes the behavior subtly - nodes added via addgroup will change `tip`
position regardless of whether the nodes exist. This might be more desirable,
since add or addgroup explicitly adding nodes should probably update the
tip position.
Offending test `test-globalrevs-requires.t` was removed since we have
forked the server-side codebase and do not need to maintain hg server
features here.
Reviewed By: sfilipco
Differential Revision: D27630090
fbshipit-source-id: cf7ecc44bf08ed756f0f1aece6655bf674171249
Summary:
The idset is not fully backed by Rust and do not batch resolve vertexes.
The nameset is backed by Rust NameSet and has proper batch prefetching.
Use nameset if possible but fallback to idset if the backend is not in
Rust (rare, only used by hgsql repos now).
Reviewed By: sfilipco
Differential Revision: D27630092
fbshipit-source-id: cf847dd1a78bd5273a8928ecb6616fe11f2c7026
Summary:
This will be useful to avoid suboptimal code paths that is very slow
with lazy vertexes.
Reviewed By: sfilipco
Differential Revision: D27630089
fbshipit-source-id: 35ee4ba79b551453de78fd22aecccf10bc43b08b
Summary:
While it is in theory "correct" going through the remote resolution even if the
protocol is "local". The overhead turns out to be something. And the tracing
message "resolve .. remotely" can be quite noisy. Let's just skip remote
resolutions early in IdConvert implementations.
Reviewed By: sfilipco
Differential Revision: D27630094
fbshipit-source-id: 7d87079876f040cf8f764f7362021dddba0d4723
Summary:
Currently the "contains vertex" check can trigger excessive
fetches for add_heads (and add_heads_and_flush used by flush).
Add a test to demonstrate the problem.
Reviewed By: sfilipco
Differential Revision: D27630091
fbshipit-source-id: ce3639c2a13226ba5681b4e8696edd7acbcb57f9
Summary:
Otherwise it can cause a lazy dag to think vertexes as "missing", insert
vertexes unnecessarily, and potentially break key graph properties (a
vertex should only have one Id).
Reviewed By: sfilipco
Differential Revision: D27629315
fbshipit-source-id: 1688d13cb94015bbc675613ecf9225556ff48372
Summary:
Also move related functions.
Similar to D27547584 (af3c3b3fd0), this allows `add_heads_and_flush` use `IdConvert`
on the `NameDag`, instead of the `IdMap` to trigger remote fetches properly.
This diff is easier to view with whitespace changes ignored.
Reviewed By: sfilipco
Differential Revision: D27629314
fbshipit-source-id: 8f79223c5d324aabfc5ab9813a9f65400fc533ec
