Summary:
Some of our upcoming repo merges will make it infeasible for someone to
use a full checkout. Let's add a config that will warn users of this. It has a
few levels, starting with a suppressable hint, then a non-suppressable warning,
then a suppressable exception, then a non-suppressable exception.
Reviewed By: ikostia
Differential Revision: D19974408
fbshipit-source-id: bad35a477ad8626dbc0977465368f5d71007e2d5
Summary:
MyRouter needs to be told which shards to watch. Since I'm adding a new shard,
it'll be easier for everyone to know that they need to update their MyRouter
configuration if we start logging the shard name we're trying to hit.
Reviewed By: ikostia
Differential Revision: D20001704
fbshipit-source-id: 8a9ff3521bc7e3c9b7ed39c6ae33d0ddc1d467b7
Summary:
On Windows, there are *two* 8-bit encodings for each process.
* The ANSI code page is used for all `...A` system calls, and this is what
Mercurial uses internally. It can be overridden using the `--encoding`
command line option.
* The OEM code page is used when outputing to the console. Mercurial has no
concept of this, and instead renders to the console using the ANSI code page,
which results in mojibake like "Θ" instead of "é".
Add the concept of an `outputencoding`. If this differs from `encoding`, we
convert from the local encoding to the output encoding before writing to the
console.
On non-Windows platforms, this defaults to the same encoding as the local encoding,
so this is a no-op unless `--outputencoding` is manually specified.
On Windows, this defaults to the codepage given by `GetOEMCP`, causing output
to be converted to the OEM codepage before being printed.
For ordinary strings, the local encoded version is wrapped by `localstr` if the
encoding does not round-trip cleanly. This means the output encoding works
even if the character is not represented in the local encoding.
Unfortunately, the templater is not localstr-clean, which means strings can get
flattened down to the local encoding and the original code points are lost. In
this case we can only output characters which are in the intersection of the
encoding and the output encoding.
Most US English Windows systems use cp1252 for the ANSI code page and cp437 for
the OEM code page. These both contain many accented characters, so users with
accented characters in their names will now see them correctly rendered.
All of this only applies to Python 2.7. In Python 3, everything is Unicode,
the `--encoding` and `--outputencoding` options do nothing, and it just works.
Reviewed By: quark-zju, ikostia
Differential Revision: D19951381
fbshipit-source-id: d5cb8b5bfe2bc131b2e6c3b892137a48b2139ca9
Summary:
`hg rage` generates the rage in the user's encoding. Since pastes are expected
to be in UTF-8, non-UTF-8 encodings result in garbled pastes.
Similarly, the lines-dec graph renderer uses escape sequences that won't work
on web pages, and the lines graph renderer uses curved lines which don't
render very well either. Force the use of the lines-square graph renderer,
which renders well.
Reviewed By: quark-zju
Differential Revision: D19951382
fbshipit-source-id: d1a5fd2ef195658f9bf10210088031474355f168
Summary:
The Rust graph renderer expects the message to be a unicode string, so ensure
we convert it from the local encoding before passing it to Rust.
Reviewed By: quark-zju
Differential Revision: D19951383
fbshipit-source-id: 644862c63873079364cb9902bd1bb49de8aa1ab9
Summary:
This adds a file hook to limit the file length we are willing to allow in
commits. This is necessary for now since Mercurial does have a limit on its
end, and we shouldn't allow commits that we cannot sync to Mercurial.
Reviewed By: HarveyHunt
Differential Revision: D19969689
fbshipit-source-id: 1da8a62d54e98b047d381a9d073ac148c9af84b0
Summary:
See later in this stack for motivation. This seems to work fine, and it allows
characters that don't fit latin1 when rendering diffs.
Reviewed By: markbt
Differential Revision: D19969743
fbshipit-source-id: 79c4afce5a19822d9b075d23ff4c88aa76ce2f42
Summary:
This adds some basic logging for input size for Gettreepack and Getpack. This
might make it easier to understand "poison pill" requests that take out the
host before it has a chance to finish the request.
Reviewed By: StanislavGlebik
Differential Revision: D19974661
fbshipit-source-id: deae13428ae2d1857872185de2b6c0a8bcaf3334
Summary:
I'm going to modify it in the next diff, so let's make it async.
Note that we used `spawn_future()` before which I replaced with tokio::spawn()
here. It's not really clear if we need it at all - I'll experiment with later.
Removing it will make the code cleaner.
Reviewed By: krallin
Differential Revision: D19973315
fbshipit-source-id: cbbb9a88f4424e6e717caf1face6807ab6c32438
Summary:
As of 63c471ad8a4ba0bebd1acf70569bcdcefc3fffbf in upstream Dulwich, it
now turns commands into unicode. Unfortunately, _ssh.py in hggit sees that the
type is no longer str or bytes and thinks it's an array and puts spaces between
every letter, causing it to break.
Let's allow unicode. This broke because dulwich was recently upgraded.
Reviewed By: sfilipco
Differential Revision: D19983215
fbshipit-source-id: 059756905bf4b2c73009001b078c8723ae378246
Summary: Not very valuable, if it just prints the constant name.
Reviewed By: StanislavGlebik
Differential Revision: D19978690
fbshipit-source-id: ae2b648f50098b479cb3719fd9b9d4b82bac3d3c
Summary: This should get rid of the extraneous uninitialized attribute errors related to `setUp` and abstract classes.
Reviewed By: simpkins
Differential Revision: D19964487
fbshipit-source-id: 52d5a6496e372d99d4398473f9ed7672228a76f5
Summary:
This is a revised version of D19887220.
D19887220 has 2 problems:
- It can silently ignore the mt.exe error after failures of all retries.
- There is another place that `mt.exe` runs that is not covered by retry.
This diff fixes them by wrapping the `set_long_paths_manifest` function
directly so it covers two `mt.exe` places, and makes sure all retry failure
is still a failure.
Reviewed By: sfilipco
Differential Revision: D19977802
fbshipit-source-id: 774d0c42b247a7e111841cd69f71760a5544d685
Summary:
Update includes to the third-party xdiff.h file to use absolute includes
from the repository root. This allows many parts of our internal build
tooling to work better, including automatic dependency processing.
Reviewed By: xavierd
Differential Revision: D19958228
fbshipit-source-id: 341dd8c94f1138cf4a387b92e1817b2a286d6aa1
Summary:
Update the C files under edenscm/mercurial/cext to use absolute includes from
the repository root. Also update a few of the libraries in edenscm/mercurial
that the cext code depends on.
This makes these files easier to build with Buck in fbsource, and reduces the
number of places where we have to use deprecated Buck functionality to help
find these headers. This also allows autodeps to work with the build targets
for these rules.
Reviewed By: xavierd
Differential Revision: D19958221
fbshipit-source-id: e6e471583a795ba5773bae5f16ed582c9c5fd57e
Summary:
Remove `thirdparty/pyre2/__init__.py` from the `libhg` sources list.
We don't compile the `thirdparty/pyre2/_re2.cc` file in the fbcode build, so
importing the `__init__.py` module from this package just triggers an
ImportError when the code tries to use it. The code then always falls back to
using the version of pyre2 included from the `fb-re2` wheel.
Dropping the `__init__.py` module from our library should simply trigger an
ImportError earlier when we can't even find this file, and the code will still
fall back to using `fb-re2`.
Including this `__init__.py` file just causes issues for type checking, since
it causes us to try and type check this file even though its dependencies are
not present.
Reviewed By: xavierd
Differential Revision: D19958224
fbshipit-source-id: 34ea8806b6ee9377f17a9318c64c91ec242225df
Summary:
Some of the methods in eden_dirstate_map.py had comments that were close to
type annotations that were added a couple years ago. Update them to proper
type comments that can be recognized by Pyre and mypy.
Also remove the unused create_clone_of_internal_map() method.
Reviewed By: chadaustin, xavierd
Differential Revision: D19958225
fbshipit-source-id: b753c030acb15cf4f8d8c536614e657ee1bcba52
Summary:
Update the `eden_dirstate_map` class to store `dirstatetuple` objects instead
of plain tuples in its `_map` member variable. Without this the `filefoldmap`
code that is used on Windows fails, as it directly accesses `self._map` and
expects it to contain `dirstatetuple` objects.
Reviewed By: DurhamG, pkaush
Differential Revision: D19841881
fbshipit-source-id: ddb7523b598cfd8ec8719a8a74446cefcb411358
Summary:
Eden SCM expects that DRY_RUN reports the same conflicts as a normal
checkout, but EdenFS would skip traversing deleted trees in dry run
mode. Fix that and add a test.
Reviewed By: genevievehelsel
Differential Revision: D19782543
fbshipit-source-id: 7a269e67a41b7ad6ce6c54fde37e8f74fcc1ef51
Summary:
bonsai_verify occasionally visits the same commit twice (I found out by adding
logging and noting that it occasionally visits the same commit twice). Let's
allow this here.
Reviewed By: StanislavGlebik
Differential Revision: D19951390
fbshipit-source-id: 3e470476c6bc43ffd62cf24c3486dfcc7133de6c
Summary: We're about to start adding more handlers to the server. Rather than putting them all in the same file, let's create a submodule for them.
Reviewed By: krallin
Differential Revision: D19957012
fbshipit-source-id: 38192664371f0b0ef5eadb4969739f7cb6e5c54c
Summary: Add a `RequestContext` type that stores per-request state, along with a `Middleware` implementation that injects a `RequestContext` into Gotham's `State` object for each request. This is essentially a stripped-down version of the `RequestContextMiddleware` used in the LFS server. Given that the RequestContext contains application-specific functionality, this Middleware lives alongside the rest of the EdenAPI server code rather than in the `gotham_ext` crate (where all of the generic Middleware lives).
Reviewed By: krallin
Differential Revision: D19957013
fbshipit-source-id: 6fad2b92aea0b3662403a69e6a6598e4cd26f083
Summary:
Currently if derivation of a particular derived data type is disabled, but a
client makes a request that requires that derived data type, we will fail with
an internal error.
This is not ideal, as internal errors should indicate something is wrong, but
in this case Mononoke is behaving correctly as configured.
Convert these errors to a new `DeriveError` type, and plumb this back up to
the SCS server. The SCS server converts these to a new `RequestError`
variant: `NOT_AVAILABLE`.
Reviewed By: krallin
Differential Revision: D19943548
fbshipit-source-id: 964ad0aec3ab294e4bce789e6f38de224bed54fa
Summary: fork exec wait in `daemon.dameon_exec` so we can get exit code of child process in order to log.
Reviewed By: simpkins
Differential Revision: D19861810
fbshipit-source-id: 85fce52b2e2d252bb4dec779f5f975e3712b6bb5
Summary:
Prepare configs locally that can be passed to any Mononoke binary where things
/just work/.
Reviewed By: HarveyHunt
Differential Revision: D19952512
fbshipit-source-id: 14a3b520972b0bdf4fa7810805066ba746bbef1a
Summary: Adds the Cargo.toml files for blobstore, this is a step towards covering mononoke-types, so only the blobstore traits are covered by this diff.
Reviewed By: aslpavel
Differential Revision: D19948739
fbshipit-source-id: c945a9ca16ccceb0e50a50d941dec65ea74fe78f
Summary: Generate the Cargo.toml files inside xdiff with autocargo. This will enable Mononoke to depend on this code easily without sacrificing anything on eden/scm side.
Reviewed By: aslpavel
Differential Revision: D19948741
fbshipit-source-id: 905ff3d64b90830e5f075e4c6ed2b3de959e3f00
Summary:
Not being able to prefetch draft parent trees should not be considered as a
fatal error.
This code path is causing trouble with narrow-heads clone:
1. Streaming clone. The client gets a changelog.
2. The client runs "pull" to get new commits. The prefetchdraftparents code path runs.
3. The client has stale remote names, and public() is lagging. `prefetchdraftparents`
will try to fetch trees at the old master, but the repo is not configured properly.
That causes a stacktrace like:
$ /usr/bin/hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/www.rc ssh://hg.fb.com/repo repo
connected to hg.fb.com
streaming all changes
searching for changes
adding commits
adding manifests
adding file changes
added 1 commits with 0 changes to 0 files # <<<< No traceback if this says "0 commit".
Traceback (most recent call last):
File "edenscm/hgext/remotenames.py", line 1464, in exclonecmd
orig(ui, *args, **opts)
File "edenscm/hgext/remotefilelog/__init__.py", line 433, in cloneshallow
orig(ui, repo, *args, **opts)
File "edenscm/mercurial/commands/__init__.py", line 1615, in clone
shareopts=shareopts,
# shareopts = {'mode': 'identity'}
File "edenscm/mercurial/hg.py", line 741, in clone
exchange.pull(local, srcpeer, revs, streamclonerequested=stream)
File "edenscm/mercurial/util.py", line 621, in __exit__
self.close()
File "edenscm/mercurial/transaction.py", line 46, in _active
return func(self, *args, **kwds)
File "edenscm/mercurial/transaction.py", line 543, in close
self._postclosecallback[cat](self)
# cat = bin('6472616674706172656e74747265656665746368')
File "edenscm/hgext/treemanifest/__init__.py", line 490, in _parenttreefetch
self.prefetchtrees([c.manifestnode() for c in draftparents])
# c = <changectx b5ad643b3009>
# draftparents = [<changectx b5ad643b3009>]
File "edenscm/hgext/treemanifest/__init__.py", line 522, in prefetchtrees
self._prefetchtrees("", mfnodes, basemfnodes, [], depth)
# basemfnodes = [bin('a25f17018d7cd07f1f6bc3076f95c5980ba087a9')]
# mfnodes = [bin('ad717aac7700e783a1d84f3330d13a7731a4726a')]
File "edenscm/hgext/treemanifest/__init__.py", line 529, in _prefetchtrees
fallbackpath = getfallbackpath(self)
File "edenscm/hgext/treemanifest/__init__.py", line 2173, in getfallbackpath
if util.safehasattr(repo, "fallbackpath"):
File "edenscm/mercurial/util.py", line 190, in safehasattr
return getattr(thing, attr, _notset) is not _notset
# attr = 'fallbackpath'
File "edenscm/mercurial/util.py", line 904, in __get__
result = self.func(obj)
File "edenscm/hgext/remotefilelog/shallowrepo.py", line 42, in fallbackpath
"no remotefilelog server " "configured - is your .hg/hgrc trusted?"
Abort: no remotefilelog server configured - is your .hg/hgrc trusted?
abort: no remotefilelog server configured - is your .hg/hgrc trusted?
Fix it by making prefetchdraftparents non-fatal. This would hopefully unblock
narrow-heads rollout.
Reviewed By: xavierd
Differential Revision: D19957251
fbshipit-source-id: e65bbe6bf422776effe49055f7332ec538177a41
Summary: Notifications is using folly Subprocess which doesn't work on Windows.
Reviewed By: genevievehelsel
Differential Revision: D19863375
fbshipit-source-id: 63b047253c0f8a48b1b0ccc767f5820e77a28d80
Summary:
This will allow us to improve our dashboards filtering out errors we are
responsible for, like missing certs on the machines.
Reviewed By: mitrandir77
Differential Revision: D19950614
fbshipit-source-id: 73503e984dfe8513a700fdcb2fc36b1618c20a4f
Summary:
Commit messages and extras can be unbounded in size. This can cause problems if users create commits with exceptionally large messages or extras. Mercurial will commit these to the changelog, increasing its size. On Mononoke, large commit messages may go over the cacheing threshold, resulting in poor performance for requests involving these commits as Mononoke will need to reload on every access.
Commit messages should not usually be that large. Mostly likely it will happen by accident, e.g. through use of `hg commit -l some-large-file`. Prevent this from happening by accident by adding configuration for soft limits when creating commits.
If a user really does need to create a commit with a very large message or extras, they can override using the config option.
Reviewed By: xavierd
Differential Revision: D19942522
fbshipit-source-id: 09b9fe1f470467237acc1b20286d2b1d2ab25613
Summary:
This parameter was originally removed in D12811551, but re-added in D12855935
due to the fact that at the time the `eden_dirstate.py` and `dirstate.py`
files were deployed in separate RPMs and could not be updated together
atomically. We now deploy these files together, so we can drop this extra
unnecessary argument.
Reviewed By: chadaustin
Differential Revision: D19913057
fbshipit-source-id: 0f0b4fde4b3124a8fc5bb568551b4e67de14d410
Summary:
- Pushing .compat down from main into run function and switch to 0.3 timed function
Note: Possible next level of pushing down: pushing .compact into derive_fn and get rid of BoxFuture run's signature.
Reviewed By: ikostia
Differential Revision: D19943392
fbshipit-source-id: 65bd84492855d3e2e560299a586af6dd4fe9c3ea
Summary:
Sometimes the treestate points to an unknown commit (ex. aborted transaction
might strip commits). While `debugrebuilddirstate -r HASH --hidden` is able to
fix it, it is too slow.
This diff adds treestate repair logic to the `doctor` command. It scans through
the treestate files, find a most recent `Root` entry with `p1` pointing to a
known commit.
This can be much faster than `debugrebuilddirstate` in some cases, because the
watchman clock might still be valid, and the NEED_CHECK file list might still
be small. In that case, `status` can still be fast.
Since treestate atomically updates all information needed for `status`
calculation (parents, need-check-files (or, "non-normal files"), watchman-clock
(only with fsmonitor), and stat for clean files). Reverting to a previous state
is still atomic. Correctness-wise, this is equivalent to aborting a "large"
transaction, and restoring treestate data to the state before the transaction.
It should be consistent, and the next `status` call won't mis-report files like
the dangerous `debugsetparents` command.
Reviewed By: DurhamG
Differential Revision: D19864422
fbshipit-source-id: d5d2f8b43a0c15ea2ac0e3c164edec7deeb8451f
Summary:
See the test change. Without this change repairing the changelog won't give the
user back a working repo.
Reviewed By: markbt
Differential Revision: D19864421
fbshipit-source-id: b84582c5302469828c8cfcb3db362ea82f2eea63
Summary:
Reuse utilities in the fixcorrupt extension to repair changelog.
This is better than fixcorrupt because `hg doctor` does not require a repo
object. Some messages are updated so they become more consistent with the
rest of `hg doctor`.
The main motivation is to get changelog fixed early, so other repair logic can
check if a commit hash is known by changelog or not.
Reviewed By: markbt
Differential Revision: D19864418
fbshipit-source-id: 6f95c6c6191d7db2a474a07a5278a857cf41d8e2
Summary:
Run 'edenfsctl doctor' on an edenfs repo. If there is no current repo, it might
be caused by edenfs daemon stopped running. So let's also run edenfsctl doctor
in that case.
Reviewed By: markbt
Differential Revision: D19864419
fbshipit-source-id: d2a49a126a040845b88b4883d214162326d08d8d
Summary:
We're seeing a user have issues because their username contains unicode
characters and sampling's use of json doesn't handle it well. I've not been able
to repro it unfortunately, but let's go ahead and switch sampling to use
mercurial.json.
Differential Revision: D19895419
fbshipit-source-id: a1f087d1e2c7568488c2b8d54f267bd5c8266202
Summary: This will be used in the LFS store.
Reviewed By: DurhamG
Differential Revision: D19895803
fbshipit-source-id: 4cf447987c10fed0b5c98904f20c841428965d89
Summary:
In some cases, higher level stores may want to store data in either a plain
IndexedLog, or in a RotateLog, for local and shared data. Due to slight
difference between the 2, they can't easily be adapted into a common trait.
Instead let's just wrap both into an enum and implement the main functions that
the higher level stores need.
The first use of this will be the LfsStore, future use will include the
IndexedLogDataStore and the IndexedLogHistoryStores.
Reviewed By: DurhamG
Differential Revision: D19859292
fbshipit-source-id: 920572e0cf5f69bda4901a727a6b0dc0f08fc8d0
Summary: records if a start was successful or not
Reviewed By: simpkins
Differential Revision: D19817810
fbshipit-source-id: b67253099781bb534b7e2fb26a09ba41c1f0bd69
Summary: Since we cannot log this case from the daemon because we can't catch sigkill, log failed stop from CLI layer.
Reviewed By: simpkins
Differential Revision: D19826140
fbshipit-source-id: eb3aa27802db0206a13e552c4cb1384f856905d2
Summary:
this is used up the stack. This introduces generic scuba logging for the cli layer. In case of the open source build, `log` will be a no-op as suggested in `cli/telemetry.py`.
this is used as so:
```
from .telemetry import build_base_sample, log
# for example, I am adding the field "status" to know that this is a status call.
sample = instance.build_sample("status").add_string("something", "another")
instance.log(sample)
```
Reviewed By: simpkins
Differential Revision: D19816913
fbshipit-source-id: b055d4d1e29456e3549292e6f5047b935f11e4e2
Summary: Add the max_jitter_ms field to the rate limiting config struct, and to the integration test.
Reviewed By: HarveyHunt
Differential Revision: D19905068
fbshipit-source-id: b44251c456a45bc494d1080e405f2d009becc0d2
Summary:
This is required for 0.2 timers or runtime reliant code to work within the sync
job. To achieve this, we need to get of Tokio 0.1 fs code, which is
incompatible with Tokio 0.2 because it uses `blocking()`.
Reviewed By: ikostia
Differential Revision: D19909434
fbshipit-source-id: 58781e858dd55a9a5fc10a004e8ebdace1a533a4
Summary:
This update the warm_bookmarks_cache's constructor to use the passed in
blobrepo's derived data configuration (instead of whatever the caller is
passing in), since we now have that information.
Reviewed By: HarveyHunt
Differential Revision: D19949725
fbshipit-source-id: 575a1b9ff48f06003dbf9e0230b7cca723ad68f5
Summary: Add hash::GitSha1 as a pure hash-only key for git Aliases, so one no longer needs to know the size or type to load by Alias::GitSha1.
Reviewed By: krallin
Differential Revision: D19903578
fbshipit-source-id: bf919b197da2976bf31073ef04d12e0edfce0f9b
Summary:
Rename GitSha1 to RichGitSha1 in preparation for introducing hash::GitSha1 as a pure sha1 without extra fields in next in stack.
Motivation for this is that currently one can't load content aliased by Alias::GitSha1 give just the hash, one has to know the type and size as well.
Once the next couple stack are done we will be able to load via just the git hash.
Reviewed By: krallin
Differential Revision: D19903280
fbshipit-source-id: ab2b8b841206a550c45b1e7f16ad83bfef0c2094
Summary:
When max concurrency is 1, we should process at most one request concurrently,
not 2! This had resulted in a flaky test since we're processing traffic out of
order there.
Reviewed By: HarveyHunt
Differential Revision: D19948594
fbshipit-source-id: 00268926095fdbbfdfd5a23366aafcfb763580f4
Summary:
It would be better to make the underlying implementation faster for full hash
cases than check when it is used.
Reviewed By: krallin
Differential Revision: D19905033
fbshipit-source-id: 2d9a77099dc614e80fdb1c0ee715c576a56ba09c
Summary:
Right now, Mononoke code in apiserver executes on Actix's runtime. That's a 0.1
runtime, which means that we're calling into code that might just fail if e.g.
it uses Tokio 0.2 timers.
This is a pretty big footgun, so let's fix it. As it turns out, we already have
a Tokio compat runtime in process, which is (was — this is mostly in SCS now)
used for Thrift calls.
So, let's use that runtime to call into Mononoke code. This ensures we don't
get any nasty surprises of the panicky kind at runtime.
Reviewed By: markbt
Differential Revision: D19902538
fbshipit-source-id: d9d7307b8cf75c3e7e1ecf04c0e10076b3eaef3d
Summary:
This allows code that is being exercised under async_unit to call into code
that expects a Tokio 0.2 environment (e.g. 0.2 timers).
Unfortunately, this requires turning off LSAN for the async_unit tests, since
it looks like LSAN and Tokio 0.2 don't work very well together, resulting in
LSAN reporting leaked memory for some TLS structures that were initialized by
tokio-preview (regardless of whether the Runtime is being dropped):
https://fb.workplace.com/groups/rust.language/permalink/3249964938385432/
Considering async_unit is effectively only used in Mononoke, and Mononoke
already turns off LSAN in tests for precisely this reason ... it's probably
reasonable to do the same here.
The main body of changes here is also about updating the majority of our
changes to stop calling wait(), and use this new async unit everywhere. This is
effectively a pretty big batch conversion of all of our tests to use async fns
instead of the former approaches. I've also updated a substantial number of
utility functions to be async fns.
A few notable changes here:
- Some pushrebase tests were pretty flaky — the race they look for isn't
deterministic. I added some actual waiting (using pushrebase hooks) to make
it more deterministic. This is kinda copy pasted from the globalrev hook
(where I had introduced this first), but this will do for now.
- The multiplexblob tests don't work at all with new futures, because they call
`poll()` all over the place. I've updated them to new futures, which required
a bit of reworking.
- I took out a couple tests in async unit that were broken anyway.
Reviewed By: StanislavGlebik
Differential Revision: D19902539
fbshipit-source-id: 352b4a531ef5fa855114c1dd8bb4d70ed967dd55
Summary: There's still some issues, but it's a lot closer.
Reviewed By: quark-zju
Differential Revision: D19802023
fbshipit-source-id: da539094cbc0ba3542e4b5fd3d49f5f80455ec23
Summary:
There was a spot where we returned bytes for a filepath. Fix this to
make dirstate tests pass more.
Reviewed By: quark-zju
Differential Revision: D19786274
fbshipit-source-id: 7465cae8bb2e3be7758abc6279ed3f5f59581732
Summary: Not sure where these got fixed.
Reviewed By: quark-zju
Differential Revision: D19772617
fbshipit-source-id: 7bebd15ad080e1fc224d8c1e78f645877551ac92
Summary:
These are a random batch of stack traces that show up in remotefilelog
tests.
Reviewed By: quark-zju
Differential Revision: D19772619
fbshipit-source-id: a8b9ce188cb7a5a2c9ccaeb62f6744f1c4083e38
Summary:
assertEquals is deprecated and shows warnings in the tests in python 3
ignore-conflict-markers
Reviewed By: quark-zju, sfilipco
Differential Revision: D19907385
fbshipit-source-id: 7d261489856a4eeb6719eae581ed986d0415d99e
Summary:
archive uses a formatter to produce it's metadat file. We need to use a
string io stream instead of bytes.
Reviewed By: quark-zju
Differential Revision: D19748163
fbshipit-source-id: fbd8c32066cfc4a234d9b51691717c7fce4c7c9a
Summary: They need to be bytes to match the nodes.
Reviewed By: xavierd
Differential Revision: D19746027
fbshipit-source-id: 41e9cc390f4aa97d8c8b378144e64100811665c2
Summary:
sshpeer._calltwowaystream used iter(fp, ""), where "" was the sentinel
indicating the fp was empty. Since fp was an iterator of bytes, the sentinel was
never hit and this code kept sending 0\n to each other until the pipes filled
and it hung.
Reviewed By: xavierd
Differential Revision: D19746028
fbshipit-source-id: 0daada0ae2356b5c99cc2c39c121cd88af8f750a
Summary:
Fixes test-cat.t by changing some formatter encoding choices. The
formatter is bit awkward here. hg cat needs to support outputting raw binary,
but also json and templated output.
For now I've set it up so json and templated output can't output non-utf8 data.
Reviewed By: quark-zju
Differential Revision: D19786542
fbshipit-source-id: 84060928103b396b23e3173b715aed996074fa3e
Summary:
treedirstate was using bytes for state and for file paths. This does
the appropriate conversions. Note, I don't use strings for all state in rust,
because it's a pain in the butt in some places. We're going to delete
treedirstate eventually anyone, so just I'm getting the test to pass for now.
Reviewed By: quark-zju
Differential Revision: D19715102
fbshipit-source-id: 4f9eac372bee7884d36aa19e3a3ed253392fa7dd
Summary:
When I run make local it's creating changes in our checked in thrift
types. I guess I need to check these in?
Reviewed By: quark-zju
Differential Revision: D19848706
fbshipit-source-id: 8a2e9a2617734eda41eade1f2645689362b1d75d
Summary:
Those tests broke after D19830033 landed, which removed the functionality that
generates the logs they were capturing. This fixes that.
Reviewed By: farnz
Differential Revision: D19941478
fbshipit-source-id: 2b422b1d2252e92bb75ae688962fdd66ed33910c
Summary:
Add `debugmutation` output for recent draft commits to `hg rage`. This will
allow us to easily see the recent history of the draft commits in a user's
repo.
Reviewed By: DurhamG
Differential Revision: D19903999
fbshipit-source-id: f6ebd729812c63d3760f9dab031414df6b24ab28
Summary:
Make it possible to limit the time range of mutation info being displayed by hg
debugmutation.
Reviewed By: DurhamG
Differential Revision: D19904000
fbshipit-source-id: 365f54fdd861661961bba1a0ea96fce772623a23
Summary: The correct template for the program name is `prog@`.
Reviewed By: HarveyHunt
Differential Revision: D19904938
fbshipit-source-id: 19a3ac18f85e2b090f53d0423fce3c6982cf32fd
Summary: The error should be request error, not internal error
Reviewed By: krallin
Differential Revision: D19856120
fbshipit-source-id: fc66ac4eaeb27941de3b4ba769fe123b28685d14
Summary:
Now let's fail if we try to derive data that's not enabled in the config.
Note that this diff also adds `derive_unsafe()` method which should be used in backfiller.
Reviewed By: krallin
Differential Revision: D19807956
fbshipit-source-id: c39af8555164314cafa9691197629ab9ddb956f1
Summary: See title. We want the name of the cache to live in `Allocator` - and be used in Open Source as well.
Reviewed By: sathyaphoenix
Differential Revision: D19663603
fbshipit-source-id: 2abef67b56a37cb551e925121813f4b4c8c6e9db
Summary:
Up to now, this has been done in chef, and thus for repos that we do not list,
they may share the memcache keys, with potential unintended consequences. Let's
always add the repo name to the key, so we can simplify the code in chef.
One small negative effect of this change is that while it is being rolled out,
the cache hit rate will be impacted. This should resolve itself quickly.
Reviewed By: DurhamG
Differential Revision: D19885775
fbshipit-source-id: 0b59ce9e378b0ab70f696a39d19d27cd89921098
Summary:
- Conditionally enable warmup cache based on derived data configuration
- Blame is temporarily disabled for ovrsource, it is currently not used, and will be enabled again once new blame is generated for this repository
Reviewed By: StanislavGlebik
Differential Revision: D19902902
fbshipit-source-id: 7b782a32f7bfd76024686aea027bb15223079b9d
Summary:
When using the `--date` parameter to `hg update` or `hg revert`,
`cmdutil.finddate` returns the binary node for the target commit. Passing this
to `scmutil.revsingle` sometimes works, however it's unreliable, as if the
binary node happens to look vaguely like a revset, we will attempt to parse it,
and perhaps fail.
Resolve the ambiguity by converting the binary node to hex.
Reviewed By: quark-zju
Differential Revision: D19902595
fbshipit-source-id: 5eb7b9b029e292a02ccc00f5a465ab7807cd56d5
Summary:
Failing means that we fallback to the Python importer. Let's simply warn about
it.
Reviewed By: fanzeyi
Differential Revision: D19897274
fbshipit-source-id: f9c63f5aa76015c28b31f00bba98244f5c86e923
Summary: adds a `get_installed_eden_version_parts` separately from `get_installed_eden_version`, and generally makes `installed_version` mirror then `running_version` parts
Reviewed By: simpkins
Differential Revision: D19846545
fbshipit-source-id: 35bc2e94fd5659e532bb56297d66501b9bb843d8
Summary: The load_limiter was extracted from server/context into its own crate and the server/context itself was refactored into multiple modules, one of which contains facebook-specific code.
Reviewed By: StanislavGlebik
Differential Revision: D19902972
fbshipit-source-id: d577492b4fe01ccfe11b3e092e0521b190516268
Summary: common/sql_ext will now be buildable in OSS
Reviewed By: krallin
Differential Revision: D19876764
fbshipit-source-id: 0f51abd1169f6b8108e7e4cab85b5f193c28e2cd
Summary: Move several pieces of Middleware related to TLS from the LFS server to `gotham_ext`, for reuse in the EdenAPI server. These modules are already well-abstracted and consequently require no modification to be moved out of the LFS server.
Reviewed By: xavierd
Differential Revision: D19890537
fbshipit-source-id: 8de420183e7bd5dd0de10a75c8cfe83825f0c23c
Summary:
Some Mononoke tests test about the backtrace. Having the environment variables
set break those tests. Clear them.
Reviewed By: xavierd
Differential Revision: D19887219
fbshipit-source-id: 529a17282c40730ee95bffbed00070edd1f0823a
Summary:
`mt.exe` can fail when Windows Anti-Virus scans the same file. Retry on such
failures. This hopefully can improve our build success rate on the Windows
platform.
Reviewed By: xavierd
Differential Revision: D19887220
fbshipit-source-id: b725d5fe883bd52697b58b74dbb4a338c02c3ed6
Summary:
This is an attempt to fix the following error:
```
--- test-fb-hgext-treemanifest-server.t
+++ test-fb-hgext-treemanifest-server.t.simplecachestore.err
@@ -167,8 +167,9 @@
$ hg push --to mybook
pushing to ssh://user@dummy/master
searching for changes
- remote: prepushrebase.myhook hook exited with status 1
+ remote: prepushrebase.myhook hook exited with status 2
abort: push failed on remote
+ remote: $TESTTMP/myhook.sh: 2: $TESTTMP/myhook.sh: [[: not found
[255]
```
which I suspect is caused by a non-bash default shell.
Reviewed By: xavierd
Differential Revision: D19887222
fbshipit-source-id: b6fe5d89e4c41ff49fca86da927c4e702ed1e7c1
Summary: On Windows we will verify both Windows and POSIX path separators, because we have both types of paths.
Reviewed By: simpkins
Differential Revision: D19562772
fbshipit-source-id: a6d8280e3bae4f6cd32b1f379bb59e22dd584211
Summary:
This is needed by the next diff. Otherwise Python 3 ssh tests in the next diff
will hang.
Reviewed By: DurhamG
Differential Revision: D19879882
fbshipit-source-id: ecc317d0685993c6b1bef8c72068bf4315030d0f
Summary:
I'm going to change the connection pool logic but I'm not sure where it gets
used. This change exposes at least one test using it.
Reviewed By: xavierd
Differential Revision: D19872614
fbshipit-source-id: 4921b92c3fe3fd7ba1a72de17eef92604964eb2e
Summary: There is a bug in WinStore::getTreeEntry(). It fetches the shared_ptr to the Tree and returns the raw pointer to the entry in the Tree. Using the entry without holding the Tree shared_ptr is unsafe. This bug was introduced while refactoring the code in D19377522.
Reviewed By: simpkins
Differential Revision: D19762486
fbshipit-source-id: 6f0a849f0dc9731bada9a56bdc2a6c18740b0b18
Summary:
The SCS monitoring future should run forever and never return, even if there is
an error.
If an error does occur, we should just log it, and try again next time.
Also convert to new futures.
Reviewed By: krallin
Differential Revision: D19879621
fbshipit-source-id: 45b7097d1c5af61f3c390f6aa6ada5832062fbc9
Summary: This diff extend `Blame` structure with `origin_line` (line at the moment of introduction of a hunk).
Reviewed By: mitrandir77
Differential Revision: D19499125
fbshipit-source-id: 2fa6bc4ee62d484dd6dc6bee17cb1d04ba1db158
Summary:
Regressed since D19702533. This test breakage is only visible if
lfs-test-server exists.
Reviewed By: farnz
Differential Revision: D19872608
fbshipit-source-id: 74ace3eb7363bb1bb773e6b448685e9a3874086f
Summary: The script does not run with the stock python since it depends on edenscm.
Reviewed By: farnz
Differential Revision: D19872953
fbshipit-source-id: 4c5b2d2935a0c9e8cf0a654f541600d7a4fd7211
Summary:
Right now, if the client establishes connection to more than one peer, the
last one to proces `clienttelemetry` wireproto command gets the honor to set
`server_realhostname`. This is not desirable. Specifically, when we have
`fallbackpath` set up for remotefilelog/treemanifest and the prefetch happens
after pulling a commit, we get the hostname of the fallbackpath server,
while losing the hostname of the original `getbundle` server.
Reviewed By: DurhamG
Differential Revision: D19837570
fbshipit-source-id: fdc41565a5dfe670df3caf3b034196c4b7bdf6d9
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
In the next diff I'm going to refactor BlobRepo::get_manifest_from_bonsai()
function and remove IncompleteFilenodes structure. While doing so I noticed
that the behaviour of get_manifest_from_bonsai() and FilenodesOnlyPublic is
different with regards to generating hg filenodes for octopus merges. In
particular, get_manifest_from_bonsai() at the moment doesn't generate filenodes
for paths that came from stepparents, but FilenodesOnlyPublic does generate
them.
I'd say both of this approach are equally reasonable, however let's try to
preserve the behaviour
Reviewed By: krallin
Differential Revision: D19856420
fbshipit-source-id: 9f8ae656205f39536c450b885fc4d8d6a2534456
Summary: It is helpful in CI to be able to remove all eden managed volumes at the end of a job, so that is what this command does.
Reviewed By: simpkins
Differential Revision: D19794482
fbshipit-source-id: d832d093d0a6369a4b533d66afbdce626fa78e2c
Summary:
In the initial iteration of eden_apfs_mount_helper I parsed the human
readable version of the `diskutil apfs list` output to get all of the useful
information from it. Later, I switched to using `diskutil apfs list -plist`
because it simplified the parsing dramatically.
However, it was overlooked that the `mount_point` field was not present in
the plist output. That was usually fine, except in the one case that I
didn't test after changing the parser: after reboot macOS picks a volume
name to remount these volumes. The lack of the mount_point field meant
that we would simply skip the logic that deals with unmounting and remounting
the volume in the correct place.
In order to address this gap we now need to parse the mount table to determine
where the device is mounted, so that is what this diff does.
Reviewed By: simpkins
Differential Revision: D19794478
fbshipit-source-id: ace2df145a46aad7df78c3f4b15fb2198aef3e6f
Summary:
This is done on a sort of best effort basis; the problem
we're aiming to solve (or at least avoid in the majority of cases)
is in these scenarios:
* `buck clean` gives up cleaning as soon as it finds a directory that
it doesn't have permissions to remove.
* `yarn` tries to look for node modules inside the `.Trashes` dir,
which it doesn't have permissions to access
macOS doesn't give us a solid way to indicate that these things should
be disabled at the time that we mount the volume; the various docs
and suggestions online all involve creating marker files to block the
system processes from performing their usual actions, and we could
set those up here, but if we create them with root permissions we'll
trigger the bad behavior in buck and yarn. If we create them with
regular user permissions then eg: running buck clean would remove them
and allow the system to recreate them.
The system will recreate the `.Trashes` directory when the user sends
something to the trash via Finder. Similarly, macOS will recreate
the fsevents directories when an application establishes a watch
on the volume using fsevents.
So we can't permanently prevent this problem, but by deleting these
things at mount time we should at least avoid it bubbling up for
the majority of users, and if those things come back and get in the
way, running `eden redirect unmount ; eden redirect fixup` should
remount and re-remove the problematic things.
Reviewed By: fanzeyi
Differential Revision: D19841514
fbshipit-source-id: f530ab3d68edfa643096bd27efae71c80b505184
Summary: Later in this stack, we're going to have to introduce a few more context types, including Mononoke's per-session `CoreContext` as well as an LFS-server inspired per-request `RequestContext`. To make the scope of the `EdenApiContext` more clear (namely, that unlike the other contexts, this one persists for the entirety of the server's execution), let's rename this to `EdenApiServerContext`. This mirrors the naming convention for these contexts used in the LFS server.
Reviewed By: xavierd
Differential Revision: D19863296
fbshipit-source-id: 8ad785070328523d0beaf824c86c7350ff6a2697
Summary: Use `ServerIdentityMiddleware` from `gotham_ext` in the EdenAPI server to provide server information in the HTTP response headers returned by the server. This is useful for debugging.
Reviewed By: xavierd
Differential Revision: D19863297
fbshipit-source-id: 6ed8bac35e05af580e062423e6f6389a18d17c2a
Summary:
This makes it possible to use `Bytes` for mmap buffers.
The changes are because `minibytes::Bytes` does not implement `From<&[u8]>`
with the intention to make slice copy explicit.
Reviewed By: xavierd
Differential Revision: D19818719
fbshipit-source-id: c34ee451bfd2dc7bcbbcebd52a76444b6c236849
Summary:
EdenFS will now be able to fetch blobs directly from memcache. This won't have
any big benefits as no blobs are in memcache right now, but over time, this
will significantly reduce the cost of fetching blobs.
Reviewed By: fanzeyi
Differential Revision: D19861643
fbshipit-source-id: c2e9d317bd30d4656bf0b3f8897794161697761a
Summary:
This improves scs client ux as you can see in the test.
Cross scheme collisions have been already supported in the code, and there will
be a proper message.
For the ambiguity within a single scheme, I added a message.
Reviewed By: mitrandir77
Differential Revision: D19834142
fbshipit-source-id: a2cff44f6ef031b1224ebf13d38d76c66c9ee4b9
Summary: The exact format of the GPL license header changed when the Mononoke directory moved from `/scm/mononoke` to `/eden/mononoke`. This file ended up with the old header somehow (was created prior to the rename but landed after), so let's fix it to make the linter happy.
Reviewed By: farnz
Differential Revision: D19848640
fbshipit-source-id: 39c5b49850e5a3cba1951bf4e5b813cd08940f01
Summary: Move the generally-useful ServerIdentityMiddleware into gotham_ext so we can use it in the EdenAPI server.
Reviewed By: xavierd
Differential Revision: D19845282
fbshipit-source-id: 3a01b15dc64cee99cefafcdac229c0b70a4db683
Summary:
These tracing points will help us understand the memcache hit rate as well as
the fetching speed.
Reviewed By: quark-zju
Differential Revision: D19836499
fbshipit-source-id: 1936c44efc3e7715069e6a959f5331139d591d5c
Summary:
Everytime a cache miss is seen, the data fetched from the server will be sent
directly to memcache for future use. Unfortunately, doing so in a blocking
manner severely impact the overall fetching speed from the server. Since
memcache is purely an optimization, we can afford to send data to it
asynchronously.
Let's move as much as possible of the code to a background thread to reduce the
overhead of memcache.
Reviewed By: DurhamG
Differential Revision: D19836011
fbshipit-source-id: 68e506ef7464d6e99d98457d0d37178f514be1a9
Summary:
Instead of fetching data one-by-one, let's prefetch data concurrently by using
the new get_iter function.
Reviewed By: DurhamG
Differential Revision: D19836009
fbshipit-source-id: 4a50328c0cbbba677c2de3777ebe4c34cb10c1e2
Summary: remove the need to pass mapping to `::derive` method
Reviewed By: StanislavGlebik
Differential Revision: D19856560
fbshipit-source-id: 219af827ea7e077a4c3e678a85c51dc0e3822d79
Summary: Test that mercurial can accept a unicode commit message.
Reviewed By: xavierd
Differential Revision: D19838221
fbshipit-source-id: b6333e587004b358a3883db70e40d2f32af4da29
Summary:
Currently admission to fastreplay is controlled at a global level - a
percentage of all repos being replayed are admitted without the ability to turn
off an individual repo.
Update fastreplay to use a config option that controls which repos should be skipped.
Reviewed By: krallin
Differential Revision: D19835304
fbshipit-source-id: 63b7c66b35eaa2cb1370ad96b1c6e98a2c93b712
Summary: Rechunker uses tokio 0.1.0 and legacy futures 0.1.0 . This diff changes rechunker.rs to use new std::future and tokio 0.2.0
Reviewed By: farnz
Differential Revision: D19801966
fbshipit-source-id: 3ba7936e8981c87906a881216ffd66332cdcc761
Summary:
This will allow us to distinguish `getbundle` for a normal `pull` from the one
for infinitepush pull.
Reviewed By: StanislavGlebik
Differential Revision: D19833206
fbshipit-source-id: 86534320fbb4d60bac04d458a0953701201cba87
Summary:
Even when memcache would be able to prefetch everything, this would always call
into the underlying remote store with an empt key set. For things like `hg
prefetch` and a large number of keys, the effect of doing that is minimum, but
for EdenFS or `hg log -p`, the roundtrip to the server for every file/revision
would add a significant amount of overhead. Let's simply stop iterating when we
no longer need to fetch anything.
Reviewed By: DurhamG
Differential Revision: D19835797
fbshipit-source-id: 54ad704428c3b20d973cfa87f7171899ec44b3f9
Summary:
Don't rebuild the memcache store everytime. This avoid spawning a thread and
reconnecting to memcache everytime.
Reviewed By: DurhamG
Differential Revision: D19797911
fbshipit-source-id: 7ecfa111a98ab67c4791b91410a22e458fe169f6
