Summary:
This enables an in-process memcache client for the Rust
ContentStore/MetadataStore. For now, this implementation is lacking several
necessary optimization:
- Start-up time is always slowed down by ~0.7s, the initialization will be
moved to a background thread
- Writing data to memcache is blocking and will be moved to a background
thread too.
- Prefetching data does a roundtrip to memcache for every key, batching
memcache APIs will be added.
Compared to the existing hg_memcache_client, this implementation is both
significantly shorter and do not exhibit some of the pathological behavior of
having to flush the indexedlog for every fetched blob when used in Eden.
Reviewed By: DurhamG
Differential Revision: D19518696
fbshipit-source-id: 4725447d13e7eddd9586135c2511e13ddb921771
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
Partially backport upstream
https://www.mercurial-scm.org/repo/hg/rev/f81c17ec303c to enable lazy loading
of python code contained in edenscmdeps3.zip.
Also, temporarily disabling the demandimport on Python3 is a bit tricky, for
the reasons mentioned in the deactivated function. Thus, instead of using the
disabled function, let's use the deactivated one.
Reviewed By: DurhamG
Differential Revision: D19672866
fbshipit-source-id: c9e39ed044121d962af1cc46745bdec72629c579
Summary:
cpptoml has traversal functionality for table reads, but not for
writes. Add a helper function for reading a config value and updating
the TOML table if it's unset.
Reviewed By: fanzeyi
Differential Revision: D19671264
fbshipit-source-id: e2b78d338af35d51fddaa258b7f45f8966d00a26
Summary: This diff updates the states transitions in the EdenMount on Windows. It starts as State::UNINITIALIZED and transitions to State::RUNNING when the start is called. It will transition to State::SHUT_DOWN on stop or destroy. Destroy will put it in State::DESTROYING, from which it should not return.
Reviewed By: chadaustin
Differential Revision: D19559271
fbshipit-source-id: d76983cab610cb9b2c896807cf1fe49c357f8095
Summary:
The Mercurial convert extension passes around parameters to indicate a commit that needs to be converted from source to sink. For existing converters like Git, this is a simple 1:1 conversion: a commit in the source gets mapped to a commit in the sink, and so they use the source commit hash (sometimes called rev or version in the API) to represent the commit to be converted.
Our converter is much more complicated. Source commits get converted multiple times to account for different ways of mounting it into the destination file system and commit history. The commits are also coming from multiple source Git projects. This means that we need multiple pieces of data to represent a single commit conversion action.
Thus far, we've been trying to meet part of this need by using concatenated strings of (variant, commithash). This logic is breaking down as we add more fields. This commit adds a new immutable object called "conversionrevision" that represents the (variant, source commit hash, source project name, destination path) that is the unique identifier for the individual commit conversions we need to perform. This commit also includes logic for serializing and deserializing these objects as strings (useful because the converter seems to require commit IDs to be strings) and unit tests for all of the new logic.
Reviewed By: tchebb
Differential Revision: D19606867
fbshipit-source-id: 77815ca858f841d452874e95dfa3b351bafde306
Summary:
When I removed an hggit test case from this test last week, it caused
it to stop being skipped and therefore runs on Windows. The filterpwd magic
doesn't work there, and it's unnecessary, so let's just drop it.
Reviewed By: singhsrb, xavierd
Differential Revision: D19744329
fbshipit-source-id: 21f5c67d4fa7a61f14bbacd78756e5397fd6c819
Summary:
This updates fastreplay to support sampling of the samples we log to Scuba.
This lets us control the volume we log to Scuba.
Reviewed By: farnz
Differential Revision: D19722223
fbshipit-source-id: 2e43f201a3e5930f5f6a29749c35e0e0dea341d2
Summary:
This adds support for sampling scuba samples, and assigns their weight properly
so that Scuba will report the right number of hits. The sampling is determined
at the point where `sampled()` is called, which means we can e.g. create a
Scuba sample and thread it through for a whole request and either everything
will be sampled, or nothing.
I haven't refactored our Multiplex blob to use this just yet, since it's
architecturally a bit trickier to use this there.
Note: I'll rebase on D19599013 if we land that first.
Reviewed By: ahornby
Differential Revision: D19722224
fbshipit-source-id: b5464937a29f73868fbdc8396d507bac1555af78
Summary:
Use `abc.ABC` from Python 3 stdlib directly. The definition matches
`pycompat3.py`:
class ABC(metaclass=ABCMeta):
pass
The following changes are reverted since they're no longer necessary:
D19732319 "[hg] py3: fix windows build"
D19703778 "[hg] py3: exclude mercurial/pycompat3.py from Python 2 builds"
D19703779 "[hg] py3: exclude pycompat3.py from Buck-based Python 2 builds"
Reviewed By: simpkins, singhsrb
Differential Revision: D19739075
fbshipit-source-id: 8c1e3727e8a88ff5f7232270d528d690523b1824
Summary:
Printing via `ui` object instead of directly to the `stderr` makes the
output more consistent.
Reviewed By: quark-zju
Differential Revision: D19738355
fbshipit-source-id: 384bf2de11f2ae7b5264ec5d94b041502c4ef7fb
Summary:
The library already has a way to wrap a Python object into a Rust object that
exposes the Rust Read/Write interface. This is the reverse direction for
the Write interface.
The initial intention is to expose Rust stdout as described in D19702533.
However, I found Python's `sys.stdout.buffer` also enforces utf-8 encoding
on Windows (unless PYTHONLEGACYWINDOWSSTDIO is set). So Python's
stdout actually behaves similarly with Rust's stdout on Windows and is okay
to use. That said, it's still useful to have this abstraction, for streampager [1]
integration.
[1]: https://github.com/markbt/streampager/
Reviewed By: sfilipco
Differential Revision: D19716127
fbshipit-source-id: ba39898122561d9a49b7080ee95d7c940540eb40
Summary:
1fb027d759
changed the kernel behavior to reject reads smaller than 8KB,
even for requests that would never need to be that large.
That causes eden to fail to start up on eg: Fedora 31 with a 5.4 kernel.
This commit adds some padding to satisfy this new check.
Reviewed By: chadaustin
Differential Revision: D19736893
fbshipit-source-id: 926456d72124b186976ee9a8a21242e93c26f790
Summary:
On python 2, os.environ is expected to be bytes, so let's remove the
forced decoding. This broke run-tests.py on Windows.
Reviewed By: singhsrb
Differential Revision: D19737218
fbshipit-source-id: d4782eea279275f57cc91bc412a2d74857002cc1
Summary: Add a handler to the EdenAPI server for basic request routing.
Reviewed By: xavierd
Differential Revision: D19726776
fbshipit-source-id: 906a538ebfd48532c2488feb9f99747690f3351b
Summary: Make the LFS server depend on `gotham_ext` for its Middleware trait and Handler struct. The code itself was previously copied from here, so there should be no functional changes.
Reviewed By: krallin
Differential Revision: D19714661
fbshipit-source-id: 64f3b230c97219c7419cec4e92d30c2ee537c724
Summary:
A lot of the code in the LFS server is generally useful for any Gotham-based HTTP server, so let's create a place to put the common pieces. Mononoke's HTTP-based services like the LFS server and EdenAPI server can then depend on this common crate.
In this initial diff, I've simply factored out the `Middleware` trait, as well as the custom Handler struct, which right now appears to simply be a mechanism to run the middleware before and after each request.
Given that none of this code appears to be LFS-specific, I hope this is reasonable. I'm open to suggestions for naming things; for now I've named the new crate `gotham_ext` since it's built on Gotham, but I could see us using another name if we intend to move in the direction of plain `hyper` in the future.
In later diffs, I intend to start moving over some of the more general Middleware from the LFS server to the new crate (though LFS-specific Middleware will remain a part of the LFS server code).
Reviewed By: krallin
Differential Revision: D19714662
fbshipit-source-id: 21494d18be659a8ec399c61eb22ee10185efcc14
Summary:
We added pycompat3.py which only parses in Python 3. We need to
exclude it from the Windows build as well.
Unfortunately, the Windows build users PyZipFile.writepy which writes an entire
directory. Let's copy in the implementation from Python 3 which allows us to
filter out certain files.
Reviewed By: xavierd
Differential Revision: D19732319
fbshipit-source-id: 4cebb434d052c51d9ae0dccdec5eadd1f412f9e5
Summary: More replacements of dict.iteritems with pycompat.iteritems(dict) for py3 compatibility.
Reviewed By: singhsrb
Differential Revision: D19704211
fbshipit-source-id: 544cb292c0241a293fc4ab83c3d9472dcf4328b6
Summary:
`timed` has been replaced by the `timefunction` in util.
I have also removed the existing usages because they seemed relevant only
during the development.
Reviewed By: quark-zju
Differential Revision: D19727919
fbshipit-source-id: 58c466f02ac2a5bf7096948b17b49ceb26fc1fd9
Summary:
The path normalization OSX logic was using unicode, and when it
switched to use pycompat in D19613690 it stopped using unicode, which broke it.
Let's roll it back.
Reviewed By: singhsrb
Differential Revision: D19726322
fbshipit-source-id: fa7bfacdf41dfedbcfef2fdb5ccb826196101e61
Summary:
This diff causes problems when new code (i.e. the one that generated linknodes
during hg changeset generation) and old code (i.e. the one that doesn't
generate linknodes) run at the same time.
Reviewed By: ikostia
Differential Revision: D19720991
fbshipit-source-id: b2278543020ed8e356b0ba95b61e1c91f5cd5de7
Summary:
We start Fastreplay with some cache warmup. During this time, it doesn't make
sense to go and start ptail, since we're not ready to process its output.
Instead, let's start ptail when we're ready. Piping from stdin is still
possible by not providing a command at all.
Reviewed By: ahornby
Differential Revision: D19723470
fbshipit-source-id: 64f58d9eced4a68fd8238b03b9234f3569215237
Summary:
At the moment missing linknodes causes cache warmup to fail. While missing
linknode is not great, failing to start up is much worse - we can make Mononoke
completely unusuable.
This diff instead adds logging of missing filenodes. It also makes sure that we
derive filenodes before using them.
Reviewed By: ikostia
Differential Revision: D19722924
fbshipit-source-id: 10d23506731c45cfaaadee911be62189104c8036
Summary:
This makes it possible to use `--debug` to enter ipdb interactive debug
sessions on crash (which was a new run-tests.py feature added by
D19581174).
For example, I was able to use it to debug a crash in test-lfs-to-mononoke.t
when debugging D19702533:
Traceback (most recent call last):
...
File "buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/ui.py", line 648, in _write
self.fout.write(encodeutf8("".join(msgs)))
File "buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/pycompat.py", line 168, in encodeutf8
assert isinstance(s, bytes)
AssertionError
> buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/pycompat.py(168)encodeutf8()
167 # type: (bytes) -> bytes
--> 168 assert isinstance(s, bytes)
169 return s
ipdb> up
ipdb> p msgs
(u'lfs: uploading e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d (1.95 KB)\n',)
ipdb> up (a few times)
ipdb> p obj
{u'oid': u'e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d', u'size': 2000, u'authenticated': False, u'actions': {u'upload': {u'href': u'http://localhost:46705/repo/upload/e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d/2000'}}}
Without this diff, the `ipdb>` prompt will be ended immediately since it's
reading from /dev/null.
Reviewed By: krallin
Differential Revision: D19709731
fbshipit-source-id: 76cefbf192c4592884e05afee81146963954c5d6
Summary:
The underlying Thrift client code is already async / await, but the layer on
top of it is not. This isn't ideal, since it results in a lot of undesirable
clones happening all over the place (e.g. every key is cloned multiple times).
This fixes that by updating the wrapper code to async / await.
Reviewed By: farnz
Differential Revision: D19695708
fbshipit-source-id: b4c1b6fd6bf4d998ca6187c4cdde67294329d181
