Summary:
This updates fastreplay to support sampling of the samples we log to Scuba.
This lets us control the volume we log to Scuba.
Reviewed By: farnz
Differential Revision: D19722223
fbshipit-source-id: 2e43f201a3e5930f5f6a29749c35e0e0dea341d2
Summary:
This adds support for sampling scuba samples, and assigns their weight properly
so that Scuba will report the right number of hits. The sampling is determined
at the point where `sampled()` is called, which means we can e.g. create a
Scuba sample and thread it through for a whole request and either everything
will be sampled, or nothing.
I haven't refactored our Multiplex blob to use this just yet, since it's
architecturally a bit trickier to use this there.
Note: I'll rebase on D19599013 if we land that first.
Reviewed By: ahornby
Differential Revision: D19722224
fbshipit-source-id: b5464937a29f73868fbdc8396d507bac1555af78
Summary:
Use `abc.ABC` from Python 3 stdlib directly. The definition matches
`pycompat3.py`:
class ABC(metaclass=ABCMeta):
pass
The following changes are reverted since they're no longer necessary:
D19732319 "[hg] py3: fix windows build"
D19703778 "[hg] py3: exclude mercurial/pycompat3.py from Python 2 builds"
D19703779 "[hg] py3: exclude pycompat3.py from Buck-based Python 2 builds"
Reviewed By: simpkins, singhsrb
Differential Revision: D19739075
fbshipit-source-id: 8c1e3727e8a88ff5f7232270d528d690523b1824
Summary:
Printing via `ui` object instead of directly to the `stderr` makes the
output more consistent.
Reviewed By: quark-zju
Differential Revision: D19738355
fbshipit-source-id: 384bf2de11f2ae7b5264ec5d94b041502c4ef7fb
Summary:
The library already has a way to wrap a Python object into a Rust object that
exposes the Rust Read/Write interface. This is the reverse direction for
the Write interface.
The initial intention is to expose Rust stdout as described in D19702533.
However, I found Python's `sys.stdout.buffer` also enforces utf-8 encoding
on Windows (unless PYTHONLEGACYWINDOWSSTDIO is set). So Python's
stdout actually behaves similarly with Rust's stdout on Windows and is okay
to use. That said, it's still useful to have this abstraction, for streampager [1]
integration.
[1]: https://github.com/markbt/streampager/
Reviewed By: sfilipco
Differential Revision: D19716127
fbshipit-source-id: ba39898122561d9a49b7080ee95d7c940540eb40
Summary:
1fb027d759
changed the kernel behavior to reject reads smaller than 8KB,
even for requests that would never need to be that large.
That causes eden to fail to start up on eg: Fedora 31 with a 5.4 kernel.
This commit adds some padding to satisfy this new check.
Reviewed By: chadaustin
Differential Revision: D19736893
fbshipit-source-id: 926456d72124b186976ee9a8a21242e93c26f790
Summary:
On python 2, os.environ is expected to be bytes, so let's remove the
forced decoding. This broke run-tests.py on Windows.
Reviewed By: singhsrb
Differential Revision: D19737218
fbshipit-source-id: d4782eea279275f57cc91bc412a2d74857002cc1
Summary: Add a handler to the EdenAPI server for basic request routing.
Reviewed By: xavierd
Differential Revision: D19726776
fbshipit-source-id: 906a538ebfd48532c2488feb9f99747690f3351b
Summary: Make the LFS server depend on `gotham_ext` for its Middleware trait and Handler struct. The code itself was previously copied from here, so there should be no functional changes.
Reviewed By: krallin
Differential Revision: D19714661
fbshipit-source-id: 64f3b230c97219c7419cec4e92d30c2ee537c724
Summary:
A lot of the code in the LFS server is generally useful for any Gotham-based HTTP server, so let's create a place to put the common pieces. Mononoke's HTTP-based services like the LFS server and EdenAPI server can then depend on this common crate.
In this initial diff, I've simply factored out the `Middleware` trait, as well as the custom Handler struct, which right now appears to simply be a mechanism to run the middleware before and after each request.
Given that none of this code appears to be LFS-specific, I hope this is reasonable. I'm open to suggestions for naming things; for now I've named the new crate `gotham_ext` since it's built on Gotham, but I could see us using another name if we intend to move in the direction of plain `hyper` in the future.
In later diffs, I intend to start moving over some of the more general Middleware from the LFS server to the new crate (though LFS-specific Middleware will remain a part of the LFS server code).
Reviewed By: krallin
Differential Revision: D19714662
fbshipit-source-id: 21494d18be659a8ec399c61eb22ee10185efcc14
Summary:
We added pycompat3.py which only parses in Python 3. We need to
exclude it from the Windows build as well.
Unfortunately, the Windows build users PyZipFile.writepy which writes an entire
directory. Let's copy in the implementation from Python 3 which allows us to
filter out certain files.
Reviewed By: xavierd
Differential Revision: D19732319
fbshipit-source-id: 4cebb434d052c51d9ae0dccdec5eadd1f412f9e5
Summary: More replacements of dict.iteritems with pycompat.iteritems(dict) for py3 compatibility.
Reviewed By: singhsrb
Differential Revision: D19704211
fbshipit-source-id: 544cb292c0241a293fc4ab83c3d9472dcf4328b6
Summary:
`timed` has been replaced by the `timefunction` in util.
I have also removed the existing usages because they seemed relevant only
during the development.
Reviewed By: quark-zju
Differential Revision: D19727919
fbshipit-source-id: 58c466f02ac2a5bf7096948b17b49ceb26fc1fd9
Summary:
The path normalization OSX logic was using unicode, and when it
switched to use pycompat in D19613690 it stopped using unicode, which broke it.
Let's roll it back.
Reviewed By: singhsrb
Differential Revision: D19726322
fbshipit-source-id: fa7bfacdf41dfedbcfef2fdb5ccb826196101e61
Summary:
This diff causes problems when new code (i.e. the one that generated linknodes
during hg changeset generation) and old code (i.e. the one that doesn't
generate linknodes) run at the same time.
Reviewed By: ikostia
Differential Revision: D19720991
fbshipit-source-id: b2278543020ed8e356b0ba95b61e1c91f5cd5de7
Summary:
We start Fastreplay with some cache warmup. During this time, it doesn't make
sense to go and start ptail, since we're not ready to process its output.
Instead, let's start ptail when we're ready. Piping from stdin is still
possible by not providing a command at all.
Reviewed By: ahornby
Differential Revision: D19723470
fbshipit-source-id: 64f58d9eced4a68fd8238b03b9234f3569215237
Summary:
At the moment missing linknodes causes cache warmup to fail. While missing
linknode is not great, failing to start up is much worse - we can make Mononoke
completely unusuable.
This diff instead adds logging of missing filenodes. It also makes sure that we
derive filenodes before using them.
Reviewed By: ikostia
Differential Revision: D19722924
fbshipit-source-id: 10d23506731c45cfaaadee911be62189104c8036
Summary:
This makes it possible to use `--debug` to enter ipdb interactive debug
sessions on crash (which was a new run-tests.py feature added by
D19581174).
For example, I was able to use it to debug a crash in test-lfs-to-mononoke.t
when debugging D19702533:
Traceback (most recent call last):
...
File "buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/ui.py", line 648, in _write
self.fout.write(encodeutf8("".join(msgs)))
File "buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/pycompat.py", line 168, in encodeutf8
assert isinstance(s, bytes)
AssertionError
> buck-out/dev/gen/eden/scm/__hg__/runtime_resources/__default__/eden/scm/hglib/edenscm/mercurial/pycompat.py(168)encodeutf8()
167 # type: (bytes) -> bytes
--> 168 assert isinstance(s, bytes)
169 return s
ipdb> up
ipdb> p msgs
(u'lfs: uploading e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d (1.95 KB)\n',)
ipdb> up (a few times)
ipdb> p obj
{u'oid': u'e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d', u'size': 2000, u'authenticated': False, u'actions': {u'upload': {u'href': u'http://localhost:46705/repo/upload/e2fff2ce58d585b4b0572e0a323f9e7e5f98cc641489e12c03c401d05d0e350d/2000'}}}
Without this diff, the `ipdb>` prompt will be ended immediately since it's
reading from /dev/null.
Reviewed By: krallin
Differential Revision: D19709731
fbshipit-source-id: 76cefbf192c4592884e05afee81146963954c5d6
Summary:
The underlying Thrift client code is already async / await, but the layer on
top of it is not. This isn't ideal, since it results in a lot of undesirable
clones happening all over the place (e.g. every key is cloned multiple times).
This fixes that by updating the wrapper code to async / await.
Reviewed By: farnz
Differential Revision: D19695708
fbshipit-source-id: b4c1b6fd6bf4d998ca6187c4cdde67294329d181
Summary:
Add a few additional type annotations on some functions.
This did require some minor code restructuring in `filestore.setfile()` to
work around pyre's unwillingness to unwrap `Optional` member variables.
Reviewed By: quark-zju
Differential Revision: D19715243
fbshipit-source-id: 8078329c1e4a50ad0aa6c765d42a89b5ed58e7bf
Summary:
Add more type annotations to vfs.py and some of the platform-specific
functions that it calls.
This did catch a couple of cases where string/bytes conversion were not being
done properly in Python 3.
Reviewed By: quark-zju
Differential Revision: D19678717
fbshipit-source-id: b3979cc5e0668ddbb93372dee25ef02ea1867d6f
Summary:
Fix various type issues when running chg under Python 3.
Enable chg in setup3.py build.
This should make tests run faster. For example, test-rebase-detach.t
now completes in 8 seconds, down from 29 seconds.
Reviewed By: xavierd
Differential Revision: D19702535
fbshipit-source-id: 8928b1b920b9b52fd03dc86f996da18f2405f146
Summary:
Drop stdoutbytes/stdinbytes. They make things unnecessarily complicated
(especially for chg / Rust dispatch entry point).
The new idea is IO are using bytes. Text are written in utf-8 (Python 3) or
local encoding (Python 2). To make stdout behave reasonably on systems not
using utf-8 locale (ex. Windows), we might add a Rust binding to Rust's stdout,
which does the right thing:
- When writing to stdout console, expect text to be utf-8 encoded and do proper decoding.
- Wehn writing to stdout file, write the raw bytes without translation.
Note Python's `sys.stdout.buffer` does not do translation when writing to stdout console
like Rust's stdout.
For now, my main motivation of this change is to fix chg on Python 3.
Reviewed By: xavierd
Differential Revision: D19702533
fbshipit-source-id: 74704c83e1b200ff66fb3a2d23d97ff21c7239c8
Summary:
My earlier refactor moved a watchmanclient.getcurrentclock() call out
of a "try/except Exception" block. This meant any watchman errors broke the hg
status entirely, instead of falling back to the old status path.
Let's add the logic around this watchman path.
Reviewed By: sfilipco
Differential Revision: D19709391
fbshipit-source-id: bfb0221ba405a926babddaf73c8e70924e4d9b25
Summary:
Implementation based on what we do in bcm layer
for reporting (non histogram) port stat counters. Abstracting
this lib out so we can reuse this code in both SAI and bcm
layers. Plus pulling this out of BcmPort makes it much
easier to unit test this code.
Reviewed By: boryas
Differential Revision: D19676641
fbshipit-source-id: 88aa20f20339729f5387500bcf6084871f4e7caa
