Summary:
On Windows, `pathlib.Path.resolve` has an use of uninitialized memory bug in Python 3.6.2. Lego Windows has this version of Python 3 installed.
When `eden prefetch` runs, it will first attempt to read the config file at the root of the repository (`$REPOROOT/.eden/config`). Because of this bug, when we normalize the path in edenfsctl, it will get some random bytes as a result. This subsequently causes `toml.read` to fail as it is unable to read a path containing NUL byte.
As you can see, this is the same exception as we see on Windows:
```
In [1]: open("test\x00")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-57764b20d660> in <module>
----> 1 open("test\x00")
ValueError: embedded null byte
```
Reviewed By: xavierd
Differential Revision: D22177997
fbshipit-source-id: ab2565c8946d809bc15bc1597b749fb5e9440ca0
Summary:
The dynamicconfig logic had an assert checking that a config was not
added by the dynamicconfig twice. This isn't actually a valid assert since the
caller could load the config as many times as it wanted.
This happened in hg doctor, where it loaded the ui once manually (without a repo
object) then it loaded it again during the creation of the repo object. I went
ahead and cleaned up hg doctor to not do this, but let's get rid of the assert
anyway.
Reviewed By: quark-zju
Differential Revision: D22194273
fbshipit-source-id: 9491d35fe14523ad3d9cb69b4ca0d615149dd0f0
Summary:
D22009699 recently introduced some logic that access repo.localvfs
during repo setup. This doesn't exist for peer repos, so let's exist early if
this is a peer.
Reviewed By: xavierd
Differential Revision: D22197586
fbshipit-source-id: 8f717aa21a0b45f8f85a1ff800ee4a1f86b94bdf
Summary: This makes it easier to run tests with non-buck build.
Reviewed By: sfilipco
Differential Revision: D22174992
fbshipit-source-id: 11100d608cb78973ae7a63056667f0be4a86ae00
Summary: This might make it easier for us to see what processes are hanging.
Reviewed By: sfilipco
Differential Revision: D22009696
fbshipit-source-id: a1c1f97976084160a1b4e4a3ff7d2f9781240f6f
Summary:
There are lots of "hanging" questions in the support group. While not all of
them are our fault (ex. mergedriver), we don't have an easy reply to tell
users what's actually going on.
This diff adds a way to write sigtrace periodically so we can include it in
rage output.
Reviewed By: sfilipco
Differential Revision: D22009699
fbshipit-source-id: 5349f613b08507ed02cc0e3fa00963fd7d4c3d21
Summary:
A lot of extensions wrap runcommand just to get the command name.
Let's define a proper API for it - ui.cmdname provides the resolved
command name.
Reviewed By: sfilipco
Differential Revision: D22009694
fbshipit-source-id: 4b8744ee5314924ea09c9f325570ce53c849fef5
Summary: The tracing APIs and error context APIs can achieve similar effects.
Reviewed By: xavierd
Differential Revision: D22129585
fbshipit-source-id: 0626e3f4c1a552c69c046ff06ac36f5e98a6c3d8
Summary:
`eden top` often adds extra quotes to the end of the commands in the process
table, this removes those pesky quotes
Reviewed By: genevievehelsel
Differential Revision: D21440316
fbshipit-source-id: f1200a28a5345691fcce517526d119f44e5993d0
Summary: Make test-test-commit-interactive.t python 3 compatible. Note I wasn't sure if it was worthwhile to spend too much time on the translation portion of the test. If someone objects to the deletion, I can spend more time on it.
Reviewed By: DurhamG
Differential Revision: D22122960
fbshipit-source-id: d232971581fca0532ed05c8ca1fdb7b5e8bd99e0
Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
We have seen that eden will unexpectedly fetch data, we want to know why.
This adds the plumbing to interact with edens current logging to be able to
log when eden fetches data from the server and what caused eden to do this
fetch. Later changes will use the classes created here to log the cause of data
fetches.
Reviewed By: chadaustin
Differential Revision: D22051013
fbshipit-source-id: 27d377d7057e66f3e7a304cd7004f8aa44f8ba62
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: Let's not heal 10000 blobs in parallel, that's a little too much data.
Reviewed By: farnz
Differential Revision: D22186543
fbshipit-source-id: 939fb5bc83b283090e979ac5fe3efc96191826d3
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: This makes it easier to understand why we're fetching so much data.
Reviewed By: quark-zju
Differential Revision: D22114905
fbshipit-source-id: 7cb6d5c9aebcc8c9089891e030b02176e208bd0f
Summary: This got broken in D22115015 — this fixes it.
Reviewed By: farnz
Differential Revision: D22186138
fbshipit-source-id: 54c05466cdbd3be4f6887a852f099351ea5e891e
Summary: This will be used in the following diffs. It just adds commitsync fixtures in a single place, so that we can later play with them in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21952665
fbshipit-source-id: 2933a9f7ea8343d5d52e6c3207e7d78a3ef0be25
Summary:
We had to fork OpenNSA and clone from it, see D19437386 for details.
Broadcom has now started hosting OpenNSA as a tarball.
Thus, we no longer need to maintain a fork (yay!)
This patch points opennsa manifest to fetch opennsa from this new location.
Reviewed By: bkoray
Differential Revision: D22175932
fbshipit-source-id: 51cd777ab836e4f191d78fbb2312925e446ca38f
Summary: This bug can be triggered when your computer name contains emoji. getdeps.py will fail to create this file due to Python attempts to write the file as cp1252 (Windows's default encoding)
Reviewed By: wez
Differential Revision: D22171935
fbshipit-source-id: fc3be2d1050c17ddbe05a0fc91d6613865f092ce
Summary:
This test appears to fail when run in hgbuild. It's not clear why it
doesn't repro locally, but it appears to be a locale issue, so let's just for it
to use C.
Reviewed By: singhsrb
Differential Revision: D22178365
fbshipit-source-id: 007c86681ff1bd4845b1feb41b7bbb031edf172b
Summary:
The server response is not strictly typed. Accessing fields might error out.
Protect against those errors.
Reviewed By: chadaustin
Differential Revision: D22169401
fbshipit-source-id: 4e57220d03c322b5b40d81ac63f4391e286525e4
Summary:
On platform the code is expected to run (Windows), the performance difference
is significant (ex. 1.3s -> 0.8s).
The following benchmark emulates truncating the last 4 revisions from changelog:
```
# hg debugshell on Windows
In [2]: clsize = repo.svfs.stat("00changelog.i").st_size
In [3]: def f1():
...: util.tryunlink(repo.svfs.join("new"))
...: shutil.copyfile(repo.svfs.join("00changelog.i"), repo.svfs.join("new"))
...: with repo.svfs.open("new","a") as f:
...: f.truncate(clsize - 256)
In [5]: def g1():
...: util.tryunlink(repo.svfs.join("new"))
...: with repo.svfs.open("00changelog.i") as f1, repo.svfs.open("new", "w") as f2:
...: size = clsize - 256
...: while size > 0:
...: bufsize = min(size, 1 << 24)
...: f2.write(f1.read(bufsize))
...: size -= bufsize
In [6]: %timeit f1()
1 loop, best of 3: 1.13 s per loop
In [7]: %timeit g1()
1 loop, best of 3: 716 ms per loop
In [8]: %timeit f1()
1 loop, best of 3: 1.51 s per loop
In [9]: %timeit g1()
1 loop, best of 3: 853 ms per loop
In [30]: %timeit f1()
1 loop, best of 3: 1.28 s per loop
In [31]: %timeit g1()
1 loop, best of 3: 755 ms per loop
```
I also removed the `vfs.exists` test as the function should raise if the file
does not exist.
Reviewed By: DurhamG
Differential Revision: D21974771
fbshipit-source-id: 358579c057eb694ce6e5b135cf95067dbc66f6f3
Summary: Add the EdenAPI server to the `hg_mononoke_run_tests` test target. This will make it possible to start an EdenAPI server in Mercurial's Buck tests.
Reviewed By: quark-zju
Differential Revision: D22147098
fbshipit-source-id: 1e29b7c06f1dd74f26d821943cd44ef09a7d505f
Summary: `HgPhase` type is redundant and was adding dependency on mercurial in phases crate.
Reviewed By: farnz
Differential Revision: D22162716
fbshipit-source-id: 1c21841d34897d0072ff6fe5e4ac89adddeb3c68
Summary: In advance of peppering async everywhere, allow subcmd's run method to be async.
Reviewed By: genevievehelsel
Differential Revision: D21892187
fbshipit-source-id: f611faacf95649d8bb5588aeefc4546bd5f63984
Summary:
During expensive getbundle request traversing skiplist uses a lot of cpu, and
in fact it's blocking the whole CPU. krallin suggested to yield since it
should avoid blocking cpus, and that's what this diff is doing.
Reviewed By: krallin
Differential Revision: D22160477
fbshipit-source-id: 5bd881d5c50f4d8e64f2cc90389abc8568ec1df6
Summary:
Fix error "AttributeError: module 'concurrent.futures' has no attribute
'_base'" when loading asyncio.base_futures module. There very well may be a more appropriate fix, but this allows ipdb python3 debugging.
Reviewed By: quark-zju
Differential Revision: D22144225
fbshipit-source-id: eb6a96f79c97e3bd40c1efd9fc75660d6fc372ea
