Summary:
If the nodemap file is seriously out of sync, it's harder to recover on Windows,
because nodemap mmap in other processes prevents `atomic_write`.
Use `util::path::remove_file` to remove the old mmap files and improve the
success rate of nodemap rewrites.
Reviewed By: xavierd
Differential Revision: D22168134
fbshipit-source-id: 577df978b3175eac3257794f5e5ff7522cf21871
Summary:
Make it possible to delete files that are mmap-ed. This avoids the extra
complexity of cleaning leftover files.
Reviewed By: xavierd
Differential Revision: D22168135
fbshipit-source-id: edb33a75638c668945feae49608c3fff25e742a4
Summary:
Update snapshot command for python3 compatibility.
This one was a bit weird, and there are some future changes we could make if we care. First, our CBOR implementation doesn't support CBOR strings (utf8 strings), only CBOR bytestrings (just byte sequences). This would be easy to add, but would be binary-incompatible with previous snapshots, if we made no other changes.
We also don't take full advantage of the supported types we already have, ie we encoded version (an integer in Python) as a string, then wrote that string as a CBOR bytestring, which makes the very awkward encodeutf8(str(version)) you see in this change, instead of just plain version, which would work round-trip perfectly fine as far as I can tell, but would break compatibility with existing snapshots.
As is, I made only the most conservatve change, Python2 snapshots should be binary-identical to Python3 snapshots, and they should both be compatible with snapshots made before this change. We could get rid of the vast majority of the conversions by simply adding CBOR string support, which would be a breaking change without special handling.
Reviewed By: quark-zju
Differential Revision: D22195085
fbshipit-source-id: cead42db0c5f84b99e305e5a7f1d839a35becac6
Summary:
Move utility function _getoptbwrapper out of Python2-gated block.
Convert undo.py to use integer division in both python2 and python3.
Reviewed By: xavierd
Differential Revision: D22125861
fbshipit-source-id: 9a283065c98eb122c4488285870e026748257b3c
Summary:
With the Rust stores being enabled by default in the code, time to actually
remove the code that wasn't using it. A lot of code still remains as the
treemanifest code still uses it. Migrating the treemanifest code to use the
Rust store would allow for even more code to be removed!
Reviewed By: DurhamG
Differential Revision: D21987918
fbshipit-source-id: 8e0becd4a1fb2ac81e26b8517d13e6d06ab06f65
Summary:
Now that the workers are in Rust, we no longer need the forker version in
Python. For now, the Python LFS extension still uses the threaded worker so
keep this one for now, when that extension will be removed we can remove the
rest of the worker code.
In theory, not all repository would benefit from the Rust workers, but these
are not supported at FB due to not being remotefilelog based.
Reviewed By: DurhamG
Differential Revision: D21987295
fbshipit-source-id: d17b9730651671608cf13f7abe6a9bb32251e140
Summary:
The Rust store code has been enabled everywhere for a few weeks now, let's
enable it by default in the code. Future changes will remove the config as well
as all the code associated with the non Rust store code.
The various tests changes are due to small difference between the Rust code and
the Python one, the biggest one being it's handling of corrupted packfiles. The
old code silently ignored them, while the new one errors out for local
packfiles. The test-lfs-bundle.t difference is just due to an ordering
difference between Python and Rust.
Reviewed By: kulshrax
Differential Revision: D21985744
fbshipit-source-id: 10410560193476bc303a72e7583f84924a6de820
Summary:
Whenever data is redacted on the server, a specific tombstone is returned when
fetching it. Make sure that whenever we update the file on disk, we write a
nice message to the user instead of the tombstone itself.
While this code could have been moved into the Rust store code itself, I prefer
to leave it to its users to decide what to do with redacted data. EdenFS for
instance may want to prevent access to it instead of showing the redacted
message.
Reviewed By: kulshrax
Differential Revision: D21999345
fbshipit-source-id: 39a83cdf5ea4567628a13fbd59520b9677aba749
Summary:
On Windows, `pathlib.Path.resolve` has an use of uninitialized memory bug in Python 3.6.2. Lego Windows has this version of Python 3 installed.
When `eden prefetch` runs, it will first attempt to read the config file at the root of the repository (`$REPOROOT/.eden/config`). Because of this bug, when we normalize the path in edenfsctl, it will get some random bytes as a result. This subsequently causes `toml.read` to fail as it is unable to read a path containing NUL byte.
As you can see, this is the same exception as we see on Windows:
```
In [1]: open("test\x00")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-57764b20d660> in <module>
----> 1 open("test\x00")
ValueError: embedded null byte
```
Reviewed By: xavierd
Differential Revision: D22177997
fbshipit-source-id: ab2565c8946d809bc15bc1597b749fb5e9440ca0
Summary:
The dynamicconfig logic had an assert checking that a config was not
added by the dynamicconfig twice. This isn't actually a valid assert since the
caller could load the config as many times as it wanted.
This happened in hg doctor, where it loaded the ui once manually (without a repo
object) then it loaded it again during the creation of the repo object. I went
ahead and cleaned up hg doctor to not do this, but let's get rid of the assert
anyway.
Reviewed By: quark-zju
Differential Revision: D22194273
fbshipit-source-id: 9491d35fe14523ad3d9cb69b4ca0d615149dd0f0
Summary:
D22009699 recently introduced some logic that access repo.localvfs
during repo setup. This doesn't exist for peer repos, so let's exist early if
this is a peer.
Reviewed By: xavierd
Differential Revision: D22197586
fbshipit-source-id: 8f717aa21a0b45f8f85a1ff800ee4a1f86b94bdf
Summary: This makes it easier to run tests with non-buck build.
Reviewed By: sfilipco
Differential Revision: D22174992
fbshipit-source-id: 11100d608cb78973ae7a63056667f0be4a86ae00
Summary: This might make it easier for us to see what processes are hanging.
Reviewed By: sfilipco
Differential Revision: D22009696
fbshipit-source-id: a1c1f97976084160a1b4e4a3ff7d2f9781240f6f
Summary:
There are lots of "hanging" questions in the support group. While not all of
them are our fault (ex. mergedriver), we don't have an easy reply to tell
users what's actually going on.
This diff adds a way to write sigtrace periodically so we can include it in
rage output.
Reviewed By: sfilipco
Differential Revision: D22009699
fbshipit-source-id: 5349f613b08507ed02cc0e3fa00963fd7d4c3d21
Summary:
A lot of extensions wrap runcommand just to get the command name.
Let's define a proper API for it - ui.cmdname provides the resolved
command name.
Reviewed By: sfilipco
Differential Revision: D22009694
fbshipit-source-id: 4b8744ee5314924ea09c9f325570ce53c849fef5
Summary: The tracing APIs and error context APIs can achieve similar effects.
Reviewed By: xavierd
Differential Revision: D22129585
fbshipit-source-id: 0626e3f4c1a552c69c046ff06ac36f5e98a6c3d8
Summary:
`eden top` often adds extra quotes to the end of the commands in the process
table, this removes those pesky quotes
Reviewed By: genevievehelsel
Differential Revision: D21440316
fbshipit-source-id: f1200a28a5345691fcce517526d119f44e5993d0
Summary: Make test-test-commit-interactive.t python 3 compatible. Note I wasn't sure if it was worthwhile to spend too much time on the translation portion of the test. If someone objects to the deletion, I can spend more time on it.
Reviewed By: DurhamG
Differential Revision: D22122960
fbshipit-source-id: d232971581fca0532ed05c8ca1fdb7b5e8bd99e0
Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
We have seen that eden will unexpectedly fetch data, we want to know why.
This adds the plumbing to interact with edens current logging to be able to
log when eden fetches data from the server and what caused eden to do this
fetch. Later changes will use the classes created here to log the cause of data
fetches.
Reviewed By: chadaustin
Differential Revision: D22051013
fbshipit-source-id: 27d377d7057e66f3e7a304cd7004f8aa44f8ba62
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: Let's not heal 10000 blobs in parallel, that's a little too much data.
Reviewed By: farnz
Differential Revision: D22186543
fbshipit-source-id: 939fb5bc83b283090e979ac5fe3efc96191826d3
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: This makes it easier to understand why we're fetching so much data.
Reviewed By: quark-zju
Differential Revision: D22114905
fbshipit-source-id: 7cb6d5c9aebcc8c9089891e030b02176e208bd0f
Summary: This got broken in D22115015 — this fixes it.
Reviewed By: farnz
Differential Revision: D22186138
fbshipit-source-id: 54c05466cdbd3be4f6887a852f099351ea5e891e
Summary: This will be used in the following diffs. It just adds commitsync fixtures in a single place, so that we can later play with them in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21952665
fbshipit-source-id: 2933a9f7ea8343d5d52e6c3207e7d78a3ef0be25
Summary:
We had to fork OpenNSA and clone from it, see D19437386 for details.
Broadcom has now started hosting OpenNSA as a tarball.
Thus, we no longer need to maintain a fork (yay!)
This patch points opennsa manifest to fetch opennsa from this new location.
Reviewed By: bkoray
Differential Revision: D22175932
fbshipit-source-id: 51cd777ab836e4f191d78fbb2312925e446ca38f
Summary: This bug can be triggered when your computer name contains emoji. getdeps.py will fail to create this file due to Python attempts to write the file as cp1252 (Windows's default encoding)
Reviewed By: wez
Differential Revision: D22171935
fbshipit-source-id: fc3be2d1050c17ddbe05a0fc91d6613865f092ce
Summary:
This test appears to fail when run in hgbuild. It's not clear why it
doesn't repro locally, but it appears to be a locale issue, so let's just for it
to use C.
Reviewed By: singhsrb
Differential Revision: D22178365
fbshipit-source-id: 007c86681ff1bd4845b1feb41b7bbb031edf172b
