Summary: I want to enable this in production to share changelog object between Rust and Python. Let's turn it on for tests, first.
Reviewed By: sggutier
Differential Revision: D46849672
fbshipit-source-id: 8a27c7310253edc767a333b3026a5f1b7dfd1f5f
Summary:
There were three things wrong, in order of discovery:
1. It wasn't testing anything because the "cl" changelog in-hand was empty. Fix by getting cl after importing commits. Didn't investigate further.
2. ancestors.py was broken due to a previous change in D43839498. Reverted that in preceding commit D46961670.
2. Revision numbers were too big for ancestors.py. Set devel.segmented-changelog-rev-compat=true to get smaller numbers.
Reviewed By: zzl0
Differential Revision: D46898400
fbshipit-source-id: 9b2bb25b457d1757851ed05d9a79da185593910d
Summary:
Previously in D43839498 I tried to do add naive copy trace support for eager repos, but that wasn't correct as evidenced by the commit hash changes in some subequent test changes. It also broke the algorithm in ancestors.py, but that code wasn't being invoked at all due to some test issues.
Anyway, revert some parts of D43839498, instead adding a naive, separate commonancestorsheads implementation for eagerfilelog. Seems to be an improvement because the test output reverts to the previous commit hashes.
Reviewed By: quark-zju
Differential Revision: D46961670
fbshipit-source-id: e3e6e4821bbdda18d3c844d00ac742f84544ed46
Summary: Adds a new EdenAPI endpoint commit/graph_segments that returns a segmented representation of ancestors of one set of commits (heads) excluding ancestors of another set ot commits (common).
Reviewed By: markbt
Differential Revision: D46796194
fbshipit-source-id: 45513e97a1b42f3fbdbe14ed94961512011cb0d5
Summary: This fixes compatibility if both the Python LFS extension and remotefilelog.lfs are enabled. In particular, if we find an LFS pointer in the non-LFS indexedlog storage (i.e. inserted by Python LFS extension when remotefilelog.lfs was not enabled), we now upgrade it to the the scmstore pointer store.
Reviewed By: quark-zju
Differential Revision: D46714084
fbshipit-source-id: 4761f1c05017897544b33b0c322d4c3bce81274d
Summary: Tweak things so scmstore is compatible with the python lfs extension. In particular, if remotefilelog lfs support isn't enabled, don't treat lfs pointers specially (i.e. just store them as regular file data). This is what contentstore does, and it is required for compatibility tests that use remotefilelog with the python extension. (This is not used in production.)
Reviewed By: zzl0
Differential Revision: D46674241
fbshipit-source-id: faa5b5b992a7e2cc61bf98ba2919753e9825bb0a
Summary:
Use a different strategy to avoid "undo" on nested runcomand calls.
Previously it would remove the runcommand wrapper, but that effectively disables the undo extension for subsequent commands under debugruntest. Instead, we now have some global state that we unset when a command finishes so we can tell the difference between nested commands in the same process, and serial commands in the same process.
Reviewed By: zzl0
Differential Revision: D46646009
fbshipit-source-id: 47387c412da246e13b2b7c947ca04f3e419f5a87
Summary:
Now we propagate invalid path errors (e.g. invalid utf-8, invalid path components) to Python, and warn about them.
This is a change in behavior in that we don't bail out on invalid paths.
Reviewed By: quark-zju
Differential Revision: D46646008
fbshipit-source-id: af7fabdfcb559b987cd8254e28429256baaf34da
Summary:
D45510982 enabled rust status for debugruntest, but some tests are not
ready yet. This diff disable rust status explicitly for those tests.
Reviewed By: quark-zju
Differential Revision: D46985910
fbshipit-source-id: 92e1c825d420f7897f165dce93297991901ad6db
Summary:
avoid establishing wire proto connections for fully edenapi pulls (segmented changelog `hg pull`)
we can speed up the command by not sending clinettelemetry and hello for this
use case that will significantly reduce wireproto traffic: https://fburl.com/scuba/mononoke_test_perf/hlllh1id
we will aslo save time not establishing wire proto connection (this is always a
separate connection than edenapi protocol)
at the moment this only helps for master segmented changelog pulls but this is
our top used command
Reviewed By: quark-zju
Differential Revision: D46313581
fbshipit-source-id: 275de25ab751f8339afe28ea937d5e7c5eac8984
Summary:
Make the eden dirstate map use a treestate object under the hood instead of a plain Python dictionary. This way Python and Rust can stay in-sync wrt the dirstate.
I'm working on making everything use the Rust status internally (e.g. "hg commit" uses the Rust status). For this to work, pending dirstate changes from Python need to be visible to the Rust status. Non-EdenFS handles this by sharing the same treestate object between Rust and Python. This doesn't work for EdenFS since EdenFS uses its own dirstate format completely separate from the Rust workingcopy.
However, the Rust workingcopy already converts the EdenFS dirstate into an in-memory treestate object for compatibility. In this commit, I pass this treestate object along from Rust to the eden_dirstate_map to act as the underling _map. This way, mutations from Python will be available to the Rust status code.
This isn't as nice as completely migrating to treestate, but it is much easier and is a step in the right direction.
Reviewed By: quark-zju
Differential Revision: D46840436
fbshipit-source-id: 99904622507645001cc5a9f2096da9d83cda98d4
Summary:
On macOS, some of these types are u32. On glibc, usize. Add more
compatible casts.
This fixes the Watchman build on macOS.
Reviewed By: zzl0
Differential Revision: D46959090
fbshipit-source-id: a15635d10bb0ff687d2aab8fb608b6d841c157fd
Summary:
D45510982 enabled rust status for debugruntest, but this test is not
ready yet. This diff disable rust status explicitly in the test.
Reviewed By: quark-zju
Differential Revision: D46938285
fbshipit-source-id: b9cb514d8721328a65ac88229aeff8f85dbaf922
Summary:
Add "connection pool" logic that emulates a pool using a directory.
This can be useful on Windows where `fork()` is unavailable.
Reviewed By: zzl0
Differential Revision: D46885307
fbshipit-source-id: 26c85f6836fc7a9437ccaef0aaf4a7f0a6e8561f
Summary:
Provide APIs to obtain the `NodeIpc` instance instead of raw sockets.
This makes it possible to use `NodeIpc` features like sending the
file descriptors, node channel, etc.
Reviewed By: muirdm
Differential Revision: D46812533
fbshipit-source-id: 7b3d51a6027c9dbacffcf74476f9b8a4a3440313
Summary:
Add a crate to make it easier to use UDS (unix-domain-socket) communication.
For now, it only contains the low-level bind, and connect functions.
Reviewed By: muirdm
Differential Revision: D46812534
fbshipit-source-id: efbf31dbb1286ae64051b570c20ca6f41c736afa
Summary:
The `FileDescriptor` type will close the underlying fd on drop. In our
use-cases `r` and `w` refer to the same fd so we only need to close it
once. Note the reason we use separate `Mutex`es for r and w is that we
want `recv()` and `send()` running from different threads to not block
by each other.
Reviewed By: zzl0
Differential Revision: D46885308
fbshipit-source-id: b2a5644251ac4773581a995ae8c3d955b0b36861
Summary: The crate is no longer used.
Reviewed By: muirdm
Differential Revision: D46812535
fbshipit-source-id: 659b73a563bf14cc17cb1b09e9a57422d21a6377
Summary: `recv_stdio` seems a better place for the logic.
Reviewed By: muirdm
Differential Revision: D46811171
fbshipit-source-id: ddd91489610fcf6a148897bbd4f35b1e404cbd0b
Summary:
This makes it easier to use. In addition, the "singleton" IPC initially decided
by NODE_CHANNEL_FD is handled internally so the callsite is less error-prone,
since it does not need to care about maintaining the NODE_CHANNEL_FD env var,
or the initialization order of the singleton.
Reviewed By: muirdm
Differential Revision: D46811166
fbshipit-source-id: a2a8a7a36986422a8d7c967d6fb5b820bedce0c9
Summary:
Remove the side effect when constructing the NodeIpc. It does not seem
necessary. This allows us to construct NodeIpc without checking the underlying
fd until first send/recv.
Reviewed By: muirdm
Differential Revision: D46811162
fbshipit-source-id: 4fc7941a1e9d6b9a9c47daa699de4283bfcec5c3
Summary:
This allows us to update it dynamically, without going through the
`NODE_CHANNEL_FD` environment variable.
Reviewed By: muirdm
Differential Revision: D46811169
fbshipit-source-id: bf52b5597fcf69291da81675d2917dc0dc010904
Summary: This makes it more flexible so we can replace the singleton if we want.
Reviewed By: muirdm
Differential Revision: D46811160
fbshipit-source-id: 902b7ff05a4fb3f8a7d7218afd3d07e0f3a89a9f
Summary:
Make it possible to send fds on POSIX. This is similar to what chg
(hgclient.c:attachio and osutil.c:recvfds) does. Note the `sendfd`
crate seems to have a solid implementation but its public API is
on the `std::os::unix::net::UnixStream` type, which is incompatible
with our struct that uses the `FileDescriptor`.
Example:
% cargo run --example spawn_sendfd
Parent: spawning child
Parent: sending hello
Parent: sending stdio and a.txt file descriptors
Parent: waiting for child to exit
Child: started with IPC handle 4
Child: got message Some("hello")
Child: got sendfd payload SendFdPayload { raw_fds: [3, 5, 6, 7] }
Child: writing "something\n" to fd 3
something
Child: writing "something\n" to fd 5
something
Child: writing "something\n" to fd 6
something
Child: writing "something\n" to fd 7
% cat a.txt
something
Reviewed By: muirdm
Differential Revision: D46811163
fbshipit-source-id: 37686327f02a7fa0b0e8768d025094a7fcb0e2b5
Summary:
This avoids over reading which might drop special messages from sendmsg(),
on POSIX systems.
The behavior disables itself if we need libuv compatibility on POSIX.
Reviewed By: muirdm
Differential Revision: D46811164
fbshipit-source-id: 3aed4bb4d63ad10ecc6a7bf76007cc7263c1c580
Summary:
This avoids over reading which might drop special messages (like file descriptors)
from sendmsg(). Buffered reader might turn those special messages into regular
messages and will lose the file descriptors.
Reviewed By: muirdm
Differential Revision: D46811158
fbshipit-source-id: 2294c5c2b132c25c21950f26e9f0d44094743fe0
Summary:
We're going to use winapi to implement sending and receiving file descriptors
on Windows.
Reviewed By: muirdm
Differential Revision: D46811161
fbshipit-source-id: dca85722da9f96e15f1ec8bb4676058487c51872
Summary: They will be used by upcoming changes.
Reviewed By: muirdm
Differential Revision: D46811168
fbshipit-source-id: fdd60a2aa96b979dd2dc5bb2d016189435bbf695
Summary: This makes it a bit easier to use in upcoming changes.
Reviewed By: zzl0
Differential Revision: D46811157
fbshipit-source-id: 333e17415d4a67fcb58bec61153b15e4cad03d23
Summary:
Exposes the ability to construct `NodeIpc` from a Windows HANDLE, not a
msvcrt/ucrt's fd.
Reviewed By: zzl0
Differential Revision: D46811167
fbshipit-source-id: 94f48dc0d059bca109c1792dceb0b733e4737adf
Summary:
In the next change I'm adding "raw ..." concept. Let's rename `raw_fd` to
`libc_fd` to make it obvious that the raw something is different from the raw
"HANDLE" on Windows.
Reviewed By: zzl0
Differential Revision: D46811165
fbshipit-source-id: e0fe48633fd8d928ce7d7ae0652f041a90aec922
Summary: We were forbidding \n but not \r. A couple Python spots validate against "\r" and "\n" in filenames (scmutil.checkfilename and manifest._checkforbidden), so let's do it in Rust as well.
Reviewed By: zzl0
Differential Revision: D46646011
fbshipit-source-id: 41b5cbad057e2b966e3c00011d75cd6cd6e804fc
Summary:
This test needed significant changes to use remotefilelog (unrelated to Rust status).
In particular:
- "hg cat" doesn't report "no such file" for non-existing files. Previously that came from py manifest, but Rust manifest walk doesn't support.
- Kill tampered bundle check. I tried for a while to get this working, but everything fails really early with the Rust manifest since bad paths are immediately rejected by the RepoPath type.
- Various update-over-symlink tests now work since the Rust vfs will remove symlinks in order to write the new file (thus avoiding the "xxx traverses symbolic link yyy" audit error).
Also this test revealed the need to process directories in the Rust status to catch a file being replaced by a directory. I changed the (Watchman) code path to no longer skip directories returned from watchman.
Reviewed By: zzl0
Differential Revision: D46646010
fbshipit-source-id: a2f8a92e85a97611212d917b2a6118f0dd1f5eed
Summary:
The x2p auth advice header is handled/displayed by the network doctor in practice now.
I'm making tests support the Rust status, and opting to remove this one instead of update it.
Reviewed By: zzl0
Differential Revision: D46646012
fbshipit-source-id: 72bd074b0e8d76ecbc4206920f6c8867d6d794cd
Summary: I had to tweak the case collision auditor to exclude untracked files. The Rust status (using Watchman) seem so to add untracked files to the dirstate earlier than Python, which was causing extra "possible case-folding" warnings.
Reviewed By: quark-zju
Differential Revision: D46832704
fbshipit-source-id: fa0e30df757804804ca437df94a4fb4b56dcbeda
Summary:
Some Python commands depend on the "bad" matcher callback getting called by dirstate.status(). Tweak things so this happens when we use the Rust status as well. This also adds support for the "clean" option when using the Rust status since clean files and "bad" are somewhat coupled.
I moved invalid file type detection from filesystem.py into dirstate.py so it covers Rust status as well.
Reviewed By: quark-zju
Differential Revision: D46646006
fbshipit-source-id: 28278af15e02e0f794aefc7bde850ce58b319ac6
Summary: It aleady passes for these, so turn it on.
Reviewed By: quark-zju
Differential Revision: D45510980
fbshipit-source-id: f45a2dbe11186411adc259863b88c6ff5ee1914e
Summary: It's annoying to have to remember when this setting is and isn't enabled by default.
Reviewed By: quark-zju
Differential Revision: D45510982
fbshipit-source-id: 7d54c0c95d7edf61f358692b1a7c487e5889b5fa
Summary:
add option for consistent routing of single tree requests and single blob requests
Meta:
Background:
We are having a lot of traffic for single tree blob or single file blob:
https://fburl.com/scuba/mononoke_test_perf/0eainxd3https://fburl.com/scuba/mononoke_test_perf/0ye4eduq
Let's route them consistently!
Reviewed By: quark-zju
Differential Revision: D46860178
fbshipit-source-id: 50298c583d82585b3345ec4c4ca6b7ab4ab57ab1
Summary:
The blackbox was reporting a mode mismatch. The cause was the filesysytem walk yielding a mode with permission bits set (e.g. 0666), but os.lstat within dirstate.normal() yields a mode without permission bits set (e.g. 0000). Fix by only reporting a mode change when the executable bit changes (which seems like the intention of the code).
I don't think this is a recent change in behavior, so I'm not going to look further into it. Hopefully the Python filesystem code will go away soon with the Rust status.
Reviewed By: zzl0
Differential Revision: D46848722
fbshipit-source-id: cdd73137ae891bc8038d7d7ddcd09d8c95170051
Summary: To avoid showing a bunch of "?" files, we need to ignore things within submodules. For watchman, we do similar to Python and add submodules to the ignored dirs list. For the manual walker, we similarly skip directories in the ignore list. Note that the walker already skipped the submodules due to the presence of the "dot dir" (e.g. ".sl") in the submodules, but I piped through the ignored dirs skipping to be consistent.
Reviewed By: quark-zju
Differential Revision: D46547238
fbshipit-source-id: b001dc02a3c73f69d74fe36615a812f9ac983944
Summary: Python status for Git has submodule aware logic we don't have in Rust, so we need to defer to Python. However, Python will still be able to use the internal Rust status API.
Reviewed By: zzl0
Differential Revision: D46547239
fbshipit-source-id: 2caf1436819219cea007307f602627fd1a38ef51
Summary: It wasn't used from Rust (only MultiWalker(threads=8) was used), and Rust walker is not enabled from Python. It's annoying to have to make changes twice, so let's kill the non-parallel walker.
Reviewed By: zzl0
Differential Revision: D46547240
fbshipit-source-id: 4a57222ec705059acbc6865f8e1a9701ae584ccc
