Summary:
Now that we have the plumbing we can start registering our NFS RPC servers (
mountd and nfsd) with the portmapper/rpcbind.
I am taing a short cut for prototyping. I am skipping general purpose
rpc registration (i.e. making these rpc calls to register a service). This is
because (1) I am trying to prototype quickly, so I don't want to implement the
portmapper endpoints that allow registering/ unregistering a server and (2)
there are maybe security considerations, and I don't want to think about those
rn.
Reviewed By: chadaustin
Differential Revision: D44987884
fbshipit-source-id: 4b849d1dd060d2e033ab48df883e228015906a7d
Summary:
getport is the primary endpoint on the rpcbind/portmap server. NFS clients
make a request on this endpoint to find the NFS RPC servers.
In this diff we keep a mapping of RPC servers and respond to the getport
request based on this mapping.
Currently no rpc severs are registered, so we reply no port to all
requests.
Reviewed By: chadaustin
Differential Revision: D44987893
fbshipit-source-id: 920171c6007b82609ae84fb7a9aff43d898e43dd
Summary:
The msft client can't be pointed at an NFS server on a particular port, so
we have to register our RPC servers (mountd and nfsd) with the port mapper
service.
There is no portmapper service running on Windows by default.
There is a port mapper implementation from msft that can be installed on
Windows Server, but not on Windows desktop.
There are some third-party implementations of the portmapper for Windows. Most
of them are part of an NFS Server implementation. But none that I have found
both run on modern Windows versions, are a general purpose port mapper, and
support port mapper v2. General purpose means they allow processes to set a
port mapping. Most the port mappers I found must register their NFS server in
process, because they don't support calling the set mapping endpoint. The rest
only run on Windows versions from before I was born or only run v4.
I tried porting the linux gnu implementation of port mapper to Windows. But despite
a few hours of trying I could not even get it to build even on Linux. Writing our own port mapper is faster.
This is the skeleton of the server. In the next diffs I will register our
RPC servers and support telling the msft NFS client about our servers.
Reviewed By: chadaustin
Differential Revision: D44987863
fbshipit-source-id: bff065795a9f4b7b6c13ef3e3ce603646e1ce364
Summary:
The msft NFS client likes to speak portmapper v2. It does not seem to know v4.
Perhaps there is a way to configure that, but I have not found it yet.
v4 is the latest version of the protocol and so what we should prefer using on
mac and linux.
Here I am introducing the v2 protocol and renaming all the v4 types to
explicitly reference v4.
Reviewed By: chadaustin
Differential Revision: D44987848
fbshipit-source-id: 97f9e2365736b49a5888af204609598b2566de55
Summary:
There is no portmapper service running on Windows by default, so we should
not be trying to connect to one by default on Windows.
Though I am adding a port mapper service that eden it self runs in the coming
diffs it is not compatible with this client. The msft NFS client likes to
speak portmapper v2 and this client only knows portmapper v4 (otherwise known
as rpc bind). So I am stubbing these out for now.
Reviewed By: chadaustin
Differential Revision: D44987834
fbshipit-source-id: 078ed1761cffbe9621ab39892c2135504a6aeb1b
Summary: it has been enabled in prod for 5 months, let's enable it by default
Reviewed By: sggutier
Differential Revision: D45086467
fbshipit-source-id: 3b0df83105ad76b67f6e841e81cd7223b68e3e9b
Summary:
[sl] bump macOS python CI to 3.11.3
We still depend on pointing to the specific Python combination that GitHub CI and Homebrew have, which currently is 3.11.3.
Hashes were taken from d39875e212/Formula/python@3.11.rb
Pull Request resolved: https://github.com/facebook/sapling/pull/605
Test Plan:
Downloaded bottles from:
https://github.com/sggutier/sapling/actions/runs/4727040284https://github.com/sggutier/sapling/actions/runs/4727040323
and installed the arm64 version. Also, verified that the dumps from `otool -L` looked correct (they usually don't when there is some when building)
Dump from arm64 version:
```
$ otool -L sapling\ 8/0.2.20230417-165855+6db1e950/bin/sl && file sapling\ 8/0.2.20230417-165855+6db1e950/bin/sl
sapling 8/0.2.20230417-165855+6db1e950/bin/sl:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60158.100.133)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1858.112.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1163.100.19)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
@HOMEBREW_PREFIX@@/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
sapling 8/0.2.20230417-165855+6db1e950/bin/sl: Mach-O 64-bit executable arm64
```
Dump from x86 version:
```
$ otool -L sapling\ 9/0.2.20230417-165855+6db1e950/bin/sl && file sapling\ 9/0.2.20230417-165855+6db1e950/bin/sl
sapling 9/0.2.20230417-165855+6db1e950/bin/sl:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60158.100.133)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1858.112.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1163.100.19)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
@HOMEBREW_PREFIX@@/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
sapling 9/0.2.20230417-165855+6db1e950/bin/sl: Mach-O 64-bit executable x86_64
```
---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/facebook/sapling/pull/605).
* https://github.com/facebook/sapling/issues/606
* __->__ https://github.com/facebook/sapling/issues/605
Reviewed By: zzl0
Differential Revision: D45072742
Pulled By: sggutier
fbshipit-source-id: 4ced2d609d9cf4098cd4bbea5d5df45659588ad8
Summary: Look at at all the lines of code we could delete!
Reviewed By: YousefSalama
Differential Revision: D45057185
fbshipit-source-id: c454a4bd70527460bf4a727aa3ebe57b3becab4d
Summary:
I didn't want to repeat myself too much so I'm introducing macro that will do
that for me.
Reviewed By: YousefSalama
Differential Revision: D45057186
fbshipit-source-id: 6e13a5a72c6ad463982611ce11a16f937dcd9386
Summary:
The add_recursive would try to access the parents from changesets table
even if they're not there yet (in some places we're racing to inster them to
both tables at the same time).
For that we need to:
* process changesets in reverse topological order
* check edges_map before trying to fetch parrents from legacy changesets table
* skip fetching parents if the parent will be processed in the same function call
Reviewed By: YousefSalama
Differential Revision: D45057189
fbshipit-source-id: 7906c610c7a4f8473647796e0c56962e074a035f
Summary:
`add_recursive` has problem when adding many changesets not present in the
legacy changesets table. This test showcases the problem, the next diffs will
attempt to fix it.
Reviewed By: YousefSalama
Differential Revision: D45057188
fbshipit-source-id: 56297b7e2d47c6ec65023e9782ac0e18012a4319
Summary: Those helped me to localize the source of errors. I might as well commmit them.
Reviewed By: YousefSalama
Differential Revision: D45057187
fbshipit-source-id: 694d9596dec030e65b0d29590068103fb1b70ca9
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Implement the modified version of file_info endpoint using the ContentMetadataV2
Reviewed By: markbt
Differential Revision: D43537343
fbshipit-source-id: 98554366a725636d4cd6989fb7e2d9b5fe29d5e4
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Modify Thrift data type, and the function signature for the endpoint
Reviewed By: markbt
Differential Revision: D43537344
fbshipit-source-id: 6076703f67923e0fa95676b0db812a75fb6c3650
Summary:
These have broken production as the behaviour was changed so that `--foo` would no longer be valid when it used to be.
The behaviour changed inadvertently when migrating these CLIs to clap 4.
Differential Revision: D45081836
fbshipit-source-id: ffa7a2a2b1c04c860082bb2770eaa2cde1e7519f
Summary:
Fixes some warnings about unused imports and struct fields when building on
Windows.
Reviewed By: MichaelCuevas
Differential Revision: D45061519
fbshipit-source-id: 322768bf4e0d2af47129bd8f5c376211ee98d0d8
Summary:
My goal is to decouple our inode layer, including EdenMount, from the
specific filesystem implementation. To that end, remove several FUSE
references from EdenMount's comments and variable names.
Reviewed By: kmancini
Differential Revision: D44737102
fbshipit-source-id: 9956cf818599d508a897b92936350f40207d4b40
Summary:
Small refactorings that don't have a particular theme. I'm working
towards removing FUSE- and PrjFS-isms from EdenMount and friends.
Reviewed By: kmancini
Differential Revision: D44728015
fbshipit-source-id: 328abae1043a48f440836b98f1a1ee20a1b091d1
Summary: I think I addressed this one already in D44187187 but forgot to remove the FIXME.
Reviewed By: evangrayk
Differential Revision: D45050467
fbshipit-source-id: eb408ae2a9e1ba873e7f28ec3ff1cb755a1d2560
Summary: Fixed the cont-builds command and updated the tests to actually work instead of just passing all the time
Reviewed By: skotchvail
Differential Revision: D44898873
fbshipit-source-id: dbbc7429942a2d400346004e13bf7ba3cb00476d
Summary:
On Windows most applications, including Windows Explorer, are not able to open symlinks if they have forward slashes (e.g., `eden/scm/Makefile`). Windows users creating symlinks will probably create symlinks with backward slashes instead because of this (e.g., `eden\scm\Makefile`). Analogously, symlinks with backward slashes will be of not use on POSIX systems.
This diff tries to alleviate this by making symlinks added to commits on Windows via `hg commit` usable everywhere else.
Reviewed By: zzl0
Differential Revision: D44358129
fbshipit-source-id: 2835652b62104e22f10cb422586fae9998db40ad
Summary:
Makes `hg diff` and `hg commit` work as one would expect on Windows. This is done by modifying some constants in `fscap.py`.
This change should not impact end users, as explained in previous diffs.
Reviewed By: muirdm
Differential Revision: D43741223
fbshipit-source-id: 203878f64c11364257cef4bb82395713456f3625
Summary: The working copy uses the VFS object for determining whether a file is a symlink or not. Previously this behavior was disabled as there was no support for symlinks at all when this was written.
Reviewed By: muirdm
Differential Revision: D44592907
fbshipit-source-id: 6b9041a5045a336292a2b641da30cf3ae22f441c
Summary:
I noticed when reading the EdenServer startup and shutdown logic that
we did not log the structured DaemonStop event on Windows. The ifdef
must have been vestigial, so remove it.
Reviewed By: xavierd
Differential Revision: D44699330
fbshipit-source-id: 529aa2b2933b1ef3b8337fa19c1b88b9033653ba
Summary:
introduce cloud tidyup that can cleanup unnessesary remote bookmarks
this command can be used by ondemand or clients to get rid of unnessesary
remote bookmarks in a safe way server side
we can also use it in hg doctor command
Reviewed By: zzl0
Differential Revision: D44954711
fbshipit-source-id: 5d4e157c9b668f26cd39794d7f752828e4fc8b3f
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which were modified in the previous diff to support the new hash and are used here.
Reviewed By: muirdm, markbt
Differential Revision: D44010967
fbshipit-source-id: a755f0750ed39719b1f0e6895b7ca81a405fcfed
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which are modified in this diff to support the new hash.
Reviewed By: quark-zju
Differential Revision: D43977390
fbshipit-source-id: bd9a264dc4483574a7a46e7ce4f9109910f18f2d
Summary: If rust code writes to the "real" stderr/stdout in a debugruntest test (e.g. println!()), that output was getting dropped by run-tests.py. Now it at least prints it out, albeit not interleaved temporally.
Reviewed By: quark-zju
Differential Revision: D44768929
fbshipit-source-id: 9520229dbf885503cd53a21cc682df7135d72fb7
Summary: Use dagcopytrace to replace the current heuristic based copy tracing.
Reviewed By: quark-zju
Differential Revision: D44336238
fbshipit-source-id: 74fa131e78531d06ae2c263534a4a56b2081c041
Summary: The clash between `-h` for both `--help` and `--host` was inadvertent. Switch to `-H` for `--host` to disambiguate. It seems clap4 handles this clash less well than clap3 did.
Reviewed By: clara-9
Differential Revision: D44994925
fbshipit-source-id: c4195500cecd2c796e4765c2fbe6047df2f57593
Summary:
This "repo" determination was incorrect. In particular, it emitted "fb:foo" for a path of "fb:foo" (should be just "foo"). This code is totally superfluous, though, since the Rust dispatch also logs "repo" (correctly).
I also killed logging of "dirstate_format" since I think that is always "treestate" now (except for some tests and test repos).
Reviewed By: zzl0
Differential Revision: D44978910
fbshipit-source-id: 8a1478dc0dc7bfda6bdab683c5c5e57e6060ce78
Summary: The walker was migrated to the new cmdlib, but still depended on the old one for a couple of structs. Remove these dependencies.
Differential Revision: D44886561
fbshipit-source-id: 122d8aa01269b49651b95bd431b52045f91b88bc
Summary: LFS server uses the new command library, so no need to import the old one just for this struct.
Differential Revision: D44886560
fbshipit-source-id: 8b8869c9cf717aa622f374468ab061dff0ef349f
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44712484
fbshipit-source-id: e81a383ca1b89b87dc64577bbc42c56740599bf4
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44711493
fbshipit-source-id: adb9ce812423a11fae4f6c6a368d755780606944
Summary:
It's strange that BlobCache's configuration came from command line
arguments and TreeCache from the configuration file. To my knowledge,
neither configurations have ever been adjusted, so unify their naming
and replace BlobCache's command line flags with EdenConfig settings.
Reviewed By: xavierd
Differential Revision: D44699297
fbshipit-source-id: 88b0edf830e8dfc8f9eaa9ff04c563c54992f8bb
Summary:
I'm not sure how BlobCache and TreeCache grew to have different
ownership patterns. Allocate them together and prepare for
initializing BlobCache with EdenConfig.
Reviewed By: xavierd
Differential Revision: D44699259
fbshipit-source-id: fc91bb9fe80423e036bb7856d9f3b8863ba71ce8
Summary:
The error handling in EdenServer::mount was unnecessarily
complicated. Rely on thenValue and thenError to simplify the control
flow.
Reviewed By: xavierd
Differential Revision: D44699195
fbshipit-source-id: b94e513bf198f38e4614ccfa18ffab37bb43ff87
Summary:
A common FsChannel base class allows removing some ifdefs from
EdenServiceHandler. With some effort, we should be able to find the
correct abstractions that allow us to entirely decouple particular
FsChannel implementations from the rest of the inode layer.
Reviewed By: kmancini
Differential Revision: D44657395
fbshipit-source-id: 9702186f1d0c9ce76a11e766010f5899e2b517af
Summary: Don't hardcode remotenames.hoist to "remote". I didn't end up using a different value for this config, but probably an improvement worth committing anyway.
Reviewed By: quark-zju
Differential Revision: D44857509
fbshipit-source-id: 68a3660827d16083231ad98dc57321ac1ce3a377
Summary: Now if the clone source URL contains a fragment, we persist the fragment as the repo's remotenames.selectivepulldefault. This gives a user friendly way of specifying the default bookmark you are interested in.
Reviewed By: quark-zju
Differential Revision: D44857510
fbshipit-source-id: 54cffeebb76d6a9dc603adcc6ad58ab17568050c
Summary: If the user specifies --debug, don't delete the repo after an error when cloning. This makes it easier to debug clone failures.
Reviewed By: quark-zju
Differential Revision: D44857512
fbshipit-source-id: 921dcfdf472826aae407564b9f8b788e53058d20
Summary:
Now if you specify "--config remotenames.selectivepulldefault=whatever", we will persist it to the repo's config.
Note that this is still not good UX. See later diff that adds support for specifying default branch as URL fragment.
Reviewed By: zzl0
Differential Revision: D44857513
fbshipit-source-id: b8474f5d54e5d597c0ff44e9d689ccbfcc3068b3
Summary: I'm sure the Python code was useful during development, but it adds an unnecessary layer of complexity/noise to the reader.
Reviewed By: quark-zju
Differential Revision: D44857515
fbshipit-source-id: 19a5165b402cd2c1e8a367edb64862c2bdeae18f
Summary:
Canonical repo names can/will have slashes in them. We shouldn't convert slashes to directories on disk; just a bad idea.
When choosing the clone destination, use the basename. It could make sense to clone "foo/bar" into "foo_bar" instead of just "bar", but let's maintain the existing/expected behavior for now.
For internal uses, we percent encode the reponame. For example, the hgcache directory for "foo/bar" will be "foo%2Fbar". This is to minimize chance of collision. If we named the cache dir "foo_bar", that would collide unpleasantly with a repo named "foo_bar".
I relaxed reponame inference a bit to allow slashes in URLs like "test:foo/bar". This was purely so I could test with a reponame containing slashes.
A better long term approach is to use the hash of a canonicalized URL, but that is a bigger change.
Reviewed By: quark-zju
Differential Revision: D44857514
fbshipit-source-id: 4fa0b83a0660b81e89fc96b50ecbeb93869dd2b2
Summary: Make the debugruntest "ls" impl report an error if an argument doesn't exist on disk. This matches real "ls" behavior, and is important feedback when you've messed up the arguments.
Reviewed By: quark-zju
Differential Revision: D44857511
fbshipit-source-id: 0467e2c5e175882bb8033186235be36e5798bf7f
Summary:
must_get is like get_nonempty_opt but returns an error if the value is unset or
empty. This makes it easy to propagate an error if the desired config is not
set.
I also updated a few spots to use must_get and got rid of a superfluous error type.
Reviewed By: quark-zju
Differential Revision: D44857516
fbshipit-source-id: cdecd7a68d4e3d8248cae977f4cdf7b553b6af74
Summary:
Xavier just taught me about `cat -tail -wait`. I was watching the logs in vscode
on windows. Let's make a tail version of `eden debug log` so everyone else can
easily figure out how to tail the logs on all platforms.
Reviewed By: chadaustin
Differential Revision: D44941688
fbshipit-source-id: e86b18025f780dda69005ad9a31b1cb11d395261
Summary:
Fixes the bug discussed in Discord here: https://discord.com/channels/1042527964224557107/1042527965256364157/1095764831480590467
Running the ignore matcher on an empty path produces incorrect results. In the rust implementation, this isn't an issue, because there are directory matchers that run on the directory result. In the Python treestate bindings, we don't have that functionality, so it returns an empty string.
This makes this logic match the code in `dirstate.py` ([line ~925](https://github.com/facebook/sapling/blob/main/eden/scm/edenscm/dirstate.py#L925-L926)) where we check for path being an empty string.
This fixes `sl status` (and related functionality) when fsmonitor is enabled and there's a complex gitignore file that ignores files in the root directory.
Pull Request resolved: https://github.com/facebook/sapling/pull/601
Test Plan: Added Mercurial integration test and manually tested locally
Reviewed By: quark-zju
Differential Revision: D44961756
Pulled By: zzl0
fbshipit-source-id: 57dd0d052ceeb184694c6f3b1a71ac53bf0e1028
Summary:
This continues my slashing of fetch retries. This diff is 2x (for a total of
-2 now) fetch retry.
The importer prefetches the tree and then gets the tree. both can cause network
fetches, so when the prefetch fails, the get will retry the network request.
If I am guessing again I think the cause was similar to the last diff:
get use to be a local only check, but now its not anymore so we are exposed to
more potential network fetch retries.
It should be safe to remove this retry because we still kill and restart the
importer if the importer has errors, which should cover any successful re-trys.
With this change we are now down to what looks like 3 network request attempts
per request.
Reviewed By: xavierd
Differential Revision: D44526625
fbshipit-source-id: 455903bdc73c9513106c4e048274d5bb08085a38
Summary: We don't have plan to support this one-to-many mapping, since it will make copytrace complexity increase exponentially in theory. For now, we just pick the last copy in ascending alphabetical order. The reason why we do this is to make the selection deterministic, so the behavior will be consistent on different platforms and Rust implementations.
Reviewed By: quark-zju
Differential Revision: D44901629
fbshipit-source-id: 154b515c7043d4a5e45e1f2077f6930baf354e81
Summary:
In order to test new copytrace logic, we need eagerepo for finding
renames from the file content header.
Reviewed By: quark-zju
Differential Revision: D44816781
fbshipit-source-id: 129b729e1508d4e60e7d0c994c3163cdbdf09353
Summary: This benchmark was used during development of Mercurial data derivation, however it is very synthetic and we haven't used it in a long time. Remove it to ease the burden of maintenance.
Reviewed By: mitrandir77
Differential Revision: D44707860
fbshipit-source-id: 6dd2ac893b4020a0c77847c4f10848ba20d96e38
Summary: Replace the usage of `benchmark/simulated_repo`'s repository with an ordinary test repository with instrumented blobstore (via `DelayedBlobstore`) and database (via `SqliteCallbacks`). This removes this dependency on this code.
Reviewed By: mitrandir77
Differential Revision: D44707861
fbshipit-source-id: 49d4c2b43aa4dc64b10d94e21d8d242c97e05415
Summary:
We want the ability to observe and change the behaviour for sqlite connections. Add the ability to register callbacks that are called whenever the multithreaded sqlite connection is acquired.
No change in behaviour yet. We will use these in a future diff.
Reviewed By: mitrandir77
Differential Revision: D44707859
fbshipit-source-id: f8156e294bbc51b612a6e38604804b4f815ed919
Summary: The blobrepo test is actually a test of mercurial derivation, so move it there.
Differential Revision: D44676336
fbshipit-source-id: 2f7f002a91171a7081ceacb3c6f10b9c62824bfd
Summary: Move the library down into its own test directory with a `src` directory.
Differential Revision: D44676334
fbshipit-source-id: cd65adaf2b4a445f13b05d700be7a30492cadb97
Summary: Reorganise filenodes derivation so that it matches mercurial derivation.
Differential Revision: D44676333
fbshipit-source-id: bb5e11cb5a8344f3f7e812e1e53360355b1ce36b
Summary: Rename to `mercurial_derivation` and create a proper `src` directory. I plan to make this change consistently across all derived data types, but will start with Mercurial derived data so that I can add a test.
Differential Revision: D44676335
fbshipit-source-id: aae67df807575e532636dd46c4655f135d2d57b0
Summary:
follow up clean up dead code
forgotten in earlier diff
Reviewed By: YousefSalama
Differential Revision: D44931222
fbshipit-source-id: 6ec965ae5c7851693432fadc9b71df6a967259bb
Summary: `prefetch-profile` has been completely removed from the experimental command list for some time now, and there's no way any environment could be using it. It's safe to delete this code now!
Reviewed By: xavierd
Differential Revision: D44113592
fbshipit-source-id: f144b713c6e35de4691071afebbe7a829886b893
Summary:
We've seen several cases on Windows where `hg status` is extremely slow due to
the working copy having 10k+ materialized files. Reading through the code,
EdenFS ends up computing the sha1 of all of these files for every `hg status`
call which is quite costly.
On Linux and macOS, both the sha1 and the filesize of materialized files were
cached in the OverlayFileAccess code, but this had the downside of being very
small (100 entries) and not compilable on Windows. Instead of storing it in the
OverlayFileAccess, we can make use of the newly added
`FileInodeState::MaterializedState` to store both the size and the sha1. Beyond
working on Windows, this also comes at no-additional memory usage.
Reviewed By: chadaustin
Differential Revision: D44737138
fbshipit-source-id: 28cf14928de4e685cb21819915f7be7e7a64047b
Summary:
deprecate skip list based version
this is part of the effort to deprecate skip lists infra known as **Skip Lists Deprecation**
this endpoint is not called, it is safe to remove it either client first or
server first
also, we don't need to maintain two endpoints to check correctness, we passed that phase.
Reviewed By: zzl0
Differential Revision: D44916440
fbshipit-source-id: 05f623cc07718f1c9aa3d835d7654b47d3728d2d
Summary:
deprecate skip list based version
this is part of the effort to deprecate skip lists infra known as **Skip Lists Deprecation**
this endpoint is not called, it is safe to remove it server side first
we don't need to keep both for checking correctness, we passed that phase!
Reviewed By: YousefSalama
Differential Revision: D44916005
fbshipit-source-id: 662cfa20ad44346011126e0ae3aae69deb719dbc
Summary:
fbcode_builder doesn't currently know how to build a Python extension module,
which means the Windows oss build fails on trying to load ntapi if we don't
pass the `--no-tests` flag to getdeps. Ideally we add extension module support
to the builder, but for now let's just disable the test case if the extension
module is unavailable.
Reviewed By: chadaustin
Differential Revision: D44804035
fbshipit-source-id: 10da54f6ea7d3699ac42421ee5cf57d883509bdd
Summary: Log the first mononoke host we connect to. Under normal circumstances all connections from a particular client should go to a host in the same datacenter, so this will allow us to look at performance trends taking into account cross datacenter factors.
Reviewed By: zzl0
Differential Revision: D44900086
fbshipit-source-id: c9bb38dac419a8faf468c6bb79a44af3c152c502
Summary:
log duration for the origin request
so that we will be able configure replay in a more flexible way
Reviewed By: clara-9
Differential Revision: D44910760
fbshipit-source-id: 8d716376702283e24bb4151eb0a68b8bc2e57e6f
Summary:
An invalid bonsai changeset or mutable rename could lead to a copy source that does not exist.
For Mercurial derived data, we already ignore this. Do the same for general diffing.
Reviewed By: mitrandir77
Differential Revision: D44909673
fbshipit-source-id: ac754e0e483e75976afb07078e57bc1f33008a17
Summary:
In the repo factory, we do different things for sql connection creation in tests and prod. In prod we cache `MetadataSqlFactory` instances for each metadata database configuration, and request a new connection from the cached factory each time. In tests, this doesn't work, as requesting a new sqlite connection from the factory is the same as opening the file again, which can be slow for large numbers of connections, so instead we cache connections directly in the repo factory, which isn't really the right place.
We can fix this by making the `MetadataSqlFactory` handle the caching of sqlite connections for us. This makes the repo factory consistent between prod and test. It also means the `MetadataSqlFactory` can hold on to the original connection to the sqlite database and perform the creation queries directly, rather than having to hold it separately in the wrapper around `SqlConnections`, and we can deduplicate the code for opening the database, too.
Reviewed By: mitrandir77
Differential Revision: D44665118
fbshipit-source-id: 1c7d45ac1900d5f2b36285e01159924d0cdb09eb
Summary: These connections are no longer used, so they and their setup code can all be removed.
Differential Revision: D44635160
fbshipit-source-id: 4407a9318086b61942523cf551988e394c6df2e1
Summary: This is already used by a couple of other tests, so it makes sense to move it out of the benchmark.
Differential Revision: D44635161
fbshipit-source-id: 41414b66b380ea1d5635c6131df4fa3f3084fcfc
Summary:
This adds the configuration `doctor.ignored-problem-class-names` so that we can
make doctor ignore individual problem classes via config rollout.
This doesn't actually stop the checks from running, but it stops reporting (or
attempting fixes for) ignored problems. Ignored tests will still be logged in the
edenfs_events table's
Reviewed By: xavierd
Differential Revision: D44720584
fbshipit-source-id: 954d1131dfbbf6a3264abf55b5e0f01ab61836d0
Summary: Warn user about potentially confusing duplicate config entries when we are making automated edits to repo/user config file.
Reviewed By: quark-zju
Differential Revision: D44876243
fbshipit-source-id: 7d50df7f55e68f3655f6086156e803e52a2b4154
Summary: When there are duplicate config keys present, we now update the final (load bearing) one instead of the first one.
Reviewed By: quark-zju
Differential Revision: D44875407
fbshipit-source-id: fa546482cbbdb432fb7d02a716e4692022388dc5
Summary:
We've seen cases where invalidation is happening concurrently to
`PrjfsChannel::stop`. In that case, `getInner` would return a `nullptr` and
thus crash EdenFS as the code doesn't handle `nullptr` gracefully.
The root cause of why `stop` is being called on a mount that is still being
used is not entirely clear at this point due to the complexity of the
mount/unmount code which chadaustin is currently looking into. While this diff
should be revisited once Chad has reworked mount/unmount, we can make the code
correct by simply handling the `nullptr` case to avoid crashing.
Reviewed By: chadaustin
Differential Revision: D44875003
fbshipit-source-id: 977c59af9fcf0ae2c66929f22489370c18b92f94
Summary: If auth_proxy.unix_socket_path is configured, use that when testing for Mononoke HTTP connectivity.
Reviewed By: johansglock
Differential Revision: D44883873
fbshipit-source-id: 1c30645072842391ad9aa6b2548ee87bd1449896
Summary:
Added a config option `experimental.rebase-long-labels`. Once set to true,
produce more descriptive conflict markers "dest (rebasing onto)" and
"source (being rebased)" on conflicts.
Eventually we probably want this to be the default everywhere. For now, not
all users are compatible with this change so it's not yet the default.
Reviewed By: muirdm
Differential Revision: D44847924
fbshipit-source-id: b832f362cc1dd80e25a3fd80e0f334bffc4b12d8
Summary: This avoids revlog and its problematic code paths for all `pull` commands.
Reviewed By: sggutier
Differential Revision: D44805076
fbshipit-source-id: a4efacf111f03cda5752fe1b25d26a278c25f242
Summary: The previous diffs added support in SCS and remote-gitimport for including the `git-extra-headers` as part of the mirrored commits in Mononoke. This diff adds similar support for gitimport.
Differential Revision: D44869271
fbshipit-source-id: 5941b0c8729cd972188cc758837c6c1f1b0f72a1
Summary: The current healing logs just mention that a CMv2 blob was healed, but there is no mention of the actual blobstore key that it healed. This diff updates that.
Reviewed By: mitrandir77
Differential Revision: D44164682
fbshipit-source-id: c156a930716499fb1b5c7b248ce11d6bf88f8281
Summary: To maintain backwards compatibility for a while, we will be writing both the new `ContentMetadataV2` and the old `ContentMetadata` blobs. This will be behind a tunable which will be removed once we can verify that the code works as expected with `ContentMetadataV2`
Reviewed By: markbt
Differential Revision: D44221076
fbshipit-source-id: 95c01a96198a99e53ab3ea4f1b04b1ba78f4b298
Summary: D44136799 was the backout diff for backing out all the ContentMetadataV2 changes made so far. This was due to the reason that the changes were landed before the backfilling was completed. This diff is the revert of the backout, to be landed when the backfilling is properly completed.
Reviewed By: YousefSalama
Differential Revision: D44195175
fbshipit-source-id: 2872c699cc7c41cf436a1c7123e01a5190db7ece
Summary:
Eden's readdir and getfileAttribute endpoint returns an error when an entry in
a directory has a type that is not: regular file, directory, or symlink.
This causes issues for Buck2 because it propagates this error to the user. For
Buck2 a directory having a file that isn't a regular file, directory, or
symlink isn't an error case, it's just a file Buck2 wants to skip over. Buck2
would like to be able to differentiate real errors getting the filetype (like
say a network error) and having a weird file in some directory.
From chatting with Thomas, Buck2 is unlikely to ever care what type the file
is (if its not a file, dir or symlink). So it's sufficient just let buck2
know it's some "other" type of file. I think it makes sense to just add a non
source control type here. I also considered adding dtype as an attribute, but
I don't think we need it, but we could add that too.
In some cases it can be dangerous to add values to thrift enumeration
(SourceControlType enum we change below)
(reference post: https://fb.workplace.com/groups/thriftusers/permalink/785884732120941/).
But in our case, rust + Buck2 handles new enum types gracefully
(and with exactly the behavior we want):
https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/fbcode/buck2/app/buck2_common/src/io/eden.rs?lines=157
so adding a value to the enum is safe (for buck2).
hack is our other client. they are going to handle it less gracefully:
https://www.internalfb.com/code/fbsource/[65673fd318750984372aeb5b44036a259a0d85d2]/fbcode/hphp/hack/src/facebook/hh_distc/package/package.rs?lines=441 but from what I can tell hack would also
error if they tried to list a directory with a socket in it with out this
change. Will confirm with them that this change is ok with them.
Reviewed By: chadaustin
Differential Revision: D44794698
fbshipit-source-id: 4e3ab7964fa2c0932b0363fb9ad62f24af74480c
Summary:
I saw builds/tests were failing for centos due to a warning "egrep: warning: egrep is obsolescent; using grep -E".
We can instead use grep -E as recommended.
Note: this isn't every single test, but somehow only these tests had failures in the Centos CI job.
Reviewed By: muirdm
Differential Revision: D44848131
fbshipit-source-id: ba182e8c866ac6bd649f820b76097e4677295105
Summary: Configure post pull "debugmarklanded" hook in the requisite extension to avoid trying to run the hook when the extension is disabled.
Reviewed By: zzl0
Differential Revision: D44646572
fbshipit-source-id: 1ac2dd97d35616683b87223a97ebbe7aecaa0a91
Summary:
Don't crash on the "m" flag used to mark submodules in the manifest.
Also, deprecrate "sl manifest" in favor of "sl files". The "sl manifest --all" flag doesn't work anymore, anyway.
Reviewed By: sggutier
Differential Revision: D44772327
fbshipit-source-id: bad76f0defe95c4a11df9b606e56983583dc5e32
Summary: We offer the `--force` flag with the python implementation, this adds it for the C++ implementation in preparation for migration
Reviewed By: kmancini
Differential Revision: D44473785
fbshipit-source-id: 041613ff0d4048b40b02f7d8f024c78d659516f0
Summary:
Noticed that this was running while looking at a log on Linux. This does nothing on Linux/macOS besides loading inodes (ie: extra overhead), thus disable it.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D44835074
fbshipit-source-id: 3b80e4ccf0f63d50de395002c6a8d05a5e99952c
Summary: Fixes a wacky bug with one of our internal tools
Reviewed By: pushpakrajgautam
Differential Revision: D44811936
fbshipit-source-id: f3b6f07ff035c8d535792dec5add5146c1314f36
Summary:
Grammar update:
"eden debugedenimporthelper command command will be sent to this file."
Removed repeated ‘command’
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: chadaustin
Differential Revision: D44799690
fbshipit-source-id: 03e70a8ec300d2b635ea02f9e03802a4f44e9d32
Summary:
We have a force-unmount-all.sh script as a last resort when a Linux
machine has a pile of stuck FUSE mounts.
Introduce a new script that only unmounts the ones under $TMPDIR,
which were probably created by failed integration tests.
Reviewed By: mshroyer
Differential Revision: D44797173
fbshipit-source-id: 41f31cf1e2e57ad07076cf1b2571c739b67b77f0
Summary: fix clippy warning by using replacing match with if let
Reviewed By: quark-zju
Differential Revision: D44728168
fbshipit-source-id: 27a944267263595ea9367c08f6716a3d2f6820bb
Summary:
we need to check if the path is in the target commit when no rename
commits found, this fixes a bug found in the previous test
Reviewed By: quark-zju
Differential Revision: D44728170
fbshipit-source-id: 7a039b0f9fee80671653e1557b80b218b58f4e99
Summary:
refactor vertex_to_tree_manifest to only process one commit, so it can be used in the following
diff to check if a path is in a commit
Reviewed By: quark-zju
Differential Revision: D44728172
fbshipit-source-id: fc9530b38757d836ca8a5a46e463a5ade510839d
Summary:
this diff adds more unit tests to cover: multiple renames, non-linear
comit graph, deletes.
This diff contain tests catch a bug in the current implementation, which
will be fixed in the following diffs.
Reviewed By: quark-zju
Differential Revision: D44728169
fbshipit-source-id: b0aaeeffa9008a70bbfe3105cceed3afb3d65d45
Summary: Makes it explicitly mention that `sort` method sorts the set in topo descending order, this is different than the `sort` function in revset, which sort the set in ascending order by default.
Reviewed By: quark-zju
Differential Revision: D44728173
fbshipit-source-id: aa95993b237ebacc587147da9caccdf43ff48b80
Summary: Add test utilities for testing copy trace logic, also added a simple test case as an example.
Reviewed By: quark-zju
Differential Revision: D44559585
fbshipit-source-id: b03796fa3869d94edc528d570d511b1d84b15861
Summary:
add vertex_fn parameter to ImportAscii trait, so that we can control
how to generate Vertex from a string: for example HgId expects 20-bytes vertex.
Reviewed By: quark-zju
Differential Revision: D44739947
fbshipit-source-id: c82c95ce4d6c45562c32ad4ccc3e5b40fd517770
Summary:
Even though we only need the Enumeration in the future's callback, if we don't
move the Enumerator it can get dereferenced while we're awaiting the future
returned by prepareEnumeration().
Reviewed By: xavierd
Differential Revision: D44770398
fbshipit-source-id: a9fc91c0766b4a9f43f566887d714cbde66b1cf3
Summary:
This allows errors to be caught and raised as well as getting file sizes bigger
than off_t which is 32-bits on Windows.
Reviewed By: chadaustin
Differential Revision: D44737139
fbshipit-source-id: 16b00c7f436da361b41ffbe8074958519e9bfc81
Summary:
When the FileInode is materialized, the NonMaterializedState is entirely unused
but still consume memory. On Windows, we've seen cases where the working copy
contains 100k modified files leading to diff taking a really long time due to
EdenFS spending most of its time computing the sha1 of materialized files,
since these are not cached, this computation would be done repeatadly.
For now, this merely creates the scaffolding to enable this caching, future
diffs will move the sha1 and sizes currently stored in the Linux/macOS
OverlayFileAccess in the MaterializedState and care will be taken to invalidate
these on materialization/writes.
For now I've gone with a `union` approach as the existing `Tag` is sufficient
to know which branch of the `union` should be looked at. This is however
potentially error prone, reviewers' opinion on whether I should revamp this and
switch to a `std::variant` would be very welcome.
Reviewed By: chadaustin
Differential Revision: D44652779
fbshipit-source-id: e769f4cc0cd7e8eea0c4cde863cd37f4ceadb01f
Summary:
Tweak revlog commit import to maintain a stable order (this broke after recent change to top sort the commits).
Unfortunately the new stable order is inconsistent w/ fullsegments output in test-eager-exchange.t, so separate the tests.
Reviewed By: quark-zju
Differential Revision: D44768930
fbshipit-source-id: 6757450d7d446373d1b8c943d6cbcc6029b07443
Summary:
This works better with Git. The old filenode-based logic does not seem correct
since it might not trigger adjustlinkrev.
Reviewed By: zzl0
Differential Revision: D44751613
fbshipit-source-id: e5a2367e544f1d55c6a2c149fd33db13af623e46
Summary:
There is a collision between Blob and BlobMetadata ObjectId causing crashes in
EdenFS. To avoid these collision, let's just split the request tracker. Kudos
to chadaustin for finding the root cause of the crashes.
Reviewed By: chadaustin
Differential Revision: D44728974
fbshipit-source-id: 52bbc3f5317342ed975507f0a431bf4d3f712d8b
Summary:
return usage of commit graph for revlog clones
Disabling is no longer needed after D44734552
Reviewed By: quark-zju
Differential Revision: D44760863
fbshipit-source-id: 0ba65f63bf8c980fc0ab9b0dce1ec895f6b5f10d
Summary:
I'm getting nfs to run on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change theoretically we should be able to mount an NFS repo.
This isn't totally true, because I cannot get the Windows NFS client to point
at a specific port, so some port mapper changes are probably going to be needed
before that really works.
NOTE: this one is more than removing ifdefs, please review carefully
Reviewed By: xavierd
Differential Revision: D44155041
fbshipit-source-id: ae424a01dc2073138070253d80e3d101863840fd
Summary:
I'm getting nfs to run on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change the NFS Server opens a socket to accept incoming connections!
Reviewed By: xavierd
Differential Revision: D44154505
fbshipit-source-id: 10c679cb390c2bab99591656d6fe96ccdf4443fe
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change NFS builds on Windows!
NOTE: this one is more than removing ifdefs, probably the most non trivial in the stack, please review carefully
Reviewed By: xavierd
Differential Revision: D44153443
fbshipit-source-id: b07e19f8bde1aa455eec97647ea450849133041f
Summary: Sometimes it's desirable to set the value for a by-repo tunable for all repos (e.g. to roll out a change everywhere after testing on some of the largest repos, or to quickly roll back) or to specify a default value, and right now the only two ways of doing this is either to specify the value for every single repo separately, or to have another non-by-repo tuanble that overrides it. Instead let's have two special keys: ":override:" and ":default:", so whenever we try to lookup a by-repo tunable we first check if it's set for ":override:" and use its value if so, otherwise we lookup the repo specific value, otherwise we lookup the value for ":default:".
Reviewed By: markbt, RajivTS
Differential Revision: D44750394
fbshipit-source-id: d8fe074ec700f69e52129dfa7de87b7972d11012
Summary:
This continues my slashing of fetch retries. This diff is -1 fetch retry.
DatapackStore::getTree is not a local only tree access. It may fallback to
fetching from edenAPI.
DatapackStore::getTree is used in the hg importer fallback code path for when
normal edenAPI rust binding fetching fails. It's unnessecary to be doing this
extra network fetch here, because we only want one edenAPI attempt and then to
fall back to the importer.
If I had to guess why this is here:
IIRC this method use to be a local only access to the hgcache. That seems to
line up with the logging comments around this fetch. So I think when the
datapack store gained remote fetching capabilities this one just wasn't changed.
With this change we are now down to what looks like 5 network fetch attempts.
Reviewed By: xavierd
Differential Revision: D44523824
fbshipit-source-id: f5e784660b396ad3a8f1e7effcd7551c3b126521
Summary:
intro high level transaction for cloud sync
this is for syncs that are not running with best effort option
the transaction is needed to eliminate repo changed while backing up errors
this is a follow up after https://www.internalfb.com/diff/D41871718 that hasn't resolved the errors fully
Reviewed By: markbt
Differential Revision: D44664729
fbshipit-source-id: da411cd20fb909c22ff8976dd31bd46034a28226
Summary:
In certain edge cases, during pull we are using EdenAPI to fetch nodes and feeding them into a revlog changelog (via doublewrite). This was erroring out since the RevlogCommits.add_commits expected nodes to be sorted parent-before-child. Work around by sorting the nodes.
[MSDK] Update icsp_domain_config_type_linter component on ICSP:master
```
Reviewed By: quark-zju
Differential Revision: D44734552
fbshipit-source-id: 7f00a93b93921baa827a55c7c1ec9388b74ac0f7
Summary:
There's no need to wrap TraceSubscriptionHandle in another struct or
manage its lifetime with shared_ptr. It looks that was done in one
place and then cargo-culted elsewhere.
Reviewed By: xavierd
Differential Revision: D44657251
fbshipit-source-id: d5563626c52192be3e2fa44de2f4819ca3ebe1ec
Summary:
TraceBus isn't easy to use correctly. Clarify some comments to make
the rules more explicit and unify the two constructors to allow
(noexcept) entry construction in place.
Reviewed By: xavierd
Differential Revision: D44657217
fbshipit-source-id: 1fa7c0e3c4ffb169be2b7b0cd1ffa2ea07dfeeb1
Summary: Buck1 is EOL and these tests are thus no longer necessary.
Reviewed By: chadaustin
Differential Revision: D44717980
fbshipit-source-id: bfe0d9977243c35405e1b5cc988b687369488d0c
Summary:
There is a new `Target::Pipe` that could be used to redirect logs to a file. So updating to the latest version which doesn't add any breaking changes.
Also, updating the deprecated call sites:
```
xbgs -sl 'env_logger::from_env' \
| arc linttool debugfilterpaths --take RUSTFMT \
| xargs sed -i 's/env_logger::from_env/env_logger::Builder::from_env/'
arc f
```
Reviewed By: dtolnay
Differential Revision: D44671457
fbshipit-source-id: 6734f2839731b189564d3b7ceffcd95410fe02e8
Summary: There is a typo in `_clonetotmp` that makes it crash. It should be `ident.configrepofile` instead of `ident.configerpofile`
Reviewed By: muirdm
Differential Revision: D44709074
fbshipit-source-id: 60160d68eea6b5cfafa10c5a053ade44a54724a5
Summary: This is deprecated and Thrift is warning about it, remove it.
Reviewed By: fanzeyi
Differential Revision: D44695237
fbshipit-source-id: 6f91f50574d2898d98cfb9c732c2d15e5789087a
Summary:
This experimental option allowed you to automatically "jump" past the specified revisions when annotating a file. The idea is you could skip big codemods/renames etc. that touch a lot of lines without meaningful changes.
However, no one uses it. The idea is good, but to use it you need to know a priori what revs you want to skip. A better integration might let you configure known annoying revisions, or maybe let you filter commits by their commit message.
Plus, any decent blame UX allows you to easily jump passed boring commits.
Reviewed By: zzl0
Differential Revision: D44596603
fbshipit-source-id: 3e5707bb2e14aecaf4f56388afb7eb49951b6d31
Summary:
If experimental.edenapi-blame is configured and the repo has an edenapi client, the annotate command will fetch blame data from the new blame endpoint.
Currently this doesn't support any of the whitespace options since the server only provides whitespace sensitive blame data. We can probably emulate the whitespace insensitive behaviors locally with more work.
Reviewed By: mitrandir77
Differential Revision: D44596604
fbshipit-source-id: 73d7462d0ded2d6941b704dfb0294e52f462879a
Summary:
This handler simply exposes the existing blame derived data over EdenAPI.
The main work is re-indexing the commit index so it can be a simple list instead of a VecMap.
Any error is propagated in-band in the results. I was a little surprised to learn that errors within the HandlerResult stream are ignored (i.e. not propagated to client). It seems like the stream items should have an envelope type to allow automatic Result propagation from server to client.
Reviewed By: quark-zju
Differential Revision: D44596606
fbshipit-source-id: 0ae9d501a35e3e930fd4cbe710fd3e76f493312c
Summary:
The one existing user has MPath in hand (and so does my upcoming code).
Also, defer the error context to avoid copying bytes (again) when there is no error. Note that the ParseError returned by RepoPathBuf::from_utf8 itself contains the bad bytes, but there doesn't seem to be a good way to share those bytes without losing context.
Reviewed By: mitrandir77
Differential Revision: D44596605
fbshipit-source-id: d9b1f635a7e5b90828f934707845a2efcb5c08a5
Summary:
These mirror the Mononoke BlameV2 derived data.
I tweaked the #[auto_wire] macro to support a new #[no_default] attribute. This made it possible to stick a Result<_, ServerError> into my type without having to hand implement anything else (since Result doesn't implement Default, which otherwise was required for #[auto_wire]).
Reviewed By: quark-zju
Differential Revision: D44596607
fbshipit-source-id: 9f2cc22d9ef43deefd32d4cfff67d350045ea845
Summary:
Original commit changeset: e0a130c04c80
Original Phabricator Diff: D44694398
Solving the `arc feature` problem in favor of D44695559
Reviewed By: muirdm
Differential Revision: D44713896
fbshipit-source-id: 4b870d0288e6e81577d8d398b1fb42d4e8b68dc9
Summary:
Currently WBC on each bookmark change will try to find all underived cs_ids and will request derivation for them one-by-one. This is redundand because both old path (using derivation manager) and new path (DDS) doing that discovery internally and will derive all dependency.
This diff just removing the part when WBC asks for derivation of each commits and just requests the latest underived commit, however it will still doing discovery of underived commits.
The next step would be to add an API for DDS to answer queries about derived bookmarks locations. This way we could remove discovery part completely.
Reviewed By: RajivTS
Differential Revision: D44536860
fbshipit-source-id: 3867abdf095141b89da44242a35327da02ab05f8
Summary: Recently D44515578 removed remotenames from the set of names, but internal internal tooling was depending on that.
Reviewed By: fanzeyi
Differential Revision: D44694398
fbshipit-source-id: e0a130c04c803e51a6e7f8a284b572047ffc0ee9
Summary: Now that the dependent logic is in place, this diff parses the git commits and extracts the git headers. It then passes these git headers while creating the corresponding changeset at Mononoke end using SCS's `repo_create_commit` method.
Differential Revision: D44705976
fbshipit-source-id: 464695e273908ef7642514b08158be4affc32ac2
Summary:
Introduce **h2** support into mononoke server.
Remove alpn crate that repeats the standard "select_next_proto" method
Server now offers the following protocols: hgcli, h2, http/1.1 in this order in protocol negotiation.
We will select the first protocol supported by the server which is also supported by the client.
This will add h2 support server side but won't change protocols currently used (because h2 is not offered client side right now, while http/1.1 is explicitly set).
We then later will switch **only** Eden Api Traffic to H2 by enabling the following config client side via slow rollout:
```
[edenapi]
http-version=2
```
We might need to tune some settings on proxygen side before that.
Wireproto traffic will continue to be obliged to set http/1.1 client side explicitly, which is totally fine because it needs web socket upgrade.
### At a high level, HTTP/2:
* is binary, instead of textual
* is fully multiplexed, instead of ordered and blocking
can therefore use one connection for parallelism
* uses header compression to reduce overhead
* allows servers to “push” responses proactively into client caches
Reviewed By: mzr
Differential Revision: D44540283
fbshipit-source-id: 560d80d68d5d364adee122881d12cf61ba9f1695
Summary:
Double writing is enabled for all repos for a long time, so it's safe to delete enable_writing_to_new_commit_graph.
enable_reading_from_new_commit_graph was never used since we needed more granular tunables.
Reviewed By: Croohand
Differential Revision: D44624559
fbshipit-source-id: a06a0f2079bb735e93d72d0d24bbc7ae684600a7
Summary:
Our logs are currently full of messages about the duration of
semifuture_getScmStatusV2(), which isn't very helpful. This drops the default
level to DBG3 so that we only log at DBG2 when the request takes longer than
one second.
Reviewed By: xavierd
Differential Revision: D44693481
fbshipit-source-id: 9ff8e04ab39e960050e64930a023949962360b9c
Summary: As title says. Compiling on Windows has been kind of annoying lately due to the number of warnings I've seen, which this cleans.
Reviewed By: muirdm
Differential Revision: D44690961
fbshipit-source-id: 37ae8e48e4aa181ac909b3ec90c76a1438b24489
Summary:
There is an issue where children sl processes inherit SCM_SAMPLING_FILEPATH and write to the same file as the parent process. This messed up the sampled data since the child's "command_duration", for example, could clobber the parent's.
Fix by only allowing one process to create SCM_SAMPLING_FILEPATH. This way the child (e.g. backgroundfsync hook) will not be able to output metrics.
Reviewed By: zzl0
Differential Revision: D44645799
fbshipit-source-id: 11f41343107802f9fd9ae2dc0fc7a9e1925859d8
Summary: Expose sampling-category-for-key and append-sampling-event methods from Rust to Python to replace relevant logic in ui.py. This consolidates things, and may fix the issue where corrupted keys like "foo{" were getting into the sampled data.
Reviewed By: zzl0
Differential Revision: D44645798
fbshipit-source-id: e158ed7434435e1d29222cda2fc1cf0ee345010a
Summary: This makes it easier to share the sampling config with Python. In particular I want to get down to only having a single place open the SCM_SAMPLING_FILEPATH file to make it easy to fix a couple bugs (see next diff).
Reviewed By: zzl0
Differential Revision: D44645800
fbshipit-source-id: 12a0da6d1356b6c116224a3c3552692bd5aa656a
Summary: I want to expose the sampling file to Python, so first let's move it to a non-tracing related crate.
Reviewed By: zzl0
Differential Revision: D44645801
fbshipit-source-id: 14da917acdc3529e17502d81846229e3ae8854e2
Summary:
On Windows, this should always be set to true. Unfortunately, we've rolled out
a bad EdenFS release that overwrote all of these configs and set them to false,
breaking several users.
Reviewed By: fanzeyi
Differential Revision: D44683911
fbshipit-source-id: 4d8efb3402f967b2e35fd333c858fe939307e6f0
Summary:
On Windows, the sqlite overlay is always enabled despite what the config says,
encode this in the code.
Reviewed By: MichaelCuevas
Differential Revision: D44682918
fbshipit-source-id: 751a763a59f43ebcac30b8884eaf1fbd4033d1cf
Summary:
The latest "main" contains all the patches that we were carrying on my
fork so far. Bonus point, hyperx is not a depedency anymore so we can also get
rid of this.
Reviewed By: zertosh
Differential Revision: D44682428
fbshipit-source-id: cbb1da18edce7478a5454d1a3ba70bd8767e089c
Summary: The hg stable commands fail to work on Windows. This is because the hg code calls a shell script to run the necessary hg commands. This diff adds a port of the shell script into python to let it work on Windows.
Reviewed By: skotchvail
Differential Revision: D44591662
fbshipit-source-id: 5d376702ccc648fe01f541b2d673ae39f954739b
Summary:
[Code Tour](https://github.com/microsoft/codetour) is an extension which allows you to annotate an ordered list of code pointers with descriptions, as a way of touring you through a codebase.
This diff adds a couple of tour files which are loadable by anyone who has the extension and loads this tour.
This tour walks through how we process "write requests" in the PrjFS version of Eden.
Reviewed By: chadaustin
Differential Revision: D44654254
fbshipit-source-id: e7e00e301e9445cc4f5875689a35c40bf5ad8b99
Summary:
[Code Tour](https://github.com/microsoft/codetour) is an extension which allows you to annotate an ordered list of code pointers with descriptions, as a way of touring you through a codebase.
This diff adds a couple of tour files which are loadable by anyone who has the extension and loads this tour.
This tour walks through how we process "read requests" in the PrjFS version of Eden.
Reviewed By: chadaustin
Differential Revision: D44645249
fbshipit-source-id: 5a3a620139d95f9298d9b0ce81064d49c1c297d2
Summary:
This is a slight behavior change, but I need queue events published to
ActivityBuffer so I can track the live set of fetches.
The impact will be that `eden trace hg --retroactive` will show 2/3 of
the number of events that it did. We can bump the capacity if that's
problematic.
Reviewed By: xavierd
Differential Revision: D44594520
fbshipit-source-id: 5b22169403402d729ff929b4b4bec6bb76d54737
Summary:
ActivityBuffer is enabled by default and we rely on `eden trace hg
--retroactive` working. For HgQueuedBackingStore, unconditionally
enable ActivityBuffer, and give a capacity config setting independent
from the others.
Reviewed By: xavierd
Differential Revision: D44593656
fbshipit-source-id: 284bad0d19315d250a4ab69f06439393dbeaf5d1
Summary: We only capitalize type names.
Reviewed By: xavierd
Differential Revision: D44593407
fbshipit-source-id: 9f0d841e6ff5c40d6a6c6ca3637a40fefcf5d084
Summary:
Note: this is a re-land of D44623815, which was was reverted because of a land race with D44626072
In particular this fixes `FlattenUnordered` having a nasty deadlock, but also
some other bugs:
```
# 0.3.28 - 2023-03-30
* Update to syn 2. This raises MSRV of utility crates to 1.56. (#2730, #2733)
* Fix bug in `FlattenUnordered` (#2726, #2728)
# 0.3.27 - 2023-03-11
* Add `TryFlattenUnordered` (#2577, #2590, #2606, #2607)
* Add `AbortHandle::is_aborted` (#2710)
* Add `AbortRegistration::handle` (#2712)
* Make `BiLock` strict-provenance compatible (#2716)
# 0.3.26 - 2023-01-30
* Add `Either::as_pin_mut` and `Either::as_pin_ref` (#2691)
* Add `Shared::ptr_eq` and `Shared::ptr_hash` (#2691)
* Implement `FusedStream` for `Buffered` (#2676)
* Implement `FusedStream` for all streams in `ReadyChunks` (#2693)
* Fix bug in `FuturesOrdered::push_front` (#2664)
* Remove `Fut::Output: Clone` bounds from some `Shared` methods (#2662)
* Remove `T: Debug` bounds from `Debug` implementations of `mpsc` and `oneshot` types (#2666, #2667)
# 0.3.25 - 2022-10-20
* Fix soundness issue in `join!` and `try_join!` macros (#2649)
* Implement `Clone` for `sink::Drain` (#2650)
# 0.3.24 - 2022-08-29
* Fix incorrect termination of `select_with_strategy` streams (#2635)
# 0.3.23 - 2022-08-14
* Work around MSRV increase due to a cargo bug.
```
Reviewed By: zertosh
Differential Revision: D44632588
fbshipit-source-id: bdd87cb02b3aef63a65b1f9b852579225adfedbd
Summary:
When this check was previously enabled, we saw 2 issues from users:
- The check failing due to status taking more than 5s,
- The check having false positive.
The first one can be fixed by increasing the timeout on the Thrift client, the
second one is anything but clear at first sight. Digging a bit deeper, one
issue became apparent. In:
if modified_file not in diff:
The type of `modified_file` is a `Path`, while the type of `diff` was a
`Set[str]` (the type annotation was wrong). If we manually test this, here is
what we get:
% python3
Python 3.8.6 (default, Feb 10 2023, 17:15:29)
[GCC 11.x 20221024 (Facebook) 11.2.1+] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> a = {"a/b.c", "c/d.c"}
>>> a
{'c/d.c', 'a/b.c'}
>>> Path('c/d.c') in a
False
>>> a = {Path("a/b.c"), Path("c/d.c")}
>>> Path('c/d.c') in a
True
>>>
The check would have thus failed, which would have led to false positives.
Lastly, this also fixes a potential false negative, where the number of
modified files could be the same as the files in diff when the latter includes
an added/remove file but is missing one of the modified file. Removing the
check comparing both is sufficient to avoid this issue.
Reviewed By: chadaustin
Differential Revision: D44558090
fbshipit-source-id: 0cc83a87758a5feeff78c38b210ebd91fa5d58f5
Summary:
Context
==
When the code being changed is called from a test or program using Python 3.10, deprecation warnings appear.
`DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead`
Python 3.10 is rolling out in fbcode, so it will be nice to have one less DeprecationWarning.
Implementation
==
* sed -i -E 's/(\s+)(.*)\.setDaemon\(True\)/\1\2.daemon = True/'
Reviewed By: muirdm
Differential Revision: D44620221
fbshipit-source-id: b243b5e8947c918f865efa3c46ffdc8c0e530243
Summary: These are the tests that passed after adding "$ eagerepo". I attempted to turn on eagerepo for all tests that didn't contain "$ eagerepo" or "configure modern".
Reviewed By: quark-zju
Differential Revision: D44445379
fbshipit-source-id: b283be6d5249f74e11a3205622961e457587e93b
Summary:
While reading through our uses of ActivityBuffer, I noticed an
opportunity to amortize the cost of publishing import events.
Add some comments to note that.
Reviewed By: xavierd
Differential Revision: D44643270
fbshipit-source-id: 1b68f904b86d91fab91af01a4e8b169dc381ad33
Summary:
This should reduce the number of allocations when pushing and popping
events into ActivityBuffer.
Also, remove -inl.h because our clangd does not support it.
Reviewed By: xavierd
Differential Revision: D44593366
fbshipit-source-id: 7dc0f2aa457b44bebe9471edd3c7e688d09534f5
Summary:
`std::deque` has poor performance and bucketing in GNU libstdc++, and
sometimes we just need a ring buffer without paying for folly's
LockFreeRingBuffer.
Introduce a simple RingBuffer atop `std::vector` for later use in
ActivityBuffer.
Reviewed By: xavierd
Differential Revision: D44593248
fbshipit-source-id: 13303c27b184002fc74aec9ae6e9133c70bab6bb
Summary:
based on the doc of writing extensions, post-* hooks should be setup in the `uisetup`
```
10.1. uisetup
* Changes to ui.__class__ . The ui object that will be used to run the command has not yet been created.
...
* Setup of pre-* and post-* hooks
10.2. extsetup
* Changes depending on the status of other extensions. (if extensions.find('mq'))
* Add a global option to all commands
* Extend revset
10.3. reposetup
* All hooks but pre-* and post-*
* Modify configuration variables
* Changes to repo.__class__, repo.dirstate.__class__
```
Reviewed By: sggutier
Differential Revision: D44632035
fbshipit-source-id: f37bb052a2ab225d43cf7f305b0bcccedb4add9c
Summary: When calling `fill` methods on a no-op memcache, we do all the work to serialize the data and then throw it away. Add a quick check at the start of the fill method to exit early.
Reviewed By: YousefSalama
Differential Revision: D44503801
fbshipit-source-id: fd490bb5cce70ee1bca931e55e1671faec09e894
Summary: The `Prefetch::Hint` variant indicates that prefetching is possible, but should not be acted upon unless an intermediate caching layer determines that the prefetch should be included. That means when we ask for the target, we shouldn't return a hinted target.
Reviewed By: liubov-dmitrieva
Differential Revision: D44498446
fbshipit-source-id: 56a0fb3b09de985cca9989335375daf972e43276
Summary: If we have missed in cachelib, and been asked to prefetch, then memcache fetches are actually slower than the prefetch would be, even if memcache is warm. Instead, go straight to the database and prefetch from there.
Reviewed By: mitrandir77
Differential Revision: D44498103
fbshipit-source-id: 944c26d553a770e054f38f478cf2bd475afa413f
Summary: When computing ancestors difference, we need all of the commits on the `head` branches, not just the skew binary steps. Prefetch all of them.
Reviewed By: YousefSalama
Differential Revision: D44498105
fbshipit-source-id: acf08930361f03c76685a94fff2c5a41a99089b3
Summary: Make the number of steps taken a parameter of prefetching, so that we can vary it based on prefetch type.
Reviewed By: YousefSalama
Differential Revision: D44498104
fbshipit-source-id: 9d24af68ea620fd11a18d9dbaa822fc9848fdd3c
Summary: If this config was not set, it would default to `false`, and we would not use it. Instead, make these configs optional and hardcode the default.
Reviewed By: fanzeyi
Differential Revision: D44635468
fbshipit-source-id: 2464439dc20ba90745b9783f491514c5d63e2c76
Summary: In preparation for extending this to a paramatarized test for different `InodeCatalog` types, this generalizes the object types in the already existing test
Reviewed By: chadaustin
Differential Revision: D44633728
fbshipit-source-id: d06a9ed146ad5ea65f11f0aa16bd2725aec604dc
Summary:
We made 'pr_marker' run inline instead of as hook in D43276199, but this can cause
nested event loop runtime error in `sl pr pull`. This diff makes 'pr_marker' run as a
Python hook to avoid the nested event loop.
The difference between current hook (this diff) and previous hook is that we are running
it as a Python object instead of shelling out to `sl` command.
https://github.com/facebook/sapling/issues/58
Reviewed By: quark-zju
Differential Revision: D44531268
fbshipit-source-id: 5270b7172d7b2c22a9c9e715d7cb5b42181cff53
Summary:
Some Python versions don't return the correct macOS version from
`platform.mac_ver()`, so let's use `/usr/bin/sw_vers` instead.
Reviewed By: xavierd
Differential Revision: D44594569
fbshipit-source-id: ee52f11aad76361b780845de8218b2f365f0ecfe
Summary:
Using an EdenConfig instead of a GFlags allows for rolling out config changes
automatically while GFlags do not allow this.
Reviewed By: chadaustin
Differential Revision: D44593575
fbshipit-source-id: 2b3e6858c031ba63b89787c3456542e79671d4ea
Summary:
`eden top` would previously crash if its terminal window was resized too small,
due to incorrectly specifying a negative padding width in a format specifier.
Reviewed By: xavierd
Differential Revision: D44564714
fbshipit-source-id: 936f346ce7e7f5cb0f18cdbffa269cb0fd06be91
Summary:
I'm introducing a new Rust-only command implementation, and I never
want it to fall back to Python help, even if it's run without a
required option. Therefore, handle
DisplayHelpOnMissingArgumentOrSubcommand as well as DisplayHelp.
Reviewed By: fanzeyi
Differential Revision: D44560239
fbshipit-source-id: 43c418da3c980a34a11277c82bf350822b0bc71b
Summary:
This config is already hardcoded to default to true on Windows in the C++ where it is consumed, and is set to true during clone, but in the case where the config is not set on disk, we should also default it to true in the python.
```
def _get_enable_sqlite_overlay(
self, instance: EdenInstance, overlay_type: Optional[str]
) -> bool:
if overlay_type is None:
# The sqlite backed overlay is default only on Windows
return sys.platform == "win32"
return overlay_type == "sqlite"
```
```
auto enableSqliteOverlay =
repository->get_as<bool>(kEnableSqliteOverlay.str());
// SqliteOverlay is default on Windows
config->enableSqliteOverlay_ =
enableSqliteOverlay.value_or(folly::kIsWindows);
```
Reviewed By: chadaustin
Differential Revision: D44544561
fbshipit-source-id: 78f3b7c72934a377ffedcd6503b6948d282d2f49
Summary:
They were in random order. Sort them alphabetically before adding a
new one.
Reviewed By: fanzeyi
Differential Revision: D44525570
fbshipit-source-id: a574b6c377f76b4fbad5d7210f772a215ad35cf1
Summary:
Display is a stronger constraint than what we need, which is a way to
get `&`static str` from a subcommand.
Reviewed By: fanzeyi
Differential Revision: D44525506
fbshipit-source-id: 7fa2f8665ed4c986c25aa58197083e93b5b059cc
Summary:
This test used to be detected as flaky in mode/dev-rust-oss and flaky, but not detected as such in mode/opt.
This was due to a race condition between fresh and warm bookmarks, which take a while to update due to WBC.
Using the helper functions to wait for bookmarks to get warm, or directly querying fresh ids, fixes the issue.
Reviewed By: RajivTS
Differential Revision: D44543648
fbshipit-source-id: f70b16609fe995341b84a77e590efdc3d71b4fdf
Summary:
D44297278 was an attempt at fixing this test, but the regex for lines with DEBUG
wasn't actually correct as it's not actually the first pattern in the strings being grepped.
It may be preceded by some spaces.
Yesterday, we saw a contbuild breakage caused by this.
https://www.internalfb.com/intern/sandcastle/job/9007200167370789/insights
Relax the grep step to achieve the intent of the previous diff.
Reviewed By: RajivTS
Differential Revision: D44536438
fbshipit-source-id: a40719f4669b55b2442715bc9350b8dcce7a9747
Summary:
Running tests with network access disabled is much slower. Let's separate those
test targets out so on CI we run with and without network access disabled. For
iterating during developement the tests with network access enabled are
usually enough.
In this diff I:
* add `disable_all_network_access_target` option to `dott_test` rule that adds the extra test target disabling network access
* convert existing opt-out list into options on test target - this required separating some targets as the config is now on per-target not per-test level
Reviewed By: clara-9
Differential Revision: D44474156
fbshipit-source-id: 780fc81c24b22e2dcae98901cd604dc50f16c70c
Summary: This rollout has long been finished (the initial rollout from sqlite -> tree). However, I recently renamed tree -> sqlite, which can trigger incorrectly trigger this check since we assume `enable-tree-overlay/enable-sqlite-overlay` is false if its not in the config, but the config is only set to true upon clone. This behavior does not affect the C++ b/c we only write the config during clone.
Reviewed By: xavierd
Differential Revision: D44523497
fbshipit-source-id: ef19af0932fb8f7747bd7b90b8eabe92c1ddc13c
Summary:
This removes unnecessary quadratic work during directory enumeration. Instead of iterating over the entire set of directory entries and performing string matching on each of PrjFS's invocations of `getEnumerationData`, we now compute this once per enumeration by keeping state in a new `Enumeration` type.
With this change, listing with cmd.exe's `dir` becomes fast. However, PowerShell core 7's `ls` can still be slow because it seems to devolve to single-entry (PRJ_CB_DATA_FLAG_ENUM_RETURN_SINGLE_ENTRY) fetches past a certain directory size. Further work will be needed to make `ls` fast in PowerShell.
This improves performance listing a directory with 151433 entries:
| | Dev `dir` | Opt `dir` |
|--------|-----------|-----------|
| Before | 15m23s | 20s |
| After | 13s | 5s |
Reviewed By: chadaustin, xavierd
Differential Revision: D44150897
fbshipit-source-id: 251171df800eec6f1f917b65abeea37c3808f2fa
Summary:
On Windows, the `NtQueryDirectoryFile` low-level API exposes some behaviors—such as restarting partway through a directory enumeration, setting filename filters, and requesting single entries at a time—that are implemented by our ProjFS provider, but were not previously covered by our integration tests.
This change adds coverage by exposing `NtQueryDirectoryFile` via a Python extension module, then exercising it on an EdenFS mount in a new test.
Reviewed By: xavierd
Differential Revision: D44356956
fbshipit-source-id: 4114a0be95092b8276156ba7fd895f64d9e64c3a
Summary: Tweak Rust status code to explicitly treat symlinks as regular files if the VFS doesn't support symlinks. This fixes weirdness where they were previously considered "untrackable" and wouldn't show up at all in "hg status".
Reviewed By: sggutier, zzl0
Differential Revision: D44521987
fbshipit-source-id: 8944ab115635268c997e5cb443d870b71c353f83
Summary:
Now that EdenFS can fetch blob metadata from the server, let's make sure to
plug it to the `eden debug blobmeta` command.
Reviewed By: kmancini
Differential Revision: D44186943
fbshipit-source-id: a64f1384cf312e3c677505c330cfc82469fb83f3
Summary:
Now that the Mercurial backingstore knows how to fetch aux data, let's thread
this through the ObjectStore.
Reviewed By: kmancini
Differential Revision: D44110102
fbshipit-source-id: c57da05066d80fee199e45b4a4223168a196e3de
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44153341
fbshipit-source-id: 8351c72e1ad4dd41090a6289aadebc6749bdf056
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44153119
fbshipit-source-id: 4d6034ea3c7c417d69da5ce54256ae8751ba2a73
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44153042
fbshipit-source-id: 012adea3c89e9c4b37c08cff685e3937d2c6678b
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44152440
fbshipit-source-id: 70de1a7dfd986b60e57079dd9f1d20a6b63dbc0f
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44152981
fbshipit-source-id: fb2788fd2334ad0debf3d45bcb6bb04ca5667b1f
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44152888
fbshipit-source-id: d213f87921d4505429aea42bfa1007a97f89f9dd
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
NOTE: this one is more than removing ifdefs, please review carefully
Reviewed By: xavierd
Differential Revision: D44152305
fbshipit-source-id: ee39db08a53762f23bf184cc6dec41ec28627cda
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
NOTE: this one is more than removing ifdefs, please review carefully
Reviewed By: xavierd
Differential Revision: D44152041
fbshipit-source-id: 21affcc417dce492c2846902225686334eb2dc30
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44151904
fbshipit-source-id: c14fa313b2817c3d55f616f4252a0611965ead78
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44151833
fbshipit-source-id: 5d39f9d3de527cd21506b8473555f9dbd2a587bb
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
Reviewed By: xavierd
Differential Revision: D44151636
fbshipit-source-id: 0f13b997064d96995f48b658ccf0ecac6477dad8
Summary:
The external OSS build is broken because thrift has a really long path and
cargo/git on Windows do not lick such long paths.
I'm not fixing that. But the external OSS build is the only one we run in CI any
more. In the meantime the internal build has broken because no one has been
watching.
There were 4 different breakages.
Reviewed By: chadaustin
Differential Revision: D44189633
fbshipit-source-id: 2eedbc2b3bbf5d1def075d99f11f2273dbb1f4ab
Summary:
Currently a single eden fetch causes 13 attempts to fetch the data.
By default hg hs 3 configured retries for edenapi fetch operations.
In the direct rust fetching code we have disabled these retries because we
use the importer as a retry, so these are not worth it.
We should disable them in the importer too.
There are still many retries. The importer internally will attempt 2 retries
and eden retries the importer once when it fails. So there are at least 5
attempts to fetch data still.
Those other retries are perhaps ensuring correctness, so we need to be more
careful with them. But these should be safe to remove.
This is a 3x reduction in retry attempts.
Reviewed By: fanzeyi
Differential Revision: D44482325
fbshipit-source-id: 96a501c8e9421e5d8aef92b697a6ed5e19c69194
Summary:
Fixing the asciitransform issue, unblocks a lot more tests.
This enables all the ones we know that were blocked by this.
Reviewed By: mshroyer
Differential Revision: D43966515
fbshipit-source-id: dfd988f81ec9b931f2d3bc6f34ad4ad82c2f2a61
Summary:
These tests seem fine? Maybe someone already fixed them?
But even with stress runs I can't get them to fail. I'll try on my intel too
to make sure it's not that.
Reviewed By: genevievehelsel
Differential Revision: D43986058
fbshipit-source-id: 7e3702a7efbd4637efbb235e626b421e9a3d28a1
Summary:
The takeover tests are failing in a couple ways.
First, there are failures:
multiprocessing seems to behave differently on mac than linux.
The process calls cause locking issues when "pickling". multiprocessing seems
kind un reliable, and we don't really need it in either of the used places.
Second. there are timeouts:
accessing an fd that was open before takeover seems to hang sometimes.
I can not manually repro on my M1, but don't have time to dig in right now, so
I will just leave a comment with some info on the issue and leave these disabled
for now.
Reviewed By: mshroyer
Differential Revision: D44000288
fbshipit-source-id: 76ef085967a495ffd3ab0a8aae337960368d75e0
Summary:
Building EdenFS with `-Werror` option reveals a number of errors:
* shadow
* unused parameters
* sign-compare
This diff fixes unused param warnings and sign comparison issues. For now, we'll ignore the shadowing error since it's in folly/portability and would be quite tough to fix.
Reviewed By: fanzeyi
Differential Revision: D44440807
fbshipit-source-id: 9d061e6cd7689cca65433ffa2f0fa738fa0aee00
Summary:
Add a way to debug copytrace binding, this can be used in .t test for
testing copytrace binding as well
Reviewed By: quark-zju
Differential Revision: D44449566
fbshipit-source-id: 1b5cc6bbf747edd449fb6e98c1bdba9527515353
Summary:
Allows nesting `#ifs` macros on debugruntest.
This can be useful if we have something like
```
#if no-windows
...
#if fsmonitor
...
#endif
...
#endif
```
Reviewed By: quark-zju, zzl0
Differential Revision: D44447876
fbshipit-source-id: 0f5a136cbed7f235a53b619a5bb3b593ffff7d8c
Summary: Recently we added the `experimental.windows-symlinks` config to make repos initialized with that config add `windowssymlinks` to the working copy requirements. That, however, didn't apply to the `eagerrepo`. This meant that tests initializing repos using eagerrepo would not have this behavior, which made writing `.t` tests expecting that in the requirements somewhat awkward.
Reviewed By: quark-zju
Differential Revision: D44444534
fbshipit-source-id: def5c30267aa91bf6d2a72514644538285f80b8b
Summary: It could be useful to make `eagerrepo` to take in a config option so that it could set in certain values on the working repo or in the backing store requirements. For instance, we might want to make `eagerrepo` able to set Windows symlinks.
Reviewed By: quark-zju
Differential Revision: D44465174
fbshipit-source-id: 122f2428dcaa2e866fa7d7774e12dee2b3fe5eee
Summary:
We've largely migrated everyone off of macFUSE. Now we can remove the
doctor check.
Reviewed By: fanzeyi
Differential Revision: D44479361
fbshipit-source-id: 6332c5d47a043797ce7e6b823f6c81e0f6ae33a7
Summary: These tests fail on my laptop because they search the actual filesystem for certs.
Reviewed By: genevievehelsel
Differential Revision: D44478611
fbshipit-source-id: 2d53fc642f7d5fbda2f82d493e20b67c1dd32513
Summary:
I accidentally referenced a non-existent "edenscm.httpclient.HTTPSConnection" when switching http libraries in the graphql client. Tweak code to work with the new library.
I also killed the ca_bundle arg since it was never actually passed from graphql.py into anything.
Reviewed By: sggutier
Differential Revision: D44478636
fbshipit-source-id: 50e78ab52f1db98fcb6b14f0fc1cac9ea6662abf
Summary:
🐛(chg): don't crash when user has more than 16 groups on macOS
D43676809 (OSS: 49ee17f2bf) introduced a check for whether the chg server and
sl process have different permissions. We run `getgroups` in two different
places:
1. using `os.getgroups` in sl client in Python
2. using libc's `unistd/getgroups` in chg server in C
Essentially the way this works is we call `os.getgroups` in the client (Python)
and `unistd/getgroups` on the chg server (C). Then, the server compares the
results. If they're the same, then both processes have the same permissions,
otherwise they might be different.
However, `getgroups` on macOS has a key limitation that it will only return a
max of `NGROUPS_MAX` groups (16 groups on my machine, defined in
`sys/param.h`). You can remove this limit by defining
`_DARWIN_UNLIMITED_GETGROUPS` or `_DARWIN_C_SOURCE`[^1].
`os.getgroups` in Python is compiled with `_DARWIN_C_SOURCE`[^2] but chg does
not, meaning that the result of `getgroups` is different in our Python and C
code.
This commit defines `_DARWIN_UNLIMITED_GETGROUPS` for chg so `getgroups` will
have consistent behavior. (the alternative is patching CPython to unset
`_DARWIN_C_SOURCE` which is worse)
Special thanks to strager for helping me debug this 👍🏻
Pull Request resolved: https://github.com/facebook/sapling/pull/573
Test Plan:
1. have user with <=16 groups
2. run `sl --version`
3. confirm it works
4. have user with >16 groups
5. run `sl --version`
6. notice sl failing with error "chg: abort: too many redirections."
7. apply this patch
8. repeat steps 4-5
9. confirm it works
[^1]: https://opensource.apple.com/source/xnu/xnu-3247.1.106/bsd/man/man2/getgroups.2.auto.html
[^2]: 28d369e070/configure.ac (L267)
Reviewed By: sggutier
Differential Revision: D44378466
Pulled By: quark-zju
fbshipit-source-id: 402aa56bcacd2d6d2af9bed66e2c3be13738c435
Co-authored-by: strager <Strager.nds@gmail.com>
Summary:
Our current practice is only using `block_on` in Python bindings, so this diff is to change the `read_rename_metadata` to an Async function and also remove the `block_on`s in its implementations, so it is consistent with other APIs.
In the longterm, we'd like to remove those Async APIs, but we will to use a lib to help us migrate them in an automatic way, consistent APIs will make the migration process eaiser.
Reviewed By: quark-zju
Differential Revision: D44199939
fbshipit-source-id: a75d1356e373f3d883da4ab972520a665587eb23
Summary: Python binding for copytrace, so it can be used in Python world
Reviewed By: quark-zju
Differential Revision: D44152527
fbshipit-source-id: 233986efddb855188ce9ada6cace70a57f834b9c
Summary: make `cl.dag` able to be used as a `dagalgo` -> `DagAlgorithm` converter, this will be used by pycopytrace.
Reviewed By: quark-zju
Differential Revision: D44152513
fbshipit-source-id: 1e211063160509587fe807789716f23787e2705e
Summary: Fill the missing part of DagCopyTrace::new
Reviewed By: quark-zju
Differential Revision: D44110425
fbshipit-source-id: 32f9cfc949a56eb60d5589258ae6da87ea26cb5a
Summary: trace_rename API will do forward search, backward search, or both based on the relationship of src and dst commits
Reviewed By: quark-zju
Differential Revision: D44106975
fbshipit-source-id: 69dc8c16cea32f060b94dd6306a346a2a953e0a4
Summary:
This will be used for searching renames from `base` to `dst` commits
```
dst
|
| src
| /
base
```
Reviewed By: quark-zju
Differential Revision: D44094783
fbshipit-source-id: 5943b5aa6bc9b1a22b75738749d934887475dd5e
Summary:
refacotr the trace_rename_backward API, so that we can reuse the
common parts in trace_rename_forward API in the following diff
Reviewed By: quark-zju
Differential Revision: D44448494
fbshipit-source-id: 997a338c5f2e9e52d1d7bb3d54641980d9119797
Summary:
Whenever there's a path or the `-d` flag, `git clean` implies that cleaning should be done recursively, into directories, removing empty ones as it goes.
The equivalent in Sapling is `sl clean --dirs`.
Closes https://github.com/facebook/sapling/issues/564.
Pull Request resolved: https://github.com/facebook/sapling/pull/575
Test Plan: .t test
Reviewed By: quark-zju
Differential Revision: D44350217
Pulled By: zzl0
fbshipit-source-id: 72bef3d23c5d92ffbd275b4e57bf218f570ca71f
Summary:
`scmutil.cleanupnode` is the "common" code path for commit rewrites (except for
"split"). Update it to notify ISL about the commit rewrite source and
destination.
Reviewed By: muirdm
Differential Revision: D44439674
fbshipit-source-id: 9e8bb7b12aba31619d0a843fdf9939682afe3e53
Summary:
See the previous diff for context. This diff adds Python bindings for the node
IPC feature. Right now only the singleton's send and recv features are exposed
driven by use-case.
Reviewed By: muirdm
Differential Revision: D44435527
fbshipit-source-id: 6343a3ecf633f7f1a12aca85eaf973ae86a1be10
Summary:
We want a way to report information (like what commits are changed to what
other commits) to ISL. This diff adds a way to communicate JSON values with
nodejs parent process using the `'ipc'` feature defined in nodejs stdlib.
This way, we can communicate using structured data instead of plain text, and
preserve the existing stdout/stderr (comparing to other solutions like
`rebase|metaedit -q -T '{nodechanges|json}'`).
Reviewed By: muirdm
Differential Revision: D44435528
fbshipit-source-id: e804a25a25dfff31c25b13b16f9bc86ab4e8306f
Summary:
Implement prefetching of commit graph edges via recursive common table expressions.
Prefetching can be performed either by the first parent edge, or by the skip tree skew ancestor edge. We can add more edges later on, however the nature of the `mononoke_queries!` macro means each one needs its own, only slightly customized, version of the SQL query that performs the operation.
Reviewed By: liubov-dmitrieva
Differential Revision: D43455724
fbshipit-source-id: 75f19533f1f4e9d118a098469bbe3f8afb5f7f92
Summary:
The `native_push_only_deny_patterns` contains the list of paths that should be
changed only via pushrebase operations.
I think the `native_push_only_deny_patterns` should be moved into its own hook
but let's start from fixing current behaviour and refactor later.
Reviewed By: liubov-dmitrieva
Differential Revision: D44433734
fbshipit-source-id: 2da93e84763cdfb7ee997cb148d4eac288fba27f
Summary:
Use nameset instead of idset to underly the fullreposet. The idset does not prefetch nodes, so there was a serial location_to_hash call for each "repo[rev]" context lookup. The nameset prefetches nodes under the hood, so "repo[rev]" is fast and the next layer of commit text prefetching fills up its 10k batch quickly.
Note that anything hitting this code will still be extremely slow since it scans all commits in the repo.
Reviewed By: zzl0
Differential Revision: D44438396
fbshipit-source-id: e1275a7a45f54cae1978820eb36d99a212fc4236
Summary: There was the classic same second race condition that left files NEED_CHECK after commit, and the tracing message had changed.
Reviewed By: quark-zju
Differential Revision: D44442614
fbshipit-source-id: 97486163301045825922f9b68ec00ff12e5432a7
Summary:
Use the "exists" field from watchman to detect when a file has been moved behind a symlink. Watchman reports it properly as not-exists, but if we naively vfs.metadata(path), it may appear to be unchanged.
I added another layer of Option<> to the file metadata so watchman fs can explicitly say "this file doesn't exist", as opposed to only "I don't have metadata".
The alternative approach is to "audit" paths before we lstat them, but that is slow and we want to avoid if possible.
Reviewed By: quark-zju
Differential Revision: D44187187
fbshipit-source-id: e24a43cbdb36d9aabf2b0b7ae0c4d2eab66394c4
Summary: Now the Rust status requests file mode, size and mtime from watchman and uses them for file comparison in lieu of lstatting the files. This way, if nothing has changed (think fresh instance), or if file size has changed, we don't need to hit the disk at all!
Reviewed By: quark-zju
Differential Revision: D44033614
fbshipit-source-id: 54cd8bd6dddd7fb88e9538d72174ae88b2118550
Summary: Add a test showing we fetch file metadata after watchmant tells us a file changed. I'm going to optimize this away buy querying and using watchman's file metadata.
Reviewed By: quark-zju
Differential Revision: D44033616
fbshipit-source-id: 8b098a5868e20283d2fc98b198c70c4b086b9657
Summary: Add a Metadata type to abstract across the various places we get file metadata from. This doesn't buy us much right now, but will allow us to easily inject file metadata from Watchman into our file comparison logic.
Reviewed By: quark-zju
Differential Revision: D44033617
fbshipit-source-id: ed08e44df9ff44bb23e8fff031d431c4a519eaa4
Summary: Use some #pragmas to ignore unreachable code warnings because sigemptyset on Mac can't return -1.
Reviewed By: quark-zju
Differential Revision: D44344187
fbshipit-source-id: bf650d20c38c950899aec3d8e7f91d18db4c031d
Summary: Going to be adding more "metadata" related stuff that depends on HgModifiedTime, so move this type first.
Reviewed By: quark-zju
Differential Revision: D44033613
fbshipit-source-id: 30c6173fc069fbb7553d6d9fffaea0fc774cb72f
Summary: This can take a few seconds if there are a ton of files (e.g. watchman fresh instance).
Reviewed By: quark-zju
Differential Revision: D44033611
fbshipit-source-id: 8e033eef68e2f4560b859d35f78da139f86abc97
Summary:
When using watchman, we show a progress bar as we compare the contents of files. Due to how the file detector works, this progress bar was staying at zero and suddenly jumping to the end. Fix by moving the progress bar into the ParallelDetector. I had to add a stupid method to the detector trait to set the total number of files.
I don't think the FileDetectorTrait is quite right. Ideally the user of the detector would own the progress bar. It would set "total" based on the work it has, and increment "position" as it receives results. However, currently you can't receive results until you have finished publishing work. We should tweak things to make this possible.
Reviewed By: quark-zju
Differential Revision: D44033612
fbshipit-source-id: a22b237eaf98f3bd55f5f79eccc3fe42e733a297
Summary:
While we are waiting for watchman to crawl the repo, call "debug-root-status" and pipe the file count into a progress bar.
The similar Python logic renders the "crawling" progress bar until the crawl is done and then proceeds with the query. In Rust I made it render the progress bar concurrently with the query. This should show the progress bar even if the repo has never been watched before (i.e. when there is no crawl until the query happens).
Reviewed By: quark-zju
Differential Revision: D44033615
fbshipit-source-id: 3c97fa7a63cd090b5ace82d7385f887e564eb9cf
Summary: The parallelized MutliWalker was skipping nested repo dot dirs (e.g. ".sl"), but wansn't skipping sibling files to the dot dir.
Reviewed By: quark-zju
Differential Revision: D43970586
fbshipit-source-id: 0cfd703a067f7db2087a6e692316eab192330b54
Summary:
Fix detection of a directory being replaced by a symlink. When the walker iterates tracked files not seen during the walk it now directly emits "deleted" events instead of going through the file detector. If tracked files had the same size and mtime "through" the symlink, the file detector's lstat() call would make it seem like there were no changes.
The alternative approach is to "audit" paths before we lstat them, but that is slow and we want to avoid if possible.
Reviewed By: quark-zju
Differential Revision: D44187186
fbshipit-source-id: 5678d6027b74283608bd0430e7acd3ae16cfb046
Summary: Show a warning about fifo etc. files instead of aborting the command.
Reviewed By: quark-zju
Differential Revision: D44187185
fbshipit-source-id: c4c01436ad21e4e15fcae9c5d04f1aaacc7e5c80
Summary:
The manifest stores a file as executable or symlink (mutually exclusive). A symlink on disk also has the executable bit(s) set, so the fs metadata (and by extension the mode in the dirstate) appears to be both symlink and executable. This is a recipe for confusion.
In Python it seems to work because the dirstate mode and fs mode both store the execute bit for symlinks, so they match when compared for executableness. (Although I wouldn't be surprised if they could get out of sync somehow.) The symlinkness check is achieved through the "flags" which does not include "x" if the file is "l".
Fix in Rust be applying the flags logic to the dirstate and fs metadata as well. In other words, if a file is a symlink, don't report it as executable. This makes the manifest file mode compare naturally with the dirstate and filesystem.
Reviewed By: quark-zju
Differential Revision: D43970590
fbshipit-source-id: b459e466dace5ee1c9cc7505d3c780be6ecd4685
Summary:
Supporting exporting the wdir() virtual commit could make it easier for ISL to
obtain file contents for interactive commit or amend.
Before this change it will error out in the Rust dag layer, since the Rust dag
layer does not know about the special wdir commit (yet). This diff works around
it by adding some special logic about it. Note in the long term we would like
to add the virtual commits directly to the Rust dag logic so they can be
calculated without special casing.
Reviewed By: evangrayk
Differential Revision: D44231011
fbshipit-source-id: 27fce76afd2399502b30e15cca6c8f8986e1bc34
Summary:
This is the reversed direction of `debugexportstack`, to create commits from
automation. It also supports `goto` and `reset` to support partial commit
(commit, reset), absorb (commit*, reset), amend (commit*, goto) cases.
Reviewed By: muirdm
Differential Revision: D44231010
fbshipit-source-id: 03d2e5a7d577a2c6becb5e7d9f162fea191ad9af
Summary: Check that the test cases actually cover various logic in debugexportstack.
Reviewed By: zzl0
Differential Revision: D44231012
fbshipit-source-id: dbb24c7f87ecd11d23aae1707b68f77486177ef7
Summary:
This makes it easier to verify that a complex Python function is fully
exercised.
Reviewed By: zzl0
Differential Revision: D44231013
fbshipit-source-id: 6269dae56acff4f29a6aa5b7ad0ab89c33601ee1
Summary: Add debug info for edenapi configs to verify if it load the static_system configs correctly
Reviewed By: quark-zju
Differential Revision: D44298145
fbshipit-source-id: 96efb86c4e96a945b7329ae1e9cba3fdf57a5128
Summary:
During cross-repo pushrebases, we need to log new commits both in large and small repos. But currently, due to the code organization, it is not too hard to forget logging in the small repo. Which is what happened in case of SCS land_stack method.
This land path was not covered, and only large repo commits were logged. For example, this led to most public opsfiles commits being missing from the category (sync to opsfiles_backup uses different code path and exposes the discrepancy):
{F912075962}
(https://fburl.com/scuba/mononoke_new_commit/8q0nbq8i)
Reviewed By: markbt
Differential Revision: D44187074
fbshipit-source-id: bba2bf112fc9326ae5fc177d421ec936a2bb5322
Summary:
The environment is passed into the async runtime via an `Arc`. If the last reference to the environment is dropped from within the runtime, then we end up destroying the runtime from within the runtime, which is an error.
Move the runtime out of the environment, and store it in the `MononokeApp` (new) or on the stack in `main` (old). This means the runtime will live on the stack outside of the async runtime, and so execution of all async code should be complete when they are dropped.
Reviewed By: YousefSalama
Differential Revision: D44299775
fbshipit-source-id: 9a93ba49a8bbf376defdd0b5ddb1d6546b5ff12b
Summary:
Curently, `debugcrdump` returns an empty string for `branch` if there are multiple downstream remote bookmarks. This diff returns the first bookmark as the best guess, which should be the bookmark (commit) that the current local commit is based on.
Caveat: for multiple downstream bookmarks, it's a best effort guess, the algorithm is like:
1. For a given commit, find the first public ancestor.
2. For that ancestor commit, find all descendant commits that have remote
bookmarks sorted in ascending order.
3. If the descendant remote bookmarks contain "master", return that.
4. Otherwise report the first remote bookmark as the current branch. For draft commit,
this should be (best guess) the remote bookmark on which the draft commit was based
if user didn't run pull from remote server.
For example, if your commit graph looks like this, it will return `bookmark1` (first one) as the branch name, this is probably what you are expecting.
```
Y: public bookmark2
|
| a: draft (your local commit)
|/
X: public bookmark1
```
But if you run a `hg pull`, then the graph might look like below, it's impossible to know for certain which remote bookmark the local commit was made against. For now we also pick `bookmark1` as a guess.
```
Z: public bookmark2
|
|
Y: public bookmark1
|
| a: draft (your local commit)
|/
X: public
```
Related diff: D17814487, this was the diff that first added `branch` field.
Reviewed By: sggutier
Differential Revision: D44355664
fbshipit-source-id: b5287578d82c756f2d7035ae5ffa9a542ab957a0
Summary: this is for reproducing the issue of T148764976
Reviewed By: sggutier
Differential Revision: D44360807
fbshipit-source-id: 14400f9abc3ef6ca8e72d1c4a1f20b6ada6361fa
Summary:
This test has been flaky practically since its creation, although below the threshold to be detected as such some of the times.
Looking at its journal, it failed its first stress run and more than half
of the stress runs thereafter (failed 1920 of them, passed 1519 of them).
The assumption that the test makes that we will always take the local path first
is not stable and has never been.
Match the test's expectations to the actual behaviour of mononoke for now.
If the behaviour is problematic, let's tackle it in its own time.
Note that this test was written here D34276252 in reaction to a change to
segmented changelog here D34041066 and the discussion on that diff is relevant
to establish the correctness of the code.
Reviewed By: RajivTS
Differential Revision: D44297278
fbshipit-source-id: 07f42fc11ca0aa637b59b5fc46263b65d624e0a4
Summary: After enabling remote derivation we've lost related to derivation logging in the parent scuba table (like SCS server). This diffs add log tags, which indicates when we requested remoted derivation and when it was finished. In case of the error we already had the logging in place.
Reviewed By: Croohand
Differential Revision: D44297304
fbshipit-source-id: 1e2f6e279e111ce01b2feaf950a11c7b69a314cf
Summary: useful for checking if an object is in the hgcache or not
Reviewed By: quark-zju
Differential Revision: D44316641
fbshipit-source-id: f16093ec1659a8f139b88e702dbd1d8646319c8a
Summary: In the next diff, we'll use this util function to acquire an exclusive handle to a file for a prjfs integration test.
Reviewed By: xavierd
Differential Revision: D44349708
fbshipit-source-id: ded7037266f539c681569f0c90f875717760fed5
Summary: We are running into fileNotification() failures way more often than we expect on Sandcastle (5% of certain jobs are failing). Let's add some logging here to see if the failures are also happening to users.
Reviewed By: xavierd
Differential Revision: D44350819
fbshipit-source-id: 3be7578dccc297071280f528b53270a63994a99d
Summary:
DurationScope has a pair of atomic reference count operations at the
beginning and end of every recorded duration. To avoid that overhead,
reference EdenStats with RefPtr and, in production EdenFS, store a
global, unowned EdenStats object.
This gives us the benefits of explicit dependencies and no global
state in tests, while avoiding atomic RMWs in the release build.
Reviewed By: xavierd
Differential Revision: D44323723
fbshipit-source-id: 1b3384d2e6a0a2959fd79774a8ba46afc4c176ca
Summary:
In Rust 1.68.0, the `derivable_impls` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10161).
Fix all instances of that lint in Mononoke, mute one that's impractical to address.
Reviewed By: markbt
Differential Revision: D44378068
fbshipit-source-id: 473a051a7001d9596db43f47b56cbad5f5db7efe
Summary:
In Rust 1.68.0, the `useless_conversion` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10020).
Fix all instances of that lint in Mononoke.
Reviewed By: markbt
Differential Revision: D44378069
fbshipit-source-id: acfd6c77c6a400830c378b4040661323e7232441
Summary:
Original commit changeset: 8c2f66457761
Original Phabricator Diff: D44111246
This seems to be causing stdout ordering issues too, possibly because of PYTHONUNBUFFERED environment variable
Reviewed By: fanzeyi
Differential Revision: D44356269
fbshipit-source-id: 707495d785987308d9c4617a582ccb8613ed5016
Summary: If an EdenAPI request is taking too long, we currently have no signal into this until it either completes or is cancelled. As for wireproto, add logs for long-running requests.
Reviewed By: liubov-dmitrieva
Differential Revision: D44340299
fbshipit-source-id: e24bed96505964c519ea85748207670ded3baacb
Summary:
When streams are cancelled we lose all of their stats, as they only get logged after the stream is polled to completion.
Make the stream stats callback synchronous (we don't actually use its async nature), which means we can call it from a `Drop` implementation, allowing us to log the stats even when the stream is not completed.
To distinguish the two cases, we add a "completed" flag, which is true if the stream was polled to completion.
This also makes `completion_time` optional, as we don't have a value for this if the stream was never polled at all when it got dropped.
`TimedStream` had an implicit fuse-like nature (once completed the stream would never yield anything again), which we rely upon, so make sure this property is maintained.
Reviewed By: liubov-dmitrieva
Differential Revision: D44331959
fbshipit-source-id: 5a7d9875a4d7fd4276c304b6c30ff45c6b990b38
Summary:
As Croohand pointed out on D44183928, if the shared future is cancelled, the `WeakShared` will linger in the collection until the next read of that key. If the read of that key never comes, it will never be cleaned up.
Switch to a scopeguard, that we move into the future. This will execute the code to remove the shared future whether the future succeeds, fails, or is dropped.
Reviewed By: Croohand
Differential Revision: D44342772
fbshipit-source-id: 37fb15d126d8289d61ab33f64f6e1cf9f7a8550d
Summary:
In production we use Logger to log to Scribe (e.g. to "mononoke_new_commit" category). It has WhenceScribeLogged enum field which can be "default", "prod", "sandbox", etc.
Without override, Logger sets "default" value that usually resolves to "prod", **but** it resolves to "sandbox" if code is run from devserver. "sandbox" means we log data to "/sandbox" category instead of usual place.
This is confusing/harmful for us since we only configure to use Logger while working with prod data. For example, if we run SCS locally and land to prod using "land_stack", the commit will go to prod while logging will go to sandbox category. On the other hand, if we run Mononoke Bootstrap to work with a test repo, we don't use Logger at all so the "sandbox" category has no benefit.
We can add more complexity to the Mononoke logging configuration, so we are able to override WhenceScribeLogged in any way we want. But for now, I suggest overriding it in place with "prod" value since we only use Logger in prod. Later we can even simplify the config by removing the old way of logging directly to Scribe and leave just two options: test (log to file) and prod (log via Logger using the "prod" override).
Reviewed By: markbt
Differential Revision: D44118890
fbshipit-source-id: 8d4b9faa6f1f2da1faa96db35dd567102ccde34d
Summary:
We use interngraph GraphQL queries for a few things (phabricator diff info, scmquery log integration).
First, move the graphql endpoint config from static to dynamic config and vary it based on domain.
Finally, set x-x2pagentd-inject-cat in GraphQL request header when using x2pagentd. This replaces the oauth token mechanism, even for existing VPN/corp use. Note that the off-vpn endpoint rejects requests with both x-x2pagentd-inject-cat header and access_token in the body. We can delete the oauth stuff once we are confident in the x2pagent usage here.
Reviewed By: quark-zju
Differential Revision: D44317674
fbshipit-source-id: 329d63207fd9a5b19a145e12b491e270b5924571
Summary:
Send graphql queries over the local proxy socket, if configured. This is a prerequisite for off-VPN compatible access.
I added an escape hatch config phabricator.use-unix-socket we can use to disable this if someone breaks.
Also, kill apparently dead phabricator_graphql_client_requests.py.
Reviewed By: quark-zju
Differential Revision: D44317673
fbshipit-source-id: 52b58ce7899ce7291b700a9fe206da6011f740e5
Summary: Compare file mtimes so we synchronously regenerate dynamic config if the user has modified /etc/mercurial/vpnless. This way things will work off-VPN immediately after writing "1" to /etc/mercurial/vpnless.
Reviewed By: evangrayk
Differential Revision: D44263217
fbshipit-source-id: 1f09591336b7dc1294e1f21fafebce7962859d70
Summary:
To support VPNless access, we need to fetch the remote config through x2pagentd, setting a CAT related header.
I tweaked the order of config loading so the remote config fetch has the system "auth_proxy.unix_socket_path" config setting available.
Reviewed By: quark-zju
Differential Revision: D44263216
fbshipit-source-id: 5b8e6c4c9c7a5ab22ecfcfa62a537623931a63bf
Summary:
ObjectStore.h includes too many headers. Replace several with forward
declarations.
Reviewed By: xavierd
Differential Revision: D44323736
fbshipit-source-id: 944cd72fd758df4363493f87a573d91245a45b6a
Summary:
All of these configs have been enabled by default for a bit, let's change their
default before we can fully remove them from the code.
Reviewed By: genevievehelsel
Differential Revision: D44234768
fbshipit-source-id: 922093a64b91c9a45f84349ea50afc1c08fd6a39
Summary:
XLOG(DFATAL) is non-fatal in production builds, so it's possible we'll fall through here. It's unclear what the compiler will do in this scenario, so let's make sure we return folly::unit to avoid any compiler weirdness.
This may be causing a frequent crash on Sandcastle hosts where we're panicking after logging this DFATAL error. The crash looks like it's coming from an invalid instruction, so falling through here could be the root cause.
Reviewed By: xavierd
Differential Revision: D44314543
fbshipit-source-id: 6d23ffa221d7af4e7be641349eab1eb9027443a1
Summary: Basename suffix skeleton manifests are not been warmed by the warm bookmark cache. This means requests that need these are slowed down while they are derived. Add a warmer for them.
Reviewed By: yancouto
Differential Revision: D43906545
fbshipit-source-id: e1f77b5fe2a7a4512bfc49bf8047afef9b441fa9
Summary: This doesn't appear used anymore (superseded by phabdiff and phabstatus extensions).
Reviewed By: evangrayk
Differential Revision: D44263215
fbshipit-source-id: 323da45474a6cd1f9df730fdce23f4f42e836fb8
Summary:
do not sample out slow requests in edenapi replay
we already don't sample them in mononoke test perf
if we also don't sample here, we will have more insight in those and also we will be able to replay
Reviewed By: clara-9
Differential Revision: D44258777
fbshipit-source-id: 597fc475062b23eb24b0c61024eea9046b808420
Summary: Deduplicate reads of large blobs by using a shared future for any in-flight reads. If another read for the same key comes in after then we will read it again, but there should be at most one read in flight for a particular key at a time.
Reviewed By: Croohand
Differential Revision: D44183928
fbshipit-source-id: d6911422a6e16b3dd246dd48cbce4390e6856940
Summary: Make it possible for tickets to become `'static` by using `std::borrow::Cow` for the borrowed values, and converting to the owned variant if needed. We will use this to shared futures that hold a ticket.
Reviewed By: Croohand
Differential Revision: D44262842
fbshipit-source-id: 2c549b2f2f3724c6485b41463b973273211dfe77
Summary:
The `awaited` boolean on the `Ticket` structure and corresponding `Drop` makes it impossible to move out of the fields of `Ticket`. This is inconvenient, especially since it only enables a unit test check.
Move the field to a structure that's internal to the `Ticket`. We still check for failure to await, as dropping the `Ticket` with a `CheckAwait` that hasn't been marked as awaited will still panic during tests when the inner `CheckAwait` gets dropped.
Reviewed By: Croohand
Differential Revision: D44270321
fbshipit-source-id: 29429c40243b31dcc1a5046969b85fb7a15fd0aa
Summary: This log was useful while we were testing the new commit graph, but now it is too noisy and should be removed.
Reviewed By: liubov-dmitrieva
Differential Revision: D44297423
fbshipit-source-id: c9969d4a0b4709ea2065f19508ae6b51a7ee6baf
Summary: We previously only logged the error code of any failed Python commands. This means we have no insight into why Python commands were failing.
Reviewed By: chadaustin
Differential Revision: D44111246
fbshipit-source-id: 8c2f66457761ddd8b90ff4fb307152cd27ffa00a
Summary:
During checkout, these 2 are called a large amount of time, let's make sure we
have some telemetry so we can understand how much time is spent doing
invalidation during checkout.
Reviewed By: chadaustin
Differential Revision: D44311711
fbshipit-source-id: 5af62fe7fd4b37972458bc545bfaa2f4b4d2ca53
Summary:
This makes lifetime easier to reason about, and allows easier use of
DurationScope (see next diff).
Reviewed By: chadaustin
Differential Revision: D44311712
fbshipit-source-id: 1f9f4cbcc59bafeb464e2b39a248958cedaf9ca6
Summary:
Running the benchmarks/random_writes.cpp benchmark on all power of 2 from 4KiB
to 1MiB shows that 16KiB has the best tradeoff between fast random writes and
fast streaming writes.
Reviewed By: chadaustin
Differential Revision: D44320693
fbshipit-source-id: 2d1839caca3c30acdb6ffc675710a68359d03a76
Summary:
This one slipped through D44263797 and allows for integration tests to be run
with Buck2 on macOS.
Reviewed By: fanzeyi
Differential Revision: D44315942
fbshipit-source-id: d2de0773ba68f13fca9e8d5c067b82653646c757
Summary:
I would like to refer to some performance data of `gca` in the Segmented
Changelog doc.
Initially I was trying to make `drawdag` fast since it can use an elegant
syntax `A001..A999` and can work for both git and hg formats
(`debugexportrevlog` can be used to create repos that the upstream Mercurial
can understand). However it turns out to be a lot of work to make `drawdag`
"just" fast so I ended up writing dedicated scripts to create repos with
millions of commits.
Reviewed By: muirdm
Differential Revision: D43373717
fbshipit-source-id: 5ea1753027cb7487099ed00007e4cc0beae8a09a
Summary:
Parents outside the requested set mislead PathHistory to skip a segment
incorrectly. Detect that and do not skip segment in this case.
Reviewed By: zzl0
Differential Revision: D44305349
fbshipit-source-id: 9c75c4f830e97ff0f54c992bef3ac3aa331edbe1
Summary:
Previously the pathhistory tests always use the entire commit graph to
initialize PathHistory. This diff adds a way to test it on a subset of
the commit graph.
Reviewed By: zzl0
Differential Revision: D44305350
fbshipit-source-id: 955af0243a89f4f6e8f13da970170021c51dafbf
Summary:
Sometimes the tests run so fast that we're being served stale version of the
bookmarks.
This may cause the test to occasionally take the fast path where it expects to take the slow path, and has caused this test to be flaky since its creation (although not always flaky enough to be marked as such).
Differential Revision: D44298928
fbshipit-source-id: 046ce5bc4e005f3d443794202352bfdc5c80895a
Summary: Youssef accidentally discovered that config hot reloading was not working for SCS server when working on an oncall task. He then went ahead and fixed that issue in D44206206. However, we did not get any notification or alert about this problem and it would have stayed hidden much longer had we not accidentally detected it. This diff adds a liveness metric for the `ConfigUpdater` which will keep publishing every 5 mins as long as the `MononokeConfigs` object is live.
Reviewed By: YousefSalama
Differential Revision: D44293235
fbshipit-source-id: 6f54831c8b1f2154a23c021a7d2d8f01615a7d70
Summary:
A gotham request consists of two parts. The first part is a future, in which the handler resolves the request and produces the response headers. The second part is the body which may optionally be a stream. We added collection of future stats for the first part, but the majority of work may actually be done when computing the stream for the body.
In this diff we add stats collection for that body stream, if it is used. Since this is separate from the future stats we already collected, we prefix the scuba columns with "stream_" to make it clear this is the stream.
Reviewed By: liubov-dmitrieva
Differential Revision: D44260233
fbshipit-source-id: 9820d3f30fb147740a95d8afc2fc1bf1fcc16aab
Summary: The counter should be has_overlay_file, not has_overlay_dir.
Reviewed By: fanzeyi
Differential Revision: D44274455
fbshipit-source-id: 55563a0da14ec0ef1a089e3073cc8fd290d2f35d
Summary: Add this so we know where events in edenfs_events came from (Python CLI, Rust CLI, or the daemon).
Reviewed By: mshroyer
Differential Revision: D44044474
fbshipit-source-id: e23af5e186121657dfabf25c5c50882cc9aec923
Summary:
Removes need for patch.
It had already been updated to 3.4 but this just formalizes it.
Reviewed By: jasonwhite
Differential Revision: D44274378
fbshipit-source-id: d17ecbbee06c7df30f689eb859fb31fdf07d8d44
Summary:
Generelizing the Hash structure and adding Hash32 support.
Also, added a few basic keyed blake3 methods for that.
This is mainly a preparation to start supporting blake3.
bypass-github-export-checks
Reviewed By: chadaustin
Differential Revision: D44173515
fbshipit-source-id: 87c55d47dabe50c7104f09ee0078f29513068862
Summary:
Adding blake3 for the upcoming migration to it.
bypass-github-export-checks
Reviewed By: chadaustin
Differential Revision: D43567300
fbshipit-source-id: a0c81b42309cab65cf68be9be0530c86e0e91e95
Summary:
X-link: https://github.com/facebookincubator/velox/pull/4373
We are now planning to remove the Sapling getdeps build since Sapling
doesn't use it for open source anyway. The EdenFS getdeps build will adapt one
way or another.
This diff only remove the test step of the Sapling getdeps, which is failing
now. Will remove the entire Sapling manifest the following diff.
Reviewed By: quark-zju
Differential Revision: D44233463
fbshipit-source-id: c29419ab316fb1d0c3131d00b2fe3b4e3bb879e6
Summary: This needs to be generalized for using multiple different `InodeCatalog` types besides `FSInodeCatalog`
Reviewed By: xavierd
Differential Revision: D44236703
fbshipit-source-id: c38b017c0b7d1587fba90d40b527a016f1984fcf
Summary:
bump request_id len
Currently there are conflicts even within the same date https://fburl.com/scuba/mononoke_test_perf/ti2v0ka8
This is confusing. Let's bump.
Differential Revision: D44255577
fbshipit-source-id: 12d4f0ff8a7589da2057d1557fb651583551e329
Summary: This is just a tiny typo fix, I saw it today when I ran `$ eden status` after restarting my Mac.
Differential Revision: D44144242
fbshipit-source-id: fd721e41d9e3a3dcd73870a0819618e43e4d089c
Summary:
As of the libgit2 upgrade from 0.27.2 to 0.28.5 in D43195307, asan has been complaining about memory leaks coming from libgit2.
It initially caused 468 test failures.
D44096350 fixed 403 of these tests, but 65 are still failing.
Running one of these, we can see two other memory leaks (see test plan) being higlighted.
Mute them.
Differential Revision: D44254572
fbshipit-source-id: 8ae49c49542dda954f66de18cb756709d0c624d3
Summary: Somehow I missed this one in D44118779.
Reviewed By: genevievehelsel
Differential Revision: D44236425
fbshipit-source-id: 241f574d61572faa75c864f81279305f50b45abe
Summary: In preparation for making `OverlayChecker` work for both `FsInodeCatalog` and `SqliteInodeCatalog`, this migrates `OverlayChecker` to use `InodeCatalog`. Currently this code path can only ever be hit if we're using a `FsInodeCatalog`, so this should not be a behavior change.
Reviewed By: chadaustin
Differential Revision: D44105571
fbshipit-source-id: 0000319a5e78a6adb32ed0f602af173562a1b35a
Summary: as far as I can tell, we no longer have any callsites using `--overlay-type`, so this diff changes "tree" -> "sqlite" since that more accurately describes the difference between the different overlay options.
Reviewed By: chadaustin
Differential Revision: D44105266
fbshipit-source-id: 40d036222ca3e07c14f3f78aec374d4b6d54eb8a
Summary: Previously we had some requirements validation logic, but that was only available for Python commands. This makes requirements validation for both Python and Rust commands work.
Reviewed By: zzl0
Differential Revision: D44106560
fbshipit-source-id: 9b435017d336a314f06dc8fff96ec50a6aef8fcf
Summary:
We need to copy redacted commits as well, to maintain consistency, otherwise we can't progress the sync.
This is stopping the www import to fbsource.
Reviewed By: RajivTS
Differential Revision: D44217272
fbshipit-source-id: c6c03df1ebc6310688a134b1bfdcb3e19fc3daa7
Summary:
As of the `libgit2` upgrade from `0.27.2` to `0.28.5` in D43195307, `asan` has been complaining about memory leaks coming from `libgit2`.
This caused 468 tests failures as reported in this task: T147770506.
An initial investigation hasn't quickly led to a resolution of the source of the memory leaks.
Get the tests back to green by suppressing the memory leaks detection from ASAN.
AddressSanitizer: 480 byte(s) leaked in 4 allocation(s).
/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py:92: === clean up done at 93.718s (+10.623s)
ERROR
======================================================================
ERROR: test_eden_list (eden.integration.rc_test.RCTestNFSGit)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py", line 158, in <lambda>
self.addCleanup(lambda: self.eden.cleanup())
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 134, in cleanup
self.kill()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 143, in kill
self.shutdown()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 434, in shutdown
raise Exception(
Exception: eden exited unsuccessfully with status 1
----------------------------------------------------------------------
Ran 1 test in 93.725s
FAILED (errors=1)
Exception: eden exited unsuccessfully with status 1
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/case.py", line 717, in doCleanups
self._callCleanup(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py", line 158, in <lambda>
self.addCleanup(lambda: self.eden.cleanup())
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 134, in cleanup
self.kill()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 143, in kill
self.shutdown()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 434, in shutdown
raise Exception(
Buck UI: https://www.internalfb.com/buck2/947a3357-9a80-4a15-b067-fbad6a615fd0
Test UI: https://www.internalfb.com/intern/testinfra/testrun/3096224848534397
RE: reSessionID-765a77c0-c485-4182-be99-f7431d285752 Up: 312 KiB Down: 498 KiB
Jobs completed: 31. Time elapsed: 109.0s. Cache hits: 100%. Commands: 2 (cached: 2, remote: 0, local: 0)
Tests finished: Pass 4. Fail 4. Fatal 0. Skip 0. 0 builds failed
4 TESTS FAILED
✗ fbcode//eden/integration:rc - test_unmount_rmdir (eden.integration.rc_test.RCTestNFSGit)
✗ fbcode//eden/integration:rc - test_eden_list (eden.integration.rc_test.RCTestGit)
✗ fbcode//eden/integration:rc - test_unmount_rmdir (eden.integration.rc_test.RCTestGit)
✗ fbcode//eden/integration:rc - test_eden_list (eden.integration.rc_test.RCTestNFSGit)B
```
After:
```
Buck UI: https://www.internalfb.com/buck2/7091ae16-eb5f-425d-8922-cf278987acb3
Test UI: https://www.internalfb.com/intern/testinfra/testrun/3940649832205800
RE: reSessionID-a06bbda7-b3c4-46c6-998f-7e2280bb7ad8 Up: 558 B Down: 0 B
Jobs completed: 11. Time elapsed: 113.5s.
Tests finished: Pass 8. Fail 0. Fatal 0. Skip 0. 0 builds failed
```
Reviewed By: kmancini
Differential Revision: D44096350
fbshipit-source-id: 1c8a5308759f2dc90e00b33b8f4e19f2ca53f54f
Summary:
Show topics up to 20 chars if terminal width is >= 100.
This is a stop gap until I land my new renderer that uses space more efficiently.
Reviewed By: quark-zju
Differential Revision: D44136359
fbshipit-source-id: 5d59c8b07d930d689b73c3e0fe1c3ee8ce020118
Summary:
When MononokeApp is dropped, the ConfigHandle inside of it gets dropped, and the config updater thread stops receiving updates causing config hot reloading to stop working. This was happening in scs, the land service, and the derived data service (not sure if this is also happening in Mononoke server).
This diff adds an explicit drop in the server closure in wait_until_terminated to make sure MononokeApp is not dropped before receiving a termination signal.
Reviewed By: Croohand
Differential Revision: D44206206
fbshipit-source-id: 7322f516c0dcbf4668cba811bdb582f0e5e2f11c
Summary:
remove dead code
left over from earlier clean up
Reviewed By: YousefSalama
Differential Revision: D44131387
fbshipit-source-id: 20601b5e32777d8b2952120e95a24930ef4d2159
Summary:
vmagro deleted his forked and now broke Reindeer importing. Please be
more careful.
https://github.com/graphql-rust/graphql-parser/pull/66 has been merged
but not released.
Reviewed By: dtolnay
Differential Revision: D44198593
fbshipit-source-id: 287f0f3f6860cdf16deb7be80448c2962a9f10a6
Summary:
We've seen several cases where the sqlite database is corrupted causing EdenFS
to fail to start and requiring manual remediation. On Windows, we can always
reconstruct the sqlite database from scratch due to FSCK being able to build it
from scratch. Thus, we can simply delete the database on disk and continue
starting up.
Reviewed By: chadaustin
Differential Revision: D44155034
fbshipit-source-id: de05c814796ab8f76fd3cd9a3e98df438431c657
Summary:
Nothing is being enqueued just yet, but this teaches EdenFS how to fetch
batches of aux data.
In the case where the aux data cannot be fetched, a nullptr auxdata is returned
to the higher level which will then trigger a blob fetch to compute aux data.
Reviewed By: chadaustin
Differential Revision: D44110104
fbshipit-source-id: 53df1496addf3a9dae521ffcdba5b060b8fce16a
Summary:
As a first step towards enabling aux data fetching, we need to allow aux data
to be batch fetched.
Reviewed By: chadaustin
Differential Revision: D44110105
fbshipit-source-id: f9c3f55ffdc5554784b5f38457ef675689664263
Summary:
Currently, EdenFS doesn't perform any aux data fetching, instead it fallback to
fetching blobs. In some cases, this can be extremely expensive when the aux
data store is emptied/flushed/rotated leading to non deterministic performance.
On Windows in particular, directory listing always need blob sizes, and thus
getting aux data reliably is critical.
As a first step, let's add some watches through the stack and expose these to
Thrift. This will allow `eden top` and `eden trace hg` to display these
fetches correctly.
Reviewed By: chadaustin
Differential Revision: D44105497
fbshipit-source-id: a3dc5cce1bc3115a2a4effcece6fa0cf0b16f6c8
Summary: By using reference everywhere, we can save on several moves.
Reviewed By: chadaustin
Differential Revision: D44101377
fbshipit-source-id: 9aa9d89570fb0432477e632d108109d292fa63bb
Summary: I don't think we need file history when diffing commits, so skip it. This can be really slow for files with a lot of history.
Reviewed By: quark-zju
Differential Revision: D44175699
fbshipit-source-id: 945517f58ea9baf9d5e22ca04699d72e8dad72f7
Summary:
Our telemetry doesn't capture any data regarding how long Overlay operations
are taking. Since there is a suspicion that checkout is bounded on the Overlay,
let's add these counters to better understand the cost of the Overlay.
Reviewed By: genevievehelsel
Differential Revision: D44118779
fbshipit-source-id: 2e8658c9b0629a0a4f9c6f4535ec876527386b5f
Summary: Backing out ContentMetadataV2 stuff since the backfilling for that data has not yet been done and live traffic would see quite laggy performance if we did it directly in Prod Mononoke. So backing out for now until backfilling is fully complete.
Reviewed By: YousefSalama
Differential Revision: D44136799
fbshipit-source-id: ca403f89b8215f2dfd5582dda7499f1618944f2a
Summary: Due to backfilling `ContentMetadataV2` in phases, there are some cases where the metadata exists but the `Blake3` alias doesn't. When copying the alias, if the alias is not present, let's regenerate it.
Reviewed By: markbt
Differential Revision: D44132513
fbshipit-source-id: 4185ece38db350b0069dff485a40166bb0cb1c4d
Summary:
It turned out `split_test` is failing for the same reason as other hg tests.
The assert raise hid the actual error so it wasn't very clear. So this diff adds some better reporting for that as well.
Reviewed By: kmancini
Differential Revision: D43962607
fbshipit-source-id: 137016260f286ee9577576d90e9e4372d4db960e
Summary: Most of these tests are already passing, except few of them are having issues with `_asciitransform`.
Reviewed By: mshroyer
Differential Revision: D43960522
fbshipit-source-id: b3bae8d3df1acc6f9b32057367309aff44c93de3
Summary: After the `._` fix, these tests are passing and we can enable them.
Reviewed By: chadaustin
Differential Revision: D43931347
fbshipit-source-id: c568eeaff5802901a20036c81faac58e44b74820
Summary:
This makes `test-invalid-windows-filepath.t` pass with recent pathauditor error
type changes.
Reviewed By: sggutier
Differential Revision: D44153212
fbshipit-source-id: 81e71c48a42ef29eecf3a3a3e117141f43232efb
Summary:
Not sure what changed, but we got:
--- test-blackbox.t
+++ test-blackbox.t.err
@@ -31,6 +31,7 @@
[process_tree] (this process)
[command] [*, "confuse"] started by uid 0 as pid 0 with nice 0 (glob) (?)
[process_tree] (this process) (?)
+ [legacy][command_info]
[legacy][command_info]
[legacy][env_vars]
[legacy][command_info] (?)
The alias expansion seems broken. Update the test text to mention it.
Reviewed By: sggutier
Differential Revision: D44139337
fbshipit-source-id: b4012eb3ee3340c52e3514dd49c617bdf851a572
Summary:
Move macOS OPENSSL_DIR logic from onetime-setup.py to distutils_rust.
Hopefully this makes it work for hgbuild which does not run onetime-setup.
`HG_LINKER_FRAMEWORK` is removed since code search shows it is not used.
Reviewed By: sggutier
Differential Revision: D44142250
fbshipit-source-id: b7af87bbdd9eac48b7b00fbd14ca062bc953c727
Summary:
Understanding how many trees and blobs were accessed but not fetched during
checkout is a useful metric that informs us as to how much was served from
caches. This is useful as checkout is more or less proportional in time to the
number of accessed inodes and the number of accessed trees+blobs will thus tell
us how many TreeInode and FileInode were loaded and traversed.
Reviewed By: chadaustin
Differential Revision: D44117153
fbshipit-source-id: 56991c51f2e4f501486d64ab5598f149f2708b77
Summary: This was never utilized beyond the "warn" stage, and I don't want to implement in Rust, so let's kill it.
Reviewed By: quark-zju
Differential Revision: D43970585
fbshipit-source-id: cea765197a5f547441865a055deb1f13ad9ea28a
Summary: Sparse profiles supported configuring a list of "unsafe" marker files that would cause an error if they accidentally were included in a sparse profile (presumably to avoid unneeded sparse profile growth). However, this feature is not in use anymore, and I don't want to implement it in Rust, so kill it.
Reviewed By: quark-zju
Differential Revision: D43970582
fbshipit-source-id: a6576b8004cc41b3e6f2f0e0040b54e54af2bdc4
Summary:
Fix rust status when dirstate does not contain executable/symlink flags for a tracked file, and the file's flags have diverged from the manifest.
To achieve this we pipe the filesystem metadata long to the content check since the content check already looks up all the files in the manifest. Before actually doing the content check, it compares the on-disk flags to the manifest flags.
Reviewed By: quark-zju
Differential Revision: D43970583
fbshipit-source-id: 544acfa355cf9d0d2acf59ad9143de6dac349235
Summary:
Handle case where a directory with tracked files is replaced with a symlink.
In paticular, during status if we get a access-file-through-symlink audit error trying to read the contents of a file for comparison, we now treat the file as deleted.
Reviewed By: quark-zju
Differential Revision: D43970589
fbshipit-source-id: e145d36275e4af18ec77f4fb3b018f85df8f39f6
Summary: I need to differentiate the symlink error in the status code path.
Reviewed By: quark-zju
Differential Revision: D43970579
fbshipit-source-id: 7931f83f828352a8f5ca8f90ff037702b0c64369
Summary:
Don't error out for files with negative or huge mtimes. We use the same masking approach as Python for now.
Note that Rust status does not write the mtime back to dirstate in general right now, so success is merely that "hg status" doesn't crash in this case.
Reviewed By: zzl0
Differential Revision: D43970588
fbshipit-source-id: f1ea50eaeeee2fc9b564934ec587c9f02abaf651
Summary: Previously this diff fixed eagerepo file store to strip copy metadata, but that was handled better by D43972516. Now this diff just enables rust status in affected tests.
Reviewed By: zzl0
Differential Revision: D43970584
fbshipit-source-id: 8d2f87128a06f33524078d476bd9fb68cf6b642e
Summary: These will help me figure out "status" discrepancies.
Reviewed By: zzl0
Differential Revision: D43970581
fbshipit-source-id: 63ef7578b6aec100b172da9516292aa549b8d318
Summary:
The new status doesn't work w/ legacy repos, so the tests need to be migrated. For each test, first I tried migrating to eagerepo, but in many cases that didn't work. Next I tried a remotefilelog repo (AKA "newclientrepo"). If I still couldn't get the test to work I just disabled rust status. This was mostly in tests that disabled treemanifest or were explicitly coupled with revlogs somehow.
Some tests had acute "hg status" discrepancies, so I left a FIXME(status) and disabled rust status.
I added a new "eagerepo" tinit.sh function that turns on eager format (for client repos). With this, tests doing "hg init" might just work with features that expect a modern repo.
Reviewed By: quark-zju
Differential Revision: D43839499
fbshipit-source-id: ce6d3b2747b90d31bd246c30551bdb55c446112f
Summary: Recycles our POSIX `checklink` implementation so that it can be used with Windows. This should be safe given that D43456961 gates the Windows symlink functionality.
Reviewed By: muirdm
Differential Revision: D43456959
fbshipit-source-id: eaf20b692b118642606cb59ffbc01c3052f05522
Summary: This should allow in later diffs to enable symlinks on Windows if users have `windowssymlinks` on their repo store requirements. Additionally, it adds the aforementioned requirement to new repos cloned with the `symlinks.force-enable` config.
Reviewed By: muirdm
Differential Revision: D43456961
fbshipit-source-id: bd2f5a1cb6c3b647ebccb0de556959758a4d7fe2
Summary: Some of these stats counters don't exist for NFS/PrjFS mounts. I filed tasks (T147665665, T147669123) to add these counters in the future, but until then we will disable some tests.
Reviewed By: kmancini
Differential Revision: D43963755
fbshipit-source-id: 4fd9ab9fa5bc123a6d24c669b655bfb18cf3a0a5
Summary: The check appears to be flaky for an unknown reason.
Reviewed By: chadaustin
Differential Revision: D44104911
fbshipit-source-id: 390a0d11e16d09c5204d829727ee20db37665815
Summary:
Create changeset uses `FuturesUnordered` and `FuturesOrdered`. These are unbounded, so with very large changesets, we may try to do too much work at once. Switch to `buffered` and `try_for_each_concurrent` so that we can add bounds.
As always, the numbers are somewhat arbitrary: 10 commits at a time, and up to 1000 files at a time.
Reviewed By: mitrandir77
Differential Revision: D44056862
fbshipit-source-id: 6284c207e3768ec0b16371dcd24db53afc0d98d6
Summary: Remove another dependency on the blobrepo crate by migrating to the underlying feature code.
Reviewed By: yancouto
Differential Revision: D44054973
fbshipit-source-id: 612de351beb2d7bffcd98cff629eb787c0184890
Summary: Extract the code that converts create commit parameters to the internal types. These methods will be used for creating stacks as well as for creating single commits.
Reviewed By: yancouto
Differential Revision: D44032677
fbshipit-source-id: 25aaeeb129d4523d51bfc27d375180f09c282555
Summary:
Add `create_changeset_stack` to `RepoContext` to allow creation of a stack of commits in one operation.
In addition to previous checks, we must build a stack of the accumulated changes for each of the earlier commits in the stack so that we can check that the stack is internally consistent. For example, a file can only be deleted once in the stack, or it can be created and then deleted in the same stack.
Reviewed By: mitrandir77
Differential Revision: D44032676
fbshipit-source-id: f08a7df39c52c9de9d6efc728321112ac5ee60f8
Summary:
Let's see if we have long poll problem anywhere which could *greatly*
contribute to any latency spikes.
Reviewed By: markbt
Differential Revision: D44059679
fbshipit-source-id: c8cb37d65e084de1500bee14773ca8229fb07a0c
