Summary: Tweak the manifest BFS iter to take a list of initial nodes. I'm going to use this to allow prefetching trees for multiple commits at once.
Reviewed By: quark-zju
Differential Revision: D45273890
fbshipit-source-id: d3487a1a2ee71f602e535b8b7f673aa5d6a91aea
Summary:
First, kill the "depth" and "path" args since neither were set by any callers.
Then, use the bfs_iter to walk the tree instead of the naive level-by-level walk. This should be much faster, but more importantly is a step towards making prefetch support fetching multiple root trees/paths at once.
Reviewed By: quark-zju
Differential Revision: D45273891
fbshipit-source-id: 8b93ac1f5a327555d80c22410c7c95ab56af9984
Summary: removed python 2 specific references along with sys.version.info and dependencies.
Reviewed By: sggutier
Differential Revision: D44715340
fbshipit-source-id: 0dae8cf1bd9cb21a8ca336d22efecf796930bccc
Summary: I will reuse this `SearchDirection` in GitRenameFinder in the following diff.
Reviewed By: sggutier
Differential Revision: D45233573
fbshipit-source-id: 4885fc5df8beda729d23843471d63aa8e870c8f4
Summary:
Code42 was known to crawl the repository without taking
into account that files are virtual. It seems that this has been corrected in the new CrashPlan 11.0.1 agent, but we will land this diff as a safety precaution to not allow them to read the repository
Reviewed By: xavierd
Differential Revision: D45282206
fbshipit-source-id: 32f77aa96745bccb36e43d4df8ff47755a7670bc
Summary:
This data may be useful for analysing the blame history of the file.
* `approx_commit_count` is the total number of commits in the blame history of the file. It is approximate as some merged in commits may be omitted.
* `distinct_range_count` is the number of distinct ranges in the blame, which may indicate the file's complexity.
Reviewed By: RajivTS
Differential Revision: D45218574
fbshipit-source-id: c0c399f525f8c047a0415cf9d6f6e3483c033700
Summary: bunch of unit tests that relied on the specific error formatting weren't caught by CI and are failing. this fixes them by changing their error formatting to use `{:#}`.
Reviewed By: markbt
Differential Revision: D45245281
fbshipit-source-id: 6b0c9c5f3a3ecb594292d2a658ecce457f007e7e
Summary: Directs DifferenceOfUnionsOfAncestorsNodeStream callsites to use commit_graph().ancestors_difference_stream instead conditional on a tunable (enable_new_commit_graph_ancestors_difference_stream) being set to true.
Reviewed By: markbt
Differential Revision: D45042733
fbshipit-source-id: ca508d9b1a2e9efd7b45d1ee834f7587a7d886c3
Summary: Implements a streamable variant of ancestors_difference to be used as a replacement for DifferenceOfUnionsOfAncestorsStream
Reviewed By: Croohand
Differential Revision: D45042732
fbshipit-source-id: aa7377512866038172899545ee9c43a2d6db39f7
Summary: Use ChangesetsCommitGraphCompat when constructing Changesets in TestRepoFactory to enable double writing to the new commit graph
Reviewed By: mitrandir77, singhsrb
Differential Revision: D45227317
fbshipit-source-id: 6ce1809a2e30d930f8edef1efda0f9bef53b2a7c
Summary: Replaced by using TestRepoFactory directly and adding the extra config to the default config.
Reviewed By: RajivTS
Differential Revision: D45227320
fbshipit-source-id: bd93d1a2fd3a10398ca1cfd4b647a32627b83179
Summary: Has the same behaviour as the contruction of warm bookmarks cache in Repo::new_test_common. To keep the same behaviour this required making TestRepoFactory into an async builder and modifying all of its callsites.
Reviewed By: RajivTS
Differential Revision: D45227318
fbshipit-source-id: 51303d48bebc0c4b915a359a7d0942bf485e8db5
Summary: Adds a sync method that awaits the completion of all ongoing updates. Mostly useful for tests to avoid adding artificial sleeps.
Reviewed By: RajivTS
Differential Revision: D45227319
fbshipit-source-id: 9f3a9c7d063ad065b913db521a27624041b2400a
Summary:
fix 'repo changed while backing up'
the current logic is still incorrect for non besteffort (regular) mode
we introduced a high level transaction in D44664729, so now it is safe to drop
the check for non besteffort (regular) mode entirely
Why is it incorrect?
This is because the check is inside the retry loop (have lost the race),
inside the loop we apply commit cloud changes from the currect sync,
so basically if we haven't synced yet at this stage we will always fail the check.
We are loosing our retry logic that we should keep for the regular mode.
After a high level transaction has been added, we can drop the check and
resuscitate our retry logic.
The only possibility for another transaction to change the repo is that the transaction belongs to the previous iteration of the loop in the current cloud sync.
Reviewed By: quark-zju
Differential Revision: D45271716
fbshipit-source-id: 78de00c512f6ac737b107053563bd74b43080d32
Summary:
✨(clone): set publicheads to remote HEAD when cloning
c8b66c6899 added support for cloning the remote git repo's HEAD (in this
case `develop`), instead of main/master by default.
However, this doesn't mark `remote/develop` and its ancestors as public
commits.
This commit will also set remote git repo's HEAD as a public commit
Closes https://github.com/facebook/sapling/issues/600
Pull Request resolved: https://github.com/facebook/sapling/issues/607
Pull Request resolved: https://github.com/facebook/sapling/pull/607
Test Plan: - Added test `test-git-clone-sets-publicheads.t`
Reviewed By: sggutier
Differential Revision: D45249747
Pulled By: zzl0
fbshipit-source-id: 8c3a36bd21a25a856d13cc3b2c78f9cbdd9a94c5
Summary:
When creating a stack, make it possible to request that a derived data type is prepared as part of creating the stack. Since the data for the stack is in local cache, this should be faster than it would otherwise be.
This is a simple initial implementation. It requires that the data is already derived for the parents of the stack, and does not support parallel derivation of the different types, nor types that include interdependencies. In fact, we only support fsnodes and skeleton manifests right now.
Differential Revision: D45216982
fbshipit-source-id: c6ca7b265b1e6219a1193c51d0a984794aa34965
Summary:
Add support for skeleton manifests as a type of derived data that can be prepared.
The current enum for derived data types does not follow best practices and uses zero. Fix this now by adding a new variant for `FSNODE`. Once server support is available, we will remove `FSNODE_NEW` and switch to that value for `FSNODE`.
Differential Revision: D45216983
fbshipit-source-id: b0714ef6f9121139cc8d4602a5a8d1beaac00a3f
Summary: Move the preparation of derived data into a `mononoke_api` method in preparation for re-use.
Differential Revision: D45216984
fbshipit-source-id: ee99379685a202018b4c32b4b56e2930755d723a
Summary:
After my previous refactoring, the memory safety bug in Nfsd3 no
longer manifests, and we can remove the unnecessary defer call.
Reviewed By: kmancini
Differential Revision: D44988749
fbshipit-source-id: 2ebd5fbf96d05ce6850fc248a642409f827ffeec
Summary:
All of the comments in RpcServer are helpful. I found a couple
opportunities to improve their clarity.
It also helps to decouple RpcStopReason from the connection status.
Reviewed By: kmancini
Differential Revision: D44988608
fbshipit-source-id: 967f35ba09533a9c8c953ef33cd595b51e7c3111
Summary:
I'm carefully reading the takeover code and noticed a few things to
improve.
Reviewed By: kmancini
Differential Revision: D44946740
fbshipit-source-id: 558441a47425b15c19ca2a6816ae2e8bf3d68019
Summary: When tests are run in parallel, it can take more than a couple of seconds for the server update to become visible. Increase the wait to up to to 10 seconds.
Reviewed By: RajivTS
Differential Revision: D45229687
fbshipit-source-id: 24bc9607c285a4098c35a4105b8deac7916a7591
Summary:
add `debugcloudstatus` command
this command is to be used by automation, for example, on demand to get a name
of commit cloud workspace
currently they have to parse the name from the text output of the `hg cloud status` but providing json output is less error prune.
Reviewed By: quark-zju
Differential Revision: D45227570
fbshipit-source-id: 9e1a67edcefa7a2f7f11c7a80e99d061e5ca4bad
Summary:
Fix a typo. The word capabilities hurts my eyes when I look at it too
long, but it's the correct spelling, so sed it into place.
Reviewed By: kmancini
Differential Revision: D44945525
fbshipit-source-id: 05e82c058c3015b0590b07d6458840b4c56730b6
Summary:
Out of curiosity, I'd removed a .via(getEventBase()) from
TakeoverServer, and it manifested as hard-to-understand segmentation
faults in the integration tests. The issue is that FutureUnixSocket
has EventBase affinity and must only be accessed from it.
To make this safer and more obvious, put FutureUnixSocket and the
other takeover connection state in EventBaseState.
Reviewed By: kmancini
Differential Revision: D44945469
fbshipit-source-id: add3af84a35d451385a73afb622f9b63c7cdb217
Summary:
I want to use EventBaseState in TakeoverServer, so move it from
RpcServer to eden/fs/utils.
Reviewed By: kmancini
Differential Revision: D44945418
fbshipit-source-id: b9738162c84ca20da800d3662be6271f4b492acc
Summary:
Some dead code, a judicious use of makeFutureWith, and replacement of
thenTryInline+makeAsyncTask with via+thenTry makes the implementation
of TakeoverServer easier to follow.
Reviewed By: kmancini
Differential Revision: D44945165
fbshipit-source-id: 67c7b68b892c5112620bdc1ab2a0feb7df4d1864
Summary:
For small handle types, passing by value is the correct choice,
because it indicates unconditional ownership transfer. (It also
generates slightly smaller code and is less typing.)
Reviewed By: xavierd
Differential Revision: D44927835
fbshipit-source-id: 5fefee7889e8216e46548f4801e83f51cde8b8c4
Summary:
Simplify ServerState a bit by returning a const reference to the
member shared_ptr.
Reviewed By: xavierd
Differential Revision: D44924418
fbshipit-source-id: cd5f3a17d92b677ec7be43d9a07ed38dca1a3c24
Summary:
No need to have an intermediate object to forward callbacks from
ReadCallback to RpcConnectionHandler. This simplifies both memory
management and data flow.
Reviewed By: kmancini
Differential Revision: D44860344
fbshipit-source-id: af02f7d7f331d6ff3a8c4f49abb5ba1b95db5f85
Summary:
Adding the below cli outputs to eden rage
`eden trace inode --retroactive` (this one is empty though because we don't publish events right now)
`eden trace thrift --retroactive`
`eden trace hg --retroactive`
Reviewed By: kmancini
Differential Revision: D45165616
fbshipit-source-id: f4a89377fdc489e10bd78dc4175bdf89e2992cbf
Summary: Backing out the revert of D44173515 to unblock RE with their development. I will revert my own diff instead.
Reviewed By: genevievehelsel
Differential Revision: D45190063
fbshipit-source-id: 2b14fdc2fa118719aed0f3215393d172a162639f
Summary:
Based on statiscal data, most renames have either same basename or
same directory, so we will use this information when checking potential
renamed-to files.
Reviewed By: quark-zju
Differential Revision: D45184854
fbshipit-source-id: 7b4ed45d27781725c372beb33d24a0960d442816
Summary:
add file_path_similarity function, we will use this to reduce the
number of files to compare for Git use case; and also use it for sorting
the potential files for Sapling use case.
Reviewed By: quark-zju
Differential Revision: D45157537
fbshipit-source-id: 476c9fa07f2409a2cf0d1a6a72c57069a42ad347
Summary:
UDS don't have port numbers, so trying to register their port with the
portmapper is going to fail. This can cause eden to fail to start when we
enable uds. The current portmapper version we use can't support
this, so let's just skip attempting to register uds servers with the
portmapper.
Reviewed By: xavierd
Differential Revision: D45196379
fbshipit-source-id: 584e6e04129d1ddbcc756ed66c55a03c3955705a
Summary:
The case that I met was like the following DAG, and I'd like to skip those `Landed` commits when running `hg ab` since they already landed.
```
╷ o zzz Apr 21 at 18:18 zhaolong
╷ │ commit zzz
╷ │
╷ o yyy Apr 21 at 17:18 zhaolong
╷ │ commit yyy
╷ │
╷ x ccc [Landed as ddd] Apr 21 at 9:00 zhaolong
╷ ╷ commit bbb
╷ ╷
╷ x aaa [Landed as bbb] Apr 12 at 08:00 zhaolong
╭─╯ commit aaa
```
Reviewed By: sggutier
Differential Revision: D45184140
fbshipit-source-id: a539468843032d41ab4d61e5c461b9a1de53cb40
Summary: Like in the previous diffs, this diff uploads the raw git tags during repo import through `gitimport`.
Differential Revision: D45118571
fbshipit-source-id: f5a6bb9b5805235e9f66f97ab3a58bd165ef475c
Summary: The previous diff uploaded raw git commit data while the commit gets imported in Mononoke through gitimport or remote-gitimport. This diff ensures that the git `tree` associated with the commit also gets uploaded in Mononoke.
Differential Revision: D45085959
fbshipit-source-id: cf4f4a5ebd0d077972f75fe9c6a8d1dfc5067180
Summary:
With the `upload_object` method in place, this diff utilizes the new uploader method to upload the raw git commit in Mononoke store before creating its corresponding Changeset at Mononoke end.
This diff has the following changes:
- Modified the `get_object` method in `GitReader` to return the raw git content (i.e. header + body bytes of git object) along with the parsed `GitObject`
- Added the `read_raw_commit` method in the `gitimport_objects` library to read the raw commit bytes
- Modified the `ExtractedCommit` type to contain the raw bytes of the commit as well
- Modified all modes of `gitimport` execution to upload the raw git commit before creating the corresponding changeset at Mononoke end
- Made changes in integration test to validate the commit upload behavior
Differential Revision: D45082135
fbshipit-source-id: f339b2851646996be7f43f02458ea5afabb9ca48
Summary:
This diff includes the following changes:
- Introduces the `upload_object` method in the `GitUploader` trait for uploading raw git-objects (e.g. commit, tree, tag) in the mercurial mirror of the repo
- Implements the `upload_object` method in `Direct` and `Remote` implementations of the `GitUploader` trait allowing the objects to uploaded from both `gitimport` and `remote-gitimport`
This diff just introduces this method, follow-up diffs will use it for uploading different commit objects.
Differential Revision: D45046811
fbshipit-source-id: 9b4e6a295ec22e06294e410735e67877e6d0ef10
Summary:
The existing `mononoke_api` git methods were defined under `RepoContext` which relies on a specific type of `Repo` facet container as defined in `mononoke_api` crate. These methods can easily be used through SCS which serves as an external entry point into the `mononoke_api` crate. `remote-gitimport` uses SCS and thus can leverage the necessary endpoints.
However, `gitimport` (or `direct-gitimport`) directly references the logic behind these methods. To ensure that we don't repeat ourselves, there are two alternatives.
- Make `gitimport` use the same `Repo` facet container as the one in `mononoke_api` crate
- Decouple the git methods from `RepoContext` and define them with explicit `facet` dependencies
I chose the later since using the same facet container would have made `gitimport` use a much bulkier `Repo` definition than is necessary. I also made some changes in the `create_annotated_tag` method which are as follows:
- Removed the `start_write()` statement which was essentially an authorization check to validate if writing to the repo is allowed. This check makes sense when validating an external caller (e.g. SCS) but direct references of the library should be allowed
- Removed `CreateChangeset` permission check since we are not actually creating a changeset that corresponds to a commit. Additionally, any auth checks need to be performed at the call-site (e.g. SCS) and not in the API
- Removed the `allowed_no_parents` check since any changeset created for representing a `Tag` object will not have a parent. Infact the concept of parent is invalid for a tag.
- Replaced the returned `MononokeError` into `GitError` to maintain consistency with other methods
- Replaced the instance `save_changeset` method with the struct-level `save_changeset` method. The new method doesn't log to the scribe category thus preventing users tailing it from receiving an invalid commit creation notification. Additionally, the new method doesn't consider any bubble changes but they are anyway irrelevant in the context of these git methods.
Differential Revision: D45046639
fbshipit-source-id: 7ff02cde3fd356eb7a18f5ca12e6c478e23ee315
Summary:
Previously, the nfs.allow-apple-double was tentatively rolled out but had to be
rolled back as XCode disliked when an AppleDouble file was present on disk but
a new one couldn't be created. To avoid this issue, let's prevent AppleDouble
from being loaded from the Overlay and re-write the Overlay for directories
that contains one at the same time.
As a slight behavior change, the nfs.allow-apple-double is no longer read on
the fly but only read at startup time to avoid cases where a TreeInode is
already loaded with an AppleDouble and the config is flipped which would cause
XCode to fail to save files.
Reviewed By: chadaustin
Differential Revision: D44854105
fbshipit-source-id: 2c263411719d19eb1d457e26661620e8d6c905ce
Summary: In preparation for deleting blame v1, make blame v2 the default.
Differential Revision: D45044338
fbshipit-source-id: c9c71a148b42f4a3301d6c4fd0de5ad7ed321a3e
Summary:
Allow the walker to walk the blame v2 graph.
Since the walker is the main user of blame in the integration tests, we can make v2 the default for the integration tests, too.
Differential Revision: D45044294
fbshipit-source-id: 52c991dd81be78053ffc0a6953caf8a068ef59e3
Summary: This is a benchmark, so put it with the other benchmarks.
Differential Revision: D45114045
fbshipit-source-id: c524c76671d4183f6baa71ca9cd4c3cdfc402dac
Summary:
We don't need folly::Synchronized for the RpcServer state
machine. It's only touched from the EventBase, so make it
EventBaseState.
Reviewed By: xavierd
Differential Revision: D44860240
fbshipit-source-id: bcbc3d02e7cc5f2dc377718065c7f95eed30202d
Summary:
There's nothing TCP-specific about RpcTcpHandler, so give it a more
accurate name.
Reviewed By: xavierd
Differential Revision: D44860198
fbshipit-source-id: 55b9dc95b2caa9d87179a17435313b91bacfee49
Summary:
RpcServer can handle the accept callbacks directly. There's no need
for an intermediate object.
Reviewed By: xavierd
Differential Revision: D44860163
fbshipit-source-id: fd2d96735810a52cd287c357534c8f5bae9eee90
Summary:
I plan to use StateWrapper elsewhere so rename it to EventBaseState
and move it to the top level.
Reviewed By: xavierd
Differential Revision: D44860040
fbshipit-source-id: fb03b3b726df50099bcce144e343be0caeeaf1be
Summary:
There is no need for an intermediate object: RpcTcpHandler can handle
the write callback methods directly.
Reviewed By: xavierd
Differential Revision: D44860027
fbshipit-source-id: fe1f44b83a371291a21a3b1b26422bef5cbbf2a0
Summary:
To simplify RpcServer further, assert that removal of the accept
callback is a synchronous operation. Previously, it wasn't, because
specifying any EventBase in addAcceptCallback created a RemoteAcceptor
which scheduled operations on the other EventBase.
Reviewed By: xavierd
Differential Revision: D44853408
fbshipit-source-id: 9369cdcb0bd8450d42e195c15738362d4a53d2e7
Summary:
I'm trying to track down a subtle lifetime rule violation in
EdenMount. While reading RpcServer, I noticed a few possible
simplifications.
Reviewed By: xavierd
Differential Revision: D44849773
fbshipit-source-id: 4c27c47a7e2c211dcd040acce955e9b2f617b55b
Summary:
This removes an ifdef _WIN32 in mount shutdown by standardizing the
takeover logic on all three platforms.
Projected FS doesn't support takeover, but removing ifdefs allows
simplifying the control flow and unblocking NFS or other FsChannel
implementations on Windows.
Reviewed By: kmancini
Differential Revision: D44737273
fbshipit-source-id: fc6b718e01698ef3700b5c198750c0e3b1f62063
Summary:
By moving NFS's StopData behind FsStopData, the std::variant and
runtime checks during takeover can be unified into one path. The next
diff will do the same for Windows.
Reviewed By: kmancini
Differential Revision: D44737160
fbshipit-source-id: 19ca623f90b367d25385f177cafa44d7bf7ebb62
Summary:
Adds a buck target to drop us into a Python repl, to ease interactive
development of Python commands in the CLI.
Reviewed By: genevievehelsel
Differential Revision: D45159857
fbshipit-source-id: 31ff61ea29ab080cc1131905c6cf363368f3c617
Summary:
Original commit changeset: a755f0750ed3
Original Phabricator Diff: D44010967
need to revert to fix our release
https://www.internalfb.com/intern/sandcastle/job/36028797946731051/insights
Reviewed By: xavierd
Differential Revision: D45156142
fbshipit-source-id: 6e5362cf609f69cdbfeb7da30981227f93d42b65
Summary: The original `find_renames` finds all the renames of the commit and checks if it contains the path that we are interested in. This diff optimizes that logic to make it return early when found the target rename. In order to do that we split `find_renames` into two functions `find_rename_forward` and `find_rename_backward`, since they will have different optimization logics.
Reviewed By: quark-zju
Differential Revision: D45149506
fbshipit-source-id: b77b3b9ef4700581e6d044644d40bf405d66d263
Summary: fix a typo in the doc of read_file_contents function
Reviewed By: quark-zju
Differential Revision: D45126147
fbshipit-source-id: 6f541acb41c5de71f56c5573a5f56b536b669959
Summary:
Practically the graph log and the find_renames can use different impls
independently and form different combinations. This will be used for suppoting
Git use case in following diffs.
Reviewed By: quark-zju
Differential Revision: D45125851
fbshipit-source-id: 785800a0e4b1c74c48b97cf47132b45b268dc0bb
Summary: This tool was used for the original import of globalrevs. We are now the source of truth for globalrevs, so it is no longer needed.
Differential Revision: D45112829
fbshipit-source-id: 8ff106c66de74a79e6cd963ca48bd1875d6ee44c
Summary: This tool was used to rechunk the filestore. The filestore is always chunked now, so this tool is no longer needed.
Differential Revision: D45112830
fbshipit-source-id: cd03dd06f7dc55a6b47b8fc338accf9d300977d2
Summary: This tool was used for the initial population of the XDB blobstore. It has been superseded by the walker, and can now be deleted.
Differential Revision: D45112828
fbshipit-source-id: 134e2c2b761fd27a9b096df4d64404bc14c89c43
Summary: This tool lints toml-format configs which we don't use in production any more, and as such the tool is unused.
Differential Revision: D45112827
fbshipit-source-id: a4f7783144ab782adba5f0a0fb346e8cb542ad97
Summary: This tool was used early on to compute commit stats, but is no longer needed as it has been superseded by other data.
Differential Revision: D45112824
fbshipit-source-id: e88282647e8e3ed85a95ff2a9b15d9212851b8ff
Summary: This tool was introduced to fix up the problem with linknodes pointing at commit cloud commits, however that has long since been resolved, so the tool is no longer needed.
Differential Revision: D45112822
fbshipit-source-id: 350b770e9a3499a8bad8aca29badc3e20a02f168
Summary: General-purpose running of hooks has been superseded by the source control service `run_hooks` method.
Differential Revision: D45112823
fbshipit-source-id: 0093953da8fbf4289260058e7db076c2537bf262
Summary: This was used for an experiment with very large repos which has now concluded. Remove the code so that it doesn't need to be maintained.
Differential Revision: D45112825
fbshipit-source-id: 2591ab469ddf40ab8155c9d3713208f9bc3e5463
Summary:
When a user has more than 8 mounts we summarize the output of eden start. But
we have an underflow in the number of "finished" mounts.
The issue is that we do:
"number of successfully mounted mounts" - (
"number of mounts that have not yet started that were already printed" +
"number of mounts that failed that were already printed" +
"number of mounts that successfully mounted that were already printed")
and stick that in an unsigned int.
When there are not yet started mounts or failed mounts this number underflows.
This makes the output confusing.
This diff introduces a new category for failed mounts and displays this in the
summary.
And not yet starting mounts are considered "in progress".
With these two fixes that number that unerflowed is now just:
"number of successfully mounted mounts" -
"number of mounts that successfully mounted that were already printed"\
which should not underflow.
Additionally, while testing I noticed we have some output issues with the
errors we display when a mount fails. So I fixed the ordering of our output.
Reviewed By: chadaustin
Differential Revision: D45128100
fbshipit-source-id: 181adaaa7c9dbaba367cb6793ecf648515f4d511
Summary:
`eden config` prints the CLI configuration, which is different from
the daemon configuration.
The fact that both exist is confusing. Nonetheless, add an `fsconfig`
command which prints the current value of every config setting, as
well as the file that sets it.
Reviewed By: kmancini
Differential Revision: D41896506
fbshipit-source-id: 7a32daf04c5f4da31d50508ba4012585b0b6c069
Summary:
There are cases where the case of a file changes but not changes are done to
it, in which case this doctor check would fail.
Reviewed By: mshroyer
Differential Revision: D45129119
fbshipit-source-id: b0473485b760127715ec4372e8511936189c30a3
Summary:
We overuse make*FutureWith. As a rule, leaf functions should not need
it, because callers must already consider the possibility that an
exception is thrown prior to the asynchronous function's first
`await`. (The same rules apply to Rust async functions, C++
coroutines, Future chains, and Python async.)
Remove an unnecessary makeSemiFutureWith in performBindMounts..
Reviewed By: kmancini
Differential Revision: D44932401
fbshipit-source-id: 5a2a2773cc92faa59a981ee7b1e01324b5d3d5d5
Summary:
Adds a doctor check to warn the user if they're running a Windows build and
revision older than KB5022906, which has the fix for a known issue with ProjFS.
Reviewed By: xavierd
Differential Revision: D45104879
fbshipit-source-id: 57cae39c0f488f7b5c295b642a96f76bccbbb3ce
Summary:
Since EdenFS doesn't explicitely sets the timestamps of files/directory when
writing them on disk, ProjectedFS sets these to the current time. Prior to
running the background GC, this was an appropriate behavior: the timestamps of
files would only get changed on checkout when the file changed. However this
behavior changed slightly since background GC got introduced as placeholder
will get invalidated when not accessed for a while. The next time the
placeholder is written on disk, it's timestamps would now be different from
what it was before GC.
To avoid this issue, we need to consistently write the same timestamp before
and after GC. This is more or less the last checkout time. The one current
downside is that the last checkout time is reset across restarts, to fix this,
we could write the last checkout time to disk and restore it at startup time.
This is left as a future improvement.
Reviewed By: MichaelCuevas
Differential Revision: D44936789
fbshipit-source-id: a21ef3f2f0ef1c0d7ecb57658ae99647dc2bd99b
Summary:
StopData is slightly different for each filesystem implementation. But
it's used in the same way. Introduce a standard interface and port
FUSE to it.
Reviewed By: kmancini
Differential Revision: D44737143
fbshipit-source-id: 93ea567b6bc4d20a89ea1e285ab7240f533f6854
Summary:
It's been a long time since EdenFS has managed bind mounts in the
daemon. We no longer need to populate the bindmounts field in the
takeover protocol.
This is a small simplification for later changes.
Reviewed By: kmancini
Differential Revision: D44737123
fbshipit-source-id: d9127445083d83626d214efbbf271cd58df96ac6
Summary: Switch from the deprecated cmdlib to `mononoke_app` as a dependency. The type originates in `mononoke_app` and is simply re-exported from cmdlib, so this is a no-op change.
Reviewed By: RajivTS
Differential Revision: D45092104
fbshipit-source-id: ff8999a68c9707131a5e499b67f6d96492066b28
Summary: This library doesn't seem to be used anymore, so let's delete it.
Reviewed By: RajivTS
Differential Revision: D45092106
fbshipit-source-id: 46d477e6847b876ee2ece74d5e7ca69c787ce32a
Summary: The `--skip-caching` arg is replaced by `--cache-mode=disabled`.
Differential Revision: D45089768
fbshipit-source-id: 22b176b992351368fe96865c8559f83c75015409
Summary:
Now that Windows is going to support NFS. We need a reliable way to check at
runtime if a file should be read from disk (Prjfs) or it should be read from
the overlay (nfs).
We read inodes during fsck which is before the mount is fully initialized, so
we need to be able to detect the mount type earlier than how we do right now.
Here I am moving the NFS detection before eden mount initialization.
While moving this I realized that the InodeMap's view if NFS is incorrect
in the takeover case, so I am also fixing that here.
Reviewed By: chadaustin
Differential Revision: D45020849
fbshipit-source-id: b0a8fd431a37174c81b0d053d92b8bac026bd0f1
Summary:
Now that we have the plumbing we can start registering our NFS RPC servers (
mountd and nfsd) with the portmapper/rpcbind.
I am taing a short cut for prototyping. I am skipping general purpose
rpc registration (i.e. making these rpc calls to register a service). This is
because (1) I am trying to prototype quickly, so I don't want to implement the
portmapper endpoints that allow registering/ unregistering a server and (2)
there are maybe security considerations, and I don't want to think about those
rn.
Reviewed By: chadaustin
Differential Revision: D44987884
fbshipit-source-id: 4b849d1dd060d2e033ab48df883e228015906a7d
Summary:
getport is the primary endpoint on the rpcbind/portmap server. NFS clients
make a request on this endpoint to find the NFS RPC servers.
In this diff we keep a mapping of RPC servers and respond to the getport
request based on this mapping.
Currently no rpc severs are registered, so we reply no port to all
requests.
Reviewed By: chadaustin
Differential Revision: D44987893
fbshipit-source-id: 920171c6007b82609ae84fb7a9aff43d898e43dd
Summary:
The msft client can't be pointed at an NFS server on a particular port, so
we have to register our RPC servers (mountd and nfsd) with the port mapper
service.
There is no portmapper service running on Windows by default.
There is a port mapper implementation from msft that can be installed on
Windows Server, but not on Windows desktop.
There are some third-party implementations of the portmapper for Windows. Most
of them are part of an NFS Server implementation. But none that I have found
both run on modern Windows versions, are a general purpose port mapper, and
support port mapper v2. General purpose means they allow processes to set a
port mapping. Most the port mappers I found must register their NFS server in
process, because they don't support calling the set mapping endpoint. The rest
only run on Windows versions from before I was born or only run v4.
I tried porting the linux gnu implementation of port mapper to Windows. But despite
a few hours of trying I could not even get it to build even on Linux. Writing our own port mapper is faster.
This is the skeleton of the server. In the next diffs I will register our
RPC servers and support telling the msft NFS client about our servers.
Reviewed By: chadaustin
Differential Revision: D44987863
fbshipit-source-id: bff065795a9f4b7b6c13ef3e3ce603646e1ce364
Summary:
The msft NFS client likes to speak portmapper v2. It does not seem to know v4.
Perhaps there is a way to configure that, but I have not found it yet.
v4 is the latest version of the protocol and so what we should prefer using on
mac and linux.
Here I am introducing the v2 protocol and renaming all the v4 types to
explicitly reference v4.
Reviewed By: chadaustin
Differential Revision: D44987848
fbshipit-source-id: 97f9e2365736b49a5888af204609598b2566de55
Summary:
There is no portmapper service running on Windows by default, so we should
not be trying to connect to one by default on Windows.
Though I am adding a port mapper service that eden it self runs in the coming
diffs it is not compatible with this client. The msft NFS client likes to
speak portmapper v2 and this client only knows portmapper v4 (otherwise known
as rpc bind). So I am stubbing these out for now.
Reviewed By: chadaustin
Differential Revision: D44987834
fbshipit-source-id: 078ed1761cffbe9621ab39892c2135504a6aeb1b
Summary: it has been enabled in prod for 5 months, let's enable it by default
Reviewed By: sggutier
Differential Revision: D45086467
fbshipit-source-id: 3b0df83105ad76b67f6e841e81cd7223b68e3e9b
Summary:
[sl] bump macOS python CI to 3.11.3
We still depend on pointing to the specific Python combination that GitHub CI and Homebrew have, which currently is 3.11.3.
Hashes were taken from d39875e212/Formula/python@3.11.rb
Pull Request resolved: https://github.com/facebook/sapling/pull/605
Test Plan:
Downloaded bottles from:
https://github.com/sggutier/sapling/actions/runs/4727040284https://github.com/sggutier/sapling/actions/runs/4727040323
and installed the arm64 version. Also, verified that the dumps from `otool -L` looked correct (they usually don't when there is some when building)
Dump from arm64 version:
```
$ otool -L sapling\ 8/0.2.20230417-165855+6db1e950/bin/sl && file sapling\ 8/0.2.20230417-165855+6db1e950/bin/sl
sapling 8/0.2.20230417-165855+6db1e950/bin/sl:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60158.100.133)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1858.112.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1163.100.19)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
@HOMEBREW_PREFIX@@/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
sapling 8/0.2.20230417-165855+6db1e950/bin/sl: Mach-O 64-bit executable arm64
```
Dump from x86 version:
```
$ otool -L sapling\ 9/0.2.20230417-165855+6db1e950/bin/sl && file sapling\ 9/0.2.20230417-165855+6db1e950/bin/sl
sapling 9/0.2.20230417-165855+6db1e950/bin/sl:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60158.100.133)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1858.112.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
@HOMEBREW_PREFIX@@/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1163.100.19)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
@HOMEBREW_PREFIX@@/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
sapling 9/0.2.20230417-165855+6db1e950/bin/sl: Mach-O 64-bit executable x86_64
```
---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/facebook/sapling/pull/605).
* https://github.com/facebook/sapling/issues/606
* __->__ https://github.com/facebook/sapling/issues/605
Reviewed By: zzl0
Differential Revision: D45072742
Pulled By: sggutier
fbshipit-source-id: 4ced2d609d9cf4098cd4bbea5d5df45659588ad8
Summary: Look at at all the lines of code we could delete!
Reviewed By: YousefSalama
Differential Revision: D45057185
fbshipit-source-id: c454a4bd70527460bf4a727aa3ebe57b3becab4d
Summary:
I didn't want to repeat myself too much so I'm introducing macro that will do
that for me.
Reviewed By: YousefSalama
Differential Revision: D45057186
fbshipit-source-id: 6e13a5a72c6ad463982611ce11a16f937dcd9386
Summary:
The add_recursive would try to access the parents from changesets table
even if they're not there yet (in some places we're racing to inster them to
both tables at the same time).
For that we need to:
* process changesets in reverse topological order
* check edges_map before trying to fetch parrents from legacy changesets table
* skip fetching parents if the parent will be processed in the same function call
Reviewed By: YousefSalama
Differential Revision: D45057189
fbshipit-source-id: 7906c610c7a4f8473647796e0c56962e074a035f
Summary:
`add_recursive` has problem when adding many changesets not present in the
legacy changesets table. This test showcases the problem, the next diffs will
attempt to fix it.
Reviewed By: YousefSalama
Differential Revision: D45057188
fbshipit-source-id: 56297b7e2d47c6ec65023e9782ac0e18012a4319
Summary: Those helped me to localize the source of errors. I might as well commmit them.
Reviewed By: YousefSalama
Differential Revision: D45057187
fbshipit-source-id: 694d9596dec030e65b0d29590068103fb1b70ca9
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Implement the modified version of file_info endpoint using the ContentMetadataV2
Reviewed By: markbt
Differential Revision: D43537343
fbshipit-source-id: 98554366a725636d4cd6989fb7e2d9b5fe29d5e4
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Modify Thrift data type, and the function signature for the endpoint
Reviewed By: markbt
Differential Revision: D43537344
fbshipit-source-id: 6076703f67923e0fa95676b0db812a75fb6c3650
Summary:
These have broken production as the behaviour was changed so that `--foo` would no longer be valid when it used to be.
The behaviour changed inadvertently when migrating these CLIs to clap 4.
Differential Revision: D45081836
fbshipit-source-id: ffa7a2a2b1c04c860082bb2770eaa2cde1e7519f
Summary:
Fixes some warnings about unused imports and struct fields when building on
Windows.
Reviewed By: MichaelCuevas
Differential Revision: D45061519
fbshipit-source-id: 322768bf4e0d2af47129bd8f5c376211ee98d0d8
Summary:
My goal is to decouple our inode layer, including EdenMount, from the
specific filesystem implementation. To that end, remove several FUSE
references from EdenMount's comments and variable names.
Reviewed By: kmancini
Differential Revision: D44737102
fbshipit-source-id: 9956cf818599d508a897b92936350f40207d4b40
Summary:
Small refactorings that don't have a particular theme. I'm working
towards removing FUSE- and PrjFS-isms from EdenMount and friends.
Reviewed By: kmancini
Differential Revision: D44728015
fbshipit-source-id: 328abae1043a48f440836b98f1a1ee20a1b091d1
Summary: I think I addressed this one already in D44187187 but forgot to remove the FIXME.
Reviewed By: evangrayk
Differential Revision: D45050467
fbshipit-source-id: eb408ae2a9e1ba873e7f28ec3ff1cb755a1d2560
Summary: Fixed the cont-builds command and updated the tests to actually work instead of just passing all the time
Reviewed By: skotchvail
Differential Revision: D44898873
fbshipit-source-id: dbbc7429942a2d400346004e13bf7ba3cb00476d
Summary:
On Windows most applications, including Windows Explorer, are not able to open symlinks if they have forward slashes (e.g., `eden/scm/Makefile`). Windows users creating symlinks will probably create symlinks with backward slashes instead because of this (e.g., `eden\scm\Makefile`). Analogously, symlinks with backward slashes will be of not use on POSIX systems.
This diff tries to alleviate this by making symlinks added to commits on Windows via `hg commit` usable everywhere else.
Reviewed By: zzl0
Differential Revision: D44358129
fbshipit-source-id: 2835652b62104e22f10cb422586fae9998db40ad
Summary:
Makes `hg diff` and `hg commit` work as one would expect on Windows. This is done by modifying some constants in `fscap.py`.
This change should not impact end users, as explained in previous diffs.
Reviewed By: muirdm
Differential Revision: D43741223
fbshipit-source-id: 203878f64c11364257cef4bb82395713456f3625
Summary: The working copy uses the VFS object for determining whether a file is a symlink or not. Previously this behavior was disabled as there was no support for symlinks at all when this was written.
Reviewed By: muirdm
Differential Revision: D44592907
fbshipit-source-id: 6b9041a5045a336292a2b641da30cf3ae22f441c
Summary:
I noticed when reading the EdenServer startup and shutdown logic that
we did not log the structured DaemonStop event on Windows. The ifdef
must have been vestigial, so remove it.
Reviewed By: xavierd
Differential Revision: D44699330
fbshipit-source-id: 529aa2b2933b1ef3b8337fa19c1b88b9033653ba
Summary:
introduce cloud tidyup that can cleanup unnessesary remote bookmarks
this command can be used by ondemand or clients to get rid of unnessesary
remote bookmarks in a safe way server side
we can also use it in hg doctor command
Reviewed By: zzl0
Differential Revision: D44954711
fbshipit-source-id: 5d4e157c9b668f26cd39794d7f752828e4fc8b3f
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which were modified in the previous diff to support the new hash and are used here.
Reviewed By: muirdm, markbt
Differential Revision: D44010967
fbshipit-source-id: a755f0750ed39719b1f0e6895b7ca81a405fcfed
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which are modified in this diff to support the new hash.
Reviewed By: quark-zju
Differential Revision: D43977390
fbshipit-source-id: bd9a264dc4483574a7a46e7ce4f9109910f18f2d
Summary: If rust code writes to the "real" stderr/stdout in a debugruntest test (e.g. println!()), that output was getting dropped by run-tests.py. Now it at least prints it out, albeit not interleaved temporally.
Reviewed By: quark-zju
Differential Revision: D44768929
fbshipit-source-id: 9520229dbf885503cd53a21cc682df7135d72fb7
Summary: Use dagcopytrace to replace the current heuristic based copy tracing.
Reviewed By: quark-zju
Differential Revision: D44336238
fbshipit-source-id: 74fa131e78531d06ae2c263534a4a56b2081c041
Summary: The clash between `-h` for both `--help` and `--host` was inadvertent. Switch to `-H` for `--host` to disambiguate. It seems clap4 handles this clash less well than clap3 did.
Reviewed By: clara-9
Differential Revision: D44994925
fbshipit-source-id: c4195500cecd2c796e4765c2fbe6047df2f57593
Summary:
This "repo" determination was incorrect. In particular, it emitted "fb:foo" for a path of "fb:foo" (should be just "foo"). This code is totally superfluous, though, since the Rust dispatch also logs "repo" (correctly).
I also killed logging of "dirstate_format" since I think that is always "treestate" now (except for some tests and test repos).
Reviewed By: zzl0
Differential Revision: D44978910
fbshipit-source-id: 8a1478dc0dc7bfda6bdab683c5c5e57e6060ce78
Summary: The walker was migrated to the new cmdlib, but still depended on the old one for a couple of structs. Remove these dependencies.
Differential Revision: D44886561
fbshipit-source-id: 122d8aa01269b49651b95bd431b52045f91b88bc
Summary: LFS server uses the new command library, so no need to import the old one just for this struct.
Differential Revision: D44886560
fbshipit-source-id: 8b8869c9cf717aa622f374468ab061dff0ef349f
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44712484
fbshipit-source-id: e81a383ca1b89b87dc64577bbc42c56740599bf4
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44711493
fbshipit-source-id: adb9ce812423a11fae4f6c6a368d755780606944
Summary:
It's strange that BlobCache's configuration came from command line
arguments and TreeCache from the configuration file. To my knowledge,
neither configurations have ever been adjusted, so unify their naming
and replace BlobCache's command line flags with EdenConfig settings.
Reviewed By: xavierd
Differential Revision: D44699297
fbshipit-source-id: 88b0edf830e8dfc8f9eaa9ff04c563c54992f8bb
Summary:
I'm not sure how BlobCache and TreeCache grew to have different
ownership patterns. Allocate them together and prepare for
initializing BlobCache with EdenConfig.
Reviewed By: xavierd
Differential Revision: D44699259
fbshipit-source-id: fc91bb9fe80423e036bb7856d9f3b8863ba71ce8
Summary:
The error handling in EdenServer::mount was unnecessarily
complicated. Rely on thenValue and thenError to simplify the control
flow.
Reviewed By: xavierd
Differential Revision: D44699195
fbshipit-source-id: b94e513bf198f38e4614ccfa18ffab37bb43ff87
Summary:
A common FsChannel base class allows removing some ifdefs from
EdenServiceHandler. With some effort, we should be able to find the
correct abstractions that allow us to entirely decouple particular
FsChannel implementations from the rest of the inode layer.
Reviewed By: kmancini
Differential Revision: D44657395
fbshipit-source-id: 9702186f1d0c9ce76a11e766010f5899e2b517af
Summary: Don't hardcode remotenames.hoist to "remote". I didn't end up using a different value for this config, but probably an improvement worth committing anyway.
Reviewed By: quark-zju
Differential Revision: D44857509
fbshipit-source-id: 68a3660827d16083231ad98dc57321ac1ce3a377
Summary: Now if the clone source URL contains a fragment, we persist the fragment as the repo's remotenames.selectivepulldefault. This gives a user friendly way of specifying the default bookmark you are interested in.
Reviewed By: quark-zju
Differential Revision: D44857510
fbshipit-source-id: 54cffeebb76d6a9dc603adcc6ad58ab17568050c
Summary: If the user specifies --debug, don't delete the repo after an error when cloning. This makes it easier to debug clone failures.
Reviewed By: quark-zju
Differential Revision: D44857512
fbshipit-source-id: 921dcfdf472826aae407564b9f8b788e53058d20
Summary:
Now if you specify "--config remotenames.selectivepulldefault=whatever", we will persist it to the repo's config.
Note that this is still not good UX. See later diff that adds support for specifying default branch as URL fragment.
Reviewed By: zzl0
Differential Revision: D44857513
fbshipit-source-id: b8474f5d54e5d597c0ff44e9d689ccbfcc3068b3
Summary: I'm sure the Python code was useful during development, but it adds an unnecessary layer of complexity/noise to the reader.
Reviewed By: quark-zju
Differential Revision: D44857515
fbshipit-source-id: 19a5165b402cd2c1e8a367edb64862c2bdeae18f
Summary:
Canonical repo names can/will have slashes in them. We shouldn't convert slashes to directories on disk; just a bad idea.
When choosing the clone destination, use the basename. It could make sense to clone "foo/bar" into "foo_bar" instead of just "bar", but let's maintain the existing/expected behavior for now.
For internal uses, we percent encode the reponame. For example, the hgcache directory for "foo/bar" will be "foo%2Fbar". This is to minimize chance of collision. If we named the cache dir "foo_bar", that would collide unpleasantly with a repo named "foo_bar".
I relaxed reponame inference a bit to allow slashes in URLs like "test:foo/bar". This was purely so I could test with a reponame containing slashes.
A better long term approach is to use the hash of a canonicalized URL, but that is a bigger change.
Reviewed By: quark-zju
Differential Revision: D44857514
fbshipit-source-id: 4fa0b83a0660b81e89fc96b50ecbeb93869dd2b2
Summary: Make the debugruntest "ls" impl report an error if an argument doesn't exist on disk. This matches real "ls" behavior, and is important feedback when you've messed up the arguments.
Reviewed By: quark-zju
Differential Revision: D44857511
fbshipit-source-id: 0467e2c5e175882bb8033186235be36e5798bf7f
Summary:
must_get is like get_nonempty_opt but returns an error if the value is unset or
empty. This makes it easy to propagate an error if the desired config is not
set.
I also updated a few spots to use must_get and got rid of a superfluous error type.
Reviewed By: quark-zju
Differential Revision: D44857516
fbshipit-source-id: cdecd7a68d4e3d8248cae977f4cdf7b553b6af74
Summary:
Xavier just taught me about `cat -tail -wait`. I was watching the logs in vscode
on windows. Let's make a tail version of `eden debug log` so everyone else can
easily figure out how to tail the logs on all platforms.
Reviewed By: chadaustin
Differential Revision: D44941688
fbshipit-source-id: e86b18025f780dda69005ad9a31b1cb11d395261
Summary:
Fixes the bug discussed in Discord here: https://discord.com/channels/1042527964224557107/1042527965256364157/1095764831480590467
Running the ignore matcher on an empty path produces incorrect results. In the rust implementation, this isn't an issue, because there are directory matchers that run on the directory result. In the Python treestate bindings, we don't have that functionality, so it returns an empty string.
This makes this logic match the code in `dirstate.py` ([line ~925](https://github.com/facebook/sapling/blob/main/eden/scm/edenscm/dirstate.py#L925-L926)) where we check for path being an empty string.
This fixes `sl status` (and related functionality) when fsmonitor is enabled and there's a complex gitignore file that ignores files in the root directory.
Pull Request resolved: https://github.com/facebook/sapling/pull/601
Test Plan: Added Mercurial integration test and manually tested locally
Reviewed By: quark-zju
Differential Revision: D44961756
Pulled By: zzl0
fbshipit-source-id: 57dd0d052ceeb184694c6f3b1a71ac53bf0e1028
Summary:
This continues my slashing of fetch retries. This diff is 2x (for a total of
-2 now) fetch retry.
The importer prefetches the tree and then gets the tree. both can cause network
fetches, so when the prefetch fails, the get will retry the network request.
If I am guessing again I think the cause was similar to the last diff:
get use to be a local only check, but now its not anymore so we are exposed to
more potential network fetch retries.
It should be safe to remove this retry because we still kill and restart the
importer if the importer has errors, which should cover any successful re-trys.
With this change we are now down to what looks like 3 network request attempts
per request.
Reviewed By: xavierd
Differential Revision: D44526625
fbshipit-source-id: 455903bdc73c9513106c4e048274d5bb08085a38
Summary: We don't have plan to support this one-to-many mapping, since it will make copytrace complexity increase exponentially in theory. For now, we just pick the last copy in ascending alphabetical order. The reason why we do this is to make the selection deterministic, so the behavior will be consistent on different platforms and Rust implementations.
Reviewed By: quark-zju
Differential Revision: D44901629
fbshipit-source-id: 154b515c7043d4a5e45e1f2077f6930baf354e81
Summary:
In order to test new copytrace logic, we need eagerepo for finding
renames from the file content header.
Reviewed By: quark-zju
Differential Revision: D44816781
fbshipit-source-id: 129b729e1508d4e60e7d0c994c3163cdbdf09353
Summary: This benchmark was used during development of Mercurial data derivation, however it is very synthetic and we haven't used it in a long time. Remove it to ease the burden of maintenance.
Reviewed By: mitrandir77
Differential Revision: D44707860
fbshipit-source-id: 6dd2ac893b4020a0c77847c4f10848ba20d96e38
Summary: Replace the usage of `benchmark/simulated_repo`'s repository with an ordinary test repository with instrumented blobstore (via `DelayedBlobstore`) and database (via `SqliteCallbacks`). This removes this dependency on this code.
Reviewed By: mitrandir77
Differential Revision: D44707861
fbshipit-source-id: 49d4c2b43aa4dc64b10d94e21d8d242c97e05415
Summary:
We want the ability to observe and change the behaviour for sqlite connections. Add the ability to register callbacks that are called whenever the multithreaded sqlite connection is acquired.
No change in behaviour yet. We will use these in a future diff.
Reviewed By: mitrandir77
Differential Revision: D44707859
fbshipit-source-id: f8156e294bbc51b612a6e38604804b4f815ed919
Summary: The blobrepo test is actually a test of mercurial derivation, so move it there.
Differential Revision: D44676336
fbshipit-source-id: 2f7f002a91171a7081ceacb3c6f10b9c62824bfd
Summary: Move the library down into its own test directory with a `src` directory.
Differential Revision: D44676334
fbshipit-source-id: cd65adaf2b4a445f13b05d700be7a30492cadb97
Summary: Reorganise filenodes derivation so that it matches mercurial derivation.
Differential Revision: D44676333
fbshipit-source-id: bb5e11cb5a8344f3f7e812e1e53360355b1ce36b
Summary: Rename to `mercurial_derivation` and create a proper `src` directory. I plan to make this change consistently across all derived data types, but will start with Mercurial derived data so that I can add a test.
Differential Revision: D44676335
fbshipit-source-id: aae67df807575e532636dd46c4655f135d2d57b0
Summary:
follow up clean up dead code
forgotten in earlier diff
Reviewed By: YousefSalama
Differential Revision: D44931222
fbshipit-source-id: 6ec965ae5c7851693432fadc9b71df6a967259bb
Summary: `prefetch-profile` has been completely removed from the experimental command list for some time now, and there's no way any environment could be using it. It's safe to delete this code now!
Reviewed By: xavierd
Differential Revision: D44113592
fbshipit-source-id: f144b713c6e35de4691071afebbe7a829886b893
Summary:
We've seen several cases on Windows where `hg status` is extremely slow due to
the working copy having 10k+ materialized files. Reading through the code,
EdenFS ends up computing the sha1 of all of these files for every `hg status`
call which is quite costly.
On Linux and macOS, both the sha1 and the filesize of materialized files were
cached in the OverlayFileAccess code, but this had the downside of being very
small (100 entries) and not compilable on Windows. Instead of storing it in the
OverlayFileAccess, we can make use of the newly added
`FileInodeState::MaterializedState` to store both the size and the sha1. Beyond
working on Windows, this also comes at no-additional memory usage.
Reviewed By: chadaustin
Differential Revision: D44737138
fbshipit-source-id: 28cf14928de4e685cb21819915f7be7e7a64047b
Summary:
deprecate skip list based version
this is part of the effort to deprecate skip lists infra known as **Skip Lists Deprecation**
this endpoint is not called, it is safe to remove it either client first or
server first
also, we don't need to maintain two endpoints to check correctness, we passed that phase.
Reviewed By: zzl0
Differential Revision: D44916440
fbshipit-source-id: 05f623cc07718f1c9aa3d835d7654b47d3728d2d
Summary:
deprecate skip list based version
this is part of the effort to deprecate skip lists infra known as **Skip Lists Deprecation**
this endpoint is not called, it is safe to remove it server side first
we don't need to keep both for checking correctness, we passed that phase!
Reviewed By: YousefSalama
Differential Revision: D44916005
fbshipit-source-id: 662cfa20ad44346011126e0ae3aae69deb719dbc
Summary:
fbcode_builder doesn't currently know how to build a Python extension module,
which means the Windows oss build fails on trying to load ntapi if we don't
pass the `--no-tests` flag to getdeps. Ideally we add extension module support
to the builder, but for now let's just disable the test case if the extension
module is unavailable.
Reviewed By: chadaustin
Differential Revision: D44804035
fbshipit-source-id: 10da54f6ea7d3699ac42421ee5cf57d883509bdd
Summary: Log the first mononoke host we connect to. Under normal circumstances all connections from a particular client should go to a host in the same datacenter, so this will allow us to look at performance trends taking into account cross datacenter factors.
Reviewed By: zzl0
Differential Revision: D44900086
fbshipit-source-id: c9bb38dac419a8faf468c6bb79a44af3c152c502
Summary:
log duration for the origin request
so that we will be able configure replay in a more flexible way
Reviewed By: clara-9
Differential Revision: D44910760
fbshipit-source-id: 8d716376702283e24bb4151eb0a68b8bc2e57e6f
Summary:
An invalid bonsai changeset or mutable rename could lead to a copy source that does not exist.
For Mercurial derived data, we already ignore this. Do the same for general diffing.
Reviewed By: mitrandir77
Differential Revision: D44909673
fbshipit-source-id: ac754e0e483e75976afb07078e57bc1f33008a17
Summary:
In the repo factory, we do different things for sql connection creation in tests and prod. In prod we cache `MetadataSqlFactory` instances for each metadata database configuration, and request a new connection from the cached factory each time. In tests, this doesn't work, as requesting a new sqlite connection from the factory is the same as opening the file again, which can be slow for large numbers of connections, so instead we cache connections directly in the repo factory, which isn't really the right place.
We can fix this by making the `MetadataSqlFactory` handle the caching of sqlite connections for us. This makes the repo factory consistent between prod and test. It also means the `MetadataSqlFactory` can hold on to the original connection to the sqlite database and perform the creation queries directly, rather than having to hold it separately in the wrapper around `SqlConnections`, and we can deduplicate the code for opening the database, too.
Reviewed By: mitrandir77
Differential Revision: D44665118
fbshipit-source-id: 1c7d45ac1900d5f2b36285e01159924d0cdb09eb
Summary: These connections are no longer used, so they and their setup code can all be removed.
Differential Revision: D44635160
fbshipit-source-id: 4407a9318086b61942523cf551988e394c6df2e1
Summary: This is already used by a couple of other tests, so it makes sense to move it out of the benchmark.
Differential Revision: D44635161
fbshipit-source-id: 41414b66b380ea1d5635c6131df4fa3f3084fcfc
Summary:
This adds the configuration `doctor.ignored-problem-class-names` so that we can
make doctor ignore individual problem classes via config rollout.
This doesn't actually stop the checks from running, but it stops reporting (or
attempting fixes for) ignored problems. Ignored tests will still be logged in the
edenfs_events table's
Reviewed By: xavierd
Differential Revision: D44720584
fbshipit-source-id: 954d1131dfbbf6a3264abf55b5e0f01ab61836d0
Summary: Warn user about potentially confusing duplicate config entries when we are making automated edits to repo/user config file.
Reviewed By: quark-zju
Differential Revision: D44876243
fbshipit-source-id: 7d50df7f55e68f3655f6086156e803e52a2b4154
Summary: When there are duplicate config keys present, we now update the final (load bearing) one instead of the first one.
Reviewed By: quark-zju
Differential Revision: D44875407
fbshipit-source-id: fa546482cbbdb432fb7d02a716e4692022388dc5
Summary:
We've seen cases where invalidation is happening concurrently to
`PrjfsChannel::stop`. In that case, `getInner` would return a `nullptr` and
thus crash EdenFS as the code doesn't handle `nullptr` gracefully.
The root cause of why `stop` is being called on a mount that is still being
used is not entirely clear at this point due to the complexity of the
mount/unmount code which chadaustin is currently looking into. While this diff
should be revisited once Chad has reworked mount/unmount, we can make the code
correct by simply handling the `nullptr` case to avoid crashing.
Reviewed By: chadaustin
Differential Revision: D44875003
fbshipit-source-id: 977c59af9fcf0ae2c66929f22489370c18b92f94
Summary: If auth_proxy.unix_socket_path is configured, use that when testing for Mononoke HTTP connectivity.
Reviewed By: johansglock
Differential Revision: D44883873
fbshipit-source-id: 1c30645072842391ad9aa6b2548ee87bd1449896
Summary:
Added a config option `experimental.rebase-long-labels`. Once set to true,
produce more descriptive conflict markers "dest (rebasing onto)" and
"source (being rebased)" on conflicts.
Eventually we probably want this to be the default everywhere. For now, not
all users are compatible with this change so it's not yet the default.
Reviewed By: muirdm
Differential Revision: D44847924
fbshipit-source-id: b832f362cc1dd80e25a3fd80e0f334bffc4b12d8
Summary: This avoids revlog and its problematic code paths for all `pull` commands.
Reviewed By: sggutier
Differential Revision: D44805076
fbshipit-source-id: a4efacf111f03cda5752fe1b25d26a278c25f242
Summary: The previous diffs added support in SCS and remote-gitimport for including the `git-extra-headers` as part of the mirrored commits in Mononoke. This diff adds similar support for gitimport.
Differential Revision: D44869271
fbshipit-source-id: 5941b0c8729cd972188cc758837c6c1f1b0f72a1
Summary: The current healing logs just mention that a CMv2 blob was healed, but there is no mention of the actual blobstore key that it healed. This diff updates that.
Reviewed By: mitrandir77
Differential Revision: D44164682
fbshipit-source-id: c156a930716499fb1b5c7b248ce11d6bf88f8281
Summary: To maintain backwards compatibility for a while, we will be writing both the new `ContentMetadataV2` and the old `ContentMetadata` blobs. This will be behind a tunable which will be removed once we can verify that the code works as expected with `ContentMetadataV2`
Reviewed By: markbt
Differential Revision: D44221076
fbshipit-source-id: 95c01a96198a99e53ab3ea4f1b04b1ba78f4b298
Summary: D44136799 was the backout diff for backing out all the ContentMetadataV2 changes made so far. This was due to the reason that the changes were landed before the backfilling was completed. This diff is the revert of the backout, to be landed when the backfilling is properly completed.
Reviewed By: YousefSalama
Differential Revision: D44195175
fbshipit-source-id: 2872c699cc7c41cf436a1c7123e01a5190db7ece
Summary:
Eden's readdir and getfileAttribute endpoint returns an error when an entry in
a directory has a type that is not: regular file, directory, or symlink.
This causes issues for Buck2 because it propagates this error to the user. For
Buck2 a directory having a file that isn't a regular file, directory, or
symlink isn't an error case, it's just a file Buck2 wants to skip over. Buck2
would like to be able to differentiate real errors getting the filetype (like
say a network error) and having a weird file in some directory.
From chatting with Thomas, Buck2 is unlikely to ever care what type the file
is (if its not a file, dir or symlink). So it's sufficient just let buck2
know it's some "other" type of file. I think it makes sense to just add a non
source control type here. I also considered adding dtype as an attribute, but
I don't think we need it, but we could add that too.
In some cases it can be dangerous to add values to thrift enumeration
(SourceControlType enum we change below)
(reference post: https://fb.workplace.com/groups/thriftusers/permalink/785884732120941/).
But in our case, rust + Buck2 handles new enum types gracefully
(and with exactly the behavior we want):
https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/fbcode/buck2/app/buck2_common/src/io/eden.rs?lines=157
so adding a value to the enum is safe (for buck2).
hack is our other client. they are going to handle it less gracefully:
https://www.internalfb.com/code/fbsource/[65673fd318750984372aeb5b44036a259a0d85d2]/fbcode/hphp/hack/src/facebook/hh_distc/package/package.rs?lines=441 but from what I can tell hack would also
error if they tried to list a directory with a socket in it with out this
change. Will confirm with them that this change is ok with them.
Reviewed By: chadaustin
Differential Revision: D44794698
fbshipit-source-id: 4e3ab7964fa2c0932b0363fb9ad62f24af74480c
Summary:
I saw builds/tests were failing for centos due to a warning "egrep: warning: egrep is obsolescent; using grep -E".
We can instead use grep -E as recommended.
Note: this isn't every single test, but somehow only these tests had failures in the Centos CI job.
Reviewed By: muirdm
Differential Revision: D44848131
fbshipit-source-id: ba182e8c866ac6bd649f820b76097e4677295105
Summary: Configure post pull "debugmarklanded" hook in the requisite extension to avoid trying to run the hook when the extension is disabled.
Reviewed By: zzl0
Differential Revision: D44646572
fbshipit-source-id: 1ac2dd97d35616683b87223a97ebbe7aecaa0a91
Summary:
Don't crash on the "m" flag used to mark submodules in the manifest.
Also, deprecrate "sl manifest" in favor of "sl files". The "sl manifest --all" flag doesn't work anymore, anyway.
Reviewed By: sggutier
Differential Revision: D44772327
fbshipit-source-id: bad76f0defe95c4a11df9b606e56983583dc5e32
Summary: We offer the `--force` flag with the python implementation, this adds it for the C++ implementation in preparation for migration
Reviewed By: kmancini
Differential Revision: D44473785
fbshipit-source-id: 041613ff0d4048b40b02f7d8f024c78d659516f0
Summary:
Noticed that this was running while looking at a log on Linux. This does nothing on Linux/macOS besides loading inodes (ie: extra overhead), thus disable it.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D44835074
fbshipit-source-id: 3b80e4ccf0f63d50de395002c6a8d05a5e99952c
Summary: Fixes a wacky bug with one of our internal tools
Reviewed By: pushpakrajgautam
Differential Revision: D44811936
fbshipit-source-id: f3b6f07ff035c8d535792dec5add5146c1314f36
Summary:
Grammar update:
"eden debugedenimporthelper command command will be sent to this file."
Removed repeated ‘command’
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: chadaustin
Differential Revision: D44799690
fbshipit-source-id: 03e70a8ec300d2b635ea02f9e03802a4f44e9d32
Summary:
We have a force-unmount-all.sh script as a last resort when a Linux
machine has a pile of stuck FUSE mounts.
Introduce a new script that only unmounts the ones under $TMPDIR,
which were probably created by failed integration tests.
Reviewed By: mshroyer
Differential Revision: D44797173
fbshipit-source-id: 41f31cf1e2e57ad07076cf1b2571c739b67b77f0
Summary: fix clippy warning by using replacing match with if let
Reviewed By: quark-zju
Differential Revision: D44728168
fbshipit-source-id: 27a944267263595ea9367c08f6716a3d2f6820bb
Summary:
we need to check if the path is in the target commit when no rename
commits found, this fixes a bug found in the previous test
Reviewed By: quark-zju
Differential Revision: D44728170
fbshipit-source-id: 7a039b0f9fee80671653e1557b80b218b58f4e99
Summary:
refactor vertex_to_tree_manifest to only process one commit, so it can be used in the following
diff to check if a path is in a commit
Reviewed By: quark-zju
Differential Revision: D44728172
fbshipit-source-id: fc9530b38757d836ca8a5a46e463a5ade510839d
Summary:
this diff adds more unit tests to cover: multiple renames, non-linear
comit graph, deletes.
This diff contain tests catch a bug in the current implementation, which
will be fixed in the following diffs.
Reviewed By: quark-zju
Differential Revision: D44728169
fbshipit-source-id: b0aaeeffa9008a70bbfe3105cceed3afb3d65d45
Summary: Makes it explicitly mention that `sort` method sorts the set in topo descending order, this is different than the `sort` function in revset, which sort the set in ascending order by default.
Reviewed By: quark-zju
Differential Revision: D44728173
fbshipit-source-id: aa95993b237ebacc587147da9caccdf43ff48b80
Summary: Add test utilities for testing copy trace logic, also added a simple test case as an example.
Reviewed By: quark-zju
Differential Revision: D44559585
fbshipit-source-id: b03796fa3869d94edc528d570d511b1d84b15861
Summary:
add vertex_fn parameter to ImportAscii trait, so that we can control
how to generate Vertex from a string: for example HgId expects 20-bytes vertex.
Reviewed By: quark-zju
Differential Revision: D44739947
fbshipit-source-id: c82c95ce4d6c45562c32ad4ccc3e5b40fd517770
Summary:
Even though we only need the Enumeration in the future's callback, if we don't
move the Enumerator it can get dereferenced while we're awaiting the future
returned by prepareEnumeration().
Reviewed By: xavierd
Differential Revision: D44770398
fbshipit-source-id: a9fc91c0766b4a9f43f566887d714cbde66b1cf3
Summary:
This allows errors to be caught and raised as well as getting file sizes bigger
than off_t which is 32-bits on Windows.
Reviewed By: chadaustin
Differential Revision: D44737139
fbshipit-source-id: 16b00c7f436da361b41ffbe8074958519e9bfc81
Summary:
When the FileInode is materialized, the NonMaterializedState is entirely unused
but still consume memory. On Windows, we've seen cases where the working copy
contains 100k modified files leading to diff taking a really long time due to
EdenFS spending most of its time computing the sha1 of materialized files,
since these are not cached, this computation would be done repeatadly.
For now, this merely creates the scaffolding to enable this caching, future
diffs will move the sha1 and sizes currently stored in the Linux/macOS
OverlayFileAccess in the MaterializedState and care will be taken to invalidate
these on materialization/writes.
For now I've gone with a `union` approach as the existing `Tag` is sufficient
to know which branch of the `union` should be looked at. This is however
potentially error prone, reviewers' opinion on whether I should revamp this and
switch to a `std::variant` would be very welcome.
Reviewed By: chadaustin
Differential Revision: D44652779
fbshipit-source-id: e769f4cc0cd7e8eea0c4cde863cd37f4ceadb01f
Summary:
Tweak revlog commit import to maintain a stable order (this broke after recent change to top sort the commits).
Unfortunately the new stable order is inconsistent w/ fullsegments output in test-eager-exchange.t, so separate the tests.
Reviewed By: quark-zju
Differential Revision: D44768930
fbshipit-source-id: 6757450d7d446373d1b8c943d6cbcc6029b07443
Summary:
This works better with Git. The old filenode-based logic does not seem correct
since it might not trigger adjustlinkrev.
Reviewed By: zzl0
Differential Revision: D44751613
fbshipit-source-id: e5a2367e544f1d55c6a2c149fd33db13af623e46
Summary:
There is a collision between Blob and BlobMetadata ObjectId causing crashes in
EdenFS. To avoid these collision, let's just split the request tracker. Kudos
to chadaustin for finding the root cause of the crashes.
Reviewed By: chadaustin
Differential Revision: D44728974
fbshipit-source-id: 52bbc3f5317342ed975507f0a431bf4d3f712d8b
Summary:
return usage of commit graph for revlog clones
Disabling is no longer needed after D44734552
Reviewed By: quark-zju
Differential Revision: D44760863
fbshipit-source-id: 0ba65f63bf8c980fc0ab9b0dce1ec895f6b5f10d
Summary:
I'm getting nfs to run on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change theoretically we should be able to mount an NFS repo.
This isn't totally true, because I cannot get the Windows NFS client to point
at a specific port, so some port mapper changes are probably going to be needed
before that really works.
NOTE: this one is more than removing ifdefs, please review carefully
Reviewed By: xavierd
Differential Revision: D44155041
fbshipit-source-id: ae424a01dc2073138070253d80e3d101863840fd
Summary:
I'm getting nfs to run on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change the NFS Server opens a socket to accept incoming connections!
Reviewed By: xavierd
Differential Revision: D44154505
fbshipit-source-id: 10c679cb390c2bab99591656d6fe96ccdf4443fe
Summary:
I'm getting nfs to build on windows to prototype it and see how feasible it
might be as an option on Windows. PrjFS has a very different model than EdenFS,
and that has made EdenFS correctness on Windows very difficult. NFS may be
easier to get correct, though the performance is suspect. Just exploring
options here.
After this change NFS builds on Windows!
NOTE: this one is more than removing ifdefs, probably the most non trivial in the stack, please review carefully
Reviewed By: xavierd
Differential Revision: D44153443
fbshipit-source-id: b07e19f8bde1aa455eec97647ea450849133041f
Summary: Sometimes it's desirable to set the value for a by-repo tunable for all repos (e.g. to roll out a change everywhere after testing on some of the largest repos, or to quickly roll back) or to specify a default value, and right now the only two ways of doing this is either to specify the value for every single repo separately, or to have another non-by-repo tuanble that overrides it. Instead let's have two special keys: ":override:" and ":default:", so whenever we try to lookup a by-repo tunable we first check if it's set for ":override:" and use its value if so, otherwise we lookup the repo specific value, otherwise we lookup the value for ":default:".
Reviewed By: markbt, RajivTS
Differential Revision: D44750394
fbshipit-source-id: d8fe074ec700f69e52129dfa7de87b7972d11012
Summary:
This continues my slashing of fetch retries. This diff is -1 fetch retry.
DatapackStore::getTree is not a local only tree access. It may fallback to
fetching from edenAPI.
DatapackStore::getTree is used in the hg importer fallback code path for when
normal edenAPI rust binding fetching fails. It's unnessecary to be doing this
extra network fetch here, because we only want one edenAPI attempt and then to
fall back to the importer.
If I had to guess why this is here:
IIRC this method use to be a local only access to the hgcache. That seems to
line up with the logging comments around this fetch. So I think when the
datapack store gained remote fetching capabilities this one just wasn't changed.
With this change we are now down to what looks like 5 network fetch attempts.
Reviewed By: xavierd
Differential Revision: D44523824
fbshipit-source-id: f5e784660b396ad3a8f1e7effcd7551c3b126521
Summary:
intro high level transaction for cloud sync
this is for syncs that are not running with best effort option
the transaction is needed to eliminate repo changed while backing up errors
this is a follow up after https://www.internalfb.com/diff/D41871718 that hasn't resolved the errors fully
Reviewed By: markbt
Differential Revision: D44664729
fbshipit-source-id: da411cd20fb909c22ff8976dd31bd46034a28226
Summary:
In certain edge cases, during pull we are using EdenAPI to fetch nodes and feeding them into a revlog changelog (via doublewrite). This was erroring out since the RevlogCommits.add_commits expected nodes to be sorted parent-before-child. Work around by sorting the nodes.
[MSDK] Update icsp_domain_config_type_linter component on ICSP:master
```
Reviewed By: quark-zju
Differential Revision: D44734552
fbshipit-source-id: 7f00a93b93921baa827a55c7c1ec9388b74ac0f7
Summary:
There's no need to wrap TraceSubscriptionHandle in another struct or
manage its lifetime with shared_ptr. It looks that was done in one
place and then cargo-culted elsewhere.
Reviewed By: xavierd
Differential Revision: D44657251
fbshipit-source-id: d5563626c52192be3e2fa44de2f4819ca3ebe1ec
Summary:
TraceBus isn't easy to use correctly. Clarify some comments to make
the rules more explicit and unify the two constructors to allow
(noexcept) entry construction in place.
Reviewed By: xavierd
Differential Revision: D44657217
fbshipit-source-id: 1fa7c0e3c4ffb169be2b7b0cd1ffa2ea07dfeeb1
Summary: Buck1 is EOL and these tests are thus no longer necessary.
Reviewed By: chadaustin
Differential Revision: D44717980
fbshipit-source-id: bfe0d9977243c35405e1b5cc988b687369488d0c
Summary:
There is a new `Target::Pipe` that could be used to redirect logs to a file. So updating to the latest version which doesn't add any breaking changes.
Also, updating the deprecated call sites:
```
xbgs -sl 'env_logger::from_env' \
| arc linttool debugfilterpaths --take RUSTFMT \
| xargs sed -i 's/env_logger::from_env/env_logger::Builder::from_env/'
arc f
```
Reviewed By: dtolnay
Differential Revision: D44671457
fbshipit-source-id: 6734f2839731b189564d3b7ceffcd95410fe02e8
Summary: There is a typo in `_clonetotmp` that makes it crash. It should be `ident.configrepofile` instead of `ident.configerpofile`
Reviewed By: muirdm
Differential Revision: D44709074
fbshipit-source-id: 60160d68eea6b5cfafa10c5a053ade44a54724a5
Summary: This is deprecated and Thrift is warning about it, remove it.
Reviewed By: fanzeyi
Differential Revision: D44695237
fbshipit-source-id: 6f91f50574d2898d98cfb9c732c2d15e5789087a
Summary:
This experimental option allowed you to automatically "jump" past the specified revisions when annotating a file. The idea is you could skip big codemods/renames etc. that touch a lot of lines without meaningful changes.
However, no one uses it. The idea is good, but to use it you need to know a priori what revs you want to skip. A better integration might let you configure known annoying revisions, or maybe let you filter commits by their commit message.
Plus, any decent blame UX allows you to easily jump passed boring commits.
Reviewed By: zzl0
Differential Revision: D44596603
fbshipit-source-id: 3e5707bb2e14aecaf4f56388afb7eb49951b6d31
Summary:
If experimental.edenapi-blame is configured and the repo has an edenapi client, the annotate command will fetch blame data from the new blame endpoint.
Currently this doesn't support any of the whitespace options since the server only provides whitespace sensitive blame data. We can probably emulate the whitespace insensitive behaviors locally with more work.
Reviewed By: mitrandir77
Differential Revision: D44596604
fbshipit-source-id: 73d7462d0ded2d6941b704dfb0294e52f462879a
Summary:
This handler simply exposes the existing blame derived data over EdenAPI.
The main work is re-indexing the commit index so it can be a simple list instead of a VecMap.
Any error is propagated in-band in the results. I was a little surprised to learn that errors within the HandlerResult stream are ignored (i.e. not propagated to client). It seems like the stream items should have an envelope type to allow automatic Result propagation from server to client.
Reviewed By: quark-zju
Differential Revision: D44596606
fbshipit-source-id: 0ae9d501a35e3e930fd4cbe710fd3e76f493312c
Summary:
The one existing user has MPath in hand (and so does my upcoming code).
Also, defer the error context to avoid copying bytes (again) when there is no error. Note that the ParseError returned by RepoPathBuf::from_utf8 itself contains the bad bytes, but there doesn't seem to be a good way to share those bytes without losing context.
Reviewed By: mitrandir77
Differential Revision: D44596605
fbshipit-source-id: d9b1f635a7e5b90828f934707845a2efcb5c08a5
Summary:
These mirror the Mononoke BlameV2 derived data.
I tweaked the #[auto_wire] macro to support a new #[no_default] attribute. This made it possible to stick a Result<_, ServerError> into my type without having to hand implement anything else (since Result doesn't implement Default, which otherwise was required for #[auto_wire]).
Reviewed By: quark-zju
Differential Revision: D44596607
fbshipit-source-id: 9f2cc22d9ef43deefd32d4cfff67d350045ea845
Summary:
Original commit changeset: e0a130c04c80
Original Phabricator Diff: D44694398
Solving the `arc feature` problem in favor of D44695559
Reviewed By: muirdm
Differential Revision: D44713896
fbshipit-source-id: 4b870d0288e6e81577d8d398b1fb42d4e8b68dc9
Summary:
Currently WBC on each bookmark change will try to find all underived cs_ids and will request derivation for them one-by-one. This is redundand because both old path (using derivation manager) and new path (DDS) doing that discovery internally and will derive all dependency.
This diff just removing the part when WBC asks for derivation of each commits and just requests the latest underived commit, however it will still doing discovery of underived commits.
The next step would be to add an API for DDS to answer queries about derived bookmarks locations. This way we could remove discovery part completely.
Reviewed By: RajivTS
Differential Revision: D44536860
fbshipit-source-id: 3867abdf095141b89da44242a35327da02ab05f8
Summary: Recently D44515578 removed remotenames from the set of names, but internal internal tooling was depending on that.
Reviewed By: fanzeyi
Differential Revision: D44694398
fbshipit-source-id: e0a130c04c803e51a6e7f8a284b572047ffc0ee9
Summary: Now that the dependent logic is in place, this diff parses the git commits and extracts the git headers. It then passes these git headers while creating the corresponding changeset at Mononoke end using SCS's `repo_create_commit` method.
Differential Revision: D44705976
fbshipit-source-id: 464695e273908ef7642514b08158be4affc32ac2
Summary:
Introduce **h2** support into mononoke server.
Remove alpn crate that repeats the standard "select_next_proto" method
Server now offers the following protocols: hgcli, h2, http/1.1 in this order in protocol negotiation.
We will select the first protocol supported by the server which is also supported by the client.
This will add h2 support server side but won't change protocols currently used (because h2 is not offered client side right now, while http/1.1 is explicitly set).
We then later will switch **only** Eden Api Traffic to H2 by enabling the following config client side via slow rollout:
```
[edenapi]
http-version=2
```
We might need to tune some settings on proxygen side before that.
Wireproto traffic will continue to be obliged to set http/1.1 client side explicitly, which is totally fine because it needs web socket upgrade.
### At a high level, HTTP/2:
* is binary, instead of textual
* is fully multiplexed, instead of ordered and blocking
can therefore use one connection for parallelism
* uses header compression to reduce overhead
* allows servers to “push” responses proactively into client caches
Reviewed By: mzr
Differential Revision: D44540283
fbshipit-source-id: 560d80d68d5d364adee122881d12cf61ba9f1695
Summary:
Double writing is enabled for all repos for a long time, so it's safe to delete enable_writing_to_new_commit_graph.
enable_reading_from_new_commit_graph was never used since we needed more granular tunables.
Reviewed By: Croohand
Differential Revision: D44624559
fbshipit-source-id: a06a0f2079bb735e93d72d0d24bbc7ae684600a7
Summary:
Our logs are currently full of messages about the duration of
semifuture_getScmStatusV2(), which isn't very helpful. This drops the default
level to DBG3 so that we only log at DBG2 when the request takes longer than
one second.
Reviewed By: xavierd
Differential Revision: D44693481
fbshipit-source-id: 9ff8e04ab39e960050e64930a023949962360b9c
Summary: As title says. Compiling on Windows has been kind of annoying lately due to the number of warnings I've seen, which this cleans.
Reviewed By: muirdm
Differential Revision: D44690961
fbshipit-source-id: 37ae8e48e4aa181ac909b3ec90c76a1438b24489
Summary:
There is an issue where children sl processes inherit SCM_SAMPLING_FILEPATH and write to the same file as the parent process. This messed up the sampled data since the child's "command_duration", for example, could clobber the parent's.
Fix by only allowing one process to create SCM_SAMPLING_FILEPATH. This way the child (e.g. backgroundfsync hook) will not be able to output metrics.
Reviewed By: zzl0
Differential Revision: D44645799
fbshipit-source-id: 11f41343107802f9fd9ae2dc0fc7a9e1925859d8
Summary: Expose sampling-category-for-key and append-sampling-event methods from Rust to Python to replace relevant logic in ui.py. This consolidates things, and may fix the issue where corrupted keys like "foo{" were getting into the sampled data.
Reviewed By: zzl0
Differential Revision: D44645798
fbshipit-source-id: e158ed7434435e1d29222cda2fc1cf0ee345010a
Summary: This makes it easier to share the sampling config with Python. In particular I want to get down to only having a single place open the SCM_SAMPLING_FILEPATH file to make it easy to fix a couple bugs (see next diff).
Reviewed By: zzl0
Differential Revision: D44645800
fbshipit-source-id: 12a0da6d1356b6c116224a3c3552692bd5aa656a
Summary: I want to expose the sampling file to Python, so first let's move it to a non-tracing related crate.
Reviewed By: zzl0
Differential Revision: D44645801
fbshipit-source-id: 14da917acdc3529e17502d81846229e3ae8854e2
Summary:
On Windows, this should always be set to true. Unfortunately, we've rolled out
a bad EdenFS release that overwrote all of these configs and set them to false,
breaking several users.
Reviewed By: fanzeyi
Differential Revision: D44683911
fbshipit-source-id: 4d8efb3402f967b2e35fd333c858fe939307e6f0
Summary:
On Windows, the sqlite overlay is always enabled despite what the config says,
encode this in the code.
Reviewed By: MichaelCuevas
Differential Revision: D44682918
fbshipit-source-id: 751a763a59f43ebcac30b8884eaf1fbd4033d1cf
Summary:
The latest "main" contains all the patches that we were carrying on my
fork so far. Bonus point, hyperx is not a depedency anymore so we can also get
rid of this.
Reviewed By: zertosh
Differential Revision: D44682428
fbshipit-source-id: cbb1da18edce7478a5454d1a3ba70bd8767e089c
Summary: The hg stable commands fail to work on Windows. This is because the hg code calls a shell script to run the necessary hg commands. This diff adds a port of the shell script into python to let it work on Windows.
Reviewed By: skotchvail
Differential Revision: D44591662
fbshipit-source-id: 5d376702ccc648fe01f541b2d673ae39f954739b
Summary:
[Code Tour](https://github.com/microsoft/codetour) is an extension which allows you to annotate an ordered list of code pointers with descriptions, as a way of touring you through a codebase.
This diff adds a couple of tour files which are loadable by anyone who has the extension and loads this tour.
This tour walks through how we process "write requests" in the PrjFS version of Eden.
Reviewed By: chadaustin
Differential Revision: D44654254
fbshipit-source-id: e7e00e301e9445cc4f5875689a35c40bf5ad8b99
Summary:
[Code Tour](https://github.com/microsoft/codetour) is an extension which allows you to annotate an ordered list of code pointers with descriptions, as a way of touring you through a codebase.
This diff adds a couple of tour files which are loadable by anyone who has the extension and loads this tour.
This tour walks through how we process "read requests" in the PrjFS version of Eden.
Reviewed By: chadaustin
Differential Revision: D44645249
fbshipit-source-id: 5a3a620139d95f9298d9b0ce81064d49c1c297d2
Summary:
This is a slight behavior change, but I need queue events published to
ActivityBuffer so I can track the live set of fetches.
The impact will be that `eden trace hg --retroactive` will show 2/3 of
the number of events that it did. We can bump the capacity if that's
problematic.
Reviewed By: xavierd
Differential Revision: D44594520
fbshipit-source-id: 5b22169403402d729ff929b4b4bec6bb76d54737
Summary:
ActivityBuffer is enabled by default and we rely on `eden trace hg
--retroactive` working. For HgQueuedBackingStore, unconditionally
enable ActivityBuffer, and give a capacity config setting independent
from the others.
Reviewed By: xavierd
Differential Revision: D44593656
fbshipit-source-id: 284bad0d19315d250a4ab69f06439393dbeaf5d1
Summary: We only capitalize type names.
Reviewed By: xavierd
Differential Revision: D44593407
fbshipit-source-id: 9f0d841e6ff5c40d6a6c6ca3637a40fefcf5d084
Summary:
Note: this is a re-land of D44623815, which was was reverted because of a land race with D44626072
In particular this fixes `FlattenUnordered` having a nasty deadlock, but also
some other bugs:
```
# 0.3.28 - 2023-03-30
* Update to syn 2. This raises MSRV of utility crates to 1.56. (#2730, #2733)
* Fix bug in `FlattenUnordered` (#2726, #2728)
# 0.3.27 - 2023-03-11
* Add `TryFlattenUnordered` (#2577, #2590, #2606, #2607)
* Add `AbortHandle::is_aborted` (#2710)
* Add `AbortRegistration::handle` (#2712)
* Make `BiLock` strict-provenance compatible (#2716)
# 0.3.26 - 2023-01-30
* Add `Either::as_pin_mut` and `Either::as_pin_ref` (#2691)
* Add `Shared::ptr_eq` and `Shared::ptr_hash` (#2691)
* Implement `FusedStream` for `Buffered` (#2676)
* Implement `FusedStream` for all streams in `ReadyChunks` (#2693)
* Fix bug in `FuturesOrdered::push_front` (#2664)
* Remove `Fut::Output: Clone` bounds from some `Shared` methods (#2662)
* Remove `T: Debug` bounds from `Debug` implementations of `mpsc` and `oneshot` types (#2666, #2667)
# 0.3.25 - 2022-10-20
* Fix soundness issue in `join!` and `try_join!` macros (#2649)
* Implement `Clone` for `sink::Drain` (#2650)
# 0.3.24 - 2022-08-29
* Fix incorrect termination of `select_with_strategy` streams (#2635)
# 0.3.23 - 2022-08-14
* Work around MSRV increase due to a cargo bug.
```
Reviewed By: zertosh
Differential Revision: D44632588
fbshipit-source-id: bdd87cb02b3aef63a65b1f9b852579225adfedbd
Summary:
When this check was previously enabled, we saw 2 issues from users:
- The check failing due to status taking more than 5s,
- The check having false positive.
The first one can be fixed by increasing the timeout on the Thrift client, the
second one is anything but clear at first sight. Digging a bit deeper, one
issue became apparent. In:
if modified_file not in diff:
The type of `modified_file` is a `Path`, while the type of `diff` was a
`Set[str]` (the type annotation was wrong). If we manually test this, here is
what we get:
% python3
Python 3.8.6 (default, Feb 10 2023, 17:15:29)
[GCC 11.x 20221024 (Facebook) 11.2.1+] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> a = {"a/b.c", "c/d.c"}
>>> a
{'c/d.c', 'a/b.c'}
>>> Path('c/d.c') in a
False
>>> a = {Path("a/b.c"), Path("c/d.c")}
>>> Path('c/d.c') in a
True
>>>
The check would have thus failed, which would have led to false positives.
Lastly, this also fixes a potential false negative, where the number of
modified files could be the same as the files in diff when the latter includes
an added/remove file but is missing one of the modified file. Removing the
check comparing both is sufficient to avoid this issue.
Reviewed By: chadaustin
Differential Revision: D44558090
fbshipit-source-id: 0cc83a87758a5feeff78c38b210ebd91fa5d58f5
Summary:
Context
==
When the code being changed is called from a test or program using Python 3.10, deprecation warnings appear.
`DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead`
Python 3.10 is rolling out in fbcode, so it will be nice to have one less DeprecationWarning.
Implementation
==
* sed -i -E 's/(\s+)(.*)\.setDaemon\(True\)/\1\2.daemon = True/'
Reviewed By: muirdm
Differential Revision: D44620221
fbshipit-source-id: b243b5e8947c918f865efa3c46ffdc8c0e530243
Summary: These are the tests that passed after adding "$ eagerepo". I attempted to turn on eagerepo for all tests that didn't contain "$ eagerepo" or "configure modern".
Reviewed By: quark-zju
Differential Revision: D44445379
fbshipit-source-id: b283be6d5249f74e11a3205622961e457587e93b
Summary:
While reading through our uses of ActivityBuffer, I noticed an
opportunity to amortize the cost of publishing import events.
Add some comments to note that.
Reviewed By: xavierd
Differential Revision: D44643270
fbshipit-source-id: 1b68f904b86d91fab91af01a4e8b169dc381ad33
Summary:
This should reduce the number of allocations when pushing and popping
events into ActivityBuffer.
Also, remove -inl.h because our clangd does not support it.
Reviewed By: xavierd
Differential Revision: D44593366
fbshipit-source-id: 7dc0f2aa457b44bebe9471edd3c7e688d09534f5
Summary:
`std::deque` has poor performance and bucketing in GNU libstdc++, and
sometimes we just need a ring buffer without paying for folly's
LockFreeRingBuffer.
Introduce a simple RingBuffer atop `std::vector` for later use in
ActivityBuffer.
Reviewed By: xavierd
Differential Revision: D44593248
fbshipit-source-id: 13303c27b184002fc74aec9ae6e9133c70bab6bb
Summary:
based on the doc of writing extensions, post-* hooks should be setup in the `uisetup`
```
10.1. uisetup
* Changes to ui.__class__ . The ui object that will be used to run the command has not yet been created.
...
* Setup of pre-* and post-* hooks
10.2. extsetup
* Changes depending on the status of other extensions. (if extensions.find('mq'))
* Add a global option to all commands
* Extend revset
10.3. reposetup
* All hooks but pre-* and post-*
* Modify configuration variables
* Changes to repo.__class__, repo.dirstate.__class__
```
Reviewed By: sggutier
Differential Revision: D44632035
fbshipit-source-id: f37bb052a2ab225d43cf7f305b0bcccedb4add9c
Summary: When calling `fill` methods on a no-op memcache, we do all the work to serialize the data and then throw it away. Add a quick check at the start of the fill method to exit early.
Reviewed By: YousefSalama
Differential Revision: D44503801
fbshipit-source-id: fd490bb5cce70ee1bca931e55e1671faec09e894
Summary: The `Prefetch::Hint` variant indicates that prefetching is possible, but should not be acted upon unless an intermediate caching layer determines that the prefetch should be included. That means when we ask for the target, we shouldn't return a hinted target.
Reviewed By: liubov-dmitrieva
Differential Revision: D44498446
fbshipit-source-id: 56a0fb3b09de985cca9989335375daf972e43276
Summary: If we have missed in cachelib, and been asked to prefetch, then memcache fetches are actually slower than the prefetch would be, even if memcache is warm. Instead, go straight to the database and prefetch from there.
Reviewed By: mitrandir77
Differential Revision: D44498103
fbshipit-source-id: 944c26d553a770e054f38f478cf2bd475afa413f
Summary: When computing ancestors difference, we need all of the commits on the `head` branches, not just the skew binary steps. Prefetch all of them.
Reviewed By: YousefSalama
Differential Revision: D44498105
fbshipit-source-id: acf08930361f03c76685a94fff2c5a41a99089b3
Summary: Make the number of steps taken a parameter of prefetching, so that we can vary it based on prefetch type.
Reviewed By: YousefSalama
Differential Revision: D44498104
fbshipit-source-id: 9d24af68ea620fd11a18d9dbaa822fc9848fdd3c
Summary: If this config was not set, it would default to `false`, and we would not use it. Instead, make these configs optional and hardcode the default.
Reviewed By: fanzeyi
Differential Revision: D44635468
fbshipit-source-id: 2464439dc20ba90745b9783f491514c5d63e2c76
Summary: In preparation for extending this to a paramatarized test for different `InodeCatalog` types, this generalizes the object types in the already existing test
Reviewed By: chadaustin
Differential Revision: D44633728
fbshipit-source-id: d06a9ed146ad5ea65f11f0aa16bd2725aec604dc
Summary:
We made 'pr_marker' run inline instead of as hook in D43276199, but this can cause
nested event loop runtime error in `sl pr pull`. This diff makes 'pr_marker' run as a
Python hook to avoid the nested event loop.
The difference between current hook (this diff) and previous hook is that we are running
it as a Python object instead of shelling out to `sl` command.
https://github.com/facebook/sapling/issues/58
Reviewed By: quark-zju
Differential Revision: D44531268
fbshipit-source-id: 5270b7172d7b2c22a9c9e715d7cb5b42181cff53
Summary:
Some Python versions don't return the correct macOS version from
`platform.mac_ver()`, so let's use `/usr/bin/sw_vers` instead.
Reviewed By: xavierd
Differential Revision: D44594569
fbshipit-source-id: ee52f11aad76361b780845de8218b2f365f0ecfe
Summary:
Using an EdenConfig instead of a GFlags allows for rolling out config changes
automatically while GFlags do not allow this.
Reviewed By: chadaustin
Differential Revision: D44593575
fbshipit-source-id: 2b3e6858c031ba63b89787c3456542e79671d4ea
Summary:
`eden top` would previously crash if its terminal window was resized too small,
due to incorrectly specifying a negative padding width in a format specifier.
Reviewed By: xavierd
Differential Revision: D44564714
fbshipit-source-id: 936f346ce7e7f5cb0f18cdbffa269cb0fd06be91
Summary:
I'm introducing a new Rust-only command implementation, and I never
want it to fall back to Python help, even if it's run without a
required option. Therefore, handle
DisplayHelpOnMissingArgumentOrSubcommand as well as DisplayHelp.
Reviewed By: fanzeyi
Differential Revision: D44560239
fbshipit-source-id: 43c418da3c980a34a11277c82bf350822b0bc71b
