Summary:
This command drops all pending requests in the HgImportRequestQueue. This does not prevent future requests from being added.
It is intended to be used when there are a large number of import requests and no progress is being made. Users can use this instead of killing EdenFS.
Reviewed By: chadaustin
Differential Revision: D36026713
fbshipit-source-id: b5da2c0a5996f6022b888d9fcb086c76a88998ca
Summary: This thrift endpoint causes all ImportRequests to be dropped from the HgImportRequestQueue. In the future, this can be modified to drop requests from any BackingStore type, although each BackingStore will need to implement dropAllPendingRequestsFromQueue() beforehand.
Reviewed By: chadaustin
Differential Revision: D36026528
fbshipit-source-id: 8f98ba18af2c9ffc97b31385eaa634082c0ab2bf
Summary: Users often kill EdenFS to force all requests to be dropped. Instead, we should give them to option to forcibly drop all requests without having to kill EdenFS. This should help prevent some of the bad state we've seen users run into.
Reviewed By: chadaustin
Differential Revision: D36025396
fbshipit-source-id: 1cc6b75fef83b0e5501baee1b908f11bc3962ae8
Summary: Updated the source of Move after the diem separation
Reviewed By: vgao1996
Differential Revision: D35915627
fbshipit-source-id: 4f6155276922aa52d74b93018b183e78cbcc19d3
Summary:
This ports the default Mononoke server test configuration from Bash to
Python and allows the new test framework to run a Mononoke server in place of an
EagerRepo server.
Reviewed By: mitrandir77
Differential Revision: D35899361
fbshipit-source-id: a4f9b166b0e2b0b63b257d65dc7e22249c823290
Summary:
We want the tests to run with production configs as much as possible.
Let's allow them to load the non-remote portions of dynamicconfig. This should
also help encourage people to move configs from remote-configuration to the
static dynamicconfig.
There is a bit of an issue right now where the static dynamicconfig is fb-only.
So these tests may behave differently when run against an open source build. We
probably need to move a bunch of the fb-only configs to be public configs.
Reviewed By: quark-zju
Differential Revision: D35820978
fbshipit-source-id: 24c8fe73198244b99e0beeae22b1e93cdccb9872
Summary:
Allow for transparently using a Eden FS checkout instead of a physical
checkout.
Reviewed By: chadaustin
Differential Revision: D35695520
fbshipit-source-id: 79827815e4557ae99df05dc62c31cc5697676a38
Summary:
On windows, the results of python's `os.lstat` (used in test cases) doesn't match the values returned by `Inode->stat()` within edenfs.
This provides fake values that match the `os.stat` values.
We haven't caught this because some of the integration tests that would have (`materialize_query_test.py`) are disabled/skipped on windows.
Reviewed By: chadaustin
Differential Revision: D35686609
fbshipit-source-id: ff5aeb6932edd06f69fba46bc17cfea3f1d3600b
Summary:
I believe both of the addressed issues below are windows-only, as it's the only platform that does this weird "redirect edenfsctl to something else".
`EDENFSCTL_REAL` wasn't being preserved when the daemon was launched, because it wasn't being prefixed with `EDENFS_`. As a result, the attempt by `edenfs.exe` to run `edenfsctl` to process redirects fails at mount time.
Additionally edenfs trys to run the `edenfsctl.par` file by default. Because that doesn't work directly on windows, the daemon-launch command needs to additional tell `edenfs.exe` where to find the actual `edenfsctl` via `--edenfsctlPath`.
I don't know that this causes any problems, but it's not intendend.
Reviewed By: fanzeyi
Differential Revision: D35677466
fbshipit-source-id: 8b67d490019bb353704d565db75ae244dc0e6d9b
Summary:
This imports the configerator changes from D35613093 and adds support on Mononoke to changing the deleted manifest version.
It changes a bit from the approach I was using previously, where we had `deleted_manifest` name for V1 and `deleted_manifest2` for V2. Now it uses the same approach blame used, which is having the same name for both, but always changing the underlying implementation based on the config.
Since there was not much usage of deleted manifests directly (only on 3 files), I went with checking the config and using static types on each of them, instead of adding a wrapper type around DMv1 or v2, as that would involve creating a lot of duplicated code.
Reviewed By: kris1319
Differential Revision: D35814371
fbshipit-source-id: 4edf87296e2ea9986f9af18cb2a7e6914ea5ffbe
Summary:
The Watchman<->EdenFS interaction has a couple of issues currently especially
regarding the handling of since queries. In particular, resolving since queries
correctly across checkout operations involves collecting all the files changed
in between the commits. This also includes all the files that aren't even in
the working copy for correctness reasons.
One of the major issue with this is that this is effectively a O(repo)
operation which tools like Watchman and EdenFS were designed to avoid in the
first place. One easy way to see this is to think about what data
EdenFS/Watchman will need to load when a checkout between the null commit and
the last commit in the repository involves: all the trees in the repository
will need to be loaded and thus fetched.
In a previous diff, this got partially hacked around by short-circuiting some
of the work above by adding a threshold on the number of files Watchman will
start returning fresh instances to clients. Unfortunately, this doesn't removes
the need for Watchman to compute the changes in the first place and thus the
tree fetches.
To solve this a new streaming API is introduced which will compute the set of
files changed (including the ones from updates) between 2 journal positions.
The client will be expected to first get the current journal position and call
this API with it and its last known journal position.
Reviewed By: chadaustin
Differential Revision: D35629448
fbshipit-source-id: 844174c0b41228c81993a3e11265a5e87e7e4690
Summary:
Currently we rely on an implicit log when we obtain the service identity
permissions. That will be going away, so move to an explicit log from the
request parameters.
Reviewed By: kris1319
Differential Revision: D36127862
fbshipit-source-id: 30b47533657b3437ec5a2beb189aeb1d972bb29a
Summary:
Simplify permission checker builder by removing `always_allow_arc`. It's only
used in one place, and we can convert from `always_allow()` with `.into()`.
Reviewed By: mzr
Differential Revision: D36127448
fbshipit-source-id: c4f6caed04a925691091be4c9b2c7a61ce668912
Summary:
On the traversal algorithms, we use BoxFuture on the types in order to make things more efficient.
That is very easy to miss and try to "fix it" like I did in D35848173, so this diff is adding comments explaining this. I think it is necessary to explain it because it is not clear from context, I thought we were using BoxFuture just because generics back in the past weren't strong enough.
Reviewed By: farnz
Differential Revision: D36131287
fbshipit-source-id: 7f741098387433bd5c73f2d30b7bd3f4785701b7
Summary:
On a case insensitive mount, changing the case of a directory is done by
removing the old directory first, then by adding the new directory. Thus
asserting that the new entry must not be a file is too aggressive, let's change
it to only fire on case insensitive mounts.
Reviewed By: genevievehelsel
Differential Revision: D36085219
fbshipit-source-id: ca5767cef48245b8bc1fdb314fe4c10222638a30
Summary:
Most of these were simply calling `.get` on the resulting future, which is also
exactly what TestMount::getInode does. Thus let's switch to the TestMount
version.
Reviewed By: chadaustin
Differential Revision: D35906051
fbshipit-source-id: ada131ad650a90b04f208bc28109025507e74073
Summary:
Now that we have a similar version that doesn't force inodes to be loaded,
let's make sure that version is recommended.
Reviewed By: genevievehelsel
Differential Revision: D35906052
fbshipit-source-id: 1923245bf03b53a77105234b618afe5bfc104701
Summary:
Fastreplay has been gone for a while, but it's still mentioned in
integration tests. Remove the buck targets and the bash function that call it.
Reviewed By: mzr
Differential Revision: D36128480
fbshipit-source-id: aea1f1c9cb43f2c15d24cf314500fdf658553c77
Summary: Adds `get_target_dir`, which will be used later to selecting the proper target directory and creating its ancestors if they don't exist in the Rust clone command. This behavior is intended to replace the similar target directory selection that fbclone does.
Differential Revision: D35781015
fbshipit-source-id: 57b9735d241ce8b55b504b36a14952fed32c403b
Summary:
We often get reports complaining `treeoverlay.db-WAL` is too big on user's machine.
The issue here is that, we never get a chance to let SQLite run wal checkpoint because we never close the connection to SQLite database.
In which case, we can run the maintenance job ourselves to keep the journal file small. This diff adds a periodical task in EdenServer to give tree overlay a chance to run `wal_checkpoint`.
Reviewed By: xavierd
Differential Revision: D35321408
fbshipit-source-id: ac900be288d0919785876b93a9bde2166871c56a
Summary:
In my next diff, I am adding a maintenance task for SQLite to run wal checkpoints periodically. However, prepared select statements will prevent wal checkpoint from succeeding if they are not reset.
This diff introduces a `SqliteStatement::Guard` to reset the prepared statement correctly, and this has proven to unblock wal checkpoint for overlays.
Reviewed By: xavierd
Differential Revision: D35624964
fbshipit-source-id: 9f97d166f62131ae5cff7e29afe9e49d6398e7ef
Summary: It has been a year, we can remove this now.
Reviewed By: genevievehelsel
Differential Revision: D35321702
fbshipit-source-id: d078cbd1c8dfb74f573c395c3ea891294de20d72
Summary:
On Windows, dbghelp.dll does not look at the executable path for debug symbol files by default. This has caused us to not be able to see stack traces when iterating / debugging EdenFS on Windows.
This diff fixes it by manually adding the binary directory to the search list, and explicitly load it when reporting a crash.
This will also trigger loading from the debug symbol server **if it is configured in the environment variable**.
Reviewed By: xavierd
Differential Revision: D35623865
fbshipit-source-id: 72822bf38af93632fbd3a20757863d6bea4b078a
Summary:
Commands calculates total size of profiles (all or provided list) by walking through the repo, matching paths to sparse profile config and using fsnodes info to get the size of files/directories.
Since in the directory with the sparse profile configs we have other files as well (validators configs, readme), I've decided to skip profile config if it is in bad format and just log it.
It took almost 27 minutes to process the request. In the next diffs I'm going to implement method to calculate delta size of profiles between 2 commits and store the all profile sizes for public commits.
Reviewed By: mitrandir77
Differential Revision: D35293351
fbshipit-source-id: 011f2e6144e8c35da226285296c9faae48a53e39
Summary:
This method rewrite the ```removeRecursively``` to better fit buck2 use case, where
1. if a file is not loaded, fast remove it
2. remove all children in a batch way
(Note: this ignores all push blocking failures!)
Reviewed By: chadaustin
Differential Revision: D33903070
fbshipit-source-id: 0bae42d07229c6539814cd82d848c2f6fd9abb92
Summary:
Migrate off the use of linkrev-based history so this becomes more compatible
with git. The old code path can still be turned on using a config flag if
the new code path causes issues.
Differential Revision: D35201371
fbshipit-source-id: dec6c8a54f68b617d99be4568de9bc91fb8cd307
Summary:
Previously, D34725120 (ab08faf821) and D34790206 (25bcb70342) implement some features on line matching.
However, other special patterns like `(?)` and `(feature !)` are also common
in tests, which are hard to support on top of D34725120 (ab08faf821) and D34790206 (25bcb70342)'s work.
This diff adds MultiLineMatcher and ExpectLine which implements the `(?)` and
`(feature !)` and other features in D34725120 (ab08faf821) and D34790206 (25bcb70342) all together.
With this change, now passing tests increase from ~180 to ~210.
Differential Revision: D34835695
fbshipit-source-id: 446e75dbaef4cbdb2093dac5134379d0dfd5c402
Summary:
Our systemd code is unused, and is causing a bunch of tests to fail. In the
case where systemd is needed again in the future, its shape will be
significantly different than what we have today, so there even less need to
keep this version around.
Reviewed By: fanzeyi
Differential Revision: D35829902
fbshipit-source-id: 6013f5fe7c3debdcfbb9929d2748819a96403fe7
Summary: I'll wait with removing it from thrift spec once all binaries are updated but to make that diff as simple as possible let's stop using it in the code now.
Reviewed By: farnz
Differential Revision: D35934780
fbshipit-source-id: 635ad24a86430a0d46d59fab3f75643b595f1a32
Summary: This is the diff all the other diffs were leading towards.
Reviewed By: farnz
Differential Revision: D35934782
fbshipit-source-id: 5d6f2b3d8d516ad27c4e719635243d03b8641748
Summary:
This diff adds a new way to specify the commits to be included in the
server-side semgented changelog. This is meant to replace the previous means of
configuration. Added features:
* it's now possible to tail multiple bookmarks
* it's now possible to have different set of commits tailed by the live
serving jobs vs offline tailer/seeder jobs.
Configerator diff: D35905607
Reviewed By: farnz
Differential Revision: D35905650
fbshipit-source-id: f0b7002d9c0f6bbc53dc776ed5564cde638be5ed
Summary:
We want to be able to return different heads for segmented changelog for
backgorund jobs vs live server. The seedheads function needs to be aware of
that distinction.
For now all SC abstraction generated by repo factor will be for live jobs
because they're used there (the tailer and repo imported construct the SC
explicitly).
Reviewed By: farnz
Differential Revision: D35934779
fbshipit-source-id: 732027d1db5e5f31237879dded3eed6d651edbe9
Summary: I want SeedHead to be a part of config - not something we generate in SC code. This diff will make more sense once you read the next ones.
Reviewed By: farnz
Differential Revision: D35934783
fbshipit-source-id: 8a68052e299401760a95d0f4e8ebf633767cb471
Summary:
I'm about to change the data model to allow the user to ask segmented chagelog
all bookmarks except some.
Reviewed By: farnz
Differential Revision: D35934781
fbshipit-source-id: cac7829adba4dbf509006064bf88f7e1147ca390
Summary:
change the tooltip during a checkout operation. Also change it back when the checkout finishes.
The debug menu can also enable this tooltip by changing the icon color to orange. Changing the color to green or white will change the tooltip back
A similar tooltip is also implemented for when EdenFS detects an unhealthy mount
Reviewed By: kmancini
Differential Revision: D35488235
fbshipit-source-id: a33ebfd15cbfd0d46eea85f32be4ddd1bb09f357
Summary:
On Windows, EdenFS should signal that it's working on a checkout operation by changing the color of the E-Menu.
We also add a debug menu option for changing the E-Menu color
Reviewed By: chadaustin
Differential Revision: D35487175
fbshipit-source-id: fae587a223e9ce910e5558b36008a8c97b9dd3ea
Summary:
I'm about to add another stream, and this code will be duplicated a third time,
let's refactor it.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D35767107
fbshipit-source-id: 44975b72a705bd4d2d749b8caacb56743153a148
Summary: before, if we exceeded 24 hours of uptime the E-Menu would roll over to 0 hours of uptime. We can avoid this by adding days to the uptime print out. We don't print out the number of days if the uptime is less than 1 day.
Reviewed By: chadaustin
Differential Revision: D35954540
fbshipit-source-id: 9285433a827f8494b840e9d551cb6504f94417b7
Summary:
Hyper used to have stricter trait bounds for stream bodies, however
they were relaxed in https://github.com/hyperium/hyper/pull/2187.
Remove the code to spawn a task that simply connected streams up now that hyper
is more relaxed. As the spawned task was just polling for data, it shouldn't
add any extra computational work onto the task used for a request.
This change is basically identical to D27963458 (e5cc9a1f3d).
Reviewed By: mitrandir77
Differential Revision: D35931360
fbshipit-source-id: 065fecc77b4ac1218c2be21e0a3639103429c5bc
Summary:
D35892336 (9b18b0d3be) fixed a bug in the LFS server that meant it would consider
an upload successful, even if writing to internal stores failed.
Add a test that verifies writing to a failing blobstore results in upload
failure.
Reviewed By: mitrandir77
Differential Revision: D35928308
fbshipit-source-id: 296ccdddb6f4b86f5fb778f97185a8a6a0ea9d17
Summary:
The upload code can be updated to use the `?` operator, which makes
the code a little simpler.
Reviewed By: mitrandir77
Differential Revision: D35936319
fbshipit-source-id: 054f1e40798a38ff699f56dacfff055eb8c60263
Summary: I'm about to touch this function so I think it's better to have it tested.
Reviewed By: quark-zju, farnz
Differential Revision: D35897942
fbshipit-source-id: 81a141d06a4c326a37cac04a8e4cfb0c97365cf1
Summary: We can use specific types so let's use them.
Reviewed By: farnz
Differential Revision: D35897795
fbshipit-source-id: f966191d05ac7b0228b14539d88c825103b3f38f
