Summary:
The simplest fix so far is to erase accessed bookmarks state before switching
New cloud join will
Reviewed By: markbt
Differential Revision: D22791409
fbshipit-source-id: 9675ec03c275e42e640d3a95dd5eda2ae084b92b
Summary:
On unexpected errors like missing blobstore keys the walker will now log the preceding node (source) and an interesting step to this node (not necessarily the immediately preceding, e.g. the affected changeset).
Validate mode produces route information with interesting tracking enabled, scrub currently does not to save time+memory. Blobstore errors in scrub mode can be reproduced in validate mode when the extra context from the graph route is needed.
Reviewed By: farnz
Differential Revision: D22600962
fbshipit-source-id: 27d46303a2f2c07219950c20cc7f1f78773163e5
Summary:
Now that we can configure ACL checking on a per-repo basis, use the
`enforce_acl_check` config option as a killswitch to quickly disable ACL
enforcement, if required.
Further, remove the `acl_check` config flag that was always set to True.
As part of this change I've refactored the integration test a little and
replaced the phrase "ACL check" with "ACL enforcement", as we always check the
ACL inside of the LFS server.
Reviewed By: krallin
Differential Revision: D22764510
fbshipit-source-id: 8e09c743a9cd78d54b1423fd2a5cfc9bf7383d7a
Summary: It was once used by EdenFS, but is now dead code, no need to keep it around.
Reviewed By: singhsrb
Differential Revision: D22784582
fbshipit-source-id: d01cf5a99a010530166cabb0de55a5ea3c51c9c7
Summary: These were only used in tests, no need to keep it.
Reviewed By: chadaustin
Differential Revision: D22744353
fbshipit-source-id: 57596d641ab85f15e8c945327d7849a64aa73ef8
Summary:
Both unices and Windows needs to invalidate the cache in the same place, let's
avoid ifdef and consolidate the function name to clean things up a bit.
Reviewed By: chadaustin
Differential Revision: D22741709
fbshipit-source-id: 04060c0080eff9840abd22747ea48404fa50fd86
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary:
We're experimenting with enabling NO_OPEN support in our internal
build of the osxfuse kext. This commit includes the relevant capability bits
for the kernel interface (which are compatible with the linux FUSE
implementation) as well as adjusts our FUSE client code to detect and use
those bits on macOS.
Reviewed By: xavierd
Differential Revision: D22744378
fbshipit-source-id: 21376439a85b0b0f5a71916dd1af618d9627695e
Summary:
When Mercurial is outside a repo and you run a repo-required command,
it will try to construct the localrepo and fail. Unfortunately the dynamicconfig
loading logic treated it like a repo-creation event, and tried to load the
dynamicconfig in memory, expecting the repo to be created.
Since we do an http request during this, if we were offline it would hang
for 30 seconds every single time. Let's avoid this code path unless creating a
repo.
A separate diff will lower the 30 second timeout.
Reviewed By: kulshrax
Differential Revision: D22710370
fbshipit-source-id: e341e9230d2fdba80059ca086f0f12494a10c5d6
Summary: Make `store` the first argument for all of the EdenAPI Python methods. I've found this arrangement to be more ergonomic when working with the client later in the stack.
Reviewed By: quark-zju
Differential Revision: D22703915
fbshipit-source-id: b0ca900d969ec86ee91e8c62d281c2102860e9ef
Summary: Add a small Python wrapper class around the Rust EdenAPI client. The intention of this class is to allow for proper SIGINT handling during FFI calls as well as better handling of Python exceptions coming from EdenAPI.
Reviewed By: quark-zju
Differential Revision: D22703916
fbshipit-source-id: 33d80f616c55a607075d23dda448064115970b55
Summary: Some versions of sqlite don't allow using LIKE operation on BLOB data, so first cast it to TEXT. This test was failing on Linux runs on GitHub.
Reviewed By: krallin
Differential Revision: D22761041
fbshipit-source-id: 567d68050297c3a2ac781b252d3e9b21ea5b2201
Summary: Have a comprehensive list of OSS tests that do not pass yet.
Reviewed By: krallin
Differential Revision: D22762196
fbshipit-source-id: 19ab920c4c143179db65a6d8ee32974db16c5e3d
Summary:
we've seen flaky tests here on stress test runs, lets subtract time.time() by an epsilon of one to ensure that the comparison is only made within a second precision (time.time() worst case precision)
an example error: `AssertionError: 1595414015.5022793 not greater than or equal to 1595414015.5025299`
`AssertionError: 1595759609.439305 not greater than or equal to 1595759609.441682`
Reviewed By: chadaustin
Differential Revision: D22687496
fbshipit-source-id: 8e87148d620577e3198d2845d785a87a909cd1d3
Summary:
Update the LFS server to use the `enforce_lfs_acl_check` to enforce
ACL checks for specific repos and also reject clients with missing idents.
In the next diff, I will use the existing LFS server config's
`enforce_acl_check` flag as a killswitch.
Reviewed By: krallin
Differential Revision: D22762451
fbshipit-source-id: 61d26944127711f3503e04154e8c079ae75dc815
Summary:
Even though we never asked for a pre rename notification, ProjectedFS would
anyway send one to us, and since we're now failing on unrecognized
notification, that meant renames would always fail. Handle them and do nothing in them.
I'm not sure yet if we want to move the actual rename logic into the pre rename
callback, or keep it as is. The benefit of the pre rename code is that it can
fail the user request, while the current code will fail, but not prevent the
rename from happening.
Reviewed By: wez
Differential Revision: D22738388
fbshipit-source-id: 487e1f90b503bc59cff7315dd38d2a3039552eaf
Summary: This will make it easier to support PrjFS async programming model.
Reviewed By: chadaustin
Differential Revision: D22738408
fbshipit-source-id: 2e2e4f6f2f718b0226cf9fab66589c50b6db49db
Summary:
The pattern when using it is to catch the exception, then call the function,
which then would re-throw it and catch the subtyped exception. We can avoid a
re-throw entirely by having the function take the exception that was thrown in
the first place and dynamic cast it.
Reviewed By: chadaustin
Differential Revision: D22736772
fbshipit-source-id: 0efa3134bccf3ba8bdcd51d67e03c7ee4483a99f
Summary:
We're still reading the entire file at a time, but this paves the way to not do
that.
While the change looks big, a lot of it is just moving code around. The main
gist of it is removing EdenMount::readFile and writing the proper future
combinator in EdenDispatcher::getFileData.
Reviewed By: wez
Differential Revision: D22361748
fbshipit-source-id: 6391a29d25a4c9e61b91952c40c21ad52e728c8b
Summary:
This would enable the enumeration callback to use ProjectedFS asynchronous
completion.
Reviewed By: chadaustin
Differential Revision: D22361747
fbshipit-source-id: b2d31533ee5128e9dd3da7f91d5225331cf8e926
Summary:
We've had a long latent bug where for some reason a file that should be in the
repo isn't visible on the filesystem. Turning on logging and running `ls` shows
that EdenFS tells ProjectedFS about the file, but it's somehow not shown. The
reason being that ProjectedFS keeps track of all the removed files by adding a
tombstone for these, and will not list them until EdenFS removes this
tombstone.
Looking at the various places where we invalidate FUSE's cache, but not
ProjectedFS, only one stands out, and it has to do with files not being present
in the working copy, but added in the update destination. Adding the right
flush there appear to solve the simple repro listed below.
I'll remove all the ifdef around flushing in a later diff to avoid this issue
from re-appearing again.
Reviewed By: fanzeyi
Differential Revision: D22739916
fbshipit-source-id: 3a4fbc825cd21b36cbd2616882fd50e3d9741f63
Summary:
Let's by default not take a lease so that derived_data_tailer can make progress even if all other services are failing to derive.
One note - we don't remove the lease completely, but rather we use another lease that's separate from the lease used by other mononoke services. The motivation here is to make sure we don't derive unodes 4 times - blame, deleted_file_manifest and fastlog all want to derive unodes, and with no lease at all they would just all derive the same data a few times. Deriving unodes a few times seems undesirable, so I suggest to use a InProcessLease instead of no lease.
Reviewed By: krallin
Differential Revision: D22761222
fbshipit-source-id: 9595705d955f3bb2fe7efd649814fc74f9f45d54
Summary:
Add log sequence numbers to the scuba sample builder. This provides an ordering
over the logs made by an individual instance of Mononoke, allowing them to be
sorted.
Reviewed By: krallin
Differential Revision: D22728880
fbshipit-source-id: 854bde51c7bfc469677ad08bb738e5097cb05ad5
Summary:
We have two deficiencies to correct in here; modernise the code without changing behaviour first to make it easier to later fix them.
Deficiency 1 is that we always call the `on_put` handler; we need a mode that doesn't do that unless a blobstore returns an error, for jobs not waiting on a human.
Deficiency 2 is that we accept a write after one blobstore accepts it; there's a risk of that being the only copy if a certain set of race conditions are met
Reviewed By: StanislavGlebik
Differential Revision: D22701961
fbshipit-source-id: 0990d3229153cec403717fcd4383abcdf7a52e58
Summary:
Switching workspaces is good to have before depracating old style backups.
Otherwise it will be too confusing for users who would like to keep their work
on different hosts separate.
Reviewed By: markbt
Differential Revision: D22692393
fbshipit-source-id: abae7667ce24465e69613f3cdd4cd01471fc7704
Summary:
as in title.
Since we haven't tested it much yet I've added a note that this feature is
experimental
Reviewed By: krallin
Differential Revision: D22760648
fbshipit-source-id: 33f858b0021939dabbe1894b08bd495464ad0f63
Summary:
Move changeset_fetcher building to a separate function, because
build_skiplist_index is already rather large and I'm going to make it larger in
the next diff
Reviewed By: krallin
Differential Revision: D22760556
fbshipit-source-id: 800baba052f46ed817f011f71dd28d40e98245fe
Summary:
Currently our skiplists store a skip edge for almost all public commits. This
is problematic for a few reasons:
1) It uses more memory
2) It increases the startup time
3) It makes startup flakier. We've noticed a few times that our backend storage
return errors more often when try to download large blobs.
Let's change the way we build skiplist. Let's not index every public changeset
we have, but rather index it smarter. See comments for more details.
Reviewed By: farnz
Differential Revision: D22500300
fbshipit-source-id: 7e9c887595ba11da80233767dad4ec177d933f72
Summary:
This adds logging for data fetches that come from the thrift globfiles call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489512
fbshipit-source-id: 040cf1277205d08ea864a1f30f8d3b25ee7c4508
Summary:
This adds logging for files fetched in prefetch like was aleady added for
blob and tree fetches.
This is needed to log the fetches caused by the glob files thrift call. The
purpose of this to help debug the cause of unexpected data fetches (See
D22448048 for more motivation).
Reviewed By: genevievehelsel
Differential Revision: D22561619
fbshipit-source-id: 5ae78b99fb0c7d863d8223b93492b0d0210ddf9e
Summary:
This adds logging for data fetches that come from the thrift getSHA1 call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489514
fbshipit-source-id: eb2d82c206af857cc79439d2854d682641292db8
Summary:
This adds logging for data fetches that come from the thrift getFileInformation call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489513
fbshipit-source-id: ab6283476d05b06b9f9e37c6b4fd81c1282046ff
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary:
We have seen that some of the unexpected data fetches do not originate from
FUSE. This adds parity to the logging for data fetches that come from the thrift
interface. Adding this logging improves the overall observability of eden, and
will help us debug the cause of unexpected data fetching.
This introduces plumbing to allow logging data fetches that originate from
thrift requests.
Reviewed By: chadaustin
Differential Revision: D22448048
fbshipit-source-id: a39dde72467c4922c07c569c14fb499341d40258
Summary:
What does this do: allocate a ThriftLogHelper and ensure it lives for the
duration of its coresponding thrift request.
Why: By ensuring that the log helper's lifetime matches that of the thrift
request, we simplify adding to the logger. We wont have to think about the
lifetime of each of the members of the logger, we simply can stick them in the
log helper and let shared_ptr do its magic.
More specific why: This refactor helps make adding Thrift data fetch logging
cleaner (see following changes). We need to ensure the ObjectFetchContext
stays around for the duration of the thrift request since we pass it around
by reference. Sticking this in ThriftLogHelper avoids adding another disjoint
piece of plumbing and makes the code easier to maintain going forward.
Reviewed By: genevievehelsel
Differential Revision: D22632546
fbshipit-source-id: 1baa79419386947e52a386d89a65f032f1988622
