Summary: Right now this is not very useful. Let's make it more useful.
Reviewed By: DurhamG
Differential Revision: D28281653
fbshipit-source-id: ef3d7acb61522549cca397048c841d1afb089b9b
Summary:
These are undermaintained, and need an update for oncall support. Start by moving to CXX, which makes maintenance easier.
In the process, I've fixed a couple of oddities in the API that were either due to the age of the code, or due to misunderstandings propagating through bindgen that CXX blocks, and fixed up the users of those APIs.
Reviewed By: dtolnay
Differential Revision: D28264737
fbshipit-source-id: d18c3fc5bfce280bd69ea2a5205242607ef23f28
Summary:
Because cachelib is not initialised at this point, it returns `None` unconditionally.
I'm refactoring the cachelib bindings so that this returns an error - take it out completely for now, leaving room to add it back in if caching is useful here
Reviewed By: sfilipco
Differential Revision: D28286986
fbshipit-source-id: cd9f43425a9ae8f0eef6fd32b8cd0615db9af5f6
Summary: This wants to use Scuba so it needs this.
Reviewed By: StanislavGlebik
Differential Revision: D28282511
fbshipit-source-id: 6d3a2b6316084f7e16f5a2f92cfae1d101a9c2d3
Summary:
This makes it easier to see what builder functions were registered:
% EDENSCM_LOG=edenapi=debug lhg log -r .
May 06 16:40:29.355 DEBUG edenapi::builder: registered eagerepo::api::edenapi_from_config to edenapi Builder
Reviewed By: DurhamG
Differential Revision: D28271366
fbshipit-source-id: f6c7c3aa9f29c3e47c2449e3d5fc16474aa338b0
Summary:
Adding support for the stables template keyword in stablerev extension.
This keyword calls out to a script specified in the config stablerev.stables_cmd to get a list of stable aliases for a given revision.
Reviewed By: quark-zju
Differential Revision: D28204529
fbshipit-source-id: 3c5b21846ce6f686afddd00d3326a54b85be87dd
Summary:
The server1 was not used after D27629318 (ba7e1c6952) while the test intentionally wants to
exercise graph isomorphism. So let's revive server1 in the test.
Reviewed By: andll
Differential Revision: D28269926
fbshipit-source-id: 0a04031415f559f8a6eb81f1e2f2530329a2a3bc
Summary:
In blobstore factory we can end up with duplicate layers of wrapper blobstores like ReadOnlyBlobstore.
For the multiplex, its inner stores get throttling, readonly etc wrappers, and it itself only writes to its queue if an inner store succeeds, which it can't when inner store has ReadOnlyBlobstore wrapper.
Differential Revision: D28250832
fbshipit-source-id: 5a3f85584b9cce17ca7ce4b83cdb2117644850db
Summary: Add support for logging the inner parts of a multiplex blobstore stack. This helps understand what wrappers have been applied.
Differential Revision: D28230927
fbshipit-source-id: 873ee30ec00fdc2dfc79b47e5831231c51e2ce0d
Summary:
Fixing the bulkops fetch size to MAX_FETCH_STEP means we can use the chunk size option to control how many changesets are walked together without affecting query performance.
This will allow more accurate first referencing commit time lookup for files and manifests, as all commits in a chunk could possibly discover them, with smaller chunks the discovered mtime becomes less approximate, at the possible cost of some walk rate performance if one was to run with very small chunk sizes.
Differential Revision: D28120030
fbshipit-source-id: 0010d0672288c6cc4e19f5e51fd8b543a087a74a
Summary: Knowing the numeric changeset id is useful in next diff when chunking in walker is loading from bulkops in large chunks, but then walking commits in smaller chunks.
Differential Revision: D28127581
fbshipit-source-id: c5b3e6c2a94e33833d701540428e1ff4f8898225
Summary: Found this useful while debugging the pack sampling
Differential Revision: D28118243
fbshipit-source-id: d94b0b87125a9863f56f72029c484909a3696329
Summary:
We were only incrementing this on `readline`, which resulted in very low
numbers. While in there, I also removed `self._totalbytes` as that was unused.
Reviewed By: johansglock
Differential Revision: D28260141
fbshipit-source-id: 6d9008f9342adaf75eecc8ed8c872f64212cd1f7
Summary:
In an NFS mount, the InodeNumber are sent to the client as the unique
identifier for a file. For caching purposes, the client will issue a GETATTR
call on that InodeNumber prior to opening it, to see if the file changed and
thus whether its cache needs to be invalidated.
In EdenFS, the checkout process does unfortunately replace file inodes
entirely, causing new InodeNumber to be created, and thus after an update, an
NFS client would not realize that the content changed, and would thus return
the old content to the application. To solve this, we could approach it in 2
different ways:
- Build a different kind of handle to hand over to the NFS client
- Keep InodeNumber constant during checkout.
After trying the first option, it became clear that this would effectively need
to duplicate a lot of functionality from the InodeMap, but with added memory
consumption. This diff attempts to do the second one.
Reviewed By: chadaustin
Differential Revision: D28132721
fbshipit-source-id: 94d470e33174bb9ffd7db00e1b37924096aac8e9
Summary:
Add a subtree to exercise treemanifest logic. Blobs in EagerRepo are verified
so we need to disable flatcompat.
Reviewed By: DurhamG
Differential Revision: D28006550
fbshipit-source-id: ac7157a9c01ed99f703601613fb3cf06add69003
Summary: This makes it easier to use it in tests.
Reviewed By: DurhamG
Differential Revision: D28006549
fbshipit-source-id: 90e29b220453a3d7a260d0a62d697d64363d9a6c
Summary:
With remotefilelog force enabled, it's now possible to read the file content after
clone. Add a test for it.
Reviewed By: DurhamG
Differential Revision: D28006547
fbshipit-source-id: 5be93e162f352b1264a6c52852c2230726652f9d
Summary:
This makes it easier to get rid of revlog stores.
`debugindexdot` is no longer supported since it reads revlogs.
Two tests use flat manifest bundles. They are no longer supported
due to remotefilelog today has some assumptions that treemanifest
extension is also being used.
Reviewed By: DurhamG
Differential Revision: D27971126
fbshipit-source-id: fdb992a8d72bbcf562b5cb95b3f29051dd1c9464
Summary:
Disabling treemanifest is a tech debt that causes problems, especially when
enabling remotefilelog.
Reviewed By: andll
Differential Revision: D27971120
fbshipit-source-id: 1a50acc23564c2d6bad79a2e99469850b5a7d1f9
Summary:
This makes it easier to filter logs related to remote fetching.
The `DEBUG dag::protocol: resolve ids [0] remotely` means the lazy hash resolution is working.
Reviewed By: kulshrax
Differential Revision: D27971117
fbshipit-source-id: f2492204c70d793997d0c3865e500bbad56b1953
Summary:
Write commit to master group. This provides proper "CloneData" and allows us to
actually test lazy commit hash backend (since only commits in the master group
can have lazy hashes).
Reviewed By: DurhamG
Differential Revision: D27971123
fbshipit-source-id: 4e19486007ddc89de7468be65445559f34d796f5
Summary:
Add clone endpoint so we can clone from an eager test repo.
Note: the master group is empty and "cloneata" does not quite work yet due to
EagerRepo not writing to the master group. It will be fixed later.
Reviewed By: DurhamG
Differential Revision: D27971121
fbshipit-source-id: 0cc35136c6987673c2c4fbbd892c344c3586fcb7
Summary: This update makes it so that we don't log versions to scuba from tests.
Reviewed By: krallin
Differential Revision: D27449808
fbshipit-source-id: 9c79e83fbfdf3d9a02c2cfc8b6a8255edb4241fe
Summary:
This is going to enable the background update in SegmentedChangelog to log
entries to Scuba.
The scuba sample builder is not fundamentally different than other elements of
the environment. It is used slightly differently to, for example, Logger,
because it has to cloned in all places that want to log rows but otherwise it
has the same characteristics.
Reviewed By: krallin
Differential Revision: D28210008
fbshipit-source-id: 68468868d13f29dddf21095bd7526cb4ff690786
Summary:
This is where async requests are logged to be processed, and from where they
are polled later.
It will acquire more functionality when the actual request processing business
logic is implemented.
Reviewed By: StanislavGlebik
Differential Revision: D28092910
fbshipit-source-id: 00e45229aa2db73fa0ae5a1cf99b8f2d3a162006
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
Log the sizing metadata about keys that scrub has seen to the pack info logs.
This uses the sampling blobstore to see all blobstore gets and captures info from them.
Also updates relatedness_key fieldname to mtime as that way its less easily confused with similarity_key
Differential Revision: D28115620
fbshipit-source-id: 666a444c2d91b0ca5bb225cea971f9b183e6a48d
Summary:
Pass BlobstoreGetData to the sampler so that it has a chance to sample the BlobstoreMetadata as well as the BlobstoreBytes.
This is used in the next diff for sampling the sizing information.
Reviewed By: markbt
Differential Revision: D28115619
fbshipit-source-id: 7a79d482c9ba1ed8b08afab5f1c1b8fe7c4f257a
Summary: When reading from packblob we'd like to see metadata about sizes so that can log it for the packer later in this stack.
Reviewed By: markbt
Differential Revision: D28101971
fbshipit-source-id: 96dd0d5497c2bb5c27231709dbd19d00168e1a77
Summary:
This diff prepares some macros to be usable by external crates. Next diff uses
them in `megarepo_api:async_requests`.
The way it makes macros usable externally is by:
1) using abs paths for std types outside of prelude
2) importing necessary traits in methods which need them
3) re-exporting important `mononoke_types`-defined types and third-party
dependencies via `mononoke_types::private` and using `$crate::private` in macro
to access them.
This way someone needs only to depend on `mononoke_types` and only to import
the macro, not any other re-exported definitions.
Reviewed By: krallin
Differential Revision: D28033196
fbshipit-source-id: 3f5fa34388a4f6466969d06c5739c5282f007e68
Summary:
The trees endpoint is another example where we try to send errors to the
client. As it was done previously we would fail to log any errors on the
server side. This diff corrects that by using custom_cbor_stream.
Reviewed By: kulshrax
Differential Revision: D28111102
fbshipit-source-id: 468095d024647f472b8ad9a9e17ca8364605ff98
Summary:
Some EdenapiEndpoints serialize their error and send them to the client. This
robs the gotham framework from the change of seeing those errors. This custom
stream takes care of counting those errors and reporting them to gotham for
logging.
There is a slight worry in the back of my head that if we start getting errors,
CPU utilization will increase because we construct so many extra strings.
Reviewed By: kulshrax
Differential Revision: D28109314
fbshipit-source-id: ab39a05227e0f8be9205d6036d32523cb1e45ca4
Summary:
The walker does about 50% of our traffic, but it also has fairly predictable
access patterns. It seems unlikely that we really benefit from logging all
scrub activity with the same precision as we do other traffic.
So, let's sample it. This should make a lot of space in our Scuba table and
make us more resilient to sudden changes in activity.
Reviewed By: StanislavGlebik
Differential Revision: D28254057
fbshipit-source-id: da748a565954c31c2d9e087b7b07747a435427bf
Summary:
Add `--blame-v2` to `mononoke_admin blame compute`. This can be used to compute
blames in the new format and validate that they are correct.
Reviewed By: mitrandir77
Differential Revision: D28183160
fbshipit-source-id: f698a77c109bfce05aeb66cd405c6f20bf158801
Summary:
Add types and an implementation for blame_v2. This differs from the existing
blame definition as follows:
* Changeset IDs and paths are always stored as indexes into a mapping.
* The Changeset ID index is stable for the p1-parent chain of commits. Although only
the changeset IDs that are used in the blame are present, the index for a particular
changeset ID doesn't change, unless it is added in a merged-in parent.
* The notion of rejection is merged into the blame definition. This makes it more
explicit that rejected parents are simply ignored by the blame algorithm.
Some bugs in the original blame algorithm are also fixed:
* Initially empty files don't cause a zero-length blame range that can never be deleted.
* Merges correctly recalculate origin offsets.
This blame implementation is not used yet. It will be introduced in subsequent commits.
Reviewed By: StanislavGlebik
Differential Revision: D28183161
fbshipit-source-id: 3604ff7ff393220e13a69d0ad51247d7bfd80926
Summary:
Add thrift definitions for blame_v2. This uses a map from integer id to
changeset id to avoid repeating 32-byte hashes in files with many blame ranges,
and to provide a stable integer index for changesets that can be used to
provide a more human-readable id for each blamed changeset.
Reviewed By: StanislavGlebik
Differential Revision: D28183163
fbshipit-source-id: a6ae01d92c3d3c022a1d412f2ba0046f08c2d9f1
Summary:
Add debug output to rage to make sure we have the relevant information in case
we need to debug issues with schemes.
Reviewed By: quark-zju
Differential Revision: D28222910
fbshipit-source-id: 9499c736d61b2c0e4568e05a3afc0ae9730acedf
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799
Summary:
This makes it easier to use `--keep` to investigate tests by using
`--configfile`.
Reviewed By: kulshrax
Differential Revision: D27971122
fbshipit-source-id: 8adcbeab825155858499c24ca74c2979049adeda
Summary:
Rename `new_with_containers` to `new`. `new` the standard name for
what the fuction does.
Reviewed By: krallin
Differential Revision: D28210005
fbshipit-source-id: 9ad0dad5e8afd89d7538fd32d93de10837e26f9d
Summary:
Refactoring the construction logic for CoreContext.
`test_mock_class` is used in only one place. That usage can be replaced with
`test_mock_session`.
Reviewed By: krallin
Differential Revision: D28210007
fbshipit-source-id: c81ca31cb7255ef640fe641e9b38f239cfb69267
Summary:
Refactoring CoreContext construction to express purpose. We will have
a constructor for request processing, for background processing.
Bulk processing is another category that has it's own constructor
already. Renaming it to make it more prominent.
Reviewed By: krallin
Differential Revision: D28210006
fbshipit-source-id: 2bb74d97e2f3588aa539e58c3d6dd6842f898121
Summary:
We have a linker issue on Windows when building EdenFS with CMake:
```
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpSetStatusCallback referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpOpen referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpCloseHandle referenced in function winhttp_close_connection
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpConnect referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpReadData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpWriteData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpQueryOption referenced in function certificate_check
```
This fixes that.
Reviewed By: xavierd
Differential Revision: D28230163
fbshipit-source-id: f74e42ee30ec8f3b81c1f80b7cf63a21ea97732c
Summary: This should fix the test failure introduced in D28203778 (9bee308885) where fsck tests were not skipped on Windows.
Reviewed By: genevievehelsel
Differential Revision: D28227230
fbshipit-source-id: 97ef5717fbda567b48f02a60e1906e374fdc6766
Summary: The syntax is not supported by Python 2.
Reviewed By: DurhamG
Differential Revision: D28233280
fbshipit-source-id: 9f882827b1357cb339e60180acadb38842c3cf8d
Summary: The syntax is not supported by Python 2.
Reviewed By: DurhamG
Differential Revision: D28232995
fbshipit-source-id: 62058751b4f00b78a2bd56908100a7bb7a3adfde
Summary: Upstream crate has landed my PR for zstd 1.4.9 support and made a release, so can remove this patch now.
Reviewed By: ikostia
Differential Revision: D28221163
fbshipit-source-id: b95a6bee4f0c8d11f495dc17b2737c9ac9142b36
Summary: Windows path like `eagerepo:///C:\foo\bar` needs special handling.
Reviewed By: kulshrax
Differential Revision: D27971119
fbshipit-source-id: 9d4b87782eca2734b708565f0ee22a7495253cff
Summary: We needed only for the initial rollout, we can delete it now
Reviewed By: krallin
Differential Revision: D28226715
fbshipit-source-id: 20349e9775b71292604a4054162f026087511db8
Summary: `hg gc` does not do anything anymore, so in order to reduce confusion, let's just print a message that says it is no longer supported and provide a manual remediation.
Reviewed By: xavierd
Differential Revision: D28164614
fbshipit-source-id: 7ed2392cdb0091cd604a15b4c2382632706981f2
Summary:
Quicksand is about 60% of our logging, but we don't really need to log
everything it does because the workload is pretty much the same across all
hosts.
Reviewed By: HarveyHunt
Differential Revision: D28225416
fbshipit-source-id: 68c0f68c4da73e0f7543754c917d7efa4bbf374e
Summary: It was requesting a slice but always converted it to an Iterator anyway. Receiving an iterator saves constructing a temporary Vec both here and later in the stack
Reviewed By: krallin
Differential Revision: D28127582
fbshipit-source-id: 625c1f17f1ded973f8b2aa13566928af0df83aec
Summary:
This avoids issues where the tree is stored without p1, p2. It is similar to
what we do for public commits (in createtreepackpart):
if sendtrees == shallowbundle.AllTrees or ctx.phase() != phases.public:
...
Note: the trees API actually provides p1, p2, but p1, p2 are dropped when
writing to the current data store implementation.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200388
fbshipit-source-id: e1fe93d8ae8576e691077d6db432d55f7b9d498d
Summary: Add a way to fetch tree content without going through store.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200387
fbshipit-source-id: 8f5b2214aafba39c7674f0f6b27af0c985f0ea72
Summary:
The `trees` API is coupled with a store. We're going to add another API that is
not coupled with a store so let's rename `trees` to `storetrees`.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200389
fbshipit-source-id: 826116f0b461873b2f5df07e7fd35e6d1018f929
Summary:
This output is non-determenistic, and it does not seem to be important in this test
We could replace HashMap with BTreeMap to make it deterministic, as an alternative, but it is probably not justified for this test
Reviewed By: quark-zju
Differential Revision: D28204050
fbshipit-source-id: 50000671520e3bbf41849dc53c420ccab496dca0
Summary: This import is not used anywhere (since D28007850 (bb3d207e33)) and could be removed (currently it throws a warning whenever someone attempts to do a cargo check)
Reviewed By: sandeepkumarpani888
Differential Revision: D28220436
fbshipit-source-id: a7fc0fe84d485d37fe86476011a853df57a6bc03
Summary: the option has been deprecated and is not used anywhere
Reviewed By: krallin
Differential Revision: D28191314
fbshipit-source-id: f5f092b93a9644c8249628520d8d707b60854aac
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
Context: https://fb.workplace.com/groups/edenfswindows/permalink/828914994691047/
Even with D27872753 it doesn't really make sense to have `eden fsck` running on Windows since it requires EdenFS repository to **be unmounted**.
This diff changes it to generate a warning to redirect users to run `eden doctor` instead (which is likely what they need).
Reviewed By: kmancini
Differential Revision: D28203778
fbshipit-source-id: ae105678876903bcf6514252bf07189775f9b187
Summary:
This will consume `CloneData` from EdenApi and write to the graph.
Note `CloneData<Vertex>` and `CloneData<HgId>` has the same mincode
serialization result so there is no need to do extra type conversion.
This can be used like:
In [1]: v=api.clonedata('fbsource');
In [6]: d=bindings.dag.commits.openhybrid(None, '/tmp/seg', '/tmp/msg', repo
...: .edenapi, repo.name, lazyhash=True)
In [7]: d.importclonedata(v)
Reviewed By: kulshrax
Differential Revision: D27971125
fbshipit-source-id: 4d420c6ff0495dc184e7c9618b866a69f0a00002
Summary:
Expose NameDag's `import_clone_data` API so this can be then exposed via
`pydag`.
Reviewed By: kulshrax
Differential Revision: D27971118
fbshipit-source-id: c9d869ffbbc8ba5a7a6ae98d17a2b7ea713bc675
Summary: The `CloneData` is currently only exposed in Rust. Expose it in Python too.
Reviewed By: kulshrax
Differential Revision: D27971124
fbshipit-source-id: 1a9c63274b6e2d78a176869b6810acbc191ba314
Summary: We skip it in other places but missed this one. Skip it too.
Reviewed By: kulshrax
Differential Revision: D27957853
fbshipit-source-id: 429d25e8b692218c9bae6c10ad76d08495a4bc66
Summary: If ui.ssh is "false", then ssh cannot be used at all. Force using edenapi.
Reviewed By: kulshrax
Differential Revision: D27957312
fbshipit-source-id: 9860344779e6a6bab557d3f953ee38e40fadb78b
Summary: Make it easier to check whether APIs in EagerRepo is called or not.
Reviewed By: andll
Differential Revision: D27955426
fbshipit-source-id: 27ca505c63596368cff98642de010b5b5717454c
Summary: It has been enabled for a long time in our production config.
Reviewed By: kulshrax
Differential Revision: D27953636
fbshipit-source-id: 428f6e8a3e7eae6d44c61970624a75d7d1ab3e36
Summary: It has been enabled for a long time in our production config.
Reviewed By: kulshrax
Differential Revision: D27953635
fbshipit-source-id: a351342fbc8cffccd16967bd0e7032ac3e4f35cf
Summary:
Add "getbundle" alternative "commitgraph" for pulling from a EagerRepo.
This avoids tech-debt like bundle2 or linkrev. It depends on a lazy
(text) changelog backend.
Reviewed By: kulshrax
Differential Revision: D27951620
fbshipit-source-id: f21119d37da6505e68c6c5f3b33b9bd1f65e4d9a
Summary: It's not an error case. It just means all nodes are unknown to the repo.
Reviewed By: kulshrax
Differential Revision: D27951619
fbshipit-source-id: 672932af3a54ffa5adfa5cccbfff7edbf4f24022
Summary: It's okay to migrate to any backend if the repo is empty.
Reviewed By: kulshrax
Differential Revision: D27951626
fbshipit-source-id: 27c00c853bf73fa3c696d74f3c05eb620f35db0e
Summary:
Add "unbundle" alternative "addblobs" for pushing to a EagerRepo.
This avoids tech-debt like bundle2 and linkrev.
Reviewed By: kulshrax
Differential Revision: D27951628
fbshipit-source-id: 3315e0653ee12928993e4e9325fbe8e2c369307b
Summary:
Now EdenApi trait is moved to a separate crate, we can inline the EdenApi
backed by EagerRepo without using dynamic registration functions.
Reviewed By: andll
Differential Revision: D28006553
fbshipit-source-id: 427513da94db228745b1a7e90af0e62296056128
Summary: So that we don't duplicate the URL handling in Python.
Reviewed By: andll
Differential Revision: D28006552
fbshipit-source-id: 2efda622fe86787373fa4ec5978537588defec28
Summary:
`peer` is the interface in hg to support push/pull. Implement it for EagerRepo.
Note: `getbundle` and `unbundle` are not implemented yet. Push / Pull are not
working yet. They will be made work later.
Reviewed By: kulshrax
Differential Revision: D27951621
fbshipit-source-id: 71f9c26713a532a0712460fa2aa34125b2b67e35
Summary:
Our BigSur mactest machines have python3 defaulting to some internal
fbprojects python install. This is breaking our OSS builds. Let's change
pick_python to avoid that install.
Note, sys.stdout was changed to print because during my manual testing on Mac,
sys.stdout did not actually print the value, despite the flush. Using print()
did work.
Reviewed By: quark-zju
Differential Revision: D28101632
fbshipit-source-id: 2907d644b2baa8a53a9a2d7da176d33cd83dfbd5
Summary: Manual scrub success file can be very large and have fairly high IO rate. When checkpointing keys to a file its not really needed, so make it optional.
Reviewed By: farnz
Differential Revision: D28199084
fbshipit-source-id: 83d946f7ab8dc6f5f17f94b6a1c3818d9af7b0b0
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary:
Right now, if your batch size is 1K we prefetch 1K unodes in parallel. This
tends to result in e.g. timeouts as various things get starved for CPU as
a result.
Let's stop doing that.
Reviewed By: StanislavGlebik
Differential Revision: D28183685
fbshipit-source-id: d8353ae8e36921a485b982a1043b81f443258098
Summary:
Our batch sizes are a bit crazy here and causing the backfiller to OOM if there
actually is that many commits to derive. Lower them, a lot.
Reviewed By: StanislavGlebik
Differential Revision: D28183686
fbshipit-source-id: 54b546c4507f65c34a264df283516b5d62408a66
Summary: The packer was adding repo prefix as part of the pack key, which would mean that same content for different repos had different binary form. This change fixes the prefix.
Reviewed By: farnz
Differential Revision: D28119422
fbshipit-source-id: 338e17885abd8cfca12d5bb399244039dbf22e63
Summary:
Our packs no longer require strict order on write to be readable, so let's hash the list of sorted keys as the pack identity. i.e. pack will have same identity if if contains same keys, regardless of order.
This makes the packer easier to work with in general
Reviewed By: farnz
Differential Revision: D28119423
fbshipit-source-id: 71c8d4ed351bde63065047c706311a869b2beeda
Summary:
Wireproto session multiplexes wireproto commands. Counting them individually is most likely a better metric for QPS even though we wouldn't be able to offload them to a different server/region one by one.
It makes the cost of a query more even across wireproto and edenapi.
Reviewed By: krallin
Differential Revision: D28058054
fbshipit-source-id: 5d606841e07816ec8808a3b9aba4b7c0614b9cb6
Summary:
When scrubbing to collect commit times for path info logging, its much easier to get correct commit times for manifests by walking from oldest changeset first. That way when any manifest/tree is discovered its from the closest changeset chunk to its creation.
Alternative would have been using the path data from linknode associated changesets to prune out which sub-manifests to walk when walking forward, which is more complicated and would require holding more state (or reloading changesets continuall)
Differential Revision: D28092314
fbshipit-source-id: 871dc80dd88b63959501dd1018b6466afae5c6c7
Summary: This annotation got lost during the refactor of repo factory.
Reviewed By: quark-zju
Differential Revision: D28135734
fbshipit-source-id: b91d359422ac2456d7c670ae7094f20e3d6e5d7c
Summary:
The implementation of `to_cbor_bytes` does not make use of the ownership. It
works the same a reference is given. However the method is a lot more flexible
if a reference is used for the argument.
Reviewed By: kulshrax
Differential Revision: D28132732
fbshipit-source-id: 2eecd44ce9e4ff5bc42ff01fd358b0d30dde91ef
Summary:
There have been a bunch of problems with the previous approach to scmstore, so I'm going to try to start simple, make it feature complete, and then add async integration and factor out generic functionality as appropriate.
This change contains a `TreeStore` implementation with a single, synchronous, batch read method (supporting local storage, memcache, and legacy fallback, with writing missing to cache).
Add `TreeStoreBuilder`, which duplicates the existing `TreeScmStoreBuilder`, which some changes that make it easier to use for this case. I intend to unify these in the future.
Add an inherent impl for `EdenApiTreeStore` that provides subset of the `BlockingEdenApi` trait, which eliminates the need to unpack this type into a different adapter as the old `scmstore` code does. This might not be the right approach here, in reality we only need a `(client: Arc<dyn EdenApi>, repo: String)` here for trees, and that + `ExtStoredPolicy` for files, so we could take the `EdenApiAdapter` approach here too. The only reason we have to do any of this is because when `pyrevisionstore` is called to construct `scmstore` / `contentstore`, all we have is `Arc<EdenApiTreeStore>`. We could also just make the `EdenApiRemoteStore` fields public, and access them through the `Arc`.
Add `add_mcdata` method to `MemcacheStore`, `impl TryFrom<Entry> for McData`, and `impl From<McData> for Entry` for convenience when working with `MemcacheStore` (so we don't need to manually unpack the type and build `Entry`, or manually build a fake `Delta` from `Entry` to write).
Reviewed By: DurhamG
Differential Revision: D28076900
fbshipit-source-id: 7fdb5e8a42d052879eff449f60d40a83cfa7145d
Summary:
Both `get_local_path` and `get_cache_path` take suffix as as `PathBuf`, even though they only ever use it as a reference. `get_local_path` also takes a reference to a `PathBuf`, even though it always clones it internally, and takes an `Option`, even though it just maps across the contents of the option.
I modified `get_local_path` to accept a `PathBuf` by move, which it uses directly, and to not take an `Option` (instead just calling `map` externally, removing some unnecessary unwraps), and for both functions to accept `impl AsRef<Path>` for suffix.
Reviewed By: DurhamG
Differential Revision: D28100527
fbshipit-source-id: df28b51c8005f3d95acc8e082b40adaab18e31c9
Summary: Add a Read/Write Guard API to IndexedLogHgIdDataStore which allows client code outside the module to perform a series of reads and writes without locking for each individually.
Reviewed By: kulshrax
Differential Revision: D28075788
fbshipit-source-id: 2a65a426f443e1a421198ad8b4c610e4822574f7
Summary:
Add get_entry, put_entry, and flush_log inherent methods to IndexedLogHgIdDataStore. Refactor callers to use them in cases where they don't lock across multiple reads / writes (to avoid performance regressions).
This should allow `ReadStore` and `WriteStore` to be moved out of the module.
Reviewed By: DurhamG
Differential Revision: D27979828
fbshipit-source-id: c9fb8c4ac68f67b285c72396509aa17928aa54ed
Summary: It has been wrong since 2014 (tweakdefault).
Reviewed By: kulshrax
Differential Revision: D28122703
fbshipit-source-id: c83ddbac2c6162e36672649c60c2e7916dc7cbd2
Summary: This is step towards unifying native merge/rebase structs with native checkout - we now construct native checkout plan from the action map, instead of directly making it from the diff
Reviewed By: quark-zju
Differential Revision: D28078156
fbshipit-source-id: 318d7e419ca9fef15a4aebf7494451f69a3bbbe5
Summary:
This diff makes concurrency of native checkout to be configurable
This config can be used to reduce concurrency on platforms that are known to cause issues with watchman due to too many checkout operations
Reviewed By: quark-zju
Differential Revision: D28074993
fbshipit-source-id: 0a09fcf3ae48d08cead36da56c06b546aecd16b4
Summary: This diff refactors out `Checkout` component from checkout plan and allows to configure parallelism in checkout
Reviewed By: quark-zju
Differential Revision: D28074994
fbshipit-source-id: 72933c757d6e27615d1ef2bb4652bc67c9c3253d
Summary:
Vertex is old. It no longer makes sense with the current structure. The main
issue is that the dag crate now has VertexName which may confuse readers at
first glance.
When Vertex was added DagId would have been confusing because we had structs
that were named Dag that did not use DagId directly. Those structures are now
renamed and DagId is consistently used for dag crate structures.
The IdMap database would still use the vertex name until someone runs a
migration to rename the column.
I am not 100% that this is needed, but it's a change that's been on my mind.
Reviewed By: quark-zju
Differential Revision: D28110184
fbshipit-source-id: b996a7545a90acc25e2bb5326f2731b95c8740b4
Summary:
Previously there were two different paths to HgChangeset. This diff unifies them, so that when walker state.rs is checking for a previous visit it will find that it happened.
For existing walks of changesets in the NewestFirst direction this wasn't causing a problem, however the next diff in stack adds support for OldestFirst walks. In the OldestFirst case the mismatch in paths to HgChangeset was leaving a deferred edge to visit when everything should have been visited in previous chunks.
Differential Revision: D28095569
fbshipit-source-id: ccba4a679fc28bde042cfc222e5097c84fa968c0
Summary:
Right now we write straight to a logger with no filter, so no matter the log
level we print this stuff out. Let's fix it.
While we're at it, move this back to debug level.
I'd made this trace in my recent cmdlib refactoring (which resulted in us
properly initializing logging in all binaries), since I assumed we just had level
filtering working but with debug-logging enabled and I didn't want to have to
update every single test, but it turns out that the reason we didn't print it
out at trace is just because thats not enabled at all in our slog build:
D28097080.
Reviewed By: StanislavGlebik
Differential Revision: D28116053
fbshipit-source-id: f59d9a70ea3c3d834adea16f2686bfc244672b14
Summary: The precise compressed size of big blobs in zstd varies between runs. Glob out the exact size
Reviewed By: StanislavGlebik
Differential Revision: D28116066
fbshipit-source-id: 990add820de6c8cb0029805bc1de304fdf83acba
Summary:
It wasn't in warm bookmark cache, but it was an oversight - there's no reason
for it to not be here. Let's add it, since in case of crashlooping derived data
tailer (see attached task T89911396) there might be nothing to derive fastlog
data structure, and we end up with a long queue to derive.
Reviewed By: krallin
Differential Revision: D28114533
fbshipit-source-id: feb29c07d90be6250c5385ae9f2fb13eb52eedba
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
Summary: Looks like these aren't needed since these files are owned by a TARGETS file.
Reviewed By: genevievehelsel
Differential Revision: D28101197
fbshipit-source-id: d790530227641bf25e48bd96c8a95dd31f08a954
Summary:
Now that autodeps knows where to find cpptoml.h, we no longer need these
manual annotation.
Reviewed By: kmancini
Differential Revision: D28100956
fbshipit-source-id: 463b73834c500c1d16a4a769af3655938124d49d
Summary:
For no particular reason I was looking at this and saw a bunch of
unneeded `vec![]` temporaries which could be replaced with arrays or slices.
Reviewed By: krallin
Differential Revision: D28073693
fbshipit-source-id: 7fca3b4c7b40cc380b4b128e9809912b7b9ba1f7
Summary:
The original bug that resulted in empty revisions being pulled is long-fixed:
T28553115. I'm planning to make data1 nullable so I can reclaim space by removing older
revs.
Reviewed By: DurhamG
Differential Revision: D28096278
fbshipit-source-id: a57da458df115dcbdf544e2151aa327651190c1a
Summary:
This enlists hgsql tests to the lists of tests using revision numbers and
marks some racy lines as optional
Reviewed By: quark-zju
Differential Revision: D28096282
fbshipit-source-id: eb8406cb74f3338d13d4109fce35f969ff9e3b79
Summary:
This is a hg-sever backport of fix from D27659634 (8e8aaa61d6)
Those are not used. Recently we saw build issues like:
lib/third-party/sha1dc/sha1.c:8:10: fatal error: string.h: No such file or directory
#include <string.h>
^~~~~~~~~~
Possibly by some compiler flags disabling stdlib. Since we don't need
the C code let's just remove them.
Reviewed By: StanislavGlebik
Differential Revision: D28096283
fbshipit-source-id: 6c5390d26264e1e39f99b29dec8608d92e5ae572
Summary: - Like it says in the title.
Reviewed By: HarveyHunt
Differential Revision: D28092796
fbshipit-source-id: 01816f815148aca6c86078fb7dec616ecf53095c
Summary:
This updates hg to have a different amount of retry for backoffs requested by
the server and errors.
The rationale is that backoffs are fairly well understood and usually caused by
a surge in traffic where everybody wants the same data (in which case we should
be willing to wait to get it because there is literally no alternative),
whereas general errors aren't predictable in the same way.
We're now effectively at a point on the server side where _all_ our instances
have the exact same load, so if any server is telling you to backoff, that
pretty much guarantees that the whole tier has too much traffic to deal with.
This leaves us with two options:
- Tell clients to wait longer and smooth out the traffic surge.
- Add enough capacity that even our biggest surges don't result in _any_
throttling.
The latter is a bit unrealistic unrealistic given we routinely get egress
variations in excess of 5x (here's an example: https://fburl.com/ods/pidsrqnl),
so this does the former.
This also updates the client to tell the server how many attempts it has left
in addition to how many it used up so far. How many are left is more meaningful
for alerting!
Finally, it adds a bit of logging so that in debug mode you can see this
happening.
Reviewed By: quark-zju
Differential Revision: D28092797
fbshipit-source-id: f61410e39c4a3e3356371a3c7bd7892de4beacc8
Summary:
After D27144492 (48cd15ab14) we disabled revision number resolution. There is no need to
consider it when calculating shortest prefix.
Reviewed By: DurhamG
Differential Revision: D28072997
fbshipit-source-id: 832017c7b626265eb8cd2dd78946a03c4e7228f6
Summary:
This diff defines symlink type in `DirType`.
Even though it is not directly used in the FSCK diff. This will allow us to support symlink in EdenFS Windows in the future.
Reviewed By: genevievehelsel
Differential Revision: D28016305
fbshipit-source-id: 67c1aa22e39198f9c91845129695f27b8303a5f1
Summary: Add strum derivations to bulkops so we can use them in command line parsing later in stack.
Differential Revision: D28069912
fbshipit-source-id: 4d997e20e18f2011b51933ed4322c85bb7468980
Summary:
We were ignoring the return value of runWhileMaterialized, and thus we were
returning to FUSE before fallocate returned.
Reviewed By: fanzeyi
Differential Revision: D28081991
fbshipit-source-id: f398942ddb2432e48e80c148abc8edb7e5ada71d
Summary: Start logging mtime as relatedness key in the walker scrub pack info output
Differential Revision: D28055637
fbshipit-source-id: 4c24c5f2af0414ae7df17ade69bba9ff18861264
Summary:
We used to carry patches for Tokio 0.2 to add support for disabling Tokio coop
(which was necessary to make Mononoke work with it), but this was upstreamed
in Tokio 1.x (as a different implementation), so that's no longer needed. Nobody
else besides Mononoke was using this.
For Hyper we used to carry a patch with a bugfix. This was also fixed in Tokio
1.x-compatible versions of Hyper. There are still users of hyper-02 in fbcode.
However, this is only used for servers and only when accepting websocket
connections, and those users are just using Hyper as a HTTP client.
Reviewed By: farnz
Differential Revision: D28091331
fbshipit-source-id: de13b2452b654be6f3fa829404385e80a85c4420
Summary:
This used to be used by Mononoke, but we're now on Tokio 1.x and on
corresponding versions of Gotham so it's not needed anymore.
Reviewed By: farnz
Differential Revision: D28091091
fbshipit-source-id: a58bcb4ba52f3f5d2eeb77b68ee4055d80fbfce2
Summary:
Connect up the scrub stream types so they will be uniform for scrubs that log pack info and those that do not.
This is in preprepation for the next diff which connects up the pack info logging of path hashes to scrub. CI for this diff verifies its not broken the non-path tracking case.
Differential Revision: D28031868
fbshipit-source-id: 7bf91eb1778f57487f6a2847f215cf7f5cd2dff7
Summary: This moves evolve_path up to WrappedPathLike so that we can use sample route evolution logic for routes that track paths (e.g. corpus sampling) and path hashes (e.g. scrub, where path hashes take less memory than full paths).
Differential Revision: D28031867
fbshipit-source-id: cdabdc466158a8db1c770536747c996dddb27e71
Summary: Name the fields rather than leave it as a tuple struct. This makes it a bit easier to work with in the rest of the stack
Differential Revision: D28062254
fbshipit-source-id: 9e5202b4d6f1f29d44d98b86aa9b6ddb97d821eb
Summary: Makes more sense for this to be a method on NodeType
Differential Revision: D28031869
fbshipit-source-id: 1ddbafa0d7634ac67fd8d5112e6f57759ed91638
Summary: Name the fields rather than leave it as a tuple struct
Differential Revision: D28031866
fbshipit-source-id: 039f004e0b81294aa6d6b13e79cb45ee2b84567c
Summary: This new trait abstracts across WrappedPath and WrapperPathHash. Later in the stack I make path tracking use this to track either full paths (for corpus sampling) or path hashes (for logging from scrub).
Differential Revision: D28031870
fbshipit-source-id: d1c57230f68fffff179929a3cb92c82d92e0588c
Summary:
Like it says in the title. This isn't giving us the same error consistently
causing flaky failures.
Reviewed By: StanislavGlebik
Differential Revision: D28091747
fbshipit-source-id: dfc7a28b443c6577823c71cee7b006ed30fec18e
Summary: This is no longer needed, as all construction is performed by facet factories.
Reviewed By: StanislavGlebik
Differential Revision: D28001390
fbshipit-source-id: 237dd4f7b8b08bec5b85360edc3be7018d9161de
Summary:
Keeping the `Changesets` trait as well as its implementations in the same crate means that users of `Changesets` also transitively depend on everything that is needed to implement it.
Flatten the dependency graph a little by splitting it into two crates: most users of `Changesets` will only depend on the trait definition. Only the factories need depend on the implementations.
Reviewed By: krallin
Differential Revision: D27430612
fbshipit-source-id: 6b45fe4ae6b0fa1b95439be5ab491b1675c4b177
Summary:
The changesets object is only valid to access the changesets of a single repo
(other repos may have different metadata database config), so it is pointless
for all methods to require the caller to provide the correct one. Instead,
make the changesets object remember the repo id.
Reviewed By: krallin
Differential Revision: D27430611
fbshipit-source-id: bf2c398af2e5eb77c1c7c55a89752753020939ab