Summary:
This makes it easier to filter logs related to remote fetching.
The `DEBUG dag::protocol: resolve ids [0] remotely` means the lazy hash resolution is working.
Reviewed By: kulshrax
Differential Revision: D27971117
fbshipit-source-id: f2492204c70d793997d0c3865e500bbad56b1953
Summary:
Write commit to master group. This provides proper "CloneData" and allows us to
actually test lazy commit hash backend (since only commits in the master group
can have lazy hashes).
Reviewed By: DurhamG
Differential Revision: D27971123
fbshipit-source-id: 4e19486007ddc89de7468be65445559f34d796f5
Summary:
Add clone endpoint so we can clone from an eager test repo.
Note: the master group is empty and "cloneata" does not quite work yet due to
EagerRepo not writing to the master group. It will be fixed later.
Reviewed By: DurhamG
Differential Revision: D27971121
fbshipit-source-id: 0cc35136c6987673c2c4fbbd892c344c3586fcb7
Summary: This update makes it so that we don't log versions to scuba from tests.
Reviewed By: krallin
Differential Revision: D27449808
fbshipit-source-id: 9c79e83fbfdf3d9a02c2cfc8b6a8255edb4241fe
Summary:
This is going to enable the background update in SegmentedChangelog to log
entries to Scuba.
The scuba sample builder is not fundamentally different than other elements of
the environment. It is used slightly differently to, for example, Logger,
because it has to cloned in all places that want to log rows but otherwise it
has the same characteristics.
Reviewed By: krallin
Differential Revision: D28210008
fbshipit-source-id: 68468868d13f29dddf21095bd7526cb4ff690786
Summary:
This is where async requests are logged to be processed, and from where they
are polled later.
It will acquire more functionality when the actual request processing business
logic is implemented.
Reviewed By: StanislavGlebik
Differential Revision: D28092910
fbshipit-source-id: 00e45229aa2db73fa0ae5a1cf99b8f2d3a162006
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
Log the sizing metadata about keys that scrub has seen to the pack info logs.
This uses the sampling blobstore to see all blobstore gets and captures info from them.
Also updates relatedness_key fieldname to mtime as that way its less easily confused with similarity_key
Differential Revision: D28115620
fbshipit-source-id: 666a444c2d91b0ca5bb225cea971f9b183e6a48d
Summary:
Pass BlobstoreGetData to the sampler so that it has a chance to sample the BlobstoreMetadata as well as the BlobstoreBytes.
This is used in the next diff for sampling the sizing information.
Reviewed By: markbt
Differential Revision: D28115619
fbshipit-source-id: 7a79d482c9ba1ed8b08afab5f1c1b8fe7c4f257a
Summary: When reading from packblob we'd like to see metadata about sizes so that can log it for the packer later in this stack.
Reviewed By: markbt
Differential Revision: D28101971
fbshipit-source-id: 96dd0d5497c2bb5c27231709dbd19d00168e1a77
Summary:
This diff prepares some macros to be usable by external crates. Next diff uses
them in `megarepo_api:async_requests`.
The way it makes macros usable externally is by:
1) using abs paths for std types outside of prelude
2) importing necessary traits in methods which need them
3) re-exporting important `mononoke_types`-defined types and third-party
dependencies via `mononoke_types::private` and using `$crate::private` in macro
to access them.
This way someone needs only to depend on `mononoke_types` and only to import
the macro, not any other re-exported definitions.
Reviewed By: krallin
Differential Revision: D28033196
fbshipit-source-id: 3f5fa34388a4f6466969d06c5739c5282f007e68
Summary:
The trees endpoint is another example where we try to send errors to the
client. As it was done previously we would fail to log any errors on the
server side. This diff corrects that by using custom_cbor_stream.
Reviewed By: kulshrax
Differential Revision: D28111102
fbshipit-source-id: 468095d024647f472b8ad9a9e17ca8364605ff98
Summary:
Some EdenapiEndpoints serialize their error and send them to the client. This
robs the gotham framework from the change of seeing those errors. This custom
stream takes care of counting those errors and reporting them to gotham for
logging.
There is a slight worry in the back of my head that if we start getting errors,
CPU utilization will increase because we construct so many extra strings.
Reviewed By: kulshrax
Differential Revision: D28109314
fbshipit-source-id: ab39a05227e0f8be9205d6036d32523cb1e45ca4
Summary:
The walker does about 50% of our traffic, but it also has fairly predictable
access patterns. It seems unlikely that we really benefit from logging all
scrub activity with the same precision as we do other traffic.
So, let's sample it. This should make a lot of space in our Scuba table and
make us more resilient to sudden changes in activity.
Reviewed By: StanislavGlebik
Differential Revision: D28254057
fbshipit-source-id: da748a565954c31c2d9e087b7b07747a435427bf
Summary:
Add `--blame-v2` to `mononoke_admin blame compute`. This can be used to compute
blames in the new format and validate that they are correct.
Reviewed By: mitrandir77
Differential Revision: D28183160
fbshipit-source-id: f698a77c109bfce05aeb66cd405c6f20bf158801
Summary:
Add types and an implementation for blame_v2. This differs from the existing
blame definition as follows:
* Changeset IDs and paths are always stored as indexes into a mapping.
* The Changeset ID index is stable for the p1-parent chain of commits. Although only
the changeset IDs that are used in the blame are present, the index for a particular
changeset ID doesn't change, unless it is added in a merged-in parent.
* The notion of rejection is merged into the blame definition. This makes it more
explicit that rejected parents are simply ignored by the blame algorithm.
Some bugs in the original blame algorithm are also fixed:
* Initially empty files don't cause a zero-length blame range that can never be deleted.
* Merges correctly recalculate origin offsets.
This blame implementation is not used yet. It will be introduced in subsequent commits.
Reviewed By: StanislavGlebik
Differential Revision: D28183161
fbshipit-source-id: 3604ff7ff393220e13a69d0ad51247d7bfd80926
Summary:
Add thrift definitions for blame_v2. This uses a map from integer id to
changeset id to avoid repeating 32-byte hashes in files with many blame ranges,
and to provide a stable integer index for changesets that can be used to
provide a more human-readable id for each blamed changeset.
Reviewed By: StanislavGlebik
Differential Revision: D28183163
fbshipit-source-id: a6ae01d92c3d3c022a1d412f2ba0046f08c2d9f1
Summary:
Add debug output to rage to make sure we have the relevant information in case
we need to debug issues with schemes.
Reviewed By: quark-zju
Differential Revision: D28222910
fbshipit-source-id: 9499c736d61b2c0e4568e05a3afc0ae9730acedf
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799