Summary:
Pass BlobstoreGetData to the sampler so that it has a chance to sample the BlobstoreMetadata as well as the BlobstoreBytes.
This is used in the next diff for sampling the sizing information.
Reviewed By: markbt
Differential Revision: D28115619
fbshipit-source-id: 7a79d482c9ba1ed8b08afab5f1c1b8fe7c4f257a
Summary: When reading from packblob we'd like to see metadata about sizes so that can log it for the packer later in this stack.
Reviewed By: markbt
Differential Revision: D28101971
fbshipit-source-id: 96dd0d5497c2bb5c27231709dbd19d00168e1a77
Summary:
This diff prepares some macros to be usable by external crates. Next diff uses
them in `megarepo_api:async_requests`.
The way it makes macros usable externally is by:
1) using abs paths for std types outside of prelude
2) importing necessary traits in methods which need them
3) re-exporting important `mononoke_types`-defined types and third-party
dependencies via `mononoke_types::private` and using `$crate::private` in macro
to access them.
This way someone needs only to depend on `mononoke_types` and only to import
the macro, not any other re-exported definitions.
Reviewed By: krallin
Differential Revision: D28033196
fbshipit-source-id: 3f5fa34388a4f6466969d06c5739c5282f007e68
Summary:
The trees endpoint is another example where we try to send errors to the
client. As it was done previously we would fail to log any errors on the
server side. This diff corrects that by using custom_cbor_stream.
Reviewed By: kulshrax
Differential Revision: D28111102
fbshipit-source-id: 468095d024647f472b8ad9a9e17ca8364605ff98
Summary:
Some EdenapiEndpoints serialize their error and send them to the client. This
robs the gotham framework from the change of seeing those errors. This custom
stream takes care of counting those errors and reporting them to gotham for
logging.
There is a slight worry in the back of my head that if we start getting errors,
CPU utilization will increase because we construct so many extra strings.
Reviewed By: kulshrax
Differential Revision: D28109314
fbshipit-source-id: ab39a05227e0f8be9205d6036d32523cb1e45ca4
Summary:
The walker does about 50% of our traffic, but it also has fairly predictable
access patterns. It seems unlikely that we really benefit from logging all
scrub activity with the same precision as we do other traffic.
So, let's sample it. This should make a lot of space in our Scuba table and
make us more resilient to sudden changes in activity.
Reviewed By: StanislavGlebik
Differential Revision: D28254057
fbshipit-source-id: da748a565954c31c2d9e087b7b07747a435427bf
Summary:
Add `--blame-v2` to `mononoke_admin blame compute`. This can be used to compute
blames in the new format and validate that they are correct.
Reviewed By: mitrandir77
Differential Revision: D28183160
fbshipit-source-id: f698a77c109bfce05aeb66cd405c6f20bf158801
Summary:
Add types and an implementation for blame_v2. This differs from the existing
blame definition as follows:
* Changeset IDs and paths are always stored as indexes into a mapping.
* The Changeset ID index is stable for the p1-parent chain of commits. Although only
the changeset IDs that are used in the blame are present, the index for a particular
changeset ID doesn't change, unless it is added in a merged-in parent.
* The notion of rejection is merged into the blame definition. This makes it more
explicit that rejected parents are simply ignored by the blame algorithm.
Some bugs in the original blame algorithm are also fixed:
* Initially empty files don't cause a zero-length blame range that can never be deleted.
* Merges correctly recalculate origin offsets.
This blame implementation is not used yet. It will be introduced in subsequent commits.
Reviewed By: StanislavGlebik
Differential Revision: D28183161
fbshipit-source-id: 3604ff7ff393220e13a69d0ad51247d7bfd80926
Summary:
Add thrift definitions for blame_v2. This uses a map from integer id to
changeset id to avoid repeating 32-byte hashes in files with many blame ranges,
and to provide a stable integer index for changesets that can be used to
provide a more human-readable id for each blamed changeset.
Reviewed By: StanislavGlebik
Differential Revision: D28183163
fbshipit-source-id: a6ae01d92c3d3c022a1d412f2ba0046f08c2d9f1
Summary:
Add debug output to rage to make sure we have the relevant information in case
we need to debug issues with schemes.
Reviewed By: quark-zju
Differential Revision: D28222910
fbshipit-source-id: 9499c736d61b2c0e4568e05a3afc0ae9730acedf
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799
Summary:
This makes it easier to use `--keep` to investigate tests by using
`--configfile`.
Reviewed By: kulshrax
Differential Revision: D27971122
fbshipit-source-id: 8adcbeab825155858499c24ca74c2979049adeda
Summary:
Rename `new_with_containers` to `new`. `new` the standard name for
what the fuction does.
Reviewed By: krallin
Differential Revision: D28210005
fbshipit-source-id: 9ad0dad5e8afd89d7538fd32d93de10837e26f9d
Summary:
Refactoring the construction logic for CoreContext.
`test_mock_class` is used in only one place. That usage can be replaced with
`test_mock_session`.
Reviewed By: krallin
Differential Revision: D28210007
fbshipit-source-id: c81ca31cb7255ef640fe641e9b38f239cfb69267
Summary:
Refactoring CoreContext construction to express purpose. We will have
a constructor for request processing, for background processing.
Bulk processing is another category that has it's own constructor
already. Renaming it to make it more prominent.
Reviewed By: krallin
Differential Revision: D28210006
fbshipit-source-id: 2bb74d97e2f3588aa539e58c3d6dd6842f898121
Summary:
We have a linker issue on Windows when building EdenFS with CMake:
```
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpSetStatusCallback referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpOpen referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpCloseHandle referenced in function winhttp_close_connection
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpConnect referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpReadData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpWriteData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpQueryOption referenced in function certificate_check
```
This fixes that.
Reviewed By: xavierd
Differential Revision: D28230163
fbshipit-source-id: f74e42ee30ec8f3b81c1f80b7cf63a21ea97732c
Summary: This should fix the test failure introduced in D28203778 (9bee308885) where fsck tests were not skipped on Windows.
Reviewed By: genevievehelsel
Differential Revision: D28227230
fbshipit-source-id: 97ef5717fbda567b48f02a60e1906e374fdc6766
Summary: The syntax is not supported by Python 2.
Reviewed By: DurhamG
Differential Revision: D28233280
fbshipit-source-id: 9f882827b1357cb339e60180acadb38842c3cf8d
Summary: The syntax is not supported by Python 2.
Reviewed By: DurhamG
Differential Revision: D28232995
fbshipit-source-id: 62058751b4f00b78a2bd56908100a7bb7a3adfde
Summary: Upstream crate has landed my PR for zstd 1.4.9 support and made a release, so can remove this patch now.
Reviewed By: ikostia
Differential Revision: D28221163
fbshipit-source-id: b95a6bee4f0c8d11f495dc17b2737c9ac9142b36
Summary: Windows path like `eagerepo:///C:\foo\bar` needs special handling.
Reviewed By: kulshrax
Differential Revision: D27971119
fbshipit-source-id: 9d4b87782eca2734b708565f0ee22a7495253cff
Summary: We needed only for the initial rollout, we can delete it now
Reviewed By: krallin
Differential Revision: D28226715
fbshipit-source-id: 20349e9775b71292604a4054162f026087511db8
Summary: `hg gc` does not do anything anymore, so in order to reduce confusion, let's just print a message that says it is no longer supported and provide a manual remediation.
Reviewed By: xavierd
Differential Revision: D28164614
fbshipit-source-id: 7ed2392cdb0091cd604a15b4c2382632706981f2
Summary:
Quicksand is about 60% of our logging, but we don't really need to log
everything it does because the workload is pretty much the same across all
hosts.
Reviewed By: HarveyHunt
Differential Revision: D28225416
fbshipit-source-id: 68c0f68c4da73e0f7543754c917d7efa4bbf374e
Summary: It was requesting a slice but always converted it to an Iterator anyway. Receiving an iterator saves constructing a temporary Vec both here and later in the stack
Reviewed By: krallin
Differential Revision: D28127582
fbshipit-source-id: 625c1f17f1ded973f8b2aa13566928af0df83aec
Summary:
This avoids issues where the tree is stored without p1, p2. It is similar to
what we do for public commits (in createtreepackpart):
if sendtrees == shallowbundle.AllTrees or ctx.phase() != phases.public:
...
Note: the trees API actually provides p1, p2, but p1, p2 are dropped when
writing to the current data store implementation.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200388
fbshipit-source-id: e1fe93d8ae8576e691077d6db432d55f7b9d498d
Summary: Add a way to fetch tree content without going through store.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200387
fbshipit-source-id: 8f5b2214aafba39c7674f0f6b27af0c985f0ea72
Summary:
The `trees` API is coupled with a store. We're going to add another API that is
not coupled with a store so let's rename `trees` to `storetrees`.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200389
fbshipit-source-id: 826116f0b461873b2f5df07e7fd35e6d1018f929
Summary:
This output is non-determenistic, and it does not seem to be important in this test
We could replace HashMap with BTreeMap to make it deterministic, as an alternative, but it is probably not justified for this test
Reviewed By: quark-zju
Differential Revision: D28204050
fbshipit-source-id: 50000671520e3bbf41849dc53c420ccab496dca0
Summary: This import is not used anywhere (since D28007850 (bb3d207e33)) and could be removed (currently it throws a warning whenever someone attempts to do a cargo check)
Reviewed By: sandeepkumarpani888
Differential Revision: D28220436
fbshipit-source-id: a7fc0fe84d485d37fe86476011a853df57a6bc03
Summary: the option has been deprecated and is not used anywhere
Reviewed By: krallin
Differential Revision: D28191314
fbshipit-source-id: f5f092b93a9644c8249628520d8d707b60854aac
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
Context: https://fb.workplace.com/groups/edenfswindows/permalink/828914994691047/
Even with D27872753 it doesn't really make sense to have `eden fsck` running on Windows since it requires EdenFS repository to **be unmounted**.
This diff changes it to generate a warning to redirect users to run `eden doctor` instead (which is likely what they need).
Reviewed By: kmancini
Differential Revision: D28203778
fbshipit-source-id: ae105678876903bcf6514252bf07189775f9b187
Summary:
This will consume `CloneData` from EdenApi and write to the graph.
Note `CloneData<Vertex>` and `CloneData<HgId>` has the same mincode
serialization result so there is no need to do extra type conversion.
This can be used like:
In [1]: v=api.clonedata('fbsource');
In [6]: d=bindings.dag.commits.openhybrid(None, '/tmp/seg', '/tmp/msg', repo
...: .edenapi, repo.name, lazyhash=True)
In [7]: d.importclonedata(v)
Reviewed By: kulshrax
Differential Revision: D27971125
fbshipit-source-id: 4d420c6ff0495dc184e7c9618b866a69f0a00002
Summary:
Expose NameDag's `import_clone_data` API so this can be then exposed via
`pydag`.
Reviewed By: kulshrax
Differential Revision: D27971118
fbshipit-source-id: c9d869ffbbc8ba5a7a6ae98d17a2b7ea713bc675
Summary: The `CloneData` is currently only exposed in Rust. Expose it in Python too.
Reviewed By: kulshrax
Differential Revision: D27971124
fbshipit-source-id: 1a9c63274b6e2d78a176869b6810acbc191ba314
