Summary:
Similar to Rust tracing's instrument. It's a decorator for functions that will
generate spans when called.
Reviewed By: sfilipco
Differential Revision: D26021958
fbshipit-source-id: bbb648ab07d6db233cd16f56a5f0441df07e1c2e
Summary:
This allows Python to create span callsites understood by Rust tracing,
and operate on the spans (ex. enter, exit, record, etc.).
Reviewed By: sfilipco
Differential Revision: D26013747
fbshipit-source-id: 2783b29750e3279c5481422bddc83366ff7a3548
Summary:
This allows Python to create callsites for one-off events understood by Rust.
This diff adds the "EventCallsite" for logging one-off events.
Reviewed By: sfilipco
Differential Revision: D26013749
fbshipit-source-id: 2520928dc360852afe2780267036e9d22c212191
Summary:
Practically there could be a lot of duplicated strings. For example, `module`
name could be the same for all functions in a file. `cat` could be a commonly
used field name, etc. Intern the strings so we don't waste too much space.
Reviewed By: sfilipco
Differential Revision: D26013748
fbshipit-source-id: 2740319019ce5058acfd7a9d2bb972c08a9386de
Summary:
To make Python code integrate deeper with the Rust tracing eco-system (ex. be
aware of layered subscribers, and filters), add APIs to create tracing Callsite
on the fly.
Unfortunately, the tracing APIs make it a bit too verbose.
Reviewed By: sfilipco
Differential Revision: D26003017
fbshipit-source-id: c3a36bcbaf4b7bbeafc3e2a8ddfdc7193c0972d3
Summary:
Limit the span or event count that a callsite can log so we can handle highly
repetitive calls. This is similar to D23307793 (d8e775f423) but applies to the Rust tracing
APIs.
Reviewed By: sfilipco
Differential Revision: D26090876
fbshipit-source-id: 394f52d697fc5e9ba5b900cebc9405a9f29220ce
Summary: Those APIs are internal details. Public APIs do not need them.
Reviewed By: sfilipco
Differential Revision: D26090877
fbshipit-source-id: b6e3ac76de719d1db71c2b57da3e8ef76015089e
Summary:
The name is being taken by stdlib:
warning: a method with this name may be added to the standard library in the future
--> eden/scm/lib/dag/src/spanset.rs:228:14
|
228 | .binary_search_by(|probe| span.low.cmp(&probe.low))
| ^^^^^^^^^^^^^^^^
|
= note: `#[warn(unstable_name_collisions)]` on by default
= warning: once this method is added to the standard library, the ambiguity may cause an error or change in behavior!
= note: for more information, see issue #48919 <https://github.com/rust-lang/rust/issues/48919>
= help: call with fully qualified syntax `BinarySearchBy::binary_search_by(...)` to keep using the current method
= help: add `#![feature(vecdeque_binary_search)]` to the crate attributes to enable `VecDeque::<T>::binary_search_by`
Reviewed By: sfilipco
Differential Revision: D26092424
fbshipit-source-id: d2cdf7d73d2f808f038817c9dc9f4c531ff643bd
Summary:
Yesterday we had an alarm when blobstore sync queue got overloaded again. This
time it was caused by a large commit cloud commit landing and writing lots of
content and alias blobs.
As we discussed before, let's add an option that would allow us to not write to
blobstore sync queue for commit cloud pushes of content and aliases.
It would slightly increase the latency, but will protect blobstore sync queue
from overloading.
Reviewed By: farnz
Differential Revision: D26129038
fbshipit-source-id: 0e96887e3aa3cf26880899c820f556bb16c437cb
Summary:
When running large repos its interesting to be able to reduce memory usage by clearing all or part of the walk state between chunks.
This adds ability to clear the node data by node type, and to clear the interning maps by the type of interned data type
The only interned type not clearable is the bonsai changeset id as that is used to maintain the list of deferred edges for later chunks to process.
Reviewed By: krallin
Differential Revision: D25867960
fbshipit-source-id: 4b48f03b1a1b8fef1c5ded952bdcd6b1241dcc32
Summary:
Split the WalkVisitor trait into TailingWalkVisitor to allow &mut self references on the methods called from
tail.rs.
This diff also has logging changes to show the chunk bounds and introduce a chunking log tag. This helped in testing.
Reviewed By: krallin
Differential Revision: D26054258
fbshipit-source-id: c74af558f1da98689a38ca61363baf7ee52a265e
Summary:
If the LFS server is overloaded then we send 429s to clients. However,
we also do this for our health checks which results in the load balancer
thinking that we're unhealthy.
Update the throttling middleware so that it doesn't throttle health checks.
Reviewed By: krallin
Differential Revision: D26125384
fbshipit-source-id: 19260caf27c6c84b6b51e6a75a5533e220d429aa
Summary:
When SQL gc starts up, all blobs written since the last time round
have an unknown generation. Set them in a bulk operation, so that we end the
mark process with all blobs belonging to a generation
Reviewed By: ahornby
Differential Revision: D26102600
fbshipit-source-id: f6403623182f92486abdac91e5ae7322678246bb
Summary: Not much to add .. Guess we gotta update a stub here.
Reviewed By: ahornby
Differential Revision: D26124590
fbshipit-source-id: efc4f324b5fed15cff46b358c2b491480e9b73fb
Summary:
The LFS server exports ODS stats for both client and server errors for
every combination of repo and method. However, it's possible for there to be
errors without a relevant repo or method (e.g. accessing a random path or
rejecting client requests due to rate limiting).
I'd like to add a detector to catch an increase in 429s (the result of rate
limiting clients) so we can detect issues like S220624 earlier.
Update the ODS middleware to first log HTTP status before moving into per-repo
and per-method stats. I've also included explicit counters for 429s and 404s.
NOTE: I have left the repo and method counters as we use them in our detectors.
However, I intend to update our dashboards to no longer use them. In the future
we may want to delete them.
Reviewed By: krallin
Differential Revision: D26102576
fbshipit-source-id: a2ff9c4d9a80bb11f2b2cf647041b035b687d798
Summary: New warnings on the docs side - fix them up
Reviewed By: StanislavGlebik
Differential Revision: D26108490
fbshipit-source-id: d7d74837570778e392c7662fd56c770f0b52988d
Summary:
Add a command to import the whole git tree as a single bonsai changeset. This
command maybe useful if we don't need the whole git history.
Reviewed By: krallin
Differential Revision: D26076645
fbshipit-source-id: 21b712776af1906ca7b06af088d98848bab907b8
Summary: I'm going to need this functionality in the next diff, so let's move it.
Reviewed By: krallin
Differential Revision: D26105565
fbshipit-source-id: bc31242713a4a37a3a48e17f6aae7690da7087f3
Summary:
We're hitting master overload while trying to mark generations.
Make use of a read replica to short-circuit work if the replica knows it's all
OK so that we catch up over time by moving load to replicas
Reviewed By: ahornby
Differential Revision: D26003713
fbshipit-source-id: f18458c8b10d66e3a945391d226848810691762a
Summary:
Provide a command that measures per-generation sizes, to be run in
Chronos.
Reviewed By: ahornby
Differential Revision: D25981511
fbshipit-source-id: 7d9877dd5d04ac6881b5024e0ba132c234499d5a
Summary:
In prod, I see errors that imply a master move or a one-off DB
failure, like:
```
Error: While executing InsertGeneration query
Caused by:
Mysql Query failed: Failed The MySQL server is running with the --read-only
(super) option so it cannot execute this statement.
```
and
```
Error: While executing InsertGeneration query
Caused by:
Mysql Query failed: TimedOut [7002](MySQL Client) Query timed out (no rows seen, took 10000 ms)
```
Teach garbage marking to retry after a random delay,
Reviewed By: StanislavGlebik
Differential Revision: D25974428
fbshipit-source-id: d905ea2c9da19de79b1b8322193831774c3f1ce9
Summary:
We have fallback logic to go to the leader if the data we want is missing in
the replica, but right now it's backwards so we go to the leader to find data
we actually *did* find in the replica (and we don't go to the leader for
missing data).
Reviewed By: sfilipco
Differential Revision: D26103898
fbshipit-source-id: 535abab2a3093165f1d55359d102a7a7cb542a9c
Summary:
The `assert_glob()` function in the glob tests previously only checked that
the expected number of results were returned. This updates the tests to
actually verify the returned paths. This also adds a few checks that verify
the behavior when the `wantDtype` parameter is True
Reviewed By: chadaustin
Differential Revision: D18903993
fbshipit-source-id: 5ed4c88160ffbda2e5fdcf089e210fda92868d56
Summary:
When a cert is expired, it can happen that the SSL error isn't raised till
after all our TLS handling, but in performing the HTTP request. Check the
exception type to display our certificate warning in these cases as well.
Reviewed By: krallin
Differential Revision: D26101231
fbshipit-source-id: 8342db5908e1b5fac9973fe48de432d591e9f54e
Summary:
Let's use this function since it removes copy-paste and adds additional checks
- for example it checks that parent are in blobstore.
Reviewed By: krallin
Differential Revision: D26079176
fbshipit-source-id: 9cd9bd170b929fa66c691432417a952ce11028ab
Summary:
We want multiple bookmarks, but only one of them should assign new globalrevs,
so it follows that we shouldn't sync the counter when other bookmarks are being
moved.
This does that.
Reviewed By: ahornby
Differential Revision: D26076567
fbshipit-source-id: 0ccc311984d3379cb44ccf10cbcb90ac31b82ee3
Summary:
Like it says in the title. This is prep work for allowing extra
bookmarks in a Gobalrevs repo later in this stack.
Reviewed By: ahornby
Differential Revision: D26076566
fbshipit-source-id: c775d50dfaa51e0f0f64e861b6c5b7ee16d62074
Summary: See D26019054 for what this does.
Reviewed By: StanislavGlebik
Differential Revision: D26019125
fbshipit-source-id: 071f0115d5ee8435bbc7ff5208ace8ef6e948272
Summary:
Bytecode generation entry point `compile_ffi::compile_from_text_cpp_ffi(...)` and a driver to exercise it: `hh_single_compile_cpp`.
What we get here is a C FFI for generating HHAS from source text. We can tweak the interface as we need to going forward but this will do for a first version and I think has roughly feature parity with the OCaml FFI:
- `compile_ffi.rs` defines the function;
- `compile_ffi.h` defines the C interface;
- `hh_single_compile_cpp.cpp` provides a C++ CLI that calls the function.
Reviewed By: shiqicao
Differential Revision: D25979967
fbshipit-source-id: 4b46f9af23c61150dda6c33f9fa14e2c455c54c2
Summary:
I plan to use these functions in the megarepotool, so let's move them to a
library that can be used in both.
Reviewed By: krallin
Differential Revision: D26015773
fbshipit-source-id: 0d2d28d86471c417508494883b69fb64e1bea328
Summary: Add a new enum representing the types the walker interns so that we can optionally clear them between chunks. This change adds the enum and command line parsing, actual clearing follows in the next diff.
Reviewed By: krallin
Differential Revision: D25910622
fbshipit-source-id: 0226b4009bf8199498e21e52f734a9529ee7afaa
Summary:
With certain compiler flags:
```
#include <atomic>
struct ImportPriority {
ImportPriority() {}
};
std::atomic<ImportPriority> f() {
throw "";
}
```
Fails with with:
```
libgcc/include/c++/trunk/atomic:194:7: error: exception specification of explicitly defaulted default constructor does not match the calculated one
atomic() noexcept = default;
^
Demo.cpp:15:29: note: in instantiation of template class 'std::atomic<ImportPriority>' requested here
std::atomic<ImportPriority> f() {
^
1 error generated.
```
Reading the C++ spec, constructors default to noexcept, but it's not clear if that's true when they have a body. There are flags you can set that make this compile, but let's be maximally compatible and add `noexcept`.
Reviewed By: fanzeyi
Differential Revision: D26076451
fbshipit-source-id: 2f63256377fb31fd7867d7b03e7572e033f72dfc
Summary:
This diff revives D25454687 (f98273063a), which was backed out by D25792491 (b52168c4c8) because it was causing Mercurial to crash in certain environments where certificates are configured incorrectly.
I've modified the code so that by default, certificates are not validated (maintaining the old behavior), but users of the API can opt-in to validation. In the case of EdenAPI (which is the only user that opts in), this is controlled via a new `edenapi.validate-certs` config option, which defaults to false. This allows enforcing validation on platforms where the configs should be correct (such as devservers) while maintaining the old behavior on other platforms by default.
Reviewed By: DurhamG
Differential Revision: D26009207
fbshipit-source-id: 904dee61fd12fdee4a0031d14adef7fdb4801139
Summary:
Previously the LazySet only supports non-async Iterator. This makes it more
flexible useful. It will be used in upcoming changes.
Reviewed By: sfilipco
Differential Revision: D25858800
fbshipit-source-id: 8c8e874f05cfab721bc0fa55160a9337ed7c2c27
Summary:
In the future we'd like to allow building the dag crate without the indexedlog
portion. This diff adds support for that.
Reviewed By: DurhamG
Differential Revision: D25769054
fbshipit-source-id: eb5a200841f878836a9f68e65e7d50be7e6b9a79
Summary:
In the future we want to build dag without indexedlog dep for Mononoke
use-case. One of the problem is the ToWire trait implemented on dag::Id by
edenapi-types. Within buck, the dag crate will have 2 targets: dag and dag-lite
(no indexedlog). They are incompatible meaning that edenapi-types depending on
dag-lite will not provide Id::to_wire for crates using dag, or vice-versa.
To solve that, we move the Id and other types to a separate crate that only has
one buck target so edenapi-types, and segmented_changelog from Mononoke can
depend on it without issues. This also makes edenapi-types more lightweight.
Reviewed By: sfilipco
Differential Revision: D25857917
fbshipit-source-id: d3e15a2b6638cc6e15171a1e9bc37362e03df583
Summary: In upcoming changes, we're moving Id to a separate crate. This makes that easier.
Reviewed By: sfilipco
Differential Revision: D25857918
fbshipit-source-id: 6e2163f6fa171d4cd3be4fc0c4c248fd87ba739c
Summary:
Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8076
D26025779 (e282452d65) may have broken the getdeps build when logging
surrogates. Use errors=surrogateescape to try to avoid that.
Reviewed By: danobi
Differential Revision: D26079717
fbshipit-source-id: 98d938bfced417e0b39a6dcddb241225ffb69c06
Summary:
The FbWhoAmI background thread throws an exception when /etc/fbwhoami
cannot be opened, which terminates the process. This happens to people
in various spurious situations on corp. Since EdenFS doesn't care, let
FbWhoAmI return empty string values in that case.
Reviewed By: genevievehelsel
Differential Revision: D26010530
fbshipit-source-id: b0e9294e45f6fc2f7547982692de8ff09939405b
Summary:
External Data Representation (XDR) is an encoding scheme that NFS and all the
related protocols are using. As EdenFS will soon gain the ability to expose
itself via NFS, let's add some code to encode/decode data to XDR.
This code was written by wez.
Reviewed By: kmancini
Differential Revision: D25986692
fbshipit-source-id: ed4ce52ac87fc9396219b35c9e143e5471039fee
