Summary:
Copying an InodePtr/shared_ptr adds 2 atomics: one for the copy, and one when
the copy is dropped. In the case where a move can be achieved, we can avoid
these atomics and make the code more efficient.
Reviewed By: fanzeyi
Differential Revision: D28614769
fbshipit-source-id: 9c4be6ce335cc17eb889a5681aa9c13595a7909e
Summary:
Looking at the glob code, I noticed that some paths were copied into lambda and
vectors, while they could have been moved.
Reviewed By: fanzeyi
Differential Revision: D28614211
fbshipit-source-id: 5662586bcc3d9a9c5ca899c59d29b40a4e590adc
Summary:
When the caller asks to not perform sanity checking, this should be honored as
the caller can for instance tell that a path is guaranteed to be valid and
shouldn't be checked.
When running the newly added glob benchmark, EdenFS was spending 7.5% of the
time validating a path that was constructed via the operator+.
Reviewed By: fanzeyi
Differential Revision: D28612665
fbshipit-source-id: 282155d0415ab6458bd9307c89a24e9e090bf09d
Summary:
Querying RocksDB for a blob that we know is not present is expensive, and will
always construct a StoreResult::missing which does allocate memory for that to
simply be dropped. We can revamp the StoreResult a bit, but for now let's
simply not query RocksDB.
In theory, if RocksDB is populated with blobs this can be a loss as more
request will need to go to the hgcache, in practice, RocksDB local caching has
been disabled for so long that we'll always get cache misses.
Reviewed By: fanzeyi
Differential Revision: D28592666
fbshipit-source-id: 49f48097e81eddb4f9c3eba7af774baf018b0821
Summary:
C++ is full of footguns, in this case, the lockContents method either returns a
const std::vector<TreeEntry>&, or an rlock RAII for a folly::Synchronized
object. When assigning the return value of that method to an `auto` variable, a
copy is made in both cases, which isn't exactly what is desired in the first
case. Instead using the reference as-is is what we want.
To achieve the right behavior, a `const auto&` needs to be used: it will either be
a const reference in the first case, or a copy in the second case.
During an `arc focus2` while the Buck parsing phase is ongoing, the copy shows
up at the top of the profiler as one of the most expensive operation. This
significantly reduce its cost.
Reviewed By: chadaustin
Differential Revision: D28591502
fbshipit-source-id: eac1e062ed94442c90ac549ec137d91b4cb42b9c
Summary:
While Buck compiling, when a glob query is running, EdenFS uses tons of CPU,
let's add a small benchmark to profile the code in isolation and keep track of
the progress.
Reviewed By: fanzeyi
Differential Revision: D28611157
fbshipit-source-id: 81df21c8a8aba44fdefec044f9cae387b4cdd15c
Summary:
Add support to our C++ and Python SNAPSHOT parsers for version 2,
which is just an arbitrary binary string after the header.
Don't start writing the format yet. I'd like to roll one release with
read support before writing the new format.
Reviewed By: xavierd
Differential Revision: D28528235
fbshipit-source-id: 0b06596f06cafa26258ab392eaae4a8994f55f1f
Summary: The legacy format was last written in 2017, so we can remove it now.
Reviewed By: xavierd
Differential Revision: D28524663
fbshipit-source-id: 59ed437b0e5ef2e5ee3e3e9944d1049a13d64d9e
Summary:
EdenFS goes out of its way to track the second working copy parent,
but it never uses it. Stop writing it to the SNAPSHOT file.
Reviewed By: genevievehelsel
Differential Revision: D28453213
fbshipit-source-id: d7d36a1c67553f92234bec911051f4f1d4ef1d4a
Summary:
branch_forest_updater speedups.
* Allow for imports in paralell.
* Inside gitimport. Move slower things out of the main tokio-execution task.
Reviewed By: StanislavGlebik
Differential Revision: D28593816
fbshipit-source-id: 8191abe2e8ee1188651e7451fbc83267e7e88b9b
Summary: Like it says in the title. If a hook is bypassed, let's log that to Scuba.
Reviewed By: farnz
Differential Revision: D28601923
fbshipit-source-id: d27b409fc218b1aa21dd655c88c63e840ce9435c
Summary:
don't apply an old public bookmark if the commit is older than max_sync_age
there is a complicated logic because we need to make sure if we later run it with different commitcloud.max_sync_age value or hg cloud sync --full the bookmarks will be appear back.
So, the changes required in both:
* checkomission
* _mergebookmarks
but both cases covered in the tests
also, if you run with max_sync_age=1000 and later max_sync_age=0, the bookmarks will not disappear, which is expected.
Reviewed By: markbt
Differential Revision: D28572875
fbshipit-source-id: 317e897a2b81c3371dbea7eb39b8925570c1d40a
Summary:
this output is not noisy for big workspaces
if a head is omitted, don't warn about the bookmark because it is expected
Reviewed By: markbt
Differential Revision: D28568919
fbshipit-source-id: eb19e1d155f65de411c1dd41a8be6d83ca71c264
Summary:
The output is a bit too noisy for large workspaces.
We could skip older commits, we know the list coming ordered from commit cloud service.
All hashes are anyway available via `hg cloud sl`.
Also I fixed several look-ups in the list. Omittedheads are heavily used to check if something is present there.
Reviewed By: markbt
Differential Revision: D28568421
fbshipit-source-id: bcf62522798fed92df7ca546c73aa14da95f1567
Summary:
When scrubbing to a new store in a multiplex that has a recently populated store, the ctime for blobs from the recently populated store will often be inside the 7 day window for lookback of the healer queue which results in high query load on the queue.
This diff adds a command line option to override the period of lookback to the queue so that it can be made shorter and reduce or eliminate the queue queries.
Reviewed By: farnz
Differential Revision: D28533393
fbshipit-source-id: fd322364e0f595065fdc7900e86bc7c4e1cfec8a
Summary: Make the Tickable blobstore used in the muliplex blobstore test aware of ctime and PutBehaviour::IfAbsent.
Reviewed By: farnz
Differential Revision: D28571866
fbshipit-source-id: 8c8d2bf137f595f7577f491499170126dd697187
Summary: Add config pull.httpbookmarks to use the edenapi http protocol to fetch bookmarks in the central local repo pull method. This impacts the pull command, as well as other commands that pull bookmarks.
Reviewed By: quark-zju
Differential Revision: D27479112
fbshipit-source-id: 2b9821f458ec0af2579143fb2c2ed7d3ff41878a
Summary:
Add streaming compression support to Mononoke. We do the compression inside our
own framing, to make sure the compression doesn't interfere with any buffering,
flushing and keepalives.
Reviewed By: krallin
Differential Revision: D28323591
fbshipit-source-id: 7192bedcf12a2d0ec025deed8b4fa3857eefd508
Summary:
Support decompression for mononoke connections. When we request it, Mononoke
can support compression our stream, saving bandwith on low throughput
connections.
Reviewed By: StanislavGlebik
Differential Revision: D28535058
fbshipit-source-id: 7594f72978093a474efd168bb87b41c415310d6c
Summary: It can be used by `cloud ssl` template after D28000088 (b506eeea0c).
Reviewed By: liubov-dmitrieva
Differential Revision: D28561180
fbshipit-source-id: fb4bf3de85f7c320c13a2a53c6a103e85ebb5425
Summary:
Like it says in the title. The API between Bytes 1.x has changed a little bit,
but the concepts are basically the same, so we just need to change the
callsites that were calling `bytes()` and have them ask for `chunk()` instead.
This diff attempts to be as small as it can (and it's already quite big). I
didn't attempt to update *everything*: I only updated whatever was needed to
keep `common/rust/tools/scripts/check_all.sh` passing.
However, there are a few changes that fall out of this. I'll outline them here:
## `BufExt`
One little caveat is the `copy_to_bytes` we had on `BufExt`. This was
introduced into Bytes 1.x (under that name), but we can't use it here directly.
The reason we can't is because the instance we have is a `Cursor<Bytes>`, which
receives an implementation of `copy_from_bytes` via:
```
impl<T: AsRef<[u8]>> Buf for std::io::Cursor<T>
```
This means that implementation isn't capable of using the optimized
`Bytes::copy_from_bytes` which doesn't do a copy at all. So, instead, we need
to use a dedicated method on `Cursor<Bytes>`: `copy_or_reuse_bytes`.
## Calls to `Buf::to_bytes()`
This method is gone in Bytes 1.x, and replaced by the idiom
`x.copy_to_bytes(x.remaining())`, so I updated callsites of `to_bytes()`
accordingly.
## `fbthrift_ext`
This set of crates provides transports for Thrift calls that rely on Tokio 0.2
for I/O. Unfortunately, Tokio 0.2 uses Bytes 0.5, so that doesn't work well.
For now, I included a copy here (there was only one required, when reading from
the socket). This can be removed if we update the whole `fbthrift_ext` stack to
Bytes 1.x. fanzeyi had been wanting to update this to Tokio 1.x, but was blocked on `thrift/lib/rust` using Bytes 0.5, and confirmed that the overhead of a copy here is fine (besides, this code can now be updated to Tokio 1.x to remove the copy).
## Crates using both Bytes 0.5 & Bytes 1.x
This was mostly the case in Mononoke. That's no coincidence: this is why I'm
working on this. There, I had to make changes that consist of removing Bytes
0.5 to Bytes 1.x copies.
## Misuse of `Buf::bytes()`
Some places use `bytes()` when they probably mean to use `copy_to_bytes()`. For
now, I updated those to use `chunk()`, which keeps the behavior the same but
keeps the code buggy. I filed T91156115 to track fixing those (in all
likelihood I will file tasks for the relevant teams).
Reviewed By: dtolnay
Differential Revision: D28537964
fbshipit-source-id: ca42a614036bc3cb08b21a572166c4add72520ad
Summary:
This allows us to do staged rollout where some users are using "lazy" backend
and they won't be migrating down to "doublewrite" backend.
Reviewed By: liubov-dmitrieva
Differential Revision: D28554381
fbshipit-source-id: ebe2e25c96fd3b086a451c3909643d19c64a186c
Summary: Migrating from the lazy backend to lazy backend should be a no-op.
Reviewed By: liubov-dmitrieva
Differential Revision: D28554382
fbshipit-source-id: 71c06584f6f7a89096ce4a94843c88cbea542475
Summary:
I'd like to de-risk this rollout a bit. Right now, we depend on everything
properly inserting a `log_id` in the bookmarks table.
Reviewed By: StanislavGlebik
Differential Revision: D28569840
fbshipit-source-id: 9cf32303e7f5de2119925d391b1e19509f705d52
Summary:
We're expecting this to be refreshed every few seconds. If that does not
happen, there's a point at which querying the log is just not efficient
anymore, and if we get there we should just rebuild from scratch.
This does that.
Reviewed By: farnz
Differential Revision: D28505644
fbshipit-source-id: 18e6cec8d8534de742781e658005a09301d3aa5f
Summary:
Like it says in the title. This adds support for using a BookmarksSubscription
to update bookmarks. This is controlled by a killswitch.
One little downside here is that if the subscription falls too far behind, we
might have too much data to fetch from MySQL, so turning this on via a
killswitch isn't optimal.
In the next diff in this stack, I'll update the BookmarksSubscription to
support a max age on last refresh and have it rebuild from scratch when we hit
this. I still want to have a killswitch here for now though, because
re-creating a subscription is more expensive than our current bookmarks
fetching (it's 2 queries in a transaction instead of 1). The long-term
intention is to remove this, though.
Reviewed By: StanislavGlebik
Differential Revision: D28505642
fbshipit-source-id: eefd23bea095c9d0351795ab599beb51bfac373f
Summary:
Like it says in the title, this starts the work of incorporating
BookmarksSubscription into the WarmBookmarksCache. To do so, we start by using
the subscription to initialize bookmarks. Under the hood, this makes the same
SQL query that initializing the cache currently it does (except it does it in a
transaction to also get the corresponding log id).
Reviewed By: StanislavGlebik
Differential Revision: D28505643
fbshipit-source-id: 65c25acfd91bf36809602d75c55e5f74c1c54015
Summary:
This introduces a mechanism to query bookmark updates from the bookmarks update
log instead of via the bookmarks table. The end goal is to use this in the warm
bookmarks cache (WBC).
Note that the WBC cannot currently work if we just give it updates. For
example, if we give it an updated bookmark, then the update fails, the WBC
won't attempt to update it again. So, we need to maintain a view of all
bookmarks, and expose that to the WBC (instead of exposing a stream of
changes). Historically, the WBC has just materialized this from SQL every
second when it needs it, so there's not much difference in terms of peak
resource usage here anyway.
Note that we are missing some information in our update log to be able to
maintain bookmarks just from that (we don't really track what kind of update it
was, we need to infer that, and we're missing changes of kind). Conversely,
we're missing some information in the bookmarks table itself (if a bookmark
gets deleted there is no row we can use to see that).
The way this works is that we query the bookmarks update log and join with the
bookmarks table to find matching rows (which we can do via the `log_id` we
insert there). This gives us the bookmark kind and cs id. It also gives us
which bookmarks were deleted: we just won't find a match.
Note that individual queries in MySQL (or transactions with repeatable read)
don't experience tearing, so as long as the bookmark and its log entry were
inserted in the same transaction, we should find both here.
Reviewed By: StanislavGlebik
Differential Revision: D28478398
fbshipit-source-id: 1a307a90148bb60b50fa6008dff8f24ec8e1d5ea
Summary:
Like it says in the title, this adds support for querying log ids for
bookmarks. This lets us have test coverage on the log ids that we
started inserting in the previous diff.
Reviewed By: farnz
Differential Revision: D28444851
fbshipit-source-id: 10206a463d66eccc668ee18bf93dec25323357b3
Summary:
Like it says in the title. The API between Bytes 1.x has changed a little bit,
but the concepts are basically the same. This ports the implementations to
Bytes 1.x, and keeps the 0.5 traits as a compatibility shim.
Reviewed By: dtolnay
Differential Revision: D28537967
fbshipit-source-id: 59ba165078eaedf2d8c891b5b3e759ab2f942323
Summary: Remove the type alias as its not needed.
Reviewed By: krallin
Differential Revision: D28475479
fbshipit-source-id: 2bf976bf96561e24d4fab540f2673d50ab180a77
Summary:
Adds ScrubWriteMostly::PopulateIfAbsent mode to scrub blobstore. This saves a read from write mostly stores when scrubbing to a mostly empty write mostly store, which makes it quicker.
The destination store is written in PutIfAbsent mode, so if there were existing keys their contents are preserved.
Reviewed By: farnz
Differential Revision: D28473896
fbshipit-source-id: c5328ace4b5addf0e21c671096435d0652fff32e
Summary: Change from an bool to an enum in preparation for adding a third value later in stack.
Reviewed By: farnz
Differential Revision: D28472922
fbshipit-source-id: 5c25e7e03f1d29eff0455282e43e359efaf9b942
Summary: LFS can reload config dynamically and can do it per-region. Let's do this instead rolling this change on the VIP, because here a rollback could happen in no time.
Reviewed By: krallin
Differential Revision: D28534346
fbshipit-source-id: 131ef2d0d084b27719a9aaf4e29003b06c578a67
Summary: This will force future modification to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544422
fbshipit-source-id: 5ecc08a21e49535a3ed4ae9c33f219080365e609
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544427
fbshipit-source-id: 0ccfe675818a96ee8fc702317ace85d0b26ccc26
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544421
fbshipit-source-id: 67d15e4d6ed13c139adffa7cedf7795856d87c7e
Summary: This will enforce future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544426
fbshipit-source-id: d502de1e0e4c3f7910e0796b2c82ebbb9992be22
Summary:
This adds the few missing types. This will ensure that future modifications of
this file are all type checked.
Reviewed By: fanzeyi
Differential Revision: D28544424
fbshipit-source-id: 4053ede8a2901928e90c484158777aab31937600
Summary: That file was already had good types.
Reviewed By: fanzeyi
Differential Revision: D28544420
fbshipit-source-id: b1b02b7d116ca382001c318fd0626c8d50480013
Summary:
This will enable future modifications of this file to always be properly type
checked.
Reviewed By: fanzeyi
Differential Revision: D28544423
fbshipit-source-id: 73e2fe9ba386c7a0b6165424c383db7d647abcc0
Summary:
This fixes the handful of errors and warnings that Pyre threw when I tried
making this file strict. Enabling pyre strict will force future addition and
modifications to this file to have type checking, increasing the reliability of
the code.
Reviewed By: fanzeyi
Differential Revision: D28543212
fbshipit-source-id: 06b976a37e6f4c3a0bab2eb3ecc4c217497bbbd5
