Summary:
Implement `commit_list_descendant_bookmarks` by iterating over all bookmarks and
checking if the target commit is an ancestor of the bookmark's current target.
Reviewed By: mitrandir77
Differential Revision: D22357988
fbshipit-source-id: e1b1d7387742ba7133370f52c4d36c0b1a77f4e3
Summary:
This is much better than having `ObjectFetchContext` itself owns a copy of `ImportPriority`. We can actually customize how different fetch context manages these priority.
We set all FUSE requests to a higher priority and prefetch requests to a lower priority
Reviewed By: xavierd
Differential Revision: D22342802
fbshipit-source-id: b9c1d0f2ddbc7a5e5d619bc2c2222e5df0e702af
Summary:
This commit makes prefetch requests derived in `TreeInode` assigned to lower priority.
Since prefetch is not running under the original request's context, we create a new `ObjectFetchContext` specifically for this case, and attach it to `prefetchLease` since it is guaranteed to be alive during the span of prefetch.
Reviewed By: chadaustin
Differential Revision: D21872719
fbshipit-source-id: d1495b06031980d7d1c21ecf2a0b47e83fc9672d
Summary:
This commit adds `ImportPriority` to `ObjectFetchContext`. By doing so we can tweak priority for a request at different stage.
This commit also provides a default implementation for the virtual methods in `ObjectFetchContext` so we can create one to carry `ImportPriority` in some specific cases.
Reviewed By: chadaustin
Differential Revision: D21872718
fbshipit-source-id: 6e8cfd84959b368e6fe69fda2baf0debf7a88295
Summary: This commit makes `readdir()` to correct send its `RequestContext` to `EdenDispatcher` method so underlying code can know the current request being processed is from FUSE.
Reviewed By: xavierd
Differential Revision: D21821949
fbshipit-source-id: f41ba912fedbfc040e3c9267aad25e7f33f8e912
Summary:
Currently the `move_bookmark` API needs to get the old bookmark location in order
to move the bookmark. We'll fix that in general later, but for now we need to
make sure the value we use doesn't come from an out-of-date cache (e.g. the
warm_bookmarks_cache), as it may prevent the move from working.
Reviewed By: krallin
Differential Revision: D22358467
fbshipit-source-id: 4d46a6be717644b24663318326fdcd81249481c9
Summary:
The remote configs are non-minified json, so using some compression
should reduce their size tremendously.
Reviewed By: quark-zju
Differential Revision: D22336251
fbshipit-source-id: 77282aae57a8e37e3462f6379dedebd2978a4d0e
Summary: This diff made fetch threshold configurable, so we can change it later as repository size grows.
Reviewed By: fanzeyi
Differential Revision: D22337850
fbshipit-source-id: 4b46420cb4e7164a3f1080279d67fa5f90549cd8
Summary: This diff updated `ObjectStore` to send a `FetchHeavy` event to Scuba when the number of fetching requests of a process has reached 2000.
Reviewed By: fanzeyi
Differential Revision: D22292104
fbshipit-source-id: b7ac48412868216ea960c8681a5fb71c587952bc
Summary:
Bookmark requests that are truncated because the requested limit is reached now return a `continue_after` value, containing the last bookmark that was processed.
Callers can make a subsequent request with the same parameters, but `after` set to the value received in `continue_after` to continue their request where it left off.
Reviewed By: krallin
Differential Revision: D22338301
fbshipit-source-id: 81e398bee444e0960e65dc3b4cdbbe877aff926d
Summary:
Add `commit_list_descendant_bookmarks` which will list all bookmarks that are
descendants of a particular commit.
We will also use this opportunity to complete the implementation of pagination
for regular bookmark listing, so add the appropriate fields to the
`repo_list_bookmarks` request and response structs.
Reviewed By: StanislavGlebik
Differential Revision: D22338300
fbshipit-source-id: defd019795c2a2ac9e5573d58de187c10848397f
Summary:
Add a new parameter, `pagination`, to the `list` method of the `Bookmarks` trait.
This restricts the returned bookmarks to those lexicographically after the
given bookmark name (exclusive). This can be use to implement pagination:
callers can provide the last bookmark in the previous page to fetch the
next page of bookmarks.
Reviewed By: krallin
Differential Revision: D22333943
fbshipit-source-id: 686df545020d936095e29ae5fee24258511f4083
Summary:
Rework the bookmarks traits:
* Split out log functions into a separate `BookmarkUpdateLog` trait. The cache doesn't care about these methods.
* Simplify `list` down to a single method with appropriate filtering parameters. We want to add more filtering types, and adding more methods for each possible combination will be messier.
* The `Bookmarks` and `BookmarkUpdateLog` traits become `attributes` on `BlobRepo`, rather than being a named member.
Reorganise the bookmarks crate to separate out the bookmarks log and transactions into their own modules.
Reviewed By: krallin
Differential Revision: D22307781
fbshipit-source-id: 4fe514df8b7ef92ed3def80b21a16e196d916c64
Summary:
The LIKE pattern used by bookmark prefixes needs to be escaped, otherwise
users looking for bookmarks containing `\`, `_` or `%` will get the
wrong results.
Reviewed By: krallin
Differential Revision: D22336716
fbshipit-source-id: 99b0ad6097f096358e66042752e4d153359935be
Summary:
If the user specifies `always`, they know what they are doing and do not
pollute their terminal with the EnableVtMode warnings.
Reviewed By: markbt
Differential Revision: D22343823
fbshipit-source-id: 7dcff43c3fa1163a061a2173d57799d2cd2ef05b
Summary: We were monitoring the wrong lag so far.
Reviewed By: farnz
Differential Revision: D22356455
fbshipit-source-id: abe41a4154c2a8d53befed4760e2e9544797c845
Summary:
`bulk_add()` method was checking for conflicts correctly i.e. it wouldn't fail
if we try to insert the same mapping twice.
`bulk_add_git_mapping_in_transaction` wasn't doing this check i.e. it would
fail.
This caused us a few problems and this diff fixes them - now
`bulk_add_git_mapping_in_transaction` would do the same checks as bulk_add was
doing previously.
There is another important change in behaviour: if we try to insert two entries, one of them
has a conflict another don't then previously we'd insert the second entry.
Now we don't insert any, arguably that's a preferred behaviour.
Reviewed By: krallin
Differential Revision: D22332001
fbshipit-source-id: 86fff8c23c43eeca0fb36b01b10cdaa73b3ce4ab
Summary:
New Commit Cloud command to list user's own worspaces in Commit Cloud
hg cloud ls
Reviewed By: markbt
Differential Revision: D22308127
fbshipit-source-id: 756c419a9bb3d6f50ddd5b4dd344cd35b9a02d2d
Summary:
Having fbsource's thrid-party crates in a separate folder will make it easier to provide a different manifest for the third-party crates for OSS builds.
In order to not trigger any network calls during internal FB builds you cannot define dependencies with "git" urls. Even if they are being patched up in the same manifest Cargo will still make a call to github. That is why internall the Cargo.toml files in third-party folder will contain dependency definitions that will use "path" dependencies only.
In the next diff a separate set of Cargo.toml files will be provided inside `fbcode/eden/oss` that ShipIt will replace for GitHub with the proper "git" url dependencies.
Note: Why even bother with "git" and "patch" section and not use only "path" dependencies? In OSS eden builds depending on other project - like fbthrift or rust-shed. They use regular "git" dependencies. So if you try to compile Eden that uses "path" dependencies with fbthrift/rust-shed that uses "git" dependencies you will get errors from Cargo saying that e.g. rust-shed's fbthrift_ext depends on "fbthrift" (from git), but the structure X implements trait Y from "fbthrift" (from path) which is not the same as trait Y from "fbthrift" (from git).
Reviewed By: quark-zju
Differential Revision: D22335430
fbshipit-source-id: 9b73f4f0bc09fa79b02488dc9b4640b97f90683c
Summary: Add `Arbitrary` implementations for `DataRequest`, `HistoryRequest`, and `CompleteTreeRequest` so that these types can be used in quickcheck tests.
Reviewed By: quark-zju
Differential Revision: D22344600
fbshipit-source-id: c7fcbcd4648ab45f8dde00cc4bb3c1c4d203c4e1
Summary: The `#[quickcheck]` attribute is a more modern way of defining quickcheck predicates (as opposed to the older `quickcheck!` macro). Update all existing usages in the crate.
Reviewed By: quark-zju
Differential Revision: D22344601
fbshipit-source-id: b7ee4d317a64bed5fcd8b35f90f2544f6b024410
Summary: Replace use of `.ok_or_else(|| anyhow!(...))?` with `.context(...)?`, which is a cleaner way to express the same thing.
Reviewed By: quark-zju
Differential Revision: D22323298
fbshipit-source-id: 9fc1a8183a54ee0c4f30f7497b98005a18a06468
Summary:
Move JSON request parsing code out of `make_req` into `edenapi_types` so it can be reused in the EdenAPI test CLI (added later in the stack).
Note to reviewer: This diff look bigger than it actually is; it is mostly just cut-and-paste (recorded as a copy to preserve history).
Reviewed By: quark-zju
Differential Revision: D22305693
fbshipit-source-id: 13248fb29b2fbb705203f889f3d2fb3c1c760bed
Summary:
EdenAPI's `make_req` tools allows developers to create ad-hoc CBOR request payloads for debugging purposes (e.g., for use with `curl`). The tool generates requests from human-created JSON, which are particularly useful in Mercurial and Mononoke's integration tests.
Later in this stack, the use of this JSON format will be extended beyond just this one tool. As such, it is important that the representation be sufficiently extensible so accommodate future changes to the request structs. In the case of the JSON representation of `DataRequest`, this means changing from an array to a single-attribute object, so that additional fields can potentially be added in the future.
Reviewed By: quark-zju
Differential Revision: D22319314
fbshipit-source-id: 5931bc7ab01ca48ceab5ffd1c9177dd3035b643c
Summary: Use autocargo for all EdenAPI code.
Reviewed By: quark-zju
Differential Revision: D22344400
fbshipit-source-id: 522e82fcc76792ed01ca2cdbfa169c23be5bf38f
Summary:
When rebasing a long stack, it's common to have a long obsoleted (`x`) stack.
The `x` stack is likely duplicated with their successors therefore generally
not that useful. They take a lot of space, making it harder to find useful
commits. Collapse them in smartlog output by only showing their heads and
roots.
This is disabled for automation as VSCode @ FB has issues rendering the
"unstable" commits.
Screenshot of before vs after:
{F242018914}
Reviewed By: markbt
Differential Revision: D22289661
fbshipit-source-id: 1afce073e14abe8c23a05d52d60847fc5e0bfb66
Summary:
The backtrace looks like:
Traceback (most recent call last):
File "edenscm/mercurial/dispatch.py", line 688, in _callcatch
return scmutil.callcatch(ui, func)
File "edenscm/mercurial/scmutil.py", line 147, in callcatch
ui.traceback()
File "edenscm/mercurial/ui.py", line 1462, in traceback
tb = util.smarttraceback(exc[2])
File "edenscm/mercurial/util.py", line 4651, in smarttraceback
reprvalue = _render(value)
File "edenscm/mercurial/util.py", line 4572, in _render
result = repr(value)
File "edenscm/mercurial/context.py", line 100, in __repr__
return r"<%s %s>" % (type(self).__name__, self)
File "edenscm/mercurial/context.py", line 94, in __str__
return short(self.node())
File "edenscm/mercurial/node.py", line 55, in short
return hex(node[:6])
TypeError: 'NoneType' object has no attribute '__getitem__'
Reviewed By: DurhamG
Differential Revision: D22342419
fbshipit-source-id: 0b0df98903b97657acff61b2d02121dfb667be50
Summary:
The subcommands it ran returned bytes then treated them as strings.
Let's decode them to strings.
Reviewed By: quark-zju
Differential Revision: D22339397
fbshipit-source-id: d9d503bd97271c649ad67c74d098b572c1bd7dea
Summary:
The (re)construction process for the IdMap will generate millions of rows
to be inserted in our database. We want to throttle the inserts so that
the database doesn't topple over.
Reviewed By: ikostia
Differential Revision: D22104349
fbshipit-source-id: 73b7c2bab12ae0cd836080bcf1eb64586116e70f
Summary:
Simple implementation that queries the MyAdmin service to fetch replication
lag.
Caching like in sqlblob::facebook::myadmin will probably come in a follow
up change.
Reviewed By: StanislavGlebik
Differential Revision: D22104350
fbshipit-source-id: fbd90174d528ddae4045e957c343e6c213f70d26
Summary:
ReplicaLagMonitor is aimed to generalize over different stategies of fetching
the replication lag in a SQL database. Querying a set of connections is one
such strategy.
Reviewed By: ikostia
Differential Revision: D22104348
fbshipit-source-id: bbbeccb55a664e60b3c14ee17f404982d09f2b25
Summary:
Hardlinks aren't supported on EdenFS, let's not silently allow them since this
can cause unintended behavior. It's possible that this might break some
workflows, but it's better to be strict about this.
Reviewed By: genevievehelsel
Differential Revision: D22250849
fbshipit-source-id: 9f90e6a8e9c9715bb4b925c3d7f952e965683081
Summary:
Errors are usually a sign that we're doing something wrong. Ignoring them can
thus lead to weird behavior, hence let's not ignore them. This may have the
potential to breaking user applications if they were doing something we're not
handling properly, but it was likely that EdenFS was broken in a subtle way. We
should be able to better understand what's wrong and fix it properly then.
Reviewed By: genevievehelsel
Differential Revision: D22250852
fbshipit-source-id: 91daa60065b5621542aee1ca3207d1d7c54f8639
Summary:
When files are moved in and out the repo, the paths being passed in might be
empty. For code clarity, let's treat these as file creation and removal.
We also do not need to actually tell ProjectedFS about removed files, since
ProjectedFS is actually telling us that the file is moved/removed.
Reviewed By: fanzeyi
Differential Revision: D22250851
fbshipit-source-id: a678e58eb9c36d4dcbf1bb41d56b38fef8943284
Summary:
SQLite's `LIKE` operator is case insensitive by default. This doesn't match MySQL, and
also seems like a surprising default. Set the pragma on every connection to make it
case sensitive.
Reviewed By: farnz
Differential Revision: D22332419
fbshipit-source-id: 4f503eeaa874e110c03c27300467ddc02dc9b365
Summary:
Whether a bookmark is publishing or not is not specific to Mercurial - it also affects
whether a commit is draft, so it is interesting to the Bonsai world.
Rename `BookmarkHgKind` to `BookmarkKind` to make this clear.
Since negatives are more awkward to work with, rename `PublishingNotPullDefault` to
`Publishing` and `PullDefault` to `PullDefaultPublishing` to make it clearer that
pull-default bookmarks are also publishing.
We can't rename the database column, so that remains as `hg_kind`.
Reviewed By: StanislavGlebik
Differential Revision: D22307782
fbshipit-source-id: 9e686a98cc5eaf9af722fa62fac5ffd4844967fd
Summary:
If we don't wait on these futures, they might never execute, which may lead to
EdenFS not being aware of files no longer present in the overlay.
For now, just add .get() to these futures, but really, we need to make these be
futures themselves and use continuation style code.
Reviewed By: fanzeyi
Differential Revision: D22326570
fbshipit-source-id: 7584981bec284f35f8d2689a3a5dafd1a111a3d8
Summary:
Blobstore healer has a logic, which prevents it from doing busy work, when the
queue is empty. This is implemented by means of checking whether the DB query
fetched the whole `LIMIT` of values. Or that is the idea, at least. In
practice, here's what happens:
1. DB query is a nested one: first it gets at most `LIMIT` distinct
`operation_key` entries, then it gets all rows with such entries. In practice
this almost always means `# of blobstores * LIMIT` rows, as we almost always
succeed writing to every blobstore
2. Once this query is done, the rows are grouped by the `blobstore_key`, and a
future is created for each such row (for simplicity, ignore that future may not
be created).
3. We then compare the number of created futures with `LIMIT` and report an
incomplete batch if the numbers are different.
This logic has a flaw: same `blobstore_key` may be written multiple times with
different `operation_key` values. One example of this: `GitSha1` keys for
identical contents. When this happens, grouping from step 2 above will produce
fewer than `LIMIT` groups, and we'll end up sleeping for nothing.
This is not a huge deal, but let's fix it anyway.
My fix also adds some strictly speaking unnecessary logging, but I found it
helpful during this investigation, so let's keep it.
The price of this change is collecting two `unique_by` calls, both of which
allocates a temporary hash set [1] of the size `LIMIT * len(blobstore_key) * #
blobstores` (and another one with `operation_key`). For `LIMIT=100_000`
`len(blobstore_key)=255`, `# blobstores = 3` we have roughly 70 mb for the
larger one, which should be ok.
[1] https://docs.rs/itertools/0.9.0/itertools/trait.Itertools.html#method.unique
Reviewed By: ahornby
Differential Revision: D22293204
fbshipit-source-id: bafb7817359e2c867cf33c319a886653b974d43f
Summary:
Besides the obvious typos in the code, it would also never go past the first
\n, and the splitted list first element would be an empty file...
Reviewed By: DurhamG
Differential Revision: D22324297
fbshipit-source-id: 339df3a4e4acc4e0630ffa3a7b0b3d266a20b666
Summary:
To check if dynamicconfigs are being generated or not, let's log the
age of the config to dev_command_timers. We also log the age of the remote_cache
so we can detect if hosts are not fetching data from the server.
Reviewed By: quark-zju
Differential Revision: D22323516
fbshipit-source-id: 1fdeef3eaa5d58566606bfc57d8ad96e752db811
Summary: Move old EdenAPI crate to `scm/lib/edenapi/old` to make room for the new crate. This old code will eventually been deleted once all references to it are removed from the codebase.
Reviewed By: quark-zju
Differential Revision: D22305173
fbshipit-source-id: 45d211340900192d0488543ba13d9bf84909ce53
Summary: This diff adds a new `CborStream` type which wraps a `TryStream` of bytes and attempts to deserialize the data from CBOR as it is received. This allows the caller to begin processing deserialized values while the download is still going on, which is important in situations where a server might return a large streaming response composed of many small entries (as is the case with EdenAPI).
Reviewed By: quark-zju
Differential Revision: D22280745
fbshipit-source-id: 9d7fa52e4e67cf7de6bed37c28e5e7afd69449cd
Summary: It is possible to properly initialize the response buffer without using `once_cell`, so remove it to simplify the code.
Reviewed By: quark-zju
Differential Revision: D22264747
fbshipit-source-id: 5890cac7fa598fc80cfd017eae111c0b9fdc6227
Summary:
Add the methods `send_async` and `send_async_with_progress` to `HttpClient`. These methods provide a `Futures`-based async interface that will make it easy to use the client from async Rust code.
Just like in D22231555, this is built on top of the previously introduced streaming API. When a batch request is sent, the client will start a Multi session on a task in a blocking worker thread using `tokio::task::spawn_blocking`. This means that right now, the implementation is not /truly/ async, but it should be possible to change the implementation in the future to avoid using any blocking I/O without needing to change the public interface.
Since all of the requests are part of the same Multi session, they will all proceed concurrently and, if possible, be multiplexed over the same connection in the case of multiple HTTP/2 requests to the same server (which is going to be our main use-case).
Unfortunately, since libcurl does not have any internal synchronization, ownership of the Multi session needs to be passed to the worker thread, meaning that the Multi handle will be dropped once the requests are complete. This means that connections will not be re-used when these methods are called several times serially. The API should make it obvious to the user that the internal state is not preserved since these methods both consume the `HttpClient` itself.
Reviewed By: quark-zju
Differential Revision: D22251488
fbshipit-source-id: 37caf64024cb12b95df5124379209550899d093d
Summary: In order to maximize the surface area of the client that gets exercised by the `http_cli` binary, make it use the new async interface (even though it is not strictly necessary here). Since the async interface is ultimately built on top of the synchronous interface, sending requests this way will still exercise parts of the synchronous interface too.
Reviewed By: quark-zju
Differential Revision: D22242944
fbshipit-source-id: 80d73cc152d0c38673436457c1e1040e4198095e
Summary:
This diff adds `AsyncResponse`, an asynchronous version of `Response` built on top of the `Streaming` handler along with a new `ChannelReceiver` that forwards the received data into async channels. The result is essentially a shim that makes libcurl usable with async/await syntax.
`ChannelReceiver` currently uses unbounded channels. This should be OK because the space usage of the channel should increase [at most] at the same rate as if we used `Buffered`. Essentially, in the worst case (if nothing was consuming items from the channel), this would have the same behavior as if we had simply used a non-streaming handler.
Reviewed By: quark-zju
Differential Revision: D22231555
fbshipit-source-id: 6ee767355bfce6d400f447ee690d974666751f16
