Summary:
This is ... a stopgap :( There is probably some slow polling happening in
unbundle_future, and this causes us to fail to use our connection in time in
check_lock_repo...
Reviewed By: ahornby, StanislavGlebik
Differential Revision: D27620728
fbshipit-source-id: b747011405328b60419a99f0e5dbbaf64d53196a
Summary:
This one is a little bit trickier since we want to use Tokio inside a
Quickcheck function. That said, this is basically the expansion `tokio::main`
does, so we can simply use it.
Reviewed By: farnz
Differential Revision: D27619146
fbshipit-source-id: 1e3ea2d119913900d9b55c0a6d33de8a6ed5781c
Summary:
I'd like to just get rid of that library since it's one more place where we
specify the Tokio version and that's a little annoying with the Tokio 1.x
update. Besides, this library is largely obsoleted by `#[fbinit::test]` and
`#[tokio::test]`.
Reviewed By: farnz
Differential Revision: D27619147
fbshipit-source-id: 4a316b81d882ea83c43bed05e873cabd2100b758
Summary:
The intention is that the packer decides what to pack and in what order, PackBlob provides the methods needed to do the packing as requested by a packer.
Change the API so that a packer cannot make mistakes
Reviewed By: ahornby
Differential Revision: D27476427
fbshipit-source-id: 7dd534302c62b2432a2aca474f49da8ab9cbef1a
Summary:
It is useful to have latency stats grouped by the shardmap and label to easily identify where the problem comes from if something is broken.
This diff switches a single histogram used for all the MySQL use-cases into a set of histograms: one per `shardmap:label`. Ans also makes the histograms a bit smaller as we don't actually have such big numbers as 10s per conn/query.
There is only one case when the histogram is created per shard instead of a shardmap, it is `xdb.hgsql` DB with 9 shards. The reason why it happens it's because we connect to each shard as to an individual tier: https://fburl.com/diffusion/um8lt7cr.
{F582699426}
Reviewed By: farnz
Differential Revision: D27503833
fbshipit-source-id: 40c7eb64df7ae0694f63d3644231f240df8212ec
Summary: introduce a way of requesting unhydrated commits using client telemetry
Reviewed By: StanislavGlebik
Differential Revision: D27591868
fbshipit-source-id: 7616f9d3b222c2c7f43c2ba930086eb0ec9e15bc
Summary:
Only used by one test that can define the constaint itself.
The problem with having it on the trait is that it's a bit noisy when
things operate on ToApi at the generic level. It adds to the list of
constaints they these users of the ToApi trait need to add.
Reviewed By: kulshrax
Differential Revision: D27549922
fbshipit-source-id: fff9e513eb4c06862111ce6eecc84ab981eea893
Summary:
This is only used in one utility which can define the constaint itself.
I am looking to simplify the Requirements for ToWire so that we can more
easily iterate on them. Debug as a requirement produces too much noise.
There is the risk of ending up in a scenario where we want to print the Wire
type but it's more practical to annotate structs with derive Debug when that
happens than to add the constaint in the trait.
Reviewed By: kulshrax
Differential Revision: D27549925
fbshipit-source-id: aacf7c1c465c94414be02aa143187897c7084980
Summary:
There is no use for it outside of one test which can describe that constraint
itself.
I think that requiring ToWire and ToApi to use the same objects is too much
for the general use case. We regularly convert between different object types
that are the same logical thing but have different representations. A good
example for that is the HgId. It makes sense to implement ToWire for all HgId
variations.
Reviewed By: kulshrax
Differential Revision: D27549924
fbshipit-source-id: d76d7a4beb528634bed46ae93dbd634d850547e5
Summary:
For async requests, we perform a blocking request in a separate thread, and stream the results back through a channel. However, if the curl handle for the request is dropped before starting the request (for example, because of a configuration error), this function would return a `oneshot::Canceled` error (from the channel consumer) instead of the real error message from the IO thread.
This diff fixes the issue by ensuring that the function waits for and returns the error message from the IO thread in the event that the IO thread returns before starting the request.
Reviewed By: quark-zju
Differential Revision: D27584502
fbshipit-source-id: 8447c158d253c3f28f03fcc4c36a28698fe6e83d
Summary:
This adds command line argument `-I` that supplies \0-separated list of files to add to commit.
Added files can be ignored/untracked.
No limit on total size for now, still waiting to hear from mononoke team on max files size
Reviewed By: quark-zju
Differential Revision: D27547822
fbshipit-source-id: 8bb755db5dd6e557e2752381dbeb5f1035073725
Summary: This will be used in ephemeral commit, since by default it does not need to include untracked files
Reviewed By: quark-zju
Differential Revision: D27580975
fbshipit-source-id: 16c4faa92e9afe472ff1677e5b92507bebaee247
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
* use `std::nullopt`
* TODO about sandcastle_instance_id in opensource version
Reviewed By: chadaustin
Differential Revision: D27575732
fbshipit-source-id: bf76970a15fee5a3dc1e4e411ea70f5af7248496
Summary:
When creating a commit via scs api we need to validate a few things (e.g. that
the file that a commit is trying to delete existed in the parent), and in order
to do that we need to use a manifest. Previously we were using fsnodes
manifests, however fsnodes is the slowest manifest to derive, and that makes
the whole create_commit operation much slower. Let's try to use skeleton
manifest, which is the fastest to derive.
Reviewed By: markbt
Differential Revision: D27587664
fbshipit-source-id: a60cab4956063bf26c0f1ec8c9cfa05233bb3cc0
Summary:
Previously ChangesetPathContext was holding both fsnode_id and unode_id, however it made it easier to misuse api and trigger expensive fsnodes or unodes path traversal (see more info in the comments for D27587664).
This diff splits it in two separate types.
Also I noticed that ChangesetPathContext wasn't using the `unode_id` future that it stored, so I just deleted it.
Reviewed By: markbt
Differential Revision: D27590997
fbshipit-source-id: 08fc14d33c82357275413c4cf2698f97620503ea
Summary: The default limit for commit cloud interactive history should be two weeks, not two days.
Reviewed By: farnz
Differential Revision: D27589697
fbshipit-source-id: 4314621fa7f06dac9243eb9b826acc1c7b0c0b10
Summary:
Hg sync jobs were frequently failing due to the task performing MySQL query being starved.
It acquired a connection but then waited for many seconds until it could finally send a query. At that time the server returned error: the connection was open idle for >12s and now timed out:
```
I0401 11:08:32.085223 390 [main] eden/mononoke/mononoke_hg_sync_job/src/main.rs:355] error without entry
E0401 11:08:32.086126 390 [main] eden/mononoke/cmdlib/src/helpers.rs:336] Execution error: While executing ReadNextBookmarkLogEntries query
Caused by:
0: While making query 'SELECT id, repo_id, name, to_changeset_id, from_changeset_id, reason, timestamp,
replay.bundle_handle, replay.commit_hashes_json
FROM bookmarks_update_log log
LEFT JOIN bundle_replay_data replay ON log.id = replay.bookmark_update_log_id
WHERE log.id > 19896395 AND log.repo_id = 2106
ORDER BY id asc
LIMIT 20'
1: Mysql Query failed: Failed (MyRouter) Idle timeout after 12 seconds see https://fburl.com/wait_timeout
I0401 11:08:32.172088 390 ClientSingletonManager.cpp:95] Shutting down Manifold ClientSingletonManager
remote: pushkey hooks finished (after 0.00s)
Error: Execution failed
```
Link to the full logs in a timeframe: https://fburl.com/tupperware/16th1yk7 (I added a debug output when `ReadNextBookmarkLogEntries` query runs).
Hg sync job initiates an infinite loop to look for the new commits to synchronize. In the async stream it runs `ReadNextBookmarkLogEntries` query and then prepares bundle and synchronizes it. The stream is [buffered](https://fburl.com/diffusion/z1r7648f) by [5 (link)](https://fburl.com/diffusion/surn37hx).
My guess is that the `ReadNextBookmarkLogEntries` query starts executing, while the previously discovered bundles are being prepared. The query opens a connection and then gets switched, now the bundles are being synced. But sometimes those bundles take too long to sync while the query task is waiting till it be executed again.
The sync finishes and the query task finally tries to send a MySQL query but hits an idle timeout error on the server.
This diff:
* Spawns the MySQL query and `apply_bundle` call.
* Adds watchdog on futures to help debug issues if they occur later, although I couldn't see any slow polls in the logs.
Reviewed By: StanislavGlebik
Differential Revision: D27503062
fbshipit-source-id: 6d1d9166b99487c056f3fb217502f8a9d3d46228
Summary:
Some manual scrub runs can take a long time. Provide progress feedback logging.
Includes a --quiet option for when progress reporting not required.
Reviewed By: farnz
Differential Revision: D27588449
fbshipit-source-id: 00840cdf2022358bc10398f08b3bbf3eeec2b299
Summary: D27591073 (a1e2833377) made the histogram smaller, so this is sufficiently fast to call directly.
Reviewed By: krallin
Differential Revision: D27592432
fbshipit-source-id: 50d3d594b237b87cc9d0a90910a6f022b7c40f2a
Summary:
There was no reason for this to be this large, and it's causing issues with
repo construction since it's pretty expensive to construct as a result
(D27501915 (69896e90b5)).
Let's just make it much smaller.
Reviewed By: StanislavGlebik
Differential Revision: D27591073
fbshipit-source-id: 1c986cb922d70b10c39711c57ac9f5899ed7496c
Summary:
This warmer hasn't been used, and we aren't going to use it going further.
Let's remove it
Reviewed By: krallin
Differential Revision: D27533802
fbshipit-source-id: deaaf954ed535789ab6b5dc89b8da9967c40e84f
Summary:
The comment seem to be incorrect - it wasn't checking that all files from a
replaced directory were removed (in fact, we allow to not remove them, see
https://fburl.com/diffusion/wdhu5arg).
Instead it was checking whether a file that was replaced with a directory was indeed
removed.
Reviewed By: markbt
Differential Revision: D27534121
fbshipit-source-id: 4f0d53096c2665fec8bb575ee183411c8edf2ccb
Summary:
This crashes with `ui.ignorerevnum=1`.
Rev number should use `%d`. Or use node and `%n`. `%s` is not the right way.
Reviewed By: kulshrax
Differential Revision: D27527470
fbshipit-source-id: 115385d8bb8dd006fcbf62dee1099b8f9d5262c7
Summary:
Since we're rolling out native checkout and resumable checkout around
the same time, let's make resumable checkout optional so we can turn it off it
causes issues, without turning off native checkout.
Reviewed By: quark-zju
Differential Revision: D27481986
fbshipit-source-id: a0a3e68567ca2a468e852dce95c03c4b606aaf22
Summary: This makes it easier to use.
Reviewed By: kulshrax
Differential Revision: D27406589
fbshipit-source-id: 11bef407ab620859381c6ee952e6ef00494551e1
Summary:
The issue is that `mut i: usize` is no longer shared across multiple `async
move` blocks (introduced by D27308798 (0df4efa969)).
Rewrite the logic to collect `ids` first, then use `vertex_name_batch`
query instead.
Reviewed By: sfilipco
Differential Revision: D27406586
fbshipit-source-id: b41fe3a13114dc34aa5763e6e2bebe0571decc87
Summary:
Merge paths like `x~n` and `x~(n+1)` to `x~n (batch_size = 2)`.
This could be more efficient bandwidth-wise and algorithm-wise.
Reviewed By: sfilipco
Differential Revision: D27406587
fbshipit-source-id: f2a67352ad627945685e33667e8299a2bc652930
Summary: IdSet is more compact. This changes the order a bit.
Reviewed By: sfilipco
Differential Revision: D27339279
fbshipit-source-id: e9b50a47beba081b892eccd7711dbd6ab5c3a886
Summary: This will be used by the next change.
Reviewed By: sfilipco
Differential Revision: D27406591
fbshipit-source-id: fcacc35a9ae8ed96cebb2af804d26d1e5e83ad9e
Summary:
Add a way to flush the overlay map to disk so we can avoid network fetches over
and over.
Reviewed By: sfilipco
Differential Revision: D27406592
fbshipit-source-id: 7086ad665119cc3a0834f533690325c7a2363442
Summary: It will be reused elsewhere.
Reviewed By: sfilipco
Differential Revision: D27406593
fbshipit-source-id: 296cf5f50830bb7285e0cb9c7c15a9b374689819
Summary:
I spent some time thinking about how to flush the "overlay_map" to disk.
It is a bit tricky because the on-disk IdMap might have changed in an
incompatible way. I tried a few ways to verify the on-disk IdMap remains
compatible and didn't find a way that looks good (either slow - calculating
universal_ids, or is not 100% correct in corner cases).
Now I come up with this "just track x~n" idea. It trades memory usage (~2x
overlay_map) for easy-to-verify correctness, and efficient overlay_map
flush.
Reviewed By: sfilipco
Differential Revision: D27406583
fbshipit-source-id: 0b7fb3186a9c15f376c1dc4afe7f0516c25d3dec
Summary: It is not obvious. So let's add more comments.
Reviewed By: sfilipco
Differential Revision: D27406584
fbshipit-source-id: 9ce1215efc1a6d4849180c6693616613c08f2a51
Summary:
A sparse dag does not have full IdMap. Its IdMap only contains "universally known" entries.
Add a basic test about cloning from a sparse clone data and resolve vertex <-> id mapping
on the fly.
Reviewed By: sfilipco
Differential Revision: D27352018
fbshipit-source-id: 4a3f5f50be52e91bf7b2021cdc858bcab9c99e80
Summary:
The `NameDag::flush` API will actually rebuild the graph using a "parent" function.
That is not necessary if we got clone data, and won't work well for a lazy graph
(since the parent function talks about vertex names and some names are missing).
Let's bypass the `flush` function and write data directly in `import_clone_data`.
Reviewed By: sfilipco
Differential Revision: D27352019
fbshipit-source-id: a79569d25d858447b8c5eb86902b8d39ae0429a3
Summary: This will be used in tests.
Reviewed By: sfilipco
Differential Revision: D27343882
fbshipit-source-id: 5a2d94a9f755eed0fc27e5a11093b55c810dc8da
Summary:
Implement logic to export the clone data. This allows us to construct a sparse/lazy
dag via export + import CloneData.
Reviewed By: sfilipco
Differential Revision: D27343885
fbshipit-source-id: 71dc0d31e36876a8b6a8c3d7f3498be3262ce297
Summary:
Clone data can only be imported to an empty Dag and universally known vertexes
should be present in the IdMap.
Reviewed By: sfilipco
Differential Revision: D27343888
fbshipit-source-id: ba150d6afdbe15f0902ec20ff150a70657e24c80
