Summary:
Some users have reported errors of the form:
```
error.HttpError: [65] Send failed since rewinding of the data stream failed (seek callback returned error 2)
```
These are caused by the fact that we're passing the HTTP request body directly to libcurl in memory rather than via a file, but we haven't implemented the `seek()` method necessary for libcurl to retransmit the data if needed. This diff implements the method.
Reviewed By: DurhamG
Differential Revision: D30654625
fbshipit-source-id: f21a067ad02ee540b86cf2e6eff2c6f08f45a3e4
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
At the moment when segmented changelog is updated and/or reseeded mononoke
servers can pick it up only once an hour (this is a current reload schedule)
or when mononoke server is restarted. However during production issues (see
attached task for an example) it would be great to have a way to force all
servers to reload segmented changelog.
This diff makes it possible to do so with a tunable. Once tunable changes its
value then monononoke servers almost immediately (subject to jitter) reload it.
This implementation adds a special loop that polls tunables value and reloads
if it changes. Note that in theory it could avoid polling and watch for configerator
changes instead, but it would be harder to implement and I decided that it's
not worth it.
Reviewed By: farnz
Differential Revision: D30725095
fbshipit-source-id: da90ea06715c4b763d0de61e5899dfda8ffe2067
Summary:
Previously, extremely large prefetch calls could cause an OOM if the requested files were all fetched remotely from EdenApi, in which case the memory would remain in use until the entire batch had been fetched.
With this change, at most 1000 EdenApi files will be held in memory at once (or 10GB of memory). This is a stop-gap solution, a better approach would be to avoid storing all EdenApi files in memory after a certain amount, or allow the batch fetching implementation to understand we're only prefetching, and thus avoid reading anything back from disk or storing EdenApi files in memory unnecessarily.
Reviewed By: andll
Differential Revision: D30686054
fbshipit-source-id: 022e353760c515961a8956f7958b43f429143971
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary: Keep two versions of fbthrift_ext that one with tokio-0.2 and the other with tokio-1.x. This diff is just renaming.
Reviewed By: dtolnay
Differential Revision: D30558441
fbshipit-source-id: bfe7e96b95529f2745f635190df5118a0cb44014
Summary:
Having the same queue for all three makes the dequeue code overly complicated
as it needs to keep track of the kind of request that needs to be dequeued.
Incidently, the previous code had a bug where request in "putback" would be
requeued at the end of the queue, even though there were at the beginning of it
if they all had the same priorities.
This is theory should also improve the dequeue performance when the queue has a
mix of blobs/tree requests, but I haven't measured.
Reviewed By: genevievehelsel
Differential Revision: D30560490
fbshipit-source-id: b27e5429105c07e5f9eab482c12e5699ca3413f7
Summary:
Since the background condition is before the actual prefetching of files,
specifying the background option would just glob files but not prefetch them
which is equivalent to prefetching all the trees.
Reviewed By: genevievehelsel
Differential Revision: D30618753
fbshipit-source-id: 5533b1c78d614342ac3341ce033795be3850750a
Summary:
It looks like a few scmstore changes landed with warnings (probably fixed higher up in the tree unification stack).
This change fixes those warnings.
Differential Revision: D30686092
fbshipit-source-id: d80625dea64f35683f815b58c83a3e5bb7cbdfa8
Summary:
When we remove a file from a sparse profile and commit the profile, it
should delete the file on disk. There's a bug where it doesn't actually delete
the file. This fixes it by passing the correct commit parents to the refresh
function.
Reviewed By: andll
Differential Revision: D30683677
fbshipit-source-id: 7f012faa99975d8270209f2962e7f9236890daed
Summary:
Use `populate_missing_vertexes_for_add_heads` (added by D27630093 (f138b012e9)) to avoid
excessive lookups for non-master ids that remain in non-master. The function
was used in two other `flush` cases, but missed the id reassignment case. It
works basically by using the "discovery" logic to quickly rule out what's
missing and what's present (ex. if a root is missing in the server graph, then
all descendants of the root is missing).
Reviewed By: andll
Differential Revision: D30700451
fbshipit-source-id: 1f1cd88399dbffd4af75083fef1f3e363a5c60fe
Summary: During reassigning non-master ids, it might trigger too many remote lookups.
Reviewed By: StanislavGlebik
Differential Revision: D30700452
fbshipit-source-id: 2483335e466c3de8a362f7b6a15fc4ba9e2693be
Summary:
Support octopus merge defined in the following format:
- The revlog flag has `1 << 12` set.
- Extra `stepparents` is set to `hexnode1,hexnode2,...` format.
This is mainly used to support revlog from stream clone.
Reviewed By: StanislavGlebik
Differential Revision: D30686450
fbshipit-source-id: d5aa2f18a02f5f0d7aa033210fb4f79b729c0d26
Summary:
Extend the struct so we can support more than 2 parents.
The size of the sturct is now 16 bytes, from 8 bytes. This might have some
performance overhead.
Not using `Box<[u8]>` because that will make the struct 24 bytes.
Reviewed By: StanislavGlebik
Differential Revision: D30686451
fbshipit-source-id: c0f8d0472c7e578f34d771dacecffc91585650c3
Summary:
In `vertex_id_with_max_group(name, group)`, if `group` is master and the `name`
exists in the non-master group, then there is no need to lookup remotely because
a same name (vertex) cannot be present in both master and non-master group. In
that case, just return that the `name` does not exist in the master group.
Reviewed By: StanislavGlebik
Differential Revision: D30699215
fbshipit-source-id: 5170abe719aa7cc31533912e18bc0e21f133e1f4
Summary:
Added a test about excessive remote lookups when flush() reassigns vertexes
from non-master to master.
Reviewed By: StanislavGlebik
Differential Revision: D30699214
fbshipit-source-id: 0547707764855ab9a563178740612b54df4a5fc9
Summary: They are used to narrow down issues related to S242328.
Reviewed By: StanislavGlebik
Differential Revision: D30699216
fbshipit-source-id: 28f4f0bfadadb2dea5510878168c2d7b47a8641c
Summary: Split out the request ID, URL, and HTTP method from `RequestContext` into a new `RequestInfo` struct, which can be cheaply cloned and included in the response returned to the caller. This enables the caller to correlate requests and responses, which is useful when working with many concurrent requests.
Reviewed By: DurhamG
Differential Revision: D30650365
fbshipit-source-id: 68efedcf852c91387450443ebe46062809633f10
Summary:
Make it possible to call the CheckIntegrity APIs from Python such as:
In [1]: cl.inner.checkuniversalids()
Out[1]: []
In [2]: cl.inner.checksegments()
Out[2]: []
In [3]: cl.inner.checkisomorphicgraph(cl.inner, cl.dageval(lambda: heads(mastergroup())))
# take a while
Out[3]: []
Reviewed By: andll
Differential Revision: D30682536
fbshipit-source-id: 23f280bf261def3d20d5f7dc15a48c2fc2d79d77
Summary: This makes other crates easier to implement CheckIntegrity.
Reviewed By: andll
Differential Revision: D30682540
fbshipit-source-id: 4333f37fa7bafe55a8bee9f149b2f23a463c51af
Summary:
Makes the revlog index provides dummy graph integrity checks so it can
be used as a generic object in the Python bindings.
Reviewed By: andll
Differential Revision: D30682542
fbshipit-source-id: 25c6e8640de46188d7bf45a927e11e0779a8ad40
Summary: Make it possible to check a graph against a reference graph.
Reviewed By: andll
Differential Revision: D30682539
fbshipit-source-id: 57db952dcda5656ff6000e9961448c9b64afbaf0
Summary: Make it possible to check segment integrity.
Reviewed By: andll
Differential Revision: D30644243
fbshipit-source-id: 24bb0c8c8c9394d688e3e9320e59268bc2a4ed3f
Summary: Make it possible to check universal ids externally.
Reviewed By: andll
Differential Revision: D30644242
fbshipit-source-id: f312ff59dbdf68e57c5249d57c5d44da0b10e398
Summary: This will be used to verify graph integrity later.
Reviewed By: andll
Differential Revision: D30644244
fbshipit-source-id: 0d22b70121da37c411adf17200a6c752fefa80ad
Summary: This breaks all use of `hg sparse`, because `.hg*` cannot be matched.
Reviewed By: mitrandir77
Differential Revision: D30666349
fbshipit-source-id: c06d1b798a57490f2e5560f178a2839ae5425146
Summary: We've got multiple manifold parameters now, two of which are Option<i64>, so lets create a struct to name them
Reviewed By: HarveyHunt
Differential Revision: D30305462
fbshipit-source-id: 44eee00d478e4485d074a14fcccec2f0f9572ecd
Summary:
This allows to quickly identify the program that emitted the error.
Per user feedback: https://fb.workplace.com/groups/clifoundation/posts/433922134631466
Reviewed By: StanislavGlebik
Differential Revision: D30604611
fbshipit-source-id: 712bc9f466c5a7b5c97a1b83a10fbe277341a300
