Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
Previously we were using two step process of verifyng working copy - we fetched
all manifests and compared filenodes. If filenodes are different we also
fetched content ids. This was harder to read and it also was slower.
Instead let's use fsnodes instead - they can give filetypes and content ids
directly. Also move all filenode related code to commit_validator, since it's the only place that uses it
Reviewed By: krallin
Differential Revision: D26020705
fbshipit-source-id: 9c598b4ac4950c316195dd10952ee0a785854538
Summary:
I'm going to change how we do working copy validation in the next diffs,
however these function will still be useful. So let's keep them.
Reviewed By: krallin
Differential Revision: D26020704
fbshipit-source-id: 32f527cf821de692529d2d23a3cb4b7e90940a32
Summary:
Without checking response status codes, we don't really know whether our request got processed as expected.
I am also adding port to url because default path doesn't have it. HTTPS is 443 by default.
Reviewed By: johansglock
Differential Revision: D26046197
fbshipit-source-id: c6082cc221c44c6c06557bccaaebb53efd56b4bf
Summary:
A bit of background: some time ago I landed an optimization to getbundle, which
split the commits user want to fetch into "high generation numbers"
(which means commits are close to main bookmark) and "low generation numbers"
(commits that are far away from the main bookmark). User can fetch "low
generation numbers" if e.g. a commit to an old release branch was landed.
Processing them separately can yield significant speedups.
Previously we chose a threshold statically e.g. if a commit has generation
number lower than X then it's considered to have a low generation number.
Threshold was chosen arbitrarily and relatively small [1], and it didn't work
that well for commits that were above this threshold, but still not close
enough to main bookmark - we still see cpu spikes.
Instead let's define a commit as having low generation number if it's >X
commits farther from the commit with the highest generation number.
Reviewed By: ahornby
Differential Revision: D25995936
fbshipit-source-id: 57eba4ba288114f430722266a8326183b3b1f0bd
Summary:
In the next diff i'm going to tweak how we decide if low generation number is
low or not [1]. To make it easier to add this tweak let's put the logic of
figuring out low generation number in a struct.
[1] If generation number is low then we run a few heuristics to speed up the
getbundle request
Reviewed By: ahornby
Differential Revision: D25995938
fbshipit-source-id: dbb95b4321d5a4caa13c4183882e90b23020503c
Summary: Using speedtest http endpoint on mononoke when client is connecting directly to mononoke.
Reviewed By: markbt
Differential Revision: D25899297
fbshipit-source-id: b9e1fb90080c42cb2bd242a1cb3612c63a8f93a4
Summary:
This new crate is part of the new telemetry / logging effort. Its input is
tracing data, and output is aggregated NoSQL table content.
This diff is only the start, setting up the direction.
Reviewed By: DurhamG
Differential Revision: D19797702
fbshipit-source-id: bdf34461c05b5eae5e59652bc82d8ee1857dbf1e
Summary:
Schema for `xdb.mononoke_blobstore_sync_queue : blobstore_sync_queue` wasn't altered yet as AOSC [wasn't working](https://fb.workplace.com/groups/mysql.users/permalink/4991036464278262/).
This will allow the blobstore healer to make decisions about how many blobs it should copy at once. The goal is to eliminate the OOMs that were regularly seen by giving the blobstore healer a memory budget.
Reviewed By: krallin
Differential Revision: D25925522
fbshipit-source-id: d3714dbadc74274a4c4a0e66fa732b84bef89227
Summary: Suddenly prompt stopped appearing for me. Flush the stream to be sure that it's printed out.
Reviewed By: HarveyHunt
Differential Revision: D25956018
fbshipit-source-id: 83419037fa6ce672e203385b71f1403a738d0c90
Summary: Its easier to distinguish the parameters when constructing it directly rather than via a new() method.
Differential Revision: D26017347
fbshipit-source-id: a020db1133de727f217f67a05953059122e3623a
Summary: Its only called from inside MononokeAppBuilder so move it inside to save passing all the struct members as params.
Differential Revision: D25976405
fbshipit-source-id: e95d7b8f5f4474f0289d29bb7bb0a8b0780112e0
Summary: This doesn't need to be in metaconfig anymore, can move it to multiplexedblob
Reviewed By: krallin
Differential Revision: D25928061
fbshipit-source-id: 8aa6ce6aafa16f84730cf388ebf7eab6d5bf2c53
Summary:
We have a few tricky opimizations, so it's better to have more logging than
less.
Reviewed By: HarveyHunt
Differential Revision: D25995937
fbshipit-source-id: b5502708125b70f3d656be3dc1120176f5c76ce8
Summary:
The test added in previous diff showed that hg filenodes weren't being deferred between chunks in the expected way.
This is because we can't tell if a hg filenode is in a given chunk until it is loaded. This is similar to unodes, but the linked changeset in this case isHgChangesetId rather than the bonsai ChangesetId, so this change introduces hg_to_bcs mapping in the walker state, which is used for looking up whether the filenodes linked HgChangesetId is in the chunk, and if not defers the edge.
Reviewed By: krallin
Differential Revision: D25742276
fbshipit-source-id: 1f92452d012aab5b9fdf29f43fc05ebc043b2c7a
Summary: Add a test for hg filenode chunking showing its not deferring any edges to the second chunk, which is a problem. The fix is in the next diff.
Reviewed By: krallin
Differential Revision: D25742278
fbshipit-source-id: bafd59cef01c3153eb1beadccb6026d456454d6b
Summary:
Add test for walking hg non-filenode data in chunks. Expect some deferred edges to next chunk as parents point into history.
Done by deferring hg_changeset_via_bonsai_step if the bonsai is outside range of the chunk
Reviewed By: krallin
Differential Revision: D25742288
fbshipit-source-id: 385c9261151d10f7a7029f86ec10470226fc993c
Summary: Add test for walking fsnodes in chunks. Fsnodes don't point into history, so not expecting any edges to be deferred between chunks.
Reviewed By: krallin
Differential Revision: D25742291
fbshipit-source-id: dfacbffd1640713df0bc80e9306210860f9a932c
Summary: Add test for walking skeleton manifest in chunks. These manifests don't point into history, so no edges expected to be deferred between chunks
Reviewed By: krallin
Differential Revision: D25742290
fbshipit-source-id: 0bee980940d3f023392a518174aae0352d5eebda
Summary:
Add test to walk deleted manifests in chunks, no deferring expected as these manifests don't point into history.
Test showed was missing handling for this manifest type in chunking so fixed it.
Reviewed By: krallin
Differential Revision: D25742285
fbshipit-source-id: 5411f904510f9b4fd9028c7d0dde6c652a784796
Summary: Add test for walking blame in chunks, check that edges crossing chunk boundary are deferred.
Reviewed By: StanislavGlebik
Differential Revision: D25742296
fbshipit-source-id: 163b07df57ebb1c745ee0577f58a45660e6cd18d
Summary: Add test for scrubbing fastlog in chunks, make sure edges get deferred between chunks.
Reviewed By: StanislavGlebik
Differential Revision: D25742277
fbshipit-source-id: 46fd47cfc776783b713717df0ab86bae1b0873fe
Summary:
For Unodes we can't determine before loading them whether they fall within the current chunk as the linked changesetid value is not visibile until the step is executed.
This change adds the ability to defer an already executing step and uses it for unode to defer if its linked changeset is not in the chunk being processed
Deferred edges are stored in the walker state, and are checked on each chunk so that any deferred edges can be run
Reviewed By: StanislavGlebik
Differential Revision: D25742280
fbshipit-source-id: 8a0e7d96b8bf10889bf5e83fe4bee829a1a5cb4c
Summary: Add an enum for the walker step output in preparation for adding a Deferred variant to it in next diff
Reviewed By: StanislavGlebik
Differential Revision: D25742293
fbshipit-source-id: 6aabacb1cd39d16f4d36998908048fd2a10eba4d
Summary: Allow scrubbing of ChangesetInfo in chunks of public commit ids
Reviewed By: markbt
Differential Revision: D25742286
fbshipit-source-id: a5e2faed16eb60c5b7054261a74595a945e68c15
Summary:
For large repos is is desirable to walk them in chunks as a prerequisite for being able to clear state caches to reduced memory usage between chunks and to checkpoint between chunks so that an interrupted scrub can resume.
Chunks are fetched from the repo bounds of changeset id in newestfirst order, this means that we scrub newest data first. Any edges discovered from the walk that point outside the chunk are deferred until the later chunk that covers them.
This change adds chunking and tests if for core bonsai data, following diffs add it for other types.
Reviewed By: StanislavGlebik
Differential Revision: D25742295
fbshipit-source-id: b989abdf2ca367cf9b10f45d9f932eba55ee6dae
Summary: New command line args to allow scrubbing a repo in chunks of N changesets. Used in a later diff.
Reviewed By: StanislavGlebik
Differential Revision: D25742282
fbshipit-source-id: 4bcf74d26f8c2863c6e96f25eca69e01f9c2c0d5
Summary:
The main thing this change does is make sure pending roots to visit are represented in the difference between Walked and Children. Children is the sum of all child nodes discovered, both visited and unvisited. Walked is a measure of number of nodes visited. Children-Walked is used as a measure of queue depth remaining to be processed.
When not chunking this is a minor issue as usually just one bookmark root node is not counted in children, but when chunking not counting the roots means mean the chunk of several 100000 roots is not visible as waiting to be processed.
Reviewed By: StanislavGlebik
Differential Revision: D25852526
fbshipit-source-id: df5f21a37be152f0baee40d33fd7dfb7aaa763de
Summary:
It's no longer true - we're doing metalog commit in transaction.py, not lock
release. Also rename the function to clarify.
Reviewed By: DurhamG
Differential Revision: D25984806
fbshipit-source-id: b17a3f635210be7855341fc8a47fed6411599164
Summary:
This setup is more extendable than the TracingData focused approach. We can
more easily add new functionality using the Subscriber list.
The approach taken here to introduce the new collector tries to maintain
existing functionality. We can then move various logic to their own
Subscribers.
Reviewed By: quark-zju
Differential Revision: D25988580
fbshipit-source-id: 045cd355dbd499109e554a29a1439c2d490b7c40
Summary:
Dot `.` is the common separator for the metrics aggregator that we use.
This adds some form of consistency.
Reviewed By: DurhamG
Differential Revision: D25968398
fbshipit-source-id: 194d2f33fe477fe5d768a9cd8f9f46f56445e3e8
Summary:
We can see number of HTTP errors when working with darkisilon S3 https://fburl.com/scuba/mononoke_blobstore_trace/lt1itidt
While we are still investigating the root cause, it seems most of them are the result of too many connections we are trying to open to the host. Long term solution for that is to make reverse proxy in between with some fixed number persistent connection to isilon. However we can still have some errors and to increase reliability let's make some exponential retry logic.
Reviewed By: krallin
Differential Revision: D25995114
fbshipit-source-id: b19e92933416f0bee20c2fa3235052ee1aa15c89
