Summary:
Previously it was hard to tell whether the process were actually responsible
for generating derived data or it was just waiting for it to be generated.
Let's make this distinction clearer.
Reviewed By: johansglock
Differential Revision: D20138284
fbshipit-source-id: 52ae12679db2f61869f048baf2a603b456710a71
Summary:
These comments end up being a source of churn as we roll out D20125635, and anyway are not particularly meaningful after the transformations performed by autocargo. For example:
```
bytes = { version = "0.4", features = ["serde"] } # todo: remove
```
^ This doesn't mean the generated Cargo.toml intends to drop its bytes dependency altogether, but just that will be migrated to a different version that is present in the third-party/rust/Cargo.toml but not visible in the generated Cargo.toml.
Reviewed By: jsgf
Differential Revision: D20128612
fbshipit-source-id: a9e7b29ddc4b26bc47a626dd73bdaa4771ee7b18
Summary:
Since Mononoke's filenodes were migrated to derived data framework
hg_linknode_populated alarm has been firing. The main reason was that there's
now a delay between hg changeset being generated and filenodes being generated.
This diff fixes it by making sure walker won't visit hg changesets without
generated filenodes (note that walker will visit these changesets later after filenodes will be
generated).
Reviewed By: ahornby
Differential Revision: D20067615
fbshipit-source-id: 285e9a3d8c89b85441491c889a8458c86ca0e3a8
Summary:
There is no need to generate expensive file history stream if only one node is requested.
I refactored code that generated stream of history commits, so it'd first yield the nodes and only then prefetch their parents. That will help to solve latency problem for the history request for only a single commit.
I removed BFS queue and added two state variables: ready nodes and already processed:
* The last are the nodes that were return as a part of a history stream on the last iteration and now can be used to construct next BFS layer: prefetch fastlog batches, fill the commit graph, take parents in BFS order to form new bunch of nodes.
* First are used if it's the first iteration - there is no processed nodes yet but there are some that are ready to be returned.
I believe removing the queue I simplified the code and logic a little bit.
Reviewed By: StanislavGlebik
Differential Revision: D19818100
fbshipit-source-id: c30d28c623464ba3552a00e8542552f7655076ef
Summary:
During our tests we noticed that we can send too many blobstore read requests to the
mapping. Let's add exponential backoff to prevent that
Reviewed By: ikostia
Differential Revision: D20116043
fbshipit-source-id: 6fecbda4c36a5065b77ba9df561c6d9c6a969089
Summary:
Memcache doesn't care (because both old and new Bytes to `Into<IOBuf>`), but
Thrift is Bytes 0.5. We have our caching ext layer in the middle, which wants
Bytes 0.4. This means we end up copying things we don't need to copy.
Let's update to fewer copies. I didn't update apiserver, because a) it's going
away, and b) those bytes go into Actix, and Actix isn't upgrading to Bytes 0.5
any time soon! Besides, this doesn't actually need updating besides tests anyway.
Reviewed By: dtolnay
Differential Revision: D20006062
fbshipit-source-id: 42766363a0ff8494f18349bcc822b5238e1ec0cd
Summary: Moving `compat` one level down to the call sites of subcommand functions.
Reviewed By: farnz
Differential Revision: D20085398
fbshipit-source-id: 461e147d2ae6e560b3a75fb92fa6b23f9f54d13e
Summary:
During S196197 lease expired and we were rederiving the same derived data over and over again for a big commit.
this diff adds lease renewal that should help with this problem.
Reviewed By: HarveyHunt
Differential Revision: D20093323
fbshipit-source-id: d139abf6659722f47ea40d9b2f279daa03623ff4
Summary:
fetch_root_filenode is called by FilenodesOnlyPublicMapping to figure out if
filenodes were already derived. Previously it first derived hg changeset and
then fetched looked up root manifest in db. However if hg changeset is not
derived then filenodes couldn't possible be derived either and we can return an
answer faster.
This is useful in the next diff where I change walker
Reviewed By: ahornby
Differential Revision: D20068819
fbshipit-source-id: 17f066c437e0b1f7bbeb8f6e247eadc9afe94f90
Summary:
The blobstore_healer has never waited for MyRouter before querying for slave
status, but it ended up implicitly working because creating a blobstore
required a SQL factory, and creating a SQL factory would result in waiting for
MyRouter.
Now that creating a blobstore doesn't require SQL factory unless you're going
to actually use it (which the healer isn't: it doesn't use a multiplexblob, it
uses the underlying blobstores instead), we no longer wait properly for
MyRouter, so if MyRouter isn't there when we boot, we crash.
This fixes that.
Reviewed By: ahornby
Differential Revision: D20094829
fbshipit-source-id: 82b7e8d893a01049d1f434ee8dff36a877a0d2f4
Summary:
Add support for loading by GitSha1 Aliases. This relies on the change to
Alias::GitSha1 earlier in stack.
Reviewed By: ikostia
Differential Revision: D19903577
fbshipit-source-id: 73cdccc04af61fa524c3683851d8af9ae90d31dc
Summary:
This should give us a slightly better idea of what hosts are doing to
troubleshoot duplicate derivation.
Also, let's make the logging a bit less confusing.
Reviewed By: StanislavGlebik
Differential Revision: D20070619
fbshipit-source-id: 91cc264b7043b8fc8c21c007832fba328ef0017d
Summary:
This updates our multiplexed blobstore configuration to carry its own DB
config. The upshot of this change is that we can move the blobstore sync queue
(a fairly unruly table) to its own DB.
Another nice side effect of this is that it cleans up a bunch of other code, by
finally decoupling the blobstore config from the DB config. For examples,
places that need to instantiate a blobstore can now to do even without a DB
config (such as wireproto logging).
Obviously, this cannot land until we update the configs to include this. I'll
do so in Configerator prior to landing the diff.
Reviewed By: HarveyHunt
Differential Revision: D19973905
fbshipit-source-id: 79e4ff92cdb989aab4532decd3fe4fd6c55e2bb2
Summary:
I'd like to refactor our multiplex blob to store its DB using a different
shard. In preparation of doing so, let's:
- Extract parsing DB configs from storage configs
- Tidy up some related places that take a reference when they actually need
ownership (which is sort of wasteful).
Reviewed By: StanislavGlebik
Differential Revision: D19973906
fbshipit-source-id: 82baceb892e9e24e5fd0349ffa5503884c177a7a
Summary: Now it no longer depends on mononoke_types, we can build it with cargo
Reviewed By: krallin
Differential Revision: D20070438
fbshipit-source-id: 1b2f9cc3640c58fd38e962c7c738d08cbb22a71d
Summary:
The bytes 0.5 is a depencency of newer tokio, it's also newer, and thus better.
Staying on 0.4 means that copies between Bytes 0.4 and 0.5 need to be done,
this will be especially bad in the LFS code since 10+MB buffer will have to be
copied...
One main API change is for the configparser. The code used to take Into<Bytes>
for the keys, I switched it to AsRef<[u8]>.
For hg_memcache_client, an extra copy is performed to build a Delta, since this
code uses an old tokio, and is being replaced right now, the effort of
switching to a new tokio and new bytes was not deemed worth it, the copy will
do for now.
Reviewed By: dtolnay
Differential Revision: D20043137
fbshipit-source-id: 395bfc3749a3b1bdfea652262019ac6a086e61e0
Summary:
This is the second (and last) step on removing RocksDB as a blobstore.
Check the task for more description.
Context for OSS:
> The issue with rocksblob (and to some extent sqlite) is that unless we
> introduce a blobstore tier/thift api (which is something I'm hoping to avoid
> for xdb blobstore) we'd have to combine all the mononoke function like hg,
> scs, LFS etc into one binary for it to have access to rocksdb, which would be
> quite a big difference to how we deploy internally
(Note: this ignores all push blocking failures!)
Reviewed By: farnz
Differential Revision: D20001261
fbshipit-source-id: c4b2b2a393b918d17680ad483aa1d77356f1d07c
Summary:
Add option to start the roots of the walk from any graph node, rather than just bookmarks.
This is useful when reproducing issues loading a key, validating a changeset/filenode etc, or to get consistent results on things like sizing where specifying root by bookmark would result in changes between runs.
Reviewed By: farnz
Differential Revision: D19886707
fbshipit-source-id: b7361cbec894aba08b6f702ff0731b9b201224d3
Summary:
Add `scsc export`. Analogous to `svn export`, this exports the contents of a
directory within a commit to files on disk, without a local checkout.
Reviewed By: mitrandir77
Differential Revision: D20006307
fbshipit-source-id: 5870712172cd8a030e85dbff75273c28ab0c332c
Summary:
Mercurial wishes to use this crate, but pulling in mononoke_types brings way
too many dependencies. Since the only reason mononoke_types is brought in is
for the Sha256 type, let's just hardcode it to [u8; 32].
Reviewed By: krallin
Differential Revision: D20003596
fbshipit-source-id: 53434143c61cd1a1275027200e1149040d30beae
Summary:
This hook was implemented to prevent incorrect users from moving a
bookmark. However, it doesn't work and the functionality is now implemented by
`is_allowed_user` in the pushrebase pipeline.
Remove the unused hook.
Reviewed By: johansglock
Differential Revision: D20030479
fbshipit-source-id: bcbc9508eebe77cffbc7936382ba4d345b76f74f
Summary:
We're working towards sharding Bonsais. Let's make them easier to cache by also
not allowing arbitrarily large commit messages.
Reviewed By: StanislavGlebik
Differential Revision: D20002994
fbshipit-source-id: b2319ac9d5709e968121d4299396e03a90df4a06
Summary:
Let's populate the bonsai<->git mapping on pushrebase of the commits that are
coming from git. By this being a pushrebase hook we can have the accuare mappings
being available as soon as the bonsai commit is available.
Corresponding configerator change: D19951607
Reviewed By: krallin
Differential Revision: D19949472
fbshipit-source-id: b957cbcdd0f14450ceb090539814952db9872576
Summary: During the pushrebase hook phase we'll need to reuse existing transaction.
Reviewed By: krallin
Differential Revision: D19949473
fbshipit-source-id: 7c53308724bec6df6d40933405f703c86be15a7a
Summary:
By having it in blobrepo we can ensure that all parts of mononoke can access it
easily
Reviewed By: StanislavGlebik
Differential Revision: D19949474
fbshipit-source-id: ac3831d61177c4ef0ad7db248f2a0cc5edb933b1
Summary:
We need a table to store git<->bonsai mappings and a crate that would abrstract operations on it:
* it's going to be useful immediately to store git hashes for configerator
commits and doing the hash translations via SCS.
* it's going to be useful further down the line for real git support.
NOTE: I'm explicitly using the name `SHA1` all over the place to minimize the
confusion if we'll ever want to support other hashing schemes for git commits.
(Git Community is working on SHA256 support nowdays).
The corresponding AOSC diff: D19835975
Reviewed By: krallin
Differential Revision: D19835974
fbshipit-source-id: 113640f4db9681b060892a8cedd93092799ab732
Summary:
Add the `--parent` flag to `scsc blame`. This runs blame against the first
parent of the specified commit, rather than the commit itself. This allows
users to copy and paste commit hashes from previous blame output in order to
skip the commit, rather than having to look up the parent commit hash
themselves.
Reviewed By: StanislavGlebik
Differential Revision: D20006308
fbshipit-source-id: d1c25aad8f236fe27e467e29f6a96c957b6c8c8f
Summary:
The former implementation here was a little difficult to work with, and
resulted in a whole lot of cloning of closures, etc.
This updates the implementation to be a little simpler on the whole (async /
await is nicer for while loops, since you can use, well, loops)
It does slightly change a few parts of the behavior:
- The old implementation would wait for the replication lag duration. That's
not really correct. As we've observed several time this weeks, replication
lag usually drops quickly once it starts dropping. I.e. if the replication
lag is 10 seconds, it doesn't take 10 seconds to catch up. This gets more
important with big lag durations.
- I updated replication lag to be u64 instead of usize. usize doesn't really
make sense for something that has absolutely nothing to do with our pointer
size.
I also split out the logic for calculating how long we wait in a part that
cares about whether we are busy and one that cares about replication lag
(whereas the older one kinda mixed the two together). We wait for our own
throttling (i.e. sleep for a sec if we didn't do anything) before we wait for
replication lag, so the new behavior should have the desired behavior of:
- If we don't have much work to do, we sleep 1 second between each iteration
(but if we do have work, we don't).
- No matter what, if we have replication lag, we wait until that passes before
doing any work.
The old one did that too, but it mixed the two calculations together, and was
(at least in my opinion) kinda hard to reason about as a result.
Reviewed By: StanislavGlebik
Differential Revision: D19997587
fbshipit-source-id: 1de6a9f9c1ecb56e26c304d32b907103b47b4728
Summary:
We had crahsloops on this (which I'm fixing earlier in this stack), which
resulted in overloading our queue as we tried to repeatedly clear out 100K
entries at a time, rebooted, and tried again.
We can fix the root cause that caused us to die, but we should also make sure
crashloops don't result in ignoring lag altogether.
Also, while in there, convert some of this code to async / await to make it
easier to work on.
Reviewed By: HarveyHunt
Differential Revision: D19997589
fbshipit-source-id: 20747e5a37758aee68b8af2e95786430de55f7b1
Summary:
Our blobstore_sync_queue selects entries with a limit on the number of unique
keys it's going to load. Then, it tries to delete them. However, the number of
entries might be (much) bigger than the number of keys. When we try to delete
them, we time out waiting for MySQL because deleting 100K entries at once isn't
OK.
This results in crashlooping in the healer, where we start, delete 100K
entries, then time out.
This is actually double bad, because when we come back up we just go wihhout
checking replication lag first, so if we're crashlooping, we disregard the
damage we're doing in MySQL (I'm fixing this later in this stack).
So, let's be a bit more disciplined, and delete keys 10K at a time, at most.
Reviewed By: HarveyHunt
Differential Revision: D19997588
fbshipit-source-id: 2262f9ba3f7d3493d0845796ad8f841855510180
Summary:
MyRouter needs to be told which shards to watch. Since I'm adding a new shard,
it'll be easier for everyone to know that they need to update their MyRouter
configuration if we start logging the shard name we're trying to hit.
Reviewed By: ikostia
Differential Revision: D20001704
fbshipit-source-id: 8a9ff3521bc7e3c9b7ed39c6ae33d0ddc1d467b7
Summary:
This adds a file hook to limit the file length we are willing to allow in
commits. This is necessary for now since Mercurial does have a limit on its
end, and we shouldn't allow commits that we cannot sync to Mercurial.
Reviewed By: HarveyHunt
Differential Revision: D19969689
fbshipit-source-id: 1da8a62d54e98b047d381a9d073ac148c9af84b0
Summary:
This adds some basic logging for input size for Gettreepack and Getpack. This
might make it easier to understand "poison pill" requests that take out the
host before it has a chance to finish the request.
Reviewed By: StanislavGlebik
Differential Revision: D19974661
fbshipit-source-id: deae13428ae2d1857872185de2b6c0a8bcaf3334
Summary:
I'm going to modify it in the next diff, so let's make it async.
Note that we used `spawn_future()` before which I replaced with tokio::spawn()
here. It's not really clear if we need it at all - I'll experiment with later.
Removing it will make the code cleaner.
Reviewed By: krallin
Differential Revision: D19973315
fbshipit-source-id: cbbb9a88f4424e6e717caf1face6807ab6c32438
Summary: Not very valuable, if it just prints the constant name.
Reviewed By: StanislavGlebik
Differential Revision: D19978690
fbshipit-source-id: ae2b648f50098b479cb3719fd9b9d4b82bac3d3c
Summary:
bonsai_verify occasionally visits the same commit twice (I found out by adding
logging and noting that it occasionally visits the same commit twice). Let's
allow this here.
Reviewed By: StanislavGlebik
Differential Revision: D19951390
fbshipit-source-id: 3e470476c6bc43ffd62cf24c3486dfcc7133de6c
Summary: We're about to start adding more handlers to the server. Rather than putting them all in the same file, let's create a submodule for them.
Reviewed By: krallin
Differential Revision: D19957012
fbshipit-source-id: 38192664371f0b0ef5eadb4969739f7cb6e5c54c
Summary: Add a `RequestContext` type that stores per-request state, along with a `Middleware` implementation that injects a `RequestContext` into Gotham's `State` object for each request. This is essentially a stripped-down version of the `RequestContextMiddleware` used in the LFS server. Given that the RequestContext contains application-specific functionality, this Middleware lives alongside the rest of the EdenAPI server code rather than in the `gotham_ext` crate (where all of the generic Middleware lives).
Reviewed By: krallin
Differential Revision: D19957013
fbshipit-source-id: 6fad2b92aea0b3662403a69e6a6598e4cd26f083
Summary:
Currently if derivation of a particular derived data type is disabled, but a
client makes a request that requires that derived data type, we will fail with
an internal error.
This is not ideal, as internal errors should indicate something is wrong, but
in this case Mononoke is behaving correctly as configured.
Convert these errors to a new `DeriveError` type, and plumb this back up to
the SCS server. The SCS server converts these to a new `RequestError`
variant: `NOT_AVAILABLE`.
Reviewed By: krallin
Differential Revision: D19943548
fbshipit-source-id: 964ad0aec3ab294e4bce789e6f38de224bed54fa
Summary:
Prepare configs locally that can be passed to any Mononoke binary where things
/just work/.
Reviewed By: HarveyHunt
Differential Revision: D19952512
fbshipit-source-id: 14a3b520972b0bdf4fa7810805066ba746bbef1a
Summary: Adds the Cargo.toml files for blobstore, this is a step towards covering mononoke-types, so only the blobstore traits are covered by this diff.
Reviewed By: aslpavel
Differential Revision: D19948739
fbshipit-source-id: c945a9ca16ccceb0e50a50d941dec65ea74fe78f
Summary:
- Pushing .compat down from main into run function and switch to 0.3 timed function
Note: Possible next level of pushing down: pushing .compact into derive_fn and get rid of BoxFuture run's signature.
Reviewed By: ikostia
Differential Revision: D19943392
fbshipit-source-id: 65bd84492855d3e2e560299a586af6dd4fe9c3ea
Summary: Add the max_jitter_ms field to the rate limiting config struct, and to the integration test.
Reviewed By: HarveyHunt
Differential Revision: D19905068
fbshipit-source-id: b44251c456a45bc494d1080e405f2d009becc0d2
Summary:
This is required for 0.2 timers or runtime reliant code to work within the sync
job. To achieve this, we need to get of Tokio 0.1 fs code, which is
incompatible with Tokio 0.2 because it uses `blocking()`.
Reviewed By: ikostia
Differential Revision: D19909434
fbshipit-source-id: 58781e858dd55a9a5fc10a004e8ebdace1a533a4
Summary:
This update the warm_bookmarks_cache's constructor to use the passed in
blobrepo's derived data configuration (instead of whatever the caller is
passing in), since we now have that information.
Reviewed By: HarveyHunt
Differential Revision: D19949725
fbshipit-source-id: 575a1b9ff48f06003dbf9e0230b7cca723ad68f5
Summary: Add hash::GitSha1 as a pure hash-only key for git Aliases, so one no longer needs to know the size or type to load by Alias::GitSha1.
Reviewed By: krallin
Differential Revision: D19903578
fbshipit-source-id: bf919b197da2976bf31073ef04d12e0edfce0f9b
Summary:
Rename GitSha1 to RichGitSha1 in preparation for introducing hash::GitSha1 as a pure sha1 without extra fields in next in stack.
Motivation for this is that currently one can't load content aliased by Alias::GitSha1 give just the hash, one has to know the type and size as well.
Once the next couple stack are done we will be able to load via just the git hash.
Reviewed By: krallin
Differential Revision: D19903280
fbshipit-source-id: ab2b8b841206a550c45b1e7f16ad83bfef0c2094
Summary:
When max concurrency is 1, we should process at most one request concurrently,
not 2! This had resulted in a flaky test since we're processing traffic out of
order there.
Reviewed By: HarveyHunt
Differential Revision: D19948594
fbshipit-source-id: 00268926095fdbbfdfd5a23366aafcfb763580f4
Summary:
It would be better to make the underlying implementation faster for full hash
cases than check when it is used.
Reviewed By: krallin
Differential Revision: D19905033
fbshipit-source-id: 2d9a77099dc614e80fdb1c0ee715c576a56ba09c
Summary:
Right now, Mononoke code in apiserver executes on Actix's runtime. That's a 0.1
runtime, which means that we're calling into code that might just fail if e.g.
it uses Tokio 0.2 timers.
This is a pretty big footgun, so let's fix it. As it turns out, we already have
a Tokio compat runtime in process, which is (was — this is mostly in SCS now)
used for Thrift calls.
So, let's use that runtime to call into Mononoke code. This ensures we don't
get any nasty surprises of the panicky kind at runtime.
Reviewed By: markbt
Differential Revision: D19902538
fbshipit-source-id: d9d7307b8cf75c3e7e1ecf04c0e10076b3eaef3d
Summary:
This allows code that is being exercised under async_unit to call into code
that expects a Tokio 0.2 environment (e.g. 0.2 timers).
Unfortunately, this requires turning off LSAN for the async_unit tests, since
it looks like LSAN and Tokio 0.2 don't work very well together, resulting in
LSAN reporting leaked memory for some TLS structures that were initialized by
tokio-preview (regardless of whether the Runtime is being dropped):
https://fb.workplace.com/groups/rust.language/permalink/3249964938385432/
Considering async_unit is effectively only used in Mononoke, and Mononoke
already turns off LSAN in tests for precisely this reason ... it's probably
reasonable to do the same here.
The main body of changes here is also about updating the majority of our
changes to stop calling wait(), and use this new async unit everywhere. This is
effectively a pretty big batch conversion of all of our tests to use async fns
instead of the former approaches. I've also updated a substantial number of
utility functions to be async fns.
A few notable changes here:
- Some pushrebase tests were pretty flaky — the race they look for isn't
deterministic. I added some actual waiting (using pushrebase hooks) to make
it more deterministic. This is kinda copy pasted from the globalrev hook
(where I had introduced this first), but this will do for now.
- The multiplexblob tests don't work at all with new futures, because they call
`poll()` all over the place. I've updated them to new futures, which required
a bit of reworking.
- I took out a couple tests in async unit that were broken anyway.
Reviewed By: StanislavGlebik
Differential Revision: D19902539
fbshipit-source-id: 352b4a531ef5fa855114c1dd8bb4d70ed967dd55
Summary:
Those tests broke after D19830033 landed, which removed the functionality that
generates the logs they were capturing. This fixes that.
Reviewed By: farnz
Differential Revision: D19941478
fbshipit-source-id: 2b422b1d2252e92bb75ae688962fdd66ed33910c
Summary:
Make it possible to limit the time range of mutation info being displayed by hg
debugmutation.
Reviewed By: DurhamG
Differential Revision: D19904000
fbshipit-source-id: 365f54fdd861661961bba1a0ea96fce772623a23
Summary: The error should be request error, not internal error
Reviewed By: krallin
Differential Revision: D19856120
fbshipit-source-id: fc66ac4eaeb27941de3b4ba769fe123b28685d14
Summary:
Now let's fail if we try to derive data that's not enabled in the config.
Note that this diff also adds `derive_unsafe()` method which should be used in backfiller.
Reviewed By: krallin
Differential Revision: D19807956
fbshipit-source-id: c39af8555164314cafa9691197629ab9ddb956f1
Summary: See title. We want the name of the cache to live in `Allocator` - and be used in Open Source as well.
Reviewed By: sathyaphoenix
Differential Revision: D19663603
fbshipit-source-id: 2abef67b56a37cb551e925121813f4b4c8c6e9db
Summary:
- Conditionally enable warmup cache based on derived data configuration
- Blame is temporarily disabled for ovrsource, it is currently not used, and will be enabled again once new blame is generated for this repository
Reviewed By: StanislavGlebik
Differential Revision: D19902902
fbshipit-source-id: 7b782a32f7bfd76024686aea027bb15223079b9d
Summary: The load_limiter was extracted from server/context into its own crate and the server/context itself was refactored into multiple modules, one of which contains facebook-specific code.
Reviewed By: StanislavGlebik
Differential Revision: D19902972
fbshipit-source-id: d577492b4fe01ccfe11b3e092e0521b190516268
Summary: common/sql_ext will now be buildable in OSS
Reviewed By: krallin
Differential Revision: D19876764
fbshipit-source-id: 0f51abd1169f6b8108e7e4cab85b5f193c28e2cd
Summary: Move several pieces of Middleware related to TLS from the LFS server to `gotham_ext`, for reuse in the EdenAPI server. These modules are already well-abstracted and consequently require no modification to be moved out of the LFS server.
Reviewed By: xavierd
Differential Revision: D19890537
fbshipit-source-id: 8de420183e7bd5dd0de10a75c8cfe83825f0c23c
Summary:
I'm going to change the connection pool logic but I'm not sure where it gets
used. This change exposes at least one test using it.
Reviewed By: xavierd
Differential Revision: D19872614
fbshipit-source-id: 4921b92c3fe3fd7ba1a72de17eef92604964eb2e
Summary:
The SCS monitoring future should run forever and never return, even if there is
an error.
If an error does occur, we should just log it, and try again next time.
Also convert to new futures.
Reviewed By: krallin
Differential Revision: D19879621
fbshipit-source-id: 45b7097d1c5af61f3c390f6aa6ada5832062fbc9
Summary: This diff extend `Blame` structure with `origin_line` (line at the moment of introduction of a hunk).
Reviewed By: mitrandir77
Differential Revision: D19499125
fbshipit-source-id: 2fa6bc4ee62d484dd6dc6bee17cb1d04ba1db158
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
In the next diff I'm going to refactor BlobRepo::get_manifest_from_bonsai()
function and remove IncompleteFilenodes structure. While doing so I noticed
that the behaviour of get_manifest_from_bonsai() and FilenodesOnlyPublic is
different with regards to generating hg filenodes for octopus merges. In
particular, get_manifest_from_bonsai() at the moment doesn't generate filenodes
for paths that came from stepparents, but FilenodesOnlyPublic does generate
them.
I'd say both of this approach are equally reasonable, however let's try to
preserve the behaviour
Reviewed By: krallin
Differential Revision: D19856420
fbshipit-source-id: 9f8ae656205f39536c450b885fc4d8d6a2534456
Summary: Later in this stack, we're going to have to introduce a few more context types, including Mononoke's per-session `CoreContext` as well as an LFS-server inspired per-request `RequestContext`. To make the scope of the `EdenApiContext` more clear (namely, that unlike the other contexts, this one persists for the entirety of the server's execution), let's rename this to `EdenApiServerContext`. This mirrors the naming convention for these contexts used in the LFS server.
Reviewed By: xavierd
Differential Revision: D19863296
fbshipit-source-id: 8ad785070328523d0beaf824c86c7350ff6a2697
Summary: Use `ServerIdentityMiddleware` from `gotham_ext` in the EdenAPI server to provide server information in the HTTP response headers returned by the server. This is useful for debugging.
Reviewed By: xavierd
Differential Revision: D19863297
fbshipit-source-id: 6ed8bac35e05af580e062423e6f6389a18d17c2a
Summary:
This improves scs client ux as you can see in the test.
Cross scheme collisions have been already supported in the code, and there will
be a proper message.
For the ambiguity within a single scheme, I added a message.
Reviewed By: mitrandir77
Differential Revision: D19834142
fbshipit-source-id: a2cff44f6ef031b1224ebf13d38d76c66c9ee4b9
Summary: The exact format of the GPL license header changed when the Mononoke directory moved from `/scm/mononoke` to `/eden/mononoke`. This file ended up with the old header somehow (was created prior to the rename but landed after), so let's fix it to make the linter happy.
Reviewed By: farnz
Differential Revision: D19848640
fbshipit-source-id: 39c5b49850e5a3cba1951bf4e5b813cd08940f01
Summary: Move the generally-useful ServerIdentityMiddleware into gotham_ext so we can use it in the EdenAPI server.
Reviewed By: xavierd
Differential Revision: D19845282
fbshipit-source-id: 3a01b15dc64cee99cefafcdac229c0b70a4db683
Summary: remove the need to pass mapping to `::derive` method
Reviewed By: StanislavGlebik
Differential Revision: D19856560
fbshipit-source-id: 219af827ea7e077a4c3e678a85c51dc0e3822d79
Summary:
Currently admission to fastreplay is controlled at a global level - a
percentage of all repos being replayed are admitted without the ability to turn
off an individual repo.
Update fastreplay to use a config option that controls which repos should be skipped.
Reviewed By: krallin
Differential Revision: D19835304
fbshipit-source-id: 63b7c66b35eaa2cb1370ad96b1c6e98a2c93b712
Summary: Rechunker uses tokio 0.1.0 and legacy futures 0.1.0 . This diff changes rechunker.rs to use new std::future and tokio 0.2.0
Reviewed By: farnz
Differential Revision: D19801966
fbshipit-source-id: 3ba7936e8981c87906a881216ffd66332cdcc761
Summary:
This will allow us to distinguish `getbundle` for a normal `pull` from the one
for infinitepush pull.
Reviewed By: StanislavGlebik
Differential Revision: D19833206
fbshipit-source-id: 86534320fbb4d60bac04d458a0953701201cba87
Summary:
Added "rust" to the languages in the if-service and its dependent targets, added a unique `rust_crate_name` to each one.
Adjusted import names in the dependent code.
Reviewed By: dtolnay
Differential Revision: D19818989
fbshipit-source-id: d1cf5c5f7b82dd64d7c045e50b304b7a58cf2ffc
Summary: Not very useful on today's hardware.
Reviewed By: quark-zju
Differential Revision: D19783741
fbshipit-source-id: d8ff29486cdc5f8e31d929d9fbba827e37012381
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary:
Include the full error source chain on any internal errors. This will improve
debugability when internal errors are encountered, as the outermost context
might not be enough to describe the origin of the error.
Detect when backtraces are disabled, and only include them on the error if a backtrace was captured.
Reviewed By: StanislavGlebik
Differential Revision: D19813785
fbshipit-source-id: c28ffc0c44050a82cb7050afa57e976f37962432
Summary: create a bounded traversal crate with the 0.1 version of bounded_traversal_stream functionality, prior to updating to futures 0.3
Reviewed By: farnz
Differential Revision: D19804873
fbshipit-source-id: e98e00111fee5b1a9fcfc20bb054eeae1263fb26
Summary: replace use of spawn_future with tokio 0.2 join handle
Reviewed By: krallin
Differential Revision: D19770171
fbshipit-source-id: e0b7bf3da58896b223149b339a72bfec997215ee
Summary:
Update the walker step methods to use new futures, and combine them with async fn
Later in stack planning to:
* remove use of spawn_future and replace it with the tokio 0.2 join handles
* port bounded_traversal_stream to new futures so all these 0.3 futures don't immediated get compat'd back to 0.1
Reviewed By: farnz
Differential Revision: D19767902
fbshipit-source-id: 10fd6236a064efbb7d0815fadbdd32036bcafead
Summary:
D19767626 added an original_timestamp column to the
blobstore_sync_queue. Update the sqlite schema to keep it in sync.
Reviewed By: krallin
Differential Revision: D19787488
fbshipit-source-id: ad576e2ec99349953e2ab69e3defb73d1ff556c0
