Summary: This is in preparation to prevent marking visited for ChangesetInfo if it is not derived yet.
Reviewed By: mitrandir77
Differential Revision: D24838453
fbshipit-source-id: 0262b7f28754724bbfcc2407b4e649a90175cbcc
Summary: The old test counted all types, make the test case check just the changeset_info derived node types for clarity.
Reviewed By: mitrandir77
Differential Revision: D24838457
fbshipit-source-id: 0ba96fd640e700f605521cf066318a89570ee71c
Summary:
Adding automatically generated derived_xxxx node groups so that less typing is
needed and we're also checking the nodes are mapped correctly to derived data types.
Reviewed By: mitrandir77
Differential Revision: D24838738
fbshipit-source-id: 2bc8ff03a82c5d18f749affba2e67d214fb7ace7
Summary: This allows us to use -i bonsai instead of -i Bookmark -i BonsaiChangeset, which is a bit shorter
Reviewed By: mitrandir77
Differential Revision: D24838454
fbshipit-source-id: a758ad069af36fb1d1301e162bee822988cab07b
Summary: All the node types support FromStr so can generate NodeType::parse_node() rather than hand implement it.
Reviewed By: mitrandir77
Differential Revision: D24711372
fbshipit-source-id: 24e27e9cdda078c6dc66ac839cb3cfed6e93f269
Summary:
implement FromStr for BookmarkName, can use it to handle bookmarks
more uniformly with other types in the walker
Reviewed By: mitrandir77
Differential Revision: D24725786
fbshipit-source-id: e7eb7ece4a4bdc5dfd91f253f0383829c4ecc73b
Summary: Refactor from non-FromStr node parsing to FromStr, make it consistent with other node keys.
Reviewed By: mitrandir77
Differential Revision: D24711374
fbshipit-source-id: 84200b781bfad0f860acd8aecb95ff238490b92d
Summary: use PathKey for parsing of Node::HgFileNode in walker.
Reviewed By: ikostia
Differential Revision: D24711375
fbshipit-source-id: 4fe5887ba44ca9fca1dde54eaa75b30114b3b4b8
Summary: add PathKey newtype to Node so can implement FromStr and use it in parsing for HgManifest
Reviewed By: mitrandir77
Differential Revision: D24711371
fbshipit-source-id: a9879f6d2e16eb54b2ca7af4e812a4f031c9e584
Summary: Add UnitKey newtype to walker so that can implement FromStr, this is leading up to all node keys supporting from_str at which point I can generate NodeType::parse_node.
Reviewed By: mitrandir77, ikostia
Differential Revision: D24711376
fbshipit-source-id: aa4e26eb8e9206673298b632a079d2cc66d152ee
Summary: This is mostly a slight refactoring to help code reuse. However, there's a small behavior change as well (which I think is acceptable): before we compared `count` vs `max_value`, and now we compare `count + bump` vs `max_value`.
Reviewed By: krallin
Differential Revision: D24871175
fbshipit-source-id: 94e53ff2c05b4f9b236473c7e4b6d78229b64d53
Summary: Now that `derive03` is the only version available, rename it to `derive`.
Reviewed By: krallin
Differential Revision: D24900106
fbshipit-source-id: c7fbf9a00baca7d52da64f2b5c17e3fe1ddc179e
Summary:
Now that all code is using `BonsaiDerived::derive03`, we can remove the old
futures version `BonsaiDerived::derive`.
Reviewed By: krallin
Differential Revision: D24900108
fbshipit-source-id: 885d903d4a45e639e4d44e19b5d70fac26bce279
Summary:
To ensure other parts of Mononoke can fully read new blobs as soon as they've
been written, ensure their buffers are flushed and they've been synced to disk
before returning from the blob put.
Reviewed By: krallin
Differential Revision: D24921657
fbshipit-source-id: df401470aaeeebcdc9d237271b40a399115ba25f
Summary: As we are making directory structure inside the bucket anyway, it would be usefull to combine keys per repo.
Reviewed By: ahornby
Differential Revision: D24884248
fbshipit-source-id: 85efeb7009a9d211381319caa4e72aa3687c51ee
Summary:
Transfers iddag flat segments along with the head_id that should be use to
rebuild a full fledged IdDag. It also transfers idmap details. In the current
version it only transfers universal commit mappings.
Reviewed By: krallin
Differential Revision: D24808329
fbshipit-source-id: 4de9edcab56b54b901df1ca4be5985af2539ae05
Summary:
Under this configuration SegmentedChangelog Dags (IdDag + IdMap) are always
downloaded from saves. There is no real state kept in memory.
It's a simple configuration and somewhat flexible with treaks to blobstore
caching.
Reviewed By: krallin
Differential Revision: D24808330
fbshipit-source-id: 450011657c4d384b5b42e881af8a1bd008d2e005
Summary:
Constructs and returns `CloneData<ChangesetId>`. This object can then be used
to bootstrap a client dag that speaks bonsai commits.
Short term we are going to be using this data in the Mercurial client which
doesn't use bonsai. Hg MononokeRepo will convert it.
Long term we may decide that we want to download cached artifacts for
CloneData. I don't see an issue getting there, I see this as a valid path
forward that cuts down on the configuration required to get to the cached
artifacts. All that said, I think that using whatever dag is available in
memory would be a viable production option.
Reviewed By: krallin
Differential Revision: D24717915
fbshipit-source-id: 656924abb4bbfa1a11431000b6ca6ed2491cdc74
Summary: The SegmentedChangelogManager abstracts saving and loading Dags. This is currently used in the tailer and seeder processes. It will also be used to load dags while the server is running.
Reviewed By: krallin
Differential Revision: D24717925
fbshipit-source-id: 30dff7dfc957f455be6cf733b20449c804511b43
Summary:
Allow users of `tests_utils` to create paths that are not `String`, by supporting any type
that can be converted into `MPath`.
Reviewed By: StanislavGlebik
Differential Revision: D24887002
fbshipit-source-id: 47ad567507185863c1cfa3c6738f30aa9266901a
Summary:
Add type definitions for skeleton manifests.
Skeleton manifests are manifests that correspond to the shape of the repository (which directories and files exist), but do not include anything relating to the content. This means they only change when files are added or deleted.
They are used for two purposes:
* To record the count of descendant directories for each directory. This will be useful for limiting parallelism when doing an ordered traversal of a manifest. The descendant directory count is a good estimate of the amount of work required to process a directory.
* To record whether a directory, or any of its subdirectories, contains a case conflict. This will be used to enforce case-conflict requirements in repos.
Differential Revision: D24787535
fbshipit-source-id: 7cb92546ed80687d5b98a6c00f9cd10896359b8d
Summary:
Generate walker EdgeType::outgoing_type() to reduce boilerplate
When defining edges if the edge variant and destination node at the same no extra parameters needed. If they are different then the node type of destination is passed in parens, e.g. BonsaiParent(BonsaiChangeset)
Reviewed By: StanislavGlebik
Differential Revision: D24687828
fbshipit-source-id: 1616c786d78242c2b3a8c7a1ba58cc1433ea0a26
Summary: It is possible that hash of newly created bonsai_changeset will be different from what is in prod repo. In this case let's fetch bonsai from the prod, to make backup repo consistent with prod.
Reviewed By: StanislavGlebik
Differential Revision: D24593003
fbshipit-source-id: 70496c59927dae190a8508d67f0e3d5bf8d32e5c
Summary: Use create_graph to generate EdgeType enum in walker to reduce boiler plate needed when adding new derived node and edge types to the walker
Differential Revision: D24687827
fbshipit-source-id: 63337f4136c649948e0d3039529965c296c6d67e
Summary: Also use the 0.3 compatible .return_remainder in unbundle.
Reviewed By: ikostia
Differential Revision: D24729464
fbshipit-source-id: ede5cc60f4b872a3b968cf14bb0e2c5d9b85c242
Summary:
When finishing a hash computation for a blob, we currently call `format!` to allocate
and format the error string before calling `.expect` on the `write_all` result.
In practice this will never fail, so this is wasted work. From experimentation on
the playground, the Rust compiler does not appear to be smart enough to optimize this
away, either.
A small optimization, but let's get rid of this by calling panic directly, and
only in the failure path.
Reviewed By: farnz
Differential Revision: D24857833
fbshipit-source-id: e3e35b402ca3a9f6c9d8fbbd758cc486ef1c5566
Summary:
Adds `--reversefill` mode to bookmarks filler that fetches bookmark updates
from the queue and syncs them to infinitepush database.
Reviewed By: farnz
Differential Revision: D24538317
fbshipit-source-id: 5ac7ef601f2ff120c4efd8df08a416e00df0ceb9
Summary:
Summarry:
This is the first part of syncing new scratch bookmark pushes from Mononoke to
Mercurial: on each bookmark movement we log this bookmark movement to filler's
queue.
Reviewed By: liubov-dmitrieva
Differential Revision: D24480546
fbshipit-source-id: 27103b4b4f8c4600aaf485826db2936eaffcc4a9
Summary: Make the naming of bonsai fsnode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687833
fbshipit-source-id: 8d46a53c023a4b8f95c0edc42df86e467c054ebb
Summary: Make the naming of linknode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687832
fbshipit-source-id: 46525d7bebd17723a130a70f566b24104cc39656
Summary:
Use macro to implement Node::get_type() in walker.
Reduces the boiler plate when adding new types to the graph.
Reviewed By: farnz
Differential Revision: D24687826
fbshipit-source-id: 5f89c6fb69fd9df3fff25a2425a4d2035dbf5ed9
Summary: Generate NodeType::root_edge_type() so less boiler plate when adding new types to the walk.
Reviewed By: farnz
Differential Revision: D24687825
fbshipit-source-id: 083fc57aee8fe01b29ad4a6f9ebe660cc057dfab
Summary:
Define walker graph with macro to reduce repetition which should make adding new derived data types simpler.
Specifically, this removes the duplication between NodeType and Node
Reviewed By: farnz
Differential Revision: D24687831
fbshipit-source-id: 97d67faf02b2a88bb871dc0388d75d3dd3e8528d
Summary: use strum EnumCount instead of own macros, can remove some code. This needed strum upgrade to 0.19
Reviewed By: krallin
Differential Revision: D24680441
fbshipit-source-id: 56e5b66f75c3d8ff949685c26f503571873c0cde
Summary: Update to strum 0.19 as it has improved EnumCount derivation
Reviewed By: mohanz
Differential Revision: D24680442
fbshipit-source-id: 2d3d2a84e994f09bf3b1c7ea748a80a67d100c13
Summary:
`manifest/test_utils` contains test utilities that are only used by derived data, and only one
of which relates to manifests. Its name (`test_utils`) is also confusing with `tests/utils`.
Move it to `derived_data_test_utils`, and update it to new futures.
Reviewed By: mitrandir77
Differential Revision: D24787536
fbshipit-source-id: 7a4a735132ccf81e3f75683c7f44c9ada11bc9d7
Summary:
Reduce visibility of add_* functions that MononokeApp controls, no need for them to be public.
Updated a couple of binaries to use MononokApp.with_fb303_args() instead of calling the add_fb303 function directly.
Reviewed By: krallin
Differential Revision: D24757202
fbshipit-source-id: a068ca4fd976429e7c02c4049429553cc8acf3d4
Summary: benchmark is the last remaining user of args::add_cachelib_args outside of MononokeApp, switch it to use MononokeApp instead.
Reviewed By: krallin
Differential Revision: D24755785
fbshipit-source-id: c105b4443394c88b6effdac382089e7eaca65bfe
Summary: make MononokeApp arguments more configurable so binaries can opt out of them if a section does not apply, making the --help more relevant.
Reviewed By: krallin
Differential Revision: D24757007
fbshipit-source-id: eed2f321bdbd04208567ef9a45cf861e56cdd07e
Summary:
Previously it was a config knob, but they are rather hard to change because
they require a restart of the service. Let's make it a tunable instead.
Reviewed By: farnz
Differential Revision: D24682129
fbshipit-source-id: 9832927a97b1ff9da49c69755c3fbdc1871b3c5d
Summary:
This updates the last bit of the Filestore that was using 0.1 futures to 0.3.
This used to use a weighted buffered stream (which we don't have for 0.3
futures at this point), but as I started working on one I realized we don't
even need it here, so I took this out.
Reviewed By: StanislavGlebik
Differential Revision: D24735907
fbshipit-source-id: 00a55c14864b09f9c353f95f2f8cbb895cf52791
Summary:
This updates the external facing API of the filestore to use 0.3 streams.
Internally, there is still a bit of 0.3 streams, but as of this change, it's
all 0.3 outside.
This required a few changes here and there in places where it was simpler to
just update them to use 0.3 futures instead of `compat()`-ing everything.
Reviewed By: ikostia
Differential Revision: D24731298
fbshipit-source-id: 18a1dc58b27d129970a6aa2d0d23994d5c5de6aa
Summary: Like it says in the title.
Reviewed By: StanislavGlebik
Differential Revision: D24731300
fbshipit-source-id: b9c44fc1e4bd4cfe8655e1024a0547e40fb99424
Summary:
Like it says in the title. This required quite a lot of changes at callsites,
as you'd expect.
Reviewed By: StanislavGlebik
Differential Revision: D24731299
fbshipit-source-id: e58447e88dcc3ba1ab3c951f87f7042e2b03eb2c
Summary: Like it says in the title. This updates `store()` and its (many) callsites.
Reviewed By: ahornby
Differential Revision: D24728658
fbshipit-source-id: 5fccf76d25e58eaf069f3f0cf5a31d2c397687ea
Summary: Like it says in the title. Not much to be said here.
Reviewed By: ahornby
Differential Revision: D24727256
fbshipit-source-id: 1645339edf287ac7e59612589b308f08b708ae00
Summary:
This updates the metadata APIs in the filestore to futures 0.3 & async / await.
This changes the external API of the filestore, so there's quite a bit of churn
outside of that module.
Reviewed By: markbt
Differential Revision: D24727255
fbshipit-source-id: 59833f185abd6ab9c609c6bcc22ca88ada6f1b42
Summary:
Like it says in the title. This also lets us get rid of some macros we no
longer need.
Reviewed By: markbt
Differential Revision: D24727259
fbshipit-source-id: 5e3211bc08fa5376b4cfce4bea0428ab7bf3dc0f
Summary:
Like it says in the title. This also lets us remove the spawn module entirely.
Note that there is one little annoyance here: I ran into the good old "not
general enough" compiler issue, so I had to add a bit more boxing to make this
go away :(
Reviewed By: markbt
Differential Revision: D24727253
fbshipit-source-id: 73435305d39cade2f32b151734adf0969311c243
Summary:
This will simplify a bunch of refactoring, and we no longer need them not to
be. This lets us convert parts of `chunk` to new futures as well.
Reviewed By: markbt
Differential Revision: D24727254
fbshipit-source-id: de643effe2d1d42ff9bf85a48d09301e929e66de
Summary:
Like it says in the title. This updates the Filestore's multiplexer to new
futures. The change is pretty mechanical, save for:
- We don't use filestore::spawn anymore, since that's provided by
`tokio::task::spawn` now.
- We no longer need to use futures or streams with `Error = !` since in Futures
0.3 you can have an `Output` that isn't a `Result`.
- We need to make the `Stream` we accept `Send` because we can't used
`boxed().compat()` otherwise. I'd like to remove that constraint once the
conversion is complete, but considering all callsites do have a `Send` stream
(the only one that didn't was API Server but that's long gone), just adding
the bound is easiest.
Reviewed By: farnz
Differential Revision: D24708596
fbshipit-source-id: 8b278b5ae49029b7f0d0d9d4fe96c467e1343f60
Summary:
This makes some refactoring later easier. I'd like to not require this, but for
now it's a bit simpler to just do this. Those are the only callsites that
send non-static streams.
Reviewed By: markbt
Differential Revision: D24727258
fbshipit-source-id: c0e4dc86e249a08c2194a20de5a2dfd5a5933d0b
Summary: We don't need to declare a fake empty version number
Reviewed By: farnz
Differential Revision: D24757981
fbshipit-source-id: 594c97e225704d783bea723efcbb9dfc4d5d800b
Summary:
The root cause for S199754 was a file named "con.rs" was checked in onto the
repo. Since this is a reserved filename on Windows, this broke all Windows
users having it in their sparse profiles.
The rules for reserved names are defined as such by Microsoft:
"CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9,
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these
names followed immediately by an extension; for example, NUL.txt is not
recommended. For more information, see Namespaces."
Of course, since the filesystem is case insensitive, these can have any casing.
Reviewed By: krallin
Differential Revision: D24453528
fbshipit-source-id: 389f15e2b1a88e3c1e8721fb7868616acabebc64
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
In Mononoke for a sharded DB we historically used connection pool size 1 per shard. With the Mysql FFI client now it doesn't make sense, as the client's conn pool is smart enough and designed to works with sharded DBs, so currently we don't even benefit from having a pool.
In this diff I added an API to create sharded connections: a single pool is shared between all the shards.
Reviewed By: farnz
Differential Revision: D24475317
fbshipit-source-id: b7142c030a10ccfde1d5a44943b38cfa70332c6a
Summary:
Whenever a sparse profile changes (e.g. we include or exclude a directory or a file) we do a full prefetch for all trees in the revision and then for each file in a revision we check if this file has changed its state after sparse profile change (i.e. whether it was included before the change and became excluded after the change and vice versa). It can be quite expensive for large repos and looks like checking all the files is unnecessary.
For example, there might be top-level directories that are excluded in sparse profile before and after the change. In that case there's no reason to check every file in this directory, and there's no reason to prefetch manifests for this directory.
More importantly, `mf.walk()` method is already smart enough to do manifest prefetches if treemanifest.ondemandfetch is set to True, so it looks like there's no reason to do any additional prefetching at all (at least in theory).
So this diff does a few things:
1) The default mode is to use mf.walk() method with a union matcher to find all the files that were are included either in old or new sparse profile. In order for it to prefetch efficiently we force enable treemanifest.ondemandfetch config option.
2) It also adds a fallback option to full prefetch (i.e. the same thing we do right now) Hopefully this fallback option won't be necessary and we'll delete them soon. I've added them only to be able to fallback to current behaviour in case there are problems with the new behaviour
I think we can do an even more efficient fetch by using xor matcher instead of union matcher. I'll try to implement it in the next diffs
Reviewed By: sfilipco
Differential Revision: D24705823
fbshipit-source-id: 2c232a66cc74ee95bdaa84201df46448412f087f
Summary:
This seems to trip up Cargo builds
```
error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `with`
--> src/lib.rs:365:3
|
7 | S with version V1
| ^^^^ expected one of 8 possible tokens
error: aborting due to previous error
```
Reviewed By: StanislavGlebik
Differential Revision: D24754708
fbshipit-source-id: 0dc5539acf340ac409bf7b6158313c8fec16a275
Summary:
Previously we were choosing current version, and just as with backsyncer this
is not always correct. Let's instead choose not the current version but the
version of the bookmark you are importing to.
This diff also introduced an integration test for a repo import into a pushredirected repo, and turned out there were a few bugs in the repo_import code (open_source_sql was used instead of open_sql). This diff fixed them as well
Reviewed By: ikostia
Differential Revision: D24651849
fbshipit-source-id: bfe36e005170ae2f49fa3a6cb208bf6d2c351298
Summary:
This diff changes semantic of `sync_commit()` function to return an error when
trying to sync a commit with no parents. This is a small code change which has big change
in semantics, and because of that I had to change how backsyncer and
mononoke_x_repo_sync job.
Instead of using `unsafe_sync_commit()/sync_commit()` functions both backsyncer and
`x_repo_sync_job` now use `unsafe_sync_commit_with_expected_version()`
which forces them to specify which version to use for commit with no parents.
And in order to find this version I changed find_toposorted_unsynced_ancestors
to not only return unsynced ancestors but also return mapping versions of their
(i.e. of unsynced ancestors) parents. Given this mapping we can figure out what
version is supposed to be used in `unsafe_sync_commit_with_expected_version()`.
The question raises what to do when a commit doesn't have any synced ancestor and hence we can't decide
which version to use to remap it. At the moment we are using current version (i.e. preserving the existing behaviour).
However this behaviour is incorrect, and so it will be changed in the next diffs
Reviewed By: ikostia
Differential Revision: D24617936
fbshipit-source-id: 6de26c50e4dde0d054ed2cba3508e6e8568f9222
Summary:
Previously we were always choosing the current version for remapping via
pushrebase, but this is incorrect. Let's instead select the version based on
what version parent commits used to remap with.
Reviewed By: ikostia
Differential Revision: D24621128
fbshipit-source-id: 2fedc34b706f090266cd43eaf3439f8fb0360d0d
Summary: Let strum crate do this for us
Reviewed By: krallin
Differential Revision: D24680444
fbshipit-source-id: dbde0077c105d6cc572a0c863bcb4d043714d441
Summary:
Now that fsnodes is async, convert more functions to use references, and tidy
up imports and type names.
Reviewed By: krallin
Differential Revision: D24726145
fbshipit-source-id: 75a619777f19754daf494a3743d26fa2e77aef54
Summary:
Update `fsnodes::derive_fsnode` and its immediate utility functions to use new style
futures and `async`/`.await` syntax.
Reviewed By: krallin
Differential Revision: D24726146
fbshipit-source-id: 0b0d5b1162a73568ef5c47db6e8252267e760e7f
Summary: We're getting rid of old futures - remove them as a dep here
Reviewed By: StanislavGlebik
Differential Revision: D24705787
fbshipit-source-id: 83ae938be0c9f7f485c74d3e26d041e844e94a43
Summary:
We can have different bonsai changesets hash for the same hg changeset. Consider situation - we have hg repo:
```
o B (Add file "b")
│
o A (Add file "a")
```
The correct bonsai changeset for B will have only entry `(<Path_to_b>,Some(<hash_b>))` in `file_changes`. But we can also have bonsai changeset for B with 2 entries `(<Path_to_b>,Some(<hash_b>)), (<Path_to_a>,Some(<hash_a>))`. This diff provides the functionality to manually create such situation. And later it will be used for verification blobimport backups
Reviewed By: StanislavGlebik
Differential Revision: D24589387
fbshipit-source-id: 89c56fca935dffe3cbfb282995efb287726a3ca9
Summary:
This changes the methods from ones that return old `BoxFuture`s to an async method
using `async_trait`.
Reviewed By: krallin
Differential Revision: D24689506
fbshipit-source-id: 7b13010924369f81681e6590898af703c5423385
Summary:
This changes the trait method from one that returns an old `BoxFuture` to an async
method using `async_trait`.
Reviewed By: krallin
Differential Revision: D24686888
fbshipit-source-id: 0ac231cdbb60d256b6d5ad5aafbe8779b96905f3
Summary: This is to be able to see in Scuba why a given sync function was called.
Reviewed By: StanislavGlebik
Differential Revision: D24689366
fbshipit-source-id: f868fc1b6fcbf6c692e1373cbe8da8cd4a230780
Summary: This is a step towards modernizing unbundle crate to use futures 0.3.
Reviewed By: farnz
Differential Revision: D24682963
fbshipit-source-id: 55c17fd699846a24647a23ea1c22888407643dfd
Summary: The more we remove, the easier it'll be to remove the last few problem cases.
Reviewed By: StanislavGlebik
Differential Revision: D24592052
fbshipit-source-id: de6371305991d54bf2802fd904850b49aeb11bbd
Summary:
This didn't matter much in practice because we've never pushrebased a rewritten
commit where one of the parents rewrites to nothing. Nevertheless I think it's
cleaner to return an error in this case
Reviewed By: ikostia
Differential Revision: D24621105
fbshipit-source-id: 8cf62efd28e0c9aed945f727b1872db6922cfeb3
Summary:
This stack has a goal of cleaning up how commits with no parents and
consequently merges are treated in CommitSyncer. At the moment it always uses current version
to rewrite them, and this is incorrect. See example below (here N means "new
commits" i.e. commits that weren't remapped yet but we need to remap them now)
```
large repo
O <- uses current version
|
O
|
O N
|/ |
O | <- uses old version
| N
| |
O N <- this commit should be rewritten with old version!
```
With current logic all N commits will use current version for remapping, but
this is incorrect, and instead "old version" should be used. The goal is to fix
it and make it so that backsyncer and x-repo sync job pick the correct version
to use for remapping.
-----
As it was noted in the previous diff we need a function that overrides a
version for a commit with no parents. We need this function because for a
commit with no parents CommitSyncer can't decide which mapping version to use
because, well, there are no parents which could have hinted about the version.
So let's add this function and cleanup unsafe_sync_commit_impl function to
apply version override only to a commit with no parents. If commit has parents then we'd verify that the version from parents matches the expected version.
In the next diffs I'll make it so that if a version for commit with no parents is not specified then
unsafe_sync_commit_impl fails rather than uses current version.
Reviewed By: ikostia
Differential Revision: D24617953
fbshipit-source-id: a68f837da9d90bb18034f628b7880482a2e548b7
Summary:
This adds some basic Scuba logging to commit sync logic in Mononoke. This can be disabled via a tunable.
Note: of course, the wrapping of logging functions would have been pretty cool to do in a macro, but I don't have time atm to figure it out, so have just code.
Reviewed By: StanislavGlebik
Differential Revision: D24571702
fbshipit-source-id: f8830fc675a2a62d2204671e86ab2c819372c5cc
Summary: This adds a `ScubaSampleBuilder` field to the `CommitSyncer` and ensures this field is instantiated with correct values (real vs discarding sample) depending on circumstances.
Reviewed By: StanislavGlebik
Differential Revision: D24539732
fbshipit-source-id: 37aedcff9aefcfcd6b740a0491ab35f9e5ce7c77
Summary:
Functionality of this function can be replaced with
unsafe_always_rewrite_sync_commit, so let's do that.
However in the next diff we'll need a function similar to
test_unsafe_sync_commit but with a slightly different semantics - instead of
forcing override for every commit we'll just force it for a commit with no
parents. Because of that let's not remove unsafe_sync_commit_impl function.
Reviewed By: ikostia
Differential Revision: D24616824
fbshipit-source-id: 6969145d98cd23604920a6b490bf7ffe47938c08
Summary:
Our integration tests do not have "test_version" version, they have only
"TEST_VERSION_NAME". This didn't make any difference because x-repo sync job
was always use the current version, but this is going to change soon. So let's
update the tests first
Reviewed By: krallin
Differential Revision: D24620364
fbshipit-source-id: 19416f6d6aba2d9c426efa545d18d4be458d3272
Summary:
Fixing test. I updated the serialization format in another stack and didn not
update the hash before I landed.
Reviewed By: lukaspiatkowski
Differential Revision: D24643831
fbshipit-source-id: 3ec888d99cbee0aa6d804740bebaf032dd28e2c9
Summary:
MySQL doesn't like that the idmap table is renamed to `inner`. For good reason,
inner is a keyword, best to rename it.
Reviewed By: ahornby
Differential Revision: D24568914
fbshipit-source-id: 7a3790e835931b29658c7652cc89069c6b9b5bab
Summary:
I avoided this function because it interacts in a weird ways with dependencies.
At this point I am no longer concerned about that and it can help us simplify
some code.
Looking ahead I think that we will refactor things into having fewer
dependencies.
Reviewed By: krallin
Differential Revision: D24555935
fbshipit-source-id: 994b25d90da491bb5cc593b6c33085790c4fb322
Summary:
The command reads the last SegmentedChangelog that was saved for a repository
and updates head to match a given bookmark (master).
Right now this is just a command that works on one repository. Follow up
changes will look at deployment options and handling multiple repositories.
Reviewed By: krallin
Differential Revision: D24516438
fbshipit-source-id: 8f04f9426c2f2d7748c5363d2dbdf9f3acb79ddd
Summary:
I initially saw the incremental build as something that would be run in places
that had IdMap and IdDag stored side by side in process. I am reconsidering
to use incremental build in the tailing process to keeps Segmented Changelog
artifacts up to date.
Since we update the IdMap before we update the IdDag, it is likely that we
will have runs that only update the IdMap and fail to update IdDags. This diff
adds a mechanism for the IdDag to catch up.
Reviewed By: krallin
Differential Revision: D24516440
fbshipit-source-id: 3a99248451d806ae20a0ba96199a34a8a35edaa4
Summary:
Nice to have things for debugging.
This isn't an exhaustive list of places that we could add context too. I'll
probably look to complete the list after the current changes are done.
Reviewed By: krallin
Differential Revision: D24516437
fbshipit-source-id: 7f29e7afde5a5918aea419181d786f48da9b8b14
Summary:
The general goal is to allign segmented changelog blobstore usage with the
general pattern in Mononoke.
Reviewed By: quark-zju
Differential Revision: D24605796
fbshipit-source-id: 808985609f74ebc45f3fcc57583e55f3af9bce1d
Summary: Change the default blobstore put behaviour to IfAbsent, so that all binaries apart from admin tools using MononokeApp::special_put_behaviour() pick up the change.
Reviewed By: farnz
Differential Revision: D24619663
fbshipit-source-id: 98439513b985d2cde88ef99e5eb177974e9db5c9
Summary:
Extend MononokeApp so admin apps can have a special default put behaviour (typically
overwrite) vs the soon to be new default of IfAbsent
Use it from the admin tools.
Reviewed By: farnz
Differential Revision: D24623094
fbshipit-source-id: 5709c68429f8e1de0535eec132998d20411fc0e6
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.
Reviewed By: krallin
Differential Revision: D24477468
fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
Summary: Update walker to use MappedHgChangesetId derived data now that hg data is modeled like other derived data. This simplifies bonsai_to_hg_mapping_step
Reviewed By: aslpavel
Differential Revision: D24531553
fbshipit-source-id: 62663d8d47ec7145980c4cd567cba3009b1999cb
Summary: This is very old code that once acted to prototype walker-type functionality. As it's dead, delete it.
Reviewed By: ikostia, krallin
Differential Revision: D24591123
fbshipit-source-id: 663108e123d354243c2be4f00819f39d6951db93
Summary:
Some requests can result in a large number of blob fetches. Add rate limiting
so that these requests don't use up all available capacity.
Rate limits can be specified in tunables.
Reviewed By: ahornby
Differential Revision: D24592814
fbshipit-source-id: 9a73a92094d0ce01be5491b65b4808e3ebb05c11
Summary: Add ChangesetInfo derived data to the walker so that it can be scrubbed and validated
Reviewed By: farnz
Differential Revision: D24312123
fbshipit-source-id: 84b3bba87e5391339f97cd2e5ae0313761726d02
Summary: Add the ability to specify a group of node types in one go on the command line
Reviewed By: farnz
Differential Revision: D24526827
fbshipit-source-id: 59d2f0cd06dbbe2555625023be3725a528256005
Summary:
Prevent the walker from trying to walk derived data node types that are not configured for a repo.
This is done by add a mapping from walker NodeType to derived data ::NAME, connect it up for existing derived data usages and using it for validation in setup.rs
Reviewed By: farnz
Differential Revision: D24391591
fbshipit-source-id: 21ae63f4f210d2e1310b0ee2c509fb492f742db7
Summary:
We want to be able to detect garbage blobs by looking at generation numbers.
Update generation numbers on put, and have a mark command exist to mark blobs as not garbage.
Reviewed By: ahornby
Differential Revision: D23989289
fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
Summary:
I'm planning to use them from inside establish_connection function. So this
diff makes a refactoring to make scuba logger and slog logger available in
StdioRelay
Reviewed By: krallin
Differential Revision: D24590426
fbshipit-source-id: 5c20025295700aa91c685c47242618a20f89eb76
Summary:
`CommitSyncerArgs` was useful when `CommitSyncer` did not have a way to query
existing configs from the configerator. Now that `SyncDataProvider` is part of
`CommitSyncer`, we can get rid of `CommitSyncerArgs`, which will also make
further improvements more convenient.
Reviewed By: StanislavGlebik
Differential Revision: D24565773
fbshipit-source-id: 4dd507b06d946c6018a4a4e8d5e77c6b27abe195
Summary:
After we synced all the files from source directory into destination directory
the destination directory might have some files that source directory doesn't.
So let's add a command to remove them.
Reviewed By: ikostia
Differential Revision: D24541984
fbshipit-source-id: 7e0e21e4c8079d24e1e24adccd3a20a8bbc737ca
Summary:
Previously `mononoke_admin rsync` didn't overwrite files i.e. if a target
directory has a file with the same name as in source directory then it won't be overwritten.
This diff adds an option to make it possible to overwrite a file if a target
directory has a file with the same name but its content is different. However
note that if a file has to be overwritten then target file is going to be removed in one
commit and then copied in the second (i.e. we'll create two commits instead of
one). The reason for doing that is to preserve the history of the original file
(i.e. history from `from-directory`).
Reviewed By: aslpavel
Differential Revision: D24538199
fbshipit-source-id: 792162c4e5ad81fb6949dd95eb1322696ec011ea
Summary: A better fix would be to also get rid of `Option` in the struct itself, but I don't want to spend any more time on this atm, and this change is a clear improvement.
Reviewed By: StanislavGlebik
Differential Revision: D24538309
fbshipit-source-id: 6190c6936dc34d996ecd3d700e5f71282d45f651
Summary: Same as the previous diff, but for push-redirector.
Reviewed By: StanislavGlebik
Differential Revision: D24538027
fbshipit-source-id: 392aee1b9cf0e684486c274c2b54fc2fb719bb3a
Summary:
update Phases::add_reachable_as_public to futures03
With this change all the Phases methods are futures03
Reviewed By: krallin
Differential Revision: D24531552
fbshipit-source-id: e9201621b798739d4d7dd197f15188103e9d359a
Summary:
Fewer clones, better code.
Note that in some cases we would previously have a fn that takes `ctx` by
ownership and just passes it through to some other fn outside of the
`cross_repo_sync`. I triead to make all such functions borrow and clone instead
in order to push cloning to the leaf fns of `cross_repo_sync`.
Reviewed By: StanislavGlebik
Differential Revision: D24538028
fbshipit-source-id: 8a3e78d4076b34d8b1767cdee1db3fdbb7acb4f7
Summary: SQLBlob GC (next diff in stack) will need a ConfigStore in SQLBlob. Make one available to blobstore creation
Reviewed By: krallin
Differential Revision: D24460586
fbshipit-source-id: ea2d5149e0c548844f1fd2a0d241ed0647e137ae
Summary: update phases imports so that BoxFuture is the futures03 version
Reviewed By: krallin
Differential Revision: D24531551
fbshipit-source-id: 1debb007456292fed1113f8c46e019bef27255c2
Summary: Use the new BonsaiDerived::fetch_derived from the walker when scrubbing fsnode derived data. This saves a round trip to derived data and simplifies bonsai_to_fsnode_mapping_step().
Reviewed By: krallin
Differential Revision: D24311164
fbshipit-source-id: b26c6bdef548ca15c0cdbffde675f1a11d8b510e
Summary:
Add BonsaiDerived::derive03 for new callsites that are 0.3/async aware.
Plan is to migrate callsites and then rename BonsaiDerived::derive03 to derive.
Reviewed By: StanislavGlebik
Differential Revision: D24389763
fbshipit-source-id: 4a6f351b9e0c5a2e9a9c47429764f88c47caff08
Summary: No callsites when I search
Reviewed By: StanislavGlebik
Differential Revision: D24389765
fbshipit-source-id: ed1985922a9c1af51afb37fb5e331d32c29fd09e
Summary:
This diff adds a method BonsaiDerived::fetch_derived which is useful for callers currently doing BonsaiDerived::is_derived, and then BonsaiDerived::fetch_derived.
Using the new method cuts out one round trip to derived data, and simplifies calling code. Usage of this in the walker when scrubbing fs nodes is in the next diff.
Reviewed By: StanislavGlebik
Differential Revision: D24311166
fbshipit-source-id: dc52a15076768a6a15ede6dd5872aaaf3f552b66
Summary:
Before allowing a bookmark to be moved, check whether the repo is read-only or not,
and deny the bookmark move if the repo is read-only.
This can be bypassed by setting the `BYPASS_READONLY=true` pushvar.
Reviewed By: krallin
Differential Revision: D24063027
fbshipit-source-id: 7b5f66e5d89058f6c5765b020149a81b674e795c
Summary: There's no need to create an empty commit.
Reviewed By: krallin
Differential Revision: D24537264
fbshipit-source-id: 41a80fcbf5cab28b731dae53c58653ec4b434716
Summary: To make it clearer what's going on
Reviewed By: krallin
Differential Revision: D24534961
fbshipit-source-id: 049edbf1b6b648670a6c6ef68b14698a27ea1a0e
Summary:
There are two new options (and one option was renamed):
1) First option (--file-size-limit) adds a way to limit total size of moved files
2) Second option (--lfs-threshold) adds a way to exclude size of the files that
are above the threshold from --file-size-limit calculations. This option can be
useful if we want to limit the total size of the bundle that will be created in
mononoke hg sync job - files that are above or equal to lfs threshold are not included
in the bundle, so no need to count their size.
Reviewed By: krallin
Differential Revision: D24534959
fbshipit-source-id: 6323e4d52802c00dc67270b08f671b0789ab262f
Summary:
a simple command that can be used to pushrebase a commit. Note that this
command is very low level so it skips a lot of checks that we do during the
normal push process (e.g. it doesn't run hooks).
Reviewed By: krallin
Differential Revision: D24534960
fbshipit-source-id: 0f1e27005fa450c86c310d66e3215747ca6a49e2
Summary: This change adds wire types for the history API in the most straightforward way possible. In a future change, I'll move the `WireHistoryEntry` / `HistoryResponseChunk` conversion logic into the `ToApi` implementation. This implementation also doesn't add per-item errors, or standardize `history` to match the protocol evolution standards used by `trees`.
Reviewed By: kulshrax
Differential Revision: D24342046
fbshipit-source-id: d46403a823f2a1e89ad9d6d2074241d8bfe4810e
Summary:
Removes top-level metadata from `TreeEntry` and add a new, specialized type for carrying child entry metadata, `TreeChildEntry`. This change also fully removes the `revisionstore_types::Metadata` from EdenAPI trees, which is only used on files.
In a follow up change I'll optimize handlings of paths / path segments in `TreeChildEntry` keys. Right now they need to be joined manually.
Reviewed By: kulshrax
Differential Revision: D24434716
fbshipit-source-id: d0739471b1f6cef58b435e10b5fb774bfb08f7f6
Summary: Rather than silently dropping entries which cannot be fetched, this change has the `WireTreeEntry` type carry optional error information, allowing it to be (de)serialized to / from `Result<TreeEntry, EdenApiServerError>` instead of a bare `TreeEntry`. Currently, handling of these failures is up to the individual application code, but it might be useful to introduce utility functions to drop failed entries and log errors.
Reviewed By: kulshrax
Differential Revision: D24315399
fbshipit-source-id: 94e4593b77cf2dc12d0dcc93d174c8a4eda95344
Summary: In D24447404, I provided some utility functions that allowed me to avoid constructing and/or passing around a ConfigStore. Remove those functions and fix up the code to run.
Reviewed By: krallin
Differential Revision: D24502692
fbshipit-source-id: 742dbc54fbcf735895d6829745b9317af14dfa0b
Summary: It's designed as a singleton store for normal use - rather than have lots of ways to get different config stores, let's use one global store
Reviewed By: krallin
Differential Revision: D24447404
fbshipit-source-id: 6ecc46351b14794471f654cec98527a11a93d3ef
Summary:
Adding number of changed files to mononoke_commit and mononoke_commit_draft scribe categories.
I need to create scuba table for mononoke_commit scribe category.
Reviewed By: krallin
Differential Revision: D24390738
fbshipit-source-id: 80b9917751a42377b49190cbb2f650a21b086e74
Summary:
- convert ManifestOps to new style futures
- at this point `//eden/manifest:manifest` crate is completely free from old style futures
Reviewed By: krallin
Differential Revision: D24502214
fbshipit-source-id: f1cdb11bd8234f22af5c905243f71e1e9fca11f1
Summary: `derive_manifest_*` functions are now based on new futures and use new bounded_traversal
Reviewed By: krallin
Differential Revision: D24448824
fbshipit-source-id: 5af06851f9c48a73e53daafe1f4972c186a5ca80
Summary: Somewhat more convenient to work with, no need to look inside.
Reviewed By: StanislavGlebik
Differential Revision: D24474898
fbshipit-source-id: 7ee0920e7d0d5a2102c68695a5cc0d9e237d958d
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary: let's make sure we don't create too big commits by setting the limit
Reviewed By: ikostia
Differential Revision: D24451569
fbshipit-source-id: 7b4f653509fce47d921e55dddad2de49ac5672a3
Summary:
This is a command that creates a commit that copies one directory into another.
Later you can take a commit and push it.
At the moment the command is very simple and also not very useful. At the very
least we need a way to limit the size of the commit, and this option is coming
in the next diff.
Note that the command doesn't overwrite the files if they exist in the target
directory - we may add an option to do so in future. Also if a file exists in
target directory but doesn't exist in source directory then it's left
untouched.
Reviewed By: ikostia
Differential Revision: D24451568
fbshipit-source-id: 2c7b945aadc99a1ca960349992c3684ef02ba35e
Summary:
This is to be able to run the hook tailer on fbsource commits, which had been
push-redirected from ovrsource.
Reviewed By: StanislavGlebik
Differential Revision: D24447814
fbshipit-source-id: bfb164ec45dd2d4d5c3c83f133b48000a71a50a3
Summary: As well as the unixname of the pusher, log all identities provided by the pusher.
Reviewed By: krallin
Differential Revision: D24446453
fbshipit-source-id: 0854b3a10221da16d87b53fd849f033f7bb2a27e
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`
Reviewed By: johansglock
Differential Revision: D24448419
fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
Summary: The hostname is of limited use when running on Tupperware - add in the session ID, which lets us look at what happened.
Reviewed By: StanislavGlebik
Differential Revision: D24361320
fbshipit-source-id: 0ac2c64d68915288b3241320c7ee14452b754ec8
Summary:
I removed --debug flag for bookmarks validator on tw because it was too spammy,
but now we get no output at all. Let's make it info so that we can see it in tw
output
Reviewed By: ikostia
Differential Revision: D24445829
fbshipit-source-id: d37c41be46794867f91ecf66f0318e7b2d660d85
Summary: As we saw today it was showing entries in incorrect order
Reviewed By: ikostia
Differential Revision: D24418800
fbshipit-source-id: c926c7904338ae40b26fe08cb1ac973e384bad28
Summary:
Add a method to diff file contents directly, even if their paths are not known.
This provides a headerless diff, just of the content differences.
Reviewed By: ahornby
Differential Revision: D24200803
fbshipit-source-id: 32cf461d0c43a7b785ae58bec284978ff7976702
Summary:
Rather than requiring the caller to construct a changeset specifier for known changeset
ID types, accept anything that implements `Into<ChangesetSpecifier>` and add automatic
conversions for the specifier types.
This makes use of the Mononoke API a bit more ergonomic in tests.
Reviewed By: krallin
Differential Revision: D24417892
fbshipit-source-id: ff84c833442d621d71dbadca596222a2e13d5ca1
Summary:
Prior to this diff, validation logic did not work when target repo was a large
one (although this limitation wasn't advertised). This was becase we explicitly
never apply a `Mover` to the target repo paths and therefore expect every
single target repo path to have an equivalent in the source repo. This is true
when the target repo is small, but not true when it is large.
It is pretty easy to make this logic direction-agnostic: just proactively check
whether target repo paths rewrite into nothingness or not.
Reviewed By: StanislavGlebik
Differential Revision: D24399029
fbshipit-source-id: 9ca5bb03760e662ff6756c27c0c77204abdb38de
Summary: This command will help us understand if it is safe to enable push-redirection with a certain CommitSyncConfigVersion,
Reviewed By: StanislavGlebik
Differential Revision: D24345898
fbshipit-source-id: c2a4c034a9203025ce3534986b3bb986784ff2b1
Summary:
We'll use this command to change the mapping version we use when doing push
redirection.
Reviewed By: ikostia
Differential Revision: D24392308
fbshipit-source-id: 4dab01c0e58a8953a0c6c84c7c166977a6baf00f
Summary:
Previously it was just taking the current version. Instead of guessing let's
allow users to specify whatever version they want to rewrite it with.
Reviewed By: ikostia
Differential Revision: D24360918
fbshipit-source-id: d9f4c55cd66931a9f2ab7da3474d2152d77525d0
Summary:
update_mapping method was silently setting current mapping version. This was
very error-prone and we had a replacement update_mapping_with_version. This
diff removes update_mapping completely.
Reviewed By: krallin
Differential Revision: D24360105
fbshipit-source-id: 57761f4279f75032e9d4ec88a45e5199e250247a
Summary:
This diff add new mode of tailing based on derived data graph, it uses same functionality as backfill.
- `tail_batch_iteration` uses `bounded_traversal_dag` directly instead of leveraging `DeriveGraph::derive` so we could warm-up dependencies for each node before calling `DerivedUitls::backfill_batch_dangerous`
Reviewed By: StanislavGlebik
Differential Revision: D24306156
fbshipit-source-id: 006cb6d4df9424cd6501cb4a381b95f096e70551
Summary:
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/15
With the update from rustc 1.44 to 1.47 the doctests started to break when building a crate that depends on a different crate of the same name, see this: https://github.com/rust-lang/cargo/issues/6819
This diff also removes the TravisCI build as it was broken and unused for some time.
Reviewed By: krallin
Differential Revision: D24390815
fbshipit-source-id: fa9c78ef394640afbf8c419f8276bbf339960406
Summary:
I updated the client to send this earlier this week. Let's start collecting it
server side. The bottom line is this will let us identify when we reject a
client that is done retrying.
Reviewed By: HarveyHunt
Differential Revision: D24361884
fbshipit-source-id: b31e53c7dc989f1698e28b2a0bf14dc3fdbb507a
Summary: It will be much more convenient for end user to know why certain limit is enforced.
Reviewed By: krallin
Differential Revision: D24364473
fbshipit-source-id: ad86384f7f7a246bc3735028339cfd54a1d0254a
Summary:
I think that this: 6293dca1e8
"broke" (in a sense that they need to be adjusted - run with --interactive) some of our integration tests.
In fbcode seem like it was D24363513.
This commit deleted derived Debug trait of enum IpAddr in favor of implementing it with already existing fmt::Display.
I think this change landed with rust 1.7.0 which hit fbcode recently.
Reviewed By: StanislavGlebik
Differential Revision: D24382155
fbshipit-source-id: bce5998015bfdd7c0b01a5c6ee5a661f9f00df5a
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/67
With this change it will be possible to build dependencies of and run integration tests using getdeps.py.
This is the first goal of Q4 as per https://fb.quip.com/v8YzAYNSYgot: "Get Open Source version of integration tests running on Legocastle".
Before this diff:
The OSS integration tests run now on GitHub by:
- Building some test dependencies with getdeps.py
- Building some test dependencies with homebrew/apt-get
- Running tests via python script
The OSS integration tests were not running on Sandcastle.
After this diff:
The OSS integration tests run on Github by:
- Building and executing tests via getdeps.py (execution of tests happens by getdeps.py calling Make calling python script)
The OSS integration tests run on Sandcastle using the same getdeps.py setup as Github.
Reviewed By: krallin
Differential Revision: D24253268
fbshipit-source-id: cae249b72d076222673b8bbe4ec21866dcdbb253
Summary:
This updates multiplexedblob and logblob to capture perf counters for the
operations we run, and log them to Scuba. Along with the previous diffs in this
stack, this gives us the number of Manifold retries, total delay, and conflicts
all logged to blobstore trace on a per-operation basis.
Reviewed By: HarveyHunt
Differential Revision: D24333039
fbshipit-source-id: 9c7d0a467f8df08dcb2a0d3bb6b543cdb3ea1d90
Summary:
This updates ManifoldBlob to log the aformentioned data points to perf
counters. There's a bit of refactoring that also had to go into this to make
`ctx` available everywhere it's needed.
Reviewed By: aslpavel
Differential Revision: D24333040
fbshipit-source-id: 1b63bcd1e1ee36bae4dbbc1da053c7f1bdf96675
Summary:
This adds support for "forking" perf counters at a point in the stack, giving
you a CoreContext that logs to one or more sets of perf counters.
This is useful for low level operations where we want to collect granular more
logging data — in particular blobstore operations, where we'd like to collect
the time spent waiting for Manifold retries or the number of Manifold retries
in blobstore trace for each individual blobstore operation (we can't do that
using the `CoreContext` we have because that would be missing
The implementation supports a list of reference counted perf counters in the
CoreContext. When you want to add a new counter, we replace the list with a new
one, and give you a reference to the one you just added. When you write, we
write to all perf counters, and when you read, we read from the "top" perf
counter (which is always there). To read from one of the forked counters, you
use the reference you got whne you created it.
Reviewed By: aslpavel
Differential Revision: D24333041
fbshipit-source-id: ce318dfc04a1ea435b2454b53df4cae93d57c0a5
Summary:
The x-repo usually have "noop" mapping i.e. mapping that doesn't change the
paths (it might have arbitrary name though)
It's useful for commits that are identical between small and large repo to be
able to backfill this mapping. This diff adds a command to do that.
Reviewed By: krallin
Differential Revision: D24337281
fbshipit-source-id: 89a058f95677e4a5c8686122a317eadf8b1bb995
Summary: It will be used in the next diff, so let's move it to a separate function.
Reviewed By: krallin
Differential Revision: D24334717
fbshipit-source-id: e50d13d45c633397504cf08954f2ced9ace8f570
Summary: Convert derived data utils to use new style futures
Reviewed By: StanislavGlebik
Differential Revision: D24331068
fbshipit-source-id: ad658b278802afa1e4ecd44c5a24164135748790
Summary:
This is needed to be able to use `has_redaction_root_cause()` with a metadata
rebuilding error.
Reviewed By: StanislavGlebik
Differential Revision: D24360816
fbshipit-source-id: 388df8cedb769ff001bfe4ff9cd5063ccd9de9f1
Summary:
This is in line with other changes we're making to map logic now. Note that
apart from checking in-repo prefix-free-ness of the map, this also checked the
same across many small repos. It probably does not make sense to do that either
now that we allow non-prefix free maps within a repo.
Reviewed By: StanislavGlebik
Differential Revision: D24348161
fbshipit-source-id: caaa22953c8a15a08607157b99c9f0fd0edf633f
Summary:
Until we have the same standards for the native and push-redirected pushes,
these need to be automatically bypassed.
Reviewed By: krallin
Differential Revision: D24357372
fbshipit-source-id: f85459145f6a5217c07445d7017f3b11ed1284a7
Summary:
Except for the tail mode of x_repo_sync_job which we use normally there's also
"once" mode which means "sync a single commit". Previously it did just that -
synced a single commit and failed if parents of this commit weren't synced.
However this is very unpleasant to use - instead let's change the semantics to
sync the commit and all of its ancestors.
also I made target_bookmark an optional parameter - sometimes we just want to sync a commit without any bookmarks at all.
Reviewed By: mitrandir77
Differential Revision: D24135771
fbshipit-source-id: 341c1808a44c58f89536b8c07394b77d8ced3f37
Summary: These were stabilized in 1.45.0 and 1.47.0 respectively.
Reviewed By: StanislavGlebik
Differential Revision: D24353680
fbshipit-source-id: f2afe906e5260b1b360455acc20d9a806c988c9c
Summary:
Update from 1.44.0. Updates have been blocked because of a bad interaction
between platform007+mode/opt-clang-thinlto+gold. Now that the default linker is
lld, perhaps this is no longer an issue.
Various tweaks due to updates:
- `atomic_min_max`, `const_transmute`, `inner_deref`, `ptr_offset_from` and `str_strip` are now stable
- `asm` renamed to `llvm_asm`
- `intra_doc_link_resolution_failure` lint renamed to `broken_intra_doc_link` (ndmitchell I didn't fix Gazebo because you'd explicitly suppressed the warning - I'll let you work out what to do with that)
(This is caused by incompatibility between the llvm used by rustc in
platform007 and llvm-fb. In platform009, rustc uses llvm-fb directly so there's
no scope for incompatibility.)
Reviewed By: dtolnay
Differential Revision: D24288638
fbshipit-source-id: 5155d85c186fd79d3cc86cb0bb554ab77d76c12c
Summary: Rename futures01 types from Foo to Foo01 in the top level lib.rs and derive_impl.rs files in preparation for adding a trait method that returns new futures
Reviewed By: aslpavel
Differential Revision: D24311165
fbshipit-source-id: 4f3b12ba3eaf8023959d6d4bbb4568d191b1fffb
Summary: This allows the user to see how the mover of a particular version operates on any given path.
Reviewed By: StanislavGlebik
Differential Revision: D24335975
fbshipit-source-id: f67847112eb0d3c8c49584604e2f9d93579cdde4
Summary:
Those tests are kinda broken in a number of ways right now.
First, they try to connect to a prod DB to record what bundle they just pushed.
That's not ideal, so this adds a flag to have them not do this.
Second, they are racy by design, and they mostly don't pass at tall in
mode/dev.
The way we make the tests run here is by having them forwardfill for 10 seconds
then give up, and we hope that during that time, they've fetched the bundles
they should fetch from SQL, and synced them. However, that's not really
sufficient because establishing your first connection to SQL from a mode/dev
binary is quite slow, so in the 10 seconds, you might pick up your first
bundle, start replaying it, then exit before you get to the second one.
To fix this, this diff updates the fillers to expect a specific number of
bundles to replay. We still have a limit on the number of total iterations to
avoid letting the tests hang if the number of bundles isn't the one we expect.
Fixing this revealed further breakage, which I solved earlier in this stack.
Unfortunately, it's not sufficient to make test-commitcloud-reversefiller.t
work on Python 3, because the infinitepush extension appears to be broken
server side. I'll file a task for the Mercurial oncall for that.
Reviewed By: mitrandir77
Differential Revision: D24277474
fbshipit-source-id: 0a5e1f7db8dc0c0068b0fc203abc0503226107ec
Summary:
Our case conflict checking is very inefficient on large changesets. The root
cause is that we traverse the parent manifest for every single file we are
modifying in the new changeset.
This results in very poor performance on large changes since we end up
reparsing manifests and doing case comparisons a lot more than we should. In
some pathological cases, it results in us taking several *minutes* to do a case
conflict check, with all of that time being spent on CPU lower-casing strings
and deserializing manifests.
This is actually a step we do after having uploaded all the data for a commit,
so this is pure overhead that is being added to the push process (but note it's
not part of the pushrebase critical section).
I ended up looking at this issue because it is contributing to the high
latencies we are seeing in commit cloud right now. Some of the bundles I
checked had 300+ seconds of on-CPU time being spent to check for case
conflicts. The hope is that with this change, we'll get fewer pathological
cases, and might be able to root cause remaining instances of latency (or have
that finally fixed).
This is pretty easy to repro.
I added a binary that runs case conflict checks on an arbitrary commit, and
tested it on `38c845c90d59ba65e7954be001c1eda1eb76a87d` (a commit that I noted
was slow to ingest in commit cloud, despite all its data being present already,
meaning it was basically a no-op). The old code takes ~3 minutes. The new one
takes a second.
I also backtested this by rigging up the hook tailer to do case conflict checks
instead (P145550763). It is about the same speed for most commits (perhaps
marginally slower on some, but we're talking microseconds here), but for some
pathological commits, it is indeed much faster.
This notably revealed one interesting case:
473b6e21e910fcdf7338df66ee0cbeb4b8d311989385745151fa7ac38d1b46ef (~8K files)
took 118329us in the new code (~0.1s), and 86676677us in the old (~87 seconds).
There are also commits with more files in recent history, but they're
deletions, so they are just as fast in both (< 0.1 s).
Reviewed By: StanislavGlebik
Differential Revision: D24305563
fbshipit-source-id: eb548b54be14a846554fdf4c3194da8b8a466afe
Summary:
I'm reworking some of our case conflict handling, and as part of this, I'm
going to be using check_case_conflicts for all our checking of case conflicts,
and notably for the case where we introduce a new commit and check it against
its parent (which, right now, does not check for case conflicts).
To do this and provide a good user experience (i.e. indicate which files
conflicted and with what), I need `check_case_conflicts` to report what files
the change conflicts with. This is what this diff does.
This does mean a few more allocations, so I "paid those off" by updating our
case lowering to allocate one fewer Vec and one fewer String per MPathElement
being lowercased.
Reviewed By: StanislavGlebik
Differential Revision: D24305562
fbshipit-source-id: 8ac14466ba3e84a3ee3d9216a84c2d9125a51b86
Summary:
Those tests need an empty blobstore, but to do this, they `fixtures::linear`,
which a) instantiates a whole blobrepo, and b) fill it with commits we never
use.
In particular, a) is kinda annoying, because that creates a dependency on
blobrepo and therefore on most of Mononoke. This makes the tests much slower to
build than they could be.
This diff updates the test to just get their memblob directly instead of via a
fixture, thus bringing down the number of Mononoke crates we pull in from 120
to 21.
Reviewed By: StanislavGlebik
Differential Revision: D24305564
fbshipit-source-id: 91c9423f6284f985461744d9c61ce207024a65c8
Summary:
Current large to small movers do not cover some of the edge cases, and this
diff attempts to fix them. More details are in the comment
Reviewed By: aslpavel, ikostia
Differential Revision: D24328574
fbshipit-source-id: 35cceff61f21603baf446d83e8daa4bda2f17d2a
Summary:
This trait is no longer used all that much outsides of a handful of tests, the
walker, and an admin subcommand, as it has been replaced by the `Manifest`
trait, which works over all kinds of Manifests, and has stronger typing (its
sub-entries always have a path, and they are wrapped in an enum that knows if
they're leaves or trees).
This left a bunch of old legacy code here or there, which is worth removing
to make sure we don't introduce any new callsites to this. Another motivation
is that this legacy code is often not very compatible with new code, and has
historically made it a bit tricky (everything owns a blobstore in this code,
which is pretty awkward and not at all how we do things nowadays).
There is, I think, a bit more potential here since we could also perhaps try to
remove the `HgBlobEntry` struct, but that has a callsites still, so I'm not
doing this here.
Reviewed By: StanislavGlebik
Differential Revision: D24306946
fbshipit-source-id: 8a73dbbf40a904ce19ac65d791b732091c206263
Summary: We need to wait for records before checking said records.
Reviewed By: StanislavGlebik
Differential Revision: D24329119
fbshipit-source-id: c06564b98c12c805444a7e0c3fcc968a8b8cc175
Summary:
We don't want to log slow derivations in tests. To do that, let's look at at
unable to decide if we should log.
We only log if the threshold is > 0, so since the default is 0, that means we
won't log unless a config overrides it (which won't be the case in tests, but
will be the case everywhere else: D24329372).
Reviewed By: StanislavGlebik
Differential Revision: D24329377
fbshipit-source-id: e9d70236c614780e383870297d08afaf64450383
Summary:
We don't give `jq` a program here, and it looks like depending on the exact
setup, this might result in it printing the help instead of running, as
reported by Jun here:
https://fb.workplace.com/groups/scm.mononoke/permalink/1280281002334472/
I think this is the behaviro inquestion:
https://github.com/stedolan/jq/issues/1110. I'm guessing something might have
changed in jq 1.6 because I am on that version and don't get that behavior but
Jun's output uses jq 1.5.
Let's give jq a program to fix this.
Reviewed By: StanislavGlebik
Differential Revision: D24329472
fbshipit-source-id: f4ba4246770dd79630433a3c46c8bbd38aa4c842
Summary: Add support for deriving all types of derived data for a given repository
Reviewed By: StanislavGlebik
Differential Revision: D23842909
fbshipit-source-id: 2fa5c4a9444169b26c5cf70d91a6cc707cca8022
Summary: This change make it possible to find underived changesets for multiple derived data types at once, and then batch derive everything taking into account dependencies between derived data types.
Reviewed By: StanislavGlebik
Differential Revision: D23762152
fbshipit-source-id: 8112fc5deb86f3c55f2fa5079cf917a0506f045c
Summary: Introduces fetching of child entry IDs, and child file metadata for a specified tree manifest ID. The aux data lookup will only be performed if `with_file_metadata` is set, which is actually kind of wrong. Instead `with_children` from the wire type should be exposed in the API request type, and `with_*_metadata` should be hidden or used for data other than the child entry `Key`s.
Reviewed By: kulshrax
Differential Revision: D23886678
fbshipit-source-id: 0cba72cea7be47ae3348a406d407a19b60976c0c
Summary: This code predates the use of `pin_project`, which simplifies the implementation of pinned types using proc macros. We now use `pin_project` in most other places in this crate where we define custom `Future`s and `Stream`s, so let's use it in `SignalStream` as well. (This makes the code easier to modify and removes the need for `unsafe` code.)
Reviewed By: farnz
Differential Revision: D24285792
fbshipit-source-id: da018b9ed62c33d9d06ecf6caa4748969a057e66
Summary:
Add a stream combinator that causes a `TryStream` to end upon encountering the error. The resulting stream remains a `TryStream`, but will always terminate after yielding an error item.
Later in this stack, this will be used in the LFS server to cause mid-stream errors to always terminate the response.
Reviewed By: krallin
Differential Revision: D24301254
fbshipit-source-id: e71ddc16e79caeb195e23be7d6a93ee63a96a713
Summary:
We had a small issue with our logging when pushredirection was enabled. See for
example https://fburl.com/scuba/mononoke_test_perf/orsld6yh. If you count the
number of "Start processing" and "Command processed" entries you'll notice that
there's one more instance of the former. That's wrong, and turned out this
additional "Start processing" line is coming from backsyncing.
Turned out that the problem was two fold:
1) start_command() logs "Start processing" row and also it sets "Start
processing" as default message for every next scuba calls. That doesn't seem
intentional, so this diff clones scuba sample before logging "Start
processing" to avoid "poisoning" the "tag" field
2) Backsyncing weren't setting the log_tag at all. So this diff fixes it
Reviewed By: ikostia
Differential Revision: D24305167
fbshipit-source-id: 3448b88693e0c5c8f5a9b90f602bc2dfe99db15a
Summary:
In D19023924 (e9df42c192) ikostia allowed non-prefix free movers, however he left the
safeguard in just in case. Well, now we finally need to use non-prefix-free
mapping, so let's remove the safeguard.
Reviewed By: ikostia
Differential Revision: D24277040
fbshipit-source-id: 4d658ad813171ab0dcb23656e95e3e443ec9961a
