Commit Graph

54 Commits

Author SHA1 Message Date
Mark Thomas
0518d6a0fb scs_server: add commit_history to return the history of commits
Summary:
Add the `commit_history` method to the source control service.

This methods returns the history of the commit, directly from the changelog.

This differs from `commit_path_history` in that it includes all commits, whereas `commit_path_history` will skip commits that do not contain changes that affect the path.

Differential Revision: D21201705

fbshipit-source-id: dbf1f446c106620620343122176eccd5d809779c
2020-04-23 07:14:49 -07:00
Mark Thomas
6424064215 mononoke_api: add changeset history
Summary:
Add a method to get the history of a changeset.  This differs from the history
of a changeset path, even the history of the root directory, in that all
changesets are included, even empty ones.

Differential Revision: D21179877

fbshipit-source-id: e19aac75fc40d8e9a3beb134e16a8cdfe882b791
2020-04-23 07:14:49 -07:00
Mark Thomas
d878593d3d mononoke_api: add test for commit_path_history
Summary: Add a new unit test for `commit_path_history`.  We will use this test to contrast against `commit_history`.

Differential Revision: D21179878

fbshipit-source-id: 318aa34d8d80f61c1e52d4053d5aead5a71e864c
2020-04-23 07:14:49 -07:00
Aida Getoeva
25eff1c91e mononoke/scs-log: integrate deleted manifest (linear)
Summary:
Use deleted manifest to search deleted paths in the repos with linear history. For merged history it returns error as there was no such path.
Commit, where the path was deleted, is returned as a first commit in the history stream, the rest is a history before deletion.

Reviewed By: StanislavGlebik

Differential Revision: D20897083

fbshipit-source-id: e75e53f93f0ca27b51696f416b313466b9abcee8
2020-04-14 18:27:39 -07:00
Aida Getoeva
2df76d79c8 mononoke/scs-log: add history stream terminator
Summary:
`log_v2` supports time-filters and that means it needs to be able to drop the history stream if the commits got older than the given time frame. (if not it just traverses the whole history...)
However, it cannot be done from the SCS commit_path API or from changeset_path, because they already receive history stream where commits are not ordered by creation time. And the naive solution "if next commit in the stream is older than `after_ts` then drop" won't work: there might be another branch (commit after the current one) which is still **in** the time frame.

I added a terminator-function to the `list_file_history` that is called on changeset id, for which a new fastlog batch is going to be fetched. If terminator returns true, then the fastlog is not fetched and the current history branch is dropped. All ready nodes are still streamed.
```
For example, if we have a history of the file changes like this:

      A 03/03    ^|
      |           |
      B 02/03     |
      |           |  - one fastlog batch
      C 01/03     |
      | \         |
02/01 D  E 10/02 _|  - let assume, that fastlog batches for D and E ancestors are needed to prefetch
      |  |
01/01 F  G 05/02

# Example 1

We query "history from A after time 01/02"

The old version would fetch all the commits and then filter them in `commit_path`. We would fetch both fastlog batches for the D branch and E branch.

With the terminator, `list_file_history` will call terminator on commit D and get `true` in return and then will drop the D branch,
then it will call terminator on E and get `false` and proceed with fetching fastlog for the E branch.

# Example 2

We query "history from A after time 01/04"

The old version would fetch all the commits and then filter them in `commit_path`, despite the fact that
the very first commit is already older than needed.

With the terminator it will call terminator on A and get `true` and won't proceed any further.

Reviewed By: StanislavGlebik

Differential Revision: D20801029

fbshipit-source-id: e637dcfb6fddceb4a8cfc29d08b427413bf42e79
2020-04-07 07:08:24 -07:00
Aida Getoeva
2dcfcbac62 mononoke/fastlog: asyncify part of ops
Summary: Asyncified main functions of the fastlog/ops, so it'd be easier to modify them and proceed with the new features.

Reviewed By: StanislavGlebik

Differential Revision: D20801028

fbshipit-source-id: 2a03eedca776c6e1048a72c7bd613a6ef38c5c17
2020-04-07 07:08:24 -07:00
Mark Thomas
640f272598 migrate from sql_ext::SqlConstructors to sql_construct
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.

In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }

[storage.blobstore.multiplexed]
queue_db = { remote = {
    db_address = "queue_database",
    sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```

This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.

**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
    * `primary`, the database used for main metadata.
    * `filenodes`, the database used for filenodes, which may be sharded or unsharded.

More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.

New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }

[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```

The `sql_construct` crate facilitates this by providing the following traits:

* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.

Sql database managers should implement:

* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
    * **SqlConstructFromMetadataDatabaseConfig** if they are not shardable.  By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
    * **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable.  They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.

Reviewed By: StanislavGlebik

Differential Revision: D20734883

fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
2020-04-02 05:27:16 -07:00
Stanislau Hlebik
03c73035cb mononoke: use more efficient copy fetching in scs server
Summary:
This is a followup from D20766465. In D20766465 we've avoided re-traversing
fsnodes for all entries except for copy/move sources. This diff make copy/move
sources fetching more efficient as well.

It does by sending a find_entries() request to prefetch all fsnodes

Reviewed By: mitrandir77

Differential Revision: D20770182

fbshipit-source-id: 7e4a68a2ded20b2895ee4d1c4f8fd897dbe1c850
2020-04-01 06:00:26 -07:00
Stanislau Hlebik
2eebab89c5 mononoke: make scs diff path-only call faster
Reviewed By: markbt

Differential Revision: D20766465

fbshipit-source-id: fa78fa66da32caddcd582a6500b9a8393904f687
2020-03-31 07:40:14 -07:00
Stanislau Hlebik
11f891a178 mononoke: remove allow dead_code
Summary: Looks like it's not dead anymore

Reviewed By: krallin

Differential Revision: D20766497

fbshipit-source-id: c49ae3b6c8a660b33e61e65adda94f78addd1498
2020-03-31 07:30:32 -07:00
Aida Getoeva
4815913d9e mononoke/scs: use changeset info in changeset context
Summary:
Changeset info is less expensive to load than Bonsai, so we would like to use it in SCS as a source of commit info if possible.

This diff adds a method into the Repo object that checks `changeset_info` derivation is enabled for the repo in the `DerivedDataConfig`. If derivation is enabled, then SCS derives this info otherwise it awaits for bonsai and converts it into the changeset info. The bonsai fields aren't copied but moved to the `ChangesetInfo`.

Reviewed By: StanislavGlebik

Differential Revision: D20282403

fbshipit-source-id: b8ddad50dcd5c6de109728b2081ca5a13f440988
2020-03-19 12:16:40 -07:00
Arun Kulshreshtha
2e39e6e991 mononoke_api: add test for HgRepoContext::trees_under_path
Summary: Add a unit test for the new `trees_under_path` method.

Reviewed By: StanislavGlebik

Differential Revision: D20460242

fbshipit-source-id: 92ec983bc8307fbfa7862ba17b603b4150c42709
2020-03-17 15:07:54 -07:00
Arun Kulshreshtha
bc5b530959 hgproto: use Option<MPath> instead of Bytes for path in GettreepackArgs
Summary: title

Reviewed By: StanislavGlebik

Differential Revision: D20459401

fbshipit-source-id: 3473c2eef39b0d51e6133a5f575d2fd7bef3bd97
2020-03-17 15:07:54 -07:00
Arun Kulshreshtha
ff055aeea0 mononoke_api: Add trees_under_path method to HgRepoContext
Summary: Add a `trees_under_path` method to `HgRepoContext` which is basically just a way to call Mononoke's implementation of `gettreepack`. This is required for EdenAPI's `/trees/prefetch` endpoint, which basically returns the results of a `gettreepack` request via HTTP.

Reviewed By: StanislavGlebik

Differential Revision: D20414860

fbshipit-source-id: 840bd5d19a10625b78758949b290f1947bdb94be
2020-03-13 14:32:20 -07:00
Mateusz Kwapich
a522cde826 add generation numbers to the changeset API
Summary:
Let's return those to the caller. The changeset is already fetched at this
point so this should come at no extra cost.

Reviewed By: markbt

Differential Revision: D20426745

fbshipit-source-id: 7e28f5fd44efdb502b8e37a7afac3ea9113e2c5e
2020-03-13 08:31:07 -07:00
Mateusz Kwapich
e70e7a9017 rename get_generation_number_by_bonsai to get_generation_number
Summary: I'm about to introduce one more usecase of it so let's rename it first.

Reviewed By: farnz

Differential Revision: D20393776

fbshipit-source-id: d74146fa212cdc4989a18c2cbd28307f58994759
2020-03-13 08:31:07 -07:00
Arun Kulshreshtha
7a22f95631 mononoke_api: use fbinit::compat_test
Summary:
Avoid boilerplate in unit tests by consistently using `fbinit::compat_test`.

This is a very mechanical change; most of the modified lines are due to indentation changes.

Reviewed By: krallin

Differential Revision: D20414564

fbshipit-source-id: 11c6bd702de28c8a21978f0345d2b0ff8a574e72
2020-03-12 12:34:07 -07:00
Stanislau Hlebik
e3bf91e944 mononoke: add generate_placeholder_diff option
Summary:
Let's use functionality added in D20389226 so that we can generate
diffs even for large files.

I've contemplated between two approaches: either "silently" generate
placeholder diff for files that are over the limit or add a new option
where client can request these placeholders. I choose the latter for a few
reasons:

1) To me option #1 might cause surprises e.g. diffing a single large file
doesn't fail, but diffing two files whose size is (LIMIT / 2 + 1) will fail.
Option #2 let's the client be very explicit about what it needs - and it also
won't return a placeholder when the actual content is expected!
2) Option #2 makes the client think about what it wants to be returned,
and it seems in line with source control thrift API design in general i.e.
commit_file_diffs is not trivial to use by design to because force client to
think about edge cases. And it seems that forcing client to think about additional
important edge case (i.e. large file) makes sense.
3) The code change is simpler with option #2

I also thought about adding generate_placeholder_diff parameter to CommitFileDiffsParams,
but that makes integration with scs_helper.py harder.

Reviewed By: markbt

Differential Revision: D20417787

fbshipit-source-id: ab9b32fd7a4768043414ed7d8bf39e3c6f4eb53e
2020-03-12 12:00:26 -07:00
Stanislau Hlebik
ba871d3bdc xdiff: allow rendering diff for large files
Summary:
The goal of the stack is to support "rendering" diffs for large files in scs
server. Note that rendering is in quotes - we are fine with just showing a
placeholder like "Binary file ... differs". This is still better than the
current behaviour which just return an error.

In order to do that I suggest to tweak xdiff library to accept FileContentType
which can be either Normal(...) meaning that we have file content available, or
Omitted, which usually means the file is large and we don't even want to fetch it, and we
just want xdiff to generate a placeholder.

Reviewed By: markbt, krallin

Differential Revision: D20389226

fbshipit-source-id: 0b776d4f143e2ac657d664aa9911f6de8ccfea37
2020-03-12 04:27:23 -07:00
Aida Getoeva
3e510491cc scs/history: don't buffer stream if there is no need
Summary:
Time filters for file history require fetching changeset or changeset info to decide whether to include the commit into response. To speed up the process instead of sequential mapping, I buffer the map stream in batches of 100.
However it is unfortunate to fetch additional 100 history changesets if there is no time filters and only several commits were requested. Keeping also in mind that the most of the requests don't care about time.
Avoiding that would speed up the history generation,

I changed the changeset_path API so now it just returns the stream of changeset contexts. And commit_path API for history if there is no time filters collects everything to vector, otherwise applies them and also collects to vector. Then this vector is converted into response using FuturesOrdered.

Reviewed By: StanislavGlebik

Differential Revision: D20287497

fbshipit-source-id: 0c6b1315eccddb48f67bf5fa732bdf7c9a54a489
2020-03-10 10:03:05 -07:00
Stanislau Hlebik
4241ca0272 mononoke: add repo name in BlobRepo
Summary:
A lot of callsites want to know repo name. Currently they need to pass it from
the place where repo was initialized, and that's quite awkward, and in some
places even impossible (i.e. in derived data, where I want to log reponame).

This diff adds reponame in BlobRepo

Reviewed By: krallin

Differential Revision: D20363065

fbshipit-source-id: 5e2eb611fb9d58f8f78638574fdcb32234e5ca0d
2020-03-10 08:32:33 -07:00
Stanislau Hlebik
b0cb300af8 mononoke: introduce BlobrepoBuilder
Summary:
The goal of the whole stack is quite simple (add reponame field to BlobRepo), but
this stack also tries to make it easier to initialize BlobRepo.

To do that BlobrepoBuilder was added. It now accepts RepoConfig instead of 6
different fields from RepoConfig - that makes it easier to pass a field from
config into BlobRepo. It also allows to customize BlobRepo. Currently it's used
just to add redaction override, but later we can extend it for other use cases
as well, with the hope that we'll be able to remove a bunch of repo-creation
functions from cmdlib.

Because of BlobrepoBuilder we no longer need open_blobrepo function. Later we
might consider removing open_blobrepo_given_datasources as well.

Note that this diff *adds* a few new clones. I don't consider it being a big
problem, though I'm curious to hear your thoughts folks.

Note that another option for the implementation would be to take a reference to objects
instead of taking them by value. I briefly looked into how they used, and lot of them are passed to the
objects that actually take ownership of what's inside these config fields. I.e. Blobstore essentially takes ownership
of BlobstoreOptions, because it needs to store manifold bucket name.
Same for scuba_censored_table, filestore_params, bookmarks_cache_ttl etc. So unless I'm missing anything, we can
either pass them as reference and then we'll have to copy them, or we can
just pass a value from BlobrepoBuilder directly.

Reviewed By: krallin

Differential Revision: D20312567

fbshipit-source-id: 14634f5e14f103b110482557254f084da1c725e1
2020-03-09 12:05:57 -07:00
Stanislau Hlebik
c1ea56c17c mononoke: asyncify blobrepo_factory
Summary:
Note that comparing to many other asyncifying efforts, this one actually adds
one more clone instead of removing them. This is the clone of a logger field.
That shouldn't matter much because it can be cleaned up later and because this
function will be called once per repo.

Reviewed By: krallin

Differential Revision: D20311122

fbshipit-source-id: ace2a108790b1423f8525d08bdea9dc3a2e3c37c
2020-03-09 05:02:05 -07:00
Lukas Piatkowski
7ddcdd818c mononoke: make sql_ext OSS buildable
Summary: separate out the Facebook-specific pieces of the sql_ext crate

Reviewed By: ahornby

Differential Revision: D20218219

fbshipit-source-id: e933c7402b31fcd5c4af78d5e70adafd67e91ecd
2020-03-06 01:33:38 -08:00
David Tolnay
754a755eee rust: Rename tokio_preview:: to tokio::
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/

This codemod replaces all dependencies on `//common/rust/renamed:tokio-preview` with `fbsource//third-party/rust:tokio-preview` and their uses in Rust code from `tokio_preview::` to `tokio::`.

This does not introduce any collisions with `tokio::` meaning 0.1 tokio because D20235404 previously renamed all of those to `tokio_old::` in crates that depend on both 0.1 and 0.2 tokio.

This is the tokio version of what D20213432 did for futures.

Codemod performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:tokio-preview, 1))" \
| xargs sed -i 's,\btokio_preview::,tokio::,'

rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:tokio-preview,fbsource//third-party/rust:tokio-preview,'
```

Reviewed By: k21

Differential Revision: D20236557

fbshipit-source-id: 15068b93a0a944d6249a1d9f63840a4c61c9c1ba
2020-03-05 14:25:10 -08:00
Liubov Dmitrieva
bb2f81e26b mononoke_api: improve algo for stack calculation
Summary: Improvements aim to minimize number of db queries

Differential Revision: D20280711

fbshipit-source-id: 6cc06f1ac4ed8db9978e0eee956550fcd16bbe8a
2020-03-05 08:31:37 -08:00
Liubov Dmitrieva
047862c02c mononoke: add 'repo_stack_info' API
Summary:
The new API is required for migration Commit Cloud off hg servers and infinitepush database

This also can fix phases issues with `hg cloud sl`.

Reviewed By: markbt

Differential Revision: D20221913

fbshipit-source-id: 67ddceb273b8c6156c67ce5bc7e71d679e8999b6
2020-03-05 05:48:32 -08:00
Arun Kulshreshtha
78adda0589 mercurial_types: make envelope functions use generics instead of trait objects
Summary: Make these functions generic so that callers don't need to construct a trait object whenever they want to call them. Passing in a trait object should still work so existing callsites should not be affected.

Reviewed By: krallin

Differential Revision: D20225830

fbshipit-source-id: df0389b0f19aa44aaa89682198f43cb9f1d84b25
2020-03-03 15:11:04 -08:00
Arun Kulshreshtha
f8d0ad25a2 mononoke_api: add history method to HgFileContext
Summary: Add a method to `HgFileContext` to stream the history of the file. Will be used to support EdenAPI history requests.

Reviewed By: krallin

Differential Revision: D20211779

fbshipit-source-id: 49e8c235468d18b23976e64a9205cbcc86a7a1b4
2020-03-03 15:11:04 -08:00
Arun Kulshreshtha
fa999d9de1 mononoke_api: add HgTreeContext
Summary: Add an 'HgTreeContext' struct to the 'hg' module to allow querying for tree data in Mercurial-specific formats. This initial implementation's primary purpose is to enable getting the content of tree nodes in a format that can be written directly to Mercurial's storage.

Reviewed By: krallin

Differential Revision: D20159958

fbshipit-source-id: d229aee4d6c7d9ef45297c18de6e393d2a2dc83f
2020-03-03 15:11:03 -08:00
David Tolnay
e988a88be9 rust: Rename futures_preview:: to futures::
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/

This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.

This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.

Codemod performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'

rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```

Reviewed By: k21

Differential Revision: D20213432

fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
2020-03-03 11:01:20 -08:00
Stanislau Hlebik
a70ccf6f04 mononoke: make it clearer which repo is accessed in permission error
Summary:
Usually we have only one repo, but in case of xrepo_commit_lookup we actually
have two. It's nice to know which permission failed

Reviewed By: krallin

Differential Revision: D20221509

fbshipit-source-id: ee98845767e72f99027ba18a8c5b374cb6f9f3ab
2020-03-03 07:22:50 -08:00
David Tolnay
fe65402e46 rust: Move futures-old rdeps to renamed futures-old
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.

rs changes performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs,
        rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
        intersect
        rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
    )" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```

Reviewed By: jsgf

Differential Revision: D20168958

fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
2020-03-02 21:02:50 -08:00
Arun Kulshreshtha
bd4a623ccb mononoke_api: Add HgFileContext::new_check_exists
Summary: Add a `new_check_exists` method to `HgFileContext` to allow looking up potentially nonexistent filenodes.

Reviewed By: xavierd

Differential Revision: D20159085

fbshipit-source-id: f6047f7a25f59594823672373d8b35adb49586e1
2020-03-02 09:41:21 -08:00
Arun Kulshreshtha
8ec76a0bce mononoke_api: add hg module
Summary:
Add a a new `hg` module to the `mononoke_api` crate that provides a `HgRepoContext` type, which can be used to query the repo for data in Mercurial-specific formats. This will be used in the EdenAPI server.

Initially, the `HgRepoContext`'s functionality is limited to just getting the content of individual files. It will be expanded to support querying more things in later diffs.

Reviewed By: markbt

Differential Revision: D20117038

fbshipit-source-id: 23dd0c727b9e3d80bd6dc873804e41c7772f3146
2020-03-02 09:41:20 -08:00
Arun Kulshreshtha
615d8392bc mononoke_api: update doc comments on file content methods
Summary: D20121350 changed the methods for accessing file content on `FileContext` to no longer return `Stream`s. We should update the comments accordingly.

Reviewed By: ahornby

Differential Revision: D20160128

fbshipit-source-id: f5bfd7e31bc7e6db63f56b8f4fc238893aa09a90
2020-03-02 09:21:08 -08:00
Thomas Orozco
26ae726af5 mononoke: update internals to Bytes 0.5
Summary:
The Bytes 0.5 update left us in a somewhat undesirable position where every
access to our blobstore incurs an extra copy whenever we fetch data out of our
cache (by turning it from Bytes 0.5 into Bytes 0.4) — we also have quite a few
place where we convert in one direction then immediately into the other.

Internally, we can start using Bytes 0.5 now. For example, this is useful when
pulling data out of our blobstore and deserializing as Thrift (or conversely,
when serializing and putting it into our blobstore).

However, when we interface with Tokio (i.e. decoders & encoders), we still have
to use Bytes 0.4.  So, when needed, we convert our Bytes 0.5 to 0.4 there.

The tradeoff idea is that we deal with more bytes internally than we end up
sending to clients, so doing the Bytes conversion closer to the point of
sending data to clients means less copies.

We can also start removing those once we migrate to Tokio 0.2 (and newer
versions of Hyper for HTTP services).

Changes that were required:

- You can't extend new bytes (because that implicitly copies). You need to use
  BytesMut instead, which I did where that was necessary (I also added calls in
  the Filestore to do that efficiently).
- You can't create bytes from a `&'a [u8]`, unless `'a` is  `'static`. You need
  to use `copy_from_slice` instead.
- `slice_to` and `slice_from` have been replaced by a `slice()` function that
  takes ranges.

Reviewed By: StanislavGlebik

Differential Revision: D20121350

fbshipit-source-id: eb31af2051fd8c9d31c69b502e2f6f1ce2190cb1
2020-02-27 08:08:28 -08:00
Mateusz Kwapich
5825db21c6 add the git<->bonsai translation to mononoke_api crate
Reviewed By: markbt

Differential Revision: D19972871

fbshipit-source-id: 79c0c59f0bd1bd033bf2a8999dbe56b60a7ac085
2020-02-27 08:05:13 -08:00
Mark Thomas
a8f06f75c0 derived_data: add DeriveError for when derivation is disabled
Summary:
Currently if derivation of a particular derived data type is disabled, but a
client makes a request that requires that derived data type, we will fail with
an internal error.

This is not ideal, as internal errors should indicate something is wrong, but
in this case Mononoke is behaving correctly as configured.

Convert these errors to a new `DeriveError` type, and plumb this back up to
the SCS server.  The SCS server converts these to a new `RequestError`
variant: `NOT_AVAILABLE`.

Reviewed By: krallin

Differential Revision: D19943548

fbshipit-source-id: 964ad0aec3ab294e4bce789e6f38de224bed54fa
2020-02-19 09:28:09 -08:00
Thomas Orozco
b451a97878 mononoke/warm_bookmarks_cache: use the repo's configuration
Summary:
This update the warm_bookmarks_cache's constructor to use the passed in
blobrepo's derived data configuration (instead of whatever the caller is
passing in), since we now have that information.

Reviewed By: HarveyHunt

Differential Revision: D19949725

fbshipit-source-id: 575a1b9ff48f06003dbf9e0230b7cca723ad68f5
2020-02-18 07:40:25 -08:00
Mateusz Kwapich
69089c1c57 mononoke: Add hash::GitSha1 as a pure hash-only key for git Aliases
Summary: Add hash::GitSha1 as a pure hash-only key for git Aliases, so one no longer needs to know the size or type to load by Alias::GitSha1.

Reviewed By: krallin

Differential Revision: D19903578

fbshipit-source-id: bf919b197da2976bf31073ef04d12e0edfce0f9b
2020-02-18 05:02:52 -08:00
Mateusz Kwapich
f6e5098f7b mononoke: rename GitSha1 to RichGitSha1
Summary:
Rename GitSha1 to RichGitSha1 in preparation for introducing hash::GitSha1 as a pure sha1 without extra fields in next in stack.

Motivation for this is that currently one can't load content aliased by Alias::GitSha1 give just the hash, one has to know the type and size as well.

Once the next couple stack are done we will be able to load via just the git hash.

Reviewed By: krallin

Differential Revision: D19903280

fbshipit-source-id: ab2b8b841206a550c45b1e7f16ad83bfef0c2094
2020-02-18 05:02:51 -08:00
Thomas Orozco
16384599a8 mononoke (+ rust/shed/async_unit): update async_unit to expect async fn's
Summary:
This allows code that is being exercised under async_unit to call into code
that expects a Tokio 0.2 environment (e.g. 0.2 timers).

Unfortunately, this requires turning off LSAN for the async_unit tests, since
it looks like LSAN and Tokio 0.2 don't work very well together, resulting in
LSAN reporting leaked memory for some TLS structures that were initialized by
tokio-preview (regardless of whether the Runtime is being dropped):
https://fb.workplace.com/groups/rust.language/permalink/3249964938385432/

Considering async_unit is effectively only used in Mononoke, and Mononoke
already turns off LSAN in tests for precisely this reason ... it's probably
reasonable to do the same here.

The main body of changes here is also about updating the majority of our
changes to stop calling wait(), and use this new async unit everywhere. This is
effectively a pretty big batch conversion of all of our tests to use async fns
instead of the former approaches. I've also updated a substantial number of
utility functions to be async fns.

A few notable changes here:

- Some pushrebase tests were pretty flaky — the race they look for isn't
  deterministic. I added some actual waiting (using pushrebase hooks) to make
  it more deterministic.  This is kinda copy pasted from the globalrev hook
  (where I had introduced this first), but this will do for now.
- The multiplexblob tests don't work at all with new futures, because they call
  `poll()` all over the place. I've updated them to new futures, which required
  a bit of reworking.
- I took out a couple tests in async unit that were broken anyway.

Reviewed By: StanislavGlebik

Differential Revision: D19902539

fbshipit-source-id: 352b4a531ef5fa855114c1dd8bb4d70ed967dd55
2020-02-18 01:55:00 -08:00
Liubov Dmitrieva
5f4da0d6e0 mononoke: improve return error for invalid hashes in scs service
Summary: The error should be request error, not internal error

Reviewed By: krallin

Differential Revision: D19856120

fbshipit-source-id: fc66ac4eaeb27941de3b4ba769fe123b28685d14
2020-02-17 02:34:01 -08:00
Pavel Aslanov
f91d6f6b29 conditionally enable warmup chache based on derived data configuration
Summary:
- Conditionally enable warmup cache based on derived data configuration
- Blame is temporarily disabled for ovrsource, it is currently not used, and will be enabled again once new blame is generated for this repository

Reviewed By: StanislavGlebik

Differential Revision: D19902902

fbshipit-source-id: 7b782a32f7bfd76024686aea027bb15223079b9d
2020-02-14 12:07:25 -08:00
Mark Thomas
950ff5dff4 scs_server: ensure monitoring future never returns
Summary:
The SCS monitoring future should run forever and never return, even if there is
an error.

If an error does occur, we should just log it, and try again next time.

Also convert to new futures.

Reviewed By: krallin

Differential Revision: D19879621

fbshipit-source-id: 45b7097d1c5af61f3c390f6aa6ada5832062fbc9
2020-02-13 10:07:28 -08:00
Pavel Aslanov
c902acbc3f remove the need to pass mapping to ::derive method
Summary: remove the need to pass mapping to `::derive` method

Reviewed By: StanislavGlebik

Differential Revision: D19856560

fbshipit-source-id: 219af827ea7e077a4c3e678a85c51dc0e3822d79
2020-02-12 10:22:39 -08:00
Lukasz Piatkowski
542d1f93d3 Manual synchronization of fbcode/eden and facebookexperimental/eden
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.

Reviewed By: StanislavGlebik

Differential Revision: D19722832

fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
2020-02-11 11:42:43 +01:00
Stanislau Hlebik
8abe1af621 mononoke: add DerivedDataConfig
Summary:
See D19787960 for more details why we need to do it.
This diff just adds a struct in BlobRepo

Reviewed By: HarveyHunt

Differential Revision: D19788395

fbshipit-source-id: d609638432db3061f17aaa6272315f0c2efe9328
2020-02-10 01:44:09 -08:00
Stanislau Hlebik
ea8bed30a8 mononoke: add new safe sync_commit() method
Summary:
Follow up from D19718839 - let's add a function that will safely sync a commit
from one repo to another. Other function to sync a commit are prefixed with
unsafe

Reviewed By: krallin

Differential Revision: D19769762

fbshipit-source-id: 844da3e2c1cc39ef3cd86d282d275d860be55f44
2020-02-07 01:57:50 -08:00