Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Regexes stored in config need to be comparable so that config is comparable.
Normally regexes are not comparable, so wrap them in a newtype wrapper that
implements comparison, rather than manually implemented PartialEq and Eq for
anything that contains them.
Reviewed By: mitrandir77
Differential Revision: D21886533
fbshipit-source-id: 1fb39c0874daed383624eeda61c903a4731b9cb8
Summary:
Right now we're only logging hooks that outright fail, which isn't great. Let's
log rejections as well.
Reviewed By: johansglock
Differential Revision: D21522804
fbshipit-source-id: 6bfc6b12394099b04faa9d23f164b436935f9fb3
Summary:
This updates repo_client to log when hooks finished, and how many were rejecte,
if any. This required a bit of refactoring to avoid iterating twice over
whether hooks are rejected or not (and instead just filter-maps outcomes to a
rejection), but it's probably for the better since it removes a bit of
un-necessary cloning (notably of the hook name).
Reviewed By: farnz
Differential Revision: D21379690
fbshipit-source-id: 53c8368d3871620ec61db76dc35b47dd17276ac4
Summary:
Make all the things that only Lua hooks needed (hook type etc) optional.
With this done, configs can be cleaned up to not contain redundant data.
Reviewed By: ikostia
Differential Revision: D21349614
fbshipit-source-id: 1c72c2082b8c002e3feb41d1d720a41d21afaae5
Summary: The new MembershipChecker and PermissionChecker traits will generalize access to various permission/acl systems (like LDAP) and leave the implementation details hidden behind an object trait.
Reviewed By: StanislavGlebik
Differential Revision: D21067811
fbshipit-source-id: 3bccd931f8acdb6c1e0cff4cb71917c9711b590b
Summary:
In next diffs with permission_checker the panic is changed to anyhow::Error.
The previous behavior of this code was that when AclChecker updated failed
after 10s this fact was ignored and the hooks were simply not using ACLs. This
diff fixes it so that the server exits when AclChecheker update is timing out.
Reviewed By: johansglock
Differential Revision: D21155944
fbshipit-source-id: ab4a5071acbe6a1282a7bc5fdbf301b4bd53a347
Summary:
Unfortunately, `BonsaiChangeset::get_changeset_id()` is a fairly expensive
operation, since it'll clone, serialize, and hash the changeset. In hooks in
particular, since we run this once per hook execution (and therefore once per
file), that can be come a problem.
Indeed, on a commit with 1K file changes, hooks run for ~30 seconds
(P129058164). According to perf, the overwhelming majority of that time is
spent in computing hashes of bonsai changesets. For a commit with 10K changes,
it spends time there as well, it took 3.5 hours.
This diff updates hooks to compute the changeset id just once, which brings our
time down to O(N) (where N = file changes).
Reviewed By: StanislavGlebik
Differential Revision: D21039811
fbshipit-source-id: 73f9939ffc7d095e717bdb5efc46dbf4ad312c65
Summary:
This parses out the Hgsql name out of the repo config. While in there, I also
noticed that our tests force us to have a default impl right now (there are
otherwise waaaay to many fields to specify), but at the same time we don't use
it everywhere. So, in an effort to clean up, I updated hooks to use a default.
I added a newtype wrapper for the hgsql name, since this will let me update the
globalrev syncer and SQL repo lock implementation to require a HgsqlName
instead of a string and have the compiler prove that all callsites are doing
so.
Reviewed By: farnz
Differential Revision: D20942177
fbshipit-source-id: bfbba6ba17cf3e3cad0be0f8406e41e5a6e6c3d4
Summary: Not in use any more - all hooks are now Bonsai form - so remove it.
Reviewed By: krallin
Differential Revision: D20891164
fbshipit-source-id: b92f169a0ec3a4832f8e9ec8dc9696ce81f7edb3
Summary: These hooks now run on modified files, not just added files, after porting to Bonsai form.
Reviewed By: krallin
Differential Revision: D20891166
fbshipit-source-id: 93a142f91c0bea7f5fe5e541530c644d215dce3a
Summary: These hooks behave the same way in Mercurial and Bonsai form. Port them over to operating on Bonsai form
Reviewed By: krallin
Differential Revision: D20891165
fbshipit-source-id: cbcdf217398714642d2f2d6669376defe8b944d7
Summary: To use Bonsai-based hooks, we ned to be able to load them. Make it possible.
Reviewed By: krallin
Differential Revision: D20879135
fbshipit-source-id: 9b44d7ca83257c8fc30809b4b65ec27a8e9a8209
Summary: We want all hooks to run against the Bonsai form, not a Mercurial form. Create a second form of hooks (currently not used) which acts on Bonsai hooks. Later diffs in the stack will move us over to Bonsai only, and remove support for Mercurial changeset derived hooks
Reviewed By: krallin
Differential Revision: D20604846
fbshipit-source-id: 61eece8bc4ec5dcc262059c19a434d5966a8d550
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.
In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }
[storage.blobstore.multiplexed]
queue_db = { remote = {
db_address = "queue_database",
sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```
This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.
**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
* `primary`, the database used for main metadata.
* `filenodes`, the database used for filenodes, which may be sharded or unsharded.
More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.
New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }
[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```
The `sql_construct` crate facilitates this by providing the following traits:
* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.
Sql database managers should implement:
* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
* **SqlConstructFromMetadataDatabaseConfig** if they are not shardable. By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
* **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable. They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.
Reviewed By: StanislavGlebik
Differential Revision: D20734883
fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
Summary: We had hooks logic scattered around the place - move it all into the hooks crate, so that it's easier to refactor to use Bonsai changesets instead of hg.
Reviewed By: StanislavGlebik
Differential Revision: D20198725
fbshipit-source-id: fb8bdc2cdbd1714c7181a5a0562c1dacce9fcc7d
Summary: Migrate hooks to new futures and thus modern tokio. In the process, replace Lua hooks with Rust hooks, and add fixes for the few cases where Lua was too restrictive about what could be done.
Reviewed By: StanislavGlebik
Differential Revision: D20165425
fbshipit-source-id: 7bdc6820144f2fdaed653a34ff7c998913007ca2
Summary:
This is convenient because it makes it possible to tell what is going on within
Mononoke from the outside (e.g. introspect perf counters).
I'll land this after D20387115.
Reviewed By: farnz
Differential Revision: D20387125
fbshipit-source-id: ee070c4d658a0ec8f232152fe8b34bd0b56e6888
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
Summary:
Most binaries don't need hooks. Let's not require them. This might not be very
long lived since Simon is working on removing lua hooks, but this was a trivial
fix.
Reviewed By: johansglock
Differential Revision: D20140026
fbshipit-source-id: cc74b37459f63c5dd550c5779b72aa1d6531202c
Summary:
The Bytes 0.5 update left us in a somewhat undesirable position where every
access to our blobstore incurs an extra copy whenever we fetch data out of our
cache (by turning it from Bytes 0.5 into Bytes 0.4) — we also have quite a few
place where we convert in one direction then immediately into the other.
Internally, we can start using Bytes 0.5 now. For example, this is useful when
pulling data out of our blobstore and deserializing as Thrift (or conversely,
when serializing and putting it into our blobstore).
However, when we interface with Tokio (i.e. decoders & encoders), we still have
to use Bytes 0.4. So, when needed, we convert our Bytes 0.5 to 0.4 there.
The tradeoff idea is that we deal with more bytes internally than we end up
sending to clients, so doing the Bytes conversion closer to the point of
sending data to clients means less copies.
We can also start removing those once we migrate to Tokio 0.2 (and newer
versions of Hyper for HTTP services).
Changes that were required:
- You can't extend new bytes (because that implicitly copies). You need to use
BytesMut instead, which I did where that was necessary (I also added calls in
the Filestore to do that efficiently).
- You can't create bytes from a `&'a [u8]`, unless `'a` is `'static`. You need
to use `copy_from_slice` instead.
- `slice_to` and `slice_from` have been replaced by a `slice()` function that
takes ranges.
Reviewed By: StanislavGlebik
Differential Revision: D20121350
fbshipit-source-id: eb31af2051fd8c9d31c69b502e2f6f1ce2190cb1
Summary:
This hook was implemented to prevent incorrect users from moving a
bookmark. However, it doesn't work and the functionality is now implemented by
`is_allowed_user` in the pushrebase pipeline.
Remove the unused hook.
Reviewed By: johansglock
Differential Revision: D20030479
fbshipit-source-id: bcbc9508eebe77cffbc7936382ba4d345b76f74f
Summary:
We're working towards sharding Bonsais. Let's make them easier to cache by also
not allowing arbitrarily large commit messages.
Reviewed By: StanislavGlebik
Differential Revision: D20002994
fbshipit-source-id: b2319ac9d5709e968121d4299396e03a90df4a06
Summary:
This adds a file hook to limit the file length we are willing to allow in
commits. This is necessary for now since Mercurial does have a limit on its
end, and we shouldn't allow commits that we cannot sync to Mercurial.
Reviewed By: HarveyHunt
Differential Revision: D19969689
fbshipit-source-id: 1da8a62d54e98b047d381a9d073ac148c9af84b0
Summary:
This allows code that is being exercised under async_unit to call into code
that expects a Tokio 0.2 environment (e.g. 0.2 timers).
Unfortunately, this requires turning off LSAN for the async_unit tests, since
it looks like LSAN and Tokio 0.2 don't work very well together, resulting in
LSAN reporting leaked memory for some TLS structures that were initialized by
tokio-preview (regardless of whether the Runtime is being dropped):
https://fb.workplace.com/groups/rust.language/permalink/3249964938385432/
Considering async_unit is effectively only used in Mononoke, and Mononoke
already turns off LSAN in tests for precisely this reason ... it's probably
reasonable to do the same here.
The main body of changes here is also about updating the majority of our
changes to stop calling wait(), and use this new async unit everywhere. This is
effectively a pretty big batch conversion of all of our tests to use async fns
instead of the former approaches. I've also updated a substantial number of
utility functions to be async fns.
A few notable changes here:
- Some pushrebase tests were pretty flaky — the race they look for isn't
deterministic. I added some actual waiting (using pushrebase hooks) to make
it more deterministic. This is kinda copy pasted from the globalrev hook
(where I had introduced this first), but this will do for now.
- The multiplexblob tests don't work at all with new futures, because they call
`poll()` all over the place. I've updated them to new futures, which required
a bit of reworking.
- I took out a couple tests in async unit that were broken anyway.
Reviewed By: StanislavGlebik
Differential Revision: D19902539
fbshipit-source-id: 352b4a531ef5fa855114c1dd8bb4d70ed967dd55
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary:
See D19787960 for more details why we need to do it.
This diff just adds a struct in BlobRepo
Reviewed By: HarveyHunt
Differential Revision: D19788395
fbshipit-source-id: d609638432db3061f17aaa6272315f0c2efe9328
