Summary:
This change has two goals:
- Put storage configuration that's common to multiple repos in a common place,
rather than replicating it in each server.toml
- Allow tools that only operate on the blobstore level - like blobstore healing
- to be configured directly in terms of the blobstore, rather than indirectly
by using a representative repo config.
This change makes several changes to repo configuration:
1. There's a separate common/storage.toml which defines named storage
configurations (ie, a combination of a blobstore and metadata DB)
2. server.toml files can also define local storage configurations (mostly
useful for testing)
3. server.toml files now reference which storage they're using with
`storage_config = "name"`.
4. Configuration of multiplex blobstores is now explicit. Previously if a
server.toml defined multiple blobstores, it was assumed that it was a
multiplex. Now storage configuration only accepts a single blobstore config,
but that config can be explicitly a multiplexed blobstore, which has the
sub-blobstores defined within it, in the `components` field. (This is
recursive, so it could be nested, but I'm not sure if this has much value in
practice.)
5. Makes configuration parsing more strict - unknown fields will be treated as
an error rather than ignored. This helps flag problems in refactoring/updating
configs.
I've updated all the configs to the new format, both production and in
integration tests. Please review to make sure I haven't broken anything.
Reviewed By: StanislavGlebik
Differential Revision: D15065423
fbshipit-source-id: b7ce58e46e91877f4e15518c014496fb826fe03c
Summary:
This migrates the internal structures representing the repo and storage config,
while retaining the existing config file format.
The `RepoType` type has been replaced by `BlobConfig`, an enum containing all
the config information for all the supported blobstores. In addition there's
the `StorageConfig` type which includes `BlobConfig`, and also
`MetadataDBConfig` for the local or remote SQL database for metadata.
Reviewed By: StanislavGlebik
Differential Revision: D15065421
fbshipit-source-id: 47636074fceb6a7e35524f667376a5bb05bd8612
Summary:
We don't need Option<bool> or Option<Vec<T>> - in the former case, the
bool is always treated as having a default value if not present, and in the
latter, None is equivalent to Some(vec![]), so just use an empty vector for
absense.
Reviewed By: lukaspiatkowski
Differential Revision: D15051895
fbshipit-source-id: 0ac6f2e6b13357bf6e30dbfa25c7fdebd208e505
Summary:
This updates our configuration to allow using a different tier name for sharded filenodes.
One thing I'd like to call out is that we currently use the DB tier name in the keys generated by `CachingFilenodes`. Updating the tier name will therefore result in us dropping all our caches. Is this acceptable? If not, should we just continue using the old tier name.
Reviewed By: jsgf, StanislavGlebik
Differential Revision: D15243112
fbshipit-source-id: 3bfdcefcc823768f2964b4733e570e9cef57cebc
Summary: Use the existing library function to read a file into a `Vec<u8>`.
Reviewed By: aslpavel
Differential Revision: D15051894
fbshipit-source-id: 853b31450556c0a2e74a09fa06e7814ac68b1052
Summary:
The config will be used to whitelist connections with certain identities and
blacklist everything else.
Differential Revision: D15150921
fbshipit-source-id: e4090072ea6ba9714575fb8104d9f45e92c6fefb
Summary:
Disallow unknown fields. They're generally the result of a mis-editing
a file and putting the config in the wrong place, or some incomplete refactor.
Reviewed By: StanislavGlebik
Differential Revision: D15168963
fbshipit-source-id: a9c9658378cda4866e44daf6e2c6bfbdfcdb9f84
Summary: Now it is possible to configure and enable/disable bookmark cache from configs
Reviewed By: StanislavGlebik
Differential Revision: D14952840
fbshipit-source-id: 3080f7ca4639da00d413a949547705ad480772f7
Summary: Added obsmarkers to pushrebase output. This allows the client to hide commits that were rebased server-side, and check out the rebased commit.
Reviewed By: StanislavGlebik
Differential Revision: D14932842
fbshipit-source-id: f215791e86e32e6420e8bd6cd2bc25c251a7dba0
Summary: We want to have casefolding configurable for admin tier
Reviewed By: StanislavGlebik
Differential Revision: D14775535
fbshipit-source-id: 103a13b98e9c06053cf0636009c6f1e96c80f741
Summary: We want to forbid pushrebases when root node is a p2 of its parents for now, since mercurial swaps them after pushrebase which causes inconsistency
Reviewed By: StanislavGlebik
Differential Revision: D14642177
fbshipit-source-id: f8f6e9565c53958e8cff5df6f4d006ddfe5a69c0
Summary:
No longer require the write_lock_db_address field in the config
in order to make configs simpler and also prevent creating a mysql connection
on startup for repos that don't need it.
Reviewed By: aslpavel
Differential Revision: D14561984
fbshipit-source-id: 0fd8765b425e008b44d90de25c34a8d487dbfb3e
Summary:
As part of the mononoke lock testing, we realised that it would
be helpful to see why a repo is locked. Add the ability to express this
to the RepoReadOnly enum entry.
Reviewed By: aslpavel
Differential Revision: D14544801
fbshipit-source-id: ecae1460f8e6f0a8a31f4e6de3bd99c87fba8944
Summary: In Mononoke we want to be able to block merge commits on a repo per repo basis.
Reviewed By: aslpavel
Differential Revision: D14455502
fbshipit-source-id: 400e85834c20df811674405bc0c391860cf677dd
Summary:
Allow using a database entry to determine if a repository
is in read-only mode.
If a repo is marked read/write in the config then we will communicate with the db.
Reviewed By: ikostia
Differential Revision: D14279170
fbshipit-source-id: 57abd597f939e57f274079011d2fdcc7e7037854
Summary:
This is a hook in mercurial, in Mononoke it will be part of the implementation. By default all non fastforward pushes are blocked, except when using the NON_FAST_FORWARD pushvar (--non-forward-move is also needed to circumvent client side restrictions). Additionally certain bookmarks (e.g. master) shouldn't be able to be moved in a non fastforward manner at all. This can be done by setting block_non_fast_forward field in config.
Pushrebase can only move the bookmark that is actually being pushrebased so we do not need to check whether it is a fastforward move (it always is)
Reviewed By: StanislavGlebik
Differential Revision: D14405696
fbshipit-source-id: 782b49c26a753918418e02c06dcfab76e3394dc1
Summary: There is no need to have an Option<Vec<XYZ>> as None can simply be represented by an empty vector. This makes these fields easier to use.
Reviewed By: StanislavGlebik
Differential Revision: D14405687
fbshipit-source-id: e4c5ba12a1e3c6a18130026af6814d54952da4d2
Summary: See D14279065, this diff is simply to clean up the deprecated code
Reviewed By: StanislavGlebik
Differential Revision: D14279210
fbshipit-source-id: 10801fb04ad533a80bb7a2f9dcdf3ee5906aa68d
Summary:
Currently we are logging new commits from BlobRepo. This will lead to issues once CommitCloud starts using Mononoke as we cannot differentiate between phases at that level. The solution is to log commits when they are pushrebased as this guarantees that they are public commits.
Note: This only introduces the new logic, cleaning up the existing implementation is part of D14279210
Reviewed By: StanislavGlebik
Differential Revision: D14279065
fbshipit-source-id: d714fae7164a8af815fc7716379ff0b7eb4826fb
Summary: New commits should be logged to scribe, these will be used to trigger the update for the hg clone streamfile.
Reviewed By: lukaspiatkowski
Differential Revision: D14022599
fbshipit-source-id: a8a68f12a8dc1e65663d1ccf1a5eafa54ca2daf0
Summary: This is to tell Mononoke whether to preserve bundle2 raw content or not.
Reviewed By: StanislavGlebik
Differential Revision: D14038588
fbshipit-source-id: f1781ed8b4aca7e37925e7f40aeb506af60d071a
Summary:
Currently if a crate depends even on a single type from metaconfig then in
order to compile this trait buck first compiles metaconfig crate with all the
logic of parsing the configs.
This diff split metaconfig into two crates. The first one just holds the types for
"external consumption" by other crates. The second holds the parsing logic.
That makes builds faster
Reviewed By: jsgf, lukaspiatkowski
Differential Revision: D13877592
fbshipit-source-id: f353fb2d1737845bf1fa0de515ff8ef131020063
Summary:
The main reason to do it is to remove dependency from cmdlib to repo_client.
repo_client depends on a lot of other crates like bundle2-resolver, hooks etc.
And it means that in order to compile mononoke_admin we need to compile these
crates too. By moving open_blobrepo into blobrepo crate we are removing
unnecessary dependencies.
Also let's remove unused blobrepo type
Reviewed By: aslpavel
Differential Revision: D13848878
fbshipit-source-id: cd3d04354649cdb5b2947f08762051318725c781
Summary: In this diff the configs are parsed from toml and passed around to hook's execution context. The actual usage of configs will be introduced in separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D13862837
fbshipit-source-id: 60ac10aa9c25d224e703e1e55bef13dc481ba07e
Summary:
This version still misses:
- proper production-ready logging
- smarter handling of case where the queue entries related to each other do not fit in the limit or `older_than` limit, so the healer will heal much more entries without realizing it shouldn't do so.
Reviewed By: aslpavel
Differential Revision: D13528686
fbshipit-source-id: 0245becea7e4f0ac69383a7885ff3746d81c4add
Summary: the from_value_opt already knows how to properly parse an integer from sql response, no need to reinvent the wheel and introduce bugs
Reviewed By: aslpavel
Differential Revision: D13651328
fbshipit-source-id: 55af810c99b93bd2f9c67c721a9d0be6034ee466
Summary: MyRouter and cache support will come in next diffs
Reviewed By: StanislavGlebik
Differential Revision: D13465146
fbshipit-source-id: 0ede0e875d4a49794ff71173bd0d1563abb3ff08
Summary: Format files effected by next commit in a stack
Reviewed By: StanislavGlebik
Differential Revision: D13650639
fbshipit-source-id: d4e37acd2bcd29b291968a529543c202f6944e1a
Summary: These config options weren't used at all. Let's delete them from the configs
Reviewed By: aslpavel
Differential Revision: D13597545
fbshipit-source-id: 5e99b38e2e9fc4ba6d836322ceba38a8b65ad2f4
Summary:
Currently some tests are failing intermittently due to a timeout, this timeout
is caused by the integration tests making an external call to AclChecker which
is intermittently very slow.
For our integration tests we disable this call since it's not part of the test
suite to stop our tests from being flaky.
Reviewed By: StanislavGlebik
Differential Revision: D13580144
fbshipit-source-id: 0c26bb14dd222b888ca2638319071f4d99eab6df
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Previously metaconfig depended on BlobRepo, and so ManifoldArgs had to be
definided in BlobRepo. That was an weird dependency, but a necessary one
because of Mononoke config repo. In the previous diffs we got rid of the
Mononoke config repo, so now we can reverse the dependencies.
Reviewed By: lukaspiatkowski
Differential Revision: D13180160
fbshipit-source-id: efe713ce3b160c98d56fc13559c57a920146841f
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
