Summary:
SQLite's `LIKE` operator is case insensitive by default. This doesn't match MySQL, and
also seems like a surprising default. Set the pragma on every connection to make it
case sensitive.
Reviewed By: farnz
Differential Revision: D22332419
fbshipit-source-id: 4f503eeaa874e110c03c27300467ddc02dc9b365
Summary: I would like to use these utilities when building segmented changelog.
Reviewed By: krallin
Differential Revision: D21876432
fbshipit-source-id: 9022627e224bfcb155b47d696371d24e538e6f39
Summary: To make it easier to navigate the codebase the oss-only code will be from now on stored in a separate module, similarly to how the fbcode-only code is stored.
Reviewed By: markbt
Differential Revision: D21429060
fbshipit-source-id: aa7e80961de2897dae31bd0ec83488c683633b7a
Summary:
We had accumulated lots of unused dependendencies, and had several test_deps in deps instead. Clean this all up to reduce build times and speed up autocargo processing.
Net removal is of around 500 unneeded dependency lines, which represented false dependencies; by removing them, we should get more parallelism in dev builds, and less overbuilding in CI.
Reviewed By: krallin, StanislavGlebik
Differential Revision: D20999762
fbshipit-source-id: 4db3772cbc3fb2af09a16601bc075ae8ed6f0c75
Summary: Now that everything is using `sql_construct`, we can remove the old `SqlConstructors` trait.
Reviewed By: StanislavGlebik
Differential Revision: D20734881
fbshipit-source-id: af46b41d17b40f6eb0839cdb9e85b00067360fe9
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.
In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }
[storage.blobstore.multiplexed]
queue_db = { remote = {
db_address = "queue_database",
sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```
This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.
**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
* `primary`, the database used for main metadata.
* `filenodes`, the database used for filenodes, which may be sharded or unsharded.
More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.
New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }
[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```
The `sql_construct` crate facilitates this by providing the following traits:
* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.
Sql database managers should implement:
* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
* **SqlConstructFromMetadataDatabaseConfig** if they are not shardable. By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
* **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable. They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.
Reviewed By: StanislavGlebik
Differential Revision: D20734883
fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
Summary:
Refactor `sql_ext::SqlConstructors` and its related traits into a separate
crate. The new `SqlConstruct` trait is joined by `SqlShardedConstruct` which
allows construction of sharded databases.
The new crate will support a new configuration model where the distinction
between the database configuration for different repository metadata types
will be made clear.
Reviewed By: StanislavGlebik
Differential Revision: D20734882
fbshipit-source-id: b44cf9d1efd014c29df88e2ad933025e440119dc
Summary: In the process the blobstore/factory/lib.rs was split into submodules - this way it was easier to untangle the dependencies and refactor it, so I've left the split in this diff.
Reviewed By: markbt
Differential Revision: D20302068
fbshipit-source-id: caa3a2b5487c30198c62f7e4f4e9cb7c488dc8de
Summary: This diff makes it possible to relay on the thrift structures from configerator in OSS.
Reviewed By: ahornby
Differential Revision: D20279457
fbshipit-source-id: 39df1c7a6f78b10a0a5bdeeebe476249ab674cc8
Summary:
This shifts the responsibility of mocking all facebook-specific code
to mononoke's sql_ext crate. If OSS code calls into any of that code it will
most likely result in a panic.
Reviewed By: ahornby
Differential Revision: D20247580
fbshipit-source-id: 43f158d91aa32adaa5df6e3786243fb89c9ce961
Summary: separate out the Facebook-specific pieces of the sql_ext crate
Reviewed By: ahornby
Differential Revision: D20218219
fbshipit-source-id: e933c7402b31fcd5c4af78d5e70adafd67e91ecd
Summary:
This is the second (and last) step on removing RocksDB as a blobstore.
Check the task for more description.
Context for OSS:
> The issue with rocksblob (and to some extent sqlite) is that unless we
> introduce a blobstore tier/thift api (which is something I'm hoping to avoid
> for xdb blobstore) we'd have to combine all the mononoke function like hg,
> scs, LFS etc into one binary for it to have access to rocksdb, which would be
> quite a big difference to how we deploy internally
(Note: this ignores all push blocking failures!)
Reviewed By: farnz
Differential Revision: D20001261
fbshipit-source-id: c4b2b2a393b918d17680ad483aa1d77356f1d07c
Summary:
MyRouter needs to be told which shards to watch. Since I'm adding a new shard,
it'll be easier for everyone to know that they need to update their MyRouter
configuration if we start logging the shard name we're trying to hit.
Reviewed By: ikostia
Differential Revision: D20001704
fbshipit-source-id: 8a9ff3521bc7e3c9b7ed39c6ae33d0ddc1d467b7
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
