Summary: It makes pushes faster, especially on non-master regions.
Reviewed By: quark-zju
Differential Revision: D15279259
fbshipit-source-id: c184b68cc8b7509938849cd86bb15ef5d5f33bdd
Summary:
In the case of mononoke's admin tool it's annoying for users to be required to run myrouter in the background and provide myrouter port to every command.
Thanks to this change it is no longer necessary to run admin commands through myrouter - the tool will simply use a direct connection to XDB using the sql crate.
It is important to note that the raw XDB connection via sql crate doesn't have connection pooling and doesn't handle XDB failover so it is crucial that it is never used for long-lived or request heavy use cases like running mononoke server or blobimport
Reviewed By: jsgf
Differential Revision: D15174538
fbshipit-source-id: 299d3d7941ae6aec31961149f926c2a4965ed970
Summary:
Before this diff we always wrote to master even if paths already exist. This
diff changes it - first check in replica if paths are there, and only then
write to master
Reviewed By: farnz
Differential Revision: D15063307
fbshipit-source-id: 802839f340c9953c7f2812e77d81bc66917c5e77
Summary:
Add a LABEL constant to the SqlConstructors trait to make it easier to identify
which table is being used, for stats and logging.
Reviewed By: HarveyHunt
Differential Revision: D13457488
fbshipit-source-id: a061a9582bc1783604f249d5b7dcede4b1e1d3c5
Summary:
These are **not** the schemas that we use in production.
So at the moment they are not used for anything and they just create confusion.
Reviewed By: aslpavel
Differential Revision: D13986001
fbshipit-source-id: 7aae0a5da474f579c9cdf1bbf5dfe183835cae2d
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
Previously to get copy/move source we had to join `paths` and `fixedcopyinfo`
table. That worked fine when we had just one shard. However now we have many
shards, and join no longer works. The reason is that move source path is in a
different shard compared to move destination path, and join returns no data.
Consider this situation. shardA contains all the data for pathA, shardB
contains all the data for pathB. That means that sharded `paths` table will
have pathA in shardA and pathB in shardB. Then if file pathA was copied form
pathB, then `fixedcopyinfo` table in shardA contains a path_hash of pathB.
However joining shardA's `fixedcopyinfo` with shardA's `paths` to convert
path_hash to path fails because pathB is in shardB.
The only possible fix is to split fetching path_hash from `fixedcopyinfo` and
converting path_hash to path.
I don't think we'll be able to keep the logic with join that we have at the
moment. It would require us to have all paths on all shards which is
unfeasible because it'll make writes much slower.
Reviewed By: aslpavel
Differential Revision: D13690141
fbshipit-source-id: 16b5cae6f23c162bb502b65c208f3ca9e443fb04
Summary:
Going to change these files in the next diff. To make next diff smaller
splitting format changes to this diff.
Reviewed By: aslpavel
Differential Revision: D13690143
fbshipit-source-id: 124232b832d8c67ee7fe931ef174230cb09ff564
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
Removed:
cmd-line cmd tool for filenodes and bookmarks. These should be a part of
mononoke_admin script
Outdates docs folder
Commitsim crate, because it's replaced by real pushrebase
unused hooks_old crate
storage crate which wasn't used
Reviewed By: aslpavel
Differential Revision: D13301035
fbshipit-source-id: 3ae398752218915dc4eb85c11be84e48168677cc
Summary:
Now that Rust macros can be `use`d like normal symbols, `stats` can
simply import the `lazy_static!` macro without requiring its users to do it.
Reviewed By: Imxset21
Differential Revision: D13281897
fbshipit-source-id: a6780fbace07dd784308e642d4a384322a17c367
Summary:
Default ServiceType is ServiceType.Any, so it might go to master in a master
region. This diff changes it.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D13021674
fbshipit-source-id: 928cf59b095549f3048411241116c097e1193c7d
Summary:
Sharding filenodes by path should stop us knocking over databases -
make it configurable.
Reviewed By: StanislavGlebik
Differential Revision: D12894523
fbshipit-source-id: e27452f9b436842e1cb5e9e0968c1822f422b4c9
Summary:
We can already flatten a single XDB server with filenodes traffic, and
do if we start up a server instance without a warm memcache. This is only going
to get worse in the future.
Start the process of sharding across multiple servers. For now, we can only
deal with shard size == 1, but this code should be ready to handle shard sizes
greater than 1
Reviewed By: StanislavGlebik
Differential Revision: D12888927
fbshipit-source-id: 8e01694357c390837487fdb3710685fd09feaec0
Summary: This will enable doing queries like DELETE, UPDATE or REPLACE without listing all possibilites in the macros
Reviewed By: StanislavGlebik
Differential Revision: D10499501
fbshipit-source-id: 3e2ba433722bd34ffb5960840c509dc27cc9eb5d
Summary:
We have a problem with service upgrades/restarts because many servers start
sending too many requests to mysql db.
Let's add a memcache that will prevent that.
Reviewed By: jsgf
Differential Revision: D10488624
fbshipit-source-id: 4575d359bc269e29fe72b47d7f47cda22bf4acd7
Summary: Reverting the myrouter based filenodes for now as they cause some problems
Reviewed By: jsgf
Differential Revision: D10405364
fbshipit-source-id: 07da917455ae5af9ef81a24d99f516171101c8a7
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary:
In the next diff I'm going to add a separate functions to look in the cache and
insert into the cache, so rename it to avoid confusion
Reviewed By: Imxset21
Differential Revision: D9869648
fbshipit-source-id: f2bd806b14d78660518d841d90a903970028eb37
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
`asynchronize` currently does not warn the event loop that it's
running blocking code, so we can end up starving the thread pool of threads.
We can't use `blocking` directly, because it won't spawn a synchronous task
onto a fresh Tokio task, so your "parallel" futures end up running in series.
Instead, use it inside `asynchronize` so that we can pick up extra threads in
the thread pool as and when we need them due to heavy load.
While in here, fix up `asynchronize` to only work on synchronous tasks and
push the boxing out one layer. Filenodes needs a specific change that's
worth extra eyes.
Reviewed By: jsgf
Differential Revision: D9631141
fbshipit-source-id: 06f79c4cb697288d3fadc96448a9173e38df425f
Summary:
It makes startup unbearably slow, and doesn't add any benefits at all. Revert
it
Reviewed By: purplefox
Differential Revision: D9358741
fbshipit-source-id: 26469941304f737c856a6ffca5e577848ad30955
Summary:
Should be functionally equivalent and semantically more appropriate
This also makes a couple of small API changes:
- The inner function is expected to just return a Result - IntoFuture is
overkill if its supposed to be synchronous in the first place
- `asynchronize` itself returns `impl Future` rather than being intrinsically
boxed.
- Restructure dieselfilenodes::add_filenodes to only asynchronize the insert
itself.
Reviewed By: farnz
Differential Revision: D8959535
fbshipit-source-id: fef9164e3be0069bd0d93573642cd57bb5babb73
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary: The blake2_path_hash is used only in MySql right now, but I want to use it in caching as well, so moving it to a more accessible place.
Reviewed By: jsgf
Differential Revision: D9028799
fbshipit-source-id: b055550fbfca5c6c419702490504f08812592470
Summary:
This information is useful to decide if get_all_filenodes is small enough to fit into memcache.
Later it will be used to track how many elements are fitting into memcache nicely.
Reviewed By: jsgf
Differential Revision: D9028395
fbshipit-source-id: ae71486f9e9d60d7b54ce28b8c591c09b8925947
Summary: a bit of cleanup for the upcoming diffs
Reviewed By: farnz
Differential Revision: D9014639
fbshipit-source-id: d03e4969426eced70fd19b91a2c15b607bf52b39
Summary: Those structures will be used in next diffs to store the FilenodeInfo inside memcache for caching purposes
Reviewed By: farnz
Differential Revision: D9014213
fbshipit-source-id: 4952a90415d4b8ab903387fd5cdfaf08d9870c07
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Unify all uses of Sqlite and of Mysql
This superceded D8712926
Reviewed By: farnz
Differential Revision: D8732579
fbshipit-source-id: a02cd04055a915e5f97b540d6d98e2ff2d707875
Summary: we want most of the read traffic to be handled by replicas
Reviewed By: lukaspiatkowski
Differential Revision: D8548027
fbshipit-source-id: 297b070b5b33ec114026ecf3f1079fdb26f54b19
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary: this will make it easier to identify stats exported from mononoke vs external libraries
Reviewed By: StanislavGlebik
Differential Revision: D8331418
fbshipit-source-id: c151e76aa386fb13759fced7cc07b03ac67fe051
