Summary:
We can already flatten a single XDB server with filenodes traffic, and
do if we start up a server instance without a warm memcache. This is only going
to get worse in the future.
Start the process of sharding across multiple servers. For now, we can only
deal with shard size == 1, but this code should be ready to handle shard sizes
greater than 1
Reviewed By: StanislavGlebik
Differential Revision: D12888927
fbshipit-source-id: 8e01694357c390837487fdb3710685fd09feaec0
Summary: This will enable doing queries like DELETE, UPDATE or REPLACE without listing all possibilites in the macros
Reviewed By: StanislavGlebik
Differential Revision: D10499501
fbshipit-source-id: 3e2ba433722bd34ffb5960840c509dc27cc9eb5d
Summary:
We have a problem with service upgrades/restarts because many servers start
sending too many requests to mysql db.
Let's add a memcache that will prevent that.
Reviewed By: jsgf
Differential Revision: D10488624
fbshipit-source-id: 4575d359bc269e29fe72b47d7f47cda22bf4acd7
Summary: Reverting the myrouter based filenodes for now as they cause some problems
Reviewed By: jsgf
Differential Revision: D10405364
fbshipit-source-id: 07da917455ae5af9ef81a24d99f516171101c8a7
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary:
In the next diff I'm going to add a separate functions to look in the cache and
insert into the cache, so rename it to avoid confusion
Reviewed By: Imxset21
Differential Revision: D9869648
fbshipit-source-id: f2bd806b14d78660518d841d90a903970028eb37
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
`asynchronize` currently does not warn the event loop that it's
running blocking code, so we can end up starving the thread pool of threads.
We can't use `blocking` directly, because it won't spawn a synchronous task
onto a fresh Tokio task, so your "parallel" futures end up running in series.
Instead, use it inside `asynchronize` so that we can pick up extra threads in
the thread pool as and when we need them due to heavy load.
While in here, fix up `asynchronize` to only work on synchronous tasks and
push the boxing out one layer. Filenodes needs a specific change that's
worth extra eyes.
Reviewed By: jsgf
Differential Revision: D9631141
fbshipit-source-id: 06f79c4cb697288d3fadc96448a9173e38df425f
Summary:
It makes startup unbearably slow, and doesn't add any benefits at all. Revert
it
Reviewed By: purplefox
Differential Revision: D9358741
fbshipit-source-id: 26469941304f737c856a6ffca5e577848ad30955
Summary:
Should be functionally equivalent and semantically more appropriate
This also makes a couple of small API changes:
- The inner function is expected to just return a Result - IntoFuture is
overkill if its supposed to be synchronous in the first place
- `asynchronize` itself returns `impl Future` rather than being intrinsically
boxed.
- Restructure dieselfilenodes::add_filenodes to only asynchronize the insert
itself.
Reviewed By: farnz
Differential Revision: D8959535
fbshipit-source-id: fef9164e3be0069bd0d93573642cd57bb5babb73
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary: Because get_all_filenodes can go above 1MB (but still less than 3MB) I added a chunking logic to speed up fetching of even the long histories
Reviewed By: farnz
Differential Revision: D9029774
fbshipit-source-id: c2d46a4b49bfebd3c778a0b9c5cf226857f34768
Summary: The blake2_path_hash is used only in MySql right now, but I want to use it in caching as well, so moving it to a more accessible place.
Reviewed By: jsgf
Differential Revision: D9028799
fbshipit-source-id: b055550fbfca5c6c419702490504f08812592470
Summary:
This information is useful to decide if get_all_filenodes is small enough to fit into memcache.
Later it will be used to track how many elements are fitting into memcache nicely.
Reviewed By: jsgf
Differential Revision: D9028395
fbshipit-source-id: ae71486f9e9d60d7b54ce28b8c591c09b8925947
Summary: a bit of cleanup for the upcoming diffs
Reviewed By: farnz
Differential Revision: D9014639
fbshipit-source-id: d03e4969426eced70fd19b91a2c15b607bf52b39
Summary: Those structures will be used in next diffs to store the FilenodeInfo inside memcache for caching purposes
Reviewed By: farnz
Differential Revision: D9014213
fbshipit-source-id: 4952a90415d4b8ab903387fd5cdfaf08d9870c07
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Unify all uses of Sqlite and of Mysql
This superceded D8712926
Reviewed By: farnz
Differential Revision: D8732579
fbshipit-source-id: a02cd04055a915e5f97b540d6d98e2ff2d707875
Summary: we want most of the read traffic to be handled by replicas
Reviewed By: lukaspiatkowski
Differential Revision: D8548027
fbshipit-source-id: 297b070b5b33ec114026ecf3f1079fdb26f54b19
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary: this will make it easier to identify stats exported from mononoke vs external libraries
Reviewed By: StanislavGlebik
Differential Revision: D8331418
fbshipit-source-id: c151e76aa386fb13759fced7cc07b03ac67fe051
Summary: Going to take a while to get to everything, but here's a start.
Reviewed By: Imxset21
Differential Revision: D8311107
fbshipit-source-id: ada1908b320a5277eda2587d7e8f26b13b952154
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary:
The transaction is not necessary in there, since if any of the query fails then the commit is not written in changesets table.
We need to remove it, because with many writes to the filenodes table some of the transactions are timing out while waiting for the database's lock for transation
Reviewed By: farnz
Differential Revision: D8220722
fbshipit-source-id: 003b1369abb1a47e8ad37517745ab659116b95f6
Summary: It only needs to borrow them.
Reviewed By: kulshrax
Differential Revision: D8244267
fbshipit-source-id: 2a24a3b7c6eb65177e4e26c57650dd7e096b4202
Summary:
It will be used in getfiles wireproto method. It requires fetching the whole
file history, and doing lots of point lookups is too slow.
Reviewed By: lukaspiatkowski
Differential Revision: D8182050
fbshipit-source-id: b17cb2259b2237f0027bfb499fbe97d9b40b1a62
Summary:
Move functionality of converting FilenodeRow into FilenodeInfo into a separate
function. It will be used in the next diff
Reviewed By: lukaspiatkowski
Differential Revision: D8182051
fbshipit-source-id: f2276aa979200f71aa05808595842c09a9f5899f
Summary:
Motivation for this change is the following:
We want to avoid doing point lookups to the db to fetch the history, and
instead select the history for a file or a directory with a single select. That
should be many times faster.
After we select the data from filenodes table, we need to fetch copy info
data. To do that we need to either do lots of point lookups to the fixedcopyinfo (which
undermines the point of doing single select) or construct a giant select
statement. First option is inefficient, the second can introduce big code
complexity.
Instead let's add a single field that says whether we have fixed copy info or
not. Db size increase should be tiny.
For the data that we've already imported I'm planning to run a script that will
fill the has_copyinfo field correctly.
Reviewed By: jsgf
Differential Revision: D8164029
fbshipit-source-id: c91c99b065808a93a9b361914cf9b3822d78cb60
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary: It can be useful to understand how often do we hit a db, write to blobstore, etc
Reviewed By: farnz
Differential Revision: D7806954
fbshipit-source-id: 9c85cbb11e184ae9f7751e05e961980b29b7568a
Summary:
Update x86 and aarch64 toolchains, rust-crates-io update.
The `use_nested_groups` feature is now stable, so we don't need to list it in
`#![feature(...)]` lines.
Reviewed By: kulshrax
Differential Revision: D7690730
fbshipit-source-id: 5f74bb01df6bd88a66ac00a82e446c18e3a1c43d
Summary:
Just as with changesets, let's make filenodes async.
I had to make unit-tests use async_unit, otherwise tokio::spawn() failed with
SpawnedError. I'm not sure why it happened, but async_unit has fixed the issue
Reviewed By: lukaspiatkowski
Differential Revision: D7635554
fbshipit-source-id: c6130b95cee2d60d0b6eec0ab6c1b3e4f3a26ccd
Summary:
Add functions that return a db connection.
This is the first step to make filenodes truly async.
The code is very similar to Diesel Changesets implementation
Reviewed By: lukaspiatkowski
Differential Revision: D7585297
fbshipit-source-id: 93df9ccbbdf0c29d6c977fec83444bc5ba9aa707
Summary: mercurial_types::DFileNodeId should be replaced by types from mononoke_types in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619290
fbshipit-source-id: aa6a8e55ae3810c4531028c3b3db2e5730fe7846
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary:
open_or_create is useful for testing.
insert_chunk_size sets the mysql batch size
Reviewed By: farnz
Differential Revision: D7429043
fbshipit-source-id: 9d651a2f71abb9022c5390d059e4298efb49f38c
Summary: It will be used in BlobRepo, and it requires it to be Send + Sync
Reviewed By: farnz
Differential Revision: D7429058
fbshipit-source-id: 610051ba3ed7ce66dd8058e1ba9fabb3ffd12c4e
