Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary: This commit upgraded openssl, enabled alpn for actix-web and added tokio-codec with fixes due to the upgrade.
Reviewed By: StanislavGlebik
Differential Revision: D8682673
fbshipit-source-id: 8c7cadfd6c0c7b016202f6cb038eb4951d0f9333
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
Let's use the new feature in SendWrapper to use many io threads. That will help
us mitigate the high cpu usage issues we were having with blobstore requests.
Manifold blobstore now creates the io threads itself.
Reviewed By: kulshrax
Differential Revision: D7831420
fbshipit-source-id: ec9f3327347ca6bfbd23c482e69a6fee663b1da5
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Let's use it! Pass config option that set's the cache max memory usage (don't
put a limit on the number of entries, it's useless in that case).
Currently we'll set a separate size for each of the caches that we use
(blobstore, changesets, filenodes, etc). Later we'll have just one single option that
sets the cache size for all of them.
Reviewed By: lukaspiatkowski
Differential Revision: D7671814
fbshipit-source-id: f9571078e6faaa80ea4c31c76a9eebcc24d8a68a
Summary:
Pass mysql tier name to the BlobRepo, so that we can use it to connect to mysql
based storages like mysql changeset storage, filenodes storage etc.
Note that currently Filenodes storage only connects to master region. This will
be fixed in the later diffs
Reviewed By: lukaspiatkowski
Differential Revision: D7585191
fbshipit-source-id: 168082abfeb7ccba549c7a49e6269cc01c490c14
Summary:
Now that `BlobNode` no longer returns `None`:
* don't expose the `BlobNode` API outside the crate because it turns out to not be very useful (it should probably go away eventually?)
* make the `File` API not return `Option` types
* Add a new `file_contents` that returns a brand-new `FileContents` (this is the first time we're tying together Mercurial and Mononoke data structures!)
Also remove a `Symlink` API that isn't really correct honestly.
Reviewed By: StanislavGlebik
Differential Revision: D7624729
fbshipit-source-id: 38443093b8bfea91384c959f3425cf355fac9f65
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary: Scuba is able to nicely format times in microseconds but not nanoseconds. Since all of the poll times we were recording in the various stats scuba tables were all >1000ns anyway, switch to using microseconds instead.
Reviewed By: farnz
Differential Revision: D7350980
fbshipit-source-id: 0fad1ffcf21a3ee3ecfcd7062efb42fb41e0809f
Summary: Remove usage of deprecated `time` crate in `futures-stats`, and fix all callsites using the new `time-ext` crate.
Reviewed By: farnz
Differential Revision: D7349956
fbshipit-source-id: 10ef86c4942b8533a734c7daadfa895f5ef92f23
Summary:
The `Option<&MPathElement>` type is more general -- it's easy to
convert from `&Option<MPathElement>` to it, but the other way around can
require a clone.
Reviewed By: farnz
Differential Revision: D7339161
fbshipit-source-id: 0c8ab57a19bc330245c612e3e0e3651e368ab8cb
Summary:
Most places expect MPathElements to be non-empty, and it's good to
have static type checking for that.
The only uses could easily be replaced.
Reviewed By: farnz
Differential Revision: D7337587
fbshipit-source-id: cf700b609478bba1f881a77c82a53f8b9863ad7f
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I want to be able to work in a private Manifold namespace when testing
that my hunches about Manifold/Mononoke interactions are good, and when testing
changes to the ManifoldBlob that should make life better. Make that possible.
In the future, once I have a Blobstore that simulates Manifold locally, this
also makes it possible to compare the simulation to Manifold without tripping
over other users.
Reviewed By: StanislavGlebik
Differential Revision: D7194754
fbshipit-source-id: 601bf1c2ded1af5db6a123fdd05600bc3eb5d7cf
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary: This starts porting uses of Vec<u8> for file contents to the Bytes type.
Reviewed By: jsgf
Differential Revision: D7106766
fbshipit-source-id: 15d531836132317cede7a6f9d6b047a423deb5bb
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary: BlobChangeset has some methods that are useful also outside of blobrepo crate, like getting it's node.
Reviewed By: farnz
Differential Revision: D7056035
fbshipit-source-id: 197d261fd21ab9332950d3fda401e7dab3730cb5
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Give them correct names, because this functions actually fetch node from
blobstore and strip copy metadata if it exists.
Reviewed By: farnz
Differential Revision: D6871271
fbshipit-source-id: 4d8c83786b223f5cd4f55188fbcb8f438713d3b7
Summary:
Update tp2 links for rust-crates-io and add corresponding build fixes due to
API changes.
Reviewed By: kulshrax
Differential Revision: D6824642
fbshipit-source-id: 5f30fa75815a6bb2de593878c40c0fa44219c3f3
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary: It removes a bit of copy-pastes code
Reviewed By: sid0
Differential Revision: D6627250
fbshipit-source-id: b110dcad998a240b6da60bc3347c7c5c0370aae0
Summary:
It won't be used in prod anyway, and for local use rocksdb blob state is still
preferable
Reviewed By: sid0
Differential Revision: D6627197
fbshipit-source-id: d9ca55b0221c050e8e8e35914aff22906198874a
Summary:
Fetching file sizes is very expensive - it's one more round-trip to the
blobstore.
Add method that returns tree content, but doesn't fetch blob sizes.
Reviewed By: jsgf
Differential Revision: D6545108
fbshipit-source-id: 3d335eafa9dc367fb9e607f5c147b7e5c37cf133
Summary:
Don't use failure's bail!() and ensure!() macros.
Instead, failure_ext provides:
- bail_err!(err) - Converts its single parameter to the expected error and returns; ie `return Err(From::from(err));`
- bail_msg!(fmt, ...) - takes format string parameters and returns a `failure::err_msg()` error
- ensure_err!(), ensure_msg!() - corresponding changes
Also:
- remove all stray references to error-chain
- remove direct references to failure_derive (it's reexported via failure and failure_ext)
- replace uses of `Err(foo)?;` with `bail_err!()` (since `bail_err` unconditionally returns, but `Err(x)?` does not in principle, which can affect type inference)
Reviewed By: kulshrax
Differential Revision: D6507717
fbshipit-source-id: 635fb6f8c96d185b195dff171ea9c8db9e83af10
Summary:
Convert scm/mononoke to use failure, and update common/rust crates it depends on as well.
What it looks like is a lot of deleted code...
General strategy:
- common/rust/failure_ext adds some things that are in git failure that aren't yet in crates.io (`bail!` and `ensure!`, `Result<T, Error>`)
- everything returns `Result<T, failure::Error>`
- crates with real error get an error type, with a derived Fail implementation
- replicate error-chain by defining an `enum ErrorKind` where the fields match the declared errors in the error! macro
- crates with dummy error-chain (no local errors) lose it
- `.chain_err()` -> `.context()` or `.with_context()`
So far the only place I've needed to extract an error is in a unit test.
Having a single unified error type has simplified a lot of things, and removed a lot of error type parameters, error conversion, etc, etc.
Reviewed By: sid0
Differential Revision: D6446584
fbshipit-source-id: 744640ca2997d4a85513c4519017f2e2e78a73f5
Summary: This diff introduce the logging mechanism to log the request data to scuba table whenever someone hits any API to EdenServer.
Reviewed By: StanislavGlebik
Differential Revision: D6414207
fbshipit-source-id: a8ce2a1dd279b3ac576992e9423c246f3e98c7da
Summary:
This diff does a few things:
- Change the rust versions in `third-party{2,-buck}/config.py` to 1.21.
- Update the tp2 symlinks for `rust` and `rust-crates-io`
- Fix build breakages due to new errors/warnings from rustc.
Reviewed By: jsgf, Imxset21
Differential Revision: D6319954
fbshipit-source-id: cd4fe9e0d6f26c1a6c9c3f1256d84cb002bb83d6
Summary:
Config file needs to have ca file in PEM format.
It is used to authenticate client.
See code comments for more details.
Reviewed By: jsgf
Differential Revision: D6323439
fbshipit-source-id: 6b6f1fd68605b263dcb33b051843e10d3f5cb38e
Summary:
Avoid using plain HTTP and use HTTPS instead.
To do this config needs to provide paths to server certificate and private key files in PEM format.
Then they will be converted to Pkcs12 archive.
This diff adds authentication of server i.e. client can check that it talks to a real server. Next diff adds authentication of a client.
Lower-level `hyper::server::Http::bind_connection()` is used instead of `hyper::server::Http::bind()` method in order to add TLS support.
See code comments for more details.
Implementation is more complicated than I expected it to be. I need to use 3 more new crates. Lmk if there is a better way to do this.
Reviewed By: jsgf
Differential Revision: D6323440
fbshipit-source-id: 544f27e6ec210ddf840212b0c0c94145980e8be3
