Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
Summary: This is more general, and allows one to call `RepoClient` methods.
Reviewed By: farnz
Differential Revision: D9318658
fbshipit-source-id: 09b2e64bc0d423eafcb381902e03f349fc666a41
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary: This diff refactors the hook code to use the Bookmark struct instead of strings
Reviewed By: farnz
Differential Revision: D8724197
fbshipit-source-id: 920aa1266ca94b2bd8683a995e4fd781159bd5b1
Summary: This diff updates rep config to support per file hooks. We also extend the hookloader to support per file hooks.
Reviewed By: lukaspiatkowski
Differential Revision: D8713464
fbshipit-source-id: 65cf20957506adc620bf06d84707669d125366ed
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D8625178
fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.
Reviewed By: farnz
Differential Revision: D8647161
fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: This diff adds new sections in the repo config for bookmarks and hooks
Reviewed By: StanislavGlebik
Differential Revision: D8506052
fbshipit-source-id: 71c2734b1aed90e92ed3d9a327d034f3083dacad
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
Simple precaching. Reads all the manifests for a bookmark and up to
`commit_warmup_limit` of ancestors.
Warming up file content can be slow, so we don't do it now.
Reviewed By: jsgf
Differential Revision: D7863728
fbshipit-source-id: bed1508b01e4e002a399d00ea45faf8a8e228d0a
Summary:
Let's use the new feature in SendWrapper to use many io threads. That will help
us mitigate the high cpu usage issues we were having with blobstore requests.
Manifold blobstore now creates the io threads itself.
Reviewed By: kulshrax
Differential Revision: D7831420
fbshipit-source-id: ec9f3327347ca6bfbd23c482e69a6fee663b1da5
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
These have bitrotted, but I need a test case for a Scuba change. Fix
them.
Reviewed By: StanislavGlebik
Differential Revision: D7813907
fbshipit-source-id: e4e9b01a8a3c1de27f59c6d5ea695152df99d4ff
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Let's use it! Pass config option that set's the cache max memory usage (don't
put a limit on the number of entries, it's useless in that case).
Currently we'll set a separate size for each of the caches that we use
(blobstore, changesets, filenodes, etc). Later we'll have just one single option that
sets the cache size for all of them.
Reviewed By: lukaspiatkowski
Differential Revision: D7671814
fbshipit-source-id: f9571078e6faaa80ea4c31c76a9eebcc24d8a68a
Summary:
Pass mysql tier name to the BlobRepo, so that we can use it to connect to mysql
based storages like mysql changeset storage, filenodes storage etc.
Note that currently Filenodes storage only connects to master region. This will
be fixed in the later diffs
Reviewed By: lukaspiatkowski
Differential Revision: D7585191
fbshipit-source-id: 168082abfeb7ccba549c7a49e6269cc01c490c14
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary:
The current structure makes defining new manifests really verbose and
doesn't easily support nested manifests. Make it really simple: either accept a
path map or a description (a csv).
Also switch to a `BTreeMap` to allow lookups in the next diff.
This is unfortunately hard to separate out because all the bits are
interconnected.
Reviewed By: StanislavGlebik
Differential Revision: D7531115
fbshipit-source-id: ca0fa35d4e0ac6a4e23adb2a7d2b5679ce52b643
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary:
This codemod tries not to change the existing behavior of system, only introduce new types specific to Mercurial Revlogs.
It introduces a lot of copypasta intentionally and it will be cleaned in following diffs.
Reviewed By: farnz
Differential Revision: D7367191
fbshipit-source-id: 0a915f427dff431065e903b5f6fbd3cba6bc22a7
Summary:
The diff adds an extension of futures to the `failures_ext` crate, allowing futures with error type `failure::Error` or `failure::Fail` to store a context.
As a proof-of-concept, the resulting `context()` function is applied to a future in use in mononoke.
Reviewed By: lukaspiatkowski
Differential Revision: D7289640
fbshipit-source-id: 06788d03c8cb5563ae1621c8354eef651f9f1531
Summary:
RevlogRepo exposes a ton of methods that are almost equvalent to taking Revlog directly and ignoring the RevloRepo abstraction above it.
This diff cleans this up a bit, there are still some methods that the "old" blobimport uses, but the "new" one shouldn't need to do that.
Reviewed By: StanislavGlebik
Differential Revision: D7289445
fbshipit-source-id: ac7130fe41c4e4484d6986fe5b19d5adc751369a
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I want to be able to work in a private Manifold namespace when testing
that my hunches about Manifold/Mononoke interactions are good, and when testing
changes to the ManifoldBlob that should make life better. Make that possible.
In the future, once I have a Blobstore that simulates Manifold locally, this
also makes it possible to compare the simulation to Manifold without tripping
over other users.
Reviewed By: StanislavGlebik
Differential Revision: D7194754
fbshipit-source-id: 601bf1c2ded1af5db6a123fdd05600bc3eb5d7cf
Summary:
These are types that are going to be used throughout Mononoke -- I'm hoping to
avoid references to current Mercurial data structures in here.
For now, the only module I've moved is part of the `path` module. The
Mercurial-specific `fsencode` bits have been kept in mercurial-types (though
maybe they should move to `mercurial`?)
In the future, this module will also include definitions for unodes, etc.
Reviewed By: jsgf
Differential Revision: D7188722
fbshipit-source-id: fc097ca12c38a787f83e35af9b8dd308f2b910ea
Summary:
Add basic telemetry for some of hg operations that return a future. This is a draft I'm requesting feedback on (as advised by stash@).
1) The code is a bit repetitive. Is there a way to refactor it (for example, python decorator-like style)?
2) What is the best way to add telemetry for operations that return a BoxStream (and not a future)?
Reviewed By: StanislavGlebik
Differential Revision: D7056236
fbshipit-source-id: 7fd61b5533d0fe3857cc7865491b5e9e7240a886
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary: Initialize the RevlogRepo using the ChangesetId instead of the NodeHash.
Reviewed By: lukaspiatkowski
Differential Revision: D6867296
fbshipit-source-id: 12f0847324d138466421acc3b1f159a03655333b
Summary:
Config to connect to manifold. It's a test version, because it uses in-memory
bookmarks and heads.
Because Manifold BlobRepo require Remote to the tokio_core, I had to do a bit
of refactoring and delay BlobRepo creation. Now it's created after the
corresponding thread is created.
Reviewed By: jsgf
Differential Revision: D6748462
fbshipit-source-id: f14a3c98a6a00f44b5557255bc514df34325420f
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary:
Don't use failure's bail!() and ensure!() macros.
Instead, failure_ext provides:
- bail_err!(err) - Converts its single parameter to the expected error and returns; ie `return Err(From::from(err));`
- bail_msg!(fmt, ...) - takes format string parameters and returns a `failure::err_msg()` error
- ensure_err!(), ensure_msg!() - corresponding changes
Also:
- remove all stray references to error-chain
- remove direct references to failure_derive (it's reexported via failure and failure_ext)
- replace uses of `Err(foo)?;` with `bail_err!()` (since `bail_err` unconditionally returns, but `Err(x)?` does not in principle, which can affect type inference)
Reviewed By: kulshrax
Differential Revision: D6507717
fbshipit-source-id: 635fb6f8c96d185b195dff171ea9c8db9e83af10
Summary:
Convert scm/mononoke to use failure, and update common/rust crates it depends on as well.
What it looks like is a lot of deleted code...
General strategy:
- common/rust/failure_ext adds some things that are in git failure that aren't yet in crates.io (`bail!` and `ensure!`, `Result<T, Error>`)
- everything returns `Result<T, failure::Error>`
- crates with real error get an error type, with a derived Fail implementation
- replicate error-chain by defining an `enum ErrorKind` where the fields match the declared errors in the error! macro
- crates with dummy error-chain (no local errors) lose it
- `.chain_err()` -> `.context()` or `.with_context()`
So far the only place I've needed to extract an error is in a unit test.
Having a single unified error type has simplified a lot of things, and removed a lot of error type parameters, error conversion, etc, etc.
Reviewed By: sid0
Differential Revision: D6446584
fbshipit-source-id: 744640ca2997d4a85513c4519017f2e2e78a73f5
Summary: `changeset_exists` only checks that the changeset is in the blob store. Once we start accepting pushes, we'll also want to check that the changeset is reachable from a head. Use revsets to check this. Once we have a notion of "completeness" for changesets, we'll switch `known` to use that instead - this code is still useful as a way to go from `changeset_exists` to `changeset_complete` in bulk.
Reviewed By: jsgf
Differential Revision: D6205695
fbshipit-source-id: 7e3b5c30bc5e459feb95a20913d8a04f3fda7469
Summary:
There are few changes here:
- split main into more smaller functions
- add panic hook that exits the process if one of the threads paniced
- use never_type feature to better type functions that should never return
- few panics to error returning changes here and there
Reviewed By: farnz
Differential Revision: D5942238
fbshipit-source-id: 7407d95b61f64f3909c5bb14dd3aa1ddee452f3c
Summary:
Mercurial filelog entries may have metadata fields in the beginning, usually used to track copies/renames. Previously mononoke eden server returned this metadata as part of the file blob.
This diff changes it. Now `get_content()` method returns file content without metadata, and to make it consistent, both `get_content()` of the blobrepo and revlog repo do the same.
This decision certainly has it's tradeoffs, because now it's more difficult to get metadata (`get_raw_content` needs to be used).
But we'll probably change how metadata is stored anyway, that's why I think changing `get_content` method is fine.
This diff also cleans up server/src/main.rs file, because previously it had to strip metadata itself.
Also diff fixes problem in metadata parsing - it previously failed if file is less than 2 bytes
Reviewed By: farnz
Differential Revision: D5901476
fbshipit-source-id: f3ade0179710352590068c238e6a733aab68a512
Summary:
`Path` has the potential to be confused with `std::path::Path`.
`MPath` is nice, concise, and clearly different from `Path`.
Reviewed By: jsgf
Differential Revision: D5895665
fbshipit-source-id: dc5ed5c3866b227d753c6d904d3c6d213c882cd7
Summary: This parsing is only to check that the modules work together. The metaconfig module still needs to have some proper parsing implemented, Error handling and tests
Reviewed By: jsgf
Differential Revision: D5698544
fbshipit-source-id: 8cd3254c10f977a17ddc31ac2199ee4c9fa98355