Summary:
This version still misses:
- proper production-ready logging
- smarter handling of case where the queue entries related to each other do not fit in the limit or `older_than` limit, so the healer will heal much more entries without realizing it shouldn't do so.
Reviewed By: aslpavel
Differential Revision: D13528686
fbshipit-source-id: 0245becea7e4f0ac69383a7885ff3746d81c4add
Summary: the from_value_opt already knows how to properly parse an integer from sql response, no need to reinvent the wheel and introduce bugs
Reviewed By: aslpavel
Differential Revision: D13651328
fbshipit-source-id: 55af810c99b93bd2f9c67c721a9d0be6034ee466
Summary: MyRouter and cache support will come in next diffs
Reviewed By: StanislavGlebik
Differential Revision: D13465146
fbshipit-source-id: 0ede0e875d4a49794ff71173bd0d1563abb3ff08
Summary: Format files effected by next commit in a stack
Reviewed By: StanislavGlebik
Differential Revision: D13650639
fbshipit-source-id: d4e37acd2bcd29b291968a529543c202f6944e1a
Summary: These config options weren't used at all. Let's delete them from the configs
Reviewed By: aslpavel
Differential Revision: D13597545
fbshipit-source-id: 5e99b38e2e9fc4ba6d836322ceba38a8b65ad2f4
Summary:
Currently some tests are failing intermittently due to a timeout, this timeout
is caused by the integration tests making an external call to AclChecker which
is intermittently very slow.
For our integration tests we disable this call since it's not part of the test
suite to stop our tests from being flaky.
Reviewed By: StanislavGlebik
Differential Revision: D13580144
fbshipit-source-id: 0c26bb14dd222b888ca2638319071f4d99eab6df
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Previously metaconfig depended on BlobRepo, and so ManifoldArgs had to be
definided in BlobRepo. That was an weird dependency, but a necessary one
because of Mononoke config repo. In the previous diffs we got rid of the
Mononoke config repo, so now we can reverse the dependencies.
Reviewed By: lukaspiatkowski
Differential Revision: D13180160
fbshipit-source-id: efe713ce3b160c98d56fc13559c57a920146841f
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary:
As part of our read path rollout, we want to block user error from
creating new commits that will confuse blobimport. Make it possible to
configure a read-only repo
Reviewed By: StanislavGlebik
Differential Revision: D12945024
fbshipit-source-id: 4265bf57f8adac7965117b710b8285bac483b8ee
Summary:
Sharding filenodes by path should stop us knocking over databases -
make it configurable.
Reviewed By: StanislavGlebik
Differential Revision: D12894523
fbshipit-source-id: e27452f9b436842e1cb5e9e0968c1822f422b4c9
Summary:
Let's add an option to validate the getfiles content that we return to users on
some percent of requests.
It'll increase the latency so let's not enabling that by default
Reviewed By: HarveyHunt
Differential Revision: D10558180
fbshipit-source-id: 2d7ec4dfe7b37b7b5541013278006278d1df68fa
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
Summary: This is more general, and allows one to call `RepoClient` methods.
Reviewed By: farnz
Differential Revision: D9318658
fbshipit-source-id: 09b2e64bc0d423eafcb381902e03f349fc666a41
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary: This diff refactors the hook code to use the Bookmark struct instead of strings
Reviewed By: farnz
Differential Revision: D8724197
fbshipit-source-id: 920aa1266ca94b2bd8683a995e4fd781159bd5b1
Summary: This diff updates rep config to support per file hooks. We also extend the hookloader to support per file hooks.
Reviewed By: lukaspiatkowski
Differential Revision: D8713464
fbshipit-source-id: 65cf20957506adc620bf06d84707669d125366ed
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D8625178
fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.
Reviewed By: farnz
Differential Revision: D8647161
fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: This diff adds new sections in the repo config for bookmarks and hooks
Reviewed By: StanislavGlebik
Differential Revision: D8506052
fbshipit-source-id: 71c2734b1aed90e92ed3d9a327d034f3083dacad
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
