Summary: There's a hidden dependency here -- ideally it'd be made explicit.
Reviewed By: jsgf
Differential Revision: D9307939
fbshipit-source-id: e00a201e20bb8d83be09eab10cfdd2ce03a3e22f
Summary:
This diff implements a service which tails a configurable bookmark and runs hooks against it.
It uses the standard Mononoke config from the meta config repo.
Reviewed By: StanislavGlebik
Differential Revision: D8898637
fbshipit-source-id: f710fe4c9bda1b78bd17eb6cd6abf2abda4fdb8e
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
We'll be running in Tupperware, and want to shrink when we get too
large to avoid OOM due to caches. Configure cachelib appropriately
Reviewed By: StanislavGlebik
Differential Revision: D8900371
fbshipit-source-id: 4f1f64c2508c64e4ce2d201e0a0e86446f84ffef
Summary:
Since we have bonsai changesets now, let's add a subcommand to print bonsai
changesets given an hg changeset
Reviewed By: farnz
Differential Revision: D9179315
fbshipit-source-id: fad8a7454b45608482ac9fbf0dae9be16d5cbd69
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary:
This diff gets the machinery in place to allow file hooks to access file content. It uses the manifest vfs crate to do this.
Initially we just expose a simple contains_string() method which checks if the file content contains the given string (this is the most common thing file hooks want to do with content), but we can extend this with further methods as the need arises.
Reviewed By: StanislavGlebik
Differential Revision: D8889837
fbshipit-source-id: fb5191d6d1286f070a99108782ff529914aeb198
Summary: This verifier works on manifests -- we'll need other ones for filenodes etc.
Reviewed By: StanislavGlebik
Differential Revision: D9105976
fbshipit-source-id: 939b72b6a3f69b716385315f69a57e91d71a45f3
Summary: The content-fetch subcommand will try to lookup a changeset by bookmark first and then falls back to a hash.
Reviewed By: StanislavGlebik
Differential Revision: D9110531
fbshipit-source-id: 3bd0e452739fc1c8b41dc559864f73e3a1f3f782
Summary: This is a workaround for buggy commits produced by older versions of treemanifest.
Reviewed By: farnz
Differential Revision: D9018502
fbshipit-source-id: 71f2aeeaee01500a1a73814dc9c4979ef2c79746
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
The goal of this tool is to verify that bonsai changesets roundtrip with the same hashes. This uses the `ChangesetVisitor` stuff introduced in the previous diff to walk over a specific number of changesets.
I've been using this tool for a few days, and it's shaken out a number of bugs in the Bonsai support in Mononoke, and even found old bugs in Mercurial. So it's been a pretty useful exercise overall.
Reviewed By: farnz
Differential Revision: D8817725
fbshipit-source-id: b6976e8ff9024098c695e82383d98d95b2701e1b
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
I don't like glog for interactive use at all. But keep it as the
default for blobimport, and add a flag to change it.
Reviewed By: StanislavGlebik
Differential Revision: D8909674
fbshipit-source-id: d0b9c439f72f231c95e9109e16b30e87cfaa2eed
Summary:
This command will create a PROD bookmark on the currently checked out commit of the choosen mononoke-config repo, then import it into a temp directory and finally invoke fbpkg build to create a package with that imported content.
It also calls fbpkg build inside the mononoke-config repo so that the VCS info from mercurial is persisted in fbpkg metadata.
Reviewed By: farnz
Differential Revision: D8907916
fbshipit-source-id: ee1843bf23c9bffe524d2b20c7322f84825e2663
Summary: make it easier to import mononoke-config repo, can be used in development, but it also will be used in fbpkg in further diffs
Reviewed By: farnz
Differential Revision: D8887356
fbshipit-source-id: 6dbcccff7fedc6a72be1b8240e49d4fba8cf89a1
Summary: later this library will be used by admin tool to ease importing of config repo
Reviewed By: StanislavGlebik
Differential Revision: D8882178
fbshipit-source-id: 293a26b038f8d76e9fcedb72a4041a48f502a00a
Summary:
can be used to easily clone mononoke-config and required hgrc setup
later this subcomand will be used to blobimport the mononok-config as well
Reviewed By: StanislavGlebik
Differential Revision: D8879500
fbshipit-source-id: 1ad19441bce73f5a8c56f5aebd04f331ca49519a
Summary: Let's pass just one field instead of 4
Reviewed By: farnz
Differential Revision: D8889899
fbshipit-source-id: 8b30496a86950ed534439f5469f8740ee32345b8
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate
Reviewed By: StanislavGlebik
Differential Revision: D8861170
fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
Summary:
There seems to be a deadlock in the internals of the now outdated tokio_core.
After applying the modern tokio::run the deadlock is not being triggered.
Reviewed By: farnz
Differential Revision: D8783183
fbshipit-source-id: 47a7d1d8e2756ea4d40812d0b8a6c850d7f7e9f8
Summary: This will be useful for the bonsai verification tool.
Reviewed By: StanislavGlebik
Differential Revision: D8792562
fbshipit-source-id: f409d0fa042528b04462a1539fd3c2a8064a4f6e
Summary: This code can easily be shared.
Reviewed By: StanislavGlebik
Differential Revision: D8777307
fbshipit-source-id: f11314f6a63bb191dc38d07cec181a4b05b158d9
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary: This diff refactors the hook code to use the Bookmark struct instead of strings
Reviewed By: farnz
Differential Revision: D8724197
fbshipit-source-id: 920aa1266ca94b2bd8683a995e4fd781159bd5b1
Summary:
This diff extends lua hooks to support per file hooks.
We also now call the hook function via a Lua wrapper function which allows us to have a better API in the hook as we can construct lua table to pass into the hook and we can also return multiple return values from the hook, which allows us to support rich hook failure reasons properly. Both of these are hard to do when calling hooks directly using hlua.
Reviewed By: StanislavGlebik
Differential Revision: D8711280
fbshipit-source-id: b91f9e47a1f8eab302775a5bbfd61590a8635282
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
Summary: This diff implements tracking of hooks by bookmark in the hook manager and functionality to run hooks per bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D8525598
fbshipit-source-id: 7987d1f8d90a77667f120f4940f12aa3cb5aa86e
Summary:
In case the bookmarks have been updated while we are reading the revlogs we shoud read the bookmarks before reading changestes.
In this diff we are also reading bookmarks after importing so that we ensure that the newest possible version of bookmarks are persisted.
Reviewed By: farnz
Differential Revision: D8723522
fbshipit-source-id: 278382dae10a0554abc8edc398e7a15a37569676
Summary:
We need to be able to distinguish logical transaction failure from
infrastructure failure, so change the `Transaction::commit` future to
`Future<Item=bool, Error=Error>`, where `false` indicates a logical transaction
failure. This allows the caller to determine whether a retry or other recovery
logic is needed.
Reviewed By: lukaspiatkowski
Differential Revision: D8555727
fbshipit-source-id: 8ab64f3019f2644e7eaabc8d699d99aa8eb08fbb
Summary:
I want to use this code for the bonsai verification tool without
copying it around. It's also a nice simplification of the code.
Reviewed By: jsgf
Differential Revision: D8675825
fbshipit-source-id: cf9432069f154dafeb81a5e93a2fb5cbd717b981
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: This will be pretty useful while debugging.
Reviewed By: jsgf
Differential Revision: D8667770
fbshipit-source-id: 1c776741844d74529415124ab826b013414678c7
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary:
Fetching the blob is still required to compute the node hash, but we don't have
to reupload it.
Reviewed By: farnz
Differential Revision: D8508462
fbshipit-source-id: 341a1a2f82d8f8b939ebf3990b3467ed7ad9244c
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
This diff introduces the ChangesetStore trait so the HookManager does not directly depend on the BlobRepo. This means the manager can obtain changesets from a simple in memory implementation of ChangesetStore which can be populated with changesets before they are committed to the server. This means the exact same hook manager logic can be used irrespective of whether the changesets are committed or not.
The diff includes two implementations of ChangesetStore - A simple in memory implementation backed by a HashMap, and one that backs onto BlobStore.
Reviewed By: StanislavGlebik
Differential Revision: D8446430
fbshipit-source-id: 8d14e48cb562fcd10a17370e34f13a662af827df
Summary: This diff implements memoization of hooks. If a hook has already been run against the same changeset it is not run again, instead the cached result is used.
Reviewed By: StanislavGlebik
Differential Revision: D8431102
fbshipit-source-id: b4080ba48a3214e767392cbcb46425aa05bc2b64
Summary: This diff refactors the hook manager to run hooks based on a changeset_id not the actual changeset. The actual changeset is looked up by the hook manager lazily when running a hook. This is needed to make it work well for AsyncMemo as we don't want the changeset cached in the key.
Reviewed By: StanislavGlebik
Differential Revision: D8422442
fbshipit-source-id: 40bc89124942e05c5aaeb2b4ee00215afd816642
Summary: This diff refactors the runhook tests to use the linear fake blobrepo to supply changesets
Reviewed By: farnz
Differential Revision: D8418096
fbshipit-source-id: 74fd2578095dbae86ed9e96eac3ca4b344c036da
