Summary: Format files effected by next commit in a stack
Reviewed By: StanislavGlebik
Differential Revision: D13650639
fbshipit-source-id: d4e37acd2bcd29b291968a529543c202f6944e1a
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Sharding filenodes by path should stop us knocking over databases -
make it configurable.
Reviewed By: StanislavGlebik
Differential Revision: D12894523
fbshipit-source-id: e27452f9b436842e1cb5e9e0968c1822f422b4c9
Summary:
One brainless idiot decided to prune all trees from changed files calcualation.
Since it also prunes subtrees, that leaves with just files in the root
directory.
Reviewed By: lukaspiatkowski
Differential Revision: D10302299
fbshipit-source-id: 8fe2c4ad8de998dfd4083d97cd816d85b5fec604
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
Hooks need to know whether file was added, modified or removed. For example, we
can't fetch content of a removed file. Also hook authors may want to allow
modifying existing files of a particular type, but they may want to disallow
addition of new files of this type.
`cs.files()` doesn't give information about whether a file was
added/deleted/modified, so we have to use `get_changed_stream` function from
manifest_utils.
Note - currently it still returns incorrect list of changed files for merges.
It will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D10237587
fbshipit-source-id: cd7f76334070cde451b4690071d03275e40c95f3
Summary:
Previously cachelib cmdline args were added only to cmd line binaries, but not
to Mononoke this diff fixes it.
Reviewed By: farnz
Differential Revision: D10083899
fbshipit-source-id: 8febba96561c5ab9a61f60fafc7a7e56985dc038
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
This is mainly generated by
```
fbgs mononoke_test_2 -l -s | xargs sed -i 's/mononoke_test_2/mononoke_production/g'
```
mononoke config repo is still pending todo. But it's ok to do it in several run
as right now 2 names are pointing to the same shard.
Reviewed By: StanislavGlebik
Differential Revision: D9939694
fbshipit-source-id: ded772037844a220b18d99b207c976b88dafdaa5
Summary:
This diff implements a service which tails a configurable bookmark and runs hooks against it.
It uses the standard Mononoke config from the meta config repo.
Reviewed By: StanislavGlebik
Differential Revision: D8898637
fbshipit-source-id: f710fe4c9bda1b78bd17eb6cd6abf2abda4fdb8e
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
We'll be running in Tupperware, and want to shrink when we get too
large to avoid OOM due to caches. Configure cachelib appropriately
Reviewed By: StanislavGlebik
Differential Revision: D8900371
fbshipit-source-id: 4f1f64c2508c64e4ce2d201e0a0e86446f84ffef
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary:
This diff gets the machinery in place to allow file hooks to access file content. It uses the manifest vfs crate to do this.
Initially we just expose a simple contains_string() method which checks if the file content contains the given string (this is the most common thing file hooks want to do with content), but we can extend this with further methods as the need arises.
Reviewed By: StanislavGlebik
Differential Revision: D8889837
fbshipit-source-id: fb5191d6d1286f070a99108782ff529914aeb198
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary: This diff refactors the hook code to use the Bookmark struct instead of strings
Reviewed By: farnz
Differential Revision: D8724197
fbshipit-source-id: 920aa1266ca94b2bd8683a995e4fd781159bd5b1
Summary:
This diff extends lua hooks to support per file hooks.
We also now call the hook function via a Lua wrapper function which allows us to have a better API in the hook as we can construct lua table to pass into the hook and we can also return multiple return values from the hook, which allows us to support rich hook failure reasons properly. Both of these are hard to do when calling hooks directly using hlua.
Reviewed By: StanislavGlebik
Differential Revision: D8711280
fbshipit-source-id: b91f9e47a1f8eab302775a5bbfd61590a8635282
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.
The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D8707726
fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
Summary: This diff implements tracking of hooks by bookmark in the hook manager and functionality to run hooks per bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D8525598
fbshipit-source-id: 7987d1f8d90a77667f120f4940f12aa3cb5aa86e
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary:
This diff introduces the ChangesetStore trait so the HookManager does not directly depend on the BlobRepo. This means the manager can obtain changesets from a simple in memory implementation of ChangesetStore which can be populated with changesets before they are committed to the server. This means the exact same hook manager logic can be used irrespective of whether the changesets are committed or not.
The diff includes two implementations of ChangesetStore - A simple in memory implementation backed by a HashMap, and one that backs onto BlobStore.
Reviewed By: StanislavGlebik
Differential Revision: D8446430
fbshipit-source-id: 8d14e48cb562fcd10a17370e34f13a662af827df
Summary: This diff implements memoization of hooks. If a hook has already been run against the same changeset it is not run again, instead the cached result is used.
Reviewed By: StanislavGlebik
Differential Revision: D8431102
fbshipit-source-id: b4080ba48a3214e767392cbcb46425aa05bc2b64
Summary: This diff refactors the hook manager to run hooks based on a changeset_id not the actual changeset. The actual changeset is looked up by the hook manager lazily when running a hook. This is needed to make it work well for AsyncMemo as we don't want the changeset cached in the key.
Reviewed By: StanislavGlebik
Differential Revision: D8422442
fbshipit-source-id: 40bc89124942e05c5aaeb2b4ee00215afd816642
Summary: This diff refactors the runhook tests to use the linear fake blobrepo to supply changesets
Reviewed By: farnz
Differential Revision: D8418096
fbshipit-source-id: 74fd2578095dbae86ed9e96eac3ca4b344c036da
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary: This test implements the passing of all current changeset fields to hooks
Reviewed By: StanislavGlebik
Differential Revision: D8298019
fbshipit-source-id: 0e6be3c83b1e4d4c3eab95c76c9041ea2a57f0d3
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: This diff adds unit tests for the runhook command line utility
Reviewed By: jsgf
Differential Revision: D8257571
fbshipit-source-id: 5c390d2a45d895080fce28dcd7943da5d803ff92
Summary:
This diff adds more robust testing for various errors in Lua hooks.
It also contains a little bit of cleanup in the runHook command
Reviewed By: StanislavGlebik
Differential Revision: D8253525
fbshipit-source-id: de9d298e70ec647f2c13e27c9937605ac5b57485
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary:
Now hook manager returns result of running hooks as BoxFuture<HashMap<String, HookExecution> where HookExecution is a new richer type representing the result of running a hook.
This provides more info to the user as to why the hook rejected the changeset and a map is simpler to lookup a particular hook failure than a Vec.
Reviewed By: StanislavGlebik
Differential Revision: D8235970
fbshipit-source-id: 9a617b6d459f105aa9dad9782e784459dd716c45
