Summary:
Display the hash of the commit that didn't pass a hook,
which is a common occurrence in fbsource hooks using $HG_NODE.
I fixed up the tests, but test-hooks.t is broken from
the hg amend/fbamend fallout and also has some other issues. I tried to only add
the changes relevant to this commit.
Reviewed By: StanislavGlebik
Differential Revision: D10466395
fbshipit-source-id: dd1cdc994171a014c3d4806804ace14e85e726d4
Summary:
Background:
According to [git lfs protocol](https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md) HTTP POST "batch" request should return a link to
the look-aside server.
In our case Mononoke API server is a look-aside server, and process both "batch" request and "upload/download" requests.
So it need to return a link to itself.
New approach requests a separate lfs-url for "batch" request.
The previous approach requested attributes --http-host and --http-port to make a link to the instance of API server running.
Reviewed By: StanislavGlebik
Differential Revision: D10488586
fbshipit-source-id: ed9d78ee9bc78bdcec5eea813bd9aaa6e4590a5c
Summary:
For unification with commit cloud vip configuration apiserver should support the same health check api
Request needed for corp2prod
The same as : D10488369
Reviewed By: liubov-dmitrieva
Differential Revision: D10488494
fbshipit-source-id: 50b4024295c596342a8080474383de850bb7754a
Summary: Let's allow to set the number of commits processes by hook tailer.
Reviewed By: lukaspiatkowski
Differential Revision: D10361239
fbshipit-source-id: ced118d5dfca3c8aea65cb8a21f5b487f47628cd
Summary:
Bookmarks point to Bonsai changesets. So previously we were fetching bonsai
changeset for a bookmark then converting it to hg changeset in `get_bookmark`
method, then converting it back to bonsai in `pushrebase.rs`.
This diff adds method `get_bonsai_bookmark()` that removes these useless
conversions.
Reviewed By: farnz
Differential Revision: D10427433
fbshipit-source-id: 1b15911fc5d77483b5a135a8d4484fccff23c774
Summary:
This is based on my reading of the source code, so I may have gotten things
wrong. Feel free to editorialize!
By putting this in a doc, hopefully it makes it easier for us to reason about
the API at a high level. Obviously it would be great if we could keep this up
to date going forward.
Reviewed By: StanislavGlebik
Differential Revision: D10340604
fbshipit-source-id: 9f3e82d234842e06c52f8a8b4440f8e06c487c0b
Summary:
Correctly handles case conflicting renames (only change in casing).
- path can now be removed from `CaseConlictingTrie`
- `check_case_conflicts` operates on `BonsaiChangeset` in pushrebase logic
Reviewed By: StanislavGlebik
Differential Revision: D10447522
fbshipit-source-id: d5342e7aa48154debee123b38bf3168e3371baa6
Summary:
It was broken because it only matched conflict markers that were on the first
line. This diff fixes it by splitting the file content by \n first
Reviewed By: farnz
Differential Revision: D10447393
fbshipit-source-id: a2091f6bc43e8bb9a77c63536e749432d524bbff
Summary:
This hook is designed to prevent text directives in .gitattributes
from making it into the repo.
As noted in the integration test, our regex may be too loose,
but it's probably OK, in practice.
For better or worse, for now, we're just trying to maintain the
behavior of the existing hook (though perhaps the existing hook
would have been a bit stricter if it wren't written in Bash).
For easy reference, here are the Git docs on gitattributes:
https://git-scm.com/docs/gitattributes/
Reviewed By: StanislavGlebik
Differential Revision: D10387336
fbshipit-source-id: c58f689ecc0648c2cc359a818c92d701258e8f46
Summary:
We want to deny landing files whose path contains magic strings. Add a
hook to do this, with some predefined examples of how to write patterns
Reviewed By: StanislavGlebik
Differential Revision: D10446531
fbshipit-source-id: 67f1a712d923345288c8d0a4f3e5da1e8f4e29f8
Summary:
Using multiple Runtimes might be a cause of problems in future and even if it isn't it will be a cause for investigating whether it is a problem or not.
The issue I have in mind is that if someone runs a future on one runtime that calls `tokio::spawn` on it (f.e. schedule a job that works for ever) but then uses a different runtime to drive another future to complection one might not suspect that the previous spawn is already lost with the previous Runtime.
Reviewed By: farnz
Differential Revision: D10446122
fbshipit-source-id: 4bfd2a04487a70355a26f821e6348f5223901c0d
Summary:
Previously buffered() wasn't particularly useful because it buffered only
mapping from ChangesetId to HgChangesetId. The actual running of hooks was done in
`.and_then()` and that means that each future in the stream should finish
before the next one starts.
Let's put running of hooks inside a buffer, that helps with perf a lot.
Reviewed By: jsgf
Differential Revision: D10359546
fbshipit-source-id: 48b8b200d7397eef8622c32cad9cec889b96f9d0
Summary:
Let's change level of logging for each hook/commit to debug, since it was
spammy. Instead let's print a total statistics about how many hooks were
accepted/rejected
Reviewed By: jsgf
Differential Revision: D10358786
fbshipit-source-id: 2e451d482ed5549e41975f9e3b57b05d90069788
Summary:
Let's add an option `--continuous`. If it's not specified then hook tailer runs
only once. That's useful for testing the new hooks.
Reviewed By: jsgf
Differential Revision: D10358785
fbshipit-source-id: b62d01b4bf3233c3f411fc298fefb79da473d7f1
Summary: Previously it always returned 1. This diff fixes it
Reviewed By: farnz
Differential Revision: D10401362
fbshipit-source-id: feceba24ca8931555b11ad29164127c4015ec751
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary: Reverting the myrouter based filenodes for now as they cause some problems
Reviewed By: jsgf
Differential Revision: D10405364
fbshipit-source-id: 07da917455ae5af9ef81a24d99f516171101c8a7
Summary:
According to [Mercurial Lfs Plan](https://www.mercurial-scm.org/wiki/LfsPlan), on push, for files which size is above the threshold (lfs.threshold config) hg client is sending LFS metadata instead of actual files contents. The main part of LFS metadata is SHA-256 of the file content (oid).
The format requires the following mandatory fields: version, oid, size.
When lfs metadata is sent instead of a real file content then lfs_ext_stored flag is in the request's revflags.
If this flag is set, We are ignoring sha-1 hash verification inconsistency.
Later check that the content is actually loaded to the blobstore and create filenode envelope from it, load the envelope to the blobstore.
Filenode envelope requires the following info:
- size - retrieved on fetching the actual data from blobstore.
- copy_from - retrieved from the file, sent by hg client.
Mononoke still does the same checks for LFS push as for non-lfs push (i.e. checks that all the necessary manifests/filelogs were uploaded by a client)
Reviewed By: StanislavGlebik
Differential Revision: D10255314
fbshipit-source-id: efc8dac4c9f6d6f9eb3275d21b7b0cbfd354a736
Summary:
Add libnfs, libffi and starlark.
Also nom now has "verbose-errors" feature (via bindgen -> cexpr -> nom), so make some tweaks to cope.
Reviewed By: farnz
Differential Revision: D10371391
fbshipit-source-id: ba889ad16a7b662c5eddafcb1e705b068ccc9af7
Summary:
This is my first Lua code, but I think this should work?
I'm assuming there's no overhead of accessing `type` from a closure that we
need to worry about?
Reviewed By: StanislavGlebik
Differential Revision: D10338505
fbshipit-source-id: 497da36c9ce56da50fb5f5cf26fcd8d8340b18c2
Summary:
Previously `create_rebased_changesets` was passed incorrect `root` value in case
it was a second or later retry attempt. It was passed the new value of `onto`
bookmark. And then in `create_rebased_changesets` it was used to rebase from
an ancestor of a root changeset to a new bookmark value. But since it wasn't
actually a root value but a new bookmark value, then the commit wasn't rebased
at all.
This diff fixes it
Reviewed By: aslpavel
Differential Revision: D10373516
fbshipit-source-id: dd5ce696d0dd2efaab3886e5ec17d51da4c82078
Summary:
Mercurial stores executable bit as part of the manifest, so if changeset only changes that attribute of a file Hg reuses file hash. But mononoke has been creating additional file node. So this change tries to handle this special case. Note this kind of reuse only happens if file has only one parent [P60183653](P60183653)
Some of our fixtures repo were effected, hence this hashes were replaced with updated ones
```
396c60c14337b31ffd0b6aa58a026224713dc07d => a5ab070634ab9cbdfc92404b3ec648f7e29547bc
339ec3d2a986d55c5ac4670cca68cf36b8dc0b82 => c10443fa4198c6abad76dc6c69c1417b2e821508
b47ca72355a0af2c749d45a5689fd5bcce9898c7 => 6d0c1c30df4acb4e64cb4c4868d4c974097da055
```
Reviewed By: farnz
Differential Revision: D10357440
fbshipit-source-id: cdd56130925635577345b08d8ed0ae6e229a82a7
Summary:
Updates rust-crates-io to add several HTTP/2 related crates, namely, `h2`, `httpbis`, and `curl`.
Additionally, fix any breakages introduced by new crate versions. (In particular, the `blake2` crate had breaking changes from 0.7.1 -> 0.8.)
Reviewed By: DurhamG
Differential Revision: D10346164
fbshipit-source-id: 6805261542b5b9c46a34cad6cf6e9fe38f074e87
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary: It got a bit rusty, this diff adds missing stuff
Reviewed By: farnz
Differential Revision: D10302729
fbshipit-source-id: af598f8c8fdd5c938c07052c03ab0f84fc6d3c20
Summary:
One brainless idiot decided to prune all trees from changed files calcualation.
Since it also prunes subtrees, that leaves with just files in the root
directory.
Reviewed By: lukaspiatkowski
Differential Revision: D10302299
fbshipit-source-id: 8fe2c4ad8de998dfd4083d97cd816d85b5fec604
Summary:
Hooks that makes sure that there are no conflict markers in file contents.
This hook is bypassable.
Reviewed By: purplefox
Differential Revision: D10260230
fbshipit-source-id: b9d69e757f18ed3f4f889a01032ef7360cba6867
Summary: Not a final version for sure, just a small improvement
Reviewed By: lukaspiatkowski
Differential Revision: D10260231
fbshipit-source-id: 9f9f61f23da5ac9a5d1abc9ad2f50900ca434326
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary: also includes a fix for blobimportjob passing an empty argument to blobimport when was not used
Reviewed By: farnz
Differential Revision: D10298965
fbshipit-source-id: 1e9475c53a7e0b7ea211cc8fbd2c327327b66c65
Summary: Add a new method for changeset hooks to allow the content of any file to be retrieved.
Reviewed By: StanislavGlebik
Differential Revision: D10255914
fbshipit-source-id: 4ec89369835a2807675b1eda41b4399cf0c66b32
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c