Summary:
This is based on my reading of the source code, so I may have gotten things
wrong. Feel free to editorialize!
By putting this in a doc, hopefully it makes it easier for us to reason about
the API at a high level. Obviously it would be great if we could keep this up
to date going forward.
Reviewed By: StanislavGlebik
Differential Revision: D10340604
fbshipit-source-id: 9f3e82d234842e06c52f8a8b4440f8e06c487c0b
Summary:
Correctly handles case conflicting renames (only change in casing).
- path can now be removed from `CaseConlictingTrie`
- `check_case_conflicts` operates on `BonsaiChangeset` in pushrebase logic
Reviewed By: StanislavGlebik
Differential Revision: D10447522
fbshipit-source-id: d5342e7aa48154debee123b38bf3168e3371baa6
Summary:
It was broken because it only matched conflict markers that were on the first
line. This diff fixes it by splitting the file content by \n first
Reviewed By: farnz
Differential Revision: D10447393
fbshipit-source-id: a2091f6bc43e8bb9a77c63536e749432d524bbff
Summary:
This hook is designed to prevent text directives in .gitattributes
from making it into the repo.
As noted in the integration test, our regex may be too loose,
but it's probably OK, in practice.
For better or worse, for now, we're just trying to maintain the
behavior of the existing hook (though perhaps the existing hook
would have been a bit stricter if it wren't written in Bash).
For easy reference, here are the Git docs on gitattributes:
https://git-scm.com/docs/gitattributes/
Reviewed By: StanislavGlebik
Differential Revision: D10387336
fbshipit-source-id: c58f689ecc0648c2cc359a818c92d701258e8f46
Summary:
We want to deny landing files whose path contains magic strings. Add a
hook to do this, with some predefined examples of how to write patterns
Reviewed By: StanislavGlebik
Differential Revision: D10446531
fbshipit-source-id: 67f1a712d923345288c8d0a4f3e5da1e8f4e29f8
Summary:
Using multiple Runtimes might be a cause of problems in future and even if it isn't it will be a cause for investigating whether it is a problem or not.
The issue I have in mind is that if someone runs a future on one runtime that calls `tokio::spawn` on it (f.e. schedule a job that works for ever) but then uses a different runtime to drive another future to complection one might not suspect that the previous spawn is already lost with the previous Runtime.
Reviewed By: farnz
Differential Revision: D10446122
fbshipit-source-id: 4bfd2a04487a70355a26f821e6348f5223901c0d
Summary:
Previously buffered() wasn't particularly useful because it buffered only
mapping from ChangesetId to HgChangesetId. The actual running of hooks was done in
`.and_then()` and that means that each future in the stream should finish
before the next one starts.
Let's put running of hooks inside a buffer, that helps with perf a lot.
Reviewed By: jsgf
Differential Revision: D10359546
fbshipit-source-id: 48b8b200d7397eef8622c32cad9cec889b96f9d0
Summary:
Let's change level of logging for each hook/commit to debug, since it was
spammy. Instead let's print a total statistics about how many hooks were
accepted/rejected
Reviewed By: jsgf
Differential Revision: D10358786
fbshipit-source-id: 2e451d482ed5549e41975f9e3b57b05d90069788
Summary:
Let's add an option `--continuous`. If it's not specified then hook tailer runs
only once. That's useful for testing the new hooks.
Reviewed By: jsgf
Differential Revision: D10358785
fbshipit-source-id: b62d01b4bf3233c3f411fc298fefb79da473d7f1
Summary: Previously it always returned 1. This diff fixes it
Reviewed By: farnz
Differential Revision: D10401362
fbshipit-source-id: feceba24ca8931555b11ad29164127c4015ec751
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary: Reverting the myrouter based filenodes for now as they cause some problems
Reviewed By: jsgf
Differential Revision: D10405364
fbshipit-source-id: 07da917455ae5af9ef81a24d99f516171101c8a7
Summary:
According to [Mercurial Lfs Plan](https://www.mercurial-scm.org/wiki/LfsPlan), on push, for files which size is above the threshold (lfs.threshold config) hg client is sending LFS metadata instead of actual files contents. The main part of LFS metadata is SHA-256 of the file content (oid).
The format requires the following mandatory fields: version, oid, size.
When lfs metadata is sent instead of a real file content then lfs_ext_stored flag is in the request's revflags.
If this flag is set, We are ignoring sha-1 hash verification inconsistency.
Later check that the content is actually loaded to the blobstore and create filenode envelope from it, load the envelope to the blobstore.
Filenode envelope requires the following info:
- size - retrieved on fetching the actual data from blobstore.
- copy_from - retrieved from the file, sent by hg client.
Mononoke still does the same checks for LFS push as for non-lfs push (i.e. checks that all the necessary manifests/filelogs were uploaded by a client)
Reviewed By: StanislavGlebik
Differential Revision: D10255314
fbshipit-source-id: efc8dac4c9f6d6f9eb3275d21b7b0cbfd354a736
Summary:
Add libnfs, libffi and starlark.
Also nom now has "verbose-errors" feature (via bindgen -> cexpr -> nom), so make some tweaks to cope.
Reviewed By: farnz
Differential Revision: D10371391
fbshipit-source-id: ba889ad16a7b662c5eddafcb1e705b068ccc9af7
Summary:
This is my first Lua code, but I think this should work?
I'm assuming there's no overhead of accessing `type` from a closure that we
need to worry about?
Reviewed By: StanislavGlebik
Differential Revision: D10338505
fbshipit-source-id: 497da36c9ce56da50fb5f5cf26fcd8d8340b18c2
Summary:
Previously `create_rebased_changesets` was passed incorrect `root` value in case
it was a second or later retry attempt. It was passed the new value of `onto`
bookmark. And then in `create_rebased_changesets` it was used to rebase from
an ancestor of a root changeset to a new bookmark value. But since it wasn't
actually a root value but a new bookmark value, then the commit wasn't rebased
at all.
This diff fixes it
Reviewed By: aslpavel
Differential Revision: D10373516
fbshipit-source-id: dd5ce696d0dd2efaab3886e5ec17d51da4c82078
Summary:
Mercurial stores executable bit as part of the manifest, so if changeset only changes that attribute of a file Hg reuses file hash. But mononoke has been creating additional file node. So this change tries to handle this special case. Note this kind of reuse only happens if file has only one parent [P60183653](P60183653)
Some of our fixtures repo were effected, hence this hashes were replaced with updated ones
```
396c60c14337b31ffd0b6aa58a026224713dc07d => a5ab070634ab9cbdfc92404b3ec648f7e29547bc
339ec3d2a986d55c5ac4670cca68cf36b8dc0b82 => c10443fa4198c6abad76dc6c69c1417b2e821508
b47ca72355a0af2c749d45a5689fd5bcce9898c7 => 6d0c1c30df4acb4e64cb4c4868d4c974097da055
```
Reviewed By: farnz
Differential Revision: D10357440
fbshipit-source-id: cdd56130925635577345b08d8ed0ae6e229a82a7
Summary:
Updates rust-crates-io to add several HTTP/2 related crates, namely, `h2`, `httpbis`, and `curl`.
Additionally, fix any breakages introduced by new crate versions. (In particular, the `blake2` crate had breaking changes from 0.7.1 -> 0.8.)
Reviewed By: DurhamG
Differential Revision: D10346164
fbshipit-source-id: 6805261542b5b9c46a34cad6cf6e9fe38f074e87
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary: It got a bit rusty, this diff adds missing stuff
Reviewed By: farnz
Differential Revision: D10302729
fbshipit-source-id: af598f8c8fdd5c938c07052c03ab0f84fc6d3c20
Summary:
One brainless idiot decided to prune all trees from changed files calcualation.
Since it also prunes subtrees, that leaves with just files in the root
directory.
Reviewed By: lukaspiatkowski
Differential Revision: D10302299
fbshipit-source-id: 8fe2c4ad8de998dfd4083d97cd816d85b5fec604
Summary:
Hooks that makes sure that there are no conflict markers in file contents.
This hook is bypassable.
Reviewed By: purplefox
Differential Revision: D10260230
fbshipit-source-id: b9d69e757f18ed3f4f889a01032ef7360cba6867
Summary: Not a final version for sure, just a small improvement
Reviewed By: lukaspiatkowski
Differential Revision: D10260231
fbshipit-source-id: 9f9f61f23da5ac9a5d1abc9ad2f50900ca434326
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary: also includes a fix for blobimportjob passing an empty argument to blobimport when was not used
Reviewed By: farnz
Differential Revision: D10298965
fbshipit-source-id: 1e9475c53a7e0b7ea211cc8fbd2c327327b66c65
Summary: Add a new method for changeset hooks to allow the content of any file to be retrieved.
Reviewed By: StanislavGlebik
Differential Revision: D10255914
fbshipit-source-id: 4ec89369835a2807675b1eda41b4399cf0c66b32
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary:
Many editors remove trailing whitespaces on save. That makes modifying these
files annoying. Adding ` (re)` mitigates the issue
Reviewed By: farnz
Differential Revision: D10237590
fbshipit-source-id: 1473f35023b878f21ff22bd5a5ccb5f11884cef3
Summary:
Hooks need to know whether file was added, modified or removed. For example, we
can't fetch content of a removed file. Also hook authors may want to allow
modifying existing files of a particular type, but they may want to disallow
addition of new files of this type.
`cs.files()` doesn't give information about whether a file was
added/deleted/modified, so we have to use `get_changed_stream` function from
manifest_utils.
Note - currently it still returns incorrect list of changed files for merges.
It will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D10237587
fbshipit-source-id: cd7f76334070cde451b4690071d03275e40c95f3
Summary:
This assert was just broken - it's fine to call `get_full_path()` on file
entry. What's disturbing is that there were no tests that cover this behaviour
i.e. no tests returned modified file!
This diff fixes both problems
Reviewed By: farnz
Differential Revision: D10237589
fbshipit-source-id: dcb7f1977768262491b4624a30a5e861c3c1eadf
Summary:
Tests are for changegroup version3
Unitests in part_inner.rs
QuickCheck for Ch3Packer and Unpacker.
Packer has the same implementation, as Packer packs everything operating with chunks, and does not care about sections.
In quickcheck tests, my proposal is to have similar CgSeqGenerator for both of Cg versions.
But as sections are different, to introduce optional sections.
Reviewed By: StanislavGlebik
Differential Revision: D10192025
fbshipit-source-id: bef0d654724f18177461c6326324d6972943eb23
Summary:
Support ChangeGroup version 3 from Mercurial
on Push, Hg client sends bundle, which is encoded in type 2 or type 3.
The type depends on which extentions are turned on in client.
The type is written in PartHeader, part_type: B2xRebase, aparams: {"cgversion": b"03"}
The differences between type 2 and type 3:
- could be found in fbcode/scm/hg/mercurial/help/internals/changegroups.txt
- The flow of parts is different. There is a new part in type 3:
-- type 2: Changeset, Manifest, Filename, Filelog
-- type 3: Changeset, Manifest, Treemanifest, Filename, Filelog
- According to hg implementation, Treemanifest part should always be empty (consists of Section End chunk (4 bytes of zeores) )
- Different chunk header size, as flags (unsigned int) is added. (type 2 - 100 bytes, type3 - 102 bytes in total)
Mononoke should support both versions, as if LFS is turned on, hg client is sending bundles in version 3 type.
Reviewed By: StanislavGlebik, farnz
Differential Revision: D10162321
fbshipit-source-id: f2ae664206b147db5f24312942d7fcf89ccd69b3
Summary:
We now have a way for a MySQL database to tell us how to send
streaming clones to the client. Hook it all up, so that (with any luck), once
we have data in MySQL and the blobstore, we'll see working streaming clones.
Reviewed By: StanislavGlebik
Differential Revision: D10130774
fbshipit-source-id: b22ffb642d0a54b09545889779f79e7a0f81acd7
Summary:
We used generate copy metadata which mercurial understand but works a bit differently
Relevant mercurial code:
```
def packmeta(meta, text):
keys = sorted(meta)
metatext = "".join("%s: %s\n" % (k, meta[k]) for k in keys)
return "\1\n%s\1\n%s" % (metatext, text)
```
Reviewed By: StanislavGlebik
Differential Revision: D10200805
fbshipit-source-id: f17a51a6aac2e1d3671fbbf3e969ed747e2fce18
