Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary: It got a bit rusty, this diff adds missing stuff
Reviewed By: farnz
Differential Revision: D10302729
fbshipit-source-id: af598f8c8fdd5c938c07052c03ab0f84fc6d3c20
Summary:
One brainless idiot decided to prune all trees from changed files calcualation.
Since it also prunes subtrees, that leaves with just files in the root
directory.
Reviewed By: lukaspiatkowski
Differential Revision: D10302299
fbshipit-source-id: 8fe2c4ad8de998dfd4083d97cd816d85b5fec604
Summary:
Hooks that makes sure that there are no conflict markers in file contents.
This hook is bypassable.
Reviewed By: purplefox
Differential Revision: D10260230
fbshipit-source-id: b9d69e757f18ed3f4f889a01032ef7360cba6867
Summary: Not a final version for sure, just a small improvement
Reviewed By: lukaspiatkowski
Differential Revision: D10260231
fbshipit-source-id: 9f9f61f23da5ac9a5d1abc9ad2f50900ca434326
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary: also includes a fix for blobimportjob passing an empty argument to blobimport when was not used
Reviewed By: farnz
Differential Revision: D10298965
fbshipit-source-id: 1e9475c53a7e0b7ea211cc8fbd2c327327b66c65
Summary: Add a new method for changeset hooks to allow the content of any file to be retrieved.
Reviewed By: StanislavGlebik
Differential Revision: D10255914
fbshipit-source-id: 4ec89369835a2807675b1eda41b4399cf0c66b32
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary:
Many editors remove trailing whitespaces on save. That makes modifying these
files annoying. Adding ` (re)` mitigates the issue
Reviewed By: farnz
Differential Revision: D10237590
fbshipit-source-id: 1473f35023b878f21ff22bd5a5ccb5f11884cef3
Summary:
Hooks need to know whether file was added, modified or removed. For example, we
can't fetch content of a removed file. Also hook authors may want to allow
modifying existing files of a particular type, but they may want to disallow
addition of new files of this type.
`cs.files()` doesn't give information about whether a file was
added/deleted/modified, so we have to use `get_changed_stream` function from
manifest_utils.
Note - currently it still returns incorrect list of changed files for merges.
It will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D10237587
fbshipit-source-id: cd7f76334070cde451b4690071d03275e40c95f3
Summary:
This assert was just broken - it's fine to call `get_full_path()` on file
entry. What's disturbing is that there were no tests that cover this behaviour
i.e. no tests returned modified file!
This diff fixes both problems
Reviewed By: farnz
Differential Revision: D10237589
fbshipit-source-id: dcb7f1977768262491b4624a30a5e861c3c1eadf
Summary:
Tests are for changegroup version3
Unitests in part_inner.rs
QuickCheck for Ch3Packer and Unpacker.
Packer has the same implementation, as Packer packs everything operating with chunks, and does not care about sections.
In quickcheck tests, my proposal is to have similar CgSeqGenerator for both of Cg versions.
But as sections are different, to introduce optional sections.
Reviewed By: StanislavGlebik
Differential Revision: D10192025
fbshipit-source-id: bef0d654724f18177461c6326324d6972943eb23
Summary:
Support ChangeGroup version 3 from Mercurial
on Push, Hg client sends bundle, which is encoded in type 2 or type 3.
The type depends on which extentions are turned on in client.
The type is written in PartHeader, part_type: B2xRebase, aparams: {"cgversion": b"03"}
The differences between type 2 and type 3:
- could be found in fbcode/scm/hg/mercurial/help/internals/changegroups.txt
- The flow of parts is different. There is a new part in type 3:
-- type 2: Changeset, Manifest, Filename, Filelog
-- type 3: Changeset, Manifest, Treemanifest, Filename, Filelog
- According to hg implementation, Treemanifest part should always be empty (consists of Section End chunk (4 bytes of zeores) )
- Different chunk header size, as flags (unsigned int) is added. (type 2 - 100 bytes, type3 - 102 bytes in total)
Mononoke should support both versions, as if LFS is turned on, hg client is sending bundles in version 3 type.
Reviewed By: StanislavGlebik, farnz
Differential Revision: D10162321
fbshipit-source-id: f2ae664206b147db5f24312942d7fcf89ccd69b3
Summary:
We now have a way for a MySQL database to tell us how to send
streaming clones to the client. Hook it all up, so that (with any luck), once
we have data in MySQL and the blobstore, we'll see working streaming clones.
Reviewed By: StanislavGlebik
Differential Revision: D10130774
fbshipit-source-id: b22ffb642d0a54b09545889779f79e7a0f81acd7
Summary:
We used generate copy metadata which mercurial understand but works a bit differently
Relevant mercurial code:
```
def packmeta(meta, text):
keys = sorted(meta)
metatext = "".join("%s: %s\n" % (k, meta[k]) for k in keys)
return "\1\n%s\1\n%s" % (metatext, text)
```
Reviewed By: StanislavGlebik
Differential Revision: D10200805
fbshipit-source-id: f17a51a6aac2e1d3671fbbf3e969ed747e2fce18
Summary: This adds `--compare-commits` option for pushrebase replayer. Which also check that commits are close enough to treat them as equal for the purpose of pushrebase.
Reviewed By: StanislavGlebik
Differential Revision: D10084308
fbshipit-source-id: f1fd05173a9a7663125a89dd03b79b2deea40dc4
Summary:
Add a comment about the restriction of cs_id used in CopyInfo. It must match
one of parents mentioned in BonsaiChangeset.
This makes it clear the cs_id won't be the commit introducing the file
revision. It cannot be other things like an ancient commit in the history, or
a future commit in a different branch, either.
Reviewed By: sunshowers
Differential Revision: D10137729
fbshipit-source-id: 9b2afd7689bb93d45514ea9ab66667eb46a3a11f
Summary: This popped up while I was building Mononoke - fix it.
Reviewed By: StanislavGlebik
Differential Revision: D10126386
fbshipit-source-id: 117239dea88c3ecd921f852ce86691ba6aa8bb07
Summary: cachelib takes a lot of times to init, and it's not used in the tests.
Reviewed By: farnz
Differential Revision: D10123692
fbshipit-source-id: bce82af3f56aa8829127bf4c605a1d472735126a
Summary:
It makes it clear why blobimport failed. Before it required modification to the
tests to run `cat FILENAME`, now it's done automatically
Reviewed By: farnz
Differential Revision: D9850279
fbshipit-source-id: b9a252f50b6dcf4906c24aca70fa37a2cec84e46
Summary: This diff adds better logging for a warmup, and spawns changesets and manifest warmup on different tasks. Manifest warmup is cpu heavy, so changeset warmup has to just wait, that's why sending them to different tasks should help.
Reviewed By: lukaspiatkowski
Differential Revision: D10032437
fbshipit-source-id: fed61db7fdf1c09a14f5f59d54b5eb268cf21e5a
Summary:
PUT request upload to mononoke API
hg client sends a PUT request to store a file into blobstore during push supporting LFS
Upload file by alias is divied into 2 parts:
- Put alias : blobstore key
- Put blobstore_key: contents
Keep in mind, that file content is thrift encoded
host_address for batch request is from command line flags -H for host, -p for port
Reviewed By: StanislavGlebik
Differential Revision: D10026683
fbshipit-source-id: 6c2726c7fee2fb171582bdcf7ce86b22b0130660
Summary:
Previously cachelib cmdline args were added only to cmd line binaries, but not
to Mononoke this diff fixes it.
Reviewed By: farnz
Differential Revision: D10083899
fbshipit-source-id: 8febba96561c5ab9a61f60fafc7a7e56985dc038
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
We want to be able to do post-commit logging for all changesets. Set
up the data structures I intend to use for now, and arrange to discard all
logging.
A later diff will add logging to a ScribeClient.
Reviewed By: StanislavGlebik
Differential Revision: D9995984
fbshipit-source-id: 796b390f6b83ace576f73a217ac564c4251d7ec5
Summary:
Main reason is to make startup faster because ChangesetFetcher can uses bulk
caches in the same way as getbundle does.
Reviewed By: farnz
Differential Revision: D10032435
fbshipit-source-id: 717114339edf31865b498893d75695968447bb43
Summary:
Revsets should use ChangesetId instead of NodeHash. This diff cleans up
ancestors revset
Reviewed By: farnz
Differential Revision: D10032436
fbshipit-source-id: 2c7d170738826154e3b606e9e29a739a34b1840e
Summary:
From non-master region fetching many changeset entries can be slow. Bumping the
buffer size helped a lot. It shouldn't have negative impact on memory usage
because changeset objects are fairly small.
Reviewed By: farnz
Differential Revision: D10026544
fbshipit-source-id: fa015660a0f7839e73b5aff7d143e85f274dd1b0
Summary:
This diff adds a real implementation for CachingChangesetFetcher. Now it
fetches the data for the cache from the blobstore.
The rest is explained in the comments.
Reviewed By: farnz
Differential Revision: D9908320
fbshipit-source-id: 5427f3ed312cb7753434161423cb27b48744347f
Summary:
Initial implementation of ChangesetsFetcher that will use cache smarter.
At the moment it doesn't do anything special, but in the next diffs it will pre
warm cache in case it has a lot of cache misses (that's why it has to have a
reference to the cachelib CachePool).
Reviewed By: farnz
Differential Revision: D9908319
fbshipit-source-id: 6377a947696bae6b060de5a441722c28309b341c
Summary:
This diff enables access to file content via contains_string() and len() (same as in file hooks) form inside changeset hooks.
This is necessary as some changeset hooks need access to file content and length, e.g. to compute total changeset size.
Reviewed By: StanislavGlebik
Differential Revision: D9788596
fbshipit-source-id: da7bafe6f6fa17a1f25b42550d0bb1a5d871579e
Summary:
High-level goal: we want to make certain big getbundle requests faster. To do
that we'd store blobs of commits that are close to each other in the blobstore
and fetch them only if we had too many cache misses. All this logic will be
hidden in ChangesetFetcher trait implementation. ChangesetFetcher will be
created per request (hence the factory).
Reviewed By: farnz
Differential Revision: D9869659
fbshipit-source-id: 9e3ace3188b3c13f83ef1bd61b668d4f22103f74
Summary:
POST request mononoke_api/objects/batch from hg client.
According to git-lfs protocol
https://github.com/git-lfs/git-lfs/tree/master/docs/apihttps://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
In order to get url for uploading/downloading files, hg client is sending POST request mononoke_api/objects/batch of the following format.
This diff implements support for this POST request.
As an answer it returns json in the format required in git-lfs protocol (see link for more info).
Reviewed By: StanislavGlebik
Differential Revision: D9966691
fbshipit-source-id: 53bcbb4b455e61d9d344bfd9b5b6fb00bc201084
Summary:
This is mainly generated by
```
fbgs mononoke_test_2 -l -s | xargs sed -i 's/mononoke_test_2/mononoke_production/g'
```
mononoke config repo is still pending todo. But it's ok to do it in several run
as right now 2 names are pointing to the same shard.
Reviewed By: StanislavGlebik
Differential Revision: D9939694
fbshipit-source-id: ded772037844a220b18d99b207c976b88dafdaa5
