Summary: This is needed to run the scmadmin tool and lock the repo.
Reviewed By: StanislavGlebik
Differential Revision: D14497608
fbshipit-source-id: 5865b90375db29a17d462044ca4cdb87242a8209
Summary:
Previously we always restart replaying from the beginning. starting from this
diff we actually record the latest replayed id in the mutable_counters table.
Special flag `--loop-forever` was added. It should be set in production, but for tests we don't want a binary to run forever.
Reviewed By: ikostia
Differential Revision: D14226938
fbshipit-source-id: cbfde15a506ee94b0fa72015d9dcfd550f5b8ca3
Summary:
Small refactoring before the next diffs.
Before
mononoke_hg_sync ... --mode sync-once ssh://PATH --start-id 0
Now
mononoke_hg_sync ... ssh://PATH sync-once --start-id 0
Reviewed By: HarveyHunt
Differential Revision: D14226940
fbshipit-source-id: 1b6db6f194aecb2f4bdb7bbd9e846aaba180e098
Summary:
Pretty big cleanup. The biggest part is simplifying comparing logic.
Pushrebase replay compared replayed pushrebase commits with their hg
counterparts. Because previously we didn't log enough data from hg servers we
had to jump through a lot of hoops to find out which commits should be compared
to which.
Now we are logging `ordered_added_revs` i.e. which commits mercurial produced
after the hg pushrebase, which is exactly what we need for comparison. So a lot
of code can be simplified.
That does mean though that some of the recorded pushrebases we won't be able to
replay because they don't have `ordered_added_revs`. At the moment we have
~300K pushrebases without `ordered_added_revs` and ~100K with. I think this
change is worth it given how simpler the code is and I'd argue 100K is a pretty
big number. For the rest 300K pushrebases we can later manually fill in
`ordered_added_revs` field if we consider it's necessary.
After this diff pushrebase replay won't use hg repo, and so we can run it on
normal twshared jobs instead of on hg servers.
A few smaller changes:
1) Also note that for now this diff makes pushrebase replay single threaded, while
previously we ran comparison in parallel. That will be fixed in the next diffs.
2) Pushrebase replay now always compare commits i.e. `--compare-commits` option
was removed
Also change repo id in tw spec to 20
Reviewed By: HarveyHunt
Differential Revision: D14122963
fbshipit-source-id: 0f8da7cffb13899f11143a01d1a301fdf8ea7f00
Summary:
First stab at a job that will keep hg in sync with Mononoke when Mononoke
becomes a source of truth.
Reviewed By: ikostia
Differential Revision: D14018269
fbshipit-source-id: f88c5eba8bf5482f2f162b7807ca8e41a3b4291d
Summary: This is adds a metaconfig option to preserve push/pushrebase bundles in the blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D14020299
fbshipit-source-id: 94304d69e0ac5d81232f058c6d94eec61eb0020a
Summary: Passing input on command line is restricted to the maximum length of a command, it is much better to pass the file descriptor, this way one can pass very long inputs inside the bash functions.
Reviewed By: StanislavGlebik
Differential Revision: D13964484
fbshipit-source-id: da010ecfcf05da8c5860c8b5ee0860a8aeda0502
Summary:
There were lots of small stupid mistakes with pushrebase replay in past.
Since pushrebase replay will stay with us for quite some time,
let's add a test to prevent these regressions in future.
In tests we'll use filesystem+sqlite instead of everstore + xdb to fetch
requests to replay.
Reviewed By: HarveyHunt
Differential Revision: D13940390
fbshipit-source-id: a4398d1c8c22bf16a85d6391a1f5665ce4b73eb1
Summary:
I already made them the same (but copy) in a different diff. As we discussed at
war room, symlinks is a better solution.
Also removed unused import and some functions that were copied from mercurial.
This is no longer needed as we now share tinit.sh with general purpose
functions.
The only mononoke specific file left is dummyssh.py
I think later we could pack them in a separate buck target or something and share in a better way that symlinks but for now it is the easiest solution.
Reviewed By: DurhamG
Differential Revision: D13881960
fbshipit-source-id: 36f425d6f0ddbae2c9d083de35d2779669dc01e7
Summary: This test covers corner case (partially public stacks)
Reviewed By: StanislavGlebik
Differential Revision: D13750852
fbshipit-source-id: cd1a5a84dfb62951cb37f1fbdd6c510d825adb41
Summary:
Mercurial has a hack to determine if a file was renamed. If p1 is None then
copy metadata is checked. Note that this hack is purely to make finding renames
faster and we don't need it in Mononoke. So let's just read copy metadata.
This diff also removes `maybe_copied()` method and unused code like `Symlink`
Reviewed By: farnz
Differential Revision: D12826409
fbshipit-source-id: 53792218cb61fcba96144765790278d17eecdbb1
Summary:
Currently some tests are failing intermittently due to a timeout, this timeout
is caused by the integration tests making an external call to AclChecker which
is intermittently very slow.
For our integration tests we disable this call since it's not part of the test
suite to stop our tests from being flaky.
Reviewed By: StanislavGlebik
Differential Revision: D13580144
fbshipit-source-id: 0c26bb14dd222b888ca2638319071f4d99eab6df
Summary:
We already had a few diffs that added or removed
```
Ignoring setSSLLockTypes after initialization
```
line.
I'm not sure why we have them, but we don't want to see them anyway, so disable
it via minloglevel glog option (level 2 means see only ERROR and FATAL).
Reviewed By: HarveyHunt
Differential Revision: D13416156
fbshipit-source-id: 362153385b77e133e404b21faa1735a9544fe13e
Summary:
We've moved to the new config structure, I forgot to make the same changes to
apiserver
Reviewed By: lukaspiatkowski
Differential Revision: D13358407
fbshipit-source-id: cec81a21518cdb3c91dabb93e220d1ba3e25d02c
Summary: Set the proper context for sessions.
Reviewed By: liubov-dmitrieva
Differential Revision: D13258641
fbshipit-source-id: edd18d4abc8f5475e0d2ac8395dfc877b2dd5958
Summary:
Previously we manually specified blobstore type and all the necessary
parameters. That was error-prone but worked because we had only one blobstore.
Since we are going to add secondary blobstores soon configuring binaries like
blobimport will be harder because we'll need to specify parameters for all
blobstores. Let's make it so that
blobimport, mononoke admin and other binaries read the configuration the same
way as Mononoke does it i.e. via toml files.
Reviewed By: lukaspiatkowski
Differential Revision: D13183244
fbshipit-source-id: 99caa6348133acec11dd04ae44e1f9f0a8ebb197
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary: test-init.t and test-lfs-to-mononoke.t were failing. This diff fixes them
Differential Revision: D13301143
fbshipit-source-id: 1f8060d4c6b641c555ba8a5cdcfe4cb14ac89d0a
Summary:
According to [Git-LFS Plan](https://www.mercurial-scm.org/wiki/LfsPlan), `getfiles` instead of file content should return file in the [following format](https://www.mercurial-scm.org/wiki/LfsPlan#Metadata_format)
```
oid: sha256.SHA256HASH
size: size_int
```
Hg client requests files using sha1 hgfilenode hash. To calculate sha256 of the content, Mononoke is fetching the file from blobstore to memory, and calculate sha256.
It does not give any profit in time and memory consumptions, comparing to non-LFS transfer of Mononoke.
*Solution:*
To put a `key-value` to blobstore, after first request of the file. This means, that after hg client requested sha256 of the file for the first time, after calculation, put it to the blobstore.
Next request of the sha256 of the file content avoid recalcualtion of sha256 in Mononoke. It return sha256 saved in the blob.
Reviewed By: StanislavGlebik
Differential Revision: D13021826
fbshipit-source-id: 692e01e212e7d716bd822fa968e87abed5103aa7
Summary:
The file of some revision is the initial file content with applied deltas
Delta is a vector of Fragments.
Fragment is a sequential change of the file (old part of the content -> new content)
This diff is representing the implementation of optimization of the process of getting a file content of some revision.
Reviewed By: lukaspiatkowski
Differential Revision: D12928138
fbshipit-source-id: fcc28e2d0e0acf83e17887092f6593e155431c1b
Summary:
As part of our read path rollout, we want to block user error from
creating new commits that will confuse blobimport. Make it possible to
configure a read-only repo
Reviewed By: StanislavGlebik
Differential Revision: D12945024
fbshipit-source-id: 4265bf57f8adac7965117b710b8285bac483b8ee
Summary:
Purpose:
- Sha256 alias link to file_content is a required part of LFS getfiles works correct.
LFS protocol uses SHA-256 to refer to the file content. Mononoke uses Blake2.
To support LFS in Mononoke we need to set up a link from SHA-256 hash of the content to blake2 of the content.
These links are called aliases.
- Aliases are uploading together with file content blobs.
But only for new push operations.
- If repo is blobimported from somewhere, we need to make sure, that all the links are in blobstore.
If repo was blobimported before aliases were added then it may miss aliases for some blobs.
- This tool can be used to
- find if any aliases are missing
- fill missing aliases.
Implementation:
- Run in repo.
- Iterate through all changesets.
- Go through all the file_content blobs in the changesets
- Verify/generate alias256 links to file_content blobs.
Mode supported:
- verify, count the number of errors and print to console
- generate, if blob is missing to add it to the blobstore
Reviewed By: StanislavGlebik
Differential Revision: D10461827
fbshipit-source-id: c2673c139e2f2991081c4024db7b85953d2c5e35
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary:
Hooks that makes sure that there are no conflict markers in file contents.
This hook is bypassable.
Reviewed By: purplefox
Differential Revision: D10260230
fbshipit-source-id: b9d69e757f18ed3f4f889a01032ef7360cba6867
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary: cachelib takes a lot of times to init, and it's not used in the tests.
Reviewed By: farnz
Differential Revision: D10123692
fbshipit-source-id: bce82af3f56aa8829127bf4c605a1d472735126a
Summary:
It makes it clear why blobimport failed. Before it required modification to the
tests to run `cat FILENAME`, now it's done automatically
Reviewed By: farnz
Differential Revision: D9850279
fbshipit-source-id: b9a252f50b6dcf4906c24aca70fa37a2cec84e46
Summary:
PUT request upload to mononoke API
hg client sends a PUT request to store a file into blobstore during push supporting LFS
Upload file by alias is divied into 2 parts:
- Put alias : blobstore key
- Put blobstore_key: contents
Keep in mind, that file content is thrift encoded
host_address for batch request is from command line flags -H for host, -p for port
Reviewed By: StanislavGlebik
Differential Revision: D10026683
fbshipit-source-id: 6c2726c7fee2fb171582bdcf7ce86b22b0130660
Summary: ugh, yet another case of a hidden dependency.
Reviewed By: StanislavGlebik
Differential Revision: D9318498
fbshipit-source-id: 5fcd25081b5033cbef9c5f137e616348c5d6ced9
Summary:
This commit added three options to specify locations to SSL certificates so the API server will accept encrypted traffic.
Currently this only works in HTTP/1.1 due to a bug in HTTP/2 parsing in actix-web. Once they fixed the bug upstream we will be able to serve HTTP/2 traffic as well.
Reviewed By: jsgf
Differential Revision: D8703861
fbshipit-source-id: 0d4e68276013a8aeb6ee006e5175b8caeba767cb
Summary: This will be useful for the bonsai verification tool.
Reviewed By: StanislavGlebik
Differential Revision: D8792562
fbshipit-source-id: f409d0fa042528b04462a1539fd3c2a8064a4f6e
Summary: - integration test of were failing on my devserver as `{http|https}_proxy` environment variables
Reviewed By: StanislavGlebik
Differential Revision: D8804505
fbshipit-source-id: fafea55ed80897b24ad816bc26b241e64cd8b913
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D8625178
fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
Summary:
They were broken since 2646d41a64917d3e50f662b0b4b628ccfdbb05a8.
It added enabled_repo repository to config, however, no enabled_repo has ever been
blobimported. Since we don't really need enabled_repo, let's remove it
Reviewed By: farnz
Differential Revision: D8712891
fbshipit-source-id: 360f2c9564087d3a8411871335aa40fa81e38c0c
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.
Reviewed By: farnz
Differential Revision: D8647161
fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
Summary:
Use tls for connection between hgcli and Mononoke server always, even for
localhost connections[1]
The setup is similar to tls setup of Eden server.
[1] This is not necessary of course, but adding an option to bypass tls
connection may result in accidental use of it in prod. However if it turns out
to be too unusable, we can add such option in the future
Reviewed By: jsgf
Differential Revision: D8644299
fbshipit-source-id: 0898e30e33b718e13a766763479f3adf9323ffe7
