Summary: This test covers corner case (partially public stacks)
Reviewed By: StanislavGlebik
Differential Revision: D13750852
fbshipit-source-id: cd1a5a84dfb62951cb37f1fbdd6c510d825adb41
Summary:
Mercurial has a hack to determine if a file was renamed. If p1 is None then
copy metadata is checked. Note that this hack is purely to make finding renames
faster and we don't need it in Mononoke. So let's just read copy metadata.
This diff also removes `maybe_copied()` method and unused code like `Symlink`
Reviewed By: farnz
Differential Revision: D12826409
fbshipit-source-id: 53792218cb61fcba96144765790278d17eecdbb1
Summary:
Currently some tests are failing intermittently due to a timeout, this timeout
is caused by the integration tests making an external call to AclChecker which
is intermittently very slow.
For our integration tests we disable this call since it's not part of the test
suite to stop our tests from being flaky.
Reviewed By: StanislavGlebik
Differential Revision: D13580144
fbshipit-source-id: 0c26bb14dd222b888ca2638319071f4d99eab6df
Summary:
We already had a few diffs that added or removed
```
Ignoring setSSLLockTypes after initialization
```
line.
I'm not sure why we have them, but we don't want to see them anyway, so disable
it via minloglevel glog option (level 2 means see only ERROR and FATAL).
Reviewed By: HarveyHunt
Differential Revision: D13416156
fbshipit-source-id: 362153385b77e133e404b21faa1735a9544fe13e
Summary:
We've moved to the new config structure, I forgot to make the same changes to
apiserver
Reviewed By: lukaspiatkowski
Differential Revision: D13358407
fbshipit-source-id: cec81a21518cdb3c91dabb93e220d1ba3e25d02c
Summary: Set the proper context for sessions.
Reviewed By: liubov-dmitrieva
Differential Revision: D13258641
fbshipit-source-id: edd18d4abc8f5475e0d2ac8395dfc877b2dd5958
Summary:
Previously we manually specified blobstore type and all the necessary
parameters. That was error-prone but worked because we had only one blobstore.
Since we are going to add secondary blobstores soon configuring binaries like
blobimport will be harder because we'll need to specify parameters for all
blobstores. Let's make it so that
blobimport, mononoke admin and other binaries read the configuration the same
way as Mononoke does it i.e. via toml files.
Reviewed By: lukaspiatkowski
Differential Revision: D13183244
fbshipit-source-id: 99caa6348133acec11dd04ae44e1f9f0a8ebb197
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary: test-init.t and test-lfs-to-mononoke.t were failing. This diff fixes them
Differential Revision: D13301143
fbshipit-source-id: 1f8060d4c6b641c555ba8a5cdcfe4cb14ac89d0a
Summary:
According to [Git-LFS Plan](https://www.mercurial-scm.org/wiki/LfsPlan), `getfiles` instead of file content should return file in the [following format](https://www.mercurial-scm.org/wiki/LfsPlan#Metadata_format)
```
oid: sha256.SHA256HASH
size: size_int
```
Hg client requests files using sha1 hgfilenode hash. To calculate sha256 of the content, Mononoke is fetching the file from blobstore to memory, and calculate sha256.
It does not give any profit in time and memory consumptions, comparing to non-LFS transfer of Mononoke.
*Solution:*
To put a `key-value` to blobstore, after first request of the file. This means, that after hg client requested sha256 of the file for the first time, after calculation, put it to the blobstore.
Next request of the sha256 of the file content avoid recalcualtion of sha256 in Mononoke. It return sha256 saved in the blob.
Reviewed By: StanislavGlebik
Differential Revision: D13021826
fbshipit-source-id: 692e01e212e7d716bd822fa968e87abed5103aa7
Summary:
The file of some revision is the initial file content with applied deltas
Delta is a vector of Fragments.
Fragment is a sequential change of the file (old part of the content -> new content)
This diff is representing the implementation of optimization of the process of getting a file content of some revision.
Reviewed By: lukaspiatkowski
Differential Revision: D12928138
fbshipit-source-id: fcc28e2d0e0acf83e17887092f6593e155431c1b
Summary:
As part of our read path rollout, we want to block user error from
creating new commits that will confuse blobimport. Make it possible to
configure a read-only repo
Reviewed By: StanislavGlebik
Differential Revision: D12945024
fbshipit-source-id: 4265bf57f8adac7965117b710b8285bac483b8ee
Summary:
Purpose:
- Sha256 alias link to file_content is a required part of LFS getfiles works correct.
LFS protocol uses SHA-256 to refer to the file content. Mononoke uses Blake2.
To support LFS in Mononoke we need to set up a link from SHA-256 hash of the content to blake2 of the content.
These links are called aliases.
- Aliases are uploading together with file content blobs.
But only for new push operations.
- If repo is blobimported from somewhere, we need to make sure, that all the links are in blobstore.
If repo was blobimported before aliases were added then it may miss aliases for some blobs.
- This tool can be used to
- find if any aliases are missing
- fill missing aliases.
Implementation:
- Run in repo.
- Iterate through all changesets.
- Go through all the file_content blobs in the changesets
- Verify/generate alias256 links to file_content blobs.
Mode supported:
- verify, count the number of errors and print to console
- generate, if blob is missing to add it to the blobstore
Reviewed By: StanislavGlebik
Differential Revision: D10461827
fbshipit-source-id: c2673c139e2f2991081c4024db7b85953d2c5e35
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary:
Hooks that makes sure that there are no conflict markers in file contents.
This hook is bypassable.
Reviewed By: purplefox
Differential Revision: D10260230
fbshipit-source-id: b9d69e757f18ed3f4f889a01032ef7360cba6867
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
That's a bit controversial, however I think it's worth it. Many file hooks
should be run only on files that exist in the repo (for example,
https://fburl.com/1cj8wm3p, https://fburl.com/t06fjwak). If you want to do
anything on deleted files then just write a changeset hook.
Reviewed By: jsgf
Differential Revision: D10239186
fbshipit-source-id: 3cb563b81ec51298623cecaf976b5a8fe50dc71c
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary: cachelib takes a lot of times to init, and it's not used in the tests.
Reviewed By: farnz
Differential Revision: D10123692
fbshipit-source-id: bce82af3f56aa8829127bf4c605a1d472735126a
Summary:
It makes it clear why blobimport failed. Before it required modification to the
tests to run `cat FILENAME`, now it's done automatically
Reviewed By: farnz
Differential Revision: D9850279
fbshipit-source-id: b9a252f50b6dcf4906c24aca70fa37a2cec84e46
Summary:
PUT request upload to mononoke API
hg client sends a PUT request to store a file into blobstore during push supporting LFS
Upload file by alias is divied into 2 parts:
- Put alias : blobstore key
- Put blobstore_key: contents
Keep in mind, that file content is thrift encoded
host_address for batch request is from command line flags -H for host, -p for port
Reviewed By: StanislavGlebik
Differential Revision: D10026683
fbshipit-source-id: 6c2726c7fee2fb171582bdcf7ce86b22b0130660
Summary: ugh, yet another case of a hidden dependency.
Reviewed By: StanislavGlebik
Differential Revision: D9318498
fbshipit-source-id: 5fcd25081b5033cbef9c5f137e616348c5d6ced9
Summary:
This commit added three options to specify locations to SSL certificates so the API server will accept encrypted traffic.
Currently this only works in HTTP/1.1 due to a bug in HTTP/2 parsing in actix-web. Once they fixed the bug upstream we will be able to serve HTTP/2 traffic as well.
Reviewed By: jsgf
Differential Revision: D8703861
fbshipit-source-id: 0d4e68276013a8aeb6ee006e5175b8caeba767cb
Summary: This will be useful for the bonsai verification tool.
Reviewed By: StanislavGlebik
Differential Revision: D8792562
fbshipit-source-id: f409d0fa042528b04462a1539fd3c2a8064a4f6e
Summary: - integration test of were failing on my devserver as `{http|https}_proxy` environment variables
Reviewed By: StanislavGlebik
Differential Revision: D8804505
fbshipit-source-id: fafea55ed80897b24ad816bc26b241e64cd8b913
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D8625178
fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
Summary:
They were broken since 2646d41a64917d3e50f662b0b4b628ccfdbb05a8.
It added enabled_repo repository to config, however, no enabled_repo has ever been
blobimported. Since we don't really need enabled_repo, let's remove it
Reviewed By: farnz
Differential Revision: D8712891
fbshipit-source-id: 360f2c9564087d3a8411871335aa40fa81e38c0c
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.
Reviewed By: farnz
Differential Revision: D8647161
fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
Summary:
Use tls for connection between hgcli and Mononoke server always, even for
localhost connections[1]
The setup is similar to tls setup of Eden server.
[1] This is not necessary of course, but adding an option to bypass tls
connection may result in accidental use of it in prod. However if it turns out
to be too unusable, we can add such option in the future
Reviewed By: jsgf
Differential Revision: D8644299
fbshipit-source-id: 0898e30e33b718e13a766763479f3adf9323ffe7
Summary: This commit upgraded openssl, enabled alpn for actix-web and added tokio-codec with fixes due to the upgrade.
Reviewed By: StanislavGlebik
Differential Revision: D8682673
fbshipit-source-id: 8c7cadfd6c0c7b016202f6cb038eb4951d0f9333
Summary:
Previously this code was completely untested. This diff adds a simple test that
checks that warmup has at least finished
Reviewed By: farnz
Differential Revision: D8611295
fbshipit-source-id: bf55d3b8953660040b4f9a934094fb688683c7ba
Summary: This commit lets apiserver reads Mononoke's config repository and creates MononokeRepo based on it.
Reviewed By: kulshrax
Differential Revision: D8465574
fbshipit-source-id: 0af1ad5f62499f83261e21bac605725156fc22d0
Summary: This revision adds an initial binary target for Mononoke API Server as well as a few basic options and logging.
Reviewed By: kulshrax
Differential Revision: D8413745
fbshipit-source-id: 65523433284e970348efcafd724ae28102d85671
Summary: Unfortunately even 10 seems to be too little for stress runs.
Reviewed By: StanislavGlebik
Differential Revision: D8310639
fbshipit-source-id: 280824a7c2ec2283d2ef3acfb968e336ca13c93b
Summary: This allows us to put hgcli on the different host
Reviewed By: jsgf
Differential Revision: D7910980
fbshipit-source-id: 45bdd1ee42b54f8f1a425ed6950e785cdf85bfc3
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host,
because unix domain socket are used. Also Mononoke has to use separate unix
domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
This diff adds a separate thread that parses the Preamble, extracts the
reponame and sends request to a thread responsible for this repo.
Also hgcli now has a new cmd line option that specifies path to connect to.
Reviewed By: jsgf
Differential Revision: D7845156
fbshipit-source-id: a48bcfeec4755b2d5b3dfcf8e0383e6945178018
Summary: new blobimport has taken over integration tests
Reviewed By: StanislavGlebik
Differential Revision: D7710186
fbshipit-source-id: 7d61f3239d8edbb1f458dfde0abf59914a912aeb
Summary:
5 seconds appears to still not be enough for stress test runs,
unfortunately. Try bumping it up a bit more.
Reviewed By: StanislavGlebik
Differential Revision: D7663231
fbshipit-source-id: 8dc82afd64185c12fd98138b96b981b7f0e9173e
