Summary:
As part of our read path rollout, we want to block user error from
creating new commits that will confuse blobimport. Make it possible to
configure a read-only repo
Reviewed By: StanislavGlebik
Differential Revision: D12945024
fbshipit-source-id: 4265bf57f8adac7965117b710b8285bac483b8ee
Summary:
Purpose:
- Sha256 alias link to file_content is a required part of LFS getfiles works correct.
LFS protocol uses SHA-256 to refer to the file content. Mononoke uses Blake2.
To support LFS in Mononoke we need to set up a link from SHA-256 hash of the content to blake2 of the content.
These links are called aliases.
- Aliases are uploading together with file content blobs.
But only for new push operations.
- If repo is blobimported from somewhere, we need to make sure, that all the links are in blobstore.
If repo was blobimported before aliases were added then it may miss aliases for some blobs.
- This tool can be used to
- find if any aliases are missing
- fill missing aliases.
Implementation:
- Run in repo.
- Iterate through all changesets.
- Go through all the file_content blobs in the changesets
- Verify/generate alias256 links to file_content blobs.
Mode supported:
- verify, count the number of errors and print to console
- generate, if blob is missing to add it to the blobstore
Reviewed By: StanislavGlebik
Differential Revision: D10461827
fbshipit-source-id: c2673c139e2f2991081c4024db7b85953d2c5e35
Summary:
Added a get_stats() Hashmap<String, Box<Any>> method for all ChangesetFetchers.
The CachingChangesetFetcher now returns statistics for cache.misses: usize, cache.hits: usize, fetches.from.blobstore: usize and max.latency: Duration.
Reviewed By: StanislavGlebik
Differential Revision: D10852637
fbshipit-source-id: 34114fd94c47aa26ea525fcc4ff76ad60827bc71
Summary: Now that Tupperware is using `/health_check`, there are no users of `/status`, so remove this endpoint.
Reviewed By: StanislavGlebik
Differential Revision: D12908785
fbshipit-source-id: be9bfae9453143a6f4b26f7fc6cbc68a3f1adc5c
Summary:
Mononoke-apiserver now will parse http header in order to get "host" url.
Based on that Host url, batch request will return the link to the same host.
Requirement:
LFS protocol consists of two parts:
- "batch" request, GET request
- HG client sends ids of the file it wants to download/upload
- API server should return valid url for downloading and uploading objects
"upload/download" request.
- Actual upload/download of the file
This diff allows API Server to return the link to itself on "batch" request. So Hg client will go to API Server again for uploading and downloading files.
Reviewed By: kulshrax
Differential Revision: D12941635
fbshipit-source-id: e56453b0d6239daa3848c285a1df09a4a869f2c8
Summary:
rustc-1.30.0 tries to work out what the linker actually is from the name, and
gets confused because the script we pass it is actually gcc (or clang), but has
`ld` in the name. Update the command-line options to also pass
`-Zlinker-flavor=gcc` to override the linker flavor inference.
proc_macros are now stable. The `quote` crate has a dependency on the `proc_macro`
which is now a shared object as part of the rustc distribution, which means that things
using `quote` must have a runtime linkage with the rustc libraries. The Thrift `codegen`
binary is the main thing affected by this, and I've hacked around it by putting an
`LD_LIBRARY_PATH` in the Buck macros. Need to find a nicer fix though.
rustc 1.30 now implements `#[macro_export(local_inner_macros)]`, which has
strict semantics around namespaces and symbol hygiene, which doesn't work with
common/rust/sql's macros. Work around this for now by just exporting
everything.
Reviewed By: kulshrax
Differential Revision: D12811597
fbshipit-source-id: f18d12e6d99994876c29d3a6bd6ae43f1f37dd17
Summary: We need more insight into what's happening inside of actix and tokio in order to debug operational issues with the API server. Unfortunately, this sort of low-level debug logs from libraries are not captured by slog. Since many of the API server's dependencies use `env_logger`, calling `env_logger::init()` in main will allow us to see the debug output from those libraries by specifying the `RUST_LOG` environment variable.
Reviewed By: HarveyHunt
Differential Revision: D12934682
fbshipit-source-id: 417201e509de66c3340c900a4914ea3a03a791fe
Summary:
We'd like to make it so that it's possible to create BlobRepo that can only do
writes in memory. Main motivation for that is to speed up hg changeset
generation from bonsai changeset. We'd like to create all hg changesets
sequentially, but store all the blobs/db entries in memory. Then we can send
blobs to blobstore/rows to database in parallel.
At the moment we only memory-write blobstore, this diff adds the same for
bonsai hg mapping. Note that there is a method that returns all inserts in the
same order as they were made. This is necessary because we need to ensure that
ancestor parents are recorded before descendatns.
Reviewed By: farnz
Differential Revision: D10427432
fbshipit-source-id: a231c5b04734e6efeec207820d2a5b2dbef02d98
Summary:
We are a tree-only server; advertising this in our caps should ensure
that when we're source of truth at Facebook, anyone with pre-treemanifest
versions of Mercurial gets an insta-fail on connect, rather than a weird error
when they start exchanging data with us.
Reviewed By: HarveyHunt
Differential Revision: D12927232
fbshipit-source-id: 52c15c8a0f1842b6ca023f97228277d0fd9e8e38
Summary:
Quite a big hook that checks whether a commit is reviewed.
What makes it big are exceptions. They should all be covered with tests
Reviewed By: farnz
Differential Revision: D10467787
fbshipit-source-id: b9f439d540b64a5df8c8f131b3e7ba2924af4c29
Summary:
In our hooks we may need to get access to test plan, list of reviewers, summary etc.
Lua doesn't have regex, only patterns. So parsing this information in Lua is a bit tricky.
What's even worse is the fact that we can't add any unittests, so debugging and
fixing is harder. Instead let's add a rust function that would do it for us
Reviewed By: farnz
Differential Revision: D10461203
fbshipit-source-id: 328b76171d2c45ceaf5b2a1f5e1d0bd4c326c223
Summary: This will help traffic replay to distinguish between different repos
Reviewed By: farnz
Differential Revision: D12922287
fbshipit-source-id: 6eed2a0eebceca0636512baa3ee885f5d9c95ccb
Summary:
Sharding filenodes by path should stop us knocking over databases -
make it configurable.
Reviewed By: StanislavGlebik
Differential Revision: D12894523
fbshipit-source-id: e27452f9b436842e1cb5e9e0968c1822f422b4c9
Summary:
We can already flatten a single XDB server with filenodes traffic, and
do if we start up a server instance without a warm memcache. This is only going
to get worse in the future.
Start the process of sharding across multiple servers. For now, we can only
deal with shard size == 1, but this code should be ready to handle shard sizes
greater than 1
Reviewed By: StanislavGlebik
Differential Revision: D12888927
fbshipit-source-id: 8e01694357c390837487fdb3710685fd09feaec0
Summary:
Panic is useless here. It produces huge stack trace which just contains the
main function and makes it harder to debug the actual problem.
Let's just exit in case of errors.
Reviewed By: farnz
Differential Revision: D12912198
fbshipit-source-id: 1faeacfb96765ce047a801f6b072112f10b50b7b
Summary:
This augments `/tree` to yields the size and content sha1 hash for the entries.
This is important for Eden and avoids additional round trips to the server.
The content hashing is the portion that I expect some push back on,
because it doesn't appear to be cached today and the implementation
here does a simplistic fetch and hash. By doing this we hope to
squash out a potential later fetch of the entire contents when
buck build is run.
Differential Revision: D10865588
fbshipit-source-id: c020ef07b99d8a5e8b2f8f7b699bf15e750d60a5
Summary:
This diffs add a signal handler for SIGTERM signal. When it's received then a
terminate process flag is set to true. When this flag is set then no new client
connections will be accepted, and server waits until open connections finish.
The connections can take a long time, so ideally an external process should
sent SIGKILL after a timeout.
Note that this change also makes thrift server thread detached. The reason is
because making it gracefully stop is non-trivial, so for making it detached
should be fine
Reviewed By: farnz
Differential Revision: D12857453
fbshipit-source-id: 6a8f890ff529d74c21fc0c62e16951dd95a3f101
Summary:
According to [blobimport logic](diffusion/FBS/browse/master/fbcode/scm/mononoke/cmdlib/src/blobimport_lib/changeset.rs;fd16808edd6e51c1d0b82f4812fe843e797025e0$163-164) blobimport requests
for parents and node content
Previous implementation was reconstructing filenode to
current revidx for both cases: for getting raw_content and for getting parents.
New implementation avoid reconstruction of the file content to retrieve parents.
Reviewed By: quark-zju
Differential Revision: D12857440
fbshipit-source-id: e1118affe85647931dd551b9ca7be5297afe56ce
Summary:
Add one more restriction to the config repo to make sure we don't forget to
move PROD bookmark
Reviewed By: HarveyHunt
Differential Revision: D12857619
fbshipit-source-id: c4b5e65f2d0b437aad77d8ccc4b4971b60020af4
Summary:
Let's have separate config bookmarks for release candidate and prod.
That will let us customize shadow tier behaviour.
This diff also adds checking of config repo consistency. It requires that RC
bookmark is a descendant of a PROD bookmark. This topology makes it easy to see
what are the changes between PROD and RC, and verification prevents divergence of
configs i.e. sutiations when somebody updated a prod config but forget to
rebase rc config.
Reviewed By: HarveyHunt
Differential Revision: D12857131
fbshipit-source-id: b60d8f7af16e3d530e5edeb22145ec0bd473ffe4
Summary:
Let's add an option to validate the getfiles content that we return to users on
some percent of requests.
It'll increase the latency so let's not enabling that by default
Reviewed By: HarveyHunt
Differential Revision: D10558180
fbshipit-source-id: 2d7ec4dfe7b37b7b5541013278006278d1df68fa
Summary: This will enable doing queries like DELETE, UPDATE or REPLACE without listing all possibilites in the macros
Reviewed By: StanislavGlebik
Differential Revision: D10499501
fbshipit-source-id: 3e2ba433722bd34ffb5960840c509dc27cc9eb5d
Summary:
troubleshooting startup problems is overly difficult without
printing more context, so print it.
Reviewed By: Anastasiya-Zhyrkevich
Differential Revision: D12814794
fbshipit-source-id: e815a6a93b4d1d3d03370b158f6fdc93edbc4ef5
Summary:
Recently there was a change in core hg that changed the way we encode filenames - D9967059. However, it wasn't reflected in Mononoke blobimport code, so the job is constantly fails
This diff change the filename encoding process according to Mercurial
Encoding process is in 3 steps:
1. (Capital -> _lowercaseletter) + ( _ -> __).
If new file name is > 255, than go to step 2, otherwise exit
2. (Capital -> Capital) + (_ -> __)
if new filename is > 255, then got to step 3, otherwise exit
3. (Capitals -> Capitals) + (_ -> : )
Reviewed By: StanislavGlebik
Differential Revision: D10851634
fbshipit-source-id: 28b7503b2601729113326a18ede3e93c04572c6d
Summary:
On push from hg client, require sha1 check on any type of the upload
If data is provided, sha1 will be calculated from bytes [sha1 calculation place](https://fburl.com/noa99y37)
If LFSMetaData is provided, sha1 will be calculated by fetching the file from blobstore [fetching place](https://fburl.com/boj4s74f)
Reviewed By: StanislavGlebik
Differential Revision: D10509331
fbshipit-source-id: 216f59541b8adf8ab87026612e735ac1527e7cc2
Summary:
We have a problem with service upgrades/restarts because many servers start
sending too many requests to mysql db.
Let's add a memcache that will prevent that.
Reviewed By: jsgf
Differential Revision: D10488624
fbshipit-source-id: 4575d359bc269e29fe72b47d7f47cda22bf4acd7
Summary:
Display the hash of the commit that didn't pass a hook,
which is a common occurrence in fbsource hooks using $HG_NODE.
I fixed up the tests, but test-hooks.t is broken from
the hg amend/fbamend fallout and also has some other issues. I tried to only add
the changes relevant to this commit.
Reviewed By: StanislavGlebik
Differential Revision: D10466395
fbshipit-source-id: dd1cdc994171a014c3d4806804ace14e85e726d4
Summary:
Background:
According to [git lfs protocol](https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md) HTTP POST "batch" request should return a link to
the look-aside server.
In our case Mononoke API server is a look-aside server, and process both "batch" request and "upload/download" requests.
So it need to return a link to itself.
New approach requests a separate lfs-url for "batch" request.
The previous approach requested attributes --http-host and --http-port to make a link to the instance of API server running.
Reviewed By: StanislavGlebik
Differential Revision: D10488586
fbshipit-source-id: ed9d78ee9bc78bdcec5eea813bd9aaa6e4590a5c
Summary:
For unification with commit cloud vip configuration apiserver should support the same health check api
Request needed for corp2prod
The same as : D10488369
Reviewed By: liubov-dmitrieva
Differential Revision: D10488494
fbshipit-source-id: 50b4024295c596342a8080474383de850bb7754a
