Summary: In this diff the configs are parsed from toml and passed around to hook's execution context. The actual usage of configs will be introduced in separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D13862837
fbshipit-source-id: 60ac10aa9c25d224e703e1e55bef13dc481ba07e
Summary:
Previously pushrebasing an empty commit failed because we assumed that root
manifest of a commit is always sent in a bundle. This diff removes this
assumption
Reviewed By: lukaspiatkowski
Differential Revision: D13818556
fbshipit-source-id: 44e96374ae343074f48e42a90c691b21e3c41386
Summary:
Previously we rely on CachingChangesetFetcher to quickly fetch all commits into
memory, but CachingChangesetFetcher was deleted in D13695201. Instead let's use
`get_many()` method from Changesets trait to quickly fetch many changesets at
once into memory.
Reviewed By: lukaspiatkowski
Differential Revision: D13712783
fbshipit-source-id: 12e8fa148f7989028547ac8d374438e23b44b6d1
Summary:
We decided to populate phases table in 2 places: blobimport and push-rebase
(alredy done).
This diff is for blobimport. We know the commits are public.
Reviewed By: lukaspiatkowski
Differential Revision: D13731900
fbshipit-source-id: b64e5643e7cffd9e8fb842e9929f4c1ee7a66197
Summary:
This version still misses:
- proper production-ready logging
- smarter handling of case where the queue entries related to each other do not fit in the limit or `older_than` limit, so the healer will heal much more entries without realizing it shouldn't do so.
Reviewed By: aslpavel
Differential Revision: D13528686
fbshipit-source-id: 0245becea7e4f0ac69383a7885ff3746d81c4add
Summary: Instead of dumping the debug output we print the most important information: Changeset id, author, message and file changes.
Reviewed By: StanislavGlebik
Differential Revision: D13621492
fbshipit-source-id: ea0f93f58516cc759d0dc9aac14545b1827ea136
Summary: Format files effected by next commit in a stack
Reviewed By: StanislavGlebik
Differential Revision: D13650639
fbshipit-source-id: d4e37acd2bcd29b291968a529543c202f6944e1a
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
Now that mononoke's config can be built using normal fbpkg tools,
we can remove the ability for mononoke_admin to build configuration.
Reviewed By: StanislavGlebik
Differential Revision: D13399189
fbshipit-source-id: 17aff327773b9c904916f99030a732a99aa34134
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary:
Let's add a command that builds and reads a skiplist indexes. This indexes will
be used by getbundle wireproto request to decrease the latency and cpu usage.
Note that we are saving only the longest "jump" from the skiplist. This is done
in order to save space.
Reviewed By: jsgf
Differential Revision: D13169018
fbshipit-source-id: 4d654284b0c0d8a579444816781419ba6ad86baa
Summary:
Purpose:
- Sha256 alias link to file_content is a required part of LFS getfiles works correct.
LFS protocol uses SHA-256 to refer to the file content. Mononoke uses Blake2.
To support LFS in Mononoke we need to set up a link from SHA-256 hash of the content to blake2 of the content.
These links are called aliases.
- Aliases are uploading together with file content blobs.
But only for new push operations.
- If repo is blobimported from somewhere, we need to make sure, that all the links are in blobstore.
If repo was blobimported before aliases were added then it may miss aliases for some blobs.
- This tool can be used to
- find if any aliases are missing
- fill missing aliases.
Implementation:
- Run in repo.
- Iterate through all changesets.
- Go through all the file_content blobs in the changesets
- Verify/generate alias256 links to file_content blobs.
Mode supported:
- verify, count the number of errors and print to console
- generate, if blob is missing to add it to the blobstore
Reviewed By: StanislavGlebik
Differential Revision: D10461827
fbshipit-source-id: c2673c139e2f2991081c4024db7b85953d2c5e35
Summary:
Sharding filenodes by path should stop us knocking over databases -
make it configurable.
Reviewed By: StanislavGlebik
Differential Revision: D12894523
fbshipit-source-id: e27452f9b436842e1cb5e9e0968c1822f422b4c9
Summary:
Panic is useless here. It produces huge stack trace which just contains the
main function and makes it harder to debug the actual problem.
Let's just exit in case of errors.
Reviewed By: farnz
Differential Revision: D12912198
fbshipit-source-id: 1faeacfb96765ce047a801f6b072112f10b50b7b
Summary:
Add one more restriction to the config repo to make sure we don't forget to
move PROD bookmark
Reviewed By: HarveyHunt
Differential Revision: D12857619
fbshipit-source-id: c4b5e65f2d0b437aad77d8ccc4b4971b60020af4
Summary:
Let's have separate config bookmarks for release candidate and prod.
That will let us customize shadow tier behaviour.
This diff also adds checking of config repo consistency. It requires that RC
bookmark is a descendant of a PROD bookmark. This topology makes it easy to see
what are the changes between PROD and RC, and verification prevents divergence of
configs i.e. sutiations when somebody updated a prod config but forget to
rebase rc config.
Reviewed By: HarveyHunt
Differential Revision: D12857131
fbshipit-source-id: b60d8f7af16e3d530e5edeb22145ec0bd473ffe4
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary:
One brainless idiot decided to prune all trees from changed files calcualation.
Since it also prunes subtrees, that leaves with just files in the root
directory.
Reviewed By: lukaspiatkowski
Differential Revision: D10302299
fbshipit-source-id: 8fe2c4ad8de998dfd4083d97cd816d85b5fec604
Summary: Pushvars is a one more way to bypass hooks. This diff implements it
Reviewed By: purplefox
Differential Revision: D10257602
fbshipit-source-id: 1bd188239878ff917ded7db995ea2453da9f64c4
Summary:
Let's add a logic to allow users to bypass hooks.
We'll have two ways to bypass hooks. One is via a string in commit message,
another is via pushvars.
This diff implements the first one.
Reviewed By: purplefox
Differential Revision: D10255378
fbshipit-source-id: 31e803a58e2f4798294f7c807933c8e26de3cfaf
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
Hooks need to know whether file was added, modified or removed. For example, we
can't fetch content of a removed file. Also hook authors may want to allow
modifying existing files of a particular type, but they may want to disallow
addition of new files of this type.
`cs.files()` doesn't give information about whether a file was
added/deleted/modified, so we have to use `get_changed_stream` function from
manifest_utils.
Note - currently it still returns incorrect list of changed files for merges.
It will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D10237587
fbshipit-source-id: cd7f76334070cde451b4690071d03275e40c95f3
Summary: This adds `--compare-commits` option for pushrebase replayer. Which also check that commits are close enough to treat them as equal for the purpose of pushrebase.
Reviewed By: StanislavGlebik
Differential Revision: D10084308
fbshipit-source-id: f1fd05173a9a7663125a89dd03b79b2deea40dc4
Summary: This popped up while I was building Mononoke - fix it.
Reviewed By: StanislavGlebik
Differential Revision: D10126386
fbshipit-source-id: 117239dea88c3ecd921f852ce86691ba6aa8bb07
Summary:
Previously cachelib cmdline args were added only to cmd line binaries, but not
to Mononoke this diff fixes it.
Reviewed By: farnz
Differential Revision: D10083899
fbshipit-source-id: 8febba96561c5ab9a61f60fafc7a7e56985dc038
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
This is mainly generated by
```
fbgs mononoke_test_2 -l -s | xargs sed -i 's/mononoke_test_2/mononoke_production/g'
```
mononoke config repo is still pending todo. But it's ok to do it in several run
as right now 2 names are pointing to the same shard.
Reviewed By: StanislavGlebik
Differential Revision: D9939694
fbshipit-source-id: ded772037844a220b18d99b207c976b88dafdaa5
