Summary:
There is a very frustrating operation that happens often when working on the
Mononoke code base:
- You want to add a flag
- You want to consume it in the repo somewhere
Unfortunately, when we need to do this, we end up having to thread this from a
million places and parse it out in every single main() we have.
This is a mess, and it results in every single Mononoke binary starting with
heaps of useless boilerplate:
```
let matches = app.get_matches();
let (caching, logger, mut runtime) = matches.init_mononoke(fb)?;
let config_store = args::init_config_store(fb, &logger, &matches)?;
let mysql_options = args::parse_mysql_options(&matches);
let blobstore_options = args::parse_blobstore_options(&matches)?;
let readonly_storage = args::parse_readonly_storage(&matches);
```
So, this diff updates us to just use MononokeEnvironment directly in
RepoFactory, which means none of that has to happen: we can now add a flag,
parse it into MononokeEnvironment, and get going.
While we're at it, we can also remove blobstore options and all that jazz from
MononokeApiEnvironment since now it's there in the underlying RepoFactory.
Reviewed By: HarveyHunt
Differential Revision: D27767700
fbshipit-source-id: e1e359bf403b4d3d7b36e5f670aa1a7dd4f1d209
Summary:
Basically every single Mononoke binary starts with the same preamble:
- Init mononoke
- Init caching
- Init logging
- Init tunables
Some of them forget to do it, some don't, etc. This is a mess.
To make things messier, our initialization consists of a bunch of lazy statics
interacting with each other (init logging & init configerator are kinda
intertwined due to the fact that configerator wants a logger but dynamic
observability wants a logger), and methods you must only call once.
This diff attempts to clean this up by moving all this initialization into the
construction of MononokeMatches. I didn't change all the accessor methods
(though I did update those that would otherwise return things instantiated at
startup).
I'm planning to do a bit more on top of this, as my actual goal here is to make
it easier to thread arguments from MononokeMatches to RepoFactory, and to do so
I'd like to just pass my MononokeEnvironment as an input to RepoFactory.
Reviewed By: HarveyHunt
Differential Revision: D27767698
fbshipit-source-id: 00d66b07b8c69f072b92d3d3919393300dd7a392
Summary:
Initializing twice causes it to fail. Let's not do that, and also let's use
init_mononoke function instead of our adhoc logger and runtime creationg (at
the very least it also initializes tunables and sets correct tokio runtime
parameters).
Also let's add more logging to see the progress of uploading
Reviewed By: ahornby
Differential Revision: D27079673
fbshipit-source-id: 940135a9aed62f7139835b2450a1964b879e814b
Summary:
The way I plan to use new streaming_changelog in prod is by running it
periodically (say, every 15 mins or so). However some repos won't get many
commits in the last 15 mins (in fact, they might get just 1 or 2).
And even for high commit rate repos most of the times the last chunk
will not be a full chunk (i.e. it will be less that --max-data-chunk-size).
If we were just uploading last chunk regardless of its size then the size of
streaming changelog database table would've just keep growing by 1 entry every
15 mins even if it's completely unnecessary. Instead I suggest to add an option
to not upload the last chunk if it's not necessary.
Reviewed By: farnz
Differential Revision: D27045681
fbshipit-source-id: 2d0fed3094944c4ed921f36943b881af394d9c17
Summary:
This command can be used to update already existing streaming changelog.
It takes a newly cloned changelog and updates the new streaming changelog
chunks in the database.
The biggest difference from "create" command is that we first need to figure
out what's already uploaded to streaming changelog. For that two new methods
were added SqlStreamingChunksFetcher.
Reviewed By: farnz
Differential Revision: D27045386
fbshipit-source-id: 36fc9387f621e1ec8ad3eb4fbb767ab431a9d0bb
Summary:
Small refactoring that will be used in the next diff. In the next diff we'll add
"update" command, and this command will specify the chunk number's itself.
So let's move setting chunk numbers from upload_chunks_to_blobstore function
Differential Revision: D27045387
fbshipit-source-id: c5387a60841fe184c6db5edc4812ddd409eb2215
Summary:
Small refactoring that makes a few things easier to do in the later diffs:
1) Adds a verification that checks the data offset
2) We now read the first chunk's offset from revlog, instead of hardcoding it
to 0, 0. This will be useful in "update" commands which needs to skip revlog
entries that already exists in the database
Differential Revision: D27045388
fbshipit-source-id: 4ee80c96d9307c77b1108889e457f10e83c8beb7
Summary: Duplicate name caused getdeps build to fail. This diff fixes it
Reviewed By: krallin
Differential Revision: D27049661
fbshipit-source-id: b23fe52ad89cbe764e656dfe960921ff1ac92b32
Summary:
Our current straming changelog updater logic is written in python, and it has a
few downsides:
1) It writes directly to manifold, which means it bypasses all the multiplexed
blobstore logic...
2) ...more importantly, we can't write to non-manifold blobstores at all.
3) There are no tests for the streaming changelogs
This diff moves the logic of initial creation of streaming changelog entry to
rust, which should fix the issues mentioned above. I want to highligh that
this implementation only works for the initial creation case i.e. when there are no
entries in the database. Next diffs will add incremental updates functionality.
Reviewed By: krallin
Differential Revision: D27008485
fbshipit-source-id: d9583bb1b98e5c4abea11c0a43c42bc673f8ed48
