Summary:
The config will be used to whitelist connections with certain identities and
blacklist everything else.
Differential Revision: D15150921
fbshipit-source-id: e4090072ea6ba9714575fb8104d9f45e92c6fefb
Summary: Per title, `hg debughttp` now prints out the hostname that the API server reports rather than the hostname in the URL we used to connect to it. The reason for this is that if the API server is behind a VIP, we get the actual hostname rather than just the VIP URL.
Reviewed By: xavierd
Differential Revision: D15170618
fbshipit-source-id: 9af5480f9987d8ea9c914baf3b62a00ad88d1b32
Summary:
Given that users typically connect to the API server via a VIP, it is
often unclear which actual host we're connected to. As such, API server
responses should contain the actual hostname of the server.
As a first pass at adding this, let's add a simple endpoint that just returns
the hostname, which can be used to see which hosts the VIP is sending requests
to.
Reviewed By: xavierd
Differential Revision: D15168952
fbshipit-source-id: d6fee5716df975fa1588bbf069ad6e970a779b1e
Summary:
Disallow unknown fields. They're generally the result of a mis-editing
a file and putting the config in the wrong place, or some incomplete refactor.
Reviewed By: StanislavGlebik
Differential Revision: D15168963
fbshipit-source-id: a9c9658378cda4866e44daf6e2c6bfbdfcdb9f84
Summary:
Currently this checks for:
- all referenced files are present
- they're well-formed toml
- all the required config keys are set, and no unknown ones are set
- blob storage and metadata db are both remote or local
- repo ids are not duplicated
Reviewed By: farnz
Differential Revision: D15068649
fbshipit-source-id: ace0e7bc52bf853ac42384c4346c3b73591312e4
Summary: Seems like 15 mins is not enough, let's bump it
Reviewed By: farnz
Differential Revision: D15154597
fbshipit-source-id: 78b8a43bbc95845719245f71ac85fd22336f3ed6
Summary: The SHA1 cache mistakenly caching only by the node hash while it is possible to have multiple files that have the same node hash while different names. This will cause the server incorrectly returning incorrect entry from the cache.
Reviewed By: StanislavGlebik
Differential Revision: D15092963
fbshipit-source-id: 608be9f2b749d0b77687ad5cf2d76d56398a543d
Summary:
At the moment verify integrity script blocks master commits that do not pass
the check. Non-master commits are not blocked even if they don't pass the check
(for example, if they don't have a valid reviewer), but verify_integrity hook
still logs them to scuba.
Mononoke's verify_integrity was enabled only on master, meaning that the hook
won't run on non-master commits at all, and we won't log non-master commits to scuba.
This diff fixes it.
Reviewed By: farnz
Differential Revision: D15146725
fbshipit-source-id: ab432fb2eccae0fbcc10755f5c8447964c490285
Summary:
This introduces prefetching in the hg sync job, which allows us to separate two logically separate steps, and buffer between them:
- Fetching bundle data from Manifold so it can be replayed in Mercurial (we call this "preparing"). This can happen out of order.
- Submitting the changes to Mercurial. This must happen sequentially.
The upshot is that we can fetch new data from Manifold while we wait for Mercurial to apply it, which should increase our throughput.
---
This required a bit of refactoring for the main loop here. The reason for that is that, currently, we heavily rely on closures to capture the `log_entry` so we can pass it to Scuba, etc.
That doesn't work if we want to have buffering, because there isn't a single scope that represents
Instead, every step of the stream has its own scope, so we need to pass the `log_entry` through the stream itself. We also need to track the log entry in errors as well (e.g. to report them to Scuba).
Reviewed By: ikostia
Differential Revision: D15146682
fbshipit-source-id: f83284571f6ca90cb621c621c5165f7fbafd81b5
Summary:
This separates out the bundle processing into two separate steps: preparing (fetching the data) and syncing (sending it to Mercurial).
The underlying goal here is to extract the 1st part into a buffered stream. Doing so will require a little further rework because we need to attach error handling to the stream (in loop mode) or future (in sync-one mode), which requires further refactoring. I'll do so in a separate diff.
Reviewed By: ikostia
Differential Revision: D15121738
fbshipit-source-id: e88d6d730d413e9c1939eefa72ec18799c8b13be
Summary:
One of the tests for mononoke admin (which I introduced recently) appears to be flaky. Sometimes, output from stdout (the error that terminates the program) and stderr (logs emitted while the program runs) is flipped.
I suspect this is caused by buffering, so this patch flushes all output before writing the error (if there was one).
An alternative approach here might be to write the final error to stderr instead of stdosut (so everything goes to stderr). That feels cleaner, but it does change the interface a little bit, so I didn't take that approach just yet. That said, if nobody objects, I'm happy to pick that approach instead.
Reviewed By: farnz
Differential Revision: D15146392
fbshipit-source-id: 67481afd4802cb48d24d19052988be4a83433efd
Summary:
I introduced several new methods on the Bookmarks trait in:
- D15097549
- D15081759
- D15097935
At the same time, Pavel introduced a new implementation of that trait in CachedBookmarks:
- D14928419
That broke the build, this fixes it.
Reviewed By: farnz
Differential Revision: D15124562
fbshipit-source-id: 61e49261c3dc5bb396883e69c29afa329d5df9ab
Summary: Enabling it was a mistake as we can't enable it on our real hg servers.
Reviewed By: farnz
Differential Revision: D15122167
fbshipit-source-id: 5e28968354b7487bd9c965d8b3f23245f6bd265a
Summary:
This is a tentative implementation of batching for the sync job.
It works by running a peer once per batch and just sending new bundle
metadata on stdin. When the batch capacity fills up, we kill the peer
and run a new one.
Reviewed By: StanislavGlebik
Differential Revision: D15063281
fbshipit-source-id: b681594c023973a165047198d490207e35a354b8
Summary:
We might use it in the sync job to make sync job faster. Merging bundles allows
us to save on sql lock and opening/closing transactions
Reviewed By: ikostia
Differential Revision: D15080348
fbshipit-source-id: bf2e4ca3dd7ff8c399a3baa34ab542f67ea6efa9
Summary: This adds the ability to exclude blobimport entries when querying the count of remaining entries in the HG sync replay log.
Reviewed By: ikostia
Differential Revision: D15097549
fbshipit-source-id: ae1a9a31f51a044924fdebbdd219fff1e2b3d46a
Summary:
This introduces a new `--skip` flag in Mononoke admin under `hg-sync-bundle last-processed`. This will update the last-synced counter to the last blobimport that preceeds a log entry that is no a blobimport.
In other words, it makes it so that the last change to be processed is *not* a blobimport.
It will fail if:
- There is no valid log entry to jump ahead to (e.g. all the further log entries are blobimports).
- The current change to be processed is not a blobimport.
- The mutable counter was changed by someone else in the meantime.
Reviewed By: ikostia
Differential Revision: D15081759
fbshipit-source-id: 8465321b08d9c7b5bc97526400518bcf3ac77f13
Summary: This adds a command in mononoke admin to verify the consistency of remaining bundles to sync -- i.e. whether all bundles are blobimports or all of them are not blobimports.
Reviewed By: ikostia
Differential Revision: D15097935
fbshipit-source-id: a0df221c38e84897213edf232972ba420977e9d3
Summary:
Stanislau and Kostia mentioned that adding a few more commands for managing the HG sync job in the Mononoke Admin tool would be useful — in particular those commands should interact with the list of log entries and the last sync.
Notably, they were:
- Add support for skipping ahead any blobimport log entries.
- Add a verify command to check that all entries looking forward are consistent — i.e. they're either all blobimport or all non-blobimport.
However, as it stands, it's hard to test those in integration tests, because one can't easily generate stub data to exercise on the admin tool.
Notably, inserting data through the `sqlite3` database command line results in inconsistent values for the `reason` column that don't compare equal to those we query from the Rust side of things.
We could work around this by updating the Rust code to e.g. use LIKE instead of equality checks, but then we're updating the production code exclusively for the purpose of serving tests.
So, instead, this patch proposes introduces a lightweight command line tool to use in tests that basically just inserts data into the bookmarks database, but from the Rust side of things.
Reviewed By: StanislavGlebik
Differential Revision: D15081758
fbshipit-source-id: 574739d2e19e6ce59d729f1aee73c368612fb92b
Summary:
The code in admin/main.rs passes `bookmarks` as the name of the SQLBookmarks database, but other pieces of code (the server which writes this data, and the hg sync job that normally consumes it) use `books`.
This hasn't been a problem until now since no integration tests exist for the admin tool. This path mismatch was never exposed since in production we don't actually care about this name.
Reviewed By: ikostia, StanislavGlebik
Differential Revision: D15081757
fbshipit-source-id: c4fcd328568160023f3c15fa2ab7d77accf2ad68
Summary: Now it is possible to configure and enable/disable bookmark cache from configs
Reviewed By: StanislavGlebik
Differential Revision: D14952840
fbshipit-source-id: 3080f7ca4639da00d413a949547705ad480772f7
Summary:
This diff adds caching layer for bookmarks. It only effects semantics of `list_by_prefix_maybe_stale`, other should methods should not be effected by this change.
- all requests to database issued by `list_by_prefix_maybe_stale` are cached for `ttl` time
- but if happened locally cache is purged and replaced to new one which will go through master replica
Reviewed By: StanislavGlebik
Differential Revision: D14928419
fbshipit-source-id: 30994f9c31f8063e8d6b2dd0a6afb06c067aa65a
Summary:
Before this diff we always wrote to master even if paths already exist. This
diff changes it - first check in replica if paths are there, and only then
write to master
Reviewed By: farnz
Differential Revision: D15063307
fbshipit-source-id: 802839f340c9953c7f2812e77d81bc66917c5e77
Summary:
Generating hg changesets might create too many connections to mysql. Let's
how many hg changesets generation we have in-flight
Reviewed By: lukaspiatkowski
Differential Revision: D15069236
fbshipit-source-id: 2b7265d1efa7d09458ed214bad7bc6f32bda8dbf
Summary:
We'll start using memcache leases in the next diff, this diff refactors the
code to make it adding memcache leases simpler
Reviewed By: lukaspiatkowski
Differential Revision: D15069237
fbshipit-source-id: 875c398a3825d839b3453a9c77043edec496efa2
Summary: Before this change we were only marking head of the pushrebase, this change fixes this problem and marks all reachable changesets as public
Reviewed By: StanislavGlebik
Differential Revision: D15063835
fbshipit-source-id: 2a360684fc01cec0f639c1789eff8150e5ba5ebb
Summary:
In order to do more complete tests around authentication we need to provide
fb identity in to our test certificates.
Reviewed By: StanislavGlebik
Differential Revision: D15046017
fbshipit-source-id: 3f3cd450425944a2970c6f02e7eb92a878076a05
Summary: - convert phases crate to rust-2018, in preparation for the next diff in a stack
Reviewed By: StanislavGlebik
Differential Revision: D14995610
fbshipit-source-id: 30327508cee5154a8a788e8c5a2f914f7ccacfec
Summary:
Add more logging for generation of hg changesets from bonsai. We are interested
in depth (i.e. how many commits it had to generate), latency of the whole
request and latency of generation of a single hg changeset.
Reviewed By: ikostia
Differential Revision: D15046567
fbshipit-source-id: 1e9b0471fa5ed7df3d492c11bf230adc7ee0c1b8
Summary: Added obsmarkers to pushrebase output. This allows the client to hide commits that were rebased server-side, and check out the rebased commit.
Reviewed By: StanislavGlebik
Differential Revision: D14932842
fbshipit-source-id: f215791e86e32e6420e8bd6cd2bc25c251a7dba0
Summary: This adds supporting in mononoke for encoding obsmarkers, according to Mercurial's V1 format for obsmarkers.
Reviewed By: StanislavGlebik
Differential Revision: D14932828
fbshipit-source-id: fe7283554c9cdcfdeb303e19b28af60d49c26722
Summary:
Add a LABEL constant to the SqlConstructors trait to make it easier to identify
which table is being used, for stats and logging.
Reviewed By: HarveyHunt
Differential Revision: D13457488
fbshipit-source-id: a061a9582bc1783604f249d5b7dcede4b1e1d3c5
Summary: Following D14935155, let's decrease number of write connections as well
Reviewed By: ikostia
Differential Revision: D14973547
fbshipit-source-id: c344ecc568be26287e998b45b6988744cb5e0a09
Summary:
Add a new option that allows a comma separated list of changesets
that should be excluded by the hook tailer.
Also add an option that allows the hook_tailer to parse a file containing
changesets that it should ignore.
If both arguments are passed, the collection of excluded changesets are
merged.
This will be used by the conveyor node to provide better signal by ignoring known
hook failures caused by hg hook bugs.
Reviewed By: aslpavel
Differential Revision: D14894072
fbshipit-source-id: b488bb0bb5e05d7313005e04c55568dd4be5b770
Summary: Update the Eden API client and server to use the new DataEntry type for file content downloads.
Reviewed By: quark-zju
Differential Revision: D14907958
fbshipit-source-id: 8a7b1cbb54bdc119dda11179ff94d3efdb7e85c9
Summary: Removing this function in favor of using Key.path
Reviewed By: quark-zju
Differential Revision: D14945331
fbshipit-source-id: 6b6bb70375629edf37b2b04a86545f18e15b33b4
Summary:
`new_rocksdb_delayed` wasn't used at all, and the other
`new_files`/`rocksdb`/`sqlite` aren't used anywhere else, so make them private.
Reviewed By: StanislavGlebik
Differential Revision: D14862288
fbshipit-source-id: 41e02804fca37f4f27dddd53ee27e60b43798a99
