Summary: This diff adds unit tests for the runhook command line utility
Reviewed By: jsgf
Differential Revision: D8257571
fbshipit-source-id: 5c390d2a45d895080fce28dcd7943da5d803ff92
Summary:
This diff adds more robust testing for various errors in Lua hooks.
It also contains a little bit of cleanup in the runHook command
Reviewed By: StanislavGlebik
Differential Revision: D8253525
fbshipit-source-id: de9d298e70ec647f2c13e27c9937605ac5b57485
Summary: Changed the about section of new_blobimport to something descriptive.
Reviewed By: kulshrax
Differential Revision: D8307334
fbshipit-source-id: 2198d43cdfaf566b57001be5230a74206306dac1
Summary:
Replaced the help and about messages to summarize what the admin tool
is useful for
Reviewed By: jsgf
Differential Revision: D8302356
fbshipit-source-id: 77d1d4bb50825b0cc2d1ac2ee8d47ee906e04a22
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary: It only needs to borrow them.
Reviewed By: kulshrax
Differential Revision: D8244267
fbshipit-source-id: 2a24a3b7c6eb65177e4e26c57650dd7e096b4202
Summary:
Now hook manager returns result of running hooks as BoxFuture<HashMap<String, HookExecution> where HookExecution is a new richer type representing the result of running a hook.
This provides more info to the user as to why the hook rejected the changeset and a map is simpler to lookup a particular hook failure than a Vec.
Reviewed By: StanislavGlebik
Differential Revision: D8235970
fbshipit-source-id: 9a617b6d459f105aa9dad9782e784459dd716c45
Summary:
There's a bit of Rust/C FFI, but hopefully the comments make it clear
how ownership works.
As part of the lifecycle I also had to handle shutdown safely. This means that `run_service_framework` actually returns now.
Reviewed By: jsgf
Differential Revision: D8133252
fbshipit-source-id: 1e394a60c6e62f5c3d56933f3e7be7a2bfd1e2c0
Summary:
This diff introduces HookManager which knows how to install, uninstall and run hooks.
Hooks are now run in parallel on a cpu_pool.
Reviewed By: lukaspiatkowski
Differential Revision: D8208538
fbshipit-source-id: f18687c14a15cadf4d832318cd66fa400586c29f
Summary: If we're going to use memcache to cache things, we need to be able to confirm that the blobstore contains what we expect. Add an admin option that lets us check what's in memcache and/or in the blobstore when using memcache
Reviewed By: StanislavGlebik
Differential Revision: D8124624
fbshipit-source-id: 763786930cf9f4be6d2d8e346ae1ef94dc8e492c
Summary: This diff introduces a simple command line utility which allows a hook to be run against a specific changeset. The idea is to allow hooks to be easily tested without having to run a Mononoke server.
Reviewed By: StanislavGlebik
Differential Revision: D8183908
fbshipit-source-id: 2ebadf026a23ac69bc14db6794fdf760728f1d3b
Summary: This should prevent new_blobimport from updating bookmarks to non-existing changesets
Reviewed By: farnz
Differential Revision: D8122315
fbshipit-source-id: 20bcbb6887b54a88b39f9ba375884e8b9c0143f7
Summary:
Those flags will let us control the subset of commits that the new_blobimport is supposed to import.
It's a temporary solution to be used instead of the old blobimport but before "hg push" works
Reviewed By: farnz
Differential Revision: D8122318
fbshipit-source-id: a1ac0824020341cd4bb18ec46d91caee50d9606e
Summary:
Instead of writing changesets one-by-one run mutiple of them at once.
The size `100` for the buffer is arbitrary, but it shouldn't matter much since we already have backpressure on the database writes.
Reviewed By: farnz
Differential Revision: D8057268
fbshipit-source-id: ca3766505395dcb6be6684323462f1bb23222435
Summary:
This tool will be installed on all devservers in our team. It should have
useful debugging commands. Currently it has a command to fetch an entry from
the blobstore, and a command to get content by path/commit hash.
Reviewed By: farnz
Differential Revision: D8028933
fbshipit-source-id: e0c37660a24e40dd9dc8f19d1789b2f25d99bfe6
Summary:
Previously, we assumed that all content hashes came from Mercurial;
this is not going to remain true, as we will want to be able to upload manifests
that have been synthesised from Bonsai Changesets. Turn the previous boolean
into a tri-state, and fix up all callers to get the behaviour they expect.
Reviewed By: StanislavGlebik
Differential Revision: D8014911
fbshipit-source-id: 9156b9fab4542ceb269626ad005e1b28392b5329
Summary: Parsing and reading revlogs is cpu intensive, thus let's use cpupool for it
Reviewed By: StanislavGlebik
Differential Revision: D7926174
fbshipit-source-id: 7f023088941e1ad118a683da972f87607e0bfec4
Summary: printing every CS is too verbose, but we still want to see progress in non-debug mode
Reviewed By: kulshrax
Differential Revision: D7925747
fbshipit-source-id: c3ed92ef8c8fbf7714779a2bf011d31c94aefa37
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.
BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.
Reviewed By: jsgf
Differential Revision: D7909992
fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
Summary:
Let's use the new feature in SendWrapper to use many io threads. That will help
us mitigate the high cpu usage issues we were having with blobstore requests.
Manifold blobstore now creates the io threads itself.
Reviewed By: kulshrax
Differential Revision: D7831420
fbshipit-source-id: ec9f3327347ca6bfbd23c482e69a6fee663b1da5
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary: The goal is to be able to read revlogs using Rust code and also parse and serialize them in Rust formats for debugging purposes
Reviewed By: farnz
Differential Revision: D7830358
fbshipit-source-id: 95e257a4482eca22b328b174bce3fceec1b47245
Summary:
The commits that are blobimported have out of order or simply incorrect lists of changed files.
Because we have to persists Changesets as is we are passing the untouched list of files here to be used by Changeset.
Reviewed By: farnz
Differential Revision: D7830310
fbshipit-source-id: 56adec2c317896decaa9176b3a6bfb0cab187ed0
Summary: Useful utility that let's you f.e. fetch blob of data from manifold, decode it and show it to you
Reviewed By: jsgf
Differential Revision: D7779154
fbshipit-source-id: aaa4ae1d09b64f7f52c7942a51e8bb4ccc0cb700
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around
Reviewed By: StanislavGlebik
Differential Revision: D7775032
fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
Summary: For on-disk-rocksdb use cases we should persist bookmarks like any other table we use
Reviewed By: farnz
Differential Revision: D7728717
fbshipit-source-id: f63a6410f5ed254a719a16a7504d1b31da5a20a8
Summary:
This change isn't doing much on it's own since rocksdb's BlobRepo is using in memory Bookmarks ATM and they dissapear when import is finished.
Later bookmarks will be used for Head discovery, then it will be properly tested
Reviewed By: farnz
Differential Revision: D7728716
fbshipit-source-id: ad50f35b18d93aa1e38951408092e46e67fde0c7
Summary: I am planning to add importing bookmarks, doing it on current main.rs would make it unreadable, so I am splitting this file now
Reviewed By: farnz
Differential Revision: D7728185
fbshipit-source-id: fdfb4f60eecd9c8af7626bd0e892bb1bfbf7f081
Summary: The eden integration test contains a commit with no content which new_blobimport couldn't import. With this changes the commit API is capable of handling such commits.
Reviewed By: jsgf
Differential Revision: D7709243
fbshipit-source-id: 7d55eb2ec421820d189ab05b0f8cb4411f850a7b
Summary:
Let's fail only if inconsistent data was inserted - for example, same commit
hash but different parents.
This matches core hg behavior, and also it's completely normal for commit cloud
to send more parent commits than necessary.
Reviewed By: lukaspiatkowski
Differential Revision: D7722649
fbshipit-source-id: 172a0985fb3fda27d55e9dce8916ec3793de5db9
Summary:
There are a few separate steps during blobimport. One of them is inserting the
blobs and another is inserting the changesets. That worked fine if RevlogRepo was
static. However if RevlogRepo has changed between first and second step we
could've inserted blobs for N commits, but insert (N + X) changesets into
changesets table. That means that a few commits would have no blobs at all.
This diff fixes it by fixing the changesets that we want to blobimport
beforehand.
Reviewed By: farnz
Differential Revision: D7615133
fbshipit-source-id: 1a66907e34a65588b101199c8f59abda53f7bc20
Summary: As with filenodes, we also want to write changesets in a real store.
Reviewed By: lukaspiatkowski
Differential Revision: D7615101
fbshipit-source-id: 269deb8fc3c1f58afb82f453a68ea4d8a3f1f63d
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Let's use it! Pass config option that set's the cache max memory usage (don't
put a limit on the number of entries, it's useless in that case).
Currently we'll set a separate size for each of the caches that we use
(blobstore, changesets, filenodes, etc). Later we'll have just one single option that
sets the cache size for all of them.
Reviewed By: lukaspiatkowski
Differential Revision: D7671814
fbshipit-source-id: f9571078e6faaa80ea4c31c76a9eebcc24d8a68a
Summary:
We know that the hashes for non-root-tree-manifests and filenodes
should always be consistent. Verify that.
Reviewed By: farnz
Differential Revision: D7704087
fbshipit-source-id: 7f6207878c5cd372b272aa6970506dd63b5a3c7c
Summary:
As the comment explains, sometimes the hashes don't match the
contents. Accept such pushes.
Reviewed By: farnz
Differential Revision: D7699930
fbshipit-source-id: 376f01b6cf03f6cad84c2c878d192d55f8d81812
Summary:
We're going to keep this around for now as part of double-writing.
All the hashes here are definitely Mercurial hashes, so use them that way.
Reviewed By: lukaspiatkowski
Differential Revision: D7683890
fbshipit-source-id: 270091cd11f3cec7ef4cf565de5ef913fcf7adea
Summary:
file::File works entirely in the Mercurial domain, so these
conversions are good.
Reviewed By: StanislavGlebik
Differential Revision: D7665973
fbshipit-source-id: 8a192c5d1886492ad21593693b080c8e5ddf8f7e
Summary:
This is because these Mercurial entries are (at least currently) going
to be stored as they come in, and this data structure is entirely in the
Mercurial domain.
Reviewed By: lukaspiatkowski
Differential Revision: D7664972
fbshipit-source-id: 9de5475eed0d7ab7085c29fd0282f205043cfe5a
Summary:
I was trying to debug something with the new blobimport, and this was
getting in the way.
Reviewed By: StanislavGlebik
Differential Revision: D7664660
fbshipit-source-id: 2ec4ee79fbe13584f35e7dcd9e8df2b8bdf181c0
