Summary: This diff introduces a simple command line utility which allows a hook to be run against a specific changeset. The idea is to allow hooks to be easily tested without having to run a Mononoke server.
Reviewed By: StanislavGlebik
Differential Revision: D8183908
fbshipit-source-id: 2ebadf026a23ac69bc14db6794fdf760728f1d3b
Summary:
This diff introduces an initial implementation of hooks, corresponding to stage 1 in "plan of action" in https://fb.quip.com/dnHuARInjA9I
* lib.rs contains the core hook structs and traits.
* rust_hooks contains a simple implementation of hooks for rust.
* lua_hooks contains a simple implementation of hooks for lua.
* The old hooks implementation has been moved to hooks_old
Reviewed By: StanislavGlebik
Differential Revision: D8183909
fbshipit-source-id: 17840db4ebb68009efe0df90df7383c5712bd1ba
Summary:
We can hit a deadlock during getfiles requests. We first send all requests via
stdin, and then read the output via stdout. The problem is that any of the
buffers anywhere between hg and mononoke may fill up, and the whole process
deadlocks.
Unfortunately there is no good solution yet. For now let's make the buffer in
hgcli big enough to make sure it won't be a problem. Let's bump it to 400Mb.
Note that the buffer will be allocated on demand in the channel. So it won't
use too much memory.
Reviewed By: sunshowers
Differential Revision: D8195511
fbshipit-source-id: bd97509bd26f8cea7d41e44933e02bd0bc2c76c4
Summary:
I'm about to introduce a MemoizedBlobstore - let's ensure that
ambiguity is kept to a minimum by renaming CachingBlobstore
Reviewed By: jsgf
Differential Revision: D8124625
fbshipit-source-id: 301fbb31c5a772c11e84566889b8a6ac86cdae19
Summary:
In practice, caching blobstores are always wrapped around a known
concrete blob store. Make it generic in the type of the Blobstore, so that
rustc can reach in and inline appropriately if it sees an optimization
opportunity.
This is a micro-optimization, hence nice to have
Reviewed By: StanislavGlebik
Differential Revision: D8124627
fbshipit-source-id: 0d5d9b11fdede062ceaa949ace574b3d1c75e6b5
Summary:
I'm about to add a memcache caching blobstore, and the blobstore
lib.rs is getting large. Shrink it by removing the outdated comment, replacing
it with documentation comments, and moving the CachingBlobstore to its own file
Reviewed By: jsgf
Differential Revision: D8124626
fbshipit-source-id: ac081a162c2f15a26452ff708d16a108699c279b
Summary:
We want to be able to go to and from Bonsai Changesets in Mononoke, so
that we can operate with Mercurial clients that haven't yet been updated to
support Bonsai form. This RFC commit shows a shell of doing so, as an
illustration for the hackamonth task
Reviewed By: jsgf
Differential Revision: D8014909
fbshipit-source-id: 28adf18ecf80e0116290662c117731b4c1632ff9
Summary:
Loading the entire manifest tree just for a limited set of conflicts
and changes is wasteful. Lazily load the referenced parts of the tree instead
Reviewed By: jsgf
Differential Revision: D8014913
fbshipit-source-id: 07678bee39de02414fdc062cf680fcd049a28415
Summary:
It will be used in getfiles wireproto method. It requires fetching the whole
file history, and doing lots of point lookups is too slow.
Reviewed By: lukaspiatkowski
Differential Revision: D8182050
fbshipit-source-id: b17cb2259b2237f0027bfb499fbe97d9b40b1a62
Summary:
Move functionality of converting FilenodeRow into FilenodeInfo into a separate
function. It will be used in the next diff
Reviewed By: lukaspiatkowski
Differential Revision: D8182051
fbshipit-source-id: f2276aa979200f71aa05808595842c09a9f5899f
Summary:
Motivation for this change is the following:
We want to avoid doing point lookups to the db to fetch the history, and
instead select the history for a file or a directory with a single select. That
should be many times faster.
After we select the data from filenodes table, we need to fetch copy info
data. To do that we need to either do lots of point lookups to the fixedcopyinfo (which
undermines the point of doing single select) or construct a giant select
statement. First option is inefficient, the second can introduce big code
complexity.
Instead let's add a single field that says whether we have fixed copy info or
not. Db size increase should be tiny.
For the data that we've already imported I'm planning to run a script that will
fill the has_copyinfo field correctly.
Reviewed By: jsgf
Differential Revision: D8164029
fbshipit-source-id: c91c99b065808a93a9b361914cf9b3822d78cb60
Summary:
This is a binary that reads json encoded requests from cmd line and sends them
to Mononoke. Usually requests will be sent via scribe from hg servers.
Then the data will be logged to scuba table `Mononoke Mercurial Comparison`
Also change hgcli to print errors to stderr
Reviewed By: farnz
Differential Revision: D8143781
fbshipit-source-id: 8d6c5511dfd2368f5d018825502c384f2ad89d4a
Summary:
It has a few benefits. Firstly, we won't get misleading empty trace ids in the
logs. Secondly, uploading traces takes a noticeable amount of time that affects
the overall latency. That is especially noticeable for getfiles requests.
Reviewed By: farnz
Differential Revision: D8158917
fbshipit-source-id: b4fe2183d9157f4d7c52e25e4efef9a06bb1b5ac
Summary: This is required for pushes, without it the pusher assumes we don't need tree manifests
Reviewed By: farnz
Differential Revision: D8139396
fbshipit-source-id: 42ad18b2f437a0b9441c0d84e4519b87f2902c0d
Summary: This should prevent new_blobimport from updating bookmarks to non-existing changesets
Reviewed By: farnz
Differential Revision: D8122315
fbshipit-source-id: 20bcbb6887b54a88b39f9ba375884e8b9c0143f7
Summary:
Those flags will let us control the subset of commits that the new_blobimport is supposed to import.
It's a temporary solution to be used instead of the old blobimport but before "hg push" works
Reviewed By: farnz
Differential Revision: D8122318
fbshipit-source-id: a1ac0824020341cd4bb18ec46d91caee50d9606e
Summary:
This could never fail - so don't allow for failure, but use the type
system to deal with the error case.
Reviewed By: jsgf
Differential Revision: D8014910
fbshipit-source-id: 8f5f0a3ff55a96b57cd4c246d072793c986724d5
Summary:
We want to handle conflicts here - do so, but with asserts, by merging
two manifests together.
The asserts will be removed once we make this all lazy
Reviewed By: jsgf
Differential Revision: D8014912
fbshipit-source-id: 7a09186f7e24a3af93d15a859c877e3a319fb110
Summary:
Some wireproto methods may have huge argument lists. They just print too much
data to stderr, and make stderr basically useless. Let's limit it
Reviewed By: jsgf
Differential Revision: D8076949
fbshipit-source-id: 58e760f25a7d4fdc9fc7bd95635f916a08e15ed2
Summary:
It happens quite often in practice, so we have to support it.
Luckily it's not that hard to implement
Reviewed By: jsgf
Differential Revision: D8076494
fbshipit-source-id: 460ba8b4bac121b84554e89b285d5ef78cff4838
Summary:
Log parameters to scuba for getfiles and gettreepack. These are the most
important wireproto methods.
Reviewed By: jsgf
Differential Revision: D8076492
fbshipit-source-id: 031bad732ee31d254a707ddd0e8c444d23c2fb0f
Summary: There is nothing we can do with it ATM
Reviewed By: StanislavGlebik
Differential Revision: D8090873
fbshipit-source-id: 232024938fe33583e328ad3231d9213dd6792601
Summary: csv crate has been updated to 1.0.0 which has API changes compared to the previous version.
Reviewed By: sunshowers
Differential Revision: D8014794
fbshipit-source-id: 8c5f148050c042df47a76fa2f8c7634794c69ccd
Summary: Once we have a lazy loading version, we'll want a way to cope with multiple accessors all trying to update the same `children` array with identical data; make it immutable in use, so that overwriting it multiple times is harmless
Reviewed By: jsgf
Differential Revision: D8014908
fbshipit-source-id: 9a2750fb1fca54601051fede1d9a37de8cfc2a74
Summary:
Instead of writing changesets one-by-one run mutiple of them at once.
The size `100` for the buffer is arbitrary, but it shouldn't matter much since we already have backpressure on the database writes.
Reviewed By: farnz
Differential Revision: D8057268
fbshipit-source-id: ca3766505395dcb6be6684323462f1bb23222435
Summary:
We had an asyncmemo bug when keys were, for example, Strings. If we insert
String::new("key"), and then try to remove String::new("key"), then these keys
may have different weights because Strings may have different capacity.
This is not the case for values though, because inserted and deleted value is
always the same object.
Let's not take key sizes into account at all. That means that the real asyncmemo memory usage will be
higher, however values are a great deal bigger than keys, so it sholdn't be an
issue.
Also we are planning to do it anyway, because we want to split
asyncmemo hashmap into in-flight futures and completed futures, and in-flight
futures won't take key sizes into account.
Reviewed By: jsgf
Differential Revision: D8040684
fbshipit-source-id: 332dbe5663182dd84ea291ca4b33b41995ac0166
Summary:
parking_lot's primitives are apparently lighter-weight than using
standard (pthreads) ones, and are more or less drop-in replacements.
Reviewed By: farnz
Differential Revision: D7661246
fbshipit-source-id: c7558a15971bf5b30c4dadb9437586832cdad3d4
Summary:
This tool will be installed on all devservers in our team. It should have
useful debugging commands. Currently it has a command to fetch an entry from
the blobstore, and a command to get content by path/commit hash.
Reviewed By: farnz
Differential Revision: D8028933
fbshipit-source-id: e0c37660a24e40dd9dc8f19d1789b2f25d99bfe6
Summary:
We're going to need to be able to edit memory manifests. Provide
remove and set operations, to match the Bonsai Changeset data structures
Reviewed By: StanislavGlebik
Differential Revision: D7620527
fbshipit-source-id: e85459c5dbfa8855267fd2cf6578c9fc39f223f8
Summary:
Removing entries can leave us with empty manifests. Rather than clean
that up mid-flow, simply skip empty manifests when saving
Reviewed By: StanislavGlebik
Differential Revision: D7620526
fbshipit-source-id: 2f0799eba305a5295eaad28cd4ad90c9de04306f
Summary:
To do Bonsai Changesets, we're going to need to perform surgery on
manifests. Provide a mechanism to get the Tree nodes only into memory, and
write them back out, so that we can do the surgery the easy way.
Reviewed By: StanislavGlebik
Differential Revision: D7557271
fbshipit-source-id: 7afdc3ef464fc042eb758af863ade8938c4e9fc5
Summary:
Previously, we assumed that all content hashes came from Mercurial;
this is not going to remain true, as we will want to be able to upload manifests
that have been synthesised from Bonsai Changesets. Turn the previous boolean
into a tri-state, and fix up all callers to get the behaviour they expect.
Reviewed By: StanislavGlebik
Differential Revision: D8014911
fbshipit-source-id: 9156b9fab4542ceb269626ad005e1b28392b5329
Summary: Parsing and reading revlogs is cpu intensive, thus let's use cpupool for it
Reviewed By: StanislavGlebik
Differential Revision: D7926174
fbshipit-source-id: 7f023088941e1ad118a683da972f87607e0bfec4
Summary: printing every CS is too verbose, but we still want to see progress in non-debug mode
Reviewed By: kulshrax
Differential Revision: D7925747
fbshipit-source-id: c3ed92ef8c8fbf7714779a2bf011d31c94aefa37
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary: this breaks the prod since we are not able to parse clienttelemetry being send by hg
Reviewed By: jsgf
Differential Revision: D7967361
fbshipit-source-id: e10eb52852ccb8c2878db01cd63b05cd7f901ae4
Summary:
@[1636530097:mung] reported that there was a bug wherein Diesel was incorrectly using MySQL's API by incorrectly calling `mysql_store_result` after `mysql_next_result` indicates that no more results are available. This behavior [appears to be intentional](705cdbe1fe/diesel/src/mysql/connection/raw.rs (L160)), but directly contradicts what the [documentation says is correct](https://dev.mysql.com/doc/refman/8.0/en/mysql-store-result.html). In MySQL 8.0, this results in the client getting an error for incorrect usage of the API.
I've added a local patch to [our fork of Diesel](https://github.com/kulshrax/diesel) on Github to [remove the extra call to `mysql_store_result`](7ae79c193e), and rebuilt the `rust-crates-io` tp2 project to use this version of Diesel. This diff updates the symlinks in fbcode to point to the updated tp2 project.
Reviewed By: lth
Differential Revision: D7961290
fbshipit-source-id: 804c599a5688cd54fb0fcc33688c608d068d7a7b
Summary: This allows us to put hgcli on the different host
Reviewed By: jsgf
Differential Revision: D7910980
fbshipit-source-id: 45bdd1ee42b54f8f1a425ed6950e785cdf85bfc3
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host,
because unix domain socket are used. Also Mononoke has to use separate unix
domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
This diff adds a separate thread that parses the Preamble, extracts the
reponame and sends request to a thread responsible for this repo.
Also hgcli now has a new cmd line option that specifies path to connect to.
Reviewed By: jsgf
Differential Revision: D7845156
fbshipit-source-id: a48bcfeec4755b2d5b3dfcf8e0383e6945178018
Summary: put_slice panics if not enough space. This diff fixes it
Reviewed By: jsgf
Differential Revision: D7876156
fbshipit-source-id: b1e2cc520e5c598588df9a8090fb70efb413b381
