Summary:
Reading filectx info tried to use an optimization to get the filenode
without uncompressing the whole manifest. In a tree world, this optimization
actually hurts performance because it requires downloading the parent tree and
performing a diff. Since everything is now treeonly, let's just drop this
optimization.
This helps in some Eden tests where they prefetch a specific commit then attempt
to access it offline.
Reviewed By: markbt
Differential Revision: D15344433
fbshipit-source-id: 3b6ebc27ce7e263abefaa962856a4c7bea0df793
Summary:
Previously readfast was an optimization that sometimes returned a delta
against p1, but other times returned the full manifest. This was weird and
caused certain algorithms to sometimes be O(changes) but other times O(repo).
In a tree world we no longer need this optimization, so let's drop it.
readdelta is similar in that it would read the difference between a manifest and
it's delta base. This has no relationship to the delta between a manifest and
it's parent, so it's weird and we should get rid of it.
There is a legitimate use case for wanting to know what entries are new in a
manifest, like when deciding what to send over the wire. So let's add a new
readnew() function that is explicitly for reading what entries were introduced
by this tree. The implementation is basically the same as readdelta, but
deltaing against p1 instead of the deltabase.
Reviewed By: markbt
Differential Revision: D15344434
fbshipit-source-id: dc8dca326f66b2fc55cc76f93c7ce48aa7efedf3
Summary:
This makes it easier for a wrapping job to associate output with a given command it ran.
Note that creating a SSH peer creates its own copy of the `ui` object, which is why we have to push a buffer to both `ui` instances.
Reviewed By: ikostia
Differential Revision: D15468456
fbshipit-source-id: c6f1937749447e27332801577538d9874eb18898
Summary: Add progress bars to remotefilelog's debug commands for HTTP data fetching. This makes it obvious whether there is activity occurring or whether Mercurial is stuck.
Reviewed By: quark-zju
Differential Revision: D15480683
fbshipit-source-id: 8c67c306488e5ec82a32143eb1748879474e3ec9
Summary:
While mutable datapack are used in several places, most of the writes are
coming through the pendingmutablepack class, so let's start switching these
to use the Rust code.
Reviewed By: kulshrax
Differential Revision: D15482770
fbshipit-source-id: d0e4aae38a5ef2ba625a40ff907420623e6efd11
Summary:
Both the Rust and Python mutable stores support a flush method, but while
Python support an abort method, Rust will simply cleanup temporary files when
the object is destroyed. We can implement a similar scheme in Python by
implementing __del__.
Reviewed By: kulshrax
Differential Revision: D15482771
fbshipit-source-id: 4c9647ab7dd9fa078a8fc3c8b00962fbc0823dfd
Summary:
Detecting the underlying store type allows for store writes to be done fully in
Rust, with no roundtrips in Python. This also allows us to reduce our
dependency on the mutabledatapack Python code.
Reviewed By: kulshrax
Differential Revision: D15474314
fbshipit-source-id: 926d828e350e5e1c59b2f29ed11dd71db17eae56
Summary: This will be used in the edenapi binding and therefore needs to be public.
Reviewed By: kulshrax
Differential Revision: D15474312
fbshipit-source-id: fcef80ba414ae63908d4a4191003fabe0f07c254
Summary:
This will allow the object to be used in native Rust code without roundtrips to
Python.
Reviewed By: kulshrax
Differential Revision: D15474310
fbshipit-source-id: f89033ac9e449125d4563e9d42b31cb6362c3235
Summary:
While the datapack read path is currently being done in Rust, the write path is
still being performed in Python. This class is the first step towards switching
the write path to Rust.
A nice bonus of this class is that it abstract packfile/indexedlog and besides
at construction time, the users of it won't be aware of which store is used.
Reviewed By: kulshrax
Differential Revision: D15474313
fbshipit-source-id: 49454d8f089089f164d858d590f254c78d3f5eb6
Summary:
Similarly to DataStorePyExt, this trait abstract parsing Python object and
transforming them into Rust objects.
Reviewed By: kulshrax
Differential Revision: D15474316
fbshipit-source-id: 3c72606dca52743824151bbad10f20a6ed6c1b2f
Summary:
The trait object `dyn DataStore` isn't sized and therefore doesn't implement
the DataStorePyExt as the later requires a sized type. There is no reason for
this restriction, so let's allow non-sized types to implement DataStorePyExt.
Reviewed By: kulshrax
Differential Revision: D15474317
fbshipit-source-id: 06ae62ef7f18808f316da03dc87ba9a6fa986690
Summary:
While conflicting method names are allowed in Rust, it usually leads to subpar
compiler warnings and code to be written in a non-traditional way. Let's rename
the methods on these 2 traits so there isn't any conflict to avoid this.
Reviewed By: kulshrax
Differential Revision: D15474311
fbshipit-source-id: 64e4f4cef2aa89c18f51b81260eff70f57183465
Summary:
In all cases, the mutable stores also support reading from them. Let's have the
compiler enforce this, and implement the missing traits.
Reviewed By: kulshrax
Differential Revision: D15474315
fbshipit-source-id: dfcd93f319582ff3cd54aeabaa66bc6df9ce63a8
Summary:
The phases command had some naive code that looped over every revision
in the repository. Twice. This was really slow on huge repos, so let's get rid
of the need for it.
I originally attempted to optimize it by only looking at commits greater than
the ones being passed in, but this is incorrect since changing a commit to
public may affect it's ancestors. Instead of trying to optimize it further,
let's just drop the output text entirely.
Reviewed By: quark-zju
Differential Revision: D15465942
fbshipit-source-id: ba63a1096a515032fe13a8699807fe69c73c80ed
Summary:
When writing to a revlog we need to check if the revlog already
contains the node. A normal `node in self.nodemap` is quite expensive in large
repositories because it's likely the node is not in the nodemap and therefore it
must scan the entire index.
This new function uses the knowledge about the parents to limit how many nodes
are inspected when adding a new revision. In the degenerate case this could end
up traversing a large chunk of the graph in a slow way, so we catch the case
where the parents are far from the tip and revert to the old native path.
Reviewed By: quark-zju
Differential Revision: D15444548
fbshipit-source-id: 5f19ccf85fb11f1c787ebda09df9c4bef17a2a31
Summary:
Opening all the revlogs to validate them is showing up as a performance
bottleneck for hgsql transactions. Let's switch them to using mmap to avoid
reading a bunch of data from disk.
Actually making this have an effect on our servers will also require setting the
experimental.mmapindexthreshold config.
Reviewed By: quark-zju
Differential Revision: D15441867
fbshipit-source-id: 4edde0bc3419ef75f82a4234c9dfc6604c6db9f4
Summary: Print out download stats when `edenapi.debug` is set. Pretty printing of the stats has been improved to make these numbers more useful to humans.
Reviewed By: xavierd
Differential Revision: D15466702
fbshipit-source-id: 6fac9ca5976b98874fc7a5f9b89d42975e1520ce
Summary:
When EdenAPI encounters an error, the current behavior is to print an error message and then fall back to SSH. In addition to the fallback notice, we were previously printing the exception message using `ui.develwarn`, with the understanding that this information is only really useful to Mercurial developers.
Instead of using `develwarn`, let's use `ui.warn` but gate the call so that only users with `edenapi.debug` set to `True` see the error message.
I initially tried printing the entire backtrace with `ui.traceback`, but this proved too verbose to print out in the middle of normal command output. The full traceback is logged to Scuba, and can be obtained from the `hg_errors` Scuba table, so just logging the exception message should be sufficient.
Reviewed By: quark-zju
Differential Revision: D15466332
fbshipit-source-id: 43d40eff47e4f68fe860383e7bd519fbd0052830
Summary: On the Source Control team, we've been referring to the project to replace parts of the Mercurial wire protocol with the Mononoke API server as "HTTP data fetching". This has caused confusion for other teams in the past (particularly teams with a security-related focus), as this nomenclature makes it seem as if we're using unsecured HTTP. (This is not true; we are always using HTTPS.) In light of this, let's make sure that user-facing messages always use the term "HTTPS" instead of "HTTP" to avoid this confusion.
Reviewed By: quark-zju
Differential Revision: D15465390
fbshipit-source-id: b556ed222dc01979ceaa0d4a7aa921cb1c38d75e
Summary: Log metrics about HTTP data fetching to `dev_command_timers`, and log HTTP fetching errors to `hg_errors`.
Reviewed By: quark-zju
Differential Revision: D15464791
fbshipit-source-id: 1027383b6aa0dc6915351332bfbc2d20d540cc4e
Summary: Check that the user's configured TLS credentials exist during client setup and disable HTTP if they aren't present.
Reviewed By: quark-zju
Differential Revision: D15459917
fbshipit-source-id: f20664c6522e47f2960cec1f02ef1a5f4c7e2c8c
Summary:
In some cases the revlog actually computes the hash it uses. In that
situation we don't need to validate it. This is showing up as a performance
hotspot in profiles of the server receiving lots of commits. I'm not sure why
the creation of the hash is not showing up as a hotspot though, which is a bit
strange.
Reviewed By: kulshrax
Differential Revision: D15459193
fbshipit-source-id: 99823dd9e1709dbfecf2626b07f0e4c3ec43686c
Summary: Add several methods to the DownloadStats FFI wrapper to allow Python code to easily access various metrics.
Reviewed By: xavierd
Differential Revision: D15458802
fbshipit-source-id: 9e19d2a9b3fcb6e3a066f040fd110510a2f0d63e
Summary: Make all of the EdenAPI data fetching methods return an object containing metrics. These can then be logged by the Python code.
Reviewed By: xavierd
Differential Revision: D15440641
fbshipit-source-id: 4a5fd090066a9020ae32986ab45ee8fb70c8de53
Summary:
Previously we sorted the trees topologically before inserting them. On
a revlog-backed server, this may mean that trees are written in a different
order from the actual commits. hgsql-backed servers rely on the data being
written in linkrev order so they can be replayed in linkrev order on other
machines, so this broke hgsql replication.
Let's instead sort by linkrev, which will be both topological and satisfy
hgsql's requirements.
Reviewed By: quark-zju
Differential Revision: D15437953
fbshipit-source-id: d4aaaa03b392a6cb6cf1be478aed2583ecb757c5
Summary:
We had this disabled in a config we ship in rpms, but if we want the
tests to work in treeonly mode we want this disabled in all tree cases.
Reviewed By: xavierd
Differential Revision: D15296199
fbshipit-source-id: 0f9751583eefa10c275bd499bb5998adfbe644a4
Summary:
With treemanifest we no longer have manifest rev numbers, so let's drop
them from the templates. This let's us convert a few more tests to treeonly.
In theory automation may parse this, but I kinda doubt anyone parses the
manifest node from this.
Reviewed By: xavierd
Differential Revision: D15296141
fbshipit-source-id: a4d2194bd9604867cd9958509bcd2e6513a72494
Summary:
`hg pushbackup` and `hg isbackedup` can be called from scripts with `HGPLAIN`
set, and so can't be provided by aliases. Instead, provide deprecated wrapper
commands.
Reviewed By: mitrandir77
Differential Revision: D15436286
fbshipit-source-id: 3fbbf9a5fb4d0e8de2026a17c41ee11a139d645f
Summary: Improve debugability of cloud sync by logging updated sync state to the blackbox logs.
Reviewed By: mitrandir77
Differential Revision: D15434890
fbshipit-source-id: c5065455985a48777a855997a99e32ce0b31cc72
Summary:
When syncing, if a locally-available bookmark is synced to a new commit that
has been omitted, remove the local bookmark to ensure that the next cloud sync
doesn't move the bookmark back to where it used to be.
Reviewed By: mitrandir77
Differential Revision: D15414172
fbshipit-source-id: 71aaa2d89f734e4c575c24da2c0ef6b59ca4deaa
Summary:
Now that flush is implemented everywhere, we can replace the use of close with
flush.
Reviewed By: kulshrax
Differential Revision: D15416717
fbshipit-source-id: 5aea730b435e3c2073619ba676e60134f59f87c9
Summary:
Now that MutableDeltaStore::flush is implemented everywhere, let's remove the
close method and replace it with flush where necessary.
Reviewed By: kulshrax
Differential Revision: D15416716
fbshipit-source-id: e66dad66a3aff25e80efb10dc2e22c9878336699
Summary: This is the last history store that didn't implement flush.
Reviewed By: kulshrax
Differential Revision: D15416712
fbshipit-source-id: f091fe697e5015afd5d6e6dc7ae6a1de27bb9b5b
Summary:
A new flush method is added to mutablebasepack that just close and re-init
self.
Reviewed By: kulshrax
Differential Revision: D15416708
fbshipit-source-id: 79cdcb20b51b9688a5e95402057c7da27883003c
Summary:
This hook fires for every commit that is introduced in a pull. When
doing pulls with hundreds of thousands of commits, this introduces a noticable
delay. We don't use this hook anywhere, and it's not particularly scalable, so
let's delete it.
Reviewed By: singhsrb
Differential Revision: D15424697
fbshipit-source-id: 98d76bca703e625adf5be8f6234436befd260fc4
Summary:
Let's move hgsubversion to absolute_import, just to be consistent with the rest
of Mercurial codebase.
Reviewed By: markbt
Differential Revision: D15392154
fbshipit-source-id: e4c32939aff0616790828da508f3feea158669e1
Summary: The flush method is now in the MutableDeltaStore, so move it there.
Reviewed By: kulshrax
Differential Revision: D15416711
fbshipit-source-id: 55eb411e4e4cf98c51813ef29364dcba74dc7f66
Summary: As of D15154509, the data fetching functions in remotefilelog write to shared mutable packs rather than opening new packs. As such, there is no need to return pack paths. In fact, the code has already been updated so that the returned paths are always `None`, so the code removed in this diff is already dead.
Reviewed By: xavierd
Differential Revision: D15419765
fbshipit-source-id: c999d5388042b429a8bda9f72a06569364d8e2e1
Summary: This diff adds an `hg debuggettrees` command that downloads trees from the API server and stores them in a datapack.
Reviewed By: xavierd
Differential Revision: D15301607
fbshipit-source-id: 7820d82d7d021c420e911a6a2e9bfce62b69fa2e
Summary:
The postincoming checks prints out advice of the following forms:
* `(run 'hg heads' to see heads)`
* `(run 'hg heads' to see heads, 'hg merge' to merge)`
* `(run 'hg heads .' to see heads, 'hg merge' to merge)`
* `(run 'hg update' to get a working copy)`
This advice is no longer useful, so remove it.
Reviewed By: DurhamG, farnz
Differential Revision: D15317185
fbshipit-source-id: 50ba576406c96715fa058399da53462be9b7a3bf
Summary:
Improve logging of background backup commands by including the command that was
run and the time it was started in the background backup logs.
Reviewed By: mitrandir77
Differential Revision: D15334879
fbshipit-source-id: 932e91a43033c5cb06c79ede7b5224da2e34eb7d
Summary:
When pulling heads from commit cloud during sync, pull them in small groups
of heads from around the same time, to prevent overloading the server when
pulling a large number of heads.
Reviewed By: mitrandir77
Differential Revision: D15317184
fbshipit-source-id: 5e69eb970b18292a4f5d643b25fac80c90c5d537
Summary:
Lift determination of the correct remote path to connect to up to the top-level
command. This prevents the need to pass around the command-line `**opts` in
all of the commit cloud functions.
Differential Revision: D15295811
fbshipit-source-id: 0e14c1643bad96022c7a01126b447b2a6fcabaed
Summary:
Refactor how commit cloud sync works.
Sync is simplified by delegating backup processing to the existing backup code.
This happens first, which means the user's work is backed up earlier, and the
sync processing can assume that all backed up commits are available in the
cloud storage.
Sync no longer attempts to handle the case where cloud storage has changed.
Instead, backup processing should ensure that all local commits are backed up
to the current cloud storage.
If a commit can't be backed up, then treat this as a normal failure to
sync and ignore that commit for this sync attempt. If a commit can't be
downloaded from the server then the sync fails.
Reviewed By: mitrandir77
Differential Revision: D15295499
fbshipit-source-id: d371c5bf0daedbbe42e8c7d4a0c3d1a40c21a36f
Summary:
Move token location into the `token` module.
Move subscription management into the `subscription` module.
Move obsmarker management into the `obsmarkers` module.
Move everything else to a `util` module which can be disambiguated at import time.
Differential Revision: D15282859
fbshipit-source-id: 7f20c449fd79ffc33b069236a05fc73fac0e7d63
