Summary:
Now that all revlog access happens through the store, let's drop the
duplicate revlog instance on the treemanifestlog. Instead let's get the copy
from the store.
Reviewed By: ryanmce
Differential Revision: D7148819
fbshipit-source-id: 3194247de583c55400b52a3b292efa7b9f9a4649
Summary:
Previously, the hg server used the python treemanifest implementation
to represent and write treemanifests. As part of unifying the code base, and
making it easier to enable treeonly servers in a future patch, let's move all
tree server logic to use cstore treemanifest instead.
Reviewed By: ryanmce
Differential Revision: D7148826
fbshipit-source-id: 81003fbeef94e448ea3aca3380af1b4a9f8fd22d
Summary:
The cstore doesn't contain our ondemand generator, nor our mutable data
store, so it's really not usable yet. Let's disable it for now until we can fix
up the issues that prevent us from using it (like having it report metrics).
Reviewed By: ryanmce
Differential Revision: D7148823
fbshipit-source-id: 5cc46af33c049b751c1c04916aafe0768d80ce7a
Summary:
Once we move to a treeonly world, there won't be any more manifest revlogs.
During the transition we'll still use hybrid manifest's though (the class, and
it will just resolve to treemanifests every time), so we need to support hybrid
manifests that don't actually have a revlog.
This also sets manifestlog._opener. We were already storing that on the other
manifestlog implementations, so we should do it on the main one for consistency
(and because this diff depends on it).
Reviewed By: ryanmce
Differential Revision: D7148821
fbshipit-source-id: 8f278a7bf4c0909423257165df59eae400a05a9e
Summary:
In a future patch we will start using the contents of the
manifestrevlogstore as the place we call add() to add new revisions. In order to
do that, those revlogs must be manifestrevlogs. This patch just changes them to
use the right type.
Reviewed By: quark-zju
Differential Revision: D7148824
fbshipit-source-id: 0d60cd22b041db83b61ccaec3b6172624ea97e42
Summary:
Previously we passed the p1tree directly. It's actually more convient
to pass the p1node, since in a future diff we'll be adding data to the tree
revlogs and they need p1node. This is also safer, since now the caller can't
accidentally pass the wrong p1tree.
Reviewed By: quark-zju
Differential Revision: D7148820
fbshipit-source-id: a489308ebcb2960643b61ff73dc90f1285a94ae3
Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
This adds `progress.bar`, which is a context manager that provides a
time-updated progress bar, analogous to `progress.spinner`.
Reviewed By: ryanmce
Differential Revision: D7211895
fbshipit-source-id: be07d363f10c09164e2324d31a71a7c795257c8a
Summary:
The normal `ui.progress` function is noticably slow due to the debug code, even
when debug is disable. Prevent this from happening by bypassing it completely
when debug is not enabled.
Reviewed By: ryanmce
Differential Revision: D7213855
fbshipit-source-id: ecea0e610ba01ad35c44cd7bd8cac74b8e00621d
Summary:
During profiling of the `hg sparse --refresh` command, I discovered
significant time spent in producing the progress bar. At first I was skeptical,
but then I commented out the progress line and the command took half as long!
I started looking for micro-optiization opportunities.
Differential Revision: D7210617
fbshipit-source-id: 17a11ae572fee48e6d8db33ca75843be0ec2e850
Summary: It can be useful for debugging why hg status is slow
Reviewed By: phillco
Differential Revision: D7212530
fbshipit-source-id: ee1b6979717604fd3cf9819748aa97a61eec0540
Summary:
xdiff has a `xdl_trim_ends` step that removes common lines, unmatchable
lines. That is in theory good, but happens too late - after splitting,
hashing, and adjusting the hash values so they are unique. Those splitting,
hashing and adjusting hash values steps could have noticeable overhead.
For not uncommon cases like diffing two large files with minor differences,
the raw performance of those preparation steps seriously matter. Even
allocating an O(N) array and storing line offsets to it is expensive.
Therefore my previous attempts [1] [2] cannot be good enough since they do
not remove the O(N) array assignment.
This patch adds a preprocessing step - `xdl_trim_files` that runs before
other preprocessing steps. It counts common prefix and suffix and lines in
them (needed for displaying line number), without doing anything else.
Testing with a crafted large (169MB) file, with minor change:
```
open('a','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6000000))
open('b','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6003000))
```
Running xdiff by a simple binary [3], this patch improves the xdiff perf by
more than 10x for the above case:
```
# xdiff before this patch
2.41s user 1.13s system 98% cpu 3.592 total
# xdiff after this patch
0.14s user 0.16s system 98% cpu 0.309 total
# gnu diffutils
0.12s user 0.15s system 98% cpu 0.272 total
# (best of 20 runs)
```
It's still slightly slower than GNU diffutils. But it's pretty close now.
Testing with real repo data:
For the whole repo, this patch makes xdiff 25% faster:
```
# hg perfbdiff --count 100 --alldata -c d334afc585e2 --blocks [--xdiff]
# xdiff, after
! wall 0.058861 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
# xdiff, before
! wall 0.077816 comb 0.080000 user 0.080000 sys 0.000000 (best of 91)
# bdiff
! wall 0.117473 comb 0.120000 user 0.120000 sys 0.000000 (best of 67)
```
For files that are long (ex. commands.py), the speedup is more than 3x, very
significant:
```
# hg perfbdiff --count 3000 --blocks commands.py.i 1 [--xdiff]
# xdiff, after
! wall 0.690583 comb 0.690000 user 0.690000 sys 0.000000 (best of 12)
# xdiff, before
! wall 2.240361 comb 2.210000 user 2.210000 sys 0.000000 (best of 4)
# bdiff
! wall 2.469852 comb 2.440000 user 2.440000 sys 0.000000 (best of 4)
```
The improvement is also seen for the `json` test case mentioned in D7124455.
xdiff's time improves from 0.3s to 0.04s, similar to GNU diffutils.
This patch is also sent as https://phab.mercurial-scm.org/D2686.
[1]: https://phab.mercurial-scm.org/D2631
[2]: https://phab.mercurial-scm.org/D2634
[3]:
```
// Code to run xdiff from command line. No proper error handling.
mmfile_t readfile(const char *path) {
struct stat st; int fd = open(path, O_RDONLY);
fstat(fd, &st); mmfile_t f = { malloc(st.st_size), st.st_size };
ensure(read(fd, f.ptr, st.st_size) == st.st_size); close(fd); return f; }
static int xdiff_outf(void *priv_, mmbuffer_t *mb, int nbuf) { int i;
for (i = 0; i < nbuf; i++) { write(STDOUT_FILENO, mb[i].ptr, mb[i].size); }
return 0; }
int main(int argc, char const *argv[]) {
mmfile_t a = readfile(argv[1]), b = readfile(argv[2]);
xpparam_t xpp = { XDF_INDENT_HEURISTIC, 0 };
xdemitconf_t xecfg = { 3, 0 }; xdemitcb_t ecb = { 0, &xdiff_outf };
xdl_diff(&a, &b, &xpp, &xecfg, &ecb); return 0; }
```
Reviewed By: ryanmce
Differential Revision: D7151582
fbshipit-source-id: 3f2dd43b74da118bd827af4fc5e1bf65be191ad2
Summary:
`mmap_readonly` will be reused in `index.rs` so let's moved it to a shared
utils module.
Reviewed By: DurhamG
Differential Revision: D7208359
fbshipit-source-id: d98779e4e21765ce0e185281c9560245b59b174c
Summary:
Add ScopedFileLock. This is similar to Python's contextmanager.
It's easier to use than the fs2 raw API, since it guarantees the file is
unlocked.
Reviewed By: jsgf
Differential Revision: D7203684
fbshipit-source-id: 5d7beed99ff992466ab7bf1fbea0353de4dfe4f9
Summary: They are simpler than radix entry and similar.
Reviewed By: DurhamG
Differential Revision: D7191652
fbshipit-source-id: b516663567267a2e354748396b44c2ac8ebb691f
Summary: These are Rust structures that map to the file format.
Reviewed By: DurhamG
Differential Revision: D7191366
fbshipit-source-id: 23a4431383be9713e955b74306cd68108eb80536
Summary: Document the format. Actual implementation in later diffs.
Reviewed By: DurhamG
Differential Revision: D7190575
fbshipit-source-id: 243992fd052ca7a9688d54d20694e65daebb9660
Summary:
The append-only index is too different so it's cleaner to cherry-pick code
from radixbuf, instead of modifying radixbuf which would break code
depending on it.
Started by picking the base16 iterator part.
`rustc-test` does not work with buck, and seems to be in an unmaintained
state, so benchmark tests are migrated to criterion.
Reviewed By: DurhamG
Differential Revision: D7189143
fbshipit-source-id: 459a79b4cf16f35d2ff86f11a5980ba1fc627951
Summary:
Filesystem is hard. Append-only sounds like a safe way to write files, but it
only really helps with process crashes. If the OS crashes, it's possible that
other parts of the file gets corrupted. As source control, data integrity check
is important. So bytes not logically touched by appending also needs to be
checked.
Implement a `ChecksumTable` which adds integrity check ability to append-only
files. It's intended to be used by future append-only indexes.
Reviewed By: DurhamG
Differential Revision: D7108433
fbshipit-source-id: 16daf6b8d04bba464f1ee9221716beba69c1d47b
Summary:
First step of a storage-related building block that is in Rust. The goal is
to use it to replace revlog, obsstore and packfiles.
Extern crates that are likely useful are added to reduce future churns.
Reviewed By: DurhamG
Differential Revision: D7108434
fbshipit-source-id: 97ebd9ba69547d876dcecc05e604acdf9088877e
Summary:
Previously, we could be in a merge state (as from `hg update --merge`)
and morestatus did not show any information about the conflicts. Now we will
show conflict info whenever there is a merge state.
Reviewed By: phillco
Differential Revision: D7149411
fbshipit-source-id: e4e03036f3a11bda3edc3628d503a8b3aea412be
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: 9d0671c0b1cd6a63a4acecc614b255c4214328bb
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: d2e350325d7e9005c8ddd5462034040274f790ff
Summary:
It makes sure that adding new options to the commands won't break them. For
example, rage sparse output was broken, and this diff fixes it.
Note that this changes behavior of the rage - if, say, smartlog extension is not enabled on the client, then there will be no output in the rage. Previously it wasn't the case. I think that's not the big problem, because:
a) The extensions are the most common ones and enabled everywhere
b) If they are disabled, then we'll immediately see a problem in hg rage.
However I need to change a test to make it pass. I need to add extensions and change grep output, because `hg rage --preview` has a few lines with `blackbox` in it.
Differential Revision: D7193297
fbshipit-source-id: dde2752ebc7dd3e3edea5c44576d0986f7d18744
Summary:
Add a function that returns command and all the default options already
initialized. It should be used by commands that call other commands. For
example, calling pull inside of update, calling log inside of show etc.
getcmdanddefaultopts has important benefits:
1) It returns "wrapped" command i.e. command with all the overrides applied. On
the other hand, commands.pull doesn't return it.
2) It correctly initializes options to their default value and correctly
changes their name - replace '-' with '_'.
Reviewed By: ryanmce
Differential Revision: D7193296
fbshipit-source-id: e8673bd4e16aad6156498660f2a7ed788ed2cac3
Summary:
For some reason I thought we could defer both of these checks if IMM, but we can only
really skip the local changes check.
Reviewed By: quark-zju
Differential Revision: D7208076
fbshipit-source-id: 10d1ed50b7d7eadcf66cef4d11185690ccd8d07b
Summary:
1. Variable Length Arrays are not supported by MSVC, but since this is a C++ code, we can just use heap allocation
2. Replacing `inet` with portability version
Depends on D7196403
Reviewed By: quark-zju
Differential Revision: D7196605
fbshipit-source-id: a0d88b6e06f255ef648c0b35a99b42ba3bee538a
Summary:
This both fixes semantics and makes `compat.h` a bit more readable.
This because necessary, because we migtrated from external `compat.h` to `mercurial/compat.h` in D7064623.
Reviewed By: DurhamG
Differential Revision: D7196403
fbshipit-source-id: 0005cc2f4e58951adfe8f7f795067da728ad64ae
Summary: sed -i without arguments doesn't work on OSX.
Reviewed By: farnz
Differential Revision: D7195193
fbshipit-source-id: a8eead927c94404a37ce5df956de82d29bc1b6a8
Summary:
In D7001328 we've added a new feature that skips commtis if there are no
changes relative to the sparse checkout. Unfortunately that causes lots of
treepacks downloads and makes bisect unusable. Let's revert the change.
Reviewed By: ryanmce, farnz
Differential Revision: D7182016
fbshipit-source-id: 274b29ca6a7b4c3faf83883b64f5ad3b0289873e
Summary:
This is an alternative fix. But it's generally better to avoid using
generator in this case.
Reviewed By: DurhamG
Differential Revision: D7189763
fbshipit-source-id: 0697f2b80e8ba0a4da7c538e0701a150386410e5
Summary: The option has been gone from the command decorator so it's been inaccessible, in addition to be rarely used, unsupported on laptops, and highly facebook-specific.
Reviewed By: ryanmce
Differential Revision: D7142733
fbshipit-source-id: ee4c833f170e8b8036624ca28cf286e8a0b0cf2d
Summary:
Before this change `hg pullbackup` did not set correct markers on commits.
This change make possible to see what changes already landed even when we are restoring repository from backup.
Before the change `fbclone` + `hg pullbackup` of repo with `C1` commit landed would result in:
```
o o C2
| |
o o C1
| /
|
o
```
after:
```
o o C2
| |
o x C1
| /
|
o
```
Reviewed By: StanislavGlebik
Differential Revision: D7032572
fbshipit-source-id: ffee3c7cc23c24a3df9a89c999c9dd2de226dbff
Summary:
Rewriting a set of commits where there are replacement relationship among the
commits do not have an optimal UX today. For example, `rebase -s A -d Z` or
`metaedit A` in the below graph. B1, B2, C will all be replaced. But the new B1
and B2 replacement won't have the B1 -> B2 relationship, and the "new B1"
appears to be revived.
```
o C
|
x B1 (amended as B2)
|
| o B2
|/
o A o Z
```
One solution is to avoid rebasing `obsolete()::`, as implemented in D7067121
for metaedit. That would result in
```
o C
|
x B1 (amended as B2) o new B2
| |
x A o new A
```
The stack of A, B1, C is forced to break into two parts. This is fine for
power users. But n00b users would wonder why C is left behind. Per discussion
with simpkins at an internal post about the metaedit case, we think a more
linear history is more user-friendly. That is:
```
o new C
|
x new B1 (amended as *new* B2)
|
| o new B2
|/
o new A
```
The stack stays in a same shape.
This diff implements the "copying obsmarkers" behavior at the "createmarkers"
level so everything using that API would get the feature for free, including
metaedit and rebase.
D7067121 is reverted since the new UX is preferred. The test added is for
`metaedit` command, changes to rebase will be added in a later patch.
Differential Revision: D7121487
fbshipit-source-id: fd3c8a96ab434b131fb86d9882ccbdff8f63f05e
Summary:
Sparse profile files are INI files, and semicolons are the traditional comment line starter.
There are already profiles that use the semicolon as a comment, see diffusion/FBS/browse/master/tools/scm/sparse/fbobjc/sandcastle and diffusion/FBS/browse/master/tools/scm/sparse/fbandroid/sandcastle
Reviewed By: farnz
Differential Revision: D7181613
fbshipit-source-id: a42171f6bd6213147c6363f8f359f885af38b8af
Summary:
Recently we unified the client and server code paths a bit, which
can cause the treemanifest server to attempt to do a prefetch (which doesn't
make sense since it has no where to prefetch from). It ends up throwing an Abort
error about not having a remote server configured. The fix is to make the
prefetch path smarter about when it's run on the server and to throw a standard
MissingNodesError instead. That kind of error is already handled in the hybrid
repository case and we just eat it and server the flat manifests like normal.
Once we move to treeonly mode, that error handler will re-raise the exception so
real issues with missing nodes won't be hidden.
Reviewed By: phillco
Differential Revision: D7182283
fbshipit-source-id: 15ed6549d9d7da1fee0570e1fa10338545ed92b1
Summary:
There's a bug where infinitepush attempting to rebundle a bundle that
does not contain trees causes an exception because the server attempts to
prefetch those trees (which fails because there's no where to prefetch from).
This diff just adds a test for that case. The next diff will fix it.
Reviewed By: phillco
Differential Revision: D7182284
fbshipit-source-id: a3fbb576cf3318c81b18943e0f0d466aa65e54fb
Summary:
All of the repos that use commit cloud have remotenames extension enabled, so bookprevnode and pushbackbookmarks parameters are not used. Local bookmarks won't be updated after a push.
We remove "bookprevnode" and "pushbackbookmarks" and functions related to them.
Reviewed By: StanislavGlebik
Differential Revision: D7122411
fbshipit-source-id: 0c6b3bc3f41f5b03d4bb2bc297ae35d77c90fedf
Summary:
The feature that automatically converted flat manifests to trees is
dependent on the hg server not sending flat manifests to treeonly clients
(otherwise it's very, very slow). Since the server rpms got reverted, we need to
backout these changes until the server issues are fixed.
Reviewed By: farnz
Differential Revision: D7181025
fbshipit-source-id: 1e4aad04d15909a3ce4f69313419e50c14bc8c19
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: d31442f71b5a13f5afcd54b019c9bbc85f6f889e
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: 4912ec5ea5097163cede00158df821f116d92c9b
Summary:
This diff adds a config option to tweak deltabase in changegroup. It has 3
options:
- Always null - always use "null" as delta base, effectively make
everything full text
- No external - delta bases cannot be a revision outside the changegroup
- Default - the current behavior: delta bases can be anything that client
thinks the server should have.
This gives Mononoke more time to bake delta related logic, as we can
choose "always null" first, then incrementally increase the complexity.
Reviewed By: phillco
Differential Revision: D7158585
fbshipit-source-id: 5f6d9a78d1108093e8d08b9f296568f4f7e7471b
