Summary:
There was a bug where if we tried to convert a flat manifest into a
tree while the flat manifest commit was being committed, the fast path
conversion would produce incorrect results because it tried to access the commit
data. If len(changelog) == 5, then changelog.revision(5) returns the nullid
data, which caused the bug.
This is absurb behavior from the changelog, so I will change that in a future
diff.
Reviewed By: phillco, singhsrb
Differential Revision: D8270372
fbshipit-source-id: 27bbfb2e54de35553b880954f06e76fe4fc0d47b
Summary:
We were encountering issues where readonly connections where failing
because the serve was returning a Permission denied error while fetching trees.
It turns out the ondemand tree generation was attempting to open a transaction
and create trees on the server. Since the server should have all the trees
anyway, we shouldn't do this ondemand conversion.
We still need on demand conversions for infinitepush bundles, so we'll leave
that in.
Reviewed By: singhsrb
Differential Revision: D8226459
fbshipit-source-id: 2d05faefbe3ae92ccb81072e012673f29c9cf0d9
Summary:
Previously we weren't including p2 when creating trees, this resulted
in incorrect conversions.
Reviewed By: phillco
Differential Revision: D8152253
fbshipit-source-id: c4f8c79b40532c5162b15032962fbc6a78d44b5f
Summary:
This switch converting flat manifests to tree manifests to use a fast
path where we avoid decompressing the whole manifest whenever possible. This
will speed up tree conversions tremendously.
Reviewed By: phillco
Differential Revision: D8130357
fbshipit-source-id: aa91f9445216d2c5cb60d51c21ae0f6636359d9c
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
This makes the signature match _getfastflatdiff, so we can unify them
in the next diff.
Reviewed By: phillco
Differential Revision: D8130358
fbshipit-source-id: 4bc51b79d0432b3347dfd7996c9f586f0c65999f
Summary:
This refactors the new tree creation logic from a single function to
two functions, one that computes the differences in (added, removed) format, and
another that accepts adds/removes and applies them to a copy of the parent tree.
This will help in a future diff where we want to unify code paths to use the
fast diff path.
Reviewed By: phillco
Differential Revision: D8122900
fbshipit-source-id: 82699dc4bbf69dd565f8564e71f8a0726659313e
Summary:
We already had logic that could compute a new tree from a manifest node
quickly (using the manifest revdiff function to avoid reading entire manifests).
Let's refactor that to a function so in a future diff we can unify it with
_getflatdiff so all paths can take advantage of the performance.
Reviewed By: phillco
Differential Revision: D8122469
fbshipit-source-id: 5cb315327356cece9b2fed3e2c57088179ef838e
Summary:
A future diff will optimize the performance of this particular
function.
Reviewed By: phillco
Differential Revision: D8121843
fbshipit-source-id: 22187aec658cef7b2f724fae113a651c24d79851
Summary:
A future patch will improve the performance of convert flat manifests
to tree manifests by relying on the stored manifest delta instead of hydrating
the whole manifest. To do this, we need to change _getnewtree to not depend on
taking the entire flat manifests. So let's refactor it to take a diff instead,
and in a future patch we will optimize how we compute that diff.
Reviewed By: phillco
Differential Revision: D8121627
fbshipit-source-id: 41bbba6e98cd605ee185618b5f99883a9e0fd7cf
Summary:
This adds logic to catch unknown exceptions from inside certain pack
files and delete them as a way to remediate issues.
Reviewed By: phillco
Differential Revision: D8129341
fbshipit-source-id: f3d8d414de7eb4a612306d579e643c2094ca93fc
Summary:
Make the (user-facing) log for prefetching trees say how many commits (or which
commit) it is prefetching for.
When remotefilelog.debug is enabled, print out the manifest identity when
fetching a tree during normal operation.
Reviewed By: quark-zju
Differential Revision: D8056555
fbshipit-source-id: dc0c7bc5c949a0674d5f553661e736bc545134b4
Summary:
We had a bug where the cache store had a value that didn't correspond
with it's key. Let's add the key to the value so we can ensure they are always
paired correctly.
Reviewed By: phillco
Differential Revision: D8065151
fbshipit-source-id: 01be3a3432d1b4f5e9a234b35b8315c37b1869c3
Summary:
Previously if we were pulling between peers, we would not send trees
for public commits, since we assumed the receiving end could fetch them from the
server.
It turns out there are situations where public commits are actually local only
to the client (like when pulling directly from subversion before the server
does), so let's change our heuristic to send trees if they exist in the source
client's local data store.
This also required updating the server side tree pack generation logic to handle
the case when the client sends a base manifest node that the server doesn't know
how to handle. We now ignore those bad base manifests.
Reviewed By: phillco
Differential Revision: D7992502
fbshipit-source-id: 8ddd1836e6f325363ad2bb6eae34ddbbc84c2d11
Summary:
Previously we only enabled on demand tree generation when treeonly was
set, but we also need it when sendtrees is set since we'll need to generate
trees for sending.
Reviewed By: quark-zju
Differential Revision: D7927856
fbshipit-source-id: a69d6c7920a92e4f90bdcd1d04aad9ef59e9c778
Summary:
This logs timing data for:
- update
- merge.update
- mergedriver
- applyupdates
- workers
- fetches
- pulls
- remotefilelog prefetches
- treemanifest prefetches
- status
- dirstate walks
- fsmonitor walks
- watchman queries
Hopefully this will let us narrow down where time is going in a number of cases
where the profile data is ambigiuous or hard to come by.
Reviewed By: quark-zju
Differential Revision: D7681026
fbshipit-source-id: e6fe65c9a4d2f4e128f62ccb767a7cbe73b2649a
Summary:
Previously, when searching for base trees (i.e. trees that we already
have that we can tell the server so it can send us only parts we don't have) we
did a naive changelog scan for public commits. If the phases were messed up and
a local commit was marked public, this would result in us sending the server a
base commit it didn't have which would error out.
Let's make two changes to fix this.
1. Let's scan by history instead of by revision number. This will
work in the future when we don't have rev numbers, and will hopefully also get
us closer base trees. It should also mean that if we fetch a server commit we
are less likely to encounter a local accidentally-public commit.
2. Let's only use commits that are ancestors of a remotename. This will help us
avoid local accidentally-public commits as well.
Reviewed By: quark-zju
Differential Revision: D7593363
fbshipit-source-id: 1df8f4fbc56a4c11ccb6d444c3d08297206ad91f
Summary:
We weren't passing bundlecaps to the backfill pull, so we were getting
bundle1. Let's fix that
Reviewed By: quark-zju
Differential Revision: D7591167
fbshipit-source-id: 111e2967190dca17764c84e6d7be850644f222fd
Summary:
Previously the connectionpool was a remotefilelog specific concept. We
want to start sharing connections between pull and prefetches, so let's move it
to core Mercurial.
Reviewed By: ryanmce, phillco
Differential Revision: D7480670
fbshipit-source-id: 1b2eff3b0e61a815709ffaec35df802eeda0c24b
Summary:
Previously we would throw an unhandled exception which would cause
stderr output on the client. Since we sometimes want to try the server silently
from the client, let's return errors as a bundle2 part that we can handle more
gracefully.
Reviewed By: quark-zju
Differential Revision: D7448101
fbshipit-source-id: f0c5d0af56e718f0403ed9a18d66ad8be150d5b8
Summary:
Since error parts pass the message and hint as parameters instead of
payload, they are limited to 255 characters. Let's add a helper function to
enforce this.
Reviewed By: quark-zju
Differential Revision: D7448104
fbshipit-source-id: 33d47a21e7159b6c4bd72cad9669568b92a51e34
Summary:
Previously we relied on the list of files created by strip to decide
what directories to strip. It turns out this list wasn't adequate though, since
it's possible for directories to change even when files do not. For example,
during a merge if different files were added on different sides of the merge,
the files don't change during the merge but their containing directory does.
This diff special cases merge commits to make sure all the directories affected
by the merge are included in the strip.
Reviewed By: quark-zju
Differential Revision: D7409156
fbshipit-source-id: 9bf67eefb70189300c29db60d9945a7f608dfdda
Summary:
In uisetup we wrap a bunch of functions. We need to make sure they each
handle the case where treemanifest is not actually enabled for that repo. This
can happen in cases like a peer-to-peer pull where one repo has treemanifest
enabled and one does not.
Reviewed By: quark-zju
Differential Revision: D7368084
fbshipit-source-id: 152a448fb8a902e8c8f8ac4e806ab53514fadaa7
Summary:
Previously we were storing the changelog on the manifestlog and using
it to resolve linkrevs before serializing them. It turns out the changelog can
be invalidated at a different rate than the manifestlog, so we could encounter
issues where the manifestlog held a reference to the old changelog.
To fix this, let's hold a reference to the repo and access the changelog from
there when we need it. This introduces a circular reference between the
manifestlog and the repo, but it's probably fine for now until we can get rid of
the need for changelog invalidation.
Reviewed By: singhsrb
Differential Revision: D7360321
fbshipit-source-id: 2317c7fcd6b307a50b64f0c5df97dda2955f3e21
Summary:
Previously, when looking for a treemanifest we'd inspect the flat
manifest revlog to get a hint for which nearby manifests might exist. This
assumed the data was in the flat manifest though, and if it wasn't it threw an
exception which caused us to not look at the server for the tree.
This commit fixes the case where the flat manifest doesn't have the commit, so
it can still look at the server.
Reviewed By: quark-zju
Differential Revision: D7343608
fbshipit-source-id: 57d5da9a8b8d71e33df9a8a0647b7bfe8815e37b
Summary:
We have some repo's whose first commit didn't not add any files, which
means they have nullid as their manifest pointer which breaks backfilltrees.
This patch fixes that.
Reviewed By: singhsrb
Differential Revision: D7340678
fbshipit-source-id: 85b048b81c1861fd48d5cc082d414271aee7283b
Summary:
A prior diff optimized push/pull to not send public trees between
peers, since those trees can be downloaded from the main server. Let's be
careful when sending data to the main server and always send everything.
In the future we should add validation on the server that the received data is
complete, but Mercurial doesn't currently do that today.
Differential Revision: D7296253
fbshipit-source-id: 49513685d19991a70d66da1d734ddae23491ed0c
Summary:
This allows pushing a treeonly pack to a server without using
pushrebase.
Differential Revision: D7295686
fbshipit-source-id: b0bfe4fbb04bc765e57f1db82909fa1ae7b3063b
Summary:
Previously we were just putting nullid as the linknode in client side
trees, because when the trees were added the changelog hadn't been written yet,
so we didn't know the linknode. This diff updates manifestlog.add to pass
linkrevs to mutablehistorypack, which get resolved at serialization time into
the appropriate linknode.
Reviewed By: ryanmce
Differential Revision: D7280104
fbshipit-source-id: bbc8a7bfad31da296a1b65973875ba2e1e1f7a95
Summary:
The linkrev parameter is a named parameter, so let's use it's name
instead of relying on the order. This is just a cleanliness diff.
Reviewed By: ryanmce
Differential Revision: D7280108
fbshipit-source-id: 88a02d0b5ccb1f586fdc01fb3cb2cd44a70e82e9
Summary:
Previously converttotrees relied on being passed a linkrev to pass to
manifestlog.add(). If a linkrev wasn't provided though, it would pass None and
no linknode was stored. If a linknode is available, let's prefer that, and only
use the linkrev when we have to.
Reviewed By: ryanmce
Differential Revision: D7280109
fbshipit-source-id: d17015789a213fab6ff57563216e6943d2525fee
Summary:
Previously we were just putting nullid as the linknode in client side
trees, because when the trees were added the changelog hadn't been written yet,
so we didn't know the linknode. As part of fixing this, let's add linknode to
the manifestlog.add api. A future diff will fill this in with the right value.
Reviewed By: ryanmce
Differential Revision: D7280103
fbshipit-source-id: dad3d2456e0cbde39392bf77af53ceeaadb49913
Summary:
Previously pushrebase pushes would send a tree pack part as well as a
rebase pack part. The tree pack part was ignored. Let's make receiving tree pack
parts on the server an error, and instead make the client not send them if it's
doing a pushrebase push.
A future diff will come back and add support for non-pushrebase push of tree
packs. This diff just makes use sure that the new logic won't run during
pushrebase.
Reviewed By: ryanmce
Differential Revision: D7267802
fbshipit-source-id: cabff989c178afb3706b77a03a299bcf3195f289
Summary:
Currently infinitepush relies on the ability to send flat manifests to
tree clients and the clients doing the conversion. This relies on a hack where
we don't pass the bundlecaps to the infinitepush changegroup creator, so it
doesn't realize it's talking to a treeonly client and therefore sends flat
manifests to a treeonly client. This is hacky. To fix this, let's start by
making it possible for the server to generate trees on demand from the flat
manifests. This will also be useful later when the server is treeonly, so it can
ondemand generate old flat bundles into a treeonly one.
A later diff will then pass the appropriate bundlecaps to the rebundle
changegroup function.
Reviewed By: ryanmce
Differential Revision: D7256554
fbshipit-source-id: 40d5d9537a5e11b3d186c8ec9a2579261f1b772e
Summary:
In a future diff we'll want to allow converting flat manifests to tree
manifests during infinitepush. To do so, let's allow writing to in memory packs
on bundlerepositories.
Reviewed By: StanislavGlebik
Differential Revision: D7256563
fbshipit-source-id: 10ec58d1171b7882d6db9a916c50a19bc11dbcb4
Summary: This part is empty anyway, so there is no point in sending it.
Reviewed By: DurhamG
Differential Revision: D7271595
fbshipit-source-id: 53bdff76d63f925725b00be8d836ffa87bb32470
Summary:
Sometimes infinitepush has to rebundle the data it's trying to send to
the client (like if you pull only part of a bundle). Previously, this was broken
for treeonly bundles because we did not correctly extend the bundlerepo's
data/history store on treemanifest servers. This patch fixes that and updates a
test.
Reviewed By: quark-zju
Differential Revision: D7240343
fbshipit-source-id: a9dd3ae884ace3fa9f4a748fe753fc394e69d6c9
Summary:
Once all clients are in treeonly mode, we can make the server tree only
mode as well and stop producing flat manifests entirely. This will speed up
pushes, save space, etc. This patch adds support for that mode.
Reviewed By: ryanmce
Differential Revision: D7148825
fbshipit-source-id: 15c899a58dd45af0c030c887c53f66347bc32b01
Summary:
Every update to capabilities was requiring changing two spots. Let's
move that into a function. In a future diff we'll add another capability here.
Reviewed By: ryanmce
Differential Revision: D7148827
fbshipit-source-id: 5e8e95c330f2328c7e6c2b15ec708a45a72cf649
Summary:
Currently revlog writing is done through a custom external path. Let's
hide it behind manifestlog.add as well. This will make it simpler to support the
transition to a tree only server in an upcoming diff.
Reviewed By: ryanmce
Differential Revision: D7148822
fbshipit-source-id: af43a639395e4686cf4b3d654d4408e0747c1b47
Summary:
Now that all revlog access happens through the store, let's drop the
duplicate revlog instance on the treemanifestlog. Instead let's get the copy
from the store.
Reviewed By: ryanmce
Differential Revision: D7148819
fbshipit-source-id: 3194247de583c55400b52a3b292efa7b9f9a4649
Summary:
Previously, the hg server used the python treemanifest implementation
to represent and write treemanifests. As part of unifying the code base, and
making it easier to enable treeonly servers in a future patch, let's move all
tree server logic to use cstore treemanifest instead.
Reviewed By: ryanmce
Differential Revision: D7148826
fbshipit-source-id: 81003fbeef94e448ea3aca3380af1b4a9f8fd22d
Summary:
The cstore doesn't contain our ondemand generator, nor our mutable data
store, so it's really not usable yet. Let's disable it for now until we can fix
up the issues that prevent us from using it (like having it report metrics).
Reviewed By: ryanmce
Differential Revision: D7148823
fbshipit-source-id: 5cc46af33c049b751c1c04916aafe0768d80ce7a
Summary:
Previously we passed the p1tree directly. It's actually more convient
to pass the p1node, since in a future diff we'll be adding data to the tree
revlogs and they need p1node. This is also safer, since now the caller can't
accidentally pass the wrong p1tree.
Reviewed By: quark-zju
Differential Revision: D7148820
fbshipit-source-id: a489308ebcb2960643b61ff73dc90f1285a94ae3
Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: 9d0671c0b1cd6a63a4acecc614b255c4214328bb
