Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: 9d0671c0b1cd6a63a4acecc614b255c4214328bb
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: d2e350325d7e9005c8ddd5462034040274f790ff
Summary:
Recently we unified the client and server code paths a bit, which
can cause the treemanifest server to attempt to do a prefetch (which doesn't
make sense since it has no where to prefetch from). It ends up throwing an Abort
error about not having a remote server configured. The fix is to make the
prefetch path smarter about when it's run on the server and to throw a standard
MissingNodesError instead. That kind of error is already handled in the hybrid
repository case and we just eat it and server the flat manifests like normal.
Once we move to treeonly mode, that error handler will re-raise the exception so
real issues with missing nodes won't be hidden.
Reviewed By: phillco
Differential Revision: D7182283
fbshipit-source-id: 15ed6549d9d7da1fee0570e1fa10338545ed92b1
Summary:
The feature that automatically converted flat manifests to trees is
dependent on the hg server not sending flat manifests to treeonly clients
(otherwise it's very, very slow). Since the server rpms got reverted, we need to
backout these changes until the server issues are fixed.
Reviewed By: farnz
Differential Revision: D7181025
fbshipit-source-id: 1e4aad04d15909a3ce4f69313419e50c14bc8c19
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: d31442f71b5a13f5afcd54b019c9bbc85f6f889e
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: 4912ec5ea5097163cede00158df821f116d92c9b
Summary:
Verify had some logic that checked the length of the changelog and
manifest to decide if either existed. This allowed for simplifying certain error
messages (like not reporting all the broken changelog manifest pointers if the
manifest was simply gone, and just reporting the manifest was gone).
Unfortunately, in future changelog and manifest implementations len() will be an
expensive function, so let's just get rid of that optimization.
This fixes hg verify in a treeonly repository.
Reviewed By: quark-zju
Differential Revision: D7127168
fbshipit-source-id: 8ddc3dfe3c3c913efd4b7af5fc9715a3e48b60a1
Summary:
Currently if you push or pull a bunch of commits between peers we will
include all the trees as part of the push. If the source repo doesn't have all
the necessary trees, it will go to the server to get them. Since the other
machine can just as easily go to the server (and probably won't need most of
those trees anyways), lets just have the source client send all draft trees and
skip the public commits,
Reviewed By: phillco
Differential Revision: D7141623
fbshipit-source-id: 6d33ae9d4c9cc32bf6dfa76f733c87c06890d719
Summary:
A future diff will make it so we can send only draft trees instead of
all trees. To prepare for this, let's move the cansendtrees logic to
shallowbundle (since it will be used by both shallowbundle and by treemanifest)
and change it to return an enum.
Reviewed By: quark-zju
Differential Revision: D7141624
fbshipit-source-id: 34c78b0d1cdb6f8d86a99fb74665e80b2af12c5c
Summary:
The addmemtree function was originally created to allow making
pending trees readable (i.e. trees that have been written but not committed).
Now that all writes are immediately readable, we don't need addmemtree anymore.
Reviewed By: quark-zju
Differential Revision: D7101314
fbshipit-source-id: f9ecabf366ba7bc59abee42d264e17ab66b7f0dd
Summary:
Historically treeonly clients were ignoring flat manifests sent in
bundles, but with a recent change they will now try to recreate those manifests
when they receive such a bundle (so we don't lose any data when a user does 'hg
unbundle' on a flat-only bundle). This means we need to stop sending flat
manifests from the server to treeonly clients.
Differential Revision: D7083028
fbshipit-source-id: e4580b00a8be96fbef0ee624529c58f41cfa2752
Summary:
Eventually all the clients using the tree manifest will operate in
treeonly mode. In treeonly mode, we only read from tree manifests and
therefore, manifest related operations would fail for commits which only have
flat manifest. Example of such commits are old draft commits which were created
before the existence of tree manifest. One way to resolve this is to
automatically convert any flat manifests we come across to tree manifests. This
commit achieves that same.
Differential Revision: D7083033
fbshipit-source-id: 092fc7852ffc6d1b4130b5a1fc8d9e124cef4fcb
Summary:
We were already hacking the ui object onto the manifestlog via the
treemanifest extension, but it didn't cover all cases. Let's just put it on the
manifestlog via the constructor and get rid of the hack. An upcoming change
required this
Reviewed By: phillco
Differential Revision: D7083032
fbshipit-source-id: 4c577cb80193a9c4799853d75a71c26719348e8c
Summary:
Previously we added the mutable packs to the union stores directly when
the mutable packs were first created. This meant we didn't have fine grained
control over what order the mutable pack was in the store. Since we want the
mutable packs to be before the remote store and before the upcoming ondemand
store, we need to insert it into the list at store initialization time.
We can't insert the mutable packs themselves, since they are opened and closed
all the time, but we can insert a proxy class that knows how to find the current
mutable pack on the manifestlog.
Differential Revision: D7083045
fbshipit-source-id: c08a877783c6bb6b95beac4e40544880a6bd3a8f
Summary:
In some cases the generating store could result in infinite recursion
if the generate function didn't actually produce the desired value. Let's add a
context manager to guard against this.
Reviewed By: mjpieters
Differential Revision: D7083030
fbshipit-source-id: 5e0037addbf2ba9fa9d4e222291cc4543da5c659
Summary:
Adds the mutable stores to the union store when they are created. This
will allow reads to access data that has been written but not finalized. This
will be important in later patches for converting a series of trees as part of a
single transaction.
Differential Revision: D7083031
fbshipit-source-id: 25ddbb1dbd29ad6b4164733b6d893b9c69d9d65e
Summary:
Now that _writeclientmanifest is basically just calling
manifestlog.add, let's get rid of it.
Differential Revision: D7083029
fbshipit-source-id: eee18cefd5a6ae3d95bba58b419364fc9fdb15b3
Summary:
As part of moving the manifest mutable pack logic to be used optionally
without transactions, let's move the lifetime logic
out to the repo.transaction() function instead of being part of the primary manifest
write logic.
Differential Revision: D7083035
fbshipit-source-id: 9947d78db61a41896cc8bdfaaa20504ebb03a125
Summary:
Previously the mutable packs were kept on the transaction and had a
lifetime that corresponded with it. In a future patch we want to enable mutable
packs that span for longer than the lifetime of the transaction, so let's move
the mutable pack maintenance on the manifestlog. For now the lifetime is still
maintained by the transaction, but a future diff will change that as well (and
will get rid of _writeclientmanifest entirely).
Differential Revision: D7083034
fbshipit-source-id: 3735eadfc18e5dd1015bfb82dbf5b9e9e6965cdf
Summary:
As part of a series of diffs that will remove the need for addmemtree,
let's remove it from the _writeclientmanifest function so we can replace
_writeclientmanifest with a cleaner interface that doesn't call addmemtree.
Reviewed By: ryanmce
Differential Revision: D7083037
fbshipit-source-id: 51885ec547df5aa21e66afe36eb1f3224c3eae66
Summary:
We now have a unified place for writing trees to packs, so let's move
the fastmanifest code to use it.
Reviewed By: ryanmce
Differential Revision: D7083043
fbshipit-source-id: f5fc312b7614906b917fc7ca10866705fbd47aac
Summary:
There are a few tweaks necessary to keep the tests passing in upcoming
diffs. Most of these are deleted by the end of the series, but having them now
makes the diffs cleaner.
Reviewed By: ryanmce
Differential Revision: D7083040
fbshipit-source-id: 70a09ba1dd573f9cd5c24080f2811db21cc54938
Summary:
In a future patch we're going to make the manifestlog control it's own
writes and the lifetime of its write objects. We are currently hacking up a
normal manifest to attach a treemanifest to it, which makes it awkward for it to
maintain the lifetimes of both the normal manifest and the treemanifest. Let's
move the existing logic into a class of its own so we can attach more logic in
the future.
Reviewed By: ryanmce
Differential Revision: D7083039
fbshipit-source-id: 7646b787c36af76650e77f8e6192cd266093c8ea
Summary:
D6867913 refactored the server side flat manifest to tree manifest
conversion code path. This commit builds on that and introduces support for the
same on the client side.
Reviewed By: singhsrb
Differential Revision: D6867928
fbshipit-source-id: 7198d3a5ff9834482c0e7423d3ab501423d72a40
Summary:
D6867913 did a bunch of refactoring which can be extended for the write
manifest wrapper. This commit achieves the same.
Reviewed By: singhsrb
Differential Revision: D6867921
fbshipit-source-id: c86cf1d54ecd3503127d91acf7a114d4d99f50ff
Summary:
This commit is essentially a refactor to further break down the
process of converting flat manifest to tree manifest. This enables further code
refactoring and reusability in later commits.
Reviewed By: ryanmce
Differential Revision: D6867913
fbshipit-source-id: 86b2ec41932eef6297169f0667190e03caab35a1
Summary:
Introducing class generatingdatastore to represent all stores like
remotetreedatastore which generate the trees when they are not found in the
shared store, add them to the shared store, and thereafter, repeat the original
operation on the shared store expecting it to succeed.
Reviewed By: ryanmce
Differential Revision: D6867898
fbshipit-source-id: 59af9d4833e402b36c35b6247d319888d19bef66
Summary:
The plan is to eventually subclass remotetreedatastore and this
refactor sets us up nicely for that.
Reviewed By: ryanmce
Differential Revision: D6867885
fbshipit-source-id: 2eb2ad0068183b8f972e1a187a65854d331e09d9
Summary:
writepending needs to return False so the transaction mechanics know there is no
further work to be done here. This was just something caught during development.
Reviewed By: ryanmce
Differential Revision: D6867879
fbshipit-source-id: 73d49b06c5379f9a3388d42fbb454b5eba534a20
Summary:
If there is already a committed flat manifest node, then we need to
use that while writing the corresponding client tree manifest so that the tree
manifest is referable from the flat hash.
Reviewed By: ryanmce
Differential Revision: D6867873
fbshipit-source-id: e86a9aad4fa113dfc3fecc0fbb753dee77f26f24
Summary:
The logic for writing client tree manifests can be reused and thus, it
makes sense to move this out into a separate method.
Reviewed By: ryanmce
Differential Revision: D6867863
fbshipit-source-id: 5d24f181dcf4fdf8752d96c12c550c447898ce61
Summary:
A future patch will need the ability to convert a linknode to a linkrev and vice
versa without having access to the changelog (when converting a flat manifest
into a treemanifest we need to get the linknode from the flat manifest, and all
we're given is a linkrev). Let's stick the changelog on the manifestlog. Since
they are generally created together and have the same lifetime, this seems safe.
Reviewed By: ryanmce
Differential Revision: D7034091
fbshipit-source-id: df6915f894c6ef975ffd214c8e229f1085141fbc
Summary:
The union content store
- iterates through all the stores it has until the current store has the
content.
- Or, it fails eventually if none of the stores have the content.
It does so by relying on the current store throwing a KeyError if it doesn't
have the content.
`remotetreedatastore` was throwing the MissingNodesError which means any
remaining stores after it would not even get a chance to look for the content.
This commit addresses the same.
Reviewed By: ryanmce
Differential Revision: D6867854
fbshipit-source-id: 784df195efcbe16f2e716968f3d93159afff6206
Summary:
Update the treemanifest extension to also add a 'gettreepack()' function to
localpeer, and add the 'gettreepack' capability to localpeer if the repository
supports acting as a treemanifest server.
Also update the _addmanifestgroup() wrapper to only forbid it from being called
in a treemanifest server repository. When pulling from a local treemanifest
server repository this function may need to be invoked in the client
repository.
Reviewed By: DurhamG
Differential Revision: D6980931
fbshipit-source-id: 61519c7deeaa0347650323a0002b89597e91d9cc
Summary:
Previously we would only do ondemand downloading of tree blobs when
accessing data contents. When accessing just history we would just fail if the
data wasn't available locally. This adds the remote store to the history union
store so we can get history remotely.
Reviewed By: singhsrb
Differential Revision: D7003434
fbshipit-source-id: 839f8e84be35779ccb146d13ce3e1d6d1e7f46bd
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Summary:
Previously we weren't able to commit right after creating a treeonly
repository. This was caused by the code attempting to read the null tree from
the store, which doesn't exist. The fix is to handle the null case and return an
empty tree instead of trying to look it up in the store. We already do this in a
number of other cases, so this was just a missing one.
Reviewed By: singhsrb
Differential Revision: D6930919
fbshipit-source-id: e227612be2640282eb997f4d563102d86f0be43a
Summary:
When building a changegroup, sometimes we need to access the list of
files changed by each commit. To do so we need to inspect the manifest.
Previously this would end up downloading each tree one-by-one, producing a bunch
of pack files. With this patch we now do one bulk download at the very
beginning.
Reviewed By: quark-zju
Differential Revision: D6873076
fbshipit-source-id: b916c15efca0770129340f798d3e7b165da6aec9
Summary:
A future diff will add tree prefetching during changegroup creation. This
requires access to the tree prefetch function from the shallowbundle class.
let's move the prefetch functions onto the repo object in preparation for that.
Reviewed By: quark-zju
Differential Revision: D6873055
fbshipit-source-id: 18de0ee0f6ab566587509f9e23ebb2e5779ed1c8
Summary:
setuptreestores caters to a lot of configurations through the `ui`
instance. It makes sense to store the `ui` instance once instead of referring
to it through `repo.ui` every time.
Reviewed By: DurhamG, quark-zju
Differential Revision: D6867843
fbshipit-source-id: 3ffca85332ec4ae3ec8a54bf0dacc7e7cf423b53
Summary:
Many processes may be reading and writing from this cache at once, so
let's be sure to use atomic temp.
Reviewed By: quark-zju
Differential Revision: D6811428
fbshipit-source-id: 5c6c157a60e294ab77cc5cb735c5705b7cddff87
Summary:
We're seeing high load on the server when many clients are fetching treemanifest
entries. A lot of this stems from the cost of opening revlogs. Let's introduce a
simple on-disk cache that let's us avoid opening revlogs as much as possible.
Reviewed By: quark-zju
Differential Revision: D6797698
fbshipit-source-id: c131aa57be333dcb2e4453c6a1f9f4e24532a7ee
Summary:
Port of D6798134 to fbsource. It eliminates module-import failures as well as errors like this:
```
mercurial.error.ForeignImportError: hgext.extlib.treedirstate: /home/phillco/.local/lib/python2.7/site-packages/hgext/extlib/treedirstate.so lives outside /..../hg
```
....that block other tests, like test-help.t
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D6799259
fbshipit-source-id: b77d1b565dbf52165e0847002be498648658e064
Summary: Moves the extension itself.
Test Plan:
make local && ./run-tests.py
A future patch fixes the tests once
fastmanifest is also moved.
Reviewers: phillco, #mercurial
Reviewed By: phillco
Differential Revision: https://phabricator.intern.facebook.com/D6684580
Signature: 6684580:1515521896:68a4d648d9f5433d6eeda8be01740fbe12abf44e