Summary: This part is empty anyway, so there is no point in sending it.
Reviewed By: DurhamG
Differential Revision: D7271595
fbshipit-source-id: 53bdff76d63f925725b00be8d836ffa87bb32470
Summary:
Sometimes infinitepush has to rebundle the data it's trying to send to
the client (like if you pull only part of a bundle). Previously, this was broken
for treeonly bundles because we did not correctly extend the bundlerepo's
data/history store on treemanifest servers. This patch fixes that and updates a
test.
Reviewed By: quark-zju
Differential Revision: D7240343
fbshipit-source-id: a9dd3ae884ace3fa9f4a748fe753fc394e69d6c9
Summary:
Once all clients are in treeonly mode, we can make the server tree only
mode as well and stop producing flat manifests entirely. This will speed up
pushes, save space, etc. This patch adds support for that mode.
Reviewed By: ryanmce
Differential Revision: D7148825
fbshipit-source-id: 15c899a58dd45af0c030c887c53f66347bc32b01
Summary:
Every update to capabilities was requiring changing two spots. Let's
move that into a function. In a future diff we'll add another capability here.
Reviewed By: ryanmce
Differential Revision: D7148827
fbshipit-source-id: 5e8e95c330f2328c7e6c2b15ec708a45a72cf649
Summary:
Currently revlog writing is done through a custom external path. Let's
hide it behind manifestlog.add as well. This will make it simpler to support the
transition to a tree only server in an upcoming diff.
Reviewed By: ryanmce
Differential Revision: D7148822
fbshipit-source-id: af43a639395e4686cf4b3d654d4408e0747c1b47
Summary:
Now that all revlog access happens through the store, let's drop the
duplicate revlog instance on the treemanifestlog. Instead let's get the copy
from the store.
Reviewed By: ryanmce
Differential Revision: D7148819
fbshipit-source-id: 3194247de583c55400b52a3b292efa7b9f9a4649
Summary:
Previously, the hg server used the python treemanifest implementation
to represent and write treemanifests. As part of unifying the code base, and
making it easier to enable treeonly servers in a future patch, let's move all
tree server logic to use cstore treemanifest instead.
Reviewed By: ryanmce
Differential Revision: D7148826
fbshipit-source-id: 81003fbeef94e448ea3aca3380af1b4a9f8fd22d
Summary:
The cstore doesn't contain our ondemand generator, nor our mutable data
store, so it's really not usable yet. Let's disable it for now until we can fix
up the issues that prevent us from using it (like having it report metrics).
Reviewed By: ryanmce
Differential Revision: D7148823
fbshipit-source-id: 5cc46af33c049b751c1c04916aafe0768d80ce7a
Summary:
Previously we passed the p1tree directly. It's actually more convient
to pass the p1node, since in a future diff we'll be adding data to the tree
revlogs and they need p1node. This is also safer, since now the caller can't
accidentally pass the wrong p1tree.
Reviewed By: quark-zju
Differential Revision: D7148820
fbshipit-source-id: a489308ebcb2960643b61ff73dc90f1285a94ae3
Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: 9d0671c0b1cd6a63a4acecc614b255c4214328bb
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: d2e350325d7e9005c8ddd5462034040274f790ff
Summary:
Recently we unified the client and server code paths a bit, which
can cause the treemanifest server to attempt to do a prefetch (which doesn't
make sense since it has no where to prefetch from). It ends up throwing an Abort
error about not having a remote server configured. The fix is to make the
prefetch path smarter about when it's run on the server and to throw a standard
MissingNodesError instead. That kind of error is already handled in the hybrid
repository case and we just eat it and server the flat manifests like normal.
Once we move to treeonly mode, that error handler will re-raise the exception so
real issues with missing nodes won't be hidden.
Reviewed By: phillco
Differential Revision: D7182283
fbshipit-source-id: 15ed6549d9d7da1fee0570e1fa10338545ed92b1
Summary:
The feature that automatically converted flat manifests to trees is
dependent on the hg server not sending flat manifests to treeonly clients
(otherwise it's very, very slow). Since the server rpms got reverted, we need to
backout these changes until the server issues are fixed.
Reviewed By: farnz
Differential Revision: D7181025
fbshipit-source-id: 1e4aad04d15909a3ce4f69313419e50c14bc8c19
Summary:
When converting an incoming bundle, rebuilding the flat text every time
is very expensive. Since we're usually converting a series of manifests that
build upon each other, let's cache the previous flat texts.
Reviewed By: quark-zju
Differential Revision: D7126948
fbshipit-source-id: d31442f71b5a13f5afcd54b019c9bbc85f6f889e
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: 4912ec5ea5097163cede00158df821f116d92c9b
Summary:
Verify had some logic that checked the length of the changelog and
manifest to decide if either existed. This allowed for simplifying certain error
messages (like not reporting all the broken changelog manifest pointers if the
manifest was simply gone, and just reporting the manifest was gone).
Unfortunately, in future changelog and manifest implementations len() will be an
expensive function, so let's just get rid of that optimization.
This fixes hg verify in a treeonly repository.
Reviewed By: quark-zju
Differential Revision: D7127168
fbshipit-source-id: 8ddc3dfe3c3c913efd4b7af5fc9715a3e48b60a1
Summary:
Currently if you push or pull a bunch of commits between peers we will
include all the trees as part of the push. If the source repo doesn't have all
the necessary trees, it will go to the server to get them. Since the other
machine can just as easily go to the server (and probably won't need most of
those trees anyways), lets just have the source client send all draft trees and
skip the public commits,
Reviewed By: phillco
Differential Revision: D7141623
fbshipit-source-id: 6d33ae9d4c9cc32bf6dfa76f733c87c06890d719
Summary:
A future diff will make it so we can send only draft trees instead of
all trees. To prepare for this, let's move the cansendtrees logic to
shallowbundle (since it will be used by both shallowbundle and by treemanifest)
and change it to return an enum.
Reviewed By: quark-zju
Differential Revision: D7141624
fbshipit-source-id: 34c78b0d1cdb6f8d86a99fb74665e80b2af12c5c
Summary:
The addmemtree function was originally created to allow making
pending trees readable (i.e. trees that have been written but not committed).
Now that all writes are immediately readable, we don't need addmemtree anymore.
Reviewed By: quark-zju
Differential Revision: D7101314
fbshipit-source-id: f9ecabf366ba7bc59abee42d264e17ab66b7f0dd
Summary:
Historically treeonly clients were ignoring flat manifests sent in
bundles, but with a recent change they will now try to recreate those manifests
when they receive such a bundle (so we don't lose any data when a user does 'hg
unbundle' on a flat-only bundle). This means we need to stop sending flat
manifests from the server to treeonly clients.
Differential Revision: D7083028
fbshipit-source-id: e4580b00a8be96fbef0ee624529c58f41cfa2752
Summary:
Eventually all the clients using the tree manifest will operate in
treeonly mode. In treeonly mode, we only read from tree manifests and
therefore, manifest related operations would fail for commits which only have
flat manifest. Example of such commits are old draft commits which were created
before the existence of tree manifest. One way to resolve this is to
automatically convert any flat manifests we come across to tree manifests. This
commit achieves that same.
Differential Revision: D7083033
fbshipit-source-id: 092fc7852ffc6d1b4130b5a1fc8d9e124cef4fcb
Summary:
We were already hacking the ui object onto the manifestlog via the
treemanifest extension, but it didn't cover all cases. Let's just put it on the
manifestlog via the constructor and get rid of the hack. An upcoming change
required this
Reviewed By: phillco
Differential Revision: D7083032
fbshipit-source-id: 4c577cb80193a9c4799853d75a71c26719348e8c
Summary:
Previously we added the mutable packs to the union stores directly when
the mutable packs were first created. This meant we didn't have fine grained
control over what order the mutable pack was in the store. Since we want the
mutable packs to be before the remote store and before the upcoming ondemand
store, we need to insert it into the list at store initialization time.
We can't insert the mutable packs themselves, since they are opened and closed
all the time, but we can insert a proxy class that knows how to find the current
mutable pack on the manifestlog.
Differential Revision: D7083045
fbshipit-source-id: c08a877783c6bb6b95beac4e40544880a6bd3a8f
Summary:
In some cases the generating store could result in infinite recursion
if the generate function didn't actually produce the desired value. Let's add a
context manager to guard against this.
Reviewed By: mjpieters
Differential Revision: D7083030
fbshipit-source-id: 5e0037addbf2ba9fa9d4e222291cc4543da5c659
Summary:
Adds the mutable stores to the union store when they are created. This
will allow reads to access data that has been written but not finalized. This
will be important in later patches for converting a series of trees as part of a
single transaction.
Differential Revision: D7083031
fbshipit-source-id: 25ddbb1dbd29ad6b4164733b6d893b9c69d9d65e
Summary:
Now that _writeclientmanifest is basically just calling
manifestlog.add, let's get rid of it.
Differential Revision: D7083029
fbshipit-source-id: eee18cefd5a6ae3d95bba58b419364fc9fdb15b3
Summary:
As part of moving the manifest mutable pack logic to be used optionally
without transactions, let's move the lifetime logic
out to the repo.transaction() function instead of being part of the primary manifest
write logic.
Differential Revision: D7083035
fbshipit-source-id: 9947d78db61a41896cc8bdfaaa20504ebb03a125
Summary:
Previously the mutable packs were kept on the transaction and had a
lifetime that corresponded with it. In a future patch we want to enable mutable
packs that span for longer than the lifetime of the transaction, so let's move
the mutable pack maintenance on the manifestlog. For now the lifetime is still
maintained by the transaction, but a future diff will change that as well (and
will get rid of _writeclientmanifest entirely).
Differential Revision: D7083034
fbshipit-source-id: 3735eadfc18e5dd1015bfb82dbf5b9e9e6965cdf
Summary:
As part of a series of diffs that will remove the need for addmemtree,
let's remove it from the _writeclientmanifest function so we can replace
_writeclientmanifest with a cleaner interface that doesn't call addmemtree.
Reviewed By: ryanmce
Differential Revision: D7083037
fbshipit-source-id: 51885ec547df5aa21e66afe36eb1f3224c3eae66
Summary:
We now have a unified place for writing trees to packs, so let's move
the fastmanifest code to use it.
Reviewed By: ryanmce
Differential Revision: D7083043
fbshipit-source-id: f5fc312b7614906b917fc7ca10866705fbd47aac
Summary:
There are a few tweaks necessary to keep the tests passing in upcoming
diffs. Most of these are deleted by the end of the series, but having them now
makes the diffs cleaner.
Reviewed By: ryanmce
Differential Revision: D7083040
fbshipit-source-id: 70a09ba1dd573f9cd5c24080f2811db21cc54938
Summary:
In a future patch we're going to make the manifestlog control it's own
writes and the lifetime of its write objects. We are currently hacking up a
normal manifest to attach a treemanifest to it, which makes it awkward for it to
maintain the lifetimes of both the normal manifest and the treemanifest. Let's
move the existing logic into a class of its own so we can attach more logic in
the future.
Reviewed By: ryanmce
Differential Revision: D7083039
fbshipit-source-id: 7646b787c36af76650e77f8e6192cd266093c8ea
Summary:
D6867913 refactored the server side flat manifest to tree manifest
conversion code path. This commit builds on that and introduces support for the
same on the client side.
Reviewed By: singhsrb
Differential Revision: D6867928
fbshipit-source-id: 7198d3a5ff9834482c0e7423d3ab501423d72a40
Summary:
D6867913 did a bunch of refactoring which can be extended for the write
manifest wrapper. This commit achieves the same.
Reviewed By: singhsrb
Differential Revision: D6867921
fbshipit-source-id: c86cf1d54ecd3503127d91acf7a114d4d99f50ff
Summary:
This commit is essentially a refactor to further break down the
process of converting flat manifest to tree manifest. This enables further code
refactoring and reusability in later commits.
Reviewed By: ryanmce
Differential Revision: D6867913
fbshipit-source-id: 86b2ec41932eef6297169f0667190e03caab35a1
Summary:
Introducing class generatingdatastore to represent all stores like
remotetreedatastore which generate the trees when they are not found in the
shared store, add them to the shared store, and thereafter, repeat the original
operation on the shared store expecting it to succeed.
Reviewed By: ryanmce
Differential Revision: D6867898
fbshipit-source-id: 59af9d4833e402b36c35b6247d319888d19bef66
Summary:
The plan is to eventually subclass remotetreedatastore and this
refactor sets us up nicely for that.
Reviewed By: ryanmce
Differential Revision: D6867885
fbshipit-source-id: 2eb2ad0068183b8f972e1a187a65854d331e09d9
Summary:
writepending needs to return False so the transaction mechanics know there is no
further work to be done here. This was just something caught during development.
Reviewed By: ryanmce
Differential Revision: D6867879
fbshipit-source-id: 73d49b06c5379f9a3388d42fbb454b5eba534a20
Summary:
If there is already a committed flat manifest node, then we need to
use that while writing the corresponding client tree manifest so that the tree
manifest is referable from the flat hash.
Reviewed By: ryanmce
Differential Revision: D6867873
fbshipit-source-id: e86a9aad4fa113dfc3fecc0fbb753dee77f26f24
Summary:
The logic for writing client tree manifests can be reused and thus, it
makes sense to move this out into a separate method.
Reviewed By: ryanmce
Differential Revision: D6867863
fbshipit-source-id: 5d24f181dcf4fdf8752d96c12c550c447898ce61
Summary:
A future patch will need the ability to convert a linknode to a linkrev and vice
versa without having access to the changelog (when converting a flat manifest
into a treemanifest we need to get the linknode from the flat manifest, and all
we're given is a linkrev). Let's stick the changelog on the manifestlog. Since
they are generally created together and have the same lifetime, this seems safe.
Reviewed By: ryanmce
Differential Revision: D7034091
fbshipit-source-id: df6915f894c6ef975ffd214c8e229f1085141fbc
Summary:
The union content store
- iterates through all the stores it has until the current store has the
content.
- Or, it fails eventually if none of the stores have the content.
It does so by relying on the current store throwing a KeyError if it doesn't
have the content.
`remotetreedatastore` was throwing the MissingNodesError which means any
remaining stores after it would not even get a chance to look for the content.
This commit addresses the same.
Reviewed By: ryanmce
Differential Revision: D6867854
fbshipit-source-id: 784df195efcbe16f2e716968f3d93159afff6206
Summary:
Update the treemanifest extension to also add a 'gettreepack()' function to
localpeer, and add the 'gettreepack' capability to localpeer if the repository
supports acting as a treemanifest server.
Also update the _addmanifestgroup() wrapper to only forbid it from being called
in a treemanifest server repository. When pulling from a local treemanifest
server repository this function may need to be invoked in the client
repository.
Reviewed By: DurhamG
Differential Revision: D6980931
fbshipit-source-id: 61519c7deeaa0347650323a0002b89597e91d9cc
Summary:
Previously we would only do ondemand downloading of tree blobs when
accessing data contents. When accessing just history we would just fail if the
data wasn't available locally. This adds the remote store to the history union
store so we can get history remotely.
Reviewed By: singhsrb
Differential Revision: D7003434
fbshipit-source-id: 839f8e84be35779ccb146d13ce3e1d6d1e7f46bd
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Summary:
Previously we weren't able to commit right after creating a treeonly
repository. This was caused by the code attempting to read the null tree from
the store, which doesn't exist. The fix is to handle the null case and return an
empty tree instead of trying to look it up in the store. We already do this in a
number of other cases, so this was just a missing one.
Reviewed By: singhsrb
Differential Revision: D6930919
fbshipit-source-id: e227612be2640282eb997f4d563102d86f0be43a
Summary:
When building a changegroup, sometimes we need to access the list of
files changed by each commit. To do so we need to inspect the manifest.
Previously this would end up downloading each tree one-by-one, producing a bunch
of pack files. With this patch we now do one bulk download at the very
beginning.
Reviewed By: quark-zju
Differential Revision: D6873076
fbshipit-source-id: b916c15efca0770129340f798d3e7b165da6aec9
Summary:
A future diff will add tree prefetching during changegroup creation. This
requires access to the tree prefetch function from the shallowbundle class.
let's move the prefetch functions onto the repo object in preparation for that.
Reviewed By: quark-zju
Differential Revision: D6873055
fbshipit-source-id: 18de0ee0f6ab566587509f9e23ebb2e5779ed1c8
