Summary:
D15381038 made the `pyrevisionstore` crate a strongly-coupled dependency of the
`bindings` crate. For simplicity, just move the code into the bindings crate.
Reviewed By: xavierd
Differential Revision: D15392547
fbshipit-source-id: 3dfa924b33722760ef7829fbb8659ce0262e6671
Summary:
After D15266191, the bindings crate directly depends on pyrevisionstore, and
since then we have seen really weird compilation errors that magically go away
when some files are touched.
From my observations, here is what I came up with. The pyrevisionstore crate is
both compiled as a dependency of the bindings crate, and as a library to be
used in python. Therefore, its Cargo.toml contains a '[lib]' section, the
presence of this section forces the crate to be compiled into a
"libpyrevisionstore.{rlib,so}", while all the regular crates have a hashed
suffix like "libedenapi-2b9975ec9b865e87.rlib".
None of this would usually matter, but our build system re-uses the build
directory to then compile the pyrevisionstore library. While doing so, it will
re-create the "libpyrevisionstore.{rlib,so}", but not in an identical fashion.
After this, the rest of the build succeeds.
Once a file in the bindings crate is touched, recompiling will only recompile
its file, and not the pyrevisionstore crate, but since
"libpyrevisionstore.{rlib,so}" is different, it now fails to compile...
A previous effort tried to compile each top level target into a separate
directory, but that directly affected some of our tests. Since the underlying
issue is that pyrevisionstore is compiled twice, let's simply not compile it as
a top-level target and simply fold it into the bindings lib. Ideally, we would
want to do the opposite, but let's do that at a later time.
Reviewed By: DurhamG
Differential Revision: D15381038
fbshipit-source-id: 047cfab794371464942f19084ffb9240e836cb40
Summary:
When `remotefilelog.fetchpacks` is enabled, an automatic repack will be
triggered on refresh. Let's add a test to verify this behavior.
Reviewed By: singhsrb
Differential Revision: D15095200
fbshipit-source-id: 89c0a98925e4e53413cf9ea1b1862859c370e12a
Summary:
The refresh function is intended to load on-disk packfiles that aren't yet
present in the cache, but it had several issues that resulted in unexpected
behavior. Mainly, the new files would be added in the wrong order in the cache,
and runonpacks would not update the last used packs.
Differential Revision: D14013785
fbshipit-source-id: da4151b859441f0ba51863f3907852922c255f82
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
When I replace lz4 to rust.lz4, the test failed. The change fixes it. That
also means datapack corruption detection is not that reliable. However, usually
those files are not changed when they are loaded into an in-memory store, so
it's probaby fine.
Also note the python-lz4 used in production has strange behavior when
compressing an empty string:
In [6]: lz4.compressHC('')
Out[6]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [7]: lz4.compress('')
Out[7]: '\x00\x00\x00\x00\xa0#\xd9\x040\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00'
In [9]: rustlz4.compress('')
Out[9]: bytearray(b'\x00\x00\x00\x00')
In [10]: rustlz4.compresshc('')
Out[10]: bytearray(b'\x00\x00\x00\x00')
In [13]: lz4.compress('1')
Out[13]: '\x01\x00\x00\x00\x101'
In [14]: rustlz4.compress('1')
Out[14]: bytearray(b'\x01\x00\x00\x00\x101')
Reviewed By: DurhamG
Differential Revision: D13528199
fbshipit-source-id: 9b3e8674f989062928900766156a97d28262c8cb
Summary:
A future diff will expose this data from the rust pack implementation,
so we need them to be expoed as functions instead of attributes.
Reviewed By: quark-zju
Differential Revision: D8884705
fbshipit-source-id: 401e5c0d24039c68063d4e9171fefefaf7c2bdff
Summary:
Now that the rust data pack is usable from python, let's enable it in
our datapack unit tests. This requires fixing up a few tests.
Reviewed By: farnz
Differential Revision: D8775754
fbshipit-source-id: a6755ca8056169228d8ee2472980f25e3231883e
Summary:
Previously, callers could access the datapack's path via the 'path'
field. In the rust implementation fields won't be accessible, so let's change
this to an accessor function instead.
Reviewed By: singhsrb
Differential Revision: D8773923
fbshipit-source-id: faea2f99e2dff99f3b6300f575bb9ff2ac178a11
Summary:
In a previous diff we added logic to delete corrupt packs in the cache
directories, but the logic only caught issues after the pack was loaded. We also
need to catch issues while loading the pack. This happens when a pack file is 0
length.
Reviewed By: quark-zju
Differential Revision: D8324248
fbshipit-source-id: c2fcb4f69c4627cce316d626e91f752ef562a0d9
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: quark-zju
Differential Revision: D8199128
fbshipit-source-id: 90c1616061bfd7cfbba0b75f03f89683340374d5
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
Summary:
This adds logic to catch unknown exceptions from inside certain pack
files and delete them as a way to remediate issues.
Reviewed By: phillco
Differential Revision: D8129341
fbshipit-source-id: f3d8d414de7eb4a612306d579e643c2094ca93fc
Summary:
As part of producing only one pack file per transaction, we need to
change the mutabledatapack store to allow reads of written-but-not-finalized
data. This will allow us to add the mutabledatapack to the union store, and then
keep it alive for the duration of the transaction.
Reviewed By: quark-zju
Differential Revision: D6944348
fbshipit-source-id: 1e721bd8e07335a9c1f9c6b7595a765ec018c007
Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Summary:
Now they are unnecessary since `run-tests.py` will set up `PYTONPATH`
correctly.
Differential Revision: D6865042
fbshipit-source-id: ca95314f725968e14349a9d916434aa832c596f9
Summary:
Moves the remotefilelog extension into hgext/ and it's tests into
tests/.
I did not fix up all the check-module errors, since it's a ton of work for
very little impact at this point.
Test Plan: make local && ./run-tests.py
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D6680030
