Commit Graph

21 Commits

Author SHA1 Message Date
Stanislau Hlebik
2a0e8205d4 extutil: create new package and move runshellfast here
Summary:
I'm going to use runshellfast in infinitepush.
To avoid copy-paste let's move it to the separate package.
Note: fastmanifest also has runshellfast but it's implementation
is a bit different and seems that it doesn't work with remotefilelog.
So I'll leave it alone for now.

Test Plan: python run-tests.py -j20

Reviewers: #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4175836
2016-11-21 00:52:30 -08:00
Durham Goode
3707e186b6 treemanifest: add local pack store to unionstore
Summary:
In a future patch we will start writing user local tree data into a local
directory. This patches add the local store to the union store so the contents
will be accessible once we start writing it.

It also renames manifest to manifests for consistency with the other store names
(like 'packs').

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4055827

Signature: t1:4055827:1477059457:0d5b0d999b47d88c73f5ab2721d8d27deacc01bc
2016-10-21 11:02:22 -07:00
Durham Goode
4235752853 remotefilelog: rename getpackpath to getcachepackpath
Summary:
In a future diff we will be introducing packs into .hg/store, so we need to
differentiate between cache packs and local packs. This patch renames
getpackpath to getcachepackpath. A future diff will add getlocalpackpath.

This exposed a pyflakes error, so we fix that too.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4055815

Signature: t1:4055815:1477059415:e0221557bbeec6701820c826f00390d3a71cd2d3
2016-10-21 11:02:20 -07:00
Durham Goode
3b7beae747 ctree: add ctreemanifest hg extension
Summary:
Adds the initial extension that sets up the ctreemanifest. It currently relies
on the fastmanifest extension to hook into all the manifest APIs to construct
ctreemanifests.

Test Plan:
In a future patch, I was able to run 'hg manifests' on a commit and
have it return the manifest contents by reading the treemanifest.

Reviewers: #fastmanifest, ttung

Reviewed By: ttung

Subscribers: ttung

Differential Revision: https://phabricator.intern.facebook.com/D3755327

Signature: t1:3755327:1472114482:0c5862cba68ed4db643d28c2fae01f33f5352970
2016-08-29 16:19:52 -07:00
Durham Goode
15fcba5c21 cachegroup: fix directory permissions for shared cache
Summary:
The shared cache needs to be completely g+ws so that all members of the group
can write to each directory in it. The old code only applied g+ws to the leaf
directories, so other users aren't able to write to non leaf directories (like
hgcache/7a/83beca8.../ others couldn't write to 7a/)

Test Plan:
Updated a test to view group permissions for the intermediate
directories

Reviewers: #mercurial, ttung, simpkins

Reviewed By: simpkins

Subscribers: lcharignon, net-systems-diffs@, simpkins, mbolin

Differential Revision: https://phabricator.intern.facebook.com/D3523918

Signature: t1:3523918:1467930221:452b11b56a2e69896bf8d2cd0acd7131b41f90d8
2016-07-07 15:58:59 -07:00
Olivier Trempe
d8d662c766 shallowutil: windows compatibility for readonly files
On Windows, os.rename cannot rename readonly files and cannot overwrite
destination if it already exists. Create small wrappers to handle these cases.
2016-07-05 00:30:42 -07:00
Laurent Charignon
9a1fb623cc shallowutil: add missing import
Summary:
Before this patch, we were not importing mercurial.error, this was
causing a crash when calling error.Abort. This patch adds the missing import.

Test Plan: Tests pass, and add a new test

Reviewers: durham

Differential Revision: https://phabricator.intern.facebook.com/D3457086
2016-06-20 15:18:14 -07:00
Jeroen Vaelen
07efaadb9d [remotefilelog] use hashlib to compute sha1 hashes
Summary:
hg-crew's c27dc3c3122 and c27dc3c3122^ were breaking our extensions:

```
$ hg log -r c27dc3c3122^
changeset:   9010734b79911d2d2e7405d91a4df479b35b3841
user:        Augie Fackler <raf@durin42.com>
date:        Thu, 09 Jun 2016 21:12:33 -0700
s.ummary:     cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
```

```
$ hg log -r c27dc3c3122
changeset:   0d55a7b8d07bf948c935822e6eea85b044383f00
user:        Augie Fackler <raf@durin42.com>
date:        Thu, 09 Jun 2016 21:13:23 -0700
s.ummary:     util: drop local aliases for md5, sha1, sha256, and sha512
```

I did a grep over facebook-hg-rpms to see what was affected:
```
$ grep "util\.\(md5\|sha1\|sha256\|sha512\)" -r ~/facebook-hg-rpms
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basestore.py:            sha = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basestore.py:                sha = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/shallowutil.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/shallowutil.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/debugcommands.py:    filekey = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        namehash = util.sha1(name).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        node = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        files = ((util.sha1(filename).digest(), offset, size)
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/fileserverclient.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/fileserverclient.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basepack.py:        self.sha = util.sha1()
/home/jeroenv/facebook-hg-rpms/remotefilelog/tests/test-datapack.py:        return util.sha1(content).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/tests/test-histpack.py:        return util.sha1(content).digest()
Binary file /home/jeroenv/facebook-hg-rpms/hg-crew/.hg/store/data/mercurial/revlog.py.i matches
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:            return util.sha1(fh.read()).hexdigest()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:    sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/mutable-history/hgext/simple4server.py:        sha = util.sha1()
/home/jeroenv/facebook-hg-rpms/mutable-history/hgext/evolve.py:        sha = util.sha1()
```
This diff is part of the fix.

Test Plan:
Ran the tests.
```
$MERCURIALRUNTEST -S -j 48 --with-hg ~/local/facebook-hg-rpms/hg-crew/hg
```

Reviewers: #sourcecontrol, ttung

Differential Revision: https://phabricator.intern.facebook.com/D3440041

Tasks: 11762191
2016-06-15 15:48:16 -07:00
Durham Goode
d860baf210 cachegroup: fix pack path use of cachegroup
Summary:
The pack path logic did not use the correct unix group when
remotefilelog.cachegroup was specified. This fixes that.

Test Plan:
I manually tested it by deleting a pack dir and running repack. This
is hard to create an automated test for since the feature isn't really cross
platform, and we don't have a way to know what groups they have on their
machine.

Reviewers: #sourcecontrol, ttung, rmcelroy

Reviewed By: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3400756

Tasks: 11584114

Signature: t1:3400756:1465342537:ed023f6dc830117df5e85e294a41486f072714c9
2016-06-08 09:09:06 -07:00
Durham Goode
6f3d6c53f5 utils: unify cachepath access through a util function
Summary:
Previously a bunch of different places accessed the cachepath through ui.config
directly. This is a problem because we need to resolve any environment variables
in the path, and some spots didn't do this. So let's unify all accesses through
a helper function that takes care of the environment variables.

Test Plan: Added a test

Reviewers: mitrandir, lcharignon, #sourcecontrol, ttung, simonfar

Reviewed By: simonfar

Subscribers: simonfar

Differential Revision: https://phabricator.intern.facebook.com/D3385583

Signature: t1:3385583:1464971813:5b9ee5ed3d6ff9f1a78cb9e0269e433844758c9d
2016-06-03 09:45:58 -07:00
Durham Goode
02c488a9f6 util: close temporary file descriptor
When we moved away from the core mercurial atomic temp logic, we forgot to close
the file descriptor that was opened by mkstemp.
2016-05-20 08:41:04 -07:00
Olivier Trempe
9f5acf47bd windows: chmod file to be writable before deleting
On windows, you cannot delete a file if it is readonly. Since remotefilelog
makes heavy use of readonly files, we need to chmod them to be writable before
deleting them.

We only do this on windows, since on unix based systems if the current user is
not the owner of the file (and is just accessing the file via a shared group),
they cannot chmod the file.

Renames some variables to disambiguate the stat module as well.
2016-05-20 08:37:49 -07:00
Olivier Trempe
a1a32e5f37 windows: use binary mode for reading and writing files
On windows, there is text mode and binary mode for reading and writing files.
Since we're dealing with files as just blob data, we always want binary mode.
2016-05-20 08:08:57 -07:00
Durham Goode
05ceb8b419 store: basic wire protocol for bundle delivery
Summary:
This adds a new wire protocol command to allow clients to request a set
of file contents and histories from the server and receive them in pack format.
It's pretty simple and always returns all the history for every node requested
(which is a bit overkill), but it's labeled v1 and we can iterate on it.

Test Plan: Added a test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277212

Signature: t1:3277212:1463421279:459cc84265502175b47df293647aab7e7a830185
2016-05-16 10:59:09 -07:00
Durham Goode
e7a294e65f utils: change writefile to use custom atomic temp logic
Summary:
The mercurial atomic temp logic makes some assumptions about modes and copies
the mode bits from the existing file. If the existing file is readonly, that
means the atomic temp fails to open the file for writing after it's initially
created.

Since we only actually need like 4 lines from the atomic temp code, I just
implemented it on our end, with custom logic for mode handling.

Test Plan: Ran the tests

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Differential Revision: https://phabricator.intern.facebook.com/D3288589

Signature: t1:3288589:1462993609:653a0a63266b9129b0e18807858e5de02310ecf9
2016-05-11 13:33:06 -07:00
Durham Goode
ce493adf83 repack: add --background option
Summary:
This allows triggering a repack that can be run in the background. In the future
we will trigger this automatically under certain circumstances (like too many
pack files).

Test Plan: Added a test

Reviewers: #mercurial, ttung, quark

Reviewed By: quark

Subscribers: quark

Differential Revision: https://phabricator.intern.facebook.com/D3261161

Signature: t1:3261161:1462398568:5ae25f3e5a9acd0f4b34490b34a62be33cc69e3c
2016-05-04 14:53:23 -07:00
Durham Goode
c3c047f0b7 Move sortnodes into shallowutil
Summary:
This is a generic topological sort and will be useful in the upcoming repacking
code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3204124

Signature: t1:3204124:1461260520:e1cb5c9d496f11e5f44e0cdbc5ba851b1573d2e1
2016-04-26 15:10:38 -07:00
Durham Goode
84bc49f25d checkcode: fix shallowrepo, shallowutil, and setup.py
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221375

Signature: t1:3221375:1461648312:7dbdd59e6370cb32b90d864a623d8066028741e7
2016-04-26 13:00:31 -07:00
Jun Wu
ead8969797 Fix missing errno import
Summary: Apparently we need to `import errno` in `shallowutil.py`

Test Plan: Code Review

Reviewers: #sourcecontrol, ttung, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D3195117

Signature: t1:3195117:1461031210:424912a96448a2a8cb37197f006cfa95d4ab1cb1
2016-04-18 19:04:58 -07:00
Durham Goode
ece19111e0 ioutil: rename ioutil to shallowutil
The old name was not very descriptive. There's already a shallowutil, so let's
just use that.
2016-04-04 16:26:12 -07:00
Siddharth Agarwal
d731468f70 [bundle2] insert ourselves into the cg1packer class hierarchy and fix up the packermap
Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.

Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`

Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D1671738

Tasks: 5568731

Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
2014-11-07 18:35:52 -08:00