Summary:
The shared cache needs to be completely g+ws so that all members of the group
can write to each directory in it. The old code only applied g+ws to the leaf
directories, so other users aren't able to write to non leaf directories (like
hgcache/7a/83beca8.../ others couldn't write to 7a/)
Test Plan:
Updated a test to view group permissions for the intermediate
directories
Reviewers: #mercurial, ttung, simpkins
Reviewed By: simpkins
Subscribers: lcharignon, net-systems-diffs@, simpkins, mbolin
Differential Revision: https://phabricator.intern.facebook.com/D3523918
Signature: t1:3523918:1467930221:452b11b56a2e69896bf8d2cd0acd7131b41f90d8
Summary:
The pack path logic did not use the correct unix group when
remotefilelog.cachegroup was specified. This fixes that.
Test Plan:
I manually tested it by deleting a pack dir and running repack. This
is hard to create an automated test for since the feature isn't really cross
platform, and we don't have a way to know what groups they have on their
machine.
Reviewers: #sourcecontrol, ttung, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3400756
Tasks: 11584114
Signature: t1:3400756:1465342537:ed023f6dc830117df5e85e294a41486f072714c9
Summary:
Now that repack can clean up old remotefilelog blobs, let's have it also delete
any empty directories that get left behind.
Test Plan: Updated an existing test to cover it
Reviewers: mitrandir, lcharignon, #sourcecontrol, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3385546
Signature: t1:3385546:1464972782:5ca63cf0a5589bb8a537957f50b4bc5ec4e0f0f5
Summary:
Previously a bunch of different places accessed the cachepath through ui.config
directly. This is a problem because we need to resolve any environment variables
in the path, and some spots didn't do this. So let's unify all accesses through
a helper function that takes care of the environment variables.
Test Plan: Added a test
Reviewers: mitrandir, lcharignon, #sourcecontrol, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3385583
Signature: t1:3385583:1464971813:5b9ee5ed3d6ff9f1a78cb9e0269e433844758c9d
On windows, you cannot delete a file if it is readonly. Since remotefilelog
makes heavy use of readonly files, we need to chmod them to be writable before
deleting them.
We only do this on windows, since on unix based systems if the current user is
not the owner of the file (and is just accessing the file via a shared group),
they cannot chmod the file.
Renames some variables to disambiguate the stat module as well.
On windows the grp module is not present, so we need to avoid importing it. This
means the shared group feature of remotefilelog is not supported on windows.
On windows, there is text mode and binary mode for reading and writing files.
Since we're dealing with files as just blob data, we always want binary mode.
Summary:
The mercurial atomic temp logic makes some assumptions about modes and copies
the mode bits from the existing file. If the existing file is readonly, that
means the atomic temp fails to open the file for writing after it's initially
created.
Since we only actually need like 4 lines from the atomic temp code, I just
implemented it on our end, with custom logic for mode handling.
Test Plan: Ran the tests
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D3288589
Signature: t1:3288589:1462993609:653a0a63266b9129b0e18807858e5de02310ecf9
Summary:
Some miscellaneous perf improvements:
- Get rid of unnecessary sorting
- Bail early from remotefilelog reverse filename lookup if there are no keys
- Avoid unnecessary class/enum lookups in ancestor hotpath
- Fix n^2 behavior in history repacking
- Remove unnecessary set() and tuple instantiation in hotpath
- Use __slots__ on repackentry to improve access times
Test Plan:
Ran the tests. Tested the actual perf improvements via 'hg repack' in
a large repo. The n^2 fix caused a massive perf when, but the others shaved off
a number of seconds as well.
Reviewers: #mercurial, ttung, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3257595
Signature: t1:3257595:1462392755:80209fe66c2632d2c16a2840500b0655926e7513
Summary:
`hg gc` is very aggressive in that it deletes any files in the cache that it
determines aren't a needed key, including files it doesn't recognize. Let's
teach it to not delete pack files.
In the future we'll need to make `hg gc` able to garbage collect the contents of
pack files as well.
Test Plan: It's probably fine!
Reviewers: lcharignon, rmcelroy, ttung, quark, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3254744
Signature: t1:3254744:1462308257:c1f932b88abf3337370f16c05c789422ea51b0e1
Summary:
This implements the new markledger and cleanup apis on the existing
remotefilelog stores. These apis are used to tell the repacker what each store
has, and allows each store to cleanup if its data has been repacked.
Test Plan:
Ran repack in conjunction with the future diffs that make use of
these apis.
Reviewers: rmcelroy, ttung, lcharignon, quark, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3249584
Signature: t1:3249584:1462226133:1e8faffc9f6bf8f7c94e6e79aee8865e3c41648c
Summary: Fix check code for various store related files
Test Plan: Ran the tests
Reviewers: #sourcecontrol, mitrandir, ttung
Reviewed By: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3222465
Signature: t1:3222465:1461701300:34560288be4dc921f0252d4ad8fdc9c8d9357e23
Summary:
This adds a api to the store contract that allows the store to return a list of
the name/node pairs that it contains. This will be used to allow a repack
algorithm to list the contents of the store so it can repack it into another
store. The old remotefilelog blob store used namehash+node keys, which is
different from the new store API's name+node keys, so the getfiles()
implementation here has to perform a reverse namehash->name lookup so it can
satisfy the store API contract.
In the remotefilelog basestore implementation, it reads the file names from the
local data directory and the shared cache directory, and reverse resolves the
file name hashes into filenames to produce the list.
Test Plan: ran the tests
Reviewers: #sourcecontrol, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3205321
Signature: t1:3205321:1461751437:a7c44c2bbe153122a3b85b8d82907a112cf77b1a
This was meant to be part of the previous stack of commits, but I pushed the
wrong stack. This patch addresses a number of code review feedback points, the
most visible being to remain 'contains' to something else (in this case
'getmissing').
The old way of fetching from the server required the base store api expose a way
for outside callers to add fetch handlers to the store. This exposed some of the
underlying details of how data is fetched in an unnecessary way and added an
awkward subscription api.
Let's just treat our remote caches as another store we can fetch from, and
require that the over arching configure logic (in shallowrepo.py) can connect
all our stores together in a union store.
The last major piece of functionality that needs to be moved into the new store
is the gc algorithm. This is just a copy paste of the one that exists in
localcache.
