Xavier Deguillard baacd19749 remotefilelog: enforce a limit on the packfiles size ... Summary: Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to limit the size of the hgcache. This will be particularly challenging for Sandcastle/Quicksand as they already see hgcache over 100GB. The long-term plan is switching to IndexedLog based stores with a log rotate functionality to control the cache size. In the meantime, we can implement a basic logic to enforce the size of the hgcache that simply remove packfiles once the cache is over the configured size. One complication of this method is that several concurrent Mercurial processes could be running and accessing the packfiles being removed. In this case, we can split the packfiles in 2 categories: ones created a while back, and new ones. Removing packfiles from the first case, lookups will simply raise a KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is acceptable. The second category belongs to ones that were just created by downloading them from Memcache/Mononoke, and the code strongly assume that they will stick around. A failure at this point will not be recovered. One way of fixing this would be to handle these failures properly and simply retry, the other is to not remove new packfiles. A time of 10 minutes was chosen to categorize the packfiles. Reviewed By: quark-zju Differential Revision: D15014076 fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd		2019-04-23 13:00:24 -07:00
..
absorb	mutation: add absorb support	2019-04-11 02:45:16 -07:00
amend	commands: update help text for amend	2019-04-19 11:16:56 -07:00
commitcloud	visibility: support cloud sync	2019-04-11 07:49:20 -07:00
convert	subrepo: remove subrepo support	2019-03-11 10:43:55 -07:00
extlib	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
fastannotate	hgext: reformat using black	2019-03-13 18:35:09 -07:00
fastmanifest	manifest: drop manifestv2 support	2019-03-26 13:32:45 -07:00
fsmonitor	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
hgevents	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
hggit	hggit: stop git-updatemeta walk when we finish commits with extras	2019-03-20 14:33:58 -07:00
hgsubversion	codeformat - remove weird looking + for string concat on the same line	2019-03-13 11:40:26 -07:00
highlight	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
infinitepush	infinitepush: fix the help text for scratch bookmark commands	2019-04-16 13:47:36 -07:00
lfs	hg: fix lfs and journal extension postshare wrapping	2019-02-07 18:58:40 -08:00
memcommit	memcommit: enforce target parent to be original parent	2019-03-13 11:50:42 -07:00
p4fastimport	codeformat - remove weird looking + for string concat on the same line	2019-03-13 11:40:26 -07:00
perfsuite	metrics: slightly change the API	2019-03-20 22:49:18 -07:00
pushrebase	pushrebase: throw if prechangegroup fails	2019-04-22 17:10:15 -07:00
remotefilelog	remotefilelog: enforce a limit on the packfiles size	2019-04-23 13:00:24 -07:00
repogenerator	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
treemanifest	remotefilelog: avoid double close of mutable packs	2019-04-22 10:15:42 -07:00
__init__.py	codemod: move Python packages to edenscm	2019-01-28 18:35:41 -08:00
arcdiff.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
automv.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
blackbox.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
catnotate.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
checkmessagehook.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
chistedit.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
churn.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
cleanobsstore.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
clienttelemetry.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
clindex.pyx	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
clonebundles.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
commitextras.py	test-revset: do not use "hg branch" directly	2019-02-14 17:44:39 -08:00
conflictinfo.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
copytrace.py	copytrace: fix an issue where undesirable entries are reported	2019-02-25 12:43:20 -08:00
crdump.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
debugcommitmessage.py	subrepo: remove subrepo support	2019-03-11 10:43:55 -07:00
debugshell.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
dialect.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
directaccess.py	branchmap: drop revbranchcache and other unused logic	2019-02-22 21:02:41 -08:00
dirsync.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
disablesymlinks.py	disablesymlinks: add extension to disable symlinks	2019-02-06 08:41:34 -08:00
drop.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
edrecord.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
eol.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
errorredirect.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
extdiff.py	subrepo: remove subrepo support	2019-03-11 10:43:55 -07:00
extorder.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
extutil.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
fastlog.py	fastlog: patch follow revset instead	2019-04-16 22:19:19 -07:00
fbconduit.py	fbconduit: fix resolving phabricator callsign for WWW	2019-03-13 18:35:09 -07:00
fbhistedit.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
fixcorrupt.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
generic_bisect.py	codemod: move Python packages to edenscm	2019-01-28 18:35:41 -08:00
githelp.py	codeformat - remove weird looking + for string concat on the same line	2019-03-13 11:40:26 -07:00
gitlookup.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
gitrevset.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
globalrevs.py	globalrevs: fail quickly for invalid searches during globalrev fast lookup	2019-04-02 11:23:19 -07:00
gpg.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
grepdiff.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
grpcheck.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
hgsql.py	hgsql: use fetchall method in committodb	2019-04-13 03:20:27 -07:00
hiddenerror.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
histedit.py	histedit: fix call to hex in loginfo	2019-04-12 01:57:56 -07:00
infinitepushbackup.py	infinitepushbackup: exclude hidden commits from backup heads	2019-04-11 02:45:15 -07:00
interactiveui.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
journal.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
linkrevcache.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
logginghelper.py	logginghelper: log normalized repo name	2019-03-04 00:05:00 -08:00
lz4revlog.py	rust: move bindings to a single python extension	2019-02-01 17:53:22 -08:00
mergedriver.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
morecolors.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
morestatus.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
myparent.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
nointerrupt.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
ownercheck.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
patchrmdir.pyx	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
phabdiff.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
phabstatus.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
phrevset.py	phrevset: avoid crashing on null graphql reply	2019-04-09 07:53:43 -07:00
progressfile.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
pullcreatemarkers.py	visibility: pullcreatemarkers should exclude hidden commits	2019-04-11 07:49:21 -07:00
purge.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
rage.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
rebase.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
remotenames.py	remotenames: make push with tracking bookmark work with infinitepush	2019-04-17 08:41:12 -07:00
reset.py	visibility: update visibility when commits are modified	2019-03-08 03:05:58 -08:00
sampling.py	sampling: silence BaseException.message deprecation warning	2019-04-22 17:58:07 -07:00
schemes.py	hgext: reformat using black	2019-03-13 18:35:09 -07:00
sendunbundlereplay.py	mononoke: preserve more output from hg side in the sync job	2019-04-10 18:05:37 -07:00
share.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
shelve.py	shelve: add shelvename template keyword	2019-04-15 05:44:49 -07:00
sigtrace.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
simplecache.py	simplecache: add checksum to simplecache values	2019-02-20 13:56:33 -08:00
smartlog.py	smartlog: move ancestor calcuation to smartlog revset	2019-03-21 17:33:45 -07:00
sparse.py	blackbox: clean up blackbox logging and log more during rage	2019-04-23 02:48:50 -07:00
sshaskpass.py	sshaskpass: workaround sys.executable	2019-02-26 18:19:43 -08:00
stablerev.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
stat.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
traceprof.pyx	codemod: move Python packages to edenscm	2019-01-28 18:35:41 -08:00
tweakdefaults.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
undo.py	visibility: add undo support	2019-03-27 04:49:14 -07:00
whereami.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
win32mbcs.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00
win32text.py	codemod: import from the edenscm package	2019-01-29 17:25:32 -08:00