Summary:
Now that we have perftracing infra, let's trace a bunch of likely
problem spots.
Reviewed By: sfilipco
Differential Revision: D14426367
fbshipit-source-id: 354a241aa9ac5d75d34062a9838d581b4f46746f
Summary:
Most of the repacks are background repacks, and most of the complaints are
coming from laptops users. Thanks to the rust repack, most of time during
repack is now spent in repacking loosefiles. While repacking them is expensive,
testing whether data is in a loosefile and obtaining it is actually pretty
fast. Packfiles have the opposite issue, repacking them is fast, but finding
data in them may be expensive if a lot of them are present.
Based on this, it makes sense to repack packfiles more frequently than
loosefiles. At first, the newly added option will be used to turn-off loosefile
repack for laptop users. A less frequent loosefile repack will be implemented
in a later patch.
Reviewed By: DurhamG
Differential Revision: D14586986
fbshipit-source-id: 5bc5c839cf8d2d78bcc4ffa016bbe3cf1b2ef3f7
Summary: Log data about round-trip count, and object count for files, trees, and SSH calls.
Differential Revision: D14515776
fbshipit-source-id: cce416fd7dccdad3c73a9f1751a04ddac0d2c507
Summary:
Reading a line over a pipe involves reading every character of the line
individually. This is extremely inefficient and slow, which cause prefetch to
be overly slow when most of the data isn't in memcache.
Using buffered reads tries to read 4096 bytes at a time, drastically reducing
the cost of reading a missing path/node pair from memcache.
Reviewed By: ikostia
Differential Revision: D14507063
fbshipit-source-id: e0910d7a303e15fe2d3c61fe2739e6c13370058f
Summary: D14387734 added 2 new arguments to the `httprequestpacks` function, but didn't update the callsite to pass those arguments. This diff fixes the problem.
Differential Revision: D14468592
fbshipit-source-id: 7e573838916067fc2cc12204ea1da460eb3955c8
Summary:
Now that hg_memcache_client will voluntarily exit when hg terminates, we no
longer need to wait for hg_memcache_client to finish before exiting hg.
Reviewed By: DurhamG
Differential Revision: D14396510
fbshipit-source-id: 7e73d9b70d481e58a0c47cd0f408580e6d548fd9
Summary:
Operations like `hg log -p` will inherently cause many requests to
hg_memcache_client. This will force many small packfiles to be created which
will significantly slow down future invocation of hg.
Now that `hg repack --incremental --packsonly` is fast, we can afford to run it
when mercurial exit after a prefetch operation was run.
Differential Revision: D14387735
fbshipit-source-id: 45f89f1120458c8b2471a1c55cafb6bc87263dd0
Summary:
When using packfiles, history and data are in different files, and thus it's
possible to only fetch one.
For now, besides the requests coming from contentstore and metadatastore, both
will be fetched, as the code hasn't been audited to know whether we only want
history or data.
Differential Revision: D14387734
fbshipit-source-id: 6aafd477ff486b9316458ce0e80636152db45b89
Summary:
When using memcache in its packfile mode, the key no longer contains the name
of the repo, and therefore, memcache will store the downloaded packfiles under
/var/cache/hgcache instead of /var/cache/hgcache/fbsource/packs.
Reviewed By: quark-zju
Differential Revision: D14217056
fbshipit-source-id: f78ce1021985dbb71a1db21d8821e8b8fcda8179
Summary: Allow file data/history packs to be fetched via HTTP when the Eden API is enabled.
Reviewed By: quark-zju
Differential Revision: D14257368
fbshipit-source-id: 8b6823a57a6fdef546a596df20387b3fc1ccdd4a
Summary:
When pushing an empty commit, the server receives a pack part with no
data, which ends up not producing any pack files. Some newly added logic tries
to access the pack paths, which then crash.
Let's fix it so we get None for the paths in this situation, and update the only
consumers of those paths to handle the None case.
Reviewed By: quark-zju
Differential Revision: D14237452
fbshipit-source-id: 418bd30179fdb76b9de3bc2c2509079502edfef8
Summary:
To connect to the memcache client it used a path from a loose files data store (`remotefilelogcontentstore._path`), which is a path to the hg cache, but pack files store doesn't have the same field.
I replaced the path with cache path from shallowutil.
Reviewed By: singhsrb, liubov-dmitrieva
Differential Revision: D14020122
fbshipit-source-id: 29bc8a01fc42d43eddc6ae67cf4e41fc552e117c
Summary:
The super python builtin expects a type as an argument, while a string was
passed, this was causing trouble for some users.
Reviewed By: DurhamG
Differential Revision: D14012716
fbshipit-source-id: 6714eb20745428818721ec3df85588a7b4c7ebb7
Summary:
It looks like D13924759 accidentally replace `prog.value` with `prog` in one
location, which is causing crashes of the form:
TypeError: unsupported operand type(s) for +=: 'nullbar' and 'int'
Reviewed By: quark-zju
Differential Revision: D13984835
fbshipit-source-id: cbf1a52d8048b623beac6048224767f0abb7f5c0
Summary: We don't really need to check misses here as neither of request fucnctions return misses after fetching from the server. They raise exception if there are any misses.
Differential Revision: D13908252
fbshipit-source-id: 80b6e4fb79b419b5e6d3b343b3bd46d3f8dc7d3c
Summary: Implemented logic for fetching data from the scmmemcache and the server in pack files format. By default `prefetch` will still use loose files, unless the `remotefilelog.fetchpacks` config flag will be set to true.
Differential Revision: D13905733
fbshipit-source-id: cc559f5bc600067199a91efe6cf44b1bde6bf69b
Summary: New get/set protocols for pack files. Also removed hitcount from `receive` method, because it is not used.
Differential Revision: D13924759
fbshipit-source-id: 1382b6f2bc865a19125f3ccd17a779cd16f08b50
Summary: To add the data fetched from the server to the scmmemcache, we need to know pasths to the packfiles created.
Differential Revision: D13905734
fbshipit-source-id: 7c470b560e32e63e7df4f960be92bc9b6da9de3d
Summary: Moved fetching loose files from the server to the separate function
Reviewed By: DurhamG
Differential Revision: D13855807
fbshipit-source-id: 39821644abcdc7375ba1b9ef9519fd2083eb0a71
Summary: Move scmmemcache set/get commands protocol to the cache wrapper to isolate it from the main logic
Reviewed By: DurhamG
Differential Revision: D13855809
fbshipit-source-id: 95b70f7f9d10d97beff107f9f6e5e4f5a8a307c1
Summary: List of missed files is converted to a set many times. So changing the type to a set.
Differential Revision: D13924760
fbshipit-source-id: 65fe52ff6113724d8f7f7a8d5b3c1501f245eedc
Summary: Move scmmemcache receive logic to the cache wrapper to isolate it from the main prefetch logic
Reviewed By: DurhamG
Differential Revision: D13855808
fbshipit-source-id: d0da84857797e3e155507fcf284b39a146677e54
Summary:
When doing lfs prefetch, pass the filenames that correspond to each oid to lfs
so that it can include them in the progress bar.
Reviewed By: DurhamG
Differential Revision: D13716078
fbshipit-source-id: c91eca6b342996c0c3d1ee43f3b1e98db4ed64d4
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
