Summary:
Move the code that isn't proper to an historypack to a separate function. This
will be used to implement a similar functionality for the
IndexedLogHistoryStore
Reviewed By: quark-zju
Differential Revision: D17104466
fbshipit-source-id: 905a4c4c63d8bbb54611fcb9521ab3039d1b56fd
Summary:
This functionality will be needed by the IndexedLogHistoryStore, let's move the
code out of historypack.
Reviewed By: quark-zju
Differential Revision: D17104468
fbshipit-source-id: 33b6b36bb69ed09ca6f72f6f9b6531ffb15c2193
Summary:
This moves code from AsyncMutableHistoryPack. This allows abstracting the type
of history store so the IndexedLogHistoryStore can be easily added.
Reviewed By: quark-zju
Differential Revision: D17100019
fbshipit-source-id: ca226336ae9ff05385f9218e88e853bf6e5356e3
Summary:
1.) Windows doesn't support close_fds if stdin/stdout/stderr is redirected. It throws out this Exception: 'close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr'. This diff resolves this issue by importing mercurial util to use file_fd only on posix
2.) Originally, stable is implemented as a mercurial extension and gets expanded as the following (take update as an example):
```
hg up stable
-> calls getstablerev("--pick-best") in stable.py
-> calls a bash script in hg repo, e.g. ovrsource or fbsource
```
However, bash is not supported on Windows.
To resolve this, this diff sets up cwd in `subprocess.Popen` and then modifies stablerev to take in a command in string format instead of a script. This way, we can pass in either the bash script for fbsource(backward compatible) or `python <xxx.py>` for ovrsource.
Reviewed By: quark-zju
Differential Revision: D17140170
fbshipit-source-id: ea8ae76883cc34a0517fa7e9eae3cbb3ba901353
Summary:
Since the shared mutable packs contain fetched data from the network, not
commiting them means that we would need to redownload the data again. Let's
persist them on disk instead to avoid having to redownload them.
Reviewed By: quark-zju
Differential Revision: D17115796
fbshipit-source-id: 3e213461c7a864156ee4c6c68e6a042294883f9d
Summary:
The loosefiles repack was made incremental to greatly reduce the repack time
for users. Since the amount of local loosefiles should be way smaller than the
amount of shared ones, let's always run a full repack on the former. This
should allow us to kill all the local loosefiles, which will help in no longer
supporting them.
Reviewed By: quark-zju
Differential Revision: D17135975
fbshipit-source-id: 9480993b31aa57d0d6e6b7caffd282929183f782
Summary:
This illustrate that the local data repack isn't a full one. Note that the test
is a bit flaky due to 1) the underterministic nature of what folder to repack,
and 2) the choice of folder to repack being done twice (one for data, one for
history). Since I won't land this without the next one, this is probably fine.
Reviewed By: quark-zju
Differential Revision: D17135978
fbshipit-source-id: 641d257d0e308b2bb4d91fdc4c214b22e45b4911
Summary:
Now that we're past the point of being not confident about the Rust repack
code, we can remove the fallback code. Instead, let's fail hard if the Rust
repack fails.
Reviewed By: quark-zju
Differential Revision: D17135979
fbshipit-source-id: a521a4f9267361a167bba61968659557bed35e20
Summary:
Rust repack has been the default for a while now. Let's stop pretending we're
going to switch back to the Python repack by removing the config entirely.
Reviewed By: quark-zju
Differential Revision: D17135977
fbshipit-source-id: 5aaa0faa48e2b40a7314d5ab455f5eeaa4e4984d
Summary:
The inline revlog format merges `.i` and `.d` into one `.i` file. It was intended to reduce the
number of files for filelogs. For the changelog one extra file does not hurt.
This makes it easier to write native code parsing the changelog revlog index.
Reviewed By: xavierd
Differential Revision: D17125922
fbshipit-source-id: f48ffe0d2df71abec007a80e05b684dcbac71883
Summary:
I broke some svn tests in that commit, but while fixing forward
realized that we're blocked from making this change because we rely on
rolling back the last commit in the case where the filemap creates an
empty commit.
We can revisit this when we have convert this all to rust!
Reviewed By: sfilipco
Differential Revision: D17140172
fbshipit-source-id: c93911f599afd2d22e84b3327a2a9455204065a0
Summary:
We noticed that the runtime for convert was exponential
due to way that packs were managed.
This wraps the convert in a big transaction so that we reuse an
existing pack rather than writing out one for each commit.
Reviewed By: xavierd
Differential Revision: D17138073
fbshipit-source-id: 46e418a49f917e2ffc411920ace1a3d98ac6b8e4
Summary:
We've seen a handful of cases where the indexedlogdatastore becomes corrupted
which makes Mercurial unable to run properly. For now, and since we only use
the indexedlog for the hgcache, let's just remove it.
A better solution would be to harden the indexedlog code to better detect the
corruption and attempt to fix them.
Reviewed By: quark-zju
Differential Revision: D17115622
fbshipit-source-id: ee474a5df60c4414f6ea21ace7dff0f7048879c9
Summary:
Wez noticed that when /var/cache/hgcache/<repo>/packs/manifest was missing, `hg
repack` would fail, not repacking anything. Let's fix the offending code to
properly ignore the missing directory.
Reviewed By: quark-zju
Differential Revision: D17120206
fbshipit-source-id: 7a7aebac717e5f128565ea2f8d50ffaeda4563f9
Summary:
I had assumed that we store p1 and p2 in the same order that they are used in
Node computation. That is incorrect. In general p1 and p2 are assumed to have
an ordering that matters and it's Node computation that is specific.
Reviewed By: quark-zju
Differential Revision: D17125743
fbshipit-source-id: 3a2673d9c243e2d2103aba0cb4fd8f536386efa7
Summary:
In some cases the finalize algorithm is used to persist data that is received
in a bundle. The process is that it constructs a store from the bundle and
goes to construct a tree with the root node received. It then goes through
finalize to generate the entries that need to be written to local storage.
Reviewed By: quark-zju
Differential Revision: D17125149
fbshipit-source-id: de5a1e922a6aebe48e238d8473177a8d3f7a9ef5
Summary:
`listdir` returns the contents a directory in the manifest. The format
is pretty simple, containing only the simple names of the files and or
directories. I don't know if this is something that eden can use because
it seems to simple. In other words, we have something but we may want
to iterate on it before we market it broadly.
Reviewed By: quark-zju
Differential Revision: D17098082
fbshipit-source-id: d6aa42c96781cf1f8b2e916fa10bb275593bdc65
Summary:
The C++ manifest implements walksubdirtrees which is used to compute the packs
that a "client" wants for a prefetch. In terms of interface the function is very
annoying and couples with storage and tree representations without being part
of any of them.
We reproduce that functionality as a means to replace the C++ implementation.
The long term goal is to do lazy fetches using an iteration style that plays
nicer with batching downloads.
This change also includes fastmanifest updates because they are required to
enable the walksubdirtrees functionality in our tests.
Reviewed By: quark-zju
Differential Revision: D17086669
fbshipit-source-id: 6c1f9fbf975814f0a2071f8d1c8e022e5ad58e29
Summary:
ignore-conflict-markers
This updates Mercurial to add a loggetpack option for wire proto logging that
allows capturing getpack responses. This will be useful to verify correctness
for ovrsource on Mononoke (right now, Mononoke isn't serving traffic for hosts
that use getpack).
Reviewed By: StanislavGlebik
Differential Revision: D17091537
fbshipit-source-id: 755a429949d7645010dddab95202c613025f2984
Summary:
In D16866464, we changed the name of the `args` to `full_args`, which is
causing compilation to fail when buildinfo is enabled:
https://fburl.com/sandcastle/f23unjvj
That said, buildinfo hasn't actually been working (AFAICT), since argv[0] is
the name of the binary, not the first argument, so running `hg buildinfo`
doesn't work, regardless of whether it was compiled in or not. This diff also
fixes that, and reduces the scope of conditional compilation to reduce the
likelihood of this happening again.
Reviewed By: quark-zju
Differential Revision: D17111929
fbshipit-source-id: 720344721f1fe73a3ebe8330c278e4be83d80c00
Summary:
While working on a stack, I noticed that `hg amend -e` would sometimes download
a lot of history information. For a 4 files change, I saw 738 history entry
fetched individually...
Looking at the profile, this pointed towards remotefilectx.parents requesting
the entire ancestormap. Since that function really only need the nodeinfo,
let's only get that.
Reviewed By: DurhamG
Differential Revision: D17104263
fbshipit-source-id: fae1f673b2d2a641ae4f22d1099317cc5abd8447
Summary:
Someone reported an error in this function being called with a missing
argument.
Reviewed By: quark-zju
Differential Revision: D17110694
fbshipit-source-id: 65adbae7e1380652afd1cec8bd3e2bbef27268dd
Summary:
Update the doc to match the latest code.
I put an example at the top. I guess most people aren't interesting in the
implementaion details. With the latest `define_flags!` macro, not knowing
the details does not block people from writing commands.
Tailing spaces are removed.
Reviewed By: sfilipco
Differential Revision: D16733268
fbshipit-source-id: 13c965b7471dd3bc131c20261601f4850c1036c1
Summary:
This makes it possible to use explicitly named arguments. It can simplify the
code a bit by moving the error handling (incorrect number of arguments) to the
derived code.
Reviewed By: xavierd
Differential Revision: D16905460
fbshipit-source-id: 93dc58d684bb52cd2f9bddc25108ef0c500f81db
Summary:
If there is no `#[args]` field set, the command does not need arguments. In
that case, just error out directly. This can simplify command implementations.
Reviewed By: xavierd
Differential Revision: D16905461
fbshipit-source-id: f4a2d03cf032f34d1b5794010774ff80eb88e1cd
Summary: This make it possible for the flags parsing to return an error.
Reviewed By: xavierd
Differential Revision: D16905462
fbshipit-source-id: 06a70ee7c72132ebb2df453a740b58e6b46dfba2
Summary:
Most commands do not care about the "command name" (aka. arg0). Only commands
sharing a similar implementation (ex. hg up/down/bottom/top) would need it.
Make it optional and make `args` not containing the command name.
Reviewed By: xavierd
Differential Revision: D16905459
fbshipit-source-id: 0a786731ebce2580e116fd50aad75062b03a1fa3
Summary: This makes the code easier to read as we add more fields to the macro state.
Reviewed By: xavierd
Differential Revision: D16905458
fbshipit-source-id: a3bc4dc1accf21cf491921469482f86aec16321b
Summary:
This makes it possible to execute Rust commands from the Python tests.
Test changes:
- test-command-template-t: non-utf8 command-line arguments are rejected at the
function signature level
- test-dispatch-debug-prefix-t: the error is now printed by Rust code, which
uses spaces instead of tabs.
- test-root-t: the test now passes
Reviewed By: xavierd
Differential Revision: D16866459
fbshipit-source-id: 386931c5497b04c53efc08fbb4de708812517ad9
Summary:
This gives the Rust code path hints to names of Python command names.
Ideally, the Rust command dispatch logic can just load the Python command table
and figure out it more accurately, and more parts of dispatch.py (ex. extension
loading, debugger, profiler, atexit, etc) are moved to Rust clidispatch or
hgcommands. But that would be a larger change.
Reviewed By: xavierd
Differential Revision: D16866462
fbshipit-source-id: eb993091d5644710686b8f720fd07258b9a5968c
Summary:
After D7840236 stack, it's possible to have a single chg server that handles
different [extensions] configurations. The 'validate' step and 'hashstate' were
mainly designed to detect changes of [extensions], the source code of the
extensions. That becomes unnecessary with the latest design.
Remove them to simplify the logic.
chg no longer creates symlink `server2 -> server2-hash`. Bump the name
to `server3` to explicitly break compatibility.
Reviewed By: xavierd
Differential Revision: D16866463
fbshipit-source-id: 5e1d00e6f895d9b8ead0bcabefcea11756f57c94
Summary:
Re-enable chg since chg can now run native Rust commands.
In the new state, the dispatch logic tries things in this order:
- chg (both Python and Rust commands)
- bindings.commands.run
- hgcommands::run_command
- clidispatch (Rust commands)
- HgPython (Python commands)
- edenscm.mercurial.entrypoint
- dispatch.py:run
- hgcommands (both Rust and Python commands)
- clidispatch (...)
- HgPython (...)
Before this stack (D16866461), the old order is:
- hgcommands (Rust commands only)
- clidispatch (Rust commands)
- chg (Python commands only)
- dispatch.py
- dispatch.py (Python commands only)
The old order enforces `hgcommands` to not handle Python commands, because of
the undesirable Python startup time overhead comparing to chg. That is bad for
code path unification.
The new approach adds small (5ms) overhead for chg running native commands, but
it helps code unification (hgcommands take care of both Rust and Python), and
perserves the chg perf win.
Code unfication can be pushed further by making clidispatch aware of Python
commands, and move more of dispatch.py to the Rust clidispatch or hgcommands
crate.
Reviewed By: singhsrb
Differential Revision: D16866469
fbshipit-source-id: 79681cb748bd16f200424b6e30ef42aa3111d1bf
Summary:
Use hgcommands via its bindings to run commands. This allows chg to run Rust
commands.
Reviewed By: xavierd
Differential Revision: D16866466
fbshipit-source-id: 0c264a6b5441d2d0b2cdd01beefda5606b00cc61
Summary:
This makes it possible to capture output of a Python command via the Rust
hgcommands crate.
Reviewed By: xavierd
Differential Revision: D16866467
fbshipit-source-id: e64feece9b31196fabfea110015d5c45cb548046
Summary:
This allows the Rust `hgcommand` library to use the hg entry point to run
different commands.
Reviewed By: xavierd
Differential Revision: D16866465
fbshipit-source-id: 03078fe76a38c9a01dd6005e03e832162fbeb7e6
Summary: This allows the io module to be used in non-extension code (ex. hgcommands).
Reviewed By: xavierd
Differential Revision: D16866458
fbshipit-source-id: b36b535e3804ba2b8e656508c10ca7ceb3e1fdf1
Summary: An upcoming change would like to downcast IO to stdio or PyObjects.
Reviewed By: xavierd
Differential Revision: D16866456
fbshipit-source-id: 73289aab13260ec421b979280f0b9c73383ab5d0
Summary: This makes it possible for Python to execute commands implemented in Rust.
Reviewed By: xavierd
Differential Revision: D16866457
fbshipit-source-id: e7352eed0e7b722c28974cb1123975f591ac44bc
Summary:
Expose a simple method in hgcommands to run the actual (Rust or Python) command.
This makes it easier to expose the "runcommand" API via bindings.
I have to disable chg temporarily since `hg root` with chg stops working at present.
chg will be re-enabled in a later diff.
Reviewed By: xavierd
Differential Revision: D16866461
fbshipit-source-id: 46c5df9a577d748fab302a8c7157d8aa62d6f1bd
Summary:
Previously, HgPython hardcodes the arguments to be std::env::args().
This diff makes that an configurable argument.
Reviewed By: xavierd
Differential Revision: D16866464
fbshipit-source-id: 3a37dd89793e0813732ea9b088ff22c9a4c6b125
Summary:
`Py_Finalize` is not recursive. It unconditionally destroys the Python interpreter
state. To make `hgcommands` usable in a library (ex. bindings), detect the case
where the interpreter was initialized, and skip `Py_Finalize` in that case.
Reviewed By: markbt
Differential Revision: D16866454
fbshipit-source-id: ea6bb6a83bc0b0fe7b5c2ce21158ae9d9e2a779d
Summary:
After D15844921, `hg version` is not running the locally built hg.
This command does not provide much useful information, though - we
have a fixed version number and the commit hash is already known
from `hg sl` or other command output.
Therefore remove `hg version`.
Reviewed By: xavierd
Differential Revision: D17097797
fbshipit-source-id: 184a110a1682ee20c973dca7dcc0e2cfd8660976
Summary:
There are cases where Mercurial makes many serial requests to the server.
This diff limits the number of lines that a perftrace can send to blackbox
or print on stderr.
Reviewed By: xavierd
Differential Revision: D17078475
fbshipit-source-id: 8c8313ae0b602160e1018590dd6b87dfffd27662
Summary:
So far we missed the metadata if there were no changes in the WC.
A metadata file was created but was not attached to the commit.
Differential Revision: D17070211
fbshipit-source-id: 646c8bceb575f4302ec60e35472cc55de086d7e0
Summary:
Filter out legacy logs without messages and the `remotefilelog` entries so the
blackbox section can look cleaner.
Reviewed By: sfilipco
Differential Revision: D17054398
fbshipit-source-id: 6485f1832fc12d6f68ed3fb1efe3dbc853af4b1f
Summary:
The indexedlog based datastore is being rolled out more broadly, let's add a
basic historystore indexedlog to replace the historypacks. One of its first use
will be in hg_memcache_client write path to remove some pathological cases
where hg_memcache_client can write thousands of packfiles. This in turn will
remove the need to run repack to keep the amount of packfiles in control.
The IndexedLog key is the concatenation of sha1(path) and the node. Hashing the
path should be fairly cheap and makes it easy to integrate with the IndexedLog.
One of the drawback versus the histpack will be storage space usage, as the
path is always stored per entry, while it was shared with multiple entries in
the history pack. I though about having a separate index to easily translate
the hashed path to the path, but due to the potential log rotation, we could
end-up in a case where the path isn't present at all in the store.
Reviewed By: quark-zju
Differential Revision: D16616082
fbshipit-source-id: 1e47260b479f8923cc137a39dcba54b2d074f43a
Summary:
See https://fb.workplace.com/groups/corehg/permalink/448152285780007/ for context.
D16833526 was to add extras lookup for translating hg hash to git hash. However we
also have gitrevset extension which still does mapfile lookup. THis diff should port
the same improvement for it.
Reviewed By: DurhamG
Differential Revision: D17070479
fbshipit-source-id: e469707b0b21541aa2c3b90d1480b04afc4f1485
Summary:
Add a way to explicitly `fsync` data to reduce changes of data corruption in
case of hard reboots.
If `fsync` is set on a `Log`, it sets `fsync` recursively on its indexes and
checksum tables. If frequent `fsync` is an issue, it might be possible to only
`fsync` the source of truth (log and meta) and have other logic to rebuild
indexes on demand.
`fsync` on btrfs seems to be relatively cheap (comparing to ext4), so maybe
this is fine for btrfs users.
Reviewed By: markbt
Differential Revision: D16926659
fbshipit-source-id: a9de2fa352f607fe6f0b9d36047323862770f2e6
