Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.
Future diffs will move config loading in as well.
Reviewed By: quark-zju
Differential Revision: D22590208
fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.
Reviewed By: quark-zju
Differential Revision: D22587237
fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.
Reviewed By: quark-zju
Differential Revision: D23089541
fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D22992103
fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
Summary:
This makes the main command table cleaner.
I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.
Reviewed By: DurhamG
Differential Revision: D22992107
fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
Summary:
If every command lives in their module, then we can define the "module" interface:
- run(...): run the command
- doc(): the help text
- name(): command name, with aliases
Then the macro would make command registration look simpler.
This diff changes `status` to use the pattern as an example.
Reviewed By: DurhamG
Differential Revision: D22992109
fbshipit-source-id: eaf589863092ec2eb1f8c24c1c7e425492fe1e3a
Summary:
As the number of commands grows, it starts making sense to move them to
individual files. Let's create a directory for them.
Reviewed By: DurhamG
Differential Revision: D22992108
fbshipit-source-id: a0556be602b832579a8e027342d5b86d9d84d257
Summary:
Introduce taggederror-util, which provides a new trait `AnyhowEdenExt`, which provides a method `eden_metadata` for anyhow errors and results. This method works much like `AnyhowExt::common_metadata`, but additionally supports extracting default error metadata from known `Tagged` types which are listed explicitly in the method implementation.
Extend `FilteredAnyhow` to support a configuration "metadata function", which allows swapping out `eden_metadata` for the standard `common_metadata`.
Modify Rust dispatch and Python bindings to use `AnyhowEdenExt` for metadata extraction and printing.
Modify `intentional_error` to rely on `AnyhowEdenExt` for tagging (removes `.tagged` call, no tags will be visible if `AnyhowEdenExt` is not used).
Reviewed By: DurhamG
Differential Revision: D22927203
fbshipit-source-id: 04b36fdfaa24af591118acb9e418d1ed7ae33f91
Summary:
When using LFS, it's possible that a pointer may be present in the local
LfsStore, but the blob would only be in the shared one. Such scenario can
happen after an upload, when the blob is moved to the shared store for
instance. In this case, during a `get` call, the local LFS store won't be able
to find the blob and thus would return Ok(None), the shared LFS store woud not
be able to find the pointer itself and would thus return Ok(None) too. If the
server is not aware of the file node itself, the `ContentStore::get` would also
return Ok(None), even though all the information is present locally.
The main reason why this is happening is due to the `get` call operating
primarily on file node based keys, and for content-based stores (like LFS),
this means that the translation layer needs to be present in the same store,
which in some case may not be the case. By allowing stores to return a
`StoreKey` when progress was made in finding the key we can effectively solve
the problem described above, the local store would translate the file node key
onto a content key, and the shared store would read the blob properly.
Reviewed By: DurhamG
Differential Revision: D22565607
fbshipit-source-id: 94dd74a462526778f7a7e232a97b21211f95239f
Summary:
Implements based Rust-Python binding layer for error metadata propagation.
We introduce a new type, `TaggedExceptionData`, which carries CommonMetadata and the original (without metadata) error message for a Rust Anyhow error. This class is passed to RustError and can be accessed in Python (somewhat awkwardly) via indexing:
```
except error.RustError as e:
fault = e.args[0].fault()
typename = e.args[0].typename()
message = e.args[0].message()
```
As far as I can tell, due to limitations in cpython-rs, this can't be made more ergonomic without introducing a Python shim around the Rust binding layer, which could adapt the cpython-rs classes to use whatever API we'd like.
Currently, anyhow errors that are not otherwise special-cased will be converted into RustError, with both the original error message and any attached metadata printed as shown below
```
abort: intentional error for debugging with message 'intentional_error'
error has type name taggederror::IntentionalError and fault None
```
We can of course re-raise the error if desired to maintain the previous behavior for handling a RustError.
If we'd like other, specialized Rust Python Exception types to carry metadata (such as `IndexedLogError`), we'll need to modify them to accept a `TaggedExceptionData` like `RustError`.
Renamed the "cause an error in pure rust command" function to `debugcauserusterror`, and instead used the name `debugthrowrustexception` for a command which causes an error in rust which is converted to a Python exception across the binding layer.
Introduced a simple integration test which exercises `debugthrowrustexception`.
Added a basic handler for RustError to scmutil.py
Reviewed By: DurhamG
Differential Revision: D22517796
fbshipit-source-id: 0409489243fe739a26958aad48f608890eb93aa0
Summary: Adds support for sharding based on user name.
Reviewed By: quark-zju
Differential Revision: D22537540
fbshipit-source-id: 962f9582c8947335dc9d9d29c500d8c09df69878
Summary:
Introduce new rust library, taggederror, which contains utilities for attaching metadata to errors. The library provides two main methods for attaching metadata to an error, the TaggedError wrapper type, and the AnyhowExt trait methods. Provides a struct, CommonMetadata, which contains all the metadata types introduced by taggederror (fault, transience, category, and typename), which can also be attached individually (and the same pattern can be used to attach other metadata).
Introduce a new native rust command, debugthrowrustexception, which causes the command to return an error, with some attached metadata.
Modify hg rust native command dispatch error handling to use debug formatter to print anyhow::Error errors. This will print out the source chain, contexts, and backtrace if available, which will cause the metadata we attach as a wrapper error or context to be printed.
Reviewed By: DurhamG
Differential Revision: D22420941
fbshipit-source-id: d38c5a10b686d86b69a2c0a19f5bcbf4ca24dff6
Summary:
Previously you could only canary locally on a devserver by setting an
environment variable. Let's add a --canary flag to debugdynamicconfig that
accepts a host. Hg will ssh to that host and run the configerator cli to grab
the canaried config from that host.
Reviewed By: quark-zju
Differential Revision: D22535509
fbshipit-source-id: af1c21d8402c4e729769e50388d913bf52b66b89
Summary: Add a new `EdenApiBlocking` trait that exposes blocking versions of the `EdenApi` trait's methods, for use in non-async code.
Reviewed By: quark-zju
Differential Revision: D22305396
fbshipit-source-id: d0f3a73cad1a23a4f0892a17f18267374e63108e
Summary: Move old EdenAPI crate to `scm/lib/edenapi/old` to make room for the new crate. This old code will eventually been deleted once all references to it are removed from the codebase.
Reviewed By: quark-zju
Differential Revision: D22305173
fbshipit-source-id: 45d211340900192d0488543ba13d9bf84909ce53
Summary:
If we're running commands from a user that only has read access, the
debugdynamicconfig commands are going to fail. Let's exit early and quickly if
that's the case, instead of spending a lot of cpu generating a config only to
fail.
Reviewed By: quark-zju
Differential Revision: D22244127
fbshipit-source-id: 24f806772ba5c08e400efb3abc7ebda228d473a5
Summary:
Fetches configs from a remote endpoint and caches them locally. If the
remote endpoint fails to respond, we use the cached version.
Reviewed By: quark-zju
Differential Revision: D22010684
fbshipit-source-id: bd6d4349d185d7450a3d18f9db2709967edc2971
Summary:
We kick off a background process every 15 minutes or so to compute the
new dynamicconfig. If the config hasn't changed, we should just touch the file
instead of rewriting the whole thing.
Reviewed By: quark-zju
Differential Revision: D21653098
fbshipit-source-id: 9d53d2a636cff082cd048994255cc809ce1b0221
Summary:
If we release a new version of Mercurial, we want to ensure that it's
builtin configs are used immediately. To do so, let's write a version number
into the generated config file, and if the version number doesn't match, we
force a synchronous regeneration of the config file.
For now, if regeneration fails, we just log it. In the future we'll probably
throw an exception and block the user since we want to ensure people are running
with modern configuration.
Reviewed By: quark-zju
Differential Revision: D21651317
fbshipit-source-id: 3edbaf6777f4ca2363d8617fad03c21204b468a2
Summary:
Instead of trying to maintain two hgrc.dynamic's for shared repositories,
let's just always use the one in the shared repo. In the long term we may be
able to get rid of the working-copy-specific hgrc entirely.
This does remove the ability to dynamically configure individual working copies.
That could be useful in cases where we have both eden and non-eden pointed at
the same repository, but I don't think we rely on this at the moment.
Reviewed By: quark-zju
Differential Revision: D21333564
fbshipit-source-id: c1fb86af183ec6dc5d973cf45d71419bda5514fb
Summary:
We'll be adding a bunch of Facebook specific configuration and values
here. Let's move it to someplace not open source.
Reviewed By: quark-zju
Differential Revision: D21241038
fbshipit-source-id: 2ac9cdce40b1b46f15f171d9d1f6b6692dcd29bf
Summary:
We want Mercurial to become more responsible for it's own
configuration, instead of relying on chef and other means. To do so, let's
introduce a new `hg debugdynamicconfig` that can generate dynamic configs for
a given repository based on various states, like what tier it's in or what shard
that machine is in. By default it generates to '.hg/hgrc.dynamic' for the given
repository.
Currently it just sets the hostgroup config.
Future diffs will make Mercurial consume this config, and possibly have Mercurial
call this command asynchronously when it notices the file is out-of-date.
Reviewed By: quark-zju
Differential Revision: D20828132
fbshipit-source-id: 6f5bf749f5b04e0a5989d6dc19ee788c2e47f88f
Summary: This makes the tracing features easier to use.
Reviewed By: DurhamG
Differential Revision: D19797703
fbshipit-source-id: fb5cb17cd389575cf0134a708bcd9df3b90e9ab4
Summary:
Do not convert the entire output to a string. This makes `debugindexedlog dump`
a good test case for native pager support - it takes a while to write the full
output for a large input.
Reviewed By: DurhamG
Differential Revision: D20885567
fbshipit-source-id: 35ed8f68dff1916f0833577c3cf2a52cbf2a658c
Summary:
This is the start of migrating blackbox events to tracing events. The
motivation is to have a single data source for log processing (for simplicity)
and the tracing data seems a better fit, since it can represent a tree of
spans, instead of just a flat list. Eventually blackbox might be mostly
a wrapper for tracing data, with some minimal support for logging some indexed
events.
Reviewed By: DurhamG
Differential Revision: D19797710
fbshipit-source-id: 034f17fb5552242b60e759559a202fd26061f1f1
Summary:
Clippy had 3 sources of warnings in this crate:
- from_str method not in impl FromStr. We still have 2 of them in path.rs, but
this is documented as not supported by the FromStr trait due to returning a
reference. Maybe we can find a different name?
- Use of mem::transmute while casts are sufficient. I find the cast to be
ugly, but they are simply safer as the compiler can do some type checking on
them.
- Unecessary lifetime parameters
Reviewed By: quark-zju
Differential Revision: D20452257
fbshipit-source-id: 94abd8d8cd76ff7af5e0bbfc97c1e106cdd142b0
Summary: This makes it clear that these traits are dealing with Mercurial Key.
Reviewed By: quark-zju
Differential Revision: D20445626
fbshipit-source-id: d5acbf442e9407b973e95e40af69b5a61bff0a4d
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.
Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.
This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.
This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.
Reviewed By: markbt
Differential Revision: D20432938
fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
Summary: Move the scope of spans so the exit code is shown.
Reviewed By: xavierd
Differential Revision: D20286516
fbshipit-source-id: f39cbf60c86ea19a1bb0a09958748f04ff6a42e8
Summary:
Previously env_logger is only initialized if Python is initialized.
This diff makes env_logger initialized for Rust native commands.
Reviewed By: xavierd
Differential Revision: D20286517
fbshipit-source-id: 18fee96c2b41db1da9648d615d1e18809de90a63
Summary:
TreeSpans used to use `&str`, which adds a lifetime to the struct, making it
harder to be used in the Python land. Use a type parameter so TreeSpans<String>
can be used.
Reviewed By: DurhamG
Differential Revision: D19797708
fbshipit-source-id: c66429abfaf16d876151ca6f29da976bed91485d
Summary:
Drop stdoutbytes/stdinbytes. They make things unnecessarily complicated
(especially for chg / Rust dispatch entry point).
The new idea is IO are using bytes. Text are written in utf-8 (Python 3) or
local encoding (Python 2). To make stdout behave reasonably on systems not
using utf-8 locale (ex. Windows), we might add a Rust binding to Rust's stdout,
which does the right thing:
- When writing to stdout console, expect text to be utf-8 encoded and do proper decoding.
- Wehn writing to stdout file, write the raw bytes without translation.
Note Python's `sys.stdout.buffer` does not do translation when writing to stdout console
like Rust's stdout.
For now, my main motivation of this change is to fix chg on Python 3.
Reviewed By: xavierd
Differential Revision: D19702533
fbshipit-source-id: 74704c83e1b200ff66fb3a2d23d97ff21c7239c8
Summary:
All diff functions are (bytes, bytes) -> bytes to preserver
the original file encoding.
Because of that I had to add ui.writebytes output function that accepts
bytes for terminal output.
Reviewed By: farnz
Differential Revision: D19656673
fbshipit-source-id: b9a1e4361e825fc8c2313e8402c2bbe00f490dd4
Summary:
Set up the `cpython-ext` and `hgcommands` libraries so that they can compile
against py2 and py3 versions of rust-cpython. Make py2 the default so
that cargo test still works.
Reviewed By: singhsrb
Differential Revision: D19615656
fbshipit-source-id: 3403e7077deb3c0a9dfe0e3b7d4f4ad1da73bba3
Summary:
Update the Rust hgcommands code to pass the command line arguments into the
Python logic as `Str` types, so that this will be Unicode `str` objects when
using Python 3.
Reviewed By: xavierd
Differential Revision: D19596739
fbshipit-source-id: 7cdfd44a1c4ce8b0f86d20b634d9b27eab822b2d
Summary:
This makes `make hg3` work. It requires cleaning up the `build` directory when
switching between py2 and py3 build, which will be fixed later.
Reviewed By: DurhamG
Differential Revision: D19604824
fbshipit-source-id: 060ff313420126a5dba935c4451b45dc9af45f13
Summary: Update to the new version of rust-cpython. This supports `list.append`, so make use of it.
Reviewed By: xavierd
Differential Revision: D19590905
fbshipit-source-id: 03609d4f698ae8e4380e82b8144caaa205b4c2d4
Summary:
Also, add a util::path::strip_unc function that is more clear than the
normalize_for_display
Reviewed By: DurhamG
Differential Revision: D19595961
fbshipit-source-id: 330bcb708bf64320a3562d79db685d6cb1e14f16
Summary:
This allows us to incrementally abstract away parts related to filesystem from
Log. For example, instead of using std::fs to access filesystem directly, use
methods on the GenericPath instead.
Right now the main motivation is to add support for "multi-logs" sharing a single meta
file. To do that, methods reading and writing metas are moved to the
GenericPath type.
This also unifies the open functions. Now OpenOptions::create_in_memory can be
private. Empty in-memory Logs can be opened via `open(())`.
Reviewed By: DurhamG
Differential Revision: D19431784
fbshipit-source-id: dbdf94f60261e09f131c6fdd9fe3b99242a28af5
Summary:
This simplifies the error handling and makes it more compatible with things that
might capture the output, including the Python testing framework.
Reviewed By: DurhamG
Differential Revision: D19325642
fbshipit-source-id: 53de8b9a8294219e2b8e62831dce236841bd4cbb
Summary: Make `hg status` use the EdenFS Thrift path, similar to the telemetry wrapper.
Reviewed By: DurhamG
Differential Revision: D19325641
fbshipit-source-id: 14777a252d7cb433316511a2a1f1a6649e9cb020
Summary:
The environment variable `EDENSCM_TRACE_OUTPUT` specifies where to write the
tracing output for the current command. Environment variables are inherited
by subprocesses by default. That is undesirable because another hg command
(triggered by hook, or background maintaince) will rewrite the trace output.
Avoid it by unsetting EDENSCM_TRACE_OUTPUT for subprocesses.
Reviewed By: xavierd
Differential Revision: D18892973
fbshipit-source-id: 575b6c0df2e7a0775172893e4aa72ca33fa4658c
Summary:
We have seen cases on Windows where the hg process gets stuck in
CreateToolhelp32Snapshot. Let's be defensive and always exit the loop.
Reviewed By: singhsrb
Differential Revision: D18729720
fbshipit-source-id: fb8602ce231eec01b6b42c6759849d56e5db2030
