Commit Graph

130 Commits

Author SHA1 Message Date
Meyer Jacobs
ef00b68647 newstore: minimal proof of concept trait & edenapi implementation
Summary: Introduce a minimal version of an async, typed `ReadStore` trait and corresponding `EdenApi` implementation, along with a debug command, `debugnewstore` to exercise it.

Reviewed By: DurhamG

Differential Revision: D26050641

fbshipit-source-id: 2c14c3715e7067f9ecd1e649e6ca146a1ce249bf
2021-02-08 10:39:31 -08:00
Jun Wu
8e35bcc49d tracing-analyzer: log version information
Reviewed By: DurhamG

Differential Revision: D19803760

fbshipit-source-id: 802d72c1a7aab495d2b6aecd2f8bb696ce2fc522
2021-01-22 12:31:16 -08:00
Stefan Filip
a76dffcb83 tracing: update base collector to Registry + Subscriber list
Summary:
This setup is more extendable than the TracingData focused approach. We can
more easily add new functionality using the Subscriber list.

The approach taken here to introduce the new collector tries to maintain
existing functionality. We can then move various logic to their own
Subscribers.

Reviewed By: quark-zju

Differential Revision: D25988580

fbshipit-source-id: 045cd355dbd499109e554a29a1439c2d490b7c40
2021-01-21 11:35:08 -08:00
Liubov Dmitrieva
8d6f9be84c fix make local build
Summary:
Make local build is broken with

```
building 'hgmain' binary
error: failed to select a version for the requirement `indicatif = "^0.14"`
candidate versions found which didn't match: 0.15.0
location searched: directory source `/data/users/liubovd/fbsource/fbcode/eden/scm/../../../third-party/rust/vendor` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `hgcommands v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/lib/hgcommands)`
    ... which is depended on by `hgmain v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/exec/hgmain)`
perhaps a crate was updated and forgotten to be re-vendored?
error: compilation of Rust target 'hgmain' failed
make: *** [Makefile:68: local] Error 1
```

Reviewed By: krallin

Differential Revision: D25973253

fbshipit-source-id: 42c66dcfe26dc5c7713d4bdedc2b353aa5eb20ed
2021-01-20 07:23:59 -08:00
Stefan Filip
691d8f7ef2 hgcommands: downgrade to indicatif 0.14
Summary: The dependency that we want to share is 0.14 instead of 0.15.

Reviewed By: singhsrb

Differential Revision: D25871110

fbshipit-source-id: 16e9f8a858ee04a47867c2916909edfc996f8bc4
2021-01-11 10:03:51 -08:00
Stefan Filip
dcfb24ccc6 commands: add simple progress reporting for debugsegmentclone.
Summary:
Rough progress reporting. The progress bars are straight coming from the
`indicatif` crate. Integrating with the `IO` object is not trivial because
we only have a reference. It gets tricky. I think that it makes sense for
us to expand the IO object to something that is more than a `Box<dyn Write>`.
We have about 3 scenarios:
1. Write object that we need to implement interior mutability for to give out
clones.
2. Stdin/Stdout which have their own imlementation for interior mutability.
3. PyObject which has locking already implemented.

(Note: this ignores all push blocking failures!)

Reviewed By: quark-zju

Differential Revision: D25840469

fbshipit-source-id: 87f466f06f2c5d4c63ccb3bbc5c009fae41ed002
2021-01-08 12:07:45 -08:00
Jun Wu
3fc8f38541 changelog2: add "lazytext" backend
Summary: The backend is designed to be used by the "debugsegmentclone" command, which does not write revlog.

Reviewed By: sfilipco

Differential Revision: D25624786

fbshipit-source-id: e145128c7b41d78fed495f8da540169f741b674d
2020-12-18 16:47:11 -08:00
Jun Wu
a03e8f4c55 dag: make DagPersistent and DagAddHeads async
Summary: This makes it easier to make DagAlgorithm async.

Reviewed By: sfilipco

Differential Revision: D25345234

fbshipit-source-id: 5ca4bac38f5aac4c6611146a87f423a244f1f5a2
2020-12-10 12:37:33 -08:00
Durham Goode
925e457ae7 revisionstore: allow persistent IndexedLogData
Summary:
In a future diff we'll use the indexedlog stores for local data. We
want those to exist forever, so let's move IndexedLogHgIdDataStore to use a
Store under the hood, and add an enum for distinguishing between the two types
at creation time.

Reviewed By: xavierd

Differential Revision: D23915622

fbshipit-source-id: 296cf6dfcd53e5cf1ae7624fdccedf0a60a77f22
2020-12-09 07:07:40 -08:00
Stefan Filip
03c55e82d0 edenapi: add full_idmap_clone functionality
Summary: Adding `full_idmap_clone` to edenapi and usign that in `debugsegmentclone`.

Reviewed By: quark-zju

Differential Revision: D25139730

fbshipit-source-id: 682055f7c30a94a941acd16e2b8e61b9ea1d0aef
2020-12-01 09:59:27 -08:00
Stefan Filip
b991370c3f commands: add debugsegmentclone
Summary: Prototype for cloning a repository using segmented changelog.

Reviewed By: quark-zju

Differential Revision: D24954821

fbshipit-source-id: 5ff42a5d60b6ba3e85dd4eab151e902bd4b96243
2020-12-01 09:59:27 -08:00
Stefan Filip
559e152b2f clidispatch: update NoRepo commands to receive the config parameter
Summary:
The config is something that makes sense for all commands to have access to.
Commands that don't use a repo don't have access to the config that is prepared
by the dispatches. This change is a stop-gap to allow new commands that don't
require a repository to receive the config as an argument.

The construction of the config is something that we should iterate on. I see
the current implementaiton as a workaround.

Reviewed By: quark-zju

Differential Revision: D24954822

fbshipit-source-id: 42254bb201ba8838e7cc107394e8fab53a1a95c7
2020-12-01 09:59:26 -08:00
Jun Wu
2abf0ada42 version: print EdenSCM instead of Mercurial
Summary: Per team discussion.

Reviewed By: singhsrb

Differential Revision: D23719401

fbshipit-source-id: a1e9a1e9a10369c307413354054a65e6520d13e5
2020-09-15 21:03:59 -07:00
Thomas Orozco
21290702e1 third-party/rust: import async-compression + update zstd
Summary:
This imports the async-compression crate. We have an equivalent-ish in
common/rust, but it targets Tokio 0.1, whereas this community-supported crate
targets Tokio 0.2 (it offers a richer API, notably in the sense that we
can use it for Streams, whereas the async-compression crate we have is only for
AsyncWrite).

In the immediate term, I'd like to use this for transfer compression in
Mononoke's LFS Server. In the future, we might also use it in Mononoke where we
currently use our own async compression crate when all that stuff moves to
Tokio 0.2.

Finally, this also updates zstd: the version we link to from tp2 is actually
zstd 1.4.5, so it's a good idea to just get the same version of the zstd crate.

The zstd crate doesn't keep a great changelog, so it's hard to tell what has changed.
At a glance, it looks like the answer is not much, but I'm going to look to Sandcastle
to root out potential issues here.

Reviewed By: StanislavGlebik

Differential Revision: D23652335

fbshipit-source-id: e250cef7a52d640bbbcccd72448fd2d4f548a48a
2020-09-15 07:59:53 -07:00
Xavier Deguillard
ed4021b8e3 revisionstore: disallow reading LFS pointers from packfiles
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.

When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).

The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.

In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.

Reviewed By: DurhamG

Differential Revision: D22951598

fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
2020-09-09 18:27:42 -07:00
David Tolnay
e83e05ff25 Update formatter to rustfmt 2.0
Reviewed By: zertosh

Differential Revision: D23591028

fbshipit-source-id: f458503fc2b9c25023fa1643eca5e166882a4811
2020-09-09 07:52:34 -07:00
Durham Goode
2919268555 revisionstore: auto-delete when we have too much pack data
Summary:
In order to keep the hgcache size bounded we need to keep track of pack
file size even during normal operations and delete excess packs.

This has the negative side effect of deleting necessary data if the operation is
legitimately huge, but we'd rather have extra downloading time than fill up the
entire disk.

Reviewed By: quark-zju

Differential Revision: D23486922

fbshipit-source-id: d21be095a8671d2bfc794c85918f796358dc4834
2020-09-08 11:33:50 -07:00
David Tolnay
e62b176170 Prepare for rustfmt 2.0
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).

This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.

 ---

*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.

 ---

Reviewed By: zertosh

Differential Revision: D23568779

fbshipit-source-id: 477200f35b280a4f6471d8e574e37e5f57917baf
2020-09-07 20:47:59 -07:00
Jun Wu
f271d882e6 hgcommands: make commands! macro define modules
Summary: Similar to D18528858 so module names do not need to be spelled twice.

Reviewed By: markbt

Differential Revision: D23091380

fbshipit-source-id: a2a261abc9c78c8805cea62b38498ba65398796d
2020-08-27 19:02:27 -07:00
Jun Wu
d586a40ada hgcommands: add debugfsync
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.

The fsync logic is put in a separate crate to avoid slow compiles.

Reviewed By: DurhamG

Differential Revision: D23124169

fbshipit-source-id: 438296002eed14db599d6ec225183bf824096940
2020-08-27 18:26:03 -07:00
Jun Wu
ed78542610 dispatch: add --trace flag
Summary:
The `--trace` flag enables tracing Python modules.
For compatibility reasons, it also enables `--traceback`.

It can be used with debugshell to make `%trace` more useful.

Reviewed By: sfilipco

Differential Revision: D23278600

fbshipit-source-id: d6d0b34bd5c48111f8cd33d7df115f349b0e95b6
2020-08-27 18:14:28 -07:00
Arun Kulshreshtha
0b9ca4e83b hgcommands: remove unused imports in dynamicconfig module
Summary: Remove unused imports.

Reviewed By: quark-zju

Differential Revision: D23356940

fbshipit-source-id: 31b81eac11946aa8b24ec23c98ddb14716fbea3a
2020-08-27 14:06:52 -07:00
Jun Wu
039419d281 configparser: fix non-fb dependencies (#45)
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/45

Fix referring to 'version' without proper codegen by making 'version' compile
without codegen. This fixes configparser test when version/src/lib.rs was not
generated.

Make unneeded deps without 'fb' feature optional.

This would hopefully fix the "EdenSCM Rust Libraries" GitHub workflow.

Reviewed By: DurhamG

Differential Revision: D23269864

fbshipit-source-id: f9e691fe0a75159c4530177b8a96dad47d2494a9
2020-08-26 16:31:00 -07:00
Jun Wu
e7f3167810 hgcommands: show milliseconds on RUST_LOG output
Summary: This makes it a bit easier to track down perf issues printed by RUST_LOGs.

Reviewed By: sfilipco

Differential Revision: D23095463

fbshipit-source-id: 78221a1992389f512fac6e6e633be6d19123e04a
2020-08-21 13:00:45 -07:00
Durham Goode
0b123ba41d configs: move Rust dynamicconfig generation into configparser::hg
Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.

Future diffs will move config loading in as well.

Reviewed By: quark-zju

Differential Revision: D22590208

fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
2020-08-16 16:55:59 -07:00
Durham Goode
7ff28d3e1c configs: move dynamicconfig into configparser
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.

Reviewed By: quark-zju

Differential Revision: D22587237

fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
2020-08-16 16:55:59 -07:00
Durham Goode
b821ab3766 contentstore: make data rotatelog size configurable
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.

Reviewed By: quark-zju

Differential Revision: D23089541

fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
2020-08-16 16:44:16 -07:00
Jeremy Sze Wei Teo
43425f1116 Revert D22992103: hgcommands: add debugfsync
Differential Revision:
D22992103 (f6d086d13b)

Original commit changeset: b5503e498d52

fbshipit-source-id: ad8f0d9c0bba1d07edb0aebca052da10c0f8e59c
2020-08-12 19:25:24 -07:00
Jun Wu
f6d086d13b hgcommands: add debugfsync
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.

The fsync logic is put in a separate crate to avoid slow compiles.

Reviewed By: DurhamG

Differential Revision: D22992103

fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
2020-08-12 18:33:52 -07:00
Jun Wu
8ca7ab1c5a hgcommands: move debug commands to individual files
Summary:
This makes the main command table cleaner.

I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.

Reviewed By: DurhamG

Differential Revision: D22992107

fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
2020-08-12 18:33:52 -07:00
Jun Wu
bcfa8e5676 hgcommands: move version to a module
Summary: Move some native commands to independent modules.

Reviewed By: DurhamG

Differential Revision: D22992106

fbshipit-source-id: cf7751418d19d54d9dd89d9d0f79851ac11879c3
2020-08-12 18:33:52 -07:00
Jun Wu
8c51e81c97 hgcommands: move root to a module
Summary: Move some native commands to independent modules.

Reviewed By: DurhamG

Differential Revision: D22992105

fbshipit-source-id: e4fd8db3f0d6f9d2ab5be862f6d9469da7d15a93
2020-08-12 18:33:52 -07:00
Jun Wu
896671cefb hgcommands: define a macro to register command from modules
Summary:
If every command lives in their module, then we can define the "module" interface:

- run(...): run the command
- doc(): the help text
- name(): command name, with aliases

Then the macro would make command registration look simpler.

This diff changes `status` to use the pattern as an example.

Reviewed By: DurhamG

Differential Revision: D22992109

fbshipit-source-id: eaf589863092ec2eb1f8c24c1c7e425492fe1e3a
2020-08-12 18:33:52 -07:00
Jun Wu
757daa5eaf hgcommands: move commands to a directory
Summary:
As the number of commands grows, it starts making sense to move them to
individual files. Let's create a directory for them.

Reviewed By: DurhamG

Differential Revision: D22992108

fbshipit-source-id: a0556be602b832579a8e027342d5b86d9d84d257
2020-08-12 18:33:51 -07:00
Meyer Jacobs
b9f3c9c692 taggederror: Introduce taggederror-util for more ergonomic error tagging for eden error types.
Summary:
Introduce taggederror-util, which provides a new trait `AnyhowEdenExt`, which provides a method `eden_metadata` for anyhow errors and results. This method works much like `AnyhowExt::common_metadata`, but additionally supports extracting default error metadata from known `Tagged` types which are listed explicitly in the method implementation.

Extend `FilteredAnyhow` to support a configuration "metadata function", which allows swapping out `eden_metadata` for the standard `common_metadata`.

Modify Rust dispatch and Python bindings to use `AnyhowEdenExt` for metadata extraction and printing.

Modify `intentional_error` to rely on `AnyhowEdenExt` for tagging (removes `.tagged` call, no tags will be visible if `AnyhowEdenExt` is not used).

Reviewed By: DurhamG

Differential Revision: D22927203

fbshipit-source-id: 04b36fdfaa24af591118acb9e418d1ed7ae33f91
2020-08-06 19:37:25 -07:00
Xavier Deguillard
3a97764d70 revisionstore: add a new StoreResult type
Summary:
When using LFS, it's possible that a pointer may be present in the local
LfsStore, but the blob would only be in the shared one. Such scenario can
happen after an upload, when the blob is moved to the shared store for
instance. In this case, during a `get` call, the local LFS store won't be able
to find the blob and thus would return Ok(None), the shared LFS store woud not
be able to find the pointer itself and would thus return Ok(None) too. If the
server is not aware of the file node itself, the `ContentStore::get` would also
return Ok(None), even though all the information is present locally.

The main reason why this is happening is due to the `get` call operating
primarily on file node based keys, and for content-based stores (like LFS),
this means that the translation layer needs to be present in the same store,
which in some case may not be the case. By allowing stores to return a
`StoreKey` when progress was made in finding the key we can effectively solve
the problem described above, the local store would translate the file node key
onto a content key, and the shared store would read the blob properly.

Reviewed By: DurhamG

Differential Revision: D22565607

fbshipit-source-id: 94dd74a462526778f7a7e232a97b21211f95239f
2020-07-24 10:45:40 -07:00
Meyer Jacobs
e3b86cf77d debug: introduce binding layer for propagating error metadata to Python
Summary:
Implements based Rust-Python binding layer for error metadata propagation.

We introduce a new type, `TaggedExceptionData`, which carries CommonMetadata and the original (without metadata) error message for a Rust Anyhow error. This class is passed to RustError and can be accessed in Python (somewhat awkwardly) via indexing:
```
except error.RustError as e:
    fault = e.args[0].fault()
    typename = e.args[0].typename()
    message = e.args[0].message()
```
As far as I can tell, due to limitations in cpython-rs, this can't be made more ergonomic without introducing a Python shim around the Rust binding layer, which could adapt the cpython-rs classes to use whatever API we'd like.

Currently, anyhow errors that are not otherwise special-cased will be converted into RustError, with both the original error message and any attached metadata printed as shown below
```
  abort: intentional error for debugging with message 'intentional_error'
  error has type name taggederror::IntentionalError and fault None
```
We can of course re-raise the error if desired to maintain the previous behavior for handling a RustError.

If we'd like other, specialized Rust Python Exception types to carry metadata (such as `IndexedLogError`), we'll need to modify them to accept a `TaggedExceptionData` like `RustError`.

Renamed the "cause an error in pure rust command" function to `debugcauserusterror`, and instead used the name `debugthrowrustexception` for a command which causes an error in rust which is converted to a Python exception across the binding layer.

Introduced a simple integration test which exercises `debugthrowrustexception`.

Added a basic handler for RustError to scmutil.py

Reviewed By: DurhamG

Differential Revision: D22517796

fbshipit-source-id: 0409489243fe739a26958aad48f608890eb93aa0
2020-07-16 19:30:00 -07:00
Durham Goode
76df783c93 configs: implement user sharding
Summary: Adds support for sharding based on user name.

Reviewed By: quark-zju

Differential Revision: D22537540

fbshipit-source-id: 962f9582c8947335dc9d9d29c500d8c09df69878
2020-07-16 09:07:53 -07:00
Meyer Jacobs
4ccbd119d7 debug: introduce error classification and metadata propagation
Summary:
Introduce new rust library, taggederror, which contains utilities for attaching metadata to errors. The library provides two main methods for attaching metadata to an error, the TaggedError wrapper type, and the AnyhowExt trait methods. Provides a struct, CommonMetadata, which contains all the metadata types introduced by taggederror (fault, transience, category, and typename), which can also be attached individually (and the same pattern can be used to attach other metadata).

Introduce a new native rust command, debugthrowrustexception, which causes the command to return an error, with some attached metadata.

Modify hg rust native command dispatch error handling to use debug formatter to print anyhow::Error errors. This will print out the source chain, contexts, and backtrace if available, which will cause the metadata we attach as a wrapper error or context to be printed.

Reviewed By: DurhamG

Differential Revision: D22420941

fbshipit-source-id: d38c5a10b686d86b69a2c0a19f5bcbf4ca24dff6
2020-07-15 10:03:10 -07:00
Durham Goode
28ddd1d1cc configs: add hg debugdynamicconfig --canary devvmXXX.prnY support
Summary:
Previously you could only canary locally on a devserver by setting an
environment variable. Let's add a --canary flag to debugdynamicconfig that
accepts a host.  Hg will ssh to that host and run the configerator cli to grab
the canaried config from that host.

Reviewed By: quark-zju

Differential Revision: D22535509

fbshipit-source-id: af1c21d8402c4e729769e50388d913bf52b66b89
2020-07-15 01:14:30 -07:00
Arun Kulshreshtha
41e68f46d3 edenapi: add blocking API
Summary: Add a new `EdenApiBlocking` trait that exposes blocking versions of the `EdenApi` trait's methods, for use in non-async code.

Reviewed By: quark-zju

Differential Revision: D22305396

fbshipit-source-id: d0f3a73cad1a23a4f0892a17f18267374e63108e
2020-07-09 13:08:27 -07:00
Arun Kulshreshtha
fdba0b98c2 edenapi: rename to edenapi_old
Summary: Move old EdenAPI crate to `scm/lib/edenapi/old` to make room for the new crate. This old code will eventually been deleted once all references to it are removed from the codebase.

Reviewed By: quark-zju

Differential Revision: D22305173

fbshipit-source-id: 45d211340900192d0488543ba13d9bf84909ce53
2020-06-30 21:10:41 -07:00
Durham Goode
e5a9446057 configs: exit debugdynamicconfig early if .hg isn't writable
Summary:
If we're running commands from a user that only has read access, the
debugdynamicconfig commands are going to fail. Let's exit early and quickly if
that's the case, instead of spending a lot of cpu generating a config only to
fail.

Reviewed By: quark-zju

Differential Revision: D22244127

fbshipit-source-id: 24f806772ba5c08e400efb3abc7ebda228d473a5
2020-06-30 09:50:44 -07:00
Durham Goode
977155ef99 configs: fetch remote configs during dynamic config generation
Summary:
Fetches configs from a remote endpoint and caches them locally. If the
remote endpoint fails to respond, we use the cached version.

Reviewed By: quark-zju

Differential Revision: D22010684

fbshipit-source-id: bd6d4349d185d7450a3d18f9db2709967edc2971
2020-06-30 09:50:44 -07:00
Durham Goode
75c96be5ff configs: don't write dynamic config if it hasn't changed
Summary:
We kick off a background process every 15 minutes or so to compute the
new dynamicconfig. If the config hasn't changed, we should just touch the file
instead of rewriting the whole thing.

Reviewed By: quark-zju

Differential Revision: D21653098

fbshipit-source-id: 9d53d2a636cff082cd048994255cc809ce1b0221
2020-05-20 13:35:28 -07:00
Durham Goode
9f6f200a08 configs: version dynamic configs
Summary:
If we release a new version of Mercurial, we want to ensure that it's
builtin configs are used immediately. To do so, let's write a version number
into the generated config file, and if the version number doesn't match, we
force a synchronous regeneration of the config file.

For now, if regeneration fails, we just log it. In the future we'll probably
throw an exception and block the user since we want to ensure people are running
with modern configuration.

Reviewed By: quark-zju

Differential Revision: D21651317

fbshipit-source-id: 3edbaf6777f4ca2363d8617fad03c21204b468a2
2020-05-20 13:35:28 -07:00
Jun Wu
88a9982eb6 hgcommands: make version a native command
Reviewed By: DurhamG

Differential Revision: D19803761

fbshipit-source-id: 32f77cc667375e537e1ff70316329251359ae6ed
2020-05-18 18:50:42 -07:00
Durham Goode
97d84e3b5d configs: move hgrc.dynamic to always be in the shared repo
Summary:
Instead of trying to maintain two hgrc.dynamic's for shared repositories,
let's just always use the one in the shared repo. In the long term we may be
able to get rid of the working-copy-specific hgrc entirely.

This does remove the ability to dynamically configure individual working copies.
That could be useful in cases where we have both eden and non-eden pointed at
the same repository, but I don't think we rely on this at the moment.

Reviewed By: quark-zju

Differential Revision: D21333564

fbshipit-source-id: c1fb86af183ec6dc5d973cf45d71419bda5514fb
2020-05-05 18:19:10 -07:00
Durham Goode
3ac2be361a configs: move fbrules to a Facebook only part of the crate
Summary:
We'll be adding a bunch of Facebook specific configuration and values
here. Let's move it to someplace not open source.

Reviewed By: quark-zju

Differential Revision: D21241038

fbshipit-source-id: 2ac9cdce40b1b46f15f171d9d1f6b6692dcd29bf
2020-05-01 13:17:21 -07:00
Carolyn Busch
4eeab3b81b Update cpython to 0.5
Summary:
D21270958 updated the cpython, python27-sys, and python3-sys crates to 0.5. Update
the Mercurial cargo dependencies to match.

Reviewed By: xavierd

Differential Revision: D21281875

fbshipit-source-id: ccad68749a25d11240351b5faeef27cb9c693456
2020-04-28 11:47:41 -07:00
Durham Goode
bfe0c8d7ab configs: add dynamic config generator
Summary:
We want Mercurial to become more responsible for it's own
configuration, instead of relying on chef and other means. To do so, let's
introduce a new `hg debugdynamicconfig` that can generate dynamic configs for
a given repository based on various states, like what tier it's in or what shard
that machine is in.  By default it generates to '.hg/hgrc.dynamic' for the given
repository.

Currently it just sets the hostgroup config.

Future diffs will make Mercurial consume this config, and possibly have Mercurial
call this command asynchronously when it notices the file is out-of-date.

Reviewed By: quark-zju

Differential Revision: D20828132

fbshipit-source-id: 6f5bf749f5b04e0a5989d6dc19ee788c2e47f88f
2020-04-14 21:22:26 -07:00
Jun Wu
4d951d777b util: add more APIs for pytracing
Summary: This makes the tracing features easier to use.

Reviewed By: DurhamG

Differential Revision: D19797703

fbshipit-source-id: fb5cb17cd389575cf0134a708bcd9df3b90e9ab4
2020-04-07 21:35:31 -07:00
Jun Wu
817f748113 debugindexedlogdump: output in a streaming way
Summary:
Do not convert the entire output to a string. This makes `debugindexedlog dump`
a good test case for native pager support - it takes a while to write the full
output for a large input.

Reviewed By: DurhamG

Differential Revision: D20885567

fbshipit-source-id: 35ed8f68dff1916f0833577c3cf2a52cbf2a658c
2020-04-07 15:57:06 -07:00
Jun Wu
6ffdcebadf tracing: write some blackbox events as tracing events
Summary:
This is the start of migrating blackbox events to tracing events. The
motivation is to have a single data source for log processing (for simplicity)
and the tracing data seems a better fit, since it can represent a tree of
spans, instead of just a flat list. Eventually blackbox might be mostly
a wrapper for tracing data, with some minimal support for logging some indexed
events.

Reviewed By: DurhamG

Differential Revision: D19797710

fbshipit-source-id: 034f17fb5552242b60e759559a202fd26061f1f1
2020-03-19 10:23:24 -07:00
Xavier Deguillard
bb30c40375 types: address clippy warnings
Summary:
Clippy had 3 sources of warnings in this crate:
 - from_str method not in impl FromStr. We still have 2 of them in path.rs, but
   this is documented as not supported by the FromStr trait due to returning a
   reference. Maybe we can find a different name?
 - Use of mem::transmute while casts are sufficient. I find the cast to be
   ugly, but they are simply safer as the compiler can do some type checking on
   them.
 - Unecessary lifetime parameters

Reviewed By: quark-zju

Differential Revision: D20452257

fbshipit-source-id: 94abd8d8cd76ff7af5e0bbfc97c1e106cdd142b0
2020-03-16 14:58:21 -07:00
Xavier Deguillard
2e4742cefc revisionstore: rename DataStore traits to HgIdDataStore
Summary: This makes it clear that these traits are dealing with Mercurial Key.

Reviewed By: quark-zju

Differential Revision: D20445626

fbshipit-source-id: d5acbf442e9407b973e95e40af69b5a61bff0a4d
2020-03-13 19:03:27 -07:00
Jun Wu
7a7f98f1b2 configparser: migrate from Bytes to Text
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.

Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.

This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.

This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.

Reviewed By: markbt

Differential Revision: D20432938

fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
2020-03-13 10:51:41 -07:00
Jun Wu
e7ed737a64 hgcommands: make env_logger show exit code
Summary: Move the scope of spans so the exit code is shown.

Reviewed By: xavierd

Differential Revision: D20286516

fbshipit-source-id: f39cbf60c86ea19a1bb0a09958748f04ff6a42e8
2020-03-10 14:16:37 -07:00
Jun Wu
bb9023c2cb hgcommands: move env_logger initialization to hgcommands
Summary:
Previously env_logger is only initialized if Python is initialized.
This diff makes env_logger initialized for Rust native commands.

Reviewed By: xavierd

Differential Revision: D20286517

fbshipit-source-id: 18fee96c2b41db1da9648d615d1e18809de90a63
2020-03-10 14:16:37 -07:00
Genevieve Helsel
528015f9fe allow more hg fastpath cases
Reviewed By: simpkins

Differential Revision: D20143888

fbshipit-source-id: 4b1a73159bde6835626ad1766b2cf9dcd2faf6c4
2020-03-02 07:43:39 -08:00
Jun Wu
918672b106 tracing-collector: support owned strings in TreeSpans
Summary:
TreeSpans used to use `&str`, which adds a lifetime to the struct, making it
harder to be used in the Python land. Use a type parameter so TreeSpans<String>
can be used.

Reviewed By: DurhamG

Differential Revision: D19797708

fbshipit-source-id: c66429abfaf16d876151ca6f29da976bed91485d
2020-02-28 09:16:14 -08:00
Jun Wu
3e0b781197 py3: only use binary stdin/stdout/stderr
Summary:
Drop stdoutbytes/stdinbytes. They make things unnecessarily complicated
(especially for chg / Rust dispatch entry point).

The new idea is IO are using bytes. Text are written in utf-8 (Python 3) or
local encoding (Python 2). To make stdout behave reasonably on systems not
using utf-8 locale (ex. Windows), we might add a Rust binding to Rust's stdout,
which does the right thing:
- When writing to stdout console, expect text to be utf-8 encoded and do proper decoding.
- Wehn writing to stdout file, write the raw bytes without translation.

Note Python's `sys.stdout.buffer` does not do translation when writing to stdout console
like Rust's stdout.

For now, my main motivation of this change is to fix chg on Python 3.

Reviewed By: xavierd

Differential Revision: D19702533

fbshipit-source-id: 74704c83e1b200ff66fb3a2d23d97ff21c7239c8
2020-02-03 18:26:57 -08:00
Mateusz Kwapich
e2dc4e8014 diff: update for py3
Summary:
All diff functions are (bytes, bytes) -> bytes to preserver
the original file encoding.

Because of that I had to add ui.writebytes output function that accepts
bytes for terminal output.

Reviewed By: farnz

Differential Revision: D19656673

fbshipit-source-id: b9a1e4361e825fc8c2313e8402c2bbe00f490dd4
2020-01-31 13:00:23 -08:00
Mark Thomas
1e63f205f4 rust-cpython: allow compilation for both py2 and py3
Summary:
Set up the `cpython-ext` and `hgcommands` libraries so that they can compile
against py2 and py3 versions of rust-cpython.  Make py2 the default so
that cargo test still works.

Reviewed By: singhsrb

Differential Revision: D19615656

fbshipit-source-id: 3403e7077deb3c0a9dfe0e3b7d4f4ad1da73bba3
2020-01-28 20:17:20 -08:00
Adam Simpkins
ad957e7803 py3: update Rust hgcommands code to pass argv to python as Str
Summary:
Update the Rust hgcommands code to pass the command line arguments into the
Python logic as `Str` types, so that this will be Unicode `str` objects when
using Python 3.

Reviewed By: xavierd

Differential Revision: D19596739

fbshipit-source-id: 7cdfd44a1c4ce8b0f86d20b634d9b27eab822b2d
2020-01-28 15:58:37 -08:00
Jun Wu
8703970cea py3: update Cargo.toml to make py3 buildable
Summary:
This makes `make hg3` work. It requires cleaning up the `build` directory when
switching between py2 and py3 build, which will be fixed later.

Reviewed By: DurhamG

Differential Revision: D19604824

fbshipit-source-id: 060ff313420126a5dba935c4451b45dc9af45f13
2020-01-28 13:39:38 -08:00
Mark Thomas
4fe02f3607 bindings: update to rust-cpython 0.4
Summary: Update to the new version of rust-cpython.  This supports `list.append`, so make use of it.

Reviewed By: xavierd

Differential Revision: D19590905

fbshipit-source-id: 03609d4f698ae8e4380e82b8144caaa205b4c2d4
2020-01-28 10:46:33 -08:00
Xavier Deguillard
283b120bb6 pyconfigparser: use PyPath instead of PyByte
Summary:
Also, add a util::path::strip_unc function that is more clear than the
normalize_for_display

Reviewed By: DurhamG

Differential Revision: D19595961

fbshipit-source-id: 330bcb708bf64320a3562d79db685d6cb1e14f16
2020-01-28 10:14:14 -08:00
Xavier Deguillard
ad58839ca1 py3/rust: use Str type in cliparser and hgcommands
Reviewed By: sfilipco

Differential Revision: D19581176

fbshipit-source-id: e92e5c2538537ec16da25a9819c9a097a24a4d6e
2020-01-27 16:50:13 -08:00
Jun Wu
de5772cb30 indexedlog: replace Log's dir with a new GenericPath type
Summary:
This allows us to incrementally abstract away parts related to filesystem from
Log. For example, instead of using std::fs to access filesystem directly, use
methods on the GenericPath instead.

Right now the main motivation is to add support for "multi-logs" sharing a single meta
file. To do that, methods reading and writing metas are moved to the
GenericPath type.

This also unifies the open functions. Now OpenOptions::create_in_memory can be
private. Empty in-memory Logs can be opened via `open(())`.

Reviewed By: DurhamG

Differential Revision: D19431784

fbshipit-source-id: dbdf94f60261e09f131c6fdd9fe3b99242a28af5
2020-01-17 03:57:59 -08:00
Jun Wu
3895af590a edenfs-client: use clidispatch IO and Error types
Summary:
This simplifies the error handling and makes it more compatible with things that
might capture the output, including the Python testing framework.

Reviewed By: DurhamG

Differential Revision: D19325642

fbshipit-source-id: 53de8b9a8294219e2b8e62831dce236841bd4cbb
2020-01-13 14:23:34 -08:00
Jun Wu
12eaf82291 hgcommands: add edenfs status fast path
Summary: Make `hg status` use the EdenFS Thrift path, similar to the telemetry wrapper.

Reviewed By: DurhamG

Differential Revision: D19325641

fbshipit-source-id: 14777a252d7cb433316511a2a1f1a6649e9cb020
2020-01-13 14:23:34 -08:00
Jun Wu
6da344eee2 hgcommands: avoid forked processes to override trace output
Summary:
The environment variable `EDENSCM_TRACE_OUTPUT` specifies where to write the
tracing output for the current command. Environment variables are inherited
by subprocesses by default. That is undesirable because another hg command
(triggered by hook, or background maintaince) will rewrite the trace output.

Avoid it by unsetting EDENSCM_TRACE_OUTPUT for subprocesses.

Reviewed By: xavierd

Differential Revision: D18892973

fbshipit-source-id: 575b6c0df2e7a0775172893e4aa72ca33fa4658c
2019-12-09 13:09:25 -08:00
Jun Wu
541d4bf625 commands: avoid infinite loop when collecting process tree info
Summary:
We have seen cases on Windows where the hg process gets stuck in
CreateToolhelp32Snapshot. Let's be defensive and always exit the loop.

Reviewed By: singhsrb

Differential Revision: D18729720

fbshipit-source-id: fb8602ce231eec01b6b42c6759849d56e5db2030
2019-11-27 11:04:30 -08:00
David Tolnay
d1d8fb939a Switch from failure::Fail trait to std::error::Error for errors
Summary:
This diff replaces eden's dependencies on failure::Error with anyhow::Error.

Failure's error type requires all errors to have an implementation of failure's own failure::Fail trait in order for cause chains and backtraces to work. The necessary methods for this functionality have made their way into the standard library error trait, so modern error libraries build directly on std::error::Error rather than something like failure::Fail. Once we are no longer tied to failure 0.1's Fail trait, different parts of the codebase will be free to use any std::error::Error-based libraries they like while still working nicely together.

Reviewed By: xavierd

Differential Revision: D18576093

fbshipit-source-id: e2d862b659450f2969520d9b74877913fabb2e5d
2019-11-22 08:53:31 -08:00
Jun Wu
6b6b6cf36d blackbox: add Event::Tags and its index
Summary:
Add Event::Tags so we can tag commands with one or more names.
Add index for the tags so we can search them more efficiently.

As we're here, update hgcommands to read EDENSCM_BLACKBOX_TAGS and write
Event::Tags automatically just after Event::Start.

Differential Revision: D18610733

fbshipit-source-id: c69b6e3d34b317e92a1c4d76f29e832b6b01c4fa
2019-11-21 08:36:10 -08:00
Jun Wu
f19a760f5f blackbox: remove IndexFilter
Summary: Migrate callsites to use the new pattern matching filtering APIs.

Differential Revision: D18610738

fbshipit-source-id: 242182f4e6a58bbff5dddbb177287921c85519bc
2019-11-21 08:36:09 -08:00
Jun Wu
ec9d925b69 blackbox: add timestamp field to Start and Finish event
Summary:
See the previous diff. As part of migrating away from IndexFilter. There needs
to be a way to filter commands by absolute clock time using pattern matching
against Events.  To achieve that, add the timestamp field to Start and Finish
events.

Differential Revision: D18610735

fbshipit-source-id: 9e1898a87bc93e47d0f0c77342428b5c42ef6f24
2019-11-21 08:36:09 -08:00
David Tolnay
b1793a4416 rust: Rename Fallible<T> to Result<T>
Summary:
This diff is preparation for migrating off of failure::Fail / failure::Error for errors in favor of errors that implement std::error::Error. The Fallible terminology is unique to failure and in non-failure code we should be using Result<T>. To minimize the size of the eventual diff that removes failure, this codemod replaces all use of Fallible with Result by:

- In modules that do not use Result<T, E>, we import `failure::Fallible as Result`;
- In modules that use a mix of Result<T, E> and Fallible<T> (only 5) we define `type Result<T, E = failure::Error> = std::result::Result<T, E>` to allow both Result<T> and Result<T, E> to work simultaneously.

Reviewed By: Imxset21

Differential Revision: D18499758

fbshipit-source-id: 9f5a54c47f81fdeedbc6003cef42a1194eee55bf
2019-11-14 14:11:01 -08:00
Adam Simpkins
ab3a7cb21f Move fb-mercurial sources into an eden/scm subdirectory.
Summary:
In preparation for merging fb-mercurial sources to the Eden repository,
move everything from the top-level directory into an `eden/scm`
subdirectory.
2019-11-13 16:04:48 -08:00