Summary:
Move these conversion related function and trait out of `hg` module so EdenFS can use it too. Changes:
* Moved `get_opt`, `get_or` and `get_or_default` directly into `ConfigSet`.
* Moved `FromConfigValue` and `ByteCount` into `configparser::convert`.
Reviewed By: quark-zju
Differential Revision: D26355403
fbshipit-source-id: 9096b7b737bc4a0cccee1a3883e89a323f864fac
Summary:
In our upcoming migration away from chef/static rc files, we'll be
marking certain files as "allowed". Our hope is that that list only includes
things like .hg/hgrc, ~/.hgrc, etc.
There are cases however where it's convienent to continue to use chef, for
instance when we condition on machine type. To support this, let's add an
allowed_config option, which will allow configs from non-supported locations.
This will also be useful when remediating issues that come up when we start
enforcing allow_location, without rolling back the entire thing.
Reviewed By: quark-zju
Differential Revision: D26233451
fbshipit-source-id: 71789e0361923a6f80de4aef7f012afc0269440d
Summary:
The original migration strategy with dynamicconfig was to fix configs
one by one until the dynamicconfig values matched the chef/static ones, then we
can turn off chef/static configs. This looks to be too much work, so we're going
to try a different strategy of just turning off all chef/static configs on a
small number of hosts and seeing what breaks.
The legacylist and disallowlist configs were part of the old strategy, and they
make it more complicated to fix dynamicconfig mismatches, so let's get rid of
them.
Reviewed By: quark-zju
Differential Revision: D26208548
fbshipit-source-id: 63171f1f16aa0498c0eefa994dffaeb8e0cc0d72
Summary:
quark-zju made it much safer to rollout configs recently by enforcing
time-sharding, but one little downside is that it becomes harder to test
them locally (or at diff-time, which I'm working on adding) because the time
shard will usually not pass.
This diffs add an environment variable to force all shards to pass.
Reviewed By: DurhamG
Differential Revision: D25925207
fbshipit-source-id: 343b90165ad2b6bae64b6821eae95c58f7d79698
Summary:
```
warning: variable does not need to be mutable
--> eden/scm/lib/configparser/src/config.rs:448:21
|
448 | let mut values_copy = values.clone();
| ----^^^^^^^^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default
```
Reviewed By: sfilipco
Differential Revision: D25625453
fbshipit-source-id: 8475056a87095f9ba633282666e6d3fee864074b
Summary:
We want to start disallowing non-approved config files from being
loaded. To do that, let's update the config verifier to accept an optional list
of allowed locations. If it's provided, we delete any values that came from a
disallowed location.
This will enable us to prune our config sources down to rust configs,
configerator configs, .hg/hgrc, and ~/.hgrc.
Reviewed By: quark-zju
Differential Revision: D25539738
fbshipit-source-id: 0ece1c7038e4a563c92140832edfa726e879e498
Summary:
The config verifier would remove items from the values list if they
were disallowed. To do this, it iterated through the values list backwards,
removing bad items. In some cases it stored the index of a bad value for later
use, but because it was iterating backwards and removing things, the indexed it
stored might not be correct by the time the loop is done. To fix this, let's go
back to iterating forwards.
Reviewed By: quark-zju
Differential Revision: D25539737
fbshipit-source-id: 87663f3c162c690f3961b8075814f3467916cb4b
Summary: This preserves the `--noninteractive` flag used by some tools.
Reviewed By: DurhamG
Differential Revision: D24040789
fbshipit-source-id: 8d50f3f3ce6b2015f0ef6c3bd1b4fbb874d0ea7d
Summary:
This restores the compatibility of setting up merge tools using the `ui.merge`
config while still limiting the default `editmerge` tool to interactive
sessions.
Reviewed By: sfilipco
Differential Revision: D24377259
fbshipit-source-id: 3d2befba412b824fc985ddffa131e339644178c2
Summary: Make it testable by allowing specifying paths to load as user hgrc.
Reviewed By: sfilipco
Differential Revision: D24377258
fbshipit-source-id: 969028df64d55ad1f1304e35675d84595ed6a2bf
Summary:
Dynamicconfig can generate configs two ways, 1) via `hg
debugdynamicconfig` and 2) synchronously in-process in an hg command when it
detects that the dynamicconfig is completely missing or has the wrong version
number.
In the first case, dynamicconfig gets the repo name from the standard config
object loaded by the hg dispatch. In the second case, the standard config
object isn't even loaded yet, so dynamicconfig does a mini-load of the user and
repo hgrcs so it can get the repo name and user name (needed for dynamic
conditions).
Unfortunately the second code path computed the wrong path (it had two .hg/'s)
which meant the reponame and user name were always none. This meant that the
dynamicconfig on disk could randomly be either computed with or without a
reponame.
Let's fix the path computation, and add a test. We may want to make
dynamicconfig fail if no repo name is passed, but I'm not sure if we'll want to
support no-repo configuration at some point.
This didn't cause a problem for most people, since it would only happen during a
hg version number change, and 15 minutes later the background 'hg
debugdynamiconfig' process would fix it up. It did affect sandcastle though,
since it often creates new repositories and acts on them immediately.
Reviewed By: quark-zju
Differential Revision: D23955628
fbshipit-source-id: c922f4b523d19df9223aa28c97700b7011fc03eb
Summary:
This imports the async-compression crate. We have an equivalent-ish in
common/rust, but it targets Tokio 0.1, whereas this community-supported crate
targets Tokio 0.2 (it offers a richer API, notably in the sense that we
can use it for Streams, whereas the async-compression crate we have is only for
AsyncWrite).
In the immediate term, I'd like to use this for transfer compression in
Mononoke's LFS Server. In the future, we might also use it in Mononoke where we
currently use our own async compression crate when all that stuff moves to
Tokio 0.2.
Finally, this also updates zstd: the version we link to from tp2 is actually
zstd 1.4.5, so it's a good idea to just get the same version of the zstd crate.
The zstd crate doesn't keep a great changelog, so it's hard to tell what has changed.
At a glance, it looks like the answer is not much, but I'm going to look to Sandcastle
to root out potential issues here.
Reviewed By: StanislavGlebik
Differential Revision: D23652335
fbshipit-source-id: e250cef7a52d640bbbcccd72448fd2d4f548a48a
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: zertosh
Differential Revision: D23568779
fbshipit-source-id: 477200f35b280a4f6471d8e574e37e5f57917baf
Summary: This crate would fail to build without the "fb" feature because `serde_json` was listed as an optional dependency (but is used in a way that isn't conditional on the `fb` feature). This diff makes the dependency non-optional, and also silences several dead code warnings that are emitted when building without the "fb" feature.
Reviewed By: quark-zju
Differential Revision: D23386786
fbshipit-source-id: b00a8b0b8b0b978c1cfab2838629fcb388a076e9
Summary:
Corp has a different concept of tier than prod. Let's load the corp
tier into our tier set as well.
Reviewed By: quark-zju
Differential Revision: D23354056
fbshipit-source-id: c9543b8253f042c7b1224578e0687b4bdf21738e
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/45
Fix referring to 'version' without proper codegen by making 'version' compile
without codegen. This fixes configparser test when version/src/lib.rs was not
generated.
Make unneeded deps without 'fb' feature optional.
This would hopefully fix the "EdenSCM Rust Libraries" GitHub workflow.
Reviewed By: DurhamG
Differential Revision: D23269864
fbshipit-source-id: f9e691fe0a75159c4530177b8a96dad47d2494a9
Summary:
We want to eventually get rid of system and repo configs, but for now
they should take precedence over the dynamicconfigs. Previously we relied on
validation to remove any entries from dynamicconfig that interferes with a
system rc config, but in some code paths we didn't run that validation (like if
we loaded configs purely from Rust).
Let's just make dynamicconfig load before system configs. If validation doesn't
run, we might miss the case where dynamicconfig sets a value and the system rc
doesn't. But that's probably fine.
Reviewed By: quark-zju
Differential Revision: D23305711
fbshipit-source-id: 77b5f49d348cfa116694a641ed17e6d1184a81ab
Summary:
Dynamicconfigs are now part of our critical path. Let's remove the
option to not load them. This also let's us get rid of a circularl dependency
where loading dynamicconfigs required having already loaded some configs. This
will let us move dynammicconfig loading to be before system rc loading in a
later diff.
Reviewed By: sfilipco
Differential Revision: D23309090
fbshipit-source-id: 5138059a8ed944c3616007e7c1289b6a57be0e65
Summary:
Dynamicconfig was throwing errors if hgrc.dynamic wasn't writable.
Let's eat those errors for normal read operations. We still treat it as an error
for straight hg debugdynamicconfig invocations.
Reviewed By: quark-zju
Differential Revision: D23301100
fbshipit-source-id: ed0bd1282d2c7ee747f0909c238a5fa07b7bc9bc
Summary:
Dynamicconfigs compares the timestamp of config files with the current
timestamp to determine when to regenerate. If the timestamp of the config file
is newer than the current timestamp, Rust throws an exception. Let's handle that
case and treat it as if the file was just created instead of crashing.
Reviewed By: quark-zju
Differential Revision: D23230216
fbshipit-source-id: ca185de7dfca46953e04ec08c84668eda6d749bd
Summary: The python code stripped new lines but the Rust code did not.
Reviewed By: singhsrb
Differential Revision: D23167515
fbshipit-source-id: add33ec6e4cfd9169e6fef8208490e0aeede38bd
Summary:
This new disallowlist will let us specify config section.key's which
should not be accepted from old rc files. This will let us incrementally disable
loading of those configs from the old files, which will then let us delete them
from the old rc's and eventually delete the old rc's entirely.
This diff also removes hgrc.local and hgrc.od from the list of configs we
verify, since those are not on the list of configs that need to be removed in
this initiative.
Reviewed By: quark-zju
Differential Revision: D23065595
fbshipit-source-id: 5cd742d099efd651174cab5e87bb7cdc4bae8054
Summary:
This threads the calls to load_dynamic and load_repo through the Rust
layer up to the Python bindings. This diff does 2 notable things:
1. It adds a reload API for reloading configs in place, versus creating a new
one. This will be used in localrepo.__init__ to construct a new config for the
repo while still maintaining the old pinned values from the copied ui.
2. It threads a repo path and readonly config list from Python down to the Rust
code. This allows load_dynamic and load_repo to operate on the repo path, and
allows the readonly filter to applied to all configs during reloading.
Reviewed By: quark-zju
Differential Revision: D22712623
fbshipit-source-id: a0f372f4971c5feac2f20e89a0fb3fe6d4a65d6f
Summary:
In a future diff we'll enable dynamic and repo config loading purely
from Rust. To do so we need load functions for both cases. A future diff will
call these.
The dynamicconfig loading is based off the Python equivalent in uiconfig.py
Reviewed By: quark-zju
Differential Revision: D22712624
fbshipit-source-id: ff46f6315fb80d4cd9e31d875ac60264563b12f2
Summary:
Previously load_system would skip loading if HGRCPATH was present and
then load_user would actually load the HGRCPATH. In an upcoming diff I add
load_dynamic, which happens after system but before user. The tests for
dynamicconfig depend on HGRCPATH being loaded when load_dynamic runs, so let's
move HGRCPATH loading up to load_system.
Reviewed By: quark-zju
Differential Revision: D22712627
fbshipit-source-id: 91175d9d7f85b9392ffea4af815a4facebbfe7c1
Summary:
In a future diff we'll allow an outside caller to pass an Options down
to configparsers::hg::load() so that filters can be applied during loading. Inside
hg::load() we need to use the options multiple times with different values, so
let's make Options clonable.
Reviewed By: quark-zju
Differential Revision: D22712626
fbshipit-source-id: 975145f38d35afe7d4a6c8e87071b0fb0ae74797
Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.
Future diffs will move config loading in as well.
Reviewed By: quark-zju
Differential Revision: D22590208
fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.
Reviewed By: quark-zju
Differential Revision: D22587237
fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
Summary:
We had two spots that loaded system and user configs, one in the
pyconfigparser layer, and one in the pure rust config layer. In an upcoming diff
I'd like to move dynamicconfig loading down into the pure rust layer, so let's
unify these.
Reviewed By: quark-zju
Differential Revision: D22585554
fbshipit-source-id: 0cea7801ae1d5a3a3c12b80ee23b37f9e690e2bc
Summary:
Previously we would audit all configs and report them if the
dynamicconfig did not match the rc-file config. Now that dynamicconfigs are
widely deployed, let's switch this around to auditing only configs we know have
had issues. This will let us start adding new configs via dynamicconfigs instead
of via the legacy staticfiles and chef, before we've finished migrating all the
legacy configs over.
Reviewed By: quark-zju
Differential Revision: D22401865
fbshipit-source-id: 5c41c674d39c8113b2a40da61e020e8a33c39312
Summary:
The dynamicconfig logic had an assert checking that a config was not
added by the dynamicconfig twice. This isn't actually a valid assert since the
caller could load the config as many times as it wanted.
This happened in hg doctor, where it loaded the ui once manually (without a repo
object) then it loaded it again during the creation of the repo object. I went
ahead and cleaned up hg doctor to not do this, but let's get rid of the assert
anyway.
Reviewed By: quark-zju
Differential Revision: D22194273
fbshipit-source-id: 9491d35fe14523ad3d9cb69b4ca0d615149dd0f0
Summary: Replace usages of whitelist/blacklist with include/exclude/filter/allow. These terms are more descriptive and less likely to contribute to racial stereotyping. More context: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2926049127516414/
Reviewed By: kulshrax
Differential Revision: D22039298
fbshipit-source-id: 255c7389ee5ce5e54bbccdfb05ffa4cafc6958e5
Summary:
There was a bug where if the dynamicconfig chose a value that matched
one of the ported rc files, but it did not match the final value of that config
(from a not-yet-ported rc file), it would chose the dynamicconfig value instead
of the original final value. Let's drop the dynamicconfig value in this case as
well.
Reviewed By: quark-zju
Differential Revision: D21674469
fbshipit-source-id: efcd36e9602e16210999ec8c88e86b1d7ee355e4
Summary:
At clone time we apply dynamic configs in memory, instead of loading
them from disk. The validation logic operated on the config value's location
field, which isn't set for data that comes from in memory. So we need to update
the validation logic to also consider the value source if location is not set.
Reviewed By: quark-zju
Differential Revision: D21664205
fbshipit-source-id: 8460c58c6d654780048de51ada8178c70ff0a9e6
Summary:
Update `contrib/check-code.py` to Python 3.
Mostly it was already compatible, however stricter regular expression parsing
revealed a case where one of our tests wasn't working, and as a result lots of
instances of `open(file).read()` existed that this test should have caught.
I have fixed up most of the instances in the code, although there are many
in the test suite that I have ignored for now.
Reviewed By: quark-zju
Differential Revision: D21427212
fbshipit-source-id: 7461a7c391e0ade947f779a2b476ca937fd24a8d
Summary:
Implements an ensure_location_supersets function who's goal is to
verify that a given config location specifies the exact same configs as a given
set of other locations. Any inconsistencies are removed from the config and
reported to the caller.
This will be used to ensure our dynamic configs match our existing rc file
configs exactly, before we delete the file configs.
Reviewed By: quark-zju
Differential Revision: D21240837
fbshipit-source-id: e2c8ec054a3696d2cf02e65c212ad886c5117253
Summary:
A future diff will want to generate configs programmatically and write
them to a file. Let's add write support to ConfigSet.
Reviewed By: quark-zju
Differential Revision: D20828133
fbshipit-source-id: 702f6f9bdfdf99ef25c6e1c0ab33373a4b6508fe
Summary:
Since we have all the integer types, let's also allow float types in the
config.
Reviewed By: kulshrax
Differential Revision: D20697007
fbshipit-source-id: 21fa264d24c0f63c233f47c3bcfb2448b4c05c70
Summary:
Clippy complains about 3 things:
- Using raw pointers in a public function that is not declared as unsafe. This
happens for C exported ones, this feels like a warning, so I haven't changed
it.
- Using .map(...).unwrap_or(<default value constructed>). The recommendation
is to use .unwrap_or_default().
- Single match instead of if let, the latter makes code much shorter.
Reviewed By: quark-zju
Differential Revision: D20452751
fbshipit-source-id: 8eeff7581c119c651ca41d8117f1f70f15774833
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.
Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.
This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.
This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.
Reviewed By: markbt
Differential Revision: D20432938
fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
Summary:
This makes it easier to further migrate to `Text` interface.
Dependent crate (`auth`) is updated.
Reviewed By: markbt
Differential Revision: D20432941
fbshipit-source-id: 1dc29d52c9b17ce14676ef0555470c6d36a09c2b
Summary:
Compiling it on Windows produced a bunch of warning due to
`hgrc_configset_load_path` not being compiled on it. Fixed it so it no longer
depends on Unix specific imports.
Reviewed By: quark-zju
Differential Revision: D20241102
fbshipit-source-id: 3002f961191fbb9bc51aa9ac1154d6d50bd7fe23
Summary:
The bytes 0.5 is a depencency of newer tokio, it's also newer, and thus better.
Staying on 0.4 means that copies between Bytes 0.4 and 0.5 need to be done,
this will be especially bad in the LFS code since 10+MB buffer will have to be
copied...
One main API change is for the configparser. The code used to take Into<Bytes>
for the keys, I switched it to AsRef<[u8]>.
For hg_memcache_client, an extra copy is performed to build a Delta, since this
code uses an old tokio, and is being replaced right now, the effort of
switching to a new tokio and new bytes was not deemed worth it, the copy will
do for now.
Reviewed By: dtolnay
Differential Revision: D20043137
fbshipit-source-id: 395bfc3749a3b1bdfea652262019ac6a086e61e0
Summary:
In the cpython bindings, the Rust String can take both PyBytes and PyUnicode
strings, which is perfect for Python3 compatibility as string literals are
PyBytes in Python2 and PyUnicode in Python3. The return values are kept as
Bytes for now as changing this is a much larger change in itself.
Other approaches tried:
- Using PyUnicode as input/output: an extremely large codemod had to be done,
with very little benefits
- Using String as output: since we do have some configs that are unicode, on
the Python side, the output might either be bytestrings or unicode, leading
to weird bugs.
Reviewed By: DurhamG
Differential Revision: D18650466
fbshipit-source-id: aebdf30590dcae40b7df2787e5ece88e2ec9395c
Summary:
This diff replaces eden's dependencies on failure::Error with anyhow::Error.
Failure's error type requires all errors to have an implementation of failure's own failure::Fail trait in order for cause chains and backtraces to work. The necessary methods for this functionality have made their way into the standard library error trait, so modern error libraries build directly on std::error::Error rather than something like failure::Fail. Once we are no longer tied to failure 0.1's Fail trait, different parts of the codebase will be free to use any std::error::Error-based libraries they like while still working nicely together.
Reviewed By: xavierd
Differential Revision: D18576093
fbshipit-source-id: e2d862b659450f2969520d9b74877913fabb2e5d
Summary:
This diff is preparation for migrating off of failure::Fail / failure::Error for errors in favor of errors that implement std::error::Error. The Fallible terminology is unique to failure and in non-failure code we should be using Result<T>. To minimize the size of the eventual diff that removes failure, this codemod replaces all use of Fallible with Result by:
- In modules that do not use Result<T, E>, we import `failure::Fallible as Result`;
- In modules that use a mix of Result<T, E> and Fallible<T> (only 5) we define `type Result<T, E = failure::Error> = std::result::Result<T, E>` to allow both Result<T> and Result<T, E> to work simultaneously.
Reviewed By: Imxset21
Differential Revision: D18499758
fbshipit-source-id: 9f5a54c47f81fdeedbc6003cef42a1194eee55bf
