Summary: The `secure_utils` crate from common/rust/secure_utils was moved to rust-shed, the remaining crates in that folder are being refactored here into a single crate `identity_ext` for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21549861
fbshipit-source-id: 4da6566a09ba7a772e8062632f9d7520af2e09e6
Summary: Currently output from build command is decoded with "surrogateescape" error handler, but when writing to log files/stdout we don't specify error handlers to be also "surrogateescape" according to https://docs.python.org/3/library/codecs.html#error-handlers, which could cause exception when there's surrogate characters logged in message.
Reviewed By: yfeldblum
Differential Revision: D21850411
fbshipit-source-id: 21c51d1ab2132171ae29f2d1fbe42655ebee94c5
Summary:
In some rare cases, we would have hgcache that contains broader permission than we are expecting. We shouldn't be fixing it if that's the case.
We also might be in situations where hgcache directory isn't entirely created by Mercurial, and the owner of the directory will be different than the process. This will cause the `chmod` call to fail with permission error. In that case, this will cause EdenFS to panic. This is undesirable. We should be handling this case more gracefully and let the original error populate.
Reviewed By: xavierd
Differential Revision: D21854542
fbshipit-source-id: e9d11399aeb40b375725b49f4bcd54050afdcbad
Summary:
Update fmt version to 6.2.1 for better compatibility with the version used in fbcode. Among other things this fixes fbthrift build failure on Travis:
```
/home/fbthrift/thrift/lib/cpp2/async/RocketClientChannel.cpp:70:67: required from here
/home/install/include/fmt/core.h:492:3: error: static assertion failed: don't know how to format the type, include fmt/ostream.h if it provides an operator<< that should be used
```
which is caused by trying to format an enum class without a formatter - only supported as of 6.0.
Reviewed By: stevegury, avalonalex
Differential Revision: D21860076
fbshipit-source-id: 1857ab65822956b005980b8dfff7a967508f507c
Summary:
In the next diffs we'll make it possible to disable filenodes in Mononoke. See
D21787848 and attached task for more details, but TL;DR is that if xdb is down
we still want to serve "hg update" traffic.
If filenodes are disabled we obviously can't generate filenodes for new
commits. So one option would be to just return an error from
FilenodesOnlyPublic::derive(...) call. But that would mean that any attempt to
call derivation would fail, and e.g. Mononoke servers won't be able to start up
- (see https://fburl.com/diffusion/roau028d). We could change callers to always
process errors from FilenodesOnlyPublic, but I think it would be harder to
enforce and easier to forget.
So this diff changes FilenodesOnlyPublic to be an enum, and
FilenodesOnlyPublic::Disabled is returned immediately if filenodes are
disabled. For callers it means that they can't rely on filenodes being present
in db even after FilenodesOnlyPublic were derived. That's the whole of the
stack, and the next diffs will update the callers to properly deal with missing
filenodes.
One caveat is that when we re-enable filenodes back we might need to derive
them for a lot of commits.
I don't expect it to happen often (i.e. if xdb is down then we probably can't
commit anyway), but if somehow it happened, then we should be a bit more
careful with re-enabling them after the problem was fixed. For example, we can
first derive all the filenodes locally by e.g. running backfill_derived_data,
and only after that has finished successfully we can re-enable them.
Reviewed By: krallin
Differential Revision: D21840328
fbshipit-source-id: ce9594d4a21110a5cb392c3049ccaede064c1e66
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
Instead of always building from scratch, continue assiging Vertexes and
Segments from the last commit that was processed.
Reviewed By: StanislavGlebik
Differential Revision: D21634699
fbshipit-source-id: 9f8b890dcf65c59a66651343f0ccc1487efc2394
Summary:
Somehow, on Windows, the socket file that was removed during `prepare` may be
back, but not bound, preventing Thrift from binding to it, let's remove it
again since it's not supposed to be there.
Note that this happens 100% of the time when starting EdenFS via the Task
Scheduler at log on, I couldn't reproduce when triggering the task manually,
or when starting edenfs by hand.
Reviewed By: simpkins
Differential Revision: D21755498
fbshipit-source-id: 32a343d9de531b90417dac0aa382a07c85cdee5a
Summary:
The CREATE_NO_WINDOW is documented as:
The process is a console application that is being run without a console
window. Therefore, the console handle for the application is not set.
The last part is what matters here, by default stdin, stdout and stderr are
inherited from the parent process, with this flag, they won't as these 3 will
simply not be set. Removing it allows Mercurial's output to be sent directly to
the edenfs log which will greatly improve our debuggability.
Reviewed By: akrieger
Differential Revision: D21820195
fbshipit-source-id: 63496a1ad28ecf5440c0846d80e1fbda5756b971
Summary:
Previously, the NameSet has properties like "is_all", "is_topo_sorted", etc.
To make lazy sets efficient, it's important to have hints about min / max Ids
and maybe some other information.
Add a dedicated Hints structure for that.
Reviewed By: sfilipco
Differential Revision: D21626219
fbshipit-source-id: 845e88d3333f0f48f60f2739adae3dccc4a2dfc4
Summary:
Implement a small subset of DagAlgorithm by default. This makes
other implementations of DagAlgorithm slightly easier.
Reviewed By: sfilipco
Differential Revision: D21626199
fbshipit-source-id: ac6dfb5c22bf1da44f521fc9e76d59bfb95063c7
Summary:
D21479023 broke it. It should convert to Id, and check Id against the SpanSet,
instead of just checking the IdMap ignoring the SpanSet.
Reviewed By: sfilipco
Differential Revision: D21626193
fbshipit-source-id: 6daf86f292a7acfd3688893a55e2a794cfe068fe
Summary: This makes the next change easier to implement.
Reviewed By: sfilipco
Differential Revision: D21626198
fbshipit-source-id: 57ab69cba7f43350767e5d0d52ebfe66764895ca
Summary:
Implements part of the dag IdMap related traits.
It does not get used yet, but eventually I'd like `pydag` to be able to work
with an abstracted dag including RevlogIndex.
Reviewed By: sfilipco
Differential Revision: D21626210
fbshipit-source-id: 53f19622f03fd71b76073dccf8dcc9b4778b40ca
Summary:
This will allow RevLogIndex to answer node -> rev and hex lookup queries.
Also change RevlogIndex::new to take file names so it can write back the
nodemap index when the index is lagging. That part of logic currently exists in
pyindexes + clindex.pyx, which are going to be replaced by revlogindex.
Practically, this will generate a `00changelog.nodemap` file in svfs, which is
temporarily unused, but will be used once clindex.pyx gets replaced.
Reviewed By: sfilipco
Differential Revision: D21626209
fbshipit-source-id: 297d9eff26a73c26558708f7a2290d4d8ba1e777
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary: In line with other crates that contain utility binaries alongside the crate, rename the `edenapi/utils` directory to `edenapi/tools`.
Reviewed By: quark-zju
Differential Revision: D21796899
fbshipit-source-id: 058319e2756b1d596f06d6e57d17a6c07a7f1c9c
Summary:
Clean up some of the conversion functions by renaming variables that are
keywords in other languages, and simplifying error handling code.
Differential Revision: D21839019
fbshipit-source-id: d8945a14a230caa744040e134203a908ad9cef20
Summary: `ErrorKind` is not meaningful, and is an artifact of older-style error handling crates. A better name is `ConfigurationError`.
Reviewed By: krallin
Differential Revision: D21837271
fbshipit-source-id: 709d9e2ab7f18dd2f7cb2489f24e91612bc378db
Summary:
Replace the use of `RepoConfigs::read*` associated functions with free
functions. These didn't really need to be associated functions (and in the
case of the common and storage configs, really didn't belong there either).
Reviewed By: krallin
Differential Revision: D21837270
fbshipit-source-id: 2dc73a880ed66e11ea484b88b749582ebdf8a73f
Summary:
Refactor parsing of repo config using a new `Convert` trait to allow
definition of each part of parsing separately.
The wireproto logging args require access to the storage definitions, so need
to be parsed by their own special function for now.
Differential Revision: D21837269
fbshipit-source-id: 7ab0e3f4b3b8549aaefb45201388c3dfc7633ef7
Summary:
Refactor parsing of storage config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766761
fbshipit-source-id: 7e224e9d322a3a16a64f5ebba2243bbe6341c8f0
Summary:
Refactor parsing of commit sync config using a new `Convert` trait to allow
definition of each part of parsing separately.
Differential Revision: D21766760
fbshipit-source-id: 3c95d70788753316d3c1f36280e7d6dbb52a9710
Summary:
We'd like to serve read traffic even if filenodes are disabled. Let's add a
tunable that can control it.
Reviewed By: HarveyHunt
Differential Revision: D21839672
fbshipit-source-id: 4ec4dd16b9e6e3ffb1ada0d812e1153e1a33a268
Summary: It was replaced with a parameter
Reviewed By: HarveyHunt
Differential Revision: D21839397
fbshipit-source-id: e75900b3da80985cd762659993b8b285411fe928
Summary:
Now that the shared directories are created with the right permissions, we
could still have some in the wild with the incorrect permissions. Let's make
sure that we fix these up.
Reviewed By: wez
Differential Revision: D21832436
fbshipit-source-id: d8ee40f61b16857d29e1360ec6df50b2a95ea7aa
Summary:
This is required for the shared cache to avoid permissions issues when multiple
users are trying to use it.
Reviewed By: fanzeyi
Differential Revision: D21830490
fbshipit-source-id: 3db0dbd674b6d2e10b9361ff3c3a668d046f5d78
Summary: Add more information for bisect states.
Reviewed By: quark-zju
Differential Revision: D21758828
fbshipit-source-id: 4bc617e50f70277428dc4c7c1be68b652d78b8f8
Summary:
In D20286499 I added chunking of remotefilelog requests, however apparently the
place where I've added it is no longer used (xavierd mentioned it might be
because of migration to rust stores).
let's put chunking logic into getpack() function
Reviewed By: xavierd
Differential Revision: D21765414
fbshipit-source-id: b481ddf070f7bd86d0071cea7be2f9cc6ef1e5d9
Summary:
DefferedDerivedMapping was added so that we can make deriving stack of commits faster - it does it by postponing updating
derived data mapping (e.g. writing to a blobstore) until the whole stack is derived.
While it probably makes derivation a bit faster, we now think it's better to remove it. A few reasons:
1) It's confusing to understand and it already caused us ubns before
2) It's increases write amplification - because we release the lease before we wrote to a blobstore, writers will try to rederive the same commit a few times. That has caused us a ubn today
Reviewed By: farnz
Differential Revision: D20113854
fbshipit-source-id: 169e05febcd382334bf4da209a20aace0b7c2333
Summary:
See D21765065 for more context. TL;DR is that we want to control
lfs rollout from client side to make sure we don't put lfs pointers in the
shared memcache
Reviewed By: xavierd
Differential Revision: D21822159
fbshipit-source-id: daea6078d95eb4e9c040d353a20bcdf1b6ae07b1
