Summary:
When doing discovery, for repos with long master lines and infrequent
branches, picking a random set of sample commits could result in not picking the
master, and therefore having to do very long commit graph traversals to check
ancestors against the other samples.
To prevent this, let's pick the N most recent commits instead of a random
sample. This should generally get the master commit into our sample.
Reviewed By: quark-zju
Differential Revision: D21394302
fbshipit-source-id: f4b8110cd126b90553ec624e48cab0b590e124fb
Summary: Making a trait out of TimeWindowCounter will help with providing different implementations of load limiting for OSS and FB.
Reviewed By: krallin
Differential Revision: D21329265
fbshipit-source-id: 7f317f8e9118493f3dcbadb0519eaff565cbd882
Summary:
This is helpful. Also, while in there, I removed an error that wasn't used at
all.
Reviewed By: StanislavGlebik
Differential Revision: D21399489
fbshipit-source-id: 0e5ef20b842afa9ffc0bb8530c48eb48339c558e
Summary:
We have a number of error enums that wrap an existing errors, but fail to
register the underlying error as a `#[source]`. This results in truncated
context chains when we print the error. This fixes that. It also removes a
bunch of manual `From` implementation that can be provided by thiserror's
`#[from]`.
This also required updating the `Display` implementation for those errors. I've
opted for not printing the underlying error, since the context chain will
include it. This does mean that if we print one of those errors without the
context chain (i.e. `{}` as opposed to `{:#}` or `{:?}`), then we'll lose out a
bit of context. That said, this should be OK, as we really shouldn't ever being
do this, because we'd be missing the rest of the chain anyways.
Reviewed By: StanislavGlebik
Differential Revision: D21399490
fbshipit-source-id: a970a7ef0a9404e51ea3b59d783ceb7bf33f7328
Summary:
This removes our own (Mononoke's) implementation of failure chains, and instead
replaces them with usage of Anyhow. This doesn't appear to be used anywhere
besides Mononoke.
The historical motivation for failure chains was to make context introspectable
back when we were using Failure. However, we're not using Failure anymore, and
Anyhow does that out of the box with its `context` method, which you can
downcast to the original error or any of the context instances:
https://docs.rs/anyhow/1.0.28/anyhow/trait.Context.html#effect-on-downcasting
Reviewed By: StanislavGlebik
Differential Revision: D21384015
fbshipit-source-id: 1dc08b4b38edf8f9a2c69a1e1572d385c7063dbe
Summary:
I'm going to send a diff to get rid of failure chains, and the LFS Server
actually uses that quite a bit. Let's make sure we don't affect the error
rendering there.
Reviewed By: StanislavGlebik
Differential Revision: D21383032
fbshipit-source-id: e0ec9c88760e7fd48d39fa1570efd1870a9ef532
Summary: The oss jobs are causing problems to the developers now, lets disable them temporarily.
Reviewed By: StanislavGlebik
Differential Revision: D21400898
fbshipit-source-id: f7a3567056633d9eef98a8d05a37cd029c9e506c
Summary:
Looks like this broke yesterday. There was a Reindeer update yesterday IIRC, so
I'm guessing that's the cause. In any case, this is easy to fix forward.
Reviewed By: farnz
Differential Revision: D21399830
fbshipit-source-id: 5cf33411e089a8c675a8b3fdf7b6ae5ae267058d
Summary:
If config.toml exists but does not contain a valid storage engine,
edenfs would fail to start. Now, it will set a default value and write
the updated config file.
Reviewed By: simpkins
Differential Revision: D19671453
fbshipit-source-id: 32f19dbed2be02bec2a96e1349dca6e7038b9301
Summary:
Addressing issues simpkins brought up on D21207287 when we upgraded and introduced some pyre bugs.
Temporarily upgrading just this project, once we resolve some sandcastle capacity issues we'll release this via another global upgrade in fbcode.
Reviewed By: simpkins
Differential Revision: D21316793
fbshipit-source-id: f0c79f53d97f7182e7d8fe6e081c58ef53ce0c9a
Summary: When we create directory at certain places, we want these directories to be shared between different users on the same machine. This Diff uses the previously added `create_shared_dir` function to create these directories.
Reviewed By: xavierd
Differential Revision: D21322776
fbshipit-source-id: 5af01d0fc79c8d2bc5f946c105d74935ff92daf2
Summary:
All the changes we need are now in stable, so use the stable crates.io version.
I also had to do coordinated updates of git2 and rustsec to make sure they're
all using the same version of libgit2-sys. This had a couple of little API changes which affected our code:
- mononoke gitimport (krallin)
- linttool (zertosh) (BTW the old code had some very dubious lifetime stuff - a signature of the form `fn foo<'a>(&self) -> Thing<'a>` never makes any sense - output lifetimes should always be derived from the params)
Similarly, toml-rs needed to be updated because there's now a hard dependency on 0.5.6.
msdkland[rust_cargo]
msdkland[rust_reindeer]
Reviewed By: dtolnay
Differential Revision: D21311180
fbshipit-source-id: 82083c8f2bb8523e70cbe99dc0a630c4bc67a505
Summary:
When using our vendored set of crates, cmake doesn't
have any dependency information to use to invalidate the Cargo.lock
file when we update crate versions. In addition, since we're
vendoring from a local directory, cargo itself doesn't seem to
want to re-assess the dependencies in that same situation, leading
to confusing error messages like this when we want to build rust
targets:
```
error: failed to select a version for the requirement `anyhow = "= 1.0.26"`
candidate versions found which didn't match: 1.0.28
```
This commit addresses this issue by removing the `Cargo.lock` that
may be alongside the `Cargo.toml` prior to invoking `cargo`.
`cargo` is pretty quick at recomputing the deps so this has
neglible overhead.
Reviewed By: xavierd
Differential Revision: D21394363
fbshipit-source-id: 547db2e2395a47aed77d9597e659eb2d96e274dd
Summary:
When the commit hash changed in fbsource, we would correctly decide
that we'd need to rebuild first-party projects but we would incorrectly skip
running the fetcher.update method. This would mean that we'd not perform the
shipit run and that our shipit tree would diverge from the source tree.
This commit resolves this by performing the fetcher.update but ignoring
the source update status in this case.
Reviewed By: xavierd
Differential Revision: D21364131
fbshipit-source-id: b4001e549c7d3f27aa4a21b19893c9bb7c0f6d1f
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Only run cmake reconfigure for .cmake, .cmake.in and CMakeLists.txt
files changes.
Previously we would reconfigure for any change to a file with a path that
matched `cmake` which could result in false positives in cases where
you may be iterating on .py or .c files in shared cmake directories.
This also reclassifies non-cmake files under fbcode_builder/CMake as source
files so that we run cmake for those; previously they would cause a
reconfigure and build, now they just cause a build.
Reviewed By: xavierd
Differential Revision: D21364133
fbshipit-source-id: a1231f657d6c6056b269656c677d3449d8715cf6
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
Previously getdeps would remove the entire top-level `CMakeFiles` directory
from the build output when it wanted to invalidate the CMake cache. This
directory is used to keep all of the compiled object files for any libraries
or executables defined in the top-level CMakeLists.txt file. Blowing away
this directory forces all of these sources to be re-compiled, even if this was
not necessary. This is particularly problematic for folly, which compiles all
of its source files via rules in the top-level CMakeLists.txt target file.
I did have the code still blow away the CMake error and output logs in this
directory: in the past I have seen situations where CMake would not update
these files on new CMake runs if they already existed.
Reviewed By: wez
Differential Revision: D21360668
fbshipit-source-id: 6fcd1a8e371d756114fbab60d8636be8cd5f8978
Summary:
Update the manifest file for folly to indicate a dependency on lz4.
folly does not require lz4 be available, but it will use it if it is found at
configure time.
getdeps is unfortunately not strict about providing projects only with the
dependencies that they require at build time. This causes it to sometimes
make lz4 available to folly (if you are also building another project that
requires lz4), and sometimes not. This ends up causing changes in
folly-config.h depending on which projects you are building, forcing all of
the folly sources to be recompiled.
In the future we perhaps should update getdeps to consistently only pass in
include directories for dependencies actually listed in the manifest file.
However, specifying that folly depends on lz4 also works to mitigate this
particular issue for now, and it is also generally desirable to build folly
with lz4 support.
Reviewed By: wez
Differential Revision: D21359995
fbshipit-source-id: aaf61671b7750d6c47e3613c732d220b3311b5ba
Summary: These docs are all EdenFS specific so move them into the fs/ directory.
Reviewed By: genevievehelsel
Differential Revision: D21329620
fbshipit-source-id: 4090ed4ca371d01ea98e06ad6ce8f434c0660962
Summary:
The MSVC compiler complains that it doesn't have the full definition of
PrivHelper, causing the build to fail. Include the right header to fix this.
Reviewed By: genevievehelsel
Differential Revision: D21381946
fbshipit-source-id: 0d0389ee8db44a36786973404c38487a94e8c4df
Summary:
Previously we only included `basic_test.py` and `hg/status_test.py` in the
integration tests during CMake-based builds. This updates the code to now
include all of the test files, with just a few exclusions based on platform
type and what dependencies were available at build time.
Reviewed By: wez
Differential Revision: D21239912
fbshipit-source-id: b8826d249a6323ac3bcc555c9ceba54a4cbcfde9
Summary:
10 of the integration tests fail on Ubuntu. For now simply blacklist them
from running on non-RedHat-based distributions.
Reviewed By: wez
Differential Revision: D21332629
fbshipit-source-id: 3fadb74bb31e89092177afaa01ddc7f6bcd0f9de
Summary:
Add a new IntegrationTestCase base class that checks the test blacklist during
`setUp()`, and update a few remaining test classes that did not derive from
`EdenTestCase` to derive from this.
Also drop the method name argument from the `skip_test_if_blacklisted()`
function, since we can get this from the existing test case argument.
Reviewed By: genevievehelsel
Differential Revision: D21340539
fbshipit-source-id: d4fc125f119d74ab923c2cc3c9070b86c582c87e
Summary:
Run `env` as `/usr/bin/env` instead of `/bin/env`
This command is typically installed in `/usr/bin`. Recent RedHat releases
replaced `/bin/ with a symlink to `/usr/bin`, allowing this command to work
when invoked as `/bin/env`, but this isn't true on all distributions, such as
Ubuntu.
Reviewed By: chadaustin
Differential Revision: D21340540
fbshipit-source-id: d7588f8c90e9a86a0cb31fd3ab3a9067aa6e79ea
Summary: These third-party includes are edenfs-specific, so move them into eden/fs/
Reviewed By: simpkins
Differential Revision: D21314642
fbshipit-source-id: c52b0a00d5080934e1f07e4cd55373602f2f6b0a
Summary: These benchmarks are edenfs-specific, so move them into /eden/fs/
Reviewed By: genevievehelsel
Differential Revision: D21314464
fbshipit-source-id: 1dcf6adfbdea1394f222de4d462397ea531ced00
Summary:
This updates repo_client to log when hooks finished, and how many were rejecte,
if any. This required a bit of refactoring to avoid iterating twice over
whether hooks are rejected or not (and instead just filter-maps outcomes to a
rejection), but it's probably for the better since it removes a bit of
un-necessary cloning (notably of the hook name).
Reviewed By: farnz
Differential Revision: D21379690
fbshipit-source-id: 53c8368d3871620ec61db76dc35b47dd17276ac4
Summary:
This adds support for running Gitimport with `--readonly-storage`. The way we
do this is by masking the various storages we use (blobstore, changesets,
bonsai).
Reviewed By: markbt
Differential Revision: D21347939
fbshipit-source-id: 68084ba0d812dc200776c761afdfe41bab9a6d82
Summary:
The original gitimport wasn't really designed for concurrency, since it did
commits one by one. With this update, we can now derive Bonsais from multiple
commits in parallel, and use multiple threads to communicate with the Git
repository (which is actually somewhat expensive when that's all we do).
We also store Bonsais iteratively. There is a bit of extra work that could be
done also here by saving Bonsais asynchronously to the Blobstore, and inserting
a single batch in Changesets once we're finished.
Reviewed By: farnz
Differential Revision: D21347941
fbshipit-source-id: e0ea86bf4d164599df1370844d3f0301d1031801
Summary:
This adds support for deriving commits within a range in gitimport, which gets
us one step closer to resumable gitimport. The primary goal of this is to
evaluate whether using Gitimport for Configerator might be suitable.
Differential Revision: D21347942
fbshipit-source-id: aa3177466e389ceb675328999ccf836f29912698
Summary:
This adds some basic functionality for deriving hg manifests in gitimport. I'd
like to add this to do some correctness testing on importing Git manifests from
Configerator.
Differential Revision: D21347940
fbshipit-source-id: 6f819fa8a62b3088fb163138fc23910b8f2ff3ce
Summary:
- Use the same case consistently
- Log even when pushrebase fails
Reviewed By: farnz
Differential Revision: D21378033
fbshipit-source-id: 062e986151086476db9100e3d9c71aa702661032
Summary:
Currently we need to specify which derived data we need to derive, however they
are already specified in the configerator configs. Let's just read it from
there.
That means that we no longer need to update tw spec to add new derived data types - we'll just need to add them to configerator and restart the backfiller.
Reviewed By: krallin
Differential Revision: D21378640
fbshipit-source-id: f97c3f0b8bb6dbd23d5a50f479ecfccbebd33897
Summary: Making a trait out of LoadLimiter will help with providing different implementations of load limiting for OSS and FB.
Reviewed By: farnz
Differential Revision: D21302819
fbshipit-source-id: 1b982a367aa7126ca5d7772e4a2406dabbe9e13b
Summary: A result of `configerator-thrift-updater scm/mononoke/repos/repos.thrift`
Reviewed By: farnz
Differential Revision: D21350982
fbshipit-source-id: b3344c99c6f53c727ea16ebc0f81f90527de103d
