Summary:
There's an issue where hg foo could spawn hg dynamicconfig which could
spawn hg version (via the telemetry wrapper) which could spawn hg dynamicconfig.
If the config generation fails then this could enter an infiniteloop, since the
mtime of the file would not be updated appropriately, so we will always think we
need to regenerate.
Let's prevent this by setting an environment variable when spawning the
background process.
Reviewed By: quark-zju
Differential Revision: D21885363
fbshipit-source-id: ba4938c926d1219985813a521fec412097df4b4a
Summary:
I observed that for whatever reason our setting of `use_try_shorthand = true` in rustfmt.toml was causing entire functions to not get processed by rustfmt. Even files that contain neither `try` nor `?`. Remove it and reformat fbsource.
Documentation of that config:
- https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#use_try_shorthand
We don't particularly care about the value anymore because nobody writes `r#try!(...)` in 2018 edition code.
Minimized:
```
fn f() {
g(
)
// ...
.h
}
```
This function gets formatted only if use_try_shorthand is not set.
The bug is fixed in the rustfmt 2.0 release candidate.
Reviewed By: jsgf
Differential Revision: D21878162
fbshipit-source-id: b028673c0eb703984d24bf0d2983453fc2a8c212
Summary:
When I was debugging the multithreaded bug issue it came up that the issue would
only affect certain versions of macos. There may be other bugs that come up
affecting only certain os versions. Having the os version in eden rage could be
helpful to identify such issues.
Reviewed By: chadaustin
Differential Revision: D21776154
fbshipit-source-id: a493e7da1823075ca4a845bd73b21716ce884911
Summary: This will allow me to test it easily.
Reviewed By: StanislavGlebik
Differential Revision: D21840079
fbshipit-source-id: 1b3743da9c7908eb0dedd665aa24a4bf7aabd94f
Summary: We need to thread it through this layer to get it to SCS
Reviewed By: StanislavGlebik
Differential Revision: D21840078
fbshipit-source-id: d3c16a4afd87aa844628dd0aab7f85b8d38dadf8
Summary:
We need a lowest common ancestor operation for `merge_base` call
implementation.
The `lca` function returns only one of many possible ancestors in some
situation (multiple merges of the same branches). This is done to:
simplify the implementation and hide the complexity from our API users
(people don't really expect to receive more than one node from such APIs).
Mercurial arbitrarily chooses the commit with smallest hash in those situations.
Reviewed By: StanislavGlebik
Differential Revision: D21840081
fbshipit-source-id: 2dfc95a4cf549d8941fc5166e878bfee4b6b2ece
Summary: This methods will replace the fbcode_build provided functions. Also add Cargo files for crates that are now thanks to that buildable in OSS.
Reviewed By: farnz
Differential Revision: D21549860
fbshipit-source-id: 69f4179aa7a9081d33fac1f2f88aa3b32cd31e17/
Summary: I will modify in the next diffs, it's nice to have some test coverage
Reviewed By: krallin
Differential Revision: D21880996
fbshipit-source-id: 266622982e5d7d6d19ab3ac921ddc51079e51457
Summary:
The max_depth parameter was confusing - one might have thought that it returns
all entries that are `max_depth` hops away from `startnode`. This is not the
case - in reality it just limits what's the total number of entries can be
returned.
In fact, I think it can be replaced with Stream::take() method, however that's
a larger refactoring, and I'm not going to do it now.
It doesn't matter much because usually we have linear histories, but I think
it's still worth to make it clearer.
Reviewed By: krallin
Differential Revision: D21880220
fbshipit-source-id: d209e1e129383f9181ae11b489992dc4c3ce2d5c
Summary:
In the hg sync job, we need to load up the ancestors for all bookmarks known to
the server we are pushhing to, and for e.g. fbsource that might be > 10K
bookmarks. If we fetch those 1 by 1 (because e.g. cold cache), that will take a
very long time.
Unfortunately, we don't currently have a way of buffering access to changesets,
so for now let's mitigate by buffering.
Reviewed By: ikostia, HarveyHunt
Differential Revision: D21860228
fbshipit-source-id: 90977a9e00689c1df5ae53d149c267de9b2f973e
Summary:
When you call poll() from within an async context (as opposed to calling
await), awkward things happen, like this:
```
thread 'validation::test::slow_ready_validates' panicked at 'no Task is currently running',
```
So, let's stop doing that by instead checking that our stream eventually
yields. We can't count the polls this way, but that feels a bit immaterial so
that's probably OK. Also, let's clean up the code a little bit by removing a
bunch of conditional compilation.
Reviewed By: ahornby
Differential Revision: D21862830
fbshipit-source-id: 3eb49575d940ca85f59c49295dd6b0dcfb2e5d15
Summary:
We are going to start using tunables in Mononoke in the next diffs, and the
name clash between "tunables" and "newfilenodes::tunables" makes it confusing.
Let's rename newfilenodes::tunables to sql_timeout_knobs
Reviewed By: krallin
Differential Revision: D21879093
fbshipit-source-id: ab0bae4be3c319dcb6afeecdd1c13df395e79e3b
Summary: The thrift team is looking to retire `stopWorkers()` and `stopListening()`, so this refactors our code to stop using both of those methods.
Differential Revision: D21851442
fbshipit-source-id: 6d63d435cbd4a51c855bf71a3b2165e35431a4d7
Summary: we need an option to rebuild skiplists from scratch in case of corruptions.
Reviewed By: farnz
Differential Revision: D21863639
fbshipit-source-id: 56d8360a6c2c38aeb35f534758f5cde410fef421
Summary:
In 0ae204a978c11ddefafd81bd319a078239a44c1c the 'projects_dir' option
became a required constructor argument since it is called within the
constructor. However, it has not been adjusted in the subclasses that
used to set the option after instantiation. This commit fixes the
'shell_builder' and the 'debian_system_builder'.
Pull Request resolved: https://github.com/facebook/openr/pull/50
Test Plan:
1. Go to build directory: `cd build`
2. Run the `shell_builder` & `debian_system_builder`:
- `python fbcode_builder/shell_builder.py`
- `python debian_system_builder/debian_system_builder.py`
`shell_builder` output before:
```
Traceback (most recent call last):
File "fbcode_builder/shell_builder.py", line 102, in <module>
builder = ShellFBCodeBuilder()
File "/home/butjar/tu/ma/openr/build/fbcode_builder/fbcode_builder.py", line 93, in __init__
self._github_dir = self.option('projects_dir')
File "/home/butjar/tu/ma/openr/build/fbcode_builder/fbcode_builder.py", line 108, in option
raise RuntimeError('Option {0} is required'.format(name))
RuntimeError: Option projects_dir is required
```
`shell_builder` output after:
```
set -exo pipefail
export CCACHE_DIR='/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/.ccache' CC="ccache ${CC:-gcc}" CXX="ccache ${CXX:-g++}"
### Diagnostics ###
# Builder ShellFBCodeBuilder(google/googletest:cmake_defines={u'BUILD_GTEST': u'ON', u'BUILD_SHARED_LIBS': u'OFF'}, google/googletest:git_hash=u'release-1.8.1', facebook/openr:local_repo_dir='/home/butjar/tu/ma/openr', facebook/zstd:git_hash=ShellQuoted(u'$(git describe --abbrev=0 --tags origin/master)'), openr/build:cmake_defines={u'ADD_ROOT_TESTS': u'OFF'}, thom311/libnl:git_hash=u'libnl3_2_25', projects_dir=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr', fmtlib/fmt:git_hash=u'5.3.0', wangle/wangle/build:cmake_defines={u'BUILD_TESTS': u'OFF'}, prefix=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/installed', fizz/fizz/build:cmake_defines={u'BUILD_TESTS': u'ON'}, ccache_dir=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/.ccache', zeromq/libzmq:git_hash=u'v4.2.2', make_parallelism=4, jedisct1/libsodium:git_hash=u'stable')
hostname
cat /etc/issue || echo no /etc/issue
g++ --version || echo g++ not installed
cmake --version || echo cmake not installed
### Check out fmtlib/fmt, workdir build ###
mkdir -p '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr' && cd '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'
git clone https://github.com/'fmtlib/fmt'
mkdir -p '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'/'fmt'/'build' && cd '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'/'fmt'/'build'
git checkout '5.3.0'
### Build and install fmtlib/fmt ###
...
```
Reviewed By: steven1327
Differential Revision: D21865881
Pulled By: saifhhasan
fbshipit-source-id: dfd78127d3b2c78721f84a3ecafe0b7198c38f06
Summary: This makes the API more flexible.
Reviewed By: sfilipco
Differential Revision: D21626196
fbshipit-source-id: d0d28ba075ee3321a1a757f848fb72592827d75d
Summary: This allows DAG algorithms to be executed on revlog.
Reviewed By: sfilipco
Differential Revision: D21626217
fbshipit-source-id: e9fd61ad62f95be7b055a0ef8879c59cbeeb60b9
Summary:
Mostly copy-paste from code added in D19503373 and D19511574. Adjusted to match
the revlog index interface.
Reviewed By: sfilipco
Differential Revision: D21626201
fbshipit-source-id: 05d160e4c03d7e2482b6a4f2d68c3688ad78f568
Summary: The trait converts NameSet to IdSet. It'll be used by the revlog index.
Reviewed By: sfilipco
Differential Revision: D21795869
fbshipit-source-id: 55f7a238158442db9d8bdfe84e64438be504f618
Summary: Add a way to inverse the DAG (swap parent / children relations).
Reviewed By: sfilipco
Differential Revision: D21795870
fbshipit-source-id: 2d076f4ae491141aa758faa5f5f303c97f7e56dc
Summary:
Similar to LazySet, but the iterator is using Ids. This will be useful for
lazy calculations that are cheaper with Ids.
Reviewed By: sfilipco
Differential Revision: D21626208
fbshipit-source-id: 9a34fbf18f0039caeb4f6e698294c4d335354093
Summary:
The NameSet is not really about Dag. It is about using Id and is static.
Rename it to clarify. In an upcoming change we'll have IdLazySet.
Reviewed By: sfilipco
Differential Revision: D21626204
fbshipit-source-id: 84f25008f7032f6e26a26fc656ccbcd2a5880ecf
Summary:
This makes it possible to use NameIter without manually specifying out iterator
types, which might be quite long.
Reviewed By: sfilipco
Differential Revision: D21626202
fbshipit-source-id: 67b338765c09629645794cf73a9b496271524f9d
Summary: Take advantage of Hints and add fast paths.
Reviewed By: sfilipco
Differential Revision: D21626216
fbshipit-source-id: 6d43666bd6cdec7ff4b93032c1064cafd8de85cf
Summary: Update hints if they are easy to obtain or calculate.
Reviewed By: sfilipco
Differential Revision: D21626206
fbshipit-source-id: 453b7db2444406ce51d574c688fe536316fb9b0f
Summary: The `secure_utils` crate from common/rust/secure_utils was moved to rust-shed, the remaining crates in that folder are being refactored here into a single crate `identity_ext` for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21549861
fbshipit-source-id: 4da6566a09ba7a772e8062632f9d7520af2e09e6
Summary: Currently output from build command is decoded with "surrogateescape" error handler, but when writing to log files/stdout we don't specify error handlers to be also "surrogateescape" according to https://docs.python.org/3/library/codecs.html#error-handlers, which could cause exception when there's surrogate characters logged in message.
Reviewed By: yfeldblum
Differential Revision: D21850411
fbshipit-source-id: 21c51d1ab2132171ae29f2d1fbe42655ebee94c5
Summary:
In some rare cases, we would have hgcache that contains broader permission than we are expecting. We shouldn't be fixing it if that's the case.
We also might be in situations where hgcache directory isn't entirely created by Mercurial, and the owner of the directory will be different than the process. This will cause the `chmod` call to fail with permission error. In that case, this will cause EdenFS to panic. This is undesirable. We should be handling this case more gracefully and let the original error populate.
Reviewed By: xavierd
Differential Revision: D21854542
fbshipit-source-id: e9d11399aeb40b375725b49f4bcd54050afdcbad
Summary:
Update fmt version to 6.2.1 for better compatibility with the version used in fbcode. Among other things this fixes fbthrift build failure on Travis:
```
/home/fbthrift/thrift/lib/cpp2/async/RocketClientChannel.cpp:70:67: required from here
/home/install/include/fmt/core.h:492:3: error: static assertion failed: don't know how to format the type, include fmt/ostream.h if it provides an operator<< that should be used
```
which is caused by trying to format an enum class without a formatter - only supported as of 6.0.
Reviewed By: stevegury, avalonalex
Differential Revision: D21860076
fbshipit-source-id: 1857ab65822956b005980b8dfff7a967508f507c
Summary:
In the next diffs we'll make it possible to disable filenodes in Mononoke. See
D21787848 and attached task for more details, but TL;DR is that if xdb is down
we still want to serve "hg update" traffic.
If filenodes are disabled we obviously can't generate filenodes for new
commits. So one option would be to just return an error from
FilenodesOnlyPublic::derive(...) call. But that would mean that any attempt to
call derivation would fail, and e.g. Mononoke servers won't be able to start up
- (see https://fburl.com/diffusion/roau028d). We could change callers to always
process errors from FilenodesOnlyPublic, but I think it would be harder to
enforce and easier to forget.
So this diff changes FilenodesOnlyPublic to be an enum, and
FilenodesOnlyPublic::Disabled is returned immediately if filenodes are
disabled. For callers it means that they can't rely on filenodes being present
in db even after FilenodesOnlyPublic were derived. That's the whole of the
stack, and the next diffs will update the callers to properly deal with missing
filenodes.
One caveat is that when we re-enable filenodes back we might need to derive
them for a lot of commits.
I don't expect it to happen often (i.e. if xdb is down then we probably can't
commit anyway), but if somehow it happened, then we should be a bit more
careful with re-enabling them after the problem was fixed. For example, we can
first derive all the filenodes locally by e.g. running backfill_derived_data,
and only after that has finished successfully we can re-enable them.
Reviewed By: krallin
Differential Revision: D21840328
fbshipit-source-id: ce9594d4a21110a5cb392c3049ccaede064c1e66
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
Instead of always building from scratch, continue assiging Vertexes and
Segments from the last commit that was processed.
Reviewed By: StanislavGlebik
Differential Revision: D21634699
fbshipit-source-id: 9f8b890dcf65c59a66651343f0ccc1487efc2394
