Summary:
With the internal streampager, progress bars must be sent on a separate stream so that
streampager can render them correctly.
Reviewed By: quark-zju
Differential Revision: D21906173
fbshipit-source-id: eb41b0bf22807d9cae518b3f676996ab1c642c6e
Summary: Upgrade the internal streampager to the latest version, which includes line wrapping.
Reviewed By: quark-zju
Differential Revision: D21906172
fbshipit-source-id: 1bc63723f55ac115090e7ae0a2541158863056b9
Summary: The test is changed because an extra file is copied during local-repo clone.
Reviewed By: markbt
Differential Revision: D21894318
fbshipit-source-id: 4417fb5473dc2cb98eda4dcc4d0484cc3b31fae2
Summary:
`index2` is the Rust revlog index implementation that has extra requirements,
namely, the revlog cannot be inlined. We have code to make sure changelog is
not inlined, but manifest or filelog can still be inlined.
`index2` is only needed for changelog for phase and visibility calculations.
Therefore, disable it for manifest and filelog.
Reviewed By: singhsrb
Differential Revision: D21917447
fbshipit-source-id: c45f955175c623092e6f728042dbfd3b942fe911
Summary: This diff updated `eden du` to display a summary and display warnings and cleaning information with colors.
Reviewed By: kmancini
Differential Revision: D21885051
fbshipit-source-id: be127b81c92bea1051a80715682cdbccf22f22e3
Summary: This diff updated `eden du` to display an aggregated result of all mounts instead of showing all details for each mount, as users generally just want to reduce disk usage but don't really care about details.
Reviewed By: genevievehelsel
Differential Revision: D21877178
fbshipit-source-id: dde43e51e96a5c2569c9fe21ab06cc7ea4295866
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff finally connects EdenAPI with EdenFS. Previously we only had basic EdenAPI implemented but the performance was not acceptable. This is largely due to overheads in talking with the remote server. Now EdenFS is able to import multiple files at once, we efficiently use EdenAPI to import these data now.
Reviewed By: chadaustin
Differential Revision: D21576569
fbshipit-source-id: a45e832ec63d057730138551393ff7547fa2c22f
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff encapsulates the newly added batch blob import API in backingstore crate, and expose it to EdenFS. It converts incoming complex data structures down to primitive data types that can go through FFI boundary.
Reviewed By: xavierd
Differential Revision: D21697578
fbshipit-source-id: 8f5505ef4cab342dfa710798a04df4e0e94055d9
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff implements getBlobBatch in `backingstore` crate that is able to process multiple blob import request at once. This will help EdenFS to process blob import request efficiently. See following diffs in the stack for usage.
To import a list of import requests, the function first check if the file exists locally. When it is already in local store, the function will call the provided `resolve` function to send the blob back. Then it proceeds to send **ONE** request to remote store (in our case, EdenAPI) for all these missing files if this import permits remote importing.
We use the callback style `resolve` parameter because we want to awake waiting threads as soon as the blob is ready, so we don't waste time on unnecessary waiting.
Reviewed By: xavierd
Differential Revision: D21697580
fbshipit-source-id: b550accf6f6163cf6f2e9be6b628e46f44076c37
Summary:
It was turned on in production (facebook.rc) except for hg servers.
This mainly affects tests.
Reviewed By: markbt
Differential Revision: D21894313
fbshipit-source-id: 5f12620cfc11bb243e96fba12a07cbf0241c738b
Summary:
This is a followup from s195788 - this little check should be enough to prevent
us from having the paths in our DB being truncated on insert.
Reviewed By: markbt
Differential Revision: D21904281
fbshipit-source-id: c9306e0cca4a2e009bb50bc287190d8f7ad12be4
Summary:
I am planning to start migrating Eden's CLI to the new Python 3 Thrift
implementation. In preparation, slightly clean up the interface and
implementation of our Python 2 Thrift wrapper.
Reviewed By: genevievehelsel
Differential Revision: D21854539
fbshipit-source-id: d398dd3f324c12288871cf0c9db41e64ed4cf7ed
Summary:
Visibleheads (and even narrow-heads) are enabled by default except for hg
servers. No need to log them specially.
Reviewed By: markbt
Differential Revision: D21894308
fbshipit-source-id: 7ffe0977e51c17e743b62da0e2bfbe7618c64610
Summary: This is required to make some tests pass.
Reviewed By: markbt
Differential Revision: D21894312
fbshipit-source-id: 481bcc42f8f7fbb5c1707e66da31aee29932484c
Summary: Make `debugobsolete` also update visibility. This affects many tests.
Reviewed By: markbt
Differential Revision: D21894310
fbshipit-source-id: eba6ecb07fdd8f5a13085434b303581ec5acf463
Summary: This makes tests more stable when changed.
Reviewed By: markbt
Differential Revision: D21894314
fbshipit-source-id: 73c36f79763643ccbe5dd52522fa8f22fa17a08a
Summary: We don't use obsmarker but mutation records. Therefore, remove the test.
Reviewed By: markbt
Differential Revision: D21894315
fbshipit-source-id: a1ccd8f78bea8adcba419b0d8ec99d52cd14d4e4
Summary:
We don't use obsmarker exchange but mutation record exchange.
Therefore, remove the test.
Reviewed By: markbt
Differential Revision: D21894311
fbshipit-source-id: 73d1f8031b2f601d133a98ffaa92a890bb59a656
Summary: Let's log the name as well - it will help with investigation.
Reviewed By: farnz
Differential Revision: D21906595
fbshipit-source-id: 51eb49354017c17ba3304f0a66c95dfc3c695e6a
Summary: The replacement will make lfs_server, mononoke_api, edenapi_server and walker OSS buildable.
Reviewed By: krallin
Differential Revision: D21884324
fbshipit-source-id: e6cdf8356b13056a9944ab9da18dc15977b6a2ec
Summary:
This adds changes, introduced by me in D21841293.
For my stuff: I will only land this once the configerator diff is reviewed.
Reviewed By: farnz
Differential Revision: D21862276
fbshipit-source-id: d4f5ccdd4ef9d62bb1e394d4db36930413633fa6
Summary:
The CreateProcess API allows inheritable handles to be available in the spawned
process by passing a flag to it. What the documentation for this API leave to
the imagination is what happens when processes are spawned concurrently in
multiple threads and handles are opened and made inheritable while doing this.
The answer is obvious but the consequences aren't: they are inherited.
When anonymous Pipes are used, one end needs to be inheritable for the spawned
process to being able use it, but if one is created concurrently with spawning
a process, that other process may have an open handle to that unrelated pipe.
And since to detect that a pipe is closed, all handles to the other end needs
to be closed, this can lead to never being able to detect that it is closed...
This scenario is exactly what is happening in eden when spawning the Mercurial
process, and when one of these processes would die, eden would just hang trying
to write to the pipe, not knowing that the process was already gone. To unblock
eden, all the hg.real processes had to be killed, as this would close all the
pipe handles, and then eden would detect that the pipe was closed and re-spawn
Mercurial (only for the exact same thing to happen).
In order to fix this, we need to tell CreateProcess to only inherit the pipes
handles, and nothing else.
Reviewed By: wez
Differential Revision: D21895537
fbshipit-source-id: 3c84a1d0316b50b5750f554fa20f72f59a718882
Summary:
The old `py` Thrift language support doesn't correctly handle string
vs. bytes, which causes an exception to be thrown when deserializing
paths or blobs that aren't UTF-8.
We will eventually want to migrate to the py3 language implementation,
which supports streaming.
Reviewed By: genevievehelsel
Differential Revision: D21693082
fbshipit-source-id: 0ea10fd3960f5acba353bccb83b5cf539e7eeffb
Summary:
It looks like a refactoring broke the EDEN_TEST_NO_CLEANUP feature
which was useful when trying to debug the eden state directory after a
failure.
While fixing it, I went ahead and added support for only saving the
state directory upon test failure.
Reviewed By: simpkins
Differential Revision: D21780122
fbshipit-source-id: 0cd6e9f274601eebd9b4a6978c0cf61fb1b85545
Summary:
Let's return FilenodeResult from get_all_filenodes_maybe_stale and change
callers to deal with that.
The change is straightforward with the exception of `file_history.rs`.
get_all_filenodes_maybe_stale() is used here to prefetch a lot filenodes in one
go. This diff changes it to return an empty vec in case filenodes are disabled.
Unfortunately this is not a great solution - since prefetched files are empty
get_file_history_using_prefetched() falls back to fetching filenodes
sequentially from the blobstore. that might be too slow, and the next diffs in
the stack will address this problem.
Reviewed By: krallin
Differential Revision: D21881082
fbshipit-source-id: a86dfd48a92182381ab56994f6b0f4b14651ea14
Summary:
In order to start EdenFS automatically at boot, a template service was used
previously, but due to several issues, we decided to move away from it.
Thankfully microsoft supports several other ways of starting tasks at startup,
one of which is the "Task Scheduler" itself.
One of the weird part of the task scheduler is that there isn't a good way
to tell it to not show a console for a non-graphical application, and thus
plainly executing edenfsctl start in it would create a cmd window, which
would then disappear a couple of seconds later. To avoid this, a "graphical"
version of Python is used (pythonw.exe) to start edenfsctl.
Reviewed By: fanzeyi
Differential Revision: D21732281
fbshipit-source-id: 87ef3a2d5569302392bd30a4b9e7fc48807ee316
Summary:
There's an issue where hg foo could spawn hg dynamicconfig which could
spawn hg version (via the telemetry wrapper) which could spawn hg dynamicconfig.
If the config generation fails then this could enter an infiniteloop, since the
mtime of the file would not be updated appropriately, so we will always think we
need to regenerate.
Let's prevent this by setting an environment variable when spawning the
background process.
Reviewed By: quark-zju
Differential Revision: D21885363
fbshipit-source-id: ba4938c926d1219985813a521fec412097df4b4a
Summary:
I observed that for whatever reason our setting of `use_try_shorthand = true` in rustfmt.toml was causing entire functions to not get processed by rustfmt. Even files that contain neither `try` nor `?`. Remove it and reformat fbsource.
Documentation of that config:
- https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#use_try_shorthand
We don't particularly care about the value anymore because nobody writes `r#try!(...)` in 2018 edition code.
Minimized:
```
fn f() {
g(
)
// ...
.h
}
```
This function gets formatted only if use_try_shorthand is not set.
The bug is fixed in the rustfmt 2.0 release candidate.
Reviewed By: jsgf
Differential Revision: D21878162
fbshipit-source-id: b028673c0eb703984d24bf0d2983453fc2a8c212
Summary:
When I was debugging the multithreaded bug issue it came up that the issue would
only affect certain versions of macos. There may be other bugs that come up
affecting only certain os versions. Having the os version in eden rage could be
helpful to identify such issues.
Reviewed By: chadaustin
Differential Revision: D21776154
fbshipit-source-id: a493e7da1823075ca4a845bd73b21716ce884911
Summary: This will allow me to test it easily.
Reviewed By: StanislavGlebik
Differential Revision: D21840079
fbshipit-source-id: 1b3743da9c7908eb0dedd665aa24a4bf7aabd94f
Summary: We need to thread it through this layer to get it to SCS
Reviewed By: StanislavGlebik
Differential Revision: D21840078
fbshipit-source-id: d3c16a4afd87aa844628dd0aab7f85b8d38dadf8
Summary:
We need a lowest common ancestor operation for `merge_base` call
implementation.
The `lca` function returns only one of many possible ancestors in some
situation (multiple merges of the same branches). This is done to:
simplify the implementation and hide the complexity from our API users
(people don't really expect to receive more than one node from such APIs).
Mercurial arbitrarily chooses the commit with smallest hash in those situations.
Reviewed By: StanislavGlebik
Differential Revision: D21840081
fbshipit-source-id: 2dfc95a4cf549d8941fc5166e878bfee4b6b2ece
Summary: This methods will replace the fbcode_build provided functions. Also add Cargo files for crates that are now thanks to that buildable in OSS.
Reviewed By: farnz
Differential Revision: D21549860
fbshipit-source-id: 69f4179aa7a9081d33fac1f2f88aa3b32cd31e17/
Summary: I will modify in the next diffs, it's nice to have some test coverage
Reviewed By: krallin
Differential Revision: D21880996
fbshipit-source-id: 266622982e5d7d6d19ab3ac921ddc51079e51457
Summary:
The max_depth parameter was confusing - one might have thought that it returns
all entries that are `max_depth` hops away from `startnode`. This is not the
case - in reality it just limits what's the total number of entries can be
returned.
In fact, I think it can be replaced with Stream::take() method, however that's
a larger refactoring, and I'm not going to do it now.
It doesn't matter much because usually we have linear histories, but I think
it's still worth to make it clearer.
Reviewed By: krallin
Differential Revision: D21880220
fbshipit-source-id: d209e1e129383f9181ae11b489992dc4c3ce2d5c
Summary:
In the hg sync job, we need to load up the ancestors for all bookmarks known to
the server we are pushhing to, and for e.g. fbsource that might be > 10K
bookmarks. If we fetch those 1 by 1 (because e.g. cold cache), that will take a
very long time.
Unfortunately, we don't currently have a way of buffering access to changesets,
so for now let's mitigate by buffering.
Reviewed By: ikostia, HarveyHunt
Differential Revision: D21860228
fbshipit-source-id: 90977a9e00689c1df5ae53d149c267de9b2f973e
