Summary:
The messaging implies that something is already mounted, especially
if one is used to the error messages that are emitted by `mount`. In this
situation we're complaining that we already have a repo configured at that
path, so let's help point folks in the right direction in the error message.
Reviewed By: chadaustin
Differential Revision: D6344017
fbshipit-source-id: d48176f5806b3233e191567a840c88af101002cf
Summary: Otherwise these will linger forever in the eden server
Reviewed By: DurhamG
Differential Revision: D6099753
fbshipit-source-id: 77b975dcde28cd7c3d4ae2302bddb625682d1994
Summary:
Update hg_import_helper.py to include the exception type name in error
responses. Add a new HgImportPyError class in the C++ code to include both the
python exception type name and the message string.
In the future this will give us the ability to perform special handling based
on the python exception type, rather than just on the message contents.
Reviewed By: bolinfest
Differential Revision: D6333613
fbshipit-source-id: 1074bbf9fa25ee8b1abeadc38b1a4f569bc18d13
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
Add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() macros to folly/test/TestUtils.h
These allow more precise checks than the basic EXPECT_THROW() macro provided as
part of gtest.
These macros are being moved into folly from Facebook's eden repository
(https://github.com/facebookexperimental/eden)
This will allow us to use them in folly tests and in other projects that depend
on folly.
Reviewed By: yfeldblum
Differential Revision: D6301760
fbshipit-source-id: 1f434fb5bc9b7859f763171264fb0b2e1b4bda62
Summary:
This script automates the process of force unmounting all edenfs
mounts. It's very helpful when a `buck test eden/...` goes awry. :)
Reviewed By: wez
Differential Revision: D6314547
fbshipit-source-id: a2374c4f3220ae663a34b5ff252bf85ddab6d0aa
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D6265043
fbshipit-source-id: b2d4f345b772f296c5335a7fbcadfce1d93245fd
Summary:
Users often run `hg update --clean .` to get out of a bad state, but this was
not clearing the "added" or "removed" state in the dirstate as it should in
Eden.
Reviewed By: wez
Differential Revision: D6331858
fbshipit-source-id: 616f187930587a1af40a1f151e3a424d50dd8da3
Summary:
https://phab.mercurial-scm.org/D1268 changed the error message that the
treemanifest code throws when it cannot fetch tree data from the remote server.
This unfortunately is thrown as a generic `error.Abort()` so eden has to match
on the error message to tell if this was an error fetching the tree vs
something else. Changing the error message broke eden's detection of this.
This updates the string that eden searches for. In the future it would
probably be nicer to make the treemanifest code throw a different exception
type so we can detect this from the exception type rather than the error
message.
Reviewed By: wez
Differential Revision: D6332802
fbshipit-source-id: 629097914e6f783729adcd25f2fd5b6d96886ab4
Summary:
Add a new debugging flag to the hg_import_helper.py script to have it just show
the manifest node ID for a given revision and then exit.
This is information is useful when debugging import issues, and I'm not aware
of an easy way to get it from the normal hg command line.
Reviewed By: wez
Differential Revision: D6332803
fbshipit-source-id: 600feac105f277d398325d2e3c50eae3d5200940
Summary:
Override dirstate.rebuild() so that it drops all file changes without trying to
mark every file in the commit manifest as normal. We don't want to track
normal files in the eden dirstate.
Reviewed By: bolinfest
Differential Revision: D6322227
fbshipit-source-id: d81ade1cdafb5fa03c642239b0cff91308c7fc35
Summary:
Now that some parts of sparse are being upstreamed to core mercurial,
Facebook's sparse implementation has internally been renamed to "fbsparse".
Make sure the sparse extension is disabled inside eden repositories under both
names.
Reviewed By: bolinfest
Differential Revision: D6322114
fbshipit-source-id: c62b2ce25cb92519f2df1d7df6d43e2b209751d3
Summary:
Previously calling `hg debugdirstate` inside an eden repository crashed, since
it would try to iterate over the dirstate and we do not allow this.
This re-implements the `debugdirstate' command inside eden repositories to
print out the data stored in eden's dirstate file.
Reviewed By: wez
Differential Revision: D6322052
fbshipit-source-id: 92f230438a545fe83134b0d67545ebb89877d2f0
Summary:
Moving the post-clone step out of C++ makes it so that
`ClientConfig` in C++ no longer knows about `hooks` and requires
`RepoConfig` to know about `hooks`. This helps us reduce
`ClientConfig`'s dependency on `ConfigData`, as my next step
is to move the remaining information that `ClientConfig` gets from
`ConfigData` directly into the client's `edenrc` file under
`~/local/.eden`.
Reviewed By: chadaustin
Differential Revision: D6310544
fbshipit-source-id: dec7a21281ab49e0416b8872757970a4eff2d943
Summary:
This will facilitate an upcoming change where I include the `hooks` information
as part of the data returned by `get_repo_config()`
(formerly `get_repo_data()`).
Reviewed By: chadaustin
Differential Revision: D6311536
fbshipit-source-id: 2e1bc063c4a12772c9806f6880822fb15e18fc92
Summary:
I created this change by running:
```
find eden -name TARGETS | grep -v eden/fs/fuse/TARGETS | grep -v eden/fs/service/TARGETS | xargs autodeps
```
apparently `eden/fs/fuse/TARGETS` and `eden/fs/service/TARGETS` have some
constructions that `autodeps` does not understand, so I filtered those out.
Reviewed By: StanislavGlebik
Differential Revision: D6319982
fbshipit-source-id: 7b3683d1507409dde6d6570e9b13811168aa6859
Summary:
This seems like something I missed during the original code review in D3554550.
In practice, there probably isn't much contention for this lock, so it's not
surprising that we haven't knowingly been bitten by this yet.
Reviewed By: chadaustin
Differential Revision: D6310326
fbshipit-source-id: c16a07aa654240be77a38c16473a4cc1e06125b6
Summary:
This seems like something that I should have deleted as part of D6179950 when we
moved the storage of dirstate data from Eden to Hg.
Reviewed By: chadaustin
Differential Revision: D6311543
fbshipit-source-id: df00d348be9a9dbbce18fa81e2cd1015b1780b02
Summary:
This is another, more efficient way to fix the mode/opt build issues
seen in D6272580. When C++17 is released, it could be updated to
std::size.
Reviewed By: yfeldblum
Differential Revision: D6274205
fbshipit-source-id: 07f25a1a0f6575b80b7d1a8af7b7c723f0f9e4f0
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber. The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked. This change
prevents future deadlocks.
Reviewed By: wez
Differential Revision: D6281410
fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
Summary:
With the subsequent diff that enables multiple concurrent
hg importers, I was seeing this deadlock during rebase; each of
the worker threads was being blocked until it saturated the various
thread pools and locked up the server.
This removes the blocking call and replaces it with a SharedPromise
to allow multiple callers to wait for the load.
Note that this changes the semantics of ensureDataLoaded slightly:
previously the blob load was synchronous while the write lock was
held. Now we release the write lock until the load completes.
I've added some post-condition checks to validate that we don't
break any state. I believe that our usage is such that we don't
do anything that might mess things up. Am I missing anything?
Reviewed By: simpkins
Differential Revision: D6264900
fbshipit-source-id: 4aa2870d95f0f0ec48d87299978cb87af99e3969
Summary:
Tweak the `INSTRUMENT_THRIFT_CALL()` log levels in EdenServiceHandler.
For the most part this changes the code so that modifying calls (e.g.,
`mount`, `checkOutRevision`, etc) are logged at `DBG2` and higher, while
read-only calls (`getSHA1`, `getParentCommits`) are logged at `DBG3` and
lower. Since we log all eden messages at `DBG2` by default this means that
modifying calls will be enabled by default but read-only calls will be
disabled.
Some important read-only calls such as `getScmStatus` and
`getFilesChangedSince` do still log at the `DBG2` level.
The main motivation for this is that `buck build` often triggers many thousands
of separate `getSHA1` calls. It doesn't seem terribly valuable to flood the
eden logs with these messages by default. These can always be enabled at
runtime if desired when debugging buck performance.
Reviewed By: bolinfest
Differential Revision: D6295566
fbshipit-source-id: dd344c1dea773f4f2a56e2b0dbb18b8738303944
Summary:
Address some feedback on D6264900 by only storing the sha-1 in the
overlay when it's immediately available. This avoids the possibility
of the FUSE thread getting blocked.
Reviewed By: simpkins
Differential Revision: D6292175
fbshipit-source-id: 06c2372724e58c485d9a302dde36c34885109acf
Summary:
In the course of verifying a fix for `hg update --merge` in D6270272, I
discovered a new bug in our merge logic in the Python code. As expained in the
test plan, there was a case where a file was listed as "untracked" instead of
"added" after a merge with `--tool :local`.
I traced through what happens in stock Mercurial. After the call to
`applyupdates()` in `update()` in `merge.py`, there is this code:
```
stats = applyupdates(repo, actions, wc, p2, overwrite, labels=labels)
wc.flushall()
if not partial:
with repo.dirstate.parentchange():
repo.setparents(fp1, fp2)
recordupdates(repo, actions, branchmerge)
# update completed, clear state
util.unlink(repo.vfs.join('updatestate'))
if not branchmerge:
repo.dirstate.setbranch(p2.branch())
```
It turns out that `applyupdates()` can have the side-effect of adding new
entries to the `actions` dict. In this case, we have a `'cd'` action for which
an `'am'` action is generated. Our `merge_update()` function in
`eden/hg/eden/__init__.py` did not have the `recordupdates()` call that the
stock implementation of Mercurial does, so the `'am'` (for "add/merge") was not
getting applied.
It seems likely that introducing this `recordupdates()` call may fix other
subtle bugs in Eden's Mercurial extension for which we do not yet have
integration tests.
Reviewed By: wez
Differential Revision: D6279971
fbshipit-source-id: 901c1bc563a7a3910dde18cf2f0d8b8ff9cd6fbe
Summary:
The underlying issue is that we were reporting a `MODIFIED_REMOVED`
conflict as a `MODIFIED_MODIFIED` conflict. This put us in a state where
Mercurial expected to find a file in the new manifest, but failed because the
file was not present in that revision, so no such file could be found.
Somewhat surprisingly, the appropriate handler for a `MODIFIED_REMOVED`
conflict already existed in our Mercurial extension, but there was no logic on
the server that would generate a `MODIFIED_REMOVED` conflict previous to
this change.
Like D6204916, this was an issue I ran into when trying to create a repro case
for the issue that was fixed in D6199215.
Reviewed By: wez
Differential Revision: D6270272
fbshipit-source-id: 6604eea00b0794cd44b01d2ba6b9ea10db32d556
Summary: Make this function match our C++ guidelines.
Reviewed By: wez
Differential Revision: D6288591
fbshipit-source-id: 1a4f52a8c1e0497df938533fe29da10264eb1ccf
Summary: We were previously potentially deleting the include prefix. We also weren't using the cpp2 include prefix if the global one wasn't set. This makes sure we use it, and fixes a bug where 'X_types.h' was included without a full prefix.
Reviewed By: yfeldblum
Differential Revision: D6236108
fbshipit-source-id: 076747fcab2b1414bafa42c9e481ba1e1e5df4b1
Summary:
This setting is bad for a few reasons
1) there is no correct value; rejecting connections is typically something you want to do when under pressure, which no fixed number of connections can indicate
2) it's not discoverable when it trips, and pretty much always confuses people
3) the effect is generally not better than the thing it is theoretically supposed to prevent, when it trips, servers crash and exhibit other weird behaviors; I don't think I've ever seen a situation where I thought this was needed, and I've seen many were it created problems where none would have existed.
About a year ago, I removed almost all calls to this; however, I left some behind since they were slightly harder to clean up or had unique flag names. Since then, 2 things have happened
1) more copypasta instances have cropped up
2) More people have run into issues with this flag, notably up2x recently
This removes all calls except one (realtime Pylon has some somewhat more complex config here, I'll need to talk to them). This is somewhat aggressive; some of the calls are totally safe to remove, as they're either copypasted or set it to 0 or some absurdly high number. Others are less obvious. I've decided to adopt a door-in-the-face strategy here and see if anyone tells me I should be more conservative
I couldn't delete all the flags; these ones are in use
zdb_thrift_max_connection
thrift2_max_connections (commerce ranker)
max_connections (a bunch of places)
maxConnections (presence)
I'll need a separate diff/set of diffs to delete those
Reviewed By: eduardo-elizondo, yfeldblum
Differential Revision: D6182563
fbshipit-source-id: e778edd9da582f4ca90a902621cb49f1e04ca26e
Summary:
Add log messages that will help debug the behavior of `TreeInode::diff()`
I added this since I saw a case where `getScmStatus()` was returning a file as
modified when it did not appear to have mode or contents changes.
Unfortunately we did not have any existing debug log messages for this, and the
problem went away after restarting edenfs, so I wasn't able to track it down.
Having these log messages will help debug this in the future if we run into it
again.
Reviewed By: bolinfest
Differential Revision: D6281905
fbshipit-source-id: 736028d6efe46a387bed6bf98d971685f83da3ec
Summary:
Building with gcc/opt mode complains that `strlen()` is not a constant
expression, so I'm dropping the `constexpr` to fix the opt build.
Reviewed By: chadaustin
Differential Revision: D6272580
fbshipit-source-id: a8e3a93ea04e878353c6cab31cb0b1b4705276fe
Summary:
In Eden, some Thrift endpoints get a lot of hits whereas others are used less
frequently. By giving each endpoint its own logging category, we can toggle the
logging for each one independently.
Each of our Thrift endpoint methods should start with the following macro:
INSTRUMENT_THRIFT_CALL();
then within the method, the macro `TLOG()` should be used everywhere
`XLOG(LEVEL)` would normally used. `LOG()` ensures the logger with the category
specific to the method will be used.
For an endpoint named `XXX`, the logging category will be `eden.thrift.XXX`.
Reviewed By: simpkins
Differential Revision: D6108311
fbshipit-source-id: 23a34179811359b0819434de31a3601d29c3b4f0
Summary:
If a file is explicitly added via `hg add`, then it should be considered added
even if it matches a pattern in `.gitignore`. Further, if it is deleted without
running `hg forget`, it should be considered missing rather than ignored.
To make this work, I had to update `eden_dirstate.walk()`, which already
had a special case when used as part of `hg add`. The new logic ensures
that files that are specified explicitly are still considered even if they are
matched via `.gitignore`.
I also had to address a TODO in `EdenThriftClient.py` related to the
handling of ignored files that was introduced as part of the major
`eden_dirstate` changes in D6179950. It was expected that it would
be easier to handle ignored files properly after D6179950 landed.
Reviewed By: wez
Differential Revision: D6242223
fbshipit-source-id: cf1cfe97a8d2ec57bce1d524074c43978a78e4ef
Summary:
Python 3 type checking currently complains about most of our integration
testing since the tests use an `hg_test` decorator to inherit from the base
test class. This prevents the type checker from being able to figure out this
inheritance.
This updates all of the test cases to explicitly derive from the test case base
class, rather than using the decorator to do so. I also renamed the base test
case class to `EdenHgTestCase` to be slightly more succinct and to follow the
common pattern of calling `unittest.TestCase` subclasses `FooTestCase`
Reviewed By: bolinfest
Differential Revision: D6268258
fbshipit-source-id: 09eef2f8217932a6516f78d17dddcd35c83b73da
Summary:
This adds an integration test that exercises a deadlock we could encounter in
the past. An "hg status" operation could trigger many trees and files to be
imported. Unfortunately the file import code currently blocks waiting for file
import futures to complete. This could result in a state where all threads in
the pool were waiting for a file import to complete, and the file import was
waiting for a free thread to complete.
Reviewed By: bolinfest
Differential Revision: D6216871
fbshipit-source-id: e1795a543a71fccbed035febb159e126e27d1950
Summary:
This fixes an issue where the `DIRECTORY_NOT_EMPTY` conflict type reported by
the server was not handled by the client. Somewhat ironically, the fix appears
to be to explicitly "do nothing," though the important part of this revision is
the new integration test.
As this is only one test, I'm not convinced this covers all possible corner
cases, but it's certainly better than blowing up, which is what we did before.
Reviewed By: wez
Differential Revision: D6264069
fbshipit-source-id: a7c45a43776a903a4d6b6cdfb0ce75db9549c380
Summary:
A few fixes:
* Fix a bug where `date_str` would not get set when `date` was specified.
* Remove `from __future__ import` stuff since this code is Python 3.
* Add type annotations to the `commit()` method.
Reviewed By: simpkins
Differential Revision: D6261874
fbshipit-source-id: 5f942d01c107cd0265c2d6ec6e1f46295bb3ec24
Summary:
If you have an untracked file and you `hg update` to a commit that has
that file in the tracked state, then the contents of the untracked version
should be ignored, as they are replaced with the contents of the file in the
commit you are updating to. The untracked version should be backed up
as specified by `ui.origbackuppath`.
Previously, our code in `eden/hg/eden/__init__.py` mapped this to a merge action
named `c`, but we did not include that in our set of `actions`, so we were
getting a `KeyError` if you exercised this code path.
I discovered this while trying to reproduce the issue that I fixed in D6199215.
Reviewed By: simpkins
Differential Revision: D6204916
fbshipit-source-id: b70153428291bda9a8853a37c0955ad7cb3bd89d
Summary: The comments and the code did not agree on what fields were valid when, so I corrected them and added runtime enforcement.
Reviewed By: wez
Differential Revision: D6260555
fbshipit-source-id: 13e35e22c44b9a5b4cb30cdfd48e2bf7b0e116d3
Summary:
Since Eden's integration tests do not run in CI yet, this adds a
test that verifies the Eden CLI can start without Python errors.
Reviewed By: wez
Differential Revision: D6250515
fbshipit-source-id: 907bffaff122c9929a7623d97f665de5b2a6f2d3
Summary:
Opening the RocksDB store can take a long time if there is a fair amount of
data in the store. In my current .eden directory I have around 10GB of data in
the local store, and it takes RocksDB nearly 60 seconds to open the database.
These log messages help provide a little more information about what edenfs is
doing during this startup delay.
Reviewed By: bolinfest
Differential Revision: D6263603
fbshipit-source-id: a0945aa1cc020b95944b365b17869756dcc27407
Summary:
Update the eden.dirstate code to return parent information correctly even if
the .hg/dirstate file is still using the old format.
This does not return non-normal tuples correctly. If we really wanted to we
could add the hgdirstate.thrift file back so we could parse the old dirstate
file correctly, but it doesn't really seem worthwhile for now. Simply
returning the parents information will probably be good enough to help most of
our current users upgrade.
Reviewed By: bolinfest
Differential Revision: D6263602
fbshipit-source-id: 9b65335b0e98ba3fd79d2f220c82df18753ccfdc
Summary:
Unless you're going to clean up and then reraise, catching
BaseException rarely makes sense, since it swallows keyboard
interrupts.
Reviewed By: simpkins, bolinfest
Differential Revision: D6262224
fbshipit-source-id: b6b9bc963a4c75fe223c3b787ba822c3cf5da24d
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary:
Apparently `Dict[str, [int, int, int, int]]` is not a valid MyPy type.
Fixed to be `Dict[str, Tuple[int, int, int, int]]`.
This was introduced in a recent change (D6189543).
Reviewed By: danzimm
Differential Revision: D6241810
fbshipit-source-id: a32da67be0c555f985669e7fc2d8058f469bb371
Summary: Continue moving the asynchrony of loading closer to the state access so, in a future world, we still behave if unloading occurs during access.
Reviewed By: simpkins
Differential Revision: D6238523
fbshipit-source-id: 722fe5bba90b55204d50393314d225f42680409b
Summary: To fix up the invariants in FileInode's API, we intend to remove ensureDataLoaded() and materializeForWrite(). But first I'm going to see if there is any pain caused by removing their calls.
Reviewed By: simpkins
Differential Revision: D6234049
fbshipit-source-id: c39c6d018d164b32903414d3750b875a897af210
Summary: Adds an `eden stats thrift` command that prints call counts for each Thrift call, similar to `eden stats io` printing FUSE counts.
Reviewed By: bolinfest
Differential Revision: D6189543
fbshipit-source-id: 59c29a4036687e1bf75b1c0ca2a7d311b9d1399f
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary: If writing out the symlink file in the overlay fails, clean up the file before releasing the lock.
Reviewed By: wez
Differential Revision: D6234214
fbshipit-source-id: 91ccf917a26cc1a73c3963dea7dd87364857fa03
