Summary: The same as < 500 references, except this time for targets with < 1600 references.
Reviewed By: yfeldblum
Differential Revision: D6540895
fbshipit-source-id: 40fa46c32abd6bc1c3c652a0396d6478b947f69b
Summary: Ensure everything remaining in dependencies of `folly:folly` that has < 500 references is explicit referenced.
Reviewed By: yfeldblum
Differential Revision: D6540137
fbshipit-source-id: 0a2ae5cf775278eedcccdb914688890acd12dab7
Summary: And then, there was 1, left all alone, preventing the whole thing from collapsing in on itself.
Reviewed By: yfeldblum
Differential Revision: D6469584
fbshipit-source-id: 4ea1fbf97ad466bc34f2e682394d328c97e539ba
Summary:
Added to Eden capability to incorporate default user and general system level gitignore files.
NOTE: Work in progress, sending the review out to calibrate/ensure I am on right track.
Reviewed By: simpkins
Differential Revision: D6482863
fbshipit-source-id: 9834ca1a577a9599a1f8cb2243dca4e714866be8
Summary:
Use an unbounded queue for edenfs's main thread pool. This fixes a
crash where DeferredDiffEntry multigets a batch of trees and pushes
the completion callbacks back onto the server thread pool. If the
server thread pool is bounded and throws when the queue is full, then
the import fails.
There is a slight performance hit relative to LifoSemMPMCQueue but
hopefully it isn't a big deal. An unbounded lock-free queue would be
nicer.
Reviewed By: simpkins
Differential Revision: D6490979
fbshipit-source-id: bc55dd6526f0ceb9d8b5e43a1a275250a9838aca
Summary:
There's no technical reason to block an open() request until the data
load / materialization returns. This change returns immediately from
open() and then waits if necessary in a subsequent write() call.
Reviewed By: wez
Differential Revision: D6391486
fbshipit-source-id: 862f87e3c3a0d760bacb0f8ca7acc479037fec2f
Summary:
Follow-up to comments in D6466209. All access to the clock goes
through the Clock interface, making time deterministic in unit tests.
Reviewed By: simpkins
Differential Revision: D6477973
fbshipit-source-id: 24e51bdb52d0d079b34d91598d2e787d361f2525
Summary: Follow-up from D6366189. First use of the new FakeClock!
Reviewed By: simpkins
Differential Revision: D6466209
fbshipit-source-id: 4d4d8a9a83df2bee11149e7a0cbddaaf734d0e04
Summary:
Introduce a Clock seam. This will allow us to write tests around
ctime, mtime, and atime logic.
Reviewed By: wez
Differential Revision: D6392543
fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
Summary:
open() called materializeInParent unconditionally, and setattr never
called it, implying it was possible to truncate a file without
materializing the parent. This change makes sure to precisely call
materializeInParent whenever the state transitions to materialized.
Reviewed By: wez
Differential Revision: D6389794
fbshipit-source-id: 1e740e133a83d5090a6b9801154b7eaeccb07f22
Summary:
To make the materialization code paths a bit clearer, this decouples
materialization from a blob and truncation. It also caches opened
files if openCount > 0 in both the truncation and materialization from
blob paths.
Reviewed By: wez
Differential Revision: D6388318
fbshipit-source-id: c95a85f5bdaa405130f2f7260143592cdc90d45e
Summary:
Make the calls async by using a user-provided event base.
It uses quite scary error management, see the comments for details.
Reviewed By: wez
Differential Revision: D6435696
fbshipit-source-id: a148c57aa116a6dfe6254ac7c14790101b8ecd4f
Summary:
In the initial implementation of `eden doctor`, if you had a mount point like:
/data/users/mbolin/eden-fbsource
then it looked for a pair of Watchman subscriptions for
`/data/users/mbolin/eden-fbsource` with the names:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource
as a heuristic that Nuclide was being used to edit files in that Eden mount.
However, if the user is using Nuclide to edit files in a subdirectory of that
Eden mount (as determined by the Nuclide connection profile), then the
subscriptions look like this:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource/<path-to-subdirectory>
In this case, `eden doctor` was returning a false positive because it thought
that the `filewatcher-/data/users/mbolin/eden-fbsource` subscription was missing
when in fact everything was just fine.
This revision changes the logic to look for a subscription to the root or one of
its subdirectories.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6467113
fbshipit-source-id: bfe5fafaa388405f2e59e61315c5a2084c8adc56
Summary:
This makes the `expected_output` in `stats_test.py` cleaner to declare.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6465282
fbshipit-source-id: 80b697c84f28a0fece77eb1050c4364b14685c0b
Summary:
Presumably for historical reasons, we still had some tests hanging around with
`from __future__ import absolute_import` in the header even though all of our
tests should be Python 3. I converted these stragglers and added a smattering of
type annotations to enforce their Python 3-ness.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464805
fbshipit-source-id: 6177d7663ab428a0dbbecb287f340d770679551d
Summary:
In D6446057, I added a new entry to the dict returned by
`config.get_client_info()`. I only ran the cli tests while working on D6446057,
but I should have ran all of the tests because there was an integration test
(`InfoTest`) that verified the return value of this method, so it broke due to
my change.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464806
fbshipit-source-id: 1b0ac0853301ba33e5e948353e4c89c0d97c0d83
Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().
Reviewed By: bolinfest, yfeldblum
Differential Revision: D6342086
fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242
Summary:
Make sure `hg update --clean` clears the merge state data. In non-clean
updates this is performed in `mercurial.merge.applyupdates()`. However, we
never call `applyupdates()` on clean updates in eden.
Reviewed By: bolinfest
Differential Revision: D6456720
fbshipit-source-id: b40d02ca0fb677bcde82822a8eafd5fcf926dae6
Summary: This is not material to the test, so it's distracting.
Reviewed By: wez
Differential Revision: D6462338
fbshipit-source-id: f289516c7e67891f1fed74df721fa1eb24e5f368
Summary:
`eden doctor` is a new subcommand that attempts to diagnose
issues with Eden and autofix them, as appropriate. There is also
a `--dry-run` flag that will tell the user about issues without attempting
to fix them.
Initially, `eden doctor` checks for the following:
- If Watchman has an `inotify` watcher, `eden doctor` identifies this and
will attempt to replace it with an `eden` watcher.
- If there are some active Watchman subscriptions that appear to come
from Nuclide, warn the user if the mission-critical `filewatcher-` subscription
that watches the entire repo is missing.
- If p1 in `SNAPSHOT` and `.hg/dirstate` do not match, warn the user.
The code for `eden doctor` is organized so that it should be relatively
easy to add new conditions to check for going forward. Admittedly, the
UX could be better by formatting the output (color, boldness, etc.) to
make certain information stand out, but we can improve that in
subsequent revisions.
Note that I had to do a bit of cleanup in `eden/cli/TARGETS` as part of
this revision and I created `eden/cli/test/TARGETS` so the tests have
their own build file.
Reviewed By: chadaustin
Differential Revision: D6446057
fbshipit-source-id: ae23c5996dba4f7f70f118179e5556efc29c31c3
Summary:
Always call `umount2()` with both `MNT_FORCE` and `MNT_DETACH`. Without
`MNT_DETACH` I see unmount operations still fail or hang in some cases on
edenfs shutdown. This should hopefully help fully eliminate situations where
disconnected mount points are sometimes left behind when eden shuts down.
Reviewed By: bolinfest, chadaustin
Differential Revision: D6455971
fbshipit-source-id: 5abe456e17a33a0080ad94b4d315540bce0c2f82
Summary:
Update TreeInode::computeCheckoutActions() to short circuit on non-materialized
directories that are identical to both the destination and source state. This
speeds up `hg update --clean` operations by allowing us to skip portions of the
tree that do not require changes.
This does not have much effect for update operations without `--clean`: those
already short-circuited the operation in `TreeInode::processCheckoutEntry()`
when the source and destination hashes were the same. That code cannot
short-circuit in the forced update case, since the forced update may have to
revert local modifications.
Reviewed By: bolinfest
Differential Revision: D6455970
fbshipit-source-id: 393acb3272745751a56e06dba0c7505ff2bfad44
Summary:
Eden already disables sqldirstate as part of its post-clone hook. Add
treedirstate to this handling.
Reviewed By: bolinfest
Differential Revision: D6424041
fbshipit-source-id: 889c07ed32f58c50c15de3a9aa3135018c4761a9
Summary:
It simplifies next diff a bit.
Also removes unnecessary `makeFuture()` calls.
Reviewed By: wez
Differential Revision: D6435694
fbshipit-source-id: ac00bed90454fef645a4fdad254ba6c8d3bda0ab
Summary: Additional thread for http server is not necessary.
Reviewed By: wez
Differential Revision: D6435695
fbshipit-source-id: b4fc7096a87146f36876bd132db66ab964feba51
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
Add a method to LogHandler to return its current configuration. This will
make it possible to query the LoggerDB for its current configuration state.
Reviewed By: bolinfest
Differential Revision: D6200563
fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().
This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.
Reviewed By: bolinfest
Differential Revision: D6434359
fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
Summary:
Increase the eden daemon start timeout from 5 to 60 seconds.
With the 5 second timeout I sometimes see `buck test eden/...` time out in the
clone integration test which waits for eden to start. This likely takes longer
than normal when lots of other integration tests are also running in parallel.
Even when simply manually starting eden I have seen it take nearly 60 seconds
to open the RocksDB database. (I suspect RocksDB performs some extra checking
when opening a DB created by an older version of the code.) Eden requires the
RocksDB be open before it starts responding successfully to health check
requests.
Reviewed By: wez
Differential Revision: D6434357
fbshipit-source-id: 5f62ff821ecd64f7a1345e611f2299177513e411
Summary: Update the functions in cli/util.py with type annotations.
Reviewed By: bolinfest
Differential Revision: D6434356
fbshipit-source-id: 054d81427132229a390b8a133d5180be5f70bd20
Summary: Annotate more integration test functions with type information.
Reviewed By: bolinfest
Differential Revision: D6434358
fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
Summary:
This fixes a crash in EdenMount::destroy() if EdenMount::create() failed to
load the root inode. Previously the code called shutdownImpl() in this case
which tried to unload all inodes and crashed since the root inode was null.
This also fixes EdenMount::destroy() to properly handle the FUSE_ERROR and
FUSE_DONE cases.
Reviewed By: wez
Differential Revision: D6434355
fbshipit-source-id: 39c5f4472d6ebbcf881b4c9c8c8fd67686032ec1
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
The goal is to provide a fast path for watchman to flesh
out the total set of changed files when it needs relay that information
on to consumers.
We choose not to include the full list in the Journal when checking out
between revisions because it will not always be needed and may be an
expensive `O(repo)` operation to compute. This means that watchman
needs to expand that information for itself, and that is currently
a fairly slow query to invoke through mercurial.
Since watchman is responding to journal events from eden we know that
we have tree data for the old and new hashes and thus we should be
able to efficiently compute that diff.
This implementation is slightly awful because it will instantiate an
unlinked TreeInode object for one side of the query, and will in
turn populate any children that differ as it walks down the tree.
A follow on diff will look at making a flavor of the diff code that
can diff raw Tree objects instead.
Reviewed By: bolinfest
Differential Revision: D6305844
fbshipit-source-id: 7506c9ba1f4febebcdc283c414261810a3951588
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
This should have been removed as part of D6179950.
Ideally, Buck would error when this happens, but apparently `glob()` does not
complain when patterns do not match any files, even when the pattern does not
contain any wildcards. There appears to be some code at Facebook that is
exploiting this behavior.
Reviewed By: simpkins
Differential Revision: D6421529
fbshipit-source-id: c6f982624e0e12a911bc12ab1e8239ba4358ea56
Summary:
This corrects a bug where the in-memory timestamp for new files would
be set to the last checkout time instead of when the file was created.
Reviewed By: simpkins
Differential Revision: D6366189
fbshipit-source-id: c5fa8c779726d3a75c2d57b2a161293297eb9271
Summary:
Per a discussion with wez, let's make it explicit in the comments that the kernel handles
O_EXCL for us.
Reviewed By: simpkins
Differential Revision: D6365157
fbshipit-source-id: 638f67cd89a9450ff084e0ef77c731ef738bf518
Summary:
In some workloads we're seeing folks run out of file descriptors.
We forgot that we'd taken out the code that closes the underlying fds.
This diff takes a run at adding a simple counter of the open file handle
objects that is incremented when they are constructed and decremented
when they are destroyed.
When the count falls to zero we release the file handle.
Note that we unconditionally open files when we first load the inodes
from the overlay. I tried to defer that open attempt and it broke
the timestamp overlay test. I think we can revisit that aspect in
a follow on diff; for now we should be more resilient to transiently
opened files from things like ripgrep or similar.
Reviewed By: simpkins
Differential Revision: D6097090
fbshipit-source-id: 9a48220002e760fb1ffb8d7e2a68fa7036558b78
Summary: It is currently unused. Let's bring it back if/when we need it.
Reviewed By: chadaustin
Differential Revision: D6368867
fbshipit-source-id: 096015ba597a6e04f544273ba9773576429e39ce
