Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary: Fix the remaining set of errors reported by pyre and mypy.
Reviewed By: strager
Differential Revision: D13086855
fbshipit-source-id: 4c2b21352f94ef225a5555aef0f6b95b92e56f6d
Summary:
This updates the integration tests to add type annotations to most functions
that were missing annotations.
In particular this is needed to make pyre happy, as it complains if subclasses
override methods from their parent class and do not specify type annotations
if the parent class did have annotations.
This diff also contains some minor changes to hg_extension_test_base.py to
explicitly declare some abstract methods that it uses. This was also
necessary to make pyre happy about this ocde.
Reviewed By: strager
Differential Revision: D13051097
fbshipit-source-id: 77567ed2f4d3050f93acefb52e688932d276d587
Summary:
Drop the `stdout` and `stderr` arguments, so that this method always return a
string. Change callers that were previously calling this method with
`stderr=None` to use the `run_hg()` method instead of `hg()`. `run_hg()`
returns a `subprocess.CompletedProcess` object.
This change simplifies the python type checking, and fixes several existing
type checking errors in the code. Even though most call sites could be
guaranteed that this function would return a `str`, the type checker wasn't
smart enough to tell that the return type would be fixed based on the argument
values, and so it assumed the result always needed to be checked for `None`.
This also updates the `GitRepository.git()` method in a similar fashion.
However, that was a simpler change since it already returned a `str` in all
cases.
Reviewed By: strager
Differential Revision: D13078095
fbshipit-source-id: a8def2a33edc865ac40279bbcb3ada4dade68374
Summary:
Restructure storage_engine_test so that the base class derives from
EdenTestCase.
Reviewed By: strager
Differential Revision: D13051096
fbshipit-source-id: e89c4b56e361460b2457d1c2e6a22727a25d7646
Summary:
Add type annotations for class member variables. The pyre type checker has
some limited automatic type detection for member variables set in
`__init__()`, but in general it expects member variables to be explicitly
declared at the top-level of the class.
Reviewed By: strager
Differential Revision: D13051092
fbshipit-source-id: 080259ab3f422ffae2b908ed610062237105ccbe
Summary:
The `test_hg_clone_non_eden_repo_within_eden_repo()` test had been disabled
when running with treemanifest enabled since the treemanifest code did not
correctly detect which repositories should actually use treemanifest.
This issue appears to have since been fixed in the treemanifest code, so this
test passes now.
Reviewed By: quark-zju
Differential Revision: D12927801
fbshipit-source-id: b9e2e041f7eab5e24007888e2dba142e1f0b2251
Summary:
Update the HgRepository.status() function in the integration tests to return
the status information as a dictionary instead of a string. Only one test was
still using the old string API.
Reviewed By: chadaustin
Differential Revision: D10503168
fbshipit-source-id: 574e4438d23bf6612a70ae5ae3174db3d464d198
Summary:
Update the eden+hg integration test code so that it uses find_executables.py
to find the post clone hook. Previously it was still using its own one-off
helper function for this.
Moving this code into find_executables.py will make it possible have the
snapshot tool also find the post clone hook.
Reviewed By: chadaustin
Differential Revision: D10503169
fbshipit-source-id: d59e3e517d19e80e0d0c997be09865519b80d34e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
When you get an error, let's print the number of conflicts in each file. This will give the user some sense of how much work they have to do.
The code change is entirely in `filemerge.py`, and `tests/test-merge-conflict-count.t` adds a new test.
Reviewed By: quark-zju
Differential Revision: D9815243
fbshipit-source-id: 1b73a1db293902ac7242997a7d6ae09478344068
Summary:
Add a new sparse show command, which shows the current sparse configuration.
Use common templating, including colors, between sparse show and sparse list.
Reviewed By: phillco
Differential Revision: D10446705
fbshipit-source-id: acb539f80f625945716758c785703b46a81aa6f1
Summary: Some commands in extensions have incorrectly defined usage. Correct these.
Reviewed By: phillco
Differential Revision: D10446707
fbshipit-source-id: e606314892a31fa09f4a655042d66d87929fc2ec
Summary: D10335083 renamed `fbsparse` into `sparse` and I forgot to udpate this test.
Reviewed By: simpkins
Differential Revision: D10423192
fbshipit-source-id: a547c403a9461d6923ecfecc931df82bb7887597
Summary:
Note that this is more aggressive than the tweakdefaults version; it doesn't allow named branches to be created or listed at all with the config set. As we want to delete named branches entirely, this seems reasonable.
I ported the `--new` flag, but made it optional, to make it easier to support both the core and tweakdefaults callers.
Saurabh Singh and I chatted with the releng team (Craig).
**tldr:** we can remove named branches from the UI but should keep the innards for now
Today the releng team doesn't make any branches (#continuous), but in the near-term future, they might either:
- create remote bookmarks in hg (similar to fbsource)
- create branches in SVN (which are then synced as hg branches by hgsubversion)
- We'll leave all the hgsubversion code to do this alone, just remove user-facing entry points
- They'll also need to be able to query for a commit's branch (`hg log -T '{branch}'` will still work)
- Once www is on hg, this possibility goes away and we can kill named branches for good
Reviewed By: ikostia
Differential Revision: D10401485
fbshipit-source-id: 6f2f3ae28af249bae1fdf782eb14fe7bfc472bb7
Summary:
Update hg absorb to use the templater and display the first line of the
description of the affected commits.
Also change the default behaviour to display the changes and prompt the user to
confirm that absorb is going to do what they want. Use the new `-a` option to
get the old behaviour.
Reviewed By: quark-zju
Differential Revision: D10366510
fbshipit-source-id: 23df2e5ab2d21a0805a16e7118686a29634d2918
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.
Reviewed By: farnz
Differential Revision: D10320739
fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.
Reviewed By: strager
Differential Revision: D10020958
fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
Summary:
The logic we care about has been moved to core so we can delete the
`inhibit` extension.
Reviewed By: markbt
Differential Revision: D10276448
fbshipit-source-id: 44cabe5d561344cf8f196127ce52491e4654b598
Summary:
Fix the code in `eden_dirstate._call_match_callbacks()` to correctly match
ignored symlinks that are explicitly listed in the matcher's file list.
It looks like this was just a mistake in the original code on my part: I
intended to check both `S_ISREG()` and `S_ISLNK()` but instead incorrectly put
the check for `S_ISREG()` twice.
Reviewed By: wez
Differential Revision: D9476524
fbshipit-source-id: 67e0fa7c2fbaac97598a8e2d028c9ef0999ed88a
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
The eden integration tests currently run hg through the telemetry wrapper.
This wrapper script adds profiling arguments to a percentage of the commands
it runs. The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".
This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".
This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.
Reviewed By: wez
Differential Revision: D8631180
fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map. Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.
Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed. Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all. This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks. As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.
Reviewed By: bolinfest, chadaustin
Differential Revision: D7980375
fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is. This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.
Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path. That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.
Reviewed By: wez
Differential Revision: D7705655
fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
Summary: I forgot to remove this comment when I fixed the test in D7565503.
Reviewed By: chadaustin
Differential Revision: D7716106
fbshipit-source-id: 58acd71ac96560f2cc454b3a7da75bb61e6e71c7
Summary:
These were making some noise in the Nuclide diagnostics pane.
In the course of making these changes, I ended up reformatting
some files, as well. Perhaps we should flag flake8 and autoformat
violations more aggressively in CI now?
Reviewed By: chadaustin
Differential Revision: D7658030
fbshipit-source-id: b52a29b1cc242967f868dcc8ee46ec1bb9bdfbee
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:
- Report the files with conflicts. The conflicts may be non-obvious to the
user if the conflicts are in ignored files that are not tracked in the
current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
after we have performed the conflict check. Otherwise the repository will be
marked as in the middle of an unfinished update even though we never
attempted the update itself.
Reviewed By: quark-zju
Differential Revision: D7581782
fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations. We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.
Reviewed By: quark-zju
Differential Revision: D7581781
fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.
The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`
The changes to the `walk()` function do fix a number of bugs. We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest. All of the glob
matching is done purely in python now. This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial. The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory). I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.
Reviewed By: chadaustin
Differential Revision: D7565503
fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.
Reviewed By: chadaustin
Differential Revision: D7556850
fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.
Reviewed By: chadaustin
Differential Revision: D7556847
fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.
Reviewed By: wez
Differential Revision: D7512339
fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`
This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata
Reviewed By: wez
Differential Revision: D7512321
fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories. This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.
The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.
This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created. This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.
Reviewed By: chadaustin
Differential Revision: D7512320
fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
Summary:
Refactor the find_executables module to only look up executables when they are
needed, instead of performing all look-ups immediately even if the test in
question may not need all of the binaries. Also add a _find_exe() helper
function to eliminate some code duplication.
Reviewed By: ryanmce
Differential Revision: D7512038
fbshipit-source-id: fdfb8ec70b3f6292603826b3fb22c01dbd1f0d72
Summary:
We already had type annotations on most of the `hg` integration tests. This
adds them for the top-level (non-source-control-specific) tests.
typeseverywhere
Reviewed By: wez
Differential Revision: D7459281
fbshipit-source-id: 41266b232ded510d6b63dd3e62c272a0cd6a0e1a
Summary:
This integration tests appears to be frequently hanging on sandcastle, and
leaving behind unkillable zombie processes. I'm disabling it for now until we
can figure out why it hangs and fix it and/or make sandcastle smart enough to
be able to successfully tear down the FUSE mounts.
These stuck processes are likely stuck waiting on a kernel lock that can only
be released by using `/sys/fs/fuse/connections/DEVICE_ID/abort`
Reviewed By: wez
Differential Revision: D7440282
fbshipit-source-id: 70034fd65267eff18bf5a052cde8d11e0a66b46b
Summary:
Update FuseChannel to send all invalidation requests in a separate thread.
This eliminates a deadlock that could previously occur during checkout
operations. The invalidation requests would block until they could acquire the
kernel's inode lock on the inode in question. However, the inode lock may
already be held by another thread attempting to perform an unlink() or rename()
call. These FUSE unlink or rename operations would be blocked waiting on
Eden's mount point rename lock, which was acquired by the checkout operation.
Checkout operations now let the invalidations complete asynchronously, but we
wait for all invalidation operations to complete before indicating to our
caller that the checkout has succeeded.
Reviewed By: chadaustin, wez
Differential Revision: D7404971
fbshipit-source-id: 6fa20c00d054e210eb0258d247d083010557f210
Summary:
The status_deadlock_test code has a small helper function used to create
directory trees in the test. This moves that function into a helper module so
we can re-use it in other tests in the future.
Reviewed By: chadaustin
Differential Revision: D7407492
fbshipit-source-id: 257e5a2ce7543bb6cd218b412d165f0fac852970
Summary:
Previously we returned an ENOENT error in response to a FUSE lookup() call for
a name that does not exist. However, this does not allow FUSE to cache the
result, so we will continue to receive lookup() calls for this path in the
future.
This changes EdenDispatcher to return a successful response with an inode
number of 0 instead. This tells the kernel that the name does not exist, but
allows the kernel to cache this negative lookup result (for as long as
specified in the entry_valid field in the response).
Reviewed By: wez
Differential Revision: D7076811
fbshipit-source-id: a2b9977e58d6b6eecb584699b9d93b5ad29ad5ad
