Summary: This seems a little more user-friendly.
Reviewed By: bradenwatling
Differential Revision: D5686562
fbshipit-source-id: 8142fb9105a3a44823f935fc04187cf0ed2258d7
Summary:
Because `DBG2` seems to be the level we are using to log thrift calls in
`EdenServiceHandler`, this seems like a reasonable default.
Reviewed By: simpkins
Differential Revision: D5686115
fbshipit-source-id: 2d3e0173df37919b6936f73e641f880d16dc539f
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.
This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.
Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)
It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.
Reviewed By: simpkins
Differential Revision: D5686114
fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
Summary:
1.Modified `TreeInode::unloadChildrenNow()` to return number of inodes that have been unloaded.
2.Modified `EdenServiceHandler::unloadInodeForPath()` to return number of inodes that are unloaded.
Reviewed By: simpkins
Differential Revision: D5627539
fbshipit-source-id: 4cdb0433dced6bf101158b9e6f8c35de67d9abbe
Summary:
Added a test case `test_unload_free_inodes_age` to verify the behaviour of unloadChildrenNow with age parameter.
Added new parameter age to `unloadInodeForPath` in eden.thrift, and `EdenServiceHandler`.
Modified `do_unload_inodes` function in `debug.py` to support the new behaviour.
Reviewed By: simpkins
Differential Revision: D5565859
fbshipit-source-id: a35053725be26bc906cf158969cbe21db1cbadde
Summary:
When Hg tells the `dirstate` to `clear()`, we should also clear out any data we
have on the server for the `Dirstate`.
As it stands, the way we subclass `dirstate`, it does not appear like `clear()`
should be called, in practice, though one thing that could call it is
`hg debugrebuilddirstate`. It is probably good for us to have an RPC lying
around that we can use to reset the `Dirstate.`
Reviewed By: wez
Differential Revision: D5675298
fbshipit-source-id: 38926cfd93f4f83e4c28910f812a693cb32e423a
Summary: This will make subsequent changes to these files cleaner.
Reviewed By: wez
Differential Revision: D5675296
fbshipit-source-id: 06b14d55485415e3ec8a59a4bcc50e6189464b7d
Summary:
In `hg/eden/__init__.py`, we wrap `match()` in Mercurial's `match.py` in an
attempt to annotate every `basematcher` created in the system with a special
`_eden_match_info` property that we can use in `_eden_walk_helper()` to perform
walks more efficiently. Unfortunately, we missed a case where `scmutil.py`
has a `matchfiles()` function that calls `exact()` in `match.py` directly rather
than going through the generic `match()` function.
This was causing a failure when running `hg revert <filename>` in Eden because
the matcher that was created via `exact()` did not have an `_eden_match_info`.
This commit wraps `exact()` to add the property.
Reviewed By: wez
Differential Revision: D5674660
fbshipit-source-id: 16d1e7648ebd7a23b43b9b1200d3e284e5bc07b0
Summary: Provide a thrift interface to invalidate the cache for an inode denoted by path.
Reviewed By: simpkins
Differential Revision: D5655387
fbshipit-source-id: 887aa4963d216a0d8eed93b6fb8721632cc31d19
Summary:
Previously, we were overloading `hgSetDirstateTuple()` to also make it possible
to delete an entry from the internal `hgDirstateTuples` map. Now we have an
explicit method to do this, which enables us to remove some hacks/TODOs.
Reviewed By: simpkins
Differential Revision: D5665170
fbshipit-source-id: bc0753d4990c8966fd9e6c50b29a954d5023292f
Summary:
This is a convenient way to test the `hgGetDirstateTuple()` endpoint in
`eden.thrift`.
Reviewed By: quark-zju, wez
Differential Revision: D5654237
fbshipit-source-id: 0b285e056002d4556733a53293582345f36780b2
Summary:
This was helpful when debugging interactions between Mercurial's
dirstate and the related Thrift calls to Eden server.
Reviewed By: simpkins
Differential Revision: D5664872
fbshipit-source-id: 4e6ef3b034f4fc81f0d467974311a58f54b6e47b
Summary: Modified `TreeInode::unloadChildrenNow` such that inodes are unloaded whose age is greater than a specific age.
Reviewed By: simpkins
Differential Revision: D5526137
fbshipit-source-id: 91e2364d55e31befedcf43d98c26467e1a472ef9
Summary:
Update the EdenServer code to correctly aggregate the thread local stats data
from all threads. Previously it only aggregated stats from the
FunctionScheduler thread that it was running in. This thread never updates any
stats, so it never had any actual stats data.
This also adds a thrift call to trigger stats flushing immediately. This can
be called from integration tests to ensure that they get up-to-date stats
information.
Reviewed By: bolinfest
Differential Revision: D5657267
fbshipit-source-id: 060a24c00a19568b09ae8795477d73a3baab9a3c
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary:
Change the default value for --use_hg_tree_manifest to true, to re-enable
treemanifest import by default. This should work much better now that we are
using the latest treemanifest code and now support fetching treemanifest data
from the remote server when it is not found locally.
Reviewed By: bolinfest
Differential Revision: D5647204
fbshipit-source-id: 424baf4de1d3b247c1e04a838040baeb5976404b
Summary:
This refactors the treemanifest union store initialization code, and fixes a
crash introduced in D5560787 if --use_hg_tree_manifest is enabled but the
underlying repository does not support treemanifest.
This moves the treemanifest initialization code into a new importTreeManifest()
helper function, and separates it from parsing the CMD_STARTED response. We
now only initialize unionStore_ if FLAGS_use_hg_tree_manifest is enabled and
the repository supports treemanifest. Other places in the code now only check
if unionStore_ is non-null to see if they should try importing via
treemanifest.
Splitting importTreeManifest() from the CMD_STARTED response parsing will
potentially make things cleaner in the future for using multiple
hg_import_helper.py processes. Even if we start multiple of these processes,
we only need one instance of the C++ union store, and we don't need to
re-initialize the treemanifest separately each time.
Reviewed By: bolinfest
Differential Revision: D5647203
fbshipit-source-id: 71e14156f1d0ebc0880dd819c5b041b7c1146818
Summary:
A removal followed by a create should be treated as a change rather than a
no-op. Previously the code was merging this into a no-op, and was also
ignoring any subsequent modifications in the same journal period being queried
(since it processes transactions in reverse order).
Reviewed By: bolinfest
Differential Revision: D5647669
fbshipit-source-id: 29389206f0e818d6b6248fbb6697b85e93064b1f
Summary:
Recently, autodeps switched to requiring simple types for the various dep
parameters in order to support proper annotation parsing (D5387184), which
started breaking on `eden/fs/fuse/TARGETS`.
Reviewed By: simpkins
Differential Revision: D5620249
fbshipit-source-id: 16fa2c73421ff7e9929c71290a662babde1289ec
Summary: Add a command to lookup the path for an inode given the inode number and eden mount path.
Reviewed By: bolinfest
Differential Revision: D5627411
fbshipit-source-id: 25928f506d3f48d8a6784fe81fb17fa0500d6bc9
Summary:
Update the hg import tester utility to recursively import all trees under the
specified path by default. Passing in `--tree_import_recurse=no` can be used
to disable this behavior.
Reviewed By: wez
Differential Revision: D5588902
fbshipit-source-id: 1426dd18e0fb86728429f203b5be4742998886ed
Summary:
Add a flag to prevent HgImporter from trying to fetch missing tree data from
the remote mercurial server.
This is mainly useful for debugging purposes to force operations to fail if we
are missing some tree data locally.
Reviewed By: wez
Differential Revision: D5588900
fbshipit-source-id: b9f688383a0c1fceab614341473b357cda31a88b
Summary:
The `entry->flag` field in a ManifestEntry object is a `char*`, but it is not
actually a nul-terminated string. When it is non-null, only the first
character should be used. This updates the eden logging code to log just this
one character, instead of trying to log it like a normal `char*` and reading
garbage after the flag character.
Reviewed By: wez
Differential Revision: D5588899
fbshipit-source-id: 63b727a3aaee838f3a0cf6d765140964dd3ea746
Summary:
Summary
This removes the stale copy of the fb-hgext C/C++ code at eden/hg/datastorage.
The eden include paths have been updated to find the newer includes from
scm/hgext.
This will remove the treemanifest code from the eden-hg repository. This code
is available in the fb-hgext bitbucket repository instead. However, I will
likely send out a diff to `#ifdef` out the treemanifest integration from eden
in open source builds. This will ensure that open source eden builds can
build without using the treemanifest code.
Reviewed By: wez
Differential Revision: D5588675
fbshipit-source-id: ab3f26c2797e95baaedeb9ca73fee56ddb521d6a
Summary:
Updated time stamps of TreeInode accurately on mkdir,rmdir,mknode,symlink,create,unlink and readdir.
updated the `TreeInode::getattr` function to return in-memory timestamps.
Reviewed By: simpkins
Differential Revision: D5568183
fbshipit-source-id: c36f7fb767cd4342aab5cc983eea56e37cd2077e
Summary: Removed creation time from `FileInode::state` which was used for getting timestamps of files that are not materialized.Now that we added timestamps to file inodes and tree inodes we no longer need this.
Reviewed By: wez
Differential Revision: D5552761
fbshipit-source-id: 6013b1f694045e08ada7bd64114c4f2e52848fef
Summary:
updating atime,ctime,mtime of FileInode on read, write and setattr system calls.
modified `FileInode::stat` function to return accurate inmemory timestamps.
Reviewed By: simpkins
Differential Revision: D5552666
fbshipit-source-id: 86d446f72908663f8db509b7b789d9f35d17df3a
Summary:
Added `TreeInode::setInodeAttr` a helper function used in `InodeBase::setattr`. Also,added `InodeBase::setAtime` ,`InodeBase::setMtime` and implemented them in `FileInode` and `TreeInode`.
Moved updating timestamp logic to `InodeBase::setattr` from `FileInode::setInodeAttr` and `TreeInode::setInodeAttr`.
Reviewed By: simpkins
Differential Revision: D5545422
fbshipit-source-id: 597cfabb3062166a058cf32776acb50a1bc0c61c
Summary: Removed `FileInode::setattr` from `FileInode` and added a helper function `setInodeAttr` to perform FileInode or TreeInode specific setattr operations in `InodeBase::setattr`. This diff contains implementation of setattr for FileInode i.e for files, will add setattr implementation for directories in another diff.
Reviewed By: simpkins
Differential Revision: D5544968
fbshipit-source-id: 089491d07a603e111966987ef390b6e597aba28c
Summary: It wasn't worth putting these into separate diffs, so just do them all at once.
Reviewed By: yfeldblum
Differential Revision: D5619187
fbshipit-source-id: 7651c062601f4134fac4e4fea654e64d7d8cb8c8
Summary:
We're seeing that this is always set to true for eden,
which is causing buck to run slower than it should.
To make this work correctly, I've augmented our journal data structure
so that it can track create, change and remove events for the various
paths.
I've also plumbed rename events into the journal.
This requires a slightly more complex merge routine, so I've refactored the two
call sites that were merging in slightly different contexts so that they can
now share the same guts of the merge routine. Perhaps slightly
counterintuitive in the merge code is that we merge a record from the past into
the state for now and this is a bit backwards compared to how people think.
I've expanded the eden integration test to check that we don't mix up
create/change/removes for the same path in a given window.
On the watchman side, we use the presence of the filename in the createdPaths
set as a hint that the file is new. In that case we will set the watchman
`ctime` (which is not the state ctime but is really the *created clock time*)
to match the current journal position if the file is new, or leave it set
to 0 if the file is not known to be new. This will cause the `is_new`
flag to be set appropriately by the code in `watchman/query/eval.cpp`;
if the sequence is 0 then it should never be set to true. Otherwise (when
the file was in the `createPaths` set) it will be set to the current journal
position and this will be seen as newer than the `since` constraint on
the query and cause the file to show as `new`.
Reviewed By: bolinfest
Differential Revision: D5608538
fbshipit-source-id: 8d78f7da05e5e53110108aca220c3a97794f8cc2
Summary:
1. Added a new structure `InodeBase::InodeTimestamps` to wrap atime,ctime,mtime together. This new structure helps in avoiding usage of `struct stat` for timestamps.
2. Modified function `Overlay::openFile` ,`Overlay::updateTimestampToHeader`, `Overlay::deserializeOverlayDir`, `Overlay::parseHeader` to use this new structure for timestamps instead of `struct stat`. Also, modified code in places where this change is being affected.
3. Added new helper methods `FileInode::setattrTimes` and `TreeInode::setattrTimes` to set timestamps in FileInode and TreeInode during setattr. Implementation of setattr for FileInode and TreeInode is in the diffs stacked above this diff.
4. Replaced atime, ctime, mtime in `FileInode::State`, `TreeInode::Dir` to `FileInode::State::timeStamps` and `TreeInode::State::timeStamps`. Made other necessary changes to support this change.
Reviewed By: simpkins
Differential Revision: D5596854
fbshipit-source-id: 2786b7b695508a62fdf8f7829f1ce76054b61c52
Summary:
I'm trying to troubleshoot why a new dir that I added isn't showing up in the overlay and I found that the debug tool doesn't know about the overlay.
This is the dumbest minimal thing I could do to make this basically functional again
Reviewed By: simpkins
Differential Revision: D5602439
fbshipit-source-id: 4ae7fa34136697f9f915ccd95275cb3a7923ae4a
Summary: Currently we have two functions `FileInode::setattr` and `FileInode::setAttr` which are used to set given attributes to a `FileInode`. Merged both the functions in to one function called `FileInode::setattr` and removed `FileInode::setAttr`.
Reviewed By: wez
Differential Revision: D5538490
fbshipit-source-id: ec241fad25d6e4694865e5fc3c0a3500e4838bdd
Summary:
Added a new function `InodeBase::updateOverlayHeader` and implemented `FileInode::updateOverlayHeader` and `TreeInode::updateOverlayHeader` to update inmemory timestamps to overlay header when an inode is unreferenced.
Added helper functions in `Overlay` class to read and update timestamps in to the overlay file. Also,modified `Overlay::loadOverlayDir` to read and populate timestamps from overlay header in to treeinode.
Modified constructor of `FileInode::state` to read timestamps from overlay file and to populate inode timestamps.
Added test case to check if time stamps are updated and read correctly on remount.
Fixed a lint warning in TARGETS file
Reviewed By: simpkins
Differential Revision: D5535429
fbshipit-source-id: f6b758f70101c65d316a35101aacc9a3363f7aed
Summary:
Update the hg_import_helper.py script to include additional information in the
initial CMD_STARTED response that it sends after it has successfully
initialized.
This response now includes:
- A protocol version number, to help catch errors if edenfs somehow ends up
invoking an incompatible version of the hg_import_helper.py script.
- Whether treemanifest is supported in this repository.
- If treemanifest is supported, the list of treemanifest pack directories.
This eliminates the need for the separate GET_CACHE_PATH command that was
previously sent immediately after the CMD_STARTED response.
This also fixes the code to always let the python code select the pack
directories. Previously the C++ code assumed the local pack directory was
aways located at ".hg/store/packs/manifest" under the repoPath. This may not
be correct if repoPath happens to be a repository created with the share
extension.
Reviewed By: wez
Differential Revision: D5560787
fbshipit-source-id: e796222f42927a85e886227b3a7b2ccf9c1ef1bd
Summary:
This is a first pass at updating eden to fetch tree manifest data from a remote
mercurial server on demand. The code now catches MissingKeyErrors thrown by
the datastore code, and then fetches the requested tree data from the remote
server.
This makes it safe to use tree manifest for import, as we can now fetch missing
tree data as needed, instead of always returning I/O errors whenever we reach a
tree not available locally.
Unfortunately, the performance of downloading tree data is currently not very
good: it takes 90+ seconds plus to download any tree data. The current
mercurial server code appears to always provide full recursive tree data
(unless you ask for the delta between two manifests, which we do not have).
Even when asking for very small subdirectories, the server appears to send the
full tree manifest data for the entire repository, sometimes taking longer than
if we had asked for the full repository data to start with.
Note that tree manifest import is still disabled by default with this diff, and
must be explicitly enabled by running edenfs with `--use_hg_tree_manifest`.
Reviewed By: wez
Differential Revision: D5544817
fbshipit-source-id: 940e0c914f055edc1beee438b0ac50c2f8b08b03
Summary:
Add logic to the hg import tester tool to support recursively importing
specific subdirectories using the tree manifest import mechanism.
Reviewed By: wez
Differential Revision: D5544818
fbshipit-source-id: 1637d32691c30dfab8d59599891feab6fb27bcdb
Summary:
This makes several improvements to the hg import tester script:
- If no --edenDir flag is specified, initialize a new temporary directory to
keep the RocksDB data store.
- Add a `--rocksdb_options_file` flag to allow controlling the options used for
the RocksDB store.
- Add an --import_type flag to allow explicitly selecting if we should test
the flat manifest or tree manifest import code.
- Add a --flat_import_file flag, to allow testing a pre-generated flat manifest
input data file, rather than retrieving the data from mercurial. This allows
benchmarking only the C++ import code, and eliminating the python portion of
the import. The input file can be generated by running
`hg_import_helper.py --manifest <revision>`
Reviewed By: wez
Differential Revision: D5541732
fbshipit-source-id: 340af4fea872412248d41453792b2179f0afa466
Summary:
Update EdenServer to use folly::Synchronized for managing access to the
mountPoints_ map, instead of using a separate standalone mutex. This simply
makes it slightly harder to accidentally access the mount point map without
holding the lock correctly.
Reviewed By: bolinfest, wez
Differential Revision: D5541315
fbshipit-source-id: 094a941a3b8177aed4a75b91a998494f4c33fe8c
Summary:
Update `hg_import_helper.py` to take `--config` options to set mercurial
options, just like the normal `hg` command.
This makes it easier to override mercurial settings during testing. For
instance, this lets us override the `remotefilelog.cachepath` option for
testing tree manifest import without hitting the real system hgcache
directory.
Reviewed By: bolinfest
Differential Revision: D5523181
fbshipit-source-id: 8bb6f5f244fa979872275cbb6ee797d1477a9b5b
Summary:
This fixes EdenServer::unmount() to actually wait for all EdenMount cleanup
to complete, and fixes unmountAll() to return a Future that correctly waits for
all mount points to be cleaned up.
Previously `unmount()` waited for the mount point to be unmounted from the
kernel, but did not wait for EdenMount shutdown to complete. Previously
EdenMount shutdown was not triggered until the last reference to the
shared_ptr<EdenMount> was released. This often happened in the FUSE channel
thread that triggered the mountFinished() call--it would still hold a
reference to this pointer, and would not release it until after
mountFinished() returns. As a result, when the main thread was shutting down,
`main()` would call `unmountAll()`, and then return soon after it completed.
Some FUSE channel threads may still be running at this point, still performing
`EdenMount` shutdown while the main thread was exiting. This could result in
crashes and deadlocks as shutdown tried to access objects already destroyed by
the main thread.
With this change `EdenMount::shutdown()` is triggered explicitly during
`mountFinished()`, and `unmount()` will not complete until this finishes.
The `EdenMount` object may still exist at this point, and could still be
deleted by the FUSE channel thread, but the deletion now only requires freeing
the memory and does not require accessing other data that may have been cleaned
up by the main thread.
We should still clean up the FUSE channel thread handling in the future, to
make sure these threads are joined before the main thread exits. However, that
cleanup can wait until a separate diff. Ideally I would like to move more of
the mount and unmount logic from EdenServer and EdenServiceHandler and put that
code in EdenMount instead.
Reviewed By: bolinfest
Differential Revision: D5541318
fbshipit-source-id: 470332478357a85c314bc40458373cb0f827f62b
Summary:
When cleaning up an unmounted EdenMount, cancel the periodic unload inodes
function before we remove the mount from the mountPoints_ map and signal other
threads that it has been removed via mountPointsCV_. Also use
FunctionScheduler::cancelAndWait() to make sure that we wait for the function
to complete if it was currently in the middle of being run.
Reviewed By: bolinfest
Differential Revision: D5541316
fbshipit-source-id: 46046f59746b5f283a5d4af94159b5c122d43d0d
Summary:
Make all the changes in thrift to enable the use of mstch-cpp2.
For now, if any cpp2 flag is present, it will fallback to the python genenerator (i.e. t_cpp_generator.py)
Reviewed By: yfeldblum
Differential Revision: D5464216
fbshipit-source-id: 287618fbf3b2dcc3142e305e648e48b1ebb762c1
Summary: Changed the initialization of last checkout time in Tree inode constructor by grabbing a lock on `TreeInode::contents_`
Reviewed By: simpkins
Differential Revision: D5524857
fbshipit-source-id: e86ee7d986ca0c280ba156ba9146d6d1f9fa722e
Summary:
Added `EdenServer::unloadInodesScheduler_` to schedule unloading inodes on timely basis.
new function will be added to the function scheduler on each mount, so each mount point can be have different start delay and different frequency.
Reviewed By: simpkins
Differential Revision: D5501036
fbshipit-source-id: 238b9cb5747a2358c65a1508095ad672bdf87ffc
Summary:
Some test cases are crashing once in a whiled due to a race condition on lock over overlay directory.
when a mount point is unmounted, `EdenServer::mountFinished` called on `umount` where destruction of `EdenMount` object for the mount point happens. When a remount is called before the destruction of `EdenMount` a race condition is occuring on the lock of overlay directory as the undestructed `EdenMount` still holds the lock on overlay and the newly created `EdenMount` tries to grab a lock on overly.
Made the `EdenServer::unmount` asynchronous which waits for a promise that is set in `EdenServer::mountFinished`.
Reviewed By: simpkins
Differential Revision: D5515740
fbshipit-source-id: 5cbfddf13fe00c2cb1b63a6be6353e9b84fbd569
Summary:
In testing, we discovered a case where concurrent Hg operations in EdenFS would
//sometimes// fail with `ECONNREFUSED` when trying to read
`<eden-mount>/.eden/socket`.
This was very confusing, as the standard reasons for `ECONNREFUSED` did not seem
to apply:
- We verified that Eden had not crashed.
- We verified that Eden's UNIX domain socket had a sufficiently large backlog
(1024) such that we should not be at risk of exhausting it with two simultaneous
Hg commands.
It turned out that there was a race condition in our `readlink()` command, which
could cause `<eden-mount>/.eden/socket` to resolve to the wrong path. Failing to
connect to this path manifested itself as an `ECONNREFUSED` error.
It turned out that `readlink()` used `FileInode::readAll()`, which was
performing an `lseek()` followed by a `read()` on a file descriptor while the
descriptor was protected by a //read lock// instead of a //write lock//. Because
the `lseek()` causes a state change, it needs to be protected by a write lock.
Changing the type of the lock fixed the issue.
The only other caller of `FileInode::readAll()` appears to be in
`TreeInode::loadGitIgnoreThenDiff()`, so presumably this fixes a possible race
condition there, as well.
Reviewed By: simpkins
Differential Revision: D5533001
fbshipit-source-id: 86cf84c45463b2ae194d6f46909ea67c0f71d065
