Summary:
Remove some more of the double-checking code left over from debugging
T24406969. We fixed the root cause of that task and haven't seen any errors
reported by this code. The higher log levels used for reporting information
about empty files just causes extra noise in the logs.
Reviewed By: strager
Differential Revision: D13403531
fbshipit-source-id: bcb7b29a1d687ffcefd6ecf586130b96f87b4820
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.
Reviewed By: strager
Differential Revision: D13325662
fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.
FileInode's changes are coming in D13325079.
Reviewed By: strager
Differential Revision: D13324995
fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
Summary:
Add a fuzz test that randomly modifies a directory during a series of
readdir calls and verifies each unmodified entry is returned once.
Reviewed By: simpkins
Differential Revision: D13371162
fbshipit-source-id: 252b03ab0288e82b56a33c347955de129e61ae42
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
There was a bug in BlobCache where, if you had an interest handle to a
blob, but that blob was evicted anyway and then something else caused
it to be reloaded, dropping your interest handle would cause the blob
to be incorrectly evicted since the reference counts were no longer
compatible. Add a version to cache items and only decrement the
reference count on an item if the interest handle and item agree.
Reviewed By: strager
Differential Revision: D13405144
fbshipit-source-id: aee052bf777e7225551c3ae2b8b69a99f4f77691
Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.
Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.
Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.
Reviewed By: simpkins
Differential Revision: D13386450
fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
Summary:
As I'm working on ripping out file handle support, I wanted to ensure
that we had a path towards a correct and stateless readdir()
implementation. Stateful file handles require extra care during
graceful restart, and it's nice that we can avoid them. In fact, this
work paves the way towards a possible FUSE_NO_OPENDIR_SUPPORT feature.
This diff fixes concurrent rm -rf in the same directory.
Reviewed By: simpkins
Differential Revision: D13370898
fbshipit-source-id: eba650e673a7cb9559e04ba28417980f6d0c65cb
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.
Reviewed By: strager
Differential Revision: D13349220
fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
I want to add some non-trivial systemd-specific logic to fb-edenfs@.service's ExecStart command. Refactor the existing command to use sh, giving us access to `if` and other useful utilities.
Also, the shell language is likely more familiar (e.g. `${foo}` and `$foo` behave the same in sh, but behave differently in systemd), so using sh should lead to fewer headaches.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13165377
fbshipit-source-id: 2e500abf75c6f79a8b174a848f7eeb1cfaebb96c
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
Frequently, non-eden-owner-owned processes show up as `err:13` in
the `eden top` output which is a bit of a PITA. This diff pulls the data
from `/proc/PID/cmdline` instead of `/proc/PID/exe`; the former is world
readable always while the latter is restricted to process owner readable.
Reviewed By: chadaustin
Differential Revision: D13342789
fbshipit-source-id: d4e395b318107e873189a1e2039329015c4c38f8
Summary: Add a CoverageSet type that tracks non-overlapping intervals so FileInode will be able to tell when the entire blob has been read by FUSE.
Reviewed By: strager
Differential Revision: D12947146
fbshipit-source-id: fbcfd9c19b09d4a7b5364671dcdbd39b53e6d186
Summary:
Write tests for readdir's semantics (we really do want to return . and
..) and simplify the logic. It's still quadratic in large directories,
but there aren't any allocations anymore.
Reviewed By: strager
Differential Revision: D13287764
fbshipit-source-id: 5e0d4b86eb16dbd7a16cdeb324e4b43363512e25
Summary: This diff also includes build script changes to EdenDispatcher, FsChannel and EdenMount.
Reviewed By: strager
Differential Revision: D13091786
fbshipit-source-id: cecc8d849fcb9ebc8fa718e1011ef8931bebc279
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary: No need for String to import fbvector.
Reviewed By: yfeldblum
Differential Revision: D13007715
fbshipit-source-id: c61639a04273f14dd3daf230ee6ed9ade93d058e
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Change the Windows pipe read and write functions to read/write in a loop. Plus changed the functions prototype to match the POSIX version.
Reviewed By: strager
Differential Revision: D13091785
fbshipit-source-id: 375b22bd9e62d371a78d410f42068945b966a743
Summary:
Backing store works with eden strings(UTF8 + Unix path separator). The path strings we receive on Windows from FS and cli are Windows paths
(Win path separator and/or UTF16). Adding the functions to convert one to another.
Reviewed By: strager
Differential Revision: D13091788
fbshipit-source-id: f7fc8a79e360e964cf4619dfa540b57f1f18d283
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary: Create a platform independent function to compare the stats and use that to check if the file contents have changed.
Reviewed By: strager
Differential Revision: D13046269
fbshipit-source-id: c4f5bc88cec3df5cb6555d13cea2f23fd4eeb7ce
Summary: Add type annotations to all functions in eden/fs/service/client.py
Reviewed By: strager
Differential Revision: D13051091
fbshipit-source-id: 23b93008352664336ad155a7f5cc281bd5529702
Summary:
Now that we have a standardized StartingGate for benchmarks, use it
everywhere we used to use folly::Baton.
(Note: this ignores all push blocking failures!)
Differential Revision: D13012244
fbshipit-source-id: 5841ab74cfa408e87d021fe5591557e79e677e5c
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary: Update the CLI code to make the pyre type checker happy.
Reviewed By: wez
Differential Revision: D13035413
fbshipit-source-id: d201f2e65667e0ce1bf4a73fbb05878e8711ad16
Summary:
chadaustin noticed this as part of fixing up the ESTALE
handling. The issue is that we were using `inodeMap->lookupInode` and
assuming that it will always return one of our magical inodes. This
isn't guaranteed so it is better to resolve the inode by name from
the root, so that's what this diff does.
Reviewed By: chadaustin
Differential Revision: D12970034
fbshipit-source-id: 8207660cbc71577b276cb092d1ef19e1076b4946
Summary:
yfeldblum were talking about whether this code might make sense in
folly. That led to polishing it a bit more. The hot path is only six
instructions now. It's not any faster in a tight generateUniqueID loop
but uses only one thread_local slot and a bit less global storage.
Reviewed By: strager
Differential Revision: D12927275
fbshipit-source-id: 94a5872c61dfe9dd441f1f34fab65f93c12997d8
Summary:
In a later diff, I needed generateUniqueID to be
noexcept. folly::ThreadLocal does not guarantee that (and it allocates
the first time a thread calls get()), so use C++ thread_local
instead. Bonus: it's about half a nanosecond faster now.
Reviewed By: strager
Differential Revision: D12914625
fbshipit-source-id: 9ddbe65d0ba1d317907f821c03dea5a207a73a68
Summary:
Update the HgUI object used by hg_import_helper to always return true from the
`plain()` function, regardless of whether HGPLAIN is set in the environment or
not.
This should help ensure that this script is never affected by user-defined
configuration settings. This should also help ensure that mercurial won't try
to print progress bars or other strange things.
Reviewed By: quark-zju
Differential Revision: D13008639
fbshipit-source-id: afe581958470c4c4b89825a259c460ece4e20fe7
Summary:
It is desirable to be able to reference the same variable
multiple times in the RHS of a config setting. This diff makes that
possible.
Reviewed By: strager
Differential Revision: D12906500
fbshipit-source-id: 4277f12105d0a0fb3dca880d3dad6b0238acedc0
Summary:
With the mononoke service being configured with proxygen
in front of the servers we need to ensure that we're correctly
setting up our http request if we want them not to fail with a
400 bad request error.
This diff allows setting mononoke.hostname to a DNS resolvable
name rather than an IP address and passing that hostname through
so that we can set it in the HTTP request.
Reviewed By: strager
Differential Revision: D12906498
fbshipit-source-id: b5aaabfd6f2f4c48d45128eaad8406e648477f75
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
Add new helper files for creating temporary files and directories.
The main advantage of these functions is that they prefer creating files in
`/dev/shm` by default instead of `/tmp`. Some of the eden unit tests are
fairly I/O intensive (particularly the checkout tests, which create and
destroy many test mounts). This can be quite slow on hosts where `/tmp` is a
a spinning hard disk. Putting the temporary files in a ramdisk greatly speeds
up the tests in this case.
These test functions also default to using the prefix `eden_test.` for the
temporary file names.
This does not yet change any of the test code to use these functions. I will
do that in a subsequent diff.
Reviewed By: chadaustin, strager
Differential Revision: D12971161
fbshipit-source-id: 3f74be7a467e8080185d4d97d114288b4528755a
Summary:
This function behaves similarly to the python `os.path.expanduser`
function, except that it is restricted to expanding only information for the
current user.
Use this new function to process the hgcache path returned via the
rust config parsing code.
In the longer term I want to centralize and add accessors for
the rust config and move this stuff into EdenConfig, but for
now this is the one place that does this and it is OK to
process it this way here.
Reviewed By: strager
Differential Revision: D12988902
fbshipit-source-id: 96b10640359c3b8c0adac1e14cd42dd592023c3d
Summary:
This makes it possible to change configuration options
for the LocalStore while the server is running.
As you'll see in the next diff, our current layering makes using
the config a bit more awkward, but at least this diff doesn't
look gross :-p
This diff doesn't introduce any new functionality or configuration.
Reviewed By: strager
Differential Revision: D12949577
fbshipit-source-id: cf897ba676b9359f92865170faa42ff17329b85f
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary: folly/Subprocess is not compatible with Windows and has broken the Windows build. It's not used so removing it.
Reviewed By: wez
Differential Revision: D12967451
fbshipit-source-id: 54d33bf6fe2ec3ede9d68eccd99e53c5eb6ed53d
Summary:
In a later diff, I needed to make generateUniqueID() noexcept. This
made me think that C++'s thread_local might be a little faster than
folly::ThreadLocal. So I added this benchmark.
Reviewed By: yfeldblum
Differential Revision: D12914249
fbshipit-source-id: 2c4acfff2162f66d13f456439d91df2ecb4167e3
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
I've included a missing `.via` for the mononoke tree import path. It's hard to
measure the difference here, but it is more technically correct than letting
that second stage run in an event base thread. desired.
Reviewed By: chadaustin
Differential Revision: D12896023
fbshipit-source-id: 0af0f7f94c3274a3e4b570fed0af4d4d3c783bc3
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
Eden used to load materialized entries at startup. It no longer does
and this code is dead.
Reviewed By: wez
Differential Revision: D12896210
fbshipit-source-id: 398363724a661f87208cf05313e61755a451edb7
Summary:
With this the eden cli can dump tracepoints and translate
them to various formats or perform any processing
Reviewed By: chadaustin
Differential Revision: D10384072
fbshipit-source-id: 8b38e7f6b551a2bd98b3e748ba1cceafeceeec8c
Summary:
This uses the tracing infrastructure to create blocks around
the getSHA1 invocation as well as the getOrLoadChild (called for each
element of the path-walk) method.
Reviewed By: chadaustin
Differential Revision: D10384074
fbshipit-source-id: a06fbe38e8d3f35fcb248e6bd724e5572724d27d
Summary:
In order to get better insights into where time is spent on
various requests, some tracing infrastructure is helpful. This will
record tracepoints into a per-thread ring-buffer which can then be
dumped out-of-band.
The trace format is quite simple, a trace is composed of blocks with
begin and end points and each block may have a parent block. We can
then translate this format to many other formats as we desire.
Reviewed By: chadaustin, strager
Differential Revision: D10384071
fbshipit-source-id: f9e0f11521c68b96ab40d517c7a83cf89375b101
Summary:
This diff implements getBlob on top of the mercurial rust
datapack code. It adds a C++ binding on top of the rust code to
make it easier to use and hooks it up in the hg backing store.
Need to figure this out for our opensource and windows builds:
* Need to teach them how to build and link the rust code
* need to add a windows version of the methods that accept paths;
this is just a matter of adding a WCHAR version of the functions.
Reviewed By: strager
Differential Revision: D10433450
fbshipit-source-id: 45ce34fb9c383ea6018a0ca858581e0fe11ef3b5
Summary:
Nothing populates these fields yet today, but will in a later diff.
If present in the json data for an entry, populate them in our TreeEntry.
When processing trees from mononoke, check for these fields being set
and emit them into the current writeBatch so that we set the BlobMetadata.
Reviewed By: chadaustin
Differential Revision: D12814793
fbshipit-source-id: cd19d3f553c22462adc58c024a90cfeb5b8da224
Summary:
We'll start populating these in the mononoke client.
They are currently unused.
Reviewed By: chadaustin
Differential Revision: D12814791
fbshipit-source-id: f5407de5cdb9f1f3ad6ee2befed50e2a7562ec97
Summary:
I want to reuse this outside the LocalStore implementation
in a later diff. This has no functional changes.
Reviewed By: chadaustin
Differential Revision: D12814792
fbshipit-source-id: 2cc34b449a93afb2e71bd49ed6587009ebf3e419
Summary:
While this doesn't deliver any gains immediately, it is desirable
because the shortest path to getting trees with pre-populated size and
content sha1 information is via the mononoke api server fetch.
If we use the local trees we will populate the basic metadata slightly
faster but will then pay more to fetch and hash the data.
The calculus is a little more complex really, as having the contents of
a dir locally in a local datapack is probably faster, but we don't have
a good way to know if that is the case right now.
Reviewed By: strager
Differential Revision: D12813490
fbshipit-source-id: 545b90bf20940dab193bef8120712dd9227033e1
Summary:
This partially wires up handling mononoke related config
changes if they are made after starting the server. We don't currently
have logic to enable mononoke after starting the server, but can disable/enable
it if it was configured at process start.
Reviewed By: strager
Differential Revision: D12813346
fbshipit-source-id: ee29e91fe714a65a3b61c4e23929b67d6b7cb521
Summary:
To facilitate testing against a test instance rather than selecting
from a tier, add a config option to return the host override.
The semantics are that a hostname string will override the tier name setting.
An empty host name string is treated as missing and will not replace the
tier name.
Reviewed By: chadaustin
Differential Revision: D12813050
fbshipit-source-id: 67773d772d78d3f7882825ce54f4313c97ab0a33
Summary:
Remove the handful of (but growing) parameters that
are threaded down to the HgBackingStore in favor of the ReloadableConfig
object and use that to configure mononoke support.
This doesn't notice changes to the config on the fly today, but
does simplify the amount of stuff we need to pass around.
Reviewed By: chadaustin
Differential Revision: D12813053
fbshipit-source-id: b8309328fab4980889624257cfd387c8c1335001
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary: This option was added upstream and to our internal 4.16 kernel series
Reviewed By: chadaustin
Differential Revision: D12823476
fbshipit-source-id: 54d241b77eff92f4083c0f1d4c98b47a6a098e3f
Summary:
Checkout.modifyFile is the long pole when running the Eden unit
tests. Splitting it into three runs it across multiple threads. It
could be split further if needed.
Reviewed By: strager
Differential Revision: D12813684
fbshipit-source-id: 49e847d46bc54f16fcb2b273ec81e92eaf756dbb
Summary: If an error happens when opening the log file, edenfs should stop immediately and report the error. Write a test to verify that this happens with DaemonStartupLogger.
Reviewed By: chadaustin
Differential Revision: D10436926
fbshipit-source-id: 3f33ed276a668b7ecda8176a55b114b5878304e3
Summary:
When running edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdin from the parent and never closes it. This has two consequences:
* The daemon process holds a reference to stdin even after the parent exits. This causes tests to hang if they wait for EPIPE on stdin after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* edenfs could potentially read from the terminal after 'eden start' has returned. I think this behavior is surprising and could potentially cause problems.
Promptly close stdin after forking the daemon process, since nothing inside the daemon needs stdin.
Reviewed By: simpkins
Differential Revision: D10442346
fbshipit-source-id: f3d287e8b48e121807f845b73a1a550e63efae25
Summary:
getLogPath in eden/fs/service/main.cpp ensures that a non-empty log path is given to DaemonStartupLogger::daemonize. An empty log path is not possible. DaemonStartupLogger behaves differently with an empty log path, but that behavior has caused problems (see D10439299).
Make DaemonStartupLogger require a (non-empty) log path, simplifying its implementation.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10227597
fbshipit-source-id: 37931528add51ab3e6e1b941478000297cb8b693
Summary:
There may be a metadata header in the contents and we currently
do not have any code to process this, so we want to disable this.
Add a constructor boolean to disable it and default to off while we
work up a proper solution to this.
Reviewed By: chadaustin
Differential Revision: D10523393
fbshipit-source-id: ec96b63b6897841ee5a6b848834b5a0ada145223
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
D9029272 (github commit fae2056037) accidentally broke the CMake-based
build of Eden when SELinux is available: it defined `PRIVHELPER_LIBS` but
later used `PRIVHELPER_LIBRARIES` (LIBS vs LIBRARIES).
This fixes that error by simply removing this intermediate variable. We
can just use SELINUX_INCLUDE_DIR and SELINUX_LIBRARIES directly, as they
will simply be empty if SELinux is not available.
Reviewed By: quark-zju
Differential Revision: D10503905
fbshipit-source-id: 1c12bb1cad0351e4e0a77d0c7e8a83086209efee
Summary:
Our CMake-based build had gotten somewhat out of date. This fixes several
issues that were preventing it from building:
- wangle now depends on fizz, which in turn depends on libsodium
- fbthrift now depends on yarpl for the streaming functionality
- Eden now depends on libsqlite3
- The eden/fs/inode library now depends on eden/fs/config
Reviewed By: strager
Differential Revision: D10436531
fbshipit-source-id: f41065da2edb909e8eb94e03d0e62ae19ad3a84f
Summary:
Add a very cheap unique ID allocator designed for extremely
low-latency uses like D10384071.
Differential Revision: D10501712
fbshipit-source-id: 15a0b7a2d344c0f6100082850dd967c585f3e2da
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
Noticed that the filenames in the logging in the error
case were sometimes garbled; capture a copy of the filename
rather than a reference to it.
Reviewed By: bolinfest
Differential Revision: D10473798
fbshipit-source-id: ee218a2ad9d6edfad6ba7246107c152e79a9efd8
