Summary:
I am planning to start migrating Eden's CLI to the new Python 3 Thrift
implementation. In preparation, slightly clean up the interface and
implementation of our Python 2 Thrift wrapper.
Reviewed By: genevievehelsel
Differential Revision: D21854539
fbshipit-source-id: d398dd3f324c12288871cf0c9db41e64ed4cf7ed
Summary:
The CreateProcess API allows inheritable handles to be available in the spawned
process by passing a flag to it. What the documentation for this API leave to
the imagination is what happens when processes are spawned concurrently in
multiple threads and handles are opened and made inheritable while doing this.
The answer is obvious but the consequences aren't: they are inherited.
When anonymous Pipes are used, one end needs to be inheritable for the spawned
process to being able use it, but if one is created concurrently with spawning
a process, that other process may have an open handle to that unrelated pipe.
And since to detect that a pipe is closed, all handles to the other end needs
to be closed, this can lead to never being able to detect that it is closed...
This scenario is exactly what is happening in eden when spawning the Mercurial
process, and when one of these processes would die, eden would just hang trying
to write to the pipe, not knowing that the process was already gone. To unblock
eden, all the hg.real processes had to be killed, as this would close all the
pipe handles, and then eden would detect that the pipe was closed and re-spawn
Mercurial (only for the exact same thing to happen).
In order to fix this, we need to tell CreateProcess to only inherit the pipes
handles, and nothing else.
Reviewed By: wez
Differential Revision: D21895537
fbshipit-source-id: 3c84a1d0316b50b5750f554fa20f72f59a718882
Summary:
The old `py` Thrift language support doesn't correctly handle string
vs. bytes, which causes an exception to be thrown when deserializing
paths or blobs that aren't UTF-8.
We will eventually want to migrate to the py3 language implementation,
which supports streaming.
Reviewed By: genevievehelsel
Differential Revision: D21693082
fbshipit-source-id: 0ea10fd3960f5acba353bccb83b5cf539e7eeffb
Summary:
In order to start EdenFS automatically at boot, a template service was used
previously, but due to several issues, we decided to move away from it.
Thankfully microsoft supports several other ways of starting tasks at startup,
one of which is the "Task Scheduler" itself.
One of the weird part of the task scheduler is that there isn't a good way
to tell it to not show a console for a non-graphical application, and thus
plainly executing edenfsctl start in it would create a cmd window, which
would then disappear a couple of seconds later. To avoid this, a "graphical"
version of Python is used (pythonw.exe) to start edenfsctl.
Reviewed By: fanzeyi
Differential Revision: D21732281
fbshipit-source-id: 87ef3a2d5569302392bd30a4b9e7fc48807ee316
Summary:
When I was debugging the multithreaded bug issue it came up that the issue would
only affect certain versions of macos. There may be other bugs that come up
affecting only certain os versions. Having the os version in eden rage could be
helpful to identify such issues.
Reviewed By: chadaustin
Differential Revision: D21776154
fbshipit-source-id: a493e7da1823075ca4a845bd73b21716ce884911
Summary: The thrift team is looking to retire `stopWorkers()` and `stopListening()`, so this refactors our code to stop using both of those methods.
Differential Revision: D21851442
fbshipit-source-id: 6d63d435cbd4a51c855bf71a3b2165e35431a4d7
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
Somehow, on Windows, the socket file that was removed during `prepare` may be
back, but not bound, preventing Thrift from binding to it, let's remove it
again since it's not supposed to be there.
Note that this happens 100% of the time when starting EdenFS via the Task
Scheduler at log on, I couldn't reproduce when triggering the task manually,
or when starting edenfs by hand.
Reviewed By: simpkins
Differential Revision: D21755498
fbshipit-source-id: 32a343d9de531b90417dac0aa382a07c85cdee5a
Summary:
The CREATE_NO_WINDOW is documented as:
The process is a console application that is being run without a console
window. Therefore, the console handle for the application is not set.
The last part is what matters here, by default stdin, stdout and stderr are
inherited from the parent process, with this flag, they won't as these 3 will
simply not be set. Removing it allows Mercurial's output to be sent directly to
the edenfs log which will greatly improve our debuggability.
Reviewed By: akrieger
Differential Revision: D21820195
fbshipit-source-id: 63496a1ad28ecf5440c0846d80e1fbda5756b971
Summary: When calling `eden start` from the CLI layer, make sure to redirect stdin in order to daemonize.
Reviewed By: chadaustin
Differential Revision: D21675707
fbshipit-source-id: 26085cc2ff7774e86f03872030c8885bc3c3b949
Summary:
Currently when you run commands on eden on mac that both fetch data for commit
and update to that commit there is a multithreading bug. This could be a new bug
or an old bug that was not causing problems before, but is showing up a lot more
recently.
See http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
for a nice explanation of this.
The root cause of this is still unknown, but this flag should bring back the
old behavior where we were not seeing this bug.
Reviewed By: fanzeyi
Differential Revision: D21744987
fbshipit-source-id: 85092e32212e83b16bc00dc0188f03b643c48eea
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
Wanting to have EdenFS run as a service on Windows cause a couple of issues
on Windows:
- Needing to log out after installing
- Forcing Sandcastle to use --foreground, which means the edenfs.log is
empty, making it impossible to debug anything on Sandcastle,
- Services can only be started by an elevated user, while the rest of EdenFS
doesn't have this requirement,
- Uses undocumented template services
The next diffs in the stack will attempt to solve all of these by starting
EdenFS as a scheduled task instead, which should solve all the above.
Reviewed By: fanzeyi
Differential Revision: D21732280
fbshipit-source-id: b959344da1e77819220d26695ff6634f13ac4e0d
Summary:
On Windows, if edenfsctl is started by pythonw, sys.stdout will be None,
check if it is before calling isatty on it.
Reviewed By: fanzeyi
Differential Revision: D21732282
fbshipit-source-id: f0a63094f2c53b026c388f4e5b58433be4397ba9
Summary:
The FormatMessage API isn't very well documented in adding \r\n at the end of
the generated message, and the bit of code that needs to be used to remove
these is not very clear either.
We won't get gratuitous empty lines in the log with this.
Reviewed By: chadaustin
Differential Revision: D21663502
fbshipit-source-id: 203ff5015da6f3cebf7eaee9a8f989db342d096a
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: chadaustin
Differential Revision: D21712029
fbshipit-source-id: 3a332b4bf6fac6b3cf396d34e6d5ca4849181a6d
Summary:
1. Add directory existence checking logic before checking "legacy_bind_mounts_dir"
2. Still print the disk consumption returned by the du command when it fails with an error invoking du -skxc
Reviewed By: chadaustin
Differential Revision: D21683421
fbshipit-source-id: 531b6b289e9ffa445ed691611c9cbf22f393e742
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary: I've updated most messages to suggest a graceful restart, but I overlooked this instance and we got a user question about this. So lets update this message here as well.
Reviewed By: chadaustin
Differential Revision: D21681035
fbshipit-source-id: 32bf5e75b9ad1d98bcd70427101aabeee0be5046
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary:
This script deals with obtaining the right privilege to start the EdenFS
service. This effectively enables `edenfsctl start`.
Reviewed By: wez
Differential Revision: D21585739
fbshipit-source-id: 7b835434d865fa4c4c8473e13665ae669fd86108
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary: EdenFS doesnt daemonize correctly due to the privhelper not closing fd 0 (see http://www.faqs.org/faqs/unix-faq/programmer/faq/). This redirects stdin to /dev/null/ in order to do so.
Reviewed By: xavierd
Differential Revision: D21602545
fbshipit-source-id: 0aeb589efbf214ef22c0db039fbb6a436a71e360
Summary:
This checks in some design documents based on the "EdenFS Internals"
presentation that I've given internally a few times.
Reviewed By: wez
Differential Revision: D21519530
fbshipit-source-id: 3f79d38e8ccf994b2ef303d491809a91fa5b6d95
Summary:
Next step in unifying the mount path, let's make the initialize the same in
Windows and unices. The only difference is now limited to the .eden directory
which we will be able to implement once regular users can create symlinks.
For the takeover code, the #ifdef is pushed down to the actual code that does
it, this allows the rest of the code to not have to bother about Windows vs
other platforms.
Reviewed By: wez
Differential Revision: D21517478
fbshipit-source-id: d40ca2694d23031ff98e319071e610efa306008f
Summary:
In order to build a StringPiece from a C-string, the length of the that
string needs to be known, and a constexp strlen is performed on it. That
strlen is however a recursive one, causing the stack to blow up on big file.
Interestingly enough, this also means that binary files potentially had a
wrong sha1 being computed, potentially causing `hg status` to report some
files as being modified, while they aren't. By switching to using a vector
instead of a string, the intent should of this should be more obvious.
Reviewed By: simpkins
Differential Revision: D21551331
fbshipit-source-id: 2dc1f08d96f49d310593e0e934a03215be2b5cbb
Summary: Nothing should be running with this version, so this check is not needed anymore!
Reviewed By: chadaustin, wez
Differential Revision: D21513269
fbshipit-source-id: 698c9d84e87461cd0484ef997f44dc9169e87a25
Summary:
While not all stats are supported on Windows, most of them can, let's enable
them. This is a first step in unifying the mount path for Windows and Unices.
Reviewed By: chadaustin
Differential Revision: D21514732
fbshipit-source-id: 00fd89b7499f8ae18c089f98bc18cd43f68538fa
Summary:
While this isn't the right fix, this is what shipped in our packages, for the
sake of being able to reproduce the package, let's land this as it is. A
future change will remove this ifdef.
Below is pkaush original description:
In Eden Windows we treat all the files as regular files and don't have a
concept of symlinks and executable files. Fixing the TreeEntryType::getType()
to return REGULAR_FILE for executable file and symlink.
Reviewed By: wez
Differential Revision: D20481051
fbshipit-source-id: 0b0c4d7aea28134383ef45aeafc02930b420286b
Summary: add two methods calls as typehints (no real code change).
Reviewed By: zeroxoneb
Differential Revision: D21337646
fbshipit-source-id: 8079883f7f989251965d3308c5374f097023b57a
Summary:
- Added uptime field to DaemonInfo thrift struct
- Created startTime member variable in EdenServer
- Made appropriate refactoring changes to EdenMain and EdenServer
- Changed main.py and util.py to use the new uptime value
Reviewed By: genevievehelsel
Differential Revision: D21471140
fbshipit-source-id: 8868de667dfb95de93e3e71b90c0412fb3825388
Summary:
The disabled tests are due to the st_mtime stored in the struct stat only
having a second granularity (as expected), but the tests are changing these
files faster than that, causing them to fail. I'll attempt to fix the file
change detection mechanism on a later diff.
Reviewed By: wez
Differential Revision: D21319025
fbshipit-source-id: a3f2d62a3ce56fea37a62bfad73306ac1556f772
Summary:
Same as the previous one, paths are different on Windows, and thus we need to
test against a different path.
Reviewed By: wez
Differential Revision: D21319021
fbshipit-source-id: 849c86574e69c3f638ea180c3b594aaeae2970ad
Summary:
The use of realpath is needed to resolve a unix-style path to a Windows style
one. This helps keep the test generic with no `#ifdef _WIN32`
Reviewed By: wez
Differential Revision: D21319019
fbshipit-source-id: b8cdd81f0afdd135849a5b850d854399cef8cef8
Summary:
In glibc, pthread cancellation support adds two atomic CAS operations
to each "cancellation point" syscall (see pthreads(7)). This includes
read() and write(). We can avoid that overhead by disabling pthread
cancellation at the start of the FUSE worker threads.
This saves two CAS operations (~40 ns) in the critical FUSE request
processing loop.
Reviewed By: simpkins
Differential Revision: D21469690
fbshipit-source-id: 7f28a2a8e831006351657981e901dc572c58cf48
Summary:
Migrating buck integration to use configurations, which is more performant and is the primary supported way to run Pyre.
Mypy is no longer supported at Facebook, so mypy buck integration is covered in this migration as well.
Configuration target automatically expanded to include all subtargets, expanding type coverage while introducing no more than 2 fixmes per file.
Reviewed By: dkgi
Differential Revision: D21474723
fbshipit-source-id: 2501af8250b4998fe331350b80c9dc8b4ba2dd1a
Summary:
Sometimes, in shell one-liners, it's convenient to insert the current
eden pid. We print the pid as part of `eden status` (like nginx and
many other daemonizing programs), but that is not useful for shell
substitution. So a pid subcommand that succeeds and prints the pid to
stdout if running and fails if edenfs is not running.
An example use would be `strace -fyp $(eden pid)`
Reviewed By: genevievehelsel
Differential Revision: D21397118
fbshipit-source-id: 06010150557ec882b452180665b84452fa06ebbd
Summary:
The information available in `eden top` is growing and as we add more and
more to eden top, we want to make sure that eden top users can understand
what `eden top` is telling them. Particularly if we want eden end users to be
able to debug themselves, providing them enough context on the
metrics eden top provides is important.
The help page explains what data is included, why this data is helpful and
when this data indicates a concern.
Reviewed By: wez
Differential Revision: D20974262
fbshipit-source-id: caaef70e76d283968bee8533c7c34d63a95c5490
Summary:
this is preperation for adding a help page to eden top (D20974262)
This allows adding a scrollable section to `eden top`. This is particuarly needed
so that the help page is readable when it does not fit all in one screen. This
also allows scrolling the process table on the main page to see all the
processes listed (this is added here).
Reviewed By: chadaustin
Differential Revision: D20974245
fbshipit-source-id: 402409efdc451db32e7a267b0750204f2322dc4f
Summary:
this is preparation for adding a help page to eden top (D20974262)
This moves all code for writing to the screen to its own class so that
it is not scattered through out the rest of the eden top logic. This is
primarlity so that it is easier to make changes the screen writing logic to
allow scrolling, which is needed so that the full help screen can be read
when it takes up more space than is avaiable.
Reviewed By: chadaustin
Differential Revision: D20965954
fbshipit-source-id: c2690a293579706605d26be9d3bbe491104e6847
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.
Reviewed By: simpkins
Differential Revision: D21395159
fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
Summary:
Add new fields to EdenConfig to control log rotation settings, and update
edenfs_monitor to set up the log rotation strategy using these settings.
Reviewed By: chadaustin
Differential Revision: D20427271
fbshipit-source-id: 9960bdb6f4d077a4e21fb4a6209aa02ab21ad653
Summary:
Update the LogFile class in edenfs_monitor to support log rotation when the
file size exceeds a set limit.
There is a generic API to abstract out the log rotation details. For now
I have only implemented a relatively straightforward implementation that
appends the current timestamp to the log path, and prunes old log files.
Reviewed By: chadaustin
Differential Revision: D20427310
fbshipit-source-id: f6aefc8f2fe604fdd1d6a72c3d30666612912f41
