Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary: Not checking whether journal duration was None caused old Edens to have an error when running "eden stats", this should fix it such that it does not print out duration info if Eden does not have it.
Reviewed By: strager
Differential Revision: D16502597
fbshipit-source-id: 1e28732208a3ad660b021b9e19c2694302a569f4
Summary: Now that Eden depends on open source fb303, EDEN_HAVE_STATS is unnecessary. Remove it.
Reviewed By: simpkins, strager
Differential Revision: D15526905
fbshipit-source-id: 2354f1b92545a089de0e91e7c33515fa0b74b067
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
From andrewjcg:
> Basically, rocksdb headers exist at both e.g. rocksdb/src/include/rocksdb/flush_block_policy.h and rocksdb//flush_block_policy.h. The latter are never meant to be used, but Apple platforms by default add the repo root to the include dir before anything else. This means depending on file-relative inclusion from within rocksdb, we end up getting different versions of the header, which busts through #pragma once causing multiple definition errors.
This is split out from D15125826.
Reviewed By: andrewjcg
Differential Revision: D16353321
fbshipit-source-id: 095e3b7fb4f56b0be42d18fe4009336b65c8eb5e
Summary:
To save on memory the journal will now compact the same action repeated multiple times into the same action. This means that modifying the same file 100 times in a row results in 1 Journal delta instead of 100. [The results will cause Watchman to act the same since all queries are down from the current time, changes should only be visible by the number of deltas in the journal, how much memory the deltas are using, "eden debug journal" which will show that sequenceID's were skipped, and the fromSequence/fromTime returned by accumulateRange might be different]
**Memory Improvements:**
For buck commands, 1 run was conducted for each with a buck clean done before each build and then eden being restarted (so the clean did not affect the outcome) [results are formatted as 'with compaction' / 'without compaction']
“buck build mode/opt eden”
Entries: 154145 / 206108 [25.2% reduction]
Memory: 46.2 MB / 61.4 MB [24.7% reduction]
“buck build mode/opt warm_storage/common/...”
Entries: 318820 / 405016 [21.3% reduction]
Memory: 95.8 MB / 121.5 MB [21.2% reduction]
For Nuclide the result was calculated by getting the number of entries in the journal vs the last sequence ID in the journal ('entries we actually have' / 'entries we would have without compaction')
Using Nuclide’s Smart Log and Checking Out various commits / arc pulling:
Entries: 6091 / 23671 [74.3% reduction]
Reviewed By: chadaustin
Differential Revision: D16096960
fbshipit-source-id: f542ae32c889ebc9da442285d808ce75247f7e65
Summary:
Some people want to use getdeps with Python 2.7. This looks easy to do, so take a step toward Python 2.7 support by fixing getdeps' tests when run with Python 2.7.
For Python 3, this diff should not change behavior.
This diff should address https://github.com/facebook/bistro/issues/35.
Reviewed By: snarkmaster
Differential Revision: D16435667
fbshipit-source-id: f5c262b12995b609263341c4de26dac7f9b12b70
Summary:
This is just https://github.com/facebook/bistro/pull/37/, backported to `fbcode_builder`.
snarkmaster
Pull Request resolved: https://github.com/facebook/bistro/pull/37
Test Plan: Push and watch Travis
Reviewed By: simpkins
Differential Revision: D16453080
Pulled By: snarkmaster
fbshipit-source-id: a15eaead931f046c41e50f8e3b412ef68b172d65
Summary:
Don't print a status message about finding LZ4 twice: one is already printed
by `find_package_handle_standard_args()`.
Also change a few function calls to lower-case to be more consistent with the
rest of our CMake files.
Reviewed By: chadaustin
Differential Revision: D16435455
fbshipit-source-id: bc2c66399823162f93e62d1d0dbdba422fe63b24
Summary:
getdeps allowed common arguments to be specified either before or after
the subcommand name. However, the arguments were only actually honored
if they were specified on the command line after the subcommand name.
Specifying any of these arguments before the subcommand would simply be
ignored, and the default value would be used. This happened since the
subcommand argument's default value overwrote the value specified by the
user from the main command parser.
This fixes the behavior by using `argparse.SUPPRESS` so that argparse
won't set the value at all if it was not specified. This allows the
value from the main parser to be used correctly if the argument was not
seen by the subcommand parser.
Reviewed By: chadaustin
Differential Revision: D16435358
fbshipit-source-id: f3893ef7a27fb018247f249f313d6438472e0140
Summary:
This diff updates Eden's journal to be bounded in terms of memory usage which should help lessen the likelihood of Eden OOMing and taking up a large amount of our users' resources.
The memory limit is set to be 1 GB per journal [so a user with 3 mounts could expect the journals to possibly use up to 3 GB of memory].
The landing of this diff will need to wait until a version of Watchman that can handle truncation is deployed on all machines using Eden. This means we need to wait for a version of Watchman with D16219267 to fully land to machines.
Reviewed By: strager
Differential Revision: D15954994
fbshipit-source-id: 9a6425527f10a1ce051feb8fc7d092a84712f338
Summary:
Enable the `edenfsctl debug` subcommands on Windows.
The `overlay` subcommand is the only debug subcommand that requires modules
not available on Windows. The overlay code uses the `fcntl` module for
locking the overlay. However, EdenFS on Windows does not use the overlay
directory structure used on Linux and Mac, so this subcommand isn't relevant
on Windows.
This moves the `overlay` subcommand to its own module, and changes the code to
avoid importing it on Windows. This lets us enable the rest of the debug
subcommands on Windows.
Reviewed By: pkaush
Differential Revision: D16354620
fbshipit-source-id: ee7ea01fa3a21018850f5ba590eb143d7a9718cf
Summary:
Update `make-client.py` so that it does not require the python interpreter to
be explicitly specified on the command line. It now uses `sys.executable`
as the default python interpreter if one was not specified on the command
line.
Also add a `--output` flag to control the output file path.
Reviewed By: pkaush
Differential Revision: D16388625
fbshipit-source-id: 6587acb0ea8c083faf76f74c77c7c09ef0b62865
Summary:
The `curses` module is part of the standard Python library on Linux and Mac,
but not on Windows.
This updates the CLI's `ui` module to avoid trying to import the `curses`
module on all platforms. This adds a new `WindowsOutput` class for Windows.
For now this class is just a stub that uses the existing `PlainOutput` class
instead.
Reviewed By: pkaush
Differential Revision: D16354626
fbshipit-source-id: 262637030febd6893a94e19712a07cd3d5d39bbb
Summary:
Centered clock between title and hostname
Removed bar
Defaulted update speed to 1s, with command line option to change
Gave more/less space to certain columns
Decreased padding to 1 character
Reviewed By: chadaustin
Differential Revision: D16195428
fbshipit-source-id: 6dbd6710727bcd3e65e87fc5bd269a39507dca9b
Summary:
If `path_search()` returns that CMake is not available, raise an Exception and
fail the build. This makes the failure somewhat easier to to identify.
Without this the code would continue and would try to invoke `subprocess` with
a value of `None` in the command argument list. This error is slightly harder
to debug, since it isn't clear which command argument or environment variable
is not a string.
Reviewed By: chadaustin
Differential Revision: D16354623
fbshipit-source-id: be972b02cb13bc70db0f867da70e5bf4c6cec46d
Summary:
The scratch path is used as part of the hash computation for each project. We
need to make sure this path is always normalized to ensure that we compute the
hashes consistently.
Reviewed By: chadaustin
Differential Revision: D16354624
fbshipit-source-id: 39b5362620bdc247cd7e7f1333dac319b354dc6f
Summary:
getdeps currently ends up calling `path_search()` repeatedly searching for
various C/C++ compilers in $PATH. It ends up doing this multiple times for
each dependency as it computes the project hashes. This updates the
`path_search()` function to cache its results so that we don't keep performing
the same searches on the file system over and over again.
Reviewed By: chadaustin
Differential Revision: D16354625
fbshipit-source-id: 116293bd2f636632517d26436b2332e6c10624f1
Summary:
We install the python files under `eden/fs/service` in a package named
`eden/thrift` in the built python binaries that use these modules. This
moves the source files into an `eden/thrift` subdirectory so that the source
directory layout more closely matches the final binary layout.
This will make it easier to run several of Eden's Python-based tools directly
from the source tree, without having to do as much directory layout
transformation. This is particularly helpful on platforms like Windows, which
don't currently have an equivalent of "live PARs" which can be run from the
source tree without requiring a rebuild after each file edit.
Reviewed By: chadaustin
Differential Revision: D16354622
fbshipit-source-id: 4b58cc96451b1ee5441714aaf74f5e3b6ada9eaa
Summary:
We install `eden/py/dirstate.py` as `eden/dirstate.py` in built python
binaries that use this module. This moves the source file into an `eden/`
subdirectory so that the source directory layout more closely matches the
final binary layout.
This will make it easier to run several of Eden's Python-based tools directly
from the source tree, without having to do as much directory layout
transformation. This is particularly helpful on platforms like Windows, which
don't currently have an equivalent of "live PARs" which can be run from the
source tree without requiring a rebuild after each file edit.
Reviewed By: chadaustin
Differential Revision: D16354627
fbshipit-source-id: 578748e76b730db33cf3ea555df48aa94d15019f
Summary:
Let's minimize the amount of work we do before dropping privileges.
This shouldn't matter too much, since it is just a memory allocation and copy,
but I think we should keep the `dropPrivileges()` call as the very first thing
in `main()` to help prevent people from adding new functionaliy before it in
the future.
This also adds some block comments to help encourage people to only put code
after the "Root privileged dropped" comment moving forwards.
Reviewed By: chadaustin
Differential Revision: D16371783
fbshipit-source-id: 1ce5115bfa71565f0fe12ac9c9442bddb771ed48
Summary: In a previous diff we added the support to pass the command line to EdenServer but missed from the Windows main(). Until we unify the main function we need to add it at two places.
Reviewed By: simpkins
Differential Revision: D16370228
fbshipit-source-id: 47b9842fbca709e1d3c832db3d82765f2e5f8930
Summary:
I made the mistake of migrating the Thrift API for edenfs process
checks without a fallback for older edenfs running processes.
If getDaemonInfo() is not available, fall back on getPid().
Reviewed By: strager, pkaush
Differential Revision: D16365848
fbshipit-source-id: aee5c43c8c40db9a38fef7ca601f7639daa4adb8
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary: The CountersTest would previously fail if by chance the counters prefixed by "thrift" and "thrift_client" were accounted for between getting "counters" and "counters2", since these counters should not be modified when mounting/unmounting mounts we will just filter them out.
Reviewed By: chadaustin
Differential Revision: D16265511
fbshipit-source-id: 21af0dff345977692785136ca0333d23d5c77e0d
