0de0ba4f3b
Summary:
This patch enables BST stats by default, and works as follows:
- it uses quantile stats to maintain per queue per port stats.
- these stats are exposed to ODS.
- experiments showed that a significant CPU cycles were spent in doing
bcm_cosq_bst_stat_sync which syncs the hw stat value to sw copy.
For example, calling bcm_cosq_bst_stat_sync while retrieving BST stats
for for every port/every queue/every second resulted into CPU utilization
going up from ~ 50% for Wedge 100 to ~100-120%. Thus, this patch calls
bcm_cosq_bst_stat_sync only once per minute and then retrieves all BST stats.
- The above optimization also helps when bufferstatlogging is enabled.
- when bufferstatlogging is enabled, the BST stats will be retrieved every
second for scuba logging, and quantile stats will also be exported every
second.
- The bufferstatlogging is enabled using fboss CLI (over thrift).
We no longer need enable_fine_grained_buffer_stats option. But, a new option
viz. update_bststats_interval_s is introduced that keeps the frequency of BST
stats updates configurable.
The virtual memory overhead of always maintaining BST stats is < 5%, and
physical memory overhead is < 0.1%. Thanks to the sync optimization above,
the CPU overhead is negligible. Refer Test Plan for raw data and math.
An alternative is to explore using TimeSeriesMinMax (to maintain max over a
time window) and TLCounter (to expose stats to ODS) instead of quantile stats.
One benefit of using quantile stats is that we don't need to maintain our own
TimeSeriesMinMax but can rely on common facebook wide infra. Moreoever,
quantile stats interface is (arguably) simpler than using combination of
TimeSeriesMinMax + TlCounter'.
A separate diff against configerator is sent out to expose the new bst_stats: D8902636
Differential Revision: D8764329
fbshipit-source-id: 78db3de5340497f2f5780863035dd9ff28dcd79b
|
||
|---|---|---|
| CMake | ||
| common | ||
| eden | ||
| .gitignore | ||
| CMakeLists.txt | ||
| CONTRIBUTING.md | ||
| getdeps.py | ||
| LICENSE | ||
| make-client.py | ||
| PATENTS | ||
| README.md | ||
EdenFS is a FUSE virtual filesystem for source control repositories.
EdenFS speeds up operations in large repositories by only populating working
directory files on demand, as they are accessed. This makes operations like
checkout much faster, in exchange for a small performance hit when first
accessing new files. This is quite beneficial in large repositories where
developers often only work with a small subset of the repository at a time.
EdenFS has similar performance advantages to using sparse checkouts, but a much better user experience. Unlike with sparse checkouts, EdenFS does not require manually curating the list of files to check out, and users can transparently access any file without needing to update the profile.
EdenFS also keeps track of which files have been modified, allowing very
efficient status queries that do not need to scan the working directory.
The filesystem monitoring tool Watchman
also integrates with EdenFS, allowing it to more efficiently track updates to
the filesystem.
Building EdenFS
EdenFS currently only builds on Linux. We have primarily tested building it on Ubuntu 18.04.
TL;DR
[eden]$ ./getdeps.py --system-deps
[eden]$ mkdir _build && cd _build
[eden/_build]$ cmake ..
[eden/_build]$ make
Dependencies
EdenFS depends on several other third-party projects. Some of these are commonly available as part of most Linux distributions, while others need to be downloaded and built from GitHub.
The getdeps.py script can be used to help download and build EdenFS's
dependencies.
Operating System Dependencies
Running getdeps.py with --system-deps will make it install third-party
dependencies available from your operating system's package management system.
Without this argument it assumes you already have correct OS dependencies
installed, and it only updates and builds dependencies that must be compiled
from source.
GitHub Dependencies
By default getdeps.py will check out third-party dependencies into the
eden/external/ directory, then build and install them into
eden/external/install/
If repositories for some of the dependencies are already present in
eden/external/ getdeps.py does not automatically fetch the latest upstream
changes from GitHub. You can explicitly run ./getdeps.py --update if you
want it to fetch the latest updates for each dependency and rebuild them from
scratch.