Commit Graph

2954 Commits

Author SHA1 Message Date
Adam Simpkins
df9a1bf374 getdeps: fix handling of the --host-type command line flag
Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.

Reviewed By: wez

Differential Revision: D16779579

fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f
2019-08-13 14:11:44 -07:00
svcscm
918adfd3de Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 7471bc5bf9df3e2f08351927ad03d182298f96a2
2019-08-13 14:11:43 -07:00
svcscm
435f65bc9a Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 2b77081840a0d035ff5adb12ca3c6748632b9a45
2019-08-13 11:41:08 -07:00
Wojciech Lis
c5d54d79ea Basic tool to collect read/written files on eden
Summary:
Basic tool to collect read/written files on eden
This is v0. The next steps are:
* Add collection of sizes for all paths
* Add as an optional step in duplo/jobs
* Add some smarter analysis options as we discover need

Reviewed By: strager

Differential Revision: D16663820

fbshipit-source-id: 1c25289c9840fc23318e0653bc632b097256a5bd
2019-08-13 11:34:17 -07:00
Matt Mullins
fbd214a01c DBusAddress: re-enable peer-to-peer DBus connections
Summary: Peer-to-peer DBus sockets do not have an org.freedesktop.DBus object on the bus, nor anything (or any reason) to accept the Hello message.   Enable users of DBusAddress to explicitly use this behavior if desired.

Reviewed By: zeroxoneb, simpkins, aleivag

Differential Revision: D16780064

fbshipit-source-id: e10e51aef7754f5b022b8d7455b51baee3db8db8
2019-08-13 10:33:19 -07:00
svcscm
4a61dce954 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 4a1cc66bae12d200ef14ab959149243f89017f32
2019-08-13 02:15:46 -07:00
Adam Simpkins
07a28e2f5c implement path computation during fsck
Summary:
Implement OverlayChecker::computePath().

This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.

Reviewed By: chadaustin

Differential Revision: D16577697

fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
2019-08-12 19:53:44 -07:00
Adam Simpkins
84de49b69b initial C++ fsck implementation
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.

This initial diff just contains the code to scan for errors, without any
repair logic.

This also includes some improvements to the logic for efficiency.  For
instance, we compute less state during the initial scan phase.  This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found.  This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code.  (On an SSD with the files in question already warm in the
kernel cache.)

Reviewed By: strager

Differential Revision: D16577699

fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
2019-08-12 19:53:44 -07:00
svcscm
314b1601b0 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: ab67038c3a545e3170097760f7be762c21d73175
2019-08-12 19:53:43 -07:00
svcscm
a435c91890 Updating submodules
Reviewed By: yns88

fbshipit-source-id: a9c2fdb1a03dde53b8b2be9d71a3a44a0ec77f5b
2019-08-11 23:48:03 -07:00
svcscm
71a5963449 Updating submodules
Reviewed By: yns88

fbshipit-source-id: e81e5cec02db0d44779f6ff1879e32b7cb4aac23
2019-08-11 14:10:25 -07:00
svcscm
ae515972e9 Updating submodules
Reviewed By: yns88

fbshipit-source-id: e0619359256bac755e0f2b5351dd38beb94f7ed5
2019-08-10 14:04:19 -07:00
Puneet Kaushik
78079981f8 Redirect the log messages to the eden log file
Summary: Redirecting the stdout and stderr to the eden log file. The log categories are not working as expected on Windows and will need to be fixed. At this point this doesn't log the "eden=DBG2" on Windows. One workaround for now is to remove "FOLLY_INIT_LOGGING_CONFIG()" to get the default logs.

Reviewed By: chadaustin

Differential Revision: D16653632

fbshipit-source-id: 028cdf1d698798ff3fb0284b3f0fa2df5570eaf0
2019-08-10 11:19:24 -07:00
Puneet Kaushik
a48cd827f1 Move getLogPath() and dependent functions to EdenInit
Summary: Moving these functions to EdenInit so we could use them on Windows. This diff doesn't change the behavior.

Reviewed By: chadaustin

Differential Revision: D16653498

fbshipit-source-id: 5cc8c2f4b12f54035fb86ae20f3274245642d032
2019-08-10 11:19:23 -07:00
svcscm
9077a3421f Updating submodules
Reviewed By: yns88

fbshipit-source-id: 93d3fafe5c7655e34e581aef9ecec84b9c09c3eb
2019-08-10 11:19:23 -07:00
svcscm
874f70a627 Updating submodules
Reviewed By: yns88

fbshipit-source-id: c525db5ec7c34f3cfa66530dad6d8b24077c94c8
2019-08-09 13:02:03 -07:00
Adam Simpkins
9cfbf61798 getdeps: make sure ManifestLoader never reloads manifests
Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded.  This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.

Reviewed By: pkaush

Differential Revision: D16586682

fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f
2019-08-09 11:22:24 -07:00
svcscm
967a85a126 Updating submodules
Reviewed By: yns88

fbshipit-source-id: fc980e902f8b03b48d5801488987f32732fe426c
2019-08-09 11:22:24 -07:00
svcscm
6f3f4d5363 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 8800c8078ca856e5fa6343db537390bcdfbf89c3
2019-08-08 13:12:03 -07:00
Brian Strauch
2dee36afea Move cmd column to the far right
Summary:
Moves the CMD column to the right, to prevent long commands from being truncated.
https://pxl.cl/GG37

Reviewed By: chadaustin

Differential Revision: D16626771

fbshipit-source-id: 291e862d2fcff6c252ae6263536a799428577137
2019-08-08 12:50:57 -07:00
svcscm
ae0d10759f Updating submodules
Reviewed By: yns88

fbshipit-source-id: 6aba866b7e145e39133997780c391012bdf3799d
2019-08-08 12:50:57 -07:00
svcscm
5dd4f90d4e Updating submodules
Reviewed By: yns88

fbshipit-source-id: d555f46e52db5b1f040050ec6d11bc187f299bc1
2019-08-08 12:29:12 -07:00
svcscm
a193926e1f Updating submodules
Reviewed By: yns88

fbshipit-source-id: ab99c2a9f4ecd3098dbbdb7d2fa3c4e5fbab0904
2019-08-07 12:23:43 -07:00
Adam Simpkins
8dc58acda1 rename the fb303_thrift_cpp2 build rule to fb303_thrift_cpp
Summary:
Drop the `2` from the `fb303_thrift_cpp2` rule name.

The old `cpp` version of fbthrift is gone, `cpp2` is the only supported C++
version, so lets drop the `2` from this build rule name.

I plan to change some of the CMake thrift rule generation code soon, to
support multiple languages (namely Python).  As part of this I plan to
automatically add a language-specific suffix(`_py` or `_cpp`) to the build
rules.  It seems better to use `_cpp` rather than `_cpp2` for C++ libraries.
Changing the name for this fb303 rule will make it fit this planned naming
scheme.

Reviewed By: chadaustin

Differential Revision: D16653264

fbshipit-source-id: e4da70c79ff14e9981717ab909d982770d104512
2019-08-07 11:22:34 -07:00
svcscm
6da3985cd3 Updating submodules
Reviewed By: yns88

fbshipit-source-id: b3d84f6df7db1e400d6ca3c9597b069966328378
2019-08-07 11:22:34 -07:00
svcscm
8c4ec5b169 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 39679982cb27ffc42ca42220359bf3be7ebb01f8
2019-08-06 20:30:02 -07:00
svcscm
473fca0af1 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 382d8cfeba1a7d7d77afe8dada10953bb6690413
2019-08-05 20:22:29 -07:00
svcscm
d23dc1ebb5 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 254178e08042af082a0b84bd1cc9843499af5cae
2019-08-05 20:22:29 -07:00
svcscm
2c34f8aa73 Updating submodules
Reviewed By: yns88

fbshipit-source-id: d86a9d51656213c4979cf7ce357bc6dba1558a97
2019-08-05 16:35:52 -07:00
Chad Austin
eab4acd9e8 fix TSAN false positive in InodeMap::updateOverlayForUnload
Summary:
Now that we have unsafe access to the contents of a
folly::Synchronized, InodeMap::updateOverlayForUnload can just access
the storage because it knows nobody else has access.

Reviewed By: simpkins

Differential Revision: D16632778

fbshipit-source-id: 4b2c1d6679e2655671d43899407bac1a19f2cab9
2019-08-05 15:43:55 -07:00
Adam Simpkins
67003591aa suppress a gcc warning about noreturn
Summary:
`StartupLogger::exitUnsuccessfully()` is declared as `noreturn`, and its last
statement is to call another `noreturn` method.  However, this other method is
virtual, and the behavior of the `noreturn` attribute on virtual functions
doesn't seem consistently implemented across compilers.  gcc doesn't seem to
treat the virtual method as `noreturn`, and therefore still warns that
`exitUnsuccessfully()` may return.

This suppresses that warning by adding a `folly::assume_unreachable()`
statement.

Reviewed By: fanzeyi

Differential Revision: D16653184

fbshipit-source-id: e0a2d8a4420d3582f849e93568a5081827a788f5
2019-08-05 15:03:18 -07:00
svcscm
76963f33f8 Updating submodules
Reviewed By: yns88

fbshipit-source-id: feb1b0318fa3292429459928d0dce1bb088a4aa5
2019-08-05 15:03:17 -07:00
svcscm
7981258f94 Updating submodules
Reviewed By: zpao

fbshipit-source-id: a3841e21c3ba1726d5961be16894daf13f44723c
2019-08-05 01:24:48 -07:00
svcscm
0eb33a5b52 Updating submodules
Reviewed By: zpao

fbshipit-source-id: c335360154dbc64b0e1ddbe50503c67fceba85b9
2019-08-04 15:24:35 -07:00
svcscm
e819ff8a4a Updating submodules
Reviewed By: zpao

fbshipit-source-id: 7806761cd6701805a7acedffa8ae6a543260fab5
2019-08-03 14:56:30 -07:00
Puneet Kaushik
fc4ba6257b Fixing Eden Windows build
Summary: Windows build was broken by D16461868.

Reviewed By: simpkins, strager

Differential Revision: D16635408

fbshipit-source-id: 928c78042ed2814c0032164592545422c868196a
2019-08-03 08:09:14 -07:00
svcscm
9687a5fb46 Updating submodules
Reviewed By: zpao

fbshipit-source-id: 08014df1718a72eac798421be9462a9f30257d05
2019-08-03 08:09:13 -07:00
svcscm
8260a7cf0f Updating submodules
Reviewed By: zpao

fbshipit-source-id: a854ff01facfbd447476889c9641eeee21d2fc53
2019-08-02 22:06:44 -07:00
Puneet Kaushik
60dca8f396 Throw TTransportException in Edencli when the Edenfs not running
Summary:
Edencli will throw and crash the cli when the edenfs is not running.

Before:
```
PS C:\open\fbsource\fbcode\eden\facebook\win> C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe status
Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\__main__.py", line 3, in <module>
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\cli\main.py", line 1546, in main
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\cli\main.py", line 140, in run
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\cli\config.py", line 604, in check_health
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\cli\util.py", line 164, in check_health
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\thrift\client.py", line 74, in __enter__
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\thrift\client.py", line 86, in open
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\thrift\transport\THeaderTransport.py", line 247, in open
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\thrift\windows_thrift.py", line 109, in open
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\thrift\windows_thrift.py", line 66, in connect
  File "C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe\eden\thrift\windows_thrift.py", line 45, in _checkReturnCode
eden.thrift.windows_thrift.WindowsSocketException: Windows Socket Error: 10061
```
After:
```
PS C:\open\fbsource\fbcode\eden\facebook\win> C:\open\fbsource\fbcode\eden\win\build\package\edencli\edenfsctl.exe status
ps: illegal option -- p
Usage ps [-aefl] [-u uid]
-f = show process uids, ppids
-l = show process uids, ppids, pgids, winpids
-u uid = list processes owned by uid
-a, -e = show processes of all users
-s = show process summary
-W = show windows as well as cygwin processes

edenfs not healthy: edenfs not running
```
It internally uses a bash ps command to look for edenfs when the thrift connection fails. Still looking into good ways to fix that.

Reviewed By: strager

Differential Revision: D16565180

fbshipit-source-id: 3c12c538451293e20fabc51c2fe4f1a9bd5b84c0
2019-08-02 17:49:16 -07:00
svcscm
dd1a800b45 Updating submodules
Reviewed By: zpao

fbshipit-source-id: b2b1dcccbee8eec99d2874638cbeeaa96613a3a0
2019-08-02 17:49:16 -07:00
svcscm
2c53f7d1b7 Updating submodules
Reviewed By: zpao

fbshipit-source-id: 436df19ac1a4bf8e7090771b33f457b8b8dc7682
2019-08-02 16:27:10 -07:00
Jake Crouch
ca9e79a791 Remove toHash from HashUpdateJournalDelta
Summary:
The toHash field is unnecessary in HashUpdateDeltas since we only ever iterate in reverse. We instead keep track of the current hash in the journal itself instead of looking at the toHash of the latest HashUpdateDelta.

Delta Struct Sizes:
> File Change: 88
> Hash Update: 96

Reviewed By: strager

Differential Revision: D16522519

fbshipit-source-id: 43baccc8ef2579f72609cc84e81e218794b11725
2019-08-02 15:48:01 -07:00
Jake Crouch
33ccbd62e8 Split Deltas into File Changes and Hash Updates
Summary: Create the second delta type HashUpdateJournalDelta that keeps track of changes to the hash [and the unclean files associated with that hash change]. Journal methods were updated to account for the different delta types.

Reviewed By: strager

Differential Revision: D16520444

fbshipit-source-id: 2a5cea11c9e70e30f6db55d9c8e33f9322ae91fc
2019-08-02 15:48:00 -07:00
Jake Crouch
6dc9783049 Add in tests for hash updating deltas
Summary: Add in two tests for deltas that change commit hashes since a test for this did not exist before. Previously a bug in setting the fromHash of a result only showed up as a failure in Watchman's integration tests with Eden.

Reviewed By: chadaustin

Differential Revision: D16528248

fbshipit-source-id: 56eede749ef2da4dc492a1f7376dc07ca8aa3050
2019-08-02 15:48:00 -07:00
Jake Crouch
c605d7058b Journal Delta Memory Estimation Fixes
Summary: Deltas are stores in a deque and therefore not malloced independently so therefore there is no reason to be getting the goodMallocSize of them. We also need to do accounting for the buckets in the deque that the deltas fall into.

Reviewed By: strager

Differential Revision: D16566675

fbshipit-source-id: 4506fbbcc2044b8fdfe6244313ef7480cfa5151e
2019-08-02 15:48:00 -07:00
svcscm
f5fec45c68 Updating submodules
Reviewed By: zpao

fbshipit-source-id: ab041e3c578c2f0b542ee093830e2988de6126bb
2019-08-02 15:48:00 -07:00
svcscm
60c83ff74d Updating submodules
Reviewed By: zpao

fbshipit-source-id: c7a80571238669f97a32fdc390ffd0dee8631cd8
2019-08-02 13:31:31 -07:00
Brian Strauch
a9bcf08d67 Fixed command name bugs
Summary: The original `eden top` trimmed `chg[worker/12345]` to `12345]` since the code trims the basenames of commands like `/usr/bin/ls` to `ls`. Now `eden top` only trims commands with exact paths and leaves commands with relative paths as is.

Reviewed By: simpkins, strager

Differential Revision: D16572655

fbshipit-source-id: a563ed5d4eb311244211e7db903fb64a37dcc4a4
2019-08-02 09:44:03 -07:00
Brian Strauch
1556dec257 Total FUSE access time
Summary: Record a rolling sum of the time taken by any FUSE call on a per-process basis

Reviewed By: strager

Differential Revision: D16553149

fbshipit-source-id: 54f1e453916727a40f245b294239dc1b232a8967
2019-08-02 09:44:03 -07:00
svcscm
032745a144 Updating submodules
Reviewed By: zpao

fbshipit-source-id: 20835b06ba65c4d4289e811db9e890646b359637
2019-08-02 09:44:03 -07:00