Commit Graph

2978 Commits

Author SHA1 Message Date
svcscm
2d1616b88b Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: e757c7a80edc58430b7019c7e1ecbc20e278ceeb
2019-08-15 18:37:22 -07:00
Adam Simpkins
117f32ca05 getdeps: distinguish build vs install steps in run_cmake.py
Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.

Reviewed By: wez

Differential Revision: D16778007

fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0
2019-08-15 17:58:29 -07:00
Adam Simpkins
e4fca4936d getdeps: honor INSTALL_DIR correctly in the generated run_cmake.py script
Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.

We now emit each entry in these variables on separate lines, just to improve
readability.  (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)

This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`.  This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.

Reviewed By: wez

Differential Revision: D16778006

fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee
2019-08-15 17:58:29 -07:00
Adam Simpkins
39fdad4d3a getdeps: allow overriding project source, build, and install directories
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis.  The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.

In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.

Reviewed By: wez

Differential Revision: D16778011

fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
2019-08-15 17:58:29 -07:00
Adam Simpkins
c7a0ad4b36 getdeps: fix a crash if the project_hashes directory does not exist
Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.

Reviewed By: wez

Differential Revision: D16778009

fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe
2019-08-15 17:58:28 -07:00
Adam Simpkins
78a214307e getdeps: consolidate code for project subcommands
Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.

This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects.  Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.

Reviewed By: wez

Differential Revision: D16778010

fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a
2019-08-15 17:58:28 -07:00
Wojciech Lis
4fe5642b14 Formatting the output of the eden debug file_stats as json
Summary:
Formatting the output of the eden debug file_stats as json
This will make it easier to use with further tooling to analyze and build
actionable statistics.

Reviewed By: strager

Differential Revision: D16832739

fbshipit-source-id: fb90ff59f121eeac3bf4865956ed5314ce243844
2019-08-15 16:24:47 -07:00
svcscm
c3641decf8 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: a9887337afc578680e3c6a43afd51246caca685e
2019-08-15 14:36:54 -07:00
Wez Furlong
6e127b8bd8 eden: fixup spurious errors when using prefetch --pattern-file
Summary:
our argparse stuff was configured to require 1+ patterns
which meant that an invocation like:

```
echo 'fbcode/eden/fs/TARGETS' | eden prefetch --pattern-file /dev/stdin
```

would emit:

```
eden prefetch: error: the following arguments are required: PATTERN
```

the workaround was to specify a dummy pattern argument.

This diff fixes up the argparse config to make the pattern argument be 0+ patterns.

Reviewed By: pkaush

Differential Revision: D16831732

fbshipit-source-id: 93cb4c4677b1af8b9d4824bcad60708c40867375
2019-08-15 14:02:17 -07:00
Wez Furlong
87dad4e133 eden: remove b'filename' from eden prefetch output
Summary:
This was an unanticipated casualty of some unicode vs bytes
work in the past, and we hadn't gotten around to fixing this... until now!

Reviewed By: pkaush

Differential Revision: D16831603

fbshipit-source-id: dd38751e6d0b1c242937a7dbb150f3aa15d1c8ae
2019-08-15 14:02:17 -07:00
svcscm
a241a6941f Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 15cf80855f4d10a242c7ed38f39de9d8409ed6cc
2019-08-15 14:02:16 -07:00
svcscm
15de398467 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: f6ffa442b1541648352985c520155ee1a15926ca
2019-08-14 23:56:31 -07:00
Wez Furlong
a7ddbe513b eden: fixup typechecker errors on linux
Summary: The type checker doesn't understand `windll`.  Sanest path is to leave the kernel of this function untyped.

Reviewed By: strager

Differential Revision: D16820160

fbshipit-source-id: 47c1bb5f29b790b48dd28d3b1b1e21b50722eb12
2019-08-14 19:55:33 -07:00
svcscm
02b35131b4 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 731dc637a4f543f912bf7a55051c8163f21e97c6
2019-08-14 19:31:40 -07:00
Puneet Kaushik
07958c15ff Implement eden prefetch for Windows
Summary: On POSIX system we use InodeTree structure to implement prefetch. These structures are missing on the Eden Windows. We will instead use mercurial's prefetch to prefetch the contents for us.

Reviewed By: strager

Differential Revision: D16742336

fbshipit-source-id: db0a389503cb98469707c07b6da412056f4dd72e
2019-08-14 19:11:33 -07:00
Puneet Kaushik
a043a4bb35 Updated get_eden_mount_name() to find the eden mount name on Windows
Summary: On Windows, we create a .eden on the root of the repo as compared to in each directory. Updating get_eden_mount_name() to look for .eden in the current parent path.

Reviewed By: strager

Differential Revision: D16742335

fbshipit-source-id: 2d27c54895917524d3d9c233ccc723b51f33f032
2019-08-14 19:11:33 -07:00
Puneet Kaushik
5bd4a8c788 Create .eden at the repo root and store config in it.
Summary: On Linux and Mac, we provision .eden inside every directory, which contains symlinks to other locations. On Windows this won't work as well. First, Projected FS will cache all the .eden and will bump up the number of files on the disk. Second, it would be hard to simulate symlinks from Projected FS.

Reviewed By: strager

Differential Revision: D16724354

fbshipit-source-id: a560698fae8c7f14e475f1c1fb9ecc53e60c189f
2019-08-14 19:11:33 -07:00
svcscm
49882e6dbe Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: b75992be2b9a97040275972abed6a6d25af0a8c0
2019-08-14 19:11:33 -07:00
svcscm
4ab4b081d2 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: a85a5b8258ce777e5001b0973e173707c729b8e4
2019-08-14 16:39:21 -07:00
svcscm
685418899d Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 559f1bf10ad130b2139dbf3f9fa06dafed658365
2019-08-14 16:39:21 -07:00
svcscm
c4c460c3e2 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: c762765b41313e645b8a25516197c7cac293fded
2019-08-14 12:55:11 -07:00
Puneet Kaushik
19aca3a6b2 edenfsctl starts edenfs in background on Windows
Summary: Until now we were running Edenfs on the console. Now we could use edenfsctl start.

Reviewed By: chadaustin

Differential Revision: D16565624

fbshipit-source-id: 9a12e52234d36d7b8ce4e3741ffb62655ae334b2
2019-08-14 12:11:05 -07:00
svcscm
16ddf64dd9 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 45a666d17339c7e458f476768e96c2c6e8302b62
2019-08-14 12:11:05 -07:00
svcscm
2ab3e8ef4a Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 62dc21b9ccbc77b59ce2881ce82b5e5dc105b511
2019-08-13 14:38:43 -07:00
Adam Simpkins
df9a1bf374 getdeps: fix handling of the --host-type command line flag
Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.

Reviewed By: wez

Differential Revision: D16779579

fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f
2019-08-13 14:11:44 -07:00
svcscm
918adfd3de Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 7471bc5bf9df3e2f08351927ad03d182298f96a2
2019-08-13 14:11:43 -07:00
svcscm
435f65bc9a Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 2b77081840a0d035ff5adb12ca3c6748632b9a45
2019-08-13 11:41:08 -07:00
Wojciech Lis
c5d54d79ea Basic tool to collect read/written files on eden
Summary:
Basic tool to collect read/written files on eden
This is v0. The next steps are:
* Add collection of sizes for all paths
* Add as an optional step in duplo/jobs
* Add some smarter analysis options as we discover need

Reviewed By: strager

Differential Revision: D16663820

fbshipit-source-id: 1c25289c9840fc23318e0653bc632b097256a5bd
2019-08-13 11:34:17 -07:00
Matt Mullins
fbd214a01c DBusAddress: re-enable peer-to-peer DBus connections
Summary: Peer-to-peer DBus sockets do not have an org.freedesktop.DBus object on the bus, nor anything (or any reason) to accept the Hello message.   Enable users of DBusAddress to explicitly use this behavior if desired.

Reviewed By: zeroxoneb, simpkins, aleivag

Differential Revision: D16780064

fbshipit-source-id: e10e51aef7754f5b022b8d7455b51baee3db8db8
2019-08-13 10:33:19 -07:00
svcscm
4a61dce954 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: 4a1cc66bae12d200ef14ab959149243f89017f32
2019-08-13 02:15:46 -07:00
Adam Simpkins
07a28e2f5c implement path computation during fsck
Summary:
Implement OverlayChecker::computePath().

This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.

Reviewed By: chadaustin

Differential Revision: D16577697

fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
2019-08-12 19:53:44 -07:00
Adam Simpkins
84de49b69b initial C++ fsck implementation
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.

This initial diff just contains the code to scan for errors, without any
repair logic.

This also includes some improvements to the logic for efficiency.  For
instance, we compute less state during the initial scan phase.  This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found.  This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code.  (On an SSD with the files in question already warm in the
kernel cache.)

Reviewed By: strager

Differential Revision: D16577699

fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
2019-08-12 19:53:44 -07:00
svcscm
314b1601b0 Updating submodules
Reviewed By: cdelahousse

fbshipit-source-id: ab67038c3a545e3170097760f7be762c21d73175
2019-08-12 19:53:43 -07:00
svcscm
a435c91890 Updating submodules
Reviewed By: yns88

fbshipit-source-id: a9c2fdb1a03dde53b8b2be9d71a3a44a0ec77f5b
2019-08-11 23:48:03 -07:00
svcscm
71a5963449 Updating submodules
Reviewed By: yns88

fbshipit-source-id: e81e5cec02db0d44779f6ff1879e32b7cb4aac23
2019-08-11 14:10:25 -07:00
svcscm
ae515972e9 Updating submodules
Reviewed By: yns88

fbshipit-source-id: e0619359256bac755e0f2b5351dd38beb94f7ed5
2019-08-10 14:04:19 -07:00
Puneet Kaushik
78079981f8 Redirect the log messages to the eden log file
Summary: Redirecting the stdout and stderr to the eden log file. The log categories are not working as expected on Windows and will need to be fixed. At this point this doesn't log the "eden=DBG2" on Windows. One workaround for now is to remove "FOLLY_INIT_LOGGING_CONFIG()" to get the default logs.

Reviewed By: chadaustin

Differential Revision: D16653632

fbshipit-source-id: 028cdf1d698798ff3fb0284b3f0fa2df5570eaf0
2019-08-10 11:19:24 -07:00
Puneet Kaushik
a48cd827f1 Move getLogPath() and dependent functions to EdenInit
Summary: Moving these functions to EdenInit so we could use them on Windows. This diff doesn't change the behavior.

Reviewed By: chadaustin

Differential Revision: D16653498

fbshipit-source-id: 5cc8c2f4b12f54035fb86ae20f3274245642d032
2019-08-10 11:19:23 -07:00
svcscm
9077a3421f Updating submodules
Reviewed By: yns88

fbshipit-source-id: 93d3fafe5c7655e34e581aef9ecec84b9c09c3eb
2019-08-10 11:19:23 -07:00
svcscm
874f70a627 Updating submodules
Reviewed By: yns88

fbshipit-source-id: c525db5ec7c34f3cfa66530dad6d8b24077c94c8
2019-08-09 13:02:03 -07:00
Adam Simpkins
9cfbf61798 getdeps: make sure ManifestLoader never reloads manifests
Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded.  This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.

Reviewed By: pkaush

Differential Revision: D16586682

fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f
2019-08-09 11:22:24 -07:00
svcscm
967a85a126 Updating submodules
Reviewed By: yns88

fbshipit-source-id: fc980e902f8b03b48d5801488987f32732fe426c
2019-08-09 11:22:24 -07:00
svcscm
6f3f4d5363 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 8800c8078ca856e5fa6343db537390bcdfbf89c3
2019-08-08 13:12:03 -07:00
Brian Strauch
2dee36afea Move cmd column to the far right
Summary:
Moves the CMD column to the right, to prevent long commands from being truncated.
https://pxl.cl/GG37

Reviewed By: chadaustin

Differential Revision: D16626771

fbshipit-source-id: 291e862d2fcff6c252ae6263536a799428577137
2019-08-08 12:50:57 -07:00
svcscm
ae0d10759f Updating submodules
Reviewed By: yns88

fbshipit-source-id: 6aba866b7e145e39133997780c391012bdf3799d
2019-08-08 12:50:57 -07:00
svcscm
5dd4f90d4e Updating submodules
Reviewed By: yns88

fbshipit-source-id: d555f46e52db5b1f040050ec6d11bc187f299bc1
2019-08-08 12:29:12 -07:00
svcscm
a193926e1f Updating submodules
Reviewed By: yns88

fbshipit-source-id: ab99c2a9f4ecd3098dbbdb7d2fa3c4e5fbab0904
2019-08-07 12:23:43 -07:00
Adam Simpkins
8dc58acda1 rename the fb303_thrift_cpp2 build rule to fb303_thrift_cpp
Summary:
Drop the `2` from the `fb303_thrift_cpp2` rule name.

The old `cpp` version of fbthrift is gone, `cpp2` is the only supported C++
version, so lets drop the `2` from this build rule name.

I plan to change some of the CMake thrift rule generation code soon, to
support multiple languages (namely Python).  As part of this I plan to
automatically add a language-specific suffix(`_py` or `_cpp`) to the build
rules.  It seems better to use `_cpp` rather than `_cpp2` for C++ libraries.
Changing the name for this fb303 rule will make it fit this planned naming
scheme.

Reviewed By: chadaustin

Differential Revision: D16653264

fbshipit-source-id: e4da70c79ff14e9981717ab909d982770d104512
2019-08-07 11:22:34 -07:00
svcscm
6da3985cd3 Updating submodules
Reviewed By: yns88

fbshipit-source-id: b3d84f6df7db1e400d6ca3c9597b069966328378
2019-08-07 11:22:34 -07:00
svcscm
8c4ec5b169 Updating submodules
Reviewed By: yns88

fbshipit-source-id: 39679982cb27ffc42ca42220359bf3be7ebb01f8
2019-08-06 20:30:02 -07:00