Summary:
Recently I found that its impossible to access elevated Watchman daemon from non-elevated process on Windows.
```
events.js:174
throw er; // Unhandled 'error' event
^
Error: connect EPERM \\.\pipe\watchman
at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Emitted 'error' event at:
at Socket.<anonymous> (C:\open\ovrsource\unity\socialvr\_js\node_modules\fb-watchman\index.js:118:12)
at Socket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
```
To fix this, it was suggested by wez to use [his library](https://github.com/wez/EleDo) to force Watchman daemon always start in normal mode on Windows. In this stack of commits I did integrated his library into project and used it to force daemon restart in normal mode when spawned from elevated terminal.
To make it happen, I checked-in library sources and created proxy project which depends on the initial library and contains header bindings and cmake configuration. I did copy pasted Rust cmake macroses from another facebook project - eden, and also created analogue of autogen.sh but for Windows - autogen.cmd.
Pull Request resolved: https://github.com/facebook/watchman/pull/878
Test Plan:
Launch elevated terminal
Start watchman.exe produced from sources
Observe daemon starting and answering
In process monitor, observe watchman.exe process running under user group
Reviewed By: fanzeyi
Differential Revision: D25595879
Pulled By: arhelmus
fbshipit-source-id: 15eb29adcf5bd4a5708b6533a1b2bacbf13f431c
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary:
- I find the ability to make a branch and play with CI on GitHub super handy
- Removing the `- master` limitation gives me this ability
Reviewed By: yi-xian
Differential Revision: D22771835
fbshipit-source-id: 8e8839cb860ab4d1dfa0dda590afaf165127f60d
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
The breakage has been fixed, so bring back the manifest, but only the Linux one, because the Mac version is failing due to another issue.
Also to make it easier to debug issues on GitHub Actions separate out the dependencies build from Mononoke and rust-shed builds.
Reviewed By: krallin
Differential Revision: D21448412
fbshipit-source-id: 68d89c858d1692727a7fd66bca114920e6dfb4dc
Summary:
D21364132 accidentally broke this; we can't run the fetcher
for projects for which we pulled the build out of cache, because there
is no source to update in that case.
This commit adjusts the logic so that we write out a marker file
to indicate that we installed a build from cache and to look for
that file being present to gate the new update logic.
Reviewed By: lnicco
Differential Revision: D21419122
fbshipit-source-id: 304670848add22531d88549d66f22c40ff255140
Summary:
When the commit hash changed in fbsource, we would correctly decide
that we'd need to rebuild first-party projects but we would incorrectly skip
running the fetcher.update method. This would mean that we'd not perform the
shipit run and that our shipit tree would diverge from the source tree.
This commit resolves this by performing the fetcher.update but ignoring
the source update status in this case.
Reviewed By: xavierd
Differential Revision: D21364131
fbshipit-source-id: b4001e549c7d3f27aa4a21b19893c9bb7c0f6d1f
Summary:
We didn't do a great job of recognizing that we'd need to
build a project when one of its dependencies had changed: we relied
chiefly on the dependency hash for this and could fail to handle
changes in individual source files.
This commit helps to improve this situation by checking to see if
any installed files in the dependencies of a manifest are newer than
the most recent built time of a given manifest. If so, we'll trigger
a build. We try to be reasonably smart about deciding when to trigger
a cmake reconfigure if it looks like cmake files in the deps have
been changed.
Reviewed By: xavierd
Differential Revision: D21364132
fbshipit-source-id: 7534496e10d1f532aa9cf865900ace84a8785327
Summary:
Our linter really wants to include formatting changes unrelated
to my diff stack.
This is a formatting only change to avoid clouding my diffs; no functional
effect.
Reviewed By: xavierd
Differential Revision: D21364519
fbshipit-source-id: 7670dd4154e788f593f256aabdfdeef6d17aeec4
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3. There is no such thing as `python3` in that environment, although there is a `python2`.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20740411
fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff
Summary:
The GH actions defaults for git prevent it from being able
to checkout the fbthrift repo due the length of the java related
files in the fbthrift repo.
This commit tells git to use long filenames and allows the checkout
to succeed.
Reviewed By: fanzeyi
Differential Revision: D20659750
fbshipit-source-id: 060b36d312d52a0769cf2f2dd9af60f7446f94a8
Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
Summary:
This updates fbcode_builder to try and automatically detect the current
repository's project name by looking for a `.projectid` file in the repository
root. If the project name can be detected:
- The current repository will automatically be used as the source directory
for this project (instead of fetching the sources into the scratch
directory).
- If an explicit project name was not specified on the command line, the
current project will be built by default.
This also changes the repository detection logic to use the current working
directory, rather than the directory where the fbcode_builder code lives.
This will allow this logic to work even if we move fbcode_builder into its own
repository in the future, and have projects depend on it using git submodules.
This does mean that callers need to invoke fbcode_builder.py from inside the
repository.
Reviewed By: wez
Differential Revision: D17088938
fbshipit-source-id: f14d28fdcfaa330ff837ea52b8bdd4e358b81c61
Summary:
Never cache first-party projects that use ShipIt. Previously the code checked
the `shipit_fbcode_builder` property, which controlled whether or not
the `fbcode_builder` sources should be included in the project's ShipIt
mapping. This setting is enabled for most but not all projects that use
ShipIt.
This resulted in projects that use ShipIt but that do not include the fbcode
builder sources being incorrectly cached. This caused getdeps.py to not
run the SimpleShipitTransformerFetcher properly when their sources changed.
Reviewed By: wez
Differential Revision: D17493522
fbshipit-source-id: 57be5ac94ae44f56ccb3ce60ba23fac5d68bce0f
Summary:
This is useful when working on changes to some of the builder functions,
to skip ever trying to use cached results.
Reviewed By: chadaustin
Differential Revision: D17401219
fbshipit-source-id: fb7d5ea45618653957b9bd6a62eed91d8334824d
Summary:
This commit adds a getdeps command that is able to generate
a workflow file for the GitHub Actions CI environment.
The workflow file could be expressed more simply using the matrix
syntax and with three steps (checkout, build, test), but I chose to
break out the steps for each of the dependencies because the UX
while waiting on the build is much nicer that way: the steps show
during and live log tailing for the section of the build that is
underway. If they were all lumped into a single build step then
the logs from the boost section of the build dominate and make
the github UI work very hard.
Pull Request resolved: https://github.com/facebook/watchman/pull/743
Test Plan:
https://github.com/facebook/watchman/pull/743 successfully
executes the github actions CI flow.
```
$ opensource/fbcode_builder/getdeps.py generate-github-actions --output-file watchman/.github/workflows/main.yml watchman
```
Reviewed By: simpkins
Differential Revision: D17384915
Pulled By: wez
fbshipit-source-id: 9a9e5a3e806c18f6cc38ba1cb7059740cda01ad4
Summary:
This diff adds a small abstraction that allows for uploading
and downloading from an artifact cache.
We try to download from this cache at build time, but will only
try to populate it for continuous builds--those are built from
code that has been reviewed and landed on master. This restriction
helps to avoid thrashing the cache with works in progress and
results in a slightly more trustworthy state of the cache contents.
In addition to this, we choose only to cache third party projects.
The rationale is that our first party projects move too quickly to
be worth caching, especially since the cache granularity is for
the whole project rather than just changed elements of a given
project.
In a later diff I will introduce some implementations of the
cache class that work with eg: Travis or Circle CI caching.
Reviewed By: simpkins
Differential Revision: D16873307
fbshipit-source-id: 2bfb69e36615791747b499073586562f2ca48be9
Summary:
Feed the first-party oncall data through to the testpilot
invocation. This will set the owner of the test in continuous runs.
The oncall is passed through via the sandcastle module, which means
that you will have to manually pass it through for local testing
(see test plan), but it should automatically get picked up when
sandcastle schedules the `test` step of the job.
Reviewed By: chadaustin
Differential Revision: D17146802
fbshipit-source-id: a4f0e65853a46ed8709594c96db859ede2530b00
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis. The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.
In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.
Reviewed By: wez
Differential Revision: D16778011
fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.
This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects. Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.
Reviewed By: wez
Differential Revision: D16778010
fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a
Summary:
Move code that computes project hashes to ManifestLoader. ManifestLoader is
the only class that has all of the information necessary to compute the
project hashes correctly. The ManifestLoader object can also cache previously
computed hashes, so that we don't have to keep computing hashes for projects
over and over again. Previously the `BuildOptions.compute_dirs()` function
would end up re-computing hashes for all dependencies each time it was called.
Reviewed By: strager
Differential Revision: D16477401
fbshipit-source-id: ce03642114f91ce4f859f612e6b2e747cf1653be
Summary:
The ManifestLoader contains all of the state needed to create a fetcher
object, so define a helper method on this object to create a fetcher.
Reviewed By: strager
Differential Revision: D16477395
fbshipit-source-id: 6de0942fe6b8de26c18c82bf99343f5467dc006a
Summary:
Add a new ManifestLoader class to handle loading manifests and computing
dependencies.
For now the main thing this class does is maintain the `manifest_by_name`
mapping. In subsequent diffs we should be able to move some additional logic
into this class, which will help clean up the code and eliminate some redudant
work. In particular, we can have this class cache project hashes, which will
avoid re-computing hashes over and over again for the same projects as we do
in many cases today. We should also be able to save and re-use some of the
project dependency ordering computation in some cases as well.
Reviewed By: strager
Differential Revision: D16477400
fbshipit-source-id: f06f62f77d8443fccaa69fe4c1306e39c395b325
Summary:
Update `BuildOpts.compute_dirs()` to use the correct project-specific manifest
context when computing project hashes. Previously it was incorrectly using
the initial project's context when evaluating all dependencies. This would
result in some projects potentially seeing the wrong values for variables that
may change from project to project (like `test`).
Reviewed By: pkaush
Differential Revision: D16477398
fbshipit-source-id: 6c23f5e5e19b2402000a138b3920b79044446041
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
getdeps allowed common arguments to be specified either before or after
the subcommand name. However, the arguments were only actually honored
if they were specified on the command line after the subcommand name.
Specifying any of these arguments before the subcommand would simply be
ignored, and the default value would be used. This happened since the
subcommand argument's default value overwrote the value specified by the
user from the main command parser.
This fixes the behavior by using `argparse.SUPPRESS` so that argparse
won't set the value at all if it was not specified. This allows the
value from the main parser to be used correctly if the argument was not
seen by the subcommand parser.
Reviewed By: chadaustin
Differential Revision: D16435358
fbshipit-source-id: f3893ef7a27fb018247f249f313d6438472e0140
Summary:
This diff adds a `fixup-dyn-deps` subcommand that is intended to
aid in packaging on multiple platforms.
Its purpose is to copy a set of executable object files from the getdeps
installation directories and place them into an installation staging
area that will then be used to create some kind of package (rpm, tarball etc.).
The dynamic dependencies of the executables are determined and also copied
into the destination area, and the important part: the execute is rewritten
such that it will load the deps out of an alternate installation prefix.
The implementation of this command draws on similar scripts in use for
the watchman and eden packaging on windows and macos. This diff adds
linux support using the `patchelf` utility.
Reviewed By: pkaush
Differential Revision: D16101902
fbshipit-source-id: 5885125971947139407841e08c0cf9f35fdf5895
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
with the reshuffling around getdeps.py, and changes in folly/thrift/wangle,
our internal package stopped building. This diff brings it back to life. It is
a little bit interwined:
* Introduce a `--facebook-internal` switch to getdeps that causes `fb` in the
build context to be set to `on` rather than `off`. This allows specifying
options in the manifest that are FB specific.
* Introduce cmake options to control the WATCHMAN_STATE_DIR and whether cmake
should attempt to create WATCHMAN_STATE_DIR when it is installed.
* For FB specific builds on macOS, set the state dir to match our existing location
and disable creation of WATCHMAN_STATE_DIR because that will ultimately be
handling during package installation and we don't have privs for that on
the machine generating the package.
* Adjust Facebook CI scripts to use getdeps for building and testing watchman
Reviewed By: simpkins
Differential Revision: D15337970
fbshipit-source-id: 8ab3c1f810decf5f0b16337dc1cbe6dc73d26426
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
This controls whether tests are built or not.
They are not built by default. When `--enable-tests` is turned on,
tests are enabled for the named project only, not all of the deps.
This results in a faster build, because eg: the number of tests in
folly is very large and consumers of folly don't want to spend so
much time waiting to build tests when really all they want to do
is build their own project.
Reviewed By: strager
Differential Revision: D15246336
fbshipit-source-id: 2e22fd60c983b059f71639602712087f2a188ec6
Summary:
This is needed to correctly pick up changes made to
eg: cmake.defines sections in the manifest for first-party
projects.
Reviewed By: strager
Differential Revision: D15246337
fbshipit-source-id: 35e525e885f87d6136d5ff3b94ebf34516ab947c
Summary:
This is useful especially on Windows where the up-to-date
checks for the dependencies take a long time.
The idea is that you might run this to start:
```
$ getdeps.py build eden
```
and then while in the edit/debug/build iteration cycle:
```
$ getdeps.py build --no-deps eden
```
Reviewed By: pkaush
Differential Revision: D15200352
fbshipit-source-id: 086f2f49db967ef4d1914a69fa80067104d79136
Summary:
This prints out the installation or source prefix for a given project.
This is useful in eg: packaging scripts to figure out where they can
find the built artifacts.
Reviewed By: simpkins
Differential Revision: D14967378
fbshipit-source-id: 7e1b5de2ca7219af24cfb07b4b42de22aa410469
