Summary: Debugging an issue where watchman returns a state error during histedit.
Reviewed By: quark-zju
Differential Revision: D20785858
fbshipit-source-id: 5baf762d1a5588573df9d01c63a24e751e04a811
Summary:
During the roll-out of commitcloud remotebookmarks sync, some clients may have it enabled, and others may have it disabled.
This can result in some clients having an empty set of remotebookmarks in their sync state for a version which actually
does have remotenames in the cloud workspace.
This becomes a problem when those clients start syncing remotebookmarks, for two reasons:
* They are unable to correctly apply the new cloud remotebookmarks, as they do not have the previous state with which to break ties, and the tie-breaking code doesn't work correctly if the old cloud remotebookmark is `None`.
* When they try to send the remotebookmarks delta to the server, they don't have the old state, and so compute the delta incorrectly. This results in `Duplicate entry for key` errors in the service.
We can handle this as follows:
* When syncing remote bookmarks from the cloud, ensure we correctly apply the remote bookmark updates, even if the old cloud remotebookmark is `None`.
* If the local repo is up-to-date with respect to the workspace, but the last sync state had no remote bookmarks recorded, sync the remote bookmarks again. This may involve pulling some more commits if the cloud remote bookmarks are ahead of the local remote bookmarks. If there truly were no remote bookmarks then this will be a no-op.
Reviewed By: quark-zju
Differential Revision: D20792360
fbshipit-source-id: cd1ed371d8d5b1be2767c8a8d4836ea870a4467b
Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.
Reviewed By: genevievehelsel
Differential Revision: D20795118
fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6
Summary:
For the initial rollout of lfs on fbsource we want to rollout just for our
team using rollout_smc_tier option. This diff adds a support for that in
Mononoke.
It spawns a future that periodically updates list of enabled hosts in smc tier.
I had a slight concern about listing all the available services and storing
them in memory - what if smc tier have too many services? I decided to go ahead
with that because
1) [Smc antipatterns](https://fburl.com/wiki/ox43ni3a) wiki page doesn't seem
to list it as a concern.
2) We are unlikely to use for large tier - most likely we'll use it just for
hg-dev which contains < 100 hosts.
Reviewed By: krallin
Differential Revision: D20789751
fbshipit-source-id: d35323e49530df6983e159e2ed5bce205cc5666d
Summary:
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Because it is a strong indicator (specifically stronger than the number of imports) for when there is an issue effecting imports, we use it as a warning signal to the user, meaning we display warning colors when the duration of the longest import is long enough to warrant concern.
We have 2 levels of concern to differentiate when there may be something strange happening and when there is almost definitely something strange happening. These correspond to displaying the import metrics in yellow and red respectively. The current threshold for something strange possibly happing is imports taking longer than 10s and something strange definitely happening is imports taking longer than 30s.
Reviewed By: chadaustin
Differential Revision: D20627754
fbshipit-source-id: 7931d2c0ab53952c1291fd1d411dad14d142734b
Summary:
When there is an issue in the import process, there is not an easy way to detect this currently. As a first step towards making it possible to detect when there is an issue here, we expose the number if imports pending to `eden top`.
If there is a problem blocking imports from happening, then they will start to queue up and the number of pending imports will keep growing. When there are a lot of pending imports it indicates that there may be a problem in the import process.
note: there are cases when there are many pending imports under normal operation. It is common to batch fetch a bunch of files at once, and submitting all these prefetches at once will mean a lot of pending imports though there is no problem. Thus number of pending imports only indicates there *may* be an issue in the import process. (Thus we couple it with the duration of the longest import D20627754 for stronger signal)
The other value in showing the number of pending imports is giving users more transparency into eden. When there is a big batch of imports this may leave the user waiting a while, and this gives a way for them to know why. They can see that eden is making progress rather than being stuck.
Reviewed By: fanzeyi
Differential Revision: D20527273
fbshipit-source-id: 59e0fdff6e7b154b409dfab385b4fdb02062b066
Summary:
The primary purpose of exposing counters for the maximum length of duration is to add this to `eden top`.
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Additionally we choose to use these counters to expose this data because we want to log this data. This data can be used to measure performance of the queueing and import process. In the future this data could be used to set up alerting for regressions and allow more proactive fixes.
Reviewed By: chadaustin
Differential Revision: D20611728
fbshipit-source-id: 9307c1ad749ac5fe356ba9eaf868de41b1a8a3a7
Summary: removing duplication to prepare for adding more counters in `EdenServer`
Reviewed By: chadaustin
Differential Revision: D20611713
fbshipit-source-id: 30dd59dff46731b53b08d5681b2add4c94cd049a
Summary:
Here we are tracking the duration of each of the current imports in `HgBackingStore`. This changes is setup to display the duration of the longest outstanding import in `eden top`.
Displaying the duration of the longest import in eden top will help debug where the problem is when eden hangs. These specifically help determine if eden is hanging because of a queueing issue or not being able to fetch an object from the server.
This is a stronger signal to the user that there is an issue in importing over just a count of imports (the previous addition) because many pending imports is not always abnormal. For example a program may prefetch a big batch of files to read, and this could cause a large number of pending imports. A single import should not hang due to a big prefetch and other normal uses.
Thus we want to display this in addition to the number of imports and use this as our warning indicator. (i.e. a high duration of import will mark the metrics as yellow or red in `eden top` to give a warning to the user)
*code changes*:
- adding lists to store watches to time the duration of each of the imports in `HgBackingStore`
- removing the counters previously used to count the number of imports in `HgBackingStore` in favor of using the size of these lists as this is constant time, so there is no benefit to tracking this value ourselves
Reviewed By: chadaustin
Differential Revision: D20611704
fbshipit-source-id: 8bbe0aed8d10688f7bfb7ecfa2fa5ac181945a76
Summary:
git started printing a "+" sign in front of a branch checked out in another worktree:
ab3138146f
and it breaks "git sl"
removing "+" the same way as "*" when parsing "git branch" result to resolve.
Reviewed By: krallin
Differential Revision: D20793452
fbshipit-source-id: 5c0b6be5afbed607b144c652f68a3ab93052f76a
Summary:
This is a followup from D20766465. In D20766465 we've avoided re-traversing
fsnodes for all entries except for copy/move sources. This diff make copy/move
sources fetching more efficient as well.
It does by sending a find_entries() request to prefetch all fsnodes
Reviewed By: mitrandir77
Differential Revision: D20770182
fbshipit-source-id: 7e4a68a2ded20b2895ee4d1c4f8fd897dbe1c850
Summary:
We are currently having problems with streaming clone:
```
$ hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/fbsource.rc 'ssh://hg.vip.facebook.com//data/scm/fbsource?force_mononoke' "$(pwd)/fbsource-clone-test"
remote: server: https://fburl.com/mononoke
remote: session: vJ3qkiQIm9FT7mCp
connected to twshared11499.02.cln2.facebook.com
streaming all changes
2 files to transfer, 5.42 GB of data
abort: unexpected response from remote server:
'\x00\x01B?AB\x00\x00\x00\x00\x02U\x00\x00\x02\xc7\x00b\xf0\xd5\x00b\xf0\xd5\x00b\xf0\xd4\xff\xff\xff\xff\xa8z\xc7W\xd0&\xab\xb2\xf1{\xbfq\xac<\xaf6W\x06q\x81\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?C\x97\x00\x00\x00\x00\x053\x00\x00\x06\xce\x00b\xf0\xd6\x00b\xf0\xd6\x00b\xf0\xd5\xff\xff\xff\xff\xa3I\x19+\xe2\x0f\xae\xd2\x95\x14\x8a\xde\x19\x18\xf0\x8cUQu\xf1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?H\xca\x00\x00\x00\x00\x02\xe4\x00\x00\x03\x9e\x00b\xf0\xd7\x00b\xf0\xd7\x00b\xf0\xd6\xff\xff\xff\xffx\xd6}\x12nt\xb9\xbc(\x83\xfb\xfa\xcc\xc1o?\xde\xcc\x06L\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?K\xae\x00\x00\x00\x00\x02j\x00\x00\x02\xb5\x00b\xf0\xd8\x00b\xf0\xd8\x00b\xf0\xd7\xff\xff\xff\xff\x04"\xfcw6\'M\xba\xf1f\xdb\x02\xbeE\x93:\xc8\x17\x88P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?N\x18\x00\x00\x00\x00\x03\xbb\x00\x00\x04\xb8\x00b\xf0\xd9\x00b\xf0\xd9\x00b\xf0\xd8\xff\xff\xff\xff\xb9\x15*p/\xa4*\x00\x9dZw\x01B\x87L\x8f\x08\x11\x89\xe0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0000changelog.d\x005406413267\n'
```
as the result of the debugging it is turned out that we are sending more data than expected, to have better error next time if we have any corruption of the `streaming_changelog_chunks` table
Reviewed By: StanislavGlebik
Differential Revision: D20763738
fbshipit-source-id: 6f6fa9f9a29909e044d9ba42fe84916ddcb62e8f
Summary: Each benchmark takes about 3 minutes to run. We've already got 16 benchmarks, and we're going to grow. Allow you to limit the number of benchmarks we run at once..
Reviewed By: ahornby
Differential Revision: D20735795
fbshipit-source-id: 241184085b35da8ab85314fef1c6a08404bdb769
Summary: We're going to be doing a variety of changes to sqlblob - let's enable working against a known baseline each time, instead of incremental changes.
Reviewed By: ahornby
Differential Revision: D20735796
fbshipit-source-id: 86f15dac1f004b2f3c83ced829a65f3f6e111d6b
Summary:
We want to be able to benchmark blobstore stacks to allow us to compare blobstore stacks, and ensure that we're not regressing as we improve SQLblob to production state.
Use Criterion to benchmark a few basic workflows - starting with writes, but reads can be added later.
Reviewed By: ahornby
Differential Revision: D20720789
fbshipit-source-id: e8b10664a9d08a1aa7e646e1ebde251bec0db991
Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them. The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.
This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.
Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.
Reviewed By: wez
Differential Revision: D20688290
fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0
Summary: Use the client identity middleware from gotham_ext in the EdenAPI server. This middleware parses validated client identities from HTTP headers inserted by Proxygen; these identities can then be used to enforce repo ACLs.
Reviewed By: HarveyHunt
Differential Revision: D20744887
fbshipit-source-id: 651e171d1b20448b3e99bfc938d118fb6dddea91
Summary:
It turns out that eden integration tests do not always set HGRCPATH for hg
commands it executes. The testing library (integration/lib/hgrepo.py) sets
HGRCPATH for hg commands it runs. However there are other code paths that
bypass the HGRCPATH setting. For example:
- hg debugimporthelper started by edenfs does not have HGRCPATH set.
- hg log -r . -T{node} started by fs/cli/util.py does not have HGRCPATH set.
That caused issues. For example, without HGRCPATH, the real system hgrc is
loaded, and that can be different from the in-repo hgrc, causing issues like
migrating "narrow-heads" down and up, which pollutes stderr and fails tests.
Increase test robustness by adding an `%include $HGRCPATH` in the repo hgrc
so commands without HGRCPATH still load the desired system hgrc.
Reviewed By: chadaustin
Differential Revision: D20769184
fbshipit-source-id: 8992bfb95250b6cccb9ebcf4bbaa0a76d50480dd
Summary:
This still requires support from EdenFS in order to do much
of use, but it takes us a step closer:
* Pull in cpptoml when building with Eden support
* On Windows, when we locate the `.eden` directory, load and parse
the config file in order to determine the socket path
* If the EdenView constructor throws, treat it as a terminal error
so that we don't fallback to the regular filesystem watcher.
This is important because current EdenFS builds don't implement
the journal thrift API endpoint yet.
Reviewed By: pkaush
Differential Revision: D20504752
fbshipit-source-id: 48bbad49f1641698aa7d7b85674e3ddf4d4e617d
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
Update the CMake build to emit a LIBRARY_DEP_DIRS.txt file in the build output
directory. This file should contain additional directories that may contain
runtime library dependencies for the generated build artifacts, one directory
per line. This file will be parsed by getdeps.py so that it can set `$PATH`
properly when running tests in order to find all of the necessary library
dependencies.
At the moment the runtime location of the Python DLL is the main thing that
needs to be stored in this directory. CMake can find the Python location from the
registry, even if that location is not in the user's `$PATH`. On its own
getdeps.py won't be able to find the location of the Python DLL unless we
explicitly give it this information through this file.
Reviewed By: wez
Differential Revision: D20688289
fbshipit-source-id: 078d07d860de8604c97bf6adede464c0d95fc079
Summary: Add a hint that points to customization and troubleshooting guides for the new graph renderer.
Reviewed By: xavierd
Differential Revision: D20763338
fbshipit-source-id: ee6d2464ae5955f0f0bf52d1994adfa2b74b3367
Summary:
Instead of using Sha256::from_slice, just use Sha256::from with a correctly
sized array.
Reviewed By: quark-zju
Differential Revision: D20756181
fbshipit-source-id: 17c869325025078e4c91a564fc57ac1d9345dd15
Summary: We are changing some of the internals of how the status is working. Few tests are broken and would need some work to get them working. Will enable them once all the Inode support is landed and the tests are fixed.
Reviewed By: wez
Differential Revision: D20602112
fbshipit-source-id: eadae818deaf62bcf0f9ec51d9dd531a6d452aa4
Summary:
Eden on Windows doesn't support setting a file as executable or creating symlinks.
Windows doesn't need executable mode bit to execute. It can execute the files with executable extension, or the responsible program can run it like Python3.exe can run python script.
Reviewed By: chadaustin
Differential Revision: D19956268
fbshipit-source-id: c22416db2a9da78e3a5c4392d1537eb7cbf9bfd0
Summary: Looks like it's not dead anymore
Reviewed By: krallin
Differential Revision: D20766497
fbshipit-source-id: c49ae3b6c8a660b33e61e65adda94f78addd1498
Summary:
It is preferable to use the higher-level API of cached_config instead of ConfigeratorAPI whenever possible since the higher-level API supports OSS builds.
For `ConfigStore` let `poll_interval` be None so that for one-off reading of configs the ConfigStore doesn't needlessly spawn an updating thread.
Also this update is with compliance to the discussion in D19026190.
Reviewed By: ahornby
Differential Revision: D20670224
fbshipit-source-id: 24fc124d440fd458a9fa88a906fc3a1cfdbd827e
Summary: In the process the blobstore/factory/lib.rs was split into submodules - this way it was easier to untangle the dependencies and refactor it, so I've left the split in this diff.
Reviewed By: markbt
Differential Revision: D20302068
fbshipit-source-id: caa3a2b5487c30198c62f7e4f4e9cb7c488dc8de
Summary:
Replace explicit casts working around T38947910 (Pyre not understanding
decorators) with `pyre-ignore` comments. This will hopefully help ensure that
these are tracked more visibly, and are removed when this task is fixed.
Reviewed By: wez
Differential Revision: D20434082
fbshipit-source-id: da4e4d11e4e029a11984c0efcad2aecd0d3094bf
Summary:
Now that the `eden` CLI code, Mononoke, and EdenFS are all available in the
same github repository, update the top-level README to describe all 3 of these
components.
Reviewed By: JoelMarcey
Differential Revision: D20688288
fbshipit-source-id: 40110ba05f16dedbcb6ab2a8e56eef7da16bb567
Summary:
Add a `build.bat` file for building on Windows, to parallel the `build.sh`
file for Linux + Mac systems.
Reviewed By: wez
Differential Revision: D20715564
fbshipit-source-id: 8e792d2d41baaf6366e8f259b5adf700cb07ad3a
Summary:
Replace Eden's old custom `getdeps.py` script with a simple `build.sh` wrapper
that invokes the modern `getdeps.py` tool.
Note that this build script was largely cribbed from the `getdeps.sh` script
in the [fboss](https://github.com/facebook/fboss/) repository.
Reviewed By: wez
Differential Revision: D20688291
fbshipit-source-id: 6f119479280e82a0c28b8b1481fa01611908c566
Summary:
Update `hg status` to print errors that were returned by EdenFS's
getScmStatusV2() call, and to exit unsuccessfully if there were any errors.
Previously errors were silently ignored.
Reviewed By: quark-zju
Differential Revision: D19958503
fbshipit-source-id: cb3109df40eb86a5bf7e3818ddfb8da74d670405
Summary:
Fix the `test_status()` function to properly canonicalize the input paths, so
that it works successfully on Windows. Previously this function would fail on
Windows as some paths would end up as UNC-style paths and some would be plain
paths, causing the equality comparison to fail even though the paths were
equivalent. This canonicalizes the repo path so that they both use the same
format.
Reviewed By: quark-zju
Differential Revision: D20662921
fbshipit-source-id: fdd36bac755f9694b4a482615d3dca43ff21e05e
