Commit Graph

55894 Commits

Author SHA1 Message Date
Durham Goode
430f047eda py3: fix flat dirstate parsing/packing
Summary: Updates the C code to support unicode filenames and states.

Reviewed By: simpkins

Differential Revision: D19786275

fbshipit-source-id: e7aeb029b792818b1b1a9c5d3028640b56522235
2020-02-26 12:53:25 -08:00
svcscm
febb8abb2b Updating submodules
Summary:
GitHub commits:

bfa19443c1
988d2da78c

Reviewed By: wittgenst

fbshipit-source-id: 92df36c60f07ae6c3ecce7e440699691fb3f610d
2020-02-26 12:53:24 -08:00
svcscm
2ca853452b Updating submodules
Summary:
GitHub commits:

6bdc921b4c
8be06739a3
78e76dd20c
d3445251df
cc1c0335f6
b4b967cab8
dac952e1ac
69f790c54f

Reviewed By: wittgenst

fbshipit-source-id: 24071451b97152f459ebd7e01e34d7035f86e9b6
2020-02-26 10:53:41 -08:00
Xavier Deguillard
76dd52a310 infinitepush: only open a transaction when deleting bookmarks
Summary: There is no need to open a transaction otherwise.

Reviewed By: DurhamG

Differential Revision: D20109840

fbshipit-source-id: e47adaaeea2d7565f3629701d8de4a67d4b55182
2020-02-26 10:27:05 -08:00
svcscm
f6d95a9977 Updating submodules
Summary:
GitHub commits:

deb715f095
8df25d6ea6
0cb581bc0b
c3015ed06a
0b257ca985
a2a3407d5a
00847c49c5
068f5ed66d
a396f62335

Reviewed By: wittgenst

fbshipit-source-id: d202148a2753754b41d1928f35ac6ad6bdcc1053
2020-02-26 09:57:02 -08:00
Durham Goode
f188acb4e0 recover: don't verify the repo
Summary:
Verifying the changelog is quite slow and we've had more users needing
to run hg recover these days. Let's finally get rid of the verify step.

Reviewed By: simpkins

Differential Revision: D20109706

fbshipit-source-id: a512d9e11716514bce986b0e3a26347fe6afd955
2020-02-26 09:07:08 -08:00
Aida Getoeva
8f09d5a51b hg-py3: fix the last amend commands
Summary: Most of the fixes related to encoding in `patch.py`

Reviewed By: DurhamG

Differential Revision: D19713378

fbshipit-source-id: 66ccbd0fc7826ab2d4c05173c7e9edb96700d106
2020-02-26 08:26:13 -08:00
Aida Getoeva
585899f419 mononoke/scs: use last change in file history
Summary:
There is no need to generate expensive file history stream if only one node is requested.

I refactored code that generated stream of history commits, so it'd first yield the nodes and only then prefetch their parents. That will help to solve latency problem for the history request for only a single commit.

I removed BFS queue and added two state variables: ready nodes and already processed:
* The last are the nodes that were return as a part of a history stream on the last iteration and now can be used to construct next BFS layer: prefetch fastlog batches, fill the commit graph, take parents in BFS order to form new bunch of nodes.
* First are used if it's the first iteration - there is no processed nodes yet but there are some that are ready to be returned.

I believe removing the queue I simplified the code and logic a little bit.

Reviewed By: StanislavGlebik

Differential Revision: D19818100

fbshipit-source-id: c30d28c623464ba3552a00e8542552f7655076ef
2020-02-26 08:09:12 -08:00
Alex Hornby
04e011525a mononoke: walker: test validate scuba logging for non-public commits
Summary: add test for scuba logging for non-public commits

Reviewed By: StanislavGlebik

Differential Revision: D20093721

fbshipit-source-id: eb0792bcae8ea27c11709181390efb0ac0c817ee
2020-02-26 06:16:29 -08:00
svcscm
a8595b791e Updating submodules
Summary:
GitHub commits:

10d0852e5a
2c04d99cda

Reviewed By: wittgenst

fbshipit-source-id: e92cad277ecee5fa958a29ebfe19f623cd676e0b
2020-02-26 05:14:09 -08:00
Stanislau Hlebik
7076fac933 mononoke: add exponential backoff
Summary:
During our tests we noticed that we can send too many blobstore read requests to the
mapping. Let's add exponential backoff to prevent that

Reviewed By: ikostia

Differential Revision: D20116043

fbshipit-source-id: 6fecbda4c36a5065b77ba9df561c6d9c6a969089
2020-02-26 05:05:33 -08:00
Thomas Orozco
4ca1333b8a mononoke/hooks: use a smaller test group for faster tests
Reviewed By: ikostia

Differential Revision: D20115985

fbshipit-source-id: 4f69fc84eee352bcc689918527c6d460fcf672ba
2020-02-26 04:44:39 -08:00
Thomas Orozco
c14a88bbef mononoke: convert places that talk to Memcache to Bytes 0.5
Summary:
Memcache doesn't care (because both old and new Bytes to `Into<IOBuf>`), but
Thrift is Bytes 0.5. We have our caching ext layer in the middle, which wants
Bytes 0.4. This means we end up copying things we don't need to copy.

Let's update to fewer copies. I didn't update apiserver, because a) it's going
away, and b) those bytes go into Actix, and Actix isn't upgrading to Bytes 0.5
any time soon! Besides, this doesn't actually need updating besides tests anyway.

Reviewed By: dtolnay

Differential Revision: D20006062

fbshipit-source-id: 42766363a0ff8494f18349bcc822b5238e1ec0cd
2020-02-26 03:30:47 -08:00
svcscm
288e74f809 Updating submodules
Summary:
GitHub commits:

ced43fb76b
c173f5d7e6

Reviewed By: wittgenst

fbshipit-source-id: 6564fe3c3220bd3bea86ed08a8c1b0c1ad99adfb
2020-02-26 03:30:47 -08:00
svcscm
a43ebde766 Updating submodules
Summary:
GitHub commits:

4e3000a470
85117a7339

Reviewed By: wittgenst

fbshipit-source-id: 87a8eb75422ee6f4b8eacfcca84a475525cf12b9
2020-02-25 21:28:24 -08:00
Adam Simpkins
08f86af0a4 enable strict type checking in telemetry.py
Summary: Enable `pyre-strict` mode in eden/cli/telemetry.py

Reviewed By: genevievehelsel

Differential Revision: D20102260

fbshipit-source-id: 0e5030f99852eb07dc427ba80cc30334adea4bfb
2020-02-25 19:01:10 -08:00
Adam Simpkins
f0cf7fec98 update the telemetry wrapper to log the current code version
Summary:
Add methods to `version.py` to get the version of the current running Eden CLI
code, rather than looking for the current installed RPM version.  This means
that we no longer have to execute a separate subprocess that examines the RPM
database.  This also makes sure we log the correct version information in
cases where developers are testing local development code even though they
have a different RPM version currently installed.

Reviewed By: genevievehelsel

Differential Revision: D20102259

fbshipit-source-id: ba9eb0c563c7f7c929170b130566946a67f679a5
2020-02-25 19:01:10 -08:00
Adam Simpkins
9ee7b23604 update RPM version code to return Optional[Tuple[str, str]]
Summary:
Update `get_installed_eden_rpm_version_parts()` to simplify the return type
from `Tuple[Optional[str], Optional[str]]` to `Optional[Tuple[str, str]]`

This also improves the output of `get_installed_eden_rpm_version()` when the
RPM is not installed so that it returns `<Not Installed>` rather than
`<Not Installed>-` with a trailing dash.

Additionally this updates the telemetry logging to include the full
version+release string.  With our current version number scheme there can be
multiple packages with the same version but different release numbers if we
release multiple packages within a single day.

Reviewed By: genevievehelsel

Differential Revision: D20102263

fbshipit-source-id: 24d2df4cdca6ac576267be66b85422c3e50f1229
2020-02-25 19:01:09 -08:00
Adam Simpkins
4ee1a29578 move code to get the running EdenFS version to EdenInstance
Summary:
Move the `get_running_eden_version()` functions from the `version.py` module
into the `EdenInstance` class in `config.py`.  This helps eliminate some
circular dependency cycles in the code, so I can start breaking a few modules
out of the main CLI `lib` library.

I also changed the return type of `get_running_version_parts()` from
`Tuple[Optional[str], Optional[str]]` to just `Tuple[str, str]`.  A dev build
of EdenFS already returns empty strings (rather than `None`) for the version
and release fields).  There shouldn't really be any cases where `None` is
returned here, and even if there were I don't think we would ever care to
distinguish this from the empty string case.

Reviewed By: genevievehelsel

Differential Revision: D20102262

fbshipit-source-id: 564ec5ee820026a0c86c70ad0d7cfd3750ad94f5
2020-02-25 19:01:09 -08:00
svcscm
e9a197f19c Updating submodules
Summary:
GitHub commits:

898bbb7a72
05105ff4f9
69679e7375
ab1825a965

Reviewed By: wittgenst

fbshipit-source-id: 86cb59d667e3fe8d293740f5cefbec998879604f
2020-02-25 16:39:54 -08:00
Genevieve Helsel
35c8305d13 scuba logging eden full restart
Summary: Log when a user runs a normal (full) restart, including success or not. Success is determined by the return code of `start_daemon()` (which calls `subprocess.call()`), similar to the success critera for graceful restart logging

Reviewed By: fanzeyi

Differential Revision: D20098949

fbshipit-source-id: 0c6f4927571f686ed6b678d5c814f76c78322274
2020-02-25 15:31:44 -08:00
Genevieve Helsel
0c908acc0d scuba logging eden doctor calls
Summary: log when a user runs eden doctor, and log how many errors they encounter

Reviewed By: fanzeyi

Differential Revision: D20084617

fbshipit-source-id: 122a062c538931eb906cbfcd515ec1e8093efc38
2020-02-25 15:31:43 -08:00
Genevieve Helsel
88851bc88d add no-op logging to FakeEdenInstance
Summary: This is required for eden doctor cli tests when adding logging to the eden doctor code path. This can just be a stub since we don't consume these scuba log statements during testing

Reviewed By: fanzeyi

Differential Revision: D20087861

fbshipit-source-id: 6805ae8d9c51e33a118cbda76461483962e876f3
2020-02-25 15:31:43 -08:00
Genevieve Helsel
0528daf796 add type annotation in check_filesystems
Summary: the TypeCheck test cases were yelling at me because of this annotation missing when running locally, so adding it to fix those tests.

Reviewed By: fanzeyi

Differential Revision: D20098619

fbshipit-source-id: 630e7bca2b63033b34d72d1c739184819d3d86a3
2020-02-25 15:31:43 -08:00
svcscm
0687e287b5 Updating submodules
Summary:
GitHub commits:

63d131bff1
82909d517a
57774941bd
e3f2681e45
a365382473
851123919a
63899b41af

Reviewed By: wittgenst

fbshipit-source-id: 08c0b0414b0def0d0a04ed80fbf9216a18373262
2020-02-25 15:31:42 -08:00
svcscm
e80f942895 Updating submodules
Summary:
GitHub commits:

484cd048cc
331b273040
144741c29a

Reviewed By: wittgenst

fbshipit-source-id: 3180b37c0490c10b0bf4f3c980c4a22fab1980c8
2020-02-25 11:01:54 -08:00
Jeff Zhang
33140b117c Push compat down one level in eden/mononoke/cmds/admin/main.rs
Summary: Moving `compat` one level down to the call sites of subcommand functions.

Reviewed By: farnz

Differential Revision: D20085398

fbshipit-source-id: 461e147d2ae6e560b3a75fb92fa6b23f9f54d13e
2020-02-25 10:22:03 -08:00
Zeyi (Rice) Fan
2222dbc1a5 fix HgPrefetchTest
Summary:
The problem is that the datapack files are not flushed to disk when it is prefetched. By having a pair of brackets around the `HgBackingStore`, it will ensure the `HgImporter` is closed by the time when we verify the prefetch with `hg cat` since it will terminate the `debugedenimporthelper` process in its destructor, which flushes the datapack files.

The real cause of the test failure is still unclear but I believe this is the correct way of doing this test.

Reviewed By: xavierd

Differential Revision: D20090249

fbshipit-source-id: 8e3966936a402c92311919433282027846d065e8
2020-02-25 10:14:29 -08:00
Puneet Kaushik
2b19eb7c17 Define directory types for Windows
Summary: Windows SDK doesn't define dirent. Defining it here for adding Inodes support on Edenfs on Windows.

Reviewed By: simpkins

Differential Revision: D19956272

fbshipit-source-id: 1bdf9a7563c194fe38008741b09668242ffa64ee
2020-02-25 10:14:29 -08:00
Puneet Kaushik
ca40c6f0f4 Update log level and remove async
Summary:
Logging on Windows doesn't work when the async is set. We haven't debugged it yet. Removing the async mode flag until we fix that.

Also bumping up the log level to 4. This would help to get more info while we are running in beta.

Reviewed By: simpkins

Differential Revision: D19776609

fbshipit-source-id: ccd6a6ed4d81f4a2edd550c6bb7195ac8b8b4d16
2020-02-25 10:14:28 -08:00
Stanislau Hlebik
19e1e94984 mononoke: add lease renewing to derived data
Summary:
During S196197 lease expired and we were rederiving the same derived data over and over again for a big commit.
this diff adds lease renewal that should help with this problem.

Reviewed By: HarveyHunt

Differential Revision: D20093323

fbshipit-source-id: d139abf6659722f47ea40d9b2f279daa03623ff4
2020-02-25 09:22:46 -08:00
Stanislau Hlebik
4bd758289b mononoke: async/await derive_may_panic() function
Reviewed By: HarveyHunt

Differential Revision: D20092945

fbshipit-source-id: 70ec1a8e5b9c99f3853a13bebe3657ece5ff9e9e
2020-02-25 09:22:46 -08:00
svcscm
746869b82c Updating submodules
Summary:
GitHub commits:

97c7ab4212

Reviewed By: wittgenst

fbshipit-source-id: 9d204b092999e263ae4ff226d9c6fa4d19d4f832
2020-02-25 09:12:36 -08:00
Genevieve Helsel
887de5105d scuba log eden rage calls
Summary: log when a user runs eden rage

Reviewed By: simpkins

Differential Revision: D20084529

fbshipit-source-id: a92c5472554cd541c9a7d340edcf6845c1c9c0c0
2020-02-25 08:11:28 -08:00
Stanislau Hlebik
3418318883 mononoke: do not generate hgchangesets unnecessarily in FilenodesOnlyPublicMapping
Summary:
fetch_root_filenode is called by FilenodesOnlyPublicMapping to figure out if
filenodes were already derived. Previously it first derived hg changeset and
then fetched looked up root manifest in db. However if hg changeset is not
derived then filenodes couldn't possible be derived either and we can return an
answer faster.

This is useful in the next diff where I change walker

Reviewed By: ahornby

Differential Revision: D20068819

fbshipit-source-id: 17f066c437e0b1f7bbeb8f6e247eadc9afe94f90
2020-02-25 08:07:07 -08:00
Thomas Orozco
f8fcbc9723 mononoke/blobstore_healer: wait for MyRouter properly
Summary:
The blobstore_healer has never waited for MyRouter before querying for slave
status, but it ended up implicitly working because creating a blobstore
required a SQL factory, and creating a SQL factory would result in waiting for
MyRouter.

Now that creating a blobstore doesn't require SQL factory unless you're going
to actually use it (which the healer isn't: it doesn't use a multiplexblob, it
uses the underlying blobstores instead), we no longer wait properly for
MyRouter, so if MyRouter isn't there when we boot, we crash.

This fixes that.

Reviewed By: ahornby

Differential Revision: D20094829

fbshipit-source-id: 82b7e8d893a01049d1f434ee8dff36a877a0d2f4
2020-02-25 07:03:28 -08:00
Alex Hornby
693e8dee0a mononoke: walker: add support for loading by GitSha1 Aliases
Summary:
Add support for loading by GitSha1 Aliases.  This relies on the change to
Alias::GitSha1 earlier in stack.

Reviewed By: ikostia

Differential Revision: D19903577

fbshipit-source-id: 73cdccc04af61fa524c3683851d8af9ae90d31dc
2020-02-25 03:36:06 -08:00
svcscm
98632c3ab3 Updating submodules
Summary:
GitHub commits:

476964303e
6bd0c1c502
d87c10c6ab
bcd6b1986d

Reviewed By: wittgenst

fbshipit-source-id: 085d10bab0f7b3893a3a6b131f4ac72afb8b88b4
2020-02-24 20:16:51 -08:00
Adam Simpkins
ef04ccf546 replace a bunch of pyre-fixme comments with pyre-ignoree
Summary:
D17135557 added a bunch of `pyre-fixme` comments to the EdenFS integration
tests for cases where Pyre cannot detect that some attributes are initialized
by the test case `setUp()` method.

It looks like Pyre's handling of `setUp()` is somewhat incorrect: it looks
like if a class has a `setUp()` method this currently suppresses all
uninitialized attribute errors (even if some attributes really are never
initialized).  However, Pyre does not detect `setUp()` methods inherited from
parent classes, and always warns about uninitialized attributes in this case
even they are initialized.

Lets change these comments from `pyre-fixme` to `pyre-ignore` since this
appears to be an issue with Pyre rather than with this code.  T62487924 is
open to track adding support for annotating custom constructor methods, which
might help here.  I've also posted in Pyre Q&A about incorrect handling of
`setUp()` in derived classes.

Reviewed By: grievejia

Differential Revision: D19963118

fbshipit-source-id: 9fd13fc8665367e0780f871a5a0d9a8fe50cc687
2020-02-24 18:55:19 -08:00
svcscm
933157e61f Updating submodules
Summary:
GitHub commits:

e5ea58f283
eed5a89eed
08db16e3cb
9d9e30d2f8

Reviewed By: wittgenst

fbshipit-source-id: 2c7599f689e033b67def4185b3b4726cf2b5b63c
2020-02-24 18:17:06 -08:00
Michael Devine
69e9601f71 Refactor convert repo into directory
Summary: As I work, it's getting harder and harder to keep my multiple changes from introducing merge conflicts between different branches. We need to break out the repo_source's implementation in to a bunch of different files to make it easier to keep things separate.

Reviewed By: zhonglowu, tchebb

Differential Revision: D20015946

fbshipit-source-id: bf954ac581e5ca9e43c091b6b1b4c539c14471f2
2020-02-24 18:07:11 -08:00
svcscm
2a9bd47507 Updating submodules
Summary:
GitHub commits:

ea8bae1f0f
134472ee45
37e6cf9d62
eb367d45c0
76de6e15c0
e1b1a55309

Reviewed By: wittgenst

fbshipit-source-id: 9d0d688d81be822900475223a787c5649e143e85
2020-02-24 17:34:07 -08:00
generatedunixname89002005287564
d801a85055 eden/integration/persistence_test.py
Reviewed By: simpkins

Differential Revision: D19995899

fbshipit-source-id: 28cf25cb5a4cde8b15f8a4f3199aaa249aade2a3
2020-02-24 15:42:45 -08:00
Adam Simpkins
b22fc79e4b clean up PathRelativizer API usage of Path vs PathBuf
Summary:
Fix the PathRelativizer APIs to accept `Path` and even `str` arguments instead
of just `PathBuf`.  The old code required a `PathBuf`, which often forced
callers to make a copy of the path data.

Reviewed By: quark-zju

Differential Revision: D19958505

fbshipit-source-id: 6fa40dd4b75df4e3faf9ad2ae4f0e4e6595669f6
2020-02-24 15:38:36 -08:00
Thomas Orozco
2a12e2beb6 mononoke/derived_data: log when we start deriving
Summary:
This should give us a slightly better idea of what hosts are doing to
troubleshoot duplicate derivation.

Also, let's make the logging a bit less confusing.

Reviewed By: StanislavGlebik

Differential Revision: D20070619

fbshipit-source-id: 91cc264b7043b8fc8c21c007832fba328ef0017d
2020-02-24 12:03:41 -08:00
svcscm
958c753391 Updating submodules
Summary:
GitHub commits:

f90a29b9a1
027453773e
0ffb23f1fd
8bf7938808
833f9d4b1a
eab7a0e4e3
be3daaf876
b01da6ae23
ba4a00006a

Reviewed By: wittgenst

fbshipit-source-id: 4191ea08dd3566dc5d39586cec1d5401f942b5e1
2020-02-24 11:58:08 -08:00
Thomas Orozco
b3bebee0b4 mononoke: include DB config in multiplexed blobstore configuration
Summary:
This updates our multiplexed blobstore configuration to carry its own DB
config. The upshot of this change is that we can move the blobstore sync queue
(a fairly unruly table) to its own DB.

Another nice side effect of this is that it cleans up a bunch of other code, by
finally decoupling the blobstore config from the DB config. For examples,
places that need to instantiate a blobstore can now to do even without a DB
config (such as wireproto logging).

Obviously, this cannot land until we update the configs to include this. I'll
do so in Configerator prior to landing the diff.

Reviewed By: HarveyHunt

Differential Revision: D19973905

fbshipit-source-id: 79e4ff92cdb989aab4532decd3fe4fd6c55e2bb2
2020-02-24 11:54:45 -08:00
Thomas Orozco
b7185f0f13 mononoke/metaconfig: tidy up blobstore creation
Summary:
I'd like to refactor our multiplex blob to store its DB using a different
shard. In preparation of doing so, let's:

- Extract parsing DB configs from storage configs
- Tidy up some related places that take a reference when they actually need
  ownership (which is sort of wasteful).

Reviewed By: StanislavGlebik

Differential Revision: D19973906

fbshipit-source-id: 82baceb892e9e24e5fd0349ffa5503884c177a7a
2020-02-24 11:54:44 -08:00
Adam Simpkins
8c9899a197 reduce the glog logging level to info (1)
Summary:
Most of EdenFS's main logging is done through folly::logging, however a number
of libraries that we use do logging through glog.  Previously we set glog's
`--minloglevel` setting to `0`, and we use the default `--v=0` setting.
This enabled glog `VLOG` messages, only for at VLOG level `0` messages.

Now that the Rust backing store code can fetch directly from memcache this now
links in some additional memcache library code that has some `VLOG(0)`
messages that are logged fairly frequently.  These aren't useful for us to
have in our logs, so reduce the `minloglevel` to `1` for now, which disables
all `VLOG` messages.

Reviewed By: genevievehelsel

Differential Revision: D20050589

fbshipit-source-id: 167e301d61e46ae3c19975e0c9233eda371495c0
2020-02-24 11:34:01 -08:00
Xavier Deguillard
401d44916b add lfs_protocol to autocargo
Summary: Now it no longer depends on mononoke_types, we can build it with cargo

Reviewed By: krallin

Differential Revision: D20070438

fbshipit-source-id: 1b2f9cc3640c58fd38e962c7c738d08cbb22a71d
2020-02-24 11:12:45 -08:00