Commit Graph

51355 Commits

Author SHA1 Message Date
Zeyi (Rice) Fan
2f15f80957 revisionstore: make EdenApiRemoteStore to be able to fetch trees from remote
Summary: This will allow us to use EdenApi in EdenFS to fetch trees and blobs.

Reviewed By: xavierd

Differential Revision: D18622844

fbshipit-source-id: 59a9091e9f2fdbcae078da2fb24ee9c0dd18505b
2019-12-10 13:40:55 -08:00
Zeyi (Rice) Fan
ae1dae6b96 eden: add experimental:use-edenapi for optionally turn on EdenApi importing
Summary:
Add an option `experimental:use-edenapi` to `EdenConfig`.

See the next diff for usage.

Reviewed By: chadaustin

Differential Revision: D18605549

fbshipit-source-id: 2786c21bb38a76229078662cc5c1ddf906d1be4a
2019-12-10 13:40:55 -08:00
Zeyi (Rice) Fan
ed9e7abca2 eden: add FakeEdenConfig for testing
Summary:
I think we need to make `EdenConfig` available for testing as more things are using `EdenConfig` to configure behaviors.

Right now it does not do much things other than just construct an `EdenConfig` instance without manually providing these parameters.

Reviewed By: chadaustin

Differential Revision: D18798484

fbshipit-source-id: b4a85d966a9b4f425c547bef9eb6e7570d7e2170
2019-12-10 13:40:55 -08:00
svcscm
4fe7ad486c Updating submodules
Summary:
GitHub commits:

b459fcc89f
2b060c1498
13a2c072c4

Reviewed By: zpao

fbshipit-source-id: 59fb11a977dcb7b2c09acb7fe997b0d5e52f27c4
2019-12-10 12:50:56 -08:00
Xavier Deguillard
d21a108280 tests: fix test-atomictempfile.py
Summary:
os.stat uses float values, use util.stat instead.

(Note: this ignores all push blocking failures!)

Reviewed By: singhsrb

Differential Revision: D18913982

fbshipit-source-id: 2df5ad0efbba3b18ef90669b4d1910ab39730e18
2019-12-10 11:57:12 -08:00
svcscm
8df1bf08d1 Updating submodules
Summary:
GitHub commits:

fd8771904e
6bf51e234f
6380df5e10
696c2a2359

Reviewed By: zpao

fbshipit-source-id: 188670fcdc50ccf060eea137698ecfb45484e059
2019-12-10 11:23:05 -08:00
Chad Austin
b47bf0289e ensure fuse capability flags survive graceful restart
Summary:
Two bugs conspired to cause edenfs after a graceful restart to think
the kernel supported FUSE_NO_OPENDIR_SUPPORT when it didn't: the
connection info struct wasn't zeroed, and FUSE connection capabilities
weren't properly mirrored into the Dispatcher upon graceful
restart. Fix both and add an integration test.

Reviewed By: simpkins

Differential Revision: D18903761

fbshipit-source-id: 23f4db3e240ee7d035f707820072c606a45f1138
2019-12-10 10:25:30 -08:00
Stefan Filip
1d61f2dbfd manifest: add TODOs with ideas about improvements on Manifest
Summary: Ideas for improvements. Put them in the code for future consideration.

Reviewed By: quark-zju

Differential Revision: D18870140

fbshipit-source-id: a2a84d94d72303adc64614b06e36588ff7778aab
2019-12-10 10:18:19 -08:00
Stefan Filip
ef220f3f7c manifest: add the list method to the Manifest trait
Summary:
The list functionality is required by EdenFs. We want this functionality
to be well supported by the Manifest.

Reviewed By: quark-zju

Differential Revision: D18870143

fbshipit-source-id: 1ebaa713ff521226e6ace22cbd35cc841d967298
2019-12-10 10:18:19 -08:00
Stefan Filip
67b0e3a600 manifest: rename FsNode to FsNodeMetadata
Summary:
Consistency in naming. The general idea is to have Metadata types that don't
contain paths. Then we will have File, Directory and eventually FsNode that
will contain paths.

Reviewed By: quark-zju

Differential Revision: D18870141

fbshipit-source-id: a1f09add7f1c3dd4fa0348693cd3ce2fd5767fa7
2019-12-10 10:18:18 -08:00
Stefan Filip
05aa630a18 manifest-tree: rename Tree to TreeManifest
Summary:
This rename is going to make it easier to import and use outside of
manifest specific crates.

Reviewed By: quark-zju

Differential Revision: D18870142

fbshipit-source-id: 2f3ea460170308162ee834efc038b2dcedd9e233
2019-12-10 10:18:18 -08:00
svcscm
2cdda7f4d8 Updating submodules
Summary:
GitHub commits:

e63b40cb4b
cb5f0670a6
470a664def
6e8f70b2d9
0fb026ca58
3595e0cf38
79b171ffa3
fb5322d98d
cd48fc606b

Reviewed By: zpao

fbshipit-source-id: 99bee659ea0fca0247d67d2dac12a821e1bd402d
2019-12-10 10:18:18 -08:00
svcscm
b774054dee Updating submodules
Summary:
GitHub commits:

6c87dc4d3c
5ec43afc1d
1e3cb8283f
3af1c72471
dc8e6e6e68
405e596d50
f40ae54a52
479a143912

Reviewed By: zpao

fbshipit-source-id: 932226ea90129f6c337d9fe833e2c3e4e1f955ee
2019-12-10 05:58:56 -08:00
Lukasz Piatkowski
d2520b62e2 Fixes for travic CI build (#3)
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3

Reviewed By: krallin

Differential Revision: D18905608

Pulled By: lukaspiatkowski

fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
2019-12-10 04:56:56 -08:00
svcscm
49c3720863 Updating submodules
Summary:
GitHub commits:

aab1edd0df
ed15255982

Reviewed By: zpao

fbshipit-source-id: 613c5af2a8601afbd65c11eef8c8087d95420942
2019-12-10 04:56:56 -08:00
svcscm
f887d792a7 Updating submodules
Summary:
GitHub commits:

1e7ac08e2a

Reviewed By: zpao

fbshipit-source-id: 8fcf7b1799d6c639178a375d0306363662883209
2019-12-09 20:58:11 -08:00
Jun Wu
2781801365 doctor: attempt to fix mutationstore, metalog and allheads
Summary:
Since we got `repair` APIs in the Python land, use them to attempt to repair
things.

This would hopefully be able to fix some user-reported issues, usually
caused by hard reboots.

Reviewed By: xavierd

Differential Revision: D18737907

fbshipit-source-id: ef3deccbf4a1c959cff7e3eb78771acd9ae9bd01
2019-12-09 20:02:09 -08:00
Chad Austin
40e6e4bde2 enable eden debug logging in open source and macOS builds
Summary:
Register the logging and logging_full option handlers in the open
source and macOS builds of Eden.

Reviewed By: simpkins

Differential Revision: D18870545

fbshipit-source-id: 140debe77be071a9fa2d628ad305517273e095e3
2019-12-09 19:41:18 -08:00
svcscm
ba49b7f7f6 Updating submodules
Summary:
GitHub commits:

b221a6b5bb

Reviewed By: zpao

fbshipit-source-id: 6dd4252a106f8c6f6cb35613a1d10d346006d8a1
2019-12-09 18:22:40 -08:00
Stefan Filip
ef336812a7 hggit: update dulwich usage to match 0.19+
Summary:
From Dulwich changelog:
0.19.0 2018-03-10
API CHANGES
GitClient.send_pack now accepts a generate_pack_data rather than a
generate_pack_contents function for performance reasons.

(Note: this ignores all push blocking failures!)

Reviewed By: quark-zju

Differential Revision: D18901269

fbshipit-source-id: 9510b3a516cb2fcbeae9f1f2072d8445e442a9ed
2019-12-09 17:59:14 -08:00
svcscm
61a9c9c6f8 Updating submodules
Summary:
GitHub commits:

8b2d8468c9
840bcfbdb8
0a702dcd44
e45e35f7e1

Reviewed By: zpao

fbshipit-source-id: 524ed22a0723ac5b669ef4c1b00b6b4e511a2d58
2019-12-09 17:59:14 -08:00
svcscm
d2999e8999 Updating submodules
Summary:
GitHub commits:

a072a44b4d
59c0f06905
c39a3e42b8
a68dff5c35
f4b1e6532c
d0790d3a38

Reviewed By: zpao

fbshipit-source-id: 29e7660c1d2981dcd61821dc79c8cc088f96ae5c
2019-12-09 16:46:21 -08:00
Genevieve Helsel
504a255355 add getTreeForManifest
Summary: Adds a function which takes both the manifestID and the commitID to get a Tree. This will be used in `checkOutRevision()` and this allows us to skip looking up the manifestID since the caller can just pass it in themselves.

Reviewed By: wez

Differential Revision: D18719405

fbshipit-source-id: 919f0a7c84bff4a2f0bc20110c45bd272f9e9107
2019-12-09 16:25:27 -08:00
Chad Austin
4963352d64 temporarily mark trees persistent
Summary:
It is very painful for Eden's automatic GC to delete trees from the
local store. An unfortunate number of programs scan the entire repo,
causing every tree to be loaded. If Eden's automatic GC kicks in, it
causes all trees to be downloaded again, blocking both FUSE and Thrift
requests.

Temporarily mark trees as persistent.

Reviewed By: fanzeyi

Differential Revision: D18842066

fbshipit-source-id: 4a22defcb1151912337d4167ac49852519c7789d
2019-12-09 16:11:36 -08:00
Jun Wu
a3129104fe tests: fix buck run :run_tests
Summary:
`buck run :run_tests -- ...` was added to be a buck version of `run-tests.py`.
However it does not work because the watchman path provided was invalided.

Reviewed By: xavierd

Differential Revision: D18892361

fbshipit-source-id: 8745a33e0d19b14f042ef1cd255d10c5737dffb1
2019-12-09 15:26:30 -08:00
Jun Wu
75a8173a10 tests: enable chg for 572 tests
Summary:
Add `#chg-compatible` to 572 tests that seem to pass with chg enabled.
This should make them run faster.

Reviewed By: xavierd

Differential Revision: D18870507

fbshipit-source-id: fe895e733efffc9286cd3d17c7a156c803124395
2019-12-09 15:26:29 -08:00
Jun Wu
84a4da85ca metalog: implement bi-directional migration
Summary:
Previously, it's fine to migrate up to metalog, and fine to migrate down once
since we double write data. However, re-enabling metalog can be problematic
since there is no code path to do "migrate up" again.

This diff fixes the issue by tracking what files (or "keys") are using metalog
as the source of truth in metalog. So we can accurately figure out whether to
migrate svfs files to metalog on demand.

Reviewed By: xavierd

Differential Revision: D18864424

fbshipit-source-id: e61e1790c231f9c88de869f413f27bb954a29920
2019-12-09 14:18:18 -08:00
Jun Wu
d754747d01 bindings: add repair to structures
Summary: Expose the repair API to the Python world.

Reviewed By: xavierd

Differential Revision: D18737913

fbshipit-source-id: c31085727589b6938c2fafb28897925aea617bc4
2019-12-09 14:15:47 -08:00
Jun Wu
02b4902319 metalog: implement Repair
Summary:
Provide a way to repair the MetaLog. It's not just repairing 2 indexedlog
structures, but also checking the relationships between them.

Reviewed By: xavierd

Differential Revision: D18737904

fbshipit-source-id: fcf8ae56a1fdbb0561765701d962dfad4a8b5bd4
2019-12-09 14:15:47 -08:00
svcscm
039e5a1f9d Updating submodules
Summary:
GitHub commits:

aba4498a76
3fa5699d02

Reviewed By: zpao

fbshipit-source-id: 0109e104147cf3eeff04bcc79e72542a1bb89056
2019-12-09 14:15:47 -08:00
svcscm
fa464c1e21 Updating submodules
Summary:
GitHub commits:

cfa22893ac

Reviewed By: zpao

fbshipit-source-id: 2b1b1ec6c532eef5f54c04fc50a57f1644972524
2019-12-09 13:39:34 -08:00
Jun Wu
083bf3f1c3 run-tests: support '#chg-compatible' header
Summary:
Detect `#chg-compatible` in test header and force enable chg. This would allow
us to mark tests as chg-compatible to get some speed ups.

Reviewed By: xavierd

Differential Revision: D18870508

fbshipit-source-id: 923d6cb10b0c621d9995809b60ae535126f6f2a5
2019-12-09 13:20:25 -08:00
Jun Wu
fb13bc5ecf run-tests: make --chg work again
Summary:
The chg logic is inlined in the "hg" binary so the old logic running "chg" as
an external binary needs to be changed.

The version check part can be annoying - it requires more frequent `make local`
to run the tests while `make local` is actually not necessary. So let's just
skip the version check if running in tests.

Reviewed By: ahornby

Differential Revision: D18870510

fbshipit-source-id: 2343549c4080a0425098698b5423c6cfc1f60c77
2019-12-09 13:20:24 -08:00
Xavier Deguillard
82082bfaf8 policy: simplify it
Summary:
We only support the "C" policy, therefore let's inline it and simplify the code
significantly.

Reviewed By: quark-zju

Differential Revision: D18819680

fbshipit-source-id: 0bc0a12b85ddc42af007d3a20c0f4e8497bedb21
2019-12-09 13:13:27 -08:00
Xavier Deguillard
24d99a9178 util: remove os.stat_float_times(False)
Summary:
This is not available on Python3, instead stat.ST_MTIME and friends should be
used to get an integer time. This is achieved by wrapping the output of the
stat function and overriding the st_[acm]time properties.

Reviewed By: quark-zju

Differential Revision: D18819679

fbshipit-source-id: 2911f3d47506456e927872d57f69253d903618e2
2019-12-09 13:13:26 -08:00
Jun Wu
6da344eee2 hgcommands: avoid forked processes to override trace output
Summary:
The environment variable `EDENSCM_TRACE_OUTPUT` specifies where to write the
tracing output for the current command. Environment variables are inherited
by subprocesses by default. That is undesirable because another hg command
(triggered by hook, or background maintaince) will rewrite the trace output.

Avoid it by unsetting EDENSCM_TRACE_OUTPUT for subprocesses.

Reviewed By: xavierd

Differential Revision: D18892973

fbshipit-source-id: 575b6c0df2e7a0775172893e4aa72ca33fa4658c
2019-12-09 13:09:25 -08:00
svcscm
60ce49ea9e Updating submodules
Summary:
GitHub commits:

0f96b98cec
8090b337a4
e43d2c4424
70d1c268bf
fc6140865b
4caba2ed65

Reviewed By: zpao

fbshipit-source-id: 5b4edf4267942ab0cbd2980dc500227e3ce353e3
2019-12-09 13:02:00 -08:00
Durham Goode
dd6f76863b lfs: use unfiltered repo for lfs gc
Summary:
Some users were reporting hg gc failures because it couldn't access a
hidden commit.  It looks like there's a `repo.unfiltered().revs("draft()")`
line to get the list of draft commits, so we need to also use an unfiltered repo
when extracting all the pointers.

Reviewed By: quark-zju

Differential Revision: D18863471

fbshipit-source-id: 61220c1ec3dcbf55879211bb271b4d9b686d66a1
2019-12-09 12:34:22 -08:00
Durham Goode
ad43fdcaf6 fastlog: fix fastlog directory histories
Summary:
D18732257 changed our fastlog integration from using conduit to using
scmquery. This changed the result type from a hash to a dictionary of {"hash":
"xxx"}, but the consumer code wasn't updated appropriately. This fixes it.

Reviewed By: mitrandir77

Differential Revision: D18833566

fbshipit-source-id: 2573cbf4f93c5e4aeb2515d42696526ceb51428e
2019-12-09 12:32:07 -08:00
svcscm
3799971fb1 Updating submodules
Summary:
GitHub commits:

a568051d01
b575961859
7e2f831924

Reviewed By: zpao

fbshipit-source-id: b18f507d9951b6b23e9514aa40cc8e5f4df97ed8
2019-12-09 11:43:08 -08:00
Genevieve Helsel
0907d3adda save loadFileFromPath in DiffContext
Summary: Saves loadFileFromPath in the DiffContext. This allows the source control differ ( at `eden/fs/store/Diff.cpp`) to call this function while it is trying to load gitIgnores and resolve symlinks.

Reviewed By: simpkins

Differential Revision: D18647088

fbshipit-source-id: bddff0158130f0c96ac263bcce81dc51fc017800
2019-12-09 11:33:23 -08:00
Genevieve Helsel
c8bdf32bc9 introduce EdenMount::loadFileContentsFromPath
Summary: Adds a helper function that can take a path to an Inode and load a file by simply getting the Inode then calling the existing `loadFile()` function

Reviewed By: simpkins

Differential Revision: D18647090

fbshipit-source-id: 89c25bef10a7a0d0ffd660293f85479ac0e9f9a9
2019-12-09 11:33:23 -08:00
Genevieve Helsel
32fb62cedf move gitignore loading to EdenMount as EdenMount::loadFileContents
Summary: Move the bulk of the gitignore loading to EdenMount, this is needed later in the stack to pass this function as a callback through the DiffContext to enable gitignore loading from the source control tree differ.

Reviewed By: simpkins

Differential Revision: D18565265

fbshipit-source-id: 062349d81bbfb78235ccb34d6ec72ad445a6e2b0
2019-12-09 11:33:23 -08:00
svcscm
8422adbb1f Updating submodules
Summary:
GitHub commits:

a34673429a
d7535e2e5c
3a6d9436e8
610fa46702
547bdd2955
b122356fd1
a7e933e480

Reviewed By: zpao

fbshipit-source-id: 7c3bc5d3e3e555c0545ce20f9df733c67526a331
2019-12-09 10:20:20 -08:00
svcscm
a20e07f467 Updating submodules
Summary:
GitHub commits:

9b94248d1e
485b5e487f

Reviewed By: yns88

fbshipit-source-id: d6fd911290a174166ede7fc683bd9c8f9ab366b2
2019-12-08 23:22:36 -08:00
svcscm
56bf5d694c Updating submodules
Summary:
GitHub commits:

bb1c9e524b
e3a82bb934

Reviewed By: yns88

fbshipit-source-id: 24cde5a92ecf398cfefc5cf6066e160243ebc1dc
2019-12-07 23:23:30 -08:00
svcscm
b745b4421b Updating submodules
Summary:
GitHub commits:

0fedb49305

Reviewed By: yns88

fbshipit-source-id: 4a7af38531667a9a10ba5466f20ac65b7bc14be4
2019-12-06 22:50:04 -08:00
Jun Wu
8f749d32f2 metalog: handle missing Root blob errors correctly
Summary:
The previously behavior silently uses the empty Root upon "id not found" cases,
which hides data corruption issues captured by the next changes.

Reviewed By: xavierd

Differential Revision: D18737906

fbshipit-source-id: 0f2cfe1b05f11e9d2584d96a14afcb1aed80da4d
2019-12-06 21:08:33 -08:00
Jun Wu
91884d1fab tests: mark tests as skipped on MySQL error
Summary:
Recently many tests become unreliable due to MySQL errors like:

  OperationalError: 2055: Lost connection to MySQL server at '<host:port>', system error: 32 Broken pipe

That makes the testing infra think our tests are broken and cause:

  # Test Failure Bot
  Test is now disabled because it has been consistently broken

Avoid that by detecting MySQL errors and mark tests as skipped.

Reviewed By: simpkins

Differential Revision: D18668303

fbshipit-source-id: 02cb09f31c712fed41903f1e8c4ad45a1d277c30
2019-12-06 21:01:44 -08:00
Jun Wu
35f85f7864 morecolors: remove extension
Summary:
It was used to highlight "more interesting" (aka. extension) code. Nowadays the
non-extension part of code is also interesting. Therefore drop the extension.

Reviewed By: simpkins

Differential Revision: D18666053

fbshipit-source-id: 027190f742015bdb9e7dba109e7a28987af3cf68
2019-12-06 21:01:44 -08:00