Commit Graph

63050 Commits

Author SHA1 Message Date
Jun Wu
06accd8674 minibytes: implement From<bytes::Bytes>
Summary:
Make it possible to construct minibytes::Bytes via bytes::Bytes.
This will be used later.

Reviewed By: kulshrax

Differential Revision: D26232990

fbshipit-source-id: 36af6f28fd08eb457de8b9223235ec038ac3ef14
2021-02-03 20:22:32 -08:00
Jun Wu
67b2d8e536 revisionstore: make redacted constants plain &[u8]
Summary:
Previously they were `Bytes`, which relies on `Bytes::from_static` being
a const_fn.  Going to migrate from bytes to minibytes, which has a type
parameter on `Bytes` that makes it harder to have const_fn from_static.

Switch the constants to plain `&[u8]` to avoid the const_fn issue.

Reviewed By: kulshrax

Differential Revision: D26218290

fbshipit-source-id: 728c9b3831e551f41fb42ec257ca5fe75b7e93a3
2021-02-03 20:22:31 -08:00
Jun Wu
e094e4dabd revisionstore: collect::<Bytes> -> collect::<Vec>().into()
Summary: Going to migrate from bytes to minibytes. Avoid using bytes' specific API.

Reviewed By: kulshrax

Differential Revision: D26218826

fbshipit-source-id: ba41697eab8fc5fb7bf73bc565c05a7c1b29464c
2021-02-03 20:22:31 -08:00
Jun Wu
5c6512f255 revisionstore: Bytes::split_at -> Bytes::slice
Summary: Going to migrate from bytes to minibytes. Avoid using bytes' specific API.

Reviewed By: kulshrax

Differential Revision: D26218292

fbshipit-source-id: c69c2a1e0d1bde37f49e7ad542bf5e952deebb7a
2021-02-03 20:22:31 -08:00
Jun Wu
a75e25d756 revisionstore: bytes::ByteMut -> Vec::<u8>
Summary: We're going to drop dependency on `bytes`.

Reviewed By: kulshrax

Differential Revision: D26218291

fbshipit-source-id: fcb58bfae3e80c37e2a3c7df2a2ca73c8266c70b
2021-02-03 20:22:30 -08:00
Xavier Deguillard
2bb4f7739f nfs: properly reply to MNT RPC call
Summary:
Now that we have the list of available mount points, we can properly reply to
the MNT RPC call.

Reviewed By: kmancini

Differential Revision: D26152810

fbshipit-source-id: f364292ead321689fb20f858aee45efdfbd7287d
2021-02-03 17:54:55 -08:00
Xavier Deguillard
ab0c780f52 nfs: controls whether Mountd should register itself with rpcbind
Summary:
For a proper NFS server, rpcbind is used to query and collect the port/protocol
of the mountd and nfs programs at mount time. In EdenFS, the mount.nfs command
will be run by EdenFS directly, against itself, and we can manually specify
what the ports are for both the mountd and nfs programs. Thus we can use these
directly to avoid polluting the rpcbind database, and potentially conflicting
with a real NFS server.

This will be particularly useful when testing EdenFS as this allows multiple
tests to run at the same time.

Reviewed By: kmancini

Differential Revision: D26140416

fbshipit-source-id: 6b8aa22e69f580a9d6f6edf775d268809a909118
2021-02-03 17:54:55 -08:00
Xavier Deguillard
22075c1159 inodes: start plumbing Mountd in the mount code.
Summary:
Once everything is plugged in, EdenFS will first register the mount point with
Mountd, and then execute the mount.nfs CLI against itself to mount the
repository with the OS.

For now, this does the first part by simply informing Mountd of a new mount
point during mount.

Reviewed By: kmancini

Differential Revision: D26139990

fbshipit-source-id: e5097b8e90c032c8c7bdd2cd03b69695074d7874
2021-02-03 17:54:54 -08:00
Xavier Deguillard
f20536cbaa inodes: add Mountd to the ServerState
Summary:
When NFS is enabled, we will need to have one mountd RPC program running to
be able to serve the mount requests, let's make sure that we have one by adding
it to the ServerState.

On top of the per-mount protocol setting added previously, I've also added a
global setting to make sure that we're not advertising ourself as an NFS server
before the code is more functional.

Reviewed By: genevievehelsel

Differential Revision: D26139991

fbshipit-source-id: 0d9e02618c931a0401394888ea8f3680974966b5
2021-02-03 17:54:54 -08:00
Xavier Deguillard
fc4b26c0ae nfs: make it compile with getdeps builds
Summary:
As the NFS bits are integrated into the rest of EdenFS, we need to make sure
the code compiles properly. The only major change was the removal of the
//common dependencies as these aren't available to OSS builds.

Reviewed By: genevievehelsel

Differential Revision: D26181750

fbshipit-source-id: 44f8ff335e543ab282f40fa7f8d28fc19cf4581c
2021-02-03 17:54:54 -08:00
svcscm
163921d4f2 Updating submodules
Summary:
GitHub commits:

c9de80eee3
eec303f4af

Reviewed By: yns88

fbshipit-source-id: 35f49eeb56e392a8ef14d636ee2e2441e7c049a5
2021-02-03 17:54:54 -08:00
svcscm
33578d2882 Updating submodules
Summary:
GitHub commits:

37a33264aa
23ee162208
c302f08861

Reviewed By: yns88

fbshipit-source-id: 8c2612642d357515c2500b7cffb4e37e0c9dbe3e
2021-02-03 16:56:07 -08:00
Xavier Deguillard
990af745e2 config: add a per-repo mount protocol
Summary:
On macOS, EdenFS will soon be exposed as an NFS server. To ease the transition,
let's have a per-mount config to decide whether this repository should be
mounted via FUSE or NFS

Reviewed By: genevievehelsel

Differential Revision: D26139992

fbshipit-source-id: d28b0e2327c1101f3555ab0bbc9a5b03d82fd7ff
2021-02-03 16:45:37 -08:00
Xavier Deguillard
86114ab338 nfs: switch to the rpcbind protocol
Summary:
The portmap protocol is limited to IPv4, while the rpcbind not only does
support IPv4 and IPv6, but also unix sockets. While we still bind only to an
ipv4, this should allow us to support IPv6 in the future.

Reviewed By: kmancini

Differential Revision: D26114218

fbshipit-source-id: 46e4dfe60a9c1ddbccbb586bf4e3d24012fd26ee
2021-02-03 16:45:37 -08:00
Xavier Deguillard
c39674f372 nfs: use an XdrTrait instead of overloaded functions
Summary:
As I'm going through the various NFS specs, it becomes clear that variant are
widely used, and the macros to generate them is awkward and making working with
them harder than it should be. Instead writing their deserialization would be
more flexible.

While doing this, I also revamped how serialization/deserialization is done by
using a trait that contains 2 static function: serialize and deserialize, new
types are expected to specialize the XdrTrait with their own
serialization/deserialization.

Reviewed By: kmancini

Differential Revision: D26160026

fbshipit-source-id: 951bdd20620cc5715e780e99b64e15397688570d
2021-02-03 16:45:37 -08:00
svcscm
097b2abca7 Updating submodules
Summary:
GitHub commits:

d99240c3dd
5ecfd4a680
58e79191b6
3ef33ffc04
a4de38e3ff
3c772487cb
af65809bd6

Reviewed By: yns88

fbshipit-source-id: 0c8b7a6323a4eeec176cdcd9f41242ea47581437
2021-02-03 15:59:22 -08:00
Genevieve Helsel
a56153126b remove advice to run deprecated hg gc in eden du
Summary: `hg gc` is deprecated and does nothing anymore. Lets just remove this section talking about running `hg gc`

Reviewed By: chadaustin

Differential Revision: D26178391

fbshipit-source-id: 2bb16bbac38465a524ed50454d548564324d35bf
2021-02-03 15:32:08 -08:00
svcscm
e10b784ae0 Updating submodules
Summary:
GitHub commits:

6098ac0543
42b8fb831d
051913b4fe
d6d84f349d
a9f8f1fe38
bd65117055
d08e1984cb

Reviewed By: yns88

fbshipit-source-id: d68a749e98b06ee57bd279a958b815d5dcd0bb3a
2021-02-03 13:20:33 -08:00
Durham Goode
88900aaf93 dynamicconfig: remove legacylist and disallowedlist deprecation logic
Summary:
The original migration strategy with dynamicconfig was to fix configs
one by one until the dynamicconfig values matched the chef/static ones, then we
can turn off chef/static configs. This looks to be too much work, so we're going
to try a different strategy of just turning off all chef/static configs on a
small number of hosts and seeing what breaks.

The legacylist and disallowlist configs were part of the old strategy, and they
make it more complicated to fix dynamicconfig mismatches, so let's get rid of
them.

Reviewed By: quark-zju

Differential Revision: D26208548

fbshipit-source-id: 63171f1f16aa0498c0eefa994dffaeb8e0cc0d72
2021-02-03 09:53:00 -08:00
Jan Mazur
45599b883b add server hostname to response header
Summary: It will be printed out by hg client so users could better understand output of `hg debugnetwork`, especially latencies.

Reviewed By: johansglock

Differential Revision: D26197877

fbshipit-source-id: 3f57bd1b8d51ad4786b5603f8124bf424f2dbf4f
2021-02-03 09:19:44 -08:00
Mark Juggurnauth-Thomas
b643f7d17d exclude thrift/server/test/handler.py
Summary: This file uses new async syntax that is not valid on Python 2.

Reviewed By: singhsrb

Differential Revision: D26223827

fbshipit-source-id: 98ecd60d6f21eb91f5e3781539be7da33327667e
2021-02-03 08:50:02 -08:00
Alex Hornby
5e82a6a2cd mononoke: reduce qps in test-walker-throttle.t
Summary: This test was flaky in the oss builds due to completing slightly to fast.  Update it prior to the switch to governor library.

Reviewed By: krallin

Differential Revision: D26173804

fbshipit-source-id: 24169b6ef045945448095fffb1f1d85c155705b8
2021-02-03 08:14:47 -08:00
Alex Hornby
e3124a43d2 mononoke: throttledblob inner store doesn't need to be Clone
Summary:
Throttledblob can be simplified now that blobstore api is new futures based.

This change removes the constraint that the inner store is cloneable, and use the limiters via async/await directly in the methods that need it.

Reviewed By: krallin

Differential Revision: D26153158

fbshipit-source-id: 6acb7e6cad5aeded17ea61f9e4103a7aacd491f7
2021-02-03 08:14:47 -08:00
svcscm
afeecbbb6c Updating submodules
Summary:
GitHub commits:

723f79fe75

Reviewed By: yns88

fbshipit-source-id: 9c982b6227dcdf6a7c02f3470815de11ebeda434
2021-02-03 08:04:11 -08:00
Mark Juggurnauth-Thomas
ec6c993570 bookmarks_filler: add option to skip connecting to scmquery
Summary:
Add an option to the bookmarks filler to skip checking SCMQuery.  Use this in
tests so that tests don't attempt to connect to SCMQuery.

Ensure the client id and type is filled in for the SCMQuery requests so that
these requests have appropriate values when requests do get sent to SCMQuery.

Reviewed By: mitrandir77

Differential Revision: D26222850

fbshipit-source-id: 620c30d1418f4e951687f244ec769b3e464462b4
2021-02-03 06:00:37 -08:00
Jan Mazur
27f37a6b5e mononoke/lfs_server: remove unused struct member
Summary:
Syncing thrift structure modified in D26221917 to fbcode.
I think it's safe to land D26221917 first and then land this diff.

Reviewed By: ahornby

Differential Revision: D26195881

fbshipit-source-id: 14d29a9374e71e55d93818a0362ee8514710295f
2021-02-03 04:49:08 -08:00
Stanislau Hlebik
7115cf31d2 mononoke: getbundle optimization for many heads with low gen number
Reviewed By: markbt

Differential Revision: D26221250

fbshipit-source-id: dbc2dd4f181d22c30c6061f5b5de95b0be1ea19f
2021-02-03 03:55:46 -08:00
svcscm
0ca147f4c0 Updating submodules
Summary:
GitHub commits:

eb1954c77a

Reviewed By: yns88

fbshipit-source-id: bf14c5ba8d0b55aae8bbe422a68d9d10f8f709d4
2021-02-03 03:55:45 -08:00
svcscm
f58b9a1e4a Updating submodules
Summary:
GitHub commits:

1f1d679dc6

Reviewed By: yns88

fbshipit-source-id: 0e8b2487793ef4ad04bae839d7b45ab0a943700f
2021-02-02 23:19:00 -08:00
Andrey Chursin
c890d2a322 checkout: update exec flag
Summary: Metadata update happens in parallel with blob fetch and content update

Reviewed By: quark-zju

Differential Revision: D26213197

fbshipit-source-id: 45ca20106123e1b6cb85d1dda2f3effa0748cf20
2021-02-02 23:14:34 -08:00
Andrey Chursin
9a89bd4057 vfs: add set_executable api
Summary: set_executable is a pub function of VFS that set exec permissions on simple file

Reviewed By: quark-zju

Differential Revision: D26212713

fbshipit-source-id: 4c3ef477fc8d61362285654dda0b006342e046ee
2021-02-02 23:14:34 -08:00
Andrey Chursin
199f1cc1b9 checkout: basic implementation for file removal
Reviewed By: quark-zju

Differential Revision: D26212172

fbshipit-source-id: 2d3d8c79023b1dfd069c9cc7b759362e3bfc24ae
2021-02-02 23:14:34 -08:00
Andrey Chursin
45e4a30651 checkout: basic CheckoutPlan::apply implementation
Summary:
Basic implementation only process file updates for now.
See todos on CheckoutPlan::apply for more detail

Reviewed By: quark-zju

Differential Revision: D26209984

fbshipit-source-id: fcfbf568359d553a51ea02ea194634048d093d0e
2021-02-02 23:14:34 -08:00
Andrey Chursin
21360d6075 remove fbcode_only from revisionstore
Reviewed By: quark-zju

Differential Revision: D26210868

fbshipit-source-id: 694cd7155c08561cc1a6a6ad6a3fbe01f5847cf5
2021-02-02 23:14:33 -08:00
Andrey Chursin
3992818136 checkout: split updates with and without content fetch
Summary: This is needed becase later during checkout list of updates that needs fetch will go into storage stream

Reviewed By: quark-zju

Differential Revision: D26209983

fbshipit-source-id: 9fb54d48c6f0afc4fb67320aafc2e981c96ab5a9
2021-02-02 23:14:33 -08:00
svcscm
1d5093195a Updating submodules
Summary:
GitHub commits:

5ab2e68c95
8f284aae93
3bd3846977

Reviewed By: yns88

fbshipit-source-id: ea154860a34fe4ef78692d9400906b36a516c217
2021-02-02 23:14:33 -08:00
svcscm
b0e7aafa33 Updating submodules
Summary:
GitHub commits:

229a0c46d6

Reviewed By: yns88

fbshipit-source-id: 42b07bfec5f2783369553e3d36dc337953b9b1e5
2021-02-02 21:19:40 -08:00
Jun Wu
290257aea5 dag: expose LazySet::from_stream API
Summary:
Expose the async LazySet API via NameSet constructor so users won't need to
care about the LazySet type.

Reviewed By: sfilipco

Differential Revision: D26142170

fbshipit-source-id: 178383684981e81e43f2a5610c45a7ebbd354ab4
2021-02-02 19:58:31 -08:00
svcscm
88259fb116 Updating submodules
Summary:
GitHub commits:

79fa39f0c8
488e1cc763
4f320d8e7b

Reviewed By: yns88

fbshipit-source-id: 5f3f2cf3a01cc8577d40b7db66dca602d4ae6a72
2021-02-02 19:58:31 -08:00
svcscm
503fd9ddba Updating submodules
Summary:
GitHub commits:

59f5145e2a
bdbdd4b3f4

Reviewed By: yns88

fbshipit-source-id: 63a0bb9ad09fe740b2a543178345150f7ada361e
2021-02-02 17:46:30 -08:00
Durham Goode
db93813b04 phabricator: supply ssl context in Python 3
Summary:
Phabricator integration was broken in Python 3 on OSX because we didn't
supply the default context so it could verify against the ca_certs.

Reviewed By: quark-zju

Differential Revision: D26183713

fbshipit-source-id: ac395c002da0fb343d1d0b999a57d2059f66f0c7
2021-02-02 17:27:20 -08:00
Jun Wu
92e1c70f2b dev-logger: traced(func) -> [log] for testing purpose
Summary:
Add a way to capture tracing output in tests to verify certain logs are
emitted.

Reviewed By: sfilipco

Differential Revision: D26142174

fbshipit-source-id: 9267ffbe413973b8c54c54db75fe037c05614b1a
2021-02-02 16:39:46 -08:00
Jun Wu
324f47b1f0 dev-logger: switch to tracing_subscriber
Summary:
`tracing_subscriber` has a fmt subscriber that satisfies the "log to stderr in
tests" need. It does not depend on `env_logger` or `TracingData` forwarding to
`env_logger`.

Reviewed By: sfilipco

Differential Revision: D26142175

fbshipit-source-id: 6e7dcd1585cb8431855322493d93bc49a8d57b76
2021-02-02 16:31:47 -08:00
Jun Wu
855122ad66 gracefully ignore invalid JSON in app_samples data
Summary: This would allow us to ignore bogus data and still get something useful for logging.

Reviewed By: DurhamG

Differential Revision: D20343844

fbshipit-source-id: 763d294bc44bb203c1f206ca80e0839396e8de6e
2021-02-02 16:28:21 -08:00
Durham Goode
13de9f801a doctor: repair treemanifest indexedlogs as well
Summary: Previously we only repaired the file indexedlogs.

Reviewed By: xavierd

Differential Revision: D26202423

fbshipit-source-id: b1c673ae69a357d66ab2baf5c36985a3b0597427
2021-02-02 16:21:01 -08:00
Andrey Chursin
6be082f3bd checkout: implement CheckoutPlan::from_diff
Reviewed By: quark-zju

Differential Revision: D26191428

fbshipit-source-id: d26dd58a1bd9a3bab27f38abbd15f0405c646294
2021-02-02 15:34:33 -08:00
Andrey Chursin
a0fd97854b checkout: data structure for CheckoutPlan
Reviewed By: quark-zju

Differential Revision: D26189792

fbshipit-source-id: 9afa1682055d088e5c6e4c166f73d6e0b836806a
2021-02-02 15:34:32 -08:00
svcscm
67c8f79f3b Updating submodules
Summary:
GitHub commits:

ce101ff93e
ea4c25f6f0
d3f6fe6b1f
0492aa7718
1f09694523
87a08974f5
ce2fd6cda1

Reviewed By: yns88

fbshipit-source-id: 68cf943dcbc618875ded890c08f662714906ff92
2021-02-02 15:34:32 -08:00
svcscm
7707a18e8b Updating submodules
Summary:
GitHub commits:

69164b8de6
536086dc8d
8530b7a221

Reviewed By: yns88

fbshipit-source-id: 7545d1dee5b302dba62780ab58298482f2110349
2021-02-02 14:31:06 -08:00
Jun Wu
00253e009c clidispatch: make '--version' and 'foo --version' a native command
Reviewed By: DurhamG

Differential Revision: D19803759

fbshipit-source-id: dd55e492c5cd9485294ecd1c02a7e1a4d98c4989
2021-02-02 13:38:55 -08:00