Summary: The script does not run with the stock python since it depends on edenscm.
Reviewed By: farnz
Differential Revision: D19872953
fbshipit-source-id: 4c5b2d2935a0c9e8cf0a654f541600d7a4fd7211
Summary:
Right now, if the client establishes connection to more than one peer, the
last one to proces `clienttelemetry` wireproto command gets the honor to set
`server_realhostname`. This is not desirable. Specifically, when we have
`fallbackpath` set up for remotefilelog/treemanifest and the prefetch happens
after pulling a commit, we get the hostname of the fallbackpath server,
while losing the hostname of the original `getbundle` server.
Reviewed By: DurhamG
Differential Revision: D19837570
fbshipit-source-id: fdc41565a5dfe670df3caf3b034196c4b7bdf6d9
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
In the next diff I'm going to refactor BlobRepo::get_manifest_from_bonsai()
function and remove IncompleteFilenodes structure. While doing so I noticed
that the behaviour of get_manifest_from_bonsai() and FilenodesOnlyPublic is
different with regards to generating hg filenodes for octopus merges. In
particular, get_manifest_from_bonsai() at the moment doesn't generate filenodes
for paths that came from stepparents, but FilenodesOnlyPublic does generate
them.
I'd say both of this approach are equally reasonable, however let's try to
preserve the behaviour
Reviewed By: krallin
Differential Revision: D19856420
fbshipit-source-id: 9f8ae656205f39536c450b885fc4d8d6a2534456
Summary: It is helpful in CI to be able to remove all eden managed volumes at the end of a job, so that is what this command does.
Reviewed By: simpkins
Differential Revision: D19794482
fbshipit-source-id: d832d093d0a6369a4b533d66afbdce626fa78e2c
Summary:
In the initial iteration of eden_apfs_mount_helper I parsed the human
readable version of the `diskutil apfs list` output to get all of the useful
information from it. Later, I switched to using `diskutil apfs list -plist`
because it simplified the parsing dramatically.
However, it was overlooked that the `mount_point` field was not present in
the plist output. That was usually fine, except in the one case that I
didn't test after changing the parser: after reboot macOS picks a volume
name to remount these volumes. The lack of the mount_point field meant
that we would simply skip the logic that deals with unmounting and remounting
the volume in the correct place.
In order to address this gap we now need to parse the mount table to determine
where the device is mounted, so that is what this diff does.
Reviewed By: simpkins
Differential Revision: D19794478
fbshipit-source-id: ace2df145a46aad7df78c3f4b15fb2198aef3e6f
Summary:
This is done on a sort of best effort basis; the problem
we're aiming to solve (or at least avoid in the majority of cases)
is in these scenarios:
* `buck clean` gives up cleaning as soon as it finds a directory that
it doesn't have permissions to remove.
* `yarn` tries to look for node modules inside the `.Trashes` dir,
which it doesn't have permissions to access
macOS doesn't give us a solid way to indicate that these things should
be disabled at the time that we mount the volume; the various docs
and suggestions online all involve creating marker files to block the
system processes from performing their usual actions, and we could
set those up here, but if we create them with root permissions we'll
trigger the bad behavior in buck and yarn. If we create them with
regular user permissions then eg: running buck clean would remove them
and allow the system to recreate them.
The system will recreate the `.Trashes` directory when the user sends
something to the trash via Finder. Similarly, macOS will recreate
the fsevents directories when an application establishes a watch
on the volume using fsevents.
So we can't permanently prevent this problem, but by deleting these
things at mount time we should at least avoid it bubbling up for
the majority of users, and if those things come back and get in the
way, running `eden redirect unmount ; eden redirect fixup` should
remount and re-remove the problematic things.
Reviewed By: fanzeyi
Differential Revision: D19841514
fbshipit-source-id: f530ab3d68edfa643096bd27efae71c80b505184
Summary: Later in this stack, we're going to have to introduce a few more context types, including Mononoke's per-session `CoreContext` as well as an LFS-server inspired per-request `RequestContext`. To make the scope of the `EdenApiContext` more clear (namely, that unlike the other contexts, this one persists for the entirety of the server's execution), let's rename this to `EdenApiServerContext`. This mirrors the naming convention for these contexts used in the LFS server.
Reviewed By: xavierd
Differential Revision: D19863296
fbshipit-source-id: 8ad785070328523d0beaf824c86c7350ff6a2697
Summary: Use `ServerIdentityMiddleware` from `gotham_ext` in the EdenAPI server to provide server information in the HTTP response headers returned by the server. This is useful for debugging.
Reviewed By: xavierd
Differential Revision: D19863297
fbshipit-source-id: 6ed8bac35e05af580e062423e6f6389a18d17c2a
Summary:
This makes it possible to use `Bytes` for mmap buffers.
The changes are because `minibytes::Bytes` does not implement `From<&[u8]>`
with the intention to make slice copy explicit.
Reviewed By: xavierd
Differential Revision: D19818719
fbshipit-source-id: c34ee451bfd2dc7bcbbcebd52a76444b6c236849
Summary:
EdenFS will now be able to fetch blobs directly from memcache. This won't have
any big benefits as no blobs are in memcache right now, but over time, this
will significantly reduce the cost of fetching blobs.
Reviewed By: fanzeyi
Differential Revision: D19861643
fbshipit-source-id: c2e9d317bd30d4656bf0b3f8897794161697761a
Summary:
This improves scs client ux as you can see in the test.
Cross scheme collisions have been already supported in the code, and there will
be a proper message.
For the ambiguity within a single scheme, I added a message.
Reviewed By: mitrandir77
Differential Revision: D19834142
fbshipit-source-id: a2cff44f6ef031b1224ebf13d38d76c66c9ee4b9
Summary: The exact format of the GPL license header changed when the Mononoke directory moved from `/scm/mononoke` to `/eden/mononoke`. This file ended up with the old header somehow (was created prior to the rename but landed after), so let's fix it to make the linter happy.
Reviewed By: farnz
Differential Revision: D19848640
fbshipit-source-id: 39c5b49850e5a3cba1951bf4e5b813cd08940f01
Summary: Move the generally-useful ServerIdentityMiddleware into gotham_ext so we can use it in the EdenAPI server.
Reviewed By: xavierd
Differential Revision: D19845282
fbshipit-source-id: 3a01b15dc64cee99cefafcdac229c0b70a4db683
Summary:
These tracing points will help us understand the memcache hit rate as well as
the fetching speed.
Reviewed By: quark-zju
Differential Revision: D19836499
fbshipit-source-id: 1936c44efc3e7715069e6a959f5331139d591d5c
Summary:
Everytime a cache miss is seen, the data fetched from the server will be sent
directly to memcache for future use. Unfortunately, doing so in a blocking
manner severely impact the overall fetching speed from the server. Since
memcache is purely an optimization, we can afford to send data to it
asynchronously.
Let's move as much as possible of the code to a background thread to reduce the
overhead of memcache.
Reviewed By: DurhamG
Differential Revision: D19836011
fbshipit-source-id: 68e506ef7464d6e99d98457d0d37178f514be1a9
Summary:
Instead of fetching data one-by-one, let's prefetch data concurrently by using
the new get_iter function.
Reviewed By: DurhamG
Differential Revision: D19836009
fbshipit-source-id: 4a50328c0cbbba677c2de3777ebe4c34cb10c1e2
Summary: remove the need to pass mapping to `::derive` method
Reviewed By: StanislavGlebik
Differential Revision: D19856560
fbshipit-source-id: 219af827ea7e077a4c3e678a85c51dc0e3822d79
Summary: Test that mercurial can accept a unicode commit message.
Reviewed By: xavierd
Differential Revision: D19838221
fbshipit-source-id: b6333e587004b358a3883db70e40d2f32af4da29
Summary:
Currently admission to fastreplay is controlled at a global level - a
percentage of all repos being replayed are admitted without the ability to turn
off an individual repo.
Update fastreplay to use a config option that controls which repos should be skipped.
Reviewed By: krallin
Differential Revision: D19835304
fbshipit-source-id: 63b7c66b35eaa2cb1370ad96b1c6e98a2c93b712
Summary: Rechunker uses tokio 0.1.0 and legacy futures 0.1.0 . This diff changes rechunker.rs to use new std::future and tokio 0.2.0
Reviewed By: farnz
Differential Revision: D19801966
fbshipit-source-id: 3ba7936e8981c87906a881216ffd66332cdcc761
Summary:
This will allow us to distinguish `getbundle` for a normal `pull` from the one
for infinitepush pull.
Reviewed By: StanislavGlebik
Differential Revision: D19833206
fbshipit-source-id: 86534320fbb4d60bac04d458a0953701201cba87
Summary:
Even when memcache would be able to prefetch everything, this would always call
into the underlying remote store with an empt key set. For things like `hg
prefetch` and a large number of keys, the effect of doing that is minimum, but
for EdenFS or `hg log -p`, the roundtrip to the server for every file/revision
would add a significant amount of overhead. Let's simply stop iterating when we
no longer need to fetch anything.
Reviewed By: DurhamG
Differential Revision: D19835797
fbshipit-source-id: 54ad704428c3b20d973cfa87f7171899ec44b3f9
Summary:
Don't rebuild the memcache store everytime. This avoid spawning a thread and
reconnecting to memcache everytime.
Reviewed By: DurhamG
Differential Revision: D19797911
fbshipit-source-id: 7ecfa111a98ab67c4791b91410a22e458fe169f6
Summary:
Added "rust" to the languages in the if-service and its dependent targets, added a unique `rust_crate_name` to each one.
Adjusted import names in the dependent code.
Reviewed By: dtolnay
Differential Revision: D19818989
fbshipit-source-id: d1cf5c5f7b82dd64d7c045e50b304b7a58cf2ffc
Summary:
We don't use the feature in production. The main motivation is to remove the
flaky test. People wanting this feature can use normal hooks instead.
Reviewed By: xavierd
Differential Revision: D19830033
fbshipit-source-id: 0b1df7bb6c5de116ee27df4c38071912568d893e
Summary:
We never actually enabled it in production, and our new storage (ex. metalog,
indexedlog) are SIGKILL-safe. Only the changelog (revlog) might be subject to
double Ctrl+C issues. And we have debugfixcorrupt which can fix changelog
corruptions. So nointerrupt becomes less important.
The direct motivation is the test has been too flaky recently (and it's hard to
reproduce locally).
Differential Revision: D19830032
fbshipit-source-id: 87722d730fab49efbfd231772b1a5dddf8ec0d7b
Summary: Not very useful on today's hardware.
Reviewed By: quark-zju
Differential Revision: D19783741
fbshipit-source-id: d8ff29486cdc5f8e31d929d9fbba827e37012381
Summary:
We don't have confidence that python 2 is only using bytes (vs unicode
or other byte-like objects). These asserts break users, so let's disable them
except for in tests. We should uncover issues here as we port to python 3.
Reviewed By: xavierd
Differential Revision: D19819699
fbshipit-source-id: 9e3a9c34e7661fac4db9ee2a79f65be3d5e48cb3
Summary:
As part of the py3 migration we've started enforcing that encodeutf8 is
only called on bytes. graphql is returning unicode, so we need to change it to
return bytes on python 2.
Reviewed By: xavierd
Differential Revision: D19819701
fbshipit-source-id: 433ced2da87d522e4234ff453d11cacfaeebdd8d
Summary:
ui.log apparently receives non-utf8 data occasionally, which makes the
encoding fail. Let's change this to not encode on python2 (by using pycompat).
As we migrate to python 3 all the strings should become unicode, and this will once
again enforce utf8 encoding
Reviewed By: xavierd
Differential Revision: D19819700
fbshipit-source-id: 66da26d00d1b1c4286e325f0574ad7ff478e5d1f
Summary: logs information about daemon shutdowns, including duration, if it was a graceful restart or not, and if it was successful or not.
Reviewed By: fanzeyi
Differential Revision: D19817811
fbshipit-source-id: a851de8872f98952d046fb148a27fbf9f05b2212
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary:
Attach the content to the exception object directly so if the exception is
handled, no verbose message will be printed.
Reviewed By: DurhamG
Differential Revision: D19817404
fbshipit-source-id: 9b79952559d07eb62bd7eb2398c6dd022ffe7b28
Summary:
I think I might want to call this from more than one place in
a later diff, so extract it into a helper function.
Reviewed By: chadaustin
Differential Revision: D19770166
fbshipit-source-id: e044003736c6ba21984a9129da1df50ce92b2f35
Summary:
This commit causes a desktop notification to be shown if we generate
EIO or ETIMEDOUT responses via fuse; the prompt is intended to make it obvious
to the user that they need to connect to the VPN.
The commit by itself doesn't show a notification, it allows configuring a
command that can be run to do something to show a notification.
The test plan includes one such configuration for our corp environment.
* It doesn't trigger for thrift-originated downloads (eg: prefetch), only for
VFS operations through FUSE.
* Ideally we'd know exactly when we have a network related error in the store
code and use that to trigger the notification. However, we have a rather
convoluted set of importers and fallbacks today, one of which is interpreting
a generic response returned from a pipe, so it is not especially clear
exactly where we should locate the logic
Reviewed By: chadaustin
Differential Revision: D17513364
fbshipit-source-id: 45134f3672679cb5580cb0c1bc12a0d6e38525ca
Summary:
Include the full error source chain on any internal errors. This will improve
debugability when internal errors are encountered, as the outermost context
might not be enough to describe the origin of the error.
Detect when backtraces are disabled, and only include them on the error if a backtrace was captured.
Reviewed By: StanislavGlebik
Differential Revision: D19813785
fbshipit-source-id: c28ffc0c44050a82cb7050afa57e976f37962432
Summary: create a bounded traversal crate with the 0.1 version of bounded_traversal_stream functionality, prior to updating to futures 0.3
Reviewed By: farnz
Differential Revision: D19804873
fbshipit-source-id: e98e00111fee5b1a9fcfc20bb054eeae1263fb26
Summary: replace use of spawn_future with tokio 0.2 join handle
Reviewed By: krallin
Differential Revision: D19770171
fbshipit-source-id: e0b7bf3da58896b223149b339a72bfec997215ee
Summary:
Update the walker step methods to use new futures, and combine them with async fn
Later in stack planning to:
* remove use of spawn_future and replace it with the tokio 0.2 join handles
* port bounded_traversal_stream to new futures so all these 0.3 futures don't immediated get compat'd back to 0.1
Reviewed By: farnz
Differential Revision: D19767902
fbshipit-source-id: 10fd6236a064efbb7d0815fadbdd32036bcafead
Summary:
D19767626 added an original_timestamp column to the
blobstore_sync_queue. Update the sqlite schema to keep it in sync.
Reviewed By: krallin
Differential Revision: D19787488
fbshipit-source-id: ad576e2ec99349953e2ab69e3defb73d1ff556c0
Summary:
Modify the multiplexed blobstore implementation so that the
multiplex_id is written to the healer queue after a put. Further, update the
blobstore healer to only look at entries with the same multiplex ID as it's
configured to run with.
Reviewed By: ahornby
Differential Revision: D19770057
fbshipit-source-id: 41db19f0b0f84c048d49ab9e6258cccc89cf4195
Summary:
This test deliberately races itself. Unfortunately, tokio's scheduler is sufficiently quick that if we spawn all the futures as they're created, they sometimes don't race each other.
Fix this by spawning in the join line instead.
Reviewed By: ahornby
Differential Revision: D19812651
fbshipit-source-id: 86685bbb71c451e9c2a96100c83ddff28d0718dd
Summary:
Before we start blocking generation of derived data let's start with logging if
derived data is not specified.
Reviewed By: farnz
Differential Revision: D19791523
fbshipit-source-id: 15bed8463f8a021de76a2878f06ec95d9fef877f
Summary:
See D19787960 for more details why we need to do it.
This diff just adds a struct in BlobRepo
Reviewed By: HarveyHunt
Differential Revision: D19788395
fbshipit-source-id: d609638432db3061f17aaa6272315f0c2efe9328
Summary:
Looks like most of our tests got slower recently, and this particular test
started failing as a result since it's sensitive to timing. Unlike when this
test was written, we can now get a little more info from Mononoke LFS by
looking at Scuba logs, to know if we went to upstream or not. So, let's do
that.
Reviewed By: HarveyHunt
Differential Revision: D19790000
fbshipit-source-id: 5617b088595c911018166d2c13eb43dc6adca60b
Summary: Make the EdenAPI server report that it is exiting when asked to shut down. This ensures that Proxygen will stop sending traffic to servers that are about to be shut down by Tupperware. This diff is basically the same as krallin's diff D17626009 for the LFS server.
Reviewed By: quark-zju
Differential Revision: D19782432
fbshipit-source-id: 41b9e6761145402e7dcf18c53a2b33799588594c
Summary: This diff sets up the Mononoke API (from the `mononoke_api` crate) in the EdenAPI server, and makes it available to route handlers by adding it to a new `EdenApiContext` struct that is maintained as part of the server's global state. The server's route handlers should use the Mononoke API for accessing source control data, and any new source control business logic should be incorporated into that crate rather than being part of the EdenAPI server itself.
Reviewed By: xavierd
Differential Revision: D19778441
fbshipit-source-id: bc2efb82e0276d75c49980c52fa0d3017a4ce2f1
Summary:
See also https://github.com/serde-rs/bytes/.
This will be used in the `dag` crate.
Reviewed By: DurhamG
Differential Revision: D19770858
fbshipit-source-id: 2a870a564e0ceecdc7a4667853b2b2a5ea4ce6e3
Summary:
This crate provides the core features of the commonly known `Bytes` crate:
zero-copy slicing and cloning, while also supports mmap-backed buffers.
The main motivation is to replace `Mmap` in `indexedlog`. That has multiple
benefits:
- Handles 0-sized mmap more cleanly.
- Handles clones more cleanly.
- Gain the flexibility to zero-copy data without lifetime / reference.
- Gain the flexibility to switch to non-mmap data.
The `bytes::Bytes` crate does not yet support mmap buffers as of its latest
release (0.5.4).
Implementation wise, `minibytes::Bytes` uses `Option<Arc<dyn Trait>>` for the
"trait object". This makes implementing the mmap storage just one line.
`bytes 0.5.4` re-invents the "trait object" manually using unsafe code. It requires
about 50 lines to implement the mmap storage (in D19756122).
Reviewed By: xavierd
Differential Revision: D19770856
fbshipit-source-id: 8cfa7052a18ac2e0cd6348b77d5e2a4acc61195c
Summary:
Our main data table has no information about the actual failure. Let's
upload a stack trace to blob storage and link to it from the main table.
Reviewed By: quark-zju
Differential Revision: D19780653
fbshipit-source-id: acaab29a47d32dd6ab81fed0c3ae50c8eb44b760
Summary:
If the main process forks, the connection to the cache client is likely
unusable. Let's drop that connection and we can recreate it later if needed.
Reviewed By: quark-zju
Differential Revision: D19796387
fbshipit-source-id: 59b0d3db9655d2233b55effcdf18cbd74a3f5edd
Summary:
We're seeing an error where worker processes fail to fetch data from
the server. Our theory is that the forking messes up the ssh connection, so
let's not reuse ssh connections across forks.
Reviewed By: xavierd
Differential Revision: D19794281
fbshipit-source-id: 3954cd96617fd18ecf0d0aa9b9a6ce774c494067
Summary:
In D19581174 I made `--debug` trigger ipdb session for `.t` tests.
This diff adds similar feature for `.py` tests.
Reviewed By: DurhamG
Differential Revision: D19784795
fbshipit-source-id: 9118c74ea465320f15aa6ad5e1e04d8bc1a49966
Summary:
Currently existing validation won't catch a bug where commits `a <- b` get
replayed as `b <- a` as long as they don't touch the same files. Let's add
such check.
Reviewed By: StanislavGlebik
Differential Revision: D19723150
fbshipit-source-id: ddc15063b9ae4fc38416ab9b96681da302fec8d4
Summary: after reviewing my stack D18647089 , chadaustin recommended some documentation surrounding this area since it is a bit more complex now, and since it was easy to deadlock and there are a lot of moving parts, I added documentation about how the diff path works, including the new code path, and other non-trivial parts.
Reviewed By: chadaustin
Differential Revision: D19688435
fbshipit-source-id: f6512340421e7c25d4fb4d27177c6187eb57ccbf
Summary:
In order to uniquely identify a blobstore multiplexer configuration,
add an ID.
Reviewed By: krallin
Differential Revision: D19770058
fbshipit-source-id: 8e09d5531d1d27b337cf62a6126f88ce15de341b
Summary: fixes printing for no match found in revlog match
Reviewed By: quark-zju
Differential Revision: D19652374
fbshipit-source-id: 55bfcbff7919ad817173241e2792df27311b931e
Summary:
Follow up from D19718839 - let's add a function that will safely sync a commit
from one repo to another. Other function to sync a commit are prefixed with
unsafe
Reviewed By: krallin
Differential Revision: D19769762
fbshipit-source-id: 844da3e2c1cc39ef3cd86d282d275d860be55f44
Summary:
If we e.g. a getpack for path like "foo\"bar", then we can't decode it into a
`&str` because we need to allocate a new `String` to hold it. At the same time,
if the path is "foo bar", then having a reference into the JSON we received is
nicer.
Right now, we expect a `&str`, so the latter case. But, if we find command args
from the first case, we can't deserialize them. To fix this, let's use
`Cow<...>`, which lets us either have a referenced or an owned string.
Also, let's add tests to confirm this works.
Reviewed By: ikostia
Differential Revision: D19767689
fbshipit-source-id: bf9e06d4a885638073c819a25a68810ff44f2546
Summary:
Print out the name of the commit and the stack.
Hopefully this can help making debugging KeyErrors easier.
Reviewed By: DurhamG
Differential Revision: D19776181
fbshipit-source-id: 2eb985dd5355732a4d7728af68eb16173c48caa5
Summary: This makes the output more readable even if the "name" of a span is very long.
Reviewed By: DurhamG
Differential Revision: D19780536
fbshipit-source-id: dce0d3777409c32b0752db51341a572addb823ea
Summary:
The use of json meant the progress step was coming out as unicode when
it should be str. Use the mercurial.json functions to solve this for python 2
and 3.
Reviewed By: xavierd
Differential Revision: D19777255
fbshipit-source-id: 15c8e45425fc8742b6e118249104fc1fb2f3345d
Summary:
Fetching things from MySQL sequentially in a buffered fashion is a bad
practice, since we might end up saturating the underlying MySQL pool, and
starving other MySQL clients.
Instead, let's make fewer, bigger queries.
Reviewed By: ahornby
Differential Revision: D19766787
fbshipit-source-id: 1cf9102eaca8cc1ab55b7b85039ca99627a86b71
Summary:
Fetching things from MySQL sequentially in a buffered fashion is a bad
practice, since we might end up saturating the underlying MySQL pool with a lot
of requests. Doing so will result in other queries being delayed as they wait
behind our batch of queries, which results in higher dispatch latency.
Instead, let's make fewer, bigger queries. Also, while we're in here, let's
update blobrepo to have an up-to-date comment.
Reviewed By: StanislavGlebik
Differential Revision: D19766788
fbshipit-source-id: 318ec4778ca259b210d431fc2add8b327bfce99a
Summary: We don't need to log so many blob fetches. Let's not.
Reviewed By: HarveyHunt
Differential Revision: D19766017
fbshipit-source-id: 674dee276234f96938a9459af18dd78d09243350
Summary: This will let us lower Scuba utilization from Fastreplay.
Reviewed By: HarveyHunt
Differential Revision: D19766018
fbshipit-source-id: 4eac19b929914db910ed13096b2a5910c134ed3a
Summary:
If the user requests blame information for a file where the blame was rejected
(either becuase the file is too big, or because it is binary), this should be
considered a request error.
Reviewed By: farnz
Differential Revision: D19768261
fbshipit-source-id: 7f0d7ba53fe1087b68f4432ec0c6de0353dc3885
Summary: They are not used much - let's use new futures instead
Reviewed By: krallin
Differential Revision: D19767952
fbshipit-source-id: c04bcf5efc6f8ee6f1d31254fcb2cb4603769b91
Summary:
Suggestions come in the error message as it is currently implemented in
Mercurial code. Format of suggestions also stays the same.
We give the hash, time, author and the title.
All suggestions are ordered (most recent go first).
We don't show them if there are two many.
Reviewed By: krallin
Differential Revision: D19732053
fbshipit-source-id: b94154cbc5a4f440a0053fc3fac2bca2ae0b7119
Summary:
Useful for debugging.
I also fixed how we open a SqlSyncedCommitMapping, because we used incorrect path for that.
Reviewed By: ikostia
Differential Revision: D19767148
fbshipit-source-id: baf67bceceb7b22429b05b41020cf4350e3c87bd
Summary:
This is the api that will be used by Sandcastle to remap a commit from one repo
to another.
Previously the implementation api was just looking in the commit mapping table,
but that's not enough - draft commit cloud commits are not in this table, so we
actually need to sync them.
There's a caveat though - we allow syncing public commits from a large repo to
a small repo, but not the other way around. Comment in the code has more info
about it.
Reviewed By: ikostia
Differential Revision: D19718839
fbshipit-source-id: 9939530f818fafd22bc3838b4647dd9cbc1c8c07
Summary:
Jump from "generating filenodes while generating hg changeset" to "generate
filenodes separately" is tricky to do without breaking production. This diff
adds additional logic in IncompleteFilenodes that should make this transition
smoother. See code comment for more details.
Reviewed By: krallin
Differential Revision: D19741913
fbshipit-source-id: 48987c15fc4144c50afcee7ae34072f6cd634271
Summary:
Now that the source of truth for third-party crates is in fbsource, let's use
it in our cargo build system. This removes the need to fetch a tarball and
untar it, which should have the benefit of speeding up the build.
A small caveat is the first build on EdenFS will be slightly slower, due to
crates needing to be individually fetched, subsequent builds will be faster.
Reviewed By: jsgf
Differential Revision: D19726217
fbshipit-source-id: 24f484d1e3118a76e052f07ff3eea0c66cccce96
Summary: In addition to duration and success, log object fetch counts and checkout type to Scuba.
Reviewed By: fanzeyi
Differential Revision: D19334276
fbshipit-source-id: dabf52427f2ebda2b58df93194df39d52f4fcb4f
Summary: Log the number of object lookups and cache hit rates for a checkout operation.
Reviewed By: simpkins
Differential Revision: D19191201
fbshipit-source-id: 5e9ad501e704810f072dabcda3fce86d027c452e
Summary:
During checkout and stats, count every object fetch and which level of
cache it was served from.
Reviewed By: simpkins
Differential Revision: D19186333
fbshipit-source-id: fc0a74db297b9c723682e245996a7befd762f933
Summary:
As initializing the memcache client takes ~0.7s, let's move it to a background
thread as to not impact Mercurial startup time. This diff uses ArcSwap in
order to reduce the overhead of the very common read paths as much as possible.
Using Mutex or RwLock instead would have caused unecessary contention.
Reviewed By: DurhamG
Differential Revision: D19518693
fbshipit-source-id: 886e9b86813fda6ff005ccce99659890026f643a
Summary:
This allows the Python code to build a memcache client and build ContentStore
and MetadataStore with it.
Reviewed By: DurhamG
Differential Revision: D19518694
fbshipit-source-id: d932fd5223ccfdf37db69cbb54a11a6571312709
Summary:
This enables an in-process memcache client for the Rust
ContentStore/MetadataStore. For now, this implementation is lacking several
necessary optimization:
- Start-up time is always slowed down by ~0.7s, the initialization will be
moved to a background thread
- Writing data to memcache is blocking and will be moved to a background
thread too.
- Prefetching data does a roundtrip to memcache for every key, batching
memcache APIs will be added.
Compared to the existing hg_memcache_client, this implementation is both
significantly shorter and do not exhibit some of the pathological behavior of
having to flush the indexedlog for every fetched blob when used in Eden.
Reviewed By: DurhamG
Differential Revision: D19518696
fbshipit-source-id: 4725447d13e7eddd9586135c2511e13ddb921771
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
Partially backport upstream
https://www.mercurial-scm.org/repo/hg/rev/f81c17ec303c to enable lazy loading
of python code contained in edenscmdeps3.zip.
Also, temporarily disabling the demandimport on Python3 is a bit tricky, for
the reasons mentioned in the deactivated function. Thus, instead of using the
disabled function, let's use the deactivated one.
Reviewed By: DurhamG
Differential Revision: D19672866
fbshipit-source-id: c9e39ed044121d962af1cc46745bdec72629c579
Summary:
cpptoml has traversal functionality for table reads, but not for
writes. Add a helper function for reading a config value and updating
the TOML table if it's unset.
Reviewed By: fanzeyi
Differential Revision: D19671264
fbshipit-source-id: e2b78d338af35d51fddaa258b7f45f8966d00a26
Summary: This diff updates the states transitions in the EdenMount on Windows. It starts as State::UNINITIALIZED and transitions to State::RUNNING when the start is called. It will transition to State::SHUT_DOWN on stop or destroy. Destroy will put it in State::DESTROYING, from which it should not return.
Reviewed By: chadaustin
Differential Revision: D19559271
fbshipit-source-id: d76983cab610cb9b2c896807cf1fe49c357f8095
Summary:
The Mercurial convert extension passes around parameters to indicate a commit that needs to be converted from source to sink. For existing converters like Git, this is a simple 1:1 conversion: a commit in the source gets mapped to a commit in the sink, and so they use the source commit hash (sometimes called rev or version in the API) to represent the commit to be converted.
Our converter is much more complicated. Source commits get converted multiple times to account for different ways of mounting it into the destination file system and commit history. The commits are also coming from multiple source Git projects. This means that we need multiple pieces of data to represent a single commit conversion action.
Thus far, we've been trying to meet part of this need by using concatenated strings of (variant, commithash). This logic is breaking down as we add more fields. This commit adds a new immutable object called "conversionrevision" that represents the (variant, source commit hash, source project name, destination path) that is the unique identifier for the individual commit conversions we need to perform. This commit also includes logic for serializing and deserializing these objects as strings (useful because the converter seems to require commit IDs to be strings) and unit tests for all of the new logic.
Reviewed By: tchebb
Differential Revision: D19606867
fbshipit-source-id: 77815ca858f841d452874e95dfa3b351bafde306
Summary:
When I removed an hggit test case from this test last week, it caused
it to stop being skipped and therefore runs on Windows. The filterpwd magic
doesn't work there, and it's unnecessary, so let's just drop it.
Reviewed By: singhsrb, xavierd
Differential Revision: D19744329
fbshipit-source-id: 21f5c67d4fa7a61f14bbacd78756e5397fd6c819
Summary:
Use `abc.ABC` from Python 3 stdlib directly. The definition matches
`pycompat3.py`:
class ABC(metaclass=ABCMeta):
pass
The following changes are reverted since they're no longer necessary:
D19732319 "[hg] py3: fix windows build"
D19703778 "[hg] py3: exclude mercurial/pycompat3.py from Python 2 builds"
D19703779 "[hg] py3: exclude pycompat3.py from Buck-based Python 2 builds"
Reviewed By: simpkins, singhsrb
Differential Revision: D19739075
fbshipit-source-id: 8c1e3727e8a88ff5f7232270d528d690523b1824
Summary:
Printing via `ui` object instead of directly to the `stderr` makes the
output more consistent.
Reviewed By: quark-zju
Differential Revision: D19738355
fbshipit-source-id: 384bf2de11f2ae7b5264ec5d94b041502c4ef7fb
Summary:
The library already has a way to wrap a Python object into a Rust object that
exposes the Rust Read/Write interface. This is the reverse direction for
the Write interface.
The initial intention is to expose Rust stdout as described in D19702533.
However, I found Python's `sys.stdout.buffer` also enforces utf-8 encoding
on Windows (unless PYTHONLEGACYWINDOWSSTDIO is set). So Python's
stdout actually behaves similarly with Rust's stdout on Windows and is okay
to use. That said, it's still useful to have this abstraction, for streampager [1]
integration.
[1]: https://github.com/markbt/streampager/
Reviewed By: sfilipco
Differential Revision: D19716127
fbshipit-source-id: ba39898122561d9a49b7080ee95d7c940540eb40
Summary:
1fb027d759
changed the kernel behavior to reject reads smaller than 8KB,
even for requests that would never need to be that large.
That causes eden to fail to start up on eg: Fedora 31 with a 5.4 kernel.
This commit adds some padding to satisfy this new check.
Reviewed By: chadaustin
Differential Revision: D19736893
fbshipit-source-id: 926456d72124b186976ee9a8a21242e93c26f790
Summary:
On python 2, os.environ is expected to be bytes, so let's remove the
forced decoding. This broke run-tests.py on Windows.
Reviewed By: singhsrb
Differential Revision: D19737218
fbshipit-source-id: d4782eea279275f57cc91bc412a2d74857002cc1
Summary:
We added pycompat3.py which only parses in Python 3. We need to
exclude it from the Windows build as well.
Unfortunately, the Windows build users PyZipFile.writepy which writes an entire
directory. Let's copy in the implementation from Python 3 which allows us to
filter out certain files.
Reviewed By: xavierd
Differential Revision: D19732319
fbshipit-source-id: 4cebb434d052c51d9ae0dccdec5eadd1f412f9e5
Summary: More replacements of dict.iteritems with pycompat.iteritems(dict) for py3 compatibility.
Reviewed By: singhsrb
Differential Revision: D19704211
fbshipit-source-id: 544cb292c0241a293fc4ab83c3d9472dcf4328b6
Summary:
`timed` has been replaced by the `timefunction` in util.
I have also removed the existing usages because they seemed relevant only
during the development.
Reviewed By: quark-zju
Differential Revision: D19727919
fbshipit-source-id: 58c466f02ac2a5bf7096948b17b49ceb26fc1fd9
Summary:
The path normalization OSX logic was using unicode, and when it
switched to use pycompat in D19613690 it stopped using unicode, which broke it.
Let's roll it back.
Reviewed By: singhsrb
Differential Revision: D19726322
fbshipit-source-id: fa7bfacdf41dfedbcfef2fdb5ccb826196101e61
Summary:
Add a few additional type annotations on some functions.
This did require some minor code restructuring in `filestore.setfile()` to
work around pyre's unwillingness to unwrap `Optional` member variables.
Reviewed By: quark-zju
Differential Revision: D19715243
fbshipit-source-id: 8078329c1e4a50ad0aa6c765d42a89b5ed58e7bf
Summary:
Add more type annotations to vfs.py and some of the platform-specific
functions that it calls.
This did catch a couple of cases where string/bytes conversion were not being
done properly in Python 3.
Reviewed By: quark-zju
Differential Revision: D19678717
fbshipit-source-id: b3979cc5e0668ddbb93372dee25ef02ea1867d6f
Summary:
Fix various type issues when running chg under Python 3.
Enable chg in setup3.py build.
This should make tests run faster. For example, test-rebase-detach.t
now completes in 8 seconds, down from 29 seconds.
Reviewed By: xavierd
Differential Revision: D19702535
fbshipit-source-id: 8928b1b920b9b52fd03dc86f996da18f2405f146
Summary:
Drop stdoutbytes/stdinbytes. They make things unnecessarily complicated
(especially for chg / Rust dispatch entry point).
The new idea is IO are using bytes. Text are written in utf-8 (Python 3) or
local encoding (Python 2). To make stdout behave reasonably on systems not
using utf-8 locale (ex. Windows), we might add a Rust binding to Rust's stdout,
which does the right thing:
- When writing to stdout console, expect text to be utf-8 encoded and do proper decoding.
- Wehn writing to stdout file, write the raw bytes without translation.
Note Python's `sys.stdout.buffer` does not do translation when writing to stdout console
like Rust's stdout.
For now, my main motivation of this change is to fix chg on Python 3.
Reviewed By: xavierd
Differential Revision: D19702533
fbshipit-source-id: 74704c83e1b200ff66fb3a2d23d97ff21c7239c8
Summary:
My earlier refactor moved a watchmanclient.getcurrentclock() call out
of a "try/except Exception" block. This meant any watchman errors broke the hg
status entirely, instead of falling back to the old status path.
Let's add the logic around this watchman path.
Reviewed By: sfilipco
Differential Revision: D19709391
fbshipit-source-id: bfb0221ba405a926babddaf73c8e70924e4d9b25
Summary:
The "repo" type is a bit hard to type since it might be wrapped by extensions,
and the decorators like "unfilteredmethod" are too hard for type checkers.
Provide a stub for it so typecheck can work. The stub was generated by `pytype`,
I made a small change to `close` to make pyre happy.
Reviewed By: DurhamG
Differential Revision: D19677410
fbshipit-source-id: 71afb3fe78a75ee269c8005d18eee3a807b50df6
Summary:
This commit enables mounting and unmounting using the
new helper, and fixes up the behavior of `eden redirect list`
and `eden gc` so that they have reasonable behavior when APFS
is in the mix.
The semantics of the `TARGET` field in the redirect list output
become a bit hazy when using APFS volumes; it's a bit less important
because the data isn't effectively moved to a different part of
the filesystem namespace: we're just mounting a different filesystem
over the top and it can only be accessed through its mountpoint.
Reviewed By: chadaustin
Differential Revision: D19403162
fbshipit-source-id: 136a7b02f605cde0a74016608179924be801eda9
Summary:
We've had a small proportion of our users run into problems
with hdiutil and diskimages-helper, where those components get into
an unhappy state and effectively block operating on the redirections.
This diff introduces a new utility that is intended to replace the
use of disk image files with APFS volumes that we mount in the
appropriate places.
The intention is that we will teach `eden redirect` to use this tool
when available, rather than disk images.
macOS's security model is weird: it is perfectly valid for a non-privileged
user to create and delete APFS volumes in the APFS storage container,
but root privs are required to mount it into the VFS.
The intent is that we deploy this utility setuid root to minimize
the fan out--this way we won't need to teach the priv helper about
this kind of redirection.
There are a couple of subcommands demonstrated in the test plan.
Reviewed By: chadaustin
Differential Revision: D19323850
fbshipit-source-id: 35556f841e49e5c4b77679b756af9093222f4500
Summary:
To avoid accidents when people run `eden rm` with a subdirectory of
the mount, require that it be given the checkout root (or a symlink to
it).
Reviewed By: genevievehelsel
Differential Revision: D19549912
fbshipit-source-id: ae0a43e9164f73f5ad9c2f40f1b7226a1e85a567
Summary:
Add type hints for `vfs.join()`. Not all subclasses supported joining
multiple arguments, so fix them all to actually implement the full API.
Reviewed By: farnz
Differential Revision: D19678712
fbshipit-source-id: f7f77eee416cd5edda756a8423cbe3630e195d4a
Summary:
Update setup.py to exclude `edenscm/mercurial/pycompat3.py` when building
for Python 2.
Reviewed By: quark-zju
Differential Revision: D19703778
fbshipit-source-id: 269f1d4e746eb5112a9e577f43c040f2946b07d1
Summary:
There were some unsafe uses of folly::collect, especially in
the diff code. Switch to collectSafe.
Reviewed By: simpkins
Differential Revision: D19529784
fbshipit-source-id: e0ac920a2d2b44767a453f2c41a9166e72246ab6
Summary:
folly::collect's Future is completed immediately when any of its input
future is completed with an exception. This makes forking off subtasks
and then joining with folly::collect dangerous when closures don't
hold pointers or references to parameters, including `this`. Introduce
a collectSafe function, which has the same signature as folly::collect
but unconditionally waits until all futures are completed.
Reviewed By: simpkins
Differential Revision: D19529772
fbshipit-source-id: b3b493fcbb0d9058d2d6e5c8a064dfdae3a43e57
Summary: Matchers were mixing str and bytes - use decodeutf8 on the place using bytes, and enable all the tests that now pass.
Reviewed By: quark-zju
Differential Revision: D19679076
fbshipit-source-id: 2b90cfbf690b2365a1302efd9db72347a295c977
Summary: This gives type checker some hints. Pyre can now look into more areas.
Reviewed By: markbt
Differential Revision: D19676085
fbshipit-source-id: db05b4238051953eb7f8af44da220825ec589c3c
Summary:
This gives type checker some hints. Pyre was able to find some issues,
which are fixed.
Reviewed By: markbt
Differential Revision: D19674436
fbshipit-source-id: 44d6df6037f27e2edeff4d343a9fdec5e51420b3
Summary: This allows type checkers to understand the i18n._ calls.
Reviewed By: markbt
Differential Revision: D19673677
fbshipit-source-id: be2cc72f70e56925007485c786f0d72b5cd40a1a
Summary:
Begin adding type hints to many of the vfs methods.
These type hints indicate that the vfs objects are always opened in binary
mode. While it is possible to open a vfs in text mode if you explicitly
specify the argument `text=True`, this is very rare. If we run into cases
in the future that do need type hints to indicate TextIO mode I think it would
be better to split those out into completely separate functions, rather than
having functions that sometimes return bytes and sometimes return str based on
the arguments used.
Reviewed By: markbt
Differential Revision: D19673538
fbshipit-source-id: 8683223f28964d4b43fe131d4f16f8877dcbe777
Summary:
Make `atomictempfile` derive from `typing.BinaryIO` so that the type checker
understands it can be used like a file.
This required defining all `BinaryIO` methods on this class. This does result
in a fair amount of additional boilerplate code, but seems like the easiest
approach for now.
Reviewed By: markbt
Differential Revision: D19673539
fbshipit-source-id: b5775ca79b3af456b45e4ef5480711d7095e7949
Summary: The share command was reaching into the filesystem directly - use the right utility functions instead
Reviewed By: quark-zju
Differential Revision: D19672980
fbshipit-source-id: a14323fd5419c3ea00d9c009b9a77f63862a7b2a
Summary:
Python 3 is more strict about bad escape characters. As far as I can tell,
this was always wrong, and when corrected, works both in Python 2 and Python 3.
Reviewed By: quark-zju
Differential Revision: D19673184
fbshipit-source-id: b8b358327db8ce2e51761f8f98784bdd4b396423
Summary:
This moves `checkwinfilename()` from util.py to a new winutil.py module.
This allows the windows.py module to import winutil.py and re-expose
`checkwinfilename()` as `checkosfliename()`. This makes pyre happier, since
all platform modules now contain a `checkosfilename` method.
This also adds type hints to `checkwinfilename()`, and updates several parts
of the code that were still incorrectly assuming the input was bytes in
Python 3. However, note that I haven't done much testing of this on Windows.
Reviewed By: quark-zju
Differential Revision: D19672077
fbshipit-source-id: d78ed5b344a2e24c2a361b56465dedc516171d28
Summary:
While the payload of wire commands are bytes, we should treat the
command and args as utf8.
Reviewed By: quark-zju
Differential Revision: D19675217
fbshipit-source-id: df64c842e0c099d77dec1313aa0639e46e539194
