Summary:
It would be nice to see if there was a fsck on startup, the duration of the fsck, and if it was able to repair all of the problems or not. This diff adds external logging for fsck runs on daemon start.
duration: how long the fsck took
success: false if was not able to repair errors, true if repaired all errors or didn't have to repair at all
attempted_repair: true if we found problems, false otherwise
Reviewed By: chadaustin
Differential Revision: D24774065
fbshipit-source-id: 2fa911652abec889299c74aaa2d53718ed3b4f92
Summary:
To ensure other parts of Mononoke can fully read new blobs as soon as they've
been written, ensure their buffers are flushed and they've been synced to disk
before returning from the blob put.
Reviewed By: krallin
Differential Revision: D24921657
fbshipit-source-id: df401470aaeeebcdc9d237271b40a399115ba25f
Summary:
We've seen http 2 potentially causing hangs for users. Let's make this
configurable for lfs, so we can disable it and see if things get fixed.
Reviewed By: krallin
Differential Revision: D24898322
fbshipit-source-id: dc7842c0247dc6b9590a1f160076b17788aab1b9
Summary:
As discussed in a group thread (see link below), HTTP 2 may be causing
hangs for users. Let's start by making the http-client configurable. In
subsequent diffs we'll make edenapi and lfs configurable as well.
Reviewed By: krallin
Differential Revision: D24898323
fbshipit-source-id: f0035a1b8df3cee626ebe519e9e99358c1b3f043
Summary:
This isn't code that compiles, but the convention in Rust is that code actually
is doctests unless annotated otherwise, so if tested with Cargo, those fail.
This fixes that.
Reviewed By: farnz
Differential Revision: D24917364
fbshipit-source-id: 62fe11700ce561c13dc5498e01d15894b17b5b22
Summary:
Thread Pool fails with py3 hg build. Let's replace with a loop.
Most of the usage for the command will be for a single head anyway.
Reviewed By: krallin
Differential Revision: D24902167
fbshipit-source-id: c7af46d0d63ddd074c98788bf55520ae3f2550b8
Summary: As we are making directory structure inside the bucket anyway, it would be usefull to combine keys per repo.
Reviewed By: ahornby
Differential Revision: D24884248
fbshipit-source-id: 85efeb7009a9d211381319caa4e72aa3687c51ee
Summary:
Transfers iddag flat segments along with the head_id that should be use to
rebuild a full fledged IdDag. It also transfers idmap details. In the current
version it only transfers universal commit mappings.
Reviewed By: krallin
Differential Revision: D24808329
fbshipit-source-id: 4de9edcab56b54b901df1ca4be5985af2539ae05
Summary:
Under this configuration SegmentedChangelog Dags (IdDag + IdMap) are always
downloaded from saves. There is no real state kept in memory.
It's a simple configuration and somewhat flexible with treaks to blobstore
caching.
Reviewed By: krallin
Differential Revision: D24808330
fbshipit-source-id: 450011657c4d384b5b42e881af8a1bd008d2e005
Summary:
Constructs and returns `CloneData<ChangesetId>`. This object can then be used
to bootstrap a client dag that speaks bonsai commits.
Short term we are going to be using this data in the Mercurial client which
doesn't use bonsai. Hg MononokeRepo will convert it.
Long term we may decide that we want to download cached artifacts for
CloneData. I don't see an issue getting there, I see this as a valid path
forward that cuts down on the configuration required to get to the cached
artifacts. All that said, I think that using whatever dag is available in
memory would be a viable production option.
Reviewed By: krallin
Differential Revision: D24717915
fbshipit-source-id: 656924abb4bbfa1a11431000b6ca6ed2491cdc74
Summary: The SegmentedChangelogManager abstracts saving and loading Dags. This is currently used in the tailer and seeder processes. It will also be used to load dags while the server is running.
Reviewed By: krallin
Differential Revision: D24717925
fbshipit-source-id: 30dff7dfc957f455be6cf733b20449c804511b43
Summary:
The XLOG_EVERY_MS doesn't use the 3rd argument as a format string, it just
prints it verbatim. To format it, we need to use fmt::format.
Reviewed By: genevievehelsel
Differential Revision: D24906819
fbshipit-source-id: 7d45787301086fb87dd8f5d478af8007df82c0b6
Summary:
The move constructor needs to be noexcept and should also initialize the
members in the right order.
Reviewed By: genevievehelsel
Differential Revision: D24874304
fbshipit-source-id: a3db5dcdab1397b872b8f13ec5c7fd45baad5e6f
Summary:
The components iterator return pieces of the original path, using a reference
makes little sense and the compiler complains.
Reviewed By: genevievehelsel
Differential Revision: D24873851
fbshipit-source-id: 40d414dcb4a0539167ab4760dfc0095af8245b3a
Summary:
The documentation for PrjFillDirEntryBuffer states that if no entries could be
added, then the ERROR_INSUFFICIENT_BUFFER errors need to be returned as is, the
code didn't do that.
Reviewed By: chadaustin
Differential Revision: D24764566
fbshipit-source-id: d6411822eac71b2f9aa7cf07858d09115767cc59
Summary:
This is the plumbing to allow us to skip Metadata prefetching during eden
prefetches. These can trigger a bunch of wasted network requests
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
We won't necessarily want to skip metadata prefetching always, we will still want it
for the watchman queries, but for `eden prefetch` will probably want to skip it. This
is why we are making it an option in the GlobParams.
Reviewed By: chadaustin
Differential Revision: D24640754
fbshipit-source-id: 20db62d4c0e59fe17cb6535c86ac8f1e3877879c
Summary:
We will start opting-in and rolling prefetch profiles mvp out to users soon.
This is a switch to allow users to opt-in, us to gradually rollout, and to
quickly turn prefetch profiles off if this causes issues for users.
Reviewed By: genevievehelsel
Differential Revision: D24803728
fbshipit-source-id: 0456f2a733958b495e5d84f7177c99f3ef481f57
Summary:
Allow users of `tests_utils` to create paths that are not `String`, by supporting any type
that can be converted into `MPath`.
Reviewed By: StanislavGlebik
Differential Revision: D24887002
fbshipit-source-id: 47ad567507185863c1cfa3c6738f30aa9266901a
Summary:
Add type definitions for skeleton manifests.
Skeleton manifests are manifests that correspond to the shape of the repository (which directories and files exist), but do not include anything relating to the content. This means they only change when files are added or deleted.
They are used for two purposes:
* To record the count of descendant directories for each directory. This will be useful for limiting parallelism when doing an ordered traversal of a manifest. The descendant directory count is a good estimate of the amount of work required to process a directory.
* To record whether a directory, or any of its subdirectories, contains a case conflict. This will be used to enforce case-conflict requirements in repos.
Differential Revision: D24787535
fbshipit-source-id: 7cb92546ed80687d5b98a6c00f9cd10896359b8d
Summary:
On Windows, /bin/sh doesn't exist. To spawn a command in a shell, we need to
use Powershell.
Reviewed By: genevievehelsel
Differential Revision: D24864355
fbshipit-source-id: 3bcf630a90e644a31ff9db8fea9891476cad641d
Summary:
While doing notifications, I struggled a bit to get them working and thought
the special quoting on Windows didn't work as expected. It turns out the error
was cmd related and using a modern shell (PowerShell) fixed it.
Having a test for the quoting is a good idea nonetheless, so let's have one.
Reviewed By: genevievehelsel
Differential Revision: D24864357
fbshipit-source-id: 6b1ac50f3b7b1ef469378d5de21f56c24c0945f9
Summary:
BE: remove old subscription to save resources in IceBreaker. The client code will recreate it anyway if missing but cleaning up will help us to reduce number of unused subscriptions.
Classic example: repo opsfiles or configerator maybe needed once and then a user don't use
Another example: switching workspaces failed and it could be result in subscriptions are not cleaned up properly
Reviewed By: markbt
Differential Revision: D24859931
fbshipit-source-id: 6df6c7e5f95859946726e04bce8bc8f3ac2d03df
Summary:
Those are the tweaks I've made to make `--config devel.bundle2.debug` more
verbose to aid with my investigation. This might help somebody else in the
future so let's comit it:
* added "params" decoding to debugsendbundle
* added "message" to `error:unsupportedcontent` part (we already send it with
some other error parts)
Reviewed By: sfilipco
Differential Revision: D24840405
fbshipit-source-id: b25d5823d05f3d50230c078e8db459dc66256707
Summary:
Generate walker EdgeType::outgoing_type() to reduce boilerplate
When defining edges if the edge variant and destination node at the same no extra parameters needed. If they are different then the node type of destination is passed in parens, e.g. BonsaiParent(BonsaiChangeset)
Reviewed By: StanislavGlebik
Differential Revision: D24687828
fbshipit-source-id: 1616c786d78242c2b3a8c7a1ba58cc1433ea0a26
Summary:
This function is useful in the mononoke to compute the universal commit idmap
that is required for clone.
Reviewed By: quark-zju
Differential Revision: D24808327
fbshipit-source-id: 0cccd59bd7982dd0bc024d5fc85fb5aa5eafb831
Summary:
`flat_segments` are going to be used to generate CloneData. These segments will
be sent to a client repository and are going to bootstrap the iddag.
Reviewed By: quark-zju
Differential Revision: D24808331
fbshipit-source-id: 00bf9723a43bb159cd98304c2c4c6583988d75aa
Summary: This is the object that will be used to bootstrap a Dag after a clone.
Reviewed By: quark-zju
Differential Revision: D24808328
fbshipit-source-id: 2c7e97c027c84a11e8716f2e288500474990169b
Summary:
The goal is to reused the functionality provided by AssignHeadOutcome for clone
purposes.
Reviewed By: quark-zju
Differential Revision: D24717924
fbshipit-source-id: e88f21ee0d8210e805e9d6896bc8992009bd7975
Summary:
The EdenFS codebase uses folly/logging/xlog to log, but we were still relying
on glog for the various CHECK macros. Since xlog also contains equivalent CHECK
macros, let's just rely on them instead.
This is mostly codemodded + arc lint + various fixes to get it compile.
Reviewed By: chadaustin
Differential Revision: D24871174
fbshipit-source-id: 4d2a691df235d6dbd0fbd8f7c19d5a956e86b31c
Summary:
There were `eden top` issues on MacOS that I thought had been fixed a while ago,
but it doesn't look like we caught them all. This should catch the remaining bug
in `eden top`.
Reviewed By: genevievehelsel
Differential Revision: D23743199
fbshipit-source-id: ca66748c7a8a26062caf934c8f2c1fd13d9ae69e
Summary:
In order to allow request to timeout to display notifications to the user, the
`within` future method will need to be called on the various callback futures.
Unfortunately, once the timeout expires, the underlying future isn't cancelled
and stopped, but the unique pointer holding the context will be reclaimed.
Whenever the future actually completes, it will try to use an invalid pointer,
crashing EdenFS.
To solve this, switch to using a shared_ptr and copy it in the right places so
it will only be freed once all futures holding a reference to it will be gone.
I also took the opportunity to reduce the nesting a bit to make the code more
readable.
Reviewed By: kmancini
Differential Revision: D24809647
fbshipit-source-id: 987d6e5763106fabc6bed3ea00d28b129b5285a1
Summary: These errors are Win32 errors, we need to wrap them into a HRESULT.
Reviewed By: chadaustin
Differential Revision: D24809646
fbshipit-source-id: 9f42b9d0c43474967dc26cb2c14cbee463768b79
Summary: It is possible that hash of newly created bonsai_changeset will be different from what is in prod repo. In this case let's fetch bonsai from the prod, to make backup repo consistent with prod.
Reviewed By: StanislavGlebik
Differential Revision: D24593003
fbshipit-source-id: 70496c59927dae190a8508d67f0e3d5bf8d32e5c
Summary: Use create_graph to generate EdgeType enum in walker to reduce boiler plate needed when adding new derived node and edge types to the walker
Differential Revision: D24687827
fbshipit-source-id: 63337f4136c649948e0d3039529965c296c6d67e
Summary: Also use the 0.3 compatible .return_remainder in unbundle.
Reviewed By: ikostia
Differential Revision: D24729464
fbshipit-source-id: ede5cc60f4b872a3b968cf14bb0e2c5d9b85c242
Summary:
When finishing a hash computation for a blob, we currently call `format!` to allocate
and format the error string before calling `.expect` on the `write_all` result.
In practice this will never fail, so this is wasted work. From experimentation on
the playground, the Rust compiler does not appear to be smart enough to optimize this
away, either.
A small optimization, but let's get rid of this by calling panic directly, and
only in the failure path.
Reviewed By: farnz
Differential Revision: D24857833
fbshipit-source-id: e3e35b402ca3a9f6c9d8fbbd758cc486ef1c5566
Summary:
Adds `--reversefill` mode to bookmarks filler that fetches bookmark updates
from the queue and syncs them to infinitepush database.
Reviewed By: farnz
Differential Revision: D24538317
fbshipit-source-id: 5ac7ef601f2ff120c4efd8df08a416e00df0ceb9
Summary:
Summarry:
This is the first part of syncing new scratch bookmark pushes from Mononoke to
Mercurial: on each bookmark movement we log this bookmark movement to filler's
queue.
Reviewed By: liubov-dmitrieva
Differential Revision: D24480546
fbshipit-source-id: 27103b4b4f8c4600aaf485826db2936eaffcc4a9
Summary: Make the naming of bonsai fsnode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687833
fbshipit-source-id: 8d46a53c023a4b8f95c0edc42df86e467c054ebb
Summary: Make the naming of linknode edge variants consistent with the other edges in preparation for building them programmatically from a macro
Reviewed By: krallin
Differential Revision: D24687832
fbshipit-source-id: 46525d7bebd17723a130a70f566b24104cc39656
Summary:
Use macro to implement Node::get_type() in walker.
Reduces the boiler plate when adding new types to the graph.
Reviewed By: farnz
Differential Revision: D24687826
fbshipit-source-id: 5f89c6fb69fd9df3fff25a2425a4d2035dbf5ed9
Summary: Generate NodeType::root_edge_type() so less boiler plate when adding new types to the walk.
Reviewed By: farnz
Differential Revision: D24687825
fbshipit-source-id: 083fc57aee8fe01b29ad4a6f9ebe660cc057dfab
Summary:
Define walker graph with macro to reduce repetition which should make adding new derived data types simpler.
Specifically, this removes the duplication between NodeType and Node
Reviewed By: farnz
Differential Revision: D24687831
fbshipit-source-id: 97d67faf02b2a88bb871dc0388d75d3dd3e8528d
Summary: use strum EnumCount instead of own macros, can remove some code. This needed strum upgrade to 0.19
Reviewed By: krallin
Differential Revision: D24680441
fbshipit-source-id: 56e5b66f75c3d8ff949685c26f503571873c0cde
Summary: Update to strum 0.19 as it has improved EnumCount derivation
Reviewed By: mohanz
Differential Revision: D24680442
fbshipit-source-id: 2d3d2a84e994f09bf3b1c7ea748a80a67d100c13
Summary:
`manifest/test_utils` contains test utilities that are only used by derived data, and only one
of which relates to manifests. Its name (`test_utils`) is also confusing with `tests/utils`.
Move it to `derived_data_test_utils`, and update it to new futures.
Reviewed By: mitrandir77
Differential Revision: D24787536
fbshipit-source-id: 7a4a735132ccf81e3f75683c7f44c9ada11bc9d7
Summary:
Reduce visibility of add_* functions that MononokeApp controls, no need for them to be public.
Updated a couple of binaries to use MononokApp.with_fb303_args() instead of calling the add_fb303 function directly.
Reviewed By: krallin
Differential Revision: D24757202
fbshipit-source-id: a068ca4fd976429e7c02c4049429553cc8acf3d4
Summary: benchmark is the last remaining user of args::add_cachelib_args outside of MononokeApp, switch it to use MononokeApp instead.
Reviewed By: krallin
Differential Revision: D24755785
fbshipit-source-id: c105b4443394c88b6effdac382089e7eaca65bfe
Summary: make MononokeApp arguments more configurable so binaries can opt out of them if a section does not apply, making the --help more relevant.
Reviewed By: krallin
Differential Revision: D24757007
fbshipit-source-id: eed2f321bdbd04208567ef9a45cf861e56cdd07e
Summary:
Previously it was a config knob, but they are rather hard to change because
they require a restart of the service. Let's make it a tunable instead.
Reviewed By: farnz
Differential Revision: D24682129
fbshipit-source-id: 9832927a97b1ff9da49c69755c3fbdc1871b3c5d
Summary:
The original problem was a fastlog bug, solved by D24513444 (c3bcc1ab88).
Restores prefetching for phabricator status so `hg ssl` and `hg fssl` become fast again.
Original commit changeset: b10c4caf8fda
Reviewed By: sfilipco
Differential Revision: D24749774
fbshipit-source-id: fa14f7dde9c922733525a7ff014efc32875426fa
Summary:
The original issue was a rust-cpython bug, solved by D24698226, or https://github.com/dgrunwald/rust-cpython/pull/244.
Original commit changeset: 08f598df0892
Reviewed By: sfilipco
Differential Revision: D24759765
fbshipit-source-id: f9a1359cfce68c8754ddd1bcb8bfc54bf75af7ff
Summary:
This updates the last bit of the Filestore that was using 0.1 futures to 0.3.
This used to use a weighted buffered stream (which we don't have for 0.3
futures at this point), but as I started working on one I realized we don't
even need it here, so I took this out.
Reviewed By: StanislavGlebik
Differential Revision: D24735907
fbshipit-source-id: 00a55c14864b09f9c353f95f2f8cbb895cf52791
Summary:
This updates the external facing API of the filestore to use 0.3 streams.
Internally, there is still a bit of 0.3 streams, but as of this change, it's
all 0.3 outside.
This required a few changes here and there in places where it was simpler to
just update them to use 0.3 futures instead of `compat()`-ing everything.
Reviewed By: ikostia
Differential Revision: D24731298
fbshipit-source-id: 18a1dc58b27d129970a6aa2d0d23994d5c5de6aa
Summary: Like it says in the title.
Reviewed By: StanislavGlebik
Differential Revision: D24731300
fbshipit-source-id: b9c44fc1e4bd4cfe8655e1024a0547e40fb99424
Summary:
Like it says in the title. This required quite a lot of changes at callsites,
as you'd expect.
Reviewed By: StanislavGlebik
Differential Revision: D24731299
fbshipit-source-id: e58447e88dcc3ba1ab3c951f87f7042e2b03eb2c
Summary: Like it says in the title. This updates `store()` and its (many) callsites.
Reviewed By: ahornby
Differential Revision: D24728658
fbshipit-source-id: 5fccf76d25e58eaf069f3f0cf5a31d2c397687ea
Summary: Like it says in the title. Not much to be said here.
Reviewed By: ahornby
Differential Revision: D24727256
fbshipit-source-id: 1645339edf287ac7e59612589b308f08b708ae00
Summary:
This updates the metadata APIs in the filestore to futures 0.3 & async / await.
This changes the external API of the filestore, so there's quite a bit of churn
outside of that module.
Reviewed By: markbt
Differential Revision: D24727255
fbshipit-source-id: 59833f185abd6ab9c609c6bcc22ca88ada6f1b42
Summary:
Like it says in the title. This also lets us get rid of some macros we no
longer need.
Reviewed By: markbt
Differential Revision: D24727259
fbshipit-source-id: 5e3211bc08fa5376b4cfce4bea0428ab7bf3dc0f
Summary:
Like it says in the title. This also lets us remove the spawn module entirely.
Note that there is one little annoyance here: I ran into the good old "not
general enough" compiler issue, so I had to add a bit more boxing to make this
go away :(
Reviewed By: markbt
Differential Revision: D24727253
fbshipit-source-id: 73435305d39cade2f32b151734adf0969311c243
Summary:
This will simplify a bunch of refactoring, and we no longer need them not to
be. This lets us convert parts of `chunk` to new futures as well.
Reviewed By: markbt
Differential Revision: D24727254
fbshipit-source-id: de643effe2d1d42ff9bf85a48d09301e929e66de
Summary:
Like it says in the title. This updates the Filestore's multiplexer to new
futures. The change is pretty mechanical, save for:
- We don't use filestore::spawn anymore, since that's provided by
`tokio::task::spawn` now.
- We no longer need to use futures or streams with `Error = !` since in Futures
0.3 you can have an `Output` that isn't a `Result`.
- We need to make the `Stream` we accept `Send` because we can't used
`boxed().compat()` otherwise. I'd like to remove that constraint once the
conversion is complete, but considering all callsites do have a `Send` stream
(the only one that didn't was API Server but that's long gone), just adding
the bound is easiest.
Reviewed By: farnz
Differential Revision: D24708596
fbshipit-source-id: 8b278b5ae49029b7f0d0d9d4fe96c467e1343f60
Summary:
This makes some refactoring later easier. I'd like to not require this, but for
now it's a bit simpler to just do this. Those are the only callsites that
send non-static streams.
Reviewed By: markbt
Differential Revision: D24727258
fbshipit-source-id: c0e4dc86e249a08c2194a20de5a2dfd5a5933d0b
Summary: We don't need to declare a fake empty version number
Reviewed By: farnz
Differential Revision: D24757981
fbshipit-source-id: 594c97e225704d783bea723efcbb9dfc4d5d800b
Summary:
The root cause for S199754 was a file named "con.rs" was checked in onto the
repo. Since this is a reserved filename on Windows, this broke all Windows
users having it in their sparse profiles.
The rules for reserved names are defined as such by Microsoft:
"CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9,
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these
names followed immediately by an extension; for example, NUL.txt is not
recommended. For more information, see Namespaces."
Of course, since the filesystem is case insensitive, these can have any casing.
Reviewed By: krallin
Differential Revision: D24453528
fbshipit-source-id: 389f15e2b1a88e3c1e8721fb7868616acabebc64
Summary:
On Windows terminal, with light color schemes, crecord text was barely visible
(sometimes invisible) due to low contrast on either the background, or the
foreground. Making the text bold makes it brighter and thus more readable.
As a bonus, I've also made the hunk lines magenta to mimic what `hg diff` does.
Reviewed By: DurhamG
Differential Revision: D24718598
fbshipit-source-id: 18c2ff03fc2a46ca45808d5061db21e1f1b501ae
Summary: This makes it clean up stale files more aggressively.
Reviewed By: DurhamG
Differential Revision: D24744461
fbshipit-source-id: 76d163c9f16d8f8d1bf628e9197a3086d7cd48aa
Summary:
The goal of this code was to divide the cache limit by the number of
logs. Instead it divided the cache limit by the default per-log size (2GB). That
results in a very small max-bytes-per-log so data was being thrown out
constantly. This fixes it and updates tests to actually demonstrate the issue.
Reviewed By: kulshrax
Differential Revision: D24712842
fbshipit-source-id: 8062758b5bfa40493e2003d5a9028d601b1522b1
Summary:
Python 3 doesn't support line buffering for binary file descriptors.
Let's stop setting it in chg.
This was causing warnings to pop up during prompts for users.
```
.../python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
return io.open(fd, *args, **kwargs)
```
Reviewed By: singhsrb
Differential Revision: D24747777
fbshipit-source-id: 0b881b4067e8c7086fe73380f81d526a2ecc364a
Summary:
Downloading and applying mercurial bundles directly for list of given heads.
Backup store stores commits as mercurial bundles that can be fetched directly from the store and applied (everstore).
The command could be useful when we migrate our server from one backend to another (Mononoke) and some commits can be missing in Mononoke.
The command could probably be deleted after a while once we migrate completely...
Reviewed By: mitrandir77
Differential Revision: D24756583
fbshipit-source-id: 1629c3756f244621efb965dfe15b75c7509a1cd1
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
In Mononoke for a sharded DB we historically used connection pool size 1 per shard. With the Mysql FFI client now it doesn't make sense, as the client's conn pool is smart enough and designed to works with sharded DBs, so currently we don't even benefit from having a pool.
In this diff I added an API to create sharded connections: a single pool is shared between all the shards.
Reviewed By: farnz
Differential Revision: D24475317
fbshipit-source-id: b7142c030a10ccfde1d5a44943b38cfa70332c6a
Summary:
This diff makes "Calculating additional actions for sparse profile update" more
efficient by using xormatcher instead of unionmatcher. Indeed, we are
interested only in files that changed their "state" after sparse profile change
e.g. either a file was included in sparse profile and then became excluded.
Reviewed By: sfilipco
Differential Revision: D24725902
fbshipit-source-id: ee611e7c123b95937652ced828b5bea6d75a3daf
Summary:
At the moment differencematcher.visitdir never returns "all".
This diff changes it to return all in the case if self._m2 doesn't visit the directory at all and
self.m1.visitdir(dir) returns "all". This makes sense - if m1 visits all files
in the directory and m2 doesn't exclude any file then it's safe to return all
in this case.
This optimization will be used in the next diff.
Reviewed By: sfilipco
Differential Revision: D24725903
fbshipit-source-id: 2a049cfb1ea4878331e8640cbb20af74da86a1a1
Summary:
Whenever a sparse profile changes (e.g. we include or exclude a directory or a file) we do a full prefetch for all trees in the revision and then for each file in a revision we check if this file has changed its state after sparse profile change (i.e. whether it was included before the change and became excluded after the change and vice versa). It can be quite expensive for large repos and looks like checking all the files is unnecessary.
For example, there might be top-level directories that are excluded in sparse profile before and after the change. In that case there's no reason to check every file in this directory, and there's no reason to prefetch manifests for this directory.
More importantly, `mf.walk()` method is already smart enough to do manifest prefetches if treemanifest.ondemandfetch is set to True, so it looks like there's no reason to do any additional prefetching at all (at least in theory).
So this diff does a few things:
1) The default mode is to use mf.walk() method with a union matcher to find all the files that were are included either in old or new sparse profile. In order for it to prefetch efficiently we force enable treemanifest.ondemandfetch config option.
2) It also adds a fallback option to full prefetch (i.e. the same thing we do right now) Hopefully this fallback option won't be necessary and we'll delete them soon. I've added them only to be able to fallback to current behaviour in case there are problems with the new behaviour
I think we can do an even more efficient fetch by using xor matcher instead of union matcher. I'll try to implement it in the next diffs
Reviewed By: sfilipco
Differential Revision: D24705823
fbshipit-source-id: 2c232a66cc74ee95bdaa84201df46448412f087f
Summary:
This seems to trip up Cargo builds
```
error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `with`
--> src/lib.rs:365:3
|
7 | S with version V1
| ^^^^ expected one of 8 possible tokens
error: aborting due to previous error
```
Reviewed By: StanislavGlebik
Differential Revision: D24754708
fbshipit-source-id: 0dc5539acf340ac409bf7b6158313c8fec16a275
Summary: force-unmount-all.sh is a convenience script for edenfs, so move it into eden/fs/.
Reviewed By: fanzeyi
Differential Revision: D24745361
fbshipit-source-id: 661a6f09b73911411fbb8a00bc016757ad19eb2a
Summary: This is unecessary, remove it.
Reviewed By: chadaustin
Differential Revision: D24743519
fbshipit-source-id: 5e10eafcd3f84d9ad053be35798df86b21f97d4f
Summary:
One of the issue that EdenFS on Windows is currently facing is around
invalidation during an update. In effect, EdenFS is over invalidating, which
causes update to be slower than it should be, as well as EdenFS recursively
triggering ProjectedFS callbacks during invalidation. Both of these are a
sub-par UX.
The reason this issue exist is multi-faceted. First, the update code follows
the "kPreciseInodeNumberMemory" path which enforces that a directory that is
present in the overlay needs to be invalidated, even if it isn't materialized.
The second reason is that no reclamation is done for the overlay, combine the
two and you get an update that gets both slower over time and will issue
significantly more invalidation that is needed.
Solving this is a bit involved. We could for instance start by reclaiming
inodes from the overlay, but this wouldn't be effective as we use the fact that
an inode is present in the overlay as a way to know that the file is cached in
the overlay. If we reclaim from the overlay we simply won't be invalidating
enough and some files will be out of date.
It turns out that we already have a mechanism to track what is cached by the
kernel: the fuse refcount. On Linux/macOS, everytime an inode is returned to
the kernel, this refcount incremented, and the kernel then notifies us when it
forgot about it, at which point the refcount can be decremented. On Windows,
the rules are a bit different, and a simple flag is sufficient: set when we
write a placeholder on disk (either during a directory listing, or when
ProjectedFS asks for it), and unset at invalidation time during update. There
is however a small snag in this plan. On Linux, the refcount starts at 0 when
EdenFS starts as a mount/unmount will clear all the kernel references on the
inodes. On Windows, the placeholder aren't disappearing when EdenFS dies or is
stopped, so we need a way to scan the working copy when EdenFS starts to know
which inodes should be loaded (an UnloadedInode really).
The astute reader will have noticed that this last part is effectively a
O(materialized) operation that needs to happen at startup, which would be
fairly expensive in itself. It turns out that we really don't have choice and
we need to do it regardless due to Windows not disallowing writes to the
working copy when EdenFS is stopped, and thus for EdenFS to be aware of the
actual state of the working copy, it needs to scan it at startup...
The first step in doing all of this is to simply rename the various places that
uses "fuse refcount" to "fs refcount" which is what this diff does.
Reviewed By: chadaustin
Differential Revision: D24716801
fbshipit-source-id: e9e6ccff14c454e9f2626fab23daeb3930554b1a
Summary:
The revlog changelog has incompatible rev numbers with changelog2 backends. Do
not construct it. Instead, just use the current changelog.
Reviewed By: DurhamG
Differential Revision: D24513444
fbshipit-source-id: 35d9326cd9fde4af8b98d628f6df66bd80883f92
Summary:
Previously we were choosing current version, and just as with backsyncer this
is not always correct. Let's instead choose not the current version but the
version of the bookmark you are importing to.
This diff also introduced an integration test for a repo import into a pushredirected repo, and turned out there were a few bugs in the repo_import code (open_source_sql was used instead of open_sql). This diff fixed them as well
Reviewed By: ikostia
Differential Revision: D24651849
fbshipit-source-id: bfe36e005170ae2f49fa3a6cb208bf6d2c351298
Summary:
This diff changes semantic of `sync_commit()` function to return an error when
trying to sync a commit with no parents. This is a small code change which has big change
in semantics, and because of that I had to change how backsyncer and
mononoke_x_repo_sync job.
Instead of using `unsafe_sync_commit()/sync_commit()` functions both backsyncer and
`x_repo_sync_job` now use `unsafe_sync_commit_with_expected_version()`
which forces them to specify which version to use for commit with no parents.
And in order to find this version I changed find_toposorted_unsynced_ancestors
to not only return unsynced ancestors but also return mapping versions of their
(i.e. of unsynced ancestors) parents. Given this mapping we can figure out what
version is supposed to be used in `unsafe_sync_commit_with_expected_version()`.
The question raises what to do when a commit doesn't have any synced ancestor and hence we can't decide
which version to use to remap it. At the moment we are using current version (i.e. preserving the existing behaviour).
However this behaviour is incorrect, and so it will be changed in the next diffs
Reviewed By: ikostia
Differential Revision: D24617936
fbshipit-source-id: 6de26c50e4dde0d054ed2cba3508e6e8568f9222
Summary:
Previously we were always choosing the current version for remapping via
pushrebase, but this is incorrect. Let's instead select the version based on
what version parent commits used to remap with.
Reviewed By: ikostia
Differential Revision: D24621128
fbshipit-source-id: 2fedc34b706f090266cd43eaf3439f8fb0360d0d
Summary: Let strum crate do this for us
Reviewed By: krallin
Differential Revision: D24680444
fbshipit-source-id: dbde0077c105d6cc572a0c863bcb4d043714d441
Summary:
Now that fsnodes is async, convert more functions to use references, and tidy
up imports and type names.
Reviewed By: krallin
Differential Revision: D24726145
fbshipit-source-id: 75a619777f19754daf494a3743d26fa2e77aef54
Summary:
Update `fsnodes::derive_fsnode` and its immediate utility functions to use new style
futures and `async`/`.await` syntax.
Reviewed By: krallin
Differential Revision: D24726146
fbshipit-source-id: 0b0d5b1162a73568ef5c47db6e8252267e760e7f
Summary:
The goal of this diff is to provide more visibility into how long the client
takes to create/upload an infinitepush bundle. This is done in two ways:
- by adding more `perftrace` calls (useful when invistigating individual slow
pushes)
- by adding `ui.timesection` scopes (useful for aggregation purposes)
Two main things that are measured:
- creation of the bundle purely on the client
- sending of the bundle over the wire
In addition, in the perftrace recording, this measures how long it takes to
process the reply handlers, how much bytes are sent over the wire, what are the
part names and sizes (when available). These changes mostly do not distinguish
whether the code is infinitepush push or not, but they are always related to
some sort of a wireproto scenario, which means that the performance impact is
negligible (writing things to thread-local storage is *much* cheaper than
sending them over the network).
Reviewed By: DurhamG
Differential Revision: D24683484
fbshipit-source-id: 53fdfb63dcdfcf38924237c59a1e8f5e24ff96c0
Summary: We're getting rid of old futures - remove them as a dep here
Reviewed By: StanislavGlebik
Differential Revision: D24705787
fbshipit-source-id: 83ae938be0c9f7f485c74d3e26d041e844e94a43
Summary:
We can have different bonsai changesets hash for the same hg changeset. Consider situation - we have hg repo:
```
o B (Add file "b")
│
o A (Add file "a")
```
The correct bonsai changeset for B will have only entry `(<Path_to_b>,Some(<hash_b>))` in `file_changes`. But we can also have bonsai changeset for B with 2 entries `(<Path_to_b>,Some(<hash_b>)), (<Path_to_a>,Some(<hash_a>))`. This diff provides the functionality to manually create such situation. And later it will be used for verification blobimport backups
Reviewed By: StanislavGlebik
Differential Revision: D24589387
fbshipit-source-id: 89c56fca935dffe3cbfb282995efb287726a3ca9
Summary: We were incorrectly marking reverts as landed during pullcreatemarkers.
Reviewed By: quark-zju
Differential Revision: D24608217
fbshipit-source-id: f919f49469d6933c17894b3b0926ba2430a5947a
Summary:
As part of getting buck build to work on OSX, we need procinfo to
include it's OSX specific library.
Reviewed By: sfilipco
Differential Revision: D24513234
fbshipit-source-id: 69d8dd546e28b4403718351ff7984ee6b2ed3d1d
Summary:
Improve ux for setting the token.
This extra bits would help to figure out prompt issues on Windows
Reviewed By: markbt
Differential Revision: D24706085
fbshipit-source-id: 6101b8d7b90aad2d687465a09cc69670ca4a46f6
Summary:
Eden can often overfetch files during checkout. There may be multiple things
going on here, but at least one of them is tied to aux data prefetching.
Responding to a FUSE request for aux data will result in the inode for that
file being loaded. During checkout if we don't short circuit the checkout
then we will create a CheckoutAction for each loaded inode. And previously
CheckoutActions will always load their inodes contents.
To fix this we have two options
1. avoid creating the CheckoutAction in the first place.
2. Avoid downloading in CheckoutAction.
1 Turns out to be more difficult. I did some thinking through this here if
you want to see: https://fb.quip.com/LeqSAb3OlpWb
2 We don't actually need to be downloading blob data here, we only really
need the eden hash and content sha1 of the blob in this code path (this is
all that is used to compare the old and new file). So we should really only
be fetching the metadata (which should be avaiable locally since the inode
was loaded unless the caches have been cleared).
This implements fix 2.
I think there is likely still over fetching having to do with trees in checkout.
And potentially we can avoid looking at metadata at all in some cases.
I will look at this next.
Reviewed By: chadaustin
Differential Revision: D23432810
fbshipit-source-id: 17c0503bf95eb360de902a370948338bf04c1887
Summary:
This changes the methods from ones that return old `BoxFuture`s to an async method
using `async_trait`.
Reviewed By: krallin
Differential Revision: D24689506
fbshipit-source-id: 7b13010924369f81681e6590898af703c5423385
Summary:
This changes the trait method from one that returns an old `BoxFuture` to an async
method using `async_trait`.
Reviewed By: krallin
Differential Revision: D24686888
fbshipit-source-id: 0ac231cdbb60d256b6d5ad5aafbe8779b96905f3
Summary: This is to be able to see in Scuba why a given sync function was called.
Reviewed By: StanislavGlebik
Differential Revision: D24689366
fbshipit-source-id: f868fc1b6fcbf6c692e1373cbe8da8cd4a230780
Summary: This is a step towards modernizing unbundle crate to use futures 0.3.
Reviewed By: farnz
Differential Revision: D24682963
fbshipit-source-id: 55c17fd699846a24647a23ea1c22888407643dfd
Summary:
On Windows, we never unload any inodes until EdenFS is restarted, thus,
checkout times go up over time as more and more inodes are being loaded. While
on Windows we don't keep track of what is referenced by the kernel, the
checkout code will use the "precise inodes" code path when deciding what to
update. This means that every inode that is in the overlay will get updated
properly, and since the overlay is a superset of what is hydrated, we are
guaranteed to always invalidate what we need to.
Due to the above, this shouldn't result in any changes as we never gc the
overlay, but that will come later, at which point checkout times will stop
being more and more expensive as time goes.
Reviewed By: chadaustin
Differential Revision: D24634253
fbshipit-source-id: c7b838edc20589bbf92ff4e2b3abd079b9a4443d
Summary:
Futures are by default run inline, meaning that when the previous future is
completed, the future will run in the same context as the previous one. In the
case where the previous one is completed by another thread setting up its
promise, the future will be completed in the context of that other thread.
In most cases, this is OK, in others, this can cause a deadlock. And this is
exactly what we're seeing here. When a file is renamed concurrently to an `hg
update`, the inode the rename operates on might not be loaded, and thus both
update and the rename callback will race to load that inode. When update wins
that race, the rename callback will wait on a promise that update will then
set. When that happens, the rest of the rename callback will be run in the
update context, but that will in turn cause update to try to re-acquire the
rename lock that it already holds...
To fix this, we need to make sure that the rename callback doesn't run inline.
Reviewed By: chadaustin
Differential Revision: D24657422
fbshipit-source-id: 23b08765afae7bda4a628f0c23675bff9f486b6b
Summary:
I'm not entirely sure why these started failing, but enabling ui.allowmerge
made these run again.
Reviewed By: chadaustin
Differential Revision: D24697462
fbshipit-source-id: ec5ca987e7116edb12658eb7b4d03f1cf0f876d3
Summary:
For logging and analytics, it's convenient for the
HgQueuedBackingStore to know the manifest hash and path early in the
import process. Since every object fetch requires looking up the
HgProxyHash anyway, fetch it immediately and thread it down to the
importer.
Reviewed By: kmancini
Differential Revision: D24524319
fbshipit-source-id: 0d91d55655e5ee25a010f7664e80125b7c50cf84
Summary:
Use the empty string to indicate the moved-from state, which makes
HgProxyHash moves noexcept. I plan to look up HgProxyHash early and
move it into HgImportRequest.
Reviewed By: kmancini
Differential Revision: D24522612
fbshipit-source-id: 037b4012ad6a51ad7ebd6a96de2e391cd570771b
Summary:
Stop pretending that HgBackingStore is a standalone backing store
implementation, and instead indicate it's an implementation class used
by HgQueuedBackingStore.
Reviewed By: kmancini
Differential Revision: D24514247
fbshipit-source-id: 90c3a442d01647fa6d1337cfd814f5bf4b480137
Summary:
We had some unnamed threads that made profiling performance on macOS a
bit harder. Give them a semblance of a useful name, at least.
Reviewed By: kmancini
Differential Revision: D24640223
fbshipit-source-id: 7dd74b30a081753006df681bf97ac96147b1896c
Summary:
Until fanzeyi gets InodeMetadata moved over to SQLite, remove some
excessive logging when the InodeMetadataTable contains zeroes because
its buffers weren't flushed to disk.
Reviewed By: genevievehelsel
Differential Revision: D24377026
fbshipit-source-id: e6ffa54244730388aaf66dc53cd29a0069fba685
Summary:
Add a very simple debug command that simply prints all materialized
inodes under the given path. It is a quick way to uncover unexpected
writes (or unexpected failed dematerializations after checkou).
Reviewed By: xavierd
Differential Revision: D24378759
fbshipit-source-id: dc393d65506c74dbc0779732cdefd915cbbf9948
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary:
Apply automated Thrift formatter. Even if the format is a bit off in a
couple places, and the bikeshed is still being painted, this avoids
unrelated formatting changes later in the stack.
Reviewed By: xavierd
Differential Revision: D24511057
fbshipit-source-id: f1b23578733a8ecf788509e407bc419fa073428d
Summary:
In Mercurial we're finding use cases for conditioning on buck vs
non-buck builds. For instance, in some cases we specify #[link] directives for
non-buck builds, but don't need to for buck builds
(https://fb.workplace.com/groups/rust.language/permalink/4568487309866515/). In
other cases we want to depend on internal buck libraries when building with
buck, and not otherwise.
Let's add a fb_buck_build cfg so other projects may distinguish the same.
Reviewed By: jsgf
Differential Revision: D24493974
fbshipit-source-id: 1d558cbe0ae01ab4a7b4b5d6d4be75bb8ab0f41a
Summary: This method is an override of its parent class, thus the need for the override.
Reviewed By: chadaustin
Differential Revision: D24657424
fbshipit-source-id: 5ce7200eeb4ef28fb51fabbe0ebb38ded51ae34e
Summary:
Dealing with non-owned path in futures is mind bending and can lead to use
after free fairly easily if not careful. It turns out that this code is
actually a bit buggy and in some cases will attempt to use a path that is
already freed. Since the callsite doesn't need to hold onto the paths, let's
just move them, which resolves the issuue.
Reviewed By: chadaustin
Differential Revision: D24657423
fbshipit-source-id: 47bbaccf18cd86e53860491e3cbfeadb4363499c
Summary:
The rage local config output was getting polluted by dynamic config.
Let's filter them out.
Reviewed By: farnz
Differential Revision: D24626564
fbshipit-source-id: df5ac04cd549595ecdccc0b2438d4e7c72b80e88
Summary: When I added `eden uptime`, I did not see that we already had `eden debug uptime`! Instead of having two, lets just use the one and remove it from debug. There seems to be a regression of the uptime I had implemented, so thats why I opted to use the implementation of the debug version.
Reviewed By: kmancini
Differential Revision: D24566844
fbshipit-source-id: d948a5303d475a543f51abbaea59f9c481dfeca2
Summary:
Previously, reading from SSH subprocesses did’t return on Python 3. bufsize=0
is default on Python 2, but not Python 3.
This is a backport of: 0fc8020ebe
Reviewed By: krallin
Differential Revision: D24652418
fbshipit-source-id: 1140c76b6f711bfe1726108bd4fe6948e6ee41a0
Summary:
We don't need an ifdef here as folly::kIsWindows works just as well. The astute
reader will notice that we now call isInodeRemembered while we weren't before.
On Windows, this will pretty much always return false as this can only return
true if the fuse refcount is positive. Since we don't keep track of a fuse
refcount on Windows, this is OK to do.
Reviewed By: genevievehelsel
Differential Revision: D24636432
fbshipit-source-id: c6db7c66f7eb27894cdd276a9368149ec8056cf4
Summary:
One of the unforseen effect of D24393690 (c5d631fd09) is that EdenFS was tied to the console
that ran `edenfsctl start`. As a result, killing that console would also kill
EdenFS, which is a bit unexpected.
Console in Windows are a bit complicated, and in theory, EdenFS could simply
call `FreeConsole` once it is started, but the side effect of this is that any
process started by EdenFS (say Mercurial), would create its own console as its
parent doesn't have one, specifying CREATE_NO_WINDOW when creating these would
lead to not having any output in the edenfs logs from Mercurial (D21820195 (223846d313))...
We could solve this if we could tell Windows to allocate a hidden console
window, but that API unfortunately doesn't exist.
So, the best we can do for now is to simply start EdenFS with the
CREATE_NO_WINDOW flag, this fixes the above issue, allows for the original
console to be killed without affecting EdenFS, but it brings another issue:
`eden start` no longer shows what is going on...
For now, this is the least worst solution, so we'll go with that. In the
future, we can imagine fixing EdenFS startup code to send its output to a pipe
that edenfsctl would read and print to its standard output.
Reviewed By: chadaustin
Differential Revision: D24607540
fbshipit-source-id: c18590052a96b7dd2938b589e92e808f38b5ef59
Summary: The more we remove, the easier it'll be to remove the last few problem cases.
Reviewed By: StanislavGlebik
Differential Revision: D24592052
fbshipit-source-id: de6371305991d54bf2802fd904850b49aeb11bbd
Summary:
This didn't matter much in practice because we've never pushrebased a rewritten
commit where one of the parents rewrites to nothing. Nevertheless I think it's
cleaner to return an error in this case
Reviewed By: ikostia
Differential Revision: D24621105
fbshipit-source-id: 8cf62efd28e0c9aed945f727b1872db6922cfeb3
Summary:
This stack has a goal of cleaning up how commits with no parents and
consequently merges are treated in CommitSyncer. At the moment it always uses current version
to rewrite them, and this is incorrect. See example below (here N means "new
commits" i.e. commits that weren't remapped yet but we need to remap them now)
```
large repo
O <- uses current version
|
O
|
O N
|/ |
O | <- uses old version
| N
| |
O N <- this commit should be rewritten with old version!
```
With current logic all N commits will use current version for remapping, but
this is incorrect, and instead "old version" should be used. The goal is to fix
it and make it so that backsyncer and x-repo sync job pick the correct version
to use for remapping.
-----
As it was noted in the previous diff we need a function that overrides a
version for a commit with no parents. We need this function because for a
commit with no parents CommitSyncer can't decide which mapping version to use
because, well, there are no parents which could have hinted about the version.
So let's add this function and cleanup unsafe_sync_commit_impl function to
apply version override only to a commit with no parents. If commit has parents then we'd verify that the version from parents matches the expected version.
In the next diffs I'll make it so that if a version for commit with no parents is not specified then
unsafe_sync_commit_impl fails rather than uses current version.
Reviewed By: ikostia
Differential Revision: D24617953
fbshipit-source-id: a68f837da9d90bb18034f628b7880482a2e548b7
Summary:
This adds some basic Scuba logging to commit sync logic in Mononoke. This can be disabled via a tunable.
Note: of course, the wrapping of logging functions would have been pretty cool to do in a macro, but I don't have time atm to figure it out, so have just code.
Reviewed By: StanislavGlebik
Differential Revision: D24571702
fbshipit-source-id: f8830fc675a2a62d2204671e86ab2c819372c5cc
Summary: This adds a `ScubaSampleBuilder` field to the `CommitSyncer` and ensures this field is instantiated with correct values (real vs discarding sample) depending on circumstances.
Reviewed By: StanislavGlebik
Differential Revision: D24539732
fbshipit-source-id: 37aedcff9aefcfcd6b740a0491ab35f9e5ce7c77
Summary:
Functionality of this function can be replaced with
unsafe_always_rewrite_sync_commit, so let's do that.
However in the next diff we'll need a function similar to
test_unsafe_sync_commit but with a slightly different semantics - instead of
forcing override for every commit we'll just force it for a commit with no
parents. Because of that let's not remove unsafe_sync_commit_impl function.
Reviewed By: ikostia
Differential Revision: D24616824
fbshipit-source-id: 6969145d98cd23604920a6b490bf7ffe47938c08
Summary:
Our integration tests do not have "test_version" version, they have only
"TEST_VERSION_NAME". This didn't make any difference because x-repo sync job
was always use the current version, but this is going to change soon. So let's
update the tests first
Reviewed By: krallin
Differential Revision: D24620364
fbshipit-source-id: 19416f6d6aba2d9c426efa545d18d4be458d3272
Summary:
Fixing test. I updated the serialization format in another stack and didn not
update the hash before I landed.
Reviewed By: lukaspiatkowski
Differential Revision: D24643831
fbshipit-source-id: 3ec888d99cbee0aa6d804740bebaf032dd28e2c9
Summary:
MySQL doesn't like that the idmap table is renamed to `inner`. For good reason,
inner is a keyword, best to rename it.
Reviewed By: ahornby
Differential Revision: D24568914
fbshipit-source-id: 7a3790e835931b29658c7652cc89069c6b9b5bab
Summary:
I avoided this function because it interacts in a weird ways with dependencies.
At this point I am no longer concerned about that and it can help us simplify
some code.
Looking ahead I think that we will refactor things into having fewer
dependencies.
Reviewed By: krallin
Differential Revision: D24555935
fbshipit-source-id: 994b25d90da491bb5cc593b6c33085790c4fb322
Summary:
The command reads the last SegmentedChangelog that was saved for a repository
and updates head to match a given bookmark (master).
Right now this is just a command that works on one repository. Follow up
changes will look at deployment options and handling multiple repositories.
Reviewed By: krallin
Differential Revision: D24516438
fbshipit-source-id: 8f04f9426c2f2d7748c5363d2dbdf9f3acb79ddd
Summary:
I initially saw the incremental build as something that would be run in places
that had IdMap and IdDag stored side by side in process. I am reconsidering
to use incremental build in the tailing process to keeps Segmented Changelog
artifacts up to date.
Since we update the IdMap before we update the IdDag, it is likely that we
will have runs that only update the IdMap and fail to update IdDags. This diff
adds a mechanism for the IdDag to catch up.
Reviewed By: krallin
Differential Revision: D24516440
fbshipit-source-id: 3a99248451d806ae20a0ba96199a34a8a35edaa4
Summary:
Nice to have things for debugging.
This isn't an exhaustive list of places that we could add context too. I'll
probably look to complete the list after the current changes are done.
Reviewed By: krallin
Differential Revision: D24516437
fbshipit-source-id: 7f29e7afde5a5918aea419181d786f48da9b8b14
Summary:
The general goal is to allign segmented changelog blobstore usage with the
general pattern in Mononoke.
Reviewed By: quark-zju
Differential Revision: D24605796
fbshipit-source-id: 808985609f74ebc45f3fcc57583e55f3af9bce1d
Summary: Change the default blobstore put behaviour to IfAbsent, so that all binaries apart from admin tools using MononokeApp::special_put_behaviour() pick up the change.
Reviewed By: farnz
Differential Revision: D24619663
fbshipit-source-id: 98439513b985d2cde88ef99e5eb177974e9db5c9
Summary:
Extend MononokeApp so admin apps can have a special default put behaviour (typically
overwrite) vs the soon to be new default of IfAbsent
Use it from the admin tools.
Reviewed By: farnz
Differential Revision: D24623094
fbshipit-source-id: 5709c68429f8e1de0535eec132998d20411fc0e6
Summary:
mitrandir77 pointed me to this field which is exactly what I want.
Leave alias handling in place for configerator, which has a different name.
Reviewed By: mitrandir77
Differential Revision: D24628323
fbshipit-source-id: ccd01e6314feefd8ab75e8052a0a8a98b3119691
Summary:
We want to be able to prefetch profiles on pull. That means we will need
to pick commits that will be good to prefetch for. This adds a flag --commits
that allows specifiying commits explicitly (mostly for testing), and a
flag `--predict-commits` which will pick good commits to prefetch for.
Reviewed By: genevievehelsel
Differential Revision: D23889882
fbshipit-source-id: 74e61c0c9d443ca396f326b0d547b9fc21a6364b
Summary:
We want to be able to fetch prefetch profiles on pull. That means we will need
to prefetch the contents of prefetch profiles for commits that we are not
currently on. Thus globFiles (the thrift endpoint used for prefetch profiles
fetching) needs to be able to take commit hashes to match and fetch against.
Why fetch prefetch profiles on pull? This would get the prefetch started earlier so
the files are hopefully fetched by the time the user needs them.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D23858659
fbshipit-source-id: 123e423d5117274b92405dbb5c2df690298a1c18
Summary:
Often a prefetch profile will be activated by a tool that uses that profile, so
we will by default prefetch a profile when activating it.
Reviewed By: genevievehelsel
Differential Revision: D23772446
fbshipit-source-id: 0b2e14cfa6ce079e8f04790ac2e0d196db5966ee
Summary:
Blocking checkout and pull to prefetch files could make these commands very
slow when large prefetches are triggered. To keep slow prefetches from causing
poor experiences we can background them by default.
Reviewed By: genevievehelsel
Differential Revision: D23831663
fbshipit-source-id: 4fba5ab894b927a6aa3359b516cff604ad524485
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.
Reviewed By: krallin
Differential Revision: D24477468
fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
Summary: This diff removes reading the token from the "cert" property in the `~/.arcrc` and forces the caller to use either an OAuth token or CATs.
Reviewed By: quark-zju
Differential Revision: D24242614
fbshipit-source-id: 18538270102b7aa28731e82c8dd21f5da9e2f2d6
Summary: Update walker to use MappedHgChangesetId derived data now that hg data is modeled like other derived data. This simplifies bonsai_to_hg_mapping_step
Reviewed By: aslpavel
Differential Revision: D24531553
fbshipit-source-id: 62663d8d47ec7145980c4cd567cba3009b1999cb
Summary:
We have automation that wants to use this to hold the lock while doing
some maintenance. They want the ability to wait for the lock so they don't have
to busy loop.
Reviewed By: snarkmaster
Differential Revision: D24604466
fbshipit-source-id: be02539908655e183f334865718b68b633b069a5
Summary: Introduces an `eden logs` command to read a large chunk (1M by default) file into a paste. This is also added to the the eden rage report to get more insight into systems in which we cannot log in to view the logs (like laptops).
Reviewed By: kmancini
Differential Revision: D24146812
fbshipit-source-id: 991f1595b974eb01f77e86559a8413b0b09a24a4
Summary:
I am wondering whether we should customize the serialization format for the
InProcessStore. I want to have a basis for the comparison before I proceed.
Reviewed By: quark-zju
Differential Revision: D24580273
fbshipit-source-id: d3ddfdc029dbdd84f60acace06fddc80b4d005f4
Summary: This is very old code that once acted to prototype walker-type functionality. As it's dead, delete it.
Reviewed By: ikostia, krallin
Differential Revision: D24591123
fbshipit-source-id: 663108e123d354243c2be4f00819f39d6951db93
Summary:
Some requests can result in a large number of blob fetches. Add rate limiting
so that these requests don't use up all available capacity.
Rate limits can be specified in tunables.
Reviewed By: ahornby
Differential Revision: D24592814
fbshipit-source-id: 9a73a92094d0ce01be5491b65b4808e3ebb05c11
Summary: Add ChangesetInfo derived data to the walker so that it can be scrubbed and validated
Reviewed By: farnz
Differential Revision: D24312123
fbshipit-source-id: 84b3bba87e5391339f97cd2e5ae0313761726d02
Summary: Add the ability to specify a group of node types in one go on the command line
Reviewed By: farnz
Differential Revision: D24526827
fbshipit-source-id: 59d2f0cd06dbbe2555625023be3725a528256005
Summary:
Prevent the walker from trying to walk derived data node types that are not configured for a repo.
This is done by add a mapping from walker NodeType to derived data ::NAME, connect it up for existing derived data usages and using it for validation in setup.rs
Reviewed By: farnz
Differential Revision: D24391591
fbshipit-source-id: 21ae63f4f210d2e1310b0ee2c509fb492f742db7
Summary:
We want to be able to detect garbage blobs by looking at generation numbers.
Update generation numbers on put, and have a mark command exist to mark blobs as not garbage.
Reviewed By: ahornby
Differential Revision: D23989289
fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
Summary:
I'm planning to use them from inside establish_connection function. So this
diff makes a refactoring to make scuba logger and slog logger available in
StdioRelay
Reviewed By: krallin
Differential Revision: D24590426
fbshipit-source-id: 5c20025295700aa91c685c47242618a20f89eb76
Summary: We had some aliases for fbsource, and `fbcode` is still in use. Teach phrevset to recognise all aliases via config for ease of patching.
Reviewed By: markbt
Differential Revision: D24589906
fbshipit-source-id: bd61e86135d63ae07fa62d741e16cea4882f691b
Summary:
The LookupProcessor class is built with the purpose of iterating all the inodes
in the passed in path. However, the LookupProcessor object may outlive the
lifetime of the path, and thus we need to build an iterator on the copied path,
not on the argument.
Differential Revision: D24581874
fbshipit-source-id: b66dc007920b7adad5272bf56d3034acb211fec6
Summary:
`CommitSyncerArgs` was useful when `CommitSyncer` did not have a way to query
existing configs from the configerator. Now that `SyncDataProvider` is part of
`CommitSyncer`, we can get rid of `CommitSyncerArgs`, which will also make
further improvements more convenient.
Reviewed By: StanislavGlebik
Differential Revision: D24565773
fbshipit-source-id: 4dd507b06d946c6018a4a4e8d5e77c6b27abe195
Summary:
After we synced all the files from source directory into destination directory
the destination directory might have some files that source directory doesn't.
So let's add a command to remove them.
Reviewed By: ikostia
Differential Revision: D24541984
fbshipit-source-id: 7e0e21e4c8079d24e1e24adccd3a20a8bbc737ca
Summary:
Previously `mononoke_admin rsync` didn't overwrite files i.e. if a target
directory has a file with the same name as in source directory then it won't be overwritten.
This diff adds an option to make it possible to overwrite a file if a target
directory has a file with the same name but its content is different. However
note that if a file has to be overwritten then target file is going to be removed in one
commit and then copied in the second (i.e. we'll create two commits instead of
one). The reason for doing that is to preserve the history of the original file
(i.e. history from `from-directory`).
Reviewed By: aslpavel
Differential Revision: D24538199
fbshipit-source-id: 792162c4e5ad81fb6949dd95eb1322696ec011ea
Summary: A better fix would be to also get rid of `Option` in the struct itself, but I don't want to spend any more time on this atm, and this change is a clear improvement.
Reviewed By: StanislavGlebik
Differential Revision: D24538309
fbshipit-source-id: 6190c6936dc34d996ecd3d700e5f71282d45f651
Summary: Same as the previous diff, but for push-redirector.
Reviewed By: StanislavGlebik
Differential Revision: D24538027
fbshipit-source-id: 392aee1b9cf0e684486c274c2b54fc2fb719bb3a
Summary:
update Phases::add_reachable_as_public to futures03
With this change all the Phases methods are futures03
Reviewed By: krallin
Differential Revision: D24531552
fbshipit-source-id: e9201621b798739d4d7dd197f15188103e9d359a
Summary:
Fewer clones, better code.
Note that in some cases we would previously have a fn that takes `ctx` by
ownership and just passes it through to some other fn outside of the
`cross_repo_sync`. I triead to make all such functions borrow and clone instead
in order to push cloning to the leaf fns of `cross_repo_sync`.
Reviewed By: StanislavGlebik
Differential Revision: D24538028
fbshipit-source-id: 8a3e78d4076b34d8b1767cdee1db3fdbb7acb4f7
Summary: SQLBlob GC (next diff in stack) will need a ConfigStore in SQLBlob. Make one available to blobstore creation
Reviewed By: krallin
Differential Revision: D24460586
fbshipit-source-id: ea2d5149e0c548844f1fd2a0d241ed0647e137ae
Summary: update phases imports so that BoxFuture is the futures03 version
Reviewed By: krallin
Differential Revision: D24531551
fbshipit-source-id: 1debb007456292fed1113f8c46e019bef27255c2
Summary: Use the new BonsaiDerived::fetch_derived from the walker when scrubbing fsnode derived data. This saves a round trip to derived data and simplifies bonsai_to_fsnode_mapping_step().
Reviewed By: krallin
Differential Revision: D24311164
fbshipit-source-id: b26c6bdef548ca15c0cdbffde675f1a11d8b510e
Summary:
Add BonsaiDerived::derive03 for new callsites that are 0.3/async aware.
Plan is to migrate callsites and then rename BonsaiDerived::derive03 to derive.
Reviewed By: StanislavGlebik
Differential Revision: D24389763
fbshipit-source-id: 4a6f351b9e0c5a2e9a9c47429764f88c47caff08
Summary: No callsites when I search
Reviewed By: StanislavGlebik
Differential Revision: D24389765
fbshipit-source-id: ed1985922a9c1af51afb37fb5e331d32c29fd09e
Summary:
This diff adds a method BonsaiDerived::fetch_derived which is useful for callers currently doing BonsaiDerived::is_derived, and then BonsaiDerived::fetch_derived.
Using the new method cuts out one round trip to derived data, and simplifies calling code. Usage of this in the walker when scrubbing fs nodes is in the next diff.
Reviewed By: StanislavGlebik
Differential Revision: D24311166
fbshipit-source-id: dc52a15076768a6a15ede6dd5872aaaf3f552b66
Summary:
When built with buck, cmake isn't available, switch to a method of echoing that
should be available on every system: powershell.
Reviewed By: genevievehelsel
Differential Revision: D24493404
fbshipit-source-id: 5986b846a211be59f1057b1e7ecd924f3f7380a5
Summary:
D24293498 (ea03d62698) is causing 'hg log scm/metagit' to fail with weird revision
number lookup errors. Let's back it out for now.
Reviewed By: singhsrb
Differential Revision: D24549796
fbshipit-source-id: b10c4caf8fdadd2955ae1f829381c66d788042e6
Summary:
We've had a handful of reports of access denied errors seen during an `hg
update`, ignore them due to:
```
// TODO(T78476916): The access denied are coming from
// PrjMarkDirectoryAsPlaceholder recursively calling into EdenFS, which
// is denied by the BAIL_ON_RECURSIVE_CALL macro.
//
// In theory this means that EdenFS is invalidating a directory that
// isn't materialized, ie: doing useless work. Despite having a negative
// performance impact, this doesn't affect correctness, so ignore for now.
//
// A long term fix will need to not issue invalidation on directories
// that aren't materialized.
```
Reviewed By: wez
Differential Revision: D24544622
fbshipit-source-id: 4f421836b6d8c7b527e860dcf55103e0e239e3ae
Summary: Fix some build warnings that show up on macOS.
Reviewed By: wez
Differential Revision: D24436572
fbshipit-source-id: bd300110df79fe9d259c8f5ce2b1085cb85a0cac
Summary: This is essentially a backout of D24365328 (9f664a8b30).
Reviewed By: DurhamG
Differential Revision: D24544495
fbshipit-source-id: 08f598df0892a8479fac563096f9782038e18dfe
Summary:
Before allowing a bookmark to be moved, check whether the repo is read-only or not,
and deny the bookmark move if the repo is read-only.
This can be bypassed by setting the `BYPASS_READONLY=true` pushvar.
Reviewed By: krallin
Differential Revision: D24063027
fbshipit-source-id: 7b5f66e5d89058f6c5765b020149a81b674e795c
Summary: There's no need to create an empty commit.
Reviewed By: krallin
Differential Revision: D24537264
fbshipit-source-id: 41a80fcbf5cab28b731dae53c58653ec4b434716
Summary: To make it clearer what's going on
Reviewed By: krallin
Differential Revision: D24534961
fbshipit-source-id: 049edbf1b6b648670a6c6ef68b14698a27ea1a0e
Summary:
There are two new options (and one option was renamed):
1) First option (--file-size-limit) adds a way to limit total size of moved files
2) Second option (--lfs-threshold) adds a way to exclude size of the files that
are above the threshold from --file-size-limit calculations. This option can be
useful if we want to limit the total size of the bundle that will be created in
mononoke hg sync job - files that are above or equal to lfs threshold are not included
in the bundle, so no need to count their size.
Reviewed By: krallin
Differential Revision: D24534959
fbshipit-source-id: 6323e4d52802c00dc67270b08f671b0789ab262f
Summary:
a simple command that can be used to pushrebase a commit. Note that this
command is very low level so it skips a lot of checks that we do during the
normal push process (e.g. it doesn't run hooks).
Reviewed By: krallin
Differential Revision: D24534960
fbshipit-source-id: 0f1e27005fa450c86c310d66e3215747ca6a49e2
Summary: This change adds wire types for the history API in the most straightforward way possible. In a future change, I'll move the `WireHistoryEntry` / `HistoryResponseChunk` conversion logic into the `ToApi` implementation. This implementation also doesn't add per-item errors, or standardize `history` to match the protocol evolution standards used by `trees`.
Reviewed By: kulshrax
Differential Revision: D24342046
fbshipit-source-id: d46403a823f2a1e89ad9d6d2074241d8bfe4810e
Summary: This change introduces custo `Serialize` and `Deserialize` implementations for wire types containing a fixed-length byte array. This change is more verbose than should be necessary. Because `TryFrom` is implemented to convert `&[T]` to `[T; N]` and `AsRef` to convert `[T; N]` to `&[T]`, we should be able to use a generic serde `with = ` attribute on the byte array fields of the appropriate structs. Unfortunately, I'm getting a lifetime issue that I haven't been able to resolve, and const generics aren't available on stable to implement the trait bounds without an explicit lifetime.
Reviewed By: kulshrax
Differential Revision: D24460527
fbshipit-source-id: d3a4179b833a523ce164d3df934e4cbdc2202546
Summary:
Removes top-level metadata from `TreeEntry` and add a new, specialized type for carrying child entry metadata, `TreeChildEntry`. This change also fully removes the `revisionstore_types::Metadata` from EdenAPI trees, which is only used on files.
In a follow up change I'll optimize handlings of paths / path segments in `TreeChildEntry` keys. Right now they need to be joined manually.
Reviewed By: kulshrax
Differential Revision: D24434716
fbshipit-source-id: d0739471b1f6cef58b435e10b5fb774bfb08f7f6
Summary: Rather than silently dropping entries which cannot be fetched, this change has the `WireTreeEntry` type carry optional error information, allowing it to be (de)serialized to / from `Result<TreeEntry, EdenApiServerError>` instead of a bare `TreeEntry`. Currently, handling of these failures is up to the individual application code, but it might be useful to introduce utility functions to drop failed entries and log errors.
Reviewed By: kulshrax
Differential Revision: D24315399
fbshipit-source-id: 94e4593b77cf2dc12d0dcc93d174c8a4eda95344
Summary:
`message` is not longer an attribute of an exception object in Python 3. Avoid
using it. This avoids turning rendering `{mirrornode}` template into a crash like:
Traceback (most recent call last):
File "edenscm/hgext/fbscmquery.py", line 78, in mirrornode
return client.getmirroredrev(reponame, "hg", torepo, totype, node)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 399, in getmirroredrev
self._raise_errors(ret)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 504, in _raise_errors
raise ClientError(None, errormsg)
edenscm.hgext.extlib.phabricator.graphql.ClientError: The provided crypto auth token(s) are expired. Check to make sure you have specified a long enough expiration time for your crypto auth token(s).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "edenscm/mercurial/commands/__init__.py", line 4225, in log
displayer.show(
....
File "edenscm/mercurial/templatefilters.py", line 422, in byteify
return b"".join([byteify(t) for t in thing if t is not None])
....
File "edenscm/mercurial/templater.py", line 391, in evalstring
return stringify(evalrawexp(context, mapping, arg))
File "edenscm/mercurial/templater.py", line 350, in evalrawexp
return func(context, mapping, data)
File "edenscm/hgext/fbscmquery.py", line 83, in mirrornode
mapping["repo"].ui.warn(_x(str(e.message) + "\n"))
AttributeError: 'ClientError' object has no attribute 'message'
Reported by: arnabde03
Reviewed By: mofarrell
Differential Revision: D24528362
fbshipit-source-id: 57d093811986f3a65d1201de1f9aed3770db2ad1
Summary: In D24447404, I provided some utility functions that allowed me to avoid constructing and/or passing around a ConfigStore. Remove those functions and fix up the code to run.
Reviewed By: krallin
Differential Revision: D24502692
fbshipit-source-id: 742dbc54fbcf735895d6829745b9317af14dfa0b
Summary: It's designed as a singleton store for normal use - rather than have lots of ways to get different config stores, let's use one global store
Reviewed By: krallin
Differential Revision: D24447404
fbshipit-source-id: 6ecc46351b14794471f654cec98527a11a93d3ef
Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Before replacing the existing debugInodeStatus changes, make some
small refactorings to help me understand it.
Reviewed By: xavierd
Differential Revision: D24299970
fbshipit-source-id: 3b60aa8365bb9a45e12357374282ba837ccc36eb
Summary:
Add a function that makes it easy to traverse EdenFS's view of the
traversed-so-far filesystem.
Reviewed By: xavierd
Differential Revision: D24299962
fbshipit-source-id: 275c849846bf45a2e80780411d60266b961d825b
Summary:
Adding number of changed files to mononoke_commit and mononoke_commit_draft scribe categories.
I need to create scuba table for mononoke_commit scribe category.
Reviewed By: krallin
Differential Revision: D24390738
fbshipit-source-id: 80b9917751a42377b49190cbb2f650a21b086e74
Summary:
- convert ManifestOps to new style futures
- at this point `//eden/manifest:manifest` crate is completely free from old style futures
Reviewed By: krallin
Differential Revision: D24502214
fbshipit-source-id: f1cdb11bd8234f22af5c905243f71e1e9fca11f1
Summary: `derive_manifest_*` functions are now based on new futures and use new bounded_traversal
Reviewed By: krallin
Differential Revision: D24448824
fbshipit-source-id: 5af06851f9c48a73e53daafe1f4972c186a5ca80
Summary: Somewhat more convenient to work with, no need to look inside.
Reviewed By: StanislavGlebik
Differential Revision: D24474898
fbshipit-source-id: 7ee0920e7d0d5a2102c68695a5cc0d9e237d958d
Summary:
Users get confused and try to pull from the wrong repo - e.g. `hg checkout D23665606` in `fbsource` - and find the resulting output about missing hashes scary.
Give them a nice clear error instead.
Reviewed By: wez
Differential Revision: D24485097
fbshipit-source-id: e9aeb7ed6aaf494f828d736ca5203100807fb321
Summary:
indexedlogdatastore is supposed to use remotefilelog.cachelimit to set
the max size, but instead it was setting the max per-log size, which means the
max size was N times bigger. Let's fix that.
Reviewed By: xavierd
Differential Revision: D24483181
fbshipit-source-id: f33cedbfdbb318e9d5eb9fda497645050b93e9fe
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary:
In the next diff in this stack, I'm changing the Rust thrift library as well as
the codegen, and this is causing quite a bit of pain with regard to this
codegen that is checked in to the hg build:
- Regenerating the codegen isn't super obvious.
- The Sandcastle build fails because it uses the current codegen with the old
library.
According to lukaspiatkowski, this has also been a problem in the past when Eden got
migrated to Tokio 0.2, so I'd like to save myself and others some pain by just
not generating the server side codegen and not checking it in, since it's
unused. This reduces the surface of stuff that might go out of sync.
#forcetdhashing
Reviewed By: markbt
Differential Revision: D23649604
fbshipit-source-id: d684bec427431a366de42c88e53072caa98d5b2f
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Previously, when that code was ported on Windows, paths separator were
converted from '\' to '/' when a wide string was provided, all the other paths
were treated as is.
The main issue with this strategy is that not all paths can be converted, the
non-stored ones for instance are immutable, which leads to some subtle bugs
down the line. For instance, the paths: "Z:/foo/bar/baz" and "Z:\foo/bar\baz"
would not be equal as the path separator isn't the same, but both of these are
actually the same path underneath.
To solve this, this diff first introduce a Windows path separator, and then
modifies the path comparison functions to ignore the path separator and only
compare the components.
I'm definitively not a fan of the pattern I use for searching for both / and \
in paths, suggestions are welcome for how to improve that.
Reviewed By: chadaustin
Differential Revision: D24376980
fbshipit-source-id: 0702bf775c7c3937b2138abd5a63d339ac80aaed
Summary:
The path iterator functions are skipping sanity checking of paths as they
assume that the path has already been validated. Unfortunately, that isn't the
case as the only sanity checking we are doing is on the beginning and end of
the string that is passed in. Notably, `RelativePathPiece{"./foo"}` would not
fail the sanity checking even though it it's not made of valid components.
Reviewed By: chadaustin
Differential Revision: D24439804
fbshipit-source-id: 852b3a1180b185cb0bfb96bf5bcdc98b231f32c5
Summary:
As the string is created with MAX_PATH wide characters, a lot of them are going
to stay nul. A future diff adds a sanity check to the PathComponent that
validates that it doesn't contain any nul bytes, which chokes on these paths.
Reviewed By: chadaustin
Differential Revision: D24479715
fbshipit-source-id: e51c4b1c53a3f375664c0c0a1e325ebe302b0cd3
Summary:
When the new rust store is enabled, it can throw exceptions if the rust
code tries to fetch multiple trees when the server doesn't support bfs fetching.
The python code path ignores these requests, letting the eventual single tree
lookup path (i.e. not the prefetch path) perform the download. Let's let the
rust code do the same for rust stores.
Eventually every repo will support bfs fetching and we can delete this code path
entirely.
Reviewed By: quark-zju
Differential Revision: D24459106
fbshipit-source-id: d9a931364bd3235113b0dbcf09cba35a16e16e8e
Summary: Comment out the tests that do not compile on Windows.
Reviewed By: fanzeyi
Differential Revision: D24458438
fbshipit-source-id: 14bd4803ccd68ea6e02f95f868b8a801161518ea
Summary: let's make sure we don't create too big commits by setting the limit
Reviewed By: ikostia
Differential Revision: D24451569
fbshipit-source-id: 7b4f653509fce47d921e55dddad2de49ac5672a3
Summary:
This is a command that creates a commit that copies one directory into another.
Later you can take a commit and push it.
At the moment the command is very simple and also not very useful. At the very
least we need a way to limit the size of the commit, and this option is coming
in the next diff.
Note that the command doesn't overwrite the files if they exist in the target
directory - we may add an option to do so in future. Also if a file exists in
target directory but doesn't exist in source directory then it's left
untouched.
Reviewed By: ikostia
Differential Revision: D24451568
fbshipit-source-id: 2c7b945aadc99a1ca960349992c3684ef02ba35e
Summary: This merely filters out the files that don't compile on mode/win.
Reviewed By: chadaustin
Differential Revision: D24456917
fbshipit-source-id: d314ca2936e064133f2fc6a6199b38d17f2d9ef7
Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.
Reviewed By: wez
Differential Revision: D24437458
fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.
FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.
Reviewed By: wez
Differential Revision: D24039130
fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
This is to be able to run the hook tailer on fbsource commits, which had been
push-redirected from ovrsource.
Reviewed By: StanislavGlebik
Differential Revision: D24447814
fbshipit-source-id: bfb164ec45dd2d4d5c3c83f133b48000a71a50a3
Summary: As well as the unixname of the pusher, log all identities provided by the pusher.
Reviewed By: krallin
Differential Revision: D24446453
fbshipit-source-id: 0854b3a10221da16d87b53fd849f033f7bb2a27e
Summary:
Use the new API `smartset.prefetch` for the prefetching logic for `log`
commands.
Prefetching is only enabled if the template contains related symbols.
Reviewed By: DurhamG
Differential Revision: D24289052
fbshipit-source-id: 810da3b249d45a7516aaf39a0b74f6965b87f96f
Summary: This makes `log` command more efficient using the `hybrid` backend.
Reviewed By: DurhamG
Differential Revision: D24293498
fbshipit-source-id: db3bac28a0b04ba6af449357bc08b285cd62780d
Summary:
Add prefetch APIs on smartset to make it easier to specify what need to be
prefetched.
This is intended to be used by `{phabstatus}`, log with lazy commit text backend,
and revsets like `author(x)`, etc.
Reviewed By: sfilipco
Differential Revision: D24321292
fbshipit-source-id: 7e645ccbe39feeae1e2da7bfb115623945bf9cef
Summary:
This makes it possible to "fork" sets and set different prefetch properties on
them.
Reviewed By: sfilipco
Differential Revision: D24365329
fbshipit-source-id: c4bdd6ab21e6a891a02d450f9e6edab3a3781eca
Summary:
The generatorset has a pure Python implementation for rewindable generator.
However it is not thread-safe, and we want thread-safety since the set
iteration will be driven by async Rust in multiple threads.
One of the issues is, during thread 1 `next(gen)`, thread 2 might call
`next(gen)`, and that's not allowed by Python.
Fix them by switching to the Rust RGenerator.
Reviewed By: DurhamG
Differential Revision: D24365328
fbshipit-source-id: 2785e80c7c460a7f754ed23e3af99f4a5c9fbcdf
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary:
Similarly to the indexedlogdatastore, the LFS stores can become corrupted on
power loss. This is happening fairly frequently on Windows Sandcastle due to
the OS being virtualized and power being cut abruptly.
For now this only attempts to repair the shared stores, in theory we could also
try to repair the local stores but haven't looked into it.
Reviewed By: DurhamG
Differential Revision: D24449202
fbshipit-source-id: 605a7943a0850b625bf00c514879b3da1ab2b406
Summary:
The ContentStore/Metadatastore are made of several different stores, attempting
to expose all of them to Python to drive the repair logic from there would leak
implementation detail of how the stores are implemented.
Instead, let's simply expose a single `repair` function out of the
pyrevisionstore crate that takes care of repairing all of the underlying
stores. For now, this is just moving code around, but a future diff will
integrate the LFS stores.
Reviewed By: DurhamG
Differential Revision: D24449203
fbshipit-source-id: 1631ced9068716453cb404bf7e65cefbf2db5247
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`
Reviewed By: johansglock
Differential Revision: D24448419
fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
Summary: The hostname is of limited use when running on Tupperware - add in the session ID, which lets us look at what happened.
Reviewed By: StanislavGlebik
Differential Revision: D24361320
fbshipit-source-id: 0ac2c64d68915288b3241320c7ee14452b754ec8
Summary:
I removed --debug flag for bookmarks validator on tw because it was too spammy,
but now we get no output at all. Let's make it info so that we can see it in tw
output
Reviewed By: ikostia
Differential Revision: D24445829
fbshipit-source-id: d37c41be46794867f91ecf66f0318e7b2d660d85
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.
Reviewed By: sfilipco
Differential Revision: D24321293
fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.
This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.
Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.
* Technically, if there were a multitude of subscribeStreamTemporary
streams, and all but one was not calling getFilesChangedSince, it's
possible that notifications could back up on the stuck
subscriptions.
Reviewed By: wez
Differential Revision: D24090247
fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
Summary: This will be used to test if a keyword is in an AST.
Reviewed By: DurhamG
Differential Revision: D24293499
fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
Summary:
There were 2 recent reports about "visibleheads" not being enabled.
Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.
There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.
Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.
Reviewed By: DurhamG
Differential Revision: D24340586
fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.
Reviewed By: DurhamG
Differential Revision: D24340585
fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.
Reviewed By: DurhamG
Differential Revision: D24340584
fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).
Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.
Reviewed By: singhsrb
Differential Revision: D24259759
fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary: It will be used to test whether `{phabstatus}` is in template.
Reviewed By: sfilipco
Differential Revision: D24293496
fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
Summary:
There are 2 issues:
- len(repo) is no longer the "max rev" + 1. Use 1<<63 instead.
- If invalidatelinkrev is a repo requirement, linkrev can no longer be trusted.
Reviewed By: DurhamG
Differential Revision: D24270423
fbshipit-source-id: 4e5c999772fad6ef2b1dabd2d6363d4226242c4d
Summary:
If linkrevs are known unreliable, do not use them. This fixes some cases with
segmented changelog, file revisions are incorrectly filtered out and not sent
to client.
Reviewed By: DurhamG
Differential Revision: D24233228
fbshipit-source-id: 268a95b44c0217658b849317cb0a680450b91d23
Summary: This makes the output less verbose.
Reviewed By: DurhamG
Differential Revision: D24411259
fbshipit-source-id: 50b175f2e31ffda62e78dd226775edaf53893bce
Summary:
With segmented changelog, the maximum revision is not always the "tip".
Therefore, track "tip" explicitly in svfs / metalog. This improves
compatibility.
Reviewed By: DurhamG
Differential Revision: D24214711
fbshipit-source-id: 7b22de7ac684a90965a695924bdb9114758aae43
Summary: Avoid HashSet or HashMap order to preserve the order of inserting commits.
Reviewed By: DurhamG
Differential Revision: D24214460
fbshipit-source-id: 66df2e0aba1820e6585f8da66897078f38abf82f
Summary:
Previously bundlerepo only supports revlog changelog. Detect repo configuration
and support non-revlog changelog. This is done by adding commits from bundle in
memory, and updating visibleheads.
Reviewed By: DurhamG
Differential Revision: D24214461
fbshipit-source-id: 63f9af49a0fe0ed2581b611f3cf7002d1b89d2fd
Summary:
The nullid and wdirid are special hg hashes that do not respect SHA1. They were
handled at the bindings layer. However the bindings layer cannot handle them
in a stream. Therefore move it in hgcommits.
Reviewed By: DurhamG
Differential Revision: D24365330
fbshipit-source-id: e8dc6205351ec1a2304252b9ec446dda010e6295
Summary:
Pure Python generator has a few limitations:
- Can only be iterated once.
- Cannot be iterated by multiple threads concurrently.
By converting revset iterators to Rust Stream, they run in different threads
and can context switch in `next()`, breaking the program.
Provide a native reentrant generator implementation to address the issue.
This is sound because Rust methods cannot be interrupted by Python.
Reviewed By: DurhamG
Differential Revision: D24365331
fbshipit-source-id: 885dade922b7863a73203b206a96b492d55bccd0
Summary:
Update the return type of streamcommitrawtext from
`{"vertex": v, "raw_text": t}` to `(v, t)`. This makes it easier to use
in Python, as Python supports `for v, t in ...` but does not support
`for {"vertex": v, "raw_text": t} in ...`.
Reviewed By: DurhamG
Differential Revision: D24295457
fbshipit-source-id: 284a29b9deae2d8509d3afea0fcbcaadbfebbae8
Summary:
In case the server does not respect the input contract and missed
some items without returning errors. The current logic would retry
forever. Change it to detect the issue and raise an error.
Reviewed By: DurhamG
Differential Revision: D24293497
fbshipit-source-id: 09421c7743078a488a9c81ce66fd92c12b39543c
Summary: The store stores sorted(p1,p2)+text to match SHA1 hashes. It's not just `text`.
Reviewed By: DurhamG
Differential Revision: D24325554
fbshipit-source-id: 8a91970f60fb535ca1a5a2d30c7d27f2714f28de
Summary: It's no longer useful as the new abstract interface does not need it.
Reviewed By: sfilipco
Differential Revision: D24399516
fbshipit-source-id: 2b6735d2a26706c6a3e6b592d2f3ecfc874c94cb
Summary:
This verifies the abstraction and simplifies the code.
The new code will use non-master segments for add_heads. Therefore the test
changes.
Reviewed By: sfilipco
Differential Revision: D24399496
fbshipit-source-id: 39067ad88ade79b4f7758bcdaafc03e5f34ced91
Summary: This makes the main namedag.rs cleaner. The next step is to move MemNameDag.
Reviewed By: sfilipco
Differential Revision: D24399495
fbshipit-source-id: c1e79a60edd8597fe7264f04548e5312414241a7
Summary: This is the last non-abstract interface of NameDag.
Reviewed By: sfilipco
Differential Revision: D24399514
fbshipit-source-id: f39bb84a1851a4fe4d1f29e6b0961e6a153c943d
