Commit Graph

6742 Commits

Author SHA1 Message Date
Xavier Deguillard
88c3bf4826 revisionstore: remove translate_lfs_missing
Summary:
As noted in the documentation for it, this can be removed once get and prefetch
return a continuation. This is now done, and thus we can remove it entirely.

Mis-use of it caused data to be fetched twice: once by memcache, and the second
one by getpackv2.

Reviewed By: singhsrb

Differential Revision: D23123344

fbshipit-source-id: 9ac0594faaba94ead04a8bb9035e14809a706641
2020-08-17 17:05:58 -07:00
Durham Goode
fe6cb9dc13 configs: fix handling shared path with trailing new lines
Summary: The python code stripped new lines but the Rust code did not.

Reviewed By: singhsrb

Differential Revision: D23167515

fbshipit-source-id: add33ec6e4cfd9169e6fef8208490e0aeede38bd
2020-08-17 15:53:08 -07:00
Durham Goode
36a49b9b2b rotatelog: fix tests on windows
Summary:
The tests weren't windows compatible. Let's get rid of the path
separators.

Reviewed By: quark-zju, singhsrb

Differential Revision: D23170573

fbshipit-source-id: f934691cd2891205442885a12debe3a28d275fc5
2020-08-17 14:08:26 -07:00
Mark Thomas
526aac3aa2 scribe_commit_queue: include received_timestamp
Summary:
Subscribers to the commit tailer categories would like to know when Mononoke
received the commit.

Reviewed By: StanislavGlebik

Differential Revision: D23162447

fbshipit-source-id: 747214f1964a643f59c491aa08cdbd5c8fe331c8
2020-08-17 13:13:12 -07:00
Liubov Dmitrieva
e6509babef enable automated migration from old backups to cloud sync
Reviewed By: markbt

Differential Revision: D22802064

fbshipit-source-id: c56e65e1a93e8cc94296cf69227bc51eb52f59c3
2020-08-17 10:58:38 -07:00
Mark Thomas
23dee0c931 mononoke_api: add freshness parameter to resolve_bookmark
Summary:
Allow callers of `resolve_bookmark` to specify whether they'd like the most recent value of
the bookmark, rather than one which may be stale.

Use this in the repo_move_bookmark test to avoid flakiness caused by the test code racing against
the warm bookmark cache.

Reviewed By: StanislavGlebik

Differential Revision: D23151427

fbshipit-source-id: 4b8358be1cf103479ccc23a41b2505776543ee49
2020-08-17 09:09:07 -07:00
Mark Thomas
053abf7919 hook_manager_factory: extract construction of the hook manager
Summary:
Extract construction of the hook manager to its own crate, so that we can re-use it.

Eventually the hook manager will become a repo attribute and will be constructed by
the repo attribute factory, but for now it needs its own factory method.

Differential Revision: D23129407

fbshipit-source-id: 302fde4d1ae38c6f61032a32c880018ebf84dee2
2020-08-17 09:09:07 -07:00
Mark Thomas
563137e6f7 hooks: create HookRejection struct
Summary:
Convert hook rejections from a tuple to a named struct.  This will be used in
the bookmarks_movement public interface.

Reviewed By: krallin

Differential Revision: D23077550

fbshipit-source-id: a35476817660c38b8df879ba603b927a7e39be21
2020-08-17 09:09:07 -07:00
Viet Hung Nguyen
d3f3cffe13 mononoke/repo_import: check if repo pushredirects
Summary: Some repos are push-redirected repos: pushes go to another repos, but then synced into this repo. Because of this, when we import a repo into a smaller repo that push-redirects to a large repo, we need to make sure we don't break the large repo with the imported code, since merges, pushes, imports etc. are redirected to the large repo. For now, in order to avoid breaking the large repo, we added a simple check that returns error, if the small repo push-redirects to the large one.

Reviewed By: ikostia

Differential Revision: D23158826

fbshipit-source-id: f722790441d641f67293e78c5d1ea5d1102bbb9b
2020-08-17 06:13:21 -07:00
Mark Thomas
2117806dc7 tests: convert dummyssh and get_free_socket to python3
Summary: Convert dummyssh and get_free_socket to full python3 binaries.

Reviewed By: johansglock

Differential Revision: D23105490

fbshipit-source-id: 6c39c32ba0728cde108b42245acece1d7828ac7c
2020-08-17 02:42:14 -07:00
Durham Goode
33a634167e dynamicconfig: support a disallowlist config
Summary:
This new disallowlist will let us specify config section.key's which
should not be accepted from old rc files. This will let us incrementally disable
loading of those configs from the old files, which will then let us delete them
from the old rc's and eventually delete the old rc's entirely.

This diff also removes hgrc.local and hgrc.od from the list of configs we
verify, since those are not on the list of configs that need to be removed in
this initiative.

Reviewed By: quark-zju

Differential Revision: D23065595

fbshipit-source-id: 5cd742d099efd651174cab5e87bb7cdc4bae8054
2020-08-16 16:56:00 -07:00
Durham Goode
0cf7ebeffe configs: make backingstore load hg configs through the approved path
Summary:
Previously the backing store was loading configs manually. Now that
system, dynamic, user, and repo config loading are unified, let's go through
that approved path.

Reviewed By: kulshrax

Differential Revision: D22736338

fbshipit-source-id: 232023e660107a096691e9d99bf89c04c218dfbd
2020-08-16 16:56:00 -07:00
Durham Goode
6da00020eb configs: move all dynamic and repo config loading out of Python
Summary:
The last few diffs prepared Rust for loading dynamic and repo configs.
This diff finally changes Python so that we're no longer doing any of this work
in Python.

Reviewed By: quark-zju

Differential Revision: D22712625

fbshipit-source-id: 0b71e81d79d10ea3ce7a1b31f315ada5728af9a9
2020-08-16 16:56:00 -07:00
Durham Goode
2da121cb60 configs: add rust support for loading dynamic and repo configs
Summary:
This threads the calls to load_dynamic and load_repo through the Rust
layer up to the Python bindings. This diff does 2 notable things:

1. It adds a reload API for reloading configs in place, versus creating a new
one. This will be used in localrepo.__init__ to construct a new config for the
repo while still maintaining the old pinned values from the copied ui.
2. It threads a repo path and readonly config list from Python down to the Rust
code. This allows load_dynamic and load_repo to operate on the repo path, and
allows the readonly filter to applied to all configs during reloading.

Reviewed By: quark-zju

Differential Revision: D22712623

fbshipit-source-id: a0f372f4971c5feac2f20e89a0fb3fe6d4a65d6f
2020-08-16 16:56:00 -07:00
Durham Goode
6b0014490c configs: implement dynamic and repo config loading in Rust
Summary:
In a future diff we'll enable dynamic and repo config loading purely
from Rust. To do so we need load functions for both cases.  A future diff will
call these.

The dynamicconfig loading is based off the Python equivalent in uiconfig.py

Reviewed By: quark-zju

Differential Revision: D22712624

fbshipit-source-id: ff46f6315fb80d4cd9e31d875ac60264563b12f2
2020-08-16 16:56:00 -07:00
Durham Goode
194e815245 configs: move HGRCPATH loading to load_system
Summary:
Previously load_system would skip loading if HGRCPATH was present and
then load_user would actually load the HGRCPATH. In an upcoming diff I add
load_dynamic, which happens after system but before user. The tests for
dynamicconfig depend on HGRCPATH being loaded when load_dynamic runs, so let's
move HGRCPATH loading up to load_system.

Reviewed By: quark-zju

Differential Revision: D22712627

fbshipit-source-id: 91175d9d7f85b9392ffea4af815a4facebbfe7c1
2020-08-16 16:56:00 -07:00
Durham Goode
ef9ba19dc5 configs: make Options clonable
Summary:
In a future diff we'll allow an outside caller to pass an Options down
to configparsers::hg::load() so that filters can be applied during loading. Inside
hg::load() we need to use the options multiple times with different values, so
let's make Options clonable.

Reviewed By: quark-zju

Differential Revision: D22712626

fbshipit-source-id: 975145f38d35afe7d4a6c8e87071b0fb0ae74797
2020-08-16 16:55:59 -07:00
Durham Goode
0cea385252 configs: remove config from repo.rs API
Summary:
A future diff will move all dynamic and repo config loading to be in
configparser. As part of this, let's simplify the repo.rs API to not pass
configs around everywhere.

Reviewed By: quark-zju

Differential Revision: D22712628

fbshipit-source-id: 79f23991aa826ce8b4f7430b45d7702efdc6b982
2020-08-16 16:55:59 -07:00
Durham Goode
26564596a1 utils: add background process utility
Summary:
Similar to the Python runbgcommand (extutil.py), this is a Rust utility that runs a
detached background process in a cross platform way.

This will be used in a later diff to run dynamicconfig generation in the
background.

Reviewed By: quark-zju

Differential Revision: D22712629

fbshipit-source-id: a317465bf03c96d977a203678e2bef13ce57cc12
2020-08-16 16:55:59 -07:00
Durham Goode
0b123ba41d configs: move Rust dynamicconfig generation into configparser::hg
Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.

Future diffs will move config loading in as well.

Reviewed By: quark-zju

Differential Revision: D22590208

fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
2020-08-16 16:55:59 -07:00
Durham Goode
7ff28d3e1c configs: move dynamicconfig into configparser
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.

Reviewed By: quark-zju

Differential Revision: D22587237

fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
2020-08-16 16:55:59 -07:00
Durham Goode
a40331be8d configs: unify system+user config loading into pure rust layer
Summary:
We had two spots that loaded system and user configs, one in the
pyconfigparser layer, and one in the pure rust config layer. In an upcoming diff
I'd like to move dynamicconfig loading down into the pure rust layer, so let's
unify these.

Reviewed By: quark-zju

Differential Revision: D22585554

fbshipit-source-id: 0cea7801ae1d5a3a3c12b80ee23b37f9e690e2bc
2020-08-16 16:55:59 -07:00
Durham Goode
3129f032a4 contentstore: make history rotatelog size configurable
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.

Reviewed By: quark-zju

Differential Revision: D23089539

fbshipit-source-id: ebfc3beaf3c0fe5b01b87d97c19455b0a24afa72
2020-08-16 16:44:16 -07:00
Durham Goode
b821ab3766 contentstore: make data rotatelog size configurable
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.

Reviewed By: quark-zju

Differential Revision: D23089541

fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
2020-08-16 16:44:16 -07:00
Durham Goode
76d3d46837 revisionstore: remove from_path from LocalStore
Summary:
from_path is an awkward constructor because it doesn't pass any other
information, like a config object. It also requires that the constructor be very
generic across all the stores. Right now it's only needed for pack files, so
let's move it to it's own trait that is limited to pack files.

This will allow us to make the indexedlog store constructors more versatile in a
later diff. Once we get rid of pack files we can delete the StoreFromPath trait
entirely.

Reviewed By: xavierd

Differential Revision: D23089542

fbshipit-source-id: ea6c50853e5d5390a029002ef5d15c74fe41fe69
2020-08-16 16:44:16 -07:00
Jun Wu
f61aaf3a15 repo: add dageval API
Summary:
Similar to `changelog.dageval`, but provides extra functions like `public`,
`draft`, `obsolete`, etc.

Reviewed By: sfilipco

Differential Revision: D23036070

fbshipit-source-id: b985f2b338a3dce11bddf53c00c30e4887762676
2020-08-14 22:00:26 -07:00
Jun Wu
a3531f9a8e changelog2: improve dageval
Summary:
Make it work with closure and `lambda dag: dag.only(...)`.
This is useful when there is a name conflict with a local variable, like:

   only = ...
   cl.dageval(lambda: only(...)) # only refers to the local variable.
   cl.dageval(lambda dag: dag.only(...)) # only refers to the dag operation.

Besides, drop `func_` so they are Py3 compatible.

Reviewed By: sfilipco

Differential Revision: D23036064

fbshipit-source-id: 8d32a34c51b6ba945cda5be313f9d71464f813b7
2020-08-14 22:00:26 -07:00
Jun Wu
2db783bed8 revlogindex: make parent_revs fallible
Summary: If parent_revs gets an out-of-bound rev, it should fail.

Reviewed By: sfilipco

Differential Revision: D23036071

fbshipit-source-id: 7fae0fd5adf07ac3c933a29d7d06289d8d740c60
2020-08-14 22:00:26 -07:00
Jun Wu
0f838d7abf revlogindex: fix \0 header handling
Summary:
If the text starts with `\0`, the `\0` should be considered as part of the
uncompressed text instead of a separated header.

Reviewed By: sfilipco

Differential Revision: D22970575

fbshipit-source-id: 49e8a1a1ea42a3c4cf153b70f59fd0558dcfcede
2020-08-14 22:00:26 -07:00
Jun Wu
54a1a620d0 revlogindex: fix parent handling
Summary:
The parent handling is unsound when there are revs that are skipped. Fix it by
reasoning about commit hashes for parents.

Reviewed By: sfilipco

Differential Revision: D23036078

fbshipit-source-id: 8f710171471025cd48b3bd8f6ea57c68330eb8b8
2020-08-14 22:00:26 -07:00
Xavier Deguillard
f4f159537f utils: add a platform independent FileUtils
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.

To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.

We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.

Reviewed By: wez

Differential Revision: D23037325

fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
2020-08-14 18:56:33 -07:00
Arun Kulshreshtha
523013c808 cmdlib: remove extra comment slashes
Reviewed By: quark-zju

Differential Revision: D23111025

fbshipit-source-id: b8606c322439c41097d739df59b551a8432e7fe4
2020-08-14 18:04:26 -07:00
Xavier Deguillard
17776fbbc1 win: various micro-optimization for opendir/readdir
Summary:
After enabling strace profiling for the dispatcher, it became very clear that
once files are populated on disk, most of the notification in edenfs come from
listing directories. While looking at the code to find any improvements, I
picked up a couple of low hanging fruits and cleanup that should help (not
measured).

Reviewed By: chadaustin

Differential Revision: D23121434

fbshipit-source-id: 8a8b369f3be6ffb6097e6e12ab0f07af0a6d56b7
2020-08-14 17:35:50 -07:00
Xavier Deguillard
76ea3054de win: conditionally enable negative path caching
Summary:
From a quick experiment, this greatly cuts down on the amount requests to
nonexistent files. For instance, the .hg directory in folders is now only
looked up once and no longer afterwards.

Reviewed By: wez

Differential Revision: D23112343

fbshipit-source-id: 223134ca591054ae9ac2e839033bbd1b714443da
2020-08-14 17:35:50 -07:00
Xavier Deguillard
3fb1b56ea7 win: remove PrjfsChannel::removeCachedFile
Summary: This function was unused, remove it.

Reviewed By: wez

Differential Revision: D23112344

fbshipit-source-id: e37dd1e07067cf457eeb5d6c0af35db3dfab9b43
2020-08-14 17:35:50 -07:00
Xavier Deguillard
79c8f0e86e win: unify the EdenDispatcher lifetime management
Summary:
This is purely moving code around to be more similar between Unices/Windows.
At some point in the future, I'd like to see just one EdenDispatcher class in
the shared code with as few platform specific code (ie: ifdef) as possible,
this is small step in this direction.

Reviewed By: wez

Differential Revision: D23112345

fbshipit-source-id: feb276040aea106d5951af30c5d445b4cb654e01
2020-08-14 17:35:50 -07:00
Xavier Deguillard
f1da417cea win: use the strace logger
Summary:
On Linux, all the notifications are sent to the strace logger, let's do the
same on Windows. Whenever possible, I've tried to use the name of the Linux
syscalls that correspond to the notification.

Reviewed By: chadaustin

Differential Revision: D23105536

fbshipit-source-id: 31c6e545cdec93b0f751682e34c34b894b2890c4
2020-08-14 16:01:27 -07:00
Xavier Deguillard
e0ec7d8896 fsinfo: fix cargo test
Summary:
Somehow `make local` doesn't complain about missing features, but `cargo test`
does.

Reviewed By: singhsrb

Differential Revision: D23132496

fbshipit-source-id: cdbfe1faf194d61d86493a760e45fd38087d2956
2020-08-14 13:38:41 -07:00
Skotch Vail
0db33b30c5 sparse: debugsparsematch accepts filesets
Summary: Use filesets so that we don't have to worry about exceeding command line

Reviewed By: quark-zju

Differential Revision: D23008329

fbshipit-source-id: bbfef9ecf23b810f4fa6ca8a6de1bbeb9330ae09
2020-08-14 09:46:32 -07:00
Skotch Vail
8780a3f66b sparse: debugsparsematch temp, multiple, exclude sparse profiles
Summary: This diff has a unit test to make sure on the fly sparse profiles work. We also allow for more than one sparse profile, and for multiple exclusion sparse profiles.

Reviewed By: quark-zju

Differential Revision: D23004062

fbshipit-source-id: 15f00401084c89a77d8e7bdd8c7eb688e20d131b
2020-08-14 09:46:32 -07:00
Tao Chen
f55d4fc261 Remove old getSelection APIs from ServiceSelectorCache
Summary: Remove old getSelection APIs from ServiceSelectorCache

Reviewed By: ovoietsa

Differential Revision: D23070503

fbshipit-source-id: ea013e43215b8bc7aa463efe7b374d67749dda97
2020-08-14 03:21:12 -07:00
Alex Hornby
213edc10ce mononoke: limit queue peeking from scrub blobstore when one store has missing value
Summary: When running manual scrub for a large repo with one empty store, we are doing one peek per key.  For keys that have existed for some time this is unnecessary as we know the key should exist and slows down the scrub.

Reviewed By: farnz

Differential Revision: D23054582

fbshipit-source-id: d2222350157ca37aa31b7792214af4446129c692
2020-08-14 02:37:45 -07:00
Mark Thomas
dba3ef35ca repo_client: extract common calls to bookmarks_movement
Summary:
Extract the calls to bookmarks_movement to separate functions to avoid duplication and
make the post-resolve action functions easier to read.

Reviewed By: StanislavGlebik

Differential Revision: D23057045

fbshipit-source-id: c6b5a8cdb2399e89c174c3df844529d4b5309edf
2020-08-14 02:28:56 -07:00
Mark Thomas
c529e6a527 bookmarks_movement: refactor bookmark movement for pushrebase
Summary: Refactor control of movement of non-scratch bookmarks through pushrebase.

Reviewed By: krallin

Differential Revision: D22920694

fbshipit-source-id: 347777045b4995b69973118781511686cf34bdba
2020-08-14 02:28:55 -07:00
Mark Thomas
a16b88d1c5 pushrebase: remove OntoBookmarkParams and clean up interface
Summary:
Some parts of the `pushrebase` public interface will be re-exported from `bookmarks_movement`.

Clean these up in preparation:

* Remove `OntoBookmarkParams` as it is now a simple wrapper around `BookmarkName` that
  prevents us from using a reference.

* Make the bundle replay data `Option<&T>` rather than `&Option<T>`, allowing us to
  use the former when available.  The latter can be readily converted with `.as_ref()`.

* Rename `SuccessResult` to `Outcome` and `ErrorKind` to `InternalError`.

Reviewed By: krallin

Differential Revision: D23055580

fbshipit-source-id: 1208a934f979a9d5eb73310fb8711b1291393ecf
2020-08-14 02:28:55 -07:00
Mark Thomas
c59c2979d2 bookmarks_movement: refactor bookmark movement for force-pushrebase
Summary:
Refactor control of movement of non-scratch bookmarks through force-pushrebase
or bookmark-only pushrebase.  These are equivalent to ordinary pushes, and so
can use the same code path for moving the bookmarks.

This has the side-effect of enabling some patterns that were previously not
possible, like populating git mappings with a force-pushrebase.

Reviewed By: ikostia

Differential Revision: D22844828

fbshipit-source-id: 4ef71fa4cef69cc2f1d124837631e8304644ca06
2020-08-14 02:28:54 -07:00
Mark Thomas
279c3dcd8f bookmarks_movement: refactor bookmark movement for push
Summary: Refactor control of movement of non-scratch bookmarks through plain pushes.

Reviewed By: krallin

Differential Revision: D22844829

fbshipit-source-id: 2f1a89e1d0f69880f74b7bc135144bfb305a918e
2020-08-14 02:28:54 -07:00
Mark Thomas
e0bdff5188 bookmarks_movement: refactor scratch bookmark movement
Summary:
Refactor control of movement of scratch bookmarks to a new `bookmark_movement` crate
that will contain all bookmark movement controls.

Reviewed By: krallin

Differential Revision: D22844830

fbshipit-source-id: 56d25ad45a9328eaa079c13466b4b802f033d1dd
2020-08-14 02:28:53 -07:00
Alex Hornby
61046e3adb rust: point interment crate to internment master
Summary: Update internment to point at its latest master branch commit.  Upstream has merged my PR to use DashMap inside internment, but they haven't cut a new crates release yet.

Reviewed By: jsgf, krallin

Differential Revision: D23075070

fbshipit-source-id: 8f4ec0e3ddbefd672c3040fb174d1cf5f6c1a94a
2020-08-14 02:15:24 -07:00
Alex Hornby
d59dd787c5 mononoke: make blobstore ctime a bit easier to use
Summary: Ctime is an Option<i64>, so rather than as_ctime()/into_ctime() use the fact that it's fairly small and Copy to just .ctime()

Reviewed By: krallin

Differential Revision: D23081739

fbshipit-source-id: be62912eca02e5c29d7473d6f386d98df11000dd
2020-08-14 02:09:46 -07:00