Commit Graph

59967 Commits

Author SHA1 Message Date
Meyer Jacobs
a01e4e3503 filemerge: Fix regex to use bytestring pattern for bytestring data.
Summary: In py3, a string pattern cannot be used to match against a bytestring text. Convert the pattern to a bytestring, which should be compatible with python2 without additional changes.

Reviewed By: DurhamG

Differential Revision: D23135472

fbshipit-source-id: fcc0b111c646011f5b556f032ef85ef326cb9511
2020-08-18 09:14:07 -07:00
Viet Hung Nguyen
e1b088d315 mononoke/repo_import: moved tests
Summary: Minor refactor: moved the tests from main to their own file, because main was getting too large and found it hard to navigate through the code.

Reviewed By: StanislavGlebik

Differential Revision: D23188766

fbshipit-source-id: a2b2e32c77587f95c07a0bb02a4957e3671dd2c6
2020-08-18 09:09:14 -07:00
Johan Schuijt-Li
4e0660a94c asyncify connection accepting
Summary:
This largely moves connection accepting from old style bytes, futures and tokio
to updated versions, while keeping some parts at old bytes/futures in order to
remain compatible with the rest of the Mononoke codebase.

Division lies on `Stdio` which maintains old channels, stream and futures,
while the socket handling, connection acception and wire encoding is updated.

With the updated futures, we now wait for the forwarding stream to have
succeeded before considering a connection fully handled.

Other notable changes:
 - futures_ext now a mini codec Decoder instead of relying on NetstringDecoder,
   which has been updated to use bytes 0.5
 - hgcli has been modified to use updated NetstringDecoder
 - netstring now requires the updated bytes 0.5 crate
 - the part in connection_acceptor was handling repo/security logic is now part of repo_handler (as it should have been), connection_acceptor now only handles networking and framing
 - tests now verify that the shutdown handler is triggered

Reviewed By: krallin

Differential Revision: D22526867

fbshipit-source-id: 34e43af4a0c8b84de0000f2093d7fffd3fb0e20d
2020-08-18 09:09:14 -07:00
Koray Polat
ad6393d8d4 Add option to run tests without Testpilot even when it's available
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.

Reviewed By: wez

Differential Revision: D23161362

fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
2020-08-18 09:06:53 -07:00
svcscm
ea519fe4d9 Updating submodules
Summary:
GitHub commits:

696d36e4c8
d6e4753c2c

Reviewed By: jurajh-fb

fbshipit-source-id: 7468f781c81566ca01037e4d9c2c9a04f4de82de
2020-08-18 09:06:53 -07:00
svcscm
ab3a42d278 Updating submodules
Summary:
GitHub commits:

475f4d573d
93b4b9ee46
8766680a2f
5e4f665cd6
e72195fd60
32837139c6
92593d511a
e363323bdd
f1dbf24b94
5a2fa04d2a
9daffaea48
ab6ea35582
25a061b5c9
8fe2b7d60c
ba0384954a

Reviewed By: jurajh-fb

fbshipit-source-id: 4aa31bbb4718c847131ad9521ad71be1a0b16cf7
2020-08-17 18:14:32 -07:00
Chad Austin
c12f60afb5 only use symbolizer if libunwind is found
Summary: folly/experimental/symbolizer requires libunwind. Do not enable it unless libunwind is available.

Reviewed By: yfeldblum, luciang

Differential Revision: D22964401

fbshipit-source-id: d71991346927fbf18167637770b2143c03b16476
2020-08-17 17:08:55 -07:00
Xavier Deguillard
88c3bf4826 revisionstore: remove translate_lfs_missing
Summary:
As noted in the documentation for it, this can be removed once get and prefetch
return a continuation. This is now done, and thus we can remove it entirely.

Mis-use of it caused data to be fetched twice: once by memcache, and the second
one by getpackv2.

Reviewed By: singhsrb

Differential Revision: D23123344

fbshipit-source-id: 9ac0594faaba94ead04a8bb9035e14809a706641
2020-08-17 17:05:58 -07:00
Durham Goode
fe6cb9dc13 configs: fix handling shared path with trailing new lines
Summary: The python code stripped new lines but the Rust code did not.

Reviewed By: singhsrb

Differential Revision: D23167515

fbshipit-source-id: add33ec6e4cfd9169e6fef8208490e0aeede38bd
2020-08-17 15:53:08 -07:00
Durham Goode
36a49b9b2b rotatelog: fix tests on windows
Summary:
The tests weren't windows compatible. Let's get rid of the path
separators.

Reviewed By: quark-zju, singhsrb

Differential Revision: D23170573

fbshipit-source-id: f934691cd2891205442885a12debe3a28d275fc5
2020-08-17 14:08:26 -07:00
Mark Thomas
526aac3aa2 scribe_commit_queue: include received_timestamp
Summary:
Subscribers to the commit tailer categories would like to know when Mononoke
received the commit.

Reviewed By: StanislavGlebik

Differential Revision: D23162447

fbshipit-source-id: 747214f1964a643f59c491aa08cdbd5c8fe331c8
2020-08-17 13:13:12 -07:00
svcscm
e29b75beae Updating submodules
Summary:
GitHub commits:

562bbaa2c7
6d492e885d
227f041e35
49b1f90bdd
1babfa857a
919b390d81
0ebfa43666
37050483ca
c68a800622
d6729b40bd
7b85e28fa8
3eaffcca42

Reviewed By: jurajh-fb

fbshipit-source-id: 657232b45379446deaa3a8ec8605aaecce9ad163
2020-08-17 11:57:19 -07:00
Jonathan Kim
5cb2331899 Update folly's source of truth to xplat
Reviewed By: mzlee

Differential Revision: D23066827

fbshipit-source-id: 6362acb7a6680a24871859d7874a9caec0a1538e
2020-08-17 11:01:52 -07:00
Liubov Dmitrieva
e6509babef enable automated migration from old backups to cloud sync
Reviewed By: markbt

Differential Revision: D22802064

fbshipit-source-id: c56e65e1a93e8cc94296cf69227bc51eb52f59c3
2020-08-17 10:58:38 -07:00
Mark Thomas
23dee0c931 mononoke_api: add freshness parameter to resolve_bookmark
Summary:
Allow callers of `resolve_bookmark` to specify whether they'd like the most recent value of
the bookmark, rather than one which may be stale.

Use this in the repo_move_bookmark test to avoid flakiness caused by the test code racing against
the warm bookmark cache.

Reviewed By: StanislavGlebik

Differential Revision: D23151427

fbshipit-source-id: 4b8358be1cf103479ccc23a41b2505776543ee49
2020-08-17 09:09:07 -07:00
Mark Thomas
053abf7919 hook_manager_factory: extract construction of the hook manager
Summary:
Extract construction of the hook manager to its own crate, so that we can re-use it.

Eventually the hook manager will become a repo attribute and will be constructed by
the repo attribute factory, but for now it needs its own factory method.

Differential Revision: D23129407

fbshipit-source-id: 302fde4d1ae38c6f61032a32c880018ebf84dee2
2020-08-17 09:09:07 -07:00
Mark Thomas
563137e6f7 hooks: create HookRejection struct
Summary:
Convert hook rejections from a tuple to a named struct.  This will be used in
the bookmarks_movement public interface.

Reviewed By: krallin

Differential Revision: D23077550

fbshipit-source-id: a35476817660c38b8df879ba603b927a7e39be21
2020-08-17 09:09:07 -07:00
Viet Hung Nguyen
d3f3cffe13 mononoke/repo_import: check if repo pushredirects
Summary: Some repos are push-redirected repos: pushes go to another repos, but then synced into this repo. Because of this, when we import a repo into a smaller repo that push-redirects to a large repo, we need to make sure we don't break the large repo with the imported code, since merges, pushes, imports etc. are redirected to the large repo. For now, in order to avoid breaking the large repo, we added a simple check that returns error, if the small repo push-redirects to the large one.

Reviewed By: ikostia

Differential Revision: D23158826

fbshipit-source-id: f722790441d641f67293e78c5d1ea5d1102bbb9b
2020-08-17 06:13:21 -07:00
svcscm
7b916a0c52 Updating submodules
Summary:
GitHub commits:

e4be93cbb8

Reviewed By: yns88

fbshipit-source-id: 6cd4078dcd2652904f5a8fa0a84b8b83c4564f21
2020-08-17 03:13:33 -07:00
Mark Thomas
2117806dc7 tests: convert dummyssh and get_free_socket to python3
Summary: Convert dummyssh and get_free_socket to full python3 binaries.

Reviewed By: johansglock

Differential Revision: D23105490

fbshipit-source-id: 6c39c32ba0728cde108b42245acece1d7828ac7c
2020-08-17 02:42:14 -07:00
svcscm
166a45e975 Updating submodules
Summary:
GitHub commits:

c8aa853cb5
2436435602

Reviewed By: yns88

fbshipit-source-id: 9438ece6b6a91a723d7fb9b16488b4984865f783
2020-08-17 02:42:14 -07:00
svcscm
32f5a2a7fd Updating submodules
Summary:
GitHub commits:

9fe9e1ed32
72530db795
97ea5d6bb1
516b76e65e
37c450a75b
3d3c2f89f5
5545fd377c
f50740edd8

Reviewed By: yns88

fbshipit-source-id: 651f44eded2da6d7b96d886a018a60c81578aed2
2020-08-16 20:43:44 -07:00
Durham Goode
33a634167e dynamicconfig: support a disallowlist config
Summary:
This new disallowlist will let us specify config section.key's which
should not be accepted from old rc files. This will let us incrementally disable
loading of those configs from the old files, which will then let us delete them
from the old rc's and eventually delete the old rc's entirely.

This diff also removes hgrc.local and hgrc.od from the list of configs we
verify, since those are not on the list of configs that need to be removed in
this initiative.

Reviewed By: quark-zju

Differential Revision: D23065595

fbshipit-source-id: 5cd742d099efd651174cab5e87bb7cdc4bae8054
2020-08-16 16:56:00 -07:00
Durham Goode
0cf7ebeffe configs: make backingstore load hg configs through the approved path
Summary:
Previously the backing store was loading configs manually. Now that
system, dynamic, user, and repo config loading are unified, let's go through
that approved path.

Reviewed By: kulshrax

Differential Revision: D22736338

fbshipit-source-id: 232023e660107a096691e9d99bf89c04c218dfbd
2020-08-16 16:56:00 -07:00
Durham Goode
6da00020eb configs: move all dynamic and repo config loading out of Python
Summary:
The last few diffs prepared Rust for loading dynamic and repo configs.
This diff finally changes Python so that we're no longer doing any of this work
in Python.

Reviewed By: quark-zju

Differential Revision: D22712625

fbshipit-source-id: 0b71e81d79d10ea3ce7a1b31f315ada5728af9a9
2020-08-16 16:56:00 -07:00
Durham Goode
2da121cb60 configs: add rust support for loading dynamic and repo configs
Summary:
This threads the calls to load_dynamic and load_repo through the Rust
layer up to the Python bindings. This diff does 2 notable things:

1. It adds a reload API for reloading configs in place, versus creating a new
one. This will be used in localrepo.__init__ to construct a new config for the
repo while still maintaining the old pinned values from the copied ui.
2. It threads a repo path and readonly config list from Python down to the Rust
code. This allows load_dynamic and load_repo to operate on the repo path, and
allows the readonly filter to applied to all configs during reloading.

Reviewed By: quark-zju

Differential Revision: D22712623

fbshipit-source-id: a0f372f4971c5feac2f20e89a0fb3fe6d4a65d6f
2020-08-16 16:56:00 -07:00
Durham Goode
6b0014490c configs: implement dynamic and repo config loading in Rust
Summary:
In a future diff we'll enable dynamic and repo config loading purely
from Rust. To do so we need load functions for both cases.  A future diff will
call these.

The dynamicconfig loading is based off the Python equivalent in uiconfig.py

Reviewed By: quark-zju

Differential Revision: D22712624

fbshipit-source-id: ff46f6315fb80d4cd9e31d875ac60264563b12f2
2020-08-16 16:56:00 -07:00
Durham Goode
194e815245 configs: move HGRCPATH loading to load_system
Summary:
Previously load_system would skip loading if HGRCPATH was present and
then load_user would actually load the HGRCPATH. In an upcoming diff I add
load_dynamic, which happens after system but before user. The tests for
dynamicconfig depend on HGRCPATH being loaded when load_dynamic runs, so let's
move HGRCPATH loading up to load_system.

Reviewed By: quark-zju

Differential Revision: D22712627

fbshipit-source-id: 91175d9d7f85b9392ffea4af815a4facebbfe7c1
2020-08-16 16:56:00 -07:00
Durham Goode
ef9ba19dc5 configs: make Options clonable
Summary:
In a future diff we'll allow an outside caller to pass an Options down
to configparsers::hg::load() so that filters can be applied during loading. Inside
hg::load() we need to use the options multiple times with different values, so
let's make Options clonable.

Reviewed By: quark-zju

Differential Revision: D22712626

fbshipit-source-id: 975145f38d35afe7d4a6c8e87071b0fb0ae74797
2020-08-16 16:55:59 -07:00
Durham Goode
0cea385252 configs: remove config from repo.rs API
Summary:
A future diff will move all dynamic and repo config loading to be in
configparser. As part of this, let's simplify the repo.rs API to not pass
configs around everywhere.

Reviewed By: quark-zju

Differential Revision: D22712628

fbshipit-source-id: 79f23991aa826ce8b4f7430b45d7702efdc6b982
2020-08-16 16:55:59 -07:00
Durham Goode
26564596a1 utils: add background process utility
Summary:
Similar to the Python runbgcommand (extutil.py), this is a Rust utility that runs a
detached background process in a cross platform way.

This will be used in a later diff to run dynamicconfig generation in the
background.

Reviewed By: quark-zju

Differential Revision: D22712629

fbshipit-source-id: a317465bf03c96d977a203678e2bef13ce57cc12
2020-08-16 16:55:59 -07:00
Durham Goode
0b123ba41d configs: move Rust dynamicconfig generation into configparser::hg
Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.

Future diffs will move config loading in as well.

Reviewed By: quark-zju

Differential Revision: D22590208

fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
2020-08-16 16:55:59 -07:00
Durham Goode
7ff28d3e1c configs: move dynamicconfig into configparser
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.

Reviewed By: quark-zju

Differential Revision: D22587237

fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
2020-08-16 16:55:59 -07:00
Durham Goode
a40331be8d configs: unify system+user config loading into pure rust layer
Summary:
We had two spots that loaded system and user configs, one in the
pyconfigparser layer, and one in the pure rust config layer. In an upcoming diff
I'd like to move dynamicconfig loading down into the pure rust layer, so let's
unify these.

Reviewed By: quark-zju

Differential Revision: D22585554

fbshipit-source-id: 0cea7801ae1d5a3a3c12b80ee23b37f9e690e2bc
2020-08-16 16:55:59 -07:00
Durham Goode
3129f032a4 contentstore: make history rotatelog size configurable
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.

Reviewed By: quark-zju

Differential Revision: D23089539

fbshipit-source-id: ebfc3beaf3c0fe5b01b87d97c19455b0a24afa72
2020-08-16 16:44:16 -07:00
Durham Goode
b821ab3766 contentstore: make data rotatelog size configurable
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.

Reviewed By: quark-zju

Differential Revision: D23089541

fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
2020-08-16 16:44:16 -07:00
Durham Goode
76d3d46837 revisionstore: remove from_path from LocalStore
Summary:
from_path is an awkward constructor because it doesn't pass any other
information, like a config object. It also requires that the constructor be very
generic across all the stores. Right now it's only needed for pack files, so
let's move it to it's own trait that is limited to pack files.

This will allow us to make the indexedlog store constructors more versatile in a
later diff. Once we get rid of pack files we can delete the StoreFromPath trait
entirely.

Reviewed By: xavierd

Differential Revision: D23089542

fbshipit-source-id: ea6c50853e5d5390a029002ef5d15c74fe41fe69
2020-08-16 16:44:16 -07:00
svcscm
5dc7a2a7e8 Updating submodules
Summary:
GitHub commits:

013b287cd9
7d1e1f8b53

Reviewed By: yns88

fbshipit-source-id: ba621446ab7f0d12d933f40897dd7fd35232beb2
2020-08-16 13:22:12 -07:00
svcscm
e418d3ad19 Updating submodules
Summary:
GitHub commits:

cf43d53b74

Reviewed By: yns88

fbshipit-source-id: 266cb07d4b6af56db60dc8dfe1b7015a4c316c7f
2020-08-15 13:11:36 -07:00
svcscm
195e0ed0a5 Updating submodules
Summary:
GitHub commits:

40df0f8a64

Reviewed By: yns88

fbshipit-source-id: 405213b1db6e7cd56d0f7dbdccead9eaa7d19490
2020-08-15 13:11:36 -07:00
svcscm
6df02c3aa6 Updating submodules
Summary:
GitHub commits:

03e2b8b90e

Reviewed By: yns88

fbshipit-source-id: 9de29685d96ec96fbdf7eb23a2789968ffe49dc3
2020-08-15 03:55:48 -07:00
Jun Wu
f61aaf3a15 repo: add dageval API
Summary:
Similar to `changelog.dageval`, but provides extra functions like `public`,
`draft`, `obsolete`, etc.

Reviewed By: sfilipco

Differential Revision: D23036070

fbshipit-source-id: b985f2b338a3dce11bddf53c00c30e4887762676
2020-08-14 22:00:26 -07:00
Jun Wu
a3531f9a8e changelog2: improve dageval
Summary:
Make it work with closure and `lambda dag: dag.only(...)`.
This is useful when there is a name conflict with a local variable, like:

   only = ...
   cl.dageval(lambda: only(...)) # only refers to the local variable.
   cl.dageval(lambda dag: dag.only(...)) # only refers to the dag operation.

Besides, drop `func_` so they are Py3 compatible.

Reviewed By: sfilipco

Differential Revision: D23036064

fbshipit-source-id: 8d32a34c51b6ba945cda5be313f9d71464f813b7
2020-08-14 22:00:26 -07:00
Jun Wu
2db783bed8 revlogindex: make parent_revs fallible
Summary: If parent_revs gets an out-of-bound rev, it should fail.

Reviewed By: sfilipco

Differential Revision: D23036071

fbshipit-source-id: 7fae0fd5adf07ac3c933a29d7d06289d8d740c60
2020-08-14 22:00:26 -07:00
Jun Wu
0f838d7abf revlogindex: fix \0 header handling
Summary:
If the text starts with `\0`, the `\0` should be considered as part of the
uncompressed text instead of a separated header.

Reviewed By: sfilipco

Differential Revision: D22970575

fbshipit-source-id: 49e8a1a1ea42a3c4cf153b70f59fd0558dcfcede
2020-08-14 22:00:26 -07:00
Jun Wu
54a1a620d0 revlogindex: fix parent handling
Summary:
The parent handling is unsound when there are revs that are skipped. Fix it by
reasoning about commit hashes for parents.

Reviewed By: sfilipco

Differential Revision: D23036078

fbshipit-source-id: 8f710171471025cd48b3bd8f6ea57c68330eb8b8
2020-08-14 22:00:26 -07:00
svcscm
6e8017fde9 Updating submodules
Summary:
GitHub commits:

bc59f8b96f
69760b4d05

Reviewed By: yns88

fbshipit-source-id: b0bfc8b29cecbaa946a5cb16029a1ee6538ee1cd
2020-08-14 22:00:25 -07:00
svcscm
ca69914ed7 Updating submodules
Summary:
GitHub commits:

0e46975f32
e93f14fb7f
dc5a4252e1
209bde495e
01f4612acd

Reviewed By: yns88

fbshipit-source-id: ac83f388b01dc9745600c8ea3a36e1eb4380c189
2020-08-14 18:59:36 -07:00
Xavier Deguillard
f4f159537f utils: add a platform independent FileUtils
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.

To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.

We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.

Reviewed By: wez

Differential Revision: D23037325

fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
2020-08-14 18:56:33 -07:00
Arun Kulshreshtha
523013c808 cmdlib: remove extra comment slashes
Reviewed By: quark-zju

Differential Revision: D23111025

fbshipit-source-id: b8606c322439c41097d739df59b551a8432e7fe4
2020-08-14 18:04:26 -07:00