Commit Graph

1501 Commits

Author SHA1 Message Date
Jun Wu
51dde1bcef interactiveui: do not import termios on Windows
Summary: `termios` is not avialable on Windows. Do not import it.

Reviewed By: DurhamG

Differential Revision: D21258999

fbshipit-source-id: f4390b69fe9abceea8b1959e7506c1558778f980
2020-04-28 10:26:21 -07:00
Jun Wu
d479053954 metalog: support exporting to a git repo
Summary:
I wanted to figure out "who added this visible head", "what is the difference
between this metalog root and that root". Those are actually source control
operations (blame, diff). Add a git export feature so we can export metalog
to git to run those queries.

Choosing git here as we don't have native Rust utilities to create a more
efficient hg repo yet.

Ideally we can also make hg operate on a metalog directory as a "metalogrepo"
directly. However that seems to be quite difficult right now due to poor
abstractions.

Reviewed By: DurhamG

Differential Revision: D21213073

fbshipit-source-id: 4cc0331fbad6e1586907c0a66c18bcc25608ea49
2020-04-27 20:25:25 -07:00
Jun Wu
dc8773033e phrevset: use unfiltered repo for lookups
Summary:
This would avoid issues where the Phabricator commit hash points
to a filtered commit locally. There are 2 user complaints already.

Reviewed By: DurhamG

Differential Revision: D21180833

fbshipit-source-id: 7374e236bcf5e3f3e62bae59fa53604869e22c6f
2020-04-27 20:04:18 -07:00
Jun Wu
3df5fcf779 pymetalog: add handy APIs for debugshell
Summary:
This makes metalog easier to use in debugshell context. For example, to
investigate the "bookmarks" in the past, the code gets simplified from:

  roots = b.metalog.metalog.listroots(repo.svfs.join('metalog'))
  past_ml = b.metalog.metalog(repo.svfs.join('metalog'), root[10])
  past_ml.get("bookmarks")

to:

  roots = ml.roots()
  past_ml = ml.checkout(roots[10])
  past_ml.get("bookmarks")

Reviewed By: DurhamG

Differential Revision: D21162568

fbshipit-source-id: 7cc5581afe596a3d2696311a36ac11caa718428a
2020-04-27 20:04:18 -07:00
Jun Wu
838cc44624 commitcloud: log metalog root during update references
Summary: This uploads more details that might be useful for debugging.

Reviewed By: DurhamG

Differential Revision: D21179514

fbshipit-source-id: ba24395fc4fd153c3ceb4957f822aab70821e6ef
2020-04-27 19:50:58 -07:00
Jun Wu
a0207c4542 metalog: expose root id API
Summary: This allows the Python world to obtain the root ID for logging purpose.

Reviewed By: DurhamG

Differential Revision: D21179513

fbshipit-source-id: 3f289c06d3d470ff492de39fa985203b3facbf00
2020-04-27 19:50:58 -07:00
Jun Wu
7a60f8ac27 commitcloud: log update reference calls
Summary:
This would hopefully give us more insights about when and what host uploads
10k+ remote bookmarks.

Reviewed By: DurhamG

Differential Revision: D21179515

fbshipit-source-id: 3e5c5559e2e739268add05e40f71bea08c29662f
2020-04-27 19:50:57 -07:00
Jun Wu
d8bdae0449 indexedlog: remove chown feature
Summary:
We removed the feature in D20704618 and it does not cause complaints.
Let's remove the code supporting the chown feature.

Reviewed By: DurhamG

Differential Revision: D21170307

fbshipit-source-id: c845016219e8c681930bb1780b94e6d31ca99730
2020-04-27 15:47:59 -07:00
Xavier Deguillard
86965b2f80 revisionstore: query store before fetching
Summary:
While the change looks fairly mechanical and simple, the why is a bit tricky.
If we follow the calls of `ContentStore::get`, we can see that it first goes
through every on-disk stores, and then switches to the remote ones, thanks to
that, when we reach the remote stores there is no reason to believe that the
local store attached to them contains the data we're fetching. Thus the
code used to always prefetch the data, before reading from the store what was
just written.

While this is true for regular stores (packstore, indexedlog, etc), it starts
to break down for the LFS store. The reason being that the LFS store is
internally represented as 2 halves: a pointer store, and a blob store.  It is
entirely possible that the LFS store contains a pointer, but not the actual
blob. In that case, the `get` executed on the LFS store will simply return
`Ok(None)` as the blob just isn't present, which will cause us to fallback to
the remote stores. Since we do have the pointer locally, we shouldn't try to
refetch it from the remote store, and thus why a `get_missing` needs to be run
before fetching from the remote store.

As I was writing this, I realized that all of this subtle behavior is basically
the same between all the stores, but unfortunately, doing a:
  impl<T: RemoteDataStore + ?Sized> HgIdDataStore for T
Conflicts with the one for `Deref<Target=HgIdDataStore>`. Macros could be used
to avoid code duplication, but for now let's not stray into them.

Reviewed By: DurhamG

Differential Revision: D21132667

fbshipit-source-id: 67a2544c36c2979dbac70dac5c1d055845509746
2020-04-27 12:53:11 -07:00
Qing Dong
b4edb7ff7f revisionstore: implement the get() functions on the various LocalDataStore interface
Summary: implement the get() functions on the various LocalDataStore interface implementations

Reviewed By: quark-zju

Differential Revision: D21220723

fbshipit-source-id: d69e805c40fb47db6970934e53a7cc8ac057b62b
2020-04-27 12:35:24 -07:00
Jun Wu
90c062ca08 debugshell: accept file name
Summary:
This makes debugshell easier to use. Especially when the script wants to access
`__file__`.

Reviewed By: DurhamG

Differential Revision: D21169556

fbshipit-source-id: 88b3ebb1ca9a39fe26bc7cc5ea8e250c28fa0d6f
2020-04-27 10:58:56 -07:00
Jun Wu
af1ad137bf remotenames: normalize infinitepush/ to default/
Summary:
Similar to part of D20804854, normalize infinitepush/ to default/ in
remotenames code paths.

This might not be strictly necessary, but we never want to write out
infinitepush/ remote bookmarks and we have seen that OnDemand writes out
infinitepush/ remote bookmarks (which got synced to commit cloud and caused
user complaints) recently.

Reviewed By: DurhamG

Differential Revision: D21233658

fbshipit-source-id: cf8c12ac40348c7b475a7f3d5d5223775491650e
2020-04-27 10:37:27 -07:00
Saurabh Singh
79f5217ccc test-bisect-t: use debugshell instead of debugpython
Summary:
The python script run using `debugpython` within the test was importing
Mercurial internal modules which causes failures on some platforms like
Windows. Instead, lets use the `debugshell` command which already imports the
common objects like `repo` correctly.

Reviewed By: xavierd

Differential Revision: D21251818

fbshipit-source-id: db1b9e92df99b736a28bc9e89fb08ae77d6e82fc
2020-04-27 07:55:30 -07:00
Xavier Deguillard
39d49b694a revisionstore: remove memcache dependency on @mode/mac
Summary:
Memcache isn't available for Mac, but we can build the revisionstore with Buck
on macOS when building EdenFS. Let's only use Memcache for fbcode builds on
Linux for now.

Reviewed By: chadaustin

Differential Revision: D21235247

fbshipit-source-id: 5943ad84f6442e4dabbd2a44ae105457f5bb9d21
2020-04-24 17:29:36 -07:00
Zeyi (Rice) Fan
3374b99f28 util: ensure correct permission is set when creating directories
Summary:
When creates directories sometime we want to make sure other users within the same group have the write access to it to enable data sharing. Previously we rely on setting umask for the entire process to make sure the newly created directories have the correct permission bit. This is kind fragile and error-prone when running in a multi-thread environment.

This diff introduces an internal function `create_dir_with_mode` to create directory with specified permission mode. It first creates a temporary directory within the parent of the directory being created, setting up the correct permission bit, then attempts to rename the temporary directory to the desired name. This ensures that we never leave a directory without the correct permission in the place we need and without changing umask for the process.

Reviewed By: xavierd

Differential Revision: D21188903

fbshipit-source-id: 381bff7d3aaca097b9d50150e86cbbf70a90a0a5
2020-04-24 17:17:05 -07:00
Durham Goode
344837cca4 filesystem: python bindings for new rust pending changes
Summary:
Adds initial python bindings for the rust pending changes. This is not
ready for production usage yet, but having the bindings let's me test changes
more easily until we're ready for automated tests.

Reviewed By: xavierd

Differential Revision: D20546896

fbshipit-source-id: c0ad7155e5068f45bf9c987030746e6c5f35c26a
2020-04-24 13:58:53 -07:00
Durham Goode
092d350800 filesystem: add treestate walking logic
Summary:
The second phase of pending changes is to iterate over the treestate
and figure out what files were not seen in the filesystem walk. This diff
implements that.

Reviewed By: xavierd

Differential Revision: D20546899

fbshipit-source-id: 3523fbc7e31ef0ed09c4937c72264b64e2a3db5b
2020-04-24 13:58:53 -07:00
Durham Goode
73a45b695b filesystem: add filesystem walking to PendingChanges
Summary:
The first phase of pending changes is inspecting the filesystem for
changes. This diff adds that logic.

Reviewed By: xavierd

Differential Revision: D20546909

fbshipit-source-id: 1c2c0fa7f700dbff4acfce4d5271b4472a13571f
2020-04-24 13:58:53 -07:00
Jun Wu
3de6513f0e revset: make autopull commit pull seletivepull bookmarks
Summary: This ensures lagged master issue does not happen by pulling a single commit.

Reviewed By: DurhamG

Differential Revision: D20845384

fbshipit-source-id: 3ba16c07fe264fe2b6aecd494bbb832af7b390a0
2020-04-24 11:16:23 -07:00
Jun Wu
3514ac0de8 remotenames: autopullprefix -> autopullpattern
Summary:
Patterns are more powerful than prefixes. Some full revset names are
obviously not valid remote name - for example, `remote/master :: .`.
They cannot be ruled out using prefixes but can using patterns.

As we're here, add some caching for the regex compiler.

Reviewed By: DurhamG

Differential Revision: D20831015

fbshipit-source-id: af8c4eed4a3153fd71480b8972c55feed4641392
2020-04-24 11:16:22 -07:00
Jun Wu
368d59e436 revset: attempt to pull the revset expression as a single name
Summary:
For names like `a-b-c`, it can be parsed in multiple ways:

- `"a-b-c"`
- `"a-b" - "c"`
- `"a" - "b-c"`

Mercurial uses `repo.lookup` in the parser to accept names like `"a-b-c"`.
Do it for the whole revset expression too.

But do not do it for every lookup (ex. testing `"a-b"` or `"b-c"` in the above
case), because that can be exceedingly expensive.

Reviewed By: DurhamG

Differential Revision: D20831014

fbshipit-source-id: f507e04ce24c953b096ccd836c356f50f11d2006
2020-04-24 11:16:22 -07:00
Jun Wu
47f73cc5e8 revset: calculate autopull names before evaluating revset
Summary:
Add a function to figure out unknown symbols of a revset AST without executing
the revset, then try to autopull unknown symbols.

This has pros and cons comparing to the runtime (stringset) approach.

Pros:
- The fullreposet restriction is lifted.
- We can have some ideas about whether the symbols are "probably" locally
  known. If everything is locally known, then we can skip all remote lookups
  (see the next diff for details).
- We can fetch multiple symbols in batch if we want. Although it's difficult
  to do so right now due to different paths (paths.default, infinitepush,
  infinitepushbookmark, etc). Revisit once the paths are unified.

Cons:
- Functions and argument types must be statically known. Might miss
  some functions added by some extensions.

Reviewed By: DurhamG

Differential Revision: D20909535

fbshipit-source-id: a567ca0aa80aa7f2930dd75637ef3ff8cef1bdef
2020-04-24 11:16:22 -07:00
Jun Wu
e26f2eeacb tests: add a test about autopull on remote names with dashes
Summary:
Dashes are also revset operators. Add a test so we can verify if we can also
autopull those special names.

Reviewed By: DurhamG

Differential Revision: D20831016

fbshipit-source-id: 97e772053dae873ebaa529ac9eb84ea9d04f0f63
2020-04-24 11:16:22 -07:00
Jun Wu
a2d9f6604e infinitepush: disable old autopull if the new autopull is enabled
Summary:
Similar to the previous diff, let's use the new autopull logic, which
uses the tech-debt free repo.pull API.

In rare cases this can cause more pulls due to the path selection (ex. use
"infinitepush" for "remote/scratch/x" but "default" for "remote/stable/x").
With Mononoke serving repos, we might be able to remote the special paths,
and just use "default" for all cases.

If it goes well, we can then delete the old code.

Reviewed By: DurhamG

Differential Revision: D20804854

fbshipit-source-id: 75e68582a29b613c8626a119b85064e3c0ba9462
2020-04-24 11:16:21 -07:00
Jun Wu
d90be19909 remotenames: disable old autopull if the new autopull is enabled
Summary:
The new auto pull logic can replace the one in remotenames. It it goes well, we
can then remove the code in remotenames doing the auto pull.

Reviewed By: sfilipco

Differential Revision: D20804853

fbshipit-source-id: c87b6b382f4cce3b306648b305a7b6bbaec05df1
2020-04-24 11:16:21 -07:00
Jun Wu
614b349628 revset: implement auto pull for hoisted remote bookmarks
Summary:
Attempt to auto pull bookmark names without the `remote/` or `default/` prefix
if hoist is set. This would hopefully be enough to allow us to enable
selectivepull globally without breaking existing users.

Reviewed By: sfilipco

Differential Revision: D20804856

fbshipit-source-id: 72601ac5e3545523cbfd7087d1fc822ef33c2f2e
2020-04-24 11:16:21 -07:00
Jun Wu
880b33b0b7 revset: implement auto pull for commit hashes using repo.pull API
Summary: This might remove less clean logic in infinitepush.

Reviewed By: sfilipco

Differential Revision: D20804855

fbshipit-source-id: da5ed24dd912b5f86839e671a9d6b0e9356b7c86
2020-04-24 11:16:20 -07:00
Xavier Deguillard
19bfd35298 revisionstore: multiplex stores should return a path on flush
Summary:
On repack, when the Rust stores are in use, the repack code relies on
ContentStore::commit_pending to return the path of a newly created packfile, so
it won't delete it when going over the repacked ones. When LFS is enabled, both
the shared and the local stores are behind the LfsMultiplexer store that
unfortunately would always return `Ok(None)`. In this situation, the repack
code would delete all the repacked packfiles, which usually is the expected
behvior, unless only one packfile is being repacked, in which case the repack
code effectively re-creates the same packfile, and is then subsequently
deleted.

The solution is for the multiplex stores to properly return a path if one was
returned from the underlying stores.

Reviewed By: DurhamG

Differential Revision: D21211981

fbshipit-source-id: 74e4b9e5d2f5d9409ce732935552a02bdde85b93
2020-04-23 15:14:28 -07:00
Xavier Deguillard
413d2b3aba remotefilelog: enable uploading LFS blobs
Summary:
This adds the proper hooks in the right place to upload the LFS blobs and write
to the bundle as LFS pointers. That last part is a bit hacky as we're writing
the pointer manually, but until that code is fully Rust, I don't really see a
good way of doing it.

Reviewed By: DurhamG

Differential Revision: D20843139

fbshipit-source-id: f2ef7b045c6604398b89580b468c354d14de1660
2020-04-23 14:00:23 -07:00
Shannon Zhu
6ec7546cab Update pyre version for eden
Summary: Automatic upgrade to remove `version` override and silence errors.

Reviewed By: grievejia

Differential Revision: D21207287

fbshipit-source-id: ab82cf81690cb847bd130ff3448345f5a0ea19af
2020-04-23 12:09:39 -07:00
Arun Kulshreshtha
6d3cacf9fd edenapi: add utility programs
Summary:
Add two utility programs for ad-hoc debugging of EdenAPI. EdenAPI requests and responses are encoded as CBOR, which is not easy to work with manually on the command line. In order to allow debugging the HTTP API using tools like `curl`, we need tools that can generate raw request payloads and interpret CBOR responses.

The utility programs included in this diff are:

- `make_req` - Can construct EdenAPI request payloads from a human-editable JSON representation.
- `data_util` - Can list, validate, and extract the contents of an EdenAPI data response.

These tools can be used by themselves or as part of a pipeline. See test plan for examples.

Reviewed By: xavierd

Differential Revision: D21136575

fbshipit-source-id: d1ac8d92964614005078a6ac76dd0835c29a80a5
2020-04-23 11:43:51 -07:00
Jun Wu
25ce8b7b9f importhelper: avoid resolving commit hashes as revsets
Summary:
`scmutil.revsingle` resolves the commit using the revset language layer,
which will trigger auto pull logic. The import helper does not want any
kind of auto pull logic. Therefore use `repo[rev]` instead, which resolves
the commit hash directly and bypasses the revset / auto pull logic.

Reviewed By: singhsrb

Differential Revision: D21196495

fbshipit-source-id: 5a51057a731523bbb643c7e264d6902dcfbb9059
2020-04-23 10:10:01 -07:00
Mark Thomas
c05efd8a5c mutationstore: move MutationEntry type to types crate
Summary: Move the MutationEntry type to the Mercurial types crate.  This will allow us to use it from Mononoke.

Reviewed By: quark-zju

Differential Revision: D20871338

fbshipit-source-id: 8de3bb8a2673673bc4c8a6cc7578a0a76358c14a
2020-04-23 08:58:10 -07:00
Durham Goode
dbff6c6b9a filesystem: add initial PendingChanges stubs
Summary:
The part of status that lists what files have changed is called
PendingChanges. This diff introduces the initial stub for PendingChanges. The
pending changes algorithm involves three parts:

1. Looking at files on the filesystem for changes.
2. Looking at files in the dirstate map for changes.
3. Looking at the content for any files that we were unsure of during steps 1
and 2.

This diff puts the basic state machine in place, and accepts the basic
information about the working copy (the root and what type of filesystem it is).
In the future we might have it detect what type of filesystem it is, but for now
this makes it easy.

Reviewed By: xavierd

Differential Revision: D20546898

fbshipit-source-id: a3030b7c846b3cb2fcba805b7fe4744df7c5764e
2020-04-22 19:55:50 -07:00
Durham Goode
701273d08f treestate: trim separators off get_filtered_keys inputs and outputs
Summary:
treestate.get_filtered_keys passes directory paths to the filter
function and returns directory matches with a trailing '/' on the end. This
makes it difficult to act as a path normalization function when the caller
doesn't know if the path is a file or directory.

It seems like we can just strip the trailing '/' before exposing the strings to
the caller (both as filter inputs and as get_filtered_keys outputs).

This is useful in the following diff that adds a case normalization crate.

Reviewed By: xavierd

Differential Revision: D20880881

fbshipit-source-id: 6e9f419178b4e278844244bd6aff2fc10e09d2cd
2020-04-22 19:55:50 -07:00
Durham Goode
e97d8d8895 vfs: move vfs logic into its own crate
Summary:
This logic will be used in a variety of places (update workers, status,
etc). Let's move it somewhere common.

Reviewed By: xavierd

Differential Revision: D20771623

fbshipit-source-id: b4de7c1d20055a10bbc1143d44c55ea1045ec62a
2020-04-22 19:55:49 -07:00
Durham Goode
c1b8f86359 treestate: store TreeState in an Arc<Mutex<_>>
Summary:
In a later diff we'll need to be able to hand a reference to the
TreeState to the pending changes iterator. We'd like to be able to hand a
Rc<RefCell<TreeState>> but cpython requires that its fields implement Send. The
simplest solution is to use Arc<Mutex<_>>. Once we finish Rustifying all of this
code we can drop the cpython requirement that this work across threads and
downgrade this to a Rc<RefCell<_>>.

Reviewed By: xavierd

Differential Revision: D20546904

fbshipit-source-id: a4a1ce6973f53b3bb95f227616149f98fcd780e0
2020-04-22 19:55:49 -07:00
Durham Goode
a6e2b90c2e pathauditor: move into workingcopy crate
Summary:
PathAuditor will be needed for native status soon. Let's move it into
the workingcopy crate.

Reviewed By: xavierd

Differential Revision: D20546906

fbshipit-source-id: ef69f88ee828a72e82b5e944cc7913f391bd8a2f
2020-04-22 19:55:49 -07:00
Jun Wu
ae8046b036 tests: update test-hgsql-strip.t
Summary: Update the test to match the latest output.

Reviewed By: xavierd

Differential Revision: D21181146

fbshipit-source-id: 3b6190c9ae8e73309c0b7cb3c48d8a6646bc7fb4
2020-04-22 16:15:46 -07:00
Durham Goode
faced01356 tracing: add more trace values
Summary: This will help us debug slow commands

Reviewed By: xavierd

Differential Revision: D21075895

fbshipit-source-id: 3e7667bb0e4426d743841d8fda00fa4a315f0120
2020-04-22 15:35:17 -07:00
Mark Thomas
a22fa5a2db localrepo: don't revive if obsmarkers are disabled
Summary:
If a commit operation results in a commit that is already present in the
repository, we must revive the commit, rather than create a new one.  In this
case, if obsmarkers are disabled, we can't create a revive marker.

Reviewed By: quark-zju

Differential Revision: D21179879

fbshipit-source-id: 320c70b0ca1e0f384733e09e0e1dd77ac3a255f9
2020-04-22 13:46:41 -07:00
Andres Suarez
7ebf66c8b8 chg: avoid needless allocations
Summary:
* Unnecessary clone of `osstring_to_local_cstring` value.
* Unnecessary UTF-8 conversion in `file_decision`.
* Neater "debugpython" if.
* Unnecessary `CHGDISABLE` `OsString` allocation.

Reviewed By: quark-zju

Differential Revision: D21171329

fbshipit-source-id: 1241066b4a01600c436709d1a5fe95d6a13634ba
2020-04-22 09:34:27 -07:00
Jun Wu
f8e642092c debugshell: fix global variables
Summary:
Without this change the following code will fail surprisingly:

    In [1]: def f():
       ...:     return m
       ...:

    In [2]: f()
    NameError: global name 'm' is not defined

Reviewed By: singhsrb

Differential Revision: D21140299

fbshipit-source-id: f0a6509badcd9314a33a5ca0c78a60bb847d63c7
2020-04-22 09:21:07 -07:00
Jun Wu
361bd06aac debugshell: make more variables avilable
Summary:
Make `ml`, `serv`, `bin`, `hex`, `util` available as they are handy.

Remove less useful names like `e`.

Reviewed By: singhsrb

Differential Revision: D21140300

fbshipit-source-id: ca012da6e33fad64b352c117595538e10d1fad83
2020-04-22 09:21:07 -07:00
Xavier Deguillard
51438d13e7 revisionstore: write data to store when reading from memcache
Summary:
The Memcache store is voluntarily added to the ContentStore read store, first
as a regular store, and then as a remote one. The regular store is added to
enable the slower remote store to write to it so that blobs are uploaded to
Memcache as we read them from the network. The subtle part of this is that the
HgIdDataStore methods should not do anything, or the data fetched won't be
written to any on-disk store, forcing a refetch next time the blob is needed.

Reviewed By: DurhamG

Differential Revision: D21132669

fbshipit-source-id: 96e963c7bb4209add5a51a5fc48bc38f6bcd2cd9
2020-04-21 18:35:38 -07:00
Jun Wu
9aba51ad82 visibility: avoid calculating realnewheads if len(heads) <= 1
Summary:
realnewheads removes ancestors from the heads list. It is not needed
if len(heads) <= 1.

This is an optimization that makes the following change a bit faster.

Reviewed By: DurhamG

Differential Revision: D21011402

fbshipit-source-id: 308b4b5b6f2b8d6e053e51e4a10e0fda2ac949db
2020-04-21 18:28:31 -07:00
Jun Wu
b72002f91d tests: update tests with bookmarks behavior change
Summary:
Update tests. They are caused by the use of deprecated command "rollback".
Normally this kind of issues shouldn't happen.

Reviewed By: DurhamG

Differential Revision: D21164084

fbshipit-source-id: 7d847166f999e2bc472bb8276c1ebb66ad7c45c2
2020-04-21 14:31:46 -07:00
Adam Simpkins
5ed9871d31 don't print background repack messages to stdout
Summary:
D17677229 unintentionally changed the remotefilelog repack code to print
messages about the fact that it was doing a repack to stdout instead of
stderr.

This results in garbage messages showing up in the output of commands that are
intended for programmatic consumption, like `HGPLAIN=1 hg status -0mardu`

Reviewed By: DurhamG

Differential Revision: D21149364

fbshipit-source-id: 104d719af35835be098a375f9b9aebcd1f7a6eca
2020-04-21 14:15:54 -07:00
Durham Goode
3588633bb7 py3: revert mysql-connector-python upgrade and hgsql tests
Summary:
The tests are broken on non-fbcode builds because
mysql-connector-python is the old version. We're unable to upgrade
mysql-connector-python on centos 7/8 because yummy is broken. This blocks us
from getting a release.

Let's roll back the mysql-connector-python upgrade for now, until yummy is
fixed.

Reviewed By: singhsrb

Differential Revision: D21162119

fbshipit-source-id: 5d0b266d72274540eca5b9a76bf151a0fbbee059
2020-04-21 13:45:23 -07:00
Durham Goode
f764f12f72 tracing: fix function tracing
Summary:
The old pytracing logic walked the stack looking for the most recent
spanid. This was fragile and missed a bunch of spots because the function name
wasn't present in f_globals. Let's make this explicit by tracking the stack of
spanids for each python thread.

Reviewed By: quark-zju

Differential Revision: D21068332

fbshipit-source-id: 98759640fa1081bc5bc0805cc620e35a2de9dae3
2020-04-21 13:23:50 -07:00
Jun Wu
73c5cb89de bookmarks: reload changelog if unknown node is seen
Summary: This mitigates loading order issues demonstrated by the previous diff.

Reviewed By: DurhamG

Differential Revision: D21148446

fbshipit-source-id: 40e4861055822b4676f3ac38d0f004b365efe86d
2020-04-21 09:19:26 -07:00
Jun Wu
b4ff907c2a bookmarks: add tests showing loading ordering issue without metalog
Summary: Demonstrate that bookmark loading order can cause "unknown name" errors.

Reviewed By: DurhamG

Differential Revision: D21148445

fbshipit-source-id: d12417ad150904bda14df47cb6e9fe464f392588
2020-04-21 09:19:26 -07:00
Mateusz Kwapich
7ca58332d0 fix xdiff behaviour for empty files
Summary:
When comparing empty file with file with content our xdiff wrongly included
warning about missing newline, which also made the line counter in the hunk
header off-by-one.

Empty files are quite rare in our repos, that's why I discovered this bug only
now (it broke phabricator parsing of this single commit).

Reviewed By: markedson1024

Differential Revision: D21141341

fbshipit-source-id: 9d3e0d8a61ac4ee2cf27978b99b3a092259ee186
2020-04-21 05:30:21 -07:00
Durham Goode
cc6e9b6ace hgsql: disable ssl
Summary:
We don't use ssl to connect to myrouter, and the fact that we even try
causes things to break because we don't have ssl certs.

Reviewed By: sfilipco

Differential Revision: D21139692

fbshipit-source-id: 4a53de929f0114e4307276b8feffcaafebc5d5ce
2020-04-20 22:23:12 -07:00
Xavier Deguillard
1d231ebac7 revisionstore: return non-uploaded keys
Summary:
Ideally, either the ContentStore, or the upper layer should verify that we
haven't missed uploading a blob, which could lead to weird behavior down the
line. For now, all the stores will return the keys of the blobs that weren't
uploaded, which allows us to return these keys to Python.

Reviewed By: DurhamG

Differential Revision: D21103998

fbshipit-source-id: 5bab0bbec32244291c65a07aa2a13aec344e715e
2020-04-20 21:02:35 -07:00
Xavier Deguillard
af1ab52527 merge: add a "rg" merge action
Summary:
When a file goes from being a symlink to a regular file, a regular update
action ("g") is used, and the Python code implicitely remove the symlink before
writing to it. In the Rust code, we don't and as a consequence write through
the symlink, not the intended behavior.

An alternative way of fixing this would be to perform an lstat(2) before
writing to a file, but the cost of doing that will be fairly high for a very
unlikely situation especially since the manifest diff can give us exactly this
information.

This whole merge code feels extremely fragile, so I'm definitively not sure if
I got all the places that needs updating :(.

Reviewed By: DurhamG

Differential Revision: D21082733

fbshipit-source-id: 4f36a67363915c9b67d5a0b290a226075a9f1d31
2020-04-19 22:48:19 -07:00
Durham Goode
7edc29d07c filesystem: move rust walker to it's own file
Summary:
We'll be adding more data to the filesystem layer, so let's move this
out of lib.rs.

Also made a slight tweak to expose File metadata in the walk results, which will be used by the future pending changes logic to avoid re-stating the file.

Reviewed By: xavierd

Differential Revision: D20546903

fbshipit-source-id: 70456055b0da601990e6d6ff535678d2df6c50ba
2020-04-16 16:51:21 -07:00
Xavier Deguillard
10e9a56434 treemanifest: do a repack instead of purging treemanifest
Summary:
Purging treemanifest packfiles always made little sense to me, as the reason we
have tons of them, is because we fetched tons of them. Remove all of them would
just lead to the same situation again. Let's just do a repack instead.

Reviewed By: markbt

Differential Revision: D21065723

fbshipit-source-id: a7cdee1de1d7897481f75c13d8aceeb8c1a68550
2020-04-16 13:06:50 -07:00
Durham Goode
856262080e py3: enable a number of hgsql related tests
Summary:
Now that we've upgraded mysql-connector-python, we can enable these
tests.

Reviewed By: quark-zju

Differential Revision: D21009185

fbshipit-source-id: e9ae62cadcc8d0a291381ab2cfb5c7bc04606d9e
2020-04-16 12:24:41 -07:00
Xavier Deguillard
4973c55030 exchange: always call prepushoutgoing hooks
Summary:
Previously, an extension adding the "changeset" pushop might forget to call the
prepushoutgoing hooks, preventing them from being called.

Reviewed By: DurhamG

Differential Revision: D21008487

fbshipit-source-id: a6bc506c7e1695854aca3d3b2cd118ef1c390c52
2020-04-15 20:22:18 -07:00
Jun Wu
f1c9a39922 doctor: check and hide less relevant branches
Summary:
Sometimes due to wrong configuration, branches that are not relevant to the
user become visible. Detect that and provide a way to hide them.

Reviewed By: xavierd

Differential Revision: D21055464

fbshipit-source-id: 19da498a137fdd08e363d6f689b6de31bc884dbd
2020-04-15 19:21:13 -07:00
Jun Wu
c0b5ad2294 doctor: fix too many remotenames
Summary:
Sometimes due to wrong configuration, cloud sync or legacy pull without
selectivepull will pollute the remote bookmark namespace with too many remote
bookmarks.  Detect that and provide a way to reset remotenames to selected
bookmarks.

Reviewed By: xavierd

Differential Revision: D21055466

fbshipit-source-id: f96c6ba79dd8646fa017947e48e80259ed77de67
2020-04-15 19:21:12 -07:00
Jun Wu
491279b140 doctor: fix incorrect visibility header
Summary:
This is rare. But if visibleheads has a broken header, attempt to fix instead
of skipping.

Reviewed By: xavierd

Differential Revision: D21055468

fbshipit-source-id: 8907c84b69eadb50afae1940862db01a926a38f5
2020-04-15 19:21:12 -07:00
Jun Wu
065ca1b066 doctor: add checks about lagged remote/master
Summary: Trigger a `hg pull` if the remote bookmark appears to be lagging.

Reviewed By: xavierd

Differential Revision: D21055465

fbshipit-source-id: e5ea33ca15f50b504368bec0f349a4abe895eee0
2020-04-15 19:21:12 -07:00
Jun Wu
7cf55dceda doctor: be less verbose
Summary: Remove "looks okay" messages. Only report issues and what changed.

Reviewed By: xavierd

Differential Revision: D21055467

fbshipit-source-id: ee7e732934b88bcdead8cb34ac4cd26b9c7aab42
2020-04-15 19:21:11 -07:00
Jun Wu
ebef9f0ab9 tests: remove test-check-config
Summary:
It was for preventing undocumented configs. Practically it does not achieve the
goal and we might prefer not documenting some of the advanced features to reduce
support burden of unusual configs. Therefore remove it.

We can revisit once we want to improve the documentation of the product.

Reviewed By: xavierd

Differential Revision: D21055469

fbshipit-source-id: 2c26791d2284071b1ac26d1190ec5c4876546f77
2020-04-15 19:21:11 -07:00
Jun Wu
2883e9e20d pushrebase: correctly detect the "rebased / obsolete" case
Summary:
`obsolete()` alone shouldn't prevent push. Check if a successor in the
destination branch instead.

Reviewed By: DurhamG

Differential Revision: D20911924

fbshipit-source-id: 0f983f3440a5bcff13e1a98b28aac2df80b419e0
2020-04-15 19:02:36 -07:00
Jun Wu
ed78133581 phrevset: use GraphQL result if possible
Summary:
Change the fork search to always respect GraphQL if possible. This will result
in more deterministic result, and can avoid full changelog scan if the diff
points to public commits.

Reviewed By: DurhamG

Differential Revision: D21045430

fbshipit-source-id: 2e2b13f5aad3670b9018e5c4ce769b330d14531e
2020-04-15 18:37:18 -07:00
Jun Wu
11a733e700 phrevset: split changelog scan into draft + full scans
Summary:
The local draft scan turns out to be needed by cfgr `jf land` for reasons like
callsign not set (s199694). The whole changelog scan is already slow and will
be slower with segmented changelog so let's log it and see if we can remove it.

Reviewed By: DurhamG

Differential Revision: D21045429

fbshipit-source-id: 999c8479662b3476a09ec852de98cbecebd7f31a
2020-04-15 18:37:18 -07:00
Jun Wu
4ea261914c pager: replace fout and ferr of all ui objects
Summary:
The `ui` object is a bit messy - there are repo specific ui, and non-repo ui,
and ui can be copied around.

This diff changes `fout` and `ferr` to an indirect type that can be mutated
globally so the streampager can affect the `fout` and `ferr` of all related `ui`
objects instead of just one single `ui`.

Reviewed By: DurhamG

Differential Revision: D20903568

fbshipit-source-id: b2f139b91a39c20c22261e03ce50f3cc59074291
2020-04-15 18:23:12 -07:00
Jun Wu
c2ffda7622 pager: configure streampager using hg configs
Summary:
This allows the streampager to be configured via hgrc files.

Default are picked so the behavior is closer to the current default pager
(`less -FRX`).

Reviewed By: DurhamG

Differential Revision: D20902034

fbshipit-source-id: 994ab963ceace02eeb1d18cfa5768e411ca3610b
2020-04-15 18:23:11 -07:00
Jun Wu
08d893fbf0 clidispatch: change pager to use stdio
Summary: This makes it work with chg, since `/dev/tty` is not available for chg.

Reviewed By: DurhamG

Differential Revision: D20936967

fbshipit-source-id: f3ded1aa5552f321ff7043a039f4e35a88160a51
2020-04-15 17:12:47 -07:00
Xavier Deguillard
5184a72b56 merge: only enable rustworkers when remotefilelog is enabled too
Summary:
The Rust worker code will only work with remotefilelog, and when the Rust
ContentStore is enabled, make sure to not enable it if these 2 conditions
aren't met.

Reviewed By: quark-zju

Differential Revision: D21033428

fbshipit-source-id: c34c1b39ddb81be399463712216fa2cd68771f41
2020-04-15 16:16:30 -07:00
Durham Goode
6078dad33e py3: fix more hggit tests
Summary: These tests now pass with python 3.

Reviewed By: xavierd

Differential Revision: D21011004

fbshipit-source-id: 65ac8f864f6eaf24fa7d3c8b25a013b5e9f28ffe
2020-04-15 11:35:39 -07:00
Xavier Deguillard
643e69e045 remotefilelog: do not write delta in bundle2
Summary:
Computing delta force the client to have the previous version locally, which it
may not have, forcing a full fetch of the blob, to then compute a delta. Since
delta are a way to save on bandwidth usage, fetching a blob to compute it
negate its benefits.

Reviewed By: DurhamG

Differential Revision: D20999424

fbshipit-source-id: ae958bb71e6a16cfc77f9ccebd82eec00ffda0db
2020-04-15 10:26:39 -07:00
Durham Goode
bfe0c8d7ab configs: add dynamic config generator
Summary:
We want Mercurial to become more responsible for it's own
configuration, instead of relying on chef and other means. To do so, let's
introduce a new `hg debugdynamicconfig` that can generate dynamic configs for
a given repository based on various states, like what tier it's in or what shard
that machine is in.  By default it generates to '.hg/hgrc.dynamic' for the given
repository.

Currently it just sets the hostgroup config.

Future diffs will make Mercurial consume this config, and possibly have Mercurial
call this command asynchronously when it notices the file is out-of-date.

Reviewed By: quark-zju

Differential Revision: D20828132

fbshipit-source-id: 6f5bf749f5b04e0a5989d6dc19ee788c2e47f88f
2020-04-14 21:22:26 -07:00
Durham Goode
232658cc81 configparser: add function for serializing configs
Summary:
A future diff will want to generate configs programmatically and write
them to a file. Let's add write support to ConfigSet.

Reviewed By: quark-zju

Differential Revision: D20828133

fbshipit-source-id: 702f6f9bdfdf99ef25c6e1c0ab33373a4b6508fe
2020-04-14 21:22:26 -07:00
Durham Goode
910ff8f180 py3: redisable failing py3 tests
Summary: These tests are breaking hgbuild.

Reviewed By: xavierd

Differential Revision: D21029157

fbshipit-source-id: cafcc74a646e18ec531a8f29dc791fa4837bb7eb
2020-04-14 15:49:55 -07:00
Arun Kulshreshtha
845bbc45c7 arcdiff: delete reference to removed since-last-arc-diff flag
Summary: D21017515 removed the `since-last-arc-diff` flag; this was the last reference to it in the hg codebase.

Reviewed By: singhsrb

Differential Revision: D21025752

fbshipit-source-id: 083cb821263fa844ec74773d8636e19a6e82c550
2020-04-14 15:45:20 -07:00
Chad Austin
77d8cd3aae remove a dead eden thrift client method
Summary:
I noticed we removed the call to getManifestEntry, but didn't remove
the method from the EdenThriftClient.

Reviewed By: singhsrb

Differential Revision: D21025523

fbshipit-source-id: 409eb52bad08b305ecfd0f8918cb71fec6a71aab
2020-04-14 15:13:24 -07:00
David Pletcher
147de1f274 Eliminate obsolete since-last-arc-diff flag
Summary:
This flag is an obsolete duplicate of since-last-submit.
Eliminate it to facilitate convenient tab autocompletion and make
the world less complicated.

Reviewed By: xavierd

Differential Revision: D21017515

fbshipit-source-id: f3e4db602a69f1c7da886d455ac36c4a2ddb6b01
2020-04-14 09:09:12 -07:00
Lukas Piatkowski
6afe62eeaa eden/scm: split revisionstore into types and rest of logic
Summary:
The revisionstore is a large crate with many dependencies, split out the types part which is most likely to be shared between different pieces of eden/mononoke infrastructure.

With this split it was easy to get eden/mononoke/mercurial/bundles

Reviewed By: farnz

Differential Revision: D20869220

fbshipit-source-id: e9ee4144e7f6250af44802e43221a5b6521d965d
2020-04-14 07:50:19 -07:00
Jun Wu
4413d92cad pushrebase: add a test showing suboptimal abort
Reviewed By: simpkins

Differential Revision: D20911925

fbshipit-source-id: 81c7fe27b5cb85d6bdf5be385d3f1db84ed1b73b
2020-04-13 20:46:21 -07:00
Genevieve Helsel
a98dd1fa6b suggest graceful restart in hg status / old edenfs warnings
Summary: Since the old Edenfs warning is usually for simply picking up new eden releases, we can suggest the user runs a graceful restart instead of a normal restart to avoid them running into `Transport not connected` errors. This path is only hit in unix environments, so windows users will not see this (since graceful restart isn't supported there yet). Since this is a manual step as well, it will be easier for a user to see if they run into an issue here. This can also enable us to get more telemetry from users running graceful restarts.

Reviewed By: wez

Differential Revision: D20901597

fbshipit-source-id: 9e5c9a90313901be159f66afcbbadc5d7af4fe28
2020-04-13 16:25:58 -07:00
Durham Goode
955013fdbc py3: fix test-bundle2-format.t
Summary: It now passes

Reviewed By: xavierd

Differential Revision: D20995130

fbshipit-source-id: c262cd8f3ef1c41cd52fd5ec5894be145059f045
2020-04-13 15:28:25 -07:00
Durham Goode
2f8cb4b6ec py3: fix test-bundle2-exchange.t
Summary: It now passes

Reviewed By: xavierd

Differential Revision: D20971712

fbshipit-source-id: d962c5f0171c056edfadb5cc795d1c8662456afa
2020-04-13 15:28:25 -07:00
Durham Goode
d9a508dfba py3: fix various wireprotocol related tests
Summary: They now pass

Reviewed By: xavierd

Differential Revision: D20970190

fbshipit-source-id: 8610e0877181c5e8b8842559f7a3e4a943c0c3fe
2020-04-13 15:28:24 -07:00
Xavier Deguillard
69ad493984 color: always specify alternate colors
Summary:
Having some colors, even dull is better than no colors. For every "bright"
color found in the repo, also specify their non-bright variant.

Reviewed By: DurhamG

Differential Revision: D20972198

fbshipit-source-id: ca10c0eac13129f120fc45623d134f279a75b3c2
2020-04-13 15:04:16 -07:00
Xavier Deguillard
b6cb2f37f2 tests: fix test-parse-date.t in python3
Reviewed By: DurhamG

Differential Revision: D20972880

fbshipit-source-id: 6436fb9af8bb8adf48d6c23a0e8cc453d9336c47
2020-04-13 14:59:41 -07:00
Xavier Deguillard
e8c2e7a280 tests: fix test-perftweaks-remotenames.t in python3
Summary: Reading needs to be done via readutf8.

Reviewed By: DurhamG

Differential Revision: D20972835

fbshipit-source-id: 0705c04f1db8b20cc4679852d75ea1790a5cf5f9
2020-04-13 14:59:41 -07:00
Xavier Deguillard
6b85ee4edb remotenames: decode the content of selectivepullenabled
Summary: We're writing utf-8 data to it, decode it before returning it.

Reviewed By: DurhamG

Differential Revision: D20972741

fbshipit-source-id: fb23a6d1c3188e10598b839639cdd5e74322089a
2020-04-13 14:59:41 -07:00
Xavier Deguillard
d2c56495e4 metalog: fix for python3
Summary:
While keys are strings, values are bytes buffer and thus needs to be converted
sometimes.

Reviewed By: DurhamG

Differential Revision: D20974484

fbshipit-source-id: 13394f5dc43191e85e4b1d350cc4fbbd8489572a
2020-04-13 14:55:11 -07:00
Xavier Deguillard
8178dfd89b context: fix symlink validation
Summary:
Symlinks are pointers to path, that are required to be utf-8, thus we can
decode the data.

Reviewed By: DurhamG

Differential Revision: D20973701

fbshipit-source-id: 8399dfb3462fea907758a01ed2fcea508833fbaa
2020-04-13 14:55:11 -07:00
Xavier Deguillard
ffd876d967 tests: fix test-filelog.py for python3
Reviewed By: DurhamG

Differential Revision: D20973502

fbshipit-source-id: dc3201dcc25509d569442a240fa8361a819f1f5f
2020-04-13 14:55:11 -07:00
Xavier Deguillard
17855deb17 tests: fix test-extdiff.t for python3
Reviewed By: DurhamG

Differential Revision: D20973286

fbshipit-source-id: 3a2d3cfc61567934d53e58ce561c367746c02403
2020-04-13 14:55:10 -07:00
Xavier Deguillard
4d5515e6ef tests: fix test-revlog.t for python3
Reviewed By: DurhamG

Differential Revision: D20973212

fbshipit-source-id: 1535d50a0cb2a7350083600e4e6c6506e0749072
2020-04-13 14:55:10 -07:00
Xavier Deguillard
5f4d53cb54 tests: fix test-hggit-illegal-contents.t
Summary:
With the upgraded git, some error message aren't the same as before, update the
test to reflect that.

Reviewed By: quark-zju

Differential Revision: D20993735

fbshipit-source-id: d9b8bfc9c6d10f7c0898299659e4322589ea8077
2020-04-13 14:25:35 -07:00
Nitzan Shaked
7833fd08d5 fix "empty link" bug in hyperlink
Summary: `hyperlink(url, title)` returns a terminal escape sequence for the url even if the title provided is empty. While the terminal displays nothing, the actual string in hg-template-land is non-empty (it contains a url), and all the template functions that check for empty strings don't see it as empty.

Reviewed By: fanzeyi, xavierd

Differential Revision: D20970733

fbshipit-source-id: 04436714f3827b98b8fb249b3eb1da82f366807c
2020-04-13 11:45:39 -07:00
Xavier Deguillard
cdb0fbdb71 convert: do not try to re-encode in python3
Summary: The encoding is utf-8 already.

Reviewed By: sfilipco

Differential Revision: D20966808

fbshipit-source-id: ade012588bf14df07772db0819be8670ecdb3b09
2020-04-13 08:53:51 -07:00
Xavier Deguillard
aa836d38ea tests: enable histedit for python3
Summary: These are now passing

Reviewed By: sfilipco

Differential Revision: D20966083

fbshipit-source-id: c9a34ee23a1552200756a4e1f75a29574c37960d
2020-04-13 08:53:51 -07:00