Commit Graph

14436 Commits

Author SHA1 Message Date
Jun Wu
7d013975ca remotefilelog: remove repo.fileslog side effect in reposetup
Summary:
The side effects are undesirable as an error in the store can crash any repo command.
That makes it harder to implement a `doctor` command to fix the stores.

Reviewed By: xavierd

Differential Revision: D17755609

fbshipit-source-id: 3ba1774de965c4d896178adc47df805f6e465071
2019-10-04 20:37:06 -07:00
Jun Wu
fd7a6868a3 testutil/dott: fix setconfig with multiple args
Summary: The old code incorrectly only respects the last argument of `setconfig`.

Reviewed By: xavierd

Differential Revision: D17766375

fbshipit-source-id: ec70c10eb62803c9f3d8dde5d88f91d076a55049
2019-10-04 18:12:10 -07:00
Stefan Filip
1d616cebe0 manifest: update test configs to enable rustmanifest by default
Summary:
Rust manifests are the future. Tests work well with them and we are looking
ahead at rolling them out. To protect against regressions on the Rust manifest
side we need to have an automated test story.

I think that the most effective thing we can do is to enable Rust Manifest for
all tests. The C++ codebase is not seeing any kind of modification and we plan
to delete as soon as Rust manifests roll out.

Reviewed By: xavierd

Differential Revision: D17728414

fbshipit-source-id: 59979d02e3cece83e73569a43d6fdbb7a29dc66d
2019-10-04 10:52:13 -07:00
Aleksei Kulikov
1c2460479e snapshot: clean the internal files given the --clean flag
Summary: `merge/*` and `rebasestate` files.

Reviewed By: mitrandir77

Differential Revision: D17719134

fbshipit-source-id: a7ed2258396e8b758debdb9fa28c953b0a22e749
2019-10-04 10:07:55 -07:00
Aleksei Kulikov
b802a42448 snapshot: store metadata content
Summary:
Make it possible to store small files directly inside the metadata blob:
```
{
  "files" : {
    "unknown": {
      "some/big/untracked/file": {"oid": "oid0"},
      "some/small/untracked/file": {"content": <binary file contents>},
    },
    . . .
  },
  "version" : "version of metadata format"
}
```

Reviewed By: mitrandir77

Differential Revision: D17689055

fbshipit-source-id: 7ab18d8f012a20be04d65fcbe2365ff0157386f1
2019-10-04 10:07:54 -07:00
Aleksei Kulikov
a45cf73590 snapshot: fix smartlog predicate for snapshots
Summary: This is a better way to wrap smartlog functionality.

Reviewed By: mitrandir77

Differential Revision: D17686738

fbshipit-source-id: 770b32c19c89a59aee0f8b25251faaae7685a4dd
2019-10-04 10:07:54 -07:00
Xavier Deguillard
5d0043c385 check-code: remove more duplicates with black
Summary: These 4 checks are done by black, no need to have duplicated code.

Reviewed By: quark-zju

Differential Revision: D17714260

fbshipit-source-id: 9cb4aa97e97216435e68eb9ed726a26631f04875
2019-10-03 15:32:25 -07:00
Thomas Orozco
1a5e6756e4 make auth logging debug level
Summary:
This error is not particularly useful for users to see, and eventually there
will be cases where it is in fact expected that some paths will be invalid
(this is particularly useful on hosts where the same config is read by
different processes with a different environment and / or mount namespace).

Reviewed By: HarveyHunt

Differential Revision: D17710604

fbshipit-source-id: 588c3653df91a84035f044af9455c6ac83305090
2019-10-03 07:49:29 -07:00
Stefan Filip
b5e80dba83 tests: update test-globalrevs to use assert
Summary:
test-check complains about the test using `raise Exception.`
For the purposes of the test, `assert` seems more appropriate.

Reviewed By: DurhamG

Differential Revision: D17724080

fbshipit-source-id: 2831a384c1e91012ee58bced47f026cd74995e86
2019-10-02 16:50:02 -07:00
Aleksei Kulikov
e635171709 snapshot: fix rebasestate show snapshot
Reviewed By: markbt

Differential Revision: D17666034

fbshipit-source-id: 67005c07c1c583b709310861c3ece38c4be825a3
2019-10-02 11:54:25 -07:00
Aleksei Kulikov
1a6762b57d snapshot: match files correctly during checkout, update tests
Summary: The previous matcher constructed wrong (with the extra `cwd` part) paths.

Reviewed By: markbt

Differential Revision: D17659134

fbshipit-source-id: 8556b8acda515ac68c2a72f2ede6408172d42575
2019-10-02 11:54:25 -07:00
Aleksei Kulikov
21837851bd commitcloud: sync snapshots on the client side
Summary:
Essentially, snapshots are being synced exactly like heads are.
A dedicated table in CC DB etc.

Reviewed By: markbt

Differential Revision: D17345491

fbshipit-source-id: a1e65c2a4815f437a11da68f9d46c27f046453a9
2019-10-02 11:54:24 -07:00
Durham Goode
8598048f01 match: fix patternmatcher.visitdir
Summary:
patternmatcher can be used for non-exact and non-prefix patterns, but
the current visitdir implementation will return False unless a prefix pattern or
exact patterns are provided. A future change makes dirstate.status more reliant
on correct matcher behavior, and this breaks tests.

The fix is to return True (meaning that the directory should be visited)
whenever the pattern is not an explicit list of files. Hopefully we can move
this to our rust matcher in the future which will be able to optimize this case
around globs and regular expressions.

Reviewed By: quark-zju

Differential Revision: D17116369

fbshipit-source-id: d65b08060143efc3f4bbde3198b0c505a8ff4ec7
2019-10-02 11:00:26 -07:00
Durham Goode
2bb8d905d4 context: remove use of dirstate.walk
Summary:
In a future diff we'll be restricting the dirstate interface. We'd like
to drop dirstate.walk, so let's remove it's use from commitablectx.walk.

Reviewed By: quark-zju

Differential Revision: D17082284

fbshipit-source-id: f6034281c10123c28c37ee93844ac7e05418973f
2019-10-02 11:00:25 -07:00
Durham Goode
7b4ce229db grep: remove use of dirstate.walk
Summary:
In a future diff we'll be restricting the dirstate interface. We'd like
to drop dirstate.walk, so let's remove it's use from `hg grep`

Reviewed By: quark-zju

Differential Revision: D17082286

fbshipit-source-id: ef7713db5288658d3fefc9ae038077570ed39bb4
2019-10-02 11:00:25 -07:00
Durham Goode
c10c1d74e4 add: remove usage of dirstate.walk
Summary:
In a future diff we'll be restricting the dirstate interface. We'd like
to drop dirstate.walk, so let's remove it's use from `hg add`.

Note, the old code was actually a bit buggy. It had `full=False` which meant the
walk might not return clean files, which meant we might not warn about the user
attempting to add clean files, depending on the dirstate implementation. The new
code avoids that bug.

Reviewed By: quark-zju

Differential Revision: D17082285

fbshipit-source-id: 946f4edc852d8c62667fb50820f4b0c1c8f229a9
2019-10-02 11:00:24 -07:00
Durham Goode
0f3585ee6d dirstate: move lookup resolution into the dirstate
Summary:
Previously, dirstate.status() would produce a status result and a list
of files that it wasn't sure of. Then the workingctx class would inspect the
unsure files, update the status result, and update the dirstate. This was a
strange logic split, since the dirstate should really be responsible for all
status result computation.

In a future diff we will be moving the "what files are changed" computation
behind a new filesystem layer, so in this diff let's move this logic into the
dirstate so it can later be hidden behind the new layer.

This adds the repo object as a attribute on the dirstate, which introduces a
cyclic reference. I think this is fine because:
1. We already do this in many of our dirstate extensions
2. We use a weak reference
3. The dependency direction should probably be workingcopy -> repo anyway,
since repo should generally not have knowledge of the working copies, but the
working copies probably need knowledge of the repo (like to read tree contents
for status computation).

Reviewed By: quark-zju

Differential Revision: D16816274

fbshipit-source-id: ad4e83d489bbe94c131a844ae380a565d7018f89
2019-10-02 11:00:22 -07:00
Durham Goode
6b9cdf9f67 tests: convert test-globalrevs.t into -t.py test
Summary:
Converts the test-globalrevs.t test into the new format. Since the new
test format doesn't restart the process between invocations, we have to make the
globalrevs extension smarter about only executing its bits when the repository
in question actually has the extension enabled (instead of depending on the
process only living for the duration of a single command on a single repo).

Reviewed By: quark-zju

Differential Revision: D17693594

fbshipit-source-id: cae3c934af3c93ee2b5b83dcf82ba00b323dccac
2019-10-02 10:15:42 -07:00
Durham Goode
8b0b788ba8 hgcommands: print errors to io instead of stderr
Summary:
HgPython::run_hg was printing errors directly to stderr instead of to
the provided io.error. This caused unhandlable output in the -t.py tests. Let's
fix it to output to the provided pipe.

Reviewed By: quark-zju

Differential Revision: D17634721

fbshipit-source-id: f441e7be461193ef54db25e0939b2e67cdf06126
2019-10-02 10:12:19 -07:00
Durham Goode
00b6214529 tests: convert 5 more hgsql tests
Summary: These work with just a simple auto-conversion.

Reviewed By: xavierd

Differential Revision: D17611533

fbshipit-source-id: 4be1ace941f6271d94318d88db456fb4cf91ef7f
2019-10-02 10:12:18 -07:00
Durham Goode
413b536a21 tests: convert test-hgsql-filenames.t
Summary:
Converts test-hgsql-filenames.t to be a -t.py test and converts the
hgsql test infra to work in -t.py tests.

Reviewed By: quark-zju

Differential Revision: D17611277

fbshipit-source-id: 8c75ad01a6af743e912312bdc603529dd0f2cf4b
2019-10-02 10:12:18 -07:00
Xavier Deguillard
396ce90580 test-contrib-check-code.t: fixup after D17695858
Summary:
D17695858 removed a couple of rules from the code checker, leading to this test
failing.

Reviewed By: quark-zju

Differential Revision: D17713409

fbshipit-source-id: f37d26ebfc16541119113ce5ea22ba9f2a48c60a
2019-10-02 08:32:03 -07:00
Stefan Filip
829d5fa23e tests: fix test-hg-absorb diff settings
Summary:
It seems that on Windows the execbit feature is not enabled.
This means that git diff format is not set. I think that the
test could do without this update but I'll keep it for now.

Reviewed By: quark-zju

Differential Revision: D17703428

fbshipit-source-id: ff44c3187ec957509a07ae99ac340450d3008656
2019-10-01 18:35:04 -07:00
Stefan Filip
34cf9b976b tests: remove whitespace from test-absorb
Summary: Fixes test-check-code.t

Reviewed By: singhsrb

Differential Revision: D17695548

fbshipit-source-id: 6f9713bef8e3fb4842dda7bf8ba2633ada96126a
2019-10-01 14:49:59 -07:00
Xavier Deguillard
a3360b4cde remotefilelog: automatically repack when many transaction are commited
Summary:
We've seen several cases where the number of packfiles explodes and causes
Mercurial to slowly becoming unusable due to spending most of its time scanning
the filesytem for new packfiles. In the case of tree, Mercurial will blow aways
the hgcache/manifest directory entirely, causing concurrent Mercurial processes
to fail to commit packfiles to disk.

Let's just count the number of commits we've had so far, and trigger a repack
when it goes over a threshold.

Reviewed By: quark-zju

Differential Revision: D17677222

fbshipit-source-id: 99c31c6137d792be1f6de3c298ebff8a4372926e
2019-10-01 14:34:04 -07:00
Xavier Deguillard
c97df843a2 dott: manually expand TESTDIR in hgrc
Summary:
On Windows, subprocess.Popen(shell=True) will not expand environment
variables, thus, the hgcloneshallow function would fail due to
$TESTDIR/dummyssh not found. Since we know the value of TESTDIR
when the config is written, let's just expand it.

Reviewed By: quark-zju

Differential Revision: D17691487

fbshipit-source-id: 149dcf03b652960fe0e5eff34541cb533d65fa3d
2019-10-01 14:34:04 -07:00
Xavier Deguillard
a6c6909237 remotefilelog: honor --quiet for background repack
Summary:
We would always print that a background repack is started, regardless of
whether --quiet was being passed in.

Reviewed By: quark-zju

Differential Revision: D17677229

fbshipit-source-id: 4e7d2fa355f11fdeea0e4cb59f78bc8f0c56b242
2019-10-01 14:34:03 -07:00
Jun Wu
c6a432e186 testutil: remove an unused reload() function
Summary:
It does not always work cleanly because some GC/__del__ might not be able to
clean up states cleanly.

Reviewed By: kulshrax

Differential Revision: D17634974

fbshipit-source-id: 39264224ee74b05ff4f4e031729d97975ecd4d18
2019-10-01 13:05:18 -07:00
Jun Wu
edbaa1b522 test-absorb: fix the test
Summary:
- Implement a basic grep.
- Implement a naive `sedi`.
- Replace some shell for loops using Python for loops.
- Replace some shell code in Python.

Reviewed By: mitrandir77

Differential Revision: D17616051

fbshipit-source-id: 2fcc19a928989b66e5e9502e10aa07c37b8d1056
2019-10-01 10:45:44 -07:00
Jun Wu
43a605ef7b tests: auto translate "test-absorb.t"
Summary:
This is done by running:

  ./translate "test-absorb.t"

Reviewed By: xavierd

Differential Revision: D17616050

fbshipit-source-id: 7d2e9e4bce88f7702af5a512ee3b4d49c89002f5
2019-10-01 10:45:43 -07:00
Xavier Deguillard
811ea67178 tests: fix test-bisect-t.py on windows
Summary:
$TESTTMP wasn't being expanded due to being in between '', since we're
already in the right directory, there is no need to use $TESTTMP.

Reviewed By: DurhamG

Differential Revision: D17687911

fbshipit-source-id: d9f7048cc95955f59e8a492335a90d48773bc6d2
2019-10-01 09:57:16 -07:00
Xavier Deguillard
728e04deda tests: fix test-revert-t.py on Windows
Summary: The glob wasn't being expanded properly.

Reviewed By: DurhamG

Differential Revision: D17687708

fbshipit-source-id: 7a41fb94a81994a3c6119a7663b1378b51f07a79
2019-10-01 09:57:16 -07:00
Xavier Deguillard
7445aa4701 testutil: fix style in shlex
Summary: The test test-check-code.t was complaining about it.

Reviewed By: quark-zju, singhsrb

Differential Revision: D17687037

fbshipit-source-id: b61351a239554d73fd3285dedca613e72d1d3e59
2019-10-01 09:03:37 -07:00
Aleksei Kulikov
8a79cfa7e8 snapshot: add the unionstore to handle the rebundling of snapshot bundles
Summary:
If some bundle consisted of several snapshots, and the client requests a part of that data, the server can trigger the rebundling process.
This diff teaches snapshots to the rebundling process.

The `bundlerepo` object should store data in memory, so a `unionstore` class is made up.
It allows to "pair" in-memory store with a regular local store and query them via the same API.

Reviewed By: markbt

Differential Revision: D17684945

fbshipit-source-id: 0bdb1f70b777fb2d8193b3bfa46b5915c2019663
2019-10-01 06:46:43 -07:00
Jun Wu
5e5465c313 testutil/dott: match shell's behavior on quote handling
Summary:
This diff makes the code closer to shell behavior.

For example, globs are disabled for both single and double quotes:

  ~ % echo /bin/bash*
  /bin/bash /bin/bashbug
  ~ % echo "/bin/bash*"
  /bin/bash*
  ~ % echo '/bin/bash*'
  /bin/bash*

Environment variables are not expanded for single qutoes:

  ~ % echo $PWD
  /home/quark
  ~ % echo "$PWD"
  /home/quark
  ~ % echo '$PWD'
  $PWD

Tests using single quoted environment varialbes are updated to use double
quotes, mostly by using this vim command:

  %s/\(sh % ".*\)'\([^'$]*\$[^']*\)'/\1\\"\2\\"

The translation script was updated to prefer double quotes to preserve
environment variable expansion behavior.

Reviewed By: xavierd

Differential Revision: D17675351

fbshipit-source-id: d5c8d5f23ea8e29fe093c4e6ae89ddacda97141e
2019-09-30 16:53:25 -07:00
Jun Wu
78c06706ca shlex: add an expandfunc argument
Summary: This allows customization about how to expand an argument with different quotes.

Reviewed By: xavierd

Differential Revision: D17675350

fbshipit-source-id: 1dc4b2e2b7ea0296dceda0d8654c2278a602e4f8
2019-09-30 16:53:24 -07:00
Jun Wu
b118e96ab2 testutil/dott: vendor Python 3's shlex library
Summary:
We need to get "whether an argument is single-quoted or not" information to
decide whether to run extra logic (ex. expanding globs) on it. `shlex` does not
really have public interface for that and it's hard to `wrapfunction` patch it.
Therefore vendor the script so we can customize it.

Some Python 2 compatibility work was done:
- use `cStringIO` instead of `io`
- mark the file as utf-8
- remove `quote`, which uses an incompatible `re.ASCII`

Reviewed By: xavierd

Differential Revision: D17675349

fbshipit-source-id: eb99a9b69ca15f770c410bd3845052cb0c6f1b17
2019-09-30 16:53:24 -07:00
Xavier Deguillard
f48990a312 tests: fix missing depth initialization in generateworkingcopystates
Summary: The variable was used but sometimes not initialized.

Reviewed By: quark-zju

Differential Revision: D17667256

fbshipit-source-id: 9efa3b160cb23205baaa614827e5990475b956c9
2019-09-30 11:23:55 -07:00
Jun Wu
e4e4cbb639 testutil/dott: avoid crash expanding empty argument
Summary:
`arg` can be empty in this context. Avoid crashing with IndexError.

Also fix test-sparse-issues-t.py so it quotes the pattern correctly.

Reviewed By: xavierd

Differential Revision: D17669371

fbshipit-source-id: ea9e4e7c09a88308a650442c22e133684805738e
2019-09-30 11:14:59 -07:00
Jun Wu
98795c9257 test-import: fix on Windows
Summary: Avoid using `rm -rf` but just create new repos aggressively.

Reviewed By: singhsrb

Differential Revision: D17669063

fbshipit-source-id: f5ea142b08f16dcbdcf99a937fba803e1d8d9958
2019-09-30 10:56:16 -07:00
Jun Wu
60b58132d1 phases: point out manual edits are no-ops if narrow-heads is enabled
Summary: This makes it clear that the new behavior is different.

Reviewed By: sfilipco

Differential Revision: D17584605

fbshipit-source-id: 59573dfa14b4eb7a4a9c5bce6ae7340f408035da
2019-09-27 17:24:00 -07:00
Jun Wu
640a873e9c narrow-heads: add bi-directional migration
Summary:
When migrating down from narrow-heads, we need to restore phase roots.  Use a
repo requirement to track the migration state, and perform migrations on
demand.

Reviewed By: sfilipco

Differential Revision: D17584604

fbshipit-source-id: 36c04b0385dd94481f5340a3ecd8eeef3506c765
2019-09-27 17:24:00 -07:00
Xavier Deguillard
646314794b test-bisect-t.py: fix it
Reviewed By: quark-zju

Differential Revision: D17620102

fbshipit-source-id: b73398846b38ce13c34d4df5d7c5779c1d8641ad
2019-09-27 17:24:00 -07:00
Xavier Deguillard
f0e98e6104 tests: auto translate "test-bisect.t"
Summary:
This is done by running:

  ./translate "test-bisect.t"

Reviewed By: quark-zju

Differential Revision: D17620101

fbshipit-source-id: 461299cb6b74eac7a4febc2fa495ea65e65eea8e
2019-09-27 17:24:00 -07:00
Saurabh Singh
16da1518e5 perforce: remove p4fastimport
Summary:
I think we moved the last perforce based repository into `fbsource`.
From the Scuba logs, this doesn't seem to be used any more:
https://fburl.com/scuba/lc6vgy3v. We can revive the extension if required for a
future Perforce repository import.

Reviewed By: quark-zju

Differential Revision: D17630314

fbshipit-source-id: e3ca1a97974aff5fc262b8c724b70d4548280325
2019-09-27 16:31:34 -07:00
Stefan Filip
e96a1efb2d tests: migrate test-revert to dott
Summary:
The test is mostly automatic conversion and easy fixes.
No refactoring to simplify code, on the contrary, more
direct printing.

Reviewed By: quark-zju

Differential Revision: D17630869

fbshipit-source-id: 45e57de5fd965d0cc913c2612a6f52cb3c8bd977
2019-09-27 13:44:59 -07:00
Stefan Filip
45784815c1 dott: update shobj to expand filename patterns
Summary:
Filename expansion is probably used a lot in our tests. I think that our
migration path is going to be simplified if we do filename expansion as
part of shobj. The current implementation works for the test that I was
migrating, test-revert.

Reviewed By: quark-zju

Differential Revision: D17630872

fbshipit-source-id: b8312e3d1fee9d331a661849a4cc5ef06ed5b091
2019-09-27 13:44:59 -07:00
Stefan Filip
9454158308 dott: update chmod to support -x
Summary: Following a similar implementation to `chmod +x`.

Reviewed By: quark-zju

Differential Revision: D17630871

fbshipit-source-id: 115fe8fd0cca214f7f739bec66301c9116edb250
2019-09-27 13:44:59 -07:00
Stefan Filip
e16d63bb2d tests: update generateworkingcopystates to work as a module
Summary: I would like to call into this script directly from dott tests.

Reviewed By: quark-zju

Differential Revision: D17630870

fbshipit-source-id: 169ab3f1706d77cb1cdc1461b2a6fce2bc77a877
2019-09-27 13:44:58 -07:00
Stefan Filip
40e989ef9a tests: rename generate-working-copy-states
Summary:
Renaming it to `generateworkingcopystates`. This is a valid module name.
The goal is to convert this script to a module. While we are converting
tests it needs to be accessible in both script form and module.

Reviewed By: singhsrb

Differential Revision: D17611370

fbshipit-source-id: 0b2d0de71f0da7cd6efd7874c505d9aa77f7b296
2019-09-27 13:44:58 -07:00