Commit Graph

2951 Commits

Author SHA1 Message Date
Jun Wu
d86802a980 parser: add a function to walk through a tree
Summary: This will be used to test if a keyword is in an AST.

Reviewed By: DurhamG

Differential Revision: D24293499

fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
2020-10-20 16:45:43 -07:00
Jun Wu
5a6161f140 repo: unconditionally migrate to specified visibility and dirstate format
Summary:
There were 2 recent reports about "visibleheads" not being enabled.

Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.

There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.

Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.

Reviewed By: DurhamG

Differential Revision: D24340586

fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
2020-10-20 16:36:50 -07:00
Jun Wu
9de7c7a22d repo: drop format.usestore=false support
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.

Reviewed By: DurhamG

Differential Revision: D24340585

fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
2020-10-20 16:36:50 -07:00
Jun Wu
e179789b7b tests: remove test cases about legacy dirstates
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.

Reviewed By: DurhamG

Differential Revision: D24340584

fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
2020-10-20 16:36:50 -07:00
Jun Wu
a3ac43a23a rollout: use doublewrite changelog backend for hg-dev
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).

Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.

Reviewed By: singhsrb

Differential Revision: D24259759

fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
2020-10-20 15:53:57 -07:00
Jun Wu
7db31b57b6 templater: add API to parse string to AST with aliases expanded
Summary: It will be used to test whether `{phabstatus}` is in template.

Reviewed By: sfilipco

Differential Revision: D24293496

fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
2020-10-20 15:24:29 -07:00
Jun Wu
c9d716fd51 test-rebase-emptycommit: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270422

fbshipit-source-id: d2dcf6db2a0c8ebd90a2edb3ee1149d8f9bd4fbe
2020-10-20 15:24:29 -07:00
Jun Wu
063bc373af test-rebase-check-restore: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270424

fbshipit-source-id: 2833dbbdc7ec34375f070fc6ce273265272a205e
2020-10-20 15:24:29 -07:00
Jun Wu
293d05dd40 test-rebase-conflicts: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258215

fbshipit-source-id: 9a9bba13c3963c79df5e672e5ac011ab7c7e5698
2020-10-20 15:24:29 -07:00
Jun Wu
20ac277bc5 test-rebase-brute-force: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258213

fbshipit-source-id: aceb6e0792b4646eb0805f56d981751eb85ae340
2020-10-20 15:24:29 -07:00
Jun Wu
af38cf3cb9 test-rebase-abort: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258214

fbshipit-source-id: db7ecb6b874da81d0e553cd9f1514ee5559635b7
2020-10-20 15:24:29 -07:00
Jun Wu
d39be1d667 tests: mark more tests as compatible with segmented changelog
Reviewed By: singhsrb

Differential Revision: D24214459

fbshipit-source-id: 7c125cbc2c0d559d337bd01107b784002e2dc9ea
2020-10-20 15:24:29 -07:00
Jun Wu
b46a5b6192 copies: improve compatibility with segmented changelog
Summary:
There are 2 issues:
- len(repo) is no longer the "max rev" + 1. Use 1<<63 instead.
- If invalidatelinkrev is a repo requirement, linkrev can no longer be trusted.

Reviewed By: DurhamG

Differential Revision: D24270423

fbshipit-source-id: 4e5c999772fad6ef2b1dabd2d6363d4226242c4d
2020-10-20 15:24:28 -07:00
Jun Wu
62f30ea23e changegroup: do not use bad linkrev to filter out file revisions
Summary:
If linkrevs are known unreliable, do not use them. This fixes some cases with
segmented changelog, file revisions are incorrectly filtered out and not sent
to client.

Reviewed By: DurhamG

Differential Revision: D24233228

fbshipit-source-id: 268a95b44c0217658b849317cb0a680450b91d23
2020-10-20 15:24:28 -07:00
Jun Wu
a7ba754d88 rebase: remove more uses of revnums
Summary: Replace revnum with commit hashes.

Reviewed By: singhsrb

Differential Revision: D24233229

fbshipit-source-id: 98b4d45b542b65c375cef405b45b66cd2dad22e4
2020-10-20 15:24:28 -07:00
Jun Wu
a4b3f573ba changelog: change "ambiguous prefix" to RepoLookupError
Summary: `LookupError` is too broad.

Reviewed By: DurhamG

Differential Revision: D24411203

fbshipit-source-id: 3e7c51618115c11f87f0fed670d18b4b5a03e20d
2020-10-20 15:24:28 -07:00
Jun Wu
8fe79db4b1 context: do not warn broken dirstate repetitively
Summary: This makes the output less verbose.

Reviewed By: DurhamG

Differential Revision: D24411259

fbshipit-source-id: 50b175f2e31ffda62e78dd226775edaf53893bce
2020-10-20 15:24:28 -07:00
Jun Wu
79764c4e04 changelog2: track "tip" explicitly
Summary:
With segmented changelog, the maximum revision is not always the "tip".
Therefore, track "tip" explicitly in svfs / metalog. This improves
compatibility.

Reviewed By: DurhamG

Differential Revision: D24214711

fbshipit-source-id: 7b22de7ac684a90965a695924bdb9114758aae43
2020-10-20 15:24:28 -07:00
Jun Wu
3a96a2a3ee hgcommits: stabilize order of inserting commits
Summary: Avoid HashSet or HashMap order to preserve the order of inserting commits.

Reviewed By: DurhamG

Differential Revision: D24214460

fbshipit-source-id: 66df2e0aba1820e6585f8da66897078f38abf82f
2020-10-20 15:24:28 -07:00
Jun Wu
0ef48f98f1 bundlerepo: support non-revlog changelog
Summary:
Previously bundlerepo only supports revlog changelog. Detect repo configuration
and support non-revlog changelog. This is done by adding commits from bundle in
memory, and updating visibleheads.

Reviewed By: DurhamG

Differential Revision: D24214461

fbshipit-source-id: 63f9af49a0fe0ed2581b611f3cf7002d1b89d2fd
2020-10-20 15:24:27 -07:00
Jun Wu
2d60f9edc8 hgcommits: handle nullid and wdirid in streaming APIs
Summary:
The nullid and wdirid are special hg hashes that do not respect SHA1. They were
handled at the bindings layer. However the bindings layer cannot handle them
in a stream.  Therefore move it in hgcommits.

Reviewed By: DurhamG

Differential Revision: D24365330

fbshipit-source-id: e8dc6205351ec1a2304252b9ec446dda010e6295
2020-10-20 15:24:27 -07:00
Jun Wu
229713757d pythreading: add a reentrant generator
Summary:
Pure Python generator has a few limitations:
- Can only be iterated once.
- Cannot be iterated by multiple threads concurrently.

By converting revset iterators to Rust Stream, they run in different threads
and can context switch in `next()`, breaking the program.

Provide a native reentrant generator implementation to address the issue.
This is sound because Rust methods cannot be interrupted by Python.

Reviewed By: DurhamG

Differential Revision: D24365331

fbshipit-source-id: 885dade922b7863a73203b206a96b492d55bccd0
2020-10-20 15:24:27 -07:00
Jun Wu
06249e957f pydag: update return type of streamcommitrawtext from dict to tuple
Summary:
Update the return type of streamcommitrawtext from
`{"vertex": v, "raw_text": t}` to `(v, t)`.  This makes it easier to use
in Python, as Python supports `for v, t in ...` but does not support
`for {"vertex": v, "raw_text": t} in ...`.

Reviewed By: DurhamG

Differential Revision: D24295457

fbshipit-source-id: 284a29b9deae2d8509d3afea0fcbcaadbfebbae8
2020-10-20 15:24:27 -07:00
Jun Wu
b35886ecdf streams: add logic to handle missing items
Summary:
In case the server does not respect the input contract and missed
some items without returning errors. The current logic would retry
forever. Change it to detect the issue and raise an error.

Reviewed By: DurhamG

Differential Revision: D24293497

fbshipit-source-id: 09421c7743078a488a9c81ce66fd92c12b39543c
2020-10-20 15:24:27 -07:00
Jun Wu
3942b0cbe6 hgcommits: fix commit writing for hybrid backend
Summary: The store stores sorted(p1,p2)+text to match SHA1 hashes. It's not just `text`.

Reviewed By: DurhamG

Differential Revision: D24325554

fbshipit-source-id: 8a91970f60fb535ca1a5a2d30c7d27f2714f28de
2020-10-20 15:24:27 -07:00
Jun Wu
75ae217850 dag: remove NameDagStorage
Summary: It's no longer useful as the new abstract interface does not need it.

Reviewed By: sfilipco

Differential Revision: D24399516

fbshipit-source-id: 2b6735d2a26706c6a3e6b592d2f3ecfc874c94cb
2020-10-20 15:19:31 -07:00
Jun Wu
24edf32eac dag: impl MemNameDag using AbstractNameDag
Summary:
This verifies the abstraction and simplifies the code.

The new code will use non-master segments for add_heads. Therefore the test
changes.

Reviewed By: sfilipco

Differential Revision: D24399496

fbshipit-source-id: 39067ad88ade79b4f7758bcdaafc03e5f34ced91
2020-10-20 15:19:31 -07:00
Jun Wu
72c4a10e7e dag: move indexedlog NameDag to a module
Summary: This makes the main namedag.rs cleaner. The next step is to move MemNameDag.

Reviewed By: sfilipco

Differential Revision: D24399495

fbshipit-source-id: c1e79a60edd8597fe7264f04548e5312414241a7
2020-10-20 15:19:31 -07:00
Jun Wu
ddf2468f00 dag: impl Debug for AbstractNameDag
Summary: This is the last non-abstract interface of NameDag.

Reviewed By: sfilipco

Differential Revision: D24399514

fbshipit-source-id: f39bb84a1851a4fe4d1f29e6b0961e6a153c943d
2020-10-20 15:19:31 -07:00
Jun Wu
f4e4fb9342 dag: impl DagPersistent for AbstractNameDag
Summary: Implement DagPersistent using the abstracted interface.

Reviewed By: sfilipco

Differential Revision: D24399499

fbshipit-source-id: 3e2c3c776d4ff8d84238a4675d443e36ce212819
2020-10-20 15:19:31 -07:00
Jun Wu
c611f18d28 dag: add 'path' to AbstractNameDag
Summary:
There is a need to open AbstractNameDag cleanly from a path.
Abstract that.

Reviewed By: sfilipco

Differential Revision: D24399498

fbshipit-source-id: ca242cd929e8f5580120c01eeaa928f630c21ed7
2020-10-20 15:19:31 -07:00
Jun Wu
c41904b765 dag: impl DagAlgorithm for AbstractNameDag
Summary:
I copied the code since it's hard to implement using the macros.
In the future I plan to merge MemNameDag into AbstractNameDag
and remove the macros.

Reviewed By: sfilipco

Differential Revision: D24399517

fbshipit-source-id: 326e76cd06a6e1ad26b39bcb51ba0ff24106c984
2020-10-20 15:19:31 -07:00
Jun Wu
1b27600778 dag: impl IdConvert and PrefixLookup for AbstractNameDag
Summary: The `delegate!` is updated to support complex `impl`s.

Reviewed By: sfilipco

Differential Revision: D24399518

fbshipit-source-id: b9ba31174472cce4248e9644611cfc207abc3c1d
2020-10-20 15:19:30 -07:00
Jun Wu
0695b17375 dag: abstract some implementations
Reviewed By: sfilipco

Differential Revision: D24399504

fbshipit-source-id: 388b6788fbe7bbb30a34dcb91b4ca488d49ac8af
2020-10-20 15:19:30 -07:00
Jun Wu
558ed0466f dag: Persist::lock takes &mut self
Summary: It'll satisfy a future change.

Reviewed By: sfilipco

Differential Revision: D24399505

fbshipit-source-id: 6ee2cd0d2b4bd20003082e2733423647cb99619b
2020-10-20 15:19:30 -07:00
Jun Wu
f8e3d67631 dag: add TryClone trait
Summary: Will be used as bounds for abstraction.

Reviewed By: sfilipco

Differential Revision: D24399497

fbshipit-source-id: 343be12237d4850fbde9ebbe4034469527bd77fc
2020-10-20 15:19:30 -07:00
Jun Wu
f04e0fa60e namedag: define an abstract NameDag struct
Summary: It's not yet abstract. But one step closer.

Reviewed By: sfilipco

Differential Revision: D24399510

fbshipit-source-id: 32969733babd41b221170ee440f5d7ced1f7490a
2020-10-20 15:19:30 -07:00
Jun Wu
c49caf64b4 dag: drop snapshot_map from NameDag
Summary: The `snapshot` field can be used instead.

Reviewed By: sfilipco

Differential Revision: D24399507

fbshipit-source-id: 67de20d897b8b763f724f3ccbd46618dec7911b9
2020-10-20 15:19:30 -07:00
Jun Wu
465ac6c5df dag: drop IdMapEq trait
Summary:
The trait requires an `IdMap` snapshot to be locally ready. That's not easy for
all possible implementations. Drop it to simplify things.

Reviewed By: sfilipco

Differential Revision: D24399501

fbshipit-source-id: 4d85f77c99208cda30b2a543a0bb5b295f49a65c
2020-10-20 15:19:30 -07:00
Jun Wu
5f00c7984b dag: unify prepare_filesystem_sync
Summary: There were 2 prepare_filesystem_sync. Unify them into one implementation.

Reviewed By: sfilipco

Differential Revision: D24399513

fbshipit-source-id: 80d009c33b7f23dc2c4225da6fd0fb09589ba061
2020-10-20 15:19:30 -07:00
Jun Wu
4a7ffc4bd7 dag: use Locked for SyncableIdMap
Summary: Simplifies some code.

Reviewed By: sfilipco

Differential Revision: D24399500

fbshipit-source-id: a1317149da066617c4060b7efdae5234e5bd7262
2020-10-20 15:19:30 -07:00
Jun Wu
282c6800cb dag: use Locked for SyncableIdDag
Reviewed By: sfilipco

Differential Revision: D24399506

fbshipit-source-id: 91cfa176b8cfeca3f96dfeb211bf9d46a3d95bd5
2020-10-20 15:19:30 -07:00
Jun Wu
5c7b169e0e dag: add Locked type
Summary: More general purposed type for Syncable{IdDag,IdMap}.

Reviewed By: sfilipco

Differential Revision: D24399502

fbshipit-source-id: 0599db6dd07fe3d430458f86a33a9144d850fca1
2020-10-20 15:19:29 -07:00
Jun Wu
77790e9f49 dag: move non-master write APIs to IdMapWrite trait
Summary: This makes it more generic.

Reviewed By: sfilipco

Differential Revision: D24399493

fbshipit-source-id: 8a1d0a13dd29989b17fe3ef1497b10b6fa0629d6
2020-10-20 15:19:29 -07:00
Jun Wu
b249950984 dag: impl Persist on IdMaps
Summary: IdMap fits the Persist trait.

Reviewed By: sfilipco

Differential Revision: D24399494

fbshipit-source-id: 97b84d155f4b9bb3006bfad116defa4fca6330d6
2020-10-20 15:19:29 -07:00
Jun Wu
625b8ab4d5 dag: move IdMap impls to separate files
Summary: Similar to IdDag change, move impls to separate files.

Reviewed By: sfilipco

Differential Revision: D24399508

fbshipit-source-id: 575b6e7194677b67b6755b0a30ae7d014d498b10
2020-10-20 15:19:29 -07:00
Jun Wu
34aa41f24f dag: iddagstore::GetLock -> ops::Persist
Summary:
The lock, reload, mutate, persist pattern is general. It can be used for IdMap
too.

Reviewed By: sfilipco

Differential Revision: D24399512

fbshipit-source-id: d25e51ba735061ca101101d75aff95deb88b1d36
2020-10-20 15:19:29 -07:00
Jun Wu
f01dca1995 dag: drop build_segments_persistent APIs
Summary:
Now `build_segments_persistent` and `build_segments_volatile` are the same.
Just keep one of them.

Reviewed By: sfilipco

Differential Revision: D24399511

fbshipit-source-id: a9f1ac920cdf5b448bd99bf9b6d4ca4160ba0304
2020-10-20 15:19:29 -07:00
Jun Wu
5fad63b010 dag: drop last high level segment by default
Summary:
Previously, we keep the last high level segment per level in memory, and
drop it on disk. When we cross the memory / disk boundary, we had to
maintain such properties carefully. That was needed because some DAG
algorithms rely on complete high level segments.

Now that no DAG algorithms depend on such properties, let's just drop
the logic adding the last segment back to simplify the code.

This removes the need of building segments after open() and sync().

Reviewed By: sfilipco

Differential Revision: D24399515

fbshipit-source-id: 4c640d9aa03c050fcd97f70ee386e32d3a8ee26d
2020-10-20 15:19:29 -07:00
Jun Wu
496724e45e dag: make children work with missing high-level segments
Summary:
This makes the algorithm a bit more robust. Now none of the DAG algorithms
depend on high-level segments are complete and cover all low-level segments.

This also removes constraints. For example, SyncableIdDag can now just
deref() to the normal IdDag for queries without worrying about correctness.

Reviewed By: sfilipco

Differential Revision: D24399503

fbshipit-source-id: e6a91010cff82264cf423e2f24dee1d372822ef6
2020-10-20 15:19:29 -07:00
Jun Wu
f290afe421 dag: remove {range,descendants}_old algorithms
Summary:
They depend on high-level segments covering low-level segments, which
adds extra complexities. Remove them to simplify logic.

Reviewed By: sfilipco

Differential Revision: D24399509

fbshipit-source-id: 56a8e06c263107d1da4d6754b884ce51e18e30bf
2020-10-20 15:19:29 -07:00
Durham Goode
c80aaa4b56 ui: disable ui.threaded by default
Summary:
Even if ui.threaded is off, we're seeing sys call interrupt issues. I
think this is because ui.threaded defaults to true, and we check it's value
before loading the repo config (which includes the dynamicconfig to turn it
off). It's already off in dynamicconfig, so let's turn it off by default.

This highlights an issue with dynamicconfig, in that it can't deploy pre-repo
configs at the moment.

As for the actual interrupted syscall, it looks to be caused by SIGCHLD signals
from child processes we spawn (possibly the dynamicconfig child process).

Reviewed By: quark-zju

Differential Revision: D24401521

fbshipit-source-id: 0372ffc7297fa6fe6a7e4f0ae69299b252c3632e
2020-10-20 12:18:44 -07:00
Jun Wu
0f2c7b4abe localrepo: move changelog backend selection to a separate method
Summary:
Move logic to select a changelog backend to a separate method so it can be used
by bundlerepo.

Reviewed By: DurhamG

Differential Revision: D24214463

fbshipit-source-id: 6667317e9d51076b4a5d4a07779792f27cbd6d5e
2020-10-19 17:07:31 -07:00
Jun Wu
0498f87da1 lock: detect deadlock in a same process
Summary:
This is for cases like:

    repo1 = hg.repository(path)
    repo2 = hg.repository(path)  # same path
    with repo1.lock():
        repo2.lock()  # deadlock

Previously it simply deadlocks with confusing message:

    waiting for lock held by process X  # X is the current process

With this change it will raise a ProgrammingError about deadlock.

Reviewed By: DurhamG

Differential Revision: D24214458

fbshipit-source-id: 57faddaed6d2f1d36fab067c2e1f63b3cda0a1fe
2020-10-19 17:07:30 -07:00
Jun Wu
696883188b cmdutil: rename rustdisplaygraph to displaygraph
Summary:
Replace the legacy `displaygraph` with `rustdisplaygraph`. The `edegfn`
parameter is no longer used and is dropped.

Reviewed By: DurhamG

Differential Revision: D24299592

fbshipit-source-id: 8336da1c8831b3352a950544d72cbdcf0203858d
2020-10-19 17:07:30 -07:00
Jun Wu
d394108655 graphmod: drop legacy rendering logic
Summary: Only keep the new Rust renderer.

Reviewed By: DurhamG

Differential Revision: D24299593

fbshipit-source-id: a0705ce477a5e1166a71fbadac051ae093b4598d
2020-10-19 17:07:30 -07:00
Jun Wu
2af1daa778 smartlog: remove support for legacy graph renderer
Summary: Always use the Rust graph renderer.

Reviewed By: DurhamG

Differential Revision: D24299594

fbshipit-source-id: 0f9b9688d2447ca4f05bfd50fc133e2606d2020b
2020-10-19 17:07:30 -07:00
Jun Wu
9bf6b674a6 config: use Rust graph render as default
Summary: Change the legacy Python graph render to Rust renderer.

Reviewed By: DurhamG

Differential Revision: D24317802

fbshipit-source-id: 4c3dc3a6dd02b7ebe79596a8e77f4b6b139d2e20
2020-10-19 17:07:30 -07:00
Jun Wu
9743d2f06a changelog2: keep laziness of findcommonmissing
Summary:
`onlyboth` returns a lazy set for revlog backend.
`torevs` does not work well with such lazy set.
Use `torevset` instead.

Reviewed By: DurhamG

Differential Revision: D24397872

fbshipit-source-id: e7f5c6412b4120305b67103ceff1b1922a67ec7d
2020-10-19 13:17:35 -07:00
Jun Wu
6dac514fae continue: pass --noninteractive to continued command
Summary: This preserves the `--noninteractive` flag used by some tools.

Reviewed By: DurhamG

Differential Revision: D24040789

fbshipit-source-id: 8d50f3f3ce6b2015f0ef6c3bd1b4fbb874d0ea7d
2020-10-16 18:40:51 -07:00
Jun Wu
b36584e704 configparser: set ui.merge:interactive from ui.merge in user config
Summary:
This restores the compatibility of setting up merge tools using the `ui.merge`
config while still limiting the default `editmerge` tool to interactive
sessions.

Reviewed By: sfilipco

Differential Revision: D24377259

fbshipit-source-id: 3d2befba412b824fc985ddffa131e339644178c2
2020-10-16 18:33:06 -07:00
Jun Wu
122108e46e configparser: move load_user to a testable method
Summary: Make it testable by allowing specifying paths to load as user hgrc.

Reviewed By: sfilipco

Differential Revision: D24377258

fbshipit-source-id: 969028df64d55ad1f1304e35675d84595ed6a2bf
2020-10-16 18:33:06 -07:00
Arun Kulshreshtha
d16a62ce06 edenapi: send user agent string
Summary:
Include a `User-Agent` header in EdenAPI requests from Mercurial. This will allow us to see the version in Scuba, and in the future, will allow us to distinguish between requests send by Mercurial and those sent directly by EdenFS.

Keeping with the current output of `hg version`, the application is specified as "EdenSCM" rather than "Mercurial".

Reviewed By: singhsrb

Differential Revision: D24347021

fbshipit-source-id: e323cfc945c9d95d8b2a0490e22c2b2505a620dc
2020-10-16 11:05:24 -07:00
Simon Farnsworth
3c7ce4eed1 hg histedit: document autoverb badly
Summary: This was competely undocumented, and thus undiscoverable. Add some documentation, so that it's at least clear the feature exists

Reviewed By: singhsrb

Differential Revision: D24332961

fbshipit-source-id: 9e73163a9314ceb7f953a3b1ac0f58c9a6e6d4d9
2020-10-16 09:30:28 -07:00
Jun Wu
7b132fb6e2 smartlog: stop filtering commits by date
Summary:
The hide-before config was added to encourage people to actively hide unused
drafts for repo performance, instead of keeping unused draft forever, since a
lot of code paths assume `len(draft)` is small. See D13993584 (28b4dfbb38) for more context.

Now our hide-before data is set to 2.5 years ago (2018-2-25), this change
probably only affects a very small number of users.

Reviewed By: DurhamG

Differential Revision: D24298198

fbshipit-source-id: 938aca1222b55e09fdb058ff01bc063733f201dc
2020-10-15 22:48:41 -07:00
Jun Wu
e39f3bc233 revisionstore: add mutex for tests related to env vars
Summary:
Rust tests run in multiple threads. Setting environment variables affects other
tests running in other threads and causes random test failures.
Protect env vars using a lock.

Reviewed By: DurhamG

Differential Revision: D24296639

fbshipit-source-id: db0bee85625a7b63e07b95ea76d96029487881d4
2020-10-15 22:48:41 -07:00
Jun Wu
263d1c5a7f hg: reduce flakiness of cargo tests
Summary:
The shell-script cargo tests seem very flaky. Use a dedicated Python script to
run the tests, with a more concise output that only includes failures, and run
tests in parallel.

Reviewed By: DurhamG

Differential Revision: D24296433

fbshipit-source-id: 1d63146c6c84f1035dded24fcd3d79f116c2e740
2020-10-15 22:48:41 -07:00
Durham Goode
174f05f3fb convert: disable p4 tests on Python 3
Summary:
Ideally we'd just delete the p4 convert functionality, but I'm too lazy
to go through and extract it right now.

This was recently enabled when I enabled all the convert tests.  We don't use the p4 logic, so it's safe to just turn back off to get a release out.

Reviewed By: quark-zju

Differential Revision: D24352068

fbshipit-source-id: 6f3a1f88739b2e2348aff00e8cae333473bbe71a
2020-10-15 22:23:24 -07:00
Durham Goode
335988858e ui: fix prompt on Windows
Summary: My recent change accidentally returned early when reading the prompt input, which skipped the \r truncation needed for Windows.

Reviewed By: sfilipco

Differential Revision: D24350672

fbshipit-source-id: 4a589d76bf41cda7fda2518003ef272f9a6ead48
2020-10-15 22:19:38 -07:00
Durham Goode
4bebe853fd remotefilelog: add debug output for pack fetching
Summary:
We're seeing slow pack fetches in some cases. Let's add some extra
debug output to get more information.

Reviewed By: quark-zju

Differential Revision: D24295593

fbshipit-source-id: b5a5bdf169a8c05a3143da09d69646a7a742ef08
2020-10-15 11:19:24 -07:00
Durham Goode
9da5707131 checkout: disable fetching history during checkout
Summary:
We're seeing history fetching take quite a while during checkout and
rebase, but it's not really necessary for a checkout. In addition to it being
slow, if memcache doesn't have the history entry we'll fall back to a full
revision fetch from the server, which includes data. Let's disable prefetching
history during checkout.

Reviewed By: quark-zju

Differential Revision: D24295594

fbshipit-source-id: 70aa6e1925074b6546626a5192a7562d6da31f2b
2020-10-15 11:19:23 -07:00
Durham Goode
e60d023263 infinitepush: fix sql integration on Python 2
Summary:
My recent diff changed these lines from `bytes(value)` to
`pycompat.decodeutf8(value)`, since we want these to be strings. Unfortunately,
on python 2 decodeutf8() just passes the value straight through, so in cases
where we're handed bytearray, we pass that through instead of converting it to
bytes. Some down stream consumers requires it to be bytes.

Let's conditionally turn it to bytes.

Reviewed By: krallin

Differential Revision: D24307818

fbshipit-source-id: a0cc64b7e2cf7645586e633e7a4a382b69390e15
2020-10-15 11:16:46 -07:00
Thomas Orozco
35ed8fe3dc revisionstore/lfs: retry 429
Summary:
When the server returns a 429, the intention is that the client will wait for a
little bit then try again later (there is no harm in that, as we haven't really
used many server resources for this). However, it turned out that right now we
just abort. Let's fix it!

Note that this changes the behavior a bit for the error cases, in the sense
that we no longer return `Ok(None)` but instead return an `Err`. Xavier noted
this should make sense here.

I've also had the client send its retry attempt via a header, because who
knows, that might be useful.

Reviewed By: kulshrax

Differential Revision: D24308127

fbshipit-source-id: 35639956f36342dfb0056b0d348dc4ad56bd576c
2020-10-15 01:11:32 -07:00
John Reese
737849c9dd apply black 20.8b1 formatting update
Summary:
allow-large-files

black_any_style

Reviewed By: zertosh

Differential Revision: D24325133

fbshipit-source-id: b4afe80d1e8b2bc993f4b8e3822c02964df47462
2020-10-14 20:21:52 -07:00
Jun Wu
0d7ae7e7cd util: move eachslice to util
Summary:
This is Ruby's `iter.each_slice(n)` or Rust's `slice.chunks(n)`.
It will be used by phabstatus to batch requests.

Reviewed By: singhsrb

Differential Revision: D24289050

fbshipit-source-id: 528b35d43874222f1b6783738e965d9a3b159deb
2020-10-14 15:09:12 -07:00
Jun Wu
2dd8aa33b0 smartset: add iterctx API
Summary: Make it easier to iterate contexes instead of revs.

Reviewed By: simpkins

Differential Revision: D24289053

fbshipit-source-id: deffdc4a5d4e7e3644a98f8112f5c8b8bdbd154d
2020-10-14 15:09:12 -07:00
Jun Wu
49dd754fa6 smartlog: do not use null as --master fallback
Summary:
Previously `--master` falls back to `null` if `-r` is provided for
compatibility. That could be surprising as smartlog's sorting does not work
well without a master branch.

Change `--master` to a consistent value regardless of `-r` is provided or not.

Reviewed By: simpkins

Differential Revision: D24297586

fbshipit-source-id: c6e695333ae17db11dcb98d9cdb57cc2f298ff64
2020-10-14 12:57:36 -07:00
Meyer Jacobs
120fbd3280 trees: port SCS aux data request method to edenapi
Summary: Introduces fetching of child entry IDs, and child file metadata for a specified tree manifest ID. The aux data lookup will only be performed if `with_file_metadata` is set, which is actually kind of wrong. Instead `with_children` from the wire type should be exposed in the API request type, and `with_*_metadata` should be hidden or used for data other than the child entry `Key`s.

Reviewed By: kulshrax

Differential Revision: D23886678

fbshipit-source-id: 0cba72cea7be47ae3348a406d407a19b60976c0c
2020-10-14 11:12:59 -07:00
Durham Goode
0fed091abb py3: fix treemanifest server tests
Summary:
simplecache didn't support binary values, which treemanifest uses.
Let's fix that.

Reviewed By: quark-zju

Differential Revision: D24290634

fbshipit-source-id: 2fd5306e10c3354d87832eb6765baca5dbb26d3e
2020-10-14 10:30:32 -07:00
Durham Goode
bff14ff181 py3: fix more tests
Summary: Some of these tests were already passing. Some required minor fixes to pass.

Reviewed By: quark-zju

Differential Revision: D24290635

fbshipit-source-id: 789d3033142466c868057da8f9f23d27aede8ea4
2020-10-14 10:30:32 -07:00
Durham Goode
0cbe0887f6 py3: fix test-convert*
Summary:
They now pass. Didn't test the p4 tests since I don't have p4
available, but I don't think we use it anymore.

Reviewed By: quark-zju

Differential Revision: D24284643

fbshipit-source-id: 184697052c0b7000ad9765b4033964b4d382b811
2020-10-14 10:30:32 -07:00
Xavier Deguillard
bc351a5891 templater: decode the result of diff()
Summary:
Template filters are expecting the output of the template to be a unicode
string, not a byte string. Thus we need to ensure that diff return a unicode
string.

Reviewed By: DurhamG

Differential Revision: D24290908

fbshipit-source-id: 70899249c3e26edd1714fd901fa6af405b61e544
2020-10-14 09:00:53 -07:00
Adam Simpkins
a1c72ad436 revert phabstatus iterator changes from D24066918
Summary:
This reverts D24066918 (a684e53a31).  That diff changed the phabstatus code to use the last
created revset iterator rather than the first for peekahead purposes.

The first revset iterator does appear to be the correct one to wrap.  After
D24066918 (a684e53a31) I consistently see `hg log -r 'draft()' -T'{phabstatus}\n'` make
Phabricator queries for only one diff at a time.  Reverting the changes from
that diff fixes the issue.

Note that the revset argument passed to `hg log` appears to make a difference
in the behavior here.  When using simple revsets including just specific
commit ranges then the peekahead behavior works as expected both before and
after D24066918 (a684e53a31).  However, when using `draft()` in the revset specifier then
the behavior works before D24066918 (a684e53a31), but not after it.

Reviewed By: singhsrb

Differential Revision: D24269910

fbshipit-source-id: c2c2abf6360fe3129a1fc73b5b433b763bce1fd2
2020-10-14 00:31:30 -07:00
Durham Goode
fe6e10c024 infinitepush: convert sql results to string
Summary:
The infinitepush sqlindex was returning bytes instead of strings, which
caused various formatters to serialize as "b'sadf'" instead of "sadf". Let's fix
the type converter to produce strings on python 3. Note, this is the opposite to
what we do in the hgsql converter. hgsql works with binary data, so we default
to bytes there. The sqlindex doesn't deal with any binary data, so defaulting to
strings is better.

Also updates a comment on the hgsql converter to be accurate. I confirmed there
are no other customer converters in the code base that need updating.

Reviewed By: krallin

Differential Revision: D24282899

fbshipit-source-id: 708732c17626f6ebc2c3cbe7f00afa5771f31d7b
2020-10-13 21:21:12 -07:00
Arun Kulshreshtha
32c109d955 edenapi: send client correlator to server
Summary:
Include the client correlator string from the `clienttelemetry` extension in each EdenAPI HTTP request via the  `X-Client-Correlator` header.

The `ClientIdentityMiddleware` in `gotham_ext` already understands this header (as it is already used by the LFS server), and `gotham_ext`'s `ScubaMiddleware` will automatically include the provided correlator in the server's Scuba samples.

Reviewed By: farnz

Differential Revision: D24282244

fbshipit-source-id: 13d04e706eda38893cff6e740bd1d7bf104e43dd
2020-10-13 13:25:52 -07:00
Lukas Piatkowski
715351d576 mononoke/integration tests: fix test-clienttelemetry.t by disabling features
Summary:
The logic inside `eden/scm/tests/features.py` script enables certain
features of HG on tests based on the name of the test. Mononoke's integration
tests suit reuses eden/scm's tests suit and as a consequence it triggers the
same `features.py` logic. It was fine until D24201934 (f5988c415c) introduced a feature for
test that is named in the same way as one of the Mononoke tests - the
`test-clienttelemetry.t`.

In order to fix this problem Mononoke will pass a `--nofeatures` flag to the
`eden/scm/run-tests.py` invocations that will turn off the usage of
`features.py`.

Reviewed By: farnz

Differential Revision: D24276294

fbshipit-source-id: eb28ed55a05de3b012e37407603c2370adaaad16
2020-10-13 05:28:27 -07:00
Adam Simpkins
25cb27f395 handle timeouts reading from phabricator
Summary:
If Phabricator takes more than 10 seconds to respond, a `socket.timeout`
exception may be thrown.  Treat this like other networking errors, and simply
report the diff phabricator status as `Error`.  Previously this exception was
unhandled, causing the entire command to abort.

Reviewed By: singhsrb

Differential Revision: D24272274

fbshipit-source-id: f646d111a91f901e09d9f94a1e0102d6dd4d0952
2020-10-13 03:04:02 -07:00
Meyer Jacobs
f9958ca35a taggederror: introduce category and transience metadata and precedence
Summary:
This change introduces two new metadata types, Category and Transience, and a mechanism for Category to provide a default Fault and Transience, which can be overriden by the user.

Also introduces a mechanism for attempting to log exceptions which occur during exception logging, falling back to the previous behavior of just swallowing the exception on failure.

Reviewed By: DurhamG

Differential Revision: D22677565

fbshipit-source-id: 1cf75ca1e2a65964a0ede1f072439378a46bd391
2020-10-12 17:17:34 -07:00
Jun Wu
bd5cfe49b2 commitstore: remove it
Summary:
It only has benchmark code that led to the use of mincode. Now hgcommits is the
main crate for commit storage. `commitstore` without `hg` in its name was
initially planned to support other kinds of commits including git and bonsai.
However we don't have immediate goal for that at present. So let's just remove
the commitstore directory.

Reviewed By: singhsrb

Differential Revision: D24263618

fbshipit-source-id: 84b4861ae490817377e69d8c2006c63331e3db1f
2020-10-12 16:42:58 -07:00
Xavier Deguillard
437031872f fix test-import-t.py take 2
Summary: Whitespaces are needed.

Reviewed By: quark-zju

Differential Revision: D24264452

fbshipit-source-id: a7ecfbe1bba7d14f92d5218f18d55361a3f4319e
2020-10-12 15:43:57 -07:00
Durham Goode
b0e341c311 signals: move signal registering to be behind ui.threaded
Summary:
On the hg servers we're seeing crashes due to signals during syscalls.
Rolling back to prior to the signal changes seems to have fixed it, though we
haven't bisected enough to be sure this is the cause.

The ui.threaded option is already used to opt-out of running on a background
thread, let's also use it to opt-out of the custom signal registering in hopes
that it fixes the server issue, until they can be deprecated.

Reviewed By: quark-zju

Differential Revision: D24254804

fbshipit-source-id: 50e4fc8c7e3a88b5baa49394f6f1edffc946187d
2020-10-12 14:36:47 -07:00
Meyer Jacobs
87cc599161 edenapi: add aux data to FileMetadata and DirectoryMetadata, and recursive children field to TreeEntry
Summary: Need to add new quickcheck tests, verify that remove `Serialize` from `TreeEntry` is okay.

Reviewed By: kulshrax

Differential Revision: D23457777

fbshipit-source-id: aa94ed7aa81b41924eba4a8bd1bdc2c737365b77
2020-10-12 14:05:23 -07:00
Jun Wu
a35f8765c0 lock: print actual lockinfo on error
Summary:
Change

  abort: repository repo: timed out waiting for lock held by <lockinfo ...>

to:

  abort: repository repo: timed out waiting for lock held by process '842210' on host 'hostname'

Reviewed By: singhsrb

Differential Revision: D24214462

fbshipit-source-id: 65056ebb9764651b2f0126061fafdfdefaa4e9c6
2020-10-12 11:22:20 -07:00
Xavier Deguillard
964e0106c9 tests: fix test-import-t.py
Summary: The rev numbers are almost gone, remove them from the test.

Reviewed By: sfilipco

Differential Revision: D24255156

fbshipit-source-id: 5cbc4a71c2d3f773c5b474d1edce84ceceb50bf9
2020-10-12 10:18:45 -07:00
Durham Goode
d10157794d eden: add flush command
Summary:
This will allow Eden to control what data is in Mercurial's memory and
what is on disk. This will let it persist the hg_import_helper process longer,
and prevent slow startup times when needing to restart it.

Reviewed By: xavierd

Differential Revision: D24231131

fbshipit-source-id: a4f743740b44957e8d2dd93f07e9831eadfee7ab
2020-10-09 18:18:00 -07:00
Durham Goode
d4f21a8ec3 atomicfiles: remove fsync from atomic file close
Summary:
This was added in D16078908 (1bc6ecf8fe). It turns out fsync followed by a hardlink
on btrfs results in very slow hardlink performance (100-600ms). Since fewer and
fewer of our files use this atomic write code path and since this affects almost
every hg write command, let's roll this back.

This will increase the chance of data loss during a hard reboot, but commit
cloud is good enough to allow recovering from that in almost every situation.

Reviewed By: sfilipco

Differential Revision: D24230056

fbshipit-source-id: aae1a5612eda5f62bb5ec1442b1312ed45c42786
2020-10-09 16:41:04 -07:00
Jun Wu
e4b677c6e1 remotefilelog: respect invalidatelinkrev store requirement
Summary:
With invalidatelinkrev, linkrev stored in revlog shouldn't be used.
This makes some tests pass with segmented changelog.

Reviewed By: singhsrb

Differential Revision: D24201944

fbshipit-source-id: 6473c30266c79aa97a955b1a6c867411cc67de2b
2020-10-09 13:53:18 -07:00
Jun Wu
7e3dabaab0 context: handle string negative revs with segmented changelog
Summary:
The old code assumes `0..len(repo)` are valid revs, which is no longer true
with segmented changelog.

Reviewed By: singhsrb

Differential Revision: D24201948

fbshipit-source-id: b882a215701c57ccdf4af340c889586b040772da
2020-10-09 13:53:18 -07:00
Jun Wu
2f1c54ae27 clone: copy modern store files
Summary:
Copy segmented changelog, metalog and mutation store for local clones.
This mainly affects tests.

Reviewed By: singhsrb

Differential Revision: D24201941

fbshipit-source-id: c60da9e2bf982a6f66004415e45178749157745e
2020-10-09 13:53:18 -07:00
Jun Wu
631034a2d6 gpg: remove revnum from output
Reviewed By: singhsrb

Differential Revision: D24201943

fbshipit-source-id: 6998ff68ba50480fb7feb4be3b35de1ea4a3ab4a
2020-10-09 13:53:17 -07:00