Commit Graph

61256 Commits

Author SHA1 Message Date
Egor Tkachenko
2494f1db7e Fix cache pool sizes
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`

Reviewed By: johansglock

Differential Revision: D24448419

fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
2020-10-21 10:16:45 -07:00
Simon Farnsworth
4df44dae29 Add the session ID into the clienttelemetry hostname
Summary: The hostname is of limited use when running on Tupperware - add in the session ID, which lets us look at what happened.

Reviewed By: StanislavGlebik

Differential Revision: D24361320

fbshipit-source-id: 0ac2c64d68915288b3241320c7ee14452b754ec8
2020-10-21 08:59:47 -07:00
Stanislau Hlebik
d4326273a7 mononoke: change debug logging to info
Summary:
I removed --debug flag for bookmarks validator on tw because it was too spammy,
but now we get no output at all. Let's make it info so that we can see it in tw
output

Reviewed By: ikostia

Differential Revision: D24445829

fbshipit-source-id: d37c41be46794867f91ecf66f0318e7b2d660d85
2020-10-21 05:53:09 -07:00
svcscm
e629afcbe8 Updating submodules
Summary:
GitHub commits:

d6ed6a77e2
c0a75500df
c17e84049e
afced0bf7b
e5af792870

Reviewed By: yns88

fbshipit-source-id: 0820e4ffd85614f1166e8479dd0598edbf9aad59
2020-10-21 03:41:17 -07:00
svcscm
2d7fc0dc75 Updating submodules
Summary:
GitHub commits:

fe3b43ea49
8345649615
739171fb18

Reviewed By: yns88

fbshipit-source-id: a735f937a6eb7c06a94d0a493d9d8142876a9bbb
2020-10-21 03:41:17 -07:00
svcscm
715854d105 Updating submodules
Summary:
GitHub commits:

5649047eff

Reviewed By: yns88

fbshipit-source-id: 93e56f484669653dea4589077da4cbfc88f7505c
2020-10-20 19:36:51 -07:00
Jun Wu
91ab519edb streams: add API to prefetch commit text in a streaming fashion
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.

Reviewed By: sfilipco

Differential Revision: D24321293

fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
2020-10-20 18:40:58 -07:00
Chad Austin
76409f59b7 avoid redundant change notifications when prior notifications have not been observed
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.

This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.

Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.

* Technically, if there were a multitude of subscribeStreamTemporary
  streams, and all but one was not calling getFilesChangedSince, it's
  possible that notifications could back up on the stuck
  subscriptions.

Reviewed By: wez

Differential Revision: D24090247

fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
2020-10-20 18:25:43 -07:00
Jun Wu
d86802a980 parser: add a function to walk through a tree
Summary: This will be used to test if a keyword is in an AST.

Reviewed By: DurhamG

Differential Revision: D24293499

fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
2020-10-20 16:45:43 -07:00
Jun Wu
5a6161f140 repo: unconditionally migrate to specified visibility and dirstate format
Summary:
There were 2 recent reports about "visibleheads" not being enabled.

Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.

There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.

Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.

Reviewed By: DurhamG

Differential Revision: D24340586

fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
2020-10-20 16:36:50 -07:00
Jun Wu
9de7c7a22d repo: drop format.usestore=false support
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.

Reviewed By: DurhamG

Differential Revision: D24340585

fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
2020-10-20 16:36:50 -07:00
Jun Wu
e179789b7b tests: remove test cases about legacy dirstates
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.

Reviewed By: DurhamG

Differential Revision: D24340584

fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
2020-10-20 16:36:50 -07:00
svcscm
36e5276ba9 Updating submodules
Summary:
GitHub commits:

e53bda6f10
55fdba7f48
6d56669bb1

Reviewed By: yns88

fbshipit-source-id: 257d47e6b40526dca2285f751fe53e5176590a73
2020-10-20 16:05:11 -07:00
Jun Wu
a3ac43a23a rollout: use doublewrite changelog backend for hg-dev
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).

Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.

Reviewed By: singhsrb

Differential Revision: D24259759

fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
2020-10-20 15:53:57 -07:00
Chad Austin
d5c04ab941 report opcode and process names for outstanding fuse calls
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.

Reviewed By: kmancini

Differential Revision: D24036420

fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
2020-10-20 15:35:47 -07:00
Jun Wu
7db31b57b6 templater: add API to parse string to AST with aliases expanded
Summary: It will be used to test whether `{phabstatus}` is in template.

Reviewed By: sfilipco

Differential Revision: D24293496

fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
2020-10-20 15:24:29 -07:00
Jun Wu
c9d716fd51 test-rebase-emptycommit: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270422

fbshipit-source-id: d2dcf6db2a0c8ebd90a2edb3ee1149d8f9bd4fbe
2020-10-20 15:24:29 -07:00
Jun Wu
063bc373af test-rebase-check-restore: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270424

fbshipit-source-id: 2833dbbdc7ec34375f070fc6ce273265272a205e
2020-10-20 15:24:29 -07:00
Jun Wu
293d05dd40 test-rebase-conflicts: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258215

fbshipit-source-id: 9a9bba13c3963c79df5e672e5ac011ab7c7e5698
2020-10-20 15:24:29 -07:00
Jun Wu
20ac277bc5 test-rebase-brute-force: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258213

fbshipit-source-id: aceb6e0792b4646eb0805f56d981751eb85ae340
2020-10-20 15:24:29 -07:00
Jun Wu
af38cf3cb9 test-rebase-abort: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258214

fbshipit-source-id: db7ecb6b874da81d0e553cd9f1514ee5559635b7
2020-10-20 15:24:29 -07:00
Jun Wu
d39be1d667 tests: mark more tests as compatible with segmented changelog
Reviewed By: singhsrb

Differential Revision: D24214459

fbshipit-source-id: 7c125cbc2c0d559d337bd01107b784002e2dc9ea
2020-10-20 15:24:29 -07:00
Jun Wu
b46a5b6192 copies: improve compatibility with segmented changelog
Summary:
There are 2 issues:
- len(repo) is no longer the "max rev" + 1. Use 1<<63 instead.
- If invalidatelinkrev is a repo requirement, linkrev can no longer be trusted.

Reviewed By: DurhamG

Differential Revision: D24270423

fbshipit-source-id: 4e5c999772fad6ef2b1dabd2d6363d4226242c4d
2020-10-20 15:24:28 -07:00
Jun Wu
62f30ea23e changegroup: do not use bad linkrev to filter out file revisions
Summary:
If linkrevs are known unreliable, do not use them. This fixes some cases with
segmented changelog, file revisions are incorrectly filtered out and not sent
to client.

Reviewed By: DurhamG

Differential Revision: D24233228

fbshipit-source-id: 268a95b44c0217658b849317cb0a680450b91d23
2020-10-20 15:24:28 -07:00
Jun Wu
a7ba754d88 rebase: remove more uses of revnums
Summary: Replace revnum with commit hashes.

Reviewed By: singhsrb

Differential Revision: D24233229

fbshipit-source-id: 98b4d45b542b65c375cef405b45b66cd2dad22e4
2020-10-20 15:24:28 -07:00
Jun Wu
a4b3f573ba changelog: change "ambiguous prefix" to RepoLookupError
Summary: `LookupError` is too broad.

Reviewed By: DurhamG

Differential Revision: D24411203

fbshipit-source-id: 3e7c51618115c11f87f0fed670d18b4b5a03e20d
2020-10-20 15:24:28 -07:00
Jun Wu
8fe79db4b1 context: do not warn broken dirstate repetitively
Summary: This makes the output less verbose.

Reviewed By: DurhamG

Differential Revision: D24411259

fbshipit-source-id: 50b175f2e31ffda62e78dd226775edaf53893bce
2020-10-20 15:24:28 -07:00
Jun Wu
79764c4e04 changelog2: track "tip" explicitly
Summary:
With segmented changelog, the maximum revision is not always the "tip".
Therefore, track "tip" explicitly in svfs / metalog. This improves
compatibility.

Reviewed By: DurhamG

Differential Revision: D24214711

fbshipit-source-id: 7b22de7ac684a90965a695924bdb9114758aae43
2020-10-20 15:24:28 -07:00
Jun Wu
3a96a2a3ee hgcommits: stabilize order of inserting commits
Summary: Avoid HashSet or HashMap order to preserve the order of inserting commits.

Reviewed By: DurhamG

Differential Revision: D24214460

fbshipit-source-id: 66df2e0aba1820e6585f8da66897078f38abf82f
2020-10-20 15:24:28 -07:00
Jun Wu
0ef48f98f1 bundlerepo: support non-revlog changelog
Summary:
Previously bundlerepo only supports revlog changelog. Detect repo configuration
and support non-revlog changelog. This is done by adding commits from bundle in
memory, and updating visibleheads.

Reviewed By: DurhamG

Differential Revision: D24214461

fbshipit-source-id: 63f9af49a0fe0ed2581b611f3cf7002d1b89d2fd
2020-10-20 15:24:27 -07:00
Jun Wu
2d60f9edc8 hgcommits: handle nullid and wdirid in streaming APIs
Summary:
The nullid and wdirid are special hg hashes that do not respect SHA1. They were
handled at the bindings layer. However the bindings layer cannot handle them
in a stream.  Therefore move it in hgcommits.

Reviewed By: DurhamG

Differential Revision: D24365330

fbshipit-source-id: e8dc6205351ec1a2304252b9ec446dda010e6295
2020-10-20 15:24:27 -07:00
Jun Wu
229713757d pythreading: add a reentrant generator
Summary:
Pure Python generator has a few limitations:
- Can only be iterated once.
- Cannot be iterated by multiple threads concurrently.

By converting revset iterators to Rust Stream, they run in different threads
and can context switch in `next()`, breaking the program.

Provide a native reentrant generator implementation to address the issue.
This is sound because Rust methods cannot be interrupted by Python.

Reviewed By: DurhamG

Differential Revision: D24365331

fbshipit-source-id: 885dade922b7863a73203b206a96b492d55bccd0
2020-10-20 15:24:27 -07:00
Jun Wu
06249e957f pydag: update return type of streamcommitrawtext from dict to tuple
Summary:
Update the return type of streamcommitrawtext from
`{"vertex": v, "raw_text": t}` to `(v, t)`.  This makes it easier to use
in Python, as Python supports `for v, t in ...` but does not support
`for {"vertex": v, "raw_text": t} in ...`.

Reviewed By: DurhamG

Differential Revision: D24295457

fbshipit-source-id: 284a29b9deae2d8509d3afea0fcbcaadbfebbae8
2020-10-20 15:24:27 -07:00
Jun Wu
b35886ecdf streams: add logic to handle missing items
Summary:
In case the server does not respect the input contract and missed
some items without returning errors. The current logic would retry
forever. Change it to detect the issue and raise an error.

Reviewed By: DurhamG

Differential Revision: D24293497

fbshipit-source-id: 09421c7743078a488a9c81ce66fd92c12b39543c
2020-10-20 15:24:27 -07:00
Jun Wu
3942b0cbe6 hgcommits: fix commit writing for hybrid backend
Summary: The store stores sorted(p1,p2)+text to match SHA1 hashes. It's not just `text`.

Reviewed By: DurhamG

Differential Revision: D24325554

fbshipit-source-id: 8a91970f60fb535ca1a5a2d30c7d27f2714f28de
2020-10-20 15:24:27 -07:00
Jun Wu
75ae217850 dag: remove NameDagStorage
Summary: It's no longer useful as the new abstract interface does not need it.

Reviewed By: sfilipco

Differential Revision: D24399516

fbshipit-source-id: 2b6735d2a26706c6a3e6b592d2f3ecfc874c94cb
2020-10-20 15:19:31 -07:00
Jun Wu
24edf32eac dag: impl MemNameDag using AbstractNameDag
Summary:
This verifies the abstraction and simplifies the code.

The new code will use non-master segments for add_heads. Therefore the test
changes.

Reviewed By: sfilipco

Differential Revision: D24399496

fbshipit-source-id: 39067ad88ade79b4f7758bcdaafc03e5f34ced91
2020-10-20 15:19:31 -07:00
Jun Wu
72c4a10e7e dag: move indexedlog NameDag to a module
Summary: This makes the main namedag.rs cleaner. The next step is to move MemNameDag.

Reviewed By: sfilipco

Differential Revision: D24399495

fbshipit-source-id: c1e79a60edd8597fe7264f04548e5312414241a7
2020-10-20 15:19:31 -07:00
Jun Wu
ddf2468f00 dag: impl Debug for AbstractNameDag
Summary: This is the last non-abstract interface of NameDag.

Reviewed By: sfilipco

Differential Revision: D24399514

fbshipit-source-id: f39bb84a1851a4fe4d1f29e6b0961e6a153c943d
2020-10-20 15:19:31 -07:00
Jun Wu
f4e4fb9342 dag: impl DagPersistent for AbstractNameDag
Summary: Implement DagPersistent using the abstracted interface.

Reviewed By: sfilipco

Differential Revision: D24399499

fbshipit-source-id: 3e2c3c776d4ff8d84238a4675d443e36ce212819
2020-10-20 15:19:31 -07:00
Jun Wu
c611f18d28 dag: add 'path' to AbstractNameDag
Summary:
There is a need to open AbstractNameDag cleanly from a path.
Abstract that.

Reviewed By: sfilipco

Differential Revision: D24399498

fbshipit-source-id: ca242cd929e8f5580120c01eeaa928f630c21ed7
2020-10-20 15:19:31 -07:00
Jun Wu
c41904b765 dag: impl DagAlgorithm for AbstractNameDag
Summary:
I copied the code since it's hard to implement using the macros.
In the future I plan to merge MemNameDag into AbstractNameDag
and remove the macros.

Reviewed By: sfilipco

Differential Revision: D24399517

fbshipit-source-id: 326e76cd06a6e1ad26b39bcb51ba0ff24106c984
2020-10-20 15:19:31 -07:00
Jun Wu
1b27600778 dag: impl IdConvert and PrefixLookup for AbstractNameDag
Summary: The `delegate!` is updated to support complex `impl`s.

Reviewed By: sfilipco

Differential Revision: D24399518

fbshipit-source-id: b9ba31174472cce4248e9644611cfc207abc3c1d
2020-10-20 15:19:30 -07:00
Jun Wu
0695b17375 dag: abstract some implementations
Reviewed By: sfilipco

Differential Revision: D24399504

fbshipit-source-id: 388b6788fbe7bbb30a34dcb91b4ca488d49ac8af
2020-10-20 15:19:30 -07:00
Jun Wu
558ed0466f dag: Persist::lock takes &mut self
Summary: It'll satisfy a future change.

Reviewed By: sfilipco

Differential Revision: D24399505

fbshipit-source-id: 6ee2cd0d2b4bd20003082e2733423647cb99619b
2020-10-20 15:19:30 -07:00
Jun Wu
f8e3d67631 dag: add TryClone trait
Summary: Will be used as bounds for abstraction.

Reviewed By: sfilipco

Differential Revision: D24399497

fbshipit-source-id: 343be12237d4850fbde9ebbe4034469527bd77fc
2020-10-20 15:19:30 -07:00
Jun Wu
f04e0fa60e namedag: define an abstract NameDag struct
Summary: It's not yet abstract. But one step closer.

Reviewed By: sfilipco

Differential Revision: D24399510

fbshipit-source-id: 32969733babd41b221170ee440f5d7ced1f7490a
2020-10-20 15:19:30 -07:00
Jun Wu
c49caf64b4 dag: drop snapshot_map from NameDag
Summary: The `snapshot` field can be used instead.

Reviewed By: sfilipco

Differential Revision: D24399507

fbshipit-source-id: 67de20d897b8b763f724f3ccbd46618dec7911b9
2020-10-20 15:19:30 -07:00
Jun Wu
465ac6c5df dag: drop IdMapEq trait
Summary:
The trait requires an `IdMap` snapshot to be locally ready. That's not easy for
all possible implementations. Drop it to simplify things.

Reviewed By: sfilipco

Differential Revision: D24399501

fbshipit-source-id: 4d85f77c99208cda30b2a543a0bb5b295f49a65c
2020-10-20 15:19:30 -07:00
Jun Wu
5f00c7984b dag: unify prepare_filesystem_sync
Summary: There were 2 prepare_filesystem_sync. Unify them into one implementation.

Reviewed By: sfilipco

Differential Revision: D24399513

fbshipit-source-id: 80d009c33b7f23dc2c4225da6fd0fb09589ba061
2020-10-20 15:19:30 -07:00