…_minimally_ support the new contract.
Some of the edge cases here aren't fixed, but this is good enough for a type of grammar whose days are numbered anyway.
…but only while we can afford to spend time on re-parsing.
Synchronous indentation hinting is clearly the best choice, but it can theoretically be very costly. So we'll set a time budget per transaction — currently 10ms, but could be adjusted up or down. We'll start out doing synchronous indentation, but flip to async indentation if we run out of time in a given transaction. At the end of a transaction, the time budget resets.
This allows us to balance indentation accuracy with editor responsiveness, and would even allow us to expose this tradeoff as a setting in the future. The current threshold, 10ms, would probably result in one dropped frame if exceeded, but not two.
Right now, any one parse can exceed the budget — because we don't set a timeout on the parse the way we do with an async parse. But this could be changed in the future. The main goal here is to prevent a catastrophic scenario where a complex transaction locks up the editor.
* A new predicate called `fold.invalidateOnChange` that can be used when a change should automatically invalidate the fold cache for each row in a node's range.
* The ability to use custom predicates in `folds.scm` files. (Previously, captures from `folds.scm` did not consult an instance of `ScopeResolver` in the processing phase.)
…when a project-specific config is present.
Most people don't use a project-specific config, which is why this bug has been present for ages. Read the new spec for an explanation.
…by namespacing them as `@_IGNORE_.foo`, `@_IGNORE_.bar`, etc.
It's sometimes necessary to define a capture in a query not because you want to apply a scope name, but because you need to use it as the argument to a predicate. `@_IGNORE_` was intended for that purpose, but it was the _only_ capture name with that special effect.
Now, you can specify any number of captures that don't apply scope names. As long as it equals `@_IGNORE_` or _starts with_ `@_IGNORE_.`, it'll have the same effect. This lets you target two or more nodes and use them all in predicates in the same query without any of them applying a scope name.
…to each PHP section in order to match the functionality of the TextMate grammar.
This was a gigantic endeavor and involved doing several things that were on my “ugh, I’ll get around to it” list:
* To make some of this stuff work, we were using two different PHP parser layers. The root one did the syntax highlighting and separated the HTML from the rest, and the deeper injection was there just so we could scope some ranges as `source.php`.
That doesn’t work because of rules we have about shallower layers acting before deeper layers when iterating through syntax highlighting boundaries. I was trying to be too clever. Now the _root_ layer is the one that does no highlighting, and the _deeper_ layer is the one that exists to apply `source.php` _and then_ perform highlighting and indentation hinting and folding and whatnot.
If we need to, we can move some of the queries to the root layer for whatever reason; maybe we come across a bug six months from now that could be fixed if we made the root layer in charge of folding. We have options.
* All boundaries for `HighlightIterator`s now _either_ open one or more scopes _or_ close one or more scopes. If one boundary can close some scopes and open others, then boundaries on different layers that share a buffer position cannot possibly close/open in the correct order because they can't intermingle. This needed to happen and this was as good of an excuse as any to do it.
* We needed to present sane PHP opening/closing ranges to the injection. Ironically, `tree-sitter-php` itself makes this nearly impossible because of how the tree is structured; it tries to hide the `<?php` and `?>` tags from the PHP layer and treat them as text, despite the fact that PHP is the thing that knows how to parse that.
The best way to do this — at least until I encounter a reason why it can’t work — is to find all the `<?php`s and `?>`s, sort them, group them, and build fake node ranges. This is fine! I mean, it’s ridiculous, but it’s fine! The ranges that we hand off to another parser are allowed to be completely arbitrary!
This lets us do what we were only approximating before: have _one_ PHP injection layer with _any number of_ content ranges, each of which begins with `<?php` and ends with `?>` without trying to claim any surrounding whitespace. This would be worth it even if we didn’t have to do any of this other stuff.
* Each content range of the injection needs _either_ `meta.embedded.line.php` _or_ `meta.embedded.block.php` depending on whether the range begins and ends on the same line. This is not something I was willing to regress on because it's _hard_ to distinguish between PHP and non-PHP unless your editor helps you out, and because I wasn't going to go into a bunch of themes and tell them to treat `source.php` like `meta.embedded`.
This also meant that we needed to be able to add _multiple_ “root” scope names per content range. But we already had a mode in which the `languageScope` injection option could be a callback, so it wasn't hard to make it a callback that could take extra parameters for the buffer and range.
This isn't a feature that I'm eager for other grammars to use, or even necessarily know about, but it was what we needed to deliver feature parity here. And none of it would have been necessary if `tree-sitter-php` made more sensible choices. (Whatever. Maybe what I want isn't possible for some strange reason.)
All existing tests pass. Tests have been written for the new `languageScope` features. Tests need to be written for PHP just like for all other language grammars, but that was on the to-do list anyway. If any of this turns out to be catastrophic, it’s easy to roll back, but tests are passing on my machine and I expect them to pass in CI.
This fixes a subtle breaking change in the WASM tree-sitter modes that
affected the semanticolor package. (And probably only that package.)
Although the node text is unused in `idForScope` in the core editor, the
legacy tree-sitter mode was specifically modified to pass the node text
into `idForScope` so that the semanticolor package could make use of it
to give custom scopes (and thus colors) to different symbols. (On
activation, semanticolor monkey patches that method in relevant
grammars.)
This alone is not enough to get sematicolor working with Pulsar's new
WASM tree-sitter modes, but it's one piece of the puzzle.
Ref: 97b905a2b0
Ref: https://github.com/atom/atom/pull/20212
Ref: 1bf7fc4025/src/tree-sitter-language-mode.js (L1321-L1324)
Ref: d219ef9a1f/lib/semanticolor-grammar.js (L44-L65)
…and create `useLegacyTreeSitter` for those who want to opt into the previous default behavior.
(Legacy Tree-sitter grammars will soon be removed, but this is a step toward that future!)
Previously, we threw an error when a scope adjustment violated its bounds constraints, but that's a bit disruptive for everyday use. Instead, we throw an error in dev mode (so that the grammar's author doesn't fail to notice the problem), but downgrade it to a warning outside of dev mode so that it's recoverable.
There's a chance that the warning will be _too_ subtle, but we'll give it a shot.
We also include more diagnostic information so that it's clearer exactly _where_ the violation is happening.
