…no matter how deeply they’re nested.
Also highlight `?` in an incomplete ternary.
(Also fix a lot of final segments in the TypeScript highlights file; I’ve been lazily copying-and-pasting and forgetting to change them.)
We are crossing the Rubicon.
https://github.com/orgs/pulsar-edit/discussions/249 is a discussion of specific ways that indentation is different in JavaScript because of the new grammar. We intentionally didn’t try to preserve the indentation decisions in the old TextMate grammar, but this is (predictably) disruptive to some people. We don’t want to punt like legacy Tree-sitter did and rely on the TextMate grammar’s coarse regexes to decide what gets indented, but we can at least make the new indentation system granularly configurable.
This commit makes practically every indentation decision configurable in JavaScript and TypeScript. It groups these settings under a new “Indentation” section in each package’s settings.
I might decide that this goes too far and roll back the granular settings for `{`, `[`, and `(`, but the other indentation hints are absolutely matters of taste, so it’s only proper that users can choose to disable them.
In the long run, we could _consider_ allowing users to swap in another indentation engine altogther, much like #895 proposes for folds. That would let us legitimately offer users a way to opt into “the way the TextMate grammar does it” without affecting other users.
Another stopgap option would be to add a way for a user to disable indentation hinting altogether for a given grammar.
…when a project-specific config is present.
Most people don't use a project-specific config, which is why this bug has been present for ages. Read the new spec for an explanation.
…by namespacing them as `@_IGNORE_.foo`, `@_IGNORE_.bar`, etc.
It's sometimes necessary to define a capture in a query not because you want to apply a scope name, but because you need to use it as the argument to a predicate. `@_IGNORE_` was intended for that purpose, but it was the _only_ capture name with that special effect.
Now, you can specify any number of captures that don't apply scope names. As long as it equals `@_IGNORE_` or _starts with_ `@_IGNORE_.`, it'll have the same effect. This lets you target two or more nodes and use them all in predicates in the same query without any of them applying a scope name.
There's a hard-to-grok setting for language injections that allows a deeper layer to monopolize the scope application for a range. In most cases, an injection is placed into a node that the parser doesn't know much about (like a `script` block in HTML); but Rust and C parsers needed a way to inject themselves _into themselves_ so that they could add syntax highlighting to macros. Because they were applying highlighting to a range that the base layer _already_ had plans to highlight, they needed a way to block the shallower layer from acting.
This mode has never worked briliantly, but it's been made smarter in several ways since the invention of modern Tree-sitter. And here's another one: if the highlight iterator is at a position where an injection range is _about_ to begin, it shouldn't be able to stop any other layer from _closing_ a scope; and if the highlight iterator is at a position where an injection range has just _finished_, it shouldn't be able to stop any other layer from _opening_ a scope.
Because of this, we can now fix a bug that I think might've been present for a while in the application of scopes to rust macros like `println!` — the position after the exclamation point is one of those injection-layer boundaries, to the effect that a scope name was opened that would persist until at least the end of the screen line.
…which locked its return values to integers.
The other language modes allow this method to return a non-integer, hence so should we. If this breaks something, it can be addressed some other way.
* Revert JSDoc tags to use the `keyword` namespace
* Ensure we always apply `meta.tag.jsx.js` to the area surrounding a JSX opening/closing/self-closing tag element and its punctuation.
* Declared namespaces should be `entity` only, not `support`
* `<` and `>` in type parameters should always have the proper `punctuation` scopes applied
* All TSX tags should have the surrounding `meta.tag.ts.tsx` scope applied
* TSX fragments (`<>`) should be highlighted
…when inside of a TS block within a TSX interpolation.
Some mixing of TS and TSX is too ambiguous for the “Toggle Line Comments” command, but in this case we've got a brace-delimited block inside of a TSX interpolation. That's practically guaranteed to mean we've got a line with _only_ TypeScript on it, so we should use an ordinary TS line comment.
to `markup.horizontal-rule.gfm` and add styles to all eight built-in syntax themes to account for it.
All the old GFM scopes are just named however they needed to be named in order to produce syntax highlighting to the grammar author, semantics be damned. Instead, we're making sure the semantics are correct, then adapting the syntax themes accordingly.
…in which all _value types_ are scoped under `support.storage`. The goal is to avoid drawing an arbitrary distinction between, say, `int` and `SomeCustomCType` — those things are different, but not _so_ different that they belong in separate root scope namespaces.
`storage.type` can be used for core language constructs — `var`, `class`, `function`, `struct`, `enum`, `namespace`, and so on. `support.storage.type` should be used for any _recognized_ value type — certainly core language types like `int`, but also other things if they make sense. `support.other.storage.type` can be used for user-defined types, or any other types that we don't recognize. (Using the `support.other` ordering here adheres to our new convention of reserving all of `support.other` for user-defined things.)
The point of this compromise is that both the `syntax--storage` and `syntax--support` class names are present so that many themes will still highlight these tokens as though they were in the root `storage` namespace.
This commit changes the C/C++ and TypeScript `highlights.scm` files. The results appear to be satisfactory to my eye after testing on two of the builtin themes.
I’ve updated the taxonomy document to reflect this new proposal:
https://gist.github.com/savetheclocktower/c9607b97477d4817911e4f2f8db89679#file-scope-taxonomy-md
…to each PHP section in order to match the functionality of the TextMate grammar.
This was a gigantic endeavor and involved doing several things that were on my “ugh, I’ll get around to it” list:
* To make some of this stuff work, we were using two different PHP parser layers. The root one did the syntax highlighting and separated the HTML from the rest, and the deeper injection was there just so we could scope some ranges as `source.php`.
That doesn’t work because of rules we have about shallower layers acting before deeper layers when iterating through syntax highlighting boundaries. I was trying to be too clever. Now the _root_ layer is the one that does no highlighting, and the _deeper_ layer is the one that exists to apply `source.php` _and then_ perform highlighting and indentation hinting and folding and whatnot.
If we need to, we can move some of the queries to the root layer for whatever reason; maybe we come across a bug six months from now that could be fixed if we made the root layer in charge of folding. We have options.
* All boundaries for `HighlightIterator`s now _either_ open one or more scopes _or_ close one or more scopes. If one boundary can close some scopes and open others, then boundaries on different layers that share a buffer position cannot possibly close/open in the correct order because they can't intermingle. This needed to happen and this was as good of an excuse as any to do it.
* We needed to present sane PHP opening/closing ranges to the injection. Ironically, `tree-sitter-php` itself makes this nearly impossible because of how the tree is structured; it tries to hide the `<?php` and `?>` tags from the PHP layer and treat them as text, despite the fact that PHP is the thing that knows how to parse that.
The best way to do this — at least until I encounter a reason why it can’t work — is to find all the `<?php`s and `?>`s, sort them, group them, and build fake node ranges. This is fine! I mean, it’s ridiculous, but it’s fine! The ranges that we hand off to another parser are allowed to be completely arbitrary!
This lets us do what we were only approximating before: have _one_ PHP injection layer with _any number of_ content ranges, each of which begins with `<?php` and ends with `?>` without trying to claim any surrounding whitespace. This would be worth it even if we didn’t have to do any of this other stuff.
* Each content range of the injection needs _either_ `meta.embedded.line.php` _or_ `meta.embedded.block.php` depending on whether the range begins and ends on the same line. This is not something I was willing to regress on because it's _hard_ to distinguish between PHP and non-PHP unless your editor helps you out, and because I wasn't going to go into a bunch of themes and tell them to treat `source.php` like `meta.embedded`.
This also meant that we needed to be able to add _multiple_ “root” scope names per content range. But we already had a mode in which the `languageScope` injection option could be a callback, so it wasn't hard to make it a callback that could take extra parameters for the buffer and range.
This isn't a feature that I'm eager for other grammars to use, or even necessarily know about, but it was what we needed to deliver feature parity here. And none of it would have been necessary if `tree-sitter-php` made more sensible choices. (Whatever. Maybe what I want isn't possible for some strange reason.)
All existing tests pass. Tests have been written for the new `languageScope` features. Tests need to be written for PHP just like for all other language grammars, but that was on the to-do list anyway. If any of this turns out to be catastrophic, it’s easy to roll back, but tests are passing on my machine and I expect them to pass in CI.
* Properly highlight comments like `/***/`
* Add `intdiv` as a builtin function
* Change PHPDoc tags back to a `keyword` scope rather than an `entity.name.tag` scope
* Highlight more PHPDoc types
…which will make EJS and ERB much more pleasant to use, and which I should've done ages ago.
Most notably: empty tags like `<% %>` are now valid, so the highlighting won't go nuts before you've typed anything.
…and add a distinct copy for TypeScript so that it works even if `language-javascript` is disabled or replaced.
(We might want to just make `language-regex` and have it export a service.)
