The comment in `src/web-tree-sitter.js` and the documentation in
`vendor/web-tree-sitter` explain why this change is needed. I hope it's
temporary.
I haven't removed our `web-tree-sitter` dependency in `package.json` because
it's a useful way of declaring (and remembering) which version of tree-sitter
we've built against, and because it lets us flip a single config flag in
`src/web-tree-sitter.js` to compare the behavior of our custom version with that
of the stock version. The cost of keeping the stock version around is under 300
kilobytes.
The custom builds will come from specialty branches on our tree-sitter fork.
When we upgrade to a new version, someone should follow the directions in
`vendor/web-tree-sitter/README.md` to create a new specialty branch.
When someone wants to build a `language-foo` package and finds that
`tree-sitter-foo` needs a function we don't yet export, they should be able to
see the warning we generate in the console and file a ticket. If we're on our
game, we should be able to generate a new web-tree-sitter build and get it into
a rolling release within a week or so.
I hope web-tree-sitter someday finds a way to fix this so that we no longer need
to do this chore. See https://github.com/tree-sitter/tree-sitter/issues/949.
Just to document this: I tried a HUGE amount of ideas for tests to run
reliably. They need to wait for the editor when it's ready to be
tokenized, but our callbacks are not that reliable - meaning, for the
first tokenization on tree-sitter, they are called _after_ we register
the callback, but for the second time, they are called _before_ we
register them. There's no reliable way to _actually listen_ to changes,
and none of these even were _called_ by TextMate grammars.
So, this `.ready` will stay here so we know when TreeSitter is ready to
tokenize.
A quick explanation on this - the old code was matching if the full
scope was ok. This is fine when we want to test a single grammar, but
for example, TextMate grammars always added `.ruby` on the last part of
the grammar. That was quite bad for TreeSitter, that basically didn't do
the same thing. So now it matches a fine crafted Regexp that basically
checks if the full scope is match from the beginning, OR if part of the
scope (up to just before the `.`) matches. So, for example, for
`constant.other.ruby` it'll match `constant`, `constant.other`, and
`constant.other.ruby` but it'll NOT MATCH `constant.oth`.
If no error message is given, show the filename and line number when a
test fails due to a timeout using waitsFor.
Co-authored-by: Max Brunsfeld <maxbrunsfeld@github.com>
