This commit is larger than it appears to due fanout from threading
through bidi parameters. The main changes are:
* When clustering cells, add an additional phase to resolve embedding
levels and further sub-divide a cluster based on the resolved bidi
runs; this is where we get the direction for a run and this needs
to be passed through to the shaper.
* When doing bidi, the forced cluster boundary hack that we use to
de-ligature when cursoring through text needs to be disabled,
otherwise the cursor appears to push/rotate the text in that
cluster when moving through it! We'll need to find a different
way to handle shading the cursor that eliminates the original
cursor/ligature/black issue.
* In the shaper, the logic for coalescing unresolved runs for font
fallback assumed LTR and needed to be adjusted to cluster RTL.
That meant also computing a little index of codepoint lengths.
* Added `experimental_bidi` boolean option that defaults to false.
When enabled, it activates the bidi processing phase in clustering
with a strong hint that the paragraph is LTR.
This implementation is incomplete and/or wrong for a number of cases:
* The config option should probably allow specifying the paragraph
direction hint to use by default.
* https://terminal-wg.pages.freedesktop.org/bidi/recommendation/paragraphs.html
recommends that bidi be applied to logical lines, not physical
lines (or really: ranges within physical lines) that we're doing
at the moment
* The paragraph direction hint should be overridden by cell attributes
and other escapes; see 85a6b178cf
and probably others.
However, as of this commit, if you `experimental_bidi=true` then
```
echo This is RTL -> عربي فارسی bidi
```
(that text was sourced from:
https://github.com/microsoft/terminal/issues/538#issuecomment-677017322)
then wezterm will display the text in the same order as the text
renders in Chrome for that github comment.
```
; ./target/debug/wezterm --config experimental_bidi=false ls-fonts --text "عربي فارسی ->"
LeftToRight
0 ع \u{639} x_adv=8 glyph=300 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
2 ر \u{631} x_adv=3.78125 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
4 ب \u{628} x_adv=4 glyph=244 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
6 ي \u{64a} x_adv=4 glyph=363 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
8 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
9 ف \u{641} x_adv=11 glyph=328 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
11 ا \u{627} x_adv=4 glyph=240 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
13 ر \u{631} x_adv=3.78125 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
15 س \u{633} x_adv=10 glyph=278 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
17 ی \u{6cc} x_adv=4 glyph=664 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
19 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
20 - \u{2d} x_adv=8 glyph=276 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
21 > \u{3e} x_adv=8 glyph=338 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
```
```
; ./target/debug/wezterm --config experimental_bidi=true ls-fonts --text "عربي فارسی ->"
RightToLeft
17 ی \u{6cc} x_adv=9 glyph=906 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
15 س \u{633} x_adv=10 glyph=277 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
13 ر \u{631} x_adv=4.78125 glyph=272 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
11 ا \u{627} x_adv=4 glyph=241 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
9 ف \u{641} x_adv=5 glyph=329 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
8 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
6 ي \u{64a} x_adv=9 glyph=904 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
4 ب \u{628} x_adv=4 glyph=243 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
2 ر \u{631} x_adv=5 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
0 ع \u{639} x_adv=6 glyph=301 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false})
/System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText
LeftToRight
0 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
1 - \u{2d} x_adv=8 glyph=480 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
2 > \u{3e} x_adv=8 glyph=470 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false})
/Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
;
```
refs: https://github.com/wez/wezterm/issues/784
We need to force the codepage to UTF8 this to avoid having our UTF-8
bytestreams be misinterpreted, and to restore the original before we're
done.
refs: https://github.com/wez/wezterm/issues/1435
implement missing alt-screen support and fixup building the examples
while we're in here.
refs: #1244
This is a fairly far-reaching commit. The idea is:
* Introduce a unicode_version config that specifies the default level
of unicode conformance for each newly created Terminal (each Pane)
* The unicode_version is passed down to the `grapheme_column_width`
function which interprets the width based on the version
* `Cell` records the width so that later calculations don't need to
know the unicode version
In a subsequent diff, I will introduce an escape sequence that allows
setting/pushing/popping the unicode version so that it can be overridden
via eg: a shell alias prior to launching an application that uses a
different version of unicode from the default.
This approach allows output from multiple applications with differing
understanding of unicode to coexist on the same screen a little more
sanely.
Note that the default `unicode_version` is set to 9, which means that
emoji presentation selectors are now by-default ignored. This was
selected to better match the level of support in widely deployed
applications.
I expect to raise that default version in the future.
Also worth noting: there are a number of callers of
`unicode_column_width` in things like overlays and lua helper functions
that pass `None` for the unicode version: these will assume the latest
known-to-wezterm/termwiz version of unicode to be desired. If those
overlays do things with emoji presentation selectors, then there may be
some alignment artifacts. That can be tackled in a follow up commit.
refs: #1231
refs: #997
Adds a use_image feature to termwiz that enables an optional
dep on the image crate. This in turn allows decoding of animation
formats (gif, apng) from file data, but more crucially, allows
modeling animation frames at the termwiz layer, which is a pre-req
for enabling kitty img protocol animation support.
refs: #986
Using a boxed slice means that we hold exactly the memory required
for the file data, rather than the next-power-of-two, which can
be wasteful when a large number of images are being sent to
the terminal.
This is a API breaking change for termwiz, so bump its version.
refs: #534
It's been replaced with an opaque termwiz error type instead.
This is a bit of a more conservative approach than that in (refs: #407)
and has less of an impact on the surrounding code, which appeals to
me from a maintenance perspective.
refs: #406
refs: #407
This allows us to support the kitty style underline sequence,
or the : separated form of the true color escape sequences.
refs: https://github.com/wez/wezterm/issues/415
This is one of those massive time sinks that I almost regret...
As part of recent changes to dust-off the allsorts shaper, I noticed
that the harfbuzz shaper wasn't shaping as well as the allsorts one.
This commit:
* Adds emoji-test.txt, a text file you can `cat` to see how well
the emoji are shaped and rendered.
* Fixes (or at least, improves) the column width calculation for
combining sequences such as "deaf man" which was previously calculated
at 3 cells in width when it should have just been 2 cells wide, which
resulted in a weird "prismatic" effect during rendering where the
glyph would be rendered with an extra RHS portion of the glyph across
3 cells.
* Improved/simplified the clustering logic used to compute fallbacks.
Previously we could end up with some wonky/disjoint sequence of
undefined glyphs which wouldn't be successfully resolved from a
fallback font. We now make a better effort to consolidate runs of
undefined glyphs for fallback.
* For sequences such as "woman with veil: dark skin tone" that occupy a
single cell, the shaper may return 3 clusters with 3 glyphs in the
case that the font doesn't fully support this grapheme. At render
time we'd just take the last glyph from that sequence and render it,
resulting in eg: a female symbol in this particular case. It is
generally a bit more useful to show the first glyph in the sequence
(eg: person with veil) rather than the gender or skin tone, so the
renderer now checks for this kind of overlapping sequence and renders
only the first glyph from the sequence.
Replaces SmallVec with an internal TeenyString that only
occupies a single machine word and avoids heap allocation
in the common case on most architectures. This takes the
textual portion of Cell from 32 bytes to 8 bytes.
This corrects an issue where the mode byte of the DCS sequence was
discarded from the DcsHook, making it impossible to know what sequence
is being activated.
So far this hasn't come up as these sequences are relatively rare,
but in looking at sixel parsing I noticed the error.
This commit changes the behavior on Windows:
* If $TERM is set and the `terminfo` crate is able to
successfully initialize and locate a terminfo database (this also
requires that $TERMINFO be set in the environment), then we'll
use the `TerminfoRenderer` instead of the `WindowsConsoleRenderer`
* If $TERM is set to `xterm-256color` and no terminfo database was
found, use our modern compiled-in copy (look in the `termwiz/data/`
directory for the source and compiled version of this) and use
the `TerminfoRenderer`.
* Otherwise use the `WindowsConsoleRenderer`.
In practice, this allows termwiz apps to opt in to features such as
true color support on Windows 10 build 1903 an later by setting their
`TERM=xterm-256color`. This happens to be the default behavior when
`ssh`ing in to a windows host via `wezterm`.
You can see the truecolor mode get applied by running this example:
```
cargo run --example widgets_basic --features widgets
```
with TERM set as above the background region that is painted by the app
will be a blueish/purplish color, but with it unset or set to something
invalid, it will fall back to black.
I'd like to eventually make termwiz assume the equivalent configuration
to `TERM=xterm-256color` by default on Windows 10 build 1903 and later,
but it's worth getting some feedback on how this works for clients such
as `streampager`.
cc: @quark-zju and @markbt
derive_builder has some extra dependencies that take a while to compile.
The builder feature can be expressed via a 30-line macro. So let's do
that to make termwiz compile faster.
The palette crate has a codegen step that translates svg_colors.txt to named.rs.
That makes it hard to build using buck.
Remove the palette dependency so termwiz is easier to build using buck.
I made sure the following code:
fn main() {
use termwiz::color::RgbColor;
let r = RgbColor::from_rgb_str("#02abcd").unwrap();
let r1 = r.to_tuple_rgba();
let r2 = r.to_linear_tuple_rgba();
println!("r1 = {:?}", r1);
println!("r2 = {:?}", r2);
}
prints
r1 = (0.007843138, 0.67058825, 0.8039216, 1.0)
r2 = (0.000607054, 0.4072403, 0.6104956, 1.0)
before and after the change.
Embed rgb.txt and parse it on the fly to produce the list of colors.
This list is a superset of palette's SVG color list.
refs: https://github.com/wez/wezterm/pull/144
