The intent is to reveal more context on what's happening in
https://github.com/wez/wezterm/issues/671
As a nice side benefit, this avoids the potential inability
to open paths that are not utf8 or representable as c-strings
on Windows.
And on top of that: this enables memory mapped file IO as well,
which wasn't enabled previously. This should help to reduce
extraneous copies of the font in memory, have fewer open files
and minimize the chances of racing with O_CLOEXEC.
When we process the system fallback list, we can produce a long list
of fonts to be speculatively processed by the shaper.
Until this commit, the shaper would always keep the associated
freetype face open forever, which increases the number of open
files and the amount of allocated memory.
This commit allows the shaper to release a font if it has never
produced any valid shaper results, which keeps the list down
to just the fonts that are in use.
This commit allows the x11 window implementation to detect changes
in the DPI that occur after a window is created.
These can occur when changing desktop resolution or when changing
the accessibility option for "Large Text" in gnome.
In order to avoid continually polling for the value on every resize,
we look for the `_GTK_EDGE_CONSTRAINTS` atom in our property change
notifications. This seems to be sent at least as often as the
dpi/scaling changes.
It's also worth noting that some dpi changes don't generate resize
events, so we can't just read the dpi value on every resize, because
we'd miss some of those changes.
Part of this commit changes the font scaling logic: previously
we'd keep a notion of "dpi scale" to apply. That dates from an
earlier time in wezterm where we didn't think that we knew an
actual dpi value.
The way that worked was that we'd compare our current guestimate
of the DPI against what we though the baseline OS dpi should be to
produce a scaling factor.
On X11 that dpi value is global and we'd effectively always produce
a revised scaling factor of 1 after we'd set up the initial window.
This commit changes that logic to just pass down the actual DPI value
to the font code. That DPI value already accounts for HiDPI scaling
so this is hopefully a NOP change for the other systems.
refs: https://github.com/wez/wezterm/issues/667
Replaces the last usage of ttf-parser with calling into freetype.
This removes a source of inconsistency, as ttf-parser doesn't support
all of the things that freetype does.
Notably, this prevents a weird error from blowing up codepoint coverage
calculations on a system where I have helvetica.bdf in my font dir for
long-forgotten reasons.
```
WEZTERM_LOG=wezterm_font=trace,info wezterm
```
will list fonts found in font_dirs and the builtin fonts.
It can't show all fonts found via the system locator, because
that only has an interface for finding fonts by matching name,
not listing all of them.
With this configuration:
```lua
local wezterm = require 'wezterm'
return {
font_dirs = {"/Users/wez/Downloads/Inconsolata"},
font = wezterm.font("Inconsolata"),
font_locator = "ConfigDirsOnly"
}
```
wezterm is now able to see the 74 variations that are available
in the single inconsolata ttf.
Running `WEZTERM_LOG=wezterm_font=trace wezterm` will log the
variations.
refs: https://github.com/wez/wezterm/issues/655
If shaping can't resolve some glyphs, queue the font locator
fallback resolution to another thread; meanwhile, a last resort
glyph is used.
That thread can trigger an invalidation once the fallback resolve
is complete, the window is invalidated and the last resort glyph
is replaced by the resolve glyph.
refs: https://github.com/wez/wezterm/issues/559
refs: https://github.com/wez/wezterm/issues/508
Keep track of the glyphs we've already advised about (until the
config is reloaded) so that we don't keep spamming the user.
refs: https://github.com/wez/wezterm/issues/559
* Check built-in fonts before asking the system for codepoint coverage
* If one of the earlier stages resolved some fonts, skip the remaining
stages and speculatively shape what we have. This avoids triggering
the system font lookup for fonts that are present in the font_dirs
or that are built-in, such as powerline symbols.
refs: https://github.com/wez/wezterm/issues/559
The dwrote crate offers functions that can extract the underlying
font file name(s) from the system, so let's use those to get
OnDisk font handles and save some memory.
refs: https://github.com/wez/wezterm/issues/559
In an earlier incarnation of both wezterm and freetype, FT_New_Face
would lead file descriptors into child processes because it didn't
set O_CLOEXEC. That led to the slightly pessimistic approach of
loading the font into memory for the lifetime of the wezterm process.
With the improved fallback handling on macos, this can result in
hundreds of MB of font data being loaded, in some cases multiple
times.
Since those days, freetype now sets O_CLOEXEC and wezterm has some
logic to close other random fds, so the descriptor leaking problem
is gone and we can now let freetype manage a file handle instead
of a memory-baked font.
This reduces the memory utilization by at least 1GB in the case
that a glyphs need to be resolved from the system fallback fonts.
refs: https://github.com/wez/wezterm/issues/559
terminus-bold.otb reports 0 height!
Detect and force that case to go through the bitmap strike loading
code path.
Improve the size selection heuristic for bitmap strikes: previously,
we would just pick the largest bitmap and allow it to be scaled down,
which was OK for emoji fonts that just had 128px square glyphs, but
is not ok for pre-rendered pixel strikes like terminus.
Note that IncreaseFontSize works in terms of percentages only,
so using a font like this may have "gaps" when ctrl-+ or - to change
the font size.
refs: https://github.com/wez/wezterm/issues/560
This is a bit unfortunate, but necessary, because the system fallback
list contains a handful of special fonts that apple doesn't ship on
disk in ttf/otf files.
One of those is `.AppleSymbolsFB` which would normally satisfy
the symbol lookup.
This commit hard codes the "Apple Symbols" font to use instead,
which is a disk based font. I don't know what the difference
is between it and `.AppleSymbolsFB`, but this is sufficient
to satisfy the glyph in question from the referenced issue.
refs: https://github.com/wez/wezterm/issues/506
Continuing along the same lines as the prior commit, the goal
of this commit is to remove the buffer transformation that was
part of uploading the texture to the GPU provided surface.
In order to do so:
* The sense of our local textures needs to change from bgra32 to rgba32.
bgra32 was a hangover from earlier versions of our window crate that
allowed direct-to-fb writes in software mode. We had to pick bgra32
for that for the broadest OS compatibility. I believe that that
constraint has been totally removed, although there is a chance that
this will flip the colors on macos.
* There was an additional linear-to-srgb conversion inlined in that
buffer transformation. I have no idea where that is needed because
the source data is carefully constructed as SRGB. I don't yet know
how to signal that, but for now I've moved that gamma correction
into the shader when we sample the texture.
With this change, timg playback now has vtparse as the hottest
region of code.
refs: #537
This commit expands on the prior commits to introduce the concept
of per-window configuration overrides.
Each TermWindow maintains json compatible object value holding
a map of config key -> config value overrides.
When the window notices that the config has changed, the config
file is loaded, the CLI overrides (if any) are applied, and then
finally the per-window overrides, before attempting to coerce
the resultant lua value into a Config object.
This mechanism has some important constraints:
* Only data can be assigned to the overrides. Closures or special
lua userdata object handles are not permitted. This is because
the lifetime of those objects is tied to the lua context in which
they were parsed, which doesn't really exist in the context of
the window.
* Only simple keys are supported for the per-window overrides.
That means that trying to override a very specific field of
a deeply structured value (eg: something like `font_rules[1].italic = false`
isn't able to be expressed in this scheme. Instead, you would
need to assign the entire `font_rules` key. I don't anticipate
this being a common desire at this time; if more advance manipulations
are required, then I have some thoughts on an event where arbitrary
lua modifications can be applied.
The implementation details are fairly straight-forward, but in testing
the two examplary use cases I noticed that some hangovers from
supporting overrides for a couple of font related options meant that the
window-specific config wasn't being honored. I've removed the code that
handled those overrides in favor of the newer more general CLI option
override support, and threaded the config through to the font code.
closes: #469closes: #329
for fonts located by core text, if they are singular TTF files rather
than TTC files, and the font doesn't exactly match the search critiera,
we could loop forever re-parsing the same file over and over again.
This commit restructures the logic to definitively check the number
of contained font entries and constrain the loop to that number.
In addition, it adjusts the name matching, as macos can return
names like "Cascadia Code Roman" when the underlying font is named
"Cascadia Code".
refs: https://github.com/wez/wezterm/issues/475
Connect the gui to the new shaping logic; this means that we
can now correctly render fg/bg color when the cursor moves
through the cells that comprise a ligature.
refs: https://github.com/wez/wezterm/issues/478
This function is intended to deal with certain kinds of ligatures
and certain combining sequences that don't have corresponding glyphs.
It isn't hooked up to the gui yet, but does have unit tests that
are probably mostly correct.
refs: https://github.com/wez/wezterm/issues/478
Refine the colorization logic to make it more of a blending operation.
Previously, we were relying on opengl to carry out blending between
layers on our behalf, but that wasn't perfect.
This commit is inspired by this post:
https://www.puredevsoftware.com/blog/2019/01/22/sub-pixel-gamma-correct-font-rendering/
and factors in the background color when computing the colorized
glyph.
This appears to reduce the dark fringes/edges that we were seeing
before, without noticeably changing the brightness of the result.
refs: #491
https://learnopengl.com/Advanced-Lighting/Gamma-Correction suggests
some good practices:
* Only enable SRGB output on the final draw call, so that all prior
stages can operate on linear values and avoid converting to/from
linear multiple times.
* The SRGBA textures automatically linearize when sampled, but:
* The RGB data must be SRGB (non-linear)
* The A channel is assumed to be linear!
This commit nudges us closer to that by:
* Converting the freetype coverage map from its linear value to
non-linear when rasterizing.
* Splitting the shader files into one per stage (background, lines,
glyphs) and only setting outputs_srgb for the glyph stage
refs: #491
In the earlier times wezterm supported different font rasterizers,
and the configuration was a bit vague and generic to accomodate
differences in how the rasterizers worked.
Since then, we've standardized on freetype.
One of the things that's been bothering me for a while is that
we have some fiddly logic to transform from the config to the freetype
flags.
This commit does away with the transformation and simply exposes
the two sets of freetype options.
The main thing that I expect people to play with is
`freetype_load_target` which can have one of the following values:
```
pub enum FreeTypeLoadTarget {
/// This corresponds to the default hinting algorithm, optimized
for standard gray-level rendering.
Normal,
/// A lighter hinting algorithm for non-monochrome modes. Many
generated glyphs are more fuzzy but better resemble its original
shape. A bit like rendering on Mac OS X. This target implies
FT_LOAD_FORCE_AUTOHINT.
Light,
/// Strong hinting algorithm that should only be used for
monochrome output. The result is probably unpleasant if the glyph
is rendered in non-monochrome modes.
Mono,
/// A variant of Normal optimized for horizontally decimated LCD displays.
HorizontalLcd,
/// A variant of Normal optimized for vertically decimated LCD displays.
VerticalLcd,
}
```
I expect most people will want to set this to one of `Normal`, `Light`
or `HorizontalLcd`. `HorizontalLcd` is what `font_antialias=Subpixel`
used to select.
refs: #491
If a font doesn't have any latin glyphs then we'd compute 0 as the
average width. Later, during rendering, we'd compute an `inf` scaling
factor and then subsequently fail to allocate texture space.
This commit takes the average width from a "random" selection of glyphs
(whatever the first few glyphs in the font may be) to avoid that
situation.
refs: https://github.com/wez/wezterm/issues/404
This defaults to None, taking the default from the freetype library.
You can select an integer value to tell the library to use an
alternative version.
Versions that are available in the build used by wezterm are 35, 38 and
40.
See https://freetype.org/freetype2/docs/subpixel-hinting.html for
more information.
Revise logging so that we use info level for things that we want
to always log, and adjust the logger config to always log info
level messages.
That means shifting some warning level logs down lower to debug level so
that they aren't noisy.
closes: https://github.com/wez/wezterm/issues/388
My original goal was to update to allsorts 0.5 but the API
changes are significant and not clearly described.
To make that transition easier, the prior commit moved the shaping
logic into our allsorts shaper module, leaving the name parsing
here in parser.rs.
This commit now replaces that logic with ttf_parser, which is
potentially faster (there's more emphasis on optimal code in that
crate than in allsorts) but definitely simpler.
It's not a slam-dunk transition: ttf_parser doesn't know how to
decode MacRoman encoded text, so there's a bit of logic borrowed
from allsorts here to handle that.
The root cause of this was that I'd added a fontformat=TrueType
constraint to the fontconfig pattern and since fontconfig has
fontformat=CFF for Hasklig, it wasn't the primary font candidate.
When I cut out redundant fontconfig checks in
ee1d84829a it meant that we'd never
"see" the hasklig result that turns up ~20 or so fonts into the
fallback list.
This commit removes the TrueType constraint so that the results
are ranked correctly again.
I've also switched the main font lookup path to using an alternative
font config API that returns only the best match as that more closely
aligns our intent in this function; originally, fallback was intended
to be handled in this code path, but these days it has its own separate
method.
closes: https://github.com/wez/wezterm/issues/383
refs: https://github.com/wez/wezterm/issues/379
font-config can return very long lists of fallback fonts like:
```
2020-12-16T16:23:13.306Z TRACE wezterm_font::locator::font_config > query font-config for Pattern(Operator Mono SSm Lig Medium,DejaVu Sans,PT Sans,PT Sans Caption,Bitstream Vera Sans,DejaVu Sans,Verdana,Arial,Albany AMT,Luxi Sans,Nimbus Sans L,Nimbus Sans,Nimbus Sans,Helvetica,Nimbus Sans,Lucida Sans Unicode,BPG Glaho International,Tahoma,Comfortaa,Montserrat,URW Gothic,Nimbus Sans,Nimbus Sans Narrow,Carlito,Droid Sans,Nachlieli,Lucida Sans Unicode,Yudit Unicode,Kerkis,ArmNet Helvetica,Artsounk,BPG UTF8 M,Waree,Loma,Garuda,Umpush,Saysettha Unicode,JG Lao Old Arial,GF Zemen Unicode,Pigiarniq,B Davat,B Compset,Kacst\-Qr,Urdu Nastaliq Unicode,Raghindi,Mukti Narrow,malayalam,Sampige,padmaa,Hapax Berbère,MS Gothic,UmePlus P Gothic,Microsoft YaHei,Microsoft JhengHei,WenQuanYi Zen Hei,WenQuanYi Bitmap Song,AR PL ShanHeiSun Uni,AR PL New Sung,MgOpen Modata,VL Gothic,IPAMonaGothic,IPAGothic,Sazanami Gothic,Kochi Gothic,AR PL KaitiM GB,AR PL KaitiM Big5,AR PL ShanHeiSun Uni,AR PL SungtiL GB,AR PL Mingti2L Big5,ＭＳ ゴシック,ZYSong18030,TSCu_Paranar,NanumGothic,UnDotum,Baekmuk Dotum,Baekmuk Gulim,KacstQura,Lohit Bengali,Lohit Gujarati,Lohit Hindi,Lohit Marathi,Lohit Maithili,Lohit Kashmiri,Lohit Konkani,Lohit Nepali,Lohit Sindhi,Lohit Punjabi,Lohit Tamil,Meera,Lohit Malayalam,Lohit Kannada,Lohit Telugu,Lohit Oriya,LKLUG,Mingzat,Padauk,Nuosu SIL,FreeSans,FreeSans,Arial Unicode MS,Arial Unicode,Code2000,Code2001,sans\-serif,Roya,Koodak,Terafik,sans\-serif,sans\-serif,sans\-serif,ITC Avant Garde Gothic,URW Gothic,sans\-serif,sans\-serif,Helvetica,Helvetica Narrow,Nimbus Sans Narrow,sans\-serif,sans\-serif,sans\-serif:slant=0:weight=80:spacing=100:fontformat=TrueType) took 1.344155ms
```
In the context of that particular call, we only care about whether the
first result matches what we're looking up. The fallbacks are processed
separately in a different method.
Therefore, we can skip additional processing and save a non-trivial
number of milliseconds overall parsing/re-parsing them to verify
whether they are the one we wanted to match.
refs: https://github.com/wez/wezterm/issues/379
font-config can return a long list of fallback results for a given
font family, and we parse those to see if they match; once we've
found a match there's zero chance that that errort is helpful,
so break out of the loop.
Add some more trace logging to see if that helps.
refs: https://github.com/wez/wezterm/issues/379
Teach the core text locator how to obtain the system fallback list
and add that to the fallback.
Fixup handling of ttc files on macOS; we'd always assume index 0
when extracting font info from the font descriptor. We now make
the effort to enumerate the contents of the TTC and find a match.
827d94a seems to have broken building on aarch64. The fix is pretty
much adapted from bf962c8.
I know little about rust, so I might've missed some obvious issues with
this PR - it seems to work so far, though.
This commit adds support for computing the codepoint coverage for fonts
loaded from font-dirs and the built-in, in-memory fonts.
What this means is that if you have eg: a font with chinese glyphs in
your font-dirs but not explicitly listed in your wezterm config, if
chinese text is rendered and no match from your config is found, wezterm
will be able to find the font from your font-dirs and use that
implicitly.
Computing the codepoint coverage is relatively expensive so we defer it
until we need to perform it, and cache it.
Previously, we'd enumerate the font dirs on every font resolve for
every bit of styled text.
This moves the new FontDatabase instances to be single instanced
in the FontConfiguration. The font-dirs will be scanned once
on a config reload, but the built-in in-memory fonts will only
every be enumerated once per FontConfiguration instance.
This tidies up the font-dir and built-in font management a little
bit and paves the way for codepoint -> font resolution for fonts
discovered in font-dirs.
By default, freetype doesn't include error strings and FT_Error_String
will always return NULL. Turn on the compile time option that makes
this function useful!
This commit uses a bit of DirectWrite to discover which font(s)
can be used to render a set of codepoints.
While hooking this up, I found that the method we were using
to extract the font data didn't handle TTC data so this commit
improves some parser diagnostics and handling for that.
refs: https://github.com/wez/wezterm/issues/299
Profiling showed that set_font_size was a hotspot. While there was
caching of the size info at the shaper layer, it was also needed in
the raster layer, so move it into the raster layer from the shaper
layer.
refs: #353
98f289f511 causes more metrics retrieval
than in earlier versions; each unchached glyph render would trigger
a metrics recompute for the relevant font.
Add a simple cache for this.
refs: #353
This commit makes some adjustments to FontConfiguration and LoadedFont
such that it the shaper is unable to resolve a (non-last-resort) font
for a set of codepoints, the locator can be used to try to find a
font that has coverage for those codepoints.
At the moment this is a bit limited:
* Only the font-config locator implements this function
* The directory based locator isn't actually an implementor of the
locator trait and doesn't have a way to be invoked for this.
Make an effort to explain what failed to load and where it came from,
and funnel users to the documentation on font configuration.
The message presented is slightly different depending on whether
we think that the font was their primary font, an explicitly
specified font_rule or an implicitly synthesized font_rule.
refs: #340
Use the scaling factor between the font metrics for the base font
and those of the fallback font selected for a given glyph.
The scenario is this: the base font is typically the first one selected
from the font configuration. There may be multiple fallback fonts that
are different sizes; for instance, the Font Awesome font has glyphs that
are square in aspect and are thus about twice the width of a typical
textual monospace font. Similarly, Noto Color Emoji is another square
font but that has a single set of bitmap strikes at a fixed 128 px
square.
The shaper returns advance metrics in the scale of the containing font,
and the rasterizer will target the supplied size and dpi.
We need to scale these to match the base metrics.
Previously we used a crude heuristic to decide whether to scale,
and that happened to work for Noto Color Emoji but not for Font Awesome,
whose metrics were just inside the bounds of the heuristic.
This commit allows retrieving the metrics for a given font_idx so
that we can compute the correct scale factor without any heuristics,
and applies that to the rasterized glyph.
refs: https://github.com/wez/wezterm/issues/342
Don't short circuit on just the family portion of the name;
if the criteria don't match there, we should fall back to
test against the full font name.
closes: https://github.com/wez/wezterm/issues/341
This is one of those massive time sinks that I almost regret...
As part of recent changes to dust-off the allsorts shaper, I noticed
that the harfbuzz shaper wasn't shaping as well as the allsorts one.
This commit:
* Adds emoji-test.txt, a text file you can `cat` to see how well
the emoji are shaped and rendered.
* Fixes (or at least, improves) the column width calculation for
combining sequences such as "deaf man" which was previously calculated
at 3 cells in width when it should have just been 2 cells wide, which
resulted in a weird "prismatic" effect during rendering where the
glyph would be rendered with an extra RHS portion of the glyph across
3 cells.
* Improved/simplified the clustering logic used to compute fallbacks.
Previously we could end up with some wonky/disjoint sequence of
undefined glyphs which wouldn't be successfully resolved from a
fallback font. We now make a better effort to consolidate runs of
undefined glyphs for fallback.
* For sequences such as "woman with veil: dark skin tone" that occupy a
single cell, the shaper may return 3 clusters with 3 glyphs in the
case that the font doesn't fully support this grapheme. At render
time we'd just take the last glyph from that sequence and render it,
resulting in eg: a female symbol in this particular case. It is
generally a bit more useful to show the first glyph in the sequence
(eg: person with veil) rather than the gender or skin tone, so the
renderer now checks for this kind of overlapping sequence and renders
only the first glyph from the sequence.
Bundle the *Last Resort High-Efficiency* font from
https://github.com/unicode-org/last-resort-font/
version 13.001 (Oct 22 2020).
This provides a more useful fallback glyph than we'd otherwise
produce if there is no matching glyph in any of the fonts.
Its license is OFL-1.1 which is compatible with the other
bundled fonts.
There are a number of cases where font-loader might panic on windows,
and the optional font-loader dep causes problems with `cargo vendor`
in #337, so this is a step to removing that dep.
This commit makes direct GDI calls to enumerate monospace truetype
fonts from the system and then applies our normal matching on the
result.
The current master of allsorts supports color fonts in both bitmap and
svg varieties. I'm interested to see if I can teach wezterm to render
the svg based variety in a subsequent diff.
First though, it's times to dust off our allsorts shaper logic.
This commit updates to point to the current master of allsorts at the
time of writing; there's a little bit of API fanout that makes it a bit
easier to manage font fallback.
The fallback logic has been improved so that we can now successfully
fall back to the emoji font.
The shaping logic has been improved so that we turn on the options that
enable ZWJ for combining sequences of emoji, such as "man health
worker".
Running with the allsorts shaper enabled produces generally superior
emoji/ligature substitution results compared to harfbuzz with Noto Color
Emoji; the "man health worker" and the flags (eg: `flag: England`) from
the subdivsion-flag section don't get substituted at all with harfbuzz,
but do produce appropriate glyphs with allsorts.
refs: https://github.com/wez/wezterm/issues/66
9892b16d40 adjusted how the text
colors are produced; it resulted in some ugly dark edges, especially
on lighter backgrounds.
This commit routes that tint via an alpha compositing helper which
produces smoother edges.
refs: #320
This commit more cleanly separates the load from the render flags,
and fixes up the render call; at some point this got messed up such
that we'd never end up with freetype returning subpixel format data
(LCD) and instead we'd only ever get grayscale data.
With that fixed, it's apparent that the colorization of the glyph
data was wonky in the shader so this commit also cleans this up.
refs: #320
refs: #121
This wasn't used by anything and the version was getting pretty stale.
Upgrading is awkward because newer versions pull in an incompatible
freetype library version.
This commit moves a bunch of stuff around such that `wezterm` is now a
lighter-weight executable that knows how to spawn the gui, talk to
the mux or emit some escape sequences for imgcat.
The gui portion has been moved into `wezterm-gui`, a separate executable
that doesn't know about the CLI or imgcat functionality.
Importantly, `wezterm.exe` is no longer a window subsystem executable
on windows, which makes interactions such as `wezterm -h` feel more
natural when spawned from `cmd`, and should allow
`type foo.png | wezterm imgcat` to work as expected.
That said, I've only tested this on linux so far, and there's a good
chance that something mac or windows specific is broken by this
change and will need fixing up.
refs: #301
