ladybird/Documentation/HighDPI.md

HighDPI design
==============

Background
----------

- macOS: Only integer scale factors at app level. Can stretch final composited framebuffer at very end for non-integer display scales.
  - advantages: simple programming model, still fairly flexible; scaling at end produces coherent final image
  - disadvantages: needs 4x memory even for 1.5x scale; scaling at very makes final image less detailed than it could be

- Android: Many (but discrete) scale levels (ldpi (0.75x), mdpi, hdpi (1.5x), xhdpi (2x), xxhdpi (3x), xxhdpi (4x))

- Windows: has "not recommended" free form text entry for scale factor between 100% and 500%.

Integer scale factors are needed in any case so let's get that working first. Actually, let's focus on just 2x for now.


Desired end state
-----------------

- All rects  (Window and Widget rects, mouse cursor) are in "logical" coordinates, which is the same as pixels at 1x scale, as much as possible.
- If something needs to be in pixels, its name starts with `physical_`. Physical coordinates should as much as possible not cross API boundaries.
- Jury's still out if logical coordinates should stay ints. Probably, but it means mouse cursor etc only have point resolution, not pixel resolution
- We should have something that can store a collection of (lazily-loaded?) bitmaps and fonts that each represent a single image / font at different scale levels, and at paint time the right representation is picked for the current scale

Implementation plan
-------------------

The plan is to have all applications use highdpi backbuffers eventually. It'll take some time to get there though, so here's a plan for getting there incrementally.

0. Add some scaling support to Painter. Make it do 2x nearest neighbor scaling of everything at paint time for now.
1. Add scale factor concept to WindowServer. WindowServer has a scaled framebuffer/backbuffer. All other bitmaps (both other bitmaps in WindowServer, as well as everything WindowServer-client-side) are always stored at 1x and scaled up when they're painted to the framebuffer. Things will look fine at 2x, but pixely (but window gradients will be smooth already).
2. Let DisplaySettings toggle it WindowServer scale. Now it's possible to switch to and from HighDPI dynamically, using UI.
3. Come up with a system to have scale-dependent bitmap and font resources. Use that to use a high-res cursor bitmaps and high-res menu bar text painting in window server. Menu text and cursor will look less pixely. (And window frames too, I suppose.)
4. Let apps opt in to high-res window framebuffers, and convert all apps one-by-one
5. Remove high-res window framebuffer opt-in since all apps have it now.

We're currently before point 3.
Docs: Add design doc and implementation plan for highdpi I have a local branch that gets us past implementation stage 1 in this doc, and it seems like a useful enough checkpoint to upstream it and then iterate in tree. 2021-01-12 22:41:36 +03:00			`HighDPI design`
			`==============`

			`Background`
			`----------`

			`- macOS: Only integer scale factors at app level. Can stretch final composited framebuffer at very end for non-integer display scales.`
			`- advantages: simple programming model, still fairly flexible; scaling at end produces coherent final image`
			`- disadvantages: needs 4x memory even for 1.5x scale; scaling at very makes final image less detailed than it could be`

			`- Android: Many (but discrete) scale levels (ldpi (0.75x), mdpi, hdpi (1.5x), xhdpi (2x), xxhdpi (3x), xxhdpi (4x))`

			`- Windows: has "not recommended" free form text entry for scale factor between 100% and 500%.`

			`Integer scale factors are needed in any case so let's get that working first. Actually, let's focus on just 2x for now.`


			`Desired end state`
			`-----------------`

WindowServer: Make HighDPI aware Almost all logic stays in "logical" (unscaled coordinates), which means the patch is small and things like DnD, window moving and resizing, menu handling, menuapplets, etc all work without changes. Screen knows about phyiscal coordinates and mouse handling internally is in physical coordinates (so that two 1 pixel movements in succession can translate to one 1 logical coordinate mouse movement -- only a single event is sent in this case, on the 2nd moved pixel). Compositor also knows about physical pixels for its backbuffers. This is a temporary state -- in a follow-up, I'll try to let Bitmaps know about their intrinsic scale, then Compositor won't have to know about pixels any longer. Most of Compositor's logic stays in view units, just blitting to and from back buffers and the cursor save buffer has to be done in pixels. The back buffer Painter gets a scale applied which transparently handles all drawing. (But since the backbuffer and cursor save buffer are also HighDPI, they currently need to be drawn using a hack temporary unscaled Painter object. This will also go away once Bitmaps know about their intrinsic scale.) With this, editing WindowServer.ini to say Width=800 Height=600 ScaleFactor=2 and booting brings up a fully-functional HighDPI UI. (Except for minimizing windows, which will crash the window server until #4932 is merged. And I didn't test the window switcher since the win-tab shortcut doesn't work on my system.) It's all pixel-scaled, but it looks pretty decent :^) 2021-01-15 18:58:55 +03:00			`- All rects (Window and Widget rects, mouse cursor) are in "logical" coordinates, which is the same as pixels at 1x scale, as much as possible.`
			- If something needs to be in pixels, its name starts with `physical_`. Physical coordinates should as much as possible not cross API boundaries.
Docs: Add design doc and implementation plan for highdpi I have a local branch that gets us past implementation stage 1 in this doc, and it seems like a useful enough checkpoint to upstream it and then iterate in tree. 2021-01-12 22:41:36 +03:00			`- Jury's still out if logical coordinates should stay ints. Probably, but it means mouse cursor etc only have point resolution, not pixel resolution`
			`- We should have something that can store a collection of (lazily-loaded?) bitmaps and fonts that each represent a single image / font at different scale levels, and at paint time the right representation is picked for the current scale`

			`Implementation plan`
			`-------------------`

			`The plan is to have all applications use highdpi backbuffers eventually. It'll take some time to get there though, so here's a plan for getting there incrementally.`

			`0. Add some scaling support to Painter. Make it do 2x nearest neighbor scaling of everything at paint time for now.`
WindowServer: Make HighDPI aware Almost all logic stays in "logical" (unscaled coordinates), which means the patch is small and things like DnD, window moving and resizing, menu handling, menuapplets, etc all work without changes. Screen knows about phyiscal coordinates and mouse handling internally is in physical coordinates (so that two 1 pixel movements in succession can translate to one 1 logical coordinate mouse movement -- only a single event is sent in this case, on the 2nd moved pixel). Compositor also knows about physical pixels for its backbuffers. This is a temporary state -- in a follow-up, I'll try to let Bitmaps know about their intrinsic scale, then Compositor won't have to know about pixels any longer. Most of Compositor's logic stays in view units, just blitting to and from back buffers and the cursor save buffer has to be done in pixels. The back buffer Painter gets a scale applied which transparently handles all drawing. (But since the backbuffer and cursor save buffer are also HighDPI, they currently need to be drawn using a hack temporary unscaled Painter object. This will also go away once Bitmaps know about their intrinsic scale.) With this, editing WindowServer.ini to say Width=800 Height=600 ScaleFactor=2 and booting brings up a fully-functional HighDPI UI. (Except for minimizing windows, which will crash the window server until #4932 is merged. And I didn't test the window switcher since the win-tab shortcut doesn't work on my system.) It's all pixel-scaled, but it looks pretty decent :^) 2021-01-15 18:58:55 +03:00			`1. Add scale factor concept to WindowServer. WindowServer has a scaled framebuffer/backbuffer. All other bitmaps (both other bitmaps in WindowServer, as well as everything WindowServer-client-side) are always stored at 1x and scaled up when they're painted to the framebuffer. Things will look fine at 2x, but pixely (but window gradients will be smooth already).`
			`2. Let DisplaySettings toggle it WindowServer scale. Now it's possible to switch to and from HighDPI dynamically, using UI.`
DisplaySettings: Add UI for switching the scale factor For now, only support 1x and 2x scale. I tried doing something "smarter" first where the UI would try to keep the physical resolution constant when toggling between 1x and 2x, but many of the smaller 1x resolutions map to 2x logical resolutions that Compositor rejects (e.g. 1024x768 becomes 512x384, which is less than the minimum 640x480 that Compositor wants) and it felt complicated and overly magical. So this instead just gives you a 1x/2x toggle and a dropdown with logical (!) resolutions. That is, 800x600 @ 2x gives you a physical resolution of 1600x1200. If we don't like this after trying it for a while, we can change the UI then. 2021-01-16 03:02:17 +03:00			`3. Come up with a system to have scale-dependent bitmap and font resources. Use that to use a high-res cursor bitmaps and high-res menu bar text painting in window server. Menu text and cursor will look less pixely. (And window frames too, I suppose.)`
			`4. Let apps opt in to high-res window framebuffers, and convert all apps one-by-one`
			`5. Remove high-res window framebuffer opt-in since all apps have it now.`
Docs: Add design doc and implementation plan for highdpi I have a local branch that gets us past implementation stage 1 in this doc, and it seems like a useful enough checkpoint to upstream it and then iterate in tree. 2021-01-12 22:41:36 +03:00
DisplaySettings: Add UI for switching the scale factor For now, only support 1x and 2x scale. I tried doing something "smarter" first where the UI would try to keep the physical resolution constant when toggling between 1x and 2x, but many of the smaller 1x resolutions map to 2x logical resolutions that Compositor rejects (e.g. 1024x768 becomes 512x384, which is less than the minimum 640x480 that Compositor wants) and it felt complicated and overly magical. So this instead just gives you a 1x/2x toggle and a dropdown with logical (!) resolutions. That is, 800x600 @ 2x gives you a physical resolution of 1600x1200. If we don't like this after trying it for a while, we can change the UI then. 2021-01-16 03:02:17 +03:00			`We're currently before point 3.`