@@ -0,0 +1,279 @@
1+# Ormolu
2+
3+* [Analysis of the existing solutions](#analysis-of-the-existing-solutions)
4+ * [Brittany](#brittany)
5+ * [Hindent](#hindent)
6+ * [Stylish Haskell](#stylish-haskell)
7+ * [Haskell formatter](#haskell-formatter)
8+* [Proposed design](#proposed-design)
9+ * [Parsing and CPP](#parsing-and-cpp)
10+ * [Printing](#printing)
11+ * [Configuration](#configuration)
12+ * [Handling of language extensions](#handling-of-language-extensions)
13+ * [Testing](#testing)
14+* [Roadmap](#roadmap)
15+
16+This document describes design of a new formatter for Haskell source code.
17+It also includes some recommendations for future implementers.
18+
19+## Analysis of the existing solutions
20+
21+In order to design a new formatter we need to study the existing solutions
22+so we can borrow the good bits and avoid making the same mistakes.
23+
24+### Brittany
25+
26+[Brittany][brittany] builds on top of [`ghc-exactprint`][ghc-exactprint]—a
27+library that uses parser of GHC itself for parsing and thus it guarantees
28+that at least parsing phase is bug-free (which is admittedly the cause of
29+majority of bugs in other projects, see below).
30+
31+`ghc-exactprint` is capable of dealing not only with vanilla Haskell source
32+code, but also with CPP. However it does so by running the preprocessor with
33+parameters provided by user (such as values of `#defines`). It doesn't parse
34+actual preprocessor directives in a way that would allow their
35+reconstruction, thus Brittany still lacks support for handling source code
36+with CPP in it. Which renders it almost unusable straight away.
37+
38+After parsing, Haskell AST and a collection of annotations are available.
39+The annotations are there because Haskell AST doesn't provide enough
40+information to reconstruct source code (for example it doesn't include
41+comments). Brittany's approach then amounts to manipulation of these
42+entities using a stateful monad in which very low-level transformations are
43+described.
44+
45+After the manipulations a few functions from `ghc-exactprint` are used for
46+rendering of code together with some custom logic. I do not fully understand
47+why the function `exactPrint` from `ghc-exactprint` is not used directly.
48+
49+The code is hard to read and is buggy too, as one would expect with such
50+approach. There are enough bugs in Brittany at the moment so that it's
51+hardly usable although it's now 2 years old. Looking at the opened bugs it's
52+clear that almost all of them are because of the too-low-level approach
53+which seems to be very fragile. I'm surprised it even works at all, probably
54+the author had strong will power.
55+
56+### Hindent
57+
58+[Hindent][hindent] uses [`haskell-src-exts`][haskell-src-exts] for parsing
59+like all older projects. `haskell-src-exts` does not use parser of GHC
60+itself, and is a source of endless parsing bugs. `Hindent` is affected by
61+these upstream issues as well as Stylish Haskell and Haskell formatter (see
62+below). This already makes all these projects unusable with some valid
63+Haskell source code, but let's continue studying Hindent anyway.
64+
65+Hindent is quite different from Brittany in the sense that it does not
66+attempt to do manipulations on source code and annotations to print them
67+afterwards, instead it just prints the parsed code straight away. This means
68+that maybe 70-80% of the code does printing, the package is essentially is
69+about pretty-printing parsed Haskell code.
70+
71+Perhaps surprisingly, this approach seems to be better (IMO), for several
72+reasons:
73+
74+1. Brittany cannot guarantee that the output will be “canoncial”. With given
75+ config, if we pass it different inputs expressing identical Haskell
76+ programs modulo spacing and indentation, by design it may produce
77+ different outputs:
78+
79+ ```
80+ input0 input1 ... inputN
81+ | | |
82+ output0 output1 ... outputN
83+ ```
84+
85+ While with the approach taken by Hindent we get rather this:
86+
87+ ```
88+ input0 input1 ... inputN
89+ | | |
90+ \ | /
91+ \ | /
92+ \ | /
93+ \ | /
94+ \ | /
95+ \ | /
96+ canonical output
97+ ```
98+
99+ Because there is one way to pretty print a parsed program. This means
100+ that the users won't need to think about the layout at all because it'll
101+ be 100% determined by the pretty-printer after the transformation.
102+
103+2. The code is easier to read and debug. Pretty-printing functions are very
104+ straightforward. If there is a bug (in pretty-printer, not in parser
105+ which Hindent cannot control), it's easy to fix AFAIK.
106+
107+Hindent is also notable for its ability to handle CPP and inputs that do not
108+constitute complete modules. It splits input stream into so-called “code
109+blocks” recognizing CPP macros and then only pretty-prints “normal code”
110+without touching CPP directives. After that CPP is inserted between
111+pretty-printed blocks of source code. The approach fails when CPP breaks
112+code in such a way that separate blocks do not form valid Haskell
113+expressions, see
114+[this](https://github.com/commercialhaskell/hindent/issues/383) for example.
115+
116+Looking at the bug tracker there are many bugs. Part of them is because of
117+the use of `haskell-src-exts`, the other part is because the maintainer
118+doesn't care (anymore?) and doesn't fix them. Well it's as simple as that,
119+with any sort of commercial backing the bugs in pretty printer would be
120+fixed long time ago.
121+
122+### Stylish Haskell
123+
124+[Stylish Haskell][stylish-haskell] also uses `haskell-src-exts` and suffers
125+from the same upstream problems. I haven't studied the transformations it
126+performs, but it looks like it transforms the parsed source code partially
127+by manipulating AST and partially by manipulating raw text (e.g. to drop
128+trailing whitspace from each line). CPP Macros are just filtered out
129+silently as a preprocessing step before feeding the code to
130+`haskell-src-exts`.
131+
132+Stylish Haskell is not so invasive as the other formatters and most reported
133+bugs are about parsing issues and CPP. As I understand it, people mostly use
134+it to screw their import lists.
135+
136+### Haskell formatter
137+
138+[Haskell formatter][haskell-formatter] is an older project that didn't get
139+much traction. It also uses `haskell-src-ext` and also tries to do
140+manipulations on the parsed AST. The issue tracker doesn't have many issues
141+probably because it never got popular enough (only 15 stars on GitHub). All
142+the issues are about upstream problems with `haskell-src-exts`.
143+
144+## Proposed design
145+
146+In this section I describe a solution that combines all the good things from
147+the ones above and tries to solve the CPP problem better.
148+
149+### Parsing and CPP
150+
151+It is clear that `ghc-exactprint` is better than `haskell-src-exts`, so we
152+should use that. If we go with `ghc-exactprint` though, we'll need to
153+specify which parser to use, e.g. the parser that parses whole module or the
154+one which parsers declarations/expressions/etc. It seems that in general
155+it's better to use the parser for modules because it should work with all
156+input files containing complete modules, while with other parsers it's
157+impossible to guess what they'll be called on.
158+
159+So far so good, but CPP problem remains. We cannot ignore it as does Stylish
160+Haskell, and cannot error out on CPP as does Brittany, and Hident solves the
161+problem only partially and we cannot follow that path because
162+`ghc-exactprint` won't accept anything but a complete module, as noted
163+above.
164+
165+One workaround is to do the following:
166+
167+1. Replace CPP with comments containing first some magic word and then the
168+ actual CPP directive.
169+2. Run GHC parser on that as usual. That will succeed and preserve the
170+ comments in annotations.
171+3. When printing, detect these special comments and output them as CPP
172+ directives again.
173+
174+The approach above may present some difficulties though:
175+
176+* When conditional CPP directives are turned into comments it's likely that
177+ we'll end up with invalid Haskell if code from several conditional
178+ branches is present at the same time.
179+* Restoring CPP properly and inserting it back may be triciier than we
180+ think.
181+
182+### Printing
183+
184+Just pretty-printing code (following the approach of Hindent) seems sane. It
185+is straightforward and when complemented with enough tests (see the section
186+about testing below), it should work all right.
187+
188+Implementation can be just a `Reader` monad producing something like text
189+builder. The context of `Reader` can store current indentation and
190+configuration options.
191+
192+Returning to `ghc-exactprint`, if we'll be printing the AST ourselves it
193+makes sense to depend on the [`ghc`][ghc] package directly.
194+
195+### Configuration
196+
197+There should be some configuration but at this point I'll leave this section
198+empty. Personally I'm OK with imposing “one good style” and only allowing
199+users to tweak things like page width in columns and indentation levels.
200+Others will probably disagree. I generally like the philosophy of [this
201+post][hindent-5-blog] by Chris Done (the author of Hindent) which says that
202+as long as the default style is conventional and good it doesn't really
203+matter how code gets formatted. Consistency is more important.
204+
205+### Handling of language extensions
206+
207+Some language extensions affect how parsing is done. We are going to deal
208+with those in two ways:
209+
210+* When language pragmas are present in source file, we must parse them
211+ before we run the main parser (I guess) and they should determine how the
212+ main parsing will be done.
213+* There also should be configuration file that may enable other language
214+ extensions to be used on all files.
215+* Later we could try to locate Cabal files and fetch the list of extensions
216+ that are enabled by default from there.
217+
218+### Testing
219+
220+It should be possible to add tests incrementally as we develop
221+pretty-printing code and new issues are discovered. The logic is
222+straightforward:
223+
224+1. Given input snippet of source code parse it and pretty print.
225+2. Check the output against expected output. Thus all tests should include
226+ two files: input and expected output.
227+3. Check that running the formatter on the output produces the same output
228+ again (the transformation is idempotent). This also checks that the
229+ output is still acceptable for the GHC parser.
230+4. It's a good idea to steal test cases from test suites of existing
231+ libraries like Brittany and Hindent.
232+5. After that we may add test cases for opened issues that Brittany and
233+ Hindent have.
234+6. When we're confident enough we can start “mining” new issues by running
235+ the program on real source code from Hackage till we don't get new issues
236+ anymore. For every issue that we find this way a test case should be
237+ added.
238+
239+## Roadmap
240+
241+Proposed roadmap (for a single person, about 38 full-time work days):
242+
243+* Create and setup a repo, setup CI (less than 1 day).
244+* Implement parsing, for now without CPP handling (1 day).
245+* Implement basis for pretty-printing of code. We could take
246+ [this][hindent-printer] as inspiration, although AST from the `ghc`
247+ package may be slightly different. Also, don't forget about annotations
248+ and comments (1 week).
249+* Implement executable program so we can easier fool around. Try it on
250+ simple code samples (1 day).
251+* Continue writing the pretty-printing code till we cover everything. Start
252+ adding tests, see [the section about testing](#testing). Fix bugs. At this
253+ point just repeat this till we cover everything that Brittany and Hindent
254+ can do maybe by stealing their tests (at least 2 weeks).
255+* Add support for CPP as described above. Also add the corresponding tests.
256+ Check that we're doing OK in the cases that are described in opened issues
257+ on the issue trackers of Brittany/Hindent/Stylish Haskell (1 week).
258+* Start checking how our formatter works on real-world code from Hackage.
259+ Fix bugs that we find that way (2 weeks).
260+* Write a good readme explaining why the project was created and how it is
261+ better, etc (1 day).
262+* Make and announce the first public release on Twitter/Reddit (less than 1
263+ day).
264+* Improve the executable by adding more interesting options, see other
265+ projects for inspiration. For example, it would be good to have an option
266+ which could check if running the formatter would change anything (1-3
267+ days).
268+* Figure out how to use the formatter application with major text editors
269+ and add the info to the readme (no estimate).
270+
271+[brittany]: https://hackage.haskell.org/package/brittany
272+[hindent]: https://hackage.haskell.org/package/hindent
273+[hindent-5-blog]: https://chrisdone.com/posts/hindent-5
274+[stylish-haskell]: https://hackage.haskell.org/package/stylish-haskell
275+[haskell-formatter]: https://hackage.haskell.org/package/haskell-formatter
276+[ghc]: https://hackage.haskell.org/package/ghc
277+[haskell-src-exts]: https://hackage.haskell.org/package/haskell-src-exts
278+[ghc-exactprint]: https://hackage.haskell.org/package/ghc-exactprint
279+[hindent-printer]: https://github.com/commercialhaskell/hindent/blob/master/src/HIndent/Pretty.hs