@@ -0,0 +1,230 @@
1+Introduction
2+------------
3+
4+Paths are used by file systems as well as protocols to represent paths to files
5+and other resources. We need a generic type safe way to represent paths in
6+general and file system paths in particular.
7+
8+Path limits
9+-----------
10+
11+OS include files generally define PATH_MAX to 4K and NAME_MAX to 255,
12+however, it is possible to create paths bigger than these depending on
13+the file system.
14+
15+Scalability
16+-----------
17+
18+In general directory trees could be quite deep and a directory can contain
19+millions of entries. A good benchmark to measure the efficiency of path
20+representation would be to traverse a directory tree recursively and list all
21+the nodes under the tree. We could do this many times so that we do not need a
22+really big directory tree.
23+
24+Compatibility
25+-------------
26+
27+A file system starts with just the root directory and then files are
28+created in the file system by the user or by programs storing their data
29+on the file system. When a directory or file is created, or when a directory is
30+listed, the following operations are performed:
31+
32+1. for lookups an existing directory name must be resolved based on the name
33+ supplied by the user.
34+2. for creation the file name to be created is supplied by the user
35+
36+When the user asks the file system to lookup or create a file or
37+directory in the file system:
38+
39+1) The operating system passes the name, as it is without any changes
40+ whatsoever, to the file system. or does it? Windows?
41+2) The file system may translate the name to its own conventions before a
42+ lookup or create, e.g. it may
43+
44+ * change the name to upper case
45+ * translate the name to 8.3 chars
46+ * change the character encoding?
47+ * change the unicode normalization form of the name (Apple)
48+
49+When resolving an existing directory name in the file system we need
50+to supply a path which consists of component names separated by a separator
51+byte. Separators are of no consequence to the file system, they are
52+resolved by the OS and the path components are used to lookup the paths
53+one components at a time. The path for lookup is acquired either by a
54+user input, device input or by the program which previously got the path
55+entries by traversing the file system itself.
56+
57+1) When the path is acquired by a user input, the user input could be:
58+
59+ a) a literal string in the program
60+ b) a path entered via an input device
61+ c) a path coming from the network
62+
63+2) If the path was previously acquired from the file system then the
64+ best thing to do is to never change anything in the path and store it
65+ as it is and supply exactly the same path when needed. That way we can
66+ guarantee that the path remains exactly what it was in the file system.
67+
68+Handling String Literals
69+========================
70+
71+The encoding of the source code file depends on the editor used and the
72+encoding chosen when saving it. The string literals would be parsed
73+by the GHC parser and then stored in the generated binaries as null
74+terminated C string literals encoded in UTF-8 (see GHC reference). There are
75+several possible points of failure here:
76+
77+a) GHC parser needs to interpret the source code encoding correctly.
78+b) We assume that the editor does not perform any translation on the
79+ literal as entered by the user e.g. it does not perform unicode
80+ normalization on it. If it does then the string as entered by the user
81+ won't remain the same when it reaches the file system.
82+c) GHC parser stores the parsed string literal in UTF-8 encoding. We
83+ assume that GHC does not perform any unicode normalization or any
84+ other translation on the string.
85+
86+The UTF-8 encoded string literal can be passed as a blob of bytes to the file
87+system or it can be converted to String type and re-encoded as UTF-8 both
88+should work equivalently in this case.
89+
90+Handling Input From Devices
91+===========================
92+
93+The path provided by the user would assume some encoding based on
94+the terminal settings or the encoding assumed by the sender over the
95+network. The correctness depends on the contract between the two parties
96+e.g. the locale setting. We assume that we get a sequence of raw bytes
97+from the input device. We need to use the sequence as raw bytes and send
98+it as it is to the file system without any translation.
99+
100+Manipulating Paths
101+------------------
102+
103+We need to parse the path components by the separator bytes. We assume
104+that the separator can be identified and removed correctly irrespective
105+of the encoding. We also make sure that none of the bytes in the
106+components is a separator byte.
107+
108+Also, we would join the path components by the separator byte
109+irrespective of the encoding of the components. If the OS treats the
110+path as a sequence of bytes and nothing else and the components do not
111+have the separator byte then we are good, we know that the OS would also be
112+parsing based on the separator as a raw byte.
113+
114+We may perform some validations on the paths such as the file names are
115+not "." or "..". Such validations could be optional and we could also
116+provide a way to not perform any validations and just blindly use the
117+paths as provided by the user and let the file system/OS fail.
118+
119+File System Translations
120+------------------------
121+
122+As we noted earlier, the file system may translate the paths before
123+using them. For example, it may store the path after converting it to
124+NFD unicode normalization. Translation may create some round tripping
125+issues for programs. For example, a program may use a string literal
126+which is stored in NFC and the file system converts it to NFD. Later,
127+when the same path is retrieved from the file system and compared with
128+the string literal that was to create it then it won't match. For such
129+cases the programs need to understand the file system and perform
130+comparisons by performing appropriate translations on the paths. To
131+perform matching and translations correctly the program needs to
132+correctly interpret the encoding specific to the file system.
133+
134+Displaying Paths
135+----------------
136+
137+When we display the paths to the user then we are forced to interpret
138+it according to some encoding, to display the path correctly we have to
139+know exactly how the file system stores the path. Otherwise if we display
140+it differently, the user may use the displayed result to find the file
141+and may not find it.
142+
143+Type Safety Requirements
144+------------------------
145+
146+* Safety against using an absolute path where a relative path is to be
147+ used and vice-versa.
148+
149+ * Validations for absolute or relative path when constructing a path.
150+ * We cannot append an absolute path to another path
151+* Safety against using a file name where a directory name is to be used and
152+ vice-versa.
153+
154+ * Certain validations can be performed e.g. file names cannot be "." or "..".
155+ * We should not be appending more directory components to a file path
156+
157+In don't care situations we should be easily able to use any type
158+conveniently or cast a type into another. It is desirable that the
159+programmer can choose the safety level. For example, we should be able
160+to instantiate a path type where we only worry about the distinction
161+between Absolute and Relative paths but no distinction between files and
162+directories or vice versa.
163+
164+Requirement Summary
165+-------------------
166+
167+* minimal dependencies, specifically streamly does not depend on bytestring
168+* round-tripping safety wrt to file system returned paths
169+* type safety for different path types
170+* support Posix/Windows
171+* support URI paths and other ways to represent paths where the separator could
172+ be different.
173+
174+Design Considerations
175+---------------------
176+
177+* Should we store path as separate components or single string with
178+ separators?
179+
180+* Should we validate the paths returned from the file system or trust
181+ those and use directly without any validations? Need to see if that makes
182+ any difference to path heavy benchmarks. If we want to use it directly
183+ then we have to store it as a single string.
184+
185+* Parameterize the low level APIs with the separator so that we can
186+ support arbitrary separators when parsing or reconstructing paths.
187+
188+* The low level API can support path handling in trees/DAGs/Graphs in general.
189+ For example, in trees we cannot have multiple parents of a child whereas in
190+ DAGs that is allowed, in graphs we can have cycles. We may also need ways to
191+ detect cycles.
192+
193+* Do we need to support arbitrarily long paths i.e. streaming of path? We do
194+ not need that for file system paths and file system paths are limited size
195+ and operating system anyway requires them in strict buffers. In case of
196+ graphs if we have cycles paths can be infinite, we could generate a stream of
197+ path and the consumer could be traversing the graph according to the
198+ generated stream. If we want to support streaming then we have to store paths
199+ as a stream of chunks rather than a single string.
200+
201+* In general, paths need not be strings, e.g. they can be references to
202+ locations in memory or they can be IP addresses of nodes. At an abstract
203+ level, paths are just a stream of tokens that represent a certain traversal.
204+
205+* Relative paths are the most general representation. At a low level,
206+ all paths are relative, absolute paths are relative to a specified root
207+ whereas relative paths are relative to a dynamic root which is the
208+ current directory.
209+
210+* Windows can have the root as different drive letters. So to represent paths
211+ with a root in general we can also store the specific root along with the
212+ path. In case of POSIX this will always be "/". In general, it could be a
213+ host name or IP address or dependent on the protocol whose path we are
214+ representing.
215+
216+* We can parameterize the low level path type with the type of path e.g. POSIX,
217+ WINDOWS, HTTP etc. In general, programs may have to manipulate different
218+ types of paths at the same time. High level path types can be instantiated
219+ using the low level type therefore they can be much simpler as desired.
220+
221+References
222+----------
223+
224+Some related links found by web search:
225+
226+* https://gitlab.haskell.org/ghc/ghc/issues/5218
227+* https://nodejs.org/fr/docs/guides/working-with-different-filesystems/
228+* https://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux
229+* https://docs.microsoft.com/en-us/windows/win32/intl/character-sets-used-in-file-names
230+* https://beets.io/blog/paths.html