PR: additional local documentation as html (#1229)

* docs: implemented python script to convert local documentation from .md files to .html files

* docs: filled index.md; reviewed all references in .md files

* docs: updated Embedded.md and resized carp_on_arduboy.jpg to sensible width

* docs: copy sub folders (./docs/core and ./docs/sdl) to ./docs-html/ and refer docu to it

* docs: phrased requirements more clearly

* docs: generate docs for Standard libraries before converting .md docs to .html

* docs: change index to markdown lists

* docs: index.md worked around limitation in md converter

* docs: removed modules count from Libraries.md

Co-authored-by: guberatsie <gunnar.bernhardt@siemens.com>

docs-html/zConvertMd2Html.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+
+# If you want to browse the Carp documentation offline with all the bells and whistles 
+# (e.g. working hyperlinks to other documents and embedded graphics) that are usually 
+# provided by the github webpage, you can convert it to static html with this script.
+#
+# It needs Python3 and the markdown module to work. Install the latter with:
+#     sudo pip install markdown
+# or depending on you system configuration (e.g. on Windows):
+#     pip install markdown
+# 
+# Start the conversion with:
+#     python3 zConvertMd2Html.py 
+# or depending on you system configuration (e.g. on Windows):
+#     python zConvertMd2Html.py
+# or by executing the script itself (only on *NIX / BSD):
+#     chmod u+x zConvertMd2Html.py
+#     ./zConvertMd2Html.py
+#
+# Please note: 
+#    inter-document references do not work since the markdown-library does not create 
+#    the required html-anchor elements for headings. :-/
+
+import markdown, glob, os, platform, re, shutil
+
+# some definitions
+pat = re.compile("\[(.*?)\]\((.*?)\)")	
+docPath = "../docs/"
+docPathLen = len(docPath)
+
+def change_md2html( match ):
+    # unfortunately we can not use filters inside the regular expression like this:
+    #    pat = re.compile("\[(.*?)\]\(((?!http).*?)\.md(#.*?)?\)")	
+    # ... because that would create false matches for cases like this: 
+    #    [desc1](ref1) txt1 (desc2(ref2). txt2 [desc3](https://url1/doc1#anchor1)
+    # since the match is still too greedy. 
+    mgroups = match.groups() 
+    refParts = mgroups[1].split("#")
+    refFile  = refParts[0]
+    lFile = refFile.lower()
+    if len(refParts) < 2:
+        refAnchor = ""
+    else:
+        refAnchor = "#" + refParts[1]
+    if lFile.startswith("http") or not lFile.endswith(".md") or lFile.startswith("../"):
+        # not a reference to a local .md file => return unchanged match
+        mSpan = match.span()
+        new = match.string[ mSpan[0]: mSpan[1] ]
+        print("\tkept reference: " + new)
+    else:
+        new = "[%s](%s.html%s)" % (mgroups[0], refFile[:-3], refAnchor )
+        print("\tadjusted reference: " + new)
+    return new
+
+def generate(fnInput, fnBase4Output):
+    if not os.path.isfile(fnInput):
+        print( "copying directory: %s to %s" % (fnInput,fnBase4Output) )
+        shutil.copytree( fnInput, fnBase4Output)
+    elif fnInput.endswith(".md"):
+        print( "converting to html: " + fnInput)
+        # replace any link "[desc](ref.md)" with "[desc](ref.html" if ref does not start with "http" since we only want to change locally available documentation
+        with open(fnInput) as finput:
+            content = finput.read()
+        new_content = re.sub(pat, change_md2html, content)
+        with open( fnBase4Output[:-3] + '.html', 'w') as foutput:
+            foutput.write( markdown.markdown( new_content, extensions=['fenced_code', 'codehilite'] ) )
+    else:
+        # any other files might be resources (e.g. pictures) which probably need to be copied
+        print( "copying file: %s to %s" % (fnInput,fnBase4Output) )
+        shutil.copy( fnInput, fnBase4Output)
+
+def remove_existing_dir(path):
+    if os.path.exists(path):
+        shutil.rmtree(path)
+
+
+# main program
+
+print("(re)generating html docs for Carp standard library...")
+pOld = os.getcwd()
+os.chdir("..")
+cmd = "carp -x ./docs/core/generate_core_docs.carp"
+if platform.system()=="Windows":
+    cmd = cmd.replace("/","\\")
+os.system(cmd)    
+os.chdir(pOld)
+print("... done\n")
+
+for p in glob.glob("./*/"):
+    remove_existing_dir(p)
+
+# generate html files
+for fname in glob.glob(docPath+"*"):
+    generate( fname, fname[docPathLen:] )
+generate( "./core/README.md", "./core/README.md" )
+
+print("\ndone :-)")

docs/Embedded.md

@@ -1,7 +1,5 @@
 # Embedded
 
-<img src="carp_on_arduboy.jpg">
-
 Programming for embedded devices is a bit like living in the desert. Everything
 is scarce, you have to constantly ration, and you better stay out of the sun.
 
@@ -17,6 +15,12 @@ done, great! Carp will probably work with them out of the box. Still, getting
 acquainted with the tools it provides might help you have an easier time
 getting productive.
 
+## A picture is worth...
+
+Just to give you an idea of what is possible, here's a picture to whet your appetite:
+<img src="carp_on_arduboy.jpg">
+
+
 ## Fundamentals
 
 In order to tame the compiler to do as you tell it to, a firm grasp on the

docs/LanguageGuide.md

@@ -10,7 +10,7 @@ When a value is returned or passed to another function the initial function will
 and any subsequent use will lead to a compiler error. To temporarily lend a value to another function
 (for example to print it) a reference must be created, using the `ref` special form (or the `&` reader macro).
 
-To learn more about the details of memory management, check out [Memory.md](https://github.com/carp-lang/Carp/blob/master/docs/Memory.md)
+To learn more about the details of memory management, check out [Memory.md](Memory.md)
 
 ### Comments
 ```clojure

docs/Libraries.md

@@ -1,6 +1,10 @@
 ## Working with libraries and modules
 
-Carp comes with a standard library called [Core](../core/) where only certain files are loaded by default (see [Core.carp](../core/Core.carp)). If your `CARP_DIR` environment variable is [set properly](https://github.com/carp-lang/Carp/blob/master/docs/Install.md#setting-the-carp_dir), these libraries will be easily imported using the `load` command. For example, to get access to the Bench module, do the following:
+Carp comes with a standard library called Core, sources are [here](../core/). 
+It consists of a number of modules.
+Documentation is available [online](http://carp-lang.github.io/carp-docs/core/core_index.html) as well as [locally](./core/core_index.html) for most of them.
+The majority of modules are loaded by default, see [Core.carp](../core/Core.carp) for details. 
+If your `CARP_DIR` environment variable is set up [properly](Install.md#setting-the-carp_dir), then the remaining libraries can easily be imported using the `load` command. For example, to get access to the Bench module, do the following:
 
 ```clojure
 (load "Bench.carp")
@@ -52,13 +56,14 @@ You can generate HTML documentation for a set of modules by running `save-docs`
 ```clojure
 > (save-docs Int Float String)
 ```
-
-See the file [generate_core_docs.carp](./core/generate_core_docs.carp) for an example of how to configure the result.
+See the [ReadMe](./core/README.md) for updating the entire standard library documentation or
+the [program](./core/generate_core_docs.carp) for examples of how to configure the documentation generator.
 
 
 ## Auto generated API documentation
 
-* [Core](http://carp-lang.github.io/carp-docs/core/core_index.html)
+* [online docu for Carp standard library](http://carp-lang.github.io/carp-docs/core/core_index.html)
+* [local  docu for Carp standard library](./core/core_index.html)
 * [SDL](http://carp-lang.github.io/carp-docs/sdl/SDL_index.html)
 
 ## Some External Libraries

docs/core/README.md

@@ -1 +1,9 @@
-[The Carp Core Library Reference](http://carp-lang.github.io/carp-docs/core/core_index.html)
+The Carp Standard Library Reference can be found [online](http://carp-lang.github.io/carp-docs/core/core_index.html)
+but your Carp installation should also contain a local copy in this very directory.
+
+You can call [a short Carp program](./generate_core_docs.carp) to refresh the local copy.
+Please note that the program expects to be run from the Carp installation directory not from ./docs/core:
+```
+carp -x ./docs/core/generate_core_docs.carp
+```
+

docs/core/generate_core_docs.carp

@@ -1,7 +1,9 @@
+; runs this from the Carp installation directory, NOT current directory!
+
 (Project.config "title" "core")
 (Project.config "docs-directory" "./docs/core/")
 (Project.config "docs-logo" "logo.png")
-(Project.config "docs-prelude" "This website contains the documentation for the Carp standard library. You can download Carp [here](https://github.com/carp-lang/carp).")
+(Project.config "docs-prelude" "Welcome to the documentation for the Carp standard library. Please select a library from the menu on the right to browse its documentation. Carp is available for download [here](https://github.com/carp-lang/carp).")
 (Project.config "docs-url" "https://github.com/carp-lang/carp")
 
 (load "SafeInt.carp")

docs/index.md

@@ -1 +1,29 @@
+# Help index
 
+Please start by reading the [Manual](Manual.md)
+
+You may find yourself coming back to 
+
+* [the description of the Carp language](LanguageGuide.md) 
+* [its libraries.](Libraries.md)
+
+Also there is documentation about:
+
+* [C Interop](CInterop.md); 
+* [how to contribute](Contributing.md);
+* [support for implementig interfaces for datatypes based on their members](Derive.md); 
+* [custom resource deallocation](Drop.md); 
+* [where we are going with Carps dynamic evaluator](DynamicSemantics.md); 
+* [programming embedded devices](Embedded.md); 
+* [formatted output with (fmt) and (fstr)](Format_Strings.md); 
+* [hacking the compiler itself](hacking.md); 
+* [working with the REPL](HowToRunCode.md); 
+* [how to install and set up Carp](Install.md); 
+* [working with Macros](Macros.md); 
+* [Memory Management](Memory.md); 
+* [an invitation to use Carp for sound and graphics](Multimedia.md); 
+* [common patterns in Carp programs](patterns.md); 
+* [using quasi-quotation in macros and dynamic functions](Quasiquotation.md); 
+* [things to remember for each new Carp release](ReleaseChecklist.md); 
+* [some clarification on the terminology we use to explain things](Terminology.md); 
+* [editor support and tooling](Tooling.md).