Fix keyword-based search (#323)

* Fixes directories handling in git-based repos
* Fixes a minor bug in tldr-adapter

README.md

@@ -1,4 +1,5 @@
 
+
 ![cheat.sh logo](http://cheat.sh/files/big-logo-v2-fixed.png)
 
 Unified access to the best community driven cheat sheets repositories of the world.
@@ -15,7 +16,7 @@ What features should it have?
 * **Tutoring** — It should help you to learn the subject.
 * **Inconspicuous** — It should be possible to use it completely unnoticed.
 
-Such a thing exists.
+Such a thing exists! It's easy to [install](#installation) and there's even [auto-complete](#tab-completion).
 
 
 ## Features
@@ -43,7 +44,10 @@ Such a thing exists.
   * [Installation](#installation)
   * [Client usage](#client-usage)
   * [Tab-completion](#tab-completion)
+    - [Bash Tab completion](#bash-tab-completion)
+    - [ZSH Tab completion](#zsh-tab-completion)
   * [Stealth mode](#stealth-mode)
+  * [Windows command line client](#windows-command-line-client)
 * [Self-Hosting](#self-hosting)
   * [Docker](#docker)
 * [Editors integration](#editors-integration)
@@ -187,6 +191,8 @@ Try your own queries. Follow these rules:
 
 Read more about the programming languages queries below.
 
+----
+
 ## Command line client, cht.sh
 
 The cheat.sh service has its own command line client (`cht.sh`) that
@@ -202,17 +208,18 @@ has several useful features compared to querying the service directly with `curl
 
 To install the client:
 
-```
-    mkdir -p ~/bin/
-    curl https://cht.sh/:cht.sh > ~/bin/cht.sh
-    chmod +x ~/bin/cht.sh
+```bash
+PATH_DIR="$HOME/bin"  # or another directory on your $PATH
+mkdir -p "$PATH_DIR"
+curl https://cht.sh/:cht.sh > "$PATH_DIR/cht.sh"
+chmod +x "$PATH_DIR/cht.sh"
 ```
 
 or to install it globally (for all users):
 
-```
-    curl https://cht.sh/:cht.sh | sudo tee /usr/local/bin/cht.sh
-    chmod +x /usr/local/bin/cht.sh
+```bash
+curl https://cht.sh/:cht.sh | sudo tee /usr/local/bin/cht.sh
+chmod +x /usr/local/bin/cht.sh
 ```
 
 Note: The package "rlwrap" is a required dependency to run in shell mode. Install this using `sudo apt install rlwrap`
@@ -300,13 +307,13 @@ Use it to specify query options that you would use with each query.
 For example, to switch syntax highlighting off create the file with the following
 content:
 
-```
+```bash
 CHTSH_QUERY_OPTIONS="T"
 ```
 
 Or if you want to use a special syntax highlighting theme:
 
-```
+```bash
 CHTSH_QUERY_OPTIONS="style=native"
 ```
 
@@ -314,7 +321,7 @@ CHTSH_QUERY_OPTIONS="style=native"
 
 Other cht.sh configuration parameters:
 
-```
+```bash
 CHTSH_CURL_OPTIONS="-A curl"        # curl options used for cht.sh queries
 CHTSH_URL=https://cht.sh            # URL of the cheat.sh server
 ```
@@ -326,21 +333,23 @@ CHTSH_URL=https://cht.sh            # URL of the cheat.sh server
 
 To activate tab completion support for `cht.sh`, add the `:bash_completion` script to your `~/.bashrc`:
 
-```
-    $ curl https://cheat.sh/:bash_completion > ~/.bash.d/cht.sh
-    $ . ~/.bash.d/cht.sh
-    $ # and add . ~/.bash.d/cht.sh to ~/.bashrc
+```bash
+    curl https://cheat.sh/:bash_completion > ~/.bash.d/cht.sh
+    . ~/.bash.d/cht.sh
+    # and add . ~/.bash.d/cht.sh to ~/.bashrc
 ```
 
 #### ZSH Tab completion
 
 To activate tab completion support for `cht.sh`, add the `:zsh` script to the *fpath* in your `~/.zshrc`:
 
+```zsh
+    curl https://cheat.sh/:zsh > ~/.zsh.d/_cht
+    echo 'fpath=(~/.zsh.d/ $fpath)' >> ~/.zshrc
+    # Open a new shell to load the plugin
 ```
-    $ curl https://cheat.sh/:zsh > ~/.zsh.d/_cht
-    $ echo 'fpath=(~/.zsh.d/ $fpath)' >> ~/.zshrc
-    $ # Open a new shell to load the plugin
-```
+
+----
 
 ### Stealth mode
 
@@ -442,6 +451,8 @@ You can also use [`scoop`](https://github.com/lukesampson/scoop) command-line in
 scoop install cht
 ```
 
+----
+
 ## Self-Hosting
 
 ### Docker
@@ -529,7 +540,7 @@ In this example, several Vim plugins are used:
 * [scrooloose/syntastic](https://github.com/vim-syntastic/syntastic) — Syntax checking plugin
 * [cheat.sh-vim](https://github.com/dbeniamine/cheat.sh-vim) — Vim support
 
-Syntastic shows warnings and errors (found by code analysis tools: `jshint`, `merlin`, `pylint`, `shellcheckt etc.),
+Syntastic shows warnings and errors (found by code analysis tools: `jshint`, `merlin`, `pylint`, `shellcheck` etc.),
 and `cheat.sh-vim` shows you explanations for the errors and warnings
 and answers on programming languages queries written in the editor.
 
@@ -813,15 +824,17 @@ and information sources, maintained by thousands of users, developers and author
 all over the world
 (in the *Users* column number of contributors/number of stars is shown):
 
-|Cheat sheets           |Repository                                            | Users | Creation Date |
-|-----------------------|------------------------------------------------------|------------|---------------|
-|UNIX/Linux, programming|[cheat.sheets](https://github.com/chubin/cheat.sheets)| 38/223       | May 1, 2017   |
-|UNIX/Linux commands    |[tldr-pages/tldr](https://github.com/tldr-pages/tldr) | 760/23158  | Dec 8, 2013   |
-|UNIX/Linux commands    |[chrisallenlane/cheat](https://github.com/chrisallenlane/cheat)|131/5240|Jul 28, 2013|
-|Programming languages  |[adambard/learnxinyminutes-docs](https://github.com/adambard/learnxinyminutes-docs)|1246/6748|Jun 23, 2013|
-|Go                     |[a8m/go-lang-cheat-sheet](https://github.com/a8m/go-lang-cheat-sheet)|31/4039|Feb 9, 2014|
-|Perl                   |[pkrumnis/perl1line.txt](https://github.com/pkrumins/perl1line.txt)|5/190|Nov 4, 2011|
-|Programming languages  |[StackOverflow](https://stackoverflow.com)|9M |Sep 15, 2008|
+|Cheat sheets           |Repository                                                                          |C/U*                                                                                                                  |Stars                                                                                                   |Creation Date|
+|-----------------------|------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|-------------|
+|UNIX/Linux, programming|[cheat.sheets](https://github.com/chubin/cheat.sheets)                              |![](https://img.shields.io/github/contributors-anon/chubin/cheat.sheets?label=%F0%9F%91%A5&labelColor=white)           |![](https://img.shields.io/github/stars/chubin/cheat.sheets?label=%E2%AD%90&labelColor=white)           |May 1, 2017  |
+|UNIX/Linux commands    |[tldr-pages/tldr](https://github.com/tldr-pages/tldr)                               |![](https://img.shields.io/github/contributors-anon/tldr-pages/tldr?label=%F0%9F%91%A5&labelColor=white)                           |![](https://img.shields.io/github/stars/tldr-pages/tldr?label=%E2%AD%90&labelColor=white)               |Dec 8, 2013  |
+|UNIX/Linux commands    |[chrisallenlane/cheat](https://github.com/chrisallenlane/cheat)                     |![](https://img.shields.io/github/contributors-anon/chrisallenlane/cheat?label=%F0%9F%91%A5&labelColor=white)          |![](https://img.shields.io/github/stars/chrisallenlane/cheat?label=%E2%AD%90&labelColor=white)          |Jul 28, 2013 |
+|Programming languages  |[adambard/learnxinyminutes-docs](https://github.com/adambard/learnxinyminutes-docs) |![](https://img.shields.io/github/contributors-anon/adambard/learnxinyminutes-docs?label=%F0%9F%91%A5&labelColor=white)|![](https://img.shields.io/github/stars/adambard/learnxinyminutes-docs?label=%E2%AD%90&labelColor=white)|Jun 23, 2013 |
+|Go                     |[a8m/go-lang-cheat-sheet](https://github.com/a8m/go-lang-cheat-sheet)               |![](https://img.shields.io/github/contributors-anon/a8m/go-lang-cheat-sheet?label=%F0%9F%91%A5&labelColor=white)       |![](https://img.shields.io/github/stars/a8m/go-lang-cheat-sheet?label=%E2%AD%90&labelColor=white)       |Feb 9, 2014  |
+|Perl                   |[pkrumnis/perl1line.txt](https://github.com/pkrumins/perl1line.txt)                 |![](https://img.shields.io/github/contributors-anon/pkrumins/perl1line.txt?label=%F0%9F%91%A5&labelColor=white)        |![](https://img.shields.io/github/stars/pkrumins/perl1line.txt?label=%E2%AD%90&labelColor=white)        |Nov 4, 2011  |
+|Programming languages  |[StackOverflow](https://stackoverflow.com)                                          |[14M](https://stackexchange.com/leagues/1/alltime/stackoverflow)                                                       |N/A                                                                                                     |Sep 15, 2008 |
+
+<sup>(*) C/U — contributors for GitHub repositories, Users for Stackoverflow</sup>
 
 Pie diagram reflecting cheat sheets sources distribution (by number of cheat sheets on cheat.sh originating from a repository):
 

bin/app.py

@@ -77,6 +77,7 @@ PLAIN_TEXT_AGENTS = [
     "openbsd ftp",
     "powershell",
     "fetch",
+    "aiohttp",
 ]
 
 def _is_html_needed(user_agent):

lib/adapter/git_adapter.py

@@ -43,7 +43,7 @@ class RepositoryAdapter(Adapter):
             self._cheatsheet_files_prefix,
             topic)
 
-        if os.path.exists(filename):
+        if os.path.exists(filename) and not os.path.isdir(filename):
             answer = self._format_page(open(filename, 'r').read())
         else:
             # though it should not happen
@@ -134,7 +134,7 @@ class GitRepositoryAdapter(RepositoryAdapter):    #pylint: disable=abstract-meth
         """
         local_repository_dir = cls.local_repository_location()
         state_filename = os.path.join(local_repository_dir, '.cached_revision')
-        open(state_filename, 'w').write(state)
+        open(state_filename, 'wb').write(state)
 
     @classmethod
     def get_state(cls):

lib/adapter/question.py

@@ -39,7 +39,7 @@ If the problem persists, file a GitHub issue at
 github.com/chubin/cheat.sh or ping @igor_chubin
 """
 
-NOT_AUTHORIZED_MESSAGE = """403 UNAUTHORIZED"
+NOT_AUTHORIZED_MESSAGE = """403 UNAUTHORIZED"""
 
 class Question(UpstreamAdapter):
 
@@ -113,13 +113,14 @@ class Question(UpstreamAdapter):
         else:
             topic = [topic]
 
-        if request_option.get("action") == "suggest":
-            if request_option.get("authorized"):
+        if request_options.get("action") == "suggest":
+            if request_options.get("authorized"):
                 topic = ["--action", "suggest"] + topic
             else:
                 return NOT_AUTHORIZED_MESSAGE
 
         cmd = [CONFIG["path.internal.bin.upstream"]] + topic
+        print(cmd)
         proc = Popen(cmd, stdin=open(os.devnull, "r"), stdout=PIPE, stderr=PIPE)
         answer = proc.communicate()[0].decode('utf-8')
 

lib/adapter/tldr.py

@@ -73,15 +73,16 @@ class Tldr(GitRepositoryAdapter):
         and as soon as anything is found, format and return it.
         """
 
-        search_order = ['common', 'linux', 'osx', 'sunos', 'windows']
+        search_order = ['common', 'linux', 'osx', 'sunos', 'windows', "android"]
         local_rep = self.local_repository_location()
         ext = self._cheatsheet_files_extension
 
         filename = None
         for subdir in search_order:
-            filename = os.path.join(
+            _filename = os.path.join(
                 local_rep, 'pages', subdir, "%s%s" % (topic, ext))
-            if os.path.exists(filename):
+            if os.path.exists(_filename):
+                filename = _filename
                 break
 
         if filename:

lib/cheat_wrapper.py

@@ -94,6 +94,21 @@ def cheat_wrapper(query, request_options=None, output_format='ansi'):
     # query = _strip_hyperlink(query.rstrip('/'))
     topic, keyword, search_options = _parse_query(query)
 
+    # Process special case when 'action' is specified.
+    # In this case the response must be returned as is,
+    # without any changes and postprocessing
+    #
+    # 'action' is specified for suggestion queries, link queries etc.
+    if request_options.get("action"):
+        answers = get_answers(topic, request_options=request_options)
+        if answers:
+            answer = answers[0]
+            if isinstance(answer, dict):
+                answer = answer.get("answer", "")
+            return answer, True
+        else:
+            return "", False
+
     if keyword:
         answers = find_answers_by_keyword(
             topic, keyword, options=search_options, request_options=request_options)
@@ -116,5 +131,5 @@ def cheat_wrapper(query, request_options=None, output_format='ansi'):
         answer_data['topics_list'] = get_topics_list()
         return frontend.html.visualize(answer_data, request_options)
     elif output_format == 'json':
-        return json.dumps(answer_data, indent=4)
+        return json.dumps(answer_data, indent=4), True
     return frontend.ansi.visualize(answer_data, request_options)

lib/fetch.py

@@ -136,6 +136,7 @@ def _update_adapter(adptr):
     updates = []
     if cmd:
         errorcode, output = _run_cmd(cmd)
+        output = output.decode("utf-8")
         if errorcode:
             _log("\nERROR:\n---\n" + output + "\n---\nCould not get list of pages to be updated: %s" % adptr)
             return False

lib/frontend/html.py

@@ -96,7 +96,7 @@ def _render_html(query, result, editable, repository_button, topics_list, reques
     curl_line = "<span class='pre'>$ curl cheat.sh/</span>"
     if query == ':firstpage':
         query = ""
-    form_html = ('<form action="/" method="GET"/>'
+    form_html = ('<form action="/" method="GET">'
                  '%s%s'
                  '<input'
                  ' type="text" value="%s" name="topic"'

lib/routing.py

@@ -188,6 +188,15 @@ class Router(object):
         if topic_type and topic_type in topic_types:
             topic_types = [topic_type]
 
+
+        # Process action-queries directly, without caching and postprocessing
+        if request_options.get("action"):
+            answers = []
+            for topic_type in topic_types:
+                answers.append(
+                    self._get_page_dict(topic, topic_type, request_options=request_options))
+            return answers
+
         # 'question' queries are pretty expensive, that's why they should be handled
         # in a special way:
         # we do not drop the old style cache entries and try to reuse them if possible

share/cht.sh.txt

@@ -24,8 +24,8 @@
 # count words in text counter
 # group elements list
 
-__CHTSH_VERSION=0.0.1
-__CHTSH_DATETIME="2020-08-05 09:30:30 +0200"
+__CHTSH_VERSION=0.0.4
+__CHTSH_DATETIME="2021-04-25 12:30:30 +0200"
 
 # cht.sh configuration loading
 #
@@ -199,8 +199,8 @@ EOF
   fi
 
   _say_what_i_do Creating virtual environment
-  "$python" "$(command -v virtualenv)" "${virtualenv_python3_option[@]}" ve \
-      || fatal Could not create virtual environment with "python2 $(command -v virtualenv) ve"
+  virtualenv "${virtualenv_python3_option[@]}" ve \
+      || fatal "Could not create virtual environment with 'virtualenv ve'"
 
   export CHEATSH_PATH_WORKDIR=$PWD
 
@@ -514,7 +514,11 @@ else
 fi
 
 if [ "$is_macos" != yes ]; then
-  command -v xsel >/dev/null ||   echo 'DEPENDENCY: please install "xsel" for "copy"' >&2
+  if [ "$XDG_SESSION_TYPE" = wayland ]; then
+    command -v wl-copy >/dev/null || echo 'DEPENDENCY: please install "wl-copy" for "copy"' >&2
+  else
+    command -v xsel >/dev/null ||   echo 'DEPENDENCY: please install "xsel" for "copy"' >&2
+  fi
 fi
 command -v rlwrap >/dev/null || { echo 'DEPENDENCY: install "rlwrap" to use cht.sh in the shell mode' >&2; exit 1; }
 
@@ -562,7 +566,11 @@ cmd_copy() {
   else
     curl -s "${CHTSH_URL}"/"$(get_query_options "$query"?T)" > "$TMP1"
     if [ "$is_macos" != yes ]; then
-      xsel -bi < "$TMP1"
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        wl-copy < "$TMP1"
+      else
+        xsel -bi < "$TMP1"
+      fi
     else
       pbcopy < "$TMP1"
     fi
@@ -578,7 +586,11 @@ cmd_ccopy() {
   else
     curl -s "${CHTSH_URL}"/"$(get_query_options "$query"?TQ)" > "$TMP1"
     if [ "$is_macos" != yes ]; then
-      xsel -bi < "$TMP1"
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        wl-copy < "$TMP1"
+      else
+        xsel -bi < "$TMP1"
+      fi
     else
       pbcopy < "$TMP1"
     fi
@@ -674,7 +686,11 @@ cmd_stealth() {
   if [ "$is_macos" = yes ]; then
     past=$(pbpaste)
   else
-    past=$(xsel -o)
+    if [ "$XDG_SESSION_TYPE" = wayland ]; then
+      past=$(wl-paste -p)
+    else
+      past=$(xsel -o)
+    fi
   fi
   printf "\033[0;31mstealth:\033[0m you are in the stealth mode; select any text in any window for a query\n"
   printf "\033[0;31mstealth:\033[0m selections longer than $STEALTH_MAX_SELECTION_LENGTH words are ignored\n"
@@ -686,13 +702,17 @@ cmd_stealth() {
     if [ "$is_macos" = yes ]; then
       current=$(pbpaste)
     else
-      current=$(xsel -o)
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        current=$(wl-paste -p)
+      else
+        current=$(xsel -o)
+      fi
     fi
     if [ "$past" != "$current" ]; then
       past=$current
       current_text="$(echo $current | tr -c '[a-zA-Z0-9]' ' ')"
       if [ "$(echo "$current_text" | wc -w)" -gt "$STEALTH_MAX_SELECTION_LENGTH" ]; then
-        echo "\033[0;31mstealth:\033[0m selection length is longer than $STEALTH_MAX_SELECTION_LENGTH words; ignoring"
+        printf "\033[0;31mstealth:\033[0m selection length is longer than $STEALTH_MAX_SELECTION_LENGTH words; ignoring\n"
         continue
       else
         printf "\n\033[0;31mstealth: \033[7m $current_text\033[0m\n"

share/help.txt

@@ -71,17 +71,9 @@ List of search options:
 
 Programming languages topics:
 
-each programming language topic has the following subptopics:
+Each programming language topic has the following subtopics:
 
     hello               hello world + how to start the program
     :learn              big cheat sheet for learning language from scratch
     :list               list of topics
     :random             fetches a random cheat sheet belonging to the topic
-
-Support programming languages:
-
-    go 
-    scala
-    rust
-    python
-    php

tests/results/7

@@ -71,17 +71,9 @@ List of search options:
 
 Programming languages topics:
 
-each programming language topic has the following subptopics:
+Each programming language topic has the following subtopics:
 
     hello               hello world + how to start the program
     :learn              big cheat sheet for learning language from scratch
     :list               list of topics
     :random             fetches a random cheat sheet belonging to the topic
-
-Support programming languages:
-
-    go 
-    scala
-    rust
-    python
-    php

tests/results/8

@@ -24,8 +24,8 @@
 # count words in text counter
 # group elements list
 
-__CHTSH_VERSION=0.0.1
-__CHTSH_DATETIME="2020-08-05 09:30:30 +0200"
+__CHTSH_VERSION=0.0.4
+__CHTSH_DATETIME="2021-04-25 12:30:30 +0200"
 
 # cht.sh configuration loading
 #
@@ -199,8 +199,8 @@ EOF
   fi
 
   _say_what_i_do Creating virtual environment
-  "$python" "$(command -v virtualenv)" "${virtualenv_python3_option[@]}" ve \
-      || fatal Could not create virtual environment with "python2 $(command -v virtualenv) ve"
+  virtualenv "${virtualenv_python3_option[@]}" ve \
+      || fatal "Could not create virtual environment with 'virtualenv ve'"
 
   export CHEATSH_PATH_WORKDIR=$PWD
 
@@ -514,7 +514,11 @@ else
 fi
 
 if [ "$is_macos" != yes ]; then
-  command -v xsel >/dev/null ||   echo 'DEPENDENCY: please install "xsel" for "copy"' >&2
+  if [ "$XDG_SESSION_TYPE" = wayland ]; then
+    command -v wl-copy >/dev/null || echo 'DEPENDENCY: please install "wl-copy" for "copy"' >&2
+  else
+    command -v xsel >/dev/null ||   echo 'DEPENDENCY: please install "xsel" for "copy"' >&2
+  fi
 fi
 command -v rlwrap >/dev/null || { echo 'DEPENDENCY: install "rlwrap" to use cht.sh in the shell mode' >&2; exit 1; }
 
@@ -562,7 +566,11 @@ cmd_copy() {
   else
     curl -s "${CHTSH_URL}"/"$(get_query_options "$query"?T)" > "$TMP1"
     if [ "$is_macos" != yes ]; then
-      xsel -bi < "$TMP1"
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        wl-copy < "$TMP1"
+      else
+        xsel -bi < "$TMP1"
+      fi
     else
       pbcopy < "$TMP1"
     fi
@@ -578,7 +586,11 @@ cmd_ccopy() {
   else
     curl -s "${CHTSH_URL}"/"$(get_query_options "$query"?TQ)" > "$TMP1"
     if [ "$is_macos" != yes ]; then
-      xsel -bi < "$TMP1"
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        wl-copy < "$TMP1"
+      else
+        xsel -bi < "$TMP1"
+      fi
     else
       pbcopy < "$TMP1"
     fi
@@ -674,7 +686,11 @@ cmd_stealth() {
   if [ "$is_macos" = yes ]; then
     past=$(pbpaste)
   else
-    past=$(xsel -o)
+    if [ "$XDG_SESSION_TYPE" = wayland ]; then
+      past=$(wl-paste -p)
+    else
+      past=$(xsel -o)
+    fi
   fi
   printf "\033[0;31mstealth:\033[0m you are in the stealth mode; select any text in any window for a query\n"
   printf "\033[0;31mstealth:\033[0m selections longer than $STEALTH_MAX_SELECTION_LENGTH words are ignored\n"
@@ -686,13 +702,17 @@ cmd_stealth() {
     if [ "$is_macos" = yes ]; then
       current=$(pbpaste)
     else
-      current=$(xsel -o)
+      if [ "$XDG_SESSION_TYPE" = wayland ]; then
+        current=$(wl-paste -p)
+      else
+        current=$(xsel -o)
+      fi
     fi
     if [ "$past" != "$current" ]; then
       past=$current
       current_text="$(echo $current | tr -c '[a-zA-Z0-9]' ' ')"
       if [ "$(echo "$current_text" | wc -w)" -gt "$STEALTH_MAX_SELECTION_LENGTH" ]; then
-        echo "\033[0;31mstealth:\033[0m selection length is longer than $STEALTH_MAX_SELECTION_LENGTH words; ignoring"
+        printf "\033[0;31mstealth:\033[0m selection length is longer than $STEALTH_MAX_SELECTION_LENGTH words; ignoring\n"
         continue
       else
         printf "\n\033[0;31mstealth: \033[7m $current_text\033[0m\n"