on master help sign in
docs: use make and uitls/searx.sh in containers

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

commitf27f6c6c5d372b257dfe671ec26eb73f6e12ed28 authorMarkus Heiser <markus.heiser@darmarit.de> dateTue 31 Mar 2020, 18:25 (7 years ago) parent8634eeb
6 files changed+139−48
docs/admin/buildhosts.rst+17−3 docs/build-templates/searx.rst+26 docs/utils/lxc.sh.rst+80−23 utils/lxc-searx.env−12 utils/lxc.sh+5−5 utils/searx.sh+11−5
docs/admin/buildhosts.rst+17−3
@@ -4,9 +4,6 @@
44Buildhosts
55==========
66
7−To get best results from build, its recommend to install additional packages
8−on build hosts.
9−
107.. sidebar:: This article needs some work
118
129 If you have any contribution send us your :pull:`PR <../pulls>`, see
@@ -17,6 +14,23 @@ on build hosts.
1714 :local:
1815 :backlinks: entry
1916
17+To get best results from build, its recommend to install additional packages
18+on build hosts (see :ref:`searx.sh`).::
19+
20+ sudo -H ./utils/searx.sh buildhost
21+
22+This will install packages needed by searx:
23+
24+.. include:: ../../build/docs/includes/searx.rst
25+ :start-after: START distro-packages
26+ :end-before: END distro-packages
27+
28+and packages needed to build docuemtation and run tests:
29+
30+.. include:: ../../build/docs/includes/searx.rst
31+ :start-after: START build-packages
32+ :end-before: END build-packages
33+
2034.. _docs build:
2135
2236Build docs
docs/build-templates/searx.rst+26
@@ -28,6 +28,32 @@ ${fedora}
2828
2929.. END distro-packages
3030
31+.. START build-packages
32+
33+.. tabs::
34+
35+ .. group-tab:: Ubuntu / debian
36+
37+ .. code-block:: sh
38+
39+ $ sudo -H apt-get install -y \\
40+${debian_build}
41+
42+ .. group-tab:: Arch Linux
43+
44+ .. code-block:: sh
45+
46+ $ sudo -H pacman -S --noconfirm \\
47+${arch_build}
48+
49+ .. group-tab:: Fedora / RHEL
50+
51+ .. code-block:: sh
52+
53+ $ sudo -H dnf install -y \\
54+${fedora_build}
55+
56+.. END build-packages
3157
3258.. START create user
3359
docs/utils/lxc.sh.rst+80−23
@@ -1,5 +1,4 @@
11
2−
32.. _snap: https://snapcraft.io
43.. _snapcraft LXD: https://snapcraft.io/lxd
54.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/
@@ -7,6 +6,8 @@
76.. _LXD: https://linuxcontainers.org/lxd/introduction/
87.. _`LXD@github`: https://github.com/lxc/lxd
98
9+.. _archlinux: https://www.archlinux.org/
10+
1011.. _lxc.sh:
1112
1213================
@@ -21,24 +22,31 @@
2122 - `LXD@github`_
2223
2324With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of
24−containers, what we call the: *lxc suite*. Before you can start with
25−containers, you need to install and initiate LXD_ once::
25+containers, what we call the: *lxc suite*. The *searx suite*
26+(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time
27+you start the ``lxc.sh`` script (*you do not need to care about*).
28+
29+Before you can start with containers, you need to install and initiate LXD_
30+once::
2631
2732 $ snap install lxd
2833 $ lxd init --auto
2934
30−The *searx suite* (:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by
31−default, every time you start the ``lxc.sh`` script (you do not need to care
32−about). To make use of the containers from the *searx suite*, you have to build
33−the :ref:`LXC suite containers <lxc.sh --help>` first. But be warned, this
34−might take some time::
35+To make use of the containers from the *searx suite*, you have to build the
36+:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, this might
37+take some time::
3538
3639 $ sudo -H ./utils/lxc.sh build
3740
3841A cup of coffee later, your LXC suite is build up and you can run whatever task
3942you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh
40−--help>`. Each container shares the root folder of the repository and the
41−command ``utils/lxc.sh cmd`` handles relative path names *transparent*::
43+help>`.
44+
45+*Good to know ...*
46+
47+Eeach container shares the root folder of the repository and the
48+command ``utils/lxc.sh cmd`` **handles relative path names transparent**,
49+compare output of::
4250
4351 $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile
4452 ...
@@ -46,29 +54,70 @@ command ``utils/lxc.sh cmd`` handles relative path names *transparent*::
4654 [searx-fedora31] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile
4755 [searx-archlinux] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile
4856
49−With this in mind, you can run :ref:`searx.sh` and install packages, needed by
50−searx::
57+If there comes the time you want to **get rid off all** the containers and
58+**clean up local images** just type::
5159
52− $ sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install packages
60+ $ sudo -H ./utils/lxc.sh remove
61+ $ sudo -H ./utils/lxc.sh remove images
5362
54−And run one of the :origin:`Makefile` targets::
5563
56− $ sudo -H ./utils/lxc.sh cmd -- make test.sh
64+Install suite
65+=============
5766
58−You can install a *buildhost environment* into the containers (time for another
59−cup of coffee)::
67+To install the complete :ref:`searx suite (includes searx, morty & filtron)
68+<lxc-searx.env>` into all LXC_ use::
6069
61− $ sudo -H ./utils/lxc.sh install buildhost
70+ $ sudo -H ./utils/lxc.sh install suite
6271
63−If you want to get rid off all the containers, just type::
72+The command above installs a searx suite (see :ref:`installation scripts`). To
73+get the IP (URL) of the filtron service in the containers use ``show suite``
74+command. To test instances from containers just open the URLs in your
75+WEB-Browser::
6476
65− $ sudo -H ./utils/lxc.sh remove
77+ $ sudo ./utils/lxc.sh show suite | grep filtron
78+ [searx-ubu1604] INFO: (eth0) filtron: http://n.n.n.135:4004/
79+ [searx-ubu1804] INFO: (eth0) filtron: http://n.n.n.141:4004/
80+ [searx-ubu1910] INFO: (eth0) filtron: http://n.n.n.137:4004/
81+ [searx-ubu2004] INFO: (eth0) filtron: http://n.n.n.127:4004/
82+ [searx-fedora31] INFO: (eth0) filtron: http://n.n.n.18:4004/
83+ [searx-archlinux] INFO: (eth0) filtron: http://n.n.n.12:4004/
6684
67−To clean up your local images use::
85+
86+Running commands
87+================
6888
69− $ sudo -H ./utils/lxc.sh remove images
89+**Inside containers, you can use make or run scripts** from the
90+:ref:`toolboxing`. By example: to setup a :ref:`buildhosts` and run the
91+Makefile target ``test`` in the archlinux_ container::
92+
93+ sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh buildhost
94+ sudo -H ./utils/lxc.sh cmd searx-archlinux make test
95+
96+
97+Setup searx buildhost
98+=====================
7099
71−.. _lxc.sh --help:
100+You can **install the searx buildhost environment** into one or all containers.
101+The installation procedure to set up a :ref:`build host<buildhosts>` takes its
102+time. Installation in all containers will take more time (time for another cup
103+of coffee).::
104+
105+ sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh buildhost
106+
107+To build (live) documentation inside a archlinux_ container::
108+
109+ sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live
110+ ...
111+ [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
112+
113+To get IP of the container and the port number *live docs* is listening::
114+
115+ $ sudo ./utils/lxc.sh show suite | grep docs-live
116+ ...
117+ [searx-archlinux] INFO: (eth0) docs-live: http://n.n.n.12:8080/
118+
119+
120+.. _lxc.sh help:
72121
73122Overview
74123========
@@ -77,3 +126,11 @@ The ``--help`` output of the script is largely self-explanatory:
77126
78127.. program-output:: ../utils/lxc.sh --help
79128
129+
130+.. _lxc-searx.env:
131+
132+searx suite
133+===========
134+
135+.. literalinclude:: ../../utils/lxc-searx.env
136+ :language: bash
utils/lxc-searx.env−12
@@ -49,18 +49,6 @@ lxc_suite_install() {
4949 )
5050}
5151
52−lxc_suite_prepare_buildhost() {
53− (
54− lxc_set_suite_env
55− FORCE_TIMEOUT=0
56− export FORCE_TIMEOUT
57− # "${LXC_REPO_ROOT}/utils/searx.sh* install packages
58− "${LXC_REPO_ROOT}/utils/searx.sh" install buildhost
59− rst_title "buildhost installation finished ($(hostname))" part
60− echo
61− )
62−}
63−
6452lxc_suite_info() {
6553 (
6654 lxc_set_suite_env
utils/lxc.sh+5−5
@@ -82,7 +82,7 @@ usage::
8282 $_cmd [start|stop] [containers|<name>]
8383 $_cmd show [info|config|suite|images]
8484 $_cmd cmd [--|<name>] '...'
85− $_cmd install [suite|base|buildhost]
85+ $_cmd install [suite|base]
8686
8787build
8888 :containers: build, launch and 'install basic' packages on 'containers'
@@ -106,7 +106,6 @@ cmd
106106install
107107 :suite: install LXC suite; ${LXC_SUITE_INSTALL_INFO}
108108 :base: prepare LXC; install basic packages
109− :buildhost: prepare LXC; install buildhost packages
110109
111110EOF
112111 usage_images
@@ -150,7 +149,9 @@ main() {
150149
151150 # don't check prerequisite when in recursion
152151 if [[ ! $1 == __* ]]; then
153− ! required_commands lxc && lxd_info && exit 42
152+ if ! in_container; then
153+ ! required_commands lxc && lxd_info && exit 42
154+ fi
154155 [[ -z $LXC_SUITE ]] && err_msg "missing LXC_SUITE" && exit 42
155156 fi
156157
@@ -240,7 +241,7 @@ main() {
240241 install)
241242 sudo_or_exit
242243 case $2 in
243− suite|base|buildhost)
244+ suite|base)
244245 lxc_exec "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2"
245246 ;;
246247 *) usage "$_usage"; exit 42 ;;
@@ -252,7 +253,6 @@ main() {
252253 case $2 in
253254 suite) lxc_suite_install ;;
254255 base) FORCE_TIMEOUT=0 lxc_install_base_packages ;;
255− buildhost) lxc_suite_prepare_buildhost ;;
256256 esac
257257 ;;
258258 doc)
utils/searx.sh+11−5
@@ -48,7 +48,7 @@ git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev
4848shellcheck"
4949
5050BUILD_PACKAGES_debian="\
51−graphviz imagemagick texlive-xetex librsvg2-bin
51+firefox graphviz imagemagick texlive-xetex librsvg2-bin
5252texlive-latex-recommended texlive-extra-utils ttf-dejavu"
5353
5454# pacman packages
@@ -59,7 +59,7 @@ git base-devel libxml2
5959shellcheck"
6060
6161BUILD_PACKAGES_arch="\
62−graphviz imagemagick texlive-bin extra/librsvg
62+firefox graphviz imagemagick texlive-bin extra/librsvg
6363texlive-core texlive-latexextra ttf-dejavu"
6464
6565# dnf packages
@@ -70,7 +70,7 @@ git @development-tools libxml2
7070ShellCheck"
7171
7272BUILD_PACKAGES_fedora="\
73−graphviz graphviz-gd ImageMagick librsvg2-tools
73+firefox graphviz graphviz-gd ImageMagick librsvg2-tools
7474texlive-xetex-bin texlive-collection-fontsrecommended
7575texlive-collection-latex dejavu-sans-fonts dejavu-serif-fonts
7676dejavu-sans-mono-fonts"
@@ -116,7 +116,7 @@ usage() {
116116usage::
117117
118118 $(basename "$0") shell
119− $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages]
119+ $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages|buildhost]
120120 $(basename "$0") update [searx]
121121 $(basename "$0") remove [all|user|pyenv|searx-src]
122122 $(basename "$0") activate [service]
@@ -135,7 +135,7 @@ install / remove
135135 :uwsgi: install searx uWSGI application
136136 :settings: reinstall settings from ${REPO_ROOT}/searx/settings.yml
137137 :packages: install needed packages from OS package manager
138− :buildhost: install packages from OS package manager needed by buildhost
138+ :buildhost: install packages from OS package manager needed by buildhosts
139139update searx
140140 Update searx installation ($SERVICE_HOME)
141141activate service
@@ -688,9 +688,15 @@ rst-doc() {
688688 local debian="${SEARX_PACKAGES_debian}"
689689 local arch="${SEARX_PACKAGES_arch}"
690690 local fedora="${SEARX_PACKAGES_fedora}"
691+ local debian_build="${BUILD_PACKAGES_debian}"
692+ local arch_build="${BUILD_PACKAGES_arch}"
693+ local fedora_build="${BUILD_PACKAGES_fedora}"
691694 debian="$(echo "${debian}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
692695 arch="$(echo "${arch}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
693696 fedora="$(echo "${fedora}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
697+ debian_build="$(echo "${debian_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
698+ arch_build="$(echo "${arch_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
699+ fedora_build="$(echo "${fedora_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
694700
695701 eval "echo \"$(< "${REPO_ROOT}/docs/build-templates/searx.rst")\""
696702