@@ -0,0 +1,413 @@
1+.. _blog-lxcdev-202006:
2+
3+=======================================
4+Developing in Linux containers [202006]
5+=======================================
6+
7+.. _LXC: https://linuxcontainers.org/lxc/introduction/
8+
9+.. sidebar:: Audience
10+
11+ This blog post is written for experienced admins and developers / readers
12+ should have a serious meaning about: *distributed*, *merge* and *linux
13+ container*.
14+
15+.. contents:: Contents
16+ :depth: 2
17+ :local:
18+ :backlinks: entry
19+
20+In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog
21+post I will show you, how you can make use of them in *distributed and
22+heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006
23+abstract`).
24+
25+Motivation
26+==========
27+
28+Normally in our development cycle, we edit the sources and run some test and/or
29+builds by using ``make`` before we commit. This cycle is simple and perfect but
30+might fail in some aspects we should not overlook.
31+
32+ The environment in which we run all our development processes matters!
33+
34+The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they
35+do not have access to all prerequisites. For example, there may have
36+dependencies on packages that are installed on the developer's desktop, but
37+usually are not preinstalled on a server or client system. Another examples
38+are; settings have been made to the software on the developer's host that would
39+never be set on a *production* system.
40+
41+*Linux Containers* (LXC_) are isolate environments and not to mix up on
42+developer's all the prerequisites of all the projects he contribute to, is
43+always a good choice.
44+
45+The scripts from PR :PR:`1803` can divide in those to install and maintain
46+software:
47+
48+- :ref:`searx.sh`
49+- :ref:`filtron.sh`
50+- :ref:`morty.sh`
51+
52+and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or
53+even development tasks over a stack of containers, what we call: *Searx's lxc
54+suite*.
55+
56+Gentlemen, start your engines!
57+==============================
58+
59+.. _LXD: https://linuxcontainers.org/lxd/introduction/
60+.. _archlinux: https://www.archlinux.org/
61+
62+Before you can start with containers, you need to install and initiate LXD_
63+once:
64+
65+.. tabs::
66+
67+ .. group-tab:: desktop
68+
69+ .. code:: sh
70+
71+ $ snap install lxd
72+ $ lxd init --auto
73+
74+And you need to clone from origin or if you have your own fork, clone from your
75+fork:
76+
77+.. tabs::
78+
79+ .. group-tab:: desktop
80+
81+ .. code:: sh
82+
83+ $ cd ~/Downloads
84+ $ git clone https://github.com/asciimoo/searx.git
85+ $ cd searx
86+
87+The :ref:`lxc-searx.env` consists of several images, see ``export
88+LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post
89+we exercise on a archlinux_ image. The container of this image is named
90+``searx-archlinux``. Lets build the container, but be sure that this container
91+does not already exists, so first lets remove possible old one:
92+
93+.. tabs::
94+
95+ .. group-tab:: desktop
96+
97+ .. code:: sh
98+
99+ $ sudo -H ./utils/lxc.sh remove searx-archlinux
100+ $ sudo -H ./utils/lxc.sh build searx-archlinux
101+
102+.. sidebar:: The ``searx-archlinux`` container
103+
104+ is the base of all our exercises here.
105+
106+In this container we install all services :ref:`including searx, morty & filtron
107+<lxc.sh install suite>` in once:
108+
109+.. tabs::
110+
111+ .. group-tab:: desktop
112+
113+ .. code:: sh
114+
115+ $ sudo -H ./utils/lxc.sh install suite searx-archlinux
116+
117+To proxy HTTP from filtron and morty in the container to the outside of the
118+container, install nginx into the container. Once for the bot blocker filtron:
119+
120+.. tabs::
121+
122+ .. group-tab:: desktop
123+
124+ .. code:: sh
125+
126+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
127+ ./utils/filtron.sh nginx install
128+ ...
129+ INFO: got 429 from http://10.174.184.156/searx
130+
131+and once for the content sanitizer (content proxy morty):
132+
133+.. tabs::
134+
135+ .. group-tab:: desktop
136+
137+ .. code:: sh
138+
139+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
140+ ./utils/morty.sh nginx install
141+ ...
142+ INFO: got 200 from http://10.174.184.156/morty/
143+
144+.. sidebar:: Fully functional searx suite
145+
146+ From here on you have a fully functional searx suite running with bot blocker
147+ (filtron) and Web content sanitizer (content proxy morty) needed for a
148+ *privacy protecting* search engine.
149+
150+On your system, the IP of your ``searx-archlinux`` container differs from
151+http://10.174.184.156/searx, just open the URL reported in your installation
152+protocol in your WEB browser from the desktop to test the instance from outside
153+of the container.
154+
155+In such a searx suite admins can maintain and access the debug log of the
156+different services quite easy.
157+
158+.. _working in containers:
159+
160+In containers, work as usual
161+============================
162+
163+Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers
164+open the root-bash in the container using ``./utils/lxc.sh cmd
165+searx-archlinux``:
166+
167+.. tabs::
168+
169+ .. group-tab:: desktop
170+
171+ .. code:: sh
172+
173+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash
174+ INFO: [searx-archlinux] bash
175+ [root@searx-archlinux searx]# pwd
176+ /share/searx
177+
178+The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in
179+the searx-container. To debug the running searx instance use:
180+
181+.. tabs::
182+
183+ .. group-tab:: root@searx-archlinux
184+
185+ .. code:: sh
186+
187+ $ ./utils/searx.sh inspect service
188+ ...
189+ use [CTRL-C] to stop monitoring the log
190+ ...
191+
192+Back in the browser on your desktop open the service http://10.174.184.156/searx
193+and run your application tests while the debug log is shown in the terminal from
194+above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug
195+option"* in searx's settings file and restarts the searx uwsgi application. To
196+debug services from filtron and morty analogous use:
197+
198+.. tabs::
199+
200+ .. group-tab:: root@searx-archlinux
201+
202+ .. code:: sh
203+
204+ $ ./utils/filtron.sh inspect service
205+ $ ./utils/morty.sh inspect service
206+
207+Another point we have to notice is that each service (:ref:`searx <searx.sh>`,
208+:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated
209+system user account with the same name (compare :ref:`create searx user`). To
210+get a shell from theses accounts, simply call one of the scripts:
211+
212+.. tabs::
213+
214+ .. group-tab:: root@searx-archlinux
215+
216+ .. code:: sh
217+
218+ $ ./utils/searx.sh shell
219+ $ ./utils/filtron.sh shell
220+ $ ./utils/morty.sh shell
221+
222+To get in touch, open a shell from the service user (searx@searx-archlinux):
223+
224+.. tabs::
225+
226+ .. group-tab:: desktop
227+
228+ .. code:: sh
229+
230+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
231+ ./utils/searx.sh shell
232+ // exit with [CTRL-D]
233+ (searx-pyenv) [searx@searx-archlinux ~]$ ...
234+
235+The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system
236+user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv*
237+``(searx-pyenv)`` environment is activated.
238+
239+.. tabs::
240+
241+ .. group-tab:: searx@searx-archlinux
242+
243+ .. code:: sh
244+
245+ (searx-pyenv) [searx@searx-archlinux ~]$ pwd
246+ /usr/local/searx
247+
248+
249+
250+Wrap production into developer suite
251+====================================
252+
253+In this section we will see how to change the *"Fully functional searx suite"*
254+from a LXC container (which is quite ready for production) into a developer
255+suite. For this, we have to keep an eye on the :ref:`installation basic`:
256+
257+- searx setup in: ``/etc/searx/settings.yml``
258+- searx user's home: ``/usr/local/searx``
259+- virtualenv in: ``/usr/local/searx/searx-pyenv``
260+- searx software in: ``/usr/local/searx/searx-src``
261+
262+The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and
263+the working tree is checked out from the ``GIT_BRANCH``. With the use of the
264+:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application
265+<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare
266+:ref:`service architectures on distributions <uwsgi configuration>`).
267+
268+.. tabs::
269+
270+ .. group-tab:: desktop
271+
272+ .. code:: sh
273+
274+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
275+ systemctl stop uwsgi@searx
276+
277+With the command above, we stopped the searx uWSGI-App in the archlinux
278+container.
279+
280+The uWSGI-App for the archlinux dsitros is configured in
281+:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at
282+least you should attend the settings of ``uid``, ``chdir``, ``env`` and
283+``http``::
284+
285+ env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml
286+ http = 127.0.0.1:8888
287+
288+ chdir = /usr/local/searx/searx-src/searx
289+ virtualenv = /usr/local/searx/searx-pyenv
290+ pythonpath = /usr/local/searx/searx-src
291+
292+If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that
293+each container shares the root folder of the repository and the command
294+``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the
295+searx installation into a developer one, we simple have to create a smylink to
296+the **transparent** reposetory from the desktop. Now lets replace the
297+repository at ``searx-src`` in the container with the working tree from outside
298+of the container:
299+
300+.. tabs::
301+
302+ .. group-tab:: container becomes a developer suite
303+
304+ .. code:: sh
305+
306+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
307+ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
308+
309+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
310+ ln -s /share/searx/ /usr/local/searx/searx-src
311+
312+Now we can develop as usual in the working tree of our desktop system. Every
313+time the software was changed, you have to restart the searx service (in the
314+conatiner):
315+
316+.. tabs::
317+
318+ .. group-tab:: desktop
319+
320+ .. code:: sh
321+
322+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
323+ systemctl restart uwsgi@searx
324+
325+
326+Remember: :ref:`working in containers` .. here are just some examples from my
327+daily usage:
328+
329+.. tabs::
330+
331+ .. group-tab:: desktop
332+
333+ To *inspect* the searx instance (already described above):
334+
335+ .. code:: sh
336+
337+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
338+ ./utils/searx.sh inspect service
339+
340+ Run :ref:`makefile`, e.g. to test inside the container:
341+
342+ .. code:: sh
343+
344+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
345+ make test
346+
347+ To install all prerequisites needed for a :ref:`buildhosts`:
348+
349+ .. code:: sh
350+
351+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
352+ ./utils/searx.sh install buildhost
353+
354+ To build the docs on a buildhost :ref:`buildhosts`:
355+
356+ .. code:: sh
357+
358+ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
359+ make docs
360+
361+.. _blog-lxcdev-202006 abstract:
362+
363+Abstract
364+========
365+
366+We build up a fully functional searx suite in a archlinux container:
367+
368+.. code:: sh
369+
370+ $ sudo -H ./utils/lxc.sh install suite searx-archlinux
371+
372+To access HTTP from the desktop we installed nginx for the services inside the
373+conatiner:
374+
375+.. tabs::
376+
377+ .. group-tab:: [root@searx-archlinux]
378+
379+ .. code:: sh
380+
381+ $ ./utils/filtron.sh nginx install
382+ $ ./utils/morty.sh nginx install
383+
384+To wrap the suite into a developer one, we created a symbolic link to the
385+repository which is shared **transparent** from the desktop's file system into
386+the container :
387+
388+.. tabs::
389+
390+ .. group-tab:: [root@searx-archlinux]
391+
392+ .. code:: sh
393+
394+ $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
395+ $ ln -s /share/searx/ /usr/local/searx/searx-src
396+ $ systemctl restart uwsgi@searx
397+
398+To get remarks from the suite of the archlinux container we can use:
399+
400+.. tabs::
401+
402+ .. group-tab:: desktop
403+
404+ .. code:: sh
405+
406+ $ sudo -H ./utils/lxc.sh show suite searx-archlinux
407+ ...
408+ [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx
409+ [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/
410+ [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/
411+ [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65]
412+ ...
413+