112 KiB
This repository is a visual cheatsheet on the main topics in Backend-development. All the material is divided into topics and subtopics. The structure of the material consists of three parts:
- Visual part - various images/tables/cheatsheets for better understanding (may not be available). All pictures and tables are made from scratch, specifically for this repository.
- Summary - A very brief summary with a list of key terms and concepts. The terms are hyperlinked to the appropriate section on Wikipedia or a similar reference resource.
- References to sources - resources where you may find complete information on a particular issue. If possible, the most authoritative sources are indicated, or those that provide information in as simple and comprehensible language as possible.
🛠 The repository is under active development, so it is constantly updated and supplemented
🤝 If you want to help the project, feel free to send your Pull requests
📝 Translation into English is in progress
Contents
Additional and similar resources
Network & Internet
Internet is a worldwide system that connects computer networks from around the world into a single network for storing/transferring information. The Internet was originally developed for the military. But soon it began to be implemented in universities, and then it could be used by private companies, which began to organize networks of providers that provide Internet access services to ordinary citizens. By early 2020, the number of Internet users exceeded 4.5 billion.
-
How the Internet works
Your computer has never been directly connected to the Internet. Because it can only see its local network to which other devices are connected via wired (Ethernet) or wirelessly (Wi-Fi, Bluetooth). To communicate with the Internet, you have a special minicomputer in your local network - router. It then connects you to Internet Service Provider which in turn connects to other higher-level providers. Thus, your message, transits through the network of several ISPs before reaching the destination network.
The Internet is just a long wire to which a small number of Tier 1 providers are directly connected. The ISPs below that are just renting access.
🔗 References
-
What is a domain name
Domain Names are human-readable addresses of web servers available on the Internet. They consist of parts (levels) separated from each other by a dot. Each of these parts provides specific information about the domain name. For example country, service name, localization, etc.
- Who owns domain names
The ICANN Corporation is the founder of the distributed domain registration system. It gives accreditations to companies that want to sell domains. In this way a competitive domain market is formed.
- How to buy a domain name
A domain name cannot be bought forever. It is leased for a certain period of time. It is better to buy domains from accredited registrars (you can find them in almost any country).
- Who owns domain names
-
IP address
IP address is a unique numeric address that is used to recognize a particular device on the network.
- Levels of visibility
- External and publicly accessible IP address that belongs to your ISP and is used to access the Internet by hundreds of other users.
- The IP address of your router in your ISP's local network, the same IP address from which you access the Internet.
- The IP address of your computer in the local (home) network created by the router, to which you can connect your devices. Typically, it looks like 192.168.XXX.XXX.
- The internal IP address of the computer, inaccessible from the outside and used only for communication between the running processes. It is the same for everyone - 127.0.0.1 or just localhost.
- Port
One device (computer) can run many applications that use the network. In order to correctly recognize where and which data coming over the network should be delivered (to which of the applications) a special numerical number - a port is used. That is, each running process on a computer which uses a network connection has its own personal port.
- IPv4
Version 4 of the IP protocol. It was developed in 1981 and limits the address space to about 4.3 billion (2^32) possible unique addresses.
- IPv6
Over time, the allocation of address space began to happen at a much faster rate, forcing the creation of a new version of the IP protocol to store more addresses. IPv6 is capable of issuing 2^128 (is huge number) unique addresses.
- Levels of visibility
🔗 References
-
What is DNS
DNS (Domain Name System) is a decentralized Internet address naming system that allows you to create human-readable alphabetic names (domain names) corresponding to the numeric IP addresses used by computers.
🔗 References
-
Web application design
Modern web applications consist of two parts: Frontend and Backend. Thus implementing a client-server model.
The tasks of the Frontend are:
- Implementation of the user interface (appearance of the application)
A special markup language HTML is used to create web pages.
> CSS style language is used to style fonts, layout of content, etc.
> JavaScript programming language is used to add dynamics and interactivity.
As a rule, these tools are rarely used in their pure form, as so-called frameworks and preprocessors exist for more convenient and faster development. - Creating functionality for generating requests to the server
These are usually different types of input forms that can be conveniently interacted with.
- Receives data from the server and then processes it for output to the client
Tasks of the Backend:
- Handling client requests
Checking for permissions and access, all sorts of validations, etc.
- Implementing business logic
A wide range of tasks can be implied here: working with databases, information processing, computation, etc. This is, so to speak, the heart of the Backend world. This is where all the important and interesting stuff happens.
- Generating a response and sending it to the client
- Implementation of the user interface (appearance of the application)
🔗 References
-
Browsers and how they work
Browser is a client which can be used to send requests to a server for files which can then be used to render web pages. In simple terms, a browser can be thought of as a program for viewing HTML files, which can also search for and download them from the Internet.
🔗 References
-
VPN and Proxy
The use of VPNs and Proxy is quite common in recent years. With the help of these technologies, users can get basic anonymity when surfing the web, as well as bypass various regional blockages.
- VPN (Virtual Private Network)
A technology that allows you to become a member of a private network (similar to your local network), where requests from all participants go through a single public IP address. This allows you to blend in with the general mass of requests from other participants.
- Simple procedure for connection and use.
- Reliable traffic encryption.
- There is no guarantee of 100% anonymity, because the owner of the network knows the IP-addresses of all participants.
- VPNs are useless for dealing with multi-accounts and some programs because all accounts operating from the same VPN are easily detected and blocked.
- Free VPNs tend to be heavily loaded, resulting in unstable performance and slow download speeds.
- Simple procedure for connection and use.
- Proxy (proxy server)
A proxy is a special server on the network that acts as an intermediary between you and the destination server you intend to reach. When you are connected to a proxy server all your requests will be performed on behalf of that server, that is, your IP address and location will be substituted.
- The ability to use an individual IP address, which allows you to work with multi-accounts.
- Stability of the connection due to the absence of high loads.
- Connection via proxy is provided in the operating system and browser, so no additional software is required.
- There are proxy varieties that provide a high level of anonymity.
- The unreliability of free solutions, because the proxy server can see and control everything you do on the Internet.
- The ability to use an individual IP address, which allows you to work with multi-accounts.
- VPN (Virtual Private Network)
🔗 References
-
Hosting
Hosting is a special service provided by hosting providers, which allows you to rent space on a server (which is connected to the Internet around the clock), where your data and files can be stored. There are different options for hosting, where you can use not only the disk space of the server, but also the CPU power to run your network applications.
- Virtual hosting
One physical server that distributes its resources to multiple tenants.
- VPS/VDS
Virtual servers that emulate the operation of a separate physical server and are available for rent to the client with maximum privileges.
- Dedicated server
Renting a full physical server with full access to all resources. As a rule, this is the most expensive service.
- Cloud hosting
A service that uses the resources of several servers. When renting, the user pays only for the actual resources used.
- Colocation
A service that gives the customer the opportunity to install their equipment on the provider's premises.
- Virtual hosting
🔗 References
-
OSI network model
№ Level Used protocols 7 Application layer HTTP, DNS, FTP, POP3 6 Presentation layer SSL, SSH, IMAP, JPEG 5 Session layer APIs Sockets 4 Transport layer TCP, UDP 3 Network layer IP, ICMP, IGMP 2 Data link layer Ethernet, MAC, HDLC 1 Physical layer RS-232, RJ45, DSL - Physical layer
At this level, bits (ones/zeros) are encoded into physical signals (current, light, radio waves) and transmitted further by wire (Ethernet) or wirelessly (Wi-Fi).
- Data link layer
Physical signals from layer 1 are decoded back into ones and zeros, errors and defects are corrected, and the sender and receiver MAC addresses are extracted.
- Network layer
This is where traffic routing, DNS queries and IP packet generation take place.
- Transport layer
The layer responsible for data transfer. There are two important protocols:
- TCP is a protocol that ensures reliable data transmission. TCP guarantees data delivery and preserves the order of the messages. This has an impact on the transmission speed. This protocol is used where data loss is unacceptable, such as when sending mail or loading web pages.
- UDP is a simple protocol with fast data transfer. It does not use mechanisms to guarantee the delivery and ordering of data. It is used e.g. in online games where partial packet loss is not crucial, but the speed of data transfer is much more important. Also, requests to DNS servers are made through UDP protocol.
- TCP is a protocol that ensures reliable data transmission. TCP guarantees data delivery and preserves the order of the messages. This has an impact on the transmission speed. This protocol is used where data loss is unacceptable, such as when sending mail or loading web pages.
- Session layer
Responsible for opening and closing communications (sessions) between two devices. Ensures that the session stays open long enough to transfer all necessary data, and then closes quickly to avoid wasting resources.
- Presentation layer
Transmission, encryption/decryption and data compression. This is where data that comes in the form of zeros and ones are converted into desired formats (PNG, MP3, PDF, etc.)
- Application layer
Allows the user's applications to access network services such as database query handler, file access, email forwarding.
- Physical layer
🔗 References
-
HTTP Protocol
HTTP (HyperText Transport Protocol) is the most important protocol on the Internet. It is used to transfer data of any format. The protocol itself works according to a simple principle: request -> response.
- Structure of HTTP messages
Start Line > Headers > Message Body
- Headers
Additional service information that is sent with the request/response.
Common headers: Host, User-Agent, If-Modified-Since, Cookie, Referer, Authorization, Cache-Control, Content-Type, Content-Length, Last-Modified, Set-Cookie, Content-Encoding. - Request methods
GET - data retrieval request
POST - request with data to create a new record
PUT - request with data to change existing record
DELETE - deletion request
Others: HEAD, CONNECT, OPTIONS, TRACE, PATCH. - Response status codes
Each response from the server has a special numeric code that characterizes the state of the sent request. These codes are divided into 5 main classes:
- 1хх - service information
- 2хх - successful request
- 3хх - redirect to another address
- 4хх - client side error
- 5хх - server side error
- 1хх - service information
- HTTPS
Same HTTP, but with encryption support
- Cookie
Because the HTTP protocol does not allow you to save any information about the status of previous requests/responses, you need to use cookies. Cookies allow the server to store various information on the client side, which the client can then send back to the server. In particular, cookies can be used for authorization or to save various settings/configurations.
- CORS (Cross origin resource sharing)
A technology that allows one domain to securely receive data from another domain.
- CSP (Content Security Policy)
A special header that allows you to recognize and eliminate certain types of web application vulnerabilities.
- HTTP/1.0 vs HTTP/1.1 vs HTTP/2
The main innovation in version 1.1 is the permanent connection mode, which allows you to send several requests per connection. In version 2, the protocol became binary, with the ability to transmit data from multiple streams on the same channel.
- Structure of HTTP messages
🔗 References
- 📄 How HTTP Works and Why it's Important – freeCodeCamp
- 📄 Hypertext Transfer Protocol (HTTP) – MDN
- 📺 Hyper Text Transfer Protocol Crash Course – YouTube
- 📄 HTTP vs HTTPS – What's the Difference? – freeCodeCamp
- 📺 HTTP Cookies Crash Course – YouTube
- 📺 Cross Origin Resource Sharing (Explained by Example) – YouTube
- 📺 When to use HTTP GET vs POST? – YouTube
- 📺 How HTTP/2 Works, Performance, Pros & Cons and More – YouTube
- 📺 HTTP/2 Critical Limitation that led to HTTP/3 & QUIC – YouTube
- 📺 304 Not Modified HTTP Status (Explained with Code Example and Pros & Cons) – YouTube
- 📺 What is the Largest POST Request the Server can Process? – YouTube
-
TCP/IP stack
Compared to the OSI model, the TCP/IP stack has a simpler architecture. It is widely used and was first used as the basis for the creation of a global network, and then to describe the workings of the Internet.
🔗 References
-
Network problems
The quality of networks, much less the Internet, is far from ideal. This is due to the complex and dispersed network structure in different devices. Therefore, on the functioning of the network affects a huge number of factors. For example: the stability of the connection between the client device and its router, the quality of service of the provider, the power and performance of the server, the physical distance between the client and the server, etc.
- Latency
The time it takes for a data packet to travel from sender to receiver. It depends more on the physical distance.
- Packet loss
Not all packets traveling over the network can reach their destination. This happens most often when using wireless networks or due to network congestion.
- Round Trip Time (RTT)
The time it takes for the data packet to reach its destination + the time to respond that the packet was received successfully.
- Jitter
Delay fluctuations, unstable ping (for example, 50ms, 120ms, 35ms...).
- Packet reordering
The IP protocol does not guarantee that packets are delivered in the order in which they are sent.
- Latency
🔗 References
-
Network diagnostics
- Traceroute
A procedure that allows you to trace to which nodes, with which IP addresses, a packet you send before it reaches its destination. Tracing can be used to identify computer network related problems and to examine/analyze the network.
- Ping scan
The easiest way to check the server for performance.
- Checking for packet loss
Due to dropped connections, not all packets sent over the network reach their destination.
- Wireshark
A powerful program with a graphical interface for analyzing all traffic that passes through the network in real time.
- Traceroute
🔗 References
PC device
-
Main components (hardware)
- Motherboard
The most important PC component to which all other elements are connected.
- Chipset - set of electronic components that responsible for the communication of all motherboard components.
- CPU socket - socket for mounting the processor.
- VRM (Voltage Regulator Module) – module that converts the incoming voltage (usually 12V) to a lower voltage to run the processor, integrated graphics, memory, etc.
- Slots for RAM.
- Expansion slots PCI-Express - designed for connection of video cards, external network/sound cards.
- Slots М.2 / SATA - designed to connect hard disks and SSDs.
- CPU (Central processing unit)
The most important device that executes instructions (programme code). Processors only work with 1 and 0, so all programmes are ultimately a set of binary code.
- Registers - the fastest memory in a PC, has an extremely small capacity, is built into the processor and is designed to temporarily store the data being processed.
- Cache - slightly less fast memory, which is also built into the processor and is used to store a copy of data from frequently used cells in the main memory.
- Processors can have different architectures. Currently, the most common are the x86 architecture (desktop and laptop computers) and ARM (mobile devices as well as the latest Apple computers).
- RAM (Random-access memory)
Fast, low capacity memory (4-16GB) designed to temporarily store program code, as well as input, output and intermediate data processed by the processor.
- Data storage
Large capacity memory (256GB-1TB) designed for long-term storage of files and installed programmes.
- GPU (Graphics card)
A separate card that translates and processes data into images for display on a monitor. This device is also called a discrete graphics card. Usually needed for those who do 3D modelling or play games.
> Built-in graphics card is a graphics card built into the processor. It is suitable for daily work. - Network card
A device that receives and transmits data from other devices connected to the local network.
- Sound card
A device that allows you to process sound, output it to other devices, record it with a microphone, etc.
- Power supply unit
A device designed to convert the AC voltage from the mains to DC voltage.
- Motherboard
🔗 References
- 📄 Everything You Need to Know About Computer Hardware
- 📺 What does what in your computer? Computer parts Explained – YouTube
- 📺 The Fetch-Execute Cycle: What's Your Computer Actually Doing? – YouTube
- 📺 How a CPU Works in 100 Seconds // Apple Silicon M1 vs Intel i9 – YouTube
- 📺 Arm vs x86 - Key Differences Explained – YouTube
-
Operating system design
Operating system (OS) is a comprehensive software system designed to manage a computer's resources. With operating systems, people do not have to deal directly with the processor, RAM or other parts of the PC.
OS can be thought of as an abstraction layer that manages the hardware of a computer, thereby providing a simple and convenient environment for user software to run.
- Main features
- RAM management (space allocation for individual programms)
- Loading programms into RAM and their execution
- Execution of requests from user's programms (inputting and outputting data, starting and stopping other programms, freeing up memory or allocating additional memory, etc.)
- Interaction with input and output devices (mouse, keyboard, monitor, etc.)
- Interaction with storage media (HDDs and SSDs)
- Providing a user's interface (console shell or graphical interface)
- Logging of software errors (saving logs)
- Additional functions (may not be available in all OSs)
- Organise multitasking (simultaneous execution of several programms)
- Delimiting access to resources for each process
- Inter-process communication (data exchange, synchronisation)
- Organise the protection of the operating system itself against other programms and the actions of the user
- Provide multi-user mode and differentiate rights between different OS users (admins, guests, etc.)
- OS kernel
The central part of the operating system which is used most intensively. The kernel is constantly in memory, while other parts of the OS are loaded into and unloaded from memory as needed.
- Bootloader
The system software that prepares the environment for the OS to run (puts the hardware in the right state, prepares the memory, loads the OS kernel there and transfers control to it (the kernel).
- Device drivers
Special software that allows the OS to work with a particular piece of equipment.
- Main features
🔗 References
-
Processes and threads
- Process
A kind of container in which all the resources needed to run a program are stored. As a rule, the process consists of:
- Executable program code
- Input and output data
- Call stack (order of instructions for execution)
- Heap (a structure for storing intermediate data created during the process)
- Segment descriptor
- File descriptor
- Information about the set of permissible powers
- Processor status information
- Executable program code
- Thread
An entity in which sequences of program actions (procedures) are executed. Threads are within a process and use the same address space. There can be multiple threads in a single process, allowing multiple tasks to be performed. These tasks, thanks to threads, can exchange data, use shared data or the results of other tasks.
- Process
🔗 References
-
Concurrency and parallelism
- Parallelism
The ability to perform multiple tasks simultaneously using multiple processor cores, where each individual core performs a different task.
- Concurrency
The ability to perform multiple tasks, but using a single processor core. This is achieved by dividing tasks into separate blocks of commands which are executed in turn, but switching between these blocks is so fast that for users it seems as if these processes are running simultaneously.
- Parallelism
🔗 References
-
Inter-process communication
A mechanism which allows to exchange data between threads of one or different processes. Processes can be run on the same computer or on different computers connected by a network. Inter-process communication can be done in different ways.
- File
The easiest way to exchange data. One process writes data to a certain file, another process reads the same file and thus receives data from the first process.
- Signal (IPC)
Asynchronous notification of one process about an event which occurred in another process.
- Network socket
In particular, IP addresses and ports are used to communicate between computers using the TCP/IP protocol stack. This pair defines a socket (socket corresponding to the address and port).
- Semaphore
A counter over which only 2 operations can be performed: increasing and decreasing (and for 0 the decreasing operation is blocked).
- Message passing & Message queue
- Pipelines
Redirecting the output of one process to the input of another (similar to a pipe).
- File
🔗 References
Linux Basics
Operating systems based on Linux kernel are the standard in the world of server development, since most servers run on such operating systems. Using Linux on servers is profitable because it is free.
There are a huge number of Linux distributions (preinstalled software bundles) to suit all tastes. One of the most popular is Ubuntu. This is where you can start your dive into server development.
Install Ubuntu on a separate PC or laptop. If this is not possible, you can use a special program Virtual Box where you can run other OS on top of the main OS. You can also run Docker Ubuntu image container (Docker is a separate topic that is exists in this repository).
-
Working with the terminal
Terminal is a program that uses special text commands to control your computer. Generally, servers do not have graphical interfaces, so you will definitely need terminal skills.
- Basic commands for navigating the file system
ls # list directory contents cd <path> # go to specified directory cd .. # move to a higher level (to the parent directory) touch <file> # create a file cat > <file> # enter text into the file (overwrite) cat >> <file> # enter text at the end of the file (append) cat/more/less <file> # to view the file contents head/tail <file> # view the first/last lines of a file pwd # print path to current directory mkdir <name> # create a directory rmdir <name> # delete a directory cp <file> <path> # copy a file or directory mv <file> <path># moving or renaming rm <file> # deleting a file or directory find <string># file system search du <file># output file or directory size - Commands for help information
man <command> # allows you to view a manual for any command apropos <string> # search for a command with a description that has a specified word man -k <string> # similar to the command above whatis <command> # a brief description of the command - Super user rights
Analogue to running as administrator in Windows
sudo <command> # executes a command with superuser privileges - Text editor
Study any in order to read and edit files freely through the terminal. The easiest – nano. The most advanced – Vim.
- Basic commands for navigating the file system
🔗 References
-
Package manager
The Package Manager is a utility that allows you to install/update software packages from the terminal.
Linux distributions can be divided into several groups, depending on which package manager they use: apt (in Debian based distributions), RPM (the Red Hat package management system) and Pacman (the package manager in Arch-like distributions)
Ubuntu is based on Debian, so it uses apt (advanced packaging tool) package manager.
- Basic Commands
apt install <package> # install the package apt remove <package> # remove the package, but keep the configuration apt purge <package> # remove the package along with the configuration apt update # update information about new versions of packages apt upgrade # update the packages installed in the system apt list --installed # list of packages installed on the system apt list --upgradable # list of packages that need to be updated apt search <package> # searching for packages by name on the network apt show <package> # package information
- Basic Commands
🔗 References
-
Bash scripts
You can use scripts to automate the sequential input of any number of commands. In Bash you can create different conditions (branching), loops, timers, etc. to perform all kinds of actions related to console input.
- Basics of Bash Scripts
The most basic and frequently used features such as: variables, I/O, loops, conditions, etc.
- Practice
Solve challenges on sites like HackerRank and Codewars. Start using Bash to automate routine activities on your computer. If you're already a programmer, create scripts to easily build your project, to install settings, and so on.
- ShellCheck script analysis tool
It will point out possible mistakes and teach you best practices for writing really good scripts.
- Additional resources
Repositories such as awesome bash and awesome shell have entire collections of useful resources and tools to help you develop even more skills with Bash and the terminal in general.
- Basics of Bash Scripts
🔗 References
-
Users and groups
Linux-based operating systems are multi-user. This means that several people can run many different applications at the same time on the same computer. For the Linux system to be able to "recognize" a user, he must be logged in and therefore each user must have a unique name and a secret password.
- Working with users
useradd <name> [flags] # create a new user passwd <name> # set a password for the user usermod <name> [flags] # edit a user usermod -L <name> # block a user usermod -U <name> # unblock a user userdel <name> [flags] # delete a user - Working with groups
groupadd <group> [flags] # create a group groupmod <group> [flags] # edit group groupdel <group> [flags] # delete group usermod -a -G <groups> <user> # add a user to groups gpasswd --delete <user> <groups> # remove a user from groups - System files
/etc/passwd # a file containing basic information about users /etc/shadow # a file containing encrypted passwords /etc/group # a file containing basic information about groups /etc/gshadow # a file containing encrypted group passwords
- Working with users
🔗 References
-
Permissions
In Linux, it is possible to share privileges between users, limit access to unwanted files or features, control available actions for services, and much more. In Linux, there are only three kinds of rights - read, write and execute - and three categories of users to which they can be applied - file owner, file group and everyone else.
- Basic commands for working with rights
chown <user> <file> # changes the owner and/or group for the specified files chmod <rights> <file> # changes access rights to files and directories chgrp <group> <file> # allows users to change groups - Extended rights SUID and GUID, sticky bit
- ACL (Access control list)
An advanced subsystem for managing access rights.
- Basic commands for working with rights
🔗 References
-
Working with processes
Linux processes can be described as containers in which all information about the state of a running program is stored. If a program hangs and you need to restore it, then you need the skills to manage the processes.
- Basic Commands
ps # display a snapshot of the processes of all users top # real-time task manager <command> & # running the process in the background, (without occupying the console) jobs # list of processes running in the background fg <PID> # return the process back to the active mode by its number bg <PID> # start a stopped process in the background kill <PID> # terminate the process by PID killall <programm> # terminate all processes related to one program
- Basic Commands
🔗 References
-
Working with SSH
SSH allows remote access to another computer's terminal. In the case of a personal computer, this may be needed to solve an urgent problem, and in the case of a server, it is generally the primary method of connection.
- Basic commands
apt install openssh-server # installing SSH (out of the box almost everywhere) service ssh start # start SSH service ssh stop # stop SSH ssh -p <port> user@remote_host # connecting to a remote PC via SSH ssh-keygen -t rsa # RSA key generation for passwordless login ssh-copy-id -i ~/.ssh/id_rsa user@remote_host # copying a key to a remote machine
- Basic commands
🔗 References
-
Task Scheduler
Schedulers allow you to flexibly manage the delayed running of commands and scripts. Linux has a built-in cron scheduler that can be used to easily perform necessary actions at certain intervals.
-
Main commands
crontab -e # edit the crontab file of the current user crontab -l # output the contents of the current schedule file crontab -r # deleting the current schedule file -
Config files
/etc/crontab # base config /etc/cron.d/ # crontab files used to manage the entire system # config files for automatically run programs: /etc/cron.daily/ # every day /etc/cron.weekly/ # every week /etc/cron.monthly/ # every month
-
🔗 References
-
System logs
Log files are special text files that contain all information about the operation of a computer, program, or user. They are especially useful when bugs and errors occur in the operation of a program or server. It is recommended to periodically review log files, even if nothing suspicious happens.
- Main log files
/var/log/syslog или /var/log/messages # information about the kernel, # various services detected, devices, network interfaces, etc. /var/log/auth.log или /var/log/secure # user authorization information /var/log/faillog # failed login attempts /var/log/dmesg # information about device drivers /var/log/boot.log # operating system boot information /var/log/cron # cron task scheduler report - lnav utility
Designed for easy viewing of log files (highlighting, reading different formats, searching, etc.)
- Log rotation with logrotate
Allows you to configure automatic deletion (cleaning) of log files so as not to clog memory.
- Demon journald
Collects data from all available sources and stores it in binary format for convenient and dynamic control
- Main log files
🔗 References
-
Linux problems
- Problems with commands in the terminal
Occur due to erroneous actions of the user. Often associated with typos, lack of rights, incorrectly specified options, etc.
- Driver problems
All free Linux drivers are built right into its kernel. Therefore, everything should work "out of the box" after installing the system (problems may occur with brand new hardware which has just been released on the market). Drivers whose source code is closed are considered proprietary and are not included in the kernel but are installed manually (like Nvidia graphics drivers).
- Problems with kernel
Kernel panic can occur due to an error when mounting the root file system. This is best helped by the skill of reading the logs to find problems (
dmesgcommand). - Segmentation fault
Occurs when a process accesses invalid memory locations.
- Disk and file system problems
Can occur due to lack of space.
- Problems with commands in the terminal
🔗 References
General knowledge
-
Numeral systems
Numeral system is a set of symbols and rules for denoting numbers. In computer science, it is customary to distinguish four main number systems: binary, octal, decimal, and hexadecimal. It is connected, first of all, with their use in various branches of programming.
- Binary number
The most important system for computing technology. Its use is justified by the fact that the logic of the processor is based on only two states (on/off, open/closed, high/low, true/false, yes/no, high/low).
- Octal
It is used e.g. in Linux systems to grant access rights.
- Decimal
A system that is easy to understand for most people.
- Hexadecimal
The letters A, B, C, D, E, F are additionally used for recording. It is widely used in low-level programming and computer documentation because the minimum addressable memory unit is an 8-bit byte, the values of which are conveniently written in two hexadecimal digits.
- Translation between different number systems
You can try online converter for a better understanding.
- Binary number
🔗 References
-
Logical connective
Logical connective are widely used in programming to check various conditions. The result of a logical expression is always truth or false.
🔗 References
-
Data structures
Data structures are containers in which data is stored according to certain rules. Depending on these rules, the data structure will be effective in some tasks and ineffective in others. Therefore, it is necessary to understand when and where to use this or that structure.
- Array
A data structure that allows you to store data of the same type, where each element is assigned a different sequence number.
- Linked list
A data structure where all elements, in addition to the data, contain references to the next and/or previous element. There are 3 varieties:
- A singly linked list is a list where each element stores a link to the next element only (one direction).
- A doubly linked list is a list where the items contain links to both the next item and the previous one (two directions).
- A circular linked list is a kind of bilaterally linked list, where the last element of the ring list contains a pointer to the first and the first to the last.
- Stack
Structure where data storage works on the principle of last in - first out (LIFO).
- Queue
Structure where data storage is based on the principle of first in - first out (FIFO).
- Hash table
In other words, it is an associative array. Here, each of the elements is accessed with a corresponding key value, which is calculated using hash function according to a certain algorithm.
- Tree
Structure with a hierarchical model, as a set of related elements, usually not ordered in any way.
- Heap
Similar to the tree, but in the heap, the items with the largest key is the root node (max-heap). But it may be the other way around, then it is a min heap.
- Graph
A structure that is designed to work with a large number of links.
- Array
🔗 References
-
Basic algorithms
Algorithms refer to sets of sequential instructions (steps) that lead to the solution of a given problem. Throughout human history, a huge number of algorithms have been invented to solve certain problems in the most efficient way. Accordingly, the correct choice of algorithms in programming will allow you to create the fastest and most resource-intensive solutions.
There is a very good book on algorithms for beginners – Grokking algorithms. You can start learning a programming language in parallel with it.
- Binary search
Maximum efficient search algorithm for sorted lists.
- Selection sort
At each step of the algorithm, the minimum element is searched for and then swapped with the current iteration element.
- Recursion
When a function can call itself and so on to infinity. On the one hand, recursion-based solutions look very elegant, but on the other hand, this approach quickly leads to stack overflow and is recommended to be avoided.
- Bubble sort
At each iteration neighboring elements are sequentially compared, and if the order of the pair is wrong, the elements are swapped.
- Quicksort
Improved bubble sorting method.
- Breadth-first search
Allows to find all shortest paths from a given vertex of the graph.
- Dijkstra's algorithm
Finds the shortest paths between all vertices of a graph and their length.
- Greedy algorithm
An algorithm that at each step makes locally the best choice in the hope that the final solution will be optimal.
- Binary search
🔗 References
-
Algorithm complexity
In the world of programming there is a special unit of measure Big O (or O-notation). It describes how the complexity of an algorithm increases with the amount of input data. Big O estimates how many actions (steps/iterations) it takes to execute the algorithm, while always showing the worst case scenario.
- Varieties of algorithm complexity
- Constant - O(1)
- Linear - O(n)
- Logarithmic - O(log n)
- Linearimetric - O(n * log n)
- Quadratic - O(n^2)
- Stepwise - О(2^n)
- Factorical - O(!n)
- Constant - O(1)
- Varieties of algorithm complexity
-
Data storage formats
Different file formats can be used to store and transfer data over the network. Text files are human-readable, so they are used for configuration files, for example. But transferring data in text formats over the network is not always rational, because they weigh more than their corresponding binary files.
-
Text formats
- JSON (JavaScript Object Notation)
Represents an object in which data is stored as key-value pairs.
- YAML (Yet Another Markup Language)
The format is close to markup languages like HTML. Minimalist, because it has no opening or closing tags. Easy to edit.
- XML (eXtensible Markup Language)
The format is closer to HTML. Here the data is wrapped in opening and closing tags.
- JSON (JavaScript Object Notation)
-
Binary formats
- Message Pack
Binary analog of JSON. Allows you to pack data 15-20% more efficiently.
- BSON (Binary JavaScript Object Notation)
It is a superset of JSON, including additionally regular expressions, binary data and dates.
- ProtoBuf (Protocol Buffers)
Binary alternative to XML text format. Simpler, more compact and faster.
- Message Pack
-
🔗 References
-
Text encodings
Computers work only with numbers, or more precisely, only with 0 and 1. It is already clear how to convert numbers from different number systems to binary. But you can't do that with text. That's why special tables called encodings were invented, in which text characters are assigned numeric equivalents.
- ASCII (American standard code for information interchange)
The simplest encoding created specifically for the American alphabet. Consists of 128 characters.
- Unicode
This is an international character table that, in addition to the English alphabet, contains the alphabets of almost all countries. It can hold more than a million different characters (the table is currently incomplete).
- UTF-8
Unicode is a variable-length encoding that can be used to represent any unicode character.
- UTF-16
Its main difference from UTF-8 is that its structural unit is not one but two bytes. That is, in UTF-16 any Unicode character can be encoded by either two or four bytes.
- ASCII (American standard code for information interchange)
🔗 References
Programming Language
At this stage you have to choose one programming language to study. There is plenty of information on various languages in the Internet (books, courses, thematic sites, etc.), so you should have no problem finding information.
Below is a list of specific languages that personally, in my opinion are good for backend development (⚠️ may not agree with the opinions of others, including those more competent in this matter).
- Python
A very popular language with a wide range of applications. Easy to learn due to its simple syntax.
- JavaScript
No less popular and practically the only language for full-fledged Web-development. Thanks to the platform Node.js last few years is gaining popularity in the field of backend development as well.
- Go
A language created internally by Google. It was created specifically for high-load server development. Minimalistic syntax, high performance and rich standard library.
- Kotlin
A kind of modern version of Java. Simpler and more concise syntax, better type-safety, built-in tools for multithreading. One of the best choices for Android development.
Find a good book or online tutorial in English at this repository. There is a large collection for different languages and frameworks.
Look for a special awesome repository - a resource that contains a huge number of useful links to materials for your language (libraries, cheat sheets, blogs and other various resources).
-
Classification of programming languages
There are many programming languages. They are all created for a reason. Some languages may be very specific and used only for certain purposes. Also, different languages may use different approaches to writing programs. They may even run differently on a computer. In general, there are many different classifications, which would be useful to understand.
- Depending on language level
- Low level languages
As close to machine code as possible, complex to write, but as productive as possible. As a rule, it provides access to all of the computer's resources.
- High-level languages
They have a fairly high level of abstraction, which makes them easy to write and easy to use. As a rule, they are safer because they do not provide access to all of the computer's resources.
- Low level languages
- Compiled, interpreted and embedded languages
- Compilation
Allows you to convert the source code of a program to an executable file.
- Interpretation
The source code of a program is translated and immediately executed (interpreted) by a special interpreter program.
- Compilation
- Depending on the programming paradigm
- Depending on language level
🔗 References
-
Language Basics
By foundations are meant some fundamental ideas present in every language.
- Variables and constants
- Data types
Strings, integers, floats, booleans, etc.
- Operators
Mathematical operators, comparison operators, bitwise operators.
- Functions
Working with arguments and return values.
Understanding the scope of variables. - Flow control
Cycles for, conditions if else, switch-case statement.
- Data structures
Arrays, objects, classes, etc.
- Standard Library
This refers to the language's built-in capabilities to manipulate strings, numbers, arrays, etc.
- Regular expressions
A powerful tool for working with strings. Be sure to familiarize yourself with it in your language, at least on a basic level.
- Package Manager
Sooner or later, there will be a desire to use third-party libraries.
After mastering the minimal base for writing the simplest programs, there is not much point in continuing to learn without having specific goals (without practice, everything will be forgotten). You need to think of/find something that you would like to create yourself (a game, a chatbot, a website, a mobile/desktop application, whatever). For inspiration, check out these repositories: Build your own x and Project based learning.
At this point, the most productive part of learning begins: You just look for all kinds of information to implement your project. Your best friends are Google, YouTube, and Stack Overflow.
🔗 References
- 📄 Free Interactive Python Tutorial
- 📺 Python Tutorial for Beginners – YouTube
- 📄 Python cheatsheet – Learn X in Y minutes
- 📄 Python cheatsheet – quickref.me
- 📄 Free Interactive JavaScript Tutorial
- 📺 JavaScript Programming - Full Course – YouTube
- 📄 The Modern JavaScript Tutorial
- 📄 JavaScript cheatsheet – Learn X in Y minutes
- 📄 JavaScript cheatsheet – quickref.me
- 📄 Go Tour – learn most important features of the language
- 📺 Learn Go Programming - Golang Tutorial for Beginners – YouTube
- 📄 Go cheatsheet – Learn X in Y minutes
- 📄 Go cheatsheet – quickref.me
- 📄 Learn Go by Examples
- 📄 Get started with Kotlin
- 📺 Learn Kotlin Programming – Full Course for Beginners – YouTube
- 📄 Kotlin cheatsheet – Learn X in Y minutes
- 📄 Kotlin cheatsheet – devhints.io
- 📄 Learn Regex step by step, from zero to advanced
-
Server development
- Creating and running a local HTTP server
- Handing out static files
Hosting HTML pages, pictures, PDFs, etc.
- Routing
Creation of endpoints (URLs) which will call the appropriate handler on the server when accessed.
- Processing requests
As a rule, HTTP handlers have a special object which receives all information about user request (headers, method, request body, full url with parameters, etc.)
- Processing responses
Sending an appropriate message to a received request (HTTP status and code, response body, headers, etc.)
- Error handling
You should always consider cases where the user could send invalid data, the database failed to execute the operation, or an unexpected error occurred in the application, so that the server does not crash but responds with an error message.
- Sending requests
Often, within one application, you will need to access another application over the network. That's why it's important to be able to send HTTP requests using the built-in features of the language.
- Template processor
Is a special module that uses a more convenient syntax to generate HTML based on dynamic data.
🔗 References
- 📄 Learn Django – Python-based web framework
- 📺 Python Django 7 Hour Course – YouTube
- 📄 A curated list of awesome things related to Django – GitHub
- 📺 Build servers in pure Node.js – YouTube
- 📄 Learn Express – web framework for Node.js
- 📺 Express.js 2022 Course – YouTube
- 📄 A curated list of awesome Express.js resources – GitHub
- 📄 How to build servers in Go
- 📺 Golang server development course – YouTube
- 📄 List of libraries for working with network in Go – GitHub
- 📄 Learn Ktor – web framework for Kotlin
- 📺 Ktor - REST API Tutorials – YouTube
- 📄 Kotlin for server side
-
Multithreading
Computers today have processors with several physical and virtual cores, and if we take into account server machines, their number can reach up to hundreds. All of these available resources would be good to use to the fullest, for maximum application performance. That is why modern server development cannot do without implementing multithreading and paralleling.
- Race conditions & data races
The main problems that arise when using multithreading.
- Creating processes
- Creating threads
- Corutines
Lightweight code execution threads organized on top of operating system threads. They can exist as separate libraries or be already built into the kernel.
- Linearizability
Operations that are performed completely, or not performed at all.
- Lockouts
Using semaphores and mutexes to synchronize data.
- Race conditions & data races
- 📺 Multithreading Code - Computerphile – YouTube
- 📺 Threading vs multiprocessing in Python – YouTube
- 📺 When is NodeJS Single-Threaded and when is it Multi-Threaded? – YouTube
- 📺 How to use Multithreading with "worker threads" in Node.js? – YouTube
- 📺 Concurrency in Go – YouTube
- 📺 Kotlin coroutines – YouTube
-
Advanced Topics
- Garbage collector
A process that has made high-level languages very popular - it allows the programmer not to worry about memory allocation and freeing. Be sure to familiarize yourself with the subtleties of its operation in your own language.
- Debuger
Handy tool for analyzing program code and identifying errors.
- Garbage collector
🔗 References
-
Code quality
During these long years that programming has existed, a huge amount of code, programs and entire systems have been written. And as a consequence, there have been all sorts of problems in the development of all this. First of all they were related to scaling, support, and the entry threshold for new developers. Clever people, of course, did not sit still and started to solve these problems, thus creating so-called patterns/principles/approaches for writing high-quality code.
By learning programming best practices, you will not only make things better for yourself, but also for others, because other developers will be working with your code.
Databases
Databases (DB) – a set of data that are organized according to certain rules (for example, a library is a database for books).
Database management system (DBMS) is a software that allows you to create a database and manipulate it conveniently (perform various operations on the data). An example of a DBMS is a librarian. He can easily and efficiently work with the books in the library: give out requested books, take them back, add new ones, etc.
-
Database classification
Databases can differ significantly from each other and therefore have different areas of application. To understand what database is suitable for this or that task, it is necessary to understand the classification.
- Relational DB
These are repositories where data is organized as a set of tables (with rows and columns). Interactions between data are organized on the basis of links between these tables. This type of database provides fast and efficient access to structured information.
- Object-oriented DB
Here data is represented as objects with a set of attributes and methods. Suitable for cases where you need high-performance processing of data with a complex structure.
- Distributed DB
Composed of several parts located on different computers (servers). Such databases may completely exclude information duplication, or completely duplicate it in each distributed copy (for example, as Blockchain).
- NoSQL
Stores and processes unstructured or weakly structured data. This type of database is subdivided into subtypes:
- Key–value DB
- Column family DB
- Document-oriented DB (store data as a hierarchy of documents)
- Graph DB (are used for data with a large number of links)
- Key–value DB
- Relational DB
🔗 References
-
Relational database
The most popular relational databases: MySQL, PostgreSQL, MariaDB, Oracle. A special language SQL (Structured Query Language) is used to work with these databases. It is quite simple and intuitive.
- SQL basics
Learn the basic cycle of creating/receiving/updating/deleting data. Everything else as needed.
- Merging tables
- Querying data from multiple tables
Operator
JOIN; Combinations with other operators;JOINtypes. - Relationships between tables
References from one table to another; foreign keys.
- Querying data from multiple tables
- Subquery Expressions
Query inside another SQL query.
- Indexes
Data structure that allows you to quickly determine the position of the data of interest in the database.
- Transactions
Sequences of commands that must be executed completely, or not executed at all.
- Command
START TRANSACTION - Commands
COMMITandROLLBACK
- Command
- Working with a programming language
To do this, you need to install a special driver for your language. For more convenient operation, there are ORM libraries that allow you to execute SQL queries as if you were simply calling methods on the object.
- Optimization and performance
- SQL basics
🔗 References
- 📺 SQL Crash Course - Beginner to Intermediate – YouTube
- 📺 SQL Tutorial - Full Database Course for Beginners – YouTube
- 📺 MySQL - The Basics. Learn SQL in 23 Easy Steps – YouTube
- 📄 MySQL command-line client commands
- 📺 Learn PostgreSQL Tutorial - Full Course for Beginners – YouTube
- 📄 Postgres Cheat Sheet
- 📺 Database Indexing Explained (with PostgreSQL) – YouTube
- 📺 What is a Database transaction? – YouTube
- 📺 SQL Server Performance Essentials – Full Course – YouTube
-
MongoDB
MongoDB is a document-oriented database (which is a classic example of NoSQL databases) that does not require a table schema description. Uses JSON-like documents and database schema.
- Basic commands
Learn the basic cycle of creating/receiving/updating/deleting data. Everything else as needed.
- Aggregations
Grouping values from multiple documents.
Three ways to perform aggregation: pipeline, map-reduce, and single-purpose aggregation methods. - Working with Indexes
- Working with a programming language
For this you need to install MongoDB driver for your language.
- Basic commands
🔗 References
-
Redis
Redis is a fast data storage working with key-value structures. It can be used as a database, cache, message broker or queue.
- Data types
String / Bitmap / Bitfield / List / Set / Hash / Sorted sets / Geospatial / Hyperlog / Stream
- Basic operations
SET key "value" # setting the key with the value "value" GET key # retrieve a value from the specified key SETNX key "data" # setting the value / creation of a key MSET key1 "1" key2 "2" key3 "3" # setting multiple keys MGET key1 key2 key3 # getting values for several keys at once DEL key # remove the key-value pair INCR someNumber # increase the numeric value by 1 DECR someNumber # decrease the numeric value by 1 EXPIRE key 1000 # set a key life timer of 1000 seconds TTL key # get information about the lifetime of the key-value pair # -1 the key exists, but has no expiration date # -2 the key does not exist # <another number> key lifetime in seconds SETEX key 1000 "value" # consolidation of commands SET and EXPIRE - Transactions
MULTI— start recording commands for the transaction.
EXEC— execute the recorded commands.
DISCARD— delete all recorded commands.
WATCH— command that provides execution only if other clients have not changed the value of the variable. Otherwise EXEC will not execute the written commands.
- Data types
🔗 References
-
ACID Requirements
ACID is an acronym consisting of the names of the four main properties that guarantee the reliability of transactions in the database.
- Atomicity
Guarantees that the transaction will be executed completely or not executed at all.
- Consistency
Ensures that each successful transaction captures only valid results (any inconsistencies are excluded).
- Isolation
Guarantees that one transaction cannot affect the other in any way.
- Durability
Guarantees that the changes made by the transaction are saved.
- Atomicity
🔗 References
-
Designing databases
Database design is a very important topic that is often overlooked. A well-designed database will ensure long-term scalability and ease of data maintenance. There are several basic steps in database design:
- Conceptual schema
The most abstract vision of the entire system is a description of the basic entities (tables) and the relationships between them without focusing on a particular database management system.
- Logic schema
Work with data models of the future database, the designation of necessary fields with the indication of types, the definition of relationships and relationships with the indication of the primary and external keys.
- Real implementation
This is the direct work with the DBMS, the organization of specific tables, the implementation of relationships, the creation of indexes, and so on.
- Conceptual schema
🔗 References
API development
API (Application Programming Interface) an interface which describes a certain set of rules by which different programs (applications, bots, websites...) can interact with each other. With API calls you can execute certain functions of a program without knowing how it works.
When developing server applications, different API formats can be used, depending on the tasks and requirements.
-
REST API
REST (Representational State Transfer) an architectural approach that describes a set of rules for how a programmer organizes the writing of server application code so that all systems can easily exchange data and the application can be easily scaled. When building a REST API, HTTP protocol methods are widely used.
Basic rules for writing a good REST API:
- Each URL-endpoint must be a noun
To perform different operations (get/create/update/delete), this endpoint must implement handlers for the corresponding HTTP methods (GET/POST/PUT/DELETE).
- Use of plural names
For example, the endpoint to get one record by id looks like this:
/task/42, and to get all records like this:/tasks. - Using versioning
Detailed instructions are described here.
- Sending the appropriate HTTP response codes
The most commonly used: 200, 201, 204, 304, 400, 401, 403, 404, 405, 410, 415, 422, 429.
- Each URL-endpoint must be a noun
🔗 References
-
GraphQL
🔗 References
-
WebSockets
🔗 References
-
RPC and gRPC
🔗 References
-
WebRTC
🔗 References
Software
-
Git version control system
🔗 References
-
Docker
🔗 References
-
Postman/Insomnia
🔗 References
-
Web servers
🔗 References
-
Message brokers
🔗 References
Security
-
Web application vulnerabilities
🔗 References
-
Environment variables
🔗 References
-
Hashing
🔗 References
-
Authentication and authorization
🔗 References
-
SSL/TLS
🔗 References
Testing
-
Unit Tests
🔗 References
-
Integration tests
🔗 References
-
E2E tests
🔗 References
-
Load testing
🔗 References
-
Regression testing
🔗 References
Optimization
-
Profiling
🔗 References
-
Caching
🔗 References
-
Load balancing
🔗 References
Documentation
-
Markdown
🔗 References
-
Documentation inside code
🔗 References
-
API Documentation
🔗 References
-
Static generators
🔗 References
Building architecture
-
Architectural templates
🔗 References
-
Design patterns
🔗 References
-
Monolithic and microservice architecture
🔗 References
-
Horizontal and vertical scaling
🔗 References
Additional and similar resources
- Backend Developer Roadmap: Learn to become a modern backend developer
- Hussein Nasser – YouTube channel about networking
- CS50 2022 – Harvard University's course about programming
- A curated and opinionated list of resources (English & Russian) for Backend developers