mirror of
https://github.com/borgbackup/borg.git
synced 2024-09-11 12:36:12 +03:00
[DOCS] #4587 – Make Sphinx warnings break docs build
general.rst and man_intro.rst both included usage_general.rst.inc, which resulted in three Sphinx warning "WARNING: duplicate label". To prevent this we move all sections of usage_general into own include files and add a second usage_general file without the three labels.
This commit is contained in:
parent
62ace206d6
commit
19aa9825a8
@ -16,7 +16,7 @@ of values (e.g. ``--encryption`` of :ref:`borg_init`).
|
||||
Experimental features are not stable, which means that they may be changed in incompatible
|
||||
ways or even removed entirely without prior notice in following releases.
|
||||
|
||||
.. include:: ../usage_general.rst.inc
|
||||
.. include:: usage_general.rst.inc
|
||||
|
||||
In case you are interested in more details (like formulas), please see
|
||||
:ref:`internals`. For details on the available JSON output, refer to
|
||||
|
10
docs/usage/general/date-time.rst.inc
Normal file
10
docs/usage/general/date-time.rst.inc
Normal file
@ -0,0 +1,10 @@
|
||||
Date and Time
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
We format date and time conforming to ISO-8601, that is: YYYY-MM-DD and
|
||||
HH:MM:SS (24h clock).
|
||||
|
||||
For more information about that, see: https://xkcd.com/1179/
|
||||
|
||||
Unless otherwise noted, we display local date and time.
|
||||
Internally, we store and process date and time as UTC.
|
142
docs/usage/general/environment.rst.inc
Normal file
142
docs/usage/general/environment.rst.inc
Normal file
@ -0,0 +1,142 @@
|
||||
Environment Variables
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Borg uses some environment variables for automation:
|
||||
|
||||
General:
|
||||
BORG_REPO
|
||||
When set, use the value to give the default repository location. If a command needs an archive
|
||||
parameter, you can abbreviate as ``::archive``. If a command needs a repository parameter, you
|
||||
can either leave it away or abbreviate as ``::``, if a positional parameter is required.
|
||||
BORG_PASSPHRASE
|
||||
When set, use the value to answer the passphrase question for encrypted repositories.
|
||||
It is used when a passphrase is needed to access an encrypted repo as well as when a new
|
||||
passphrase should be initially set when initializing an encrypted repo.
|
||||
See also BORG_NEW_PASSPHRASE.
|
||||
BORG_PASSCOMMAND
|
||||
When set, use the standard output of the command (trailing newlines are stripped) to answer the
|
||||
passphrase question for encrypted repositories.
|
||||
It is used when a passphrase is needed to access an encrypted repo as well as when a new
|
||||
passphrase should be initially set when initializing an encrypted repo. Note that the command
|
||||
is executed without a shell. So variables, like ``$HOME`` will work, but ``~`` won't.
|
||||
If BORG_PASSPHRASE is also set, it takes precedence.
|
||||
See also BORG_NEW_PASSPHRASE.
|
||||
BORG_PASSPHRASE_FD
|
||||
When set, specifies a file descriptor to read a passphrase
|
||||
from. Programs starting borg may choose to open an anonymous pipe
|
||||
and use it to pass a passphrase. This is safer than passing via
|
||||
BORG_PASSPHRASE, because on some systems (e.g. Linux) environment
|
||||
can be examined by other processes.
|
||||
If BORG_PASSPHRASE or BORG_PASSCOMMAND are also set, they take precedence.
|
||||
BORG_NEW_PASSPHRASE
|
||||
When set, use the value to answer the passphrase question when a **new** passphrase is asked for.
|
||||
This variable is checked first. If it is not set, BORG_PASSPHRASE and BORG_PASSCOMMAND will also
|
||||
be checked.
|
||||
Main usecase for this is to fully automate ``borg change-passphrase``.
|
||||
BORG_DISPLAY_PASSPHRASE
|
||||
When set, use the value to answer the "display the passphrase for verification" question when defining a new passphrase for encrypted repositories.
|
||||
BORG_HOSTNAME_IS_UNIQUE=no
|
||||
Borg assumes that it can derive a unique hostname / identity (see ``borg debug info``).
|
||||
If this is not the case or you do not want Borg to automatically remove stale locks,
|
||||
set this to *no*.
|
||||
BORG_HOST_ID
|
||||
Borg usually computes a host id from the FQDN plus the results of ``uuid.getnode()`` (which usually returns
|
||||
a unique id based on the MAC address of the network interface. Except if that MAC happens to be all-zero - in
|
||||
that case it returns a random value, which is not what we want (because it kills automatic stale lock removal).
|
||||
So, if you have a all-zero MAC address or other reasons to better externally control the host id, just set this
|
||||
environment variable to a unique value. If all your FQDNs are unique, you can just use the FQDN. If not,
|
||||
use fqdn@uniqueid.
|
||||
BORG_LOGGING_CONF
|
||||
When set, use the given filename as INI_-style logging configuration.
|
||||
A basic example conf can be found at ``docs/misc/logging.conf``.
|
||||
BORG_RSH
|
||||
When set, use this command instead of ``ssh``. This can be used to specify ssh options, such as
|
||||
a custom identity file ``ssh -i /path/to/private/key``. See ``man ssh`` for other options. Using
|
||||
the ``--rsh CMD`` commandline option overrides the environment variable.
|
||||
BORG_REMOTE_PATH
|
||||
When set, use the given path as borg executable on the remote (defaults to "borg" if unset).
|
||||
Using ``--remote-path PATH`` commandline option overrides the environment variable.
|
||||
BORG_FILES_CACHE_TTL
|
||||
When set to a numeric value, this determines the maximum "time to live" for the files cache
|
||||
entries (default: 20). The files cache is used to quickly determine whether a file is unchanged.
|
||||
The FAQ explains this more detailed in: :ref:`always_chunking`
|
||||
BORG_SHOW_SYSINFO
|
||||
When set to no (default: yes), system information (like OS, Python version, ...) in
|
||||
exceptions is not shown.
|
||||
Please only use for good reasons as it makes issues harder to analyze.
|
||||
BORG_WORKAROUNDS
|
||||
A list of comma separated strings that trigger workarounds in borg,
|
||||
e.g. to work around bugs in other software.
|
||||
|
||||
Currently known strings are:
|
||||
|
||||
basesyncfile
|
||||
Use the more simple BaseSyncFile code to avoid issues with sync_file_range.
|
||||
You might need this to run borg on WSL (Windows Subsystem for Linux) or
|
||||
in systemd.nspawn containers on some architectures (e.g. ARM).
|
||||
Using this does not affect data safety, but might result in a more bursty
|
||||
write to disk behaviour (not continuously streaming to disk).
|
||||
|
||||
Some automatic "answerers" (if set, they automatically answer confirmation questions):
|
||||
BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no (or =yes)
|
||||
For "Warning: Attempting to access a previously unknown unencrypted repository"
|
||||
BORG_RELOCATED_REPO_ACCESS_IS_OK=no (or =yes)
|
||||
For "Warning: The repository at location ... was previously located at ..."
|
||||
BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "Warning: 'check --repair' is an experimental feature that might result in data loss."
|
||||
BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "You requested to completely DELETE the repository *including* all archives it contains:"
|
||||
BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "recreate is an experimental feature."
|
||||
|
||||
Note: answers are case sensitive. setting an invalid answer value might either give the default
|
||||
answer or ask you interactively, depending on whether retries are allowed (they by default are
|
||||
allowed). So please test your scripts interactively before making them a non-interactive script.
|
||||
|
||||
Directories and files:
|
||||
BORG_BASE_DIR
|
||||
Defaults to '$HOME', '~$USER', '~' (in that order)'.
|
||||
If we refer to ~ below, we in fact mean BORG_BASE_DIR.
|
||||
BORG_CACHE_DIR
|
||||
Defaults to '~/.cache/borg'. This directory contains the local cache and might need a lot
|
||||
of space for dealing with big repositories. Make sure you're aware of the associated
|
||||
security aspects of the cache location: :ref:`cache_security`
|
||||
BORG_CONFIG_DIR
|
||||
Defaults to '~/.config/borg'. This directory contains the whole config directories. See FAQ
|
||||
for security advisory about the data in this directory: :ref:`home_config_borg`
|
||||
BORG_SECURITY_DIR
|
||||
Defaults to '~/.config/borg/security'. This directory contains information borg uses to
|
||||
track its usage of NONCES ("numbers used once" - usually in encryption context) and other
|
||||
security relevant data. Will move with BORG_CONFIG_DIR variable unless specified.
|
||||
BORG_KEYS_DIR
|
||||
Defaults to '~/.config/borg/keys'. This directory contains keys for encrypted repositories.
|
||||
BORG_KEY_FILE
|
||||
When set, use the given filename as repository key file.
|
||||
TMPDIR
|
||||
This is where temporary files are stored (might need a lot of temporary space for some
|
||||
operations), see tempfile_ for details.
|
||||
|
||||
Building:
|
||||
BORG_OPENSSL_PREFIX
|
||||
Adds given OpenSSL header file directory to the default locations (setup.py).
|
||||
BORG_LIBLZ4_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/lz4.h' is found Borg
|
||||
will be linked against the system liblz4 instead of a bundled implementation. (setup.py)
|
||||
BORG_LIBB2_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/blake2.h' is found Borg
|
||||
will be linked against the system libb2 instead of a bundled implementation. (setup.py)
|
||||
BORG_LIBZSTD_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/zstd.h' is found Borg
|
||||
will be linked against the system libzstd instead of a bundled implementation. (setup.py)
|
||||
|
||||
|
||||
Please note:
|
||||
|
||||
- be very careful when using the "yes" sayers, the warnings with prompt exist for your / your data's security/safety
|
||||
- also be very careful when putting your passphrase into a script, make sure it has appropriate file permissions
|
||||
(e.g. mode 600, root:root).
|
||||
|
||||
|
||||
.. _INI: https://docs.python.org/3/library/logging.config.html#configuration-file-format
|
||||
|
||||
.. _tempfile: https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir
|
61
docs/usage/general/file-metadata.rst.inc
Normal file
61
docs/usage/general/file-metadata.rst.inc
Normal file
@ -0,0 +1,61 @@
|
||||
Support for file metadata
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Besides regular file and directory structures, Borg can preserve
|
||||
|
||||
* symlinks (stored as symlink, the symlink is not followed)
|
||||
* special files:
|
||||
|
||||
* character and block device files (restored via mknod)
|
||||
* FIFOs ("named pipes")
|
||||
* special file *contents* can be backed up in ``--read-special`` mode.
|
||||
By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored.
|
||||
* hardlinked regular files, devices, FIFOs (considering all items in the same archive)
|
||||
* timestamps in nanosecond precision: mtime, atime, ctime
|
||||
* other timestamps: birthtime (on platforms supporting it)
|
||||
* permissions:
|
||||
|
||||
* IDs of owning user and owning group
|
||||
* names of owning user and owning group (if the IDs can be resolved)
|
||||
* Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky)
|
||||
|
||||
On some platforms additional features are supported:
|
||||
|
||||
.. Yes/No's are grouped by reason/mechanism/reference.
|
||||
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Platform | ACLs | xattr | Flags |
|
||||
| | [#acls]_ | [#xattr]_ | [#flags]_ |
|
||||
+=========================+==========+===========+============+
|
||||
| Linux | Yes | Yes | Yes [1]_ |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Mac OS X | Yes | Yes | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| FreeBSD | Yes | Yes | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| OpenBSD | n/a | n/a | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| NetBSD | n/a | No [2]_ | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Solaris and derivatives | No [3]_ | No [3]_ | n/a |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Windows (cygwin) | No [4]_ | No | No |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
|
||||
Other Unix-like operating systems may work as well, but have not been tested at all.
|
||||
|
||||
Note that most of the platform-dependent features also depend on the file system.
|
||||
For example, ntfs-3g on Linux isn't able to convey NTFS ACLs.
|
||||
|
||||
.. [1] Only "nodump", "immutable", "compressed" and "append" are supported.
|
||||
Feature request :issue:`618` for more flags.
|
||||
.. [2] Feature request :issue:`1332`
|
||||
.. [3] Feature request :issue:`1337`
|
||||
.. [4] Cygwin tries to map NTFS ACLs to permissions with varying degrees of success.
|
||||
|
||||
.. [#acls] The native access control list mechanism of the OS. This normally limits access to
|
||||
non-native ACLs. For example, NTFS ACLs aren't completely accessible on Linux with ntfs-3g.
|
||||
.. [#xattr] extended attributes; key-value pairs attached to a file, mainly used by the OS.
|
||||
This includes resource forks on Mac OS X.
|
||||
.. [#flags] aka *BSD flags*. The Linux set of flags [1]_ is portable across platforms.
|
||||
The BSDs define additional flags.
|
26
docs/usage/general/file-systems.rst.inc
Normal file
26
docs/usage/general/file-systems.rst.inc
Normal file
@ -0,0 +1,26 @@
|
||||
File systems
|
||||
~~~~~~~~~~~~
|
||||
|
||||
We strongly recommend against using Borg (or any other database-like
|
||||
software) on non-journaling file systems like FAT, since it is not
|
||||
possible to assume any consistency in case of power failures (or a
|
||||
sudden disconnect of an external drive or similar failures).
|
||||
|
||||
While Borg uses a data store that is resilient against these failures
|
||||
when used on journaling file systems, it is not possible to guarantee
|
||||
this with some hardware -- independent of the software used. We don't
|
||||
know a list of affected hardware.
|
||||
|
||||
If you are suspicious whether your Borg repository is still consistent
|
||||
and readable after one of the failures mentioned above occurred, run
|
||||
``borg check --verify-data`` to make sure it is consistent.
|
||||
|
||||
.. rubric:: Requirements for Borg repository file systems
|
||||
|
||||
- Long file names
|
||||
- At least three directory levels with short names
|
||||
- Typically, file sizes up to a few hundred MB.
|
||||
Large repositories may require large files (>2 GB).
|
||||
- Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
|
||||
- mkdir(2) should be atomic, since it is used for locking
|
||||
- Hardlinks are needed for :ref:`borg_upgrade` ``--inplace``
|
44
docs/usage/general/logging.rst.inc
Normal file
44
docs/usage/general/logging.rst.inc
Normal file
@ -0,0 +1,44 @@
|
||||
Logging
|
||||
~~~~~~~
|
||||
|
||||
Borg writes all log output to stderr by default. But please note that something
|
||||
showing up on stderr does *not* indicate an error condition just because it is
|
||||
on stderr. Please check the log levels of the messages and the return code of
|
||||
borg for determining error, warning or success conditions.
|
||||
|
||||
If you want to capture the log output to a file, just redirect it:
|
||||
|
||||
::
|
||||
|
||||
borg create repo::archive myfiles 2>> logfile
|
||||
|
||||
|
||||
Custom logging configurations can be implemented via BORG_LOGGING_CONF.
|
||||
|
||||
The log level of the builtin logging configuration defaults to WARNING.
|
||||
This is because we want Borg to be mostly silent and only output
|
||||
warnings, errors and critical messages, unless output has been requested
|
||||
by supplying an option that implies output (e.g. ``--list`` or ``--progress``).
|
||||
|
||||
Log levels: DEBUG < INFO < WARNING < ERROR < CRITICAL
|
||||
|
||||
Use ``--debug`` to set DEBUG log level -
|
||||
to get debug, info, warning, error and critical level output.
|
||||
|
||||
Use ``--info`` (or ``-v`` or ``--verbose``) to set INFO log level -
|
||||
to get info, warning, error and critical level output.
|
||||
|
||||
Use ``--warning`` (default) to set WARNING log level -
|
||||
to get warning, error and critical level output.
|
||||
|
||||
Use ``--error`` to set ERROR log level -
|
||||
to get error and critical level output.
|
||||
|
||||
Use ``--critical`` to set CRITICAL log level -
|
||||
to get critical level output.
|
||||
|
||||
While you can set misc. log levels, do not expect that every command will
|
||||
give different output on different log levels - it's just a possibility.
|
||||
|
||||
.. warning:: Options ``--critical`` and ``--error`` are provided for completeness,
|
||||
their usage is not recommended as you might miss important information.
|
15
docs/usage/general/positional-arguments.rst.inc
Normal file
15
docs/usage/general/positional-arguments.rst.inc
Normal file
@ -0,0 +1,15 @@
|
||||
Positional Arguments and Options: Order matters
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Borg only supports taking options (``-s`` and ``--progress`` in the example)
|
||||
to the left or right of all positional arguments (``repo::archive`` and ``path``
|
||||
in the example), but not in between them:
|
||||
|
||||
::
|
||||
|
||||
borg create -s --progress repo::archive path # good and preferred
|
||||
borg create repo::archive path -s --progress # also works
|
||||
borg create -s repo::archive path --progress # works, but ugly
|
||||
borg create repo::archive -s --progress path # BAD
|
||||
|
||||
This is due to a problem in the argparse module: http://bugs.python.org/issue15112
|
13
docs/usage/general/repository-locations.rst.inc
Normal file
13
docs/usage/general/repository-locations.rst.inc
Normal file
@ -0,0 +1,13 @@
|
||||
Repository / Archive Locations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Many commands want either a repository (just give the repo URL, see above) or
|
||||
an archive location, which is a repo URL followed by ``::archive_name``.
|
||||
|
||||
Archive names must not contain the ``/`` (slash) character. For simplicity,
|
||||
maybe also avoid blanks or other characters that have special meaning on the
|
||||
shell or in a filesystem (borg mount will use the archive name as directory
|
||||
name).
|
||||
|
||||
If you have set BORG_REPO (see above) and an archive location is needed, use
|
||||
``::archive_name`` - the repo URL part is then read from BORG_REPO.
|
54
docs/usage/general/repository-urls.rst.inc
Normal file
54
docs/usage/general/repository-urls.rst.inc
Normal file
@ -0,0 +1,54 @@
|
||||
Repository URLs
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
**Local filesystem** (or locally mounted network filesystem):
|
||||
|
||||
``/path/to/repo`` - filesystem path to repo directory, absolute path
|
||||
|
||||
``path/to/repo`` - filesystem path to repo directory, relative path
|
||||
|
||||
Also, stuff like ``~/path/to/repo`` or ``~other/path/to/repo`` works (this is
|
||||
expanded by your shell).
|
||||
|
||||
Note: you may also prepend a ``file://`` to a filesystem path to get URL style.
|
||||
|
||||
**Remote repositories** accessed via ssh user@host:
|
||||
|
||||
``user@host:/path/to/repo`` - remote repo, absolute path
|
||||
|
||||
``ssh://user@host:port/path/to/repo`` - same, alternative syntax, port can be given
|
||||
|
||||
|
||||
**Remote repositories with relative paths** can be given using this syntax:
|
||||
|
||||
``user@host:path/to/repo`` - path relative to current directory
|
||||
|
||||
``user@host:~/path/to/repo`` - path relative to user's home directory
|
||||
|
||||
``user@host:~other/path/to/repo`` - path relative to other's home directory
|
||||
|
||||
Note: giving ``user@host:/./path/to/repo`` or ``user@host:/~/path/to/repo`` or
|
||||
``user@host:/~other/path/to/repo`` is also supported, but not required here.
|
||||
|
||||
|
||||
**Remote repositories with relative paths, alternative syntax with port**:
|
||||
|
||||
``ssh://user@host:port/./path/to/repo`` - path relative to current directory
|
||||
|
||||
``ssh://user@host:port/~/path/to/repo`` - path relative to user's home directory
|
||||
|
||||
``ssh://user@host:port/~other/path/to/repo`` - path relative to other's home directory
|
||||
|
||||
|
||||
If you frequently need the same repo URL, it is a good idea to set the
|
||||
``BORG_REPO`` environment variable to set a default for the repo URL:
|
||||
|
||||
::
|
||||
|
||||
export BORG_REPO='ssh://user@host:port/path/to/repo'
|
||||
|
||||
Then just leave away the repo URL if only a repo URL is needed and you want
|
||||
to use the default - it will be read from BORG_REPO then.
|
||||
|
||||
Use ``::`` syntax to give the repo URL when syntax requires giving a positional
|
||||
argument for the repo (e.g. ``borg mount :: /mnt``).
|
95
docs/usage/general/resources.rst.inc
Normal file
95
docs/usage/general/resources.rst.inc
Normal file
@ -0,0 +1,95 @@
|
||||
Resource Usage
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Borg might use a lot of resources depending on the size of the data set it is dealing with.
|
||||
|
||||
If one uses Borg in a client/server way (with a ssh: repository),
|
||||
the resource usage occurs in part on the client and in another part on the
|
||||
server.
|
||||
|
||||
If one uses Borg as a single process (with a filesystem repo),
|
||||
all the resource usage occurs in that one process, so just add up client +
|
||||
server to get the approximate resource usage.
|
||||
|
||||
CPU client:
|
||||
- **borg create:** does chunking, hashing, compression, crypto (high CPU usage)
|
||||
- **chunks cache sync:** quite heavy on CPU, doing lots of hashtable operations.
|
||||
- **borg extract:** crypto, decompression (medium to high CPU usage)
|
||||
- **borg check:** similar to extract, but depends on options given.
|
||||
- **borg prune / borg delete archive:** low to medium CPU usage
|
||||
- **borg delete repo:** done on the server
|
||||
|
||||
It won't go beyond 100% of 1 core as the code is currently single-threaded.
|
||||
Especially higher zlib and lzma compression levels use significant amounts
|
||||
of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or
|
||||
expensive (if not).
|
||||
|
||||
CPU server:
|
||||
It usually doesn't need much CPU, it just deals with the key/value store
|
||||
(repository) and uses the repository index for that.
|
||||
|
||||
borg check: the repository check computes the checksums of all chunks
|
||||
(medium CPU usage)
|
||||
borg delete repo: low CPU usage
|
||||
|
||||
CPU (only for client/server operation):
|
||||
When using borg in a client/server way with a ssh:-type repo, the ssh
|
||||
processes used for the transport layer will need some CPU on the client and
|
||||
on the server due to the crypto they are doing - esp. if you are pumping
|
||||
big amounts of data.
|
||||
|
||||
Memory (RAM) client:
|
||||
The chunks index and the files index are read into memory for performance
|
||||
reasons. Might need big amounts of memory (see below).
|
||||
Compression, esp. lzma compression with high levels might need substantial
|
||||
amounts of memory.
|
||||
|
||||
Memory (RAM) server:
|
||||
The server process will load the repository index into memory. Might need
|
||||
considerable amounts of memory, but less than on the client (see below).
|
||||
|
||||
Chunks index (client only):
|
||||
Proportional to the amount of data chunks in your repo. Lots of chunks
|
||||
in your repo imply a big chunks index.
|
||||
It is possible to tweak the chunker params (see create options).
|
||||
|
||||
Files index (client only):
|
||||
Proportional to the amount of files in your last backups. Can be switched
|
||||
off (see create options), but next backup might be much slower if you do.
|
||||
The speed benefit of using the files cache is proportional to file size.
|
||||
|
||||
Repository index (server only):
|
||||
Proportional to the amount of data chunks in your repo. Lots of chunks
|
||||
in your repo imply a big repository index.
|
||||
It is possible to tweak the chunker params (see create options) to
|
||||
influence the amount of chunks being created.
|
||||
|
||||
Temporary files (client):
|
||||
Reading data and metadata from a FUSE mounted repository will consume up to
|
||||
the size of all deduplicated, small chunks in the repository. Big chunks
|
||||
won't be locally cached.
|
||||
|
||||
Temporary files (server):
|
||||
A non-trivial amount of data will be stored on the remote temp directory
|
||||
for each client that connects to it. For some remotes, this can fill the
|
||||
default temporary directory at /tmp. This can be remediated by ensuring the
|
||||
$TMPDIR, $TEMP, or $TMP environment variable is properly set for the sshd
|
||||
process.
|
||||
For some OSes, this can be done just by setting the correct value in the
|
||||
.bashrc (or equivalent login config file for other shells), however in
|
||||
other cases it may be necessary to first enable ``PermitUserEnvironment yes``
|
||||
in your ``sshd_config`` file, then add ``environment="TMPDIR=/my/big/tmpdir"``
|
||||
at the start of the public key to be used in the ``authorized_hosts`` file.
|
||||
|
||||
Cache files (client only):
|
||||
Contains the chunks index and files index (plus a collection of single-
|
||||
archive chunk indexes which might need huge amounts of disk space,
|
||||
depending on archive count and size - see FAQ about how to reduce).
|
||||
|
||||
Network (only for client/server operation):
|
||||
If your repository is remote, all deduplicated (and optionally compressed/
|
||||
encrypted) data of course has to go over the connection (``ssh://`` repo url).
|
||||
If you use a locally mounted network filesystem, additionally some copy
|
||||
operations used for transaction support also go over the connection. If
|
||||
you backup multiple sources to one target repository, additional traffic
|
||||
happens for cache resynchronization.
|
18
docs/usage/general/return-codes.rst.inc
Normal file
18
docs/usage/general/return-codes.rst.inc
Normal file
@ -0,0 +1,18 @@
|
||||
Return codes
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Borg can exit with the following return codes (rc):
|
||||
|
||||
=========== =======
|
||||
Return code Meaning
|
||||
=========== =======
|
||||
0 success (logged as INFO)
|
||||
1 warning (operation reached its normal end, but there were warnings --
|
||||
you should check the log, logged as WARNING)
|
||||
2 error (like a fatal error, a local or remote exception, the operation
|
||||
did not reach its normal end, logged as ERROR)
|
||||
128+N killed by signal N (e.g. 137 == kill -9)
|
||||
=========== =======
|
||||
|
||||
If you use ``--show-rc``, the return code is also logged at the indicated
|
||||
level as the last log entry.
|
11
docs/usage/general/units.rst.inc
Normal file
11
docs/usage/general/units.rst.inc
Normal file
@ -0,0 +1,11 @@
|
||||
Units
|
||||
~~~~~
|
||||
|
||||
To display quantities, Borg takes care of respecting the
|
||||
usual conventions of scale. Disk sizes are displayed in `decimal
|
||||
<https://en.wikipedia.org/wiki/Decimal>`_, using powers of ten (so
|
||||
``kB`` means 1000 bytes). For memory usage, `binary prefixes
|
||||
<https://en.wikipedia.org/wiki/Binary_prefix>`_ are used, and are
|
||||
indicated using the `IEC binary prefixes
|
||||
<https://en.wikipedia.org/wiki/IEC_80000-13#Prefixes_for_binary_multiples>`_,
|
||||
using powers of two (so ``KiB`` means 1024 bytes).
|
27
docs/usage/usage_general.rst.inc
Normal file
27
docs/usage/usage_general.rst.inc
Normal file
@ -0,0 +1,27 @@
|
||||
.. include:: general/positional-arguments.rst.inc
|
||||
|
||||
.. include:: general/repository-urls.rst.inc
|
||||
|
||||
.. include:: general/repository-locations.rst.inc
|
||||
|
||||
.. include:: general/logging.rst.inc
|
||||
|
||||
.. include:: general/return-codes.rst.inc
|
||||
|
||||
.. _env_vars:
|
||||
|
||||
.. include:: general/environment.rst.inc
|
||||
|
||||
.. _file-systems:
|
||||
|
||||
.. include:: general/file-systems.rst.inc
|
||||
|
||||
.. include:: general/units.rst.inc
|
||||
|
||||
.. include:: general/date-time.rst.inc
|
||||
|
||||
.. include:: general/resources.rst.inc
|
||||
|
||||
.. _platforms:
|
||||
|
||||
.. include:: general/file-metadata.rst.inc
|
@ -1,508 +1,21 @@
|
||||
Positional Arguments and Options: Order matters
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
.. include:: usage/general/positional-arguments.rst.inc
|
||||
|
||||
Borg only supports taking options (``-s`` and ``--progress`` in the example)
|
||||
to the left or right of all positional arguments (``repo::archive`` and ``path``
|
||||
in the example), but not in between them:
|
||||
.. include:: usage/general/repository-urls.rst.inc
|
||||
|
||||
::
|
||||
.. include:: usage/general/repository-locations.rst.inc
|
||||
|
||||
borg create -s --progress repo::archive path # good and preferred
|
||||
borg create repo::archive path -s --progress # also works
|
||||
borg create -s repo::archive path --progress # works, but ugly
|
||||
borg create repo::archive -s --progress path # BAD
|
||||
.. include:: usage/general/logging.rst.inc
|
||||
|
||||
This is due to a problem in the argparse module: http://bugs.python.org/issue15112
|
||||
.. include:: usage/general/return-codes.rst.inc
|
||||
|
||||
.. include:: usage/general/environment.rst.inc
|
||||
|
||||
Repository URLs
|
||||
~~~~~~~~~~~~~~~
|
||||
.. include:: usage/general/file-systems.rst.inc
|
||||
|
||||
**Local filesystem** (or locally mounted network filesystem):
|
||||
.. include:: usage/general/units.rst.inc
|
||||
|
||||
``/path/to/repo`` - filesystem path to repo directory, absolute path
|
||||
.. include:: usage/general/date-time.rst.inc
|
||||
|
||||
``path/to/repo`` - filesystem path to repo directory, relative path
|
||||
.. include:: usage/general/resources.rst.inc
|
||||
|
||||
Also, stuff like ``~/path/to/repo`` or ``~other/path/to/repo`` works (this is
|
||||
expanded by your shell).
|
||||
|
||||
Note: you may also prepend a ``file://`` to a filesystem path to get URL style.
|
||||
|
||||
**Remote repositories** accessed via ssh user@host:
|
||||
|
||||
``user@host:/path/to/repo`` - remote repo, absolute path
|
||||
|
||||
``ssh://user@host:port/path/to/repo`` - same, alternative syntax, port can be given
|
||||
|
||||
|
||||
**Remote repositories with relative paths** can be given using this syntax:
|
||||
|
||||
``user@host:path/to/repo`` - path relative to current directory
|
||||
|
||||
``user@host:~/path/to/repo`` - path relative to user's home directory
|
||||
|
||||
``user@host:~other/path/to/repo`` - path relative to other's home directory
|
||||
|
||||
Note: giving ``user@host:/./path/to/repo`` or ``user@host:/~/path/to/repo`` or
|
||||
``user@host:/~other/path/to/repo`` is also supported, but not required here.
|
||||
|
||||
|
||||
**Remote repositories with relative paths, alternative syntax with port**:
|
||||
|
||||
``ssh://user@host:port/./path/to/repo`` - path relative to current directory
|
||||
|
||||
``ssh://user@host:port/~/path/to/repo`` - path relative to user's home directory
|
||||
|
||||
``ssh://user@host:port/~other/path/to/repo`` - path relative to other's home directory
|
||||
|
||||
|
||||
If you frequently need the same repo URL, it is a good idea to set the
|
||||
``BORG_REPO`` environment variable to set a default for the repo URL:
|
||||
|
||||
::
|
||||
|
||||
export BORG_REPO='ssh://user@host:port/path/to/repo'
|
||||
|
||||
Then just leave away the repo URL if only a repo URL is needed and you want
|
||||
to use the default - it will be read from BORG_REPO then.
|
||||
|
||||
Use ``::`` syntax to give the repo URL when syntax requires giving a positional
|
||||
argument for the repo (e.g. ``borg mount :: /mnt``).
|
||||
|
||||
|
||||
Repository / Archive Locations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Many commands want either a repository (just give the repo URL, see above) or
|
||||
an archive location, which is a repo URL followed by ``::archive_name``.
|
||||
|
||||
Archive names must not contain the ``/`` (slash) character. For simplicity,
|
||||
maybe also avoid blanks or other characters that have special meaning on the
|
||||
shell or in a filesystem (borg mount will use the archive name as directory
|
||||
name).
|
||||
|
||||
If you have set BORG_REPO (see above) and an archive location is needed, use
|
||||
``::archive_name`` - the repo URL part is then read from BORG_REPO.
|
||||
|
||||
|
||||
Logging
|
||||
~~~~~~~
|
||||
|
||||
Borg writes all log output to stderr by default. But please note that something
|
||||
showing up on stderr does *not* indicate an error condition just because it is
|
||||
on stderr. Please check the log levels of the messages and the return code of
|
||||
borg for determining error, warning or success conditions.
|
||||
|
||||
If you want to capture the log output to a file, just redirect it:
|
||||
|
||||
::
|
||||
|
||||
borg create repo::archive myfiles 2>> logfile
|
||||
|
||||
|
||||
Custom logging configurations can be implemented via BORG_LOGGING_CONF.
|
||||
|
||||
The log level of the builtin logging configuration defaults to WARNING.
|
||||
This is because we want Borg to be mostly silent and only output
|
||||
warnings, errors and critical messages, unless output has been requested
|
||||
by supplying an option that implies output (e.g. ``--list`` or ``--progress``).
|
||||
|
||||
Log levels: DEBUG < INFO < WARNING < ERROR < CRITICAL
|
||||
|
||||
Use ``--debug`` to set DEBUG log level -
|
||||
to get debug, info, warning, error and critical level output.
|
||||
|
||||
Use ``--info`` (or ``-v`` or ``--verbose``) to set INFO log level -
|
||||
to get info, warning, error and critical level output.
|
||||
|
||||
Use ``--warning`` (default) to set WARNING log level -
|
||||
to get warning, error and critical level output.
|
||||
|
||||
Use ``--error`` to set ERROR log level -
|
||||
to get error and critical level output.
|
||||
|
||||
Use ``--critical`` to set CRITICAL log level -
|
||||
to get critical level output.
|
||||
|
||||
While you can set misc. log levels, do not expect that every command will
|
||||
give different output on different log levels - it's just a possibility.
|
||||
|
||||
.. warning:: Options ``--critical`` and ``--error`` are provided for completeness,
|
||||
their usage is not recommended as you might miss important information.
|
||||
|
||||
Return codes
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Borg can exit with the following return codes (rc):
|
||||
|
||||
=========== =======
|
||||
Return code Meaning
|
||||
=========== =======
|
||||
0 success (logged as INFO)
|
||||
1 warning (operation reached its normal end, but there were warnings --
|
||||
you should check the log, logged as WARNING)
|
||||
2 error (like a fatal error, a local or remote exception, the operation
|
||||
did not reach its normal end, logged as ERROR)
|
||||
128+N killed by signal N (e.g. 137 == kill -9)
|
||||
=========== =======
|
||||
|
||||
If you use ``--show-rc``, the return code is also logged at the indicated
|
||||
level as the last log entry.
|
||||
|
||||
.. _env_vars:
|
||||
|
||||
Environment Variables
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Borg uses some environment variables for automation:
|
||||
|
||||
General:
|
||||
BORG_REPO
|
||||
When set, use the value to give the default repository location. If a command needs an archive
|
||||
parameter, you can abbreviate as ``::archive``. If a command needs a repository parameter, you
|
||||
can either leave it away or abbreviate as ``::``, if a positional parameter is required.
|
||||
BORG_PASSPHRASE
|
||||
When set, use the value to answer the passphrase question for encrypted repositories.
|
||||
It is used when a passphrase is needed to access an encrypted repo as well as when a new
|
||||
passphrase should be initially set when initializing an encrypted repo.
|
||||
See also BORG_NEW_PASSPHRASE.
|
||||
BORG_PASSCOMMAND
|
||||
When set, use the standard output of the command (trailing newlines are stripped) to answer the
|
||||
passphrase question for encrypted repositories.
|
||||
It is used when a passphrase is needed to access an encrypted repo as well as when a new
|
||||
passphrase should be initially set when initializing an encrypted repo. Note that the command
|
||||
is executed without a shell. So variables, like ``$HOME`` will work, but ``~`` won't.
|
||||
If BORG_PASSPHRASE is also set, it takes precedence.
|
||||
See also BORG_NEW_PASSPHRASE.
|
||||
BORG_PASSPHRASE_FD
|
||||
When set, specifies a file descriptor to read a passphrase
|
||||
from. Programs starting borg may choose to open an anonymous pipe
|
||||
and use it to pass a passphrase. This is safer than passing via
|
||||
BORG_PASSPHRASE, because on some systems (e.g. Linux) environment
|
||||
can be examined by other processes.
|
||||
If BORG_PASSPHRASE or BORG_PASSCOMMAND are also set, they take precedence.
|
||||
BORG_NEW_PASSPHRASE
|
||||
When set, use the value to answer the passphrase question when a **new** passphrase is asked for.
|
||||
This variable is checked first. If it is not set, BORG_PASSPHRASE and BORG_PASSCOMMAND will also
|
||||
be checked.
|
||||
Main usecase for this is to fully automate ``borg change-passphrase``.
|
||||
BORG_DISPLAY_PASSPHRASE
|
||||
When set, use the value to answer the "display the passphrase for verification" question when defining a new passphrase for encrypted repositories.
|
||||
BORG_HOSTNAME_IS_UNIQUE=no
|
||||
Borg assumes that it can derive a unique hostname / identity (see ``borg debug info``).
|
||||
If this is not the case or you do not want Borg to automatically remove stale locks,
|
||||
set this to *no*.
|
||||
BORG_HOST_ID
|
||||
Borg usually computes a host id from the FQDN plus the results of ``uuid.getnode()`` (which usually returns
|
||||
a unique id based on the MAC address of the network interface. Except if that MAC happens to be all-zero - in
|
||||
that case it returns a random value, which is not what we want (because it kills automatic stale lock removal).
|
||||
So, if you have a all-zero MAC address or other reasons to better externally control the host id, just set this
|
||||
environment variable to a unique value. If all your FQDNs are unique, you can just use the FQDN. If not,
|
||||
use fqdn@uniqueid.
|
||||
BORG_LOGGING_CONF
|
||||
When set, use the given filename as INI_-style logging configuration.
|
||||
A basic example conf can be found at ``docs/misc/logging.conf``.
|
||||
BORG_RSH
|
||||
When set, use this command instead of ``ssh``. This can be used to specify ssh options, such as
|
||||
a custom identity file ``ssh -i /path/to/private/key``. See ``man ssh`` for other options. Using
|
||||
the ``--rsh CMD`` commandline option overrides the environment variable.
|
||||
BORG_REMOTE_PATH
|
||||
When set, use the given path as borg executable on the remote (defaults to "borg" if unset).
|
||||
Using ``--remote-path PATH`` commandline option overrides the environment variable.
|
||||
BORG_FILES_CACHE_TTL
|
||||
When set to a numeric value, this determines the maximum "time to live" for the files cache
|
||||
entries (default: 20). The files cache is used to quickly determine whether a file is unchanged.
|
||||
The FAQ explains this more detailed in: :ref:`always_chunking`
|
||||
BORG_SHOW_SYSINFO
|
||||
When set to no (default: yes), system information (like OS, Python version, ...) in
|
||||
exceptions is not shown.
|
||||
Please only use for good reasons as it makes issues harder to analyze.
|
||||
BORG_WORKAROUNDS
|
||||
A list of comma separated strings that trigger workarounds in borg,
|
||||
e.g. to work around bugs in other software.
|
||||
|
||||
Currently known strings are:
|
||||
|
||||
basesyncfile
|
||||
Use the more simple BaseSyncFile code to avoid issues with sync_file_range.
|
||||
You might need this to run borg on WSL (Windows Subsystem for Linux) or
|
||||
in systemd.nspawn containers on some architectures (e.g. ARM).
|
||||
Using this does not affect data safety, but might result in a more bursty
|
||||
write to disk behaviour (not continuously streaming to disk).
|
||||
|
||||
Some automatic "answerers" (if set, they automatically answer confirmation questions):
|
||||
BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no (or =yes)
|
||||
For "Warning: Attempting to access a previously unknown unencrypted repository"
|
||||
BORG_RELOCATED_REPO_ACCESS_IS_OK=no (or =yes)
|
||||
For "Warning: The repository at location ... was previously located at ..."
|
||||
BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "Warning: 'check --repair' is an experimental feature that might result in data loss."
|
||||
BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "You requested to completely DELETE the repository *including* all archives it contains:"
|
||||
BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
|
||||
For "recreate is an experimental feature."
|
||||
|
||||
Note: answers are case sensitive. setting an invalid answer value might either give the default
|
||||
answer or ask you interactively, depending on whether retries are allowed (they by default are
|
||||
allowed). So please test your scripts interactively before making them a non-interactive script.
|
||||
|
||||
Directories and files:
|
||||
BORG_BASE_DIR
|
||||
Defaults to '$HOME', '~$USER', '~' (in that order)'.
|
||||
If we refer to ~ below, we in fact mean BORG_BASE_DIR.
|
||||
BORG_CACHE_DIR
|
||||
Defaults to '~/.cache/borg'. This directory contains the local cache and might need a lot
|
||||
of space for dealing with big repositories. Make sure you're aware of the associated
|
||||
security aspects of the cache location: :ref:`cache_security`
|
||||
BORG_CONFIG_DIR
|
||||
Defaults to '~/.config/borg'. This directory contains the whole config directories. See FAQ
|
||||
for security advisory about the data in this directory: :ref:`home_config_borg`
|
||||
BORG_SECURITY_DIR
|
||||
Defaults to '~/.config/borg/security'. This directory contains information borg uses to
|
||||
track its usage of NONCES ("numbers used once" - usually in encryption context) and other
|
||||
security relevant data. Will move with BORG_CONFIG_DIR variable unless specified.
|
||||
BORG_KEYS_DIR
|
||||
Defaults to '~/.config/borg/keys'. This directory contains keys for encrypted repositories.
|
||||
BORG_KEY_FILE
|
||||
When set, use the given filename as repository key file.
|
||||
TMPDIR
|
||||
This is where temporary files are stored (might need a lot of temporary space for some
|
||||
operations), see tempfile_ for details.
|
||||
|
||||
Building:
|
||||
BORG_OPENSSL_PREFIX
|
||||
Adds given OpenSSL header file directory to the default locations (setup.py).
|
||||
BORG_LIBLZ4_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/lz4.h' is found Borg
|
||||
will be linked against the system liblz4 instead of a bundled implementation. (setup.py)
|
||||
BORG_LIBB2_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/blake2.h' is found Borg
|
||||
will be linked against the system libb2 instead of a bundled implementation. (setup.py)
|
||||
BORG_LIBZSTD_PREFIX
|
||||
Adds given prefix directory to the default locations. If a 'include/zstd.h' is found Borg
|
||||
will be linked against the system libzstd instead of a bundled implementation. (setup.py)
|
||||
|
||||
|
||||
Please note:
|
||||
|
||||
- be very careful when using the "yes" sayers, the warnings with prompt exist for your / your data's security/safety
|
||||
- also be very careful when putting your passphrase into a script, make sure it has appropriate file permissions
|
||||
(e.g. mode 600, root:root).
|
||||
|
||||
|
||||
.. _INI: https://docs.python.org/3/library/logging.config.html#configuration-file-format
|
||||
|
||||
.. _tempfile: https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir
|
||||
|
||||
.. _file-systems:
|
||||
|
||||
File systems
|
||||
~~~~~~~~~~~~
|
||||
|
||||
We strongly recommend against using Borg (or any other database-like
|
||||
software) on non-journaling file systems like FAT, since it is not
|
||||
possible to assume any consistency in case of power failures (or a
|
||||
sudden disconnect of an external drive or similar failures).
|
||||
|
||||
While Borg uses a data store that is resilient against these failures
|
||||
when used on journaling file systems, it is not possible to guarantee
|
||||
this with some hardware -- independent of the software used. We don't
|
||||
know a list of affected hardware.
|
||||
|
||||
If you are suspicious whether your Borg repository is still consistent
|
||||
and readable after one of the failures mentioned above occurred, run
|
||||
``borg check --verify-data`` to make sure it is consistent.
|
||||
|
||||
.. rubric:: Requirements for Borg repository file systems
|
||||
|
||||
- Long file names
|
||||
- At least three directory levels with short names
|
||||
- Typically, file sizes up to a few hundred MB.
|
||||
Large repositories may require large files (>2 GB).
|
||||
- Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
|
||||
- mkdir(2) should be atomic, since it is used for locking
|
||||
- Hardlinks are needed for :ref:`borg_upgrade` ``--inplace``
|
||||
|
||||
Units
|
||||
~~~~~
|
||||
|
||||
To display quantities, Borg takes care of respecting the
|
||||
usual conventions of scale. Disk sizes are displayed in `decimal
|
||||
<https://en.wikipedia.org/wiki/Decimal>`_, using powers of ten (so
|
||||
``kB`` means 1000 bytes). For memory usage, `binary prefixes
|
||||
<https://en.wikipedia.org/wiki/Binary_prefix>`_ are used, and are
|
||||
indicated using the `IEC binary prefixes
|
||||
<https://en.wikipedia.org/wiki/IEC_80000-13#Prefixes_for_binary_multiples>`_,
|
||||
using powers of two (so ``KiB`` means 1024 bytes).
|
||||
|
||||
Date and Time
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
We format date and time conforming to ISO-8601, that is: YYYY-MM-DD and
|
||||
HH:MM:SS (24h clock).
|
||||
|
||||
For more information about that, see: https://xkcd.com/1179/
|
||||
|
||||
Unless otherwise noted, we display local date and time.
|
||||
Internally, we store and process date and time as UTC.
|
||||
|
||||
Resource Usage
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Borg might use a lot of resources depending on the size of the data set it is dealing with.
|
||||
|
||||
If one uses Borg in a client/server way (with a ssh: repository),
|
||||
the resource usage occurs in part on the client and in another part on the
|
||||
server.
|
||||
|
||||
If one uses Borg as a single process (with a filesystem repo),
|
||||
all the resource usage occurs in that one process, so just add up client +
|
||||
server to get the approximate resource usage.
|
||||
|
||||
CPU client:
|
||||
- **borg create:** does chunking, hashing, compression, crypto (high CPU usage)
|
||||
- **chunks cache sync:** quite heavy on CPU, doing lots of hashtable operations.
|
||||
- **borg extract:** crypto, decompression (medium to high CPU usage)
|
||||
- **borg check:** similar to extract, but depends on options given.
|
||||
- **borg prune / borg delete archive:** low to medium CPU usage
|
||||
- **borg delete repo:** done on the server
|
||||
|
||||
It won't go beyond 100% of 1 core as the code is currently single-threaded.
|
||||
Especially higher zlib and lzma compression levels use significant amounts
|
||||
of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or
|
||||
expensive (if not).
|
||||
|
||||
CPU server:
|
||||
It usually doesn't need much CPU, it just deals with the key/value store
|
||||
(repository) and uses the repository index for that.
|
||||
|
||||
borg check: the repository check computes the checksums of all chunks
|
||||
(medium CPU usage)
|
||||
borg delete repo: low CPU usage
|
||||
|
||||
CPU (only for client/server operation):
|
||||
When using borg in a client/server way with a ssh:-type repo, the ssh
|
||||
processes used for the transport layer will need some CPU on the client and
|
||||
on the server due to the crypto they are doing - esp. if you are pumping
|
||||
big amounts of data.
|
||||
|
||||
Memory (RAM) client:
|
||||
The chunks index and the files index are read into memory for performance
|
||||
reasons. Might need big amounts of memory (see below).
|
||||
Compression, esp. lzma compression with high levels might need substantial
|
||||
amounts of memory.
|
||||
|
||||
Memory (RAM) server:
|
||||
The server process will load the repository index into memory. Might need
|
||||
considerable amounts of memory, but less than on the client (see below).
|
||||
|
||||
Chunks index (client only):
|
||||
Proportional to the amount of data chunks in your repo. Lots of chunks
|
||||
in your repo imply a big chunks index.
|
||||
It is possible to tweak the chunker params (see create options).
|
||||
|
||||
Files index (client only):
|
||||
Proportional to the amount of files in your last backups. Can be switched
|
||||
off (see create options), but next backup might be much slower if you do.
|
||||
The speed benefit of using the files cache is proportional to file size.
|
||||
|
||||
Repository index (server only):
|
||||
Proportional to the amount of data chunks in your repo. Lots of chunks
|
||||
in your repo imply a big repository index.
|
||||
It is possible to tweak the chunker params (see create options) to
|
||||
influence the amount of chunks being created.
|
||||
|
||||
Temporary files (client):
|
||||
Reading data and metadata from a FUSE mounted repository will consume up to
|
||||
the size of all deduplicated, small chunks in the repository. Big chunks
|
||||
won't be locally cached.
|
||||
|
||||
Temporary files (server):
|
||||
A non-trivial amount of data will be stored on the remote temp directory
|
||||
for each client that connects to it. For some remotes, this can fill the
|
||||
default temporary directory at /tmp. This can be remediated by ensuring the
|
||||
$TMPDIR, $TEMP, or $TMP environment variable is properly set for the sshd
|
||||
process.
|
||||
For some OSes, this can be done just by setting the correct value in the
|
||||
.bashrc (or equivalent login config file for other shells), however in
|
||||
other cases it may be necessary to first enable ``PermitUserEnvironment yes``
|
||||
in your ``sshd_config`` file, then add ``environment="TMPDIR=/my/big/tmpdir"``
|
||||
at the start of the public key to be used in the ``authorized_hosts`` file.
|
||||
|
||||
Cache files (client only):
|
||||
Contains the chunks index and files index (plus a collection of single-
|
||||
archive chunk indexes which might need huge amounts of disk space,
|
||||
depending on archive count and size - see FAQ about how to reduce).
|
||||
|
||||
Network (only for client/server operation):
|
||||
If your repository is remote, all deduplicated (and optionally compressed/
|
||||
encrypted) data of course has to go over the connection (``ssh://`` repo url).
|
||||
If you use a locally mounted network filesystem, additionally some copy
|
||||
operations used for transaction support also go over the connection. If
|
||||
you backup multiple sources to one target repository, additional traffic
|
||||
happens for cache resynchronization.
|
||||
|
||||
.. _platforms:
|
||||
|
||||
Support for file metadata
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Besides regular file and directory structures, Borg can preserve
|
||||
|
||||
* symlinks (stored as symlink, the symlink is not followed)
|
||||
* special files:
|
||||
|
||||
* character and block device files (restored via mknod)
|
||||
* FIFOs ("named pipes")
|
||||
* special file *contents* can be backed up in ``--read-special`` mode.
|
||||
By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored.
|
||||
* hardlinked regular files, devices, FIFOs (considering all items in the same archive)
|
||||
* timestamps in nanosecond precision: mtime, atime, ctime
|
||||
* other timestamps: birthtime (on platforms supporting it)
|
||||
* permissions:
|
||||
|
||||
* IDs of owning user and owning group
|
||||
* names of owning user and owning group (if the IDs can be resolved)
|
||||
* Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky)
|
||||
|
||||
On some platforms additional features are supported:
|
||||
|
||||
.. Yes/No's are grouped by reason/mechanism/reference.
|
||||
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Platform | ACLs | xattr | Flags |
|
||||
| | [#acls]_ | [#xattr]_ | [#flags]_ |
|
||||
+=========================+==========+===========+============+
|
||||
| Linux | Yes | Yes | Yes [1]_ |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Mac OS X | Yes | Yes | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| FreeBSD | Yes | Yes | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| OpenBSD | n/a | n/a | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| NetBSD | n/a | No [2]_ | Yes (all) |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Solaris and derivatives | No [3]_ | No [3]_ | n/a |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
| Windows (cygwin) | No [4]_ | No | No |
|
||||
+-------------------------+----------+-----------+------------+
|
||||
|
||||
Other Unix-like operating systems may work as well, but have not been tested at all.
|
||||
|
||||
Note that most of the platform-dependent features also depend on the file system.
|
||||
For example, ntfs-3g on Linux isn't able to convey NTFS ACLs.
|
||||
|
||||
.. [1] Only "nodump", "immutable", "compressed" and "append" are supported.
|
||||
Feature request :issue:`618` for more flags.
|
||||
.. [2] Feature request :issue:`1332`
|
||||
.. [3] Feature request :issue:`1337`
|
||||
.. [4] Cygwin tries to map NTFS ACLs to permissions with varying degrees of success.
|
||||
|
||||
.. [#acls] The native access control list mechanism of the OS. This normally limits access to
|
||||
non-native ACLs. For example, NTFS ACLs aren't completely accessible on Linux with ntfs-3g.
|
||||
.. [#xattr] extended attributes; key-value pairs attached to a file, mainly used by the OS.
|
||||
This includes resource forks on Mac OS X.
|
||||
.. [#flags] aka *BSD flags*. The Linux set of flags [1]_ is portable across platforms.
|
||||
The BSDs define additional flags.
|
||||
.. include:: usage/general/file-metadata.rst.inc
|
||||
|
Loading…
Reference in New Issue
Block a user