diff --git a/docs/development.rst b/docs/development.rst index 7d7e3113e5..01fe19fdf7 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -310,12 +310,22 @@ Important notes: - When using ``--`` to give options to py.test, you MUST also give ``borg.testsuite[.module]``. +Running pytest directly without tox: + +:: + + # run the tests, with parallelism, skipping remote tests + pytest -v -rs --benchmark-skip -n auto -k "not remote" + + # run the tests, skipping remote tests, only running repository test + pytest -v -rs --benchmark-skip -k "not remote and repository" + Running the tests (using the pypi package) ------------------------------------------ -Since borg 1.4, it is also possible to run the tests without a development -environment, using the borgbackup dist package (downloaded from pypi.org or -github releases page): +It is also possible to run the tests without a development environment, using +the borgbackup dist package (downloaded from pypi.org or github releases page): + :: # optional: create and use a virtual env: @@ -453,19 +463,12 @@ Creating standalone binaries ---------------------------- Make sure you have everything built and installed (including fuse stuff). -When using the Vagrant VMs, pyinstaller will already be installed. With virtual env activated:: - pip install pyinstaller # or git checkout master - pyinstaller -F -n borg-PLATFORM borg/__main__.py - for file in dist/borg-*; do gpg --armor --detach-sign $file; done - -If you encounter issues, see also our `Vagrantfile` for details. + scripts/build-borg-using-pyinstaller.sh + scripts/build-borg-using-nuitka.sh -.. note:: Standalone binaries built with pyinstaller are supposed to - work on same OS, same architecture (x86 32bit, amd64 64bit) - without external dependencies. .. _releasing: diff --git a/docs/internals.rst b/docs/internals.rst index 0ac249437b..89f9ed523f 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -35,8 +35,8 @@ hash table of all chunks that already exist. The "key" object provides both compression and authenticated encryption used by the data access layer. The "key" object represents the sole trust boundary in Borg. - The lowest layer is the repository, either accessed directly - (Repository) or remotely (RemoteRepository). + The lowest layer is the repository accessed via class Repository. + Repository uses ``borgstore`` internally. .. toctree:: :caption: Internals contents diff --git a/docs/internals/structure.odg b/docs/internals/structure.odg index d68e89d028..103a884ea4 100644 Binary files a/docs/internals/structure.odg and b/docs/internals/structure.odg differ diff --git a/docs/internals/structure.png b/docs/internals/structure.png index 5c17db6d5e..f0eb5c839e 100644 Binary files a/docs/internals/structure.png and b/docs/internals/structure.png differ diff --git a/docs/usage/general/environment.rst.inc b/docs/usage/general/environment.rst.inc index 5422cb5934..120025979c 100644 --- a/docs/usage/general/environment.rst.inc +++ b/docs/usage/general/environment.rst.inc @@ -71,9 +71,6 @@ General: BORG_FILES_CACHE_TTL When set to a numeric value, this determines the maximum "time to live" for the files cache entries (default: 2). The files cache is used to determine quickly whether a file is unchanged. - BORG_USE_CHUNKS_ARCHIVE - When set to no (default: yes), the ``chunks.archive.d`` folder will not be used. This reduces - disk space usage but slows down cache resyncs. BORG_SHOW_SYSINFO When set to no (default: yes), system information (like OS, Python version, ...) in exceptions is not shown. diff --git a/docs/usage/general/resources.rst.inc b/docs/usage/general/resources.rst.inc index 7706f69510..89be824807 100644 --- a/docs/usage/general/resources.rst.inc +++ b/docs/usage/general/resources.rst.inc @@ -13,11 +13,12 @@ server to get the approximate resource usage. CPU client: - **borg create:** chunking, hashing, compression, encryption (high CPU usage) - - **chunks cache sync:** quite heavy on CPU, doing lots of hash table operations + - **chunks index rebuild:** quite heavy on CPU, doing lots of hash table operations - **borg extract:** decryption, decompression (medium to high CPU usage) - - **borg check:** similar to extract, but depends on options given - - **borg prune/borg delete archive:** low to medium CPU usage + - **borg prune/borg delete archive:** quick, low CPU usage - **borg delete repo:** done on the server + - **borg compact:** medium CPU usage + - **borg check:** medium CPU usage, but depends on options given It will not use more than 100% of one CPU core as the code is currently single-threaded. Especially higher zlib and lzma compression levels use significant amounts @@ -26,11 +27,11 @@ CPU client: CPU server: It usually does not need much CPU; it just deals with the key/value store - (repository) and uses the repository index for that. + (repository). borg check: the repository check computes the checksums of all chunks (medium CPU usage) - borg delete repo: low CPU usage + borg compact: low to medium CPU usage CPU (only for client/server operation): When using Borg in a client/server way with an ssh-type repository, the SSH @@ -41,12 +42,11 @@ CPU (only for client/server operation): Memory (RAM) client: The chunks index and the files index are read into memory for performance reasons. Might need large amounts of memory (see below). - Compression, especially lzma compression with high levels, might need substantial + Compression, especially with high compression levels, might need substantial amounts of memory. Memory (RAM) server: - The server process will load the repository index into memory. Might need - considerable amounts of memory, but less than on the client (see below). + Usually rather low memory needs, much less than the client. Chunks index (client only): Proportional to the number of data chunks in your repo. Lots of chunks @@ -58,38 +58,16 @@ Files index (client only): off (see create options), but the next backup might be much slower if you do. The speed benefit of using the files cache is proportional to file size. -Repository index (server only): - Proportional to the number of data chunks in your repo. Lots of chunks - in your repo imply a big repository index. - It is possible to tweak the chunker parameters (see create options) to - influence the number of chunks created. - Temporary files (client): - Reading data and metadata from a FUSE-mounted repository will consume up to - the size of all deduplicated, small chunks in the repository. Big chunks - will not be locally cached. + TODO Temporary files (server): - A non-trivial amount of data will be stored in the remote temporary directory - for each client that connects to it. For some remotes, this can fill the - default temporary directory in /tmp. This can be mitigated by ensuring the - $TMPDIR, $TEMP, or $TMP environment variable is properly set for the sshd - process. - For some OSes, this can be done by setting the correct value in the - .bashrc (or equivalent login config file for other shells); however, in - other cases it may be necessary to first enable ``PermitUserEnvironment yes`` - in your ``sshd_config`` file, then add ``environment="TMPDIR=/my/big/tmpdir"`` - at the start of the public key to be used in the ``authorized_keys`` file. + TODO Cache files (client only): - Contains the chunks index and files index (plus a collection of single- - archive chunk indexes), which might need huge amounts of disk space - depending on archive count and size — see the FAQ for how to reduce this. + Contains the files cache, which might become quite large depending on the + amount and size of files. Network (only for client/server operation): If your repository is remote, all deduplicated (and optionally compressed/ - encrypted) data has to go over the connection (``ssh://`` repository URL). - If you use a locally mounted network filesystem, some additional copy - operations used for transaction support also go over the connection. If - you back up multiple sources to one target repository, additional traffic - happens for cache resynchronization. + encrypted) data has to go over the network connection. diff --git a/docs/usage/notes.rst b/docs/usage/notes.rst index b40666ebb3..6ed17e143d 100644 --- a/docs/usage/notes.rst +++ b/docs/usage/notes.rst @@ -174,24 +174,21 @@ Now, let's see how to restore some LVs from such a backup. Separate compaction ~~~~~~~~~~~~~~~~~~~ -Borg does not auto-compact the segment files in the repository at commit time -(at the end of each repository-writing command) any more (since borg 1.2.0). +Borg does not automatically compact the files in the repository when doing +borg operations. This has some notable consequences: - repository space is not freed immediately when deleting / pruning archives -- commands finish quicker -- repository is more robust and might be easier to recover after damages (as - it contains data in a more sequential manner, historic manifests, multiple - commits - until you run ``borg compact``) +- commands finish quicker, less I/O +- it is possible to undelete deleted archives as long as no compaction has been + invoked. - user can choose when to run compaction (it should be done regularly, but not necessarily after each single borg command) -- user can choose from where to invoke ``borg compact`` to do the compaction - (from client or from server, it does not need a key) - less repo sync data traffic in case you create a copy of your repository by using a sync tool (like rsync, rclone, ...) -You can manually run compaction by invoking the ``borg compact`` command. +You can run compaction by invoking the ``borg compact`` command. SSH batch mode ~~~~~~~~~~~~~~ diff --git a/docs/usage/repo-create.rst b/docs/usage/repo-create.rst index 80ccba40f3..59765205aa 100644 --- a/docs/usage/repo-create.rst +++ b/docs/usage/repo-create.rst @@ -9,20 +9,25 @@ Examples # Local repository $ export BORG_REPO=/path/to/repo # Recommended AEAD cryptographic modes (key stored in the repository by default) - $ borg repo-create --encryption=aes-ocb + $ borg repo-create --encryption=aes256-ocb $ borg repo-create --encryption=chacha20-poly1305 - $ borg repo-create --encryption=blake3-aes-ocb - $ borg repo-create --encryption=blake3-chacha20-poly1305 # No encryption (not recommended) $ borg repo-create --encryption=authenticated - $ borg repo-create --encryption=authenticated-blake3 $ borg repo-create --encryption=none - # The crypto suite (--encryption) and where the key is stored (--key-location) are - # chosen independently. --key-location defaults to repokey. + # --encryption (the cipher / AE algorithm) and --id-hash (the id hash function) are + # chosen independently. --id-hash defaults to sha256; use blake3 if it is faster on + # your hardware (run 'borg benchmark cpu' to find out). The 'none' encryption only + # supports the sha256 id hash. + $ borg repo-create --encryption=aes256-ocb --id-hash=blake3 + $ borg repo-create --encryption=chacha20-poly1305 --id-hash=blake3 + $ borg repo-create --encryption=authenticated --id-hash=blake3 + + # Where the key is stored (--key-location) is also chosen independently. + # --key-location defaults to repokey. # repokey: stores the encrypted key inside the repository - $ borg repo-create --encryption=aes-ocb --key-location=repokey + $ borg repo-create --encryption=aes256-ocb --key-location=repokey # keyfile: stores the encrypted key in the config dir's keys/ subdir # (e.g. ~/.config/borg/keys/ on Linux, ~/Library/Application Support/borg/keys/ on macOS) - $ borg repo-create --encryption=aes-ocb --key-location=keyfile + $ borg repo-create --encryption=aes256-ocb --key-location=keyfile diff --git a/docs/usage/serve.rst b/docs/usage/serve.rst index e9838796b5..693343add6 100644 --- a/docs/usage/serve.rst +++ b/docs/usage/serve.rst @@ -41,12 +41,6 @@ locations like ``/etc/environment`` or in the forced command itself (example bel This automatically blocks potentially dangerous SSH features, even when they are added in a future update. Thus, this option should be preferred. - If you are using OpenSSH server < 7.2, however, you must explicitly - specify the SSH features to restrict and cannot simply use the ``restrict`` option, as it - was introduced in v7.2. We recommend using - ``no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc`` - in this case. - Details about sshd usage: `sshd(8) `_ .. _ssh_configuration: