docs fixes (master)

docs/development.rst

@@ -310,12 +310,22 @@ Important notes:
 
 - When using ``--`` to give options to py.test, you MUST also give ``borg.testsuite[.module]``.
 
+Running pytest directly without tox:
+
+::
+
+    # run the tests, with parallelism, skipping remote tests
+    pytest -v -rs --benchmark-skip -n auto -k "not remote"
+
+    # run the tests, skipping remote tests, only running repository test
+    pytest -v -rs --benchmark-skip -k "not remote and repository"
+
 Running the tests (using the pypi package)
 ------------------------------------------
 
-Since borg 1.4, it is also possible to run the tests without a development
-environment, using the borgbackup dist package (downloaded from pypi.org or
-github releases page):
+It is also possible to run the tests without a development environment, using
+the borgbackup dist package (downloaded from pypi.org or github releases page):
+
 ::
 
     # optional: create and use a virtual env:
@@ -453,19 +463,12 @@ Creating standalone binaries
 ----------------------------
 
 Make sure you have everything built and installed (including fuse stuff).
-When using the Vagrant VMs, pyinstaller will already be installed.
 
 With virtual env activated::
 
-  pip install pyinstaller  # or git checkout master
-  pyinstaller -F -n borg-PLATFORM borg/__main__.py
-  for file in dist/borg-*; do gpg --armor --detach-sign $file; done
-
-If you encounter issues, see also our `Vagrantfile` for details.
+  scripts/build-borg-using-pyinstaller.sh
+  scripts/build-borg-using-nuitka.sh
 
-.. note:: Standalone binaries built with pyinstaller are supposed to
-          work on same OS, same architecture (x86 32bit, amd64 64bit)
-          without external dependencies.
 
 .. _releasing:
 

docs/internals.rst

@@ -35,8 +35,8 @@ hash table of all chunks that already exist.
     The "key" object provides both compression and authenticated
     encryption used by the data access layer. The "key" object represents
     the sole trust boundary in Borg.
-    The lowest layer is the repository, either accessed directly
-    (Repository) or remotely (RemoteRepository).
+    The lowest layer is the repository accessed via class Repository.
+    Repository uses ``borgstore`` internally.
 
 .. toctree::
     :caption: Internals contents

docs/usage/general/environment.rst.inc

@@ -71,9 +71,6 @@ General:
     BORG_FILES_CACHE_TTL
         When set to a numeric value, this determines the maximum "time to live" for the files cache
         entries (default: 2). The files cache is used to determine quickly whether a file is unchanged.
-    BORG_USE_CHUNKS_ARCHIVE
-        When set to no (default: yes), the ``chunks.archive.d`` folder will not be used. This reduces
-        disk space usage but slows down cache resyncs.
     BORG_SHOW_SYSINFO
         When set to no (default: yes), system information (like OS, Python version, ...) in
         exceptions is not shown.

docs/usage/general/resources.rst.inc

@@ -13,11 +13,12 @@ server to get the approximate resource usage.
 
 CPU client:
     - **borg create:** chunking, hashing, compression, encryption (high CPU usage)
-    - **chunks cache sync:** quite heavy on CPU, doing lots of hash table operations
+    - **chunks index rebuild:** quite heavy on CPU, doing lots of hash table operations
     - **borg extract:** decryption, decompression (medium to high CPU usage)
-    - **borg check:** similar to extract, but depends on options given
-    - **borg prune/borg delete archive:** low to medium CPU usage
+    - **borg prune/borg delete archive:** quick, low CPU usage
     - **borg delete repo:** done on the server
+    - **borg compact:** medium CPU usage
+    - **borg check:** medium CPU usage, but depends on options given
 
     It will not use more than 100% of one CPU core as the code is currently single-threaded.
     Especially higher zlib and lzma compression levels use significant amounts
@@ -26,11 +27,11 @@ CPU client:
 
 CPU server:
     It usually does not need much CPU; it just deals with the key/value store
-    (repository) and uses the repository index for that.
+    (repository).
 
     borg check: the repository check computes the checksums of all chunks
     (medium CPU usage)
-    borg delete repo: low CPU usage
+    borg compact: low to medium CPU usage
 
 CPU (only for client/server operation):
     When using Borg in a client/server way with an ssh-type repository, the SSH
@@ -41,12 +42,11 @@ CPU (only for client/server operation):
 Memory (RAM) client:
     The chunks index and the files index are read into memory for performance
     reasons. Might need large amounts of memory (see below).
-    Compression, especially lzma compression with high levels, might need substantial
+    Compression, especially with high compression levels, might need substantial
     amounts of memory.
 
 Memory (RAM) server:
-    The server process will load the repository index into memory. Might need
-    considerable amounts of memory, but less than on the client (see below).
+    Usually rather low memory needs, much less than the client.
 
 Chunks index (client only):
     Proportional to the number of data chunks in your repo. Lots of chunks
@@ -58,38 +58,16 @@ Files index (client only):
     off (see create options), but the next backup might be much slower if you do.
     The speed benefit of using the files cache is proportional to file size.
 
-Repository index (server only):
-    Proportional to the number of data chunks in your repo. Lots of chunks
-    in your repo imply a big repository index.
-    It is possible to tweak the chunker parameters (see create options) to
-    influence the number of chunks created.
-
 Temporary files (client):
-    Reading data and metadata from a FUSE-mounted repository will consume up to
-    the size of all deduplicated, small chunks in the repository. Big chunks
-    will not be locally cached.
+    TODO
 
 Temporary files (server):
-    A non-trivial amount of data will be stored in the remote temporary directory
-    for each client that connects to it. For some remotes, this can fill the
-    default temporary directory in /tmp. This can be mitigated by ensuring the
-    $TMPDIR, $TEMP, or $TMP environment variable is properly set for the sshd
-    process.
-    For some OSes, this can be done by setting the correct value in the
-    .bashrc (or equivalent login config file for other shells); however, in
-    other cases it may be necessary to first enable ``PermitUserEnvironment yes``
-    in your ``sshd_config`` file, then add ``environment="TMPDIR=/my/big/tmpdir"``
-    at the start of the public key to be used in the ``authorized_keys`` file.
+    TODO
 
 Cache files (client only):
-    Contains the chunks index and files index (plus a collection of single-
-    archive chunk indexes), which might need huge amounts of disk space
-    depending on archive count and size — see the FAQ for how to reduce this.
+    Contains the files cache, which might become quite large depending on the
+    amount and size of files.
 
 Network (only for client/server operation):
     If your repository is remote, all deduplicated (and optionally compressed/
-    encrypted) data has to go over the connection (``ssh://`` repository URL).
-    If you use a locally mounted network filesystem, some additional copy
-    operations used for transaction support also go over the connection. If
-    you back up multiple sources to one target repository, additional traffic
-    happens for cache resynchronization.
+    encrypted) data has to go over the network connection.

docs/usage/notes.rst

@@ -174,24 +174,21 @@ Now, let's see how to restore some LVs from such a backup.
 Separate compaction
 ~~~~~~~~~~~~~~~~~~~
 
-Borg does not auto-compact the segment files in the repository at commit time
-(at the end of each repository-writing command) any more (since borg 1.2.0).
+Borg does not automatically compact the files in the repository when doing
+borg operations.
 
 This has some notable consequences:
 
 - repository space is not freed immediately when deleting / pruning archives
-- commands finish quicker
-- repository is more robust and might be easier to recover after damages (as
-  it contains data in a more sequential manner, historic manifests, multiple
-  commits - until you run ``borg compact``)
+- commands finish quicker, less I/O
+- it is possible to undelete deleted archives as long as no compaction has been
+  invoked.
 - user can choose when to run compaction (it should be done regularly, but not
   necessarily after each single borg command)
-- user can choose from where to invoke ``borg compact`` to do the compaction
-  (from client or from server, it does not need a key)
 - less repo sync data traffic in case you create a copy of your repository by
   using a sync tool (like rsync, rclone, ...)
 
-You can manually run compaction by invoking the ``borg compact`` command.
+You can run compaction by invoking the ``borg compact`` command.
 
 SSH batch mode
 ~~~~~~~~~~~~~~

docs/usage/repo-create.rst

@@ -9,20 +9,25 @@ Examples
     # Local repository
     $ export BORG_REPO=/path/to/repo
     # Recommended AEAD cryptographic modes (key stored in the repository by default)
-    $ borg repo-create --encryption=aes-ocb
+    $ borg repo-create --encryption=aes256-ocb
     $ borg repo-create --encryption=chacha20-poly1305
-    $ borg repo-create --encryption=blake3-aes-ocb
-    $ borg repo-create --encryption=blake3-chacha20-poly1305
     # No encryption (not recommended)
     $ borg repo-create --encryption=authenticated
-    $ borg repo-create --encryption=authenticated-blake3
     $ borg repo-create --encryption=none
 
-    # The crypto suite (--encryption) and where the key is stored (--key-location) are
-    # chosen independently. --key-location defaults to repokey.
+    # --encryption (the cipher / AE algorithm) and --id-hash (the id hash function) are
+    # chosen independently. --id-hash defaults to sha256; use blake3 if it is faster on
+    # your hardware (run 'borg benchmark cpu' to find out). The 'none' encryption only
+    # supports the sha256 id hash.
+    $ borg repo-create --encryption=aes256-ocb --id-hash=blake3
+    $ borg repo-create --encryption=chacha20-poly1305 --id-hash=blake3
+    $ borg repo-create --encryption=authenticated --id-hash=blake3
+
+    # Where the key is stored (--key-location) is also chosen independently.
+    # --key-location defaults to repokey.
     # repokey: stores the encrypted key inside the repository
-    $ borg repo-create --encryption=aes-ocb --key-location=repokey
+    $ borg repo-create --encryption=aes256-ocb --key-location=repokey
     # keyfile: stores the encrypted key in the config dir's keys/ subdir
     # (e.g. ~/.config/borg/keys/ on Linux, ~/Library/Application Support/borg/keys/ on macOS)
-    $ borg repo-create --encryption=aes-ocb --key-location=keyfile
+    $ borg repo-create --encryption=aes256-ocb --key-location=keyfile
 

docs/usage/serve.rst

@@ -41,12 +41,6 @@ locations like ``/etc/environment`` or in the forced command itself (example bel
     This automatically blocks potentially dangerous SSH features, even when
     they are added in a future update. Thus, this option should be preferred.
 
-    If you are using OpenSSH server < 7.2, however, you must explicitly
-    specify the SSH features to restrict and cannot simply use the ``restrict`` option, as it
-    was introduced in v7.2. We recommend using
-    ``no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding,no-user-rc``
-    in this case.
-
 Details about sshd usage: `sshd(8) <https://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8>`_
 
 .. _ssh_configuration: