Updates regarding OCI artifacts.

docs/contributing/05-oci-artifacts.md

@@ -0,0 +1,39 @@
+---
+slug: /oci-artifacts
+title: OCI Artifacts
+sidebar_position: 5
+---
+
+# OCI Artifacts
+
+Certain artifacts of metal-stack are not shipped as Docker containers but in a more generic registry container format following the [OCI](https://opencontainers.org/) specification. Examples for these artifacts are the metal-stack release vectors as defined by the [releases](https://github.com/metal-stack/releases) repository or ansible-roles that can be used for deploying metal-stack.
+
+The OCI artifacts have an expected format convention, which is described on this page.
+
+## Release Vector Artifacts
+
+This OCI artifact expects a layer with the artifact type `application/vnd.metal-stack.release-vector.v1` including one gzipped tar file called `release.tar.gz`, which should be marked with custom media type `application/vnd.metal-stack.release-vector.v1.tar+gzip`.
+
+Inside the tar file, there is a `release.yaml` file that contains a metal-stack release vector.
+
+The metal-stack release vector has a free format but by default expects an `ansible-roles` key at the root, mapping the role names to OCI artifacts and versions, like:
+
+```
+ansible-roles:
+  <example>:
+    oci: <image-name>
+    version: <image-ref>
+  # e.g.:
+  ansible-common:
+    oci: ghcr.io/metal-stack/ansible-common
+    repository: https://github.com/metal-stack/ansible-common
+    version: v0.7.2
+```
+
+If this convention is not followed, it is not possible to install ansible-roles through the `metal_stack_release_vector` image as provided by the metal-deployment-base deployment base image.
+
+## Ansible Roles
+
+This OCI artifact expects a layer with the artifact type `application/vnd.metal-stack.release-vector.v1` including one gzipped tar file called `ansible-role.tar.gz`, which should be marked with custom media type `application/vnd.metal-stack.ansible-role.v1.tar+gzip`.
+
+Inside the tar file, there is **one folder** containing the ansible-role to install. Please do not include multiple folders as otherwise the `metal_stack_release_vector` module cannot alias role names, which is sometimes required for deployments.

docs/contributing/05-community.md → docs/contributing/06-community.md

@@ -1,7 +1,7 @@
 ---
 slug: /community
 title: Community
-sidebar_position: 5
+sidebar_position: 6
 draft: true
 ---
 

docs/docs/04-For Operators/03-deployment-guide.mdx

@@ -62,7 +62,7 @@ At the end of this section we are gonna end up with the following files and fold
 │   ├── control-plane.yaml
 │   └── group_vars
 │       ├── all
-│       │   └── images.yaml
+│       │   └── release_vector.yaml
 │       └── control-plane
 │           ├── common.yaml
 │           └── metal.yml
@@ -73,52 +73,43 @@ At the end of this section we are gonna end up with the following files and fold
             └── main.yaml
 ```
 
-You can already define the `inventories/group_vars/all/images.yaml` file. It contains the metal-stack version you are gonna deploy:
+### Releases and Ansible Role Dependencies
+
+As metal-stack consists of many microservices all having individual versions, we have come up with a [releases](https://github.com/metal-stack/releases) repository. It contains a YAML file (we often call it release vector) describing the fitting versions of all components for every release of metal-stack. Ansible role dependencies are also part of a metal-stack release. Both the metal-stack release vector and the metal-stack ansible-roles are shipped as OCI artifacts following a specific format that's described [here](../../contributing/05-oci-artifacts.md). These artifacts are signed with the CI token of the metal-stack Github organization and can be verified using [cosign](https://github.com/sigstore/cosign).
+
+In order to download the release vector and the referenced ansible-roles prior to a deployment, we provide a small helper module called `metal_stack_release_vector` as part of the [metal-deployment-base](https://github.com/metal-stack/metal-deployment-base) deployment image. It's main tasks are:
+
+- Downloading the release vector OCI artifact.
+- Downloading the ansible-role OCI artifacts referenced in the release vector.
+- Validating the release vector and the ansible-role signatures.
+- Make information from the release vector available as ansible variables that can be used during the deployment.
+
+The module picks up a magic variable called `metal_stack_release_vectors`, which can be defined in `inventories/group_vars/all/release_vector.yaml` like this:
 
 import CodeBlock from '@theme/CodeBlock';
 import latestRelease from '@site/src/version.json'
 export function ImageReleaseConfig() {
 return <><CodeBlock language="yaml">
 ---{"\n"}
 metal_stack_release_version: {latestRelease.version}
+
+metal_stack_release_vectors:
+  - url: oci://ghcr.io/metal-stack/releases:{{ metal_stack_release_version }}
+    variable_mapping_path: metal_stack_release.mapping
+    include_role_defaults: metal-roles/common/roles/defaults
+    oci_cosign_verify_key: |
+      -----BEGIN PUBLIC KEY-----
+      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEdeAXd2namgVNDT0APmogKGwaV+Q4
+      rfe4uVgmsyBbb6TrhX5Py6x1PsonDahTvdVpbSGC7QGEjxIHdi8HnJ4Okg==
+      -----END PUBLIC KEY-----
 </CodeBlock></>
 }
 
 <ImageReleaseConfig />
 
-### Releases and Ansible Role Dependencies
-
-As metal-stack consists of many microservices all having individual versions, we have come up with a [releases](https://github.com/metal-stack/releases) repository. It contains a YAML file (we often call it release vector) describing the fitting versions of all components for every release of metal-stack.
+The public key for the validation is attached to each metal-stack release.
 
-Ansible role dependencies are also part of a metal-stack release. Therefore, we will now write up a playbook, which dynamically renders a `requirements.yaml` file from the ansible-roles defined in the release repository. The `requirements.yaml` can then be used to resolve the actual role dependencies through [Ansible Galaxy](https://galaxy.ansible.com/). Define the following playbook in `generate_role_requirements.yaml`:
-
-```yaml
----
-- name: generate requirements.yaml
-  hosts: control-plane
-  connection: local
-  gather_facts: false
-  vars:
-    release_vector_url: "https://raw.githubusercontent.com/metal-stack/releases/{{ metal_stack_release_version }}/release.yaml"
-  tasks:
-    - name: download release vector
-      uri:
-        url: "{{ release_vector_url }}"
-        return_content: yes
-      register: release_vector
-
-    - name: write requirements.yaml from release vector
-      copy:
-        dest: "{{ playbook_dir }}/requirements.yaml"
-        content: |
-          {% for role_name, role_params in (release_vector.content | from_yaml).get('ansible-roles').items() %}
-          - src: {{ role_params.get('repository') }}
-            name: {{ role_name }}
-            version: {{ hostvars[inventory_hostname][role_name | lower | replace('-', '_') + '_version'] | default(role_params.get('version'), true) }}
-          {% endfor %}
-```
-
-This playbook will always be run before the actual metal-stack deployment and provide you with the proper versions of the Ansible role dependencies.
+Optional further parametrization for this module can be found its [module documentation](https://github.com/metal-stack/ansible-common/blob/a5977fc618cd34a6cebda262602b8cbc0632500d/library/metal_stack_release_vector.py).
 
 ### Inventory
 
@@ -134,24 +125,6 @@ control-plane:
 
 We do this since we are deploying to Kubernetes and do not need to SSH-connect to any hosts for the deployment (which is what Ansible typically does). This inventory is also necessary to pick up the variables inside `inventories/group_vars/control-plane` during the deployment.
 
-We recommend using the following `ansible.cfg`:
-
-```ini
-[defaults]
-retry_files_enabled = false
-force_color = true
-host_key_checking = false
-stdout_callback = yaml
-jinja2_native = true
-transport = ssh
-timeout = 30
-force_valid_group_names = ignore
-
-[ssh_connection]
-retries=3
-ssh_executable = /usr/bin/ssh
-```
-
 Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases.
 
 ### Control Plane Playbook
@@ -164,27 +137,16 @@ Next, we will define the actual deployment playbook in a file called `deploy_met
   hosts: control-plane
   connection: local
   gather_facts: no
-  vars:
-    setup_yaml:
-      - url: https://raw.githubusercontent.com/metal-stack/releases/{{ metal_stack_release_version }}/release.yaml
-        meta_var: metal_stack_release
   roles:
     - name: ansible-common
       tags: always
     - name: ingress-controller
-      tags: ingress-controller
     - name: metal-roles/control-plane/roles/prepare
-      tags: prepare
     - name: metal-roles/control-plane/roles/nsq
-      tags: nsq
     - name: metal-roles/control-plane/roles/metal-db
-      tags: metal-db
     - name: metal-roles/control-plane/roles/ipam-db
-      tags: ipam-db
     - name: metal-roles/control-plane/roles/masterdata-db
-      tags: masterdata-db
     - name: metal-roles/control-plane/roles/metal
-      tags: metal
 ```
 
 Basically, this playbook does the following:
@@ -201,7 +163,7 @@ Basically, this playbook does the following:
 
 ### Setup an ingress-controller
 
-As a next step you have to add a task for deploying an ingress-controller into your cluster. [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) is what we use. If you want to use another ingress-controller, you need to parametrize the metal roles carefully. When you just use ingress-nginx, make sure to also deploy it to the default namespace ingress-nginx.
+As a next step we write the `ingress-controller` role for deploying an ingress-controller into the cluster. [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) is what we use. If you want to use another ingress-controller, you need to parametrize the metal roles carefully. When you just use ingress-nginx, make sure to also deploy it to the default namespace ingress-nginx.
 
 This is how your `roles/ingress-controller/tasks/main.yaml` could look like:
 
@@ -460,6 +422,9 @@ export METAL_VERSION={latestRelease.version}
 Then you can spin up the deployment with docker:
 
 ```bash
+# ideally, validate the signature of the deployment image with cosign before running it:
+cosign verify ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} --certificate-oidc-issuer https://accounts.google.com --certificate-identity keyless@metal-stack.iam.gserviceaccount.com
+# then run the deployment:
 docker run --rm -it \
   -v $(pwd):/workdir \
   --workdir /workdir \
@@ -468,8 +433,7 @@ docker run --rm -it \
   -e ANSIBLE_INVENTORY=inventories/control-plane.yaml \
   ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} \
   /bin/bash -ce \
-    "ansible-playbook obtain_role_requirements.yaml
-     ansible-galaxy install -r requirements.yaml
+    "ansible -m metalstack.base.metal_stack_release_vector localhost
      ansible-playbook deploy_metal_control_plane.yaml"
 ```