# metal-stack.io > Bring the cloud to your data center. metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. This file contains all documentation content in a single document following the llmstxt.org standard. ## Home # Welcome to the metal-stack docs! metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. To categorize this product, we use the terms _Metal-as-a-Service (MaaS)_ or _bare metal cloud_. From the perspective of a user, the metal-stack does not feel any different from working with a conventional cloud provider. Users manage their resources (machines, networks and ip addresses, etc.) by themselves, which effectively turns your data center into an elastic cloud infrastructure. The major difference to other cloud providers is that compute power and data reside in your own data center. --- ## Getting Started Before starting to buy any hardware, you should try out the metal-stack on your notebook and familiarize with the software. For this, we made the [mini-lab](https://github.com/metal-stack/mini-lab). The mini-lab is a fully virtual setup of metal-stack and is supposed to be run locally on a single machine. For this reason, the setup was slightly simplified in comparison to full-blown setups on real hardware. However, the lab should help to understand all ideas behind the metal-stack. Get your hands dirty and follow the guide on how to get on with the mini-lab [here](https://github.com/metal-stack/mini-lab). --- ## Why metal-stack? Before we started with our mission to implement the metal-stack, we decided on a couple of key characteristics and constraints that we think are unique in the domain (otherwise we would definitely have chosen an existing solution). We hope that the following properties appeal to you as well. ## On-Premise Running on-premises gives you data sovereignty and usually a better price / performance ratio than with hyperscalers — especially the larger you grow your environment. Another benefit of running on-premises is an easier connectivity to existing company networks. ## Fast Provisioning Provisioning bare metal machines should not feel much different from virtual machines. metal-stack is capable of provisioning servers in less than a minute. The underlying network topology is based on BGP and allows announcing new routes to your host machines in a matter of seconds. ## No-Ops Part of the metal-stack runs on dedicated switches in your data center. This way, it is possible to automate server inventorization, permanently reconcile network configuration and automatically manage machine lifecycles. Manual configuration is neither required nor wanted. ## Security Our networking approach was designed for highest standards on security. Also, we enforce firewalling on dedicated tenant firewalls before users can establish connections to other networks than their private tenant network. API authentication and authorization is done with the help of OIDC. ## API driven The development of metal-stack is strictly API driven and offers self-service to end-users. This approach delivers the highest possible degree of automation, maintainability and performance. ## Ready for Kubernetes Not only does the metal-stack run smoothly on [Kubernetes](https://kubernetes.io/) (K8s). The major intent of metal-stack has always been to build a scalable machine infrastructure for _Kubernetes as a Service (KaaS)_. In partnership with the open-source project [Gardener](https://gardener.cloud/), we can provision Kubernetes clusters on metal-stack at scale. From the perspective of the Gardener, the metal-stack is just another cloud provider. The time savings compared to providing machines and Kubernetes by hand are significant. We actually want to be able to compete with offers of public cloud providers, especially regarding speed and usability. Of course, you can use metal-stack only for machine provisioning as well and just put something else on top of your metal infrastructure. ## Open Source The metal-stack is open source and free of constraints regarding vendors and third-party products. The stack is completely built on open source products. We have a community actively working on the metal-stack, which can assist you delivering all reasonable features you are gonna need. --- ## Why Bare Metal? Bare metal has several advantages over virtual environments and overcomes several drawbacks of virtual machines. We also listed drawbacks of the bare metal approach. Bare in mind though that it is still possible to virtualize on bare metal environments when you have your stack up and running. ## Virtual Environment Drawbacks - [Spectre and Meltdown](https://meltdownattack.com/) can only be mitigated with a "cluster per tenant" approach - Missing isolation of multi-tenant change impacts - Licensing restrictions - Noisy-neighbors ## Bare Metal Advantages - Guaranteed and fastest possible performance (especially disk i/o) - Reduced stack depth (Host / VM / Application vs. Host / Container) - Reduced attack surface - Lower costs, higher performance - No VM live-migrations - Bigger hardware configurations possible (hypervisors have restrictions, e.g. it is not possible to assign all CPUs to a single VM) ## Bare Metal Drawbacks - Hardware defects have direct impact (should be considered by design) and can not be mitigated by live-migration as in virtual environments - Capacity planning is more difficult (no resource overbooking possible) --- ## Flavors of metal-stack metal-stack bare metal provisioning offers machines, networks, firewalls and floating IPs. That`s it. You can consume it as-is with our [Plain Flavor](#plain) or use it as foundation for an orchestrator. We offer two K8s Flavors [Gardener](#gardener) and [Cluster API](#cluster-api). ## Plain All flavors start with this. This is what you get if you set up metal-stack and stop there. Using plain metal-stack without additional layer was not a focus in the past. Therefore firewall features and role management are quite basic. There is ongoing work on [improved RBAC in MEP-4](/community/MEP-4-multi-tenancy-for-the-metal-api) and [firewall configuration via metal-api in MEP-16](/community/MEP-16-metal-api-as-an-alternative-configuration-source-for-the-firewall-controller). If you want more features, keep reading. ## Gardener [Gardener](https://gardener.cloud/) is an open-source managed Kubernetes service. It provides a good "batteries-included" developer experience and should be your first choice for a Kubernetes-as-a-service solution. Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/01-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). ## Cluster API Our [Cluster API integration](../05-Concepts/04-Kubernetes/02-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. --- ## Client Libraries Our public-facing APIs are built on [swagger](https://swagger.io/), which allows you generating API clients in all sorts of programming languages. For the [metal-api](https://github.com/metal-stack/metal-api) we officially support the following client libraries: - [metal-go](https://github.com/metal-stack/metal-go) - [metal-python](https://github.com/metal-stack/metal-python) --- ## Hardware Support In order to keep the automation and maintenance overhead small, we strongly advise against building highly heterogeneous environments with metal-stack. Having a lot of different vendors and server models in your partitions will heavily increase the time and effort for introducing metal-stack in your infrastructure. From experience we can tell that the interfaces for automating hardware provisioning are usually inconsistent between vendors and even between server models of the same vendor. Therefore, we encourage adopters to start off with only a small amount of machine types. If you want to be on the safe side, you should consider buying the hardware that we officially support. We came up with a repository called [go-hal](https://github.com/metal-stack/go-hal), which includes the interface required for metal-stack to support a machine vendor. If you plan to implement support for new vendors, please check out this repository and contribute back your efforts in order to make the community benefit from extended vendor support as well. ## Servers The following server types are officially supported and verified by the metal-stack project: | Vendor | Series | Model | Board Type | Status | |------------|--------------------------------|------------------|:---------------|:-------| | Supermicro | Big-Twin | SYS-2029BT-HNR | X11DPT-B | stable | | Supermicro | Big-Twin | SYS-220BT-HNTR | X12DPT-B6 | stable | | Supermicro | SuperServer | SSG-5019D8-TR12P | X11SDV-8C-TP8F | stable | | Supermicro | SuperServer | 2029UZ-TN20R25M | X11DPU | stable | | Supermicro | SuperServer | SYS-621C-TN12R | X13DDW-A | stable | | Supermicro | Microcloud | 5039MD8-H8TNR | X11SDD-8C-F | stable | | Supermicro | Microcloud | SYS-531MC-H8TNR | X13SCD-F | stable | | Supermicro | Microcloud | 3015MR-H8TNR | H13SRD-F | stable | | Supermicro | Workstation | AS-2115HV-TNRT | H13SRH | stable | | Lenovo | ThinkSystem | SD530 | | alpha | | Gigabyte | OCP Open Rack line | | | alpha | | Dell | PowerEdge Service with iDRAC 9 | | | alpha | Other server series and models might work but were not reported to us. ## GPUs The following GPU types are officially supported and verified by the metal-stack project: | Vendor | Model | Status | |--------|----------|:-------| | NVIDIA | RTX 6000 | stable | | NVIDIA | H100 | stable | | NVIDIA | H200 | stable | Other GPU models might work but were not reported to us. For a detailed description howto use GPU support in a kubernetes cluster please check this [documentation](../05-Concepts/04-Kubernetes/06-gpu-workers.md). ## Network Cards The following network cards are officially supported and verified by the metal-stack project for usage in servers: | Vendor | Series | Model | Status | |----------|------------|------------------------------|:-------| | Intel | XXV710 | DA2 DualPort 2x25G SFP28 | stable | | Intel | E810 | DA2 DualPort 2x25G SFP28 | stable | | Intel | E810 | CQDA2 DualPort 2x100G QSFP28 | stable | | Mellanox | ConnectX-5 | MCX512A-ACAT 2x25G SFP28 | stable | ## Switches The following switch types are officially supported and verified by the metal-stack project: | Vendor | Series | Model | OS | Status | |:----------|:--------------|:------------|:---------------|:-------| | Edge-Core | AS4600 Series | AS4625-54T | Edgecore SONiC | stable | | Edge-Core | AS4600 Series | AS4630-54TE | Edgecore SONiC | stable | | Edge-Core | AS7700 Series | AS7726-32X | Cumulus 3.7.13 | stable | | Edge-Core | AS7700 Series | AS7726-32X | Edgecore SONiC | stable | Other switch series and models might work but were not reported to us. :::warning On our switches we run [SONiC](https://sonicfoundation.dev). The metal-core writes network configuration specifically implemented for this operating system. Please also consider running SONiC on your switches if you do not want to run into any issues with networking. [Cumulus Linux](https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-linux/) support in metal-stack is deprecated since Cumulus/Nvidia ended support for all of our currently supported switches. Of course, contributions for supporting other switch vendors and operating systems are highly appreciated. ::: ## Portable metal-stack Setup A minimal physical hardware setup may contain at least the following components: :::warning This setup dedicated to testing environments, getting to know the metal-stack software and discussing BOMs for production setups. ::: | # | Vendor | Series | Model | Function | |:---|:-----------|:--------------|:----------------|:-------------------------------------------------------------------------| | 1x | EdgeCore | AS5500 Series | AS4630-54x (1G) | Management Switch and Management Server | | 2x | EdgeCore | AS5500 Series | AS4625-54x (1G) | Leaf switches | | 1x | Supermicro | Microcloud | 3015MR-H8TNR | Usable machines | | 1x | Teltonika | Router | RUTXR1 | Front router for internet and out-of-band access to servers and switches | This setup will yield in 8 usable machines, one of them can be configured to provide persistent CSI storage. ![Portable metal-stack Setup](starter.jpg) --- ## Operating Systems Our operating system images are built on regular basis from the [metal-images](https://github.com/metal-stack/metal-images) repository. All images are hosted on GKE at [images.metal-stack.io](https://images.metal-stack.io). Feel free to use this as a mirror for your metal-stack partitions if you want. The metal-stack developers continuously have an eye on the supported images. They are updated regularly and scanned for vulnerabilities. ## Supported OS Images The operating system images that we build are trimmed down to their bare essentials for serving as Kubernetes worker nodes. Small image sizes make machine provisioning blazingly fast. The supported images for worker nodes currently are: | Platform | Distribution | Version | | :------- | :----------- | :------ | | Linux | Debian | 12 | | Linux | Debian | 13 | | Linux | Ubuntu | 24.04 | The supported images for firewalls are: | Platform | Distribution | Version | Based On | | :------- | :----------- | :------ | -------- | | Linux | Ubuntu | 3 | 24.04 | ## Image Lifecycle The metal-stack project provides images are derived from the official Docker images of the respective distribution. We install components required by the metal-stack infrastructure, e.g. FRR for routing-to-the-host and automation tools like [cloud-init](https://docs.cloud-init.io/en/latest/index.html) and [ignition](https://github.com/coreos/ignition) to run user provided post-install tasks. Every image registered with the metal-api carries one of the following lifecycle classifications: | Classification | Meaning | | :------------- | :--------------------------------------------------------------------------------------- | | preview | Recently introduced image, not yet recommended for production use | | supported | Regularly updated and scanned image, recommended for production use | | deprecated | Image will expire and be removed soon, users should switch to a newer image | These classifications prevent breaking changes in new images from affecting production workloads. :::info Operating system images are provided by the operators of a metal-stack installation, not by its end-users. End-users can only choose from the images that have been registered with the metal-api. If you require an additional image in an installation, get in touch with your operators. ::: :::info Official metal-stack images are validated for use as K8s Node OS. Results for the validation pipelines will be published later in 2026. ::: ## Building Your Own Images It is fully possible to build your own operating system images and provision them through the metal-stack without any in-tree implementations. There are some conventions though that you need to follow in order to make your image installable. You should understand the [machine provisioning sequence](../05-Concepts/01-architecture.mdx#machine-provisioning-sequence), the purpose of the [metal-hammer](https://github.com/metal-stack/metal-hammer) and the [os-installer](https://github.com/metal-stack/os-installer) before starting to write your own images. You can build own images as described in the following points: 1. Images need to be compressed to a tarball using the [lz4](https://de.wikipedia.org/wiki/LZ4) compression algorithm. These are unpacked by the metal-hammer after machine allocation onto the root drive of the machine. 1. An `md5` checksum file with the same name as the image archive needs to be provided in the download path along with the actual os image. This is used by the metal-hammer to check the integrity of the downloaded file. 1. A `packages.txt` containing the packages contained in the OS image should be provided in the download path (not strictly required). 1. Consider semantic image versioning, which we use in our algorithms to select latest images (e.g. `os-major.minor.patch` ➡️ `ubuntu-24.04.20191018`) 1. We strongly suggest installing some essential packages used by the metal-stack infrastructure: - [FRR](https://frrouting.org/) to enable routing-to-the-host in our network topology - [go-lldpd](https://github.com/metal-stack/go-lldpd) to enable checking if the machine is still alive after user allocation - [ignition](https://github.com/coreos/ignition) for enabling users to run user-specific initialization instructions before bootup. It's pretty small in size, which is why we use it. However, you are free to use other cloud instance initialization tools if you want to. 1. The [os-installer](https://github.com/metal-stack/os-installer) repository will be invoked by the metal-hammer in the os image's chroot during machine provisioning. You can check if you can use the existing provisioning tasks of this component for your OS image. - If the os-installer does not do what you need, it is possible to specify parameters for it in `/etc/metal/os-installer.yaml`. In there, it is possible to either use only specific functionality of the os-installer or even invoke custom provisioning scripts (total freedom). 1. For the time being, your image must be able to support [kexec](https://en.wikipedia.org/wiki/Kexec) into the new operating system kernel, the `kexec` command is issued by the metal-hammer after running the os-installer. We do this because `kexec` is _much_ faster than rebooting a machine. We recommend building images from Dockerfiles as it is done in [metal-images](https://github.com/metal-stack/metal-images) repository. :::info Building own operating system images is an advanced topic. When you have just started with metal-stack, we recommend using the public operating system images first. ::: --- ## Deployment Guide We are bootstrapping the [metal control plane](../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. In order to build up your deployment, we recommend to make use of the same Ansible roles that we are using by ourselves in order to deploy the metal-stack. You can find them in the repository called [metal-roles](https://github.com/metal-stack/metal-roles). In order to wrap up deployment dependencies there is a special [deployment base image](https://github.com/metal-stack/metal-deployment-base/pkgs/container/metal-deployment-base) hosted on GitHub that you can use for running the deployment. Using this Docker image eliminates a lot of moving parts in the deployment and should keep the footprints on your system fairly small and maintainable. This document will from now on assume that you want to use our Ansible deployment roles for setting up metal-stack. We will also use the deployment base image, so you should also have [Docker](https://docs.docker.com/get-started/get-docker/) installed. It is in the nature of software deployments to differ from site to site, company to company, user to user. Therefore, we can only describe you the way of how the deployment works for us. It is up to you to tweak the deployment described in this document to your requirements. :::warning Probably you need to learn writing Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge about Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). ::: :::info If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. However, you will probably be able to re-use some of our contents from our [metal-roles](https://github.com/metal-stack/metal-roles) repository, e.g. the Helm chart for deploying the metal control plane. ::: :::tip You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this document. ::: ## Metal Control Plane Deployment The metal control plane is typically deployed in a Kubernetes cluster. Therefore, this document will assume that you have a Kubernetes cluster ready for getting deployed. Even though it is theoretically possible to deploy metal-stack without Kubernetes, we strongly advise you to use the described method because we believe that Kubernetes gives you a lot of benefits regarding the stability and maintainability of the application deployment. :::tip For metal-stack it does not matter where your control plane Kubernetes cluster is located. You can of course use a cluster managed by a hyperscaler. This has the advantage of not having to setup Kubernetes by yourself and could even become beneficial in terms of fail-safe operation. However, we also describe a solution of how to setup metal-stack with a self-hosted, [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. The only requirement from metal-stack is that your partitions can establish network connections to the metal control plane. If you are interested, you can find a reasoning behind this deployment decision [here](../05-Concepts/01-architecture.mdx#target-deployment-platforms). ::: Let's start off with a fresh folder for your deployment: ```bash mkdir -p metal-stack-deployment cd metal-stack-deployment ``` At the end of this section we are gonna end up with the following files and folder structures: ``` . ├── ansible.cfg ├── deploy_metal_control_plane.yaml ├── files │   └── certs │      ├── ca-config.json │      ├── ca-csr.json │      ├── metal-api-grpc │      │   ├── client.json │      │   ├── server.json │      ├── masterdata-api │      │   ├── client.json │      │   ├── server.json │      └── roll_certs.sh ├── inventories │   ├── control-plane.yaml │   └── group_vars │      ├── all │      │   └── release_vector.yaml │      └── control-plane │        ├── common.yaml │         └── metal.yml ├── generate_role_requirements.yaml └── roles └── ingress-controller └── tasks └── main.yaml ``` ### Releases and Ansible Role Dependencies As metal-stack consists of many microservices all having individual versions, we have come up with a [releases](https://github.com/metal-stack/releases) repository. It contains a YAML file (we often call it release vector) describing the fitting versions of all components for every release of metal-stack. Ansible role dependencies are also part of a metal-stack release. Both the metal-stack release vector and the metal-stack ansible-roles are shipped as OCI artifacts following a specific format that's described [here](/community/oci-artifacts). These artifacts are signed with the CI token of the metal-stack Github organization and can be verified using [cosign](https://github.com/sigstore/cosign). In order to download the release vector and the referenced ansible-roles prior to a deployment, we provide a small helper module called `metal_stack_release_vector` as part of the [metal-deployment-base](https://github.com/metal-stack/metal-deployment-base) deployment image. Its main tasks are: - Downloading the release vector OCI artifact. - Downloading the ansible-role OCI artifacts referenced in the release vector. - Validating the release vector and the ansible-role signatures. - Make information from the release vector available as ansible variables that can be used during the deployment. The module picks up a magic variable called `metal_stack_release_vectors`, which can be defined in `inventories/group_vars/all/release_vector.yaml` like this: import CodeBlock from '@theme/CodeBlock'; import latestRelease from '@site/src/version.json' export function ImageReleaseConfig() { return <> ---{"\n"} metal_stack_release_vectors:{"\n"} {" "}- url: oci://ghcr.io/metal-stack/releases:{latestRelease.version}{"\n"} {" "} variable_mapping_path: metal_stack_release.mapping{"\n"} {" "} include_role_defaults: metal-roles/common/roles/defaults{"\n"} {" "} oci_cosign_verify_key: |{"\n"} {" "} -----BEGIN PUBLIC KEY-----{"\n"} {" "} MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEdeAXd2namgVNDT0APmogKGwaV+Q4{"\n"} {" "} rfe4uVgmsyBbb6TrhX5Py6x1PsonDahTvdVpbSGC7QGEjxIHdi8HnJ4Okg=={"\n"} {" "} -----END PUBLIC KEY-----{"\n"} } The public key for the validation is attached to each metal-stack release. Optional further parametrization for this module can be found in its [module documentation](https://github.com/metal-stack/ansible-common/blob/a5977fc618cd34a6cebda262602b8cbc0632500d/library/metal_stack_release_vector.py). ### Inventory Then, there will be an inventory for the control plane deployment in `inventories/control-plane.yaml` that adds `localhost` to the `control-plane` host group: ```yaml --- control-plane: hosts: localhost: ansible_python_interpreter: "{{ ansible_playbook_python }}" ``` We do this since we are deploying to Kubernetes and do not need to SSH-connect to any hosts for the deployment (which is what Ansible typically does). This inventory is also necessary to pick up the variables inside `inventories/group_vars/control-plane` during the deployment. Most of the properties in there are up to taste, but make sure you enable the [Jinja2 native environment](https://jinja.palletsprojects.com/en/2.11.x/nativetypes/) as this is needed for some of our roles in certain cases. ### Control Plane Playbook Next, we will define the actual deployment playbook in a file called `deploy_metal_control_plane.yaml`. You can start with the following lines: ```yaml --- - name: Deploy Control Plane hosts: control-plane connection: local gather_facts: no roles: - name: ansible-common tags: always - name: ingress-controller - name: metal-roles/control-plane/roles/prepare - name: metal-roles/control-plane/roles/nsq - name: metal-roles/control-plane/roles/metal-db - name: metal-roles/control-plane/roles/ipam-db - name: metal-roles/control-plane/roles/masterdata-db - name: metal-roles/control-plane/roles/metal ``` Basically, this playbook does the following: - Include all the modules, filter plugins, etc. of [ansible-common](https://github.com/metal-stack/ansible-common) into the play - Deploys an ingress-controller into your cluster - Deploys the metal-stack by - Running preparation tasks - Deploying NSQ - Deploying the rethinkdb database for the metal-api (wrapped in a backup-restore-sidecar), - Deploying the postgres database for go-ipam (wrapped in a backup-restore-sidecar) - Deploying the postgres database for the masterdata-api (wrapped in a backup-restore-sidecar) - Applying the metal control plane helm chart ### Setup an ingress-controller As a next step we write the `ingress-controller` role for deploying an ingress-controller into the cluster. [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) is what we use. If you want to use another ingress-controller, you need to parametrize the metal roles carefully. When you just use ingress-nginx, make sure to also deploy it to the default namespace ingress-nginx. This is how your `roles/ingress-controller/tasks/main.yaml` could look like: ```yaml - name: Deploy ingress-controller include_role: name: ansible-common/roles/helm-chart vars: helm_repo: "https://helm.nginx.com/stable" helm_chart: nginx-ingress helm_release_name: nginx-ingress helm_target_namespace: ingress-nginx ``` :::tip The [ansible-common](https://github.com/metal-stack/ansible-common) repository contains very general roles and modules that you can also use when extending your deployment further. ::: ### Deployment Parametrization Now you can parametrize the referenced roles to fit your environment. The role parametrization can be looked up in the role documentation on [metal-roles/control-plane](https://github.com/metal-stack/metal-roles/tree/master/control-plane). You should not need to define a lot of variables for the beginning as most values are reasonably defaulted. You can start with the following content for `group_vars/control-plane/common.yaml`: ```yaml --- metal_control_plane_ingress_dns: # if you do not have a DNS entry, you could also start with .nip.io ``` ### Providing Certificates We have several components in our stack that communicate over encrypted gRPC just like Kubernetes components do. For the very basic setup you will need to create self-signed certificates for the communication between the following components (see [architecture](../05-Concepts/01-architecture.mdx) document): - [metal-api](https://github.com/metal-stack/metal-api) and [masterdata-api](https://github.com/metal-stack/masterdata-api) (in-cluster traffic communication) - [metal-api](https://github.com/metal-stack/metal-api) and [metal-hammer](https://github.com/metal-stack/metal-hammer) (partition to control plane communication) Here is a snippet for `files/roll_certs.sh` that you can use for generating your certificates (requires [cfssl](https://github.com/cloudflare/cfssl)): ```bash #!/usr/bin/env bash set -eo pipefail for i in "$@" do case $i in -t=*|--target=*) TARGET="${i#*=}" shift ;; *) echo "unknown parameter passed: $1" exit 1 ;; esac done if [ -z "$TARGET" ]; then echo "generating ca cert" cfssl genkey -initca ca-csr.json | cfssljson -bare ca rm *.csr fi if [ -z "$TARGET" ] || [ $TARGET == "grpc" ]; then pushd metal-api-grpc echo "generating grpc certs" cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=server server.json | cfssljson -bare server cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=client client.json | cfssljson -bare client rm *.csr popd fi if [ -z "$TARGET" ] || [ $TARGET == "masterdata-api" ]; then pushd masterdata-api echo "generating masterdata-api certs" rm -f *.pem cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=client-server server.json | cfssljson -bare server cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=client client.json | cfssljson -bare client rm *.csr popd fi ``` Also define the following configurations for `cfssl`: - `files/certs/ca-config.json` ```json { "signing": { "default": { "expiry": "43800h" }, "profiles": { "server": { "expiry": "43800h", "usages": ["signing", "key encipherment", "server auth"] }, "client": { "expiry": "43800h", "usages": ["signing", "key encipherment", "client auth"] }, "client-server": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "client auth", "server auth" ] } } } } ``` - `files/certs/ca-csr.json` ```json { "CN": "metal-control-plane", "hosts": [], "key": { "algo": "rsa", "size": 4096 }, "names": [ { "C": "DE", "L": "Munich", "O": "Metal-Stack", "OU": "DevOps", "ST": "Bavaria" } ] } ``` - `files/certs/masterdata-api/client.json` ```json { "CN": "masterdata-client", "hosts": [""], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "DE", "L": "Munich", "O": "Metal-Stack", "OU": "DevOps", "ST": "Bavaria" } ] } ``` - `files/certs/masterdata-api/server.json` ```json { "CN": "masterdata-api", "hosts": [ "localhost", "masterdata-api", "masterdata-api.metal-control-plane.svc", "masterdata-api.metal-control-plane.svc.cluster.local" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "DE", "L": "Munich", "O": "Metal-Stack", "OU": "DevOps", "ST": "Bavaria" } ] } ``` - `files/certs/metal-api-grpc/client.json` ```json { "CN": "grpc-client", "hosts": [""], "key": { "algo": "rsa", "size": 4096 }, "names": [ { "C": "DE", "L": "Munich", "O": "Metal-Stack", "OU": "DevOps", "ST": "Bavaria" } ] } ``` - `files/certs/metal-api-grpc/server.json` (**Fill in your control plane ingress DNS here**) ```json { "CN": "metal-api", "hosts": [""], "key": { "algo": "rsa", "size": 4096 }, "names": [ { "C": "DE", "L": "Munich", "O": "Metal-Stack", "OU": "DevOps", "ST": "Bavaria" } ] } ``` Running the `roll_certs.sh` bash script without any arguments should generate you the required certificates. Now Provide the paths to these certificates in `group_vars/control-plane/metal.yaml`: ```yaml --- metal_masterdata_api_tls_ca: "{{ lookup('file', 'certs/ca.pem') }}" metal_masterdata_api_tls_cert: "{{ lookup('file', 'certs/masterdata-api/server.pem') }}" metal_masterdata_api_tls_cert_key: "{{ lookup('file', 'certs/masterdata-api/server-key.pem') }}" metal_masterdata_api_tls_client_cert: "{{ lookup('file', 'certs/masterdata-api/client.pem') }}" metal_masterdata_api_tls_client_key: "{{ lookup('file', 'certs/masterdata-api/client-key.pem') }}" metal_api_grpc_certs_server_key: "{{ lookup('file', 'certs/metal-api-grpc/server-key.pem') }}" metal_api_grpc_certs_server_cert: "{{ lookup('file', 'certs/metal-api-grpc/server.pem') }}" metal_api_grpc_certs_client_key: "{{ lookup('file', 'certs/metal-api-grpc/client-key.pem') }}" metal_api_grpc_certs_client_cert: "{{ lookup('file', 'certs/metal-api-grpc/client.pem') }}" metal_api_grpc_certs_ca_cert: "{{ lookup('file', 'certs/ca.pem') }}" ``` :::tip For the actual communication between the metal-api and the user clients (REST API, runs over the ingress-controller you deployed before), you can simply deploy a tool like [cert-manager](https://github.com/cert-manager/cert-manager) into your Kubernetes cluster, which will automatically provide your ingress domains with Let's Encrypt certificates. ::: ### Running the Deployment Finally, it should be possible to run the deployment through a Docker container. Make sure to have the [Kubeconfig file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) of your cluster and set the path in the following command accordingly: export function VariableConfig() { return <> export KUBECONFIG={"\<"}path-to-your-cluster-kubeconfig{"\>"}{"\n"} export METAL_VERSION={latestRelease.version} } Then you can spin up the deployment with docker: ```bash # ideally, validate the signature of the deployment image with cosign before running it: cosign verify ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} --certificate-oidc-issuer https://accounts.google.com --certificate-identity keyless@metal-stack.iam.gserviceaccount.com # then run the deployment: docker run --rm -it \ -v $(pwd):/workdir \ --workdir /workdir \ -e KUBECONFIG="${KUBECONFIG}" \ -e K8S_AUTH_KUBECONFIG="${KUBECONFIG}" \ -e ANSIBLE_INVENTORY=inventories/control-plane.yaml \ ghcr.io/metal-stack/metal-deployment-base:${METAL_VERSION} \ /bin/bash -ce \ "ansible -m metalstack.base.metal_stack_release_vector localhost ansible-playbook deploy_metal_control_plane.yaml" ``` :::tip If you are having issues regarding the deployment take a look at the [troubleshoot document](./06-troubleshoot.md). Please give feedback such that we can make the deployment of the metal-stack easier for you and for others! ::: ### Providing Images After the deployment has finished (hopefully without any issues!), you should consider deploying some masterdata entities into your metal-api. For example, you can add your first machine sizes and operating system images. You can do this by further parametrizing the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal). We will just add an operating system for demonstration purposes. Add the following variable to your `inventories/group_vars/control-plane/common.yaml`: ``` metal_api_images: - id: firewall-ubuntu-2.0.20201004 name: Firewall 2 Ubuntu 20201004 description: Firewall 2 Ubuntu 20201004 url: http://images.metal-stack.io/metal-os/master/firewall/2.0-ubuntu/20201004/img.tar.lz4 features: - firewall - id: ubuntu-20.04.20201004 name: Ubuntu 20.04 20201004 description: Ubuntu 20.04 20201004 url: http://images.metal-stack.io/metal-os/master/ubuntu/20.04/20201004/img.tar.lz4 features: - machine ``` Then, re-run the deployment to apply your changes. Our playbooks are idempotent. :::info Image versions should be regularly checked for updates. ::: ### Setting up metalctl You can now verify the existence of the operating system images in the metal-api using our CLI client called [metalctl](https://github.com/metal-stack/metalctl). The configuration for `metalctl` should look like this: ```yaml # ~/.metalctl/config.yaml --- current: test contexts: test: # the metal-api endpoint depends on your dns name specified before # you can look up the url to the metal-api via the kubernetes ingress # resource with: # $ kubectl get ingress -n metal-control-plane url: # in the future you have to change the HMAC to a strong, random string # in order to protect against unauthorized api access # the default hmac is "change-me" hmac: change-me ``` Issue the following command: ```bash $ metalctl image ls ID NAME DESCRIPTION FEATURES EXPIRATION STATUS ubuntu-19.10.20200331 Ubuntu 19.10 20200331 Ubuntu 19.10 20200331 machine 89d 23h preview ``` The basic principles of how the metal control plane can be deployed should now be clear. It is now up to you to move the deployment execution into your CI and add things like certificates for the ingress-controller and NSQ. ### Setting Up the backup-restore-sidecar The backup-restore-sidecar can come in very handy when you want to add another layer of security to the metal-stack databases in your Kubernetes cluster. The sidecar takes backups of the metal databases in small time intervals and stores them in a blobstore of a cloud provider. For each database that will be backed up, a lifecycle rule is established. The backup mechanism is deactivated by default and must be activated by the operator. This way your metal-stack setup can even survive the deletion of your Kubernetes control plane cluster (including all volumes getting lost). After re-deploying metal-stack to another Kubernetes clusters, the databases come up with the latest backup data in a matter of seconds. Encryption can be enabled for the backups by providing an AES-256 encryption key. Checkout the [role documentation](https://github.com/metal-stack/metal-roles/tree/master/control-plane) of the individual databases to find out how to configure the sidecar properly. You can also try out the mechanism from the [backup-restore-sidecar](https://github.com/metal-stack/backup-restore-sidecar) repository. ### Auth metal-stack currently supports two authentication methods: - user authentication through [OpenID Connect](https://openid.net/developers/how-connect-works/) (OIDC) - [HMAC](https://en.wikipedia.org/wiki/HMAC) auth, typically used for access by technical users (because we do not have service account tokens at the time being) If you decided to use OIDC, you can parametrize the [metal role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/metal) for this by defining the variable `metal_masterdata_api_tenants` with the following configuration: ```yaml --- metal_masterdata_api_tenants: - meta: id: kind: Tenant apiversion: v1 version: 0 name: iam_config: issuer_config: client_id: url: idm_config: idm_type: # "AD" | "UX" group_config: namespace_max_length: 20 description: ``` ## Bootstrapping a Partition ### Out-Of-Band-Network To be able to deploy and maintain a metal-stack partition, you need to bootstrap the Out-Of-Band-Network first. Some considerations must be made to fulfill the requirements of our infrastructure, a partition is designed to be: - secure - fully routable (BGP) - scalable - resilient - deployable via CI/CD jobs - accessible from the internet from specific IPs In order to accomplish this task remotely and in a nearly automatic manner, you have to bootstrap the components in this order: 1. management firewalls 2. management servers 3. management spines 4. management leaves 5. leaves, spines and exits This document assumes that all cabling is done. Here is a quick overview of the architecture: ![Out-of-Band-Network](mgmt_net_layer3.png) ### Management Firewalls As you can see, the management firewalls are the first bastion hosts in a partition to provide access to our infrastructure. There are two of them in each partition to guarantee high availability and load balancing. The very first configuration of these routers has to be done manually to solve the chicken and egg problem that you need the management firewalls in place to deploy the partition. Manually means that we generate a configuration template with ansible that we deploy with copy/paste, and the load, through the machine console. Once the management server has been deployed, we are able to deploy this configuration via CI runner and ansible. For this you need the user and the ssh-key, which is deployed with the configuration file mentioned above. The Edgerouters has to fulfill some requirements including: - provide and restrict access to the Out-Of-Band-Network from the internet with a firewall ruleset - provide destination NAT to the management server and its IPMI interface - provide Onie Boot and ztp via DHCP options for the management spine - provide DHCP management addresses for management spine, management server and ipmi interface of the management server - Hairpin-NAT for the management server to access itself via its public IP, needed by the CI runner to delegate CI Jobs. - propagate a default gateway via BGP ### Management Servers The second bastion hosts are the management servers. They are the main bootstrapping components of the Out-Of-Band-Network. They also act as jump hosts for all components in a partition. Once they are installed and deployed, we are able to bootstrap all the other components. To bootstrap the management servers, we generate an [ISO](https://github.com/metal-stack/bootstrap-debian) image which will automatically install an OS and an ansible user with ssh keys. It is preconfigured with a preseed file to allow an unattended OS installation for our needs. This is why we need remote access to the IPMI interface of the management servers: The generated [ISO](https://github.com/metal-stack/bootstrap-debian) is attached via the virtual media function of the BMC. After that, all we have to do is boot from that virtual CD-ROM and wait for the installation to finish. Deployment jobs (Gitlab-CI) in a partition are delegated to the appropriate management servers, therefore we need a CI runner active on each management server. After the CI runner has been installed, you can trigger your Playbooks from the the CI. The Ansible-Playbooks have to make sure that these functionalities are present on the management servers: - Prometheus and exporters - CI runner - metal-bmc - image-cache - simple webserver to provide images - [Onie Boot](https://opencomputeproject.github.io/onie/) and ZTP - DHCP addresses for ipmi interfaces of the workers - DHCP addresses for switches ### Management Spines :::tip If you are using SONiC switches, you should make use of Zero Touch Provisioning and Onie Boot ::: The purpose of these switches is to connect the management interfaces of all switches to the management servers. The management spine's own management interface is connected to the management firewall for the bootstrapping of the management spine itself. The management firewall will provide a DHCP address and DHCP options to start SONiC's [Zero Touch Provisioning](https://github.com/sonic-net/SONiC/blob/master/doc/ztp/ztp.md); the images for all switches are downloaded from the management server (nginx container). Each management leaf is connected to both management spines to provide redundant connectivity to both management servers. BGP is used as a routing protocol such that, when a link goes down, an alternate path is used. In the picture above you can see that there are also switch management interfaces connected to the management spine. This has to be done so that we can bootstrap these switches; the management spine relays the DHCP requests from these switches to the management servers so that they are able to Onie Boot and get their ZTP scripts. ### Management Leaves All workers have to be connected with their IPMI/BMC interface to the management leaves to get DHCP addresses from the management server. The management leaves are relaying those DHCP requests to the management server which will answer the requests and provide IPs from a given range. The management interfaces of the management leaves also have to be reachable from the management server, and need to get their IP address via DHCP for the bootstrapping process. In the example setup, these interfaces are connected to an end-of-row-switch which aggregates them and connects them to the management spines with a fiber-optics connection. If you can reach the management spines from the management leaves with copper cables, you do not need the end of row switch. After the initial bootstrapping, the management interfaces of the management leaves continue to be used for access to the switches' command line, and for subsequent OS updates. (update=reset+bootrap+deployment) ## Partition Deployment ## Gardener with metal-stack If you want to deploy metal-stack as a cloud provider for [Gardener](https://gardener.cloud/), you should follow the regular Gardener installation instructions and setup a Gardener cluster first. It's perfectly fine to setup the Gardener cluster in the same cluster that you use for hosting metal-stack. You can find installation instructions for Gardener on the Gardener website beneath [docs](https://gardener.cloud/docs/). metal-stack is an out-of-tree provider and therefore you will not find example files for metal-stack resources in the Gardener repositories. The following list describes the resources and components that you need to deploy into the Gardener cluster in order to make Gardener work with metal-stack: :::warning The following list assumes you have Gardener installed in a Kubernetes cluster and that you have a basic understanding of how Gardener works. If you need further help with the following steps, you can also come and ask in our Slack channel. ::: 1. Deploy the [validator](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/validator-metal) from the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) repository to your cluster via Helm 1. Add a [cloud profile](https://github.com/gardener/gardener/blob/v1.3.3/example/30-cloudprofile.yaml) called `metal` containing all your machine images, machine types and regions (region names can be chosen freely, the zone names need to match your partition names) together with our metal-stack-specific provider config as defined [here](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/pkg/apis/metal/v1alpha1/types_cloudprofile.go) 1. Register the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) controller by deploying the [controller-registration](https://github.com/metal-stack/gardener-extension-provider-metal/blob/v0.9.1/example/controller-registration.yaml) into your Gardener cluster, parametrize the embedded chart in the controller registration's values section if necessary ([this](https://github.com/metal-stack/gardener-extension-provider-metal/tree/v0.9.1/charts/provider-metal) is the corresponding values file) 1. metal-stack does not provide an own backup storage infrastructure for now. If you want to enable ETCD backups (which you should do because metal-stack also does not have persistent storage out of the box, which makes these backups even more valuable), you should deploy an extension-provider of another cloud provider and configure it to only reconcile the backup buckets (you can reference this backup infrastructure used for the metal shoot in the shoot spec) 1. Register the [os-extension-provider-metal](https://github.com/metal-stack/os-metal-extension) controller by deploying the [controller-registration](https://github.com/metal-stack/os-metal-extension/blob/v0.4.1/example/controller-registration.yaml) into your Gardener cluster, this controller can transform the operating system configuration from Gardener into Ignition user data 1. You need to use the Gardener's [networking-calico](https://github.com/gardener/gardener-extension-networking-calico) controller for setting up shoot CNI, you will have to put specific provider configuration into the shoot spec to make it work with metal-stack: ```yaml networking: type: calico # we can peer with the frr within 10.244.0.0/16, which we do with the metallb # the networks for the shoot need to be disjunct with the networks of the seed, otherwise the VPN connection will not work properly # the seeds are typically deployed with podCIDR 10.244.128.0/18 and serviceCIDR 10.244.192.0/18 # the shoots are typically deployed with podCIDR 10.244.0.0/18 and serviceCIDR 10.244.64.0/18 pods: 10.244.0.0/18 services: 10.244.64.0/18 providerConfig: apiVersion: calico.networking.extensions.gardener.cloud/v1alpha1 kind: NetworkConfig backend: vxlan ipv4: pool: vxlan mode: Always autoDetectionMethod: interface=lo typha: enabled: false ``` 1. For your seed cluster you will need to provide the provider secret for metal-stack containing the key `metalAPIHMac`, which is the API HMAC to grant editor access to the metal-api 1. Checkout our current provider configuration for [infrastructure](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_infrastructure.go) and [control-plane](https://github.com/metal-stack/gardener-extension-provider-metal/blob/master/pkg/apis/metal/v1alpha1/types_controlplane.go) before deploying your shoot :::tip We are officially supported by [Gardener dashboard](https://github.com/gardener/dashboard). The dashboard can also help you setting up some of the resources mentioned above. ::: --- ## Maintenance ## Update Policy For new features and breaking changes we create a new minor release of metal-stack. For every minor release we present excerpts of the changes in a corresponding blog article published on metal-stack.io. It is not strictly necessary to cycle through the patch releases if you depend on the pure metal-stack components. However, it is important to go through all the patch releases and apply all required actions from the release notes. Therefore, we recommend to just install every patch release one by one in order to minimize possible problems during the update process. In case you depend on the Gardener integration, especially when using metal-stack roles for deploying Gardener, we strongly recommend installing every patch release version. We increment our Gardener dependency version by version following the Gardener update policy. Jumping versions may lead to severe problems with the installation and should only be done if you really know what you are doing. :::info If you use the Gardener integration of metal-stack do not skip any patch releases. You may skip patch releases if you depend on metal-stack only, but we recommend to just deploy every patch release one by one for the best possible upgrade experience. ::: ## Releases Before upgrading your metal-stack installation, review the release notes carefully - they contain important information on required pre-upgrade actions and notable changes. These notes are currently shared via a dedicated Slack channel and are also available in the release on GitHub. Once you are prepared, you can deploy a new metal-stack version by updating the `metal_stack_release_version` variable in your Ansible configuration and trigger the corresponding deployment jobs in your CI. metal-stack offers prebuilt system images for firewalls and worker machines, which can be downloaded from `images.metal-stack.io`. In offline or air-gapped setups, these images must either be synced into the partition-local [image-cache](https://github.com/metal-stack/metal-image-cache-sync) after they were added to the metal-api or be manually downloaded in advance and uploaded to your local S3-compatible storage. Ensure that the image paths and metadata are correctly maintained so the system can retrieve them during provisioning. If you are using metal-stack in combination with Gardener and you do not run pre-production stages, we advise running some basic functional tests after upgrading metal-stack to ensure the installation is in a fully functional state (e.g. reconciling a bunch of shoot clusters with evaluation purpose, creating and deleting a shoot cluster). metal-images for firewalls and worker nodes follow independent release cycles, typically driven by the need for security patches or system updates. When new images are made available, the machines must be re-provisioned to apply the updates. When using metal-stack in a Kubernetes context, this results in a rolling update of the cluster worker groups. In a Gardener setup, image updates can be triggered by referencing the new image in the shoot spec. Because all outbound traffic passes through the firewall node, this results in a short downtime of around 30 seconds. This interruption only occurs if the firewall image has actually changed. The process works as follows: a new firewall node is provisioned and configured in parallel with the existing one. Once setup is complete, traffic is switched over to the new node, and the old firewall is then decommissioned. This minimizes disruption while ensuring a seamless transition. The worker nodes are rolled out one after the other and, if possible, the containers are redistributed to the machines that are still available. However, for unclustered stateful workloads like databases, temporary disruptions may occur during node restarts. ## Firmware Updates Firmware updates are intentionally not part of the automated machine lifecycle, since firmware handling is too vendor-specific for stable automation. Updating server firmware is an administrative task that must be performed manually by operators. The corresponding API endpoints are restricted to admin users. ## Repurposing Hardware metal-stack assumes a fully automated server landscape. Using servers managed by metal-stack for other purposes is not supported. To repurpose hardware, the machines have to be decommissioned, removed from the metal database and physically relocated out of the partition before they can be reused outside of metal-stack. When machines are decommissioned, metal-hammer wipes their disks to prevent data leakage. ## Rollback metal-stack employs forward-only database migrations (e.g., for RethinkDB), and each release undergoes thorough integration testing. However, rollback procedures are not included in test coverage. To maintain data integrity and system reliability, rolling back a full release is not supported and strongly discouraged. In the event of issues after an upgrade, it is possible to downgrade specific components rather than reverting the entire system. ## Backup and Disaster Recovery The databases of the metal control plane are continuously backed up by the [backup-restore-sidecar](https://github.com/metal-stack/backup-restore-sidecar), which pushes the backups to S3-compatible object storage. When a database starts up with an empty state, the sidecar automatically restores the most recent backup. Restoring a specific backup point manually is also supported, see the [manual restore documentation](../08-References/Control%20Plane/backup-restore-sidecar/manual_restore.md). The following table summarizes the recovery approach for common disaster scenarios: | Scenario | Recovery Approach | |:----------------------------|:----------------------------------------------------------------------------------------------------| | Control plane database loss | Restore the databases from the backups on the object storage, redeploy the control plane via CI/CD | | Partition loss | Rebuild the partition using the partition Ansible playbooks | | Switch failure | Mount a replacement switch and re-run the partition Ansible playbooks (the topology is redundant, so a single switch failure does not cause an outage) | During recovery, the metal-stack version referenced in the deployment Git repository is deployed — not the latest release. This ensures that the environment returns to its pre-outage state. Operators trigger this redeployment manually through the CI pipeline. Backup of local machine storage is not covered by metal-stack. Machine/Disk failure will lead to data loss. To mitigate this a clustered storage system can be used to replicate data across multiple machines in multiple failure domains (e.g. machines, racks, partitions). In addition, we recommend automated and monitored backups and regular restore drills. --- ## Monitoring # Monitoring the metal-stack ## Overview ![Monitoring Stack](monitoring-stack.svg) The diagram above shows the full monitoring and logging stack. metal-stack supports the deployment of a central monitoring control plane, with Grafana providing unified dashboards and alerting across all tiers. In a **partition**, hosts ship logs to Loki via Alloy. A partition-local Prometheus scrapes exporters in the switch plane and remote-writes the collected metrics to the centralized Thanos ingress, enabling long-term metric persistence and compaction. The **control-plane** and **Gardener** seed Alloy instances push both logs and self-metrics directly to the centralized monitoring control plane. ## Logging [Grafana Alloy](https://grafana.com/docs/alloy/latest/) collects and pushes logs to a [Loki](https://grafana.com/docs/loki/latest/) instance running in the control plane. Loki is deployed in [monolithic mode](https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/) and with storage type `'filesystem'`. You can find all logging related configuration parameters for the control plane in the [logging](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging/README.md) role. In the partitions, Alloy can be deployed inside a systemd-managed Docker container on management servers and switches. Configuration parameters can be found in the partition's [alloy](https://github.com/metal-stack/metal-roles/blob/master/partition/roles/alloy/README.md) role. ### Control-Plane Log Sources In the control plane, Alloy runs as a Kubernetes `DaemonSet` and collects logs from two sources: | Source | Description | Key labels | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Pod logs | Read from the node filesystem (`/var/log/pods`, `loki.source.file`). Each DaemonSet pod collects only pods scheduled on its own node. | `cluster`, `namespace`, `pod`, `container`, `pod_uid`, `node_name`, `app`, `instance`, `component`, `job` | | Kubernetes events | Collected natively via `loki.source.kubernetes_events` with clustering-based leader election — no separate event-exporter required | `cluster`, `job=events`, `namespace` | All control-plane log entries carry a `cluster` label (configured via `logging_alloy_cluster_label`) identifying the control-plane stage. #### Gardener Gardener ships with a built-in logging stack (Vali + fluent-bit per seed), that can be used as-is or replaced/complemented by the metal-stack's Alloy + Loki solution. The metal-stack roles provide an own centralized logging stack based on Alloy and Loki — giving platform operators a single place to query infrastructure logs across all Gardener clusters. The [gardener-logging](https://github.com/metal-stack/blob/master/control-plane/roles/gardener-logging/README.md) role deploys an Alloy instance into each Gardener shooted seed and optionally into the garden cluster itself. These instances read pod logs from the node filesystem and collect Kubernetes events, forwarding everything to the same Loki instance in the metal-stack control plane. Logs carry a `cluster` label set to the cluster name (garden name or shooted seed name), enabling per-cluster filtering in Grafana. ### Control-Plane: Querying Logs in Grafana - `{cluster=""}` — all logs from a control-plane stage - `{namespace=""}` — all logs from a specific namespace - `{job="/"}` — logs from a specific application - `{job="events"}` — Kubernetes events _(recently renamed from `monitoring/event-exporter`)_ - `{cluster=""}` — all logs from the Gardener garden cluster or a specific shooted seed ### Partition Log Sources Alloy is configured through snippets that define what logs are collected. The following built-in snippets are available: | Host type | Snippet | Description | Key labels | | ---------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | | Leaves, spines, exits | `journal` | Collects logs from the systemd journal; auto-discovers both volatile (`/run/log/journal`) and persistent (`/var/log/journal`) storage | `job=systemd-journal`, `unit`, `level` | | Management servers | `journal-file` | Collects logs from the persistent systemd journal at a configurable path; supports migrating cursor position from promtail | `job=systemd-journal`, `unit`, `level` | | Hosts without journald | `syslog` | Tails `/var/log/syslog` | `job=syslog` | | Hosts running Docker | `docker` | Collects logs from all Docker containers via the Docker socket | `job=docker`, `container` | Custom log sources can be added without modifying the role by providing your own Jinja2 snippet templates and referencing them via `alloy_config_custom_snippets` in your inventory. See the [alloy role](https://github.com/metal-stack/metal-roles/blob/master/partition/roles/alloy/README.md#customizing-the-config) for details. ### Partition: Querying Logs in Grafana All log entries carry the `host` and `partition` labels regardless of snippet, which makes it easy to filter logs in Grafana Explore by host or partition. - `{partition=""}` — all logs from a partition - `{host=""}` — all logs from a specific host - `{job="docker", container=""}` — logs from a specific Docker container - `{job="systemd-journal", unit=".service"}` — logs from a specific systemd unit - `{job="systemd-journal", level="error"}` — error-level journal entries across all units :::note Migrating from Promtail The `promtail` role is deprecated and replaced by the `alloy` role. Refer to the respective migration guides for step-by-step instructions: - [Partition](https://github.com/metal-stack/metal-roles/blob/master/partition/roles/alloy/README.md#migration-from-promtail) — partition alloy role - [Control-plane and Gardener](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#migration-from-promtail) — logging-common migration guide (applies to both the logging and gardener-logging roles) ::: ## Monitoring For monitoring we deploy the [kube-prometheus-stack](https://github.com/prometheus-operator/kube-prometheus) and a [Thanos](https://thanos.io/tip/thanos/getting-started.md/) instance in the control plane. ### Control-Plane Metrics In-cluster components are scraped by Prometheus via `ServiceMonitor` resources (pull model). Additional metrics are supplied by - `metal-metrics-exporter` - `rethinkdb-exporter` - `gardener-metrics-exporter` - `alloy` (control-plane) — self-metrics, disabled by default; see [logging-common](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#meta-monitoring) for configuration - `alloy` (gardens and seeds) — self-metrics, disabled by default, push-only (no ServiceMonitor); see [logging-common](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#meta-monitoring) for configuration The following `ServiceMonitors` are deployed: - `gardener-metrics-exporter` - `ipam-db` - `masterdata-api` - `masterdata-db` - `metal-api` - `metal-db` - `rethinkdb-exporter` - `metal-metrics-exporter` All monitoring related configuration parameters for the control plane can be found in the control plane's [monitoring](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/monitoring/README.md) role. ### Partition Metrics Partition metrics are collected via Prometheus scraping (pull model). Exporters running on partition hosts supply the metrics: - `node-exporter` - `blackbox-exporter` - `ipmi-exporter` - `sonic-exporter` - `metal-core` - `alloy` Target hosts for each exporter are defined by - `prometheus_node_exporter_targets` - `prometheus_blackbox_exporter_targets` - `prometheus_ipmi_exporter_targets` - `prometheus_sonic_exporter_targets` - `prometheus_metal_core_targets` - `prometheus_frr_exporter_targets` - `prometheus_alloy_targets` ### Dashboards To query and visualize logs, metrics and alerts we deploy several grafana dashboards to the control plane: - `grafana-dashboard-alertmanager` - `grafana-dashboard-machine-capacity` - `grafana-dashboard-metal-api` - `grafana-dashboard-rethinkdb` - `grafana-dashboard-sonic-exporter` and also some Gardener related dashboards: - `grafana-dashboard-gardener-overview` - `grafana-dashboard-shoot-cluster` - `grafana-dashboard-shoot-customizations` - `grafana-dashboard-shoot-details` - `grafana-dashboard-shoot-states` ## Alerting In addition to Grafana, alerts can optionally be sent to a [Slack](https://slack.com/) channel. For this to work, at least a valid `monitoring_slack_api_url` and a `monitoring_slack_notification_channel` must be specified. For further configuration parameters refer to the [monitoring](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/monitoring) role. Alerting rules are defined in the [rules](https://github.com/metal-stack/metal-roles/tree/master/partition/roles/monitoring/prometheus/files/rules) directory of the partition's prometheus role. --- ## Troubleshooting This document summarizes help when something goes wrong and provides advice on debugging the metal-stack in certain situations. Of course, it is also advisable to check out the issues on the Github projects for help. If you still can't find a solution to your problem, please reach out to us and our community. We have a public Slack Channel to discuss problems, but you can also reach us via mail. Check out [metal-stack.io](https://metal-stack.io) for contact information. ## Deployment ### Ansible fails when the metal control plane helm chart gets applied There can be many reasons for this. Since you are deploying the metal control plane into a Kubernetes cluster, the first step should be to install [kubectl](https://kubernetes.io/docs/tasks/tools/) and check the pods in your cluster. Depending on the metal-stack version and Kubernetes cluster, your control-plane should look something like this after the deployment (this is in a Kind cluster): ```bash kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx nginx-ingress-controller-56966f7dc7-khfp9 1/1 Running 0 2m34s kube-system coredns-66bff467f8-grn7q 1/1 Running 0 2m34s kube-system coredns-66bff467f8-n7n77 1/1 Running 0 2m34s kube-system etcd-kind-control-plane 1/1 Running 0 2m42s kube-system kindnet-4dv7m 1/1 Running 0 2m34s kube-system kube-apiserver-kind-control-plane 1/1 Running 0 2m42s kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 2m42s kube-system kube-proxy-jz7kp 1/1 Running 0 2m34s kube-system kube-scheduler-kind-control-plane 1/1 Running 0 2m42s local-path-storage local-path-provisioner-bd4bb6b75-cwfb7 1/1 Running 0 2m34s metal-control-plane ipam-db-0 2/2 Running 0 2m31s metal-control-plane masterdata-api-6dd4b54db5-rwk45 1/1 Running 0 33s metal-control-plane masterdata-db-0 2/2 Running 0 2m29s metal-control-plane metal-api-998cb46c4-jj2tt 1/1 Running 0 33s metal-control-plane metal-api-initdb-r9sc6 0/1 Completed 0 2m24s metal-control-plane metal-api-liveliness-1590479940-brhc7 0/1 Completed 0 6s metal-control-plane metal-console-7955cbb7d7-p6hxp 1/1 Running 0 33s metal-control-plane metal-db-0 2/2 Running 0 2m34s metal-control-plane nsq-lookupd-5b4ccbfb64-n6prg 1/1 Running 0 2m34s metal-control-plane nsqd-6cd87f69c4-vtn9k 2/2 Running 0 2m33s ``` If there are any failing pods, investigate those and look into container logs. This information should point you to the place where the deployment goes wrong. :::info Sometimes, you see a helm errors like "no deployed releases" or something like this. When a helm chart fails after the first deployment it could be that you have a chart installation still pending. Also, the control plane helm chart uses pre- and post-hooks, which creates [jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) that helm expects to be completed before attempting another deployment. Delete the helm chart (use Helm 3) with `helm delete -n metal-control-plane metal-control-plane` and delete the jobs in the `metal-control-plane` namespace before retrying the deployment. ::: ### In the mini-lab the control-plane deployment fails because my system can't resolve api.172.17.0.1.nip.io The control-plane deployment returns an error like this: ```bash deploy-control-plane | fatal: [localhost]: FAILED! => changed=false deploy-control-plane | attempts: 60 deploy-control-plane | content: '' deploy-control-plane | elapsed: 0 deploy-control-plane | msg: 'Status code was -1 and not [200]: Request failed: ' deploy-control-plane | redirected: false deploy-control-plane | status: -1 deploy-control-plane | url: http://api.172.17.0.1.nip.io:8080/metal/v1/health deploy-control-plane | deploy-control-plane | PLAY RECAP ********************************************************************* deploy-control-plane | localhost : ok=29 changed=4 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 deploy-control-plane | deploy-control-plane exited with code 2 ``` Some home routers have a security feature that prevents DNS Servers to resolve anything in the router's local IP range (DNS-Rebind-Protection). You need to add an exception for `nip.io` in your router configuration or add `127.0.0.1 api.172.17.0.1.nip.io` to your `/etc/hosts`. #### FritzBox `Home Network -> Network -> Network Settings -> Additional Settings -> DNS Rebind Protection -> Host name exceptions -> nip.io` ## Operations ### Fixing Machine Issues The `metalctl machine issues` command gives you an overview over machines in your metal-stack environment that are in an unusual state. :::tip Machines that are known not to function properly, should be locked through `metalctl machine lock` and annotated with a description of the problem. This way, you can mark machine for replacement without being in danger of having a user allocating the faulty machine. ::: In the following sections, you can look up the machine issues that are returned by `metalctl` and find out how to deal with them properly. #### no-event-container Every machine in the metal-stack database usually has a corresponding event container where provisioning events are stored. This database entity gets created lazily as soon as a machine is registered by the metal-hammer or a provisioning event for the machine arrives at the metal-api. When there is no event container, this means that the machine has never registered nor received a provisioning event. As an operator you should evaluate why this machine is not booting into the metal-hammer. This issue is special in a way that it prevents other issues from being evaluated for this machine because the issue calculation usually requires information from the machine event container. #### no-partition When a machine has no partition, the [metal-hammer](https://github.com/metal-stack/metal-hammer) has not yet registered the machine at the [metal-api](https://github.com/metal-stack/metal-api). Instead, the machine was created through metal-stack's event machinery, which does not have a lot of information about a machine (e.g. a PXE boot event was reported from the pixiecore), or just by the [metal-bmc](https://github.com/metal-stack/metal-bmc) which discovered the machine through DHCP. This can usually happen on the very first boot of a machine and the machine's [hardware is not supported](./01-hardware.md) by metal-stack, leading to the [metal-bmc](https://github.com/metal-stack/metal-bmc) being unable to report BMC details to the metal-api (a metal-bmc report sets the partition id of a machine) and the metal-hammer not finishing the machine registration phase. To resolve this issue, you need to identify the machine in your metal-stack partition that emits PXE boot events and find the reason why it is not properly booting into the metal-hammer. The console logs of this machine should enable you to find out the root cause. #### liveliness-dead For machines without an allocation, the metal-hammer consistently reports whether a machine is still being responsive or not. When the liveliness is `Dead`, there were no events received from this machine for longer than ~5 minutes. Reasons for this can be: - The network connection between the partition and metal-stack control plane is interrupted - The machine was removed from your data center - The machine has changed its UUID [metal-hammer#52](https://github.com/metal-stack/metal-hammer/issues/52) - The machine is turned off - The machine hangs / freezes - The machine booted to BIOS or UEFI shell and does not try to PXE boot again - The issue only appears temporarily - The machine takes longer than 5 minutes for the reboot - The machine is performing a firmware upgrade, which usually takes longer than 5 minutes to succeed :::info In order to minimize maintenance overhead, a machine which is dead for longer than an hour will be rebooted through the metal-api. In case you want to prevent this action from happening for a machine, you can lock the machine through `metalctl machine lock`. ::: If the machine is dead for a long time and you are sure that it will never come back, you can clean up the machine through `metalctl machine rm --remove-from-database`. #### liveliness-unknown For machines that are allocated by a user, the ownership has gone over to this user and as an operator you cannot access the machine anymore. This makes it harder to detect whether a machine is in a healthy state or not. Typically, all official metal-stack OS images deploy an LLDP daemon, that consistently emits alive messages. These messages are caught by the [metal-core](https://github.com/metal-stack/metal-core) and turned into a `Phoned Home` event. Internally, the metal-api uses these events as an indicator to decide whether the machine is still responsive or not. When the LLDP daemon stopped sending packages, the reasons are identical to those of [dead machines](#liveliness-dead). However, it's not possible anymore to decide whether the user is responsible for reaching this state or not. In most of the cases, there is not much that can be done from the operator's perspective. You will need to wait for the user to report an issue with the machine. When you do support, you can use this issue type to quickly identify this machine. #### liveliness-not-available This is more of a theoretical issue. When the machine liveliness is not available check that the Kubernetes `CronJob` in the metal-stack control plane for evaluating the machine liveliness is running regularly and not containing error logs. Make the machine boot into the metal-hammer and this issue should not appear. #### failed-machine-reclaim If a machine remains in the `Phoned Home` state without having an allocation, this indicates that the [metal-bmc](https://github.com/metal-stack/metal-bmc) was not able to put the machine back into PXE boot mode after `metalctl machine rm`. The machine is still running the operating system and it does not return back into the allocatable machine pool. Effectively, you lost a machine in your environment and no-one pays for it. Therefore, you should resolve this issue as soon as possible. In bad scenarios, when the machine was a firewall, the machine can still reach the internet through the PXE boot network and also attract traffic, which it cannot route anymore inside the tenant VRF. This can cause traffic loss inside a tenant network. In most of the cases, it should be sufficient to run another `metalctl machine rm` on this machine in order to retry booting into PXE mode. If this still does not succeed, you can boot the machine into the BIOS and manually and change the boot order to PXE boot. This should force booting the metal-hammer again and add the machine back into your pool of allocatable machines. For further reference, see [metal-api#145](https://github.com/metal-stack/metal-api/issues/145). #### crashloop Under bad circumstances, a machine diverges from its typical machine lifecycle. When this happens, the internal state-machine of the metal-api detects that the machine reboots unexpectedly during the provisioning phase. It is likely that the machine has entered a crash loop where it PXE boots again and again without the machine ever becoming usable. Reasons for this can be: - The machine's [hardware is not supported](./01-hardware.md) and the metal-hammer crashes during the machine discovery - The machine registration fails through the metal-hammer because an orphaned / dead machine is still present in the metal-api's data base. The machine is connected to the same switch ports that were used by the orphaned machine. In this case, you should clean up the orphaned machine through `metalctl machine rm --remove-from-database`. Please also consider console logs of the machine for investigating the issue. The incomplete cycle count is reset as soon as the machine reaches `Phoned Home` state or there is a `Planned Reboot` of the machine (planned reboot is also done by the metal-hammer once a day in order to reboot with the latest version). #### last-event-error The machine had an error during the provisioning lifecycle recently or events are arriving out of order at the metal-api. This can be an interesting hint for the operator that something during machine provisioning went wrong. You can look at the error through `metalctl machine describe` or `metalctl machine logs`. This error will disappear after a certain time period from `machine issues`. You can still look up the error as described above. #### asn-not-unique This issue was introduced by a bug in earlier versions of metal-stack and was fixed in [PR105](https://github.com/metal-stack/metal-api/pull/105.) To resolve the issue, you need to recreate the firewalls that use the same ASN. #### bmc-without-mac The [metal-bmc](https://github.com/metal-stack/metal-bmc) is responsible to report connection data for the machine's [BMC](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface#Baseboard_management_controller). If it's uncapable of discovering this information, your [hardware might not be supported](./01-hardware.md). Please investigate the logs of the metal-bmc to find out what's going wrong with this machine. #### bmc-without-ip The [metal-bmc](https://github.com/metal-stack/metal-bmc) is responsible to report connection data for the machine's [BMC](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface#Baseboard_management_controller). If it's uncapable of discovering this information, your [hardware might not be supported](./01-hardware.md). Please investigate the logs of the metal-bmc to find out what's going wrong with this machine. #### bmc-no-distinct-ip The [metal-bmc](https://github.com/metal-stack/metal-bmc) is responsible to report connection data for the machine's [BMC](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface#Baseboard_management_controller). When there is no distinct IP address for the BMC, it can be that an orphaned machine used this IP in the past. In this case, you need to clean up the orphaned machine through `metalctl machine rm --remove-from-database`. #### bmc-info-outdated The [metal-bmc](https://github.com/metal-stack/metal-bmc) is responsible to report bmc details for the machine's [BMC](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface#Baseboard_management_controller). When the metal-bmc was not able to fetch the bmc info for longer than 20 minutes, something is wrong with the BMC configuration of the machine. This can be caused by one of the following reasons: - Wrong password for the root user is configured in the BMC - ip address of the BMC is either wrong or not present - the device on the given ip address is not a machine, maybe a switch or a management component which is not managed by the metal-api In either case, please check the logs for the given machine UUID on the metal-bmc for further details. Also check that the metal-bmc is configured to only consider BMC IPs in the range they are configured from the DHCP server in the partition. This prevents grabbing unrelated BMCs. ### A machine has registered with a different UUID after reboot metal-stack heavily relies on steady machine UUIDs as the UUID is the primary key of the machine entity in the metal-api. For further reference also see [metal-stack/metal-hammer#52](https://github.com/metal-stack/metal-hammer/issues/52). #### Reasons There are some scenarios (can be vendor-specific), which can cause a machine UUID to change over time, e.g.: - When the UUID partly contains of a network card's mac address, it can happen when: - Exchanging network cards - Disabling network cards through BIOS - Changing the UUID through vendor-specific CLI tool #### Solution 1. After five minutes, the orphaned machine UUID will be marked dead (💀) because machine events will be sent only to the most recent UUID 1. Identify the dead machine through `metalctl machine ls` 1. Remove the dead machine forcefully with `metalctl machine rm --remove-from-database --yes-i-really-mean-it ` ### Fixing Switch Issues #### switch-sync-failing For your network infrastructure it is key to adapt to new configuration. In case this sync process fails for more than 10 minutes, it is likely to require manual investigation. Depending on your switch operating system, the error sources might differ a lot. Try to connect to your switch using the console or ssh and investigate the logs. Check if the hard drive is full. ### Switch Replacement and Migration There are two mechanisms to replace an existing switch with a new one, both of which will transfer existing VRF configuration and machine connections from one switch to another. Due to the redundance of the CLOS topology, a switch replacement can be performed without downtime. #### Replacing a Switch If the new switch should have the same ID as the old one you should perform a switch replacement. To find detailed information about the procedure of a switch replacement use `metalctl switch replace --help`. Basically, what you need to do is mark the switch for replacement via `metalctl switch replace`, then physically replace the switch with the new one and configure it. The last step is to deploy metal-core on the switch. Once metal-core registers the new switch at the metal-api, the old switches configuration and machine connections will be transferred to the new one. Note that the replacement only works if the new switch has the same ID as the old one. Otherwise metal-core will simply register a new switch and leave the old one untouched. #### Migrating from one Switch to another If the new switch should not or cannot have the same ID as the old one, then the `switch migrate` command can be used to achieve the same result as a switch replacement. Perform the following steps: 1. Leave the old switch in place. 1. Install the new switch in the rack without connecting it to any machines yet. 1. Adjust the metal-stack deployment in the same way as for a switch replacement. 1. Deploy metal-core on the new switch and wait for it to register at the metal-api. Once the switch is registered it will be listed when you run `metalctl switch ls`. 1. Run `metalctl switch migrate `. 1. Disconnect all machines from the old switch and connect them to the new one. In between steps 5 and 6 there is a mismatch between the switch-machine-connections known to the metal-api and the real connections. Since the metal-api learns about the connections from what a machine reports during registration, a machine registration that occurs in between steps 5 and 6 will result in a condition that looks somewhat broken. The metal-api will think that a machine is connected to three switches. This, however, should not cause any problems. Just move on to step 6 and delete the old switch from the metal-api afterwards. If the case just described really occurs, then `metalctl switch delete ` will throw an error, because deleting a switch with existing machine connections might be dangerous. If, apart from that, the migration was successful, then the old switch can be safely deleted with `metalctl switch delete --force`. #### Preconditions for Migration and Replacement An invariant that must be satisfied throughout is that the switch ports a machine is connected to must match, i.e. a machine connected to `Ethernet0` on switch 1 must be connected to `Ethernet0` on switch 2 etc. Furthermore, the breakout configurations of both switches must match and the new switch must contain at least all of the old switch's interfaces. #### Migrating from Cumulus to Edgecore SONiC Both migration and replacement can be used to move from Cumulus to Edgecore SONiC (or vice versa). Migrating to or from Broadcom SONiC or mixing Broadcom SONiC with Cumulus or Edgecore SONiC is not supported. ### Connect a Machine to Another Switch Pair As soon as a machine was connected to the management network and a pair of leaf switches, and the metal-hammer successfully registered the machine at the metal-api after PXE boot, the `switch` entity in metal-stack contains the machine ID in a data structure called _machine connections_. In case you would like to wire this machine to another pair of switches inside this partition, the metal-api would prevent the machine registration because it finds that the machine is already connected to other switches in this partition. To resolve this state, the approach for recabling a machine works as follows: 1. Free the machine if it still has an allocation. 1. Reconnect the machine to the new switch pair. 1. Leave the machine turned off or turn it off and wait until the machine reaches the dead state (💀) in the metal-api. 1. Delete the machine through `metalctl machine delete --remove-from-database --yes-i-really-mean-it`. This cleans up the existing machine connections, too. 1. The machine will soon show up again because the [metal-bmc](https://github.com/metal-stack/metal-bmc) discovers it through the DHCP address obtained by the machine BMC. 1. Power on the machine again and let the metal-hammer register the machine. --- ## Architecture The metal-stack is a compound of microservices predominantly written in [Golang](https://go.dev/). This page gives you an overview over which microservices exist, how they communicate with each other and where they are deployed. ## Target Deployment Platforms For our environments, we chose to deploy the metal-stack into a Kubernetes cluster. This means that also our entire installation was developed for metal-stack being run on Kubernetes. Running applications on Kubernetes gives you a lot of benefits regarding ease-of-deployment, scalability, reliability and so on. However, very early we decided that we do not want to depend on technical Kubernetes functionality with our software (i.e. we did not implement the stack "kube-native" by using controllers and Kubernetes CRDs and things like that). With the following paragraph we want to point out the reasoning behind this "philosophical" decision that may sound conservative at first glance. But not relying on Kubernetes technology: - Makes deployments of the stack without Kubernetes theoretically possible. - We believe that cloud providers should be able to act beneath Kubernetes - This way it is possible to use metal-stack for providing your own Kubernetes offering without relying on Kubernetes yourself (breaks the chicken-egg problem) - Follows an important claim in microservice development: "Be agnostic to your choice of technology" - For applications that are purely made for being run on Kubernetes, it does not matter to rely on this technology (we even do the same a lot with our applications that integrate the metal-stack with Gardener) but as soon as you start using things like the underlying reconciliation abilities (which admittedly are fanstatic) you are locking your code into a certain technology - We don't know what comes after Kubernetes but we believe that a cloud offering should have the potential to survive a choice of technology - By this decision we ensured that we can migrate the stack to another future technology and survive the change One more word towards determining the location for your metal control plane: It is not strictly required to run the control plane inside the same data center as your servers. It even makes sense not to do so because this way you can place your control plane and your servers into a different failure domains, which makes your installation more robust to data center meltdown. Externally hosting the control plane brings you up and running quickly plus having the advantage of higher security through geo-distribution. ## Metal Control Plane The foundation of the metal-stack is what we call the _metal control plane_. The control plane contains a couple of essential microservices for the metal-stack including: - **[metal-api](https://github.com/metal-stack/metal-api)** The API to manage control plane resources like machines, switches, operating system images, machine sizes, networks, IP addresses and more. The exposed API is an old-fashioned REST API with different authentication methods. The metal-api stores the state of these entities in a [RethinkDB](https://rethinkdb.com/) database. The metal-api also has its own IP address management ([go-ipam](https://github.com/metal-stack/go-ipam)), which writes IP address and network allocations into a PostgreSQL backend. - **[masterdata-api](https://github.com/metal-stack/masterdata-api)** Manages tenant and project entities, which can be described as entities used for company-specific resource separation and grouping. Having these "higher level entities" managed by a separate microservice was a design choice that allows to re-use the information by other microservices without having them to know the metal-api at all. The masterdata gets persisted in a dedicated PostgreSQL database. - **[metal-console](https://github.com/metal-stack/metal-console)** Provides access for users to a machine's serial console via SSH. It can be seen as an optional component. - **[nsq](https://nsq.io/)** A message queuing system (not developed by the metal-stack) used for decoupling microservices and distributing tasks. The following figure shows the relationships between these microservices: ![Metal Control Plane](./assets/metal-stack-control-plane.svg) > Figure 1: The metal control plane deployed in a Kubernetes environment with an ingress-controller exposing additional services via [service exposal](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/). Some notes on this picture: - Users can access the metal-api with the CLI client called [metalctl](https://github.com/metal-stack/metalctl). - You can programmatically access the metal-api with [client libraries](../03-For%20Users/01-client_libraries.md) (e.g. [metal-go](https://github.com/metal-stack/metal-go)). - Our databases are wrapped in a specially built [backup-restore-sidecar](https://github.com/metal-stack/backup-restore-sidecar), which is consistently backing up the databases in external blob storage. - The metal-api can be scaled out using replicas when being deployed in Kubernetes. ## Partitions A _partition_ is our term for describing hardware in the data center controlled by the metal-stack with all the hardware participating in the same network topology. Being in the same network topology causes the hardware inside a partition to build a failure domain. Even though the network topology for running the metal-stack is required to be redundant by design, you should consider setting up multiple partitions. With multiple partitions it is possible for users to maintain availability of their applications by spreading them across the partitions. Installing partitions in multiple data centers would be even better in regards of fail-safe application performance, which would even tolerate the meltdown of a data center. :::tip In our setups, we encode the name of a region and a zone name into our partition names. However, we do not have dedicated entities for regions and zones in our APIs. A **region** is a geographic area in which data centers are located. **Zones** are geographic locations in a region usually in different fire compartments. Regions can consist of several zones. A zone can consist of several **partitions**. Usually, a partition spans a rack or a group of racks. ::: We strongly advise to group your hardware into racks that are specifically assembled for running metal-stack. When using modular rack design, the amount of compute resources of a partition can easily be extended by adding more racks to your partition. :::info The hardware that we currently support to be placed inside a partition is described in the [hardware](../04-For%20Operators/01-hardware.md) document. ::: :::info How large you can grow your partitions and how the network topology inside a partition looks like is described in the [networking](../05-Concepts/03-Network/01-theory.md) document. ::: The metal-stack has microservices running on the leaf switches in a partition. For this reason, your leaf switches are required to run a Linux distribution that you have full access to. Additionally, there are a servers not added to the pool of user-allocatable machines, which are instead required for running metal-stack and we call them _management servers_. We also call the entirety of switches inside a partition the _switch plane_. The microservices running inside a partition are: - **[metal-hammer](https://github.com/metal-stack/metal-hammer)** (runs on a server when not allocated by user, often referred to as _discovery image_) An initrd, which is booted up in PXE mode, preparing and registering a machine. When a user allocates a machine, the metal-hammer will install the target operating system on this machine and kexec into the new operating system kernel. - **[metal-core](https://github.com/metal-stack/metal-core)** (runs on leaf switches) Dynamically configures the leaf switch from information provided by the metal-api. It also proxies requests from the metal-hammer to the metal-api including publishment of machine lifecycle events and machine registration requests. - **[pixiecore](https://github.com/danderson/netboot/tree/master/pixiecore)** (preferably runs on management servers, forked by metal-stack) Provides the capability of PXE booting servers in the PXE boot network. - **[metal-bmc](https://github.com/metal-stack/metal-bmc)** (runs on management servers) Reports the ip addresses that are leased to ipmi devices together with their machine uuids to the metal-api. This provides machine discovery in the partition machines and keeps all IPMI interface access data up-to-date. Also forwards metal-console requests to the actual machine, allowing user access to the machine's serial console. Furthermore it processes firmware updates and power on/off, led on/off, boot order changes. ![Partition](./assets/metal-stack-partition.svg) > Figure 2: Simplified illustration of services running inside a partition. Some notes on this picture: - This figure is slightly simplified. The switch plane consists of spine switches, exit routers, management firewalls and a bastion router with more software components deployed on these entities. Please refer to the [networking](../05-Concepts/03-Network/01-theory.md) document to see the full overview over the switch plane. - The image-cache is an optional component consisting of multiple services to allow caching images from the public image store inside a partition. This brings increased download performance on machine allocation and increases independence of a partition on the internet connection. ## Complete View The following figure shows several partitions connected to a single metal control plane. Of course, it is also possible to have multiple metal control planes, which can be useful for staging. ![metal-stack](./assets/metal-stack-architecture.svg) > Figure 3: Reduced view on the communication between the metal control plane and multiple partitions. Some notes on this picture: - By design, a partition only has very few ports open for incoming-connections from the internet. This contributes to a smaller attack surface and higher security of your infrastructure. - With the help of NSQ, it is not required to have connections from the metal control plane to the metal-core. The metal-core instances register at the message bus and can then consume partition-specific topics, e.g. when a machine deletion gets issued by a user. ## Machine Provisioning Sequence The following sequence diagram illustrates some of the main principles of the machine provisioning lifecycle. ![provisioning sequence](./assets/provisioning_sequence.svg) > Figure 4: Sequence diagram of the machine provisioning sequence. Here is a video showing a screen capture of a machine's serial console while running the metal-hammer in "wait mode". Then, a user allocates the machine and the metal-hammer installs the target operating system and the machine boots into the new operating system kernel via the kexec system call. export function MachineAllocationYT(){ return } ## Offline Resilience It is possible to use metal-stack without any external network dependencies by integrating your own DNS and NTP configuration into the stack. This feature is great for workloads requiring strong independence and reliability. Even in case of an internet connection failure, your infrastructure remains operational. Existing machines do not encounter any downtime as well as new machines can be provisioned. All you need to have in place is a DNS and NTP server configured and accessible for metal-stack. NTP servers need to be configured on the pixiecore and the metal-hammer microservices. This can be achieved by providing a list of NTP servers with the following Ansible variable through metal-roles: ```yaml pixiecore_metal_hammer_ntp_servers: [] ``` In the background, the pixiecore is taking the NTP servers and passing it via the `MetalConfig` to the metal-hammer. When booting bare-metal servers, the metal-hammer needs to configure NTP servers. It recognises the ones from the `MetalConfig` and configures itself accordingly. If no NTP servers are passed along, the following standard servers are used: - 0.de.pool.ntp.org - 1.de.pool.ntp.org - 2.de.pool.ntp.org Moreover, machine and firewall images need to be configured with your custom DNS and NTP servers. The customisation can be made via the fields `ntp_servers` an `dns_servers` and specifying a list of servers in the creation request for the machine or firewall. Within a partition default values for DNS and NTP servers can be configured. They are applied to all machines and firewalls within this partition, but can be replaced by specifying different ones inside the machine allocation request. Thus, for creating a partition as well as a machine or a firewall, the flags `dnsservers` and `ntpservers` can be provided within the `metalctl` command. In order to be fully offline resilient, make sure to check out `metal-image-cache-sync`. This component provides copies of `metal-images`, `metal-kernel` and `metal-hammer`. This feature is related to [MEP14](/community/MEP-14-independence-from-external-sources). --- ## User Management At the moment, metal-stack can more or less be seen as a low-level API that does not scope access based on projects and tenants. Fine-grained access control with full multi-tenancy support is actively worked on in [MEP4](/community/MEP-14-independence-from-external-sources). Until then projects and tenants can be created, but have no effect on access control. ## Default Users The current system provides three default users with their corresponding roles: - **Metal-Admin** is an **Admin** can perform all actions. - **Metal-Edit** has the **Edit** role and may create, edit and delete most resources. - **Metal-Viewer** is a **Viewer** and may only view resources and may access machines. Each of these users have a corresponding [HMAC](https://en.wikipedia.org/wiki/HMAC) token, which can be used to authenticate against the API. The tokens do not expire and can be used as long as the service is running. Be cautious with who you share the tokens with. ## OIDC Currently the only way to act as a different user than the default ones, is by using OIDC authentication. Here the OIDC provider decides which role the user has. ## Role Mapping The following table shows which role is required to access the endpoints of the various services at a high level. Only the minimum role required to access the group of endpoints is shown. For the more in-depth documentation of all endpoints, head over to the [API documentation](../08-References/Control%20Plane/metal-api/metal-api.md). | **Service** | **Group of Endpoints** | **Minimum Role** | | ------------------------------- | ------------------------------- | ---------------- | | **audit-service** | Reading audit traces | Viewer | | **filesystem-service** | Reading filesystem layouts | Viewer | | | Managing filesystem layouts | Admin | | **firewall-service** | Reading firewalls | Viewer | | | Allocating firewalls | Editor | | **firmware-service** | All endpoints | Admin | | **image-service** | Reading images | Viewer | | | Managing images | Admin | | **ip-service** | Reading IPs | Viewer | | | Managing IPs | Editor | | **machine-service** | Reading machines and issues | Viewer | | | Managing machines and issues | Editor | | | IPMI operations | Editor | | | Updating, deleting machines | Admin | | | Updating firmware | Admin | | **network-service** | Reading networks | Viewer | | | Allocating and freeing networks | Editor | | | Managing networks | Admin | | **partition-service** | Reading partitions | Viewer | | | Managing partitions | Admin | | **project-service** | Reading projects | Viewer | | | Managing projects | Admin | | **size-service** | Reading sizes | Viewer | | | Managing reservations | Editor | | | Managing sizes | Admin | | **sizeimageconstraint-service** | Reading size image constraints | Viewer | | | Managing size image constraints | Admin | | **switch-service** | Reading switches | Viewer | | | Managing switches | Admin | | **tenant-service** | Reading tenants | Viewer | | | Managing tenants | Admin | | **user-service** | Getting user information | Viewer | | **vpn-service** | Getting VPN auth key | Admin | --- ## Networking We spent a lot of time on trying to provide state-of-the-art networking in the data center. This document describes the requirements, ideas and implementation details of the network topology that hosts the metal-stack. The document is separated into three main sections describing the constraints, theoretical ideas and implementation details. ## Requirements Finding the requirements for this greenfield project was kicked off with a handful of design parameters that included: - Investigation of the idea of a **layer-3 based infrastructure** to overcome the drawbacks of traditional layer-2 architectures. - Application of a routing technology that involves a single stand-alone protocol **BGP** for operational simplicity. - Utilization of the overlay virtual network technology **EVPN** to support cost-effective scaling, efficient network information exchange and a manageable amount of administration effort. - Applying the routing topology on top of a completely new physical infrastructure that is designed as a CLOS network topology. Evaluation of those parameters led to more specific requirements: - Physical Wiring: - The data center is made of a leaf-spine CLOS topology containing: - leaf switches - spine switches - exit switches - management server - management switch - tenant servers - tenant firewalls. - Bare metal servers are dual-attached to leaf switches. The bare metal servers either become tenant servers or firewalls for a group of tenant servers. - All network switches are connected to a management switch. A management server provides access to this management network. - Network Operation Characteristics: - IPv4 based network. - No IPv6 deployment. - Utilization of external BGP. - Numbered BGP only for peerings at exit switches with third parties (Internet Service Provider). - Overall BGP unnumbered. - 4-byte private ASN instead of default 2-byte ASN for BGP. - Network operation relies on SONiC Linux. - Bleeding edge Routing-to-the-Host/EVPN-to-the-Host with ordinary Linux distributions. - Layer-3 routing using BGP and VXLAN/EVPN. - Every VTEP acts as a layer-3 gateway and does routing. Routing is done on both the ingress and the egress VTEP (aka distributed symmetric routing). - Tenant isolation is realized with VRF. - Internet Access is implemented with route leak on the firewall servers and during the PXE-Process with route leak on the exit switches. - MTU 9216 is used for VXLAN-facing interfaces, otherwise MTU 9000 is used. Furthermore, requirements such as *operational simplicity* and *network stability* that *a small group of people can effectively support* have been identified being a primary focus for building metal-stack. ## Concept The theoretical concept targets the aforementioned requirements. New technologies have been evaluated to apply the best solutions. The process was heavily inspired by the work of Dinesh G. Dutt regarding BGP ([bgp-ebook](https://www.nvidia.com/en-us/networking/border-gateway-protocol/)), EVPN ([evpn-ebook](https://www.nvidia.com/en-us/networking/evpn-ebook/)) and the his 2019 work "[Cloud Native Data Center Networking](https://www.oreilly.com/library/view/cloud-native-data/9781492045595/)" (O'Reilly), which teaches some interesting basics. External BGP together with network overlay concepts as EVPN can address the essential demands. These revolutionary concepts are part of the next evolutionary step in data center design. It overcomes common issues of traditional layer 2 architectures (e.g. VLAN limitations, network visibility for operations, firewall requirements) by introducing a layer 3 based network topology. ### CLOS A CLOS topology is named after the pioneer Charles Clos (short: **CLOS**) who first formalized this approach. CLOS defines a multistage network topology that is used today to improve performance and resilience while enabling a cost effective scalability. A CLOS topology comprises network switches aggregated into spine and leaf layers. Each leaf switch (short: **leaf**) is connected to all spine switches (short: **spine**) but there is no direct leaf-to-leaf or spine-to-spine connection (See: picture 1). ![2 Layer CLOS Topology](2-layer-leaf-spine.svg) > Picture 1: Fragment of CLOS to show leaf-spine layer. This data center network architecture, based on a leaf-spine architecture, is also know as "two-tier" CLOS topology. ![3 Layer CLOS Topology](3-layer-leaf-spine.svg) > Picture 2: Fragment to show a 3-stage, 2-layer CLOS topology. Tenant servers are dual-attached to the leaf layer in order to have redundancy and load balancing capability (Picture 2). The set of leaves, spine switches and tenant servers define stages. From top down each server is reachable with 3 hops (spine -> leaf -> server). This is why that CLOS design is called a 3-stage CLOS. Consistent latency throughout the data center are an outcome of this design. It is not only important to have a scalable and resilient infrastructure but also to support planning and operation teams. Visibility within the network is of significant meaning for them. Consequently layer-3 routing in favor of layer-2 bridging provides this kind of tooling. ### BGP For routing the **Border Gateway Protocol (BGP)**, more specific: External BGP was selected. Extensive testing and operational experiences have shown that External BGP is well suited as a stand-alone routing protocol (see: [RFC7938](https://datatracker.ietf.org/doc/html/rfc7938)). Not all tenant servers are connected to the same leaf. Instead they can be distributed among any of the leaves of the data center. To not let this detail restrict the intra-tenant communication it is required to interconnect those layer-2 domains. In the context of BGP there is a concept of overlay networking with VXLAN/ EVPN that was evaluated to satisfy the needs of the metal-stack. #### BGP Unnumbered In BGP traditionally each BGP peer-facing interface requires a separate IPv4 address. This consumes a lot of IP addresses. [RFC 5549](https://datatracker.ietf.org/doc/html/rfc5549) defines the BGP unnumbered standard. It allows to use interface's IPv6 link local address (LLA) to set up a BGP session with a peer. With BGP unnumbered the IPv6 LLA of the remote is automatically discovered via Router Advertisement (RA) protocol. Important: This does not (!) mean that IPv6 must be deployed in the network. BGP uses [RFC 5549](https://datatracker.ietf.org/doc/html/rfc5549) to encode IPv4 routes as reachable over IPv6 next-hop using the LLA. Having unnumbered interfaces does not mean no IPv4 address may be in place. It is a good practice to configure an IP address to the never failing and always present local loopback interface (lo). This lo address is reachable over BGP from other peers because the [RFC 5549](https://datatracker.ietf.org/doc/html/rfc5549) standard provides an encoding scheme to allow a router to advertise IPv4 routes with an IPv6 next-hop. BGP unnumbered also has an advantage from security perspective. It removes IPv4 and global IPv6 addresses from router interfaces, thus reducing the attack vector. To sum it up: - BGP unnumbered uses IPv6 next-hops to announce IPv4 routes. - There is no IPv6 deployment in the network required. - IPv6 just has to be enabled on the BGP peers to provide LLA and RA. *In External BGP, ASN is how BGP peers know each other.* #### ASN Numbering Within the data center each BGP router is identified by a private autonomous system number (ASN). This ASN is used for internal communication. The default is to have 2-byte ASN. To avoid having to find workarounds in case the ASN address space is exhausted, a 4-byte ASN (see [RFC 6793](https://datatracker.ietf.org/doc/html/rfc6793)) that supports up to 95 million private ASNs (4200000000–4294967294, see [RFC 6996](https://www.rfc-editor.org/rfc/rfc6996.html)) is used from the beginning. ASN numbering in a CLOS topology should follow a model to avoid routing problems (path hunting) due to it's redundant nature. Within a a two-tier CLOS topology the following ASN numbering model is suggested to solve path hunting problems: - Leaves have unique ASN - Spines share an ASN - Exit switches share an ASN A illustrated example of the background of this architecture decision can be inspected in the chapter "BGP’s ASN Numbering Scheme" ("BGP’S PATH HUNTING PROBLEM") of the previously mentioned "Cloud Native Data Center Networking" book. To summarize that, one can say: Since all nodes receive or know the physical connection status of all other nodes in the network, the nodes potentially have routing information that they do not know whether they still have up to date, since it takes some time before they are fully distributed in the network. Routes to nodes may actually no longer exist (because not a single link to the node, but the node itself has failed) or the path may have changed. To determine how and whether a particular node can be reached, a path search must therefore be carried out at all its communication partners or BGP routers. Essentially, the sharing of ASNs reduces the transmission of incorrect or outdated path information (this reduces path transmissions and calculations and thus saves resources). #### Address-Families As stated, BGP is a multi-protocol routing protocol. Since it is planned to use IPv4 and overlay networks using EVPN/VXLAN several address-families have to be activated for the BGP sessions to use: - IPv4 unicast address-family - L2 EVPN address-family ### EVPN Ethernet VPN (EVPN, see [RFC 7432](https://www.rfc-editor.org/rfc/rfc7432.html)) is an overlay virtual network that connects layer-2 segments over layer-3 infrastructure. EVPN is an answer to common problems of entire layer-2 data centers. #### The necessity of EVPN Challenges such as large failure domains, spanning tree complexities, difficult troubleshooting and scaling issues are addressed by EVPN: - **administration**: less routers are involved in configuration (with VLAN every switch on routing-paths needs VLAN awareness). The configuration is less error prone due to the nature of EVPN and the good support in FRR. - **scaling**: EVPN overcomes scaling issues with traditional VLANs (max. 4094 VLANs). - **cost-effectiveness**: EVPN is an overlay virtual network. Not every switch on the routing path needs EVPN awareness. This enables the use of standard routers (in contrast to traditional VLAN); e.g.: spine switches act only as EVPN information replicator and do not need to have knowledge of specific virtual networks. - **efficiency**: EVPN information is exclusively exchanged via BGP (Multiprotocol BGP, see [RFC 4760](https://datatracker.ietf.org/doc/html/rfc4760)). Only a single eBGP session is needed to advertise layer-2 reachability. No other protocols beneath BGP are involved and flood traffic is reduced to a minimum (no "flood-and-learn", no BUM traffic). Virtual routing permits multiple network paths without the need of multiple switches. Hence the servers are logically isolated by assigning their networks to dedicated virtual routers using virtual routing and forwarding (short, **VRF**, see [Linux Virtual Routing and Forwarding](https://docs.kernel.org/networking/vrf.html) and [SONiC VRF support](https://github.com/sonic-net/SONiC/blob/master/doc/vrf/sonic-vrf-hld.md)). #### The operation of EVPN EVPN (technology) is based on BGP as control plane protocol (underlay) and VXLAN as data plane protocol (overlay). As EVPN is an overlay network, only the VXLAN Tunnel End Points (VTEPs) must be configured. In the case of two-tier CLOS networks leaf switches are tunnel endpoints. As described earlier, a dedicated VRF is used for each new tenant. VRF enables true multi-tenancy/isolation for routing tables. This is why the same ip-addresses or -networks can be used for tenants with different meanings without collisions or conflicts. In EVPN routing is assumed to occur in the context of a VRF. VRF enables true multitenancy/isolatation for routing tables. Therewith, VRF is the first step for EVPN configuration and there is a 1:1 relationship between tenant and VRF. To enable layer-2 connectivity, we need a special interface to route between layer-2 networks. This interface is called Switched VLAN Interface (SVI). The SVI is realized with a VLAN. It is part of a VRF (layer-3). The VTEP configuration requires the setup of a VXLAN interface. A VLAN aware bridge interconnects the VXLAN interface and the SVI. Required resources to establish the EVPN control plane: - VRF: because routing happens in the context of this interface. - SVI: because remote host routes for symmetric routing are installed over this interface. - VLAN-aware bridge: because router MAC addresses of remote VTEPs are installed over this interface. - VXLAN Interface / VXLAN Tunnel Endpoint: because the VRF to layer-3 VNI mapping has to be consistent across all VTEPs) ![EVPN VTEP](evpn-vtep.svg) > Picture 3: Required interfaces on the switch to wire up the vrf to swp 1 connectivity with a given vxlan Integrated routing and bridging (IRB) is the most complex part of EVPN. You could choose between centralized or distributed routing, and between asymmetrical (routing on ingress) or symmetrical (routing on ingress and egress) routing. We expect a lot of traffic within the data center itself which implies the need to avoid zigzag routing. This is why we go with distributed routing model. Further it is recommended to use the symmetric model since it makes the cut in most cases and has advantages in scalability (see "EVPN in the Data Center", Dinesh G. Dutt). ### MTU In a layer-3 network it is important to associate each interface with a proper Maximum Transmission Unit (MTU) to avoid fragmentation of IP packets. Typical modern networks do not fragment IP packets and the introduction of VXLAN adds another additional header to the packets that must not exceed the MTU. If the MTU is exceeded, VXLAN might just fail without error. This already represents a difficult-to-diagnose connectivity issue that has to be avoided. It is common practice to set the MTU for VXLAN facing interfaces (e.g. inter-switch links) to a value of `9216` to compensate the additional VXLAN overhead and an MTU of `9000` as a default to other interfaces (e.g. server facing ports). The common MTU of `1500` is not sufficient for traffic inside a data center! ### VRF Routing is needed for communication between VXLAN tunnels or between a VXLAN tunnel and an external networks. VXLAN routing supports layer-3 multi-tenancy. All routing occurs in the context of a VRF. There is a 1:1 relation of a VRF to a tenant. Picture 3 illustrates this. Servers A and B belong to the same vrf VRF1. Server C is enslaved into VRF2. There is no communication possible between members of VRF1 and those of VRF2. ![Two routing tables](vrf-simple.svg) > Picture 4: Illustration of two distinct routing tables of VRF1 (enslaved: servers A and B) and VRF2 (enslaved: server C) To leaverage the potential and power of BGP, VRF, EVPN/VXLAN without a vendor lock-in the implementation relies on hardware that is supported by open network operating system: SONiC. ## Implementation Implementation of the network operation requires the data center infrastructure to be in place. To implement a functional meaning for the parts of the CLOS network, all members must be wired accordingly. ### Physical Wiring Reference: See the [CLOS overview picture](./01-theory.md#clos) | Name | Wiring | | :-------------------------- | :-------------------------------------------------------------------------------------------- | | Tenant server (aka Machine) | Bare metal server that is associated to a tenant. Dual-connected to leafs. | | Tenant firewall | Bare metal server that is associated to a tenant. Dual-connected to leafs. | | Leaf | Network Switch that interconnects tenant servers and firewalls. Connected to spines. | | Spine | Network switch that interconnects leafs and exit switches. | | Exit | Network switch that connects to spines and interconnects to external networks. | | Management Server | Jump-host to access all network switches within the CLOS topology for administrative purpose. | | Management Switch | Connected to the management port of each of the network switches. | ![Physical Wiring](network-physical-wiring.svg) > Picture 5: This illustration shows an example of a suitable physical wiring inside a metal-stack partition. Tenant servers are organized into a layer called projects. In case those tenant servers require access to or from external networks, a new tenant server to function as a firewall is created. Leaf and spine switches form the fundament of the CLOS network to facilitate redundancy, resilience and scalability. Exit switches establish connectivity to or from external networks. Management Switch and Management Server are mandatory parts that build a management network to access the network switches for administration. To operate the CLOS topology, software defined configuration to enable BGP, VRF, EVPN and VXLAN must be set up. ![Network VRFs across the different switch layers](network-vrfs.svg) > Picture 6: This illustration shows the VRF tenant separation and VRF termination happening on the firewall for the tenant VRF and external network VRFs. ### Network Operating Systems SONiC as the network operating system will be installed on all network switches (leaves, spines, exit switches) within the CLOS topology. SONiC cannot be installed on bare metal servers that require BGP/EVPN but does not have a switching silicon. Components without a switching silicon are: - tenant servers - tenant firewalls - management server There exist two paradigms to use BGP and/or VXLAN/EVPN on non switching bare metal servers: **BGP-to-the-host** and **EVPN-to-the-host**. Both describe a setup of Free Range Routing Framework (see [frrouting.org](https://frrouting.org/)) and its configuration. FRR seamlessly integrates with the native Linux IP networking stacks. Starting with an explanation of the tenant server's BGP-to-the-Host helps to get an insight into the setup of the CLOS network from a bottom-up perspective. #### Tenant Servers: BGP-to-the-Host Tenant servers are dual-connected to leaf switches. To communicate with other servers or reach out to external networks they must join a BGP session with each of the leaf switches. Thus, it is required to bring BGP to those hosts (aka BGP-to-the-Host). Each tenant server becomes a BGP router (aka BGP speaker). BGP-to-the-Host is established by installing and configuring FRR. The required FRR configuration for tenant servers is limited to a basic setup to peer with BGP next-hops: ```bash # /etc/network/interfaces auto lo iface lo inet static address 10.0.0.1/32 auto lan0 iface lan0 inet6 auto mtu 9000 auto lan1 iface lan1 inet6 auto mtu 9000 ``` > Listing 1: Network interfaces of a tenant server. Listing 1 shows the local interfaces configuration. lan0 and lan1 connect to the leaves. As described, there is no IPv4 address assigned to them (BGP unnumbered). The local loopback has an IPv4 address assigned that is announced by BGP. The required BGP configuration: ```bash # /etc/frr/frr.conf frr version 7.0 frr defaults datacenter log syslog debugging service integrated-vtysh-config ! interface lan0 ipv6 nd ra-interval 6 no ipv6 nd suppress-ra ! interface lan1 ipv6 nd ra-interval 6 no ipv6 nd suppress-ra ! router bgp 4200000001 bgp router-id 10.0.0.1 bgp bestpath as-path multipath-relax neighbor TOR peer-group neighbor TOR remote-as external neighbor TOR timers 1 3 neighbor lan0 interface peer-group TOR neighbor lan1 interface peer-group TOR neighbor LOCAL peer-group neighbor LOCAL remote-as internal neighbor LOCAL timers 1 3 neighbor LOCAL route-map local-in in bgp listen range 10.244.0.0/16 peer-group LOCAL address-family ipv4 unicast redistribute connected neighbor TOR route-map only-self-out out exit-address-family ! bgp as-path access-list SELF permit ^$ ! route-map local-in permit 10 set weight 32768 ! route-map only-self-out permit 10 match as-path SELF ! route-map only-self-out deny 99 ! ``` > Listing 2: FRR configuration of a tenant server. The frr configuration in Listing 2 starts with `frr defaults datacenter`. This is a marker that enables compile-time provided settings that e.g. set specific values for BGP session timers. This is followed by a directive to state that instead of several configuration files for different purposes a single *frr.conf* file is used: `service integrated-vtysh-config`. The two interface specific blocks starting with `interface ...` enable the RA mechanism that is required for BGP unnumbered peer discovery. There is a global BGP instance configuration `router bgp 4200000001` that sets the private ASN. The BGP router configuration contains a setup that identifies the BGP speaker `bgp router-id 10.0.0.1`. This router id should be unique. It is a good practice to assign the local loopback IPv4 as router-id. To apply the same configuration to several interfaces a peer group named `TOR` is defined via `neighbor TOR peer-group`. `remote-as external` activates external BGP for this peer group. To have a fast convergence, limits of default timers are reduced by `timer 1 3` section. The two BGP-peer-facing interfaces are enslaved into the peer-group to inherit the peer-group's setup. Activation of IPv4 unicast protocol is completed with `address-family ipv4 unicast`. To prevent a tenant server from announcing other paths than `lo` interface a route-map `only-self-out` is defined. This route map is activated within the ipv4 address family: `neighbor TOR route-map only-self-out out`. Application of the route map `only-self-out` enables to announce only local ip(s). This is to avoid that a tenant server announces paths to other servers (prevents unwanted traffic). To achieve this: - the route-map named `only-self-out` permits only matches against an access list named `SELF` - access list `SELF` permits only empty path announcements - the path of the tenant server itself has no ASN. It is always empty (see line `*> 10.0.0.2/32 0.0.0.0 0 32768 ?`): ```bash root@machine:~# vtysh -c 'show bgp ipv4 unicast' BGP table version is 7, local router ID is 10.0.0.2, vrf id 0 Default local pref 100, local AS 4200000002 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *= 0.0.0.0/0 lan1 0 4200000012 4200000040 i *> lan0 0 4200000011 4200000040 i *= 10.0.0.1/32 lan1 0 4200000012 4200000001 ? *> lan0 0 4200000011 4200000001 ? *> 10.0.0.2/32 0.0.0.0 0 32768 ? *= 10.0.0.78/32 lan1 0 4200000012 4200000001 ? *> lan0 0 4200000011 4200000001 ? Displayed 4 routes and 7 total paths ``` That is why only the self ip (loopback ip) is announced. To allow for peering between FRR and other routing daemons on a tenant server a `listen range` is specified to accept iBGP sessions on the network `10.244.0.0/16`. Therewith it gets possible that pods / containers like metal-lb with IPs of this range may peer with FRR. This is the only place where we use iBGP in our topology. For local peering this has the advantage, that we don't need an additional ASN that has to be handled / pruned in the AS-path of routes. Routes coming from other routing daemons look as if they are configured on the tenant server's lo interface from the viewpoint of the leaves. iBGP routes are differently handled than eBGP routes in BGPs best path algorithm. Generally BGP has the rule to prefer eBGP routes over iBGP routes (see ['eBGP over iBGP'](https://medium.com/netdevops/how-bgp-best-path-selection-works-80e6e7b2da2b) ). BGP adds automatically an weight based on the route type. To overcome this issue, we set the weight of iBGP routes to the same weight that eBGP routes have, namely 32768 (`set weight 32768`). Without this configuration we will only get a single route to the IPs announced via iBGP. So this setting is essential for HA/failover! Statistics of the established BGP session can be viewed locally from the tenant server via: `sudo vtysh -c 'show bgp ipv4 unicast'` To establish this BGP session a BGP setup is required on the leaves as well. #### Leaf Setup Every leaf switch is connected to every spine switch. Tenant servers can be distributed within the data center and thus be connected to different leaves. Routing for tenant servers is isolated in unique VRFs. These constraints imply several configuration requirements for the leaf and spine switches: - leaves define tenant VRFs - leaves terminate VXLAN tunnels (aka "VXLAN tunnel endpoint" = VTEP) The leaf setup requires the definition of a tenant VRF that enslaves the tenant server facing interfaces: ```bash # /etc/network/interfaces # [...] iface vrf3981 vrf-table auto iface swp1 mtu 9000 post-up sysctl -w net.ipv6.conf.swp1.disable_ipv6=0 vrf vrf3981 # [...] ``` > Listing 3: Fragment that shows swp1 being member of vrf vrf3981. There is a VRF definition `iface vrf3981` to create a distinct routing table and a section `vrf vrf3981` that enslaves swp1 (connects the tenant server) into the VRF. Those host facing ports are also called `edge ports`. Additional to the VRF definition the leaf must be configured to provide and connect a VXLAN interface to establish a VXLAN tunnel. This network virtualization begins at the leaves. Therefore, the leaves are also called Network Virtualization Edges (NVEs). The leaves encapsulate and decapsulate VXLAN packets. ```bash # /etc/network/interfaces # [...] iface bridge bridge-ports vni3981 bridge-vids 1001 bridge-vlan-aware yes iface vlan1001 mtu 9000 vlan-id 1001 vlan-raw-device bridge vrf vrf3981 iface vni3981 mtu 9000 bridge-access 1001 bridge-arp-nd-suppress on bridge-learning off mstpctl-bpduguard yes mstpctl-portbpdufilter yes vxlan-id 3981 vxlan-local-tunnelip 10.0.0.11 # [...] ``` > Listing 4: Fragment that shows VXLAN setup for vrf vrf3981. All routing happens in the context of the tenant VRF. To send and receive packets of a VRF, several interface are in place. A bridge is used to attach VXLAN interface `bridge-ports vni3981` and map its local VLAN to a VNI. Router MAC addresses of remote VTEPs are installed over this interface. The Routed VLAN Interface or Switched Virtual Interface (SVI) `iface vlan1001` is configured corresponding to the per-tenant VXLAN interface. It is attached to the tenant VRF. Remote host routes are installed over this SVI. The `vlan-raw-device bridge` is used to associate the SVI with the VLAN aware bridge. For a packet received from a locally attached host the SVI interface corresponding to the VLAN determines the VRF `vrf vrf3981`. The VXLAN interface `iface vni3981` defines a tunnel address that is used for the VXLAN tunnel header `vlxan-local-tunnelip 10.0.0.11`. This VTEP IP address is typically the loopback device address of the switch. When EVPN is provisioned, data plane MAC learning for VXLAN interfaces must be disabled because the purpose of EVPN is to exchange MACs between VTEPs in the control plane: `bridge-learning off`. EVPN is responsible for installing remote MACs. `bridge-arp-nd-suppress` suppresses ARP flooding over VXLAN tunnels. Instead, a local proxy handles ARP requests received from locally attached hosts for remote hosts. ARP suppression is the implementation for IPv4; ND suppression is the implementation for IPv6. It is recommended to enable ARP suppression on all VXLAN interfaces. Bridge Protocol Data Unit (BPDU) are not transmitted over VXLAN interfaces. So as a good practice bpduguard and pbdufilter are enabled with `mstpctl-bpduguard yes` and `mstpctl-portbpdufilter yes`. These settings filter BPDU and guard the spanning tree topology from unauthorized switches affecting the forwarding path. `vxlan-id 3981` specifies the VXLAN Network Identifier (VNI). The type of VNI can either be layer-2 (L2) or layer-3 (L3). This is an implicit thing. A VNI is a L3 VNI (L3VNI) when a mapping exists that maps the VNI to a VRF (configured in `/etc/frr/frr.conf`) otherwise it is a L2 VNI (L2VNI). ```bash # /etc/frr/frr.conf # [...] vrf vrf3981 vni 3981 exit-vrf #[...] router bgp 4200000011 # [...] address-family ipv4 unicast redistribute connected route-map LOOPBACKS # [...] address-family l2vpn evpn neighbor FABRIC activate advertise-all-vni exit-address-family # [...] router bgp 4200000011 vrf vrf3981 # [...] address-family ipv4 unicast redistribute connected neighbor MACHINE maximum-prefix 100 exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast exit-address-family # [...] route-map LOOPBACKS permit 10 match interface lo # [...] ``` > Listing 5: Leaf FRR configuration. Listing 5 shows the required FRR configuration of the BGP control plane. Only content not discussed so far is explained. The section `vrf vrf3981` contains the mapping from layer-3 VNI to VRF. This is required to be able to install EVPN IP prefix routes (type-5 routes) into the routing table. Further the file contains a global BGP instance `router bgp 4200000011` definition. A new setting `redistribute connected route-map LOOPBACKS` is in place to filter the redistribution of routes that are not matching the local loopback interface. The route-map is defined with `route-map LOOPBACKS permit 10`. With the configuration line `address-family l2vpn evpn`, the EVPN address family is enabled between BGP neighbours. `advertise-all-vni` makes the switch a VTEP configures it in such a way, that all locally configured VNIs should be advertised by the BGP control plane. The second BGP instance configuration is specific to the tenant VRF `router bgp 4200000011 vrf vrf3981`. This VRF BGP instance configures the l2vpn evpn address family with `advertise ipv4 unicast` to announce IP prefixes in BGP's routing information base (RIB). This is required to apply learned routes to the routing tables of connected hosts. The Maximum-Prefix feature is useful to avoid that a router receives more routes than the router memory can take. The maximum number of prefixes a tenant server is allowed to announce is limited to `100` with: `neighbor MACHINE maximum-prefix 100`. #### Spine setup On the spine switches the setup is quite simple. `/etc/network/interfaces` contains the loopback interface definition to support BGP unnumbered and listings for connected switch ports to provide proper MTUs (Listing 6). I.e. swp1 is configured with an MTU of 9216 as it is a VXLAN-facing interface. ```bash # /etc/network/interfaces # [...] iface swp1 mtu 9216 ``` > Listing 6: Fragment of spine interface configuration. The spines are important to forward EVPN routes and transport VXLAN packets between the VTEPs. They are not configured as VTEPs. The FRR configuration only contains the already known global BGP instance configuration `router bgp 4200000020` plus the activation of the l2vpn evpn address family `address-family l2vpn evpn` to enable EVPN type-5 route forwarding (Listing 7). ```bash hostname spine01 username admin nopassword ! # [...] interface swp1 ipv6 nd ra-interval 6 no ipv6 nd suppress-ra ! # [...] ! router bgp 4200000020 # [...] ! address-family l2vpn evpn neighbor FABRIC activate exit-address-family ! # [...] ``` > Listing 7: Fragment of spine FRR configuration to show the activated L2VPN EVPN address-family. #### Tenant Firewalls: EVPN-to-the-Host In case a tenant server needs to reach out to external networks as the Internet, a tenant firewall is provisioned. The firewall is a bare metal server without a switching silicon. Thus, there is no installation of SONiC. [FRR](https://frrouting.org) provides the BGP / EVPN functionality known as `EVPN-to-the-host`. The firewall is configured as a VTEP and applies `dynamic route-leaking` to install routes of an foreign VRF. The set of routes that are leaked are restricted with route-maps. As Listing 8 shows, the firewall is configured with VXLAN interfaces as known from the leaf setup. Additionally, a VXLAN setup for VRF `vrfInternet` is added to provide Internet access. vrfInternet contains a route to the Internet that will be leaked into the tenant VRF. Traffic that originates from the tenant network `10.0.0.0/22` will be masqueraded before leaving the interface `vlanInternet` towards the internet. ```bash # /etc/network/interfaces # [...] iface bridge # [...] iface vlan1001 # [...] iface vni3981 # [...] iface vrf3981 # [...] iface vlanInternet mtu 9000 vlan-id 4009 vlan-raw-device bridge vrf vrfInternet address 185.1.2.3 post-up iptables -t nat -A POSTROUTING -s 10.0.0.0/22 -o vlanInternet -j MASQUERADE pre-down iptables -t nat -D POSTROUTING -s 10.0.0.0/22 -o vlanInternet -j MASQUERADE iface vniInternet mtu 9000 bridge-access 4009 mstpctl-bpduguard yes mstpctl-portbpdufilter yes vxlan-id 104009 vxlan-local-tunnelip 10.0.0.40 iface vrfInternet mtu 9000 vrf-table auto ``` > Listing 8: Interfaces configuration of firewall to show the VTEP interface configuration. To install a default route into the routing table of tenant VRF vrf3981 a dynamic route leak is established for it (`import vrf vrfInternet`). With the help of a route-map `import vrf route-map vrf3981-import-map` only the default route will be leaked: ```bash root@firewall01:~# vtysh -c 'show ip route vrf vrf3981' # [...] VRF vrf3981: S>* 0.0.0.0/0 [1/0] is directly connected, vrfInternet(vrf vrfInternet), 03:19:26 B>* 10.0.0.1/32 [20/0] via 10.0.0.12, vlan1001 onlink, 02:34:48 * via 10.0.0.11, vlan1001 onlink, 02:34:48 B>* 10.0.0.2/32 [20/0] via 10.0.0.12, vlan1001 onlink, 02:34:49 * via 10.0.0.11, vlan1001 onlink, 02:34:49 ``` To receive responses from vrfInternet in vrf3981 a route is leaked into vrfInternet as well (`import vrf vrf3981`) restricted with the route-map `vrfInternet-import-map` that allows leaking of the tenant routes as well as internet prefixes used on worker nodes of the tenant. To limit the prefixes that are announced from the firewall within the global BGP instance a route-map `only-self-out` is defined and applied within the ipv4 and l2vpn evpn address family. Together with the definition of an as path access list `bgp as-path access-list` it avoids the announcement of prefixes to non VRF BGP peers. ```bash # /etc/frr/frr.conf ! vrf vrf3981 vni 3981 ! vrf vrfInternet vni 104009 ! # [...] ! router bgp 4200000040 # [...] ! address-family ipv4 unicast # [...] neighbor FABRIC route-map only-self-out out exit-address-family ! ! router bgp 4200000040 vrf vrf3981 # [...] address-family ipv4 unicast redistribute connected import vrf vrfInternet import vrf route-map vrf3981-import-map # [...] address-family l2vpn evpn advertise ipv4 unicast # [...] router bgp 4200000040 vrf vrfInternet # [...] address-family ipv4 unicast redistribute connected import vrf vrf3981 import vrf route-map vrfInternet-import-map # [...] address-family l2vpn evpn advertise ipv4 unicast # [...] bgp as-path access-list SELF permit ^$ ! route-map only-self-out permit 10 match as-path SELF ! route-map only-self-out deny 99 ! route-map LOOPBACKS permit 10 match interface lo ! ip prefix-list vrf3981-import-prefixes seq 100 permit 0.0.0.0/0 ! route-map vrf3981-import-map permit 10 match ip address prefix-list vrf3981-import-prefixes ! route-map vrf3981-import-map deny 99 ! ip prefix-list vrfInternet-import-prefixes seq 100 permit 10.0.0.0/22 le 32 ip prefix-list vrfInternet-import-prefixes seq 101 permit 185.1.2.0/24 le 32 ip prefix-list vrfInternet-import-prefixes seq 102 permit 185.27.0.0/27 le 32 ! route-map vrfInternet-import-map permit 10 match ip address prefix-list vrfInternet-import-prefixes ! route-map vrfInternet-import-map deny 99 ! line vty ! ``` > Listing 9: FRR configuration of a tenant firewall to show route leak and prefix announcement filtering. #### Exit Switch Traffic to external networks is routed via the firewalls to the exit switch. The exit switch, as an exception, connects to the Internet Service Provider using numbered BGP. Numbered BGP implies to assign IPv4 addresses to network interfaces (See Listing 10, swp1). Interface swp1 is enslaved into `vrf vrfInternet` to include the port that is connected to the ISP within the VRF that is expected to contain a way into the Internet. The exit switch is configured to be a VTEP to terminate traffic coming from the firewall VRF `vrfInternet`. ```bash # /etc/network/interfaces # [...] iface swp1 mtu 9000 vrf vrfInternet address 172.100.0.2/30 # [...] iface vlan4000 mtu 9000 address 10.0.0.71/24 vlan-id 4000 vlan-raw-device bridge # [...] iface vlanInternet # [...] iface vniInternet # [...] iface vrfInternet # [...] ``` > Listing 10: Fragment of interfaces configuration of exit switch. The configuration of FRR is equivalent to the previously discussed ones. It contains a global BGP instance configuration that enables IPv4 unicast and l2vpn evpn address families. The vrfInternet BGP instance defines `neighbor 172.100.0.1 peer-group INTERNET` to use "old style BGP" transit network. ```bash # [..] vrf vrfInternet vni 104009 ! # [...] router bgp 4200000031 bgp router-id 10.0.0.31 neighbor FABRIC peer-group neighbor FABRIC remote-as external neighbor FABRIC timers 1 3 # [...] ! address-family ipv4 unicast neighbor FABRIC activate redistribute connected route-map LOOPBACKS exit-address-family ! address-family l2vpn evpn neighbor FABRIC activate advertise-all-vni exit-address-family ! router bgp 4200000031 vrf vrfInternet bgp router-id 10.0.0.31 bgp bestpath as-path multipath-relax neighbor INTERNET peer-group neighbor INTERNET remote-as external neighbor INTERNET timers 1 3 neighbor 172.100.0.1 peer-group INTERNET ! address-family ipv4 unicast neighbor INTERNET route-map PREPEND-PATH-TO-DISFAVOR-IN in neighbor INTERNET route-map PREPEND-PATH-TO-DISFAVOR-OUT out exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast exit-address-family ! route-map LOOPBACKS permit 10 match interface lo ! route-map PREPEND-PATH-TO-DISFAVOR-IN permit 10 set as-path prepend last-as 2 ! route-map PREPEND-PATH-TO-DISFAVOR-OUT permit 10 set as-path prepend last-as 2 ! vrf mgmt ip route 10.0.0.0/24 10.0.0.71 nexthop-vrf default exit-vrf ! ip route 0.0.0.0/0 192.168.0.254 nexthop-vrf mgmt ! line vty ! ``` > Listing 11: Fragment of FRR configuration on exit switch to give an example for numbered BGP and route leak. In addition to the standard BGP setup the exit switches have configured `static route leak` to support internet access during PXE. There is one route leak from `default` VRF into the `mgmt` VRF defined with: `ip route 0.0.0.0/0 192.168.0.254 nexthop-vrf mgmt` and another one from `mgmt` VRF into the `default` VRF: `ip route 10.0.0.0/24 10.0.0.71 nexthop-vrf default`. The first one adds a default route into the `default` VRF and the second one routes traffic destined to the PXE network back from `mgmt` VRF into the `default` VRF. To reach out into external networks each of the exit nodes joins a BGP session with a distinct external router. There is a different latency to each of these routers. To favor routes of exit nodes connected with lower latency over exit nodes with higher latency two route maps `PREPEND-PATH-TO-DISFAVOR-IN` and `PREPEND-PATH-TO-DISFAVOR-OUT` are added to high latency exit nodes. These route maps apply actions to prolong the path of the incoming and outgoing routes. Because of this path extension BGP will calculate a lower weight for these paths and favors paths via other exit nodes. It is important to know that within an address family only one route map (the last) will be applied. To apply more than one actions within a route-map the required entries can be applied to a single route-map. ### PXE Boot Mode Before a bare metal server can act as tenant server or tenant firewall, it has to be provisioned. Within the Metal domain, this provisioning mode is called "PXE Mode" since it is based on Preboot eXecution Environment (PXE). PXE uses protocols like DHCP. This requires all bare metal servers that need provisioning to be located in a layer-2 domain where DHCP is available. This domain is a VLAN `vlan4000`. A DHCP server for PXE Mode is installed on the exit switches to work in this specific VLAN. ```bash # /etc/default/isc-dhcp-server INTERFACES="vlan4000" ``` > Listing 13: DHCP server configuration of exit switches. As shown in listing 13, the PXE DHCP server is located on the exit switches and enforced to bind to interface `vlan4000`. This represents a layer-2 separation that allows only DHCP clients in the same VLAN to request IP addresses. Only unprovisioned bare metal servers are configured to be member of this VLAN. Thus unwanted or accidental provisioning is impossible. To provide `vlan4000` on the leaves (that face the bare metal servers) the exit and leaf switches are configured as VTEPs and share an interface configuration that contains the required interfaces (Listing 13). Since no EVPN routing is in place `vni104000` is configured as an L2 VNI (there is no mapping for this VNI in `/etc/frr/frr.conf`). ```bash # /etc/network/interfaces # [...] iface bridge bridge-ports vni104000 [...] bridge-vids 4000 [...] bridge-vlan-aware yes iface vlan4000 # [...] iface vni104000 # [...] ``` > Listing 13: Interfaces configuration on exit and leaf switches to show DHCP/PXE related fragments. On the leaf switches the bare metal server facing ports are configured as VLAN access ports to carry the traffic for only the PXE VLAN `vlan4000` (listing 14)to separate unprovisioned from other bare metal servers. ```bash # /etc/network/interfaces # [...] auto swp1 iface swp1 mtu 9000 bridge-access 4000 # [...] ``` > Listing 14: VLAN access setup for bare metal server facing ports on leaves. Once a bare metal server is provisioned it is deconfigured from PXE VLAN `vlan4000` to avoid accidental or unwanted provisioning. During provisioning bare metal servers get internet access via the management network of the exit switches. This is because the exit switches are announced as DHCP gateway to the DHCP clients. ### Management Network To manage network switches beside the out-of-band system console access a further management access is required. For this purpose the concept of **Management VRF** is applied. The Management VRF is a subset of VRF. It provides a separation between out-of-band management network and the in-band data plane network by introducing another routing table **mgmt**. SONiC supports eth0 to be used as the management interface. To enable and use the Management VRF all switches have to be connected via their eth0 interface to a management-switch. The management switch is connected to a management server. All access is established from within the management server. Logins to the switch are set into the Management VRF context once the Management VRF is enabled. --- ## Firewalls Firewalls in metal-stack are machines that connect networks to each other or the internet and allows traffic filtering. Only firewalls can have multiple networks attached. Firewalls do not require specialized hardware. For most scenarios any of your Machine Sizes can be used as a Firewall. :::important metal-stack does not support using existing firewall appliances for east-west traffic within metal-stack. It is technically possible to use an existing firewall appliance to filter north-south traffic. Nevertheless we encourage you to use metal-stack firewalls for north-south traffic as well to get the benefits of lower TCO due to use of commodity hardware and end-to-end configurability using metal-api. ::: ## Why do we use Machines as Firewalls? Using machines as firewalls gives us a lot of advantages. Controlling the full software stack running on the firewall gives us a lot of flexibility and enables quicker feature development. Our default [firewall image](https://github.com/metal-stack/metal-images/tree/master/firewall) uses nftables for L4 filtering and [Suricata](https://suricata.io/) for network analysis and threat detection. ### Why not integrate existing firewall appliances? We see limited benefit in using firewall appliances for the following reasons: - Firewall appliances are a fragmented product category without vendor-agnostic API or common behaviour - As a result of the fragmented landscape supporting the most common firewalls would bind development time - Appliances and bandwidth would be shared between tenants, increasing management complexity and reducing tenant isolation ### Why not use Top-of-Rack Leaf switches as firewalls? SONiC can provide basic L3/L4 packet filtering capabilities using its ACLs, which can be offloaded to the switching silicon. But unfortunately that is not enough for our requirements: - Switch ASICs vary widely in supported maximum number of active rules and supported ACL features - We require NAT, which is not supported when offloading to ASICs - Going past stateless filtering to stateful tracking requires specialized DPU hardware, increasing cost and reducing accessibility - We offer Intrusion Detection as a feature, which is also not possible while utilizing offloading - Using dedicated machines allows us to provide the same great isolation guarantees as with regular machines. If you provision the Firewall, you do not share resources or access with other tenants. In short, to offer comparable features to our current solution, we would need to disable ASIC offloading and either punt all traffic to either the weak main switch CPU, causing unpredictable performance or use specialized DPUs. ## Creating a Firewall :::info TODO: metalctl/metalcli example TODO: Configuring firewalls via firewall-controller CRDs https://github.com/metal-stack/website/issues/289 ::: ## Firewall Controller The firewall-controller allows you to configure Firewalls using CRDs from inside of Kubernetes clusters. Firewall configuration provided by the firewall-controller is hot reloaded. ## Lifecycle Firewall Machines are managed by metal-stack. The local state of Firewall machines is ephemeral, as the authoritative configuration is stored in metal-api. Manual changes to the configuration are not supported and will be overridden. Use metalctl or Firewall CRDs to apply changes to firewall configuration. ## VPN integration TODO: https://github.com/metal-stack/website/issues/290 --- ## Tailscale # Expose Services with Tailscale This guide is a recommendation on how to access services from anywhere if you are evaluating the metalstack on-prem starter without a public IP-address. These steps will guide you through the process quickly. For a deeper dive, or if you want to use alternative setups, the Tailscale articles are also linked. ## What are Tailscale and Tailnets? Tailscale is a Canadian company that offers a virtual private network solution based on WireGuard. Instead of relying on centralised VPN servers to route all traffic, Tailscale creates a mesh VPN called Tailnet. It creates encrypted peer-to-peer connections between network subscribers. This approach is claimed to improve throughput and stability while reducing latency. Tailscale's solution is composed of components that are mostly open source. Find more information on their [open source sstatement](https://tailscale.com/opensource) and their [GitHub repository](https://github.com/tailscale). ## Setup an Account and Clients Please begin by [following these steps](https://login.tailscale.com/start) to use an authentication provider to create the first user account for your network. The first step is to install Tailscale clients on the devices from which you wish to access the Kubernetes services. Should you require to extend an invitation to additional users, this can be facilitated by navigating to the "Users" tab on the Admin Console. ## Setup the Operator ### Labels First, we will establish tags to categorise our services. Please open the 'Access controls' tab, where you will find a text editor containing all the Access Control settings in JSON format. Uncomment the `tagOwners` section and add the following tags: ```json "tagOwners": { "tag:k8s-operator": [], "tag:k8s": ["tag:k8s-operator"], } ``` The operator will use the `k8s-operator`-tag. Devices with this tag are now configured as owner for devices with the `k8s`-tag, which will be used for our services. ### Create OAuth-Client Credentials In the "Settings" tab at "OAuth clients", generate a new OAuth client. Set write permissions for "Devices - Core" and "Keys - Auth Keys". Select the `k8s-operator` tag for both. ![Tailscale Devices Configuration](tailscale-devices.png) ![Tailscale AuthKey Configuration](tailscale-authkeys.png) Therefore, a device that has been assigned the label `k8s-operator` will have the capability to register additional devices with the `k8s` tag. When you click "create", you get a client-id and client-secret, that you will need to setup the operator. ### Setup Operator with helm The most common and practical way is to use a Helm chart to setup the operator. Therefore, we first have to add and update the helm-repository of tailscale: ``` helm repo add tailscale https://pkgs.tailscale.com/helmcharts helm repo update ``` Now, we can install the helm-chart in a dedicated namespace using the credentials of the OAuth client ```bash helm upgrade \ --install \ tailscale-operator \ tailscale/tailscale-operator \ --namespace=tailscale \ --create-namespace \ --set-string oauth.clientId="" \ --set-string oauth.clientSecret="" \ --wait ``` Check on the administration console, if your operator appears on the Machines list. Alternative ways & troubleshooting: - Take the [operator.yaml manifest](https://github.com/tailscale/tailscale/blob/main/cmd/k8s-operator/deploy/manifests/operator.yaml) from the GitHub repository and make your adjustments to use [Static manifests with kubectl](https://tailscale.com/kb/1236/kubernetes-operator#static-manifests-with-kubectl) - If the operator does not show op in the Machines list, use the guide for [Troubleshooting the Kubernetes operator](https://tailscale.com/kb/1446/kubernetes-operator-troubleshooting) ## Expose Services on the Tailnet There are three ways to allow traffic to your pods from the tailnet. We can setup a dedicated Service or annotate an existing one. For more routing options, use an Ingress object. For detailed configuration options, review the article about [Expose a Kubernetes cluster workload to your tailnet (cluster ingress)](https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress) ### Add a Load Balancer Service The installed operator is looking for Service objects with the `spec.type`of `LoadBalancer`and the `spec.loadBalancerClass` of `tailscale`. ```yaml --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - name: https port: 443 targetPort: 443 type: LoadBalancer loadBalancerClass: tailscale ``` ### Annotate an existing Service Edit the Service and add the annotation `tailscale.com/expose` with the value "true": ```yaml --- apiVersion: v1 kind: Service metadata: annotations: tailscale.com/expose: "true" name: nginx spec: ... ``` Note that "true" is quoted because annotation values are strings, and an unquoted true will be incorrectly interpreted as a boolean. ### Use an Ingress To enable path-based routing, use an Ingress resource. Ingress routes only use TLS over HTTPS. To make this work, you have to enable the `MagicDNS` and `HTTPS` options in the [DNS-Tab on your Administration Console](https://login.tailscale.com/admin/dns). This enables helpful features: - Magic DNS automatically register your services with subdomains in your tailnet - HTTPS enables the provisioning of certificates for devices To set the Ingress up, just refer to `tailscale` as the `ingressClassName`: ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx spec: ingressClassName: tailscale rules: - http: paths: - path: / pathType: Prefix backend: ... ``` Please consider, that currently only paths with `pathType: prefix` are supported currently --- ## Inventory Management # Inventory Management and Discovery metal-stack does not rely on external inventory files or an external IPAM. Physical resources are discovered by the service that owns each one, and only the leaf switches need a predefined minimal configuration. The address space is managed by metal-api's built-in IPAM (`go-ipam`), scoped to projects and tenants. ## Physical Inventory The physical inventory is the hardware in a partition. metal-stack discovers the leaf switches, the machines, and the server BMCs automatically. The spine and exit switches are provisioned as static underlay rather than discovered. ### Discovered **Switches**: Leaf switches are discovered and registered during initial deployment. They ship running the Open Network Install Environment (ONIE), a minimal Linux environment that downloads and installs the SONiC network operating system. After installation the switch pulls a minimal configuration via ZTP to establish basic connectivity. The remaining components are installed via CI. From then on the switch is orchestrated by the `metal-core` container running on it, which registers it with metal-api. **Machines**: Tenant machines are discovered automatically during PXE boot. pixiecore intercepts the DHCP broadcast and serves metal-hammer via iPXE, which collects the machine's hardware details. **Server BMCs**: metal-bmc discovers the chassis management controllers by scanning the DHCP leases of the management network and reaching each one over IPMI/Redfish. Many of the [server types supported by metal-stack](/hardware/) are multi-node chassis (such as the Supermicro MicroCloud), so a single BMC can serve several machines rather than mapping one-to-one. Each node is identified when it PXE-boots. Each component reports its findings to metal-api: | Resource | Discovered by | Reported fields | | :--- | :--- | :--- | | Leaf switch | `metal-core` | Switch ID (hostname), ports (names + MACs), OS vendor + version, metal-core version, management IP, management user | | Machine | `metal-hammer` | Serial number, RAM, disks, MAC addresses, FRU data, LLDP neighbors (switch chassis + port per NIC) | | Server BMC | `metal-bmc` | Manufacturer, BIOS version, firmware, power state, chassis UUID | LLDP is the link between the machine inventory and the switch inventory. metal-hammer reports each NIC's neighbor, and metal-api correlates those against the registered switch ports to build the connection map between switch ports and machine ports. The physical cable layout is therefore discovered during the same boot, never maintained by hand, and it is what drives per-port VRF configuration when a machine is later allocated to a tenant network. ### Preconfigured Part of the partition is provisioned declaratively instead of being discovered. This configuration is maintained as version-controlled infrastructure-as-code using Ansible and rolled out by CI, outside metal-stack's control plane. **Leaf switches**, although discovered and then dynamically managed, still need a minimal base configuration to bootstrap metal-core, grouped here by purpose: - **Identity**: PartitionID, RackID, Name / Description - **Network underlay**: ASN (unique per leaf), LoopbackIP, metal-core CIDR (the switch's address in the PXE/provisioning VLAN), SpineUplinks, ManagementGateway - **Auth / transport**: HMACKey, GrpcAddress **Spine and exit switches** are not represented in metal-api's inventory. metal-api tracks only the switches that machines attach to, namely the leaf pair of each rack. Because spines and exits carry no machine ports, they are never registered and never run metal-core. They are configured once as static underlay: - **Spines** forward EVPN routes and transport VXLAN between the VTEPs (see [Spine setup](./01-theory.md#spine-setup)). They are not VTEPs themselves and hold no tenant state, so their configuration is uniform and rarely changes. - **Exit switches** connect the partition to the outside: they terminate the external network VRF arriving from the firewalls and peer with upstream routers over numbered BGP (see [Exit Switch](./01-theory.md#exit-switch)). The base configuration is templated like the rest, but the upstream peerings are individual and effectively maintained by hand. ## Logical Inventory Address management is performed by metal-api through its built-in IPAM (`go-ipam`). Each partition has a super network from which the tenant address space is allocated. How a network can be used follows from its type: - **Private** networks are child networks owned by a single project. A child prefix is allocated from the super network and assigned a unique VRF, which maps to a VNI in the EVPN/VXLAN overlay. IP addresses are allocated only within the owning project, and the VRF isolates the network from every other tenant. - **Shared** networks are private networks that have been marked as shared so that other projects may allocate IP addresses from them. They cover services that several projects in the partition need to reach while the traffic stays inside the cluster. - **External** networks represent connectivity beyond a metal-stack partition, such as internet egress, enterprise networks or storage systems. Their IP addresses can be allocated from different projects, they carry destination prefixes (routes) that are announced into the tenant VRFs, and traffic to external networks may be masqueraded at a firewall behind a dedicated IP address. Every private network, the IP addresses allocated within it, and its VRF belong to a project, which in turn belongs to a tenant. Each network routes in its own VRF (see [VRF](./01-theory.md#vrf)), so the same IP ranges can be reused across networks, and therefore across projects and tenants, without colliding. The logical inventory has a boundary at the edge of metal-stack's scope. metal-stack manages the underlay (loopbacks, private 4-byte ASNs, the PXE network) and the tenant overlay (VRF/VNI/EVPN) automatically, but it does not discover networks outside its own scope, so a pre-existing range that must not be allocated has to be documented manually. It also does not configure the BGP sessions with upstream providers. Those peerings (numbered BGP at the exit and border switches) are highly individual and are maintained as version-controlled infrastructure-as-code, tracked in git and deployed by CI. --- ## Network Segmentation metal-stack enforces network segmentation from a single source of truth. Tenants express intent at a high level (a project, a network, a machine allocation), and `metal-api` turns that intent into device-level configuration. The `metal-core` agent on each leaf renders the switch side into interface and FRR configuration, while machines and firewalls derive their own FRR and nftables configuration from the same network model. No segment inside a partition is wired by hand, so the isolation is a property of metal-api's records rather than of an operator remembering to add an ACL. The records themselves are not maintained by hand. As described under [Inventory Management and Discovery](./04-inventory-management.md), metal-stack fully owns its own inventory. The connection map that pins each machine port to its switch port is [discovered over LLDP](./04-inventory-management.md#physical-inventory) during PXE boot, and metal-api allocates every [prefix, VRF and VNI](./04-inventory-management.md#logical-inventory) itself. Segmentation is therefore derived from facts the platform observed rather than from configuration an operator transcribed, which is what keeps per-port VRF placement, prefix-lists and VNI mappings correct as machines are allocated and freed. ## Segmentation Model A tenant owns one or more projects, and a project owns one or more private networks. Each [private network](./04-inventory-management.md#logical-inventory) maps to its own VRF, which corresponds 1:1 to a VNI in the EVPN/VXLAN overlay. Because each network routes in its own VRF (see [VRF](./01-theory.md#vrf)), no two networks share a routing table, which isolates projects and tenants from one another and lets the same IP ranges be reused across networks without colliding. [Picture 6](./01-theory.md#physical-wiring) illustrates this separation and the VRF termination that happens on the firewall. From this model metal-stack draws and enforces the following boundaries: | Boundary | Mechanism | Enforced at | | :--- | :--- | :--- | | Project / tenant | VRF + VNI (EVPN/VXLAN) | leaves, firewalls, exit switches (VTEPs) | | Per machine port | VRF binding + inbound prefix-list route-map + `maximum-prefix` | leaf | | Unprovisioned machine | PXE VLAN (layer-2) | leaf access port | | Partition edge / external | numbered BGP + route-leak + ACLs | exit switch / firewall | The partition edge is the one boundary metal-stack deliberately does not own end to end. Connectivity that leaves the partition (internet egress, a DMZ, storage, or another availability zone) crosses an [external network](./04-inventory-management.md#logical-inventory) and is terminated on the exit switches, which peer with upstream routers using numbered BGP (see [Exit Switch](./01-theory.md#exit-switch)). Those peerings and the underlay base configuration are highly individual and maintained outside the control plane as version-controlled infrastructure-as-code, tracked in git and deployed by CI. ## Defense in Depth A packet from a tenant machine crosses several independent enforcement layers before it can reach anything, implementing the [defense in depth](../../06-For%20CISOs/Security/01-principles.md#defense-in-depth) principle, and because each layer is rendered from the same metal-api state they cannot drift out of agreement. Any one of them is sufficient to deny the traffic. ### Routing Isolation (VRFs) VRFs provide hard layer-3 isolation. The routing table of `vrf5417` contains no routes to destinations in other VRFs, so traffic cannot cross from one VRF to another. Route-leaking between VRFs happens only where it is explicitly configured, and even then it is constrained. On a tenant firewall, `import vrf` installs routes from a foreign VRF, but an `import vrf route-map` plus a prefix-list decide exactly which prefixes may cross (see [Tenant Firewalls](./01-theory.md#tenant-firewalls-evpn-to-the-host) and Listing 9). The default posture is no reachability, and every leak is a named, prefix-scoped exception. ### Route Filtering The leaf does not trust a machine to police its own announcements. When a machine is allocated to a project, the leaf ports discovered for the machine in the [physical inventory](./04-inventory-management.md#physical-inventory) are bound to its private network's VRF. The machine's BGP session is filtered inbound by a route-map whose prefix-list permits only the prefixes that network may use. A `maximum-prefix` limit bounds how many routes it can announce (see [Leaf Setup](./01-theory.md#leaf-setup)). A machine therefore cannot announce an address outside its allocation or flood the switch with routes. The same fabric constrains firewalls. Per-firewall route-maps filter both the prefixes and the EVPN/VNI routes the leaf advertises to a firewall. A firewall only has access to the networks it is entitled to. Unprovisioned machines never enter the routed fabric. They are isolated at layer-2 in the PXE VLAN (`vlan4000`, see [PXE Boot Mode](./01-theory.md#pxe-boot-mode)) and are removed from it the moment provisioning finishes, so a half-installed machine can neither reach nor be reached by tenant networks. ### Software-Defined Firewalls Each firewall is an ordinary bare-metal machine running FRR as an EVPN-to-the-host VTEP, one per group of tenant servers. A firewall is attached to several VRFs at once (the tenant network plus, for example, `vrfInternet`) and enforces inter-network policy by leaking only the permitted routes between them, masquerading tenant source addresses on the way out. Because it sits in the data path of every packet that leaves the tenant network, it is also where stateful packet filtering happens. On a Kubernetes cluster, the firewall is configured through Kubernetes resources, from inside the cluster itself. The [firewall-controller](../04-Kubernetes/04-firewall-controller-manager.md) watches `ClusterwideNetworkPolicy` (CWNP) custom resources and compiles them into native nftables rules, optionally running Suricata for intrusion detection. Two networks can exchange packets only where a route leak connects their VRFs. What is then allowed, by port, protocol, source and destination, is governed by the CWNP. Together these layers implement the defense-in-depth posture described in the [security principles](../../06-For%20CISOs/Security/01-principles.md). The result is isolation by default at layer-3, explicit and prefix-scoped exceptions, and stateful policy at the edge of every tenant network. --- ## Gardener [Gardener](https://gardener.cloud/) is an open source project for orchestrated Kubernetes cluster provisioning. It supports many different cloud providers, metal-stack being one of them. Using the Gardener project, metal-stack can act as a machine provider for Kubernetes worker nodes. The idea behind the Gardener project is to start with a dedicated set of Kubernetes clusters (this can be a single cluster, too), which are used to host Kubernetes control planes for new Kubernetes clusters. The new Kubernetes control planes reside in dedicated namespaces of the initial clusters ("Kubernetes in Kubernetes" or "underlay / overlay Kubernetes"). Where initial clusters come from is the subject of a larger debate, with suggestions made in a later section of this document. Gardener's architecture is designed for multi-tenant environments, with a strong distinction between the operator and the end users. In Gardener, Kubernetes control planes for different tenants may reside in the same operator cluster. This approach makes it very suitable for being used with bare metal because it allows taking full advantage of the server resources. Another implication is that end users do not have access to their control plane components, such as the kube-apiserver or the ETCD. These are managed by the operator and in case of metal-stack even physically divided from the end user's workload. Gardener allocates machines from a cloud provider and automatically deploys a kubelet to those nodes, which then joins the appropriate control plane. Operators can also nest clusters so that newly provisioned clusters can be used to spin up more clusters, leading to nearly infinite scalability (also known as "kubeception" model). ## Terminology We would like to explain the most important Gardener terms. The terminology used in the Gardener project has many similarities to the architecture of Kubernetes. Additional information can also be found in the [official glossary](https://github.com/gardener/documentation/blob/master/website/documentation/glossary/_index.md). ### Garden Cluster The Garden Cluster is a Kubernetes cluster that runs the Gardener Control Plane. The control plane components introduce dedicated Kubernetes API resources for provisioning new Kubernetes clusters with the Gardener. It also takes care of the validation for many of those Gardener API resources and also reconciling some of them. The components are the following: - Gardener API Server - Gardener Controller Manager - Gardener Scheduler - Gardener Admission Controller The control plane components can be deployed in the Garden Cluster through the Gardener Operator. The Garden cluster can also be used as [seed](#seeds-and-soils) cluster. ### Virtual Garden A recommended way to deploy the Gardener is running a "virtual cluster" inside the Garden cluster. It is basically a Kubernetes control plane without any worker nodes, providing the Kubernetes API in an own ETCD. Its purpose is to store all Gardener resources (such that they reside inside a dedicated ETCD) and provide an individual update lifecycle from the Garden Cluster. End users can have access to own project namespaces in the virtual garden, too. The virtual garden consists of the following components: - garden kube-apiserver - etcd - kube-controller-manager More details about the virtual garden can be found in the description of [`gardener-operator`](https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md). ### Seeds and Soils A seed cluster is a cluster in which an agent component called the `Gardenlet` is running. The gardenlet is connected to the Gardener Control Plane and is responsible for orchestrating the provisioning of new clusters inside the seed cluster. The control plane components for the new clusters run as pods in the seed cluster. A seed cluster can also be called a soil if the Gardenlet has been manually deployed by the operator and not by the Gardener. Clusters created on the soil can be turned into seed clusters by the operator using a Gardener resource called `ManagedSeed`. This resource causes Gardener to automatically deploy the Gardenlet to the new cluster, such that the resulting cluster is not called a soil. ### Shoot Every Kubernetes cluster that is fully provisioned and managed by Gardener is called a `Shoot` cluster. It consists of the shoot control plane running on the seed cluster and worker nodes running the actual workload. ## Gardener Integration Components During the provisioning flow of a cluster, Gardener emits resources that are expected to be reconciled by controllers of a cloud provider. This section briefly describes the controllers implemented by metal-stack to allow the creation of a Kubernetes cluster on metal-stack infrastructure. If you want to learn how to deploy metal-stack with Gardener, please check out the corresponding [deployment-guide section](../../04-For%20Operators/03-deployment-guide.mdx#gardener-with-metal-stack). ### gardener-extension-provider-metal The [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) contains of a set of webhooks and controllers for reconciling cloud provider specific resources of `type: Metal`, which created by Gardener during the cluster provisioning flow. Primarily, its purpose is to reconcile `Infrastructure`, `ControlPlane`, and `Worker` resources. The project also introduces an own API (`ProviderConfiguration` resources) and consists of an admission-controller to validate them. This admission controller should be deployed in the Gardener control plane cluster. ### os-metal-extension Due to the reason metal-stack initially used ignition to provision operating system images (today, cloud-init is supported as well) there is an implementation of a controller that translates the generic `OperatingSystemConfig` format of Gardener into ignition userdata. It can be found on Github in the [os-metal-extension](https://github.com/metal-stack/os-metal-extension) repository. ### machine-controller-manager-provider-metal Worker nodes are managed through Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) repository. ## Initial Cluster Setup Before creating the `garden cluster`, a base K8s cluster needs to be in place. Some suggestions for the initial K8s cluster are: - GCP/GKE - metalstack.cloud ### Initial Cluster on GCP - A GCP account needs to be in place. - The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authenticating against GCP. - The Ansible [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) can be used for creating a GKE cluster. Suggestions for default values are: - `gcp_machine_type`: e2-standard-8 - `gcp_autoscaling_min_nodes`: 1 - `gcp_autoscaling_max_nodes`: 3 ### Initial Cluster on metalstack.cloud - A Kubernetes cluster can be created on [metalstack.cloud](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster) via UI, CLI or Terraform. ## metal-stack Setup > **Attention:** Bootstrapping a metal-stack partition is out of scope and need to be done before focusing on the relationship between metal-stack and Gardener. This guide assumes a metal-stack partition (servers, switches, network, ...) is already in place. Start by deploying: - `ingress-nginx-controller` - `cert-manager` This guide assumes, that metal-stack gets deployed on the same initial cluster as Gardener. On the initial cluster, the metal-stack control plane need to be deployed. This can be done as described in the metal-stack [documentation](https://docs.metal-stack.io/stable/installation/deployment/#Metal-Control-Plane-Deployment). ### Garden Cluster Setup After setting up the initial K8s cluster and metal-stack, Gardener can be deployed with the [Gardener Ansible role](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener). This deploys the following components: - virtual garden - Gardener control plane components - soil cluster - managed seed cluster (into the metal-stack partition) In summary, this results in the following: - `Garden cluster` created in the initial cluster - `soil cluster` created in the initial cluster. This will be the `initial seed` used for spinning up `shooted seeds` in the metal-stack partition - `shooted seed` inside the metal-stack partition, where all `shoots` are derived from --- ## Cluster API [Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project that aims to simplify the management of Kubernetes clusters. It provides a declarative way to create, configure, and manage clusters using Kubernetes-style APIs. We provide the [Cluster API provider for metal-stack (CAPMS)](https://github.com/metal-stack/cluster-api-provider-metal-stack) infrastructure provider that allows the declaration of Kubernetes clusters. :::warning "Early Development Stage" This project is currently under heavy development and is not advised to be used in production any time soon. Please use our stack on top of [Gardener](./01-gardener.md) for production workloads. ::: See the [cluster-api-provider-metal-stack documentation](../../08-References/Kubernetes/cluster-api-provider-metal-stack/cluster-api-provider-metal-stack.md) for more in-depth information. --- ## Cloud Controller Manager # metal Cloud Controller Manager CCM stands for [cloud-controller-manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and is the bridge between Kubernetes and a cloud-provider. We implemented the [cloud provider interface](https://github.com/kubernetes/cloud-provider/blob/master/cloud.go) in the [metal-ccm](https://github.com/metal-stack/metal-ccm) repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties. --- ## Firewall Controller Manager To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our [firewall-controller](https://github.com/metal-stack/firewall-controller) to the firewall image. The controller watches special CRDs, enabling users to manage: - nftables rules - Intrusion-detection with [suricata](https://suricata.io/) - network metric collection Please check out the [guide](../../08-References/Kubernetes/firewall-controller-manager/firewall-controller-manager.md) on how to use it. --- ## Isolated Clusters # Isolated Kubernetes Clusters Some customers have the need to run their workloads in a very restricted environment. These restrictions are driven by regulatory requirements in some industries such as finance, healthcare, energy and more. Regulatory requirements often mandate that the workload must not be exposed to the public internet, nor is capable to reach the public internet in any case. For this purpose we implemented a possibility to start Kubernetes clusters in such a manner. This is referred to as cluster isolation. ## Design Choices When talking about highly secure Kubernetes environments people often raise the term "Air Gapped Cluster". This would mean that no physical connection exists between the Kubernetes control plane and the Kubernetes worker nodes with the outside world. This requirement exists in extreme environments such as ships, moon bases or nuclear plants. The effort to produce this in a completely automated manner is extremely challenging. We decided to follow a different approach which is more practical, still very secure but much simpler to implement and operate. The solution we created is called "Isolated Cluster" which means that there are still physical connections between the Kubernetes cluster, but guarded to prohibit malicious traffic. It is also not possible to enable malicious traffic by accident, e.g. if a cluster user configures network policies or load balancers to untrusted environments. ## Network Design In order to be able to restrict ingress and egress internet traffic, but still make it possible to create a working Kubernetes cluster we implemented the following network design. - All strictly required container images are mirrored to a registry which is only accessible from the Kubernetes clusters. - DNS and NTP servers are produced alongside the registry. - The `containerd` configuration on every worker node is configured to pull all of the strictly required container images from this private registry mirror. - DNS and NTP configuration is also adopted to use the DNS and NTP servers on this private environment. - A list of networks which are allowed to reach is managed, this list reflects the networks of the cloud provider and is not modifiable by the cluster user. This list usually contains the internet prefixes of the provider and one or more RFC address ranges. ![Network Design](isolated-kubernetes.svg) Users are advised to attach an additional network to the Kubernetes cluster in order to be able to pull container images for the application workloads from private registries. ### Strictly Required Container Images In general the creation of a Kubernetes cluster requires the ability to pull container images for several applications which are necessary to make a machine a Kubernetes worker node. To mention the most important: - Kubelet: the main controller on each worker node to manage the workload - CNI (Container Network Interface): controller and daemon set to setup and run the container networking - CSI (Container Storage Interface): controller and daemon set to setup and run the container storage - CoreDNS: DNS for containers - MetalLB: Service Type LoadBalancer Implementation - node-exporter and metrics-server: Monitoring for the worker node - Metal-Stack Addons: for firewall and auditing events ## Flavors With the introduction of Isolated Kubernetes Clusters, cluster users must decide upon cluster creation which type of isolation he needs for his workload. There are three different flavours available: - Internet access `baseline`: This is the default cluster creation mode, which does not change any aspects of network and registry access. - Internet access `forbidden`: No internet access is possible, neither ingress nor egress. - Internet access `restricted`: No internet access is possible, neither ingress nor egress, but can be enabled by the cluster user. Please see the detailed description of these flavors below. ### Cluster Wide Network Policies CWNP To restrict which egress traffic is allowed, Custom Resources `ClusterWideNetworkPolicy` are deployed and can be deployed by the cluster user. The set of deployed CWNPs differs between `baseline` and `forbidden`/`restricted`. `baseline` CWNPs: | Rule Name | Destination | Purpose | |:-------------------|:-----------------------------------------------------|:---------------------------------------------------------------------------------------------| | allow-to-http | 0.0.0.0/0 | egress via http | | allow-to-https | 0.0.0.0/0 | egress via https | | allow-to-apiserver | IP of the Kubernetes API Server on the control plane | API Server communication of kubelet and other controllers | | allow-to-dns | IP of the Google DNS Servers | DNS resolution from the Kubernetes worker nodes and containers | | allow-to-ntp | IP of the Cloudflare NTP Servers | Time synchronization | | allow-to-storage | network of the container storage | persistent volumes with the cni driver | | allow-to-vpn | IP of the vpn endpoint on the control plane | allow communication from the api server to the kubelet for container logs and container exec | `forbidden` and `restricted` CWNPs: | Rule Name | Destination | Purpose | |:-------------------|:-----------------------------------------------------|:---------------------------------------------------------------------------------------------| | allow-to-apiserver | IP of the Kubernetes API Server on the control plane | API Server communication of kubelet and other controllers | | allow-to-dns | IP of the private DNS Server | DNS resolution from the Kubernetes worker nodes and containers | | allow-to-ntp | IP of the private NTP Server | Time synchronization | | allow-to-registry | IP of the private Registry Mirror | Pulling strictly required container images | | allow-to-storage | network of the container storage | persistent volumes with the cni driver | | allow-to-vpn | IP of the vpn endpoint on the control plane | allow communication from the api server to the kubelet for container logs and container exec | All of these CWNPs are managed by the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal), every manual modification will be reverted immediately. ### Internet Access Baseline This is the default configuration of a Kubernetes cluster, egress traffic is controlled by multiple CWNPs (`ClusterWideNetworkPolicy`), ingress traffic is possible by deploying a Service Type LoadBalancer. The cluster user can add additional CWNPs without any restrictions and is responsible for them. Container images can be pulled from any reachable container registry. The `containerd` is not reconfigured to point to our private registry mirror. DNS and NTP are configured to internet DNS resolvers and NTP servers. ### Internet Access Forbidden This configuration can only be achieved by creating a new Kubernetes cluster, it is not possible to modify a existing cluster (with internet access `baseline` or `restricted`) to this configuration. It is also required to specify the most recent version of Kubernetes, older versions of Kubernetes are not supported. Every network access modification triggered by a cluster user, either by adding or modifying CWNPs or adding a Service Type LoadBalancer, is validated against the list of allowed networks. `containerd` is configured so that all required images are pulled from the private registry mirror. This registry contains only the strictly required images, therefore no additional (workload) images can be pulled from public registries. #### Egress traffic Egress traffic is only allowed to the private registry mirror and the DNS and NTP servers. Additional CWNPs can be added to reach destinations in the internal networks if specified. If a CWNP was created which points to a destination outside of the allowed networks, the CWNP will still be present but stays in the status `ignored`. ```bash > kubectl get clusterwidenetworkpolicies.metal-stack.io NAME STATUS MESSAGE allow-to-apiserver deployed allow-to-dns deployed allow-to-ntp deployed allow-to-registry deployed allow-to-storage deployed allow-to-vpn deployed allow-to-google ignored ingress/egress does not match allowed networks ``` Also an event is created which describes why the CWNP was ignored: ```bash > kubectl get events 5s Warning ForbiddenCIDR clusterwidenetworkpolicy/allow-to-google address:"8.8.8.8/32" is outside of the allowed network range:"10.0.0.0/8,100.64.0.0/10,212.34.83.0/27", ignoring ``` #### Ingress traffic Ingress traffic is only allowed from the internal networks if specified. To specify the address where the Service Type LoadBalancer is listening to, the cluster user must use one of his statically acquired ip addresses. Of course, this ip address is only considered if it is contained in the list of allowed networks. Then this ip address must be configured in the service: ```yaml apiVersion: v1 kind: Service spec: type: LoadBalancer loadBalancerIP: 10.1.1.1 # ip from the internal network ``` By default, no ip address will be automatically selected for such clusters and the ip of the service will stay in pending mode until the ip was specified as shown above. ```bash > kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-service LoadBalancer 10.244.75.171 443:32179/TCP 4s > kubectl get events 8s Warning AllocationFailed service/example-service Failed to allocate IP for "default/example-service": no available IPs 3s Warning SyncLoadBalancerFailed service/example-service Error syncing load balancer: failed to ensure load balancer: no default network for ip acquisition specified, acquire an ip for your cluster's project and specify it directly in "spec.loadBalancerIP" ``` ### Internet Access Restricted This configuration can only be achieved by creating a new Kubernetes cluster, it is not possible to modify a existing cluster (with internet access `baseline` or `forbidden`) to this configuration. It is also required to specify the most recent version of Kubernetes, older versions of Kubernetes are not supported. The same default CWNPs are deployed and the container images are pulled from the private registry. Also DNS and NTP are configured to use the private DNS and NTP servers. The only difference to the `forbidden` mode is that CWNPs and Service Type LoadBalancers can be created without the restriction that only allowed networks are allowed. Pulling container images is theoretically possible if a cluster user creates a CWNP which allows network access to an external registry. But most container registries serve the container images from large CDN networks, which have a lot of ip addresses. Simply adding the ip address of docker.io is therefore not sufficient. ## Application Container Images In order to deploy application containers into a cluster with Internet Access `forbidden` a private registry must be provided. This private registry must be located in the list of allowed networks. The DNS name of the registry must resolve in the public DNS servers. The registry must be secured with a TLS certificate that is also valid with the CA certificates from the worker node, e.g. vanilla debian ca-certificates. ## Implementation To achieve this functionality modifications have been implemented in various components in metal-stack, this includes: ### Gardener Extension Provider Metal The ControlPlane API is adopted to enable a user to configure a shoot with the internet access type `forbidden` or `restricted`. The CloudProfile can now be extended to carry the list of allowed networks, the dns and ntp servers, the registry with the mirrored registries. ControlPlane: ```go // ControlPlaneConfig contains configuration settings for the control plane. type ControlPlaneConfig struct { metav1.TypeMeta // NetworkAccessType defines how the cluster can reach external networks. // +optional NetworkAccessType *NetworkAccessType } type ( // NetworkAccessType defines how a cluster is capable of accessing external networks NetworkAccessType string ) const ( // NetworkAccessBaseline allows the cluster to access external networks in a baseline manner NetworkAccessBaseline = NetworkAccessType("baseline") // NetworkAccessRestricted access to external networks is by default restricted to registries, dns and ntp to partition only destinations. // Therefore registries, dns and ntp destinations must be specified in the cloud-profile accordingly. // If this is not the case, restricting the access must not be possible. // Image overrides for all images which are required to create such a shoot, must be specified. No other images are provided in the given registry. // customers can define own rules to access external networks as in the baseline. // Service type LoadBalancers are also not restricted. NetworkAccessRestricted = NetworkAccessType("restricted") // NetworkAccessForbidden in this configuration a customer can no longer create rules to access external networks. // which are outside of a given list of allowed networks. This is enforced by the firewall. // Service type LoadBalancers are also not possible to open a service ip which is not in the list of allowed networks. // This is also enforced by the firewall. NetworkAccessForbidden = NetworkAccessType("forbidden") ) ``` A sample Shoot Spec: ```yaml --- apiVersion: core.gardener.cloud/v1beta1 kind: Shoot metadata: name: isolated namespace: sample spec: provider: type: metal controlPlaneConfig: networkAccessType: forbidden ... ``` CloudProfile: ```go type NetworkIsolation struct { // AllowedNetworks is a list of networks which are allowed to connect in restricted or forbidden NetworkIsolated clusters. AllowedNetworks AllowedNetworks // DNSServers DNSServers []string // NTPServers NTPServers []string // The registry which serves the images required to create a shoot. RegistryMirrors []RegistryMirror } // AllowedNetworks is a list of networks which are allowed to connect in restricted or forbidden NetworkIsolated clusters. type AllowedNetworks struct { // Ingress defines a list of networks which are allowed for incoming traffic like service type LoadBalancer // to allow all you must specify 0.0.0.0/0 or ::/0 Ingress []string // Egress defines a list of networks which are allowed for outgoing traffic // to allow all you must specify 0.0.0.0/0 or ::/0 Egress []string } type RegistryMirror struct { // Name describes this server Name string // Endpoint is typically the url of the registry in the form https://hostname Endpoint string // IP is the ipv4 or ipv6 address of this server IP string // Port at which port the service is reachable Port int32 // This Registry Mirror mirrors the following registries MirrorOf []string } ``` A sample configuration in the CloudProfile would look like: ```yaml network-isolation: allowedNetworks: egress: - 1.2.3.0/24 # Internet CIDR of the Provider - 100.64.0.0/10 - 10.0.0.0/8 ingress: - 100.64.0.0/10 dnsServers: - "1.2.3.1" - "1.2.3.2" - "1.2.3.3" ntpServers: - "1.2.3.1" - "1.2.3.2" - "1.2.3.3" registryMirrors: - name: test registry endpoint: https://some.private.registry ip: "1.2.3.4" port: 443 mirrorOf: - "docker.io" - "quay.io" - "eu.gcr.io" - "ghcr.io" - "registry.k8s.io" ``` The GEPM generates machine classes for the MCM that contain the NTP and DNS configuration for the machine. The machine-controller-manager-provider-metal implements machine creation containing these properties through the metal-api. ### OS Metal Extension Based on the configuration of a cluster the configuration of the containerd must be changed to pull images from the private registry mirror. If a cluster is either configured with `restricted` or `forbidden` and for every registry mirror an additional `certs.d/$HOST/hosts.yaml` will be created. This is in line with [Gardener's containerd Registry Configuration](https://gardener.cloud/docs/gardener/advanced/containerd-registry-configuration/). ```toml # certs.d/docker.io/hosts.yaml server = "https://docker.io" [host."https://some.private.registry"] capabilities = ["pull", "resolve"] ``` ### Firewall Controller Manager and Firewall Controller The Firewall Controller Manager has extended the FirewallSpec to configure the Firewall Controller which must enforce the restrictions regarding allowed networks. ```go // FirewallSpec defines parameters for the firewall creation along with configuration for the firewall-controller. type FirewallSpec struct { // AllowedNetworks defines which networks are allowed to connect to, and allow incoming traffic from. // Is enforced with NetworkAccessForbidden. // The node network is always allowed. AllowedNetworks AllowedNetworks `json:"allowedNetworks,omitempty"` } // AllowedNetworks is a list of networks which are allowed to connect when NetworkAccessType is NetworkAccessForbidden. type AllowedNetworks struct { // Ingress defines a list of cidrs which are allowed for incoming traffic like service type LoadBalancer Ingress []string `json:"ingress,omitempty"` // Egress defines a list of cidrs which are allowed for outgoing traffic Egress []string `json:"egress,omitempty"` } ``` Also the ClusterwideNetworkPolicy in the Firewall Controller was changed to show the deployment status of a CWNP. ```go type ClusterwideNetworkPolicy struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec PolicySpec `json:"spec,omitempty"` Status PolicyStatus `json:"status,omitempty"` } // PolicyDeploymentState describes the state of a CWNP deployment type PolicyDeploymentState string const ( // PolicyDeploymentStateDeployed the CWNP was deployed to a native nftable rule PolicyDeploymentStateDeployed = PolicyDeploymentState("deployed") // PolicyDeploymentStateIgnored the CWNP was not deployed to a native nftable rule because it is outside of the allowed networks PolicyDeploymentStateIgnored = PolicyDeploymentState("ignored") ) // PolicyStatus defines the observed state for CWNP resource type PolicyStatus struct { // FQDNState stores mapping from FQDN rules to nftables sets used for a firewall rule. // Key is either MatchName or MatchPattern // +optional FQDNState FQDNState `json:"fqdn_state,omitempty"` // State of the CWNP, can be either deployed or ignored State PolicyDeploymentState `json:"state"` // Message describe why the state changed Message string `json:"message,omitempty"` } ``` ### Cloud Controller Manager This component was adopted to allow to be started without a default network specified. This was actually always the internet network and if no ip address was specified in the Service Type LoadBalancer, one ip was allocated from this default network. For isolated clusters this is not provided and a cluster user must always specify this ip to get a working load balancer. ### OCI Mirror The [OCI Mirror](https://github.com/metal-stack/oci-mirror) is a new application which acts as a scheduled job that pulls a given list of container images and pushes them to a private registry (which will then serve as the private registry mirror). The detailed description can be read on the project website. ## Related Pull Requests - [Gardener Extension Provider](https://github.com/metal-stack/gardener-extension-provider-metal/pull/361) - [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager/pull/48) - [Firewall Controller](https://github.com/metal-stack/firewall-controller/pull/172) - [OS Metal Extension](https://github.com/metal-stack/os-metal-extension/pull/38) - [Metal Cloud Controller Manager](https://github.com/metal-stack/metal-ccm/pull/87) - [Metal Networker](https://github.com/metal-stack/metal-networker/pull/101) - [Metal Images](https://github.com/metal-stack/metal-images/pull/214) - [OCI Mirror](https://github.com/metal-stack/oci-mirror) --- ## GPU Workers For workloads which require the assistance of GPUs, support for GPUs in bare metal servers was added to metal-stack.io v0.18.0. ## GPU Operator installation With the nvidia image a worker has basic GPU support. This means that the required kernel driver, the containerd shim and the required containerd configuration are already installed and configured. To enable `Pods` that require GPU support to be scheduled on a worker node with a GPU, a `gpu-operator' must be installed. This has to be done by the cluster owner after the cluster is up and running. The simplest way to install this operator is as follows: ```bash helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update kubectl create ns gpu-operator kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged helm install --wait \ --generate-name \ --namespace gpu-operator \ --create-namespace \ nvidia/gpu-operator \ --set driver.enabled=false \ --set toolkit.enabled=true ``` After that `kubectl describe node` must show the gpu in the capacity like so: ```plain ... Capacity: cpu: 64 ephemeral-storage: 100205640Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 263802860Ki nvidia.com/gpu: 1 pods: 510 ... ``` With this basic installation, the worker node is ready to process GPU workloads. :::warning However, there is a caveat - only one 'Pod' can access the GPU. If this is all you need, no additional configuration is required. On the other hand, if you are planning to deploy multiple applications that require GPU support, and there are not that many GPUs available, you will need to configure the `gpu-operator` to allow the GPU to be shared between multiple `Pods`. ::: There are several approaches to sharing GPUs, please consult the official Nvidia documentation for further reference. - [https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes](https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes) - [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) - [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) With this, happy AI processing. --- ## Storage When working with bare-metal servers, providing cloud storage is a challenge. With physical machines there is no opportunity that a hypervisor can mount storage devices into the servers and thus, we have to implement other mechanisms that are capable of dynamically mounting storage onto the machines. In the meantime, we have started to integrate third-party solutions into our metal-stack landscape. They help us to provide modern, well-integrated and scalable storage solutions to our end-users. ## Lightbits Labs NVMe over TCP Storage Integration [Lightbits Labs](https://www.lightbitslabs.com/nvme-over-tcp/) offers a proprietary implementation of persistent storage using NVMe over TCP. The solution has some very superior traits that fit very well to metal-stack. The strongest advantages are: - High performance - Built-in multi-tenant capabilities - Configurable compression and replication factors We are maintaining an open source integration for running LightOS in our [Gardener](./01-gardener.md) cluster provisioning. You can enable it through the controller registration of the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). With the integration in place, the extension-provider deploys a [duros-controller](https://github.com/metal-stack/duros-controller) along with a Duros Storage CRD into the seed's shoot namespace. The duros-controller takes care of creating projects and managing credentials at the Lightbits Duros API. It also provides storage classes as configured in the extension-provider's controller registration to the customer's shoot cluster such that users can start consuming the Lightbits storage immediately. ## Simple Node Local Storage with csi-driver-lvm If you wish to quickly start off with cluster provisioning without caring so much about complex cloud storage solutions, we recommend using a small storage driver we wrote called [csi-driver-lvm](https://github.com/metal-stack/csi-driver-lvm). It provides a storage class that manages node-local storage through [LVM](). A definition of a PVC can look like this: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Mi storageClassName: csi-driver-lvm-linear ``` The solution does not provide cloud-storage or whatsoever, but it improves the user's accessibility of local storage on bare-metal machines through Kubernetes. Check out the driver's documentation [here](../../08-References/Storage/csi-driver-lvm/csi-driver-lvm.md). --- ## Security Principles metal-stack adheres to several security principles to ensure the integrity, confidentiality and availability of its services and data. These principles guide the design and implementation of security measures across the metal-stack architecture. ## Minimal Need to Know The minimal need to know principle is a security concept that restricts access to information and resources only to those who absolutely need it for their specific role or task. This principle is implemented throughout the metal-stack architecture and operational practices to enhance security and reduce the risk of unauthorized access or data breaches. ### RBAC :::info As of now metal-stack does not implement fine-grained Role-Based Access Control (RBAC) within the `metal-api` but this is worked on in [MEP-4](/community/MEP-14-independence-from-external-sources). ::: As described in our [User Management](../../05-Concepts/02-user-management.md) concept the [metal-api](https://github.com/metal-stack/metal-api) currently offers three different user roles for authorization: - `Admin` - `Edit` - `View` To ensure that internal components interact securely with the metal-api, metal-stack assigns specific roles to each service based on the principle of least privilege. | Component | Role | | --------------------------------------------------------------------------------------------------------------------- | ----- | | [metal-image-cache-sync](https://github.com/metal-stack/metal-image-cache-sync) | View | | [machine-controller-manager-provider-metal](https://github.com/metal-stack/machine-controller-manager-provider-metal) | Edit | | [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal) | Edit | | [metal-bmc](https://github.com/metal-stack/metal-bmc) | Edit | | [metal-core](https://github.com/metal-stack/metal-core) | Edit | | [metal-hammer](https://github.com/metal-stack/metal-hammer/) | View | | [metal-metrics-exporter](https://github.com/metal-stack/metal-metrics-exporter) | View | | [metal-ccm](https://github.com/metal-stack/metal-ccm) | Admin | | [pixiecore](https://github.com/metal-stack/pixie) | View | | [metal-console](https://github.com/metal-stack/metal-console) | Admin | | [cluster-api-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack) | Edit | | [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager) | Edit | Users can interact with the metal-api using [metalctl](https://github.com/metal-stack/metalctl), the command-line interface provided by metal-stack. Depending on the required operations, users should authenticate with the appropriate role to match their level of access. ## Defense in Depth Defense in depth is a security strategy that employs multiple layers of defense to protect systems and data. By implementing various security measures at different levels, metal-stack aims to mitigate risks and enhance overall security posture. ## Redundancy Redundancy is a key principle in metal-stack's security architecture. It involves duplicating critical components and services to ensure that if one fails, others can take over, maintaining system availability and reliability. This is particularly important for data storage and processing, where redundancy helps prevent data loss and ensures continuous operation. ## BMC User Management For bare metal provisioning with metal-stack, two dedicated users to interact with a machine BMC are created. The `metal-hammer` first creates a BMC user called `root` or `superuser` with the administrator privilege. The password used, is configured with the Ansible variable [`metal_api_bmc_superuser_pwd`](https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/metal/README.md#metal-api). It is necessary e.g. for [`metal-bmc`](https://github.com/metal-stack/metal-roles/blob/master/partition/roles/metal-bmc/README.md), to perform its actions while deleting a machine and adding it to the pool of available machines again. Since metal-stack operates within a Kubernetes cluster, the `metal_api_bmc_superuser_pwd` is stored as a Kubernetes Secret in the metal-control-plane namespace, with the value represented in base64 encoding. :::info Note: The superuser feature is optional. If no superuser password is configured, it is disabled. In this case, `metal-bmc` cannot report machine data for unallocated machines. ::: Afterwards a user called `metal` with administrator privileges is created by `metal-hammer`. When a machine registers with the `metal-api`, a password for the metal user is automatically generated and not set through an Ansible variable in `metal-roles`. This password is added to the machine’s IPMI details, which are then persisted in the `metal-db`. The register event triggers the `metal-api` to store all required machine information. Each time a machine is allocated, the password is reset and updated in the database, while the user account itself remains unchanged. The applied password constraints are as follows: - Password length: 10 characters - Number of digits: 3 - Number of special characters: 0 - Uppercase allowed - Repeated characters allowed :::info Note: The `metal-db` is not encrypted. Access to the metal-control-plane namespace should therefore be carefully restricted to trusted administrators only. ::: The `metal` user is solely intended for SOL (Serial over LAN) out-of-band administrative access to the machine via [metalctl](/docs/references/metalctl). To establish this connection, the [`metal-console`](docs/references/metal-console) component is used, which transfers console output over SSH. This setup ensures secure, remote out-of-band management, allowing operators to troubleshoot and control machines even when the operating system is unavailable. To maintain security, the BMC credentials should be treated as system-managed accounts. For security and compliance, administrators are strongly advised to avoid interactive logins with them. --- ## SBOM Every container image and binary that's part of metal-stack contains an _SBOM_ (Software Bill of Materials). It provides a detailed inventory of components within container images and binaries, enabling you to manage vulnerabilities and compliance effectively. We decided to use [_SPDX_ (Software Package Data Exchange)](https://spdx.dev/), as it is among the most widely adopted standards and is natively supported in Docker. Docker utilizes the [in-toto SPDX format](https://github.com/in-toto/attestation/blob/main/spec/predicates/spdx.md), while binary-*SBOM*s are created using [Syft](https://github.com/anchore/syft). *SBOM*s are created as part of each repository's _GitHub Actions_ workflow utilizing [Anchore SBOM Action](https://github.com/marketplace/actions/anchore-sbom-action) for binaries and [Build and push Docker images](https://github.com/marketplace/actions/build-and-push-docker-images) for container images. ## Download _SBOM_ of a container image ```bash docker buildx imagetools inspect ghcr.io/metal-stack/: --format "{{ json .SBOM.SPDX }}" > sbom.json ``` For further info, refer to the [Docker docs](https://docs.docker.com/build/metadata/attestations/sbom/#inspecting-sboms). ## Download _SBOM_ of a binary from the GitHub release ```bash wget https://github.com/metal-stack//releases/latest/download/sbom.json ``` Please note, if more than one binary is released, e.g. for different platforms / architectures, you are required to include this info in the _SBOM_ file name as well. ```bash # This is an example using https://github.com/metal-stack/metalctl wget https://github.com/metal-stack/metalctl/releases/latest/download/sbom-darwin-arm64.json ``` ## Identify CVEs There are many tools that can help you to identify the CVEs with the help of an SBOM. Just to name one example, [grype](https://github.com/anchore/grype) can be used to do this, which would look like this: ```plain $ grype sbom-darwin-arm64.json ✔ Scanned for vulnerabilities [14 vulnerability matches] ├── by severity: 0 critical, 5 high, 9 medium, 0 low, 0 negligible NAME INSTALLED FIXED IN TYPE VULNERABILITY SEVERITY EPSS RISK stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-61723 High < 0.1% (23rd) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-61725 High < 0.1% (23rd) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-58186 Medium < 0.1% (17th) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-61724 Medium < 0.1% (17th) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-47912 Medium < 0.1% (16th) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-58188 High < 0.1% (8th) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-58189 Medium < 0.1% (12th) < 0.1 github.com/gorilla/csrf v1.7.3 go-module GHSA-82ff-hg59-8x73 Medium < 0.1% (8th) < 0.1 stdlib go1.24.5 1.23.12, 1.24.6 go-module CVE-2025-47907 High < 0.1% (4th) < 0.1 stdlib go1.24.5 1.23.12, 1.24.6 go-module CVE-2025-47906 Medium < 0.1% (5th) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-58185 Medium < 0.1% (6th) < 0.1 stdlib go1.24.5 1.24.9, 1.25.3 go-module CVE-2025-58187 High < 0.1% (2nd) < 0.1 stdlib go1.24.5 1.24.8, 1.25.2 go-module CVE-2025-58183 Medium < 0.1% (2nd) < 0.1 github.com/go-viper/mapstructure/v2 v2.3.0 2.4.0 go-module GHSA-2464-8j7c-4cjm Medium N/A N/A ``` Or even simpler by passing the output of `docker buildx imagetools inspect` into grype like so: ```bash docker buildx imagetools inspect ghcr.io/metal-stack/: --format "{{ json .SBOM.SPDX }}" | grype ``` --- ## Cryptography metal-stack incorporates multiple layers of cryptographic protection and secure communication to ensure system integrity and confidentiality: ### TLS Certificate Management TLS certificates used by metal-stack components - as outlined in the [architecture section](../../05-Concepts/01-architecture.mdx) - can be generated using either RSA 4096-bit or ECDSA 256-bit keys. We recommend RSA 4096. By default, in-cluster communication is not encrypted. If encryption is required within the cluster, it must be configured manually using a service mesh (e.g., Istio or Linkerd) or a similar mechanism. For outbound traffic, we recommend integrating cert-manager in combination with Let's Encrypt to handle certificate issuance and enable automated certificate rotation for ingress domains. In offline environments where Let's Encrypt cannot be used, the certificates must be issued and managed manually or via an internal CA. ### VPN & Network Encryption metal-stack employs WireGuard-based VPN technology, orchestrated via Headscale. WireGuard leverages Elliptic Curve Cryptography (ECC) for key exchange and relies on the Noise Protocol Framework to establish secure and lightweight cryptographic handshakes. ### Authentication with JWT Access to the `metal-api` is protected using JWT (JSON Web Tokens). These tokens are generated and verified using the [`go-jose`](https://github.com/go-jose/go-jose) library, which implements JOSE standards. Supported signature algorithms include: - RSA (RS256, RS384, RS512) - RSA-PSS (PS256, PS384, PS512) - ECDSA (ES256, ES384, ES512) - EdDSA --- ## Communication Matrix This matrix describes the communication between components in the metal-stack and their respective security properties. Please note that depending on your setup and configuration, some components may not be present, may have different security properties and might communicate differently than described here. The communication processes described here correspond to the standard configuration and setup. **Legend**: - `C`: Confidentiality, cryptography, encryption. Marked with an `x` if the communication is encrypted. - `I`: Integrity of data. Marked with an `x` if the communication ensures data integrity. - `Auth`: Authentication, ensures the identity of the communicating parties. Marked with an `x` if authentication is required. - `Trust`: Only trusted networks involved. Marked with an `x` if the communication is only between trusted networks. ## Plain metal-stack While metal-stack can be used in different environments and setups, the following communication is required by metal-stack components in a standard setup. This includes all components running on the control plane, partition management and machines. :::info Description The following table might not be displayed in completeness. Scroll to the right to see all entries. ::: | No. | Component | Source Zone | Protocol | Destination | Destination Zone | Port | C | I | Auth | Trust | Purpose | Notes | | ---- | ---------------------- | -------------------- | :------: | ---------------------- | -------------------- | :---: | :-: | :-: | :--: | :---: | ------------------------------ | ------------------------------------------------ | | 1.1 | metalctl | Internet | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | | 1.2 | metalctl | Internet | HTTPS | OIDC Provider | unknown | 443 | x | x | x | | Authentication & Authorization | Optional. Needs to be configured. | | 1.3 | metalctl | Internet | HTTPS | GitHub | Internet | 443 | x | x | | | Updater | Used for updates and version checks. | | 2.1 | metal-api | Metal Control Plane | TCP | metal-db | Metal Control Plane | 28015 | | | x | x | RethinkDB | Database access. | | 2.2 | metal-api | Metal Control Plane | TCP | masterdata-api | Metal Control Plane | 8443 | | | x | x | Postgres | Database access. | | 2.3 | metal-api | Metal Control Plane | HTTP | ipam | Metal Control Plane | 9090 | | | | x | Address Management | Used to manage IP addresses. | | 2.4 | metal-api | Metal Control Plane | TLS | nsq | Metal Control Plane | 4150 | x | x | x | x | Machine Operation | Used for machine operations and notifications. | | 2.5 | metal-api | Metal Control Plane | HTTP | nsq lookupd | Metal Control Plane | 4161 | | | x | x | Machine Operation | Used for machine operations and notifications. | | 2.6 | metal-api | Metal Control Plane | TCP | auditing timescaledb | Metal Control Plane | 5432 | | | x | x | Audit Logs | Logging of auditing events. Used for compliance. | | 2.7 | metal-api | Metal Control Plane | HTTPS | headscale | Metal Control Plane | 50443 | x | x | x | x | Headscale API | Headscale is used for VPN networking. | | 2.8 | metal-api | Metal Control Plane | HTTPS | S3-compatible Storage | unknown | 443 | ? | ? | ? | ? | Firmware | Optional. Needs to be configured. | | 2.9 | metal-api | Metal Control Plane | HTTPS | OIDC Provider | unknown | 443 | ? | ? | ? | ? | Authentication & Authorization | Optional. Needs to be configured. | | 3.1 | metal-apiserver | Metal Control Plane | TCP | valkey | Metal Control Plane | 6379 | | | x | x | Background Jobs | Used for background job processing and caching. | | 3.2 | metal-apiserver | Metal Control Plane | TCP | metal-db | Metal Control Plane | 28015 | x | x | x | x | RethinkDB | Database access. | | 3.3 | metal-apiserver | Metal Control Plane | TCP | masterdata-api | Metal Control Plane | 8080 | | | x | x | Postgres | Database access. | | 3.4 | metal-apiserver | Metal Control Plane | HTTP | ipam | Metal Control Plane | 9090 | | | | x | Address Management | Used to manage IP addresses. | | 3.5 | metal-apiserver | Metal Control Plane | TCP | auditing timescaledb | Metal Control Plane | 5432 | | | x | x | Audit Logs | Logging of auditing events. Used for compliance. | | 3.6 | metal-apiserver | Metal Control Plane | HTTPS | headscale | Metal Control Plane | 50443 | x | x | x | x | Headscale API | Headscale is used for VPN networking. | | 3.7 | metal-apiserver | Metal Control Plane | HTTPS | OIDC Provider | unknown | 443 | x | x | x | ? | Authentication & Authorization | Optional. Needs to be configured. | | 4.1 | masterdata-api | Metal Control Plane | TCP | masterdata-db | Metal Control Plane | 5432 | | | x | x | Postgres database access | Database access. | | 5.1 | ipam | Metal Control Plane | TCP | ipam-db | Metal Control Plane | 5432 | | | x | x | Postgres database access | Database access. | | 6.1 | backup-restore-sidecar | Metal Control Plane | HTTPS | S3-compatible Storage | unknown | 443 | ? | ? | ? | ? | Backup & Restore | Optional. Needs to be configured. | | 6.2 | backup-restore-sidecar | Metal Control Plane | HTTPS | Google API | Internet | 443 | x | x | x | | Backup & Restore | Optional. Needs to be configured. | | 6.3 | backup-restore-sidecar | Metal Control Plane | TCP | Postgres | Metal Control Plane | 5432 | | | x | x | Backup & Restore | Optional. Needs to be configured. | | 6.4 | backup-restore-sidecar | Metal Control Plane | TCP | RethinkDB | Metal Control Plane | 28015 | | | x | x | Backup & Restore | Optional. Needs to be configured. | | 6.5 | backup-restore-sidecar | Metal Control Plane | TCP | ETCD | Metal Control Plane | 2380 | | | x | x | Backup & Restore | Optional. Needs to be configured. | | 6.6 | backup-restore-sidecar | Metal Control Plane | TCP | Redis | Metal Control Plane | 6379 | | | x | x | Backup & Restore | Optional. Needs to be configured. | | 6.7 | backup-restore-sidecar | Metal Control Plane | TCP | keydb | Metal Control Plane | 6379 | | | x | x | Backup & Restore | Optional. Needs to be configured. | | 7.1 | metal-console | Partition Management | HTTP | metal-api | Metal Control Plane | 8080 | | | x | x | API Requests | Used for management operations. | | 7.2 | metal-console | Partition Management | HTTPS | metal-bmc | Partition Management | 3333 | x | x | x | x | Machine Management | Used for management operations. | | 8.1 | ssh | unknown | TCP | metal-console | Partition Management | 10001 | x | x | x | ? | Machine Access (SSH) | Used to access the metal-console via SSH. | | 9.1 | pixiecore | Partition Management | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for management operations. | | 10.1 | metal-bmc | Partition Management | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for management operations. | | 10.2 | metal-bmc | Partition Management | TLS | nsq | Partition Management | 4150 | x | x | x | x | Machine Operation | Used for machine operations and notifications. | | 10.2 | metal-bmc | Partition Management | IPMI | machine BMC | Machine | 623 | | | x | x | Machine Operation | Used for BMC management. | | 11.1 | metal-cache-image-sync | Partition Management | HTTPS | S3-compatible Storage | unknown | 443 | ? | ? | ? | | Image Caching and Sync | Optional. Needs to be configured. | | 11.2 | metal-cache-image-sync | Partition Management | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | | 12.1 | metal-hammer | Machine | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for management operations. | | 12.2 | metal-hammer | Machine | HTTPS | pixiecore | Partition Management | 443 | x | x | | x | Machine Management | Used for machine management. | | 12.3 | metal-hammer | Machine | HTTPS | Prometheus | unknown | 443 | x | x | x | x | Monitoring | Actively pushes metrics to Prometheus. | | 12.4 | metal-hammer | Machine | HTTP | HAProxy | Metal Control Plane | 9001 | | x | | x | Image Caching and Pulling | Used to pull images via HAProxy. | | 12.5 | metal-hammer | Machine | HTTPS | Container Registry | internet | 443 | x | x | ? | | Image and Pulling | Used to pull images from the registry. | | 13.1 | machine firmware | Machine | HTTPS | pixiecore | Partition Management | 443 | x | x | | x | Machine Management | Used to provision machines with iPXE. | | 13.2 | machine firmware | Machine | TFTP | pixiecore | Partition Management | 69 | | | | x | Machine OS Provisioning | Used to provision machine firmware. | | 14.1 | machine OS | Machine | DHCP | DHCP Server | Machine | 67/68 | | | | x | Machine OS Provisioning | Used to obtain an IP address. | | 14.2 | machine OS | Machine | DNS | DNS Server | Machine | 53 | | | | x | Machine OS Resolution | Used to resolve hostnames. | | 14.3 | machine OS | Machine | NTP | NTP Server | Machine | 123 | | | | x | Machine OS Time Sync | Used to synchronize time with the NTP server. | | 15.1 | metal-metrics-exporter | Metal Control Plane | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | Monitoring | Scrapes metrics from metal-api. | | 16.1 | prometheus | Metal Control Plane | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | Monitoring | Scrapes metrics from metal-api. | | 16.2 | prometheus | Metal Control Plane | HTTPS | metal-metrics-exporter | Metal Control Plane | 9080 | | | | x | Monitoring | Scrapes metrics from metal-metrics-exporter. | | 16.3 | prometheus | Metal Control Plane | HTTPS | metal-apiserver | Metal Control Plane | 443 | x | x | x | x | Monitoring | Scrapes metrics from metal-apiserver. | | 16.4 | prometheus | Metal Control Plane | HTTPS | masterdata-api | Metal Control Plane | 2113 | x | x | x | x | Monitoring | Scrapes metrics from masterdata-api. | ### Used Technologies | Technology | Parties | Notes | | --------------------------------- | ---------- | -------------------------------------------------------------------------------- | | DHCP | All | Used for obtaining IP addresses and boot configurations. | | NTP | All | Used for synchronizing time across all components. | | iPXE | Machines | Used for network-based bootstrapping of machines. | | TFTP | Machines | Used for transferring boot files to machines. | | HTTP | Multiple | Communication in trusted networks. | | HTTPS | Multiple | Cross-network communication. | | DNS | Multiple | Used for resolving hostnames to IP addresses. | | Kubernetes | Cluster | Metal-stack components running in pods. Optional, but recommended. | | Container Network Interface (CNI) | Kubernetes | Provides networking capabilities for pods in a cluster. Required for Kubernetes. | ## With SONiC While metal-stack does not directly depend on SONiC, it is the only actively maintained implementation of our networking stack. Therefore, the following communication is required by metal-stack components to interact with SONiC. Please note that every [networking setup](../../05-Concepts/03-Network/01-theory.md) has its own requirements and configurations, so the following table might not be complete for your setup. | No. | Component | Source Zone | Protocol | Destination | Destination Zone | Port | C | I | Auth | Trust | Purpose | Notes | | ---- | ---------- | ----------------- | :------: | -------------------- | ------------------- | :---: | :-: | :-: | :--: | :---: | ------------- | --------------------------------------------- | | S1.1 | metal-core | Leaf Switches | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for management operations. | | S1.2 | metal-core | Leaf Switches | TCP | SONiC ConfigDB Redis | Switch | 6379 | | | | x | API Requests | Used for management operations. | | S2.1 | DHCP Relay | Leaf Switches | TCP/UDP | DHCP Server | Management Server | 67/68 | | | | x | DHCP Requests | Used to forward DHCP requests. | | S3.1 | ssh client | unknown | SSH | ssh daemon | Management Server | 22 | x | x | x | | SSH Access | Used to access the management server via SSH. | | S3.2 | ssh client | Management Server | SSH | ssh daemon | Switch | 22 | x | x | x | x | SSH Access | Used to access the SONiC switch via SSH. | | S4.1 | FRRouting | Firewall | BGP | FRRouting | Switches | 179 | | | | x | Routing | Used for dynamic routing. | | S4.2 | FRRouting | Machine | BGP | FRRouting | Firewall | 179 | | | | x | Routing | Used for dynamic routing. | | S4.3 | FRRouting | Switches | BGP | FRRouting | Switches | 179 | | | | x | Routing | Used for dynamic routing. | | S5.1 | tailscale | Firewall | HTTPS | Headscale | Metal Control Plane | 443 | x | x | x | x | VPN Access | Used for Wireguard VPN access via Headscale. | ### Used Technologies | Technology | Parties | Notes | | ---------- | --------------------------- | ---------------------------------------------------------------------------------------------- | | VRF | Switches, Firewalls | Isolation of network segments, e.g. for management and data traffic. | | VLAN | Switches, Firewalls | Layer 2 traffic segmentation. | | VXLAN | Switches, Firewalls | Encapsulate Layer 2 frames in Layer 3 packets for network virtualization. | | EVPN | Switches, Firewalls | Overlay network technology for scalable and flexible network architectures. | | VPN | Firewalls | Management access [without open SSH ports](/community/MEP-9-no-open-ports-to-the-data-center). | | BGP | Multiple | Routing protocol for dynamic routing and network management. | | SSH | Management Server, Switches | Secure shell access for management and configuration. | | LLDP | Switches, Machines | Link Layer Discovery Protocol for network device discovery. | | ICMP | Multiple | Used for network diagnostics and reachability testing. | ## With Gardener When using metal-stack in [conjunction with Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md), the following communication is required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries. ::: | No. | Component | Source Zone | Protocol | Destination | Destination Zone | Port | C | I | Auth | Trust | Purpose | Notes | | ---- | ----------------------------------------- | ------------ | :------: | -------------- | ------------------- | :--: | :-: | :-: | :--: | :---: | ------------ | ------------------------------------------- | | G1.1 | metal-ccm | Seed Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for management operations. | | G1.2 | metal-ccm | Seed Cluster | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | x | API Requests | Used for management operations. | | G2.1 | firewall-controller-manager | Seed Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | x | API Requests | Used for firewall management. | | G2.2 | firewall-controller-manager | Seed Cluster | HTTPS | kube-apiserver | Seed Cluster | 443 | x | x | x | x | API Requests | Used for firewall management. | | G2.3 | firewall-controller-manager | Seed Cluster | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | x | API Requests | Used for firewall management. | | G3.1 | firewall-controller | Firewall | HTTPS | kube-apiserver | Seed Cluster | 443 | x | x | x | x | API Requests | Used for firewall management. | | G3.2 | firewall-controller | Firewall | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | x | API Requests | Used for firewall management. | | G3.3 | firewall-controller | Firewall | HTTPS | Controller URL | Internet | 443 | x | x | | | Self-Update | Controller URL and version provided by FCM. | | G4.1 | machine-controller-manager-provider-metal | Seed Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | | G5.1 | gardener-extension-provider-metal | Seed Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | | G5.2 | gardener-extension-provider-metal | Seed Cluster | HTTPS | kube-apiserver | Garden Cluster | 443 | x | x | x | | API Requests | Used for management operations. | | G5.3 | gardener-extension-provider-metal | Seed Cluster | HTTPS | kube-apiserver | Seed Cluster | 443 | x | x | x | | API Requests | Used for management operations. | | G5.4 | gardener-extension-provider-metal | Seed Cluster | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | | API Requests | Used for management operations. | ### Used Technologies | Technology | Parties | Notes | | ---------- | -------------------------------- | ---------------------------------------------- | | Gardener | Contains of multiple components. | Cluster management system for many Kubernetes. | ## With Cluster API By using the [Cluster API provider for metal-stack](../../05-Concepts/04-Kubernetes/02-cluster-api.md), the following communictations are required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries. ::: | No. | Component | Source Zone | Protocol | Destination | Destination Zone | Port | C | I | Auth | Trust | Purpose | Notes | | ---- | -------------------------------- | ------------------ | :------: | -------------- | ------------------- | :--: | :-: | :-: | :--: | :---: | ------------ | ------------------------------- | | C1.1 | metal-ccm | Workload Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | | C1.2 | metal-ccm | Workload Cluster | HTTPS | kube-apiserver | Workload Cluster | 443 | x | x | x | x | API Requests | Used for management operations. | | C2.1 | cluster-api-provider-metal-stack | Management Cluster | HTTPS | metal-api | Metal Control Plane | 443 | x | x | x | | API Requests | Used for management operations. | ### Used Technologies | Technology | Parties | Notes | | ----------- | --------------------------------------------------------- | --------------------------------------------------------- | | Cluster API | Contains of multiple components and additional providers. | Cluster management system for single Kubernetes clusters. | ## With Lightbits In order to use [Lightbits as a storage solution](../../05-Concepts/04-Kubernetes/07-storage.md), the following communications are required by metal-stack components. :::info The following table might not be displayed in completeness. Scroll to the right to see all entries. ::: | No. | Component | Source Zone | Protocol | Destination | Destination Zone | Port | C | I | Auth | Trust | Purpose | Notes | | ---- | ----------------- | ------------- | :------: | -------------- | ----------------- | :--: | :-: | :-: | :--: | :---: | ---------- | ------------------------------- | | L1.1 | duros-controller | Seed Cluster | HTTPS | duros-api | Lightbits Cluster | 443 | x | x | x | x | Storage | Used for management operations. | | L1.2 | duros-controller | Seed Cluster | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | x | Kubernetes | Used for management operations. | | L2.1 | lb-csi-controller | Shoot Cluster | HTTPS | duros-api | Lightbits Cluster | 443 | x | x | x | | Storage | Used for management operations. | | L2.2 | lb-csi-controller | Shoot Cluster | HTTPS | kube-apiserver | Shoot Cluster | 443 | x | x | x | x | Kubernetes | Used for management operations. | | L3.1 | lb-csi-node | Shoot Cluster | TCP | duros-api | Lightbits Cluster | 4420 | x | x | x | | Storage | Used for management operations. | | L3.2 | lb-csi-node | Shoot Cluster | TCP | duros-api | Lightbits Cluster | 8009 | x | x | x | | Storage | Used for management operations. | ### Used Technologies | Technology | Parties | Notes | | ---------- | ------- | --------------------------- | | Lightbits | Storage | Used for storage solutions. | --- ## Artifact Signing To increase trust and integrity, metal-stack introduces artifact signing for its released components. The release vector is published as an OCI artifact and signed using [cosign](https://github.com/sigstore/cosign). The images are signed using a public key that is always attached to a metal-stack release in the [releases repository](https://github.com/metal-stack/releases/blob/master/cosign.pub). To verify an image, the following command can be used: ```bash cosign verify --key files/cosign.pub ghcr.io/metal-stack/metal-deployment-base:v0.9.2 Verification for ghcr.io/metal-stack/metal-deployment-base:v0.9.2 -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The signatures were verified against the specified public key [{"critical":{"identity":{"docker-reference":"ghcr.io/metal-stack/metal-deployment-base:v0.9.2"},"image":{"docker-manifest-digest":"sha256:8b4a19650efc27f6cd29798c94eca9f1ebbab2d20004a267d6729ad69f3c095f"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}},{"critical":{"identity":{"docker-reference":"ghcr.io/metal-stack/metal-deployment-base:v0.9.2"},"image":{"docker-manifest-digest":"sha256:8b4a19650efc27f6cd29798c94eca9f1ebbab2d20004a267d6729ad69f3c095f"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}] ``` Certain images we also sign keyless in addition, such the command can also look like this: ```bash cosign verify ghcr.io/metal-stack/metal-deployment-base:v0.9.2 --certificate-oidc-issuer https://accounts.google.com --certificate-identity keyless@metal-stack.iam.gserviceaccount.com Verification for ghcr.io/metal-stack/metal-deployment-base:v0.9.2 -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates [{"critical":{"identity":{"docker-reference":"ghcr.io/metal-stack/metal-deployment-base:v0.9.2"},"image":{"docker-manifest-digest":"sha256:8b4a19650efc27f6cd29798c94eca9f1ebbab2d20004a267d6729ad69f3c095f"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}},{"critical":{"identity":{"docker-reference":"ghcr.io/metal-stack/metal-deployment-base:v0.9.2"},"image":{"docker-manifest-digest":"sha256:8b4a19650efc27f6cd29798c94eca9f1ebbab2d20004a267d6729ad69f3c095f"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}] ``` --- ## Remote Access ## Machines and Firewalls Remote access to machines and firewalls is essential for performing administrative tasks such as incident management, troubleshooting and sometimes for development. Standard SSH access is often insufficient for these purposes. In many cases, direct serial console access is required to fully manage the system. metal-stack follows a security-first approach by not offering direct SSH access to machines. This practice reduces the attack surface and prevents unauthorized access that could lead to system damage. Detailed information can be found in [MEP-9](/community/MEP-9-no-open-ports-to-the-data-center). Administrators can access machines in two primary ways. **Out-of-band management via SOL** ```bash metalctl machine console --ipmi ``` This method leverages the machine’s BMC. For detailed user configuration, see the [BMC User Management](Security/01-principles.md#bmc-user-management) section. **Via metal-console:** ```bash metalctl machine console ``` This approach uses the [`metal-console`](../08-References/Control%20Plane/metal-console/metal-console.md), which is required to establish console access. This component acts as a bridge between SSH and the console protocol of the concrete machine. Both methods ensure secure and controlled access to machines without exposing them unnecessarily to the network, maintaining the integrity and safety of the infrastructure. Connecting directly to a machine without a clear plan of action can have unintended consequences and negatively impact stability. For this reason, administrative privileges are required. This restriction ensures that only authorized personnel with the necessary expertise can perform actions that affect the underlying infrastructure. These principles will evolve with the introduction of [MEP-4](/community/MEP-14-independence-from-external-sources). --- ## v0.20.0 # metal-stack v0.20.0 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.20.0](https://github.com/metal-stack/releases/releases/tag/v0.20.0) ## General * General support for IPv6 in the metal-api * [Gardener v1.106](https://github.com/gardener/gardener/releases/tag/v1.106.0) ## Required Actions * The `ChildPrefixLength` property was removed from the `partition` entity and moved into the `network` entity. So, every tenant super network must now be configured with `DefaultChildPrefixLength` accordingly. The existing values in the database are migrated automatically, but the deployment has to be adapted. It is required to add the new property to the tenant super network, like: ``` defaultchildprefixlength: IPv4: 22 ``` Also ensure that all firewall-controllers are up-to-date in all firewalls before adding IPv6 prefixes to any of the existing networks. (metal-stack/metal-api#549) * Another audit backend based on Postgres and TimescaleDB was added as an alternative to the existing Meilisearch backend. For this, the variables `metal_auditing_enabled`, `metal_auditing_index_prefix`, `metal_auditing_index_interval`, `metal_auditing_url` were renamed to `metal_auditing_meili_enabled`, `metal_auditing_meili_index_prefix`, `metal_auditing_meili_index_interval`, `metal_auditing_meili_url`. Please adapt your deployment accordingly. If you switch the backend, be aware that backup data of the Meilisearch backend remains and manual cleanup is required. (metal-stack/metal-roles#310) * Kubernetes 1.31 for worker nodes require to run on the latest OS metal-images. (metal-stack/os-metal-extension#54) * To prevent using incompatible metal-os images in combination with Kubernetes 1.31 worker nodes, use the new deployment parameter `gardener_os_compatibility_mapping` in order to add a Kubelet version constraint to your `CloudProfile` machine images: ```yaml gardener_os_compatibility_mapping: ubuntu: when: operator: "<" version: "24.04.20250228" except: - "24.04" kubelet: "<= 1.30" debian: when: operator: "<" version: "12.0.20250228" except: - "12.0" kubelet: "<= 1.30" ``` (metal-stack/metal-roles#384) ## Component Releases ### node-init v0.1.6 * Build with go-1.24 and update go modules (metal-stack/node-init#15) @majst01 ### metal-api v0.40.3 * Dualstack Network Support (metal-stack/metal-api#549) @majst01 * Allow additional auditing backend based on TimescaleDB (metal-stack/metal-api#565) @Gerrit91 * Fix swagger spec regarding address family is required. (metal-stack/metal-api#608) @Gerrit91 * Remove unnecessary parent network ID field from network allocation. (metal-stack/metal-api#609) @Gerrit91 ### helm-charts v0.4.20 * Roll metal-api on cert changes. (metal-stack/helm-charts#115) @Gerrit91 * Auditing for metal-api with timescaledb backend. (metal-stack/helm-charts#99) @Gerrit91 ### metal-roles v0.15.2 * Role for auditing-timescaledb. (metal-stack/metal-roles#310) @Gerrit91 * Roll nsqd and metal-api on certificate roll. (metal-stack/metal-roles#377) @Gerrit91 * Allow setting the name and kubernetes version for worker groups (metal-stack/metal-roles#381) @robertvolkmann * feat: allow configuring gardeners vali (metal-stack/metal-roles#380) @vknabel * Add possibility to add kubelet constraints to cloud profile OS images. (metal-stack/metal-roles#384) @Gerrit91 ### metalctl v0.18.0 * Only try to use token from kubeconfig for machine console. (metal-stack/metalctl#259) @Gerrit91 * Dualstack support (metal-stack/metalctl#256) @majst01 * fix: bug which required optional hmac type (metal-stack/metalctl#275) @vknabel * docs: fix readme auth type (metal-stack/metalctl#274) @vknabel ### metal-console v0.7.3 * Update to go-1.24 (metal-stack/metal-console#57) @majst01 ### os-metal-extension v0.8.5 * Do not deploy containerd config for systemd cgroup driver. (metal-stack/os-metal-extension#54) @Gerrit91 ### gardener-extension-provider-metal v0.25.7 * Revendor g/g v1.106. (metal-stack/gardener-extension-provider-metal#446) @Gerrit91 * Make extension ports configurable via helm values (metal-stack/gardener-extension-provider-metal#445) @ScheererJ ### machine-controller-manager-provider-metal v0.1.23 * Revendor MCM v0.54.0. (metal-stack/machine-controller-manager-provider-metal#38) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.19.8 (metal-stack/docs#248) @metal-robot[bot] * Bump metal-api to version v0.40.0 (metal-stack/metal-python#145) @metal-robot[bot] * Bump metal-api to version v0.40.0 (metal-stack/metal-go#200) @metal-robot[bot] * Bump metal-api to version v0.40.1 (metal-stack/metal-python#146) @metal-robot[bot] * Bump metal-api to version v0.40.1 (metal-stack/metal-go#201) @metal-robot[bot] * Bump metal-api to version v0.40.2 (metal-stack/metal-python#147) @metal-robot[bot] * Bump metal-api to version v0.40.2 (metal-stack/metal-go#202) @metal-robot[bot] * Bump metal-api to version v0.40.3 (metal-stack/metal-python#148) @metal-robot[bot] * Bump metal-api to version v0.40.3 (metal-stack/metal-go#203) @metal-robot[bot] * use resource name as cluster id (metal-stack/cluster-api-provider-metal-stack#73) @vknabel * Split development guide from contributing.md. (metal-stack/cluster-api-provider-metal-stack#74) @Gerrit91 * Add patch for accton platform support (metal-stack/sonic-build#1) @iljarotar * Update node_exporter to v1.9.0 (metal-stack/metal-images#281) @majst01 * Update to go-1.24 (metal-stack/oci-mirror#12) @majst01 * Update ubuntu/firewall kernel to 6.12.16 (metal-stack/metal-images#283) @majst01 * Update to go-1.24 (metal-stack/go-lldpd#28) @majst01 * ship with default containerd config to get cgroupv2 compatibility (metal-stack/metal-images#282) @majst01 * Do not install a config.toml override for nvidia anymore (metal-stack/metal-images#285) @majst01 * Update with swagger spec from real cluster (metal-stack/ontap-go#3) @majst01 * Add files via upload (metal-stack/docs#249) @philiphaas * Update favicon (metal-stack/docs#250) @philiphaas * Fix timescaledb query (missing parenthesis). (metal-stack/metal-lib#169) @Gerrit91 * Improve consistency to meilisearch audit indexing. (metal-stack/metal-lib#170) @Gerrit91 * Next release (metal-stack/releases#218) @metal-robot[bot] --- ## v0.20.1 # metal-stack v0.20.1 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.20.1](https://github.com/metal-stack/releases/releases/tag/v0.20.1) ## General * [Gardener v1.107](https://github.com/gardener/gardener/releases/tag/v1.107.0) ## Required Actions * Before rolling out Gardener `1.107`, raise the version of the gardener-admission-controller manually prior to the deployment as otherwise the controlplane chart cannot be applied. ```bash kubectl patch deployments.apps -n garden gardener-admission-controller --patch='{"spec":{"template":{"spec":{"containers":[{"name": "gardener-admission-controller","image":"europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.107.3"}]}}}}' ``` ## Component Releases ### metal-roles v0.15.3 * Add metal_bmc_docker_network var (metal-stack/metal-roles#386) @iljarotar * add default for ztp_provisioning_script (metal-stack/metal-roles#382) @iljarotar * Fix worker group name concat of shooted seed. (metal-stack/metal-roles#387) @Gerrit91 ### metal-api v0.40.4 * Update metal-lib. (metal-stack/metal-api#610) @Gerrit91 ### metal-core v0.12.5 * update to go-1.24 and all go modules (metal-stack/metal-core#146) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.20.0 (metal-stack/docs#251) @metal-robot[bot] * Migrate to use asynq instead of homegrown tx package (metal-stack/metal-apiserver#3) @majst01 * Reconcile upstream fixes (metal-stack/metal-apiserver#4) @majst01 * Dual-stack Support (metal-stack/docs#202) @majst01 * Bump metal-api to version v0.40.4 (metal-stack/metal-python#149) @metal-robot[bot] * Bump metal-api to version v0.40.4 (metal-stack/metal-go#204) @metal-robot[bot] * fix: wait for cluster deletion until all machines are gone #67 (metal-stack/cluster-api-provider-metal-stack#77) @vknabel * Next release (metal-stack/releases#219) @metal-robot[bot] --- ## v0.20.2 # metal-stack v0.20.2 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.20.2](https://github.com/metal-stack/releases/releases/tag/v0.20.2) ## General * [Gardener v1.108](https://github.com/gardener/gardener/releases/tag/v1.108.0) ## Component Releases ### gardener-extension-csi-driver-lvm v0.1.2 * Allow specifying a default storage class. (metal-stack/gardener-extension-csi-driver-lvm#9) @Gerrit91 * Revendor g/g v1.107. (metal-stack/gardener-extension-csi-driver-lvm#8) @Gerrit91 * Do not write termination message into host `/dev`. (metal-stack/gardener-extension-csi-driver-lvm#7) @Gerrit91 * Fail reconciliation if csi-lvm still exists. (metal-stack/gardener-extension-csi-driver-lvm#6) @Gerrit91 * Allow missing access to shoot kube-apiservers. (metal-stack/gardener-extension-csi-driver-lvm#10) @Gerrit91 ### metal-ccm v0.9.5 * Update to go-1.24 and nonroot distroless (metal-stack/metal-ccm#109) @majst01 ### gardener-extension-provider-metal v0.25.8 * Update metallb to v0.14.9 (metal-stack/gardener-extension-provider-metal#450) @majst01 ### machine-controller-manager-provider-metal v0.1.24 * Fix deletion request can receive empty provider id. (metal-stack/machine-controller-manager-provider-metal#39) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * feat: reconcile associated resources (metal-stack/cluster-api-provider-metal-stack#76) @vknabel * chore: CAPI v1.9.5 (metal-stack/cluster-api-provider-metal-stack#78) @robertvolkmann * docs: quick move instructions for maintainers (metal-stack/cluster-api-provider-metal-stack#79) @vknabel * Implement image service (metal-stack/metal-apiserver#5) @majst01 * fix: deletion of untouched machines (metal-stack/cluster-api-provider-metal-stack#81) @vknabel * chore(lab): bump clusterctl (metal-stack/cluster-api-provider-metal-stack#80) @vknabel * Update ubuntu/firewall kernel to 6.12.18 (metal-stack/metal-images#287) @majst01 * Update on Hardware support (metal-stack/docs#253) @majst01 * Next release (metal-stack/releases#220) @metal-robot[bot] --- ## v0.21.0 # metal-stack v0.21.0 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.0](https://github.com/metal-stack/releases/releases/tag/v0.21.0) ## General * [Gardener v1.109](https://github.com/gardener/gardener/releases/tag/v1.109.0) ## Breaking Changes * A change in the semver library that is used by metal-stack and in the Gardener project forces us to rename the identifiers that we typically use for OS images like Ubuntu 24.04. The library now requires stricter semantic versions, not allow leading zeroes in version segments. In case you use for example `ubuntu-24.04.20250228` as an ID for an `image` in the metal-api, this needs to become `ubuntu-24.4.20250228`. In order to introduce the new identifier-format, before updating to this release of metal-stack, an image has to be created following the new version format. This image then co-exists with the old image format. After this, all machines referencing the old image must be reprovisioned with the new image ID format. After all the references were migrated to the new image format, the old versions must be removed from the metal-api before upgrading to this release. Please adapt your deployments accordingly. Unfortunately, there is no better way to migrate this ID. Another option was to fork the Gardener project, which we did not want to do. If you encounter bigger issues with this step, please contact us in our Slack channel. (metal-stack/mini-lab#229) ## Component Releases ### csi-driver-lvm v0.6.1 * update to go-1.24 and alpine-3.21 (metal-stack/csi-driver-lvm#114) @majst01 ### helm-charts v0.4.21 * Postgreslet update spilo image (metal-stack/helm-charts#116) @eberlep ### masterdata-api v0.11.7 * Update to go-1.24, simplify error handling in startup code (metal-stack/masterdata-api#110) @majst01 * Fix missing workdir when switching to distroless image. (metal-stack/masterdata-api#112) @Gerrit91 ### metal-api v0.40.5 * Update semver parser with breaking change regarding image versions (metal-stack/metal-api#612) @majst01 * If deleting a machine from database which is crashed, this hint would… (metal-stack/metal-api#611) @majst01 ### metal-image-cache-sync v0.1.8 * Go 1.24 (metal-stack/metal-image-cache-sync#18) @majst01 ### go-ipam v1.14.10 * Update dependencies, try benchmark history (metal-stack/go-ipam#174) @majst01 ### metal-roles v0.15.7 * Fix default for `ztp_provisioning_script` (metal-stack/metal-roles#394) @iljarotar * use different frr debian repo (metal-stack/metal-roles#393) @iljarotar * stop mirroring from eu.gcr.io (metal-stack/metal-roles#392) @mwennrich * Remove deprecated HVPA feature gates. (metal-stack/metal-roles#395) @Gerrit91 * Add csi livenessprobe sidecar (metal-stack/metal-roles#396) @majst01 ### metal-ansible-modules v0.2.10 * Support firewall network allocation mode. (metal-stack/metal-ansible-modules#21) @Gerrit91 * Optional fields for firewall rules. (metal-stack/metal-ansible-modules#20) @Gerrit91 ### gardener-extension-csi-driver-lvm v0.1.3 * Update csi sidecars (metal-stack/gardener-extension-csi-driver-lvm#11) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.20.1 (metal-stack/docs#252) @metal-robot[bot] * Bump releases to version v0.20.2 (metal-stack/docs#254) @metal-robot[bot] * Add database migration with some fixes and small refactoring on datastore (metal-stack/metal-apiserver#6) @Gerrit91 * try machine allocation with redis streams (metal-stack/metal-apiserver#7) @majst01 * Fix ip list (metal-stack/metal-apiserver#8) @majst01 * Move service initialization into services (metal-stack/metal-apiserver#9) @majst01 * Unify and simplify service tests (metal-stack/metal-apiserver#10) @majst01 * Update debian kernel (metal-stack/metal-images#288) @majst01 * Bump metal-api to version v0.40.5 (metal-stack/metal-go#206) @metal-robot[bot] * Bump metal-api to version v0.40.5 (metal-stack/metal-python#150) @metal-robot[bot] * Toolkit must be enabled on newer images (metal-stack/docs#256) @majst01 * Update frr for debian to 10.2.2 (metal-stack/metal-images#289) @majst01 * Next release (metal-stack/releases#221) @metal-robot[bot] --- ## v0.21.1 # metal-stack v0.21.1 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.1](https://github.com/metal-stack/releases/releases/tag/v0.21.1) ## General * Please do not forget to mitigate the [critical ingress-nginx vulnerability](https://www.wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities) by updating the ingress-controllers in your landscape. metal-stack itself does not deploy an ingress-controller and the ingress-controller deployed by Gardener for the monitoring is not really affected as the admission webhook is not enabled. If you still want to run the patched version though, you can use this override: ```yaml gardener_image_vector_overwrite: - name: nginx-ingress-controller sourceRepository: github.com/kubernetes/ingress-nginx repository: registry.k8s.io/ingress-nginx/controller-chroot tag: "v1.11.5" targetVersion: ">= 1.26" ``` * [Gardener v1.110](https://github.com/gardener/gardener/releases/tag/v1.110.0) ## Noteworthy * As part of the [MEP-4](https://docs.metal-stack.io/stable/development/proposals/MEP4/README/) implementation, it is now possible to deploy a preview version of the [metal-apiserver](https://github.com/metal-stack/metal-apiserver). Note that this is only a development preview and will undergo a lot of breaking changes in the next time, so do not deploy this for any production use cases yet. (metal-stack/metal-roles#391) ## Breaking Changes * The support for meilisearch as an audit backend was dropped. Please migrate to the TimescaleDB backend if you depend on this implementation of meilisearch support. (metal-stack/metal-lib#174) ## Component Releases ### gardener-extension-audit v0.1.12 * Revendor g/g v1.108. (metal-stack/gardener-extension-audit#38) @Gerrit91 * cleanup: introduce backend interface (metal-stack/gardener-extension-audit#36) @hown3d * Fix some regressions in Splunk backend (metal-stack/gardener-extension-audit#39) @Gerrit91 ### metal-roles v0.15.13 * Adaptions for g/g v1.110. (metal-stack/metal-roles#397) @Gerrit91 * Remove meilisearch backend from metal-api configuration. (metal-stack/metal-roles#398) @Gerrit91 * Add deployment for metal-apiserver. (metal-stack/metal-roles#391) @Gerrit91 * Remove duplicate value from release vector mapping. (metal-stack/metal-roles#399) @Gerrit91 * Add missing `metal_apiserver_redis_password` default variable. (metal-stack/metal-roles#400) @Gerrit91 * Add missing `metal_apiserver_admin_subjects` default variable. (metal-stack/metal-roles#402) @Gerrit91 ### masterdata-api v0.11.8 * Memory datastore (metal-stack/masterdata-api#111) @majst01 * Reusable postgres db (metal-stack/masterdata-api#114) @majst01 ### gardener-extension-provider-metal v0.25.10 * Create machines with NTP and DNS servers through metal-api (metal-stack/gardener-extension-provider-metal#451) @Gerrit91 * Retain DNS and NTP additional files if they were present before. (metal-stack/gardener-extension-provider-metal#452) @Gerrit91 ### os-metal-extension v0.8.7 * Do not write DNS and NTP configuration anymore. (metal-stack/os-metal-extension#57) @Gerrit91 * Go 1.24 (metal-stack/os-metal-extension#55) @majst01 * Retain DNS and NTP additional files if they were present before. (metal-stack/os-metal-extension#58) @Gerrit91 ### machine-controller-manager-provider-metal v0.1.25 * Support creating machines with NTP and DNS server configuration. (metal-stack/machine-controller-manager-provider-metal#40) @Gerrit91 ### metal-api v0.41.0 * Update dependencies. (metal-stack/metal-api#614) @Gerrit91 ### helm-charts v0.4.22 * Add deployment of metal-apiserver. (metal-stack/helm-charts#117) @Gerrit91 * Remove meilisearch from metal-api configuration. (metal-stack/helm-charts#118) @Gerrit91 ### metal-console v0.7.4 * Upgrade to golangci-lint v2 (metal-stack/metal-console#58) @majst01 ### sonic-configdb-utils v0.2.1 * Add release drafter. (metal-stack/sonic-configdb-utils#4) @Gerrit91 * Add CODEOWNERS and code contribution guidelines. (metal-stack/sonic-configdb-utils#3) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.21.0 (metal-stack/docs#258) @metal-robot[bot] * Update ubuntu and firewall kernel to 6.12.20 (metal-stack/metal-images#290) @majst01 * Remove meilisearch, update go deps (metal-stack/metal-lib#174) @majst01 * Fix issue unmarshalling internal error of audit entry struct. (metal-stack/metal-lib#175) @Gerrit91 * Bump metal-api to version v0.41.0 (metal-stack/metal-python#151) @metal-robot[bot] * Bump metal-api to version v0.41.0 (metal-stack/metal-go#207) @metal-robot[bot] * Use in memory masterdata (metal-stack/metal-apiserver#11) @majst01 * Upgrade to golangci-lint v2 and fix new issues (metal-stack/metal-apiserver#12) @majst01 * Update isolated cluster documentation. (metal-stack/docs#259) @Gerrit91 * MEP-18: Autonomous control plane (metal-stack/docs#235) @majst01 * chore(deps): upgrade golangci-lint (metal-stack/cluster-api-provider-metal-stack#83) @vknabel * Add CODEOWNERS and code contribution guidelines. (metal-stack/sonic-configdb-utils#3) @Gerrit91 * Try pushing to ghcr.io (metal-stack/metal-images#274) @majst01 * fix(ci): forgot registry login (metal-stack/metal-images#292) @vknabel * Next release (metal-stack/releases#222) @metal-robot[bot] --- ## v0.21.10 # metal-stack v0.21.10 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.10](https://github.com/metal-stack/releases/releases/tag/v0.21.10) ## General * [Gardener v1.118](https://github.com/gardener/gardener/releases/tag/v1.118.0) ## Component Releases ### go-ipam v1.14.13 * Upgrade to go-1.25 (metal-stack/go-ipam#185) @majst01 ### metal-bmc v0.6.0 * Update to go-1.25 (metal-stack/metal-bmc#86) @majst01 * Correctly set version build flags. (metal-stack/metal-bmc#82) @Gerrit91 * include sbom in container image (metal-stack/metal-bmc#81) @mac641 * Let metal-bmc configure booting from disk via redfish (metal-stack/metal-bmc#85) @simcod ### metal-api v0.42.3 * Update to go-1.25 (metal-stack/metal-api#633) @majst01 ### metal-console v0.7.5 * Update to go-1.25 and update all deps (metal-stack/metal-console#60) @majst01 * include sbom in container images (metal-stack/metal-console#59) @mac641 ### metal-hammer v0.13.15 * Update to go-1.25, u-root v0.15.0 (metal-stack/metal-hammer#166) @majst01 * Check if password change is necessary for BMC superuser (metal-stack/metal-hammer#163) @simcod * Trigger re-read of partition table (metal-stack/metal-hammer#164) @simcod * Include SBOM as release asset (metal-stack/metal-hammer#158) @mac641 * Fix missing recent nvidia gpu pci ids (metal-stack/metal-hammer#167) @majst01 ### metal-roles v0.17.15 * Adaptions for g/g v1.118. (metal-stack/metal-roles#467) @Gerrit91 ### pixie v0.3.7 * Update to go-1.25 (metal-stack/pixie#38) @majst01 ### metal-core v0.13.1 * Update to go-1.25 and alpine 3.22 (metal-stack/metal-core#162) @majst01 ### gardener-extension-provider-metal v0.26.4 * Fix firewall deployment patch update function called with empty string. (metal-stack/gardener-extension-provider-metal#472) @Gerrit91 * Allow setting explicit hash. (metal-stack/gardener-extension-provider-metal#462) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update to go-1.25 (metal-stack/go-lldpd#31) @majst01 * Add new vendor Gigabyte (metal-stack/go-hal#75) @simcod * Update dependencies (metal-stack/go-hal#77) @majst01 * Bump metal-api to version v0.42.3 (metal-stack/metal-python#158) @metal-robot[bot] * Bump metal-api to version v0.42.3 (metal-stack/metal-go#218) @metal-robot[bot] * Use systemd generator functionality for enabling getty instances (metal-stack/metal-images#338) @simcod * Check for typos (metal-stack/website#93) @simcod * Update go, kernels, lldpd, cri-tools (metal-stack/metal-images#339) @majst01 * Add information about remote access to machines and firewalls (metal-stack/website#94) @simcod * Bump lint-staged from 16.1.5 to 16.1.6 (metal-stack/website#97) @dependabot[bot] * Add information about storing BMC credentials (metal-stack/website#91) @simcod * Bind to valid loopback address if no `bind_to` config option specified (metal-stack/nftables-exporter#30) @auvred * docs: Fix some lines in network physical wiring (metal-stack/website#101) @GeertJohan * Add partition tags to accounting tags (metal-stack/metal-lib#190) @thheinel * Remove julia parts (metal-stack/website#100) @BotondGalxc * Add image url query (metal-stack/api#42) @majst01 * Bump to v0.21.9 release. (metal-stack/website#99) @Gerrit91 * Next release (metal-stack/releases#251) @metal-robot[bot] --- ## v0.21.11 # metal-stack v0.21.11 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.11](https://github.com/metal-stack/releases/releases/tag/v0.21.11) ## General * [Gardener v1.119](https://github.com/gardener/gardener/releases/tag/v1.119.0) ## Required Actions * Before deploying this version, all ONTAP storage classes in all affected shoots must be deleted, since storage classes are immutable. (metal-stack/gardener-extension-ontap#57) * For existing deployments, when updating the gardener-extension-provider-admission controller, the Helm chart upgrade will fail due to the presence of `autoscaling.k8s.io/v1beta2` resources. These were removed in Gardener. For us, it was sufficient to uninstall the admission prior to the rollout using the command `helm uninstall admission-metal`. During this time shoots cannot be created or updated by users. ## Component Releases ### metal-apiserver v0.0.7 * Fix login broken since tenant-service was refactored (metal-stack/metal-apiserver#68) @majst01 * Use synctest to leverage fake time during tests (metal-stack/metal-apiserver#69) @majst01 * Add ctx to convert funcs (metal-stack/metal-apiserver#67) @iljarotar * Since go-1.24, stdlib is able to start http2 unencrypted (metal-stack/metal-apiserver#66) @majst01 ### gardener-extension-backup-s3 v0.8.1 * fix(chart): rbac-runtime has a wrong serviceAccountName (metal-stack/gardener-extension-backup-s3#20) @nschad ### gardener-extension-provider-metal v0.26.5 * Update to go-1.25, gardener 1.119 (metal-stack/gardener-extension-provider-metal#473) @majst01 * Add IP expiration time to CWNP CRD (metal-stack/gardener-extension-provider-metal#471) @iljarotar ### metal-roles v0.17.17 * Add retries to Headscale API key creation. (metal-stack/metal-roles#471) @Gerrit91 * Fix typo in gardenlet operator resource deployment. (metal-stack/metal-roles#470) @Gerrit91 * Added vrrp to sonic_config role for gateway redundancy without vxlan (metal-stack/metal-roles#466) @mreiger * Use own helm-chart for valkey. (metal-stack/metal-roles#473) @Gerrit91 ### masterdata-api v0.13.0 * Introduce namespaces for memberships. (metal-stack/masterdata-api#132) @Gerrit91 * Update to mockery v3. (metal-stack/masterdata-api#133) @Gerrit91 * Update to go-1.25 (metal-stack/masterdata-api#129) @majst01 * Add filters for project and tenant labels. (metal-stack/masterdata-api#131) @Gerrit91 ### gardener-extension-audit v0.2.3 * Set sources in `Chart.yaml` (metal-stack/gardener-extension-audit#53) @maboehm ### gardener-extension-ontap v0.2.6 * Fix reconcile when svm is created but seed secret creation failed (metal-stack/gardener-extension-ontap#55) @Honigeintopf * Remove svm name from tbc (metal-stack/gardener-extension-ontap#59) @Honigeintopf * configure ontap-csi-node daemonset as system-critical (metal-stack/gardener-extension-ontap#58) @mwennrich * set fsType to ext4 in StorageClass (metal-stack/gardener-extension-ontap#57) @mwennrich * Delete ManagedResources (metal-stack/gardener-extension-ontap#49) @majst01 * Make the controller idempotent (metal-stack/gardener-extension-ontap#63) @Honigeintopf ### machine-controller-manager-provider-metal v0.1.27 * Revendor MCM v0.58.0. (metal-stack/machine-controller-manager-provider-metal#43) @Gerrit91 ### helm-charts v0.5.0 * Add valkey chart to remove dependency from bitnami (metal-stack/helm-charts#131) @BotondGalxc ### metal-api v0.42.4 * Consider all machines of partition for evaluating faulty machines (metal-stack/metal-api#632) @ostempel # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update to go-1.25 (metal-stack/metal-robot#90) @majst01 * Update to gardener 1.119 and matching extensions (metal-stack/releases#254) @majst01 * Update postgres to 14.19 and 16.10 (metal-stack/releases#253) @majst01 * Bump releases to version v0.21.10 (metal-stack/website#105) @metal-robot[bot] * Handle A and AAAA records TTL properly in dnscache (metal-stack/firewall-controller#187) @mreiger * docs: Add backgrounds to some diagrams for better visibility in dark mode (metal-stack/website#102) @GeertJohan * Add metal-api reference (metal-stack/website#104) @iljarotar * Add labels to issues and pull requests on creation (metal-stack/metal-robot#83) @Gerrit91 * A bunch of updates including Go 1.25. (metal-stack/cli#8) @Gerrit91 * UpdateRequests must have the Meta field set (metal-stack/api#43) @majst01 * Refactor deprecated dark mode check (metal-stack/website#92) @simcod * Generate table for planning meetings. (metal-stack/website#106) @Gerrit91 * Bump tailwindcss from 4.1.12 to 4.1.13 (metal-stack/website#96) @dependabot[bot] * Bump @mdx-js/react from 3.1.0 to 3.1.1 (metal-stack/website#86) @dependabot[bot] * Bump @carbon/icons-react from 11.65.0 to 11.67.0 (metal-stack/website#108) @dependabot[bot] * Bump @tailwindcss/postcss from 4.1.12 to 4.1.13 (metal-stack/website#109) @dependabot[bot] * Bump typescript from 5.6.3 to 5.9.2 (metal-stack/website#85) @dependabot[bot] * docs: add links to the umbrellas (metal-stack/website#107) @vknabel * fix: broken links (metal-stack/website#110) @vknabel * Detect UpdateRequests which do not specify a updated_at field (metal-stack/api#45) @majst01 * Update to lightos v3.16.x (metal-stack/duros-go#47) @majst01 * Introduce UpdateMeta for OptLock handling (metal-stack/api#46) @majst01 * Ensure defined locking strategies only (metal-stack/api#47) @majst01 * Bump axios from 1.11.0 to 1.12.2 (metal-stack/website#111) @dependabot[bot] * Dependency updates. (metal-stack/metal-go#219) @Gerrit91 * Bump metal-api to version v0.42.4 (metal-stack/metal-python#159) @metal-robot[bot] * Bump metal-api to version v0.42.4 (metal-stack/metal-go#220) @metal-robot[bot] * Next release (metal-stack/releases#252) @metal-robot[bot] --- ## v0.21.2 # metal-stack v0.21.2 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.2](https://github.com/metal-stack/releases/releases/tag/v0.21.2) ## General * [Gardener v1.111](https://github.com/gardener/gardener/releases/tag/v1.111.0) ## Component Releases ### metal-ccm v0.9.6 * on update of a LoadBalancer service, untag old ip addresses, if any exist (metal-stack/metal-ccm#111) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.21.1 (metal-stack/docs#260) @metal-robot[bot] * Update mini-lab (metal-stack/cluster-api-provider-metal-stack#84) @vknabel * Update dependencies and migrate to golangci-lint v2 (metal-stack/metal-go#209) @majst01 * Next release (metal-stack/releases#223) @metal-robot[bot] --- ## v0.21.3 # metal-stack v0.21.3 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.3](https://github.com/metal-stack/releases/releases/tag/v0.21.3) ## General * [Gardener v1.112](https://github.com/gardener/gardener/releases/tag/v1.112.0) ## Required Actions * In order to roll out Gardener v1.112, it is required to re-generate the admin kubeconfig of the virtual garden and make the admin user belong to gardener seeds group. Otherwise, the Gardenlets, which currently use this admin kubeconfig, are not able to reconcile the seeds anymore. A template of how to generate the certificate for the admin kubeconfig can be looked up in https://github.com/metal-stack/mini-lab/pull/233. Existing admin kubeconfigs with this approach still work but they cannot modify resources like `ConfigMaps` anymore in the virtual garden. Please note, that this approach is only a temporary workaround until we migrated our setup to the Gardener Operator. This is planned after we released support with Gardener v1.113. After this, the admin kubeconfig will be managed differently through the operator and it will also be rotated automatically. ## Component Releases ### node-init v0.1.7 * Wrap erros for better understanding (metal-stack/node-init#16) @majst01 ### helm-charts v0.4.23 * increase default memory limit for wal-g-exporter (metal-stack/helm-charts#119) @TLINDEN ### metal-roles v0.15.15 * Configure default value for backup objects to keep in `metal-roles` (metal-stack/metal-roles#370) @simcod * Create seed cluster role after starting up the virtual garden. (metal-stack/metal-roles#407) @Gerrit91 * Adaptions for g/g v1.111. (metal-stack/metal-roles#406) @Gerrit91 ### mini-lab v0.4.4 * Put virtual-garden admin into seeds group. (metal-stack/mini-lab#233) @Gerrit91 * Lower default for backup objects (metal-stack/mini-lab#232) @simcod * Add metal-apiserver as preview in the mini-lab (metal-stack/mini-lab#231) @majst01 * Fix semver image of ubuntu (metal-stack/mini-lab#229) @majst01 * Build images before running integration tests. (metal-stack/mini-lab#230) @Gerrit91 * fix: warning for missing metalctl hmac auth type (metal-stack/mini-lab#224) @vknabel * fix: include missing metal auth type in compose.yaml (metal-stack/mini-lab#222) @vknabel * Add target to transpile ignition userdata (metal-stack/mini-lab#220) @Gerrit91 * Configuring auditing with timescaledb. (metal-stack/mini-lab#217) @Gerrit91 * Dualstack support In Progress (metal-stack/mini-lab#208) @majst01 * Fix routing for flavor CAPMS (metal-stack/mini-lab#218) @robertvolkmann * Use the same bridge for the mgmt network and external connectivity (metal-stack/mini-lab#216) @robertvolkmann * Test on latest OS images. (metal-stack/mini-lab#214) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.21.2 (metal-stack/docs#261) @metal-robot[bot] * Update node-exporter to v1.9.1 (metal-stack/metal-images#293) @majst01 * Migrate to golangci-lint-v2 (metal-stack/metal-images#294) @majst01 * Remove default project. (metal-stack/api#3) @Gerrit91 * Go 1.24 (metal-stack/nftables-exporter#32) @majst01 * Kernel 6.12.22 for ubuntu and firewall (metal-stack/metal-images#295) @majst01 * Next release (metal-stack/releases#224) @metal-robot[bot] --- ## v0.21.4 # metal-stack v0.21.4 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.4](https://github.com/metal-stack/releases/releases/tag/v0.21.4) ## General * [Gardener v1.113](https://github.com/gardener/gardener/releases/tag/v1.113.0) ## Component Releases ### masterdata-api v0.11.9 * Do not allow to update the tenant of a project (metal-stack/masterdata-api#116) @majst01 * Fix misleading error messages (metal-stack/masterdata-api#117) @majst01 ### go-ipam v1.14.11 * Update to golangci-lint v2 (metal-stack/go-ipam#176) @majst01 ### rethinkdb-exporter v0.1.4 * Update to go-1.24 (metal-stack/rethinkdb-exporter#5) @majst01 * Migrate to slog (metal-stack/rethinkdb-exporter#6) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.21.3 (metal-stack/docs#262) @metal-robot[bot] * Bump golang.org/x/net from 0.33.0 to 0.36.0 (metal-stack/go-dhcp-relay#1) @dependabot[bot] * Add a grype workflow for scheduled security scans (metal-stack/metal-images#296) @chbmuc * Add ci/cd, smaller binary, test target, satisfy linter (metal-stack/go-dhcp-relay#2) @majst01 * Simpler runcommand, smallish style (metal-stack/go-dhcp-relay#4) @majst01 * change to urfavcli (metal-stack/go-dhcp-relay#5) @iljarotar * Next release (metal-stack/releases#225) @metal-robot[bot] --- ## v0.21.5 # metal-stack v0.21.5 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.5](https://github.com/metal-stack/releases/releases/tag/v0.21.5) ## General ## Noteworthy * The release vector is now available as an OCI artifact, which is signed using [cosign](https://github.com/sigstore/cosign). This is for now only preview but we are planning to apply signing to our metal-stack container images as well. In addition to that, we are also planning to release SBOMs along with these artifacts. Details are still being discussed, so we will keep you updated on this process in the upcoming releases. Many repositories are migrating to the docker meta action, which effectively changes the created container image tags slightly. Pull request image tags are created as `pr-`, the `latest` tag gets only produced on the latest release and not on the mainline, the mainline image is tagged after the branch name of the mainline (e.g. `main`). (metal-stack/releases#226) ## Required Actions * The `gardener-extension-admission-metal` now registers its mutating and webhook with a service reference instead of a webhook server url. Thus, it might happen that the rollout of the `gardener` role does not succeed on first try giving the following error: `tls: failed to verify certificate: x509: certificate is valid for gardener-extension-admission-metal.garden, not gardener-extension-admission-metal.garden.svc`. In this case, you might roll the virtual garden kube-apiserver instances and re-run the deployment. * A bug has been fixed in the backup-restore-sidecar that occurred when the backup-restore-sidecar was configured to store multiple backups in the same bucket with different object prefixes. The sidecar now creates individual lifecycle rules for the object prefixes such that they do not overwrite each others configurations. If you use such a configuration, it is required to clean up the old lifecycle rule that does not match an object prefix after rolling out this version of the backup-restore-sidecar. ([release notes](https://github.com/metal-stack/backup-restore-sidecar/releases/tag/v0.11.1)) * The new `sonic-config` role is to replace the existing `sonic` role. Read the [sonic-config README](https://github.com/metal-stack/metal-roles/blob/v0.15.17/partition/roles/sonic-config/README.md) for a migration guide from the old sonic role to the new sonic-config role. (metal-stack/metal-roles#390) ## cluster-api-provider-metal-stack * The new Cluster API cluster template now requires `CONTROL_PLANE_IP` to be set. The Control Plane IP will no longer be attached automatically to control plane machines. Rewrite your Cluster API cluster definition to include KubeVIP or alternatives. The images must replace `METAL_MACHINE_ASN` with the machine ASN number in the KubeVIP static pod configuration to be compatible with the default template. (metal-stack/cluster-api-provider-metal-stack#87) * Existing clusters using cluster-provider-metal-stack need to manually update the tags of all cluster associated entities in metal-stack. Before upgrading set `Cluster.spec.paused` to `true`. Then upgrade your metal-stack provider and manually patch every metal-stack machine: replace any `/` with a `.` within the value of the tags `metal-stack.infrastructure.cluster.x-k8s.io/machine-resource` and `metal-stack.infrastructure.cluster.x-k8s.io/cluster-resource` by using `metalctl machine edit $MACHINE_ID`. Now set `Cluster.spec.paused` to `false`. (metal-stack/cluster-api-provider-metal-stack#90) * Make sure the metal-ccm is at least [v0.9.7](https://github.com/metal-stack/metal-ccm/releases/tag/v0.9.7) and its `METAL_CLUSTER_ID` env is set to `${NAMESPACE}.${CLUSTER_NAME}`. (metal-stack/cluster-api-provider-metal-stack#90) ## Component Releases ### backup-restore-sidecar v0.11.1 * Use individual lifecycle policies per object prefix to allow sidecars to act on a single bucket (metal-stack/backup-restore-sidecar#111) @simcod * Fix typo in container name (metal-stack/backup-restore-sidecar#107) @simcod * Migrate to aws-sdk-go-v2 (metal-stack/backup-restore-sidecar#81) @robertvolkmann * Update go deps and proto creation (metal-stack/backup-restore-sidecar#99) @majst01 * add streaming to upload/download of backup (metal-stack/backup-restore-sidecar#102) @ostempel ### metal-api v0.41.2 * Add check to prevent migrating existing machine from one rack to another (metal-stack/metal-api#615) @iljarotar * Migrate to golangci-lint v2 (metal-stack/metal-api#616) @majst01 * Add masterdata-api version (metal-stack/metal-api#617) (#618) @BotondGalxc ### metal-roles v0.15.17 * Allow providing backup encryption key for auditing-timescaledb. (metal-stack/metal-roles#413) @Gerrit91 * Move encryption-key to secret (metal-stack/metal-roles#408) @ostempel * Configurable `deletionGracePeriodHours` for `backupEntries` (metal-stack/metal-roles#409) @simcod * Align to changes in metal-admission chart. (metal-stack/metal-roles#426) @Gerrit91 * Run DHCP tests. (metal-stack/metal-roles#423) @Gerrit91 * Add a new sonic-config role for SONiC configuration. (metal-stack/metal-roles#390) @iljarotar * Removed unsupported taint keys (metal-stack/metal-roles#418) @robertvolkmann * Add authenticationConfig for Gardener (metal-stack/metal-roles#417) @simcod * Static machine ports (metal-stack/metal-roles#414) @mreiger * Add option to configure `imagePullSecrets` when deploying valkey (metal-stack/metal-roles#412) @mac641 ### metalctl v0.18.1 * Allow setting ca certificates when connecting to metal-api (metal-stack/metalctl#279) @mac641 * Update linter to v2 and fix all new warnings (metal-stack/metalctl#283) @majst01 * Add SHA512 checksums (metal-stack/metalctl#280) @mac641 * Small improvements for switch and tenants table. (metal-stack/metalctl#282) @Gerrit91 ### metal-bmc v0.5.8 * Update to go-1.24 and golangci-lint v2 (metal-stack/metal-bmc#78) @majst01 * Update go deps and gh actions (metal-stack/metal-bmc#72) @majst01 * Try to use docker/meta for tag creation (metal-stack/metal-bmc#79) @majst01 * Push tags with v prefix (metal-stack/metal-bmc#80) @majst01 ### masterdata-api v0.11.11 * Bump golang.org/x/net from 0.37.0 to 0.38.0 (metal-stack/masterdata-api#120) @[dependabot[bot]](https://github.com/apps/dependabot) * Add version endpoint (metal-stack/masterdata-api#119) @BotondGalxc * Version grpc client (metal-stack/masterdata-api#121) @BotondGalxc ### droptailer v0.2.18 * Update to go-1.24 and golangci-lint v2 (metal-stack/droptailer#43) @majst01 ### ansible-common v0.6.13 * Add wantedby var to systemd service (metal-stack/ansible-common#31) @iljarotar ### gardener-extension-provider-metal v0.25.14 * Print error message when decoding cp config does not work. (metal-stack/gardener-extension-provider-metal#454) @Gerrit91 * Release Helm OCI artifacts (metal-stack/gardener-extension-provider-metal#455) @Gerrit91 * Push all container tags. (metal-stack/gardener-extension-provider-metal#456) @Gerrit91 * Again more fixes with the meta action. 😔 (metal-stack/gardener-extension-provider-metal#457) @Gerrit91 ### sonic-configdb-utils v0.2.2 * Output generated config_db.json to different file than the current one (metal-stack/sonic-configdb-utils#13) @iljarotar * Adjust config parameters and improve documentation (metal-stack/sonic-configdb-utils#14) @iljarotar * Correctly deal with empty mclag values (metal-stack/sonic-configdb-utils#8) @iljarotar * Allow specifying custom features (metal-stack/sonic-configdb-utils#7) @iljarotar * add split-unified mode (metal-stack/sonic-configdb-utils#5) @iljarotar ### metal-ccm v0.9.7 * fix: do not abort on tag update failure (metal-stack/metal-ccm#115) @vknabel ### helm-charts v0.4.24 * Postgreslet next release (metal-stack/helm-charts#120) @eberlep ### gardener-extension-dns-powerdns v0.7.2 * Release Helm OCI artifacts + Revendor g/g v1.113. (metal-stack/gardener-extension-dns-powerdns#10) @Gerrit91 ### gardener-extension-backup-s3 v0.7.2 * Release Helm OCI artifacts and revendor g/g v1.113. (metal-stack/gardener-extension-backup-s3#15) @Gerrit91 ### os-metal-extension v0.8.8 * Release Helm OCI artifacts. (metal-stack/os-metal-extension#59) @Gerrit91 ### gardener-extension-csi-driver-lvm v0.1.4 * Release Helm OCI artifacts. (metal-stack/gardener-extension-csi-driver-lvm#12) @Gerrit91 ### gardener-extension-audit v0.1.13 * Release Helm OCI artifacts and revendor g/g v1.113. (metal-stack/gardener-extension-audit#43) @Gerrit91 ### metal-deployment-base v0.7.7 * Prefer IPv4 dns lookups over IPv6. (metal-stack/metal-deployment-base#39) @Gerrit91 ### duros-controller v0.11.5 * Lb csi v1.18 (metal-stack/duros-controller#90) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Add section how to recable a machine to another switch pair. (metal-stack/docs#257) @Gerrit91 * Allow setting tls.ClientConfig when creating new client (metal-stack/metal-go#205) @mac641 * Bump metal-api to version v0.41.1 (metal-stack/metal-python#152) @metal-robot[bot] * Bump metal-api to version v0.41.1 (metal-stack/metal-go#210) @metal-robot[bot] * Bump releases to version v0.21.4 (metal-stack/docs#263) @metal-robot[bot] * Prepare for release vector (metal-stack/go-dhcp-relay#6) @iljarotar * Update to go-1.24 and golangci-lint v2 (metal-stack/metal-networker#122) @majst01 * Update dependencies. (metal-stack/metallb-health-sidecar#3) @Gerrit91 * Add information about backup lifecycle rules and encryption (metal-stack/docs#264) @simcod * Rename dhcp-server flag and change help text (metal-stack/go-dhcp-relay#7) @iljarotar * Cleanup worlflow file (metal-stack/go-dhcp-relay#8) @iljarotar * Bump golang.org/x/net from 0.37.0 to 0.38.0 (metal-stack/metal-lib#178) @dependabot[bot] * MEP-16 Firewall Support for Cluster API Provider (metal-stack/docs#255) @vknabel * Update debian and firewall kernel (metal-stack/metal-images#300) @majst01 * Use KubeVIP Load Balancing for Control Planes. (metal-stack/cluster-api-provider-metal-stack#87) @vknabel * Bump the metadata.yaml (metal-stack/cluster-api-provider-metal-stack#88) @vknabel * Bump metal-api to version v0.41.2 (metal-stack/metal-python#153) @metal-robot[bot] * Bump metal-api to version v0.41.2 (metal-stack/metal-go#211) @metal-robot[bot] * Implement auditing memory backend for dev purposes. (metal-stack/metal-lib#179) @Gerrit91 * Allow search for status code 0. (metal-stack/metal-lib#180) @Gerrit91 * Update cri droptailer tailscale (metal-stack/metal-images#301) @majst01 * Update to go-1.24 and linter v2, also better error message (metal-stack/security#61) @majst01 * Describe Tailscale for cluster ingress in an evaluation setup (metal-stack/docs#265) @BotondGalxc * Update debian Kernel (metal-stack/metal-images#302) @majst01 * Metallb BGPPeers are generated for each node and don't have to be created manually (metal-stack/cluster-api-provider-metal-stack#91) @vknabel * Change Cluster ID to be a valid label value (metal-stack/cluster-api-provider-metal-stack#90) @vknabel * Kernel 6.12.28 for ubuntu and firewall (metal-stack/metal-images#303) @majst01 * Update to go-1.24.3 (metal-stack/builder#82) @majst01 * Implement audit backend for Splunk. (metal-stack/metal-lib#167) @Gerrit91 * First code review (metal-stack/gardener-extension-ontap#3) @majst01 * Default add options manager (metal-stack/gardener-extension-ontap#4) @Honigeintopf * Update gardener to 1.114 (metal-stack/gardener-extension-ontap#5) @majst01 * remove fmt.Printf, use structured logging, unexport consts (metal-stack/gardener-extension-ontap#6) @majst01 * Unexport, remove unneeded parameters, remove common pkg (metal-stack/gardener-extension-ontap#7) @majst01 * Introduce svnManager to reuse clients and logger (metal-stack/gardener-extension-ontap#8) @majst01 * Update deps, more enum funcs (metal-stack/api#4) @majst01 * Remove default project part 2. (metal-stack/api#5) @Gerrit91 * Fix linting issues. (metal-stack/cli#2) @Gerrit91 * Update debian kernel (metal-stack/metal-images#305) @majst01 * Migrate to linter v2, prove registry v3 compatibility, update go modules (metal-stack/oci-mirror#13) @majst01 * Include SBOM in the container image (metal-stack/cluster-api-provider-metal-stack#89) @mac641 * Add gardener-operator related release vector components. (metal-stack/releases#229) @Gerrit91 * Fix firewall Dockerfile to include SEMVER_PATCH in base image version (metal-stack/metal-images#308) @mwennrich * remove SEMVER_PATCH_DIR from TARGET_PATH (metal-stack/metal-images#309) @mwennrich * use frr 10.3 from frr-stable (metal-stack/metal-images#307) @mwennrich * Ensure systemd-networkd-wait-online works properly (metal-stack/metal-networker#123) @robertvolkmann * Ensure systemd-networkd-wait-online works properly (metal-stack/metal-images#304) @robertvolkmann * Complete Network Services for api and admin (metal-stack/api#2) @majst01 * Fix debian-nvidia Dockerfile to include SEMVER_PATCH in base image version (metal-stack/metal-images#310) @mwennrich * Kernel 6.12.30 for ubuntu and firewall, 6.1.0-37 for debian (metal-stack/metal-images#311) @majst01 --- ## v0.21.6 # metal-stack v0.21.6 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.6](https://github.com/metal-stack/releases/releases/tag/v0.21.6) ## General * [Gardener v1.114](https://github.com/gardener/gardener/releases/tag/v1.114.0) ## Required Actions * Before rolling out this version make sure there are no shoots anymore with K8s `< 1.27`. * Before updating to this version, make sure that no shoot has the RestrictEgress feature gate set. (metal-stack/gardener-extension-provider-metal#460) ## Noteworthy * It is now possible to deploy the gardener operator through the metal-stack deployment roles. In case you use the Gardener integration of metal-stack, please check out the new deployment roles because the existing `gardener` role is going to be deprecated soon. Migration path is described in the new `gardener-operator` role and can be done with Gardener v1.119. (metal-stack/metal-roles#422) * Upgrading postgres databases from 12-alpine to 13-alpine is not possible because of library differences in icu-lib. The solution is to upgrade to a older 14.10-alpine which has the same icu-lib version as 12-alpine and then update to 14.18-alpine or newer which does not require to run pg_upgrade. It is also recommended to pin the original database to postgres:12.22-alpine to ensure the latest minor. This approach will be applied through our release vector in the upcoming releases of metal-stack. (metal-stack/backup-restore-sidecar#116) ## Breaking Changes * As symlinks that bring defaults into the role scope (including the release vector mapping) were removed from the metal-roles repository, it might be required to adjust your playbooks. In case you used simply `metal-roles` to bring these variables into the role scope, this now needs to be changed to the new defaults location at `metal-roles/common/roles/defaults`. * The release vector variables `gardener_networking_calico_image_*`, `gardener_networking_cilium_image_*`, `gardener_shoot_cert_service_image_*` and `gardener_shoot_dns_service_image_*` were renamed. Usually, these variables are being set automatically through `setup_yaml` and there is nothing to do. However, please check if you have an existing overwrite and adapt accordingly. (metal-stack/metal-roles#422) * The deprecated RestrictEgress feature gate has been removed. (metal-stack/gardener-extension-provider-metal#460) ## Component Releases ### metal-apiserver v0.0.3 * Add test coverage reporting (metal-stack/metal-apiserver#37) @majst01 * Fix log interceptor. (metal-stack/metal-apiserver#33) @Gerrit91 * Fix datastore migration not working. (metal-stack/metal-apiserver#34) @Gerrit91 * Use new enum helpers for file system layouts. (metal-stack/metal-apiserver#31) @Gerrit91 * Cleanup unnecessary json tags in image struct. (metal-stack/metal-apiserver#30) @Gerrit91 * Adding more healthchecks. (metal-stack/metal-apiserver#29) @Gerrit91 * Introduce log interceptor. (metal-stack/metal-apiserver#16) @Gerrit91 * Add useful validation helper (metal-stack/metal-apiserver#28) @majst01 * Adapt to API removing default project. (metal-stack/metal-apiserver#27) @Gerrit91 * Use uuidv7 for entity ids, speed up build (metal-stack/metal-apiserver#26) @majst01 * Add VRF and ASN pools. (metal-stack/metal-apiserver#25) @Gerrit91 * Create container tags through meta action. (metal-stack/metal-apiserver#24) @Gerrit91 * Simplify generic datastore creation for tests (metal-stack/metal-apiserver#21) @majst01 * Improve function signature for entity updates. (metal-stack/metal-apiserver#17) @Gerrit91 * Finalize asynq deletion implementation (metal-stack/metal-apiserver#13) @majst01 * Network services (metal-stack/metal-apiserver#14) @majst01 * include sbom in container image (metal-stack/metal-apiserver#38) @mac641 ### pixie v0.3.6 * include sbom in container image (metal-stack/pixie#35) @mac641 * Update to go-1.24, linter v2 and satisfy new linter warnings (metal-stack/pixie#33) @majst01 ### go-ipam v1.14.12 * Add a NewTestserver helper and fix context handling in tests (metal-stack/go-ipam#182) @majst01 * include sbom in container image (metal-stack/go-ipam#181) @mac641 * Update deps and fix cli not able to connect to server by default (metal-stack/go-ipam#179) @majst01 ### metal-roles v0.17.4 * Publish as OCI artifact. (metal-stack/metal-roles#427) @Gerrit91 * Replace symlinks by role dependencies. (metal-stack/metal-roles#429) @Gerrit91 * feat: remove unused meilisearch (metal-stack/metal-roles#428) @vknabel * fix: json-file logs for bmc #424 (metal-stack/metal-roles#425) @vknabel * Provide missing role meta for metal-python. (metal-stack/metal-roles#432) @Gerrit91 * Role to deploy the Gardener Operator (metal-stack/metal-roles#422) @Gerrit91 * Default ipv4 prefix for headscale was malformed (metal-stack/metal-roles#438) @majst01 * Allow the Soil to be visible in the Gardener Dashboard (metal-stack/metal-roles#419) @robertvolkmann * Use correct variable to reference the CSI driver LVM repository (metal-stack/metal-roles#437) @robertvolkmann * Support ACLs and make it work with newer headscale (metal-stack/metal-roles#439) @majst01 * Add patched node-agent (metal-stack/metal-roles#440) @majst01 * Remove mini-lab patches from Gardener role. (metal-stack/metal-roles#442) @Gerrit91 ### gardener-extension-audit v0.1.14 * Restart audit-webhook-backend StatefulSet on config change (metal-stack/gardener-extension-audit#45) @dergeberl * include sbom in container image (metal-stack/gardener-extension-audit#44) @mac641 ### ansible-common v0.7.0 * Actually run tests in CI and provide common Gardener Operator modules (metal-stack/ansible-common#32) @Gerrit91 ### gardener-extension-provider-metal v0.26.0 * Update to g/g v1.113. (metal-stack/gardener-extension-provider-metal#453) @Gerrit91 * include sbom in container image (metal-stack/gardener-extension-provider-metal#458) @mac641 * Remove deprecated RestrictEgress featureGate (metal-stack/gardener-extension-provider-metal#460) @mwennrich ### metal-ccm v0.9.8 * Update deps, migrate to linter v2 (metal-stack/metal-ccm#118) @majst01 * include sbom in container image (metal-stack/metal-ccm#117) @mac641 * fix: missing error propagation (metal-stack/metal-ccm#116) @vknabel ### backup-restore-sidecar v0.12.0 * Remove meilisearch support (metal-stack/backup-restore-sidecar#117) @majst01 * include sbom in container image (metal-stack/backup-restore-sidecar#115) @mac641 * Update to go-1.24 (metal-stack/backup-restore-sidecar#116) @majst01 * Bump golang.org/x/net from 0.33.0 to 0.38.0 (metal-stack/backup-restore-sidecar#114) @[dependabot[bot]](https://github.com/apps/dependabot) ### duros-controller v0.11.7 * Update duros-go to v3.15, lb-csi to v1.19 and all csi sidecars (metal-stack/duros-controller#95) @majst01 * add security contexts for csi-driver containers and pods (metal-stack/duros-controller#96) @mwennrich * include sbom in container image (metal-stack/duros-controller#94) @mac641 * pointer package is removed (metal-stack/duros-controller#97) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Next release (metal-stack/releases#226) @metal-robot[bot] * Always use full name of container images (metal-stack/releases#230) @robertvolkmann * Bump releases to version v0.21.5 (metal-stack/docs#268) @metal-robot[bot] * Use ghcr image repository reference (metal-stack/docs#269) @simcod * Unexport more, refactor to use retry-go, support iscsi (metal-stack/gardener-extension-ontap#9) @majst01 * Update kernel to 6.12.32 and frr to 10.3.1 (metal-stack/metal-images#314) @majst01 * Do not break release, actual implementation of closing the issues was wrong (metal-stack/metal-images#313) @majst01 * feat: remove unused meilisearch (metal-stack/releases#231) @vknabel * Add handler to add issues and pull requests to a project. (metal-stack/metal-robot#76) @Gerrit91 * Update protovalidate and mockery (metal-stack/api#9) @majst01 * Move to new docs structure (metal-stack/docs#273) @Gerrit91 * Update machine provisioning sequence diagram (metal-stack/docs#272) @simcod * Prevent a nil pointer access in GetStringValue (metal-stack/api#10) @majst01 * Add tenant scoped message validation (metal-stack/api#12) @majst01 * Adapt DIY configuration to starter config (metal-stack/docs#289) @mwindower * include sbom in container image (metal-stack/go-dhcp-relay#9) @mac641 * include sbom as release asset (metal-stack/nftables-exporter#33) @mac641 * include sbom as container image and as release asset (metal-stack/firewall-controller#201) @mac641 * Update Ubuntu and Firewall kernel to 6.12.34 (metal-stack/metal-images#316) @majst01 * include sbom in container image (metal-stack/metallb-health-sidecar#4) @mac641 * include sbom in container image (metal-stack/metal-robot#78) @mac641 * include sbom in container image (metal-stack/gardener-vpn-gateway#3) @mac641 * include sbom in container image (metal-stack/mini-lab-ovmf#2) @mac641 * include sbom in container image (metal-stack/chrony#4) @mac641 * Fix build and update build deps. (metal-stack/rethinkdb-backup-tools-build#3) @Gerrit91 * include sbom in container image (metal-stack/rethinkdb-backup-tools-build#2) @mac641 * Update to lightos v3.15 (metal-stack/duros-go#46) @majst01 * Information on planning meetings. (metal-stack/docs#292) @Gerrit91 * Unwrap connect response body for audit indexing. (metal-stack/metal-lib#182) @Gerrit91 * Bump github.com/gorilla/csrf from 1.7.2 to 1.7.3 (metal-stack/metal-lib#177) @dependabot[bot] * Bump github.com/containerd/containerd from 1.7.20 to 1.7.27 (metal-stack/metal-lib#176) @dependabot[bot] * Add section on release repo requirements. (metal-stack/docs#294) @Gerrit91 * chore: update svg (metal-stack/docs#295) @vknabel * include sbom in container image (metal-stack/gardener-extension-ontap#12) @mac641 * Rename section and add information regarding rollback (metal-stack/docs#287) @simcod * More tests (metal-stack/updater#12) @Azneo * Update metal-lib (metal-stack/metal-images#318) @majst01 * Update to go-1.24.4 and install typst (metal-stack/builder#84) @majst01 * Next release (metal-stack/releases#232) @metal-robot[bot] --- ## v0.21.7 # metal-stack v0.21.7 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.7](https://github.com/metal-stack/releases/releases/tag/v0.21.7) ## General * [Gardener v1.115](https://github.com/gardener/gardener/releases/tag/v1.115.0) ## Component Releases ### gardener-extension-provider-metal v0.26.2 * Update metallb to v0.15.2 (metal-stack/gardener-extension-provider-metal#464) @mwennrich * update dependencies (metal-stack/gardener-extension-provider-metal#461) @mwennrich * Configmap access for firewallcontroller (metal-stack/gardener-extension-provider-metal#449) @mreiger * tighten rbac for kube-system:duros-controller (metal-stack/gardener-extension-provider-metal#463) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Gardener v1.115.4 (metal-stack/releases#241) @majst01 * Pin postgres to 12.22 (metal-stack/releases#240) @majst01 * Remove systemd-resolved and configure DNS consistently (metal-stack/metal-images#320) @simcod * Nvme Modprobe Of Worker Node (metal-stack/gardener-extension-ontap#20) @Honigeintopf * Next release (metal-stack/releases#242) @metal-robot[bot] --- ## v0.21.8 # metal-stack v0.21.8 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.8](https://github.com/metal-stack/releases/releases/tag/v0.21.8) ## General * [Gardener v1.116](https://github.com/gardener/gardener/releases/tag/v1.116.0) ## Component Releases ### backup-restore-sidecar v0.12.1 * Allow setting a trusted CA certificate (metal-stack/backup-restore-sidecar#120) @robertvolkmann * Migrate to new tablewriter (metal-stack/backup-restore-sidecar#118) @majst01 ### metal-apiserver v0.0.4 * Update deps, adjust semver test to new version (metal-stack/metal-apiserver#50) @majst01 * Proposal for improving repository generic interface (metal-stack/metal-apiserver#18) @Gerrit91 * Enable rego linting, fix issues (metal-stack/metal-apiserver#48) @majst01 * Rename store property accordingly (metal-stack/metal-apiserver#46) @majst01 * Allow only prefixes on boundaries, prevent overlaps on update and fix update (metal-stack/metal-apiserver#42) @majst01 * Simplify rethinkdb, less valkey syncs (metal-stack/metal-apiserver#43) @majst01 * Error wrapping got lost with errorutil.Convert (metal-stack/metal-apiserver#36) @majst01 * Use existing context where available (metal-stack/metal-apiserver#40) @majst01 * Use Testserver from go-ipam (metal-stack/metal-apiserver#39) @majst01 ### metal-metrics-exporter v0.3.0 * feat: add METAL_API_AUTH_TYPE to support read-only acciess via Metal-View auth type (metal-stack/metal-metrics-exporter#24) @mwennrich * include sbom in container image (metal-stack/metal-metrics-exporter#23) @mac641 * feat: add imageId to metal_machine_allocation_info metric (metal-stack/metal-metrics-exporter#22) @mwennrich ### gardener-extension-audit v0.1.15 * Bump vpn-gateway to v0.1.1. (metal-stack/gardener-extension-audit#48) @Gerrit91 ### sonic-configdb-utils v0.3.0 * Different way to retrieve platform information (metal-stack/sonic-configdb-utils#12) @iljarotar * change license to MIT (metal-stack/sonic-configdb-utils#28) @iljarotar * do not set autoneg if not defined (metal-stack/sonic-configdb-utils#27) @iljarotar * include sbom in container image (metal-stack/sonic-configdb-utils#26) @mac641 * FIX: Add all bgp ports to interfaces (metal-stack/sonic-configdb-utils#23) @iljarotar * Pass device metadata as values instead of reading current config (metal-stack/sonic-configdb-utils#22) @iljarotar * Vlan subinterfaces (metal-stack/sonic-configdb-utils#20) @iljarotar * Adapt LLDP config keys to Edgecore 202111 branch (metal-stack/sonic-configdb-utils#19) @iljarotar * Add switch to add VTEP with no VXLAN tunnel maps configured (metal-stack/sonic-configdb-utils#18) @iljarotar * Remove vlan_members variable and always add members (metal-stack/sonic-configdb-utils#17) @iljarotar * Improve support for Accton-AS4630-54TE and Accton-AS4625-54T (metal-stack/sonic-configdb-utils#16) @iljarotar * Allow configuring ntp source interface and vrf (metal-stack/sonic-configdb-utils#15) @iljarotar ### metal-roles v0.17.7 * Allow configuring the trusted certificate authority of the S3 storage service (metal-stack/metal-roles#444) @robertvolkmann * add gepm defaults for pods and services cidr (metal-stack/metal-roles#448) @iljarotar * Allow skipping certificate check for the virtual garden etcd (metal-stack/metal-roles#445) @robertvolkmann * Sonic role: Enhancements to support multiple neighbors per interconnect (metal-stack/metal-roles#421) @mreiger * Add loopback as default src address for BGP (metal-stack/metal-roles#430) @iljarotar * disable-admission-plugins=PersistentVolumeLabels is deprecated since 1.31 (metal-stack/metal-roles#452) @majst01 * Provide endpoints for gardener admission-metal. (metal-stack/metal-roles#454) @Gerrit91 ### gardener-extension-provider-metal v0.26.3 * Revendor g/g v1.116. (metal-stack/gardener-extension-provider-metal#467) @Gerrit91 * Remove now obsolete kubelet config modification. (metal-stack/gardener-extension-provider-metal#465) @Gerrit91 ### machine-controller-manager-provider-metal v0.1.26 * Revendor MCM v0.57.1. (metal-stack/machine-controller-manager-provider-metal#41) @Gerrit91 * include sbom in container image (metal-stack/machine-controller-manager-provider-metal#42) @mac641 ### metal-hammer v0.13.13 * Changes for new supermicro support (metal-stack/metal-hammer#148) @simcod * Update to go-1.24 and all go modules (metal-stack/metal-hammer#156) @majst01 * Use `docker buildx build` for the metal-hammer image build (metal-stack/metal-hammer#155) @simcod * Configure NTP servers earlier (metal-stack/metal-hammer#149) @simcod * Fix CI, update actions (metal-stack/metal-hammer#161) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Gardener v1.116 (metal-stack/releases#244) @majst01 * Bump releases to version v0.21.6 (metal-stack/docs#296) @metal-robot[bot] * Bump releases to version v0.21.7 (metal-stack/docs#298) @metal-robot[bot] * Patch management information for operators (metal-stack/docs#297) @simcod * Release Helm OCI artifacts. (metal-stack/gardener-extension-ontap#17) @Honigeintopf * add sbom section (metal-stack/docs#293) @mac641 * Remove services, remove data lif (metal-stack/gardener-extension-ontap#24) @Honigeintopf * CISO: Information about cryptography (metal-stack/docs#299) @simcod * Introduce CISO section (metal-stack/docs#290) @simcod * Add cryptsetup to debian (metal-stack/metal-images#322) @Honigeintopf * Go-1.24 and replace zap with slog (metal-stack/gardener-vpn-gateway#2) @majst01 * Size Services for api and admin (metal-stack/api#6) @majst01 * Generate python client, add examples (metal-stack/api#16) @majst01 * add docker build step and include sboms in container image and as release asset (metal-stack/cli#3) @mac641 * Add a Token Refresh Service (metal-stack/api#15) @majst01 * Support empty tokens, and tokens without exp set (metal-stack/api#17) @majst01 * General section and slightly moving stuff (metal-stack/docs#300) @vknabel * add multiple client functionality (metal-stack/ontap-go#4) @Honigeintopf * Simplified api for token refresh config (metal-stack/api#18) @majst01 * Do not disable user before password change (metal-stack/go-hal#73) @simcod * Next release (metal-stack/releases#243) @metal-robot[bot] --- ## v0.21.9 # metal-stack v0.21.9 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.21.9](https://github.com/metal-stack/releases/releases/tag/v0.21.9) ## General * [Gardener v1.117](https://github.com/gardener/gardener/releases/tag/v1.117.0) ## Required Actions * We recommend running the `metal-metrics-exporter` with a `Metal-View` auth type. The new Ansible variable `monitoring_metal_api_authtype` was introduced. By default, `Metal-View` is used. Please make sure, to update the auth type, if you want to run the exporter with higher privileges or verify, if `monitoring_metal_api_hmac` is configured correctly to be a view HMAC. (metal-stack/metal-roles#449) * The container image `metalstack/builder` is not published on Docker Hub anymore. Please migrate to `ghcr.io/metal-stack/builder` in case you are using the image from Docker Hub. (metal-stack/builder#83) ## Breaking Changes * The `audittailer` pod is now being deployed into the `kube-system` namespace and not anymore into the dedicated `audit` namespace. (metal-stack/gardener-extension-audit#50) * In the audit extension, the splunk backend now validates the TLS hostname. Verify that your splunk backend uses a TLS certificate that matches the hostname. (metal-stack/gardener-extension-audit#51) ## Component Releases ### gardener-extension-ontap v0.2.3 * Trident deployment refactor (metal-stack/gardener-extension-ontap#32) @Honigeintopf * Create network interface on both nodes (metal-stack/gardener-extension-ontap#25) @Honigeintopf * Remove services, remove data lif (metal-stack/gardener-extension-ontap#24) @Honigeintopf * Release Helm OCI artifacts. (metal-stack/gardener-extension-ontap#17) @Honigeintopf * Nvme Modprobe Of Worker Node (metal-stack/gardener-extension-ontap#20) @Honigeintopf * include sbom in container image (metal-stack/gardener-extension-ontap#12) @mac641 * Unexport more, refactor to use retry-go, support iscsi (metal-stack/gardener-extension-ontap#9) @majst01 * Introduce svnManager to reuse clients and logger (metal-stack/gardener-extension-ontap#8) @majst01 * Unexport, remove unneeded parameters, remove common pkg (metal-stack/gardener-extension-ontap#7) @majst01 * remove fmt.Printf, use structured logging, unexport consts (metal-stack/gardener-extension-ontap#6) @majst01 * Update gardener to 1.114 (metal-stack/gardener-extension-ontap#5) @majst01 * Default add options manager (metal-stack/gardener-extension-ontap#4) @Honigeintopf * First code review (metal-stack/gardener-extension-ontap#3) @majst01 * Use new ontap-go metroclusterclient (metal-stack/gardener-extension-ontap#35) @Honigeintopf * Add ctx to all api calls to ontap (metal-stack/gardener-extension-ontap#34) @Honigeintopf * Add storage encryption labels (metal-stack/gardener-extension-ontap#22) @Honigeintopf * first approach to create cwnp as mr manifests (metal-stack/gardener-extension-ontap#38) @Honigeintopf * Bugfix for password requirement (metal-stack/gardener-extension-ontap#40) @Honigeintopf * Update dependencies (metal-stack/gardener-extension-ontap#37) @majst01 * Remove Aggregate (metal-stack/gardener-extension-ontap#41) @Honigeintopf * Simplify controller configuration (metal-stack/gardener-extension-ontap#39) @majst01 * distribute lifs on both nodes (metal-stack/gardener-extension-ontap#47) @Honigeintopf * Template Secret (metal-stack/gardener-extension-ontap#44) @majst01 * Allow volume expansion (metal-stack/gardener-extension-ontap#53) @majst01 * change pause container from busybox to pause (metal-stack/gardener-extension-ontap#51) @mwennrich * Create ontap-encrypted storageclass (metal-stack/gardener-extension-ontap#54) @majst01 ### gardener-extension-audit v0.2.2 * change shoot-namespace to kube-system (metal-stack/gardener-extension-audit#50) @ostempel * Splunk: Remove duplicate secret (metal-stack/gardener-extension-audit#46) @dergeberl * Ensure audit-webhook-backend sts restart on changes to backend configuration (metal-stack/gardener-extension-audit#49) @MichaelEischer * Add S3 as backend (metal-stack/gardener-extension-audit#47) @dergeberl * Update fluentbit and enable tls.verify_hostname for splunk (metal-stack/gardener-extension-audit#51) @MichaelEischer ### metal-ansible-modules v0.2.11 * Ship as OCI artifact. (metal-stack/metal-ansible-modules#22) @Gerrit91 ### ansible-common v0.7.2 * Ship as OCI artifact. (metal-stack/ansible-common#38) @Gerrit91 * Use appropriate shebang lines for ansible modules. (metal-stack/ansible-common#40) @Gerrit91 ### helm-charts v0.4.25 * Make postgreslet anti affinity topology key configurable (metal-stack/helm-charts#128) @eberlep * Next Postgreslet Release (metal-stack/helm-charts#127) @eberlep * Postgreslet: Add dedicated fluentbit tag for the backup log (metal-stack/helm-charts#126) @eberlep * Add pg_role metrics (metal-stack/helm-charts#124) @TLINDEN * Add query to monitor pg_stat_wal_receiver (metal-stack/helm-charts#122) @eberlep * Fix SQL error in postgres-exporter query config when executing WAL check on standby node (metal-stack/helm-charts#121) @TLINDEN * use distinct on pg_stat_statements to avoid postgres-exporter crash (metal-stack/helm-charts#114) @TLINDEN ### metal-roles v0.17.13 * add ontap to gardener-operator (metal-stack/metal-roles#446) @Honigeintopf * Provide more roles for gardener operator setup. (metal-stack/metal-roles#436) @Gerrit91 * add netapp/trident images to oci-mirror config (metal-stack/metal-roles#457) @mwennrich * Make global Prometheus scrape parameters configurable (metal-stack/metal-roles#455) @simcod * Remove unknown fields from new gardener roles. (metal-stack/metal-roles#458) @Gerrit91 * feat: alertmanager additional rules and receivers (metal-stack/metal-roles#459) @vknabel * Use `metal-view` as hmac default variable for `metal-metrics-exporter` (metal-stack/metal-roles#449) @simcod * Adjust sonic-config role to new sonic-configdb-utils version (metal-stack/metal-roles#463) @iljarotar * Fix sonic-config template (metal-stack/metal-roles#464) @iljarotar ### masterdata-api v0.12.0 * Migrate away from deprecated wrappers (metal-stack/masterdata-api#81) @majst01 * include sbom in container image (metal-stack/masterdata-api#124) @mac641 * Fix duplicate memberships in tenant delete (metal-stack/masterdata-api#123) @iljarotar ### metalctl v0.18.3 * use sha512 in updater to fix md5 checksum mismatch errors (metal-stack/metalctl#288) @mac641 * Include SBOM into container image and as release asset (metal-stack/metalctl#286) @mac641 * Upgrade metal-lib with new tableprinter version (metal-stack/metalctl#285) @BotondGalxc ### gardener-extension-backup-s3 v0.8.0 * improve resource naming of chart by using fullname rather than the static gardener-extension-backup-s3 (metal-stack/gardener-extension-backup-s3#18) @nschad * Harden deployment to comply with pod security standard "restricted" (metal-stack/gardener-extension-backup-s3#19) @MichaelEischer ### metal-api v0.42.2 * Size services (metal-stack/metal-apiserver#49) @majst01 * Log call duration if debug is true (metal-stack/metal-apiserver#56) @majst01 * Token Refresh Service (metal-stack/metal-apiserver#51) @majst01 * Spread only on same machine size. (metal-stack/metal-api#625) @Gerrit91 * Include sbom in container image (metal-stack/metal-api#622) @mac641 * Project is not used in size validation, remove it (metal-stack/metal-api#621) @majst01 * Provide network type migration as preparation for MEP-4. (metal-stack/metal-api#620) @Gerrit91 * Update masterdata-api dependency. (metal-stack/metal-api#627) @Gerrit91 * Fix broken audit search. (metal-stack/metal-api#629) @Gerrit91 * Ignore delayed buffered phoned home events during machine reclaim (metal-stack/metal-api#631) @iljarotar ### sonic-configdb-utils v0.4.1 * Support different build branches (metal-stack/sonic-configdb-utils#30) @iljarotar * yaml is not yml (metal-stack/sonic-configdb-utils#31) @iljarotar ### metal-core v0.13.0 * Add vrf to vni mapping (metal-stack/metal-core#144) @iljarotar * Read SONiC interfaces from ConfigDB (metal-stack/metal-core#161) @iljarotar * Cleanup VLANs and VRFs on SONiC (metal-stack/metal-core#155) @iljarotar * Graceful shutdown (metal-stack/metal-core#109) @robertvolkmann * include sbom in container image (metal-stack/metal-core#157) @mac641 * Bump golang.org/x/net from 0.35.0 to 0.38.0 (metal-stack/metal-core#154) @[dependabot[bot]](https://github.com/apps/dependabot) # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.21.8 (metal-stack/docs#303) @metal-robot[bot] * replace md5sum using sha512sum (metal-stack/updater#15) @mac641 * Workaround GitHub Pagination for releases (metal-stack/updater#18) @vknabel * Update Postgres to 14.10 of ipam and masterdata (metal-stack/releases#246) @majst01 * Introduce new cluster type tags (metal-stack/metal-lib#185) @thheinel * Change ontap client to slice (metal-stack/ontap-go#5) @Honigeintopf * Add OCI refs for ansible roles. (metal-stack/releases#248) @Gerrit91 * Add cluster api provider metal stack (metal-stack/releases#245) @vknabel * bump go to 1.24.x and golangci-lint-action to v8 (metal-stack/updater#17) @mac641 * update Masterminds/semver to v3.4.0 (and other deps) (metal-stack/oci-mirror#15) @mwennrich * include sbom in container image (metal-stack/gardener-extension-duros#3) @mac641 * Upgrade Tablewriter to v1 (metal-stack/metal-lib#181) @BotondGalxc * investigate and fix `invalid argument` error while checking location permissions (metal-stack/updater#19) @mac641 * Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 (metal-stack/metal-lib#183) @dependabot[bot] * More infos on contents of the planning meeting and for contributions. (metal-stack/docs#305) @Gerrit91 * Cumulus is not in release flow anymore. (metal-stack/docs#306) @Gerrit91 * frr 10.4 was released but 10.3.1 is not installable anymore (metal-stack/metal-images#324) @majst01 * Fix typo in the hardware network cards section (metal-stack/docs#308) @GeertJohan * Add a generic python driver (metal-stack/api#21) @majst01 * Fix python build if it is the first time (metal-stack/api#23) @majst01 * Update available OS Images (metal-stack/docs#309) @majst01 * Communication matrix (metal-stack/docs#307) @vknabel * Provide information about used technologies and protocols (metal-stack/docs#302) @simcod * Add infra switch service (metal-stack/api#24) @iljarotar * Update to frr 10.4.1 and kernel mainline v6.12.41 (metal-stack/metal-images#332) @simcod * machine api definitions for get, list and update (metal-stack/api#22) @majst01 * Rename machine-role to machine-allocation-type which is more appropriate (metal-stack/api#28) @majst01 * By accident machine-allocation-type was optional (metal-stack/api#29) @majst01 * Update metal-lib, adopt tablewriter (metal-stack/cli#5) @majst01 * Admin machine get and list (metal-stack/api#31) @majst01 * Bump metal-api to version v0.41.3 (metal-stack/metal-python#154) @metal-robot[bot] * Bump metal-api to version v0.41.3 (metal-stack/metal-go#214) @metal-robot[bot] * Use yq docker image instead of local install (metal-stack/metal-go#212) @iljarotar * Bump metal-api to version v0.42.0 (metal-stack/metal-python#155) @metal-robot[bot] * Bump metal-api to version v0.42.0 (metal-stack/metal-go#215) @metal-robot[bot] * MEP-17 global network view (metal-stack/docs#145) @majst01 * Unify naming of reference fields (metal-stack/api#27) @majst01 * Go 1.25 (metal-stack/builder#85) @majst01 * Add duros extension (metal-stack/gardener-extension-duros#1) @ostempel * Fix release drafter merges sections on common component prefixes. (metal-stack/metal-robot#86) @Gerrit91 * Update to go-1.25 and mockery (metal-stack/api#32) @majst01 * Small fork build improvements for issues handler. (metal-stack/metal-robot#88) @Gerrit91 * Skip empty code blocks to be added to release draft. (metal-stack/metal-robot#89) @Gerrit91 * Also support REQUIRED_ACTIONS along with ACTIONS_REQUIRED. (metal-stack/metal-robot#87) @Gerrit91 * include SBOM in container image (metal-stack/oci-mirror#16) @mac641 * Go 1.25 (metal-stack/firewall-controller#205) @majst01 * introduce standard docker workflow and include sbom in container images (metal-stack/builder#83) @mac641 * Remove useless query parameter, make partition require for admin machine list (metal-stack/api#33) @majst01 * Remove last reinstall artifact (metal-stack/api#34) @majst01 * include generating sbom in actions workflow (metal-stack/go-lldpd#30) @mac641 * Fix link formatting for go-containerregistry (metal-stack/oci-mirror#17) @guettli * Automatically remove triage label when changing item status. (metal-stack/metal-robot#82) @Gerrit91 * Bump metal-api to version v0.42.1 (metal-stack/metal-python#156) @metal-robot[bot] * Bump metal-api to version v0.42.1 (metal-stack/metal-go#216) @metal-robot[bot] * Fix linting issues of new linter version. (metal-stack/metal-lib#189) @Gerrit91 * Better deal with gRPC errors in auditing. (metal-stack/metal-lib#186) @Gerrit91 * Auto generate first release (metal-stack/website#73) @Gerrit91 * More realistic redirects (metal-stack/website#64) @vknabel * fix: force redirects for domains (metal-stack/website#76) @vknabel * Fix link to doc archive in nav. (metal-stack/website#75) @Gerrit91 * docs: link to capms ref docs (metal-stack/website#77) @vknabel * Minor updates on `README.md` (metal-stack/website#78) @Gerrit91 * Go 1.25 (metal-stack/nftables-exporter#35) @majst01 * fix: missing redirects and inconsistent urls (metal-stack/website#80) @vknabel * Allow setting labels on projects and tenant requests. (metal-stack/api#36) @Gerrit91 * Update connectpy (metal-stack/api#37) @majst01 * Try is reserved in python (metal-stack/api#38) @majst01 * fix: tag pagination and wrong month for post (metal-stack/website#84) @vknabel * Use python sync client. (metal-stack/api#39) @Gerrit91 * Bump metal-api to version v0.42.2 (metal-stack/metal-python#157) @metal-robot[bot] * Bump metal-api to version v0.42.2 (metal-stack/metal-go#217) @metal-robot[bot] * Remove unused try fsl response (metal-stack/api#40) @majst01 * Remove old repository name occurrences. (metal-stack/website#81) @Gerrit91 * Add labels to query requests where possible. (metal-stack/api#41) @Gerrit91 * docs: proposal process (metal-stack/website#89) @vknabel * Information about BMC user management (metal-stack/website#82) @simcod * Check if user password change is necessary via ipmitool (metal-stack/go-hal#76) @simcod * Next release (metal-stack/releases#247) @metal-robot[bot] --- ## v0.22.0 # metal-stack v0.22.0 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.0](https://github.com/metal-stack/releases/releases/tag/v0.22.0) ## General ## Required Actions * This release expects the migration to the Gardener Operator. Please read the migration notes [here](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/gardener-operator#migration-path). * The deployment-base image now comes with Ansible 11. The `ansible-core` was raised from `2.15.4` to `2.18.7`. We did not see any major issues with this update for our roles and playbooks. In case you encounter problems with the update, please adapt your playbooks or file an issue. You can always go back and use the previous release of `metal-deployment-base:v0.7.7` until we found a solution to your issue. Note that Python 2 is not supported anymore. You probably should take a look at the [porting guides](https://docs.ansible.com/ansible/latest/porting_guides/core_porting_guides.html) of Ansible, too. ([release notes](https://github.com/metal-stack/metal-deployment-base/releases/tag/v0.8.0)) * Loopback addresses of all leaves must be known to all routers (and servers) that need to communicate with the leaves via default VRF. BGP or static routes must be established accordingly. (metal-stack/metal-core#168) ## Noteworthy * The audit extension now scrapes the fluentbit v2 metrics, which are a superset of the v1 metrics, and also provides metrics for the storage use. Metric `process_start_time_seconds` is now called `fluentbit_process_start_time_seconds`. (metal-stack/gardener-extension-audit#56) * Add missing securityContext controls in order to comply with the restricted Pod Security Standards policy. Move default serverPort to 10443. (metal-stack/gardener-extension-audit#57) ## Component Releases ### metal-core v0.13.5 * Redistribute loopback address on SONiC (metal-stack/metal-core#165) @iljarotar * Revert "Set loopback as source address in SONiC frr.conf (metal-stack/metal-core#159)" (#167) @iljarotar * Revert "Redistribute loopback address on SONiC (metal-stack/metal-core#165)" (#166) @iljarotar * Set Loopback0 address as default source address in frr.conf on SONiC Leaves (metal-stack/metal-core#168) @iljarotar ### gardener-extension-ontap v0.2.8 * Use name of shoot to create user secret (metal-stack/gardener-extension-ontap#71) @Honigeintopf ### metal-deployment-base v0.8.0 * Add metalstack.cloud CLI. (metal-stack/metal-deployment-base#45) @Gerrit91 * Update dependencies. (metal-stack/metal-deployment-base#44) @Gerrit91 * include sbom in container image (metal-stack/metal-deployment-base#41) @mac641 ### gardener-extension-audit v0.3.0 * Helm chart: add missing securityContext to comply with restricted PSS… (metal-stack/gardener-extension-audit#57) @mstueer * Enable storage metrics (metal-stack/gardener-extension-audit#56) @MichaelEischer * Bump fluentbit version (metal-stack/gardener-extension-audit#55) @MichaelEischer * Update README (adding S3 backend + notes to Cilium). (metal-stack/gardener-extension-audit#54) @Gerrit91 ### metal-apiserver v0.0.9 * Enable end user e2e tests with multiple tenants (metal-stack/metal-apiserver#81) @majst01 * Include proto validation during tests (metal-stack/metal-apiserver#77) @majst01 * Add ip create benchmark as first step (metal-stack/metal-apiserver#83) @majst01 * Update api to main (metal-stack/metal-apiserver#82) @majst01 * Fix wrong project mapper function being used. (metal-stack/metal-apiserver#79) @Gerrit91 * Properly log app version at launch. (metal-stack/metal-apiserver#78) @Gerrit91 * Implement switch service get, create, update, delete (metal-stack/metal-apiserver#57) @iljarotar * Adopt to breaking api changes (metal-stack/metal-apiserver#76) @majst01 * Update dependencies, adopt breaking changes in validation (metal-stack/metal-apiserver#75) @majst01 * UpdateRequest must now have Meta set (metal-stack/metal-apiserver#71) @majst01 * Update to use masterdata membership namespaces. (metal-stack/metal-apiserver#72) @Gerrit91 * Migrate to simple connect go (metal-stack/metal-apiserver#85) @majst01 * Do not expose internal types from the repository package. (metal-stack/metal-apiserver#74) @Gerrit91 ### metal-roles v0.17.19 * Add proper config map ref for shoot audit policy. (metal-stack/metal-roles#482) @Gerrit91 * Fix typos (metal-stack/metal-roles#480) @majst01 * Gardener operator review (metal-stack/metal-roles#481) @Gerrit91 * Deploy seed-api-server config map on GCP. (metal-stack/metal-roles#483) @Gerrit91 ### cluster-api-provider-metal-stack v0.6.2 * feat: integration tests (metal-stack/cluster-api-provider-metal-stack#94) @vknabel * feat: bump images (metal-stack/cluster-api-provider-metal-stack#97) @vknabel # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Fix typos. (metal-stack/website#123) @Gerrit91 * Use Slack join link instead of slack workspace link. (metal-stack/website#124) @Gerrit91 * Migrate to docker bake and include sboms in generated images (metal-stack/metal-images#341) @mac641 * Bump releases to version v0.21.11 (metal-stack/website#121) @metal-robot[bot] * Fix bad link on blog article. (metal-stack/website#126) @Gerrit91 * Adopt to recent api changes (metal-stack/cli#9) @majst01 * fix build pipeline by ensuring tarball output directory is created properly (metal-stack/metal-images#353) @mac641 * ConnectPy is now moved to the connect org (metal-stack/api#48) @majst01 * Add definitions for api and admin switch services (metal-stack/api#30) @iljarotar * Predefined validation rules (metal-stack/api#52) @majst01 * Use client interceptor instead of RoundTripper for auth header injection (metal-stack/api#51) @majst01 * Log only if logger is set (metal-stack/api#53) @majst01 * Missing switch port status (metal-stack/api#54) @iljarotar * Switch cannot be updated (metal-stack/api#55) @iljarotar * Simplify and fix some proto validations (metal-stack/api#57) @majst01 * Simplify validation tests (metal-stack/api#56) @majst01 * Fix validation isssues found in apiserver (metal-stack/api#59) @majst01 * Switch from ignite to cloud-hypervisor (metal-stack/metal-images#348) @mwindower * Add switch ID to heartbeat (metal-stack/api#58) @iljarotar * Remove periodic vulnerability scans, we have sboms now (metal-stack/metal-images#356) @majst01 * Go 1.25.3 (metal-stack/builder#86) @majst01 * Token with labels, console token is user token now (metal-stack/api#60) @majst01 * IPIssues is not yet decided (metal-stack/api#61) @majst01 * update to latest main of parent project (metal-stack/monsoon#1) @ostempel * Make switch port desired state optional (metal-stack/api#62) @iljarotar * Migrate to simple (metal-stack/api#50) @majst01 * Update to simple (metal-stack/cli#10) @majst01 * generate unique files in /etc/nvme/ (metal-stack/metal-images#360) @mwennrich * Bump lint-staged from 16.1.6 to 16.2.4 (metal-stack/website#127) @dependabot[bot] * Bump @docusaurus/tsconfig from 3.8.1 to 3.9.2 (metal-stack/website#131) @dependabot[bot] * Bump @scalar/api-reference-react from 0.7.45 to 0.8.1 (metal-stack/website#130) @dependabot[bot] * Add admin tenant cmds. (metal-stack/cli#12) @Gerrit91 * fix-docs-links (metal-stack/cli#7) @BotondGalxc * Update debian and ubuntu kernel (metal-stack/metal-images#362) @majst01 * adjust to debian and add ci (metal-stack/monsoon#2) @ostempel * CAPMS images (metal-stack/metal-images#352) @mac641 * publish capms images as part of release as well (metal-stack/metal-images#363) @mac641 * Fix link to suricata (metal-stack/firewall-controller#206) @GeertJohan * Next release (metal-stack/releases#255) @metal-robot[bot] --- ## v0.22.1 # metal-stack v0.22.1 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.1](https://github.com/metal-stack/releases/releases/tag/v0.22.1) ## General * [Gardener v1.120](https://github.com/gardener/gardener/releases/tag/v1.120.0) ## Noteworthy * In case of outputs that ingest audit logs too slow, the behavior of gardener-extension-audit now depends on the configured webhookMode. The modes `batch` and `blocking` configure fluentbit such that audit log messages may be dropped if the audit log backend cannot ingest them in time. The mode `blocking-strict` instead causes all kube-apiserver requests to fail in this case. Fluentbit buffers up to storage.max_chunks_up * 2M (roughly 900MB). ([release notes](https://github.com/metal-stack/gardener-extension-audit/releases/tag/v0.4.0)) ## Breaking Changes * The `gardener-monitoring-certs` was updated to work with the Gardener operator virtual garden. The parametrization for the role has breaking changes. Please adapt your playbooks accordingly if you use this role. (metal-stack/metal-roles#487) ## Component Releases ### csi-driver-lvm v0.7.0 * Pod eviction controller (metal-stack/csi-driver-lvm#117) @ostempel * Update to go-1.25 (metal-stack/csi-driver-lvm#122) @majst01 * add matrix multiplatform build (metal-stack/csi-driver-lvm#119) @ostempel * include sbom in container image (metal-stack/csi-driver-lvm#118) @mac641 * Bump golang.org/x/net from 0.37.0 to 0.38.0 (metal-stack/csi-driver-lvm#115) @[dependabot[bot]](https://github.com/apps/dependabot) ### gardener-extension-provider-metal v0.27.1 * Allow PVC deletion on eviction with csi-driver-lvm ETCD volumes. (metal-stack/gardener-extension-provider-metal#468) @Gerrit91 * Fix metallb webhook (metal-stack/gardener-extension-provider-metal#475) @Gerrit91 * restrict ccm rbac for nodes (metal-stack/gardener-extension-provider-metal#478) @mwennrich ### gardener-extension-ontap v0.2.9 * Update Torc (metal-stack/gardener-extension-ontap#73) @Honigeintopf ### metal-roles v0.17.25 * Allow configuring csi-driver-lvm ETCD volume evictions on GEPM. (metal-stack/metal-roles#477) @Gerrit91 * Cleanup release vector mapping. (metal-stack/metal-roles#489) @Gerrit91 * Remove valkey migration steps (metal-stack/metal-roles#488) @Gerrit91 * Add capms controller to release vector mapping. (metal-stack/metal-roles#485) @Gerrit91 * Fix shoot template when auditing + csi-driver-lvm extension is enabled. (metal-stack/metal-roles#492) @Gerrit91 * Add `gardener-partition-proxy` role (metal-stack/metal-roles#491) @Gerrit91 * Update `gardener-monitoring-certs` role to work with operator. (metal-stack/metal-roles#487) @Gerrit91 * Adaptions for g/g v1.120 + drop deprecated `gardener` role. (metal-stack/metal-roles#486) @Gerrit91 * Remove debugging output from nsq role. (metal-stack/metal-roles#474) @Gerrit91 * Bump monitoring to match current Gardener version. (metal-stack/metal-roles#476) @Gerrit91 * Fix issue with removal of gardenlet image overwrites. (metal-stack/metal-roles#494) @Gerrit91 * CreateCRDs for shoot-dns-service extension. (metal-stack/metal-roles#495) @Gerrit91 * Update oci-mirror image semver constraints for isolated clusters (metal-stack/metal-roles#496) @mwennrich * Add new csi-driver-lvm-controller image to oci-mirror. (metal-stack/metal-roles#497) @Gerrit91 ### os-metal-extension v0.8.9 * Go 1.25 (metal-stack/os-metal-extension#62) @majst01 * Trim down README.md. (metal-stack/os-metal-extension#61) @Gerrit91 * include sbom in container image (metal-stack/os-metal-extension#60) @mac641 ### duros-controller v0.11.8 * Update duros-go and use go-1.25 (metal-stack/duros-controller#98) @majst01 ### gardener-extension-audit v0.4.0 * Fix storage size overflow (metal-stack/gardener-extension-audit#61) @MichaelEischer ### helm-charts v0.5.1 * Postgreslet next release (metal-stack/helm-charts#134) @eberlep ### droptailer v0.2.19 * Update to go-1.25, debian13 (metal-stack/droptailer#45) @majst01 * include sbom in container image (metal-stack/droptailer#44) @mac641 ### cluster-api-provider-metal-stack v0.6.3 * Add `--namespace` flag to watch only specific namespace if necessary. (metal-stack/cluster-api-provider-metal-stack#114) @Gerrit91 * Cluster K8s Upgrade (metal-stack/cluster-api-provider-metal-stack#105) @simcod * fix(machine): give load balancers some time after control plane machine deletion (metal-stack/cluster-api-provider-metal-stack#115) @vknabel * test: reference new templates (metal-stack/cluster-api-provider-metal-stack#113) @vknabel * Use calico flavor for tests (metal-stack/cluster-api-provider-metal-stack#112) @simcod * Add first flavors for cluster templates. (metal-stack/cluster-api-provider-metal-stack#110) @Gerrit91 * Add Calico `HelmChartProxy` and metal-ccm `ClusterResourceSet` sample. (metal-stack/cluster-api-provider-metal-stack#106) @Gerrit91 * test: teardown and slight refactorings (metal-stack/cluster-api-provider-metal-stack#104) @vknabel * docs: updated readme for current status (metal-stack/cluster-api-provider-metal-stack#109) @vknabel * Integration test for HA cluster setup (metal-stack/cluster-api-provider-metal-stack#103) @simcod * Integration Tests: `clusterctl move` (metal-stack/cluster-api-provider-metal-stack#102) @simcod ### gardener-extension-csi-driver-lvm v0.1.6 * add eviction controller to csi-driver-lvm (metal-stack/gardener-extension-csi-driver-lvm#16) @ostempel * docs: improved docs regarding migration (metal-stack/gardener-extension-csi-driver-lvm#14) @vknabel * Add more netpols required when running in distributed landscapes (metal-stack/gardener-extension-csi-driver-lvm#15) @Gerrit91 * include sbom in container image (metal-stack/gardener-extension-csi-driver-lvm#13) @mac641 * Fix image vector points to wrong csi-driver-lvm container tag. (metal-stack/gardener-extension-csi-driver-lvm#18) @Gerrit91 ### api v0.0.33 * Infra Role Handling and Admin Token Create, better Token documentation (metal-stack/api#69) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Blog article for metal-stack v0.22.0. (metal-stack/website#136) @Gerrit91 * Bump releases to version v0.22.0 (metal-stack/website#137) @metal-robot[bot] * Gardener v1.120 + drop deprecated Gardener parts. (metal-stack/releases#256) @Gerrit91 * Fix link in v0.22.0 blog article. (metal-stack/website#138) @Gerrit91 * Add blog article for OCP hardware support. (metal-stack/website#132) @Gerrit91 * GenericCLI Fix: Commands set to 'false' in OnlyCmds are now correctly disabled (metal-stack/metal-lib#193) @izvyk * Correct grammar of on-premises (metal-stack/website#140) @stmcginnis * fix capms-ubuntu release location (metal-stack/metal-images#368) @mac641 * Firewall: update node_exporter, tailscale and kernel (metal-stack/metal-images#369) @majst01 * Add vendor Gigabyte to supported hardware list (metal-stack/website#141) @simcod * Bump typescript from 5.9.2 to 5.9.3 (metal-stack/website#135) @dependabot[bot] * Bump @headlessui/react from 2.2.7 to 2.2.9 (metal-stack/website#117) @dependabot[bot] * Remove unused Dockerfile and docker-bake.hcl args (metal-stack/metal-images#357) @majst01 * Bump tailwindcss from 4.1.13 to 4.1.16 (metal-stack/website#133) @dependabot[bot] * Update to go-1.25, debian:13 (metal-stack/oci-mirror#18) @majst01 * Stay with frr-10.4.x for now (metal-stack/metal-images#372) @majst01 * use docker-bake tar exporter as default output (metal-stack/metal-images#358) @mac641 * Next attempt to calculate planning meetings table. (metal-stack/website#148) @Gerrit91 * docs(MEP16): FCM contents adjustments (metal-stack/website#142) @vknabel * Bump @tailwindcss/postcss from 4.1.13 to 4.1.17 (metal-stack/website#143) @dependabot[bot] * Bump @docusaurus/types from 3.8.1 to 3.9.2 (metal-stack/website#144) @dependabot[bot] * Bump tailwindcss from 4.1.16 to 4.1.17 (metal-stack/website#145) @dependabot[bot] * chore(deps): bump docusaurus to 3.9.2 (metal-stack/website#149) @vknabel * Update debian kernel to 6.1.0-41 (metal-stack/metal-images#373) @majst01 * Use grype instead of cve-bin-tool (metal-stack/website#150) @majst01 * Downgrade debian 12 kernel because of eBPF bug preventing calico to work (metal-stack/metal-images#374) @majst01 * Next release (metal-stack/releases#257) @metal-robot[bot] --- ## v0.22.10 # metal-stack v0.22.10 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.10](https://github.com/metal-stack/releases/releases/tag/v0.22.10) ## General * [Gardener v1.129](https://github.com/gardener/gardener/releases/tag/v1.129.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Component Releases ### metal-roles v0.20.3 * Make default src address loopback optional on SONiC (metal-stack/metal-roles#563) @iljarotar * adapt oci-mirror-config to newer tags (metal-stack/metal-roles#586) @mwennrich * add xdr provider config if extension is enabled (metal-stack/metal-roles#566) @ulrichSchreiner ### metal-api v0.43.3 * Quick fix for wrong machine connections (metal-stack/metal-api#644) @iljarotar * Use refactored auditing from metal-lib (metal-stack/metal-api#645) @majst01 * Update metal-lib to contain audit fix. (metal-stack/metal-api#647) @Gerrit91 ### metalctl v0.18.9 * metal-go v0.43.2 (metal-stack/metalctl#303) @iljarotar # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Gardener v1.129 (metal-stack/releases#285) @Gerrit91 * fix: update kernel versions for debian and ubuntu targets (copyfail fix) (metal-stack/metal-images#406) @mwennrich * Bump releases to version v0.22.9 (metal-stack/website#257) @metal-robot[bot] * update kernels (dirtyfrag, CVE-2026-43284, CVE-2026-43500) (metal-stack/metal-images#407) @mwennrich * Bump metal-api to version v0.43.2 (metal-stack/metal-python#163) @metal-robot[bot] * Bump metal-api to version v0.43.2 (metal-stack/metal-go#224) @metal-robot[bot] * Add information regarding artifact signing (metal-stack/website#88) @simcod * Forgot to push OCI artifact link not working. (metal-stack/website#264) @Gerrit91 * Go 1.26.3 (metal-stack/builder#92) @majst01 * chore(deps): bump the other-dependencies group across 1 directory with 8 updates (metal-stack/website#265) @dependabot[bot] * Update debian kernel (metal-stack/metal-images#408) @majst01 * Implement Task API (metal-stack/cli#31) @Gerrit91 * Fix auditing misses user in context (metal-stack/metal-lib#207) @majst01 * Bump metal-api to version v0.43.3 (metal-stack/metal-python#164) @metal-robot[bot] * Bump metal-api to version v0.43.3 (metal-stack/metal-go#225) @metal-robot[bot] * Next release (metal-stack/releases#286) @metal-robot[bot] --- ## v0.22.11 # metal-stack v0.22.11 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.11](https://github.com/metal-stack/releases/releases/tag/v0.22.11) ## General * [Gardener v1.130](https://github.com/gardener/gardener/releases/tag/v1.130.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * Some control plane database minor version upgrades for Postgres databases. ## Breaking Changes * Migrated to connectrpc simple, you need to adjust your client implementation. (metal-stack/go-ipam#188) ## Noteworthy * The `metal-control-plane` helm chart can now use Gateway API resources in place of `Ingress` resources and TCP port exposure using `ingress-nginx`. This marks the start of our journey to deprecate the use of `Ingress`. What does that mean for you? Right now, nothing. You will not have to move to Gateway API right now and we will support both `Ingress` and Gateway API during a transition period. During the next releases we will wire up the missing pieces and document a migration process. Now is a great time to familiarize yourself with Gateway API to ensure a smooth transistion If you want to check on the progress or leave feedback, take a look at our umbrella issue here: https://github.com/metal-stack/metal-roles/issues/562 ([release notes](https://github.com/metal-stack/helm-charts/releases/tag/v0.6.1)) ## Component Releases ### metal-roles v0.20.4 * Apply yaml fixes from ansible-lint. (metal-stack/metal-roles#531) @Gerrit91 ### helm-charts v0.6.1 * Use grpc readiness probe for ipam (metal-stack/helm-charts#158) @majst01 * feat(metal-control-plane): add GatewayAPI support (metal-stack/helm-charts#156) @l0wl3vel ### metal-ccm v0.9.10 * Health probe. Again. (metal-stack/metal-ccm#128) @l0wl3vel * feat: expose healthiness and readiness checks against metal-api (metal-stack/metal-ccm#124) @l0wl3vel * Update actions workflows and dependencies. (metal-stack/metal-ccm#123) @Gerrit91 * Go 1.26 (metal-stack/metal-ccm#122) @majst01 * Go 1.26 (metal-stack/metal-ccm#121) @majst01 ### csi-driver-lvm v0.9.0 * Support mount options from the storage class (metal-stack/csi-driver-lvm#155) @tacho * Fix existing VG not being activated (metal-stack/csi-driver-lvm#153) @tacho * Add luks encryption support (metal-stack/csi-driver-lvm#148) @chbmuc * Bump google.golang.org/grpc from 1.79.1 to 1.79.3 in the go_modules group across 1 directory (metal-stack/csi-driver-lvm#151) @[dependabot[bot]](https://github.com/apps/dependabot) * Go 1.26 (metal-stack/csi-driver-lvm#149) @majst01 ### gardener-extension-csi-driver-lvm v0.1.10 * Add LUKS encryption support with encrypted StorageClass variants (metal-stack/gardener-extension-csi-driver-lvm#25) @chbmuc ### go-ipam v1.15.0 * Remove grpc health probe from image (metal-stack/go-ipam#195) @majst01 * Use existing context where possible (metal-stack/go-ipam#194) @majst01 * Migrate to connect-simple (metal-stack/go-ipam#188) @majst01 * Update go modules and gh actions (metal-stack/go-ipam#192) @majst01 * Bump google.golang.org/grpc from 1.78.0 to 1.79.3 in the go_modules group across 1 directory (metal-stack/go-ipam#191) @[dependabot[bot]](https://github.com/apps/dependabot) * Update to go-1.26 (metal-stack/go-ipam#190) @majst01 ### metal-apiserver v0.2.1 * Upgrade go-ipam to v1.15 (metal-stack/metal-apiserver#204) @majst01 * Adopt to api changes where networktype and nattype are not optional (metal-stack/metal-apiserver#192) @majst01 * remove context.background where possible, do not log passwords (metal-stack/metal-apiserver#191) @majst01 * Switch tests datacenter (metal-stack/metal-apiserver#167) @iljarotar * Try metal lib auditing refactoring (metal-stack/metal-apiserver#186) @majst01 * Prevent race access, terminate goroutine on exit (metal-stack/metal-apiserver#190) @majst01 ### gardener-extension-provider-metal v0.27.8 * Adjustments to new metal-ccm health probe. (metal-stack/gardener-extension-provider-metal#498) @Gerrit91 * set correct netpol namespace (metal-stack/gardener-extension-provider-metal#495) @iljarotar # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * update ubuntu kernel to v6.12.89 (metal-stack/metal-images#409) @mwennrich * Update postgres (metal-stack/releases#290) @majst01 * chore: add alloy image (metal-stack/releases#287) @ma-hartma * chore: add alloy helm chart (metal-stack/releases#288) @ma-hartma * Bump releases to version v0.22.10 (metal-stack/website#269) @metal-robot[bot] * Replace unmaintained lunr plugin with local search (metal-stack/website#266) @vknabel * docs(blog): hack the garden no. 8 (metal-stack/website#270) @vknabel * fix(blog): hack the garden typos (metal-stack/website#271) @vknabel * chore(deps-dev): bump lint-staged from 16.4.0 to 17.0.5 (metal-stack/website#268) @dependabot[bot] * Next release (metal-stack/releases#291) @metal-robot[bot] --- ## v0.22.12 # metal-stack v0.22.12 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.12](https://github.com/metal-stack/releases/releases/tag/v0.22.12) ## General * [Gardener v1.131](https://github.com/gardener/gardener/releases/tag/v1.131.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Component Releases ### helm-charts v0.6.2 * Bump csi-driver-lvm chart to v0.9.0. (metal-stack/helm-charts#159) @Gerrit91 ### gardener-extension-ontap v0.2.18 * provide snapshot controller (metal-stack/gardener-extension-ontap#91) @Honigeintopf ### metal-bmc v0.7.2 * Update actions workflows and dependencies. (metal-stack/metal-bmc#94) @Gerrit91 ### metal-roles v0.20.5 * Adaptions for g/g v1.131. (metal-stack/metal-roles#600) @Gerrit91 * Fix typos in gardener-projects role README (metal-stack/metal-roles#598) @simcod ### gardener-extension-provider-metal v0.27.9 * Add missing RBAC for metrics endpoint of metal-ccm. (metal-stack/gardener-extension-provider-metal#499) @Gerrit91 * Build with go 1.26 and run go fix. (metal-stack/gardener-extension-provider-metal#500) @Gerrit91 ### virtual-garden-kubeconfig-refresher v0.1.3 * Updates. (metal-stack/virtual-garden-kubeconfig-refresher#2) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Next release (metal-stack/releases#292) @metal-robot[bot] --- ## v0.22.13 # metal-stack v0.22.13 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.13](https://github.com/metal-stack/releases/releases/tag/v0.22.13) ## General * [Gardener v1.132](https://github.com/gardener/gardener/releases/tag/v1.132.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * This release supports Kubernetes clusters with `v1.34` ## Required Actions * If you explicitly set `secret_binding_name` in the `gardener-shoots` role, migrate to `credentials_binding_name`. (metal-stack/metal-roles#602) ## Component Releases ### oci-mirror v0.3.0 * add retry with exponential backoff (metal-stack/oci-mirror#22) @mwennrich ### os-metal-extension v0.8.11 * chore(deps): update to g/g 1.132.5 (metal-stack/os-metal-extension#64) @vknabel ### gardener-extension-audit v0.6.2 * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-audit#87) @vknabel ### machine-controller-manager-provider-metal v0.1.28 * Revendor MCM v0.60.2. (metal-stack/machine-controller-manager-provider-metal#44) @Gerrit91 ### gardener-extension-dns-powerdns v0.7.4 * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-dns-powerdns#13) @vknabel ### gardener-extension-csi-driver-lvm v0.1.11 * Go 1.26. (metal-stack/gardener-extension-csi-driver-lvm#27) @Gerrit91 * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-csi-driver-lvm#26) @vknabel ### gardener-extension-backup-s3 v0.8.3 * Go 1.26. (metal-stack/gardener-extension-backup-s3#27) @Gerrit91 * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-backup-s3#26) @vknabel ### metal-roles v0.21.2 * Adaptions for g/g v1.132. (metal-stack/metal-roles#602) @Gerrit91 * Set time if unsynchronised so that initial pull of sonic-configdb-utils will not fail (metal-stack/metal-roles#536) @mreiger * Forgot to update shoot field in g/g v1.132 changes. (metal-stack/metal-roles#604) @Gerrit91 * Isolated clusters: Add `coredns-config-adapter` to oci-mirror config. (metal-stack/metal-roles#605) @Gerrit91 ### gardener-extension-provider-metal v0.27.10 * Revendor g/g v1.132. (metal-stack/gardener-extension-provider-metal#501) @Gerrit91 * Update actions workflows and dependencies. (metal-stack/gardener-extension-provider-metal#496) @Gerrit91 ### gardener-extension-ontap v0.2.20 * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-ontap#94) @vknabel * allow dns for snapshot-controller (metal-stack/gardener-extension-ontap#95) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.11 (metal-stack/website#272) @metal-robot[bot] * Bump releases to version v0.22.12 (metal-stack/website#273) @metal-robot[bot] * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-duros#5) @vknabel * chore(deps): update to g/g 1.132.5 (metal-stack/gardener-extension-csi-driver-synology#5) @vknabel * Correctly sort release notes (metal-stack/website#274) @vknabel * Next release (metal-stack/releases#293) @metal-robot[bot] --- ## v0.22.14 # metal-stack v0.22.14 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.14](https://github.com/metal-stack/releases/releases/tag/v0.22.14) ## General * [Gardener v1.133](https://github.com/gardener/gardener/releases/tag/v1.133.0) * ⚠️ Be aware that this version of Gardener does not support garden, seed, or shoot clusters with Kubernetes versions <= 1.29. * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Breaking Changes * With the introduction of the os-installer, provisioning machines with OS images older than `debian-12`, or ubuntu `24.04` is no longer supported. In addition to that, the image builder contract has changed and does not rely on `install.sh` anymore. If you build your own OS images, you can find more information in the metal-stack docs. (metal-stack/metal-hammer#181) ## Noteworthy * With this release we introduce initial support for Gateway API in Metal Stack into metal-roles as well with the goal of removing our dependency on `Ingress` and ingress-nginx. For now `Ingress` will still be supported but will be removed in a future release. You can track the Gateway API progress in our umbrella issue: https://github.com/metal-stack/metal-roles/issues/562 (metal-stack/metal-roles#594) * The new [tenant-apiserver](https://github.com/metal-stack/tenant-apiserver) is now being deployed unconditionally. This microservice will replace the masterdata-api during MEP-4. The two can coexist without any problems. Currently, the new tenant-apiserver is only used when the metal-apiserver is explicitly enabled through deployment. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.21.3)) ## Component Releases ### metal-core v0.16.0 * Fix SetSrcLoopback Config (metal-stack/metal-core#192) @iljarotar * Bump go.opentelemetry.io/otel from 1.40.0 to 1.41.0 in the go_modules group across 1 directory (metal-stack/metal-core#190) @[dependabot[bot]](https://github.com/apps/dependabot) * add test for bgp-state parsing ( frr_test.go ) (metal-stack/metal-core#191) @mwennrich ### tenant-apiserver v0.1.0 * Initial Release after masterdata-api move ### api v0.1.1 * Add tenant-apiserver health (metal-stack/api#129) @majst01 * FSL Match and Try is a admin endpoint (metal-stack/api#128) @majst01 * Fix inconsistencies found by Ai, update deps (metal-stack/api#127) @majst01 * Auditing must be specified explicitly now (metal-stack/api#126) @majst01 * Machine State Command (metal-stack/api#132) @majst01 * Update connect-python to v0.10.1 (metal-stack/api#130) @majst01 * Remove alpha stage note. (metal-stack/api#131) @Gerrit91 ### helm-charts v0.6.3 * Deploy tenant-apiserver (metal-stack/helm-charts#160) @majst01 ### metal-roles v0.21.7 * Deploy tenant-apiserver (metal-stack/metal-roles#608) @majst01 * feat: add gatewayapi support (metal-stack/metal-roles#594) @l0wl3vel * Fix defaulting for HTTP Route in gardener-cloudprofile role. (metal-stack/metal-roles#610) @Gerrit91 * Gardener adapt new cloudprofile spec. (metal-stack/metal-roles#616) @Gerrit91 * add missing gardener-project/releases/gardener/vpn-client to isolated-cluster images (metal-stack/metal-roles#617) @mwennrich * Better default for `metal_tenant_apiserver_db_password`. (metal-stack/metal-roles#618) @Gerrit91 ### metal-apiserver v0.3.0 * Migrate to tenant-api (metal-stack/metal-apiserver#199) @majst01 * Implement FSL Match (metal-stack/metal-apiserver#200) @majst01 * Better ip create validation (metal-stack/metal-apiserver#198) @majst01 * Fix issues ai found (metal-stack/metal-apiserver#203) @majst01 ### metal-hammer v0.15.0 * With apiv2 os installer in chroot (metal-stack/metal-hammer#181) @majst01 ### oci-mirror v0.3.1 * skip copying images that already exist in the destination (metal-stack/oci-mirror#23) @mwennrich ### go-ipam v1.15.1 * Try fix otel memory leak (metal-stack/go-ipam#196) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.13 (metal-stack/website#275) @metal-robot[bot] * update kernels (metal-stack/metal-images#410) @mwennrich * Debian 13 (metal-stack/metal-images#370) @majst01 * chore(deps): bump @scalar/api-reference-react from 0.9.38 to 0.9.41 in the docusaurus-dependencies group (metal-stack/website#276) @dependabot[bot] * chore(deps): bump semver from 7.8.0 to 7.8.1 in the other-dependencies group (metal-stack/website#277) @dependabot[bot] * Make repo ready for release vector. (metal-stack/tenant-api#1) @Gerrit91 * Update to api v0.1.0 (metal-stack/cli#32) @majst01 * Update image builder contract with new os-installer. (metal-stack/website#279) @Gerrit91 * Update docs and github actions (metal-stack/metal-images#413) @majst01 * Update CI part 2 (metal-stack/metal-images#414) @majst01 * Update README (metal-stack/metal-images#405) @simcod * chore(deps): bump the other-dependencies group with 4 updates (metal-stack/website#281) @dependabot[bot] * chore(deps): bump the docusaurus-dependencies group with 3 updates (metal-stack/website#280) @dependabot[bot] * Upgrade Zitadel charts to 10.0.2, Zitadel to v4.14.0 (metal-stack/releases#295) @majst01 * Client namespace interception (metal-stack/tenant-api#4) @Gerrit91 * Next release (metal-stack/releases#294) @metal-robot[bot] --- ## v0.22.15 # metal-stack v0.22.15 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.15](https://github.com/metal-stack/releases/releases/tag/v0.22.15) ## General * [Gardener v1.134](https://github.com/gardener/gardener/releases/tag/v1.134.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * [firewall-controller v2.5.0](https://github.com/metal-stack/firewall-controller/releases/tag/v2.5.0) ## Breaking Changes * The event `job` label was renamed from `monitoring/event-exporter` to `events`. Update any existing LogQL queries, dashboard filters, and alert rules that reference `{job="monitoring/event-exporter"}` to `{job="events"}`. The monitoring role's basic auth configuration for `Thanos Receive Ingress` has changed. The raw `htpasswd` string `monitoring_thanos_receive_ingress_basic_auth` has been replaced by two plaintext variables ` monitoring_thanos_receive_ingress_basic_auth_user: thanos-receive` and `monitoring_thanos_receive_ingress_basic_auth_password: mysecret`. See: https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/monitoring/README.md#thanos-receive-ingress-credentials (metal-stack/metal-roles#595) ## Noteworthy * It is now possible to annotate the `FirewallMonitor` resource with the annotation `firewall.metal-stack.io/restart-systemd-services=` in order to trigger a restart of a systemd service on the firewall. Only whitelisted services can be restarted. (metal-stack/firewall-controller#220) * The `ClusterWideNetworkPolicy` only supports ports of type `int32` now. Before that, we allowed `intstr`. The string value, however, was always prevented by validation such that the transition should not cause any issues. (metal-stack/firewall-controller#219) ## Required Actions * The partition `promtail` role is deprecated and replaced by the new `alloy` role. Operators should migrate to Alloy for partition log collection. Follow the migration instructions: https://github.com/metal-stack/metal-roles/blob/master/partition/roles/alloy/README.md#migration-from-promtail Several migration scenarios are supported — from a hard cutover to a parallel run for verification before switching over. Note that running both services simultaneously will produce duplicate log entries in Loki during the transition window. (metal-stack/metal-roles#592) * The control-plane `logging` role now deploys Grafana `Alloy` as the default log collector. Alloy needs to be configured before running the new version of the role, as the deprecated `Promtail` DaemonSet and the `event-exporter` resources are disabled and removed by default. Follow the migration instructions: - https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#migration-from-promtail Several migration scenarios are supported — from a hard cutover to a parallel run for verification before switching over. Note that running both simultaneously will produce duplicate log entries in Loki during the transition window. (metal-stack/metal-roles#595) ## Component Releases ### metal-apiserver v0.5.0 * Partition response does not include mgmt servers (metal-stack/metal-apiserver#219) @majst01 * Update dependencies (metal-stack/metal-apiserver#218) @majst01 * Fix missing machine allocation uuid query. (metal-stack/metal-apiserver#217) @Gerrit91 * Add default retention to all asynq tasks. (metal-stack/metal-apiserver#216) @Gerrit91 * Provide generic Encode/Decode payload func for asynq package. (metal-stack/metal-apiserver#215) @Gerrit91 * Introduce secure-cookie flag (metal-stack/metal-apiserver#213) @majst01 * Machine State Command (metal-stack/metal-apiserver#211) @majst01 * IP Query for IPType was wrong (metal-stack/metal-apiserver#210) @majst01 * Enhance token validation security (metal-stack/metal-apiserver#207) @majst01 * Fix Partition list (metal-stack/metal-apiserver#209) @majst01 * Fix FSL issues found during api-conformance (metal-stack/metal-apiserver#220) @majst01 * Return deletion task IDs in meta. (metal-stack/metal-apiserver#221) @Gerrit91 * Machine delete and machine update validation (metal-stack/metal-apiserver#197) @majst01 ### metal-bmc v0.7.3 * Update go-hal dependency for powersupply (metal-stack/metal-bmc#97) @simcod ### helm-charts v0.6.4 * Postgreslet next release (metal-stack/helm-charts#164) @eberlep * modify block transaction (metal-stack/helm-charts#163) @emekaponw * Secure Cookie flag for metal-apiserver (metal-stack/helm-charts#162) @majst01 ### api v0.1.3 * Return task IDs in deletion responses. (metal-stack/api#133) @Gerrit91 * Fix issues found during api-conformance (metal-stack/api#134) @majst01 * Admin machine delete (metal-stack/api#135) @majst01 * Introduce `TokenQuery`. (metal-stack/api#111) @Gerrit91 ### firewall-controller-manager v0.6.1 * Add systemd-restart annotation for firewall resource. (metal-stack/firewall-controller-manager#88) @Gerrit91 * fix: correct event message format in createFirewall function (metal-stack/firewall-controller-manager#90) @mwennrich * Fix typos in API constants. (metal-stack/firewall-controller-manager#89) @Gerrit91 ### metal-roles v0.22.0 * chore: replace control-plane promtail with alloy (metal-stack/metal-roles#595) @ma-hartma * chore: replace partition promtail with alloy (metal-stack/metal-roles#592) @ma-hartma * Adadpt sonic-config README for VLAN sub-interfaces (metal-stack/metal-roles#607) @iljarotar * Secure Cookie flag for metal-apiserver (metal-stack/metal-roles#620) @majst01 * Update actions. (metal-stack/metal-roles#619) @Gerrit91 ### sonic-configdb-utils v0.5.0 * allow multiple CIDRs for a VLAN sub-interface (metal-stack/sonic-configdb-utils#40) @iljarotar ### mini-lab v0.6.0 * chore: replace control-plane and partition promtail with alloy (metal-stack/mini-lab#298) @ma-hartma * Cosign verify (metal-stack/mini-lab#308) @Gerrit91 * Update debian, disable secure cookie (metal-stack/mini-lab#304) @majst01 * Update some release vector overwrites. (metal-stack/mini-lab#303) @Gerrit91 * fix: re-introduce bgp restart for community sonic to ensure default route (metal-stack/mini-lab#302) @ma-hartma * feat: allow using local checkouts of dependencies via env var (metal-stack/mini-lab#294) @ma-hartma * fix: install missing frr-reload.service on dell_sonic leaves (metal-stack/mini-lab#295) @ma-hartma * chore: prepare kamaji tests (metal-stack/mini-lab#289) @ma-hartma * chore(deps): use latest nginx-ingress helm chart version (metal-stack/mini-lab#288) @vknabel * add kamaji flavor (metal-stack/mini-lab#280) @mac641 * chore: remove redundant ansible.cfg and hide skipped ansible task output (metal-stack/mini-lab#282) @ma-hartma * make bash shebang use /usr/bin/env for better compatibility across systems (metal-stack/mini-lab#284) @mac641 * Allow disabling monitoring deployment through make param. (metal-stack/mini-lab#283) @Gerrit91 * Deploy monitoring in mini-lab (metal-stack/mini-lab#275) @ostempel * Migrate to new sonic-config role for community SONiC flavor (metal-stack/mini-lab#276) @iljarotar * Add zitadel OIDC (metal-stack/mini-lab#273) @ostempel * sudo-rs does not allow to preserve all env variables (metal-stack/mini-lab#277) @majst01 * Introduce new mini-lab flavor for Enterprise SONiC and fix capi-lab (metal-stack/mini-lab#272) @mwindower # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Remove trailing comma to make release happen (metal-stack/metal-images#416) @majst01 * Fix for CVE-2026-23111 (metal-stack/metal-images#417) @majst01 * Some more small refinements on building own images. (metal-stack/website#284) @Gerrit91 * Bump releases to version v0.22.14 (metal-stack/website#283) @metal-robot[bot] * Update docs. (metal-stack/metal-images#418) @Gerrit91 * chore(deps): bump @scalar/api-reference-react from 0.9.42 to 0.9.45 in the docusaurus-dependencies group (metal-stack/website#287) @dependabot[bot] * chore(deps): bump semver from 7.8.1 to 7.8.4 in the other-dependencies group (metal-stack/website#288) @dependabot[bot] * Collect powersupplies if power metrics are available (metal-stack/go-hal#90) @simcod * Always use terse mode when listing nftables ruleset (metal-stack/nftables-exporter#38) @ConnorsApps * Update deps (metal-stack/nftables-exporter#40) @majst01 * Hello API V2 conformance tests (metal-stack/releases#298) @majst01 * feat: rework flavors explanation (metal-stack/website#285) @l0wl3vel * Add possibility to restart systemd services through annotations (metal-stack/firewall-controller#220) @majst01 * Define own `NetworkPolicyPort` field. (metal-stack/firewall-controller#219) @Gerrit91 * Update go modules and gh actions (metal-stack/firewall-controller#221) @majst01 * Provide GenericCLI e2e test framework. (metal-stack/metal-lib#203) @Gerrit91 * docs(monitoring): use alloy instead of promtail (metal-stack/website#260) @ma-hartma * Ubuntu 26.04 (metal-stack/metal-images#411) @majst01 * whitelist H13SSH boards from bios modifications (metal-stack/go-hal#91) @mwennrich * Update firewall-controller and tools (metal-stack/metal-images#419) @majst01 * Next release (metal-stack/releases#297) @metal-robot[bot] --- ## v0.22.16 # metal-stack v0.22.16 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.16](https://github.com/metal-stack/releases/releases/tag/v0.22.16) ## General * [Gardener v1.135](https://github.com/gardener/gardener/releases/tag/v1.135.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Required Actions * Upgrade headscale from `0.23.0` -> `v0.28.0` with all minor versions in between prior to updating the metal-stack control plane. You can use the following approach: ```bash #!/usr/bin/env bash # async start a loop to fix broken postgres migration (see https://github.com/juanfont/headscale/issues/2351) while true; do k exec headscale-db-0 -- psql -U postgres headscale -c "alter table users add constraint uni_users_name unique (name);"; done # cycle through headscale versions, after every update check the headscale logs and see if it comes up properly NAMESPACE=metal-control-plane for VERSION in v0.24.3 v0.25.1 v0.26.1 v0.27.1 v0.28.0 do echo "upgrade to ${VERSION}" kubectl patch deployments.apps -n "${NAMESPACE}" headscale \ --patch="{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\": \"headscale\",\"image\":\"docker.io/headscale/headscale:${VERSION}\"}]}}}}" kubectl delete pod -n "${NAMESPACE}" -l app=headscale kubectl wait --for=condition=Ready pod -l app=headscale printf "%s " "check that headscale is running properly, press enter to continue" read ans done # recreate headscale api token for metal-api kubectl delete secret headscale-api-key echo "done, now run the metal-stack deployment" ``` Creating firewalls with working VPN does not work until both headscale and metal-stack control plane were updated. There is a possibility that with `v0.27.1` the connected clients loose connectivity. This can be solved by restarting tailscaled on the clients, e.g. firewalls. With https://github.com/metal-stack/firewall-controller/pull/220 this can be done fleet-wide as platform admin from all seeds. ⚠️ Firewall Images must be >= `firewall-ubuntu-3.0.20241106` because headscale `v0.28.0` requires tailscale >= `v1.74.0`. (metal-stack/metal-api#648) * The variable `gardener_managed_seed_default_dns_domain` in the `gardener-managed-seeds` role is now required. It is usually parametrized identical to the `gardener_gardenlet_default_dns_domain` in our environments. If not defined, the role execution will fail. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.23.0)) ## Component Releases ### metal-core v0.16.1 * Migrate to valkey-go (metal-stack/metal-core#184) @majst01 ### metal-api v0.44.1 * Upgrade to Headscale v0.28.0. (metal-stack/metal-api#648) @Gerrit91 * Migrate reserved to tainted (metal-stack/metal-api#646) @majst01 * fix: only enforce machine quota when explicitly configured (avoids default to zero) (metal-stack/metal-api#650) @mwennrich * Fix VPN Error user already exists (metal-stack/metal-api#652) @majst01 ### metal-roles v0.23.2 * Adaptions for g/g v1.135. (metal-stack/metal-roles#629) @Gerrit91 * ansible-lint: fix names. (metal-stack/metal-roles#632) @Gerrit91 * ansible-lint: fix FQCN. (metal-stack/metal-roles#631) @Gerrit91 * ansible-lint: fix key-order. (metal-stack/metal-roles#630) @Gerrit91 * Fix more ansible-lint errors and allow pipeline failure (metal-stack/metal-roles#628) @Gerrit91 * Stabilize `frr-reload` on sonic (metal-stack/metal-roles#603) @iljarotar * Sonic config prefix lists (metal-stack/metal-roles#570) @iljarotar * fix: gardener managed thanos receive ingress tls (metal-stack/metal-roles#606) @ma-hartma * Fix task renaming regression: Apply renaming to handler names (metal-stack/metal-roles#638) @iljarotar ### api v0.1.7 * ProviderTenant Annotation (metal-stack/api#139) @majst01 * Move more tags from metal-lib (metal-stack/api#137) @majst01 * Add ability to configure client from env variables (metal-stack/api#136) @majst01 * add MachineRoom tag (metal-stack/api#138) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.15 (metal-stack/website#292) @metal-robot[bot] * chore(deps): bump @scalar/api-reference-react from 0.9.45 to 0.9.47 in the docusaurus-dependencies group (metal-stack/website#293) @dependabot[bot] * chore(deps): bump the other-dependencies group across 1 directory with 3 updates (metal-stack/website#294) @dependabot[bot] * Bump metal-api to version v0.44.0 (metal-stack/metal-python#165) @metal-robot[bot] * Bump metal-api to version v0.44.0 (metal-stack/metal-go#226) @metal-robot[bot] * Migrate tags to api (metal-stack/metal-lib#205) @majst01 * Almalinux 10 (metal-stack/metal-images#412) @majst01 * Fix build regression. (metal-stack/metal-robot#115) @Gerrit91 * chore: update debian/ubuntu kernel versions (metal-stack/metal-images#420) @mwennrich * feat: document maintenance boundaries and disaster recovery (metal-stack/website#291) @l0wl3vel * feat: add firewall explanation (metal-stack/website#286) @l0wl3vel * Bump metal-api to version v0.44.1 (metal-stack/metal-python#166) @metal-robot[bot] * Bump metal-api to version v0.44.1 (metal-stack/metal-go#227) @metal-robot[bot] * Next release (metal-stack/releases#299) @metal-robot[bot] --- ## v0.22.17 # metal-stack v0.22.17 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.17](https://github.com/metal-stack/releases/releases/tag/v0.22.17) ## General * [Gardener v1.136](https://github.com/gardener/gardener/releases/tag/v1.136.0) * This release supports Kubernetes clusters with v1.35 ## Required Actions * The control-plane `gardener-logging` role now supports Grafana Alloy as the log collector. Alloy needs to be configured before running the new version of the role, as the deprecated `Promtail` DaemonSet is disabled and removed by default. Follow the migration instructions: - https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging-common/README.md#migration-from-promtail Several migration scenarios are supported — from a hard cutover to a parallel run for verification before switching over. Note that running both simultaneously will produce duplicate log entries in Loki during the transition window. (metal-stack/metal-roles#622) * Only for operators who deployed the metal-apiserver prematurely: The valkey deployment helm chart was removed and instead wrapped into deployment manifests using the backup-restore-sidecar project for backups. Therefore, it is necessary to uninstall the old valkey chart prior to the deployment using `helm delete valkey`. Existing tokens will not work anymore and need to be re-issued. (metal-stack/metal-roles#645) ## Breaking Changes * The deprecated `docker-on-cumulus` role was removed. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.23.3)) ## Component Releases ### metal-roles v0.25.0 * Remove `docker-on-cumulus` role. (metal-stack/metal-roles#641) @Gerrit91 * ansible-lint: Fix no-handler, package-latest, risky-octal and risky-shell-pipe (metal-stack/metal-roles#640) @Gerrit91 * chore: use print_message plugin for promtail deprecation (metal-stack/metal-roles#639) @ma-hartma * Allow passing security opts to alloy container. (metal-stack/metal-roles#649) @Gerrit91 * chore: replace gardener promtail with alloy (metal-stack/metal-roles#622) @ma-hartma * remove calico overwrites (metal-stack/metal-roles#646) @mwennrich * Adaptions for g/g v1.136 (metal-stack/metal-roles#651) @Gerrit91 * Fix regression that calico image overwrite was completely removed. (metal-stack/metal-roles#650) @Gerrit91 * Deploy valkey with backup-restore-sidecar. (metal-stack/metal-roles#645) @Gerrit91 ### metal-apiserver v0.5.1 * Prevent machine registering on other switches than before. (metal-stack/metal-apiserver#224) @Gerrit91 * Adapt to tenant query. (metal-stack/metal-apiserver#227) @Gerrit91 * Reserved is now migrated to tainted (metal-stack/metal-apiserver#226) @majst01 ### ansible-common v0.9.0 * feat: `print_message` action plugin (metal-stack/ansible-common#47) @ma-hartma * feat(systemd-docker-service): allow optional `--security-opt` parameters (metal-stack/ansible-common#48) @ma-hartma ### backup-restore-sidecar v0.13.0 * ONTAP Compatibility v2 (metal-stack/backup-restore-sidecar#141) @simcod * Update docs information (metal-stack/backup-restore-sidecar#142) @simcod * Fix postgres incompatibilities for database upgrades (metal-stack/backup-restore-sidecar#143) @simcod * Upgrade to go-1.26, run go fix, update proto (metal-stack/backup-restore-sidecar#137) @majst01 * Update to go-1.25 and most go modules (metal-stack/backup-restore-sidecar#133) @majst01 * Use absolute links (metal-stack/backup-restore-sidecar#129) @BotondGalxc * feat: add Valkey database support (metal-stack/backup-restore-sidecar#126) @olehoerb ### gardener-extension-provider-metal v0.27.11 * Revendor g/g v1.136. (metal-stack/gardener-extension-provider-metal#504) @Gerrit91 * chore(deps): update dependencies (metal-stack/gardener-extension-provider-metal#503) @mwennrich ### machine-controller-manager-provider-metal v0.1.29 * Revendor MCM v0.61.2. (metal-stack/machine-controller-manager-provider-metal#45) @Gerrit91 ### helm-charts v0.7.0 * Remove valkey chart. (metal-stack/helm-charts#166) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Fix broken link in README (metal-stack/firewall-controller#224) @simcod * Bump releases to version v0.22.16 (metal-stack/website#302) @metal-robot[bot] * Bump to gardener v1.136.7. (metal-stack/releases#300) @Gerrit91 * Debian 13 (metal-stack/frr#15) @majst01 * Zitadel 4.15.3 (metal-stack/releases#302) @majst01 * Update apiv2 (metal-stack/releases#303) @majst01 * Removed EOL switch from supported hardware list (metal-stack/website#306) @Sven-Ric * Ensure go code is fmted (metal-stack/actions-common#10) @majst01 * Fix release freeze is not always working. (metal-stack/metal-robot#114) @Gerrit91 * Update dependencies. (metal-stack/chrony#5) @Gerrit91 * Update actions. (metal-stack/actions-common#12) @Gerrit91 * Next release (metal-stack/releases#301) @metal-robot[bot] --- ## v0.22.18 # metal-stack v0.22.18 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.18](https://github.com/metal-stack/releases/releases/tag/v0.22.18) ## General * [Gardener v1.137](https://github.com/gardener/gardener/releases/tag/v1.137.0) ## Noteworthy * The backup-restore-sidecar can now be parametrized to work with ONTAP-provided S3 storage. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.26.1)) ## Required Actions * The metal-apiserver deployment from **[MEP-4](https://metal-stack.io/community/MEP-4-multi-tenancy-for-the-metal-api) is now enabled by default**. It will run in parallel with the metal-api until all clients have been migrated to the metal-stack V2 API. This process will take place over the course of many releases, allowing time for migration. For the metal-apiserver, it is required to add the [Valkey](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/valkey) into your control plane deployment playbook. In addition, we always deploy [Zitdadel](https://github.com/metal-stack/metal-roles/tree/master/control-plane/roles/zitadel) into the control plane for authentication, in order to provide a working OIDC flow without external dependencies. While this is not strictly required, we advise installing Zitadel and setting up a federation with external OIDC providers if necessary. We do this because it allows us to easily test the setup locally in the mini-lab, and we therefore keep it identical in real environments. You can check out the mini-lab to see how to set up Valkey and Zitadel. For Zitadel and the metal-apiserver **we recommend to expose them through [Gateway API](https://kubernetes.io/docs/concepts/services-networking/gateway/)**. More detailed information will be added to the release notes of this release in the next couple of days. The metal-apiserver does not yet provide a stable API, so many things are still subject to change. While it makes sense to prepare for the migration of your clients, we advise against doing this immediately with this release, but rather waiting for a few more releases until the API has stabilized. We welcome your feedback at any time! (metal-stack/metal-roles#652) * For operators who already deployed metal-apiserver prior to this release: The field `metal_apiserver_admin_subjects` was removed and should be cleaned up. Admins are now managed through the provider tenant that is automatically ensured on startup of the metal-apiserver. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.26.0)) * In [#633](https://github.com/metal-stack/metal-roles/pull/633) changes were introduced for the s3 backend configuration used by the `backup-restore-sidecar` component. Unfortunately, the defaults for the Ansible variable `*_backup_restore_sidecar_object_max_keep` were removed in the process. So if the corresponding `*_backup_restore_sidecar_object_max_keep` variable for the metal-stack databases (`ipam-db`, `masterdata-db`, `metal-db`) are not explicitly defined in your deployment, the backup lifecycle rule configuration might be modified unexpectedly. So before rolling out metal-stack `v0.22.18` please consider setting those variables explicitly in your deployment. If you want to use the defaults configure e.g. `ipam_db_backup_restore_sidecar_object_max_keep: 480` The defaults will be added back again with metal-roles pull request [#699](https://github.com/metal-stack/metal-roles/pull/633). ## Component Releases ### metal-roles v0.27.2 * ProviderTenant adoptions for metal-apiserver (metal-stack/metal-roles#642) @majst01 * ONTAP compatibility v2 (metal-stack/metal-roles#633) @simcod * Enable metal-apiserver deployment by default. (metal-stack/metal-roles#652) @Gerrit91 * ansible-lint: Fix risky-file-permissions (metal-stack/metal-roles#643) @Gerrit91 * Rename zitadel-init to zitadel-metal-init. (metal-stack/metal-roles#654) @Gerrit91 * Utilize reusable workflow from actions-common. (metal-stack/metal-roles#653) @Gerrit91 * Add new toggles for ingress rule deployment (metal-stack/metal-roles#656) @Gerrit91 * Fix partition directory permissions partially wrong (metal-stack/metal-roles#657) @Gerrit91 * Also enable zitadel login gateway httproute. (metal-stack/metal-roles#655) @Gerrit91 ### helm-charts v0.7.4 * ProviderTenant adoptions for metal-apiserver (metal-stack/helm-charts#165) @majst01 * Passing Splunk HEC token to metal-apiserver is missing. (metal-stack/helm-charts#168) @Gerrit91 * Add toggles to conditionally deploy ingress rules (metal-stack/helm-charts#169) @Gerrit91 * Do not forget to conditionally set tls hostnames. (metal-stack/helm-charts#170) @Gerrit91 ### metal-apiserver v0.6.3 * Add machine-roles handling (metal-stack/metal-apiserver#236) @majst01 * Fix some AI findings (metal-stack/metal-apiserver#235) @majst01 * Fix token create bug. (metal-stack/metal-apiserver#222) @Gerrit91 * Inherit admins from provider tenant membership (metal-stack/metal-apiserver#225) @majst01 * Write rate-limiter-interceptor tests (metal-stack/metal-apiserver#234) @majst01 * Machine Issues (metal-stack/metal-apiserver#230) @majst01 * Admin Tenant add-member (metal-stack/metal-apiserver#229) @majst01 * Project query in-line with tenant query (metal-stack/metal-apiserver#232) @majst01 * Fmt code (metal-stack/metal-apiserver#233) @majst01 * Implement `UpdateLabelStrategy` for update labels. (metal-stack/metal-apiserver#179) @Gerrit91 * Add timestamp to audit entry. (metal-stack/metal-apiserver#241) @Gerrit91 * Adapt to v0.2.2 API. (metal-stack/metal-apiserver#240) @Gerrit91 * Remove creation of default project for provider tenant. (metal-stack/metal-apiserver#244) @Gerrit91 * Adopt to Labels Update Strategie Patch (metal-stack/metal-apiserver#243) @majst01 ### virtual-garden-kubeconfig-refresher v0.1.4 * Dependency updates. (metal-stack/virtual-garden-kubeconfig-refresher#4) @Gerrit91 * Generic Garden Cluster Access (metal-stack/virtual-garden-kubeconfig-refresher#3) @simcod ### api v0.2.2 * Add `UpdateLabelStrategy` option to label updates. (metal-stack/api#121) @Gerrit91 * Fix typo. (metal-stack/api#147) @Gerrit91 ### zitadel-init v0.3.4 * Fix application already exists status code changed. (metal-stack/zitadel-init#10) @Gerrit91 **Full Changelog**: https://github.com/metal-stack/zitadel-init/compare/v0.3.3...v0.3.4 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.17 (metal-stack/website#308) @metal-robot[bot] * Update to changed tenant and project query (metal-stack/cli#34) @majst01 * e2e test framework moved to generic cli. (metal-stack/cli#28) @Gerrit91 * Add size API cmds. (metal-stack/cli#35) @Gerrit91 * chore(deps): bump the other-dependencies group with 3 updates (metal-stack/website#304) @dependabot[bot] * chore(deps): bump @scalar/api-reference-react from 0.9.47 to 0.9.50 in the docusaurus-dependencies group across 1 directory (metal-stack/website#303) @dependabot[bot] * chore(deps): bump actions/checkout from 6 to 7 in the actions group across 1 directory (metal-stack/website#311) @dependabot[bot] * Adaption for api v0.2.2. (metal-stack/cli#36) @Gerrit91 * Add `container-build.yaml` action. (metal-stack/actions-common#11) @Gerrit91 * chore(deps): bump the other-dependencies group with 4 updates (metal-stack/website#314) @dependabot[bot] * Add `oci-release-artifact.yaml` action (metal-stack/actions-common#13) @Gerrit91 * chore: update index of MEPs (metal-stack/website#313) @vknabel * Adopt to labels update strategie (metal-stack/cli#37) @majst01 * Update release drafter configuration. (metal-stack/.github#5) @Gerrit91 * Update tailscale and debian and ubuntu kernels (metal-stack/metal-images#424) @majst01 * Next release (metal-stack/releases#304) @metal-robot[bot] --- ## v0.22.19 # metal-stack v0.22.19 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.19](https://github.com/metal-stack/releases/releases/tag/v0.22.19) ## General * [Gardener v1.138](https://github.com/gardener/gardener/releases/tag/v1.138.0) ## Required Actions * Prior to rolling out this release of Gardener, remove the deprecated `enableAnonymousAuthentication` field from the `Garden` resource (if present). ## Component Releases ### go-ipam v1.15.2 * Update etcd and actions (metal-stack/go-ipam#197) @majst01 ### droptailer v0.2.21 * Update dependencies (metal-stack/droptailer#49) @majst01 ### oci-mirror v0.3.2 * Update go deps and ci actions (metal-stack/oci-mirror#25) @majst01 ### virtual-garden-kubeconfig-refresher v0.1.5 * Provide raw GARDEN_CLUSTER_KUBECONFIG (metal-stack/virtual-garden-kubeconfig-refresher#5) @simcod ### api v0.3.0 * Python client tests (metal-stack/api#155) @majst01 * Fix token create/update validation (metal-stack/api#154) @majst01 * Slight improvements on membership endpoints. (metal-stack/api#159) @Gerrit91 * Service permissions performance (metal-stack/api#157) @majst01 * Add room to machine and switch (metal-stack/api#156) @majst01 * Tokenfile reread duration defaulting was wrong (metal-stack/api#158) @majst01 ### metal-apiserver v0.7.1 * Store admin token in secret (metal-stack/metal-apiserver#231) @majst01 * Permutate token-create tests (metal-stack/metal-apiserver#238) @majst01 * Implement token delete in token repository. (metal-stack/metal-apiserver#254) @Gerrit91 * Implement token get and find in token repository. (metal-stack/metal-apiserver#253) @Gerrit91 * Implement token list in token repository. (metal-stack/metal-apiserver#252) @Gerrit91 * Add stub for token repository. (metal-stack/metal-apiserver#251) @Gerrit91 * Return proto errors from token store and improve errors in general (metal-stack/metal-apiserver#250) @Gerrit91 * Extend token store model to contain `labels` and `updated_at` (metal-stack/metal-apiserver#248) @Gerrit91 * Move max token expiration to certs package. (metal-stack/metal-apiserver#249) @Gerrit91 * Return aborted error on concurrent resource modification. (metal-stack/metal-apiserver#246) @Gerrit91 * Use errorutil from API package. (metal-stack/metal-apiserver#247) @Gerrit91 * Report correct valkey version (metal-stack/metal-apiserver#242) @majst01 * Prevent redirect url manipulation (metal-stack/metal-apiserver#245) @majst01 * Move token update and token refresh to repository. (metal-stack/metal-apiserver#264) @Gerrit91 * Move token creation without perm check to repository. (metal-stack/metal-apiserver#263) @Gerrit91 * Add more authorizer tests to ensure machine scoping works (metal-stack/metal-apiserver#259) @majst01 * Enable more tests since proto validation is adopted (metal-stack/metal-apiserver#258) @majst01 * Move user token creation to repository. (metal-stack/metal-apiserver#261) @Gerrit91 * Fix missing repo wiring in admin token service. (metal-stack/metal-apiserver#257) @Gerrit91 ### helm-charts v0.7.5 * Create apiserver admintoken (metal-stack/helm-charts#167) @majst01 ### metal-roles v0.28.1 * Read admin token from secret instead of execing into metal-apiserver. (metal-stack/metal-roles#669) @Gerrit91 * Store admin token in secret (metal-stack/metal-roles#660) @majst01 * Enable thanos receive HA mode (metal-stack/metal-roles#666) @Sven-Ric * Remove unused OIDC variables for Gardener (metal-stack/metal-roles#663) @simcod * Configuration parameter for control-plane monitoring (metal-stack/metal-roles#659) @simcod * Fix cross-referenced headscale variable (metal-stack/metal-roles#662) @simcod * Headscale ingress cleanup task. (metal-stack/metal-roles#671) @Gerrit91 * Simplify V2 python client installation. (metal-stack/metal-roles#667) @Gerrit91 * feat(monitoring): allow disabling prometheus operator (metal-stack/metal-roles#331) @vknabel ### metal-console v0.7.6 * chore(deps): update (metal-stack/metal-console#65) @vknabel ### metalctl v0.19.0 * Update go modules and gh actions (metal-stack/metalctl#309) @majst01 * Add reason flag for firewall SSH. (metal-stack/metalctl#308) @Gerrit91 * Allow port toggle for unconnected ports (metal-stack/metalctl#305) @iljarotar * add --room to switch and machine commands (metal-stack/metalctl#304) @mwennrich ### metal-api v0.45.0 * Add admin status to nic port (metal-stack/metal-api#655) @iljarotar * add RoomID to switches and machines (metal-stack/metal-api#651) @mwennrich * Adopt to go-ipam v1.15.x (metal-stack/metal-api#653) @majst01 ### zitadel-init v0.3.5 * Fix again application creation not idempotent. (metal-stack/zitadel-init#12) @Gerrit91 ### metal-core v0.17.0 * check admin status instead of actual status (metal-stack/metal-core#197) @iljarotar # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * g/g v1.138 (metal-stack/releases#305) @Gerrit91 * Bump releases to version v0.22.18 + only 3 latest minor release notes (metal-stack/website#320) @vknabel * docs: remove archived docs (metal-stack/website#321) @vknabel * chore: improved redirects (metal-stack/website#322) @vknabel * chore(deps): bump @scalar/api-reference-react from 0.9.50 to 0.9.54 in the docusaurus-dependencies group (metal-stack/website#318) @dependabot[bot] * chore(deps-dev): bump typescript from 6.0.3 to 7.0.2 (metal-stack/website#319) @dependabot[bot] * Use tag ref on tag push (metal-stack/actions-common#14) @Gerrit91 * Fix invalid bash condition. (metal-stack/actions-common#15) @Gerrit91 * Fix overwriting tag variable by mistake. (metal-stack/actions-common#16) @Gerrit91 * Attempt to fix release asset upload. (metal-stack/actions-common#17) @Gerrit91 * docs: Update link to the nftables-exporter (metal-stack/firewall-controller#225) @chbmuc * feat: initial implementation (metal-stack/metal-token-refresher#1) @vknabel * feat: metal-token-refresher (metal-stack/releases#307) @vknabel * Bump metal-api to version v0.45.0 (metal-stack/metal-go#229) @metal-robot[bot] * Bump metal-api to version v0.45.0 (metal-stack/metal-python#167) @metal-robot[bot] * Update node-exporter and droptailer (metal-stack/metal-images#427) @majst01 * chore(deps): bump the docusaurus-dependencies group with 8 updates (metal-stack/website#324) @dependabot[bot] * Update apiv2 dependency (metal-stack/releases#308) @majst01 * Bump golang.org/x/net from 0.39.0 to 0.55.0 in the go_modules group across 1 directory (metal-stack/metallb-health-sidecar#5) @dependabot[bot] * feat: add new components (metal-stack/website#327) @vknabel * chore(deps): bump the other-dependencies group across 1 directory with 3 updates (metal-stack/website#325) @dependabot[bot] * docs: document the OS image lifecycle and provisioning model (metal-stack/website#307) @l0wl3vel * Doc/network inventory management (metal-stack/website#295) @Sven-Ric * docs(network): add network segmentation and policy boundaries article (metal-stack/website#296) @Sven-Ric * chore: update debian/ubuntu kernel versions (metal-stack/metal-images#430) @mwennrich * Speedup single stream network performance (metal-stack/metal-images#429) @majst01 * Next release (metal-stack/releases#306) @metal-robot[bot] --- ## v0.22.2 # metal-stack v0.22.2 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.2](https://github.com/metal-stack/releases/releases/tag/v0.22.2) ## General * [Gardener v1.121](https://github.com/gardener/gardener/releases/tag/v1.121.0) ## Noteworthy * Starting with this release the cluster-api-provider-metal-stack supports the automatic deployment of static firewalls. All default templates are now adjusted accordingly. You can try this out by using the new `MetalStackFirewallTemplate` and `MetalStackFirewallDeployment` resources and reference the deployment within `MetalStackCluster` using the new `spec.firewallDeploymentRef.name` field. (metal-stack/cluster-api-provider-metal-stack#118) * Gracefully handle audit event batches larger than 4MB by splitting them into smaller batches and limiting the size of individual audit events to 4MB. (metal-stack/gardener-extension-audit#65) ## Required Actions * It is now possible to resolve metal-stack OCI release vectors and ansible-roles using the `metal_stack_release_vector` module that is now by default included in the metal-deployment-base image. Please check the the [README](https://github.com/metal-stack/metal-deployment-base/blob/master/README.md) or the latest mini-lab as an example for how to use it. Old deployments still continue to work, the old `setup_yaml` module will become deprecated over time. `setup_yaml` will be deprecated with metal-stack v0.23.0. (metal-stack/metal-deployment-base#40) ## Component Releases ### gardener-extension-ontap v0.2.10 * update version to 25.06.3 (metal-stack/gardener-extension-ontap#75) @Honigeintopf ### metal-ccm v0.9.9 * Update to go-1.25, debian:13 (metal-stack/metal-ccm#120) @majst01 ### gardener-extension-provider-metal v0.27.2 * Fix metallb webhook certificate. (metal-stack/gardener-extension-provider-metal#481) @Gerrit91 ### api v0.0.35 * Fix admin token create (metal-stack/api#72) @majst01 * Migrate to mockery v3 (metal-stack/api#73) @majst01 * Add admin project service (metal-stack/api#74) @AnnaSchreiner ### metal-hammer v0.13.16 * Downgrade u-root to enable ctrl-c again (metal-stack/metal-hammer#174) @majst01 ### ansible-common v0.7.4 * Provide `metal_stack_release_vector` module. (metal-stack/ansible-common#39) @Gerrit91 * Template module variables in action plugin. (metal-stack/ansible-common#41) @Gerrit91 ### metal-apiserver v0.1.0 * Authorize with expanded token permissions (metal-stack/metal-apiserver#94) @majst01 * add project/tenant leave endpoint (metal-stack/metal-apiserver#92) @mac641 * Implement switch infra get and heartbeat endpoints (metal-stack/metal-apiserver#84) @iljarotar * Implement admin tenant list. (metal-stack/metal-apiserver#88) @Gerrit91 * Add Admin Token Service (metal-stack/metal-apiserver#87) @majst01 ### metal-deployment-base v0.9.0 * Ship `metal_stack_release_vector` module. (metal-stack/metal-deployment-base#40) @Gerrit91 * sshpass is required to use passwords with the ssh connection plugin (metal-stack/metal-deployment-base#46) @robertvolkmann ### metal-roles v0.17.27 * Cleanup old gardener images from before operator migration. (metal-stack/metal-roles#499) @Gerrit91 * Fix improper API version field in namespace definition. (metal-stack/metal-roles#498) @Gerrit91 * Add oci-mirror to image vector mapping. (metal-stack/metal-roles#501) @Gerrit91 ### mini-lab v0.5.0 * Use `metal_stack_release_vector` module (metal-stack/mini-lab#247) @Gerrit91 * update mini-lab illustration (metal-stack/mini-lab#274) @AnnaSchreiner * feat: allow capms-ubuntu in fsl (metal-stack/mini-lab#264) @vknabel * Ansible update for deployment base image v0.8.0. (metal-stack/mini-lab#263) @Gerrit91 * Fix flakiness of virtual SONiC tests (metal-stack/mini-lab#262) @mwindower * Differentiate SONiC base image versions and use 202505 (metal-stack/mini-lab#260) @mwindower * Update firewall creation command in README (metal-stack/mini-lab#258) @dhilgarth * README.md: Add jq to apt install (metal-stack/mini-lab#257) @GeertJohan * Hotfix using archive.debian.org for bullseye-backports on SONiC. (metal-stack/mini-lab#253) @Gerrit91 * Add sonic base image workflow (metal-stack/mini-lab#252) @ostempel * added sonic base image workflow (metal-stack/mini-lab#251) @ostempel * Remove Cumulus (metal-stack/mini-lab#246) @robertvolkmann * Create proxy registries after kind cluster. (metal-stack/mini-lab#248) @Gerrit91 * Implement a health check for SONiC (metal-stack/mini-lab#195) @robertvolkmann * Pin ingress-nginx controller helm chart due to regression. (metal-stack/mini-lab#245) @Gerrit91 * Adapt to latest metal-roles. (metal-stack/mini-lab#244) @Gerrit91 * Add Headscale support (metal-stack/mini-lab#243) @majst01 * Don't cancel each other's job in test matrix. (metal-stack/mini-lab#240) @Gerrit91 * include sboms in container images (metal-stack/mini-lab#242) @mac641 * Gardener operator (metal-stack/mini-lab#239) @Gerrit91 * feat: remove unused meilisearch (metal-stack/mini-lab#241) @vknabel * Recreate update strategy for ingress-nginx (metal-stack/mini-lab#235) @Gerrit91 * Add pull-through caches for common container registries (metal-stack/mini-lab#227) @ostempel * Use OpenIPMI for managing the power of the virtual machines. (metal-stack/mini-lab#223) @robertvolkmann * Use new flag --abort-on-container-failure (metal-stack/mini-lab#238) @robertvolkmann * Inline IP masquerade for SONiC switches (metal-stack/mini-lab#237) @iljarotar * docs: include missing step for make dev-env (metal-stack/mini-lab#226) @vknabel ### cluster-api-provider-metal-stack v0.7.0 * Static Firewall Deployments (metal-stack/cluster-api-provider-metal-stack#118) @vknabel ### metal-core v0.14.1 * Update go dependencies (metal-stack/metal-core#174) @majst01 * Allow different VTEP name than `vtep` (metal-stack/metal-core#163) @iljarotar * Do not try to cleanup default VRF (metal-stack/metal-core#170) @iljarotar * Fix ethernet packet overflow (metal-stack/metal-core#176) @majst01 ### gardener-extension-audit v0.4.1 * Limit audit log event size to prevent dropped events (metal-stack/gardener-extension-audit#65) @MichaelEischer ### metal-api v0.42.5 * Upgrade security with jwx v3 (metal-stack/metal-api#636) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * First set of refactorings. (metal-stack/metal-robot#91) @Gerrit91 * Bump releases to version v0.22.1 (metal-stack/website#152) @metal-robot[bot] * Remove release vector repos to require CONTRIBUTING.md. (metal-stack/website#159) @Gerrit91 * Bump lint-staged from 16.2.4 to 16.2.7 (metal-stack/website#153) @dependabot[bot] * Bump semver from 7.7.2 to 7.7.3 (metal-stack/website#154) @dependabot[bot] * Bump axios from 1.12.2 to 1.13.2 (metal-stack/website#157) @dependabot[bot] * Bump @carbon/icons-react from 11.67.0 to 11.70.0 (metal-stack/website#156) @dependabot[bot] * Bump react-dom from 19.1.1 to 19.2.0 (metal-stack/website#155) @dependabot[bot] * Fix for ethernet packet overflow (metal-stack/go-lldpd#32) @majst01 * Add admin project list (metal-stack/cli#13) @AnnaSchreiner * add log message (metal-stack/zitadel-init#1) @ostempel * make init idempotent and initialize entities (metal-stack/zitadel-init#3) @ostempel * add code-owners and license (metal-stack/zitadel-init#4) @ostempel * gardener hackathon blog article (metal-stack/website#161) @Honigeintopf * Svg bg in dark mode (metal-stack/website#160) @AnnaSchreiner * Migrate to jwx v3 (metal-stack/security#64) @majst01 * Bump metal-api to version v0.42.5 (metal-stack/metal-python#160) @metal-robot[bot] * Bump metal-api to version v0.42.5 (metal-stack/metal-go#221) @metal-robot[bot] * Bump go-github to v79. (metal-stack/metal-robot#96) @Gerrit91 * Deprecate Swagger in contribution guideline. (metal-stack/website#167) @Gerrit91 * Upgrade to go 1.25.5 (metal-stack/builder#87) @thheinel * add possibility to configure http client for oidc provider (metal-stack/goth#1) @ostempel * Bump tailwind-merge from 3.3.1 to 3.4.0 (metal-stack/website#166) @dependabot[bot] * Bump yaml from 2.8.1 to 2.8.2 (metal-stack/website#162) @dependabot[bot] * Bump @carbon/icons-react from 11.70.0 to 11.71.0 (metal-stack/website#164) @dependabot[bot] * Bump react from 19.2.0 to 19.2.1 (metal-stack/website#163) @dependabot[bot] * change default provider to openid-connect (metal-stack/cli#14) @ostempel * Allow connecting through kubernetes service instead of ingress. (metal-stack/zitadel-init#5) @Gerrit91 * fix issue where line-breaks in rendered docs were missing (metal-stack/website#169) @mac641 * Bump tailwindcss from 4.1.17 to 4.1.18 (metal-stack/website#172) @dependabot[bot] * Bump @scalar/api-reference-react from 0.8.1 to 0.8.11 (metal-stack/website#171) @dependabot[bot] * Bump @tailwindcss/postcss from 4.1.17 to 4.1.18 (metal-stack/website#175) @dependabot[bot] * Bump react from 19.2.1 to 19.2.3 (metal-stack/website#173) @dependabot[bot] * chore(deps): bump google.golang.org/protobuf from 1.36.10 to 1.36.11 (metal-stack/terraform-provider-metal#268) @dependabot[bot] * Bump @scalar/api-reference-react from 0.8.11 to 0.8.15 (metal-stack/website#176) @dependabot[bot] * Announcement for FOSDEM 2026 participation. (metal-stack/website#177) @Gerrit91 * Next release (metal-stack/releases#258) @metal-robot[bot] --- ## v0.22.20 # metal-stack v0.22.20 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.20](https://github.com/metal-stack/releases/releases/tag/v0.22.20) ## General * [Gardener v1.139](https://github.com/gardener/gardener/releases/tag/v1.139.0) * This release was integration tested against the control plane that fully uses Gateway API and without anymore `Ingress` resources. We recommend to migrate to Gateway API for this release. ## Breaking Changes * It is now possible to configure `NoncurrentDays` also via flag, as follow up from #141. The s3 backend can be configured with `NewerNoncurrentVersions` as well as `NoncurrentDays`. Both can be used together, independently or completely removed from the lifecycle rule. The always applied default for `NewerNoncurrentVersions` was removed. So make sure to configure it accordingly, if not already configured in your deployment. (metal-stack/backup-restore-sidecar#148) ## Required Actions * ⚠️ Make sure the seed clusters are updated to at least K8s 1.32 before rolling out this Gardener release. Gardener writes that >= 1.30 seeds are supported but these lower versions cannot be reconciled because the perses-operator CRDs cannot be applied. * The monsoon sonic-exporter was superseded and will be removed in a future release. Use gNMI instead. More information about the migration can be found in the gnmic [README](https://github.com/metal-stack/metal-roles/blob/master/partition/roles/monitoring/gnmic/README.md). (metal-stack/metal-roles#670) ## Noteworthy * - We now create a dedicated internal mini-lab docker network (mini_lab_internal) to enable explicit reproducible IP assignments for Service type: LoadBalancer. We reserve the 172.42.0.1/16 subnet for the mini_lab_internal network. mini-lab will manage the lifecycle of this network and will destroy it during `make down`. - We made changes to the certificates. If you deployed mini-lab before you need to run `make roll-certs` to update host names in the certificates. (metal-stack/mini-lab#299) * In [#633](https://github.com/metal-stack/metal-roles/pull/633) changes were introduced for the s3 backend configuration used by the `backup-restore-sidecar` component. Unfortunately, the defaults for the Ansible variable `*_backup_restore_sidecar_object_max_keep` were removed in the process. So if the corresponding `*_backup_restore_sidecar_object_max_keep` variable for the metal-stack databases (`ipam-db`, `masterdata-db`, `metal-db`) are not explicitly defined in your deployment, the backup lifecycle rule configuration will be modified unexpectedly. So before rolling out metal-stack `v0.22.18` please consider setting those variables explicitly in your deployment. If you want to use the defaults configure e.g. `ipam_db_backup_restore_sidecar_object_max_keep: 480` The defaults will be added back again with metal-roles pull request [#699](https://github.com/metal-stack/metal-roles/pull/633). (metal-stack/metal-roles#699) ## Component Releases ### metal-ansible-modules v0.3.2 * Update v2 modules to latest API. (metal-stack/metal-ansible-modules#26) @Gerrit91 * Act on latest created resource when using identifier label. (metal-stack/metal-ansible-modules#27) @Gerrit91 * Use abstract base class to bundle common v2 module behavior + tests (metal-stack/metal-ansible-modules#29) @Gerrit91 ### virtual-garden-kubeconfig-refresher v0.1.6 * Use STATIC_KUBECONFIG as environment variable name (metal-stack/virtual-garden-kubeconfig-refresher#7) @simcod ### metal-roles v0.30.0 * Deploy gardener metrics exporter again (metal-stack/metal-roles#599) @simcod * Increase default length of metal-apiserver session secret. (metal-stack/metal-roles#674) @Gerrit91 * Optional loki deployment (metal-stack/metal-roles#672) @simcod * Enable TLS verification by default (metal-stack/metal-roles#677) @simcod * fix(alloy): allow setting docker network (metal-stack/metal-roles#676) @ma-hartma * Adaptions for g/g v1.139. (metal-stack/metal-roles#682) @Gerrit91 * sonic-exporter to gnmic migration (metal-stack/metal-roles#670) @Sven-Ric, @Gerrit91 * feat(monitoring): allow separate ingress annotations for grafana and alertmanager (metal-stack/metal-roles#685) @ma-hartma * Update partition-proxy image tag. (metal-stack/metal-roles#686) @Gerrit91 * Add missing s3 object max keep backup lifecycle defaults (metal-stack/metal-roles#699) @simcod * Enable vpn in metal-apiserver (metal-stack/metal-roles#687) @majst01 * mgmt-server: render ssh_config hosts in stable order (metal-stack/metal-roles#689) @GeertJohan * mgmt-server: escape the pubkey in the authorized_keys regexp (metal-stack/metal-roles#690) @GeertJohan * Fix handler notify/listen case mismatches (metal-stack/metal-roles#692) @GeertJohan ### backup-restore-sidecar v0.14.0 * fix: do not remove redis dump #145 (metal-stack/backup-restore-sidecar#146) @vknabel * Replace s3/manager with s3/transfermanager (metal-stack/backup-restore-sidecar#147) @simcod * Make NewerNoncurrentVersions optional in lifecycle policy for s3 backend (metal-stack/backup-restore-sidecar#148) @simcod ### helm-charts v0.7.7 * Enable TLS verification by default (metal-stack/helm-charts#172) @simcod * Enable vpn in metal-apiserver (metal-stack/helm-charts#173) @majst01 * Modify Postgres pg_stat_statement query (metal-stack/helm-charts#171) @emekaponw, @eberlep ### metal-apiserver v0.9.1 * Enable TLS verification by default (metal-stack/metal-apiserver#278) @simcod ### api v0.4.4 * Update typescript dependencies (metal-stack/api#169) @majst01 * Add test interceptor to Python client. (metal-stack/api#170) @Gerrit91 * Add missing update labels for images. (metal-stack/api#172) @Gerrit91 ### tenant-apiserver v0.1.1 * Create v7 uuids for resources. (metal-stack/tenant-apiserver#5) @Gerrit91 ### metal-hammer v0.15.2 * Strict error checking on umount, report back to metal-api (metal-stack/metal-hammer#190) @majst01 * build with go-hal@add-H13SSH (metal-stack/metal-hammer#184) @mwennrich * Fix chrony config of firewalls (metal-stack/metal-hammer#192) @majst01 ### mini-lab v0.7.0 * feat: add gateway-api support (metal-stack/mini-lab#299) @l0wl3vel * Raise Garden cluster K8s version to 1.35. (metal-stack/mini-lab#323) @Gerrit91 * Disable metal-apiserver TLS verification (metal-stack/mini-lab#319) @simcod * Add a make target to export admin editor token (metal-stack/mini-lab#314) @majst01 * Example for metal-roles override (metal-stack/mini-lab#313) @simcod * Fix route reconciliation by enabling fpm (metal-stack/mini-lab#315) @l0wl3vel * Latest adaptions for MEP-4. (metal-stack/mini-lab#311) @Gerrit91 * Export metal-apiserver url (metal-stack/mini-lab#307) @majst01 ### duros-controller v0.12.2 * Update lb-csi and csi sidecars (metal-stack/duros-controller#102) @majst01 ### gardener-extension-ontap v0.2.21 * Documentation ontap (metal-stack/gardener-extension-ontap#98) @Honigeintopf * Update go dependencies and ci actions (metal-stack/gardener-extension-ontap#99) @majst01 * provide ontap VolumeSnapshotClass (metal-stack/gardener-extension-ontap#97) @mwennrich * use dedicated names for snapshot-controller rbac, to not collide with existing rbac (metal-stack/gardener-extension-ontap#96) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update ubuntu kernel to 6.18.40 (metal-stack/metal-images#431) @majst01 * Better support for protoyaml (printer and multi-document reader) (metal-stack/metal-lib#210) @Gerrit91 * Bump releases to version v0.22.19 (metal-stack/website#329) @metal-robot[bot] * chore(deps): bump the docusaurus-dependencies group with 4 updates (metal-stack/website#330) @dependabot[bot] * chore(deps): bump the other-dependencies group across 1 directory with 4 updates (metal-stack/website#331) @dependabot[bot] * Use protoyaml for edit command (metal-stack/metal-lib#212) @Gerrit91 * Use protoyaml for everything by default and update to latest API (metal-stack/cli#38) @Gerrit91 * Partition (metal-stack/cli#19) @majst01 * Update containerd on debian (metal-stack/metal-images#425) @majst01 * Remove iptables and fever (metal-stack/metal-images#432) @majst01 * Generate admin docs. (metal-stack/cli#43) @Gerrit91 * Admin token create (metal-stack/cli#41) @majst01 * Unify update meta and labels + slight refactorings. (metal-stack/cli#42) @Gerrit91 * Add gnmic to the release vector (metal-stack/releases#313) @Sven-Ric * VPN commands. (metal-stack/cli#44) @Gerrit91 * Fix empty chrony config if ntpservers is empty (metal-stack/os-installer#9) @majst01 * Network commands (metal-stack/cli#45) @Gerrit91 * cleanup completions (metal-stack/cli#46) @majst01 * Debian kernel 6.12.100 (metal-stack/metal-images#434) @majst01 * Next release (metal-stack/releases#310) @metal-robot[bot] --- ## v0.22.21 # metal-stack v0.22.21 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.21](https://github.com/metal-stack/releases/releases/tag/v0.22.21) ## General * [Gardener v1.140](https://github.com/gardener/gardener/releases/tag/v1.140.0) * This release was integration tested against the control plane that fully uses Gateway API and without anymore Ingress resources. We recommend to migrate to Gateway API for this release. ## Noteworthy * Add environment variables support to cosign verify. (metal-stack/ansible-common#50) * Prometheus in the partition monitoring role can now be configured with additional labels. As some systems use `replica` while others use `__replica__` that may conflict, the label: `replica: '{{ inventory_hostname }}'` has been added to the defaults. If you make use of this configuration option and want to keep the `replica` label, make sure to include it explicitly. (metal-stack/metal-roles#702) ## Breaking Changes * In the `mgmt-server` role the misspelled `mgmt_server_routerid` is renamed to `mgmt_server_router_id`. Update your variables accordingly. The role will error on missing variable. (metal-stack/metal-roles#626) ## Component Releases ### csi-driver-lvm v0.9.1 * Support setting mount options via HelmChart (metal-stack/csi-driver-lvm#160) @serveric ### metal-roles v0.31.1 * fix: change default apiserver admin token expiry to 16h (metal-stack/metal-roles#715) @mwennrich * Fix mgmt-server FRR installation (metal-stack/metal-roles#668) @iljarotar * fix(mgmt-server): use mgmt_server_router_id in frr.conf template (metal-stack/metal-roles#626) @GeertJohan * mgmt-server: add optional static routes to the frr.conf template (metal-stack/metal-roles#688) @GeertJohan * add roomID variable to metal-core service (metal-stack/metal-roles#625) @mwennrich * Add metal-core interface naming schema (metal-stack/metal-roles#717) @iljarotar * Partition Prometheus Additional External Labels (metal-stack/metal-roles#702) @simcod * Service account secret for virtual-garden-kubeconfig-refresher (metal-stack/metal-roles#713) @simcod * Run the test workflow on pull requests (metal-stack/metal-roles#711) @GeertJohan ### metal-core v0.18.0 * Add option for SONiC interface naming schema (metal-stack/metal-core#208) @iljarotar * Some cleanup and formatting (metal-stack/metal-core#210) @iljarotar * register with RoomID (metal-stack/metal-core#193) @mwennrich * Support additional routes to the management network gateway (metal-stack/metal-core#204) @GeertJohan ### helm-charts v0.7.8 * Sync csi-driver-lvm chart v0.9.1. (metal-stack/helm-charts#176) @Gerrit91 ### ansible-common v0.9.1 * add existing env for proxy support, if set (metal-stack/ansible-common#50) @TLINDEN ### gardener-extension-provider-metal v0.27.12 * Update dependencies. (metal-stack/gardener-extension-provider-metal#507) @Gerrit91 * fix: change calico bpfConnectTimeLoadBalancing from default "TCP" to "Enabled", to also allow udp-connection from host to kubernetes services (metal-stack/gardener-extension-provider-metal#506) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update gardener to 1.140.5 (metal-stack/releases#314) @majst01 * Fix Reconcile on Dualstack Firewalls (metal-stack/firewall-controller#227) @majst01 * Add link to management server iso generation repository (metal-stack/website#341) @muhittink * Indicate the secrets used in the reusable workflows (metal-stack/actions-common#19) @Gerrit91 * Update to latest actions. (metal-stack/actions-common#20) @Gerrit91 * Allow to optionally pass a registry password. (metal-stack/actions-common#21) @Gerrit91 * Fix MEP-8 identity crisis (metal-stack/website#340) @GeertJohan * Remove install-go (metal-stack/metal-images#426) @majst01 * Update go deps and actions (metal-stack/security#68) @majst01 * Update go deps and actions (metal-stack/metal-lib#213) @majst01 * Fix only completion (metal-stack/cli#50) @majst01 * Admin IP command (metal-stack/cli#47) @majst01 * Allow to set firewall vpn ip (metal-stack/metal-lib#214) @majst01 * chore(deps): bump the other-dependencies group across 1 directory with 6 updates (metal-stack/website#338) @dependabot[bot] * Add `machine` cmd. (metal-stack/cli#51) @Gerrit91 * Prune components (metal-stack/cli#53) @majst01 * Metal stack v0.22.20 with sync fixes (metal-stack/website#343) @vknabel * chore: bump deps (metal-stack/website#344) @vknabel * FI-TS C5 blogpost (metal-stack/website#342) @philiphaas * Next release (metal-stack/releases#315) @metal-robot[bot] --- ## v0.22.22 # metal-stack v0.22.22 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.22](https://github.com/metal-stack/releases/releases/tag/v0.22.22) ## General * [Gardener v1.141](https://github.com/gardener/gardener/releases/tag/v1.141.0) - ⚠️ Gardener does no longer support Garden, Seed, or Shoot clusters with Kubernetes versions <= 1.30. Make sure to upgrade all existing clusters before upgrading to this Gardener version. * This release was integration tested against the control plane that fully uses Gateway API and without anymore Ingress resources. We recommend to migrate to Gateway API for this release. ## Required Actions * The metal-image-cache-sync is the first component to be migrated to use the V2 metal-stack API by communicating with the metal-apiserver. To deploy it, please take care of the following tasks: - Add the role `metal-roles/common/roles/metal-v2-client` to your partition playbook to install the Python API V2 client into the deployment runner. - Make sure the metal-apiserver is running in the control-plane. - Ensure that the new variables `defaults_partition_metal_apiserver_url` and `defaults_partition_metal_apiserver_admin_token` are provided through inventory. We advice to provide the partition deployment token by manually creating it through the metalctlv2 CLI and storing it ansible-vaulted in the inventory. Renewal cycles of this token are up to the deployment operator. (metal-stack/metal-roles#453) ## Breaking Changes * The metal-image-cache-sync has been migrated to the metal-stack V2 API. This means that it is no longer possible to connect to the metal-api, but instead to the new metal-apiserver, which was introduced in an earlier release. Please check the metal-roles release notes and adapt your deployment accordingly. (metal-stack/metal-image-cache-sync#21) * The `metal_apiserver_session_secret` variable is now required. Make sure that it set in your deployment and has a length >= 10 characters. (metal-stack/metal-roles#725) * `dhcp_relay_src_intf` and `dhcp_relay_server_vrf` are no longer set automatically when `dhcp_servers` are defined but must be set explicitly. (metal-stack/sonic-configdb-utils#50) ## Component Releases ### gardener-extension-dns-powerdns v0.8.0 * Fix RBACs for runtime clusters. (metal-stack/gardener-extension-dns-powerdns#16) @Gerrit91 * Update README and actions. (metal-stack/gardener-extension-dns-powerdns#17) @Gerrit91 ### api v0.5.1 * Make switch nic mac optional (metal-stack/api#182) @iljarotar * Machine BMC List must have MachineQuery instead of MachineBMCQuery (metal-stack/api#179) @majst01 * PlacementLabels, and b64 encoded userdata (metal-stack/api#180) @majst01 * If tokenfile instead of token is specified, no reread happens (metal-stack/api#178) @majst01 * Fix js generate (metal-stack/api#177) @iljarotar * Fix punctuation in docstrings (metal-stack/api#175) @majst01 * Add generic Typescript client (metal-stack/api#171) @majst01 * Fix npm publish (metal-stack/api#183) @iljarotar ### metal-apiserver v0.10.0 * Allow nil mac for switch nics (metal-stack/metal-apiserver#287) @iljarotar * Update err msg to match check (metal-stack/metal-apiserver#284) @eberlep * Fix machine bmc list (metal-stack/metal-apiserver#285) @majst01 * Adapt to latest API (placement labels + b64 userdata). (metal-stack/metal-apiserver#286) @Gerrit91 * Update protovalidate (metal-stack/metal-apiserver#283) @majst01 * Update deps, remove replaces (metal-stack/metal-apiserver#281) @majst01 * Allow changing port status of unconnected ports (metal-stack/metal-apiserver#282) @iljarotar * Add image labels. (metal-stack/metal-apiserver#279) @Gerrit91 ### metal-image-cache-sync v0.2.0 * Migrate to V2 API (metal-stack/metal-image-cache-sync#21) @majst01 * Update actions workflows and dependencies. (metal-stack/metal-image-cache-sync#26) @Gerrit91 * include sbom in container image (metal-stack/metal-image-cache-sync#20) @mac641 * Migrate to golangci lint v2 and satisfy new warnings (metal-stack/metal-image-cache-sync#19) @majst01 ### sonic-configdb-utils v0.6.1 * Allow adding src_intf and server_vrf to dhcp_relay (metal-stack/sonic-configdb-utils#45) @iljarotar * Allow adding additional numbered interfaces (metal-stack/sonic-configdb-utils#44) @iljarotar * Use substring matching for version checking for latest 202211 sonic version (metal-stack/sonic-configdb-utils#35) @mreiger * Dependabot mongo driver 1 17 7 (metal-stack/sonic-configdb-utils#49) @iljarotar * Cleanup vlan dhcp_relay (metal-stack/sonic-configdb-utils#48) @iljarotar * Dhcp relay src intf and server vrf (metal-stack/sonic-configdb-utils#47) @iljarotar ### metal-roles v0.32.1 * Migrate metal-image-cache-sync to V2 API (metal-stack/metal-roles#453) @majst01 * Make `metal_apiserver_session_secret` required. (metal-stack/metal-roles#725) @Gerrit91 * Add ansible variable to configure OIDC ID for metal-apiserver (metal-stack/metal-roles#718) @simcod, @Gerrit91 * Add configuration options for auditing and valkey (metal-stack/metal-roles#701) @simcod * fix(isolated clusters): add cilium to oci-mirror-config (metal-stack/metal-roles#735) @mwennrich * Refactor gardener-metrics-exporter and refresher configuration options (metal-stack/metal-roles#726) @simcod * Remove unused gardener ingress nginx variable. (metal-stack/metal-roles#727) @Gerrit91 * Increase default token expiration for image-cache. (metal-stack/metal-roles#731) @Gerrit91 ### metal-token-refresher v0.1.0 * refactor(tests): drop mocks in favor of interceptors (metal-stack/metal-token-refresher#5) @vknabel * Github actions permissions for dependabot. (metal-stack/metal-token-refresher#3) @Gerrit91 * build(deps): bump github.com/metal-stack/api from 0.2.3 to 0.2.6 in the metal-stack group (metal-stack/metal-token-refresher#2) [@dependabot[bot]](https://github.com/apps/dependabot) * chore(ci): minimal permissions for spell-check workflow (metal-stack/metal-token-refresher#4) @vknabel # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Generic enum completion, switch os in list view (metal-stack/cli#54) @majst01 * Adapt to API v0.5.0 (metal-stack/cli#55) @Gerrit91 * Gardener 1.141.6 (metal-stack/releases#317) @vknabel * Bump releases to version v0.22.21 (metal-stack/website#345) @metal-robot[bot] * Update Ubuntu kernel to v6.18.44 (metal-stack/metal-images#439) @mwennrich * Go 1.27 (metal-stack/cli#57) @majst01 * Next release (metal-stack/releases#316) @metal-robot[bot] --- ## v0.22.23 # metal-stack v0.22.23 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.23](https://github.com/metal-stack/releases/releases/tag/v0.22.23) ## General * This release was integration tested against the control plane that fully uses Gateway API and without anymore Ingress resources. We recommend to migrate to Gateway API for this release. ## Breaking Changes * [OPERATOR] ⚠️ Gardener does no longer support Garden, Seed, or Shoot clusters with Kubernetes version `1.31`. Make sure to upgrade all existing clusters before upgrading to this Gardener version. (metal-stack/releases#321) * Cluster API Provider metal-stack drops support for Kubernetes `1.32`. ## Required Actions * The logging components can now be migrated to Gateway API. You can use the following new variables: `logging_ingress_enabled`: false `logging_httproute_enabled`: true `logging_httproute_parent_refs`: (metal-stack/metal-roles#730) * The metal-core has been migrated to use the V2 metal-stack API. If you already deployed the migration of the metal-image-cache-sync from the [v0.22.22 release](https://github.com/metal-stack/releases/releases/tag/v0.22.22), you can simply run the deployment playbook without any further parametrization. Do not forget to add the V2 client installation using the role `metal-roles/common/roles/metal-v2-client` in the playbook. (metal-stack/metal-roles#506) * The monitoring components can now be migrated to Gateway API. You can use the following new variables to directly migrate prometheus, grafana, alertmanager and thanos: `monitoring_prometheus_ingress_enabled`: false `monitoring_grafana_ingress_enabled`: false `monitoring_alertmanager_ingress_enabled`: false `monitoring_thanos_receive_ingress_enabled`: false `monitoring_prometheus_httproute_enabled`: false # by default does not need exposal `monitoring_prometheus_httproute_parent_refs`: `monitoring_grafana_httproute_enabled`: true `monitoring_grafana_httproute_parent_refs`: `monitoring_alertmanager_httproute_enabled`: true `monitoring_alertmanager_httproute_parent_refs`: `monitoring_thanos_receive_httproute_enabled`: true `monitoring_thanos_receive_parent_refs`: If you use basic auth: `monitoring_alertmanager_deploy_envoy_gateway_basic_auth`: true `monitoring_thanos_deploy_envoy_gateway_basic_auth`: true (metal-stack/metal-roles#729) ## Component Releases ### gardener-extension-backup-s3 v0.8.5 * Allow accessing S3 service inside the Garden Cluster. (metal-stack/gardener-extension-backup-s3#31) @Gerrit91 * Update README and actions. (metal-stack/gardener-extension-backup-s3#32) @Gerrit91 * Fix RBAC (metal-stack/gardener-extension-backup-s3#35) @simcod * add handling of backupentry to fix #29 (metal-stack/gardener-extension-backup-s3#30) @christianhuening ### metal-roles v0.34.1 * Allow configuring additional netpol for backup-s3 extension. (metal-stack/metal-roles#721) @Gerrit91 * feat: loki upgrade, persistence and retention (metal-stack/metal-roles#647) @ma-hartma * migrate metal-core to api v2 (metal-stack/metal-roles#506) @iljarotar * Allow configuring Gateway API for `logging` role (metal-stack/metal-roles#730) @Gerrit91 * Allow configuring Gateway API for `monitoring` role (metal-stack/metal-roles#729) @Gerrit91 ### metal-core v0.19.0 * Migrate to v2 (metal-stack/metal-core#169) @iljarotar ### api v0.5.3 * Add external application component type (metal-stack/api#185) @majst01 ### tenant-apiserver v0.2.1 * Validate tenant member exists before creation. (metal-stack/tenant-apiserver#7) @Gerrit91 * Migrate to go-1.27 (metal-stack/tenant-apiserver#6) @majst01 * Use unique api package (metal-stack/tenant-apiserver#8) @majst01 * Add call duration logging (metal-stack/tenant-apiserver#9) @majst01 ### metal-apiserver v0.11.1 * Support component type external application (metal-stack/metal-apiserver#298) @majst01 * Set replace mode to operational if not set (metal-stack/metal-apiserver#288) @iljarotar * Log token uuid and duration of call (metal-stack/metal-apiserver#295) @majst01 * Admin machine delete, aka delete from database (metal-stack/metal-apiserver#292) @majst01 * No more panics (metal-stack/metal-apiserver#293) @majst01 * Migrate to jwx v4 (metal-stack/metal-apiserver#291) @majst01 * Go 1.27 (metal-stack/metal-apiserver#289) @majst01 * Adopt to tenant-api with unique proto (metal-stack/metal-apiserver#297) @majst01 ### ansible-common v0.9.2 * Update actions + remove mini-lab workaround to detect loadbalancer port. (metal-stack/ansible-common#51) @Gerrit91 ### backup-restore-sidecar v0.14.1 * Go 1.27 (metal-stack/backup-restore-sidecar#149) @majst01 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Sort components (metal-stack/cli#56) @majst01 * Bump releases to version v0.22.22 (metal-stack/website#347) @metal-robot[bot] * chore(deps): bump @scalar/api-reference-react from 0.9.63 to 0.9.65 in the docusaurus-dependencies group (metal-stack/website#346) @dependabot[bot] * Disable sbom generation for debian-nvidia (metal-stack/metal-images#442) @majst01 * chore(logging): upgrade loki to v18.5.4 of the community helm charts (metal-stack/releases#311) @ma-hartma * update debian and ubuntu kernel (metal-stack/metal-images#441) @mwennrich * Update go dependencies, replace docker with moby (metal-stack/metal-images#440) @majst01 * Make proto package unique (metal-stack/tenant-api#5) @majst01 * Update all postgres database patch versions (metal-stack/releases#320) @majst01 * Upgrade debian and ubuntu kernels (metal-stack/metal-images#444) @majst01 * frr-10.7.1 for debian and ubuntu (metal-stack/metal-images#436) @majst01 * Add a test for the testinterceptor (metal-stack/tenant-api#6) @majst01 * chore(gardener): v1.142.6 (metal-stack/releases#321) @vknabel * Next release (metal-stack/releases#319) @metal-robot[bot] --- ## v0.22.3 # metal-stack v0.22.3 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.3](https://github.com/metal-stack/releases/releases/tag/v0.22.3) ## General * [Gardener v1.122](https://github.com/gardener/gardener/releases/tag/v1.122.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Component Releases ### gardener-extension-csi-driver-lvm v0.1.9 * Bump csi-driver-lvm to `v0.8.1` (metal-stack/gardener-extension-csi-driver-lvm#22) @Gerrit91 * transfer to only-daemonset-architecture (metal-stack/gardener-extension-csi-driver-lvm#20) @ostempel * Add gardener network-policy labels (metal-stack/gardener-extension-csi-driver-lvm#23) @mwennrich * Bump csi-driver-lvm to v0.8.2. (metal-stack/gardener-extension-csi-driver-lvm#24) @Gerrit91 ### helm-charts v0.5.3 * Sync csi-driver-lvm chart to `v0.8.1` (metal-stack/helm-charts#140) @Gerrit91 ### go-ipam v1.14.14 * Update to debian:13 and all go modules (metal-stack/go-ipam#187) @majst01 ### duros-controller v0.11.9 * Add gardener network-policy labels to controller (metal-stack/duros-controller#99) @mwennrich ### api v0.0.38 * Add vpn-service (metal-stack/api#76) @majst01 * Add last_sync and last_sync_error to switch (metal-stack/api#85) @iljarotar ### metal-roles v0.17.29 * Update Github workflows actions. (metal-stack/metal-roles#508) @Gerrit91 * Fix readme and some defaults in the sonic-config role (metal-stack/metal-roles#504) @iljarotar * Undocumented variable `metal_control_plane_host_provider` (metal-stack/metal-roles#507) @AnnaSchreiner ### sonic-configdb-utils v0.4.4 * Only configure ports and breakouts if they are explicitly passed (metal-stack/sonic-configdb-utils#36) @iljarotar * Fix nil pointer when breakouts are defined and ports are not (metal-stack/sonic-configdb-utils#37) @iljarotar ### metal-core v0.15.0 * Get ports from running config (metal-stack/metal-core#177) @iljarotar * log error if lldp discovery fails (metal-stack/metal-core#175) @iljarotar ### metal-hammer v0.13.17 * Add AMD Workstation support (metal-stack/metal-hammer#176) @majst01 ### metal-bmc v0.6.1 * Dependency updates (metal-stack/metal-bmc#91) @Gerrit91 * Fix regex parsing of lease file mixing up lease entries (metal-stack/metal-bmc#90) @Gerrit91 ### gardener-extension-provider-metal v0.27.3 * Revendor g/g v1.122. (metal-stack/gardener-extension-provider-metal#484) @Gerrit91 ### gardener-extension-audit v0.4.2 * Add gardener network-policy label to audittailer (metal-stack/gardener-extension-audit#69) @mwennrich ### gardener-extension-ontap v0.2.11 * Add gardener network-policy labels to controller (metal-stack/gardener-extension-ontap#78) @Gerrit91 * remove update password (metal-stack/gardener-extension-ontap#76) @Honigeintopf # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Next release (metal-stack/releases#190) @fits-robot[bot] * Add zitadel (metal-stack/releases#260) @ostempel * Bump releases to version v0.22.2 (metal-stack/website#178) @metal-robot[bot] * fix cloud-hypervisor download url (metal-stack/metal-images#378) @mac641 * Merge pr.yaml and master.yaml into docker.yaml (metal-stack/metal-images#376) @mac641 * ci: fix issue where SEMVER_PATCH was assigned an empty string (metal-stack/metal-images#379) @mac641 * Continue with refactor (metal-stack/metal-robot#95) @Gerrit91 * Bump @scalar/api-reference-react from 0.8.15 to 0.8.20 (metal-stack/website#180) @dependabot[bot] * Bump @carbon/icons-react from 11.71.0 to 11.72.0 (metal-stack/website#179) @dependabot[bot] * fix: pin containerd version to 2.1.5 to ensure compatibility with gardener-node-agent (metal-stack/metal-images#380) @mwennrich * Newer AMD Workstation motherboard also requires less bios modifications (metal-stack/go-hal#81) @majst01 * implement tagging release images and copying them to google bucket in go (metal-stack/metal-images#365) @mac641 * Implement shared context command in genericCLI (metal-stack/metal-lib#194) @izvyk * add missing copy destination for gcs on master branch (metal-stack/metal-images#381) @mac641 * Add Supermicro AMD Workstation Support (metal-stack/website#181) @majst01 * Rename v1 to v2 (metal-stack/cli#15) @iljarotar * remove last v1 remainders (metal-stack/cli#16) @iljarotar * frr-10.4.1 (metal-stack/frr#13) @majst01 * Update all dependencies (metal-stack/metal-networker#124) @majst01 * Bump @scalar/api-reference-react from 0.8.20 to 0.8.27 (metal-stack/website#182) @dependabot[bot] * Bump @carbon/icons-react from 11.72.0 to 11.73.0 (metal-stack/website#183) @dependabot[bot] * Pin firewall to frr-10.4.1 (metal-stack/metal-images#382) @majst01 * Bump gardener-extension-networking-cilium to upstream `v1.42.3`. (metal-stack/releases#264) @Gerrit91 * Next release (metal-stack/releases#261) @metal-robot[bot] --- ## v0.22.4 # metal-stack v0.22.4 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.4](https://github.com/metal-stack/releases/releases/tag/v0.22.4) ## General * [Gardener v1.123](https://github.com/gardener/gardener/releases/tag/v1.123.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * Virtual Garden `v1.33.7` ## Noteworthy * When using the `cluster-forwarding` audit backend in combination with `blocking-strict` policy, the controller now lowers the audit policy mode to `blocking` in order to prevent a scenario that produces a deadlock with a crashing kube-apiserver. Since `cluster-forwarding` is not intended for production use cases, this behavior is appropriate. Note that the policy mode is also lowered when using `cluster-forwarding` in combination with another backend. (metal-stack/gardener-extension-audit#71) ## Breaking Changes * The `metal_stack_release_vector` module now uses `oci_registry_credentials` to provide registry credentials instead of `oci_registry_username` and `oci_registry_password`. This way, it is possible to pass credentials for multiple registries. (metal-stack/ansible-common#43) * CSI-Plugin architecture changed from a split controller-daemonset to an only `DaemonSet` architecture. Each `DaemonSet` now also provides its own controller (figure 3 of the possible csi-plugin architectures [here](https://github.com/container-storage-interface/spec/blob/master/spec.md#architecture)). With this architecture it is not necessary anymore to spawn pods for volume provisioning and deletion. For users of the Helm chart and the gardener-extension-csi-driver-lvm this migration can be deployed seamlessly. Manually deployed csi-driver-lvm installations need to adapt to the new architecture and delete the existing controller stateful set. (metal-stack/csi-driver-lvm#128) * There are some breaking changes in the `monitoring` roles in order to make them compatible with the mini-lab. If you deployed monitoring components using this role, please review the changes of introduced in metal-stack/metal-roles#502 and adapt your deployment parameters accordingly. (metal-stack/metal-roles#502) * connect.OutBand now requires to specify a connection timeout for redfish calls, if nil is specified it defaults to 10secs. (metal-stack/go-hal#82) ## Actions Required - In case you set the `NodeAgentAuthorizer` feature gate on the gardenlet, this now has to be removed. ## Component Releases ### oci-mirror v0.2.9 * Update go modules and gh actions (metal-stack/oci-mirror#19) @majst01 ### metal-core v0.15.1 * Fix waitgroup panics with negative counter (metal-stack/metal-core#179) @majst01 ### ansible-common v0.8.0 * Allow login with cosign. (metal-stack/ansible-common#43) @Gerrit91 * Add missing documentation on `helm_additional_params`. (metal-stack/ansible-common#45) @Gerrit91 ### metal-deployment-base v0.9.1 * Bump metal stack release vector plugin. (metal-stack/metal-deployment-base#47) @Gerrit91 ### csi-driver-lvm v0.8.2 * Install kustomize to local bin. (metal-stack/csi-driver-lvm#146) @Gerrit91 * Fix eviction controller evicts volume on VPA eviction (metal-stack/csi-driver-lvm#145) @Gerrit91 * Add warning admonition on working with local data. (metal-stack/csi-driver-lvm#143) @Gerrit91 ### metal-api v0.43.0 * Provide splunk audit configuration. (metal-stack/metal-api#641) @Gerrit91 * Provide reason field for retrieving VPN auth key. (metal-stack/metal-api#640) @Gerrit91 * Use newer approach to use sync.WaitGroup which prevents leaking counters (metal-stack/metal-api#637) @majst01 * Include consolepassword endpoint into auditing. (metal-stack/metal-api#638) @Gerrit91 ### helm-charts v0.5.4 * Sync csi-driver-lvm chart to `v0.8.2` (metal-stack/helm-charts#146) @Gerrit91 * Add splunk audit configuration for metal-api. (metal-stack/helm-charts#144) @Gerrit91 ### metalctl v0.18.4 * Allow field to provide a reason for a VPN key. (metal-stack/metalctl#293) @Gerrit91 * Update to go-1.25 (metal-stack/metalctl#291) @majst01 * fix issue where binary name was missing in generated sbom (metal-stack/metalctl#290) @mac641 ### gardener-extension-audit v0.5.0 * Move S3 secret key ref constants to public API package. (metal-stack/gardener-extension-audit#72) @Gerrit91 * Prevent `blocking-strict` with cluster-forwarding backend. (metal-stack/gardener-extension-audit#71) @Gerrit91 ### gardener-extension-provider-metal v0.27.4 * Revendor g/g v1.123. (metal-stack/gardener-extension-provider-metal#486) @Gerrit91 ### metal-roles v0.18.4 * Adaptions for running monitoring in the mini-lab (metal-stack/metal-roles#502) @ostempel * Configurable vali storage capacity (metal-stack/metal-roles#516) @simcod * Add splunk audit configuration for metal-api. (metal-stack/metal-roles#514) @Gerrit91 * Frr reload on all sonic switches (metal-stack/metal-roles#479) @iljarotar * Fix wrong `metal_registry_url` name (metal-stack/metal-roles#515) @AnnaSchreiner * Zitadel role: some leftovers from the last review. (metal-stack/metal-roles#505) @Gerrit91 * Add configurable `metal_registry_url` (metal-stack/metal-roles#509) @AnnaSchreiner * Make nsq tls requirement for client connections configurable (metal-stack/metal-roles#513) @AnnaSchreiner * Modify nsq and postgres backup restore namespace creation (metal-stack/metal-roles#511) @AnnaSchreiner * Monitoring promtail client configuration (metal-stack/metal-roles#518) @simcod * Fix promtail config when no timeout is configured. (metal-stack/metal-roles#520) @Gerrit91 * Allow setting `shootAdminKubeconfigMaxExpiration` in `Garden` resource. (metal-stack/metal-roles#519) @Gerrit91 * Structured authentication for Garden kube-apiserver (metal-stack/metal-roles#517) @simcod * feat: add envoy gateway to service clusters (for isolated clusters) (metal-stack/metal-roles#522) @mwennrich ### api v0.0.41 * Boot Services (metal-stack/api#83) @majst01 ### gardener-extension-ontap v0.2.12 * Set DNS policy to Default in mutator for the trident-node-linux daemonSet (metal-stack/gardener-extension-ontap#80) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.3 (metal-stack/website#184) @metal-robot[bot] * Updates regarding OCI artifacts. (metal-stack/website#146) @Gerrit91 * Register handlers by serve path. (metal-stack/metal-robot#100) @Gerrit91 * Slight refactor of comment command exec. (metal-stack/metal-robot#101) @Gerrit91 * fix typo in release pipeline (metal-stack/metal-images#384) @mac641 * Bump metal-api to version v0.43.0 (metal-stack/metal-python#161) @metal-robot[bot] * Bump metal-api to version v0.43.0 (metal-stack/metal-go#222) @metal-robot[bot] * Add section on pre-releases. (metal-stack/website#187) @Gerrit91 * Bump axios from 1.13.2 to 1.13.3 (metal-stack/website#186) @dependabot[bot] * Bump @scalar/api-reference-react from 0.8.27 to 0.8.34 (metal-stack/website#185) @dependabot[bot] * Cancel redfish calls after timeout (metal-stack/go-hal#82) @majst01 * fix container image retag in release pipeline (metal-stack/metal-images#385) @mac641 * fix typo in release pipeline (metal-stack/metal-images#386) @mac641 * Updates golang base image version to 1.25.6-bookworm (metal-stack/builder#88) @thheinel * FOSDEM 2026 recap. (metal-stack/website#193) @Gerrit91 * Bump axios from 1.13.3 to 1.13.4 (metal-stack/website#189) @dependabot[bot] * Bump @carbon/icons-react from 11.73.0 to 11.74.0 (metal-stack/website#191) @dependabot[bot] * fix: containerlab link in blog post (metal-stack/website#194) @vknabel * Bump react from 19.2.3 to 19.2.4 (metal-stack/website#188) @dependabot[bot] * Bump @scalar/api-reference-react from 0.8.36 to 0.8.46 (metal-stack/website#192) @dependabot[bot] * Update gofish and adjust for changes (metal-stack/go-hal#84) @stmcginnis * fix api version pinning and small ts errors (metal-stack/metal-ui#9) @ostempel * Update dependencies (metal-stack/firewall-controller#208) @mwennrich * docs: fix outdated links to metal-stack.io (metal-stack/metal-images#383) @vknabel * Implement gcp auth and remove unused actions in release pipeline (metal-stack/metal-images#387) @mac641 * Gov1.25.7 (metal-stack/builder#89) @thheinel * size, admin not finished yet (metal-stack/cli#4) @majst01 * Bump semver from 7.7.3 to 7.7.4 (metal-stack/website#195) @dependabot[bot] * Bump @scalar/api-reference-react from 0.8.46 to 0.8.52 (metal-stack/website#196) @dependabot[bot] * Bump axios from 1.13.4 to 1.13.5 (metal-stack/website#197) @dependabot[bot] * Fix gcs authentication, gcs object paths and rework tests for release pipeline (metal-stack/metal-images#388) @mac641 * Fix log output formatting and download url links in release pipeline (metal-stack/metal-images#389) @mac641 * build(Dockerfile): Upgrade Go version to 1.26.0 (metal-stack/builder#90) @thheinel * Next release (metal-stack/releases#265) @metal-robot[bot] --- ## v0.22.5 # metal-stack v0.22.5 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.5](https://github.com/metal-stack/releases/releases/tag/v0.22.5) ## General * [Gardener v1.124](https://github.com/gardener/gardener/releases/tag/v1.124.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Component Releases ### metalctl v0.18.5 * Add `machine ipmi chassis-list` subcommand. (metal-stack/metalctl#294) @Gerrit91 * Remove self-assignments in for-loops from older go versions. (metal-stack/metalctl#295) @Gerrit91 ### metal-metrics-exporter v0.3.1 * build with go 1.26, update deps, update gh actions, go fix (metal-stack/metal-metrics-exporter#26) @mwennrich ### api v0.0.49 * Fix python client for proto files with underscores. (metal-stack/api#101) @Gerrit91 * Partition Capacity (metal-stack/api#100) @majst01 * Add labels to admin tenant create request (metal-stack/api#103) @Gerrit91 ### helm-charts v0.5.5 * Move some metal-apiserver params to env. (metal-stack/helm-charts#147) @Gerrit91 * Update control-plane components `imagePullSecrets` (metal-stack/helm-charts#142) @AnnaSchreiner * Postgreslet next release (metal-stack/helm-charts#145) @TLINDEN ### ansible-common v0.8.1 * Add test and filter plugin for easier list handling (metal-stack/ansible-common#46) @Gerrit91 ### metal-roles v0.18.8 * Update zitadel (metal-stack/metal-roles#529) @majst01 * Add `ansible-lint` and `markdown-lint` (metal-stack/metal-roles#528) @Gerrit91 * Preparations for V2 API deployment (metal-stack/metal-roles#527) @Gerrit91 * Add `metal-v2-client` role to install V2 client library. (metal-stack/metal-roles#525) @Gerrit91 * Introduce spell checking. (metal-stack/metal-roles#524) @Gerrit91 * Fix typo and auditing backend reference (metal-stack/metal-roles#521) @simcod * only log http requests (metal-stack/metal-roles#523) @mwennrich * Fix zitadel deployment (metal-stack/metal-roles#530) @majst01 * Zitadel: Allow TLS configuration and overwrite of kubectl alpine k8s image (metal-stack/metal-roles#532) @Gerrit91 * Pass provider tenant for metal-apiserver. (metal-stack/metal-roles#533) @Gerrit91 ### metal-ansible-modules v0.2.12 * More updates on V2 modules (metal-stack/metal-ansible-modules#24) @Gerrit91 * Add first v2 modules. (metal-stack/metal-ansible-modules#23) @Gerrit91 ### metal-apiserver v0.1.4 * Fix provider tenant ensure must pass creator. (metal-stack/metal-apiserver#136) @Gerrit91 * Add labels to admin tenant create request (metal-stack/metal-apiserver#133) @Gerrit91 * Environment Variables for all configuration parameters (metal-stack/metal-apiserver#132) @majst01 * Update to go-1.26, run go fix (metal-stack/metal-apiserver#130) @majst01 * Support defining integer pool size on startup (metal-stack/metal-apiserver#129) @majst01 ### gardener-extension-audit v0.5.1 * Fix cluster-forwarding backend for g/g v1.124. (metal-stack/gardener-extension-audit#76) @Gerrit91 * Fix typos in S3 backend examples (metal-stack/gardener-extension-audit#73) @MichaelEischer # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Bump releases to version v0.22.4 (metal-stack/website#198) @metal-robot[bot] * chore(deps): Upgrade golangci-lint to v2.9.0 (metal-stack/builder#91) @thheinel * Upgrade to go-1.26, run go fix (metal-stack/metal-images#390) @majst01 * chore(deps): bump tailwind-merge from 3.4.0 to 3.4.1 (metal-stack/website#199) @dependabot[bot] * chore(deps): bump @carbon/icons-react from 11.74.0 to 11.75.0 (metal-stack/website#200) @dependabot[bot] * chore(deps): bump @scalar/api-reference-react from 0.8.52 to 0.8.57 (metal-stack/website#201) @dependabot[bot] * Update to lightos v3.18.x (metal-stack/duros-go#48) @majst01 * Add creation of generic oidc providers. (metal-stack/zitadel-init#6) @Gerrit91 * mini rack blog (metal-stack/website#203) @qrnvttrl * fix: add rlock() for fqdnToEntry map (metal-stack/firewall-controller#211) @mwennrich * Go 1.26 (metal-stack/security#65) @majst01 * Add roadmap section in contributing pages. (metal-stack/website#205) @Gerrit91 * Update to go-1.26 (metal-stack/cli#20) @majst01 * Pass auto create and auto update options to provider creation. (metal-stack/zitadel-init#7) @Gerrit91 * Fix link to roadmap. (metal-stack/website#212) @Gerrit91 * Go 1.26 (metal-stack/gardener-extension-csi-driver-synology#4) @majst01 * Component Service (metal-stack/cli#21) @majst01 * Next release (metal-stack/releases#266) @metal-robot[bot] --- ## v0.22.6 # metal-stack v0.22.6 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.6](https://github.com/metal-stack/releases/releases/tag/v0.22.6) ## General * [Gardener v1.125](https://github.com/gardener/gardener/releases/tag/v1.125.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * masterdata-db bump to Postgres 15 ## Noteworthy * The fluentbit version used by the audit extension has been downgraded to 4.0.5 to fix rare OOM kills of fluentbit. (metal-stack/gardener-extension-audit#62) * Default Calico images are intentionally pinned to `v3.29.7` in this role. Reason: Calico eBPF versions `< v3.29.7` can fail on newer kernels: https://github.com/projectcalico/calico/issues/11302#issuecomment-3663319534 (metal-stack/metal-roles#555) ## Component Releases ### helm-charts v0.5.6 * Allow passing provider tenant to metal-apiserver. (metal-stack/helm-charts#148) @Gerrit91 ### gardener-extension-audit v0.5.5 * Revert fluentbit upgrade to 4.0.12 to fix rare OOM crashes (metal-stack/gardener-extension-audit#62) @MichaelEischer * feat: make defaultWebhookMode configurable (metal-stack/gardener-extension-audit#80) @mwennrich * Feature add custom backend (metal-stack/gardener-extension-audit#70) @charleenklang * also respect blocking-override for cluster-forwarding in actuator (metal-stack/gardener-extension-audit#83) @mwennrich ### duros-controller v0.12.0 * Update to lightos 3.18 api (metal-stack/duros-controller#100) @majst01 ### metal-roles v0.18.15 * Auto-cleanup zitadel-init jobs and update of zitadel-init to v0.3.1. (metal-stack/metal-roles#534) @Gerrit91 * Fix indention of `shootAdminKubeconfigMaxExpiration`. (metal-stack/metal-roles#539) @Gerrit91 * Configurable metal-bmc lease file (metal-stack/metal-roles#538) @AnnaSchreiner * Make hosts file mount for partition prometheus optional. (metal-stack/metal-roles#542) @Gerrit91 * Little improvements for the ipmi_exporter (metal-stack/metal-roles#543) @Gerrit91 * Provide `gardener-logging` role (metal-stack/metal-roles#540) @Gerrit91 * Fix metal-api log level not using correct parameter variable. (metal-stack/metal-roles#541) @Gerrit91 * Update monitoring charts and enable thanos ruler for control plane (metal-stack/metal-roles#544) @Gerrit91 * Add parameter for `gardener_extension_audit_default_webhook_mode` (metal-stack/metal-roles#545) @mwennrich * Ensure prometheus rules directory on mgmt-server. (metal-stack/metal-roles#546) @Gerrit91 * Move thanos global section out of objstore config section. (metal-stack/metal-roles#547) @Gerrit91 * Overwrite default calico version to 3.29.7 (metal-stack/metal-roles#555) @mwennrich * Additional parameters for Gardener projects and shoots (metal-stack/metal-roles#526) @simcod * Add gardener_extension_networking_calico_image_vector_overwrite (metal-stack/metal-roles#554) @mwennrich * Allow setting provider region in gardener runtime cluster. (metal-stack/metal-roles#553) @Gerrit91 * Add ability to set storage class for thanos compactor. (metal-stack/metal-roles#550) @Gerrit91 * Set thanos compactor resource preset. (metal-stack/metal-roles#548) @Gerrit91 ### oci-mirror v0.2.10 * Update to go-1.26 (metal-stack/oci-mirror#20) @majst01 ### api v0.0.53 * Namespaced IP support for machine create and Size Image Constraint (metal-stack/api#106) @majst01 * Add switch connected-machines endpoint (metal-stack/api#93) @iljarotar ### metal-apiserver v0.1.7 * Size Image Constraint (metal-stack/metal-apiserver#140) @majst01 * Component expiration was not passed (metal-stack/metal-apiserver#139) @majst01 ### cluster-api-provider-metal-stack v0.8.0 * test: upgrade contracts (metal-stack/cluster-api-provider-metal-stack#126) @vknabel * Managed Node Network ID (metal-stack/cluster-api-provider-metal-stack#119) @vknabel * Cluster API v1.11 (metal-stack/cluster-api-provider-metal-stack#96) @simcod * docs: rolling firewalls (metal-stack/cluster-api-provider-metal-stack#121) @vknabel ### pixie v0.4.0 * Detect manufacturer with ipxe script (metal-stack/pixie#43) @majst01 * fix linting (metal-stack/pixie#41) @mac641 * Nicer logging (metal-stack/pixie#39) @majst01 ### metal-hammer v0.14.1 * Dell Support (metal-stack/metal-hammer#172) @majst01 * Update to go-1.26 and fix intel firmware download (metal-stack/metal-hammer#178) @majst01 ### metal-bmc v0.7.0 * Add new vendor Dell (metal-stack/metal-bmc#87) @simcod ### firewall-controller-manager v0.6.0 * Recreate firewall on unhealthy condition (metal-stack/firewall-controller-manager#63) @Honigeintopf * Update dependencies (metal-stack/firewall-controller-manager#83) @Gerrit91 * Propagate the No Controller Annotation on Resource Creation (metal-stack/firewall-controller-manager#80) @vknabel * SSH Key Namespace (metal-stack/firewall-controller-manager#79) @vknabel * include sbom in container image (metal-stack/firewall-controller-manager#73) @mac641 * Update to go-1.24 and golangci-lint v2 (metal-stack/firewall-controller-manager#72) @majst01 ### gardener-extension-provider-metal v0.27.5 * Allow configuration of health timeout (metal-stack/gardener-extension-provider-metal#488) @Honigeintopf * add network policies for metallb-system and csi-lvm (metal-stack/gardener-extension-provider-metal#489) @mwennrich ### gardener-extension-ontap v0.2.14 * build with go 1.26, update deps/gh-actions, go fix by @mwennrich in https://github.com/metal-stack/gardener-extension-ontap/pull/83 * Update admin client to be ready for switchover and volume limit reach by @Honigeintopf in https://github.com/metal-stack/gardener-extension-ontap/pull/82 **Full Changelog**: https://github.com/metal-stack/gardener-extension-ontap/compare/v0.2.12...v0.2.13 * Mutate init container for nvme-tcp kernel driver to trident-node-linux daemonset (metal-stack/gardener-extension-ontap#84) @mwennrich # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update to go-1.26 (metal-stack/metal-lib#199) @majst01 * refactor: community is now unversioned (metal-stack/website#207) @vknabel * chore(deps): bump @scalar/api-reference-react from 0.8.57 to 0.8.62 (metal-stack/website#211) @dependabot[bot] * chore(deps): bump tailwindcss from 4.1.18 to 4.2.0 (metal-stack/website#209) @dependabot[bot] * chore(deps): bump tailwind-merge from 3.4.1 to 3.5.0 (metal-stack/website#208) @dependabot[bot] * chore(deps): bump @tailwindcss/postcss from 4.1.18 to 4.2.0 (metal-stack/website#210) @dependabot[bot] * docs: minors only (metal-stack/website#214) @vknabel * Bump releases to version v0.22.5 (metal-stack/website#213) @metal-robot[bot] * docs(blog): rename conferences to events (metal-stack/website#206) @vknabel * fix: redirect renamed pages (metal-stack/website#202) @vknabel * Fix version bump despite release being frozen (metal-stack/metal-robot#104) @Gerrit91 * Add issue type filter for project item add. (metal-stack/metal-robot#105) @Gerrit91 * Go 1.26 (metal-stack/nftables-exporter#37) @majst01 * Go 1.26 (metal-stack/metal-robot#106) @majst01 * Upgrade masterdata db to v15.16 (metal-stack/releases#276) @majst01 * Add new vendor Dell (metal-stack/go-hal#79) @simcod * fix(metal-api): api docs unavailable (metal-stack/website#221) @vknabel * Dell Support (metal-stack/website#220) @majst01 * chore(deps): bump @scalar/api-reference-react from 0.8.62 to 0.8.66 (metal-stack/website#215) @dependabot[bot] * chore(deps-dev): bump lint-staged from 16.2.7 to 16.3.1 (metal-stack/website#216) @dependabot[bot] * chore(deps): bump tailwindcss from 4.2.0 to 4.2.1 (metal-stack/website#217) @dependabot[bot] * chore(deps): bump axios from 1.13.5 to 1.13.6 (metal-stack/website#218) @dependabot[bot] * chore(deps): bump @carbon/icons-react from 11.75.0 to 11.76.0 (metal-stack/website#219) @dependabot[bot] * update firewall-controller to v2.4.2 (metal-stack/metal-images#391) @mwennrich * Adopt to ontap v9.16 (metal-stack/ontap-go#6) @majst01 * Add README for the org. (metal-stack/.github#2) @Gerrit91 * migrate from uber-go/mock to vektra/mockery (metal-stack/firewall-controller#212) @chbmuc * Include metal-networker (metal-stack/os-installer#1) @majst01 * Go 1.26 (metal-stack/firewall-controller#213) @majst01 * Add tests to detect race conditions and fix two potential issues (metal-stack/firewall-controller#214) @chbmuc * Build debian (metal-stack/frr#14) @majst01 * chore(deps-dev): bump svgo from 4.0.0 to 4.0.1 (metal-stack/website#227) @dependabot[bot] * chore(deps): bump postcss from 8.5.6 to 8.5.8 (metal-stack/website#225) @dependabot[bot] * chore(deps-dev): bump lint-staged from 16.3.1 to 16.3.2 (metal-stack/website#224) @dependabot[bot] * chore(deps): bump @scalar/api-reference-react from 0.8.68 to 0.9.1 (metal-stack/website#226) @dependabot[bot] * downgrade debian kernel to 6.1.0-42 (metal-stack/metal-images#393) @mwennrich * Dell blog post (metal-stack/website#223) @simcod * Go 1.26 (metal-stack/go-lldpd#33) @majst01 * Increase LimitNOFILE for container 2.1 to previous value from containerd 1.7 (metal-stack/metal-images#394) @mwennrich * Add AI tools paragraph to contribution guideline. (metal-stack/website#204) @Gerrit91 * Add AI usage section to pull request template. (metal-stack/.github#3) @Gerrit91 * Next release (metal-stack/releases#275) @metal-robot[bot] --- ## v0.22.7 # metal-stack v0.22.7 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.7](https://github.com/metal-stack/releases/releases/tag/v0.22.7) ## General * [Gardener v1.126](https://github.com/gardener/gardener/releases/tag/v1.126.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * A couple of postgres databases received patch version bumps ## Breaking Changes * The customforwarding backend now expects its configuration in the fluent-bit YAML format. Please adapt your custom backends configuration if you use this feature. ([release notes](https://github.com/metal-stack/gardener-extension-audit/releases/tag/v0.6.0)) ## Noteworthy * It is now possible to configure the maximum log event size via the providerConfig. (metal-stack/gardener-extension-audit#79) * The deployment image now ships with Python 14 and includes `ansible-pylibssh`. (metal-stack/metal-deployment-base#48) * The partition-proxy role now creates DNS records for every configured proxy endpoint. (metal-stack/metal-roles#559) * The `resources.requests.cpu` is now set to 50% by default. To recreate the previous behaviour, set `postgreslet.spiloCpuRequestsPercentage` to `100`. (metal-stack/helm-charts#152) * The fluentbit used by gardener-extension-audit now uses the YAML configuration format. (metal-stack/gardener-extension-audit#78) ## Component Releases ### pixie v0.4.1 * Set console ttyS0 for QEMU manufacturer (metal-stack/pixie#44) @simcod ### metal-core v0.15.2 * Update to go-1.26 (metal-stack/metal-core#181) @majst01 * Check ctx.Done() in dbus Reload and Start (metal-stack/metal-core#183) @iljarotar * Ignore spine uplinks when evaluating down ports (metal-stack/metal-core#182) @iljarotar * Use topmost parent context for all subsequent functions during switch reconfigure (metal-stack/metal-core#180) @iljarotar ### api v0.0.58 * Fix Plural of DNS and NTPServer (metal-stack/api#109) @majst01 * Add audit API definition. (metal-stack/api#110) @Gerrit91 * More machine Queries (metal-stack/api#107) @majst01 * Add redis and tasks to health services. (metal-stack/api#113) @Gerrit91 * Simplify client tests (metal-stack/api#115) @majst01 ### gardener-extension-audit v0.6.0 * Revendor g/g v1.125.4 (metal-stack/gardener-extension-audit#82) @Gerrit91 * Make maximum audit event size configurable (metal-stack/gardener-extension-audit#79) @relusc * Convert fluentbit configuration to YAML (metal-stack/gardener-extension-audit#78) @MichaelEischer ### metal-deployment-base v0.9.2 * Dependency updates. (metal-stack/metal-deployment-base#48) @Gerrit91 ### zitadel-init v0.3.2 * Fix client secret gets emptied on every second execution (metal-stack/zitadel-init#8) @Gerrit91 ### metal-roles v0.19.5 * Delegate multiple partition proxy definitions by virtual host (metal-stack/metal-roles#559) @Honigeintopf * add timeout config parameter to metal-core (metal-stack/metal-roles#558) @iljarotar * Introduce more granular code owners. (metal-stack/metal-roles#557) @Gerrit91 * Make repeat interval for alerts configurable. (metal-stack/metal-roles#556) @Gerrit91 * Use zitadel-init from release vector. (metal-stack/metal-roles#561) @Gerrit91 * Wire up metal-apiserver splunk auditing params. (metal-stack/metal-roles#560) @Gerrit91 * fix: switch zitadel init image tag and name in release vector mapping (metal-stack/metal-roles#564) @ma-hartma * Annotation for old dns proxies confirmation (metal-stack/metal-roles#565) @Honigeintopf * Add missing variable to sonic-config readme (metal-stack/metal-roles#567) @iljarotar * feat: add UDP session idle timeout configuration for Envoy Gateway (metal-stack/metal-roles#568) @mwennrich * Add security context and resource limits (metal-stack/metal-roles#537) @AnnaSchreiner * More parameters for gardenlet and gardener-extension-provider-metal (metal-stack/metal-roles#569) @Gerrit91 * Only reload/load SONiC config if `config_db.json` changed. (metal-stack/metal-roles#572) @Gerrit91 * Fix typo in NSQd admin resources. (metal-stack/metal-roles#573) @Gerrit91 ### gardener-extension-dns-powerdns v0.7.3 * Go 1.26 (metal-stack/gardener-extension-dns-powerdns#12) @majst01 * include sbom in container image (metal-stack/gardener-extension-dns-powerdns#11) @mac641 ### helm-charts v0.5.9 * Wire up metal-apiserver splunk auditing params. (metal-stack/helm-charts#151) @Gerrit91 * Postgreslet Next Release (metal-stack/helm-charts#152) @eberlep ### metal-api v0.43.1 * Add new network properties to machineNetwork (metal-stack/metal-api#643) @majst01 * Go 1.26 (metal-stack/metal-api#642) @majst01 ### metal-apiserver v0.1.8 * Speedup tests by splitting them up (metal-stack/metal-apiserver#154) @majst01 * Implementation of the Audit API. (metal-stack/metal-apiserver#157) @Gerrit91 * Fix flaky vpn connected machines test (metal-stack/metal-apiserver#159) @majst01 * Datacenter improvements (metal-stack/metal-apiserver#147) @iljarotar * Some var blocks and alphatic sorting in `services.go` (metal-stack/metal-apiserver#158) @Gerrit91 * Fix Plural of DNS and NTPServer (metal-stack/metal-apiserver#153) @majst01 * Ensure only one child-shared per project (metal-stack/metal-apiserver#146) @majst01 * Implement switch connected machines endpoint (metal-stack/metal-apiserver#145) @iljarotar * Add task payload type to simplify `NewTask` function (metal-stack/metal-apiserver#143) @iljarotar * Simplify database initialization (metal-stack/metal-apiserver#142) @majst01 ### metalctl v0.18.7 * Update to machine-networks (metal-stack/metalctl#297) @majst01 * Upgrade to go 1.26 and run go fix (metal-stack/metalctl#296) @majst01 * Add machine filters for chassis part number and serial. (metal-stack/metalctl#298) @Gerrit91 ### gardener-extension-ontap v0.2.16 * Make svm consider aggregate idempotency (metal-stack/gardener-extension-ontap#89) @Honigeintopf * Remove node records length case (metal-stack/gardener-extension-ontap#87) @Honigeintopf * Mutate even with delete timestamp by @Honigeintopf in https://github.com/metal-stack/gardener-extension-ontap/pull/90 **Full Changelog**: https://github.com/metal-stack/gardener-extension-ontap/compare/v0.2.15...v0.2.16 ### gardener-extension-provider-metal v0.27.7 * fix metallb and csi-lvm networkpolicies for node-local-dns featuregate (metal-stack/gardener-extension-provider-metal#493) @mwennrich * Allow adding netpols for ingress-controller in Garden cluster. (metal-stack/gardener-extension-provider-metal#494) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * Update to go-1.26, run go fix (metal-stack/metal-networker#125) @majst01 * Bump releases to version v0.22.6 (metal-stack/website#230) @metal-robot[bot] * Deprecation Notice (metal-stack/metal-networker#126) @majst01 * General updates on the contribution guideline. (metal-stack/website#231) @Gerrit91 * chore(deps): bump @scalar/api-reference-react from 0.9.1 to 0.9.9 (metal-stack/website#232) @dependabot[bot] * Switch functions (metal-stack/cli#17) @iljarotar * chore(ci): configure dependabot (metal-stack/website#234) @vknabel * chore(deps): bump the actions group with 2 updates (metal-stack/website#235) @dependabot[bot] * chore(deps-dev): bump lint-staged from 16.3.2 to 16.4.0 (metal-stack/website#233) @dependabot[bot] * Update postgres patch versions (metal-stack/releases#279) @majst01 * Bump google.golang.org/grpc from 1.79.1 to 1.79.3 in the go_modules group across 1 directory (metal-stack/duros-go#49) @dependabot[bot] * Bump the go_modules group across 1 directory with 2 updates (metal-stack/metal-lib#201) @dependabot[bot] * docs(community): link to org discussions (metal-stack/website#238) @vknabel * update kernel to include latest apparmor fixes (metal-stack/metal-images#400) @mwennrich * switch from proprietary NVIDIA driver to open kernel module (metal-stack/metal-images#401) @mwennrich * Starting E2E tests (metal-stack/cli#24) @Gerrit91 * Bump metal-api to version v0.43.1 (metal-stack/metal-python#162) @metal-robot[bot] * Bump metal-api to version v0.43.1 (metal-stack/metal-go#223) @metal-robot[bot] * Implement Audit API. (metal-stack/cli#23) @Gerrit91 * Add CODEOWNERS. (metal-stack/cli#25) @Gerrit91 * Add `Health` to audit interface to check backend health. (metal-stack/metal-lib#202) @Gerrit91 * Test interceptor was moved to API repo. (metal-stack/cli#27) @Gerrit91 * chore(deps): bump the other-dependencies group with 2 updates (metal-stack/website#237) @dependabot[bot] * add goss tests for capi images (metal-stack/metal-images#402) @mac641 * chore(deps): bump the other-dependencies group with 2 updates (metal-stack/website#239) @dependabot[bot] * chore(deps): bump @scalar/api-reference-react from 0.9.9 to 0.9.11 in the docusaurus-dependencies group (metal-stack/website#236) @dependabot[bot] * chore(deps-dev): bump typescript from 5.9.3 to 6.0.2 (metal-stack/website#240) @dependabot[bot] * docs(community): skip 2026-04-06 due to easter monday (metal-stack/website#241) @vknabel * add skill for agentic coding (metal-stack/metal-ui#10) @ostempel * Systemd file units (metal-stack/os-installer#3) @majst01 * Render nftables tmpfiles with leading dot. (metal-stack/os-installer#7) @Gerrit91 * chore(deps): bump @scalar/api-reference-react from 0.9.15 to 0.9.18 in the docusaurus-dependencies group (metal-stack/website#243) @dependabot[bot] * chore(deps): bump the other-dependencies group across 1 directory with 2 updates (metal-stack/website#244) @dependabot[bot] * add kamaji blog post (metal-stack/website#229) @mac641 * Bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 in the go_modules group across 1 directory (metal-stack/metal-lib#204) @dependabot[bot] * chore(deps): bump the other-dependencies group with 4 updates (metal-stack/website#246) @dependabot[bot] * chore(deps): bump the docusaurus-dependencies group across 1 directory with 9 updates (metal-stack/website#245) @dependabot[bot] * Next release (metal-stack/releases#278) @metal-robot[bot] --- ## v0.22.8 # metal-stack v0.22.8 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.8](https://github.com/metal-stack/releases/releases/tag/v0.22.8) ## General * [Gardener v1.127](https://github.com/gardener/gardener/releases/tag/v1.127.0) * ⚠️ Be aware that this version of Gardener does not support garden, seed, or shoot clusters with Kubernetes versions `<= 1.28`. * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. * A couple of postgres databases received patch version bumps ## Required Actions * The deprecated `spec.virtualCluster.kubernetes.kubeAPIServer.oidcConfig` configuration parameters used in the `gardener-operator` role were removed. If you used them, please migrate your configuration to `gardener_operator_virtual_garden_api_server_structured_authentication`. (metal-stack/metal-roles#551) * The registry mirror config for `isolated-clusters` changed, please make sure to deploy it before upgrading the Gardener. ([release notes](https://github.com/metal-stack/metal-roles/releases/tag/v0.19.8)) ## Noteworthy * Add basic OpenTelemetry backend to allow sending audit logs to a OTLP-compatible destination. (metal-stack/gardener-extension-audit#84) * Helm charts from helm.metal-stack.io can now also be installed using OCI images from ghcr.io. (metal-stack/helm-charts#154) ## Component Releases ### gardener-extension-audit v0.6.1 * Add OpenTelemetry Backend (metal-stack/gardener-extension-audit#84) @MichaelEischer ### metalctl v0.18.8 * hide switches without matching machines in connected-machines table output (metal-stack/metalctl#301) @mwennrich * Bump the go_modules group across 1 directory with 2 updates (metal-stack/metalctl#300) @[dependabot[bot]](https://github.com/apps/dependabot) ### os-metal-extension v0.8.10 * Go 1.26 (metal-stack/os-metal-extension#63) @majst01 ### duros-controller v0.12.1 * Update go modules and ci (metal-stack/duros-controller#101) @majst01 ### metal-roles v0.19.8 * More refinements on ipmi-exporter (metal-stack/metal-roles#576) @Gerrit91 * Allow specification of machine timeout (metal-stack/metal-roles#577) @Honigeintopf * Allow dynamic generation of IPMI exporter targets. (metal-stack/metal-roles#571) @Gerrit91 * Give alertmanager a disk. (metal-stack/metal-roles#575) @Gerrit91 * Gardener operator dashboard deployment (metal-stack/metal-roles#551) @simcod * fix(metal): prevent helm templating error if resources are unset (metal-stack/metal-roles#581) @l0wl3vel * add envoy to oci-mirror (metal-stack/metal-roles#583) @mwennrich ### helm-charts v0.6.0 * Fix release drafter branch name. (metal-stack/helm-charts#155) @Gerrit91 * Update actions including OCI artifacts. (metal-stack/helm-charts#154) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * docs: llms(-full).txt (metal-stack/website#248) @vknabel * Bump releases to version v0.22.7 (metal-stack/website#249) @metal-robot[bot] * Use local unversioned paths for testing (metal-stack/actions-common#1) @Gerrit91 * Use @v1 in README. (metal-stack/actions-common#2) @Gerrit91 * Fix release asset uploads. (metal-stack/actions-common#3) @Gerrit91 * Upload entire folder not individual files. (metal-stack/actions-common#4) @Gerrit91 * Next attempt to fix artifact upload. (metal-stack/actions-common#5) @Gerrit91 * Next release (metal-stack/releases#282) @metal-robot[bot] --- ## v0.22.9 # metal-stack v0.22.9 See original release note at [https://github.com/metal-stack/releases/releases/tag/v0.22.9](https://github.com/metal-stack/releases/releases/tag/v0.22.9) ## General * [Gardener v1.128](https://github.com/gardener/gardener/releases/tag/v1.128.0) * Please note that this release contains the gardener-apiserver built from the metal-stack fork in order to prevent the defaulting of worker machine images by Gardener. This will be resolved upstream with https://github.com/gardener/gardener/pull/13785. If you do not use short image versions in the `CloudProfile` you can also use the upstream version of the gardener-apiserver. ## Required Actions * Update existing zitadel-init configurations in your deployment (`zitadel_init_config`) to use `redirect_uris` instead of `redirect_uri`. (metal-stack/zitadel-init#9) * If during reconciles of old shoots the following shoot error occurs: `ScrapeConfig.monitoring.coreos.com "shoot-blackbox-exporter-k8s-service-check is invalid: spec.kubernetesSDConfigs[0].role: Unsupported value: "service"`, it is enough to edit the `ScrapeConfig` in the shoot namespace in the `Seed` and correct `service` to `Service`. ## Component Releases ### metal-roles v0.20.1 * Use new OCI helm-charts from metal-stack. (metal-stack/metal-roles#582) @Gerrit91 * Add param for deploying `ClusterRoleBinding`s into the virtual garden. (metal-stack/metal-roles#585) @Gerrit91 ### api v0.0.61 * Machine create refinements (metal-stack/api#119) @majst01 * Map validations (metal-stack/api#122) @majst01 ### node-init v0.1.8 * Update actions workflows and dependencies. (metal-stack/node-init#20) @Gerrit91 * feat: configure HTTP transport for persistent connections to kube-apiserver (metal-stack/node-init#19) @mwennrich * include sbom in container image (metal-stack/node-init#17) @mac641 ### zitadel-init v0.3.3 * Allow configuration of multiple redirect URIs. (metal-stack/zitadel-init#9) @Gerrit91 # Merged Pull Requests This is a list of pull requests that were merged since the last release. The list does not contain pull requests from release-vector-repositories. The fact that these pull requests were merged does not necessarily imply that they have already become part of this metal-stack release. * More documentation for release-drafter action. (metal-stack/actions-common#6) @Gerrit91 * Provide common helm-chart action. (metal-stack/actions-common#7) @Gerrit91 * chore: blog maintainers (metal-stack/website#253) @vknabel * Bump releases to version v0.22.8 (metal-stack/website#252) @metal-robot[bot] * Remove unnecessary braces from if conditions. (metal-stack/actions-common#9) @Gerrit91 * Fix issue comments only working on pull requests. (metal-stack/metal-robot#108) @Gerrit91 * chore(deps): bump postcss from 8.5.9 to 8.5.10 in the other-dependencies group (metal-stack/website#251) @dependabot[bot] * chore(deps): bump the docusaurus-dependencies group with 2 updates (metal-stack/website#250) @dependabot[bot] * Allow repo maintainers to run comment actions. (metal-stack/metal-robot#109) @Gerrit91 * Support more chassistypes (metal-stack/go-hal#86) @majst01 * Improved test coverage (metal-stack/cli#29) @AnnaSchreiner * Enter serial console (metal-stack/go-hal#87) @majst01 * Remove security.User from auditing and split http/grpc/connect (metal-stack/metal-lib#206) @majst01 * chore(ci): actions-common (metal-stack/metal-robot#110) @vknabel * chore(deps): bump the docusaurus-dependencies group with 2 updates (metal-stack/website#254) @dependabot[bot] * chore(deps): bump the other-dependencies group across 1 directory with 5 updates (metal-stack/website#255) @dependabot[bot] * Migrate to web-only UI (metal-stack/metal-ui#11) @ostempel * Dependency updates. (metal-stack/cli#30) @Gerrit91 * adjust metal-ui skill and remove tauri leftovers (metal-stack/metal-ui#12) @ostempel * Add rule template. (metal-stack/website#256) @Gerrit91 * Next release (metal-stack/releases#284) @metal-robot[bot] --- ## API import { useState, useEffect } from "react"; export function VersionedLink({ children }) { const [version, setVersion] = useState(""); useEffect(() => { const path = window.location.pathname; const v = path .replace("/docs/", "") .replace("references/api", "") .replace("/", ""); if (v === "") setVersion("latest"); else setVersion(v); }, []); if (version) return {children}; return null; } We are currently working on the [metal-stack API v2](/docs/MEP-4-multi-tenancy-for-the-metal-api) which will gradually replace v1. This page provides links to both API documentations. | Version | Repository | Docs | | ------- | ----------------------------------------------------- | ------------------------------------------- | | v1 | [metal-api](https://github.com/metal-stack/metal-api) | OpenAPI Spec | | v2 | [api](https://github.com/metal-stack/api) | Coming soon | --- ## metalctl a cli to manage entities in the metal-stack api ### Options ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty -h, --help help for metalctl --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl audit](./metalctl_audit.md) - manage audit trace entities * [metalctl completion](./metalctl_completion.md) - Generate the autocompletion script for the specified shell * [metalctl context](./metalctl_context.md) - manage metalctl context * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities * [metalctl firewall](./metalctl_firewall.md) - manage firewall entities * [metalctl firmware](./metalctl_firmware.md) - manage firmwares * [metalctl health](./metalctl_health.md) - shows the server health * [metalctl image](./metalctl_image.md) - manage image entities * [metalctl login](./metalctl_login.md) - login user and receive token * [metalctl logout](./metalctl_logout.md) - logout user from OIDC SSO session * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl markdown](./metalctl_markdown.md) - create markdown documentation * [metalctl network](./metalctl_network.md) - manage network entities * [metalctl partition](./metalctl_partition.md) - manage partition entities * [metalctl project](./metalctl_project.md) - manage project entities * [metalctl size](./metalctl_size.md) - manage size entities * [metalctl switch](./metalctl_switch.md) - manage switch entities * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities * [metalctl update](./metalctl_update.md) - update the program * [metalctl version](./metalctl_version.md) - print the client and server version information * [metalctl vpn](./metalctl_vpn.md) - access VPN * [metalctl whoami](./metalctl_whoami.md) - shows current user --- ## metalctl_audit ## metalctl audit manage audit trace entities ### Synopsis show audit traces of the api. feature must be enabled on server-side. ### Options ``` -h, --help help for audit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl audit describe](./metalctl_audit_describe.md) - describes the audit trace * [metalctl audit list](./metalctl_audit_list.md) - list all audit traces --- ## metalctl_audit_describe ## metalctl audit describe describes the audit trace ``` metalctl audit describe [flags] ``` ### Options ``` -h, --help help for describe --phase string phase of the audit trace. One of [request, response, single, error, opened, closed] (default "response") --prettify-body attempts to interpret the body as json and prettifies it ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl audit](./metalctl_audit.md) - manage audit trace entities --- ## metalctl_audit_list ## metalctl audit list list all audit traces ``` metalctl audit list [flags] ``` ### Options ``` --component string component of the audit trace. --detail string detail of the audit trace. An HTTP method, unary or stream --error string error of the audit trace. --forwarded-for string forwarded for of the audit trace. --from string start of range of the audit traces. e.g. 1h, 10m, 2006-01-02 15:04:05 (default "1h") -h, --help help for list --limit int limit the number of audit traces. (default 100) --path string api path of the audit trace. --phase string phase of the audit trace. One of [request, response, single, error, opened, closed] -q, --query string filters audit trace body payloads for the given text. --remote-addr string remote address of the audit trace. --request-id string request id of the audit trace. --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: path|tenant|timestamp|user --status-code int32 HTTP status code of the audit trace. --tenant string tenant of the audit trace. --to string end of range of the audit traces. e.g. 1h, 10m, 2006-01-02 15:04:05 --type string type of the audit trace. One of [http, grpc, event]. --user string user of the audit trace. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl audit](./metalctl_audit.md) - manage audit trace entities --- ## metalctl_completion ## metalctl completion Generate the autocompletion script for the specified shell ### Synopsis Generate the autocompletion script for metalctl for the specified shell. See each sub-command's help for details on how to use the generated script. ### Options ``` -h, --help help for completion ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl completion bash](./metalctl_completion_bash.md) - Generate the autocompletion script for bash * [metalctl completion fish](./metalctl_completion_fish.md) - Generate the autocompletion script for fish * [metalctl completion powershell](./metalctl_completion_powershell.md) - Generate the autocompletion script for powershell * [metalctl completion zsh](./metalctl_completion_zsh.md) - Generate the autocompletion script for zsh --- ## metalctl_completion_bash ## metalctl completion bash Generate the autocompletion script for bash ### Synopsis Generate the autocompletion script for the bash shell. This script depends on the 'bash-completion' package. If it is not installed already, you can install it via your OS's package manager. To load completions in your current shell session: source <(metalctl completion bash) To load completions for every new session, execute once: #### Linux: metalctl completion bash > /etc/bash_completion.d/metalctl #### macOS: metalctl completion bash > $(brew --prefix)/etc/bash_completion.d/metalctl You will need to start a new shell for this setup to take effect. ``` metalctl completion bash ``` ### Options ``` -h, --help help for bash --no-descriptions disable completion descriptions ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl completion](./metalctl_completion.md) - Generate the autocompletion script for the specified shell --- ## metalctl_completion_fish ## metalctl completion fish Generate the autocompletion script for fish ### Synopsis Generate the autocompletion script for the fish shell. To load completions in your current shell session: metalctl completion fish | source To load completions for every new session, execute once: metalctl completion fish > ~/.config/fish/completions/metalctl.fish You will need to start a new shell for this setup to take effect. ``` metalctl completion fish [flags] ``` ### Options ``` -h, --help help for fish --no-descriptions disable completion descriptions ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl completion](./metalctl_completion.md) - Generate the autocompletion script for the specified shell --- ## metalctl_completion_powershell ## metalctl completion powershell Generate the autocompletion script for powershell ### Synopsis Generate the autocompletion script for powershell. To load completions in your current shell session: metalctl completion powershell | Out-String | Invoke-Expression To load completions for every new session, add the output of the above command to your powershell profile. ``` metalctl completion powershell [flags] ``` ### Options ``` -h, --help help for powershell --no-descriptions disable completion descriptions ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl completion](./metalctl_completion.md) - Generate the autocompletion script for the specified shell --- ## metalctl_completion_zsh ## metalctl completion zsh Generate the autocompletion script for zsh ### Synopsis Generate the autocompletion script for the zsh shell. If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once: echo "autoload -U compinit; compinit" >> ~/.zshrc To load completions in your current shell session: source <(metalctl completion zsh) To load completions for every new session, execute once: #### Linux: metalctl completion zsh > "${fpath[1]}/_metalctl" #### macOS: metalctl completion zsh > $(brew --prefix)/share/zsh/site-functions/_metalctl You will need to start a new shell for this setup to take effect. ``` metalctl completion zsh [flags] ``` ### Options ``` -h, --help help for zsh --no-descriptions disable completion descriptions ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl completion](./metalctl_completion.md) - Generate the autocompletion script for the specified shell --- ## metalctl_context ## metalctl context manage metalctl context ### Synopsis context defines the backend to which metalctl talks to. You can switch back and forth with "-" ``` metalctl context [flags] ``` ### Examples ``` ~/.metalctl/config.yaml --- current: prod contexts: prod: url: https://api.metal-stack.io/metal issuer_url: https://dex.metal-stack.io/dex client_id: metal_client client_secret: 456 dev: url: https://api.metal-stack.dev/metal issuer_url: https://dex.metal-stack.dev/dex client_id: metal_client client_secret: 123 ... ``` ### Options ``` -h, --help help for context ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl context short](./metalctl_context_short.md) - only show the default context name --- ## metalctl_context_short ## metalctl context short only show the default context name ``` metalctl context short [flags] ``` ### Options ``` -h, --help help for short ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl context](./metalctl_context.md) - manage metalctl context --- ## metalctl_filesystemlayout ## metalctl filesystemlayout manage filesystemlayout entities ### Synopsis a filesystemlayout is a specification how the disks in a machine are partitioned, formatted and mounted. ### Options ``` -h, --help help for filesystemlayout ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl filesystemlayout apply](./metalctl_filesystemlayout_apply.md) - applies one or more filesystemlayouts from a given file * [metalctl filesystemlayout create](./metalctl_filesystemlayout_create.md) - creates the filesystemlayout * [metalctl filesystemlayout delete](./metalctl_filesystemlayout_delete.md) - deletes the filesystemlayout * [metalctl filesystemlayout describe](./metalctl_filesystemlayout_describe.md) - describes the filesystemlayout * [metalctl filesystemlayout edit](./metalctl_filesystemlayout_edit.md) - edit the filesystemlayout through an editor and update * [metalctl filesystemlayout list](./metalctl_filesystemlayout_list.md) - list all filesystemlayouts * [metalctl filesystemlayout match](./metalctl_filesystemlayout_match.md) - check if a machine satisfies all disk requirements of a given filesystemlayout * [metalctl filesystemlayout try](./metalctl_filesystemlayout_try.md) - try to detect a filesystem by given size and image * [metalctl filesystemlayout update](./metalctl_filesystemlayout_update.md) - updates the filesystemlayout --- ## metalctl_filesystemlayout_apply ## metalctl filesystemlayout apply applies one or more filesystemlayouts from a given file ``` metalctl filesystemlayout apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl filesystemlayout describe filesystemlayout-1 -o yaml > filesystemlayout.yaml $ vi filesystemlayout.yaml $ # either via stdin $ cat filesystemlayout.yaml | metalctl filesystemlayout apply -f - $ # or via file $ metalctl filesystemlayout apply -f filesystemlayout.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_create ## metalctl filesystemlayout create creates the filesystemlayout ``` metalctl filesystemlayout create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl filesystemlayout describe filesystemlayout-1 -o yaml > filesystemlayout.yaml $ vi filesystemlayout.yaml $ # either via stdin $ cat filesystemlayout.yaml | metalctl filesystemlayout create -f - $ # or via file $ metalctl filesystemlayout create -f filesystemlayout.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_delete ## metalctl filesystemlayout delete deletes the filesystemlayout ``` metalctl filesystemlayout delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl filesystemlayout describe filesystemlayout-1 -o yaml > filesystemlayout.yaml $ vi filesystemlayout.yaml $ # either via stdin $ cat filesystemlayout.yaml | metalctl filesystemlayout delete -f - $ # or via file $ metalctl filesystemlayout delete -f filesystemlayout.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_describe ## metalctl filesystemlayout describe describes the filesystemlayout ``` metalctl filesystemlayout describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_edit ## metalctl filesystemlayout edit edit the filesystemlayout through an editor and update ``` metalctl filesystemlayout edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_list ## metalctl filesystemlayout list list all filesystemlayouts ``` metalctl filesystemlayout list [flags] ``` ### Options ``` -h, --help help for list --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_match ## metalctl filesystemlayout match check if a machine satisfies all disk requirements of a given filesystemlayout ``` metalctl filesystemlayout match [flags] ``` ### Options ``` --filesystemlayout string filesystemlayout id to check against [required] -h, --help help for match --machine string machine id to check for match [required] ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_try ## metalctl filesystemlayout try try to detect a filesystem by given size and image ``` metalctl filesystemlayout try [flags] ``` ### Options ``` -h, --help help for try --image string image to try --size string size to try ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_filesystemlayout_update ## metalctl filesystemlayout update updates the filesystemlayout ``` metalctl filesystemlayout update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl filesystemlayout describe filesystemlayout-1 -o yaml > filesystemlayout.yaml $ vi filesystemlayout.yaml $ # either via stdin $ cat filesystemlayout.yaml | metalctl filesystemlayout update -f - $ # or via file $ metalctl filesystemlayout update -f filesystemlayout.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl filesystemlayout](./metalctl_filesystemlayout.md) - manage filesystemlayout entities --- ## metalctl_firewall ## metalctl firewall manage firewall entities ### Synopsis firewalls are used to establish network connectivity between metal-stack networks. firewalls are similar to machines but are managed by the provider. almost every command of the machine command subset works on firewalls, too. ### Options ``` -h, --help help for firewall ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl firewall create](./metalctl_firewall_create.md) - creates the firewall * [metalctl firewall describe](./metalctl_firewall_describe.md) - describes the firewall * [metalctl firewall list](./metalctl_firewall_list.md) - list all firewalls * [metalctl firewall ssh](./metalctl_firewall_ssh.md) - SSH to a firewall --- ## metalctl_firewall_create ## metalctl firewall create creates the firewall ``` metalctl firewall create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -d, --description string Description of the firewall to create. [optional] --dnsservers strings dns servers to add to the machine or firewall. [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl firewall describe firewall-1 -o yaml > firewall.yaml $ vi firewall.yaml $ # either via stdin $ cat firewall.yaml | metalctl firewall create -f - $ # or via file $ metalctl firewall create -f firewall.yaml the file can also contain multiple documents and perform a bulk operation. --filesystemlayout string Filesystemlayout to use during machine installation. [optional] --firewall-rules-file string firewall rules specified in a yaml file Example: $ metalctl firewall create ..mandatory args.. --firewall-rules-file rules.yaml rules.yaml --- egress: - comment: allow outgoing https ports: - 443 protocol: TCP to: - 0.0.0.0/0 - comment: allow outgoing dns via tcp ports: - 53 protocol: TCP to: - 0.0.0.0/0 - comment: allow outgoing dns and ntp via udp ports: - 53 - 123 protocol: UDP to: - 0.0.0.0/0 ingress: - comment: allow incoming ssh only to one ip ports: - 22 protocol: TCP from: - 0.0.0.0/0 - 1.2.3.4/32 to: - 212.34.83.19/32 - comment: allow incoming https to all targets ports: - 80 - 433 protocol: TCP from: - 0.0.0.0/0 -h, --help help for create -H, --hostname string Hostname of the firewall. [required] -I, --id string ID of a specific firewall to allocate, if given, size and partition are ignored. Need to be set to reserved (--reserve) state before. -i, --image string OS Image to install. [required] --ips strings Sets the firewall's IP address. Usage: [--ips[=IPV4-ADDRESS[,IPV4-ADDRESS]...]]... IPV4-ADDRESS specifies the IPv4 address to add. It can only be used in conjunction with --networks. -n, --name string Name of the firewall. [optional] --networks strings Adds network(s). Usage: --networks NETWORK[:MODE][,NETWORK[:MODE]]... [--networks NETWORK[:MODE][, NETWORK[:MODE]]...]... NETWORK specifies the id of an existing network. MODE can be omitted or one of: auto IP address is automatically acquired from the given network noauto No automatic IP address acquisition --ntpservers strings ntp servers to add to the machine or firewall. [optional] -S, --partition string partition/datacenter where the firewall is created. [required, except for reserved machines] -P, --project string Project where the firewall should belong to. [required] -s, --size string Size of the firewall. [required, except for reserved machines] --skip-security-prompts skips security prompt for bulk operations -p, --sshpublickey string SSH public key for access via ssh and console. [optional] Can be either the public key as string, or pointing to the public key file to use e.g.: "@~/.ssh/id_rsa.pub". If ~/.ssh/[id_ed25519.pub | id_rsa.pub | id_dsa.pub] is present it will be picked as default, matching the first one in this order. --tags strings tags to add to the firewall, use it like: --tags "tag1,tag2" or --tags "tag3". --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --userdata string cloud-init.io compatible userdata. [optional] Can be either the userdata as string, or pointing to the userdata file to use e.g.: "@/tmp/userdata.cfg". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firewall](./metalctl_firewall.md) - manage firewall entities --- ## metalctl_firewall_describe ## metalctl firewall describe describes the firewall ``` metalctl firewall describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firewall](./metalctl_firewall.md) - manage firewall entities --- ## metalctl_firewall_list ## metalctl firewall list list all firewalls ``` metalctl firewall list [flags] ``` ### Options ``` -h, --help help for list --hostname string allocation hostname to filter [optional] --id string ID to filter [optional] --image string allocation image to filter [optional] --mac string mac to filter [optional] --name string allocation name to filter [optional] --partition string partition to filter [optional] --project string allocation project to filter [optional] --size string size to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|event|id|image|liveliness|partition|project|size|when --tags strings tags to filter, use it like: --tags "tag1,tag2" or --tags "tag3". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firewall](./metalctl_firewall.md) - manage firewall entities --- ## metalctl_firewall_ssh ## metalctl firewall ssh SSH to a firewall ### Synopsis SSH to a firewall via VPN. ``` metalctl firewall ssh [flags] ``` ### Options ``` -h, --help help for ssh -i, --identity string specify identity file to SSH to the firewall like: -i path/to/id_rsa (default "~/.ssh/id_rsa") --reason string the reason why to connect to the firewall through SSH ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firewall](./metalctl_firewall.md) - manage firewall entities --- ## metalctl_firmware ## metalctl firmware manage firmwares ### Synopsis list, upload and remove firmwares. ### Options ``` -h, --help help for firmware ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl firmware delete](./metalctl_firmware_delete.md) - delete a firmware * [metalctl firmware list](./metalctl_firmware_list.md) - list firmwares * [metalctl firmware upload](./metalctl_firmware_upload.md) - upload a firmware --- ## metalctl_firmware_delete ## metalctl firmware delete delete a firmware ### Synopsis deletes the specified firmware. ``` metalctl firmware delete [flags] ``` ### Options ``` --board string the board type (required) -h, --help help for delete --kind string the firmware kind [bmc|bios] (required) --revision string the firmware revision (required) --vendor string the vendor (required) ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firmware](./metalctl_firmware.md) - manage firmwares --- ## metalctl_firmware_list ## metalctl firmware list list firmwares ### Synopsis lists all available firmwares matching the given criteria. ``` metalctl firmware list [flags] ``` ### Options ``` --board string the board type -h, --help help for list --kind string the firmware kind [bmc|bios] --machineid string the machine id (ignores vendor and board flags) --vendor string the vendor ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firmware](./metalctl_firmware.md) - manage firmwares --- ## metalctl_firmware_upload ## metalctl firmware upload upload a firmware ### Options ``` -h, --help help for upload ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firmware](./metalctl_firmware.md) - manage firmwares * [metalctl firmware upload bios](./metalctl_firmware_upload_bios.md) - upload a BIOS firmware * [metalctl firmware upload bmc](./metalctl_firmware_upload_bmc.md) - upload a BMC firmware --- ## metalctl_firmware_upload_bios ## metalctl firmware upload bios upload a BIOS firmware ### Synopsis the given BIOS firmware file will be uploaded and tagged as given revision. ``` metalctl firmware upload bios [flags] ``` ### Options ``` --board string the board type (required) -h, --help help for bios --revision string the BIOS firmware revision (required) --vendor string the vendor (required) ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firmware upload](./metalctl_firmware_upload.md) - upload a firmware --- ## metalctl_firmware_upload_bmc ## metalctl firmware upload bmc upload a BMC firmware ### Synopsis the given BMC firmware file will be uploaded and tagged as given revision. ``` metalctl firmware upload bmc [flags] ``` ### Options ``` --board string the board type (required) -h, --help help for bmc --revision string the BMC firmware revision (required) --vendor string the vendor (required) ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl firmware upload](./metalctl_firmware_upload.md) - upload a firmware --- ## metalctl_health ## metalctl health shows the server health ``` metalctl health [flags] ``` ### Options ``` -h, --help help for health ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## metalctl_image ## metalctl image manage image entities ### Synopsis os images available to be installed on machines. ### Options ``` -h, --help help for image ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl image apply](./metalctl_image_apply.md) - applies one or more images from a given file * [metalctl image create](./metalctl_image_create.md) - creates the image * [metalctl image delete](./metalctl_image_delete.md) - deletes the image * [metalctl image describe](./metalctl_image_describe.md) - describes the image * [metalctl image edit](./metalctl_image_edit.md) - edit the image through an editor and update * [metalctl image list](./metalctl_image_list.md) - list all images * [metalctl image update](./metalctl_image_update.md) - updates the image --- ## metalctl_image_apply ## metalctl image apply applies one or more images from a given file ``` metalctl image apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl image describe image-1 -o yaml > image.yaml $ vi image.yaml $ # either via stdin $ cat image.yaml | metalctl image apply -f - $ # or via file $ metalctl image apply -f image.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_create ## metalctl image create creates the image ``` metalctl image create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -d, --description string Description of the image. --features strings features of the image, can be one of machine|firewall -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl image describe image-1 -o yaml > image.yaml $ vi image.yaml $ # either via stdin $ cat image.yaml | metalctl image create -f - $ # or via file $ metalctl image create -f image.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string ID of the image. -n, --name string Name of the image. --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --url string url of the image. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_delete ## metalctl image delete deletes the image ``` metalctl image delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl image describe image-1 -o yaml > image.yaml $ vi image.yaml $ # either via stdin $ cat image.yaml | metalctl image delete -f - $ # or via file $ metalctl image delete -f image.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_describe ## metalctl image describe describes the image ``` metalctl image describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_edit ## metalctl image edit edit the image through an editor and update ``` metalctl image edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_list ## metalctl image list list all images ``` metalctl image list [flags] ``` ### Options ``` --classification string Classification of this image. --features string Features of this image. -h, --help help for list --id string ID of the image. --name string Name of the image. --os string OS derivate of this image. --show-usage show from how many allocated machines every image is used --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: classification|description|expiration|id|name --version string Version of this image. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_image_update ## metalctl image update updates the image ``` metalctl image update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl image describe image-1 -o yaml > image.yaml $ vi image.yaml $ # either via stdin $ cat image.yaml | metalctl image update -f - $ # or via file $ metalctl image update -f image.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl image](./metalctl_image.md) - manage image entities --- ## metalctl_login ## metalctl login login user and receive token ### Synopsis login and receive token that will be used to authenticate commands. ``` metalctl login [flags] ``` ### Options ``` -h, --help help for login --print-only If true, the token is printed to stdout ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## metalctl_logout ## metalctl logout logout user from OIDC SSO session ``` metalctl logout [flags] ``` ### Options ``` -h, --help help for logout ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## metalctl_machine ## metalctl machine manage machine entities ### Synopsis a machine is a bare metal server provisioned through metal-stack that is intended to run user workload. ### Options ``` -h, --help help for machine ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl machine apply](./metalctl_machine_apply.md) - applies one or more machines from a given file * [metalctl machine console](./metalctl_machine_console.md) - console access to a machine * [metalctl machine consolepassword](./metalctl_machine_consolepassword.md) - fetch the consolepassword for a machine * [metalctl machine create](./metalctl_machine_create.md) - creates the machine * [metalctl machine delete](./metalctl_machine_delete.md) - deletes the machine * [metalctl machine describe](./metalctl_machine_describe.md) - describes the machine * [metalctl machine edit](./metalctl_machine_edit.md) - edit the machine through an editor and update * [metalctl machine identify](./metalctl_machine_identify.md) - manage machine chassis identify LED power * [metalctl machine ipmi](./metalctl_machine_ipmi.md) - display ipmi details of the machine, if no machine ID is given all ipmi addresses are returned. * [metalctl machine issues](./metalctl_machine_issues.md) - display machines which are in a potential bad state * [metalctl machine list](./metalctl_machine_list.md) - list all machines * [metalctl machine lock](./metalctl_machine_lock.md) - lock a machine * [metalctl machine logs](./metalctl_machine_logs.md) - display machine provisioning logs * [metalctl machine power](./metalctl_machine_power.md) - manage machine power * [metalctl machine reinstall](./metalctl_machine_reinstall.md) - reinstalls an already allocated machine * [metalctl machine reserve](./metalctl_machine_reserve.md) - reserve a machine * [metalctl machine update](./metalctl_machine_update.md) - updates the machine * [metalctl machine update-firmware](./metalctl_machine_update-firmware.md) - update a machine firmware --- ## metalctl_machine_apply ## metalctl machine apply applies one or more machines from a given file ``` metalctl machine apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl machine describe machine-1 -o yaml > machine.yaml $ vi machine.yaml $ # either via stdin $ cat machine.yaml | metalctl machine apply -f - $ # or via file $ metalctl machine apply -f machine.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_console ## metalctl machine console console access to a machine ### Synopsis console access to a machine, machine must be created with a ssh public key, authentication is done with your private key. In case the machine did not register properly a direct ipmi console access is available via the --ipmi flag. This is only for administrative access. ``` metalctl machine console [flags] ``` ### Options ``` --admin authenticate as admin (admin only). -h, --help help for console --ipmi use ipmitool with direct network access (admin only). --ipmipassword string overwrite ipmi password (admin only). --ipmiuser string overwrite ipmi user (admin only). -i, --sshidentity string SSH key file, if not given the default ssh key will be used if present [optional]. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_consolepassword ## metalctl machine consolepassword fetch the consolepassword for a machine ``` metalctl machine consolepassword [flags] ``` ### Options ``` -h, --help help for consolepassword --reason string a short description why access to the consolepassword is required ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_create ## metalctl machine create creates the machine ``` metalctl machine create [flags] ``` ### Examples ``` machine create can be done in two different ways: - default with automatic allocation: metalctl machine create \ --hostname worker01 \ --name worker \ --image ubuntu-18.04 \ # query available with: metalctl image list --size t1-small-x86 \ # query available with: metalctl size list --partition test \ # query available with: metalctl partition list --project cluster01 \ --sshpublickey "@~/.ssh/id_rsa.pub" - for metal administration with reserved machines: reserve a machine you want to allocate: metalctl machine reserve 00000000-0000-0000-0000-0cc47ae54694 --description "blocked for maintenance" allocate this machine: metalctl machine create \ --hostname worker01 \ --name worker \ --image ubuntu-18.04 \ # query available with: metalctl image list --project cluster01 \ --sshpublickey "@~/.ssh/id_rsa.pub" \ --id 00000000-0000-0000-0000-0cc47ae54694 after you do not want to use this machine exclusive, remove the reservation: metalctl machine reserve 00000000-0000-0000-0000-0cc47ae54694 --remove Once created the machine installation can not be modified anymore. ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -d, --description string Description of the machine to create. [optional] --dnsservers strings dns servers to add to the machine or firewall. [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl machine describe machine-1 -o yaml > machine.yaml $ vi machine.yaml $ # either via stdin $ cat machine.yaml | metalctl machine create -f - $ # or via file $ metalctl machine create -f machine.yaml the file can also contain multiple documents and perform a bulk operation. --filesystemlayout string Filesystemlayout to use during machine installation. [optional] -h, --help help for create -H, --hostname string Hostname of the machine. [required] -I, --id string ID of a specific machine to allocate, if given, size and partition are ignored. Need to be set to reserved (--reserve) state before. -i, --image string OS Image to install. [required] --ips strings Sets the machine's IP address. Usage: [--ips[=IPV4-ADDRESS[,IPV4-ADDRESS]...]]... IPV4-ADDRESS specifies the IPv4 address to add. It can only be used in conjunction with --networks. -n, --name string Name of the machine. [optional] --networks strings Adds a network. Usage: [--networks NETWORK[:MODE][,NETWORK[:MODE]]...]... NETWORK specifies the name or id of an existing network. MODE cane be omitted or one of: auto IP address is automatically acquired from the given network noauto IP address for the given network must be provided via --ips --ntpservers strings ntp servers to add to the machine or firewall. [optional] -S, --partition string partition/datacenter where the machine is created. [required, except for reserved machines] -P, --project string Project where the machine should belong to. [required] -s, --size string Size of the machine. [required, except for reserved machines] --skip-security-prompts skips security prompt for bulk operations -p, --sshpublickey string SSH public key for access via ssh and console. [optional] Can be either the public key as string, or pointing to the public key file to use e.g.: "@~/.ssh/id_rsa.pub". If ~/.ssh/[id_ed25519.pub | id_rsa.pub | id_dsa.pub] is present it will be picked as default, matching the first one in this order. --tags strings tags to add to the machine, use it like: --tags "tag1,tag2" or --tags "tag3". --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --userdata string cloud-init.io compatible userdata. [optional] Can be either the userdata as string, or pointing to the userdata file to use e.g.: "@/tmp/userdata.cfg". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_delete ## metalctl machine delete deletes the machine ### Synopsis delete a machine and destroy all data stored on the local disks. Once destroyed it is back for usage by other projects. A destroyed machine can not restored anymore ``` metalctl machine delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl machine describe machine-1 -o yaml > machine.yaml $ vi machine.yaml $ # either via stdin $ cat machine.yaml | metalctl machine delete -f - $ # or via file $ metalctl machine delete -f machine.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --remove-from-database remove given machine from the database, is only required for maintenance reasons [optional] (admin only). --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_describe ## metalctl machine describe describes the machine ``` metalctl machine describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_edit ## metalctl machine edit edit the machine through an editor and update ``` metalctl machine edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_identify ## metalctl machine identify manage machine chassis identify LED power ### Options ``` -h, --help help for identify ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl machine identify off](./metalctl_machine_identify_off.md) - power off the machine chassis identify LED * [metalctl machine identify on](./metalctl_machine_identify_on.md) - power on the machine chassis identify LED --- ## metalctl_machine_identify_off ## metalctl machine identify off power off the machine chassis identify LED ### Synopsis set the machine chassis identify LED to off state ``` metalctl machine identify off [flags] ``` ### Options ``` -d, --description string description of the reason for chassis identify LED turn-off. (default "Triggered by metalctl") -h, --help help for off ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine identify](./metalctl_machine_identify.md) - manage machine chassis identify LED power --- ## metalctl_machine_identify_on ## metalctl machine identify on power on the machine chassis identify LED ### Synopsis set the machine chassis identify LED to on state ``` metalctl machine identify on [flags] ``` ### Options ``` -d, --description string description of the reason for chassis identify LED turn-on. -h, --help help for on ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine identify](./metalctl_machine_identify.md) - manage machine chassis identify LED power --- ## metalctl_machine_ipmi ## metalctl machine ipmi display ipmi details of the machine, if no machine ID is given all ipmi addresses are returned. ### Synopsis display ipmi details of the machine, if no machine ID is given all ipmi addresses are returned. Meaning of the emojis: 🚧 Machine is reserved. Reserved machines are not considered for random allocation until the reservation flag is removed. 🔒 Machine is locked. Locked machines can not be deleted until the lock is removed. 💀 Machine is dead. The metal-api does not receive any events from this machine. ❗ Machine has a last event error. The machine has recently encountered an error during the provisioning lifecycle. ❓ Machine is in unknown condition. The metal-api does not receive phoned home events anymore or has never booted successfully. ⭕ Machine is in a provisioning crash loop. Flag can be reset through an API-triggered reboot or when the machine reaches the phoned home state. 🚑 Machine reclaim has failed. The machine was deleted but it is not going back into the available machine pool. 🛡 Machine is connected to our VPN, ssh access only possible via this VPN. ``` metalctl machine ipmi [] [flags] ``` ### Options ``` --bmc-address string bmc ipmi address (needs to include port) to filter [optional] --bmc-mac string bmc mac address to filter [optional] --board-part-number string fru board part number to filter [optional] --chassis-part-number string fru chassis part number to filter [optional] --chassis-part-serial string fru chassis part serial to filter [optional] -h, --help help for ipmi --hostname string allocation hostname to filter [optional] --id string ID to filter [optional] --image string allocation image to filter [optional] --last-event-error-threshold duration the duration up to how long in the past a machine last event error will be counted as an issue [optional] (default 1h0m0s) --mac string mac to filter [optional] --manufacturer string fru manufacturer to filter [optional] --name string allocation name to filter [optional] --network-destination-prefixes string network destination prefixes to filter [optional] --network-ids string network ids to filter [optional] --network-ips string network ips to filter [optional] --partition string partition to filter [optional] --product-part-number string fru product part number to filter [optional] --product-serial string fru product serial to filter [optional] --project string allocation project to filter [optional] --rack string rack to filter [optional] --role string allocation role to filter [optional] --room string room to filter [optional] --size string size to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|bios|bmc|event|id|liveliness|partition|project|rack|size|when --state string state to filter [optional] --tags strings tags to filter, use it like: --tags "tag1,tag2" or --tags "tag3". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl machine ipmi chassis-list](./metalctl_machine_ipmi_chassis-list.md) - display ipmi machines grouped by chassis serial * [metalctl machine ipmi events](./metalctl_machine_ipmi_events.md) - display machine hardware events --- ## metalctl_machine_ipmi_chassis-list ## metalctl machine ipmi chassis-list display ipmi machines grouped by chassis serial ### Synopsis display ipmi machines grouped by chassis serial Meaning of the emojis: 🚧 Machine is reserved. Reserved machines are not considered for random allocation until the reservation flag is removed. 🔒 Machine is locked. Locked machines can not be deleted until the lock is removed. 💀 Machine is dead. The metal-api does not receive any events from this machine. ❗ Machine has a last event error. The machine has recently encountered an error during the provisioning lifecycle. ❓ Machine is in unknown condition. The metal-api does not receive phoned home events anymore or has never booted successfully. ⭕ Machine is in a provisioning crash loop. Flag can be reset through an API-triggered reboot or when the machine reaches the phoned home state. 🚑 Machine reclaim has failed. The machine was deleted but it is not going back into the available machine pool. 🛡 Machine is connected to our VPN, ssh access only possible via this VPN. ``` metalctl machine ipmi chassis-list [flags] ``` ### Options ``` --bmc-address string bmc ipmi address (needs to include port) to filter [optional] --bmc-mac string bmc mac address to filter [optional] --board-part-number string fru board part number to filter [optional] --chassis-part-number string fru chassis part number to filter [optional] --chassis-part-serial string fru chassis part serial to filter [optional] -h, --help help for chassis-list --hostname string allocation hostname to filter [optional] --id string ID to filter [optional] --image string allocation image to filter [optional] --last-event-error-threshold duration the duration up to how long in the past a machine last event error will be counted as an issue [optional] (default 1h0m0s) --mac string mac to filter [optional] --manufacturer string fru manufacturer to filter [optional] --name string allocation name to filter [optional] --network-destination-prefixes string network destination prefixes to filter [optional] --network-ids string network ids to filter [optional] --network-ips string network ips to filter [optional] --partition string partition to filter [optional] --product-part-number string fru product part number to filter [optional] --product-serial string fru product serial to filter [optional] --project string allocation project to filter [optional] --rack string rack to filter [optional] --role string allocation role to filter [optional] --room string room to filter [optional] --size string size to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|bios|bmc|event|id|liveliness|partition|project|rack|size|when --state string state to filter [optional] --tags strings tags to filter, use it like: --tags "tag1,tag2" or --tags "tag3". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine ipmi](./metalctl_machine_ipmi.md) - display ipmi details of the machine, if no machine ID is given all ipmi addresses are returned. --- ## metalctl_machine_ipmi_events ## metalctl machine ipmi events display machine hardware events ``` metalctl machine ipmi events [flags] ``` ### Options ``` -h, --help help for events --ipmipassword string overwrite ipmi password (admin only). --ipmiuser string overwrite ipmi user (admin only). -n, --last string show last log entries. (default "10") ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine ipmi](./metalctl_machine_ipmi.md) - display ipmi details of the machine, if no machine ID is given all ipmi addresses are returned. --- ## metalctl_machine_issues ## metalctl machine issues display machines which are in a potential bad state ### Synopsis display machines which are in a potential bad state Meaning of the emojis: 🚧 Machine is reserved. Reserved machines are not considered for random allocation until the reservation flag is removed. 🔒 Machine is locked. Locked machines can not be deleted until the lock is removed. 💀 Machine is dead. The metal-api does not receive any events from this machine. ❗ Machine has a last event error. The machine has recently encountered an error during the provisioning lifecycle. ❓ Machine is in unknown condition. The metal-api does not receive phoned home events anymore or has never booted successfully. ⭕ Machine is in a provisioning crash loop. Flag can be reset through an API-triggered reboot or when the machine reaches the phoned home state. 🚑 Machine reclaim has failed. The machine was deleted but it is not going back into the available machine pool. 🛡 Machine is connected to our VPN, ssh access only possible via this VPN. ``` metalctl machine issues [] [flags] ``` ### Options ``` --bmc-address string bmc ipmi address (needs to include port) to filter [optional] --bmc-mac string bmc mac address to filter [optional] --board-part-number string fru board part number to filter [optional] --chassis-part-number string fru chassis part number to filter [optional] --chassis-part-serial string fru chassis part serial to filter [optional] -h, --help help for issues --hostname string allocation hostname to filter [optional] --id string ID to filter [optional] --image string allocation image to filter [optional] --last-event-error-threshold duration the duration up to how long in the past a machine last event error will be counted as an issue [optional] --mac string mac to filter [optional] --manufacturer string fru manufacturer to filter [optional] --name string allocation name to filter [optional] --network-destination-prefixes string network destination prefixes to filter [optional] --network-ids string network ids to filter [optional] --network-ips string network ips to filter [optional] --omit strings issue types to omit [optional] --only strings issue types to include [optional] --partition string partition to filter [optional] --product-part-number string fru product part number to filter [optional] --product-serial string fru product serial to filter [optional] --project string allocation project to filter [optional] --rack string rack to filter [optional] --role string allocation role to filter [optional] --room string room to filter [optional] --severity string issue severity to include [optional] --size string size to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|bios|bmc|event|id|liveliness|partition|project|rack|size|when --state string state to filter [optional] --tags strings tags to filter, use it like: --tags "tag1,tag2" or --tags "tag3". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl machine issues list](./metalctl_machine_issues_list.md) - list all machine issues that the metal-api can evaluate --- ## metalctl_machine_issues_list ## metalctl machine issues list list all machine issues that the metal-api can evaluate ``` metalctl machine issues list [flags] ``` ### Options ``` -h, --help help for list --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: id|severity ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine issues](./metalctl_machine_issues.md) - display machines which are in a potential bad state --- ## metalctl_machine_list ## metalctl machine list list all machines ### Synopsis list all machines Meaning of the emojis: 🚧 Machine is reserved. Reserved machines are not considered for random allocation until the reservation flag is removed. 🔒 Machine is locked. Locked machines can not be deleted until the lock is removed. 💀 Machine is dead. The metal-api does not receive any events from this machine. ❗ Machine has a last event error. The machine has recently encountered an error during the provisioning lifecycle. ❓ Machine is in unknown condition. The metal-api does not receive phoned home events anymore or has never booted successfully. ⭕ Machine is in a provisioning crash loop. Flag can be reset through an API-triggered reboot or when the machine reaches the phoned home state. 🚑 Machine reclaim has failed. The machine was deleted but it is not going back into the available machine pool. 🛡 Machine is connected to our VPN, ssh access only possible via this VPN. ``` metalctl machine list [flags] ``` ### Options ``` --bmc-address string bmc ipmi address (needs to include port) to filter [optional] --bmc-mac string bmc mac address to filter [optional] --board-part-number string fru board part number to filter [optional] --chassis-part-number string fru chassis part number to filter [optional] --chassis-part-serial string fru chassis part serial to filter [optional] -h, --help help for list --hostname string allocation hostname to filter [optional] --id string ID to filter [optional] --image string allocation image to filter [optional] --last-event-error-threshold duration the duration up to how long in the past a machine last event error will be counted as an issue [optional] (default 1h0m0s) --mac string mac to filter [optional] --manufacturer string fru manufacturer to filter [optional] --name string allocation name to filter [optional] --network-destination-prefixes string network destination prefixes to filter [optional] --network-ids string network ids to filter [optional] --network-ips string network ips to filter [optional] --partition string partition to filter [optional] --product-part-number string fru product part number to filter [optional] --product-serial string fru product serial to filter [optional] --project string allocation project to filter [optional] --rack string rack to filter [optional] --role string allocation role to filter [optional] --room string room to filter [optional] --size string size to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|event|id|image|liveliness|partition|project|rack|size|when --state string state to filter [optional] --tags strings tags to filter, use it like: --tags "tag1,tag2" or --tags "tag3". ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_lock ## metalctl machine lock lock a machine ### Synopsis when a machine is locked, it can not be destroyed, to destroy a machine you must first remove the lock from that machine with --remove ``` metalctl machine lock [flags] ``` ### Options ``` -d, --description string description of the reason for the lock. -h, --help help for lock -r, --remove remove the lock. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_logs ## metalctl machine logs display machine provisioning logs ``` metalctl machine logs [flags] ``` ### Options ``` -h, --help help for logs --last-event-error-threshold duration the duration up to how long in the past a machine last event error will be counted as an issue [optional] (default 168h0m0s) ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_power ## metalctl machine power manage machine power ### Options ``` -h, --help help for power ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl machine power bios](./metalctl_machine_power_bios.md) - boot a machine into BIOS * [metalctl machine power cycle](./metalctl_machine_power_cycle.md) - power cycle a machine (graceful shutdown) * [metalctl machine power disk](./metalctl_machine_power_disk.md) - boot a machine from disk * [metalctl machine power off](./metalctl_machine_power_off.md) - power off a machine * [metalctl machine power on](./metalctl_machine_power_on.md) - power on a machine * [metalctl machine power pxe](./metalctl_machine_power_pxe.md) - boot a machine from PXE * [metalctl machine power reset](./metalctl_machine_power_reset.md) - power reset a machine --- ## metalctl_machine_power_bios ## metalctl machine power bios boot a machine into BIOS ### Synopsis the machine will boot into bios. (machine does not reboot automatically) ``` metalctl machine power bios [flags] ``` ### Options ``` -h, --help help for bios ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_cycle ## metalctl machine power cycle power cycle a machine (graceful shutdown) ### Synopsis (soft) cycle the machine power. ``` metalctl machine power cycle [flags] ``` ### Options ``` -h, --help help for cycle ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_disk ## metalctl machine power disk boot a machine from disk ### Synopsis the machine will boot from disk. (machine does not reboot automatically) ``` metalctl machine power disk [flags] ``` ### Options ``` -h, --help help for disk ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_off ## metalctl machine power off power off a machine ### Synopsis set the machine to power off state, if the machine already was off nothing happens. It will usually take some time to power off the machine, depending on the machine type. Power on will therefore not work if the machine is in the powering off phase. ``` metalctl machine power off [flags] ``` ### Options ``` -h, --help help for off ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_on ## metalctl machine power on power on a machine ### Synopsis set the machine to power on state, if the machine already was on nothing happens. ``` metalctl machine power on [flags] ``` ### Options ``` -h, --help help for on ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_pxe ## metalctl machine power pxe boot a machine from PXE ### Synopsis the machine will boot from PXE. (machine does not reboot automatically) ``` metalctl machine power pxe [flags] ``` ### Options ``` -h, --help help for pxe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_power_reset ## metalctl machine power reset power reset a machine ### Synopsis (hard) reset the machine power. ``` metalctl machine power reset [flags] ``` ### Options ``` -h, --help help for reset ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine power](./metalctl_machine_power.md) - manage machine power --- ## metalctl_machine_reinstall ## metalctl machine reinstall reinstalls an already allocated machine ### Synopsis reinstalls an already allocated machine. If it is not yet allocated, nothing happens, otherwise only the machine's primary disk is wiped and the new image will subsequently be installed on that device ``` metalctl machine reinstall [flags] ``` ### Options ``` -d, --description string description of the reinstallation. [optional] -h, --help help for reinstall --image string id of the image to get installed. [required] ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_reserve ## metalctl machine reserve reserve a machine ### Synopsis reserve a machine for exclusive usage, this machine will no longer be picked by other allocations. This is useful for maintenance of the machine or testing. After the reservation is not needed anymore, the reservation should be removed with --remove. ``` metalctl machine reserve [flags] ``` ### Options ``` -d, --description string description of the reason for the reservation. -h, --help help for reserve -r, --remove remove the reservation. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_machine_update-firmware ## metalctl machine update-firmware update a machine firmware ### Options ``` -h, --help help for update-firmware ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities * [metalctl machine update-firmware bios](./metalctl_machine_update-firmware_bios.md) - update a machine BIOS * [metalctl machine update-firmware bmc](./metalctl_machine_update-firmware_bmc.md) - update a machine BMC --- ## metalctl_machine_update-firmware_bios ## metalctl machine update-firmware bios update a machine BIOS ### Synopsis the machine BIOS will be updated to given revision. If revision flag is not specified an update plan will be printed instead. ``` metalctl machine update-firmware bios [flags] ``` ### Options ``` --description string the reason why the BIOS should be updated -h, --help help for bios --revision string the BIOS revision ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine update-firmware](./metalctl_machine_update-firmware.md) - update a machine firmware --- ## metalctl_machine_update-firmware_bmc ## metalctl machine update-firmware bmc update a machine BMC ### Synopsis the machine BMC will be updated to given revision. If revision flag is not specified an update plan will be printed instead. ``` metalctl machine update-firmware bmc [flags] ``` ### Options ``` --description string the reason why the BMC should be updated -h, --help help for bmc --revision string the BMC revision ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine update-firmware](./metalctl_machine_update-firmware.md) - update a machine firmware --- ## metalctl_machine_update ## metalctl machine update updates the machine ``` metalctl machine update [flags] ``` ### Options ``` --add-tags strings tags to be added to the machine [optional] --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --description string the description of the machine [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl machine describe machine-1 -o yaml > machine.yaml $ vi machine.yaml $ # either via stdin $ cat machine.yaml | metalctl machine update -f - $ # or via file $ metalctl machine update -f machine.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --remove-tags strings tags to be removed from the machine [optional] --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl machine](./metalctl_machine.md) - manage machine entities --- ## metalctl_markdown ## metalctl markdown create markdown documentation ``` metalctl markdown [flags] ``` ### Options ``` -h, --help help for markdown ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## metalctl_network ## metalctl network manage network entities ### Synopsis networks can be attached to a machine or firewall such that they can communicate with each other. ### Options ``` -h, --help help for network ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl network allocate](./metalctl_network_allocate.md) - allocate a network * [metalctl network apply](./metalctl_network_apply.md) - applies one or more networks from a given file * [metalctl network create](./metalctl_network_create.md) - creates the network * [metalctl network delete](./metalctl_network_delete.md) - deletes the network * [metalctl network describe](./metalctl_network_describe.md) - describes the network * [metalctl network edit](./metalctl_network_edit.md) - edit the network through an editor and update * [metalctl network free](./metalctl_network_free.md) - free a network * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities * [metalctl network list](./metalctl_network_list.md) - list all networks * [metalctl network update](./metalctl_network_update.md) - updates the network --- ## metalctl_network_allocate ## metalctl network allocate allocate a network ``` metalctl network allocate [flags] ``` ### Options ``` --addressfamily string addressfamily of the network to acquire, if not specified the network inherits the address families from the parent [optional] -d, --description string description of the network to create. [optional] --dmz use this private network as dmz. [optional] -h, --help help for allocate --ipv4-prefix-length int ipv4 prefix bit length of the network to create, defaults to default child prefix length of the parent network. [optional] --ipv6-prefix-length int ipv6 prefix bit length of the network to create, defaults to default child prefix length of the parent network. [optional] --labels strings labels for this network. [optional] -n, --name string name of the network to create. [required] --partition string partition where this network should exist. [required] --project string partition where this network should exist. [required] --shared shared allows usage of this private network from other networks ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_apply ## metalctl network apply applies one or more networks from a given file ``` metalctl network apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl network describe network-1 -o yaml > network.yaml $ vi network.yaml $ # either via stdin $ cat network.yaml | metalctl network apply -f - $ # or via file $ metalctl network apply -f network.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_create ## metalctl network create creates the network ``` metalctl network create [flags] ``` ### Options ``` --additional-announcable-cidrs strings list of cidrs which are added to the route maps per tenant private network, these are typically pod- and service cidrs, can only be set in a supernetwork --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --default-ipv4-child-prefix-length int default child prefix length for ipv4 prefixes for private super networks. --default-ipv6-child-prefix-length int default child prefix length for ipv6 prefixes for private super networks. -d, --description string description of the network to create. [optional] --destination-prefixes strings destination prefixes in this network. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl network describe network-1 -o yaml > network.yaml $ vi network.yaml $ # either via stdin $ cat network.yaml | metalctl network create -f - $ # or via file $ metalctl network create -f network.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string id of the network to create. [optional] --labels strings add initial labels, must be in the form of key=value, use it like: --labels "key1=value1,key2=value2". -n, --name string name of the network to create. [optional] --nat set nat flag of network, if set to true, traffic from this network will be natted. -p, --partition string partition where this network should exist. --prefixes strings prefixes in this network. --privatesuper set private super flag of network, if set to true, this network is used to start machines there. --project string project of the network to create. [optional] --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --underlay set underlay flag of network, if set to true, this is used to transport underlay network traffic --vrf int vrf of this network --vrfshared vrf shared allows multiple networks to share a vrf ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_delete ## metalctl network delete deletes the network ``` metalctl network delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl network describe network-1 -o yaml > network.yaml $ vi network.yaml $ # either via stdin $ cat network.yaml | metalctl network delete -f - $ # or via file $ metalctl network delete -f network.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_describe ## metalctl network describe describes the network ``` metalctl network describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_edit ## metalctl network edit edit the network through an editor and update ``` metalctl network edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_free ## metalctl network free free a network ``` metalctl network free [flags] ``` ### Options ``` -h, --help help for free ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_ip ## metalctl network ip manage ip entities ### Synopsis an ip address can be attached to a machine or firewall such that network traffic can be routed to these servers. ### Options ``` -h, --help help for ip ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities * [metalctl network ip apply](./metalctl_network_ip_apply.md) - applies one or more ips from a given file * [metalctl network ip create](./metalctl_network_ip_create.md) - creates the ip * [metalctl network ip delete](./metalctl_network_ip_delete.md) - deletes the ip * [metalctl network ip describe](./metalctl_network_ip_describe.md) - describes the ip * [metalctl network ip edit](./metalctl_network_ip_edit.md) - edit the ip through an editor and update * [metalctl network ip issues](./metalctl_network_ip_issues.md) - display ips which are in a potential bad state * [metalctl network ip list](./metalctl_network_ip_list.md) - list all ips * [metalctl network ip update](./metalctl_network_ip_update.md) - updates the ip --- ## metalctl_network_ip_apply ## metalctl network ip apply applies one or more ips from a given file ``` metalctl network ip apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl ip describe ip-1 -o yaml > ip.yaml $ vi ip.yaml $ # either via stdin $ cat ip.yaml | metalctl ip apply -f - $ # or via file $ metalctl ip apply -f ip.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_create ## metalctl network ip create creates the ip ``` metalctl network ip create [flags] ``` ### Options ``` --addressfamily string addressfamily of the ip to acquire, defaults to IPv4 [optional] --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -d, --description string description of the IP to allocate. [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl ip describe ip-1 -o yaml > ip.yaml $ vi ip.yaml $ # either via stdin $ cat ip.yaml | metalctl ip create -f - $ # or via file $ metalctl ip create -f ip.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --ipaddress string a specific ip address to allocate. [optional] -n, --name string name of the IP to allocate. [optional] --network string network from where the IP should be allocated. --project string project for which the IP should be allocated. --skip-security-prompts skips security prompt for bulk operations --tags strings tags to attach to the IP. --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --type string type of the IP to allocate: ephemeral|static [optional] (default "ephemeral") ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_delete ## metalctl network ip delete deletes the ip ``` metalctl network ip delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl ip describe ip-1 -o yaml > ip.yaml $ vi ip.yaml $ # either via stdin $ cat ip.yaml | metalctl ip delete -f - $ # or via file $ metalctl ip delete -f ip.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_describe ## metalctl network ip describe describes the ip ``` metalctl network ip describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_edit ## metalctl network ip edit edit the ip through an editor and update ``` metalctl network ip edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_issues ## metalctl network ip issues display ips which are in a potential bad state ``` metalctl network ip issues [flags] ``` ### Options ``` -h, --help help for issues ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_list ## metalctl network ip list list all ips ``` metalctl network ip list [flags] ``` ### Options ``` --addressfamily string addressfamily of the ip to filter, defaults to all addressfamilies [optional] -h, --help help for list --ipaddress string ipaddress to filter [optional] --machineid string machineid to filter [optional] --name string name to filter [optional] --network string network to filter [optional] --prefix string prefix to filter [optional] --project string project to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: age|description|id|ipaddress|name|network|type --tags strings tags to filter [optional] --type string type to filter [optional] ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_ip_update ## metalctl network ip update updates the ip ``` metalctl network ip update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl ip describe ip-1 -o yaml > ip.yaml $ vi ip.yaml $ # either via stdin $ cat ip.yaml | metalctl ip update -f - $ # or via file $ metalctl ip update -f ip.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network ip](./metalctl_network_ip.md) - manage ip entities --- ## metalctl_network_list ## metalctl network list list all networks ``` metalctl network list [flags] ``` ### Options ``` --addressfamily string addressfamily to filter, either ipv4 or ipv6 [optional] --destination-prefixes strings destination prefixes to filter, use it like: --destination-prefixes prefix1,prefix2. -h, --help help for list --id string ID to filter [optional] --name string name to filter [optional] --nat nat to filter [optional] --parent string parent network to filter [optional] --partition string partition to filter [optional] --prefixes strings prefixes to filter, use it like: --prefixes prefix1,prefix2. --privatesuper privatesuper to filter [optional] --project string project to filter [optional] --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name|partition|project --underlay underlay to filter [optional] --vrf int vrf to filter [optional] ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_network_update ## metalctl network update updates the network ``` metalctl network update [flags] ``` ### Options ``` --add-destinationprefixes strings destination prefixes to be added to the network [optional] --add-prefixes strings prefixes to be added to the network [optional] --additional-announcable-cidrs strings list of cidrs which are added to the route maps per tenant private network, these are typically pod- and service cidrs, can only be set in a supernetwork --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --description string the description of the network [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl network describe network-1 -o yaml > network.yaml $ vi network.yaml $ # either via stdin $ cat network.yaml | metalctl network update -f - $ # or via file $ metalctl network update -f network.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --labels strings the labels of the network, must be in the form of key=value, use it like: --labels "key1=value1,key2=value2". [optional] --name string the name of the network [optional] --remove-destinationprefixes strings destination prefixes to be removed from the network [optional] --remove-prefixes strings prefixes to be removed from the network [optional] --shared marks a network as shared or not [optional] --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl network](./metalctl_network.md) - manage network entities --- ## metalctl_partition ## metalctl partition manage partition entities ### Synopsis a partition is a failure domain in the data center. ### Options ``` -h, --help help for partition ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl partition apply](./metalctl_partition_apply.md) - applies one or more partitions from a given file * [metalctl partition capacity](./metalctl_partition_capacity.md) - show partition capacity * [metalctl partition create](./metalctl_partition_create.md) - creates the partition * [metalctl partition delete](./metalctl_partition_delete.md) - deletes the partition * [metalctl partition describe](./metalctl_partition_describe.md) - describes the partition * [metalctl partition edit](./metalctl_partition_edit.md) - edit the partition through an editor and update * [metalctl partition list](./metalctl_partition_list.md) - list all partitions * [metalctl partition update](./metalctl_partition_update.md) - updates the partition --- ## metalctl_partition_apply ## metalctl partition apply applies one or more partitions from a given file ``` metalctl partition apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl partition describe partition-1 -o yaml > partition.yaml $ vi partition.yaml $ # either via stdin $ cat partition.yaml | metalctl partition apply -f - $ # or via file $ metalctl partition apply -f partition.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_capacity ## metalctl partition capacity show partition capacity ``` metalctl partition capacity [flags] ``` ### Options ``` -h, --help help for capacity --id string filter on partition id. [optional] --project-id string consider project-specific counts, e.g. size reservations. [optional] --size string filter on size id. [optional] --sort-by strings order by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_create ## metalctl partition create creates the partition ``` metalctl partition create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --cmdline string kernel commandline for the metal-hammer in the partition. [required] -d, --description string Description of the partition. [required] --dnsservers string dns servers for the machines and firewalls in the partition. [optional] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl partition describe partition-1 -o yaml > partition.yaml $ vi partition.yaml $ # either via stdin $ cat partition.yaml | metalctl partition create -f - $ # or via file $ metalctl partition create -f partition.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string ID of the partition. [required] --imageurl string initrd for the metal-hammer in the partition. [required] --kernelurl string kernel url for the metal-hammer in the partition. [required] --mgmtserver string management server address in the partition. [required] -n, --name string Name of the partition. [optional] --ntpservers string ntp servers for the machines and firewalls in the partition. [optional] --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_delete ## metalctl partition delete deletes the partition ``` metalctl partition delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl partition describe partition-1 -o yaml > partition.yaml $ vi partition.yaml $ # either via stdin $ cat partition.yaml | metalctl partition delete -f - $ # or via file $ metalctl partition delete -f partition.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_describe ## metalctl partition describe describes the partition ``` metalctl partition describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_edit ## metalctl partition edit edit the partition through an editor and update ``` metalctl partition edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_list ## metalctl partition list list all partitions ``` metalctl partition list [flags] ``` ### Options ``` -h, --help help for list --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_partition_update ## metalctl partition update updates the partition ``` metalctl partition update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl partition describe partition-1 -o yaml > partition.yaml $ vi partition.yaml $ # either via stdin $ cat partition.yaml | metalctl partition update -f - $ # or via file $ metalctl partition update -f partition.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl partition](./metalctl_partition.md) - manage partition entities --- ## metalctl_project ## metalctl project manage project entities ### Synopsis a project belongs to a tenant and groups together entities in metal-stack. ### Options ``` -h, --help help for project ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl project apply](./metalctl_project_apply.md) - applies one or more projects from a given file * [metalctl project create](./metalctl_project_create.md) - creates the project * [metalctl project delete](./metalctl_project_delete.md) - deletes the project * [metalctl project describe](./metalctl_project_describe.md) - describes the project * [metalctl project edit](./metalctl_project_edit.md) - edit the project through an editor and update * [metalctl project list](./metalctl_project_list.md) - list all projects * [metalctl project update](./metalctl_project_update.md) - updates the project --- ## metalctl_project_apply ## metalctl project apply applies one or more projects from a given file ``` metalctl project apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl project describe project-1 -o yaml > project.yaml $ vi project.yaml $ # either via stdin $ cat project.yaml | metalctl project apply -f - $ # or via file $ metalctl project apply -f project.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_create ## metalctl project create creates the project ``` metalctl project create [flags] ``` ### Options ``` --annotation strings add initial annotation, must be in the form of key=value, can be given multiple times to add multiple annotations, e.g. --annotation key=value --annotation foo=bar --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --cluster-quota int32 cluster quota --description string description of the project. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl project describe project-1 -o yaml > project.yaml $ vi project.yaml $ # either via stdin $ cat project.yaml | metalctl project create -f - $ # or via file $ metalctl project create -f project.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --ip-quota int32 ip quota --label strings add initial label, can be given multiple times to add multiple labels, e.g. --label=foo --label=bar --machine-quota int32 machine quota --name string name of the project, max 10 characters. --skip-security-prompts skips security prompt for bulk operations --tenant string create project for given tenant --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_delete ## metalctl project delete deletes the project ``` metalctl project delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl project describe project-1 -o yaml > project.yaml $ vi project.yaml $ # either via stdin $ cat project.yaml | metalctl project delete -f - $ # or via file $ metalctl project delete -f project.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_describe ## metalctl project describe describes the project ``` metalctl project describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_edit ## metalctl project edit edit the project through an editor and update ``` metalctl project edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_list ## metalctl project list list all projects ``` metalctl project list [flags] ``` ### Options ``` -h, --help help for list --id string ID of the project. --name string Name of the project. --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name|tenant --tenant string tenant of this project. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_project_update ## metalctl project update updates the project ``` metalctl project update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl project describe project-1 -o yaml > project.yaml $ vi project.yaml $ # either via stdin $ cat project.yaml | metalctl project update -f - $ # or via file $ metalctl project update -f project.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl project](./metalctl_project.md) - manage project entities --- ## metalctl_size ## metalctl size manage size entities ### Synopsis a size matches a machine in terms of cpu cores, ram and storage. ### Options ``` -h, --help help for size ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl size apply](./metalctl_size_apply.md) - applies one or more sizes from a given file * [metalctl size create](./metalctl_size_create.md) - creates the size * [metalctl size delete](./metalctl_size_delete.md) - deletes the size * [metalctl size describe](./metalctl_size_describe.md) - describes the size * [metalctl size edit](./metalctl_size_edit.md) - edit the size through an editor and update * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities * [metalctl size list](./metalctl_size_list.md) - list all sizes * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities * [metalctl size suggest](./metalctl_size_suggest.md) - suggest size from a given machine id * [metalctl size update](./metalctl_size_update.md) - updates the size --- ## metalctl_size_apply ## metalctl size apply applies one or more sizes from a given file ``` metalctl size apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl size describe size-1 -o yaml > size.yaml $ vi size.yaml $ # either via stdin $ cat size.yaml | metalctl size apply -f - $ # or via file $ metalctl size apply -f size.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_create ## metalctl size create creates the size ``` metalctl size create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -d, --description string Description of the size. [required] -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl size describe size-1 -o yaml > size.yaml $ vi size.yaml $ # either via stdin $ cat size.yaml | metalctl size create -f - $ # or via file $ metalctl size create -f size.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string ID of the size. [required] --max int min value of given size constraint type. [required] --min int min value of given size constraint type. [required] -n, --name string Name of the size. [optional] --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations --type string type of constraints. [required] ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_delete ## metalctl size delete deletes the size ``` metalctl size delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl size describe size-1 -o yaml > size.yaml $ vi size.yaml $ # either via stdin $ cat size.yaml | metalctl size delete -f - $ # or via file $ metalctl size delete -f size.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_describe ## metalctl size describe describes the size ``` metalctl size describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_edit ## metalctl size edit edit the size through an editor and update ``` metalctl size edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_imageconstraint ## metalctl size imageconstraint manage imageconstraint entities ### Synopsis if a size has specific requirements regarding the images which must fulfill certain constraints, this can be configured here. ### Options ``` -h, --help help for imageconstraint ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities * [metalctl size imageconstraint apply](./metalctl_size_imageconstraint_apply.md) - applies one or more imageconstraints from a given file * [metalctl size imageconstraint create](./metalctl_size_imageconstraint_create.md) - creates the imageconstraint * [metalctl size imageconstraint delete](./metalctl_size_imageconstraint_delete.md) - deletes the imageconstraint * [metalctl size imageconstraint describe](./metalctl_size_imageconstraint_describe.md) - describes the imageconstraint * [metalctl size imageconstraint edit](./metalctl_size_imageconstraint_edit.md) - edit the imageconstraint through an editor and update * [metalctl size imageconstraint list](./metalctl_size_imageconstraint_list.md) - list all imageconstraints * [metalctl size imageconstraint try](./metalctl_size_imageconstraint_try.md) - try if size and image can be allocated * [metalctl size imageconstraint update](./metalctl_size_imageconstraint_update.md) - updates the imageconstraint --- ## metalctl_size_imageconstraint_apply ## metalctl size imageconstraint apply applies one or more imageconstraints from a given file ``` metalctl size imageconstraint apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl imageconstraint describe imageconstraint-1 -o yaml > imageconstraint.yaml $ vi imageconstraint.yaml $ # either via stdin $ cat imageconstraint.yaml | metalctl imageconstraint apply -f - $ # or via file $ metalctl imageconstraint apply -f imageconstraint.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_create ## metalctl size imageconstraint create creates the imageconstraint ``` metalctl size imageconstraint create [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl imageconstraint describe imageconstraint-1 -o yaml > imageconstraint.yaml $ vi imageconstraint.yaml $ # either via stdin $ cat imageconstraint.yaml | metalctl imageconstraint create -f - $ # or via file $ metalctl imageconstraint create -f imageconstraint.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_delete ## metalctl size imageconstraint delete deletes the imageconstraint ``` metalctl size imageconstraint delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl imageconstraint describe imageconstraint-1 -o yaml > imageconstraint.yaml $ vi imageconstraint.yaml $ # either via stdin $ cat imageconstraint.yaml | metalctl imageconstraint delete -f - $ # or via file $ metalctl imageconstraint delete -f imageconstraint.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_describe ## metalctl size imageconstraint describe describes the imageconstraint ``` metalctl size imageconstraint describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_edit ## metalctl size imageconstraint edit edit the imageconstraint through an editor and update ``` metalctl size imageconstraint edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_list ## metalctl size imageconstraint list list all imageconstraints ``` metalctl size imageconstraint list [flags] ``` ### Options ``` -h, --help help for list --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_try ## metalctl size imageconstraint try try if size and image can be allocated ``` metalctl size imageconstraint try [flags] ``` ### Options ``` -h, --help help for try --image string image to check if allocaltion is possible --size string size to check if allocaltion is possible ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_imageconstraint_update ## metalctl size imageconstraint update updates the imageconstraint ``` metalctl size imageconstraint update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl imageconstraint describe imageconstraint-1 -o yaml > imageconstraint.yaml $ vi imageconstraint.yaml $ # either via stdin $ cat imageconstraint.yaml | metalctl imageconstraint update -f - $ # or via file $ metalctl imageconstraint update -f imageconstraint.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size imageconstraint](./metalctl_size_imageconstraint.md) - manage imageconstraint entities --- ## metalctl_size_list ## metalctl size list list all sizes ``` metalctl size list [flags] ``` ### Options ``` -h, --help help for list --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_reservation ## metalctl size reservation manage reservation entities ### Synopsis manage size reservations ### Options ``` -h, --help help for reservation ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities * [metalctl size reservation apply](./metalctl_size_reservation_apply.md) - applies one or more reservations from a given file * [metalctl size reservation create](./metalctl_size_reservation_create.md) - creates the reservation * [metalctl size reservation delete](./metalctl_size_reservation_delete.md) - deletes the reservation * [metalctl size reservation describe](./metalctl_size_reservation_describe.md) - describes the reservation * [metalctl size reservation edit](./metalctl_size_reservation_edit.md) - edit the reservation through an editor and update * [metalctl size reservation list](./metalctl_size_reservation_list.md) - list all reservations * [metalctl size reservation update](./metalctl_size_reservation_update.md) - updates the reservation * [metalctl size reservation usage](./metalctl_size_reservation_usage.md) - see current usage of size reservations --- ## metalctl_size_reservation_apply ## metalctl size reservation apply applies one or more reservations from a given file ``` metalctl size reservation apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl reservation describe reservation-1 -o yaml > reservation.yaml $ vi reservation.yaml $ # either via stdin $ cat reservation.yaml | metalctl reservation apply -f - $ # or via file $ metalctl reservation apply -f reservation.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_create ## metalctl size reservation create creates the reservation ``` metalctl size reservation create [flags] ``` ### Options ``` --amount int32 the amount to associate with this reservation --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --description string the description to associate with this reservation -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl reservation describe reservation-1 -o yaml > reservation.yaml $ vi reservation.yaml $ # either via stdin $ cat reservation.yaml | metalctl reservation create -f - $ # or via file $ metalctl reservation create -f reservation.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string the id to associate with this reservation --labels strings the labels to associate with this reservation --partitions strings the partition ids to associate with this reservation --project string the project id to associate with this reservation --size string the size id to associate with this reservation --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_delete ## metalctl size reservation delete deletes the reservation ``` metalctl size reservation delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl reservation describe reservation-1 -o yaml > reservation.yaml $ vi reservation.yaml $ # either via stdin $ cat reservation.yaml | metalctl reservation delete -f - $ # or via file $ metalctl reservation delete -f reservation.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_describe ## metalctl size reservation describe describes the reservation ``` metalctl size reservation describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_edit ## metalctl size reservation edit edit the reservation through an editor and update ``` metalctl size reservation edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_list ## metalctl size reservation list list all reservations ``` metalctl size reservation list [flags] ``` ### Options ``` -h, --help help for list --id string the id to filter --partition string the partition id to filter --project string the project id to filter --size string the size id to filter --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: amount|id|partition|project|size ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_update ## metalctl size reservation update updates the reservation ``` metalctl size reservation update [flags] ``` ### Options ``` --amount int32 the amount to associate with this reservation --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --description string the description to associate with this reservation -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl reservation describe reservation-1 -o yaml > reservation.yaml $ vi reservation.yaml $ # either via stdin $ cat reservation.yaml | metalctl reservation update -f - $ # or via file $ metalctl reservation update -f reservation.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --labels strings the labels to associate with this reservation --partitions strings the partition ids to associate with this reservation --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_reservation_usage ## metalctl size reservation usage see current usage of size reservations ``` metalctl size reservation usage [flags] ``` ### Options ``` -h, --help help for usage --partition string the partition to filter --project string the project to filter --size-id string the size-id to filter --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: amount|id|partition|project|size|used-amount ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size reservation](./metalctl_size_reservation.md) - manage reservation entities --- ## metalctl_size_suggest ## metalctl size suggest suggest size from a given machine id ``` metalctl size suggest [flags] ``` ### Options ``` --description string The description of the suggested size (default "a suggested size") -h, --help help for suggest --labels strings labels to add to the size --machine-id string Machine id used to create the size suggestion. [required] --name string The name of the suggested size (default "suggested-size") ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_size_update ## metalctl size update updates the size ``` metalctl size update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl size describe size-1 -o yaml > size.yaml $ vi size.yaml $ # either via stdin $ cat size.yaml | metalctl size update -f - $ # or via file $ metalctl size update -f size.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl size](./metalctl_size.md) - manage size entities --- ## metalctl_switch ## metalctl switch manage switch entities ### Synopsis switch are the leaf switches in the data center that are controlled by metal-stack. ### Options ``` -h, --help help for switch ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl switch connected-machines](./metalctl_switch_connected-machines.md) - shows switches with their connected machines * [metalctl switch console](./metalctl_switch_console.md) - connect to the switch console * [metalctl switch delete](./metalctl_switch_delete.md) - deletes the switch * [metalctl switch describe](./metalctl_switch_describe.md) - describes the switch * [metalctl switch detail](./metalctl_switch_detail.md) - switch details * [metalctl switch edit](./metalctl_switch_edit.md) - edit the switch through an editor and update * [metalctl switch list](./metalctl_switch_list.md) - list all switches * [metalctl switch migrate](./metalctl_switch_migrate.md) - migrate machine connections and other configuration from one switch to another * [metalctl switch port](./metalctl_switch_port.md) - sets the given switch port state up or down * [metalctl switch replace](./metalctl_switch_replace.md) - put a leaf switch into replace mode in preparation for physical replacement. For a description of the steps involved see the long help. * [metalctl switch ssh](./metalctl_switch_ssh.md) - connect to the switch via ssh * [metalctl switch update](./metalctl_switch_update.md) - updates the switch --- ## metalctl_switch_connected-machines ## metalctl switch connected-machines shows switches with their connected machines ``` metalctl switch connected-machines [flags] ``` ### Examples ``` The command will show the machines connected to the switch ports. Can also be used with -o template in order to generate CSV-style output: $ metalctl switch connected-machines -o template --template '{{ $machines := .machines }}{{ range .switches }}{{ $switch := . }}{{ range .connections }}{{ $switch.id }},{{ $switch.rack_id }},{{ .nic.name }},{{ .machine_id }},{{ (index $machines .machine_id).ipmi.fru.product_serial }}{{ printf "\n" }}{{ end }}{{ end }}' r01leaf01,swp1,f78cc340-e5e8-48ed-8fe7-2336c1e2ded2, r01leaf01,swp2,44e3a522-5f48-4f3c-9188-41025f9e401e, ... ``` ### Options ``` -h, --help help for connected-machines --id string ID of the switch. --machine-id string The id of the connected machine, ignores size flag if set. --name string Name of the switch. --os-vendor string OS vendor of this switch. --os-version string OS version of this switch. --partition string Partition of this switch. --rack string Rack of this switch. --room string Room of this switch. --size string Size of the connected machines. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_console ## metalctl switch console connect to the switch console ### Synopsis this requires a network connectivity to the ip address of the console server this switch is connected to. ``` metalctl switch console [flags] ``` ### Options ``` -h, --help help for console ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_delete ## metalctl switch delete deletes the switch ``` metalctl switch delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl switch describe switch-1 -o yaml > switch.yaml $ vi switch.yaml $ # either via stdin $ cat switch.yaml | metalctl switch delete -f - $ # or via file $ metalctl switch delete -f switch.yaml the file can also contain multiple documents and perform a bulk operation. --force forcefully delete the switch accepting the risk that it still has machines connected to it -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_describe ## metalctl switch describe describes the switch ``` metalctl switch describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_detail ## metalctl switch detail switch details ``` metalctl switch detail [flags] ``` ### Options ``` -h, --help help for detail --id string ID of the switch. --name string Name of the switch. --os-vendor string OS vendor of this switch. --os-version string OS version of this switch. --partition string Partition of this switch. --rack string Rack of this switch. --room string Room of this switch. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_edit ## metalctl switch edit edit the switch through an editor and update ``` metalctl switch edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_list ## metalctl switch list list all switches ``` metalctl switch list [flags] ``` ### Options ``` -h, --help help for list --id string ID of the switch. --name string Name of the switch. --os-vendor string OS vendor of this switch. --os-version string OS version of this switch. --partition string Partition of this switch. --rack string Rack of this switch. --room string Room of this switch. --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_migrate ## metalctl switch migrate migrate machine connections and other configuration from one switch to another ``` metalctl switch migrate [flags] ``` ### Options ``` -h, --help help for migrate ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_port ## metalctl switch port sets the given switch port state up or down ### Options ``` -h, --help help for port --port string the port to be changed. ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities * [metalctl switch port describe](./metalctl_switch_port_describe.md) - gets the given switch port state * [metalctl switch port down](./metalctl_switch_port_down.md) - sets the given switch port state down * [metalctl switch port up](./metalctl_switch_port_up.md) - sets the given switch port state up --- ## metalctl_switch_port_describe ## metalctl switch port describe gets the given switch port state ### Synopsis shows the current actual and desired state of the port of the given switch. ``` metalctl switch port describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --port string the port to be changed. --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch port](./metalctl_switch_port.md) - sets the given switch port state up or down --- ## metalctl_switch_port_down ## metalctl switch port down sets the given switch port state down ### Synopsis sets the port status to DOWN so the connected machine will not be able to connect to the switch. ``` metalctl switch port down [flags] ``` ### Options ``` -h, --help help for down ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --port string the port to be changed. --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch port](./metalctl_switch_port.md) - sets the given switch port state up or down --- ## metalctl_switch_port_up ## metalctl switch port up sets the given switch port state up ### Synopsis sets the port status to UP so the connected machine will be able to connect to the switch. ``` metalctl switch port up [flags] ``` ### Options ``` -h, --help help for up ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --port string the port to be changed. --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch port](./metalctl_switch_port.md) - sets the given switch port state up or down --- ## metalctl_switch_replace ## metalctl switch replace put a leaf switch into replace mode in preparation for physical replacement. For a description of the steps involved see the long help. ### Synopsis Put a leaf switch into replace mode in preparation for physical replacement Operational steps to replace a switch: - Put the switch that needs to be replaced in replace mode with this command - Replace the switch MAC address in the metal-stack deployment configuration - Make sure that interfaces on the new switch do not get connected to the PXE-bridge immediately by setting the interfaces list of the respective leaf switch to [] in the metal-stack deployment configuration - Deploy the management servers so that the dhcp servers will serve the right address and DHCP options to the new switch - Replace the switch physically. Be careful to ensure that the cabling mirrors the remaining leaf exactly because the new switch information will be cloned from the remaining switch! Also make sure to have console access to the switch so you can start and monitor the install process - If the switch is not in onie install mode but already has an operating system installed, put it into install mode with "sudo onie-select -i -f -v" and reboot it. Now the switch should be provisioned with a management IP from a management server, install itself with the right software image and receive license and ssh keys through ZTP. You can check whether that process has completed successfully with the command "sudo ztp -s". The ZTP state should be disabled and the result should be success. - Deploy the switch plane and metal-core through metal-stack deployment CI job - The switch will now register with its metal-api, and the metal-core service will receive the cloned interface and routing information. You can verify successful switch replacement by checking the interface and BGP configuration, and checking the switch status with "metalctl switch ls -o wide"; it should now be operational again ``` metalctl switch replace [flags] ``` ### Options ``` -h, --help help for replace ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_ssh ## metalctl switch ssh connect to the switch via ssh ### Synopsis this requires a network connectivity to the management ip address of the switch. ``` metalctl switch ssh [flags] ``` ### Options ``` -h, --help help for ssh ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_switch_update ## metalctl switch update updates the switch ``` metalctl switch update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl switch describe switch-1 -o yaml > switch.yaml $ vi switch.yaml $ # either via stdin $ cat switch.yaml | metalctl switch update -f - $ # or via file $ metalctl switch update -f switch.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl switch](./metalctl_switch.md) - manage switch entities --- ## metalctl_tenant ## metalctl tenant manage tenant entities ### Synopsis a tenant belongs to a tenant and groups together entities in metal-stack. ### Options ``` -h, --help help for tenant ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl tenant apply](./metalctl_tenant_apply.md) - applies one or more tenants from a given file * [metalctl tenant create](./metalctl_tenant_create.md) - creates the tenant * [metalctl tenant delete](./metalctl_tenant_delete.md) - deletes the tenant * [metalctl tenant describe](./metalctl_tenant_describe.md) - describes the tenant * [metalctl tenant edit](./metalctl_tenant_edit.md) - edit the tenant through an editor and update * [metalctl tenant list](./metalctl_tenant_list.md) - list all tenants * [metalctl tenant update](./metalctl_tenant_update.md) - updates the tenant --- ## metalctl_tenant_apply ## metalctl tenant apply applies one or more tenants from a given file ``` metalctl tenant apply [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl tenant describe tenant-1 -o yaml > tenant.yaml $ vi tenant.yaml $ # either via stdin $ cat tenant.yaml | metalctl tenant apply -f - $ # or via file $ metalctl tenant apply -f tenant.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for apply --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_create ## metalctl tenant create creates the tenant ``` metalctl tenant create [flags] ``` ### Options ``` --annotations strings add initial annotations, must be in the form of key=value, can be given multiple times to add multiple annotations, e.g. --annotation key=value --annotation foo=bar --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. --cluster-quota int32 cluster quota --description string description of the tenant. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl tenant describe tenant-1 -o yaml > tenant.yaml $ vi tenant.yaml $ # either via stdin $ cat tenant.yaml | metalctl tenant create -f - $ # or via file $ metalctl tenant create -f tenant.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for create --id string id of the tenant, max 10 characters. --ip-quota int32 ip quota --labels strings add initial label, can be given multiple times to add multiple labels, e.g. --label=foo --label=bar --machine-quota int32 machine quota --name string name of the tenant, max 10 characters. --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_delete ## metalctl tenant delete deletes the tenant ``` metalctl tenant delete [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl tenant describe tenant-1 -o yaml > tenant.yaml $ vi tenant.yaml $ # either via stdin $ cat tenant.yaml | metalctl tenant delete -f - $ # or via file $ metalctl tenant delete -f tenant.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for delete --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_describe ## metalctl tenant describe describes the tenant ``` metalctl tenant describe [flags] ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_edit ## metalctl tenant edit edit the tenant through an editor and update ``` metalctl tenant edit [flags] ``` ### Options ``` -h, --help help for edit ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_list ## metalctl tenant list list all tenants ``` metalctl tenant list [flags] ``` ### Options ``` --annotations strings annotations -h, --help help for list --id string ID of the tenant. --name string Name of the tenant. --sort-by strings sort by (comma separated) column(s), sort direction can be changed by appending :asc or :desc behind the column identifier. possible values: description|id|name ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_tenant_update ## metalctl tenant update updates the tenant ``` metalctl tenant update [flags] ``` ### Options ``` --bulk-output when used with --file (bulk operation): prints results at the end as a list. default is printing results intermediately during the operation, which causes single entities to be printed in a row. -f, --file string filename of the create or update request in yaml format, or - for stdin. Example: $ metalctl tenant describe tenant-1 -o yaml > tenant.yaml $ vi tenant.yaml $ # either via stdin $ cat tenant.yaml | metalctl tenant update -f - $ # or via file $ metalctl tenant update -f tenant.yaml the file can also contain multiple documents and perform a bulk operation. -h, --help help for update --skip-security-prompts skips security prompt for bulk operations --timestamps when used with --file (bulk operation): prints timestamps in-between the operations ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl tenant](./metalctl_tenant.md) - manage tenant entities --- ## metalctl_update ## metalctl update update the program ### Options ``` -h, --help help for update ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl update check](./metalctl_update_check.md) - check for update of the program * [metalctl update do](./metalctl_update_do.md) - do the update of the program --- ## metalctl_update_check ## metalctl update check check for update of the program ``` metalctl update check [flags] ``` ### Options ``` -h, --help help for check ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl update](./metalctl_update.md) - update the program --- ## metalctl_update_do ## metalctl update do do the update of the program ``` metalctl update do [flags] ``` ### Options ``` -h, --help help for do -v, --version string the version to update to, by default updates to the supported version, use "latest" to update to latest version ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl update](./metalctl_update.md) - update the program --- ## metalctl_version ## metalctl version print the client and server version information ``` metalctl version [flags] ``` ### Options ``` -h, --help help for version ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## metalctl_vpn ## metalctl vpn access VPN ### Synopsis access VPN ### Options ``` -h, --help help for vpn ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api * [metalctl vpn key](./metalctl_vpn_key.md) - create an auth key --- ## metalctl_vpn_key ## metalctl vpn key create an auth key ### Synopsis create an auth key to connect to VPN ``` metalctl vpn key [flags] ``` ### Examples ``` auth key for tailscale can be created by this command: metalctl vpn key \ -- project cluster01 ``` ### Options ``` --ephemeral create an ephemeral key (default true) -h, --help help for key --project string project ID for which auth key should be created --reason string a short description why access to the vpn is required ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl vpn](./metalctl_vpn.md) - access VPN --- ## metalctl_whoami ## metalctl whoami shows current user ### Synopsis shows the current user, that will be used to authenticate commands. ``` metalctl whoami [flags] ``` ### Options ``` -h, --help help for whoami ``` ### Options inherited from parent commands ``` --api-token string api token to authenticate. Can be specified with METALCTL_API_TOKEN environment variable. --api-url string api server address. Can be specified with METALCTL_API_URL environment variable. -c, --config string alternative config file path, (default is ~/.metalctl/config.yaml). Example config.yaml: --- apitoken: "alongtoken" ... --debug debug output --force-color force colored output even without tty --kubeconfig string Path to the kube-config to use for authentication and authorization. Is updated by login. Uses default path if not specified. --no-headers do not print headers of table output format (default print headers) -o, --output-format string output format (table|wide|markdown|json|yaml|template), wide is a table with more columns. (default "table") --template string output template for template output-format, go template format. For property names inspect the output of -o json or -o yaml for reference. Example for machines: metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs) ``` ### SEE ALSO * [metalctl](./metalctl.md) - a cli to manage entities in the metal-stack api --- ## backup-restore-sidecar # K8s Backup Restore Sidecar for Databases ![Go version](https://img.shields.io/github/go-mod/go-version/metal-stack/backup-restore-sidecar) [![Go Report Card](https://goreportcard.com/badge/github.com/metal-stack/backup-restore-sidecar)](https://goreportcard.com/report/github.com/metal-stack/backup-restore-sidecar) [![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/metal-stack/backup-restore-sidecar) [![Build](https://github.com/metal-stack/backup-restore-sidecar/actions/workflows/docker.yaml/badge.svg?branch=master)](https://github.com/metal-stack/backup-restore-sidecar/actions) [![Slack](https://img.shields.io/badge/slack-metal--stack-brightgreen.svg?logo=slack)](https://metal-stack.slack.com/) This project adds automatic backup and recovery to databases managed by K8s via sidecar. The idea is taken from the [etcd-backup-restore](https://github.com/gardener/etcd-backup-restore) project. Probably, it does not make sense to use this project with large databases. However, if it is certain that a database will never grow large, the auto-recovery mechanism can come in very handy. ## Supported Databases | Database | Image | Status | Upgrade Support | | --------- | ------------ | :----: | :-------------: | | postgres | >= 12-alpine | beta | ✅ | | rethinkdb | >= 2.4.0 | beta | ❌ | | ETCD | >= 3.5 | alpha | ❌ | | redis | >= 6.0 | alpha | ❌ | | keydb | >= 6.0 | alpha | ❌ | | valkey | >= 8.1 | alpha | ❌ | | localfs | | alpha | ❌ | Postgres also supports updates when using the TimescaleDB extension. Please consider the integration test for supported upgrade paths. > [!IMPORTANT] > Upgrade from 12-alpine to 13-alpine is not possible because of library differences in icu-lib. > The solution is to upgrade to a older 14.10-alpine which has the same icu-lib version as 12-alpine > and then update to 14.18-alpine or newer which does not require to run pg_upgrade. > It is also recommended to pin the original database to postgres:12.22-alpine to ensure the latest minor. > Upgrade from 14.18-alpine to 15-alpine is not possible because of version differences in ICU. > The solution is to upgrade to 15.13-alpine, followed by 15.18-alpine before upgrading to 17.10-alpine. ## Database Upgrades ### Postgres Postgres requires special treatment if a major version upgrade is planned. `pg_upgrade` needs to be called with the old and new binaries, also the old data directory and a already initialized data directory which was initialized with the new binary, e.g. `initdb `. To make this process as smooth as possible, backup-restore-sidecar will detect if the version of the database files and the version of the postgres binary. If the binary is newer than the database files it will start the upgrade process. Strict validation to ensure all prerequisites are met is done before actually starting the upgrade process. To achieve this, `backup-restore-sidecar` saves the postgres binaries in the database directory in the form of `pg-bin-v12` for postgres 12. If later the database version is upgraded, the previous postgres binaries are present for doing the actual upgrade. ## Supported Compression Methods With `--compression-method` you can define how generated backups are compressed before stored at the storage provider. Available compression methods are: | compression-method | suffix | comments | | ------------------ | -------- | -------------------------------------------------------------------------------------------- | | tar | .tar | no compression, best suited for already compressed content | | targz | .tar.gz | tar and gzip, most commonly used, best compression ratio, average performance | | tarlz4 | .tar.lz4 | tar and lz4, very fast compression/decompression speed compared to gz, slightly bigger files | ## Supported Storage Providers - GCS Buckets - S3 Buckets (tested against Ceph RADOS gateway) - Local ## Encryption For all three storage providers AES encryption is supported and can be enabled with `--encryption-key=`. The key must be 32 bytes (AES-256) long. The backups are stored at the storage provider with the `.aes` suffix. If the file does not have this suffix, decryption is skipped. ## How it works In a recovery scenario, control plane state can be restored from regular backups taken by the `backup-restore-sidecar` component to S3-compatible object storage. On startup, the affected database automatically restores from the referenced backup without manual intervention. The process is illustrated in the following diagram: ![Sequence Diagram](./assets/sequence.drawio.svg) ## Limitations - The database is deployed unclustered / standalone - The database is deployed as a statefulset and the data is backed by a PVC - No "Point in Time Recovery" (PITR) ## Using Multiple Backup-Restore-Sidecars On a Single Bucket It is possible to let multiple backup-restore-sidecars (for different databases) use the same backup bucket at an external provider. However, it has to be noted that these sidecars must all configure a dedicated object prefix in which they store the backups. Otherwise they would overwrite each other's data. Be aware that, if you change the object prefix under which the backups are stored, the old lifecycle policies matching this prefix are not automatically cleaned up and have to be removed manually. ## Try it out Requires: - [docker](https://www.docker.com/) - [kind](https://github.com/kubernetes-sigs/kind) - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) - [stern](https://github.com/wercker/stern) To start a demo / devel setup, run: `make start-postgres` or `make start-rethinkdb`. By default, the backup-restore-sidecar will start with the `local` backup provider, which is probably not very useful for most use-cases. If you want to test storing the data at a real backup provider, then: 1. Configure the backup provider secret in `deploy/provider-secret-.yaml`. 2. Run `BACKUP_PROVIDER= make start-postgres` instead. ## Manual restoration The advantage of the `backup-restore-sidecar` is that it automatically restores the latest backup automatically in case your data is lost. There can be situations though where you need to restore a specific backup from the past manually. In order to manually restore a specific backup version with the `backup-restore-sidecar`, use the following steps: Take a copy of your existing stateful set by running: ```bash kubectl get sts -o yaml ``` Now, get into a clean state, i.e. delete the existing stateful set and the pvc of your database Deploy the exact stateful set you had but only with the backup-restore-sidecar container and tail some file such that container does not die. This is your "helper" stateful set, which you can use for manual administration. - For postgres check the example [here](https://github.com/metal-stack/backup-restore-sidecar/blob/master/deploy/postgres_manual_restore.yaml) - For rethinkdb check the example [here](https://github.com/metal-stack/backup-restore-sidecar/blob/master/deploy/rethinkdb_manual_restore.yaml) Enter the container in your "helper" pod by running: ```bash kubectl exec -it -0 -c backup-restore-sidecar -- bash ``` Inside the container, you can view the existing backup versions using ```bash backup-restore-sidecar restore ls ``` Choose the version to restore by running ```bash backup-restore-sidecar restore ``` The backup was now restored, you can exit the container and remove the "helper" stateful set but keep the pvc! ```bash kubectl delete sts ``` Now, deploy the regular backup-restore-sidecar stateful set again. It will find out that all the data is in place and the database will start normally --- ## go-ipam [![Actions](https://github.com/metal-stack/go-ipam/actions/workflows/docker.yml/badge.svg?branch=master)](https://github.com/metal-stack/go-ipam/actions) [![GoDoc](https://godoc.org/github.com/metal-stack/go-ipam?status.svg)](https://godoc.org/github.com/metal-stack/go-ipam) [![Go Report Card](https://goreportcard.com/badge/github.com/metal-stack/go-ipam)](https://goreportcard.com/report/github.com/metal-stack/go-ipam) [![codecov](https://codecov.io/gh/metal-stack/go-ipam/branch/master/graph/badge.svg)](https://codecov.io/gh/metal-stack/go-ipam) [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/metal-stack/go-ipam/blob/master/LICENSE) go-ipam is a module to handle IP address management. It can operate on networks, prefixes and IPs. It also comes as a ready to go microservice which offers a grpc api. ## IP Most obvious this library is all about IP management. The main purpose is to acquire and release an IP, or a bunch of IP's from prefixes. ## Prefix A prefix is a network with IP and mask, typically in the form of *192.168.0.0/24*. To be able to manage IPs you have to create a prefix first. Library Example usage: ```go package main import ( "context" "fmt" "time" goipam "github.com/metal-stack/go-ipam" ) func main() { // The background context bgCtx := context.Background() // Create a ipamer with in memory storage ipam := goipam.New(bgCtx) // Optionally, we can pass around a context for a given namespace namespace := "tenant-a" err := ipam.CreateNamespace(bgCtx, namespace) if err != nil { panic(err) } ctx := goipam.NewContextWithNamespace(bgCtx, namespace) ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() // Create a prefix to manage some IPs prefix, err := ipam.NewPrefix(ctx, "192.168.0.0/24") if err != nil { panic(err) } // Acquire and release an IP with this prefix ip, err := ipam.AcquireIP(ctx, prefix.Cidr) if err != nil { panic(err) } fmt.Printf("got IP: %s\n", ip.IP) prefix, err = ipam.ReleaseIP(ctx, ip) if err != nil { panic(err) } fmt.Printf("IP: %s released.\n", ip.IP) // Now a IPv6 Super Prefix with Child Prefixes prefix, err = ipam.NewPrefix(ctx, "2001:aabb::/48") if err != nil { panic(err) } cp1, err := ipam.AcquireChildPrefix(ctx, prefix.Cidr, 64) if err != nil { panic(err) } fmt.Printf("got Prefix: %s\n", cp1) cp2, err := ipam.AcquireChildPrefix(ctx, prefix.Cidr, 72) if err != nil { panic(err) } fmt.Printf("got Prefix: %s\n", cp2) ip21, err := ipam.AcquireIP(ctx, cp2.Cidr) if err != nil { panic(err) } fmt.Printf("got IP: %s\n", ip21.IP) } ``` ## GRPC Service First start the go-ipam container with the database backend of your choice already up and running. For example if you have a postgres database for storing the ipam data, you could run the grpc service like so: ```bash docker run -it --rm ghcr.io/metal-stack/go-ipam postgres ``` From a client perspective you can now talk to this service via grpc. GRPC Example usage: ```go package main import ( "context" "http" v1 "github.com/metal-stack/go-ipam/api/v1" "github.com/metal-stack/go-ipam/api/v1/apiv1connect" ) func main() { c := apiv1connect.NewIpamServiceClient( http.DefaultClient, "http://localhost:9090", connect.WithGRPC(), ) bgCtx := context.Background() // Optional with Namespace ctx := goipam.NewContextWithNamespace(bgCtx, "tenant-a") result, err := c.CreatePrefix(ctx, &v1.CreatePrefixRequest{Cidr: "192.168.0.0/16"}) if err != nil { panic(err) } fmt.Println("Prefix:%q created", result.GetPrefix().GetCidr()) } ``` ## GRPC client There is also a `cli` provided in the container which can be used to make calls to the grpc endpoint manually: ```bash docker run -it --rm --entrypoint /cli ghcr.io/metal-stack/go-ipam ``` ## Metrics ```bash http://localhost:2112/metrics ``` ## pprof ```bash go tool pprof -http :8080 localhost:2113/debug/pprof/heap go tool pprof -http :8080 localhost:2113/debug/pprof/goroutine ``` ## Docker Compose example Ensure you have docker with compose support installed. Then execute the following command: ```bash docker compose up -d # check if up and running docker compose ps NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS go-ipam-ipam-1 go-ipam "/server postgres" ipam 14 seconds ago Up 13 seconds (healthy) 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp go-ipam-postgres-1 postgres:alpine "docker-entrypoint.s…" postgres 8 minutes ago Up 13 seconds 5432/tcp # Then execute the cli to create prefixes and acquire ips docker compose exec ipam /cli prefix create --cidr 192.168.0.0/16 prefix:"192.168.0.0/16" created docker compose exec ipam /cli ip acquire --prefix 192.168.0.0/16 ip:"192.168.0.1" acquired # Queries can also made against the Rest api like so: curl -v -X POST -d '{}' -H 'Content-Type: application/json' localhost:9090/api.v1.IpamService/ListPrefixes ``` ## Supported Databases & Performance | Database | Acquire Child Prefix | Acquire IP | New Prefix | Prefix Overlap | Production-Ready | Geo-Redundant | |:------------|---------------------:|------------:|------------:|---------------:|:-----------------|:--------------| | In-Memory | 106,861/sec | 196,687/sec | 330,578/sec | 248/sec | N | N | | File | | | | | N | N | | KeyDB | 777/sec | 975/sec | 2,271/sec | | Y | Y | | Redis | 773/sec | 958/sec | 2,349/sec | | Y | N | | MongoDB | 415/sec | 682/sec | 772/sec | | Y | Y | | Etcd | 258/sec | 368/sec | 533/sec | | Y | N | | Postgres | 203/sec | 331/sec | 472/sec | | Y | N | | CockroachDB | 170/sec | 300/sec | 470/sec | | Y | Y | The benchmarks above were performed using: * cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz * postgres:17-alpine * cockroach:v24.1.0 * redis:7.4-alpine * keydb:alpine_x86_64_v6.3.1 * etcd:v3.5.15 * mongodb:7 ### Database Version Compatibility | Database | Details | |-------------|---------------------------------------------------------------------------------------------------------------------------| | KeyDB | | | Redis | | | MongoDB | [mongodb-go compatibility](https://www.mongodb.com/docs/drivers/go/current/compatibility/#std-label-golang-compatibility) | | Etcd | | | Postgres | | | CockroachDB | | ## Testing individual Backends It is possible to test a individual backend only to speed up development roundtrip. `backend` can be one of `Memory`, `Postgres`, `Cockroach`, `Etcd`, `Redis`, and `MongoDB`. ```bash BACKEND=backend make test ``` --- ## masterdata-api This Microservice provides the source of truth for master data. * tenant master-data * name/desc * tenant settings * cloud * limit max projects * limit max clusters * cluster * limit * tenant project * settings * limit max clusters * tenant members * tenant project members * version ## Design The services are exposed as grpc-services. The included client can be used in other services by simply importing it. The data is stored in a generic way using a postgres database with tables consisting of id and json-document fields. Changes to the data are reflected in a history table-twin per entity. When data is created, updated or deleted, the change is also written to the history table. The main entities are generated from a `.proto`-file plus some additional mapping-code in a `.go` file. Using a go-generate-statement the db-schema and some boilerplating code is generated using naming-conventions. ## Initial Data It is possible to insert data on startup, this is done by placing one ore more yaml documents into the `initdb.d` directory. Multi document yaml files are not supported at the moment. If the given version of the entity is lower or equal the entity version stored in the database, no create or update happens. Otherwise a update is executed. On every error happening during initdb is logged, but the affected entity is not processed. ## Build ```bash make all ``` ## Run ```bash make postgres-up ``` Start client with extensive logging ```bash make clean protoc client GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info bin/client ``` Start server ```bash make clean protoc server bin/server ``` ## Metrics ```bash http://localhost:2112/metrics ``` ## pprof ```bash go tool pprof -http :8080 localhost:2113/debug/pprof/heap go tool pprof -http :8080 localhost:2113/debug/pprof/goroutine ``` ## Generics migration In order to get rid of all the reflection based logic in `postgres.go`, we decided to migrate to generics which are available since go 1.18. This leads to much nicer code to read and also brings some benefits regarding allocations. Performance is at the same level as the reflection based approach. To measure the impact, a bunch of benchmarks have been implemented for all CRUD operations provided by `postgres.go`. Results comparing old(reflection based) vs. new(generics based): ```plain name old time/op new time/op delta GetTenant-16 92.9µs ±11% 94.4µs ± 5% ~ (p=0.421 n=5+5) CreateTenant-16 3.06ms ± 9% 3.40ms ± 4% +10.95% (p=0.008 n=5+5) UpdateTenant-16 3.59ms ± 9% 3.81ms ±19% ~ (p=0.548 n=5+5) FindTenant-16 259µs ±12% 224µs ± 3% -13.75% (p=0.008 n=5+5) name old alloc/op new alloc/op delta GetTenant-16 5.68kB ± 0% 4.40kB ± 0% -22.55% (p=0.029 n=4+4) CreateTenant-16 10.8kB ± 0% 9.6kB ± 0% -11.22% (p=0.008 n=5+5) UpdateTenant-16 22.7kB ± 0% 19.0kB ± 0% -16.26% (p=0.008 n=5+5) FindTenant-16 7.15kB ± 0% 5.19kB ± 0% -27.38% (p=0.016 n=4+5) name old allocs/op new allocs/op delta GetTenant-16 118 ± 0% 92 ± 0% -22.03% (p=0.008 n=5+5) CreateTenant-16 238 ± 0% 204 ± 0% -14.29% (p=0.008 n=5+5) UpdateTenant-16 500 ± 0% 408 ± 0% -18.40% (p=0.008 n=5+5) FindTenant-16 146 ± 0% 108 ± 0% -26.03% (p=0.008 n=5+5) ``` As shown, performance is about the same, but allocations in terms of bytes and count have been reduced quite significant. --- ## metal-api [![Build](https://github.com/metal-stack/metal-api/actions/workflows/docker.yaml/badge.svg?branch=master)](https://github.com/metal-stack/metal-api/actions) [![Slack](https://img.shields.io/badge/slack-metal--stack-brightgreen.svg?logo=slack)](https://metal-stack.slack.com/) [![Go Report Card](https://goreportcard.com/badge/github.com/metal-stack/metal-api)](https://goreportcard.com/report/github.com/metal-stack/metal-api) [![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/metal-stack/metal-api) [![Docker Pulls](https://img.shields.io/docker/pulls/metalstack/metal-api.svg)](https://hub.docker.com/r/metalstack/metal-api/) The metal-api is one of the major components of the metal-stack control plane. It is both the public interface for users to manage machines, networks, ips, and so forth and it is also the interface for metal-stack components running inside a partition. The CLI tool for using the API is called `metalctl`. You can find this project [here](https://github.com/metal-stack/metalctl). --- ## metal-apiserver [![Actions](https://github.com/metal-stack/metal-apiserver/actions/workflows/build.yaml/badge.svg?branch=main)](https://github.com/metal-stack/metal-apiserver/actions) [![Go Report Card](https://goreportcard.com/badge/github.com/metal-stack/metal-apiserver)](https://goreportcard.com/report/github.com/metal-stack/metal-apiserver) [![codecov](https://codecov.io/gh/metal-stack/metal-apiserver/branch/main/graph/badge.svg)](https://codecov.io/gh/metal-stack/metal-apiserver) This is meant as a POC implementation of the metal-stack [V2 API](https://github.com/metal-stack/api) used to provide a MVP of our metal-stack offering. --- ## metal-console `metal-console` provides access to the serial console of metal machines. This is achieved by acting as a bridge between ssh and the console protocol of the concrete machine. It will support either libvirt based console access, which is used in the development environment. On real hardware ipmi based lanplus sol (Serial Over LAN) will be used. To access the console execute: ```bash ssh -i @ ``` The metal-console will then lookup the given username as machine uuid on metal-api, request which console protocol to use. If the machine uuid is a valid machine, it will then use the provided private key to authenticate against the ssh public key stored in the metal-api for this machine. If access is granted, the user will have access to the console. `metal-console` figures out in which partition the machine is located and then opens a tls socket connection to `metal-bmc` running on the management server in this partition. `metal-bmc` checks if the tls client certificate matches. If this is the case, it looks up the machine ipmi details from `metal-api` and starts a ipmi sol session to the machine. ## TODO - If a second console access starts to same machine, kill existing one --- ## tenant-apiserver This microservice provides the source of truth for tenant-related data. * Tenants * Tenant members * Project * Project members * Quotas * Version ## Design The API definition and the client are included in a dedicated repository that can be found on: https://github.com/metal-stack/tenant-api. The services are exposed as grpc-services using connectrpc. The data is stored in a generic way using a postgres database with tables consisting of id and json-document fields. Changes to the data are reflected in a history table-twin per entity. When data is created, updated or deleted, the change is also written to the history table. ## Metrics ```bash http://localhost:2112/metrics ``` ### pprof ```bash go tool pprof -http :8080 localhost:2113/debug/pprof/heap go tool pprof -http :8080 localhost:2113/debug/pprof/goroutine ``` --- ## helm-charts # Helm Charts Contains Helm Charts of metal-stack. The charts can be consumed as follows: ```bash helm install oci://ghcr.io/metal-stack/helm-charts/ --version # for pull requests: # helm install oci://ghcr.io/metal-stack/helm-charts/ --version -pull-request. # for latest: # helm install oci://ghcr.io/metal-stack/helm-charts/ --version -master ``` In addition to that, charts are also provided on https://helm.metal-stack.io. They can be used like this: 1. Add the charts repository to your helm repository index. ```bash helm repo add metal-stack https://helm.metal-stack.io ``` 1. Use the charts. ```bash helm install metal-stack/ ``` ## Contributing If you want to add modifications to a chart, make sure to also bump your chart version in the chart's `Chart.yaml` file. This is necessary because a release of this repository does not overwrite existing chart releases (as this would potentially break already released deployments). For PRs, your changes will be released to a special pull requests URL `https://helm.metal-stack.io/pull_requests/`. You can use this repository instead in order to test your changes. --- ## ARCHITECTURE(Metal-images) # Architectural Decisions ## Goals for creating OS images * minimal set of installed packages * customization has to be possible * reproducible builds ## Our approach * build an image based on the published docker images of OS vendors * add necessary packages and services in a Dockerfile for the OS to provide: * sudo functionality * an SSH-Server * the [FRR-Suite](https://frrouting.org/) for BGP (the servers act as a BGP router/speaker) * [ignition](https://www.flatcar.org/docs/latest/provisioning/ignition/) for userdata execution * `mdadm` for raid fsl configuration * `lvm` for volume group fsl provisioning * `grub` for bootloader installation * `timesyncd` for NTP time synchronization * these services will be configured by the [os-installer](https://github.com/metal-stack/os-installer) invoked during machine provisioning by the metal-hammer * for out-of-tree custom images, build owners might want to provide a `/etc/metal/os-installer.yaml` file within the image that can provide special instructions for the os-installer --- ## IMAGE_STORE # Image store The metal-images tarballs are stored on . The tarball must be compressed using lz4 and a md5 checksum must be provided as well. To be able to have an insight what packages are included in this image a `packages.txt` with the output of `dpkg -l`. The actual directory layout should look like: ```bash ///img.tar.lz4 ///img.tar.lz4.md5 ///packages.txt ``` Where `` is `stable` for the master branch, `` for releases and `pull_requests/-` for pull requests. `` is the name of the os in use, some images like `firewall` are derived from another os image (in this case the `ubuntu` image). `` specifies the major and minor number of the OS, which is case of ubuntu "19.10", "19.10", "20.04" and so on. This version must follow the semantic versioning specification, whereas we tolerate a leading zero for the minor version which is quite common for some OSes. `` must follow the semantic version requirements for `patch`, we defined that patch is always in the form of "YYYYMMDD` for example 20191018. To specify the image for machine creation the full qualified image must be in the form of: ``, e.g. `ubuntu-19.10.20191018`. From the metal-api perspective, there are two possibilities to specify a image to create a machine: 1. specify major.minor without patch, e.g. `--image ubuntu-19.10` 1. specify major.minor.patch `--image ubuntu-19.10.20191018` In the first case a most recent version resolution is taken place in the metal-api to resolve to the most recent available image for ubuntu-19.10, which will be then for example ubuntu-19.10.20191018, this image is the stored in the machine allocation. The second form guarantees the machine creation of this exact image. Images which are no longer in use by any allocated machine and are older than the specified usage period will be deleted from the metal-api and the image store. --- ## metal-images # metal-stack.io | metal-images ![Go version](https://img.shields.io/github/go-mod/go-version/metal-stack/metal-images) [![Go Report Card](https://goreportcard.com/badge/github.com/metal-stack/metal-images)](https://goreportcard.com/report/github.com/metal-stack/metal-images) [![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/metal-stack/metal-images) [![Build](https://github.com/metal-stack/metal-images/actions/workflows/docker.yaml/badge.svg?branch=master)](https://github.com/metal-stack/metal-images/actions) [![Slack](https://img.shields.io/badge/slack-metal--stack-brightgreen.svg?logo=slack)](https://metal-stack.slack.com/) This project builds officially supported operating system images that can be used for bare metal server deployments with [metal-stack](https://metal-stack.io). Every OS image is built from a `Dockerfile`, exported to a `lz4` compressed tarball, and uploaded to [images.metal-stack.io](https://images.metal-stack.io/). More information about the image store is available in [IMAGE_STORE.md](./IMAGE_STORE.md). Information about our initial architectural decisions can be found in [ARCHITECTURE.md](./ARCHITECTURE.md). ## Supported Images Currently these images are supported: 1. Debian 13 1. Ubuntu 26.04 1. Firewall 3.0-ubuntu (based on Ubuntu 26.04) 1. Nvidia (based on Debian 13) ## Unsupported Images We also publish images that we need for special purposes but do not officially support. Use at your own risk. 1. Almalinux 10 ## Building Custom Images (Out-Of-Tree) It is fully possible to build custom operating system images and provision them through the metal-stack without directly contributing them to this repository. Please find information [in the official docs](https://metal-stack.io/docs/next/operating-systems#building-your-own-images) and in the [image builder contract](./ARCHITECTURE.md). ### GPU Support GPU support for workers are available by using the `debian-nvidia` images. Please check our official documentation at [metal-stack.io](https://metal-stack.io/docs/gpu-workers) how to get this running on Kubernetes. ## How new images become usable in a metal-stack partition Images are synchronized to partitions using a service called [metal-image-cache-sync](https://github.com/metal-stack/metal-image-cache-sync). This service mirrors the public operating system images to the management servers and transparently serves the metal-images within a partition. Released images are tagged with the release date and can be accessed using the following image URL pattern: `https://images.metal-stack.io/metal-os/20240913/debian/13/img.tar.lz4` Images built from the master branch are accessible at an image URL like this: `https://images.metal-stack.io/metal-os/stable/debian/13/img.tar.lz4` For other branches, the URL pattern is as follows: `https://images.metal-stack.io/metal-os/pull_requests/${CI_COMMIT_REF_SLUG}/debian/13/img.tar.lz4` These URLs can be used to define an image at the `metal-api`. ## Local development and integration testing Please also refer to our section [Build Your Own Images](https://metal-stack.io/docs/operating-systems#building-your-own-images) in our documentation to check for the contract an OS image is expected to fulfill. Before you can start developing changes for metal-images or even introduce new operating systems, you should install the following tools: - **docker** - **golang** - **kvm**: hypervisor used for integration tests - **lz4**: to compress tarballs - enable docker's [**containerd image store**](https://docs.docker.com/engine/storage/containerd/#enable-containerd-image-store-on-docker-engine) - **[cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor)**: virtual machine monitor running on top of KVM to spin up MicroVMs for integration tests You can build metal-images like that: ```bash # for debian images make debian # for ubuntu images make ubuntu # for firewall images make firewall # for nvidia images make nvidia # for almalinux images make almalinux ``` _IMPORTANT_ if you prefer the old docker build output instead of the fancy buildx rolling behind output, do the following: ```bash BUILDKIT_PROGRESS=plain make debian ``` For integration testing the images are started as [cloud-hypervisor](https://www.cloudhypervisor.org) VMs and basic properties like interfaces to other metal-stack components, kernel parameters, internet accessibility, DNS resolution etc. are checked with [goss](https://github.com/aelsabbahy/goss) in a GitHub action workflow. Integration tests are also executed if you build an image locally. ### Debugging Image Provisioning In some cases it may be necessary to manually figure out the commands for provisioning a machine image. To do this in a real server environment, it is possible to hook into the metal-hammer through the machine's serial console. You can interrupt the metal-hammer at any time by sending a keyboard interrupt. The metal-hammer takes a short break before booting into the operating system kernel, which is a good time to send the interrupt. To prevent the machine from rebooting, you should immediately issue the following command: ```bash while true; do echo "1" > /dev/watchdog && sleep 55; done & ``` If you want to enter the operating system through `chroot`, you need to remount some file systems that were mounted by the metal-hammer during provisioning: ```bash # the mount points also depend on the file system layout of the machine, so please only take this as an example: mount /dev/sda2 /rootfs mount -t vfat /dev/sda1 /rootfs/boot/efi mount -t proc /proc /rootfs/proc mount -t sysfs /sys /rootfs/sys mount -t efivarfs /sys/firmware/efi/efivars /rootfs/sys/firmware/efi/efivars mount -t devtmpfs /dev /rootfs/dev ``` Finally, you can then enter the provisioned OS image. ```bash chroot /rootfs # maybe you can mount further file systems here, which was not possible in the u-root environment of the metal-hammer vgchange -ay mount /dev/csi-lvm/varlib /var/lib/ ``` Keep in mind that you are still running on the metal-hammer kernel, which is different from the kernel that will be run in the operating system after provisioning. For further information on the metal-stack machine provisioning sequence, check out our documentation at [metal-stack.io](https://metal-stack.io/docs/architecture/#Machine-Provisioning-Sequence). The kernel used by the metal-hammer is built on our own inside the [kernel repository](https://github.com/metal-stack/kernel). --- ## mini-lab The mini-lab is a small, virtual setup to locally run the metal-stack. It deploys the metal control plane and a metal-stack partition with two simulated leaf switches. The lab can be used for trying out metal-stack, demonstration purposes or development. ![overview components](./assets/overview.drawio.svg) > Figure 1: Simplified illustration of the mini-lab. This project can also be used as a template for writing your own metal-stack deployments. - [Requirements](#requirements) - [Known Limitations](#known-limitations) - [Try it out](#try-it-out) - [Power management](#power-management) - [Development](#development) - [Release vector](#release-vector) - [Using local checkouts of dependencies](#using-local-checkouts-of-dependencies) - [Flavors](#flavors) - [Network topology](#network-topology) - [V2 Quickstart](#v2-quickstart) ## Requirements - Linux machine with hardware virtualization support - kvm as hypervisor for the VMs (you can check through the `kvm-ok` command) - [docker](https://www.docker.com/) >= 24.x.y (for using kind and our deployment base image) - [kind](https://github.com/kubernetes-sigs/kind/releases) == v0.23.0 (for hosting the metal control plane) - [containerlab](https://containerlab.dev/install/) >= v0.56.0 - the lab creates a docker network on your host machine with the address block `203.0.113.0/24`, designated as TEST-NET-3 for documentation and examples. - (recommended) haveged to have enough random entropy (only needed if the PXE process does not work) Here is some code that should help you to set up most of the requirements: ```bash # If UFW enabled. # Disable the firewall or allow traffic through Docker network IP range. sudo ufw status sudo ufw allow from 172.17.0.0/16 # Install required tools / kvm sudo apt install -y git curl qemu qemu-kvm haveged jq # Install Docker curl -fsSL https://get.docker.com | sh # if you want to be on the safe side, follow the original installation # instructions at https://docs.docker.com/engine/install/ubuntu/ # Ensure that your user is member of the group "docker" # you need to login again in order to make this change take effect sudo usermod -G docker -a ${USER} # Install containerlab bash -c "$(curl -sL https://get.containerlab.dev)" # Install kind (kubernetes in docker), for more details see https://kind.sigs.k8s.io/docs/user/quick-start/#installation sudo curl -Lo /usr/local/bin/kind "https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64" sudo chmod +x /usr/local/bin/kind ``` The following ports are used statically on your host machine: | Port | Bind Address | Description | |:----:|:------------ |:---------------------------------- | | 6443 | 0.0.0.0 | kube-apiserver of the kind cluster | | 4443 | 0.0.0.0 | HTTPS ingress | | 4150 | 0.0.0.0 | nsqd | | 8080 | 0.0.0.0 | HTTP ingress | ## Known Limitations - to keep the demo small there is no EVPN - machines have to be restarted manually with `make power-reset-` - login to the machines is possible with virsh console, login to the firewall is possible with SSH from your local machine ## Try it out ```bash git clone https://github.com/metal-stack/mini-lab.git cd mini-lab ``` Start the mini-lab with a kind cluster, a metal-api instance as well as two containers wrapping leaf switches and another container that hosts two user-allocatable machines: ```bash make MONITORING_ENABLED=false # containerlab will ask you for root permissions (https://github.com/srl-labs/containerlab/issues/669) # if you explicitly want to see the monitoring stack you can also run "make" without passing any params ``` Before the upcoming steps, you need to bind some environment variables using the following command. This ensures `metalctl` `kubectl` are able to communicate with your mini-lab. ```bash eval $(make dev-env) ``` After the deployment and waiting for a short amount of time, two machines in status `PXE booting` become visible through `metalctl machine ls`: ```bash docker compose run --rm metalctl machine ls ID LAST EVENT WHEN AGE HOSTNAME PROJECT SIZE IMAGE PARTITION 00000000-0000-0000-0000-000000000001   PXE Booting 3s 00000000-0000-0000-0000-000000000002 PXE Booting 5s ``` Wait until the machines reach the waiting state: ```bash docker compose run --rm metalctl machine ls ID LAST EVENT WHEN AGE HOSTNAME PROJECT SIZE IMAGE PARTITION 00000000-0000-0000-0000-000000000001   Waiting 8s v1-small-x86 mini-lab 00000000-0000-0000-0000-000000000002   Waiting 8s v1-small-x86 mini-lab ``` Create a firewall and a machine with: ```bash make firewall make machine ``` __Alternatively__, you may want to issue the `metalctl` commands on your own: ```bash docker compose run --rm metalctl network allocate \ --partition mini-lab \ --project 00000000-0000-0000-0000-000000000000 \ --name user-private-network # lookup the network ID and create a machine docker compose run --rm metalctl machine create \ --description test \ --name machine \ --hostname machine \ --project 00000000-0000-0000-0000-000000000000 \ --partition mini-lab \ --image ubuntu-24.4 \ --size v1-small-x86 \ --networks # create a firewall that is also connected to the virtual internet-mini-lab network docker compose run --rm metalctl firewall create \ --description fw \ --name fw \ --hostname fw \ --project 00000000-0000-0000-0000-000000000000 \ --partition mini-lab \ --image firewall-ubuntu-3.0 \ --size v1-small-x86 \ --networks internet-mini-lab, ``` See the installation process in action ```bash make console-machine01 # or console-machine02 ... Ubuntu 24.04 machine ttyS0 machine login: ``` Two machines are now installed and have status "Phoned Home" ```bash docker compose run --rm metalctl machine ls ID LAST EVENT WHEN AGE HOSTNAME PROJECT SIZE IMAGE PARTITION 00000000-0000-0000-0000-000000000001   Phoned Home 2s 21s machine 00000000-0000-0000-0000-000000000000 v1-small-x86 Ubuntu 24.04 mini-lab 00000000-0000-0000-0000-000000000002   Phoned Home 8s 18s fw 00000000-0000-0000-0000-000000000000 v1-small-x86 Firewall 3 Ubuntu mini-lab ``` Login with user name metal and the console password from ```bash docker compose run --rm metalctl machine consolepassword 00000000-0000-0000-0000-000000000001 ``` To remove the kind cluster, the switches and machines, run: ```bash make cleanup ``` ### Power management There are make targets to handle the power state of a machine: ``` make power-- ``` ## Development If you want to contribute to the _metal-stack_ project, you can use the mini-lab as a local development environment. It allows you to quickly test changes to the _metal-stack_ components without needing any external clusters or hardware. You can also use it to test changes to the Ansible roles and modules used by the _metal-stack_. ### Release vector You can configure the `mini-lab` to deploy specific custom or unreleased images of components by adjusting the variables in [inventories/group_vars/all/release_vector.yaml](inventories/group_vars/all/release_vector.yaml). For example, to quickly deploy `metal-api` with a custom branch tag, you can add or uncomment specific release tags in that file: ```yaml metal_api_image_tag: my-feat-branch # metal_core_image_tag: v1.2.3 ``` Further overrides can be looked up in `metal-roles` where the mapping is defined in [common/roles/defaults/defaults/main.yaml](https://github.com/metal-stack/metal-roles/blob/master/common/roles/defaults/defaults/main.yaml). ### Using local checkouts of dependencies By default, the `mini-lab` runs with pre-packaged Ansible roles and modules. If you want to use local checkouts of dependencies for development, you must start the `mini-lab` with `DEV=true`: ```bash DEV=true make up ``` When `DEV=true` is set, you can provide the following environment variables to map local directories into the containers. Each variable is independent — only the ones you set will be mounted (via a matching override file from `compose.dev/`): - `MINI_LAB_METAL_ROLES`: path to local `metal-roles` (includes `compose.dev/metal-roles.yaml`) - `MINI_LAB_ANSIBLE_COMMON`: path to local `ansible-common` (includes `compose.dev/ansible-common.yaml`) - `MINI_LAB_METAL_ANSIBLE_MODULES`: path to local `metal-ansible-modules` (includes `compose.dev/metal-ansible-modules.yaml`) - `MINI_LAB_HELM_CHARTS`: path to local `helm-charts` (includes `compose.dev/helm-charts.yaml`) Example: ```bash export MINI_LAB_METAL_ROLES=${HOME}/src/github.com/metal-stack/metal-roles export DEV=true make up ``` ## Flavors All available mini-lab flavors are listed below: - `sonic`: runs two Community SONiC switches - `dell_sonic`: runs two Enterprise SONiC switches with a [locally built vrnetlab image](https://github.com/srl-labs/vrnetlab/tree/master/dell/dell_sonic) - `capms_dell_sonic`: runs the `dell_sonic` flavor but with four instead of two machines (this is used for [cluster-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack) in order to have dedicated hosts for control plane / worker / firewall) - `kamaji`: runs a variation of the `sonic` flavor. The working example is available at the [cluster-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack)'s `capi-lab`. - `gardener`: runs the `sonic` flavor and installs the [Gardener](https://gardener.cloud) in the mini-lab In order to start specific flavor, you can define the flavor as follows: ```bash export MINI_LAB_FLAVOR=sonic make ``` ## Network topology An Nginx is running inside of the www container to allow automatic testing of outgoing connections. ![Network topology](./assets/network.svg) > Figure 2: mini-lab network topology illustration. ## V2 Quickstart Login with [cli](https://github.com/metal-stack/cli): ```bash $ metalctlv2 login --provider openid-connect ``` User: olli.owner@metal-stack.io Password: Olli.Owner123! User: gerrit.guest@metal-stack.io Password: Gerrit.Guest123! Zitadel Admin: User: admin@metal-stack.auth.172.17.0.1.nip.io Password: Password1! --- ## gardener-extension-audit Provides a Gardener extension for managing kube-apiserver audit logs for a shoot cluster. The extension spins up a fluent-bit-based audit sink in the seed's shoot namespace prior to starting the shoot's API server. Therefore, it is required to run this extension with the reconcile lifecycle policy `BeforeKubeAPIServer`. Also the deletion has to happen `BeforeKubeAPIServer` as otherwise the managed resources of this extension block the shoot deletion flow. This sink has the ability to buffer audit logs to a persistent volume and send them to the supported backends. ## Specifying An Audit Policy A custom audit policy can be natively configured by Gardener in the shoot spec's API server configuration under `.spec.kubernetes.kubeAPIServer.auditConfig.auditPolicy.configMapRef.name`. ## Supported Backends - Splunk - S3 - Log (just logs to the container, only for devel-purposes) - Cluster Forwarding (forwards audit logs into a pod in the shoot cluster) > [!IMPORTANT] > The Cluster Forwarding backend is mainly intended for showcasing and not for production purposes. It is known not to work with Gardener HA Control Planes and also there were issues reported when using it in combination with the Cilium CNI configured kubeproxyless with Native-Routing (audit entries do not arrive at the `audittailer` pod). ## Development This extension can be developed in the gardener-local devel environment. 1. Start up the local devel environment 1. The extension's docker image can be pushed into Kind using `make push-to-gardener-local` 1. Install the extension `kubectl apply -k example/` 1. Parametrize the `example/shoot.yaml` and apply with `kubectl -f example/shoot.yaml` --- ## gardener-extension-csi-driver-lvm Provides a Gardener extension for managing [csi-driver-lvm](https://github.com/metal-stack/csi-driver-lvm) for a shoot cluster. As a safety measurement, the extension checks for the old [csi-lvm](https://github.com/metal-stack/csi-lvm/tree/master) and stops reconciling if the old driver is still available. If not the extension will reconcile the new `csi-driver-lvm`. The following storage classes will be created by default: - `csi-driver-lvm-linear` for linear volumes - `csi-driver-lvm-mirror` for mirrored volumes for improved redundancy on multiple physical volumes - `csi-driver-lvm-striped` for striped volumes for improved performance on multiple physical volumes - `csi-lvm` for backwards compatibility with type linear. When encryption is enabled (see below), three additional LUKS-encrypted storage classes are created: - `csi-driver-lvm-linear-encrypted` - `csi-driver-lvm-mirror-encrypted` - `csi-driver-lvm-striped-encrypted` See [docs/migration.md](./migration.md) for further information about migrating from `csi-lvm` to `csi-driver-lvm`. ## LUKS Encryption LUKS block-device encryption can be enabled per shoot by adding an `encryption` field to the `providerConfig`. When set, the three encrypted storage classes above are created. Each class instructs the CSI node plugin to open the LUKS device using a key read from a Secret in the shoot cluster. ### Setup 1. Create the LUKS key Secret in the shoot cluster: ```sh kubectl -n create secret generic csi-lvm-encryption-secret \ --from-literal=passphrase="$(openssl rand -base64 32)" ``` > **Important:** This Secret is part of your cluster's recovery data. Back it up securely. Losing it makes encrypted volumes permanently inaccessible. 2. Reference the Secret in the shoot's `providerConfig`: ```yaml extensions: - type: csi-driver-lvm providerConfig: apiVersion: csi-driver-lvm.metal.extensions.gardener.cloud/v1alpha1 kind: CsiDriverLvmConfig devicePattern: /dev/nvme[0-9]n[0-9] hostWritePath: /etc/lvm encryption: secretRef: name: csi-lvm-encryption-secret namespace: ``` 3. Create a PVC using one of the encrypted storage classes: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-encrypted-pvc spec: accessModes: [ReadWriteOnce] storageClassName: csi-driver-lvm-linear-encrypted resources: requests: storage: 10Gi ``` ### Notes - Both `secretRef.name` and `secretRef.namespace` are required when `encryption` is set; the extension will refuse to reconcile with either field empty. - The Secret is user-owned and is not managed by the extension. It will not be deleted or overwritten during reconciliation. - `StorageClass.parameters` are immutable in Kubernetes. If you need to change the secret reference, delete the encrypted storage classes manually and allow the extension to recreate them. - Removing `encryption` from `providerConfig` deletes the encrypted storage classes but does **not** affect existing PVCs or the underlying LUKS devices. ## Development This extension can be developed in the gardener-local devel environment. Before make sure you have created loop-devices on your machine (identical to how you would develop the csi-driver-lvm locally, refer to the repository [docs](https://github.com/metal-stack/csi-driver-lvm?tab=readme-ov-file#development) for further information). ```sh for i in 100 101; do fallocate -l 1G loop${i}.img ; sudo losetup /dev/loop${i} loop${i}.img; done sudo losetup -a # use this for recreation or cleanup # for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done ``` 1. Start up the local devel environment 1. The extension's docker image can be pushed into Kind using `make push-to-gardener-local` 1. Install the extension `kubectl apply -k example/` 1. Parametrize the `example/shoot.yaml` and apply with `kubectl -f example/shoot.yaml` --- ## migration # Migration from csi-lvm to csi-driver-lvm The migration from the deprecated [`csi-lvm`](https://github.com/metal-stack/csi-lvm) to the new `csi-driver-lvm` must be performed in multiple steps. In here we assume you are currently using the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal). 1. Add the the `csi-driver-lvm` extension to the shoot spec. 2. Disable `csi-lvm` in `gardener-extension-provider-metal` by setting `featureGates.disableCsiLvm` to `true`. This allows the `csi-driver-lvm` to be installed. 3. Wait until the shoot has been successfully reconciled. 4. Make sure the `csi-driver-lvm` has been installed and that the `csi-lvm` storage class has been created. 5. Make sure to roll all nodes. Otherwise volumes will stop working after the next restart of a machine. ## Issues ### Why is the Drop-in replacement not possible? Deploying the new csi-driver-lvm with the same provisioner-name as the old one is not possible, as it causes errors when using k8s sidecar images for controllers. The provisioner name contains "/", which causes problems with node registrar directories (**metal-stack.io/csi-lvm**): ```sh I1015 08:09:23.292306 1 node_register.go:53] Starting Registration Server at: /registration/metal-stack.io/csi-lvm-reg.sock E1015 08:09:23.292482 1 node_register.go:56] failed to listen on socket: /registration/metal-stack.io/csi-lvm-reg.sock with error: listen unix /registration/metal-stack.io/csi-lvm-reg.sock: bind: no such file or directory ``` This problem requires a more complex migration. ## Manual migration The migration solution so far has been tested manually: 1. create old controller & provisioner 2. create pvcs & pod 3. write files to volumes 4. delete old controller & provisioner 5. install new controller & provisioner with helm 6. add additional storage class with name `csi-lvm` and type linear 1. mimics old storage class 2. default storage class (not supported yet -> see default storage class of `gardener-extension-provider-metal`) 7. create new pvcs 8. create new pod with old and new pvcs and test --- ## README # Gardener Extension for ONTAP Documentation > Technical background documentation for the extension, ONTAP, and their integration into metal-stack. These pages supplement the concise project overview and development instructions in the [repository README](./README.md). They can be read independently as topic-based reference material. --- ## Topics | Topic | Content | Start here | |-------|---------|------------| | **NetApp ONTAP** | ONTAP concepts, HA pairs, MetroCluster, SVMs, aggregates, LIFs, NVMe/TCP, Trident, and the PVC data path | [`./././ontap-storage.md`](ontap-storage.md) | | **Gardener Extension for ONTAP** | Activation, inputs, reconciliation, SVM and credential lifecycle, ManagedResources, shoot webhook, and cleanup boundaries | [`./././gardener-extension-ontap.md`](gardener-extension-ontap.md) | --- ## Recommended Reading Paths ### I want to understand the storage architecture 1. [`ONTAP Storage`](ontap-storage.md) — from the physical MetroCluster to a project-specific SVM 2. [`Gardener Extension for ONTAP`](gardener-extension-ontap.md) — how the architecture is automated for a shoot cluster ### I am investigating a provisioning failure 1. [`Gardener Extension for ONTAP: Reconciliation`](gardener-extension-ontap.md#reconciliation) 2. [`Gardener Extension for ONTAP: State Inspection`](gardener-extension-ontap.md#state-inspection) 3. [`ONTAP Storage: Management and Data Paths`](ontap-storage.md#management-and-data-paths) ### I am investigating a mount or I/O failure 1. [`ONTAP Storage: From a PVC to a Volume`](ontap-storage.md#from-a-pvc-to-a-volume) 2. [`ONTAP Storage: Management and Data Paths`](ontap-storage.md#management-and-data-paths) 3. [`Gardener Extension for ONTAP: Shoot Webhook`](gardener-extension-ontap.md#shoot-webhook-and-node-preparation) ### I am planning maintenance or cleanup 1. [`ONTAP Storage: Failure Behavior`](ontap-storage.md#failure-behavior) 2. [`Gardener Extension for ONTAP: Delete Lifecycle`](gardener-extension-ontap.md#delete-lifecycle-and-responsibility-boundary) --- ## System Overview ![Provisioning and data paths from cluster creation to an ONTAP volume](./assets/ontap-provisioning-and-data-paths.png) The components have distinct responsibilities: | Component | Responsibility | |-----------|----------------| | **Cloud API / Metal API** | Detects the ONTAP-enabled cluster configuration and reserves project-specific LIF addresses. | | **Gardener extension** | Establishes the desired ONTAP and shoot state. | | **ONTAP MetroCluster** | Stores and replicates data and provides SVMs and network endpoints. | | **Trident** | Translates Kubernetes storage operations into ONTAP operations. | | **Kubernetes worker** | Connects the remote block device over NVMe/TCP and mounts it into the workload. | --- ## Documentation Structure [`gardener-extension-ontap.md`](gardener-extension-ontap.md) documents the behavior of this repository and links to the relevant source paths. [`ontap-storage.md`](ontap-storage.md) describes the wider platform and storage context in which the extension operates. --- ## gardener-extension-ontap # Gardener Extension for NetApp ONTAP CSI Plugin This repository contains the Gardener extension controller for managing the NetApp ONTAP CSI Plugin. ## Table of Contents - [Documentation](#documentation) - [Prerequisites](#prerequisites) - [Development Workflow](#development-workflow) - [Test Environment Setup](#test-environment-setup) - [Known Issues](#known-issues) - [TODO List](#todo-list) ## Documentation Architecture, reconciliation, credential lifecycle, data paths, and troubleshooting are documented in the [documentation index](./README.md). ## Prerequisites - A local Gardener setup ## Development Workflow ### Setup Gardener Locally 1. Clone the Gardener Repository: ```bash git clone git@github.com:gardener/gardener.git ``` 2. Start a local Kubernetes cluster: ```bash make kind-up ``` 3. Deploy Gardener: ```bash make gardener-up ``` 4. Generate Helm Charts: ```bash make generate ``` ### Deploy the Extension 1. Apply the example configuration: ```bash kubectl apply -k example/ ``` 2. Apply the shoot cluster configuration: ```bash kubectl apply -f example/shoot.yaml ``` ### Update Code Changes When making changes to the code, build and deploy locally using: ```bash make push-to-gardener-local ``` ### Access the Shoot Cluster 1. Adjust your `/etc/hosts` file: ```bash cat < admin-kubeconf.yaml ``` 3. Trigger a reconciliation if needed: ```bash kubectl -n garden- annotate shoot gardener.cloud/operation=reconcile ``` kubectl patch torc trident -n --type=merge -p '{"spec":{"debug":true}}' ## Test Environment Setup To properly set up the test environment, we need to configure network translation between the external IPs (10.x) and internal KVM network (192.168.x). ### Simulator Host Machine Configuration #### Cluster Management Interface ```bash # Port Forward rules sudo iptables -t nat -A PREROUTING -i lan0 -p tcp --dport 443 -d 10.130.184.5 -j DNAT --to-destination 192.168.10.11 sudo iptables -t nat -A PREROUTING -i lan1 -p tcp --dport 443 -d 10.130.184.5 -j DNAT --to-destination 192.168.10.11 # NAT rules sudo iptables -t nat -A POSTROUTING -o lan0 -p tcp --dport 443 -d 192.168.10.11 -j SNAT --to-source 10.130.184.5 sudo iptables -t nat -A POSTROUTING -o lan1 -p tcp --dport 443 -d 192.168.10.11 -j SNAT --to-source 10.130.184.5 # Forward rules sudo iptables -I FORWARD 1 -i lan0 -o br-ontap-data -d 192.168.10.11 -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 2 -i lan1 -o br-ontap-data -d 192.168.10.11 -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 3 -i br-ontap-data -o lan0 -s 192.168.10.11 -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 4 -i br-ontap-data -o lan1 -s 192.168.10.11 -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT ``` #### SVM Management Interface ```bash # Port Forward rules sudo iptables -t nat -A PREROUTING -i lan0 -p tcp --dport 443 -d 10.130.184.6 -j DNAT --to-destination 192.168.10.29 sudo iptables -t nat -A PREROUTING -i lan1 -p tcp --dport 443 -d 10.130.184.6 -j DNAT --to-destination 192.168.10.29 # NAT rules sudo iptables -t nat -A POSTROUTING -o lan0 -p tcp --dport 443 -d 192.168.10.29 -j SNAT --to-source 10.130.184.6 sudo iptables -t nat -A POSTROUTING -o lan1 -p tcp --dport 443 -d 192.168.10.29 -j SNAT --to-source 10.130.184.6 # Forward rules sudo iptables -I FORWARD 1 -i lan0 -o br-ontap-data -d 192.168.10.29 -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 2 -i lan1 -o br-ontap-data -d 192.168.10.29 -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 3 -i br-ontap-data -o lan0 -s 192.168.10.29 -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 4 -i br-ontap-data -o lan1 -s 192.168.10.29 -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT ``` #### SVM Data Interface ```bash # Port Forward rules sudo iptables -t nat -A PREROUTING -i lan0 -p tcp --dport 4420 -d 10.130.184.7 -j DNAT --to-destination 192.168.10.30 sudo iptables -t nat -A PREROUTING -i lan1 -p tcp --dport 4420 -d 10.130.184.7 -j DNAT --to-destination 192.168.10.30 # NAT rules sudo iptables -t nat -A POSTROUTING -o lan0 -p tcp --dport 4420 -j SNAT --to-source 10.130.184.7 sudo iptables -t nat -A POSTROUTING -o lan1 -p tcp --dport 4420 -d 192.168.10.30 -j SNAT --to-source 10.130.184.7 # Forward rules sudo iptables -I FORWARD 1 -i lan0 -o br-ontap-data -d 192.168.10.30 -p tcp --dport 4420 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 2 -i lan1 -o br-ontap-data -d 192.168.10.30 -p tcp --dport 4420 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 3 -i br-ontap-data -o lan0 -s 192.168.10.30 -p tcp --sport 4420 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT sudo iptables -I FORWARD 4 -i br-ontap-data -o lan1 -s 192.168.10.30 -p tcp --sport 4420 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT ``` ### Worker Node Configuration Configure the worker node with these rules: ```bash iptables -t nat -A OUTPUT -d 192.168.10.30 -j DNAT --to-destination 10.130.184.7 iptables -t nat -A OUTPUT -d 192.168.10.30 -p tcp --dport 4420 -j DNAT --to-destination 10.130.184.7:4420 iptables -t nat -A POSTROUTING -d 10.130.184.7 -j MASQUERADE echo "10.130.184.7 192.168.10.30" >> /etc/hosts ``` ### Required Network Policies #### Clusterwidewide Network Policy in Shoot: ```bash apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: namespace: firewall name: allow-nvme-port spec: egress: - to: - cidr: 10.130.184.7/32 ports: - protocol: TCP port: 4420 ``` #### Clusterwidewide Network Policy in Seed: ```bash apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: namespace: firewall name: allow-mgmt-port spec: egress: - to: - cidr: 10.130.184.5/32 ports: - protocol: TCP port: 443 ``` ## Known Issues - In local environments, using the "Default" broadcast domain can result in "no route to host" errors. Using "Default-1" broadcast domain resolves this issue. - On test environments, the opposite is true - "Default" works but "Default-1" fails. - In the simulator, ports e0c and e0d are not functional. Use only e0a and e0b. - The Trident NVMe driver automatically uses network interfaces that are internally assigned. See [Trident issue #1007](https://github.com/NetApp/trident/issues/1007) for details. - When an SVM already exists, the secret in the shoot isn't created because it assumes the secret is already in the seed. ## TODO List - Fix SVM secret creation when SVM already exists - Implement proper SVM deletion logic - Add default gateway/routing configuration for SVMs - Fix hardcoded password in the `GenerateSecurePassword` function - Implement network route creation after SVM setup - Add monitoring and alerting for SVM health - Create proper cleanup and lifecycle management ## Creating ONTAP Encrypted Volumes To create an encrypted volume using NetApp Trident CSI, you need three components: 1. Secret with LUKS Passphrase ```yaml apiVersion: v1 kind: Secret metadata: name: storage-encryption-key namespace: stringData: luks-passphrase-name: A luks-passphrase: secretA ``` 2. StorageClass with Encryption Annotations The StorageClass must include CSI node stage secret annotations: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ontap-encrypted provisioner: csi.trident.netapp.io parameters: selector: "luks=true" csi.storage.k8s.io/node-stage-secret-name: storage-encryption-key csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace} backendType: "ontap-san" provisioningType: "thin" fsType: "ext4" allowVolumeExpansion: true ``` 3. PVC Using the Encrypted StorageClass ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ontap-encrypted-volume spec: storageClassName: ontap-encrypted accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` ### Key Requirements - Secret name in StorageClass must match actual secret name - Secret namespace in StorageClass must match where secret is created - PVC must reference the StorageClass with encryption annotations ### **Reconcile State Matrix** | SVM | Data LIFs | Mgmt LIF | ONTAP User | Seed Secret | Action Taken | Code Path | | --- | --------- | -------- | ---------- | ----------- | ---------------------------------------------- | -------------------------------------------------------------------- | | ❌ | ❌ | ❌ | ❌ | ❌ | Create complete SVM from scratch | `EnsureCompleteSVM()` → `CreateSVM()` | | ✅ | ❌ | ❌ | ❌ | ❌ | Validate SVM + Create all LIFs + Create User | `validateAndEnsureCompleteSVMState()` | | ✅ | ✅ | ❌ | ❌ | ❌ | Create missing data LIFs + Mgmt LIF + User | `validateAndEnsureDataLIFs()` + `validateAndEnsureManagementLIF()` | | ✅ | ✅ | ❌ | ❌ | ❌ | Create management LIF + Complete user creation | `validateAndEnsureManagementLIF()` → `createCompleteUserAndSecret()` | | ✅ | ✅ | ✅ | ❌ | ❌ | Create ONTAP user and K8s secret | `createCompleteUserAndSecret()` | | ✅ | ✅ | ✅ | ✅ | ❌ | Reset ONTAP password + Create K8s secret | `resetONTAPUserPassword()` → `buildAndCreateSecretInSeed()` | | ✅ | ✅ | ✅ | ❌ | ✅ | Create ONTAP user with existing K8s password | `createONTAPUserWithPassword()` | | ✅ | ✅ | ✅ | ✅ | ✅ | Validate password consistency | `validatePasswordConsistency()` | | ✅ | ✅ | ✅ | ✅ | ❌ | Fix corrupted secret (empty password) | `resetONTAPUserPassword()` → `updateSecretInSeed()` | ### **Legend** - ✅ = Resource exists and is correct - ❌ = Resource missing or not functional --- ## ontap-storage # ONTAP Storage > How NetApp ONTAP works as a MetroCluster and how it is provided to Kubernetes through Gardener and Trident. --- ## Context ONTAP is the platform's persistent, network-attached block storage system. Kubernetes workers do not hold the volume locally. They connect to a block device in the ONTAP system over NVMe/TCP. This is the essential difference from local LVM storage: | Property | ONTAP | Local LVM | |------------------------|--------------------------------------|--------------------------------------| | Storage location | External storage system | Local worker disk | | Access | NVMe/TCP over the network | Directly on the node | | Node loss | Data remains in the storage system | Local data can be lost with the node | | Site protection | Synchronous MetroCluster replication | No cross-site protection | | Kubernetes integration | NetApp Trident CSI | CSI Driver LVM | This page explains ONTAP and the end-to-end data path. The automation is documented separately in [`./././gardener-extension-ontap.md`](gardener-extension-ontap.md). --- ## ONTAP Concepts and Object Model ONTAP separates physical capacity, tenants, network endpoints, and the volumes used by Kubernetes. The following objects are important for understanding the platform: | Term | Meaning | |--------------------------------|---------------------------------------------------------------------------------------------| | **Node / controller** | Physical ONTAP instance that provides disks, network ports, and storage services | | **HA pair** | Two controllers at one site that can take over for each other if a node fails | | **ONTAP cluster** | Administrative grouping of the local HA pair | | **MetroCluster** | Cross-site grouping of two ONTAP clusters with synchronous replication | | **Aggregate** | ONTAP capacity pool built from the physical disks of a node | | **SVM** | Storage Virtual Machine; the logical and administrative tenant boundary for a Metal project | | **Volume** | Logical storage area within an SVM that draws capacity from its assigned aggregates | | **LIF** | Logical Interface; an IP-based SVM interface for management or data access | | **NVMe subsystem / namespace** | ONTAP objects through which a block device is made available to an NVMe host | | **Trident backend** | Connection between Trident in the shoot and the project-specific SVM | The simplified hierarchy is: ```text MetroCluster ├── ONTAP cluster, site A │ ├── Controller A │ ├── Controller B │ └── Aggregates └── ONTAP cluster, site B ├── Controller A ├── Controller B └── Aggregates Project SVM ├── Management LIF ├── Data LIF 1 ├── Data LIF 2 ├── Volumes └── NVMe configuration ``` An SVM is not separate hardware and does not own exclusive disks. It is a logical view of the cluster resources assigned to it. In this integration, an SVM belongs to a Metal project and can be used by multiple shoots in that project. --- ## Physical Topology A MetroCluster consists of two ONTAP clusters at separate sites: | Location | Topology | Role | |----------------------|------------------------------------------|-------------------------------| | Site A | Two ONTAP controllers as a local HA pair | One MetroCluster site | | Site B | Two ONTAP controllers as a local HA pair | Second MetroCluster site | | Third failure domain | Optional ONTAP Mediator | Quorum and switchover support | The controllers at a site are connected through a local cluster interconnect. The two sites communicate through redundant inter-site links. These links carry synchronous replication and cross-site MetroCluster metadata. Bandwidth, switch models, and specific network segments depend on the deployment. Several management and access networks exist separately: - controller and cluster management for administration and the ONTAP REST API, - intercluster communication between the storage systems, - BGP-routed SVM LIFs for Trident and the Kubernetes workers, - BMC access for out-of-band administration. This separation is important: Kubernetes data traffic does not use the global cluster management addresses, and MetroCluster replication does not pass through the shoot cluster. --- ## How ONTAP MetroCluster Works A MetroCluster spans two sites. At each site, two controllers form a local HA pair. MetroCluster replicates data synchronously between the sites. | Layer | Responsibility | |--------------------------------|--------------------------------------------------------------| | **Local HA pair** | Handles the failure of a single controller. | | **MetroCluster between sites** | Maintains a synchronous copy of the data at the second site. | | **ONTAP Mediator** | Supports the switchover decision if an entire site fails. | A project-specific Storage Virtual Machine (SVM) is active at one site. The corresponding MetroCluster sync destination with the `-mc` suffix exists at the other site. Applications do not write to both sides simultaneously: they access the active SVM while ONTAP performs synchronous replication to the other side. ### Local HA and Cross-Site Replication Local HA and MetroCluster address different failure classes: - If one controller fails, its partner in the local HA pair takes over. - If an entire site fails, the previously passive MetroCluster site can take over the storage functions. - Until a switchover occurs, exactly one side of the SVM remains active and the other side remains the sync destination. A successful synchronous write has reached both MetroCluster sites. For replicated writes, the architecture therefore targets an RPO of 0: an acknowledged write should not be missing after the loss of a site. This protects the stored blocks, but it does not replace recovery of the Kubernetes and network paths. ### ONTAP Mediator The ONTAP Mediator runs separately from the two storage sites. It observes both sides and supports the decision to perform an automatic switchover when an entire site fails. This helps prevent both sides from becoming active simultaneously. The Mediator does not transfer application data and is not in the I/O path. If the Mediator is unavailable, some failure scenarios can require a manual switchover. The active MetroCluster site depends on the current state. When creating a new project SVM, the Gardener extension selects the ONTAP cluster with the lower current total number of volumes. --- ## Failure Behavior Three separate layers must work together during failover: 1. ONTAP performs controller takeover or a MetroCluster switchover. 2. The management and data LIFs must be reachable through the network at the active site. 3. Trident and the Linux NVMe/TCP stack reconnect the volumes in the shoot cluster. MetroCluster protects the data and makes it available at the second site. Kubernetes, Trident, and the network then restore workload access. | Failure | Primary response | What matters next | |----------------------------|-----------------------------------------|-----------------------------------------------------------------| | Single ONTAP controller | Local partner takes over | LIF and NVMe/TCP connections must remain reachable | | Entire storage site | MetroCluster switchover | BGP routing, LIF reachability, and Trident reconnect | | Kubernetes worker | Workload starts on another worker | The new worker establishes an NVMe/TCP connection to the volume | | Management path disruption | Existing I/O can continue independently | New volumes, expansion, and snapshot operations can fail | | Data path disruption | Worker cannot reach the block device | Inspect the data LIF, TCP 4420, and NVMe session | A switchover therefore does not guarantee an interruption-free application. ONTAP, routing, CSI reconnection, Kubernetes scheduling, and the application itself all contribute to the actual recovery time. MetroCluster also does not switch automatically to the other site solely because latency has increased. Latency is an operational and diagnostic condition, not an automatic switchover trigger. --- ## SVMs as the Project Boundary The SVM is the central handoff point between the platform and ONTAP: - It contains the project-specific network endpoints. - It has its own NVMe configuration. - It has its own users and roles. - Trident manages volumes only within this SVM. - It can draw capacity from every aggregate assigned to it. The Gardener extension assigns all available aggregates at the selected ONTAP site to a new SVM. The aggregates remain cluster resources and are not reserved exclusively for the project. If another aggregate is added later, reconciliation can add it to the SVM assignment. ### Project-Specific, Not Shoot-Specific The Cloud API reserves LIF addresses for the Metal project. The extension also derives the SVM name from the project ID. The resulting model is: ```text Metal project | +--> one project SVM | +--> one management LIF | +--> two data LIFs | +--> Shoot A --> dedicated SVM user + dedicated Trident backend | +--> Shoot B --> dedicated SVM user + dedicated Trident backend ``` This model avoids creating an SVM for every shoot. Their lifecycles are not identical, however: a shoot can be deleted while another shoot still requires the project SVM. --- ## How We Use ONTAP ONTAP is not provisioned directly for each Kubernetes PVC. Instead, an automated chain connects the project configuration to the mounted block device: ```text Metal project | v 1 management LIF + 2 data LIFs | v Gardener extension in the seed | +--> project-specific SVM in ONTAP | +--> Trident + backend + StorageClasses in the shoot | v Application PVC | v ONTAP volume over NVMe/TCP ``` ### Project-Specific SVM The Cloud API reserves three BGP-routed `/32` addresses for an ONTAP-enabled project: - one management LIF for ONTAP operations, - two data LIFs for NVMe/TCP traffic. The Gardener extension reads the project ID and these IP addresses from the shoot configuration. If the SVM is missing, the extension creates it, enables NVMe, assigns the ONTAP aggregates, and creates the LIFs. If the SVM already exists, the extension inspects and completes its state. The SVM belongs to the Metal project and can be used by multiple ONTAP-enabled shoot clusters in that project. The ONTAP user and corresponding credential Secret are shoot-specific. This separation provides two security boundaries: - The project determines which SVM contains its volumes. - Each shoot receives only its own Trident credentials. The complete reconciliation and credential logic is described in [`Gardener Extension for ONTAP`](gardener-extension-ontap.md). ### Trident in the Shoot Cluster The extension deploys NetApp Trident and the required resources into the shoot as Gardener `ManagedResource` objects. These include: - the Trident operator, controller, and node plugin, - `TridentBackendConfig` for the project-specific SVM, - the shoot-specific credential Secret, - the `ontap-gold` and `ontap-encrypted` StorageClasses, - snapshot resources, - a network policy for the management and data LIFs when the `firewall` namespace exists. The backend uses the Trident `ontap-san` driver with `sanType: nvme`. A shoot webhook also prepares the Trident node DaemonSet so the `nvme-tcp` kernel module is available on every worker. ### Management and Data Paths Runtime communication is deliberately separated: | Source | Destination | Protocol | Purpose | |---------------------------------|--------------------------|---------------------|------------------------------------------| | Gardener extension in the seed | ONTAP cluster management | HTTPS / TCP 443 | Manage SVMs, LIFs, aggregates, and users | | Trident controller in the shoot | SVM management LIF | HTTPS / TCP 443 | Manage volumes and snapshots | | Kubernetes worker | SVM data LIFs | NVMe/TCP / TCP 4420 | Workload block I/O | The management LIF does not carry application data. Conversely, the data LIFs are not used for administrative ONTAP operations. This separation is also the most important starting point for troubleshooting. ### Why One Management LIF and Two Data LIFs? The management LIF provides a stable, project-specific API endpoint for Trident. Trident therefore does not need to use the global ONTAP cluster administrative addresses. The data LIFs provide the SVM's NVMe/TCP endpoints. Two LIFs permit multiple network paths and distribute the endpoints across the ONTAP nodes. They are not a substitute for MetroCluster: LIF redundancy protects the access path, while MetroCluster protects the storage site and data copy. In production, the three SVM LIFs are announced through BGP as `/32` VIPs. The Trident controller and workers can therefore reach the currently routed endpoints without knowing the internal ONTAP management networks at either site. ### Which Component Uses Which Address? - The Gardener extension connects to the ONTAP cluster management addresses because it manages privileged objects such as SVMs and users. - Trident connects to the project-specific management LIF because it manages only that SVM's backend. - The worker connects to the data LIFs because the NVMe/TCP I/O terminates there. If a PVC cannot be created, inspect the management path first. If the volume exists but cannot be mounted, focus on the data path and the node. ### From a PVC to a Volume 1. An application creates a PVC with `storageClassName: ontap-gold` or `ontap-encrypted`. 2. Kubernetes calls the Trident CSI controller. 3. Trident selects the appropriate storage pool and creates the volume through the SVM management LIF. 4. The Trident node plugin connects the worker to a data LIF over NVMe/TCP. 5. The block device is made available on the worker, formatted with `ext4`, and mounted into the pod. 6. With `ontap-encrypted`, the device is additionally opened with LUKS2 before it is mounted. Kubernetes ultimately sees a `PersistentVolume`, and the pod sees a regular mount. Trident and ONTAP encapsulate the SVM, ONTAP volume, NVMe configuration, and session details. A simple unencrypted PVC looks like this: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-data spec: storageClassName: ontap-gold accessModes: - ReadWriteOnce resources: requests: storage: 10Gi ``` | StorageClass | Use | |-------------------|--------------------------------------------------------------------------------------------| | `ontap-gold` | Thin-provisioned block storage with `ext4`; volume expansion is allowed. | | `ontap-encrypted` | Client-side LUKS2-encrypted block storage; the application provides the passphrase Secret. | > **Important:** With `ontap-encrypted`, ONTAP sees only encrypted blocks. If the LUKS Secret is lost, the data cannot be recovered. ### Encrypted Storage The passphrase is stored in a Secret named `storage-encryption-key` in the PVC namespace: ```yaml apiVersion: v1 kind: Secret metadata: name: storage-encryption-key namespace: application stringData: luks-passphrase-name: key-a luks-passphrase: ``` The StorageClass references this Secret for node-stage and node-expand operations. Encryption takes place on the Kubernetes worker: ```text Application | v ext4 file system | v LUKS2 device | v NVMe/TCP | v ONTAP sees encrypted blocks ``` Secret backup, access control, and tested passphrase rotation therefore belong to the application lifecycle rather than the ONTAP MetroCluster lifecycle. ### Volume Expansion and Snapshots Both StorageClasses allow volume expansion. During expansion, Trident first enlarges the backend volume. The block device, LUKS layer when applicable, and file system are then expanded on the worker. The extension also provides the `ontap-snapshot` `VolumeSnapshotClass`. Its `deletionPolicy` is `Delete`: deleting the Kubernetes snapshot object also deletes the corresponding backend snapshot. Snapshots and MetroCluster replication solve different problems. MetroCluster protects against storage site failure. A snapshot provides an earlier data state within the storage lifecycle. --- ## The Most Important Operational Boundary Deleting a shoot cluster does not automatically remove the project-specific SVM, its LIFs, or the ONTAP volumes. This is intentional because other shoots in the project can still use the SVM. Complete project cleanup must therefore happen separately and in a controlled order. A safe project cleanup must answer at least these questions: 1. Does an ONTAP-enabled shoot still exist in the project? 2. Are any volumes or snapshots still required? 3. Can the project SVM, including its NVMe configuration and LIFs, be removed? 4. Has the MetroCluster sync destination also been cleaned up? 5. Can the three project-specific IP addresses then be released? IP addresses must not be released while LIFs with those addresses still exist in ONTAP. Otherwise, orphaned ONTAP objects can block later reuse of an address. --- ## Summary - MetroCluster replicates data synchronously between two sites; local HA pairs additionally protect against controller failures. - Each Metal project has one ONTAP SVM with one management LIF and two data LIFs. - The Gardener extension manages the ONTAP-side and shoot-side resources. - Trident translates Kubernetes PVCs into ONTAP volumes. - Management uses HTTPS on port 443; block I/O uses NVMe/TCP on port 4420. --- > Back to the [`documentation index`](./README.md) · Implementation details: [`gardener-extension-ontap.md`](gardener-extension-ontap.md) --- ## gardener-vpn-gateway # Gardener VPN Gateway This is a small piece of software that is intended to run in the shoot controlplane of a [Gardener](https://github.com/gardener/gardener) shoot cluster. It watches for a service in the shoot for incoming connections and listens for incoming connections, which it will forward through the VPN between seed and shoot to the service in the cluster. The point of this is that the VPN gateway will take care of talking to the VPN proxy, while for the client in the controlplane it looks like a transparent connection. ## Current scope for the implementation - Only one service per running VPN gateway instance ### Use with proxy (mTLS proxy with http-connect) ## Testing locally TODO this is not updated for the new vpn gateway role; will need to evaluate if local testing still makes sense and if so how to do this. --- ## os-metal-extension [![GitHub License](https://img.shields.io/github/license/metal-stack/os-metal-extension)](https://github.com/metal-stack/os-metal-extension/blob/master/LICENCE) [![Build](https://github.com/metal-stack/os-metal-extension/actions/workflows/build.yaml/badge.svg)](https://github.com/metal-stack/os-metal-extension/actions/workflows/build.yaml) This controller operates on the [`OperatingSystemConfig`](https://github.com/gardener/gardener/blob/master/docs/proposals/01-extensibility.md#cloud-config-user-data-for-bootstrapping-machines) resource in the `extensions.gardener.cloud/v1alpha1` API group. It manages those objects that are requesting: - [Ubuntu](https://www.ubuntu.com/) configuration (`.spec.type=ubuntu`) - [Debian](https://www.debian.org/) configuration (`.spec.type=debian`) In comparison to the official OSC extensions from Gardener, this extension transforms the `OperatingSystemConfig` resources into [Ignition](https://www.flatcar.org/docs/latest/provisioning/ignition/) userdata. This userdata can be applied during machine provisioning as done by the metal-stack project. This extension was made for working with operating system images built in the [metal-images](https://github.com/metal-stack/metal-images) repository. ## Example An example `ControllerRegistration` resource that can be used to register this controller to Gardener can be found [here](example/controller-registration.yaml). ## Development Development currently needs to happen against a real environment because there are many dependencies to external APIs for reconciliation. It is planned to allow development in the [mini-lab](https://github.com/metal-stack/mini-lab) soon. ## Feedback and Support Feedback and contributions are always welcome! Please report bugs or suggestions as [GitHub issues](https://github.com/metal-stack/os-metal-extension/issues) or reach out to our [community](https://metal-stack.io/community). --- ## DEVELOPMENT # Development ## Getting Started Locally This project comes with a preconfigured version of the [mini-lab](https://github.com/metal-stack/mini-lab) in [capi-lab](./capi-lab) which runs a local metal-stack instance and all prerequisites required by this provider. ```bash make -C capi-lab # allows access using metalctl and kubectl eval $(make -C capi-lab --silent dev-env) ``` Next install our CAPMS provider into the cluster. ```bash # repeat this whenever you make changes make push-to-capi-lab ``` Before creating a cluster some manual steps are required beforehand: you need to allocate a node network and a firewall. ```bash make -C capi-lab node-network firewall control-plane-ip ``` A basic cluster configuration that relies on `config/clusterctl-templates/cluster-template.yaml` and uses the aforementioned node network can be generated and applied to the management cluster using a make target. ```bash make -C capi-lab apply-sample-cluster ``` Once the control plane node has phoned home, run: ```bash make -C capi-lab mtu-fix ``` When the control plane node was provisioned, you can obtain the kubeconfig like: ```bash kubectl get secret metal-test-kubeconfig -o jsonpath='{.data.value}' | base64 -d > capi-lab/.capms-cluster-kubeconfig.yaml # alternatively: clusterctl get kubeconfig metal-test > capi-lab/.capms-cluster-kubeconfig.yaml ``` It is now expected to deploy a CNI to the cluster: ```bash kubectl --kubeconfig=capi-lab/.capms-cluster-kubeconfig.yaml create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml cat < [!note] > Actually, Calico should be configured using BGP (no overlay), eBPF and DSR. An example will be proposed in this repository at a later point in time. The node's provider ID is provided by the [metal-ccm](https://github.com/metal-stack/metal-ccm), which needs to be deployed into the cluster: ```bash make -C capi-lab deploy-metal-ccm ``` If you want to provide service's of type load balancer through MetalLB by the metal-ccm, you need to deploy MetalLB: ```bash kubectl --kubeconfig capi-lab/.capms-cluster-kubeconfig.yaml apply --kustomize capi-lab/metallb ``` That's it! ### To Deploy on the cluster **Build and push your image to the location specified by `IMG`:** ```sh make docker-build docker-push IMG=/cluster-api-provider-metal-stack:tag ``` **NOTE:** This image ought to be published in the personal registry you specified. And it is required to have access to pull the image from the working environment. Make sure you have the proper permission to the registry if the above commands don’t work. **Install the CRDs into the cluster:** ```sh make install ``` **Deploy the Manager to the cluster with the image specified by `IMG`:** ```sh make deploy IMG=/cluster-api-provider-metal-stack:tag ``` > **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin privileges or be logged in as admin. **Create instances of your solution** You can apply the sample cluster configuration: ```sh make -C capi-lab apply-sample-cluster ``` ### To Uninstall **Delete the instances (CRs) from the cluster:** ```sh make -C capi-lab delete-sample-cluster ``` **Delete the APIs(CRDs) from the cluster:** ```sh make uninstall ``` **UnDeploy the controller from the cluster:** ```sh make undeploy ``` ## Project Distribution Following are the steps to build the installer and distribute this project to users. 1. Build the installer for the image built and published in the registry: ```sh make build-installer IMG=/cluster-api-provider-metal-stack:tag ``` NOTE: The makefile target mentioned above generates an 'install.yaml' file in the dist directory. This file contains all the resources built with Kustomize, which are necessary to install this project without its dependencies. 2. Using the installer Users can just run kubectl apply -f to install the project, i.e.: ```sh kubectl apply -f https://raw.githubusercontent.com//cluster-api-provider-metal-stack//dist/install.yaml ``` ## Quick opinionated Cluster Bootstrap and move This is a short and opinionated fast track to create and move a cluster using our provider. In contrast to a guide and the README, we do not explain all commands and try to be concise. Configure your clusterctl: ```yaml # ~/.config/cluster-api/clusterctl.yaml providers: - name: "metal-stack" url: "https://github.com/metal-stack/cluster-api-provider-metal-stack/releases/latest/download/infrastructure-components.yaml" # or for PRs # url: "${HOME}/path/to/infrastructure-metal-stack/v0.4.0/infrastructure-components.yaml" # generate with: # IMG_TAG=branch-name RELEASE_DIR=${HOME}/path/to/infrastructure-metal-stack/v0.4.0 make release-manifests type: InfrastructureProvider ``` Set environment variables. Don't forget to update them along the way. ```bash export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true export METAL_API_HMAC= export METAL_API_HMAC_AUTH_TYPE= export METAL_API_URL= export METAL_PARTITION= export METAL_PROJECT_ID= export METAL_NODE_NETWORK_ID= export CONTROL_PLANE_IP= export FIREWALL_MACHINE_IMAGE= export FIREWALL_MACHINE_SIZE= export CONTROL_PLANE_MACHINE_IMAGE= export CONTROL_PLANE_MACHINE_SIZE= export WORKER_MACHINE_IMAGE= export WORKER_MACHINE_SIZE= export CLUSTER_NAME= export NAMESPACE=default export KUBERNETES_VERSION=v1.31.6 export CONTROL_PLANE_MACHINE_COUNT=1 export WORKER_MACHINE_COUNT=1 # Additional envs export repo_path=$HOME/path/to/cluster-api-provider-metal-stack export project_name= export tenant_name= export firewall_id= ``` Create firewall if needed: ```bash metalctl project create --name $project_name --tenant $tenant_name --description "Cluster API test project" metalctl network allocate --description "Node network for $CLUSTER_NAME" --name $CLUSTER_NAME --project $METAL_PROJECT_ID --partition $METAL_PARTITION metalctl network ip create --network internet --project $METAL_PROJECT_ID --name "$CLUSTER_NAME-vip" --type static -o template --template "{{ .ipaddress }}" metalctl firewall create --description "Firewall for $CLUSTER_NAME cluster" --name firewall-$CLUSTER_NAME --hostname firewall-$CLUSTER_NAME --project $METAL_PROJECT_ID --partition $METAL_PARTITION --image $FIREWALL_MACHINE_IMAGE --size $FIREWALL_MACHINE_SIZE --firewall-rules-file $repo_path/config/target-cluster/firewall-rules.yaml --networks internet,$METAL_NODE_NETWORK_ID ``` ```bash kind create cluster --name bootstrap kind export kubeconfig --name bootstrap --kubeconfig kind-bootstrap.kubeconfig clusterctl init --infrastructure metal-stack --kubeconfig kind-bootstrap.kubeconfig clusterctl generate cluster $CLUSTER_NAME --infrastructure metal-stack > cluster-$CLUSTER_NAME.yaml kubectl apply -n $NAMESPACE -f cluster-$CLUSTER_NAME.yaml # once the control plane node is in phoned home metalctl machine consolepassword $firewall_id metalctl machine console --ipmi $firewall_id # sudo systemctl restart frr # ~. kubectl --kubeconfig kind-bootstrap.kubeconfig -n $NAMESPACE get metalstackmachines.infrastructure.cluster.x-k8s.io export control_plane_machine_id= metalctl machine console --ipmi $control_plane_machine_id # ip r # sudo systemctl restart kubeadm # crictl ps # ~. clusterctl get kubeconfig > capms-cluster.kubeconfig # metal-ccm cat $repo_path/config/target-cluster/metal-ccm.yaml | envsubst | kubectl --kubeconfig capms-cluster.kubeconfig apply -f - # cni kubectl --kubeconfig=capms-cluster.kubeconfig create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml cat < [!WARNING] > As of now the CAPMS is not yet feature complete and there might be breaking changes in future releases. > In case you search for a feature stable alternative consider [Gardener on metal-stack](https://docs.metal-stack.io/stable/installation/deployment/#Gardener-with-metal-stack) instead. > For developing this project head to our [DEVELOPMENT.md](./DEVELOPMENT.md). Currently, we provide the following custom resources: - [`MetalStackCluster`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/./api/v1alpha1/metalstackcluster_types.go) can be used as [infrastructure cluster](https://cluster-api.sigs.k8s.io/developer/providers/contracts/infra-cluster) and ensures that there is a control plane IP for the cluster. - [`MetalStackMachine`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/./api/v1alpha1/metalstackmachine_types.go) bridges between [infrastructure machines](https://cluster-api.sigs.k8s.io/developer/providers/contracts/infra-machine) and metal-stack machines. - [`MetalStackMachineTemplate`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/./api/v1alpha1/metalstackmachinetemplate_types.go) can be used to define reusable machine specifications for `MetalStackMachine` resources. - [`MetalStackFirewallDeployment`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/./api/v1alpha1/metalstackfirewalldeployment_types.go) can be used to define firewall deployments for a cluster. - [`MetalStackFirewallTemplate`](https://github.com/metal-stack/cluster-api-provider-metal-stack/blob/main/./api/v1alpha1/metalstackfirewalltemplate_types.go) defines the configuration of deployed firewalls. We plan to cover more resources in the future: - Complete Firewall Deployments using the [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager) - Improved configuration suggestion of CNIs > [!note] > Currently our infrastructure provider is only tested against the [Cluster API bootstrap provider Kubeadm (CABPK)](https://cluster-api.sigs.k8s.io/tasks/bootstrap/kubeadm-bootstrap/index.html?highlight=kubeadm#cluster-api-bootstrap-provider-kubeadm). > While other providers might work, there is no guarantee nor the goal to reach compatibility. ## Getting started **Prerequisites:** - Running metal-stack installation. See our [installation](https://docs.metal-stack.io/stable/installation/deployment/) section on how to get started with metal-stack. - Operating system images available to metal-stack. See [metal-stack/metal-images](https://github.com/metal-stack/metal-images) for pre-built ones. - Management cluster (with network access to the metal-stack infrastructure). - CLI metalctl installed for communicating with the metal-api. Installation instructions can be found in the corresponding [repository](https://github.com/metal-stack/metalctl). - CLI clusterctl First, add the metal-stack infrastructure provider to your `clusterctl.yaml`: ```yaml # ~/.config/cluster-api/clusterctl.yaml providers: - name: "metal-stack" url: "https://github.com/metal-stack/cluster-api-provider-metal-stack/releases/latest/download/infrastructure-components.yaml" type: InfrastructureProvider ``` Now, you are able to install the CAPMS into your management cluster: ```bash # export the following environment variables export METAL_API_URL= export METAL_API_HMAC= export METAL_API_HMAC_AUTH_TYPE= export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true # initialize the management cluster clusterctl init --infrastructure metal-stack ``` > [!CAUTION] > **Manual steps needed:** > Due to the early development stage, manual actions are needed for the cluster to operate. Some metal-stack resources need to be created manually. Allocate a VIP for the control plane. ```bash export CLUSTER_NAME= export METAL_PARTITION= export METAL_PROJECT_ID= export CONTROL_PLANE_IP=$(metalctl network ip create --network internet --project $METAL_PROJECT_ID --name "$CLUSTER_NAME-vip" --type static -o template --template "{{ .ipaddress }}") ``` For your first cluster, it is advised to start with our generated template. Ensure that the namespaced cluster name is unique within the metal stack project. ```bash # display required environment variables clusterctl generate cluster $CLUSTER_NAME --infrastructure metal-stack --list-variables --flavor calico # set additional environment variables export CONTROL_PLANE_MACHINE_IMAGE= export CONTROL_PLANE_MACHINE_SIZE= export WORKER_MACHINE_IMAGE= export WORKER_MACHINE_SIZE= export FIREWALL_MACHINE_IMAGE= export FIREWALL_MACHINE_SIZE= # generate manifest clusterctl generate cluster $CLUSTER_NAME --kubernetes-version v1.32.9 --infrastructure metal-stack --flavor calico ``` Apply the generated manifest from the `clusterctl` output. ```bash kubectl apply -f ``` That's it! ## Frequently Asked Questions ### I need to know the Control Plane IP address in advance. Can I provide a static IP address in advance? Yes, simply create a static IP address and set it to `metalstackcluster/$CLUSTER_NAME.spec.controlPlaneIP`. ```bash metalctl network ip create --name $CLUSTER_NAME-vip --project $METAL_PROJECT_ID --type static ``` ### I'd like to have a specific Pod CIDR. How can I achieve this? When generating your cluster, set `POD_CIDR` to your desired value. ```bash export POD_CIDR=["10.240.0.0/12"] ``` ## I'd like to update the firewall and / or its rules. How can I achieve this? Unfortunately there is no automated way to update firewall deployments or rules yet. You have to manually edit the `MetalStackFirewallDeployment` resource to force it to create a new firewall. 1. Save the firewall machine ID stored in `MetalStackFirewallDeployment.spec.managedResourceRef.name`. 2. Update the `MetalStackFirewallTemplate` as desired. 3. Remove `MetalStackFirewallDeployment.spec.managedResourceRef`. 4. Wait for CAPMS to create the new firewall. 5. Wait for the new firewall to be in `Phoned Home` state using `metalctl machine list --id `. 6. Delete the old firewall machine using `metalctl machine delete ` as soon as possible. This leads to a minimized downtime of the cluster as the firewall is not available during the transition. ## Flavors You might choose from different cluster template [flavors](https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster.html?highlight=flavor#flavors) to generate manifests with clusterctl. Here is a table describing the available flavors: | Name | Description | K8s Compatibility | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | | This is the default flavor without providing the `--flavor` flag. This expects the user to deploy a CNI and a CCM. | >= v1.33 | | calico | Installs [calico](https://docs.tigera.io/calico/latest/about/) CNI along with [metal-ccm](https://github.com/metal-stack/metal-ccm). Depends on `ClusterResourceSet` and the [Add-on Provider for Helm](https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/tree/main). | >= v1.33 | | pre-v1.33 | The same as the default flavor but working for K8s versions < v1.33. | < v1.33 | --- ## droptailer # Droptailer Droptailer gathers firewall drop or accept logs from different machines, enriches them with data from kubernetes api resources and makes them accessible by kubernetes means. ## Client - reads the systemd journal for kernel log messages about packet drops or accepts - pushes them with gRPC to the `droptail` server environment variables: - `DROPTAILER_SERVER_ADDRESS`: endpoint for the server - `DROPTAILER_PREFIXES_OF_DROPS`: prefixes that identify drop messages in the journal - `DROPTAILER_PREFIXES_OF_ACCEPTS`: prefixes that identify drop messages in the journal ## Generating certificates ```bash # Install cfssl tool curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x ~/bin/{cfssl,cfssljson} # Create certificates for client and server echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca - echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json export ADDRESS=droptailer export NAME=droptailer-server echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' \ | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - \ | cfssljson -bare $NAME export ADDRESS= export NAME=droptailer-client echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' \ | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - \ | cfssljson -bare $NAME ``` ## Testing droptailer ```bash # install kind 0.6.0 or higher ! KIND_VERSION=v0.7.0 wget https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64 mv kind-linux-amd64 ~/bin/kind chmod +x ~/bin/kind # Create a k8s cluster kind create cluster # Deploy droptailer-server kubectl apply -f ./test/manifests/droptailer.yaml # Expose droptailer-server port to host podName=$(kubectl get pods -n firewall -o=jsonpath='{.items[0].metadata.name}') echo $podName kubectl port-forward -n firewall --address 0.0.0.0 pod/$podName 50051:50051 & # Run droptailer-client docker run -it \ --privileged \ --add-host droptailer:172.17.0.1 \ --env DROPTAILER_SERVER_ADDRESS=droptailer:50051 \ --volume $(pwd)/test/certs:/etc/droptailer-client:ro \ --volume /run/systemd/private:/run/systemd/private \ --volume /var/log/journal:/var/log/journal \ --volume /run/log/journal:/run/log/journal \ --volume /etc/machine-id:/etc/machine-id \ metalstack/droptailer-client # Watch for drops stern -n firewall drop # Generate sample messages for the systemd journal that is caught by the droptailer-client sudo logger -t kernel "nftables-metal-dropped: IN=vrf09 OUT= MAC=12:99:fd:3b:ce:f8:1a:ae:e9:a7:95:50:08:00 SRC=1.2.3.4 DST=4.3.2.1 LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=46474 PROTO=TCP SPT=59265 DPT=445 WINDOW=1024 RES=0x00 SYN URGP=0" sudo logger -t kernel "nftables-metal-accepted: IN=vrf10 OUT=vrf11 MAC=12:99:fd:3b:ce:f8:1a:ae:e9:a7:95:50:08:00 SRC=5.6.7.8 DST=8.7.6.5 LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=46474 PROTO=TCP SPT=59265 DPT=445 WINDOW=1024 RES=0x00 SYN URGP=0 ItIs=OnlyText" ``` --- ## firewall-controller This controller is installed on a bare-metal firewall in front of several kubernetes worker nodes and responsible to reconcile a `ClusterwideNetworkPolicy` to nftables rules to control access to and from the kubernetes cluster. It allows also to control the traffic rate going through, to limit network resources for restricted usage scenarios. Nftable and node metrics are exposed with the `nftables-exporter` and `node-exporter`, the ips are visible as service and endpoint from the kubernetes cluster. Additionally, an IDS is managed on the firewall to detect known network anomalies. [suricata](https://suricata-ids.org) is used for this purpose. Right now, only basic statistics about the amount of scanned packets is reported. In a future release, access to all alarms will be provided. This controller is typically setup through the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager) (FCM), which manages the lifecycle of metal-stack firewalls inside our [Gardener integration](https://docs.metal-stack.io/stable/overview/kubernetes/). ## Architecture The firewall-controller is acting on 3 CRDs typically running in your cluster and a provider-managed cluster (in Gardener terms "shoot" and "seed").: | CRD | API | Resides In | Purpose | | -------------------------------- | ---------------------------- | ---------- | ------------------------------------------------------------------- | | `ClusterwideNetworkPolicy` | `metal-stack.io/v1` | Shoot | Controls firewall rules and can be provided by the user | | `Firewall` defined by FCM | `firewall.metal-stack.io/v2` | Seed | Defines the firewall including rate limits, controller version, ... | | `FirewallMonitor` defined by FCM | `firewall.metal-stack.io/v2` | Shoot | Used as an overview for the user on the status of the firewall | Note that `clusterwidenetworkpolicy` resources are namespaced and must reside in the `firewall` namespace, otherwise this controller does not reconcile them. ![Architecture](./assets/architecture.drawio.svg) Example `ClusterwideNetworkPolicy`: ```yaml apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: namespace: firewall name: clusterwidenetworkpolicy-sample spec: egress: - to: - cidr: 1.1.0.0/24 except: - 1.1.1.0/16 - cidr: 8.8.8.8/32 ports: - protocol: UDP port: 53 - protocol: TCP port: 53 - protocol: TCP port: 8080 # Optional, if specified this is the way to specify a port range from port to endPort endPort: 8088 ``` ## Automatically Generated Ingress Rules For every `Service` of type `LoadBalancer` in the cluster, the corresponding ingress rules will be automatically generated. If `loadBalancerSourceRanges` is not specified, incomig traffic to this service will be allowed for any source ip addresses. ### DNS Policies Configuration The `ClusterwideNetworkPolicy` resource allows you to define DNS based egress policies as well. They allow you to filter egress traffic based either on DNS name or by matching names to the provided pattern. To filter by specific domain name you need to provide `matchName` field: ```yaml apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: namespace: firewall name: clusterwidenetworkpolicy-fqdn spec: egress: - toFQDNs: - matchName: example.com ports: - protocol: UDP port: 80 - protocol: TCP port: 80 ``` If you want to filter FQDNs that are matching certain pattern, you can use `matchPattern` field, which supports `*` wildcard. Following example allows traffic to port 80 of all resources in the `.example` top-level domain: ```yaml apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: namespace: firewall name: clusterwidenetworkpolicy-fqdn-pattern spec: egress: - toFQDNs: - matchPattern: *.example ports: - protocol: UDP port: 80 - protocol: TCP port: 80 ``` By default, DNS info is collected from Google DNS (with address 8.8.8.8:53). The preferred DNS server can be changed through the `Firewall` resource of the FCM, which is governed by the provider. ## Status Once the firewall-controller is running, it will report several statistics to the `FirewallMonitor` CRD Status. This can be inspected by running: ```bash kubectl get -n firewall fwmon NAME MACHINE ID IMAGE SIZE LAST EVENT AGE shoot--prod--seed-firewall-089f9 f4f8b200-deef-11e9-8000-3cecef22f910 firewall-ubuntu-2.0.20221025 n1-medium-x86 Phoned Home 18m ``` When showing the resource with `-o yaml`, it contains detailed information on traffic counting, package drops and IDS: ```yaml Status: Last Run: 2020-06-17T13:18:58Z Stats: # Network traffic in bytes separated into external and internal in/out/total Devices: External: In: 91696 Out: 34600 Total: 0 Internal: In: 0 Out: 0 Total: 2678671 # IDS Statistics by interface Idsstats: vrf104009: Drop: 1992 Invalidchecksums: 0 Packets: 4997276 # nftable rule statistics by rule name Rules: Accept: BGP unnumbered: Counter: Bytes: 0 Packets: 0 SSH incoming connections: Counter: Bytes: 936 Packets: 16 accept established connections: Counter: Bytes: 21211168 Packets: 39785 accept icmp: Counter: Bytes: 0 Packets: 0 accept traffic for k8s service kube-system/vpn-shoot: Counter: Bytes: 360 Packets: 6 Drop: drop invalid packets: Counter: Bytes: 52 Packets: 1 drop invalid packets from forwarding to prevent malicious activity: Counter: Bytes: 0 Packets: 0 drop invalid packets to prevent malicious activity: Counter: Bytes: 0 Packets: 0 drop packets with invalid ct state: Counter: Bytes: 0 Packets: 0 drop ping floods: Counter: Bytes: 0 Packets: 0 Other: block bgp forward to machines: Counter: Bytes: 0 Packets: 0 count and log dropped packets: Counter: Bytes: 2528 Packets: 51 snat (networkid: internet): Counter: Bytes: 36960 Packets: 486 ``` ## Prometheus Integration There are two exporters running on the firewall to report essential metrics from this machine: - node-exporter for machine specific metrics like cpu, ram and disk usage, see [node-exporter](https://github.com/prometheus/node_exporter) for details. - nftables-exporter for nftables metrics, see [nftables-exporter](https://github.com/Sheridan/nftables_exporter) Both exporters are exposed as services: ```bash kubectl get svc -n firewall NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nftables-exporter ClusterIP None 9630/TCP 13h node-exporter ClusterIP None 9100/TCP 13h ``` These services are in front of virtual endpoints: ```bash kubectl get ep -n firewall NAME ENDPOINTS AGE nftables-exporter 10.3.164.1:9630 13h node-exporter 10.3.164.1:9100 13h ``` You can scrape these services in you prometheus installation to get the metrics. To check you can run: ```bash curl nftables-exporter.firewall.svc.cluster.local:9630/metrics curl node-exporter.firewall.svc.cluster.local:9100/metrics ``` ## Firewall Logs It is also possible to tail for the dropped packets with the following command (install stern from [stern](https://github.com/stern/stern)): ```bash stern -n firewall drop ``` The output will look like: ```json droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:27 +0000 UTC {"ACTION":"Drop","DPT":"4000","DST":"1.2.3.4","ID":"54321","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"38464","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"236","URGP":"0","WINDOW":"65535","timestamp":"2020-06-17 13:23:27 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"ACTION":"Drop","DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:30 +0000 UTC {"ACTION":"Accept","DPT":"650","DST":"1.2.3.4","ID":"12399","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"241","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:30 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"ACTION":"Accept","DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:10 +0000 UTC {"ACTION":"Accept","DPT":"63351","DST":"1.2.3.4","ID":"11855","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"54589","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"245","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:10 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"ACTION":"Accept","DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"} ``` You can forward the droptailer logs to any log aggregation infrastructure you have in place. If enabled in the firewall spec, accepted connections can be logged in addition to dropped connections. --- ## firewall-controller-manager ## Overview The firewall-controller-manager (FCM) is a collection of controllers which are responsible for managing the lifecycle of firewalls in a [Gardener](https://gardener.cloud/) shoot cluster for the metal-stack provider. The FCM is typically deployed into the shoot namespace of a seed cluster. This is done by the [gardener-extension-provider-metal](https://github.com/metal-stack/gardener-extension-provider-metal/). The design of the FCM is roughly inspired by Gardener's [machine-controller-manager](https://github.com/gardener/machine-controller-manager) and Kubernetes' built-in resources `Deployment`, `ReplicaSet` and `Pod`. ## Architecture The following table is a summary over the [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) introduced by the FCM: | Custom Resource Object | Description | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FirewallDeployment` | A `FirewallDeployment` contains the spec template of a `Firewall` resource similar to a `Deployment` and implements update strategies like rolling update. | | `FirewallSet` | A `FirewallSet` is similar to `ReplicaSet`. It is typically owned by a `FirewallDeployment` and attempts to run the defined replica amount of the `Firewall`(s) | | `Firewall` | A `Firewall` is similar to a `Pod` and has a 1:1 relationship to a firewall in the metal-stack api. | | `FirewallMonitor` | Deployed into the cluster of the user (shoot cluster), which is useful for monitoring the firewall or user-triggered actions on the firewall. | ### `FirewallDeploymentController` The `FirewallDeployment` controller manages the lifecycle of `FirewallSet`s. It syncs the `Firewall` template spec and if significant changes were made, it may trigger a `FirewallSet` roll. When choosing `RollingUpdate` as a deployment strategy, the deployment controller is waiting for the firewall-controller to connect before throwing away an old `FirewallSet`. The `Recreate` strategy first releases firewalls before creating a new one (can be useful for environments which ran out of available machines but you still want to update). The controller also deploys a service account for the firewall-controller to be able to talk to the seed's kube-apiserver. ### `FirewallSetController` Creates and deletes `Firewall` objects according to the spec and the given number of firewall replicas. It also checks the status of the `Firewall` and report that in the own status. ### `FirewallController` Creates and deletes the physical firewall machine from the spec at the [metal-api](https://github.com/metal-stack/metal-api). ## Rolling a `FirewallSet` through `FirewallMonitor` Annotation A user can initiate rolling the latest firewall set by annotating a monitor in the following way: ```bash kubectl annotate fwmon firewall.metal-stack.io/roll-set=true ``` ## Restarting a systemd-service on the Firewall through Annotation A user can initiate the restart of a systemd service through annotating the `FirewallMonitor`: ```bash kubectl annotate fwmon firewall.metal-stack.io/restart-systemd-services=tailscale ``` The firewall-controller imeplements a whitelist of allowed services to restart. In addition, operators can overwrite this whitelist and also initiate a service restart by annotating the `Firewall` resource instead of the `FirewallMonitor`: ```bash kubectl annotate fw firewall.metal-stack.io/restart-systemd-services-whitelist=tailscale.service,frr.service kubectl annotate fw firewall.metal-stack.io/restart-systemd-services=frr ``` ## Development Most of the functionality is developed with the help of the [integration](integration) test suite. To play with the FCM, you can also run this controller inside the [mini-lab](https://github.com/metal-stack/mini-lab) and without a running Gardener installation: 1. Start up the mini-lab, run `eval $(make dev-env)` and change back to this project's directory 1. Deploy the FCM into the mini-lab with `make deploy` 1. Adapt the example [firewalldeployment.yaml](config/examples/firewalldeployment.yaml) and apply with `kubectl apply -f config/examples/firewalldeployment.yaml` 1. Note that the firewall-controller will not be able to connect to the mini-lab due to network restrictions, so the firewall will not get ready. - You can make the firewall become ready anyway by setting the annotation `kubectl annotate fw -n firewall firewall.metal-stack.io/no-controller-connection=true` --- ## metal-ccm # Kubernetes Cloud Controller Manager for metal metal-ccm is the Kubernetes cloud controller manager implementation for Metal. ## Deploy Read how to deploy the metal CCM [here](deploy/releases/)! ## Building To build the binary, run: ```bash make build ``` It will deposit the binary for your local architecture as `dist/bin/metal-cloud-controller-manager-$(ARCH)` By default `make build` builds the binary using a docker container. To install using your locally installed go toolchain, do: ```bash make build LOCALBUILD=true ``` ## Docker Image To build a docker image, run: ```bash make dockerimage ``` The image will be tagged with `:latest`. --- ## metal-token-refresher Refreshes metal-apiserver-tokens stored in Kubernetes secrets as a CronJob. ## Configuration To configure the metal-token-refresher to refresh a secret, set the following environment variables accordingly. - `METAL_APISERVER_URL` - `TOKEN_SECRET_NAMESPACE` - `TOKEN_SECRET_NAME` - `TOKEN_SECRET_KEY` ```yaml --- apiVersion: batch/v1 kind: CronJob metadata: name: metal-token-refresher spec: schedule: "@hourly" jobTemplate: metadata: name: metal-token-refresher spec: template: spec: serviceAccountName: metal-token-refresher restartPolicy: OnFailure containers: - name: token-refresher image: ghcr.io/metal-stack/metal-token-refresher env: - name: METAL_APISERVER_URL value: http://metal-apiserver:8080 - name: TOKEN_SECRET_NAMESPACE value: "metal-control-plane" - name: TOKEN_SECRET_NAME value: "token-secret" - name: TOKEN_SECRET_KEY value: "token" --- apiVersion: v1 kind: Secret metadata: name: token-secret data: token: "..." ``` Also make sure to configure the service account accordingly. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: metal-token-renewal namespace: metal-stack --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: metal-console-token namespace: metal-stack rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "update", "patch"] - apiGroups: [""] resources: ["secrets"] resourceNames: ["token-secret"] verbs: ["get", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: metal-token-renewal namespace: metal-stack subjects: - kind: ServiceAccount name: metal-token-renewal namespace: metal-stack roleRef: kind: Role name: metal-token-renewal apiGroup: rbac.authorization.k8s.io ``` --- ## metal-metrics-exporter A simple exporter for metal-api metrics. ## sample output ```text # HELP metal_machine_allocation_info Provide information about the machine allocation # TYPE metal_machine_allocation_info gauge metal_machine_allocation_info{clusterTag="00713837-544e-47dd-a55b-52b65a00f8e9",machineid="00000000-0000-0000-0000-ac1f6b7b77c8",machinename="shoot--pbs4kr--test-group-0-5cb6b-585wc",partition="fra-equ01",primaryASN="4210000047",role="machine",state="AVAILABLE",imageId="debian-12.0.20250402"} 1 # HELP metal_image_used_total The total number of machines using a image # TYPE metal_image_used_total gauge metal_image_used_total{classification="preview",created="1588078965",expirationDate="1598533365",features="firewall",imageID="firewall-ubuntu-2.0.20200331",name="Firewall 2 Ubuntu 20200331"} 2 # HELP metal_network_info Shows available prefixes in a network # TYPE metal_network_info gauge metal_network_info{clusterTag="d8f75911-b73f-4ae9-a419-67ead8ab0192",description="",destPrefixes="",isPrivateSuper="false",isUnderlay="false",name="ausfalltest",networkId="020966ab-18da-40d6-ba69-fea8d60e6074",parentNetworkID="tenant-super-network-nbg-w8101",partition="nbg-w8101",prefixes="10.91.112.0/22",projectId="c77daafe-58f8-44df-82eb-6ef631cee3c9",useNat="false",vrf="454"} 1 # HELP metal_network_ip_available The total number of available IPs of the network # TYPE metal_network_ip_available gauge metal_network_ip_available{networkId="020966ab-18da-40d6-ba69-fea8d60e6074"} 1024 # HELP metal_network_ip_used The total number of used IPs of the network # TYPE metal_network_ip_used gauge metal_network_ip_used{networkId="020966ab-18da-40d6-ba69-fea8d60e6074"} 2 # HELP metal_network_prefix_available The total number of available prefixes of the network # TYPE metal_network_prefix_available gauge metal_network_prefix_available{networkId="020966ab-18da-40d6-ba69-fea8d60e6074"} 256 # HELP metal_network_prefix_used The total number of used prefixes of the network # TYPE metal_network_prefix_used gauge metal_network_prefix_used{networkId="020966ab-18da-40d6-ba69-fea8d60e6074"} 0 # HELP metal_partition_capacity_allocatable The total number of waiting allocatable machines in the partition # TYPE metal_partition_capacity_allocatable gauge metal_partition_capacity_allocatable{partition="fra-equ01",size="c1-xlarge-x86"} 2 # HELP metal_partition_capacity_allocated The capacity of allocated machines in the partition # TYPE metal_partition_capacity_allocated gauge metal_partition_capacity_allocated{partition="fra-equ01",size="c1-xlarge-x86"} 1 # HELP metal_partition_capacity_faulty The capacity of faulty machines in the partition # TYPE metal_partition_capacity_faulty gauge metal_partition_capacity_faulty{partition="fra-equ01",size="c1-xlarge-x86"} 0 # HELP metal_partition_capacity_free (DEPRECATED) The total number of allocatable machines in the partition, use metal_partition_capacity_allocatable # TYPE metal_partition_capacity_free gauge metal_partition_capacity_free{partition="fra-equ01",size="c1-xlarge-x86"} 2 # HELP metal_partition_capacity_other The total number of machines in an other state in the partition # TYPE metal_partition_capacity_other gauge metal_partition_capacity_other{partition="fra-equ01",size="c1-xlarge-x86"} 0 # HELP metal_partition_capacity_phoned_home The total number of faulty machines in the partition # TYPE metal_partition_capacity_phoned_home gauge metal_partition_capacity_phoned_home{partition="fra-equ01",size="c1-xlarge-x86"} 1 # HELP metal_partition_capacity_reservations_total The sum of capacity reservations in the partition # TYPE metal_partition_capacity_reservations_total gauge metal_partition_capacity_reservations_total{partition="fra-equ01",size="c1-xlarge-x86"} 1 # HELP metal_partition_capacity_reservations_used The sum of used capacity reservations in the partition # TYPE metal_partition_capacity_reservations_used gauge metal_partition_capacity_reservations_used{partition="fra-equ01",size="c1-xlarge-x86"} 0 # HELP metal_partition_capacity_total The total number of machines in the partition # TYPE metal_partition_capacity_total gauge metal_partition_capacity_total{partition="fra-equ01",size="c1-xlarge-x86"} 3 # HELP metal_partition_capacity_unavailable The total number of faulty machines in the partition # TYPE metal_partition_capacity_unavailable gauge metal_partition_capacity_unavailable{partition="fra-equ01",size="c1-xlarge-x86"} 0 # HELP metal_partition_capacity_waiting The total number of waiting machines in the partition # TYPE metal_partition_capacity_waiting gauge metal_partition_capacity_waiting{partition="fra-equ01",size="c1-xlarge-x86"} 2 # HELP metal_switch_sync_durations The duration of the syncs # TYPE metal_switch_sync_durations gauge metal_switch_sync_durations{partition="fra-equ01",rackid="fra-equ01-rack01",switchname="fra-equ01-r01leaf01"} 2.06530044e+08 metal_switch_sync_durations{partition="fra-equ01",rackid="fra-equ01-rack01",switchname="fra-equ01-r01leaf02"} 2.24029886e+08 # HELP metal_switch_sync_failed 1 when the switch sync is failing, otherwise 0 # TYPE metal_switch_sync_failed gauge metal_switch_sync_failed{partition="fra-equ01",rackid="fra-equ01-rack01",switchname="fra-equ01-r01leaf01"} 0 metal_switch_sync_failed{partition="fra-equ01",rackid="fra-equ01-rack01",switchname="fra-equ01-r01leaf02"} 0 # HELP metal_machine_hardware_info Provide information about the machine # TYPE metal_machine_hardware_info gauge metal_machine_hardware_info{size="c1-xlarge-x86",biosVersion="3.4",bmcVersion="1.73",boardMfg="Supermicro",boardMfgSerial="",boardPartNumber="X11DPT-B",chassisPartNumber="CSE-217BHQ+-R2K22BP2",chassisPartSerial="C217BAH24AE0006",machineid="11111111-2222-3333-4444-aabbccddeeff",partition="fra-equ01",productManufacturer="Supermicro",productPartNumber="SYS-2029BT-HNR",productSerial="E262335X9100766D"} 1 # HELP metal_machine_info Provide the ipmi ip address # TYPE metal_machine_info gauge metal_machine_info{ipmiIP="10.1.1.36:623",machineid="11111111-2222-3333-4444-aabbccddeeff"} 1 # HELP metal_machine_power_state Provide information about the machine power state # TYPE metal_machine_power_state gauge metal_machine_power_state{machineid="11111111-2222-3333-4444-aabbccddeeff"} 1 # HELP metal_machine_power_supplies_healthy Provide information about the number of healthy power supplies # TYPE metal_machine_power_supplies_healthy gauge metal_machine_power_supplies_healthy{machineid="11111111-2222-3333-4444-aabbccddeeff"} 2 # HELP metal_machine_power_supplies_total Provide information about the total number of power supplies # TYPE metal_machine_power_supplies_total gauge metal_machine_power_supplies_total{machineid="11111111-2222-3333-4444-aabbccddeeff"} 2 # HELP metal_machine_power_usage Provide information about the machine power usage in watts # TYPE metal_machine_power_usage gauge metal_machine_power_usage{machineid="11111111-2222-3333-4444-aabbccddeeff"} 69 # HELP metal_switch_interface_bgp_established_timestamp Provide the unix timestamp since BGP is established on the interfaces (0 if not established) # TYPE metal_switch_interface_bgp_established_timestamp gauge metal_switch_interface_bgp_established_timestamp{device="Ethernet2",machineid="11111111-2222-3333-4444-aabbccddeeff",partition="fra-equ01",switchname="fra-equ01-r02leaf01"} 0 metal_switch_interface_bgp_established_timestamp{device="Ethernet2",machineid="11111111-2222-3333-4444-aabbccddeeff",partition="fra-equ01",switchname="fra-equ01-r02leaf02"} 1.731591734e+09 ``` --- ## nftables-exporter # nftables_exporter Export nftables statistics to prometheus, original source from [https://github.com/Sheridan/nftables_exporter](https://github.com/Sheridan/nftables_exporter) ## Need more - Create a feature request, describe the metric that you would like to have and attach exported from nftables json file ## Configuration ### Command line options - `--config=/path/to/file.yaml`: Path to configuration file, default `/etc/nftables_exporter.yaml` - `--version`: Show version and exit ### Configuration file Example content: ```yaml nftables_exporter: bind_to: "[::1]:9630" url_path: "/metrics" nft_location: /sbin/nft fake_nft_json: /path/to/nft.json log_level: warn ``` `fake_nft_json` used for debugging. I create this file with the command `nft -j list ruleset > /path/to/nft.json`. For normal exporter usage, this option is not needed. `log_level` can be one of the following: `debug`, `info`, `warn`, `error`. Default: `warn`. ## Example metrics ```config # HELP nftables_chain_rules Count rules in chain # TYPE nftables_chain_rules gauge nftables_chain_rules{family="inet",name="forward",table="filter"} 2.0 nftables_chain_rules{family="inet",name="global",table="filter"} 15.0 # HELP nftables_table_chains Count chains in table # TYPE nftables_table_chains gauge nftables_table_chains{family="inet",name="filter"} 7.0 nftables_table_chains{family="ip",name="nat"} 4.0 # HELP nftables_rule_bytes Bytes, matched by rule per rule comment # TYPE nftables_rule_bytes gauge nftables_rule_bytes{action="accept",chain="host_spc",comment="[spc->internet] Default http [tcp]",destination_addresses="any",destination_ports="http",family="inet",input_interfaces="internal_0",output_interfaces="external_kis_0",source_addresses="10.0.0.10",source_ports="any",table="filter"} 2280.0 # HELP nftables_rule_packets Packets, matched by rule per rule comment # TYPE nftables_rule_packets gauge nftables_rule_packets{action="accept",chain="host_spc",comment="[spc->internet] Default http [tcp]",destination_addresses="any",destination_ports="http",family="inet",input_interfaces="internal_0",output_interfaces="external_kis_0",source_addresses="10.0.0.10",source_ports="any",table="filter"} 38.0 ``` ## Thank to - [@onokonem](https://github.com/onokonem) --- ## rethinkdb-exporter # RethinkDB Prometheus exporter [![GitHub tag](https://img.shields.io/github/tag/rethinkdb/prometheus-exporter.svg?style=flat)](https://github.com/rethinkdb/prometheus-exporter/releases) [![GoDoc](https://godoc.org/github.com/rethinkdb/prometheus-exporter?status.svg)](https://godoc.org/github.com/rethinkdb/prometheus-exporter) [![Build status](https://travis-ci.org/rethinkdb/prometheus-exporter.svg?branch=master)](https://travis-ci.org/rethinkdb/prometheus-exporter) [RethinkDB](http://www.rethinkdb.com/) statistics exporter for [Prometheus](https://prometheus.io/). Current version: v1.0.1 (RethinkDB v2.4) ## Build and Run Local build and run: ```shell script git clone https://github.com/rethinkdb/prometheus-exporter.git cd prometheus-exporter go build -o rethinkdb-exporter ./rethinkdb-exporter ``` Docker build and run: ```shell script git clone https://github.com/rethinkdb/prometheus-exporter.git cd prometheus-exporter docker build --tag rethinkdb-exporter . docker run -d -p 9050:9050 rethinkdb-exporter ``` ## Parameters Exporter can get parameters from config file, CLI flags or Environment variables. | CLI flag | Env var name | Config key | Description | | --- | --- | --- | --- | | --config | - | - | Config file (default to prometheus-exporter.yaml) | | --web.listen-address string | WEB_LISTEN_ADDRESS | web.listen_address | Address to listen on for web interface and telemetry (default "0.0.0.0:9055") | | --web.telemetry-path string | WEB_TELEMETRY_PATH | web.telemetry_path | Path under which to expose metrics (default "/metrics") | | --db.address | DB_ADDRESSES | db.rethinkdb_addresses | Address of one or more nodes of rethinkdb (default [localhost:28015]) | | --db.enable-tls | DB_ENABLE_TLS | db.enable_tls | Enable to use tls connection | | --db.ca | DB_CA | db.ca_file | Path to CA certificate file for tls connection | | --db.cert | DB_CERT | db.certificate_file | Path to certificate file for tls connection | | --db.key | DB_KEY | db.key_file | Path to key file for tls connection | | --db.username | DB_USERNAME | db.username | Username of rethinkdb user | | --db.password | DB_PASSWORD | db.password | Password of rethinkdb user | | --db.pool-size | DB_POOL_SIZE | db.connection_pool_size | Size of connection pool to rethinkdb (default 5) | | --log.debug | LOG_DEBUG | log.debug | Verbose debug logs | | --log.json-output | LOG_JSON_OUTPUT | log.json_output | Use JSON output for logs | | --stats.table-estimates | STATS_TABLE_ESTIMATES | stats.table_docs_estimates | Collect docs count estimates for each table | Config file can be yaml or json. Example: ```yaml web: listen_address: "0.0.0.0:9050" db: rethinkdb_addresses: - "0.0.0.0:28015" - "0.0.0.0:28016" stats: table_docs_estimates: true ``` ## Metrics Most of the [RethinkDB stats table](http://rethinkdb.com/docs/system-stats/) are exported. Optionally table rows count estimates can be exported from [Table info](https://rethinkdb.com/api/javascript/info). ## Grafana dashboard [Grafana](https://grafana.com/) can be found [here](grafana-dashboard.json). ![image](./assets/grafana.png) ## Contribution Pull requests and issues are welcome. --- ## go-hal # go-hal a hardware abstraction layer for servers [![GoDoc](https://godoc.org/github.com/metal-stack/go-hal?status.svg)](https://pkg.go.dev/github.com/metal-stack/go-hal) go server hardware abstraction, tries to lower the burden of supporting different server vendors. Example usage: ```golang package main import ( "fmt" "github.com/metal-stack/go-hal/connect" ) func main() { ib, err := connect.InBand() if err != nil { panic(err) } firmware, err := ib.Firmware() if err != nil { panic(err) } fmt.Println(firmware) // UEFI err = ib.PowerOff() } ``` --- ## metal-bmc `metal-bmc` is responsible to gather data from machines via the out of band interfaces and report them back to the metal-api. It also passes commands to the machines like power on/off, led on/off, firmware update etc. Access to the console of a machine is also terminated here in conjunction with the `metal-console` running in the control-plane. More details per package as follows: ## Reporter Reporter reports the ip addresses that are leased to ipmi devices together with their machine uuids to the `metal-api`. Therewith it is possible to have knowledge about new machines very early in the `metal-api` and also get knowledge about possibly changing ipmi ip addresses. `metal-bmc` parses the DHCPD lease file and reports the mapping of machine uuids to ipmi ip address to the `metal-api`. ## BMC The `bmc` package serves the following: ### Commands Commands from the metal-api are passed via nsq and executed either through redfish or ipmi against the out-of-band interface of a machine. ### Firmware Firmware updates the firmware of the BIOS and the BMC of a machine. ### Console Console forwards the the serial console access terminated in `metal-console` to the machine. --- ## metal-core metal-core dynamically reconfigures switches based on the state held in the metal-api. Therefore, it must run on every leaf switch and have control over the configuration files for network interfaces and the routing suite (`/etc/frr/frr.config`) of the switches. In the PXE-boot process of machines `metal-core` will act as a proxy between API-requests issued by `pixiecore` and the `metal-api`. The `metal-api` will answer with a mini OS (see [metal-hammer](https://github.com/metal-stack/metal-hammer) and [kernel](https://github.com/metal-stack/kernel)). Besides that, it ensures the proper boot order (IPMI) and monitors their liveliness with [LLDP](https://github.com/metal-stack/go-lldpd)). ## Build Ensure you have `libpcap-dev` installed. ```bash make ``` ## Interface Naming on SONiC Switches On SONiC switches, there are different naming schemas for interfaces. For example, the first port could be named `Ethernet0`, or it could be `Eth1/1` or something similar. Additionally, a port can have an alias, which may be the same as its name or it may follow a different naming schema. If the port is named `Ethernet0` the alias could be `Eth1/1(Port1)`. But it could also be the other way round. The defaults for the naming schemas differ across distributions. These differences wouldn't cause any problems if it weren't for LLDP. An LLDP message carries two fields to identify the port, `portidsubtype` and `portdescription`. Depending on the distribution `portidsubtype` will be either the port's name or its alias and `portdescription`, usually, will be the other of the two. So the metal-core registers its ports at the metal-api which stores the names as `V1SwitchNic.Name` and aliases as `V1SwitchNic.Identifier`. At the same time, when a machine registers at the metal-api it reports its LLDP neighbors and identifies the neighbors' ports by `portidsubtype` and `portdescription`. When the metal-api attempts to match the machine's neighbors with the neighboring switches' ports it compares each neighboring switch's `portidsubtype` with all of its Nics' `.Identifier` fields. But this only works if the port's alias is identical to its `portidsubtype` which, as stated above, is not always the case. While it is possible to configure `portidsubtype` and `portdescription` via `lldpcli`, this configuration is not persisted. A simple change of the MTU will restore the defaults. So this is not a viable solution. To accommodate all of the possible combinations, there is an `InterfaceNamingSchema` option. This option has nothing to do with SONiC's `interface-naming`. It simply tells metal-core how it should report its ports to the metal-api. It allows the following values. - `default`: `V1SwitchNic.Name` is the interface name; `V1SwitchNic.Identifier` is the interface alias - `swap`: name and alias get swapped - `name`: both `V1SwitchNic.Name` and `V1SwitchNic.Identifier` will be the interface name - `alias`: both `V1SwitchNic.Name` and `V1SwitchNic.Identifier` will be the interface alias To determine which of these suits your setup compare the port aliases with the LLDP configuration for `portidsubtype`. If they match the correct value for `InterfaceNamingSchema` is `default`. If not, check if `portdescription` matches the alias and `portidsubtype` matches the name. In that case you can use `swap`. If you need to have both of the fields to have the same value, use either `name` or `alias`. --- ## metal-hammer # Metal Stack Hammer Hammer is used to boot a bare metal server via PXE together with the Metal Stack kernel. Hammer is a initrd which runs a small golang binary as init process. This does the following actions: - Ensures all interfaces are up - Check if the server was booted in UEFI, if not modify the bios to uefi and reboots - Wipes as existing disks by either: - run secure erase if possible by using the mechanism in modern disks, this is true for most SSD´s and NVME disks. - If not possible run mkfs.ext4 --discard on the disks. - Gather HW information and report them back to metal-api: - CPU Core count - Memory count - Disks with their size and device path - Network adapters which have an active uplink with their interface name, own mac address and mac address of the switch chassis where this network card is connected to. 2 distinct switch chassis are required. - IPMI interface with mac and ipaddress. - create a metal user on IPMI with a strong password - Set BIOS boot order to contain only PXE and Hard Disk as possible options. - Wait until a `machine create` command was issued from metal-api ## Local Testing ```bash make clean initrd vagrant-up ``` ## Create a PXE boot initrd with u-root In order to be able to create an initrd image which is suitable to boot a bare metal server with the required tools to discover and install the target os, we use u-root. ### Quickstart - download u-root: ```bash go get -u github.com/u-root/u-root ``` - build the initrd ```bash make initrd ``` ### check content ```bash cpio -itv < metal-hammer-initrd.img ``` ### start it ```bash make vagrant-up ``` --- ## pixie # Pixiecore This project is a permanent fork of: [Pixiecore](https://github.com/danderson/netboot/tree/master/pixiecore) Sample command to run `pixie` in grpc mode which talks to the metal-api and provides grpc client certificates and the metal-api-view-hmac to the metal-hammer. With this metal-hammer will be able to talk to metal-api directly. ```bash docker run -it --rm -name pixiecore \ --network host \ --dns 10.1.253.13 \ --dns 10.1.253.29 \ --volume "/certs/grpc:/certs/grpc:ro" \ ghcr.io/metal-stack/pixie grpc \ --debug \ --dhcp-no-bind \ --pixie-api-url http://the-ip-of-this-service/certs \ --grpc-address api.metal-stack.dev:50051 \ --grpc-ca-cert /certs/grpc/ca.pem \ --grpc-cert /certs/grpc/client.pem \ --grpc-key /certs/grpc/client-key.pem \ --metal-api-url https://api.metal-stack.io/metal \ --metal-api-view-hmac a-view-hmac \ --partition partition-1 --ntp-servers 0.custom.ntp,1.custom.ntp ``` --- ## csi-driver-lvm # csi-driver-lvm # csi-driver-lvm utilizes local storage of Kubernetes nodes to provide persistent storage for pods. It automatically creates hostPath based persistent volumes on the nodes. Underneath it creates a LVM logical volume on the local disks. A comma-separated list of grok pattern, which disks to use must be specified. This CSI driver is derived from [csi-driver-host-path](https://github.com/kubernetes-csi/csi-driver-host-path) and [csi-lvm](https://github.com/metal-stack/csi-lvm) > [!WARNING] > Note that there is always an inevitable risk of data loss when working with local volumes. For this reason, be sure to back up your data or implement proper data replication methods when using this CSI driver. ## Currently it can create, delete, mount, unmount and resize block and filesystem volumes via lvm ## For the special case of block volumes, the filesystem-expansion has to be performed by the app using the block device ## Automatic PVC Deletion on Pod Eviction The persistent volumes created by this CSI driver are strictly node-affine to the node on which the pod was scheduled. This is intentional and prevents pods from starting without the LV data, which resides only on the specific node in the Kubernetes cluster. Consequently, if a pod is evicted (potentially due to cluster autoscaling or updates to the worker node), the pod may become stuck. In certain scenarios, it's acceptable for the pod to start on another node, despite the potential for data loss. The csi-driver-lvm-controller can capture these events and automatically delete the PVC without requiring manual intervention by an operator. To use this functionality, the following is needed: - This only works on `StatefulSet`s with volumeClaimTemplates and volume references to the `csi-driver-lvm` storage class - In addition to that, the `Pod` or `PersistentVolumeClaim` managed by the `StatefulSet` needs the annotation: `metal-stack.io/csi-driver-lvm.is-eviction-allowed: true` ## Installation ## **For convenience, helm charts for installation are synced to a separate repository called [helm-charts](https://github.com/metal-stack/helm-charts). The source for this chart is located in the `charts` folder.** You have to set the `devicePattern` for your hardware to specify which disks should be used to create the volume group. ```bash helm install csi-driver-lvm ./charts/csi-driver-lvm --set lvm.devicePattern='/dev/nvme[0-9]n[0-9]' # or alternatively after the a release: # helm install --repo https://helm.metal-stack.io csi-driver-lvm csi-driver-lvm --set lvm.devicePattern='/dev/nvme[0-9]n[0-9]' ``` Now you can use one of following storageClasses: * `csi-driver-lvm-linear` * `csi-driver-lvm-mirror` * `csi-driver-lvm-striped` * `csi-driver-lvm-linear-encrypted` * `csi-driver-lvm-mirror-encrypted` * `csi-driver-lvm-striped-encrypted` To get the previous old and now deprecated `csi-lvm-sc-linear`, ... storageclasses, set helm-chart value `compat03x=true`. ## Encryption ## csi-driver-lvm supports LUKS2 encryption for volumes at rest. When encryption is enabled, the LVM logical volume is formatted with LUKS2 and a dm-crypt mapper device is used transparently for all I/O. ### Setup ### 1. Create a Kubernetes Secret containing the LUKS passphrase: ```bash kubectl create secret generic csi-lvm-encryption-secret \ --from-literal=passphrase='my-secret-passphrase' ``` 2. Enable the encrypted StorageClasses in your Helm values (they are disabled by default): ```yaml storageClasses: linearEncrypted: enabled: true mirrorEncrypted: enabled: true stripedEncrypted: enabled: true ``` 3. Create PVCs using one of the encrypted StorageClasses. The encryption is handled transparently by the driver. ### How it works ### - **NodeStageVolume**: LUKS-formats the LV (first use only), then opens it via `cryptsetup luksOpen`, creating a `/dev/mapper/csi-lvm-` device - **NodePublishVolume**: Mounts the mapper device (instead of the raw LV) to the target path - **NodeUnpublishVolume**: Unmounts as usual - **NodeUnstageVolume**: Closes the LUKS device via `cryptsetup luksClose` - **Volume expansion**: The LV is extended first, then the LUKS layer is resized, then the filesystem Both filesystem and raw block access types are supported with encryption. ### Encrypted Ephemeral Volumes ### Encryption is also supported for CSI ephemeral (inline) volumes. Since ephemeral volumes bypass `NodeStageVolume`, the LUKS formatting and opening is handled directly during `NodePublishVolume`, and the LUKS device is closed during `NodeUnpublishVolume`. To use an encrypted ephemeral volume, specify `encryption: "true"` in `volumeAttributes` and reference the encryption secret via `nodePublishSecretRef`: ```yaml volumes: - name: encrypted-ephemeral csi: driver: lvm.csi.metal-stack.io volumeAttributes: size: "100Mi" type: "linear" encryption: "true" nodePublishSecretRef: name: csi-lvm-encryption-secret ``` ## Migration ## If you want to migrate your existing PVC to / from csi-driver-lvm, you can use [korb](https://github.com/BeryJu/korb). ### Todo ### * implement CreateSnapshot(), ListSnapshots(), DeleteSnapshot() ### Test ### ```bash kubectl apply -f examples/csi-pvc-raw.yaml kubectl apply -f examples/csi-pod-raw.yaml kubectl apply -f examples/csi-pvc.yaml kubectl apply -f examples/csi-app.yaml kubectl delete -f examples/csi-pod-raw.yaml kubectl delete -f examples/csi-pvc-raw.yaml kubectl delete -f examples/csi-app.yaml kubectl delete -f examples/csi-pvc.yaml ``` ### Development ### In order to run the integration tests locally, you need to create to loop devices on your host machine. Make sure the loop device mount paths are not used on your system (default path is `/dev/loop10{0,1}`). You can create these loop devices like this: ```bash for i in 100 101; do fallocate -l 1G loop${i}.img ; sudo losetup /dev/loop${i} loop${i}.img; done sudo losetup -a # https://github.com/util-linux/util-linux/issues/3197 # use this for recreation or cleanup # for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done ``` You can then run the tests against a kind cluster, running: ```bash make test ``` To recreate or cleanup the kind cluster: ```bash make test-cleanup ``` --- ## MULTITENANCY # Multi tenancy with lightbits storage Multi tenancy is a crucial aspect when providing central storage. This divides into several aspects, impact to storage from one tenant to another tenant must be restricted so that neither data can be seen, destroyed or modified by another tenant, but also read and write actions at a high rate must not impact the performance of the storage of other tenants. Lightbits storage uses NVMEoTCP (NVME over TCP) as transport protocol as defined here: [NVMEoF](https://nvmexpress.org/developers/nvme-of-specification/). The storage traffic is routed over the same network as normal TCP/IP traffic. The basic setup of the components is shown here: ![Diagram](./assets/nvme-over-tcp.jpg) The current implementation prevents malicious access to data, prevention of performance impacts are subject of later lightos releases. ## Gardener and metal-stack Multi tenancy in metal-stack and gardener are based on projects. In metal-stack, projects additionally belong to a tenant entity that groups projects. A single kubernetes cluster is created in the scope of project, one project can have multiple kubernetes clusters. Every kubernetes cluster will get physically separated firewall and worker nodes in a dedicated routing domain called VRF. Every kubernetes cluster is totally separated from a physical an network perspective, nothing is shared. Lightbits storage has also the notion of a project, once a cluster is created, a new project is created in the lightos storage API, the project there matches the project from the gardener/metal-stack perspective. For every cluster an authentication token in the JWT format is created, this token is able to create/update/list/delete volumes in the lightos cluster in the given project, resp. lightos project. For every kubernetes cluster, even in the same project, an individual JWT token is created. The token is also set to have a 8 day validity, 1 day before the token will get invalid and the cluster still exists, a new token is issued. The duros-controller is responsible to create such tokens, it is deployed in the seed's shoot namespace (find details on gardener architecture [here](./architecture.md)). This namespace is fully managed by the provider and invisible for the cluster user. Once the token has been created, the token is stored in a secret in the actual user cluster alongside with the deployment of the lightbits CSI driver and storage classes. This CSI driver will then be responsible to create/update/delete volumes based on the manifests deployed in the cluster. ```bash k get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE partition-gold csi.lightbitslabs.com Delete Immediate true 7d4h partition-silver csi.lightbitslabs.com Delete Immediate true 7d4h ``` The storageclass partition-gold with 3 fold replication and the pointers to the secrets. ```bash k get sc partition-gold -o yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: "2022-01-24T08:40:03Z" name: partition-gold resourceVersion: "234" uid: 3b40edfa-ff72-4426-904c-4205b061e311 parameters: compression: enabled csi.storage.k8s.io/controller-expand-secret-name: lb-csi-creds csi.storage.k8s.io/controller-expand-secret-namespace: kube-system csi.storage.k8s.io/controller-publish-secret-name: lb-csi-creds csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: lb-csi-creds csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: lb-csi-creds csi.storage.k8s.io/node-stage-secret-namespace: kube-system csi.storage.k8s.io/provisioner-secret-name: lb-csi-creds csi.storage.k8s.io/provisioner-secret-namespace: kube-system mgmt-endpoint: 10.131.44.1:443,10.131.44.2:443,10.131.44.3:443 mgmt-scheme: grpcs project-name: 0f89286d-0429-4209-a8a9-8612befbff97 replica-count: "3" provisioner: csi.lightbitslabs.com reclaimPolicy: Delete volumeBindingMode: Immediate ``` This is the secret where the storageclass points to: ```bash k get secret lb-csi-creds -o yaml apiVersion: v1 data: jwt: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklqQm1PRGt5T0Raa0xUQTBNamt0TkRJd09TMWhPR0U1TFRnMk1USmlaV1ppWm1ZNU56cHliMjkwSWl3aWRIbHdJam9pU2xkVUluMC5leUpsZUhBaU9qRTJORFF6TURrMk1qa3NJbXAwYVNJNkltUmpNVGc0TWpVd0xUZGtORGt0TkRjMk1DMWlZVGs0TFdGbE1ESmpNR0l5WmpNeVpTSXNJbWxoZENJNk1UWTBNell4T0RReU9Td2lhWE56SWpvaVpIVnliM010WTI5dWRISnZiR3hsY2lJc0luTjFZaUk2SW5Ob2IyOTBMUzF3WkRjMmJYSXRMV2x1ZEhSbGMzUXdJaXdpY205c1pYTWlPbHNpTUdZNE9USTRObVF0TURReU9TMDBNakE1TFdFNFlUa3RPRFl4TW1KbFptSm1aamszT21Ga2JXbHVJbDE5LmIxaWo0aHV0R2lmUll1YlZMb1J5WlBKRHZ0ZWpodDZqdW1KdW1xbEMyOWpwRUxWa0JfdG4tZU9VbERPb09HUEZTN2FhRDBGOXRKSGVrOGVYQ0xqZ1R2RkdCMzI5aE5zTzlra0M5OXNQZWJvaWE1RmRLUmlUbjNBTC1KcXZZZ3pKZTNaZmZNWFdFVHhsZmxSXzFTNERpQlZFNERSc3hNczNpbWt2Nl83cU5BUEhXd2ZCdU5OUDVyMmxOdGRqdVl2VXlqN3hNWTZhODdSU1RkMUZINGlaMUx3OEZwLW9haTdyN1M1SlhnMkhBcUU2VTJ1UTMzMWhwMlREY3M2ZTJjcDdpbjJNSnhHWkRYUE5SSGFnRUJyWFdfYVVJdjJpYTU3emVGeWoyMFdhVVlUVS1rRFhxYXBUaDJfQXlPV19jQ1hhbTJWdVY0N1IwOWxqdVNocGRaXzJFZw== kind: Secret metadata: creationTimestamp: "2022-01-24T08:40:02Z" name: lb-csi-creds namespace: kube-system resourceVersion: "1837602" uid: 406c3583-d4cd-4a0b-8615-3a6f2b9b7577 ``` ## Lightbits and NVMEoTCP Once a volume is created and mounted, e.g. a PVC and PV, the csi driver will first create the volume on the lightos API with his token and set the hosts which are allowed to talk to this volume to the name of the worker nodes. ```bash k get pvc,pv,node NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/data-sampla-app-0 Bound pvc-c4b7822b-b3c8-414a-a1fa-9350d30a4f5c 1Gi RWO partition-silver 25s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-c4b7822b-b3c8-414a-a1fa-9350d30a4f5c 1Gi RWO Delete Bound sampla-app/data-sampla-app-0 partition-silver 25s NAME STATUS ROLES AGE VERSION node/shoot--pd76mr--inttest0-group-0-845b8-49r7x Ready node 6d21h v1.21.9 node/shoot--pd76mr--inttest0-group-0-845b8-ng7xh Ready node 7d1h v1.21.9 ``` The pod which mounts this volume is running on the node `shoot--pd76mr--inttest0-group-0-845b8-49r7x`. ```bash k get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sampla-app-0 1/1 Running 0 5m34s 10.244.15.152 shoot--pd76mr--inttest0-group-0-845b8-49r7x ``` Once the lightos cluster has set the ACL of this volume only a node which sets the hostnqn matching the given ACL is able to mount that volume. ```bash lbcli list volumes Name UUID State Protection State NSID Size Replicas Compression ACL Rebuild Progress pvc-c4b7822b-b3c8-414a-a1fa-9350d30a4f5c 7828aa17-2316-442d-883e-d000436d41f2 Available FullyProtected 631 1.0 GiB 2 true values:"nqn.2019-09.com.lightbitslabs:host:shoot--pd76mr--inttest0-group-0-845b8-49r7x.node" None ``` The NVMEoTCP module in the linux kernel on the worker node side and on the lightos side implements setting the host nqn to match these ACL expectations. This can be inspected on the worker node side by looking at the host nqn, e.g. the name of the nvme drive (NVMe qualified name). This nqn matches the ACL on the lightos server side. ```bash cat /sys/devices/virtual/nvme-fabrics/ctl/nvme1/hostnqn nqn.2019-09.com.lightbitslabs:host:shoot--pd76mr--inttest0-group-0-845b8-49r7x.node ``` ## Further improvements In the upcoming lightos release the performance aspects of multi tenancy are addressed. This is achieved that it will be possible to cap the maximum throughput possible per volume. This will ensure that no single tenant is able to saturate the whole lightos cluster and impact other tenants using this lightos cluster. --- ## duros-controller A Bridge between cloud-native and duros storage controller. This controller is responsible to create a project in the duros-api and an authentication token to access the duros storage. After the authentication token is created, it creates a storage class deployment in the shoot which itself authenticates against the duros storage to create/delete PV and mount them. ## Architecture ![Architecture](./assets/architecture.drawio.svg) The Dataplane is built like this: ![Dataplane](./assets/dataplane.drawio.svg) ## Configuration The configuration is done through a CustomResource in the shoot namespace in the seed. The Duros CustomResource contains a Reference to the metal-api project the Endpoints and the name of StorageClasses which should be created. The JWT Token is stored as StorageClass Secret. The Gardener Extension Provider Metal will create a Duros CustomResource on shoot creation. Example CR which will reconcile 2 StorageClasses, one with 2 replicas, and one with 3 replicas. ```yaml --- apiVersion: storage.metal-stack.io/v1 kind: Duros metadata: namespace: duros name: sample spec: metalProjectID: 5820c4e7-fbd4-4e4b-a40b-2b83eb34bbe3 storageClasses: - name: partition-silver replicas: 2 compression: "true" - name: partition-gold replicas: 3 compression: "true" ``` Ensure you also have a ClusterwideNetworkPolicy deployed to have access to the duros storage servers with the required ports ```yaml --- apiVersion: metal-stack.io/v1 kind: ClusterwideNetworkPolicy metadata: name: allow-to-storage namespace: firewall spec: egress: - ports: - port: 443 protocol: TCP - port: 4420 protocol: TCP - port: 8009 protocol: TCP to: - cidr: 10.128.0.0/14 ``` ## Use Cases There are several use cases during the lifecycle of storage consumed by customers. ### First usage from a tenants project There was no storage usage from a project before, e.g. because the customer created a new project and then a cluster is created. Then `gepm` will create the `Duros` CR and deploys the `duros-controller` into the shoot namespace in the seed. The `duros-controller` will check if there is already a project present in the `duros-api` with the metal-api project ID and create it if it does not exist. A JWT Token with project permission is created and stored as secret in the shoot namespace. With this Secret a `StorageClass` deployment in the shoot is created. Then the Cluster user is able to create PVs. ### Second cluster in the same project If a second cluster in the same project in the same partition is created, the `duros-controller` will see there is already a project present in the duros-api. A new secret for this cluster is created and stored StorageClass Secret. ### Clusters are deleted If a cluster is deleted, even if it is the latest in the project, storage volumes are not deleted. This enables customers to keep their storage and consume it in new clusters. ### Storage Volume and Project list/delete The cloud-api will add endpoints to list/delete duros volumes and list projects, this will be done through a grpc proxy as shown in the architecture. A Duros project will be deleted if the metal-api project is deleted. A check if there are no volumes present is also implemented. ## Accounting Accounting of volumes is done with the kube-counter running in every shoot in the seed. Accounting of volumes currently not in use in any of the clusters are listed from the cloud-api and reported to the accounting-api. ## Tenant separation How tenant separation works is described more detailed [here](./MULTITENANCY.md) ## TODO - check if Gardener deletes PVC's after cluster deletion.