Add scripts for integration test setup and docs generation

* Add an OpenPGP test keypair.
 * Move script parts of documentation into script, so they can be used on a fresh Ubuntu 20.04 system
   for within a github action to setup a csaf_provider and upload documents to it for an integration test.
 * Use dineshsonachalam/markdown-autodocs in github action to automatically
   insert lines from the scripts into the docs.

Co-authored-by: Bernhard Reiter <bernhard@intevation.de>

docs/scripts/Readme.md

@@ -0,0 +1,25 @@
+Scripts for assisting the Integration tests. They are written on Ubuntu 20.04 TLS amd64.
+
+- `prepareUbunutForITest.sh` installs the required packages for the csaf_distribution integration tests on a naked ubuntu 20.04 LTS amd64.
+
+- `TLSConfigsForITest.sh` generates a root CA and webserver cert by running `createRootCAForITest.sh` and `createWebserverCertForITest.sh`
+and configures nginx for serving TLS connections.
+
+- `TLSClientConfigsForITest.sh` generates client certificates by calling `createCCForITest.sh` which uses the root certificate initialized before with `createRootCAForITest.sh`. It configures nginx to enable the authentication with client certificate. (This assumes that the same folder name is used to create the root certificate)
+
+- `setupProviderForITest.sh` builds the csaf_provider, writes the required nginx configurations and create the initial folders. IT calls `uploadToProvider.sh` to upload some csaf example files to the provider.
+
+As creating the folders needs to authenticate with the csaf_provider, the configurations of TLS server and Client certificate authentication should be set. So it is recommended to call the scripts in this order: `TLSConfigsForITest.sh`, `TLSClientConfigsForITest.sh`, `setupProviderForITest.sh`
+
+Calling example (as root):
+``` bash
+    curl --fail -O https://raw.githubusercontent.com/csaf-poc/csaf_distribution/main/docs/scripts/prepareUbuntuInstanceForITests.sh
+    bash prepareUbuntuInstanceForITests.sh
+
+    git clone https://github.com/csaf-poc/csaf_distribution.git
+    pushd csaf_distribution/docs/scripts/
+
+    env FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)" ./TLSConfigsForITest.sh
+    env FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)" ./TLSClientConfigsForITest.sh
+    ./setupProviderForITest.sh
+```

docs/scripts/TLSClientConfigsForITest.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+# It sets the right nginx configurations for enabling client certificate authentication.
+# FOLDERNAME and ORGANAME variables must be set.
+# FOLDERNAME: Where to store the CAs and keys.
+# ORGANAME: The organization name used in the CA template.
+# Usage Example: env FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)" ./TLSClientConfigsForITest.sh
+
+set -e
+
+NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
+
+cd ~/csaf_distribution/docs/scripts/
+source ./createCCForITest.sh
+
+echo '
+        ssl_client_certificate '${SSL_CLIENT_CERTIFICATE}' # e.g. ssl_client_certificate /etc/ssl/rootca-cert.pem;
+        ssl_verify_client optional;
+        ssl_verify_depth 2;
+
+        # This example allows access to all three TLP locations for all certs.
+        location ~ /.well-known/csaf/(red|green|amber)/{
+
+            autoindex on;
+
+            # in this location access is only allowed with client certs
+            if  ($ssl_client_verify != SUCCESS){
+                # we use status code 404 == "Not Found", because we do not
+                # want to reveal if this location exists or not.
+                return 404;
+            }
+       }
+'> clientCertificateConfigs.txt
+
+sed -i "/^server {/r  ${HOME}/${FOLDERNAME}/clientCertificateConfigs.txt" $NGINX_CONFIG_PATH
+
+systemctl reload nginx

docs/scripts/TLSConfigsForITest.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+# This script generates webserver cert that is signed with the generated root CA.
+# It sets the right nginx configurations for serving TLS connections.
+# FOLDERNAME and ORGANAME variables must be set.
+# FOLDERNAME: Where to store the CAs and keys.
+# ORGANAME: The organization name used in the CA template.
+# Usage Example: env FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)" ./TLSConfigsForITest.sh
+
+set -e
+
+NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
+
+cd ~/csaf_distribution/docs/scripts/
+## Create Root CA
+./createRootCAForITest.sh
+
+## Create webserver cert
+source ./createWebserverCertForITest.sh
+
+# Configure nginx
+echo '
+        listen 443 ssl default_server; # ipv4
+        listen [::]:443 ssl http2 default_server;  # ipv6
+
+        ssl_certificate  '${SSL_CERTIFICATE}' # e.g. ssl_certificate /etc/ssl/csaf/bundle.crt
+        ssl_certificate_key '${SSL_CERTIFICATE_KEY}' # e.g. ssl_certificate_key /etc/ssl/csaf/testserver-key.pem;
+
+        ssl_protocols TLSv1.2 TLSv1.3;
+' > TLSConfigs.txt
+
+# a second listener port for testing setup where someone wants to tunnel access
+# to an unpriviledged port and still have the same access url
+echo '
+        listen 8443 ssl default_server; # ipv4
+        listen [::]:8443 ssl http2 default_server;  # ipv6
+' > TLS8443Configs.txt
+
+cp $NGINX_CONFIG_PATH $NGINX_CONFIG_PATH.org
+sed -i "/^server {/r ${HOME}/${FOLDERNAME}/TLSConfigs.txt" $NGINX_CONFIG_PATH
+sed -i "/^server {/r ${HOME}/${FOLDERNAME}/TLS8443Configs.txt" $NGINX_CONFIG_PATH
+sed -i "/^\s*listen.*80/d" $NGINX_CONFIG_PATH # Remove configs for listinig on port 80
+systemctl reload nginx
+

docs/scripts/createCCForITest.sh

@@ -0,0 +1,57 @@
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+# This scripts creates two client certificates. It uses for signing the root certifcate
+# created with `createRootCAForITest.sh` that must be run earlier.
+
+set -e
+
+mkdir -p ~/${FOLDERNAME}
+cd ~/${FOLDERNAME}
+
+certtool --generate-privkey --outfile testclient1-key.pem
+
+echo '
+organization = "'${ORGANAME}'"
+country = DE
+cn = "TLS Test Client 1"
+
+tls_www_client
+signing_key
+encryption_key
+
+serial = 020
+expiration_days = 50
+' > gnutls-certtool.testclient1.template
+
+certtool --generate-certificate --load-privkey testclient1-key.pem --outfile testclient1.crt --load-ca-certificate rootca-cert.pem --load-ca-privkey rootca-key.pem --template gnutls-certtool.testclient1.template --stdout | head -1
+
+certtool --load-ca-certificate rootca-cert.pem --load-certificate testclient1.crt --load-privkey testclient1-key.pem --to-p12 --p12-name "Test Client 1" --null-password --outder --outfile testclient1.p12
+
+certtool --generate-privkey --outfile testclient2-key.pem
+
+echo '
+organization = "'${ORGANAME}'"
+country = DE
+cn = "TLS Test Client 2"
+
+tls_www_client
+signing_key
+encryption_key
+
+serial = 021
+expiration_days = 1
+' > gnutls-certtool.testclient2.template
+
+certtool --generate-certificate --load-privkey testclient2-key.pem --outfile testclient2.crt --load-ca-certificate rootca-cert.pem --load-ca-privkey rootca-key.pem --template gnutls-certtool.testclient2.template --stdout | head -1
+
+certtool --load-ca-certificate rootca-cert.pem --load-certificate testclient2.crt --load-privkey testclient2-key.pem --to-p12 --p12-name "Test Client 2" --null-password --outder --outfile testclient2.p12
+
+SSL_CLIENT_CERTIFICATE=$(
+echo "$PWD/rootca-cert.pem;"
+)

docs/scripts/createRootCAForITest.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+set -e
+
+mkdir -p ~/${FOLDERNAME}
+cd ~/${FOLDERNAME}
+
+certtool --generate-privkey --outfile rootca-key.pem
+
+echo '
+organization = "'${ORGANAME}'"
+country = DE
+cn = "Tester"
+
+ca
+cert_signing_key
+crl_signing_key
+
+serial = 001
+expiration_days = 100
+' >gnutls-certtool.rootca.template
+
+certtool --generate-self-signed --load-privkey rootca-key.pem --outfile rootca-cert.pem --template gnutls-certtool.rootca.template --stdout | head -1

docs/scripts/createWebserverCertForITest.sh

@@ -0,0 +1,41 @@
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+set -e
+
+cd ~/${FOLDERNAME}
+
+certtool --generate-privkey --outfile testserver-key.pem
+
+echo '
+organization = "'${ORGANAME}'"
+country = DE
+cn = "Service Testing"
+
+tls_www_server
+signing_key
+encryption_key
+non_repudiation
+
+dns_name = "*.local"
+dns_name = "localhost"
+
+serial = 010
+expiration_days = 50
+' > gnutls-certtool.testserver.template
+
+certtool --generate-certificate --load-privkey testserver-key.pem --outfile testserver.crt --load-ca-certificate rootca-cert.pem --load-ca-privkey rootca-key.pem --template gnutls-certtool.testserver.template --stdout | head -1
+
+cat testserver.crt rootca-cert.pem >bundle.crt
+
+SSL_CERTIFICATE=$(
+echo "$PWD/bundle.crt;"
+)
+SSL_CERTIFICATE_KEY=$(
+echo "$PWD/testserver-key.pem;"
+)

docs/scripts/downloadExamples.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+#
+# Desc: Tries getting csaf 2.0 examples from api.github. Do not run too often!
+#
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+set -e
+
+# using an extended regular expression to whitelist only CSAF 2.0 filenames
+# with a sane path
+
+CSAFPATHregexp='^ *"path": "(csaf_2.0/examples/csaf/[a-z0-9+-_]+.json)",'
+
+curl --silent --show-error -H 'Accept: application/vnd.github.v3.raw' \
+ https://api.github.com/repos/oasis-tcs/csaf/contents/csaf_2.0/examples/csaf \
+ | grep -E "$CSAFPATHregexp" \
+ |  sed -E -e "s;${CSAFPATHregexp};\1;" \
+ > csaf_examples_pathnames.txt
+
+mkdir csaf_examples
+cd csaf_examples
+
+cat ../csaf_examples_pathnames.txt | \
+ xargs -I {} \
+  curl --silent --show-error -H 'Accept: application/vnd.github.v3.raw' \
+   https://api.github.com/repos/oasis-tcs/csaf/contents/{} -O

docs/scripts/prepareUbuntuInstanceForITests.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -e
+
+# This script prepares a naked Ubuntu 20.04 LTS amd64
+# for the csaf_distribution integration tests
+# by installing the required packages.
+
+
+apt install -y make git nginx fcgiwrap gnutls-bin
+
+# Install Go from binary distribution
+curl -O https://storage.googleapis.com/golang/go1.18.linux-amd64.tar.gz
+tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz

docs/scripts/setupProviderForITest.sh

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+# This script sets up the csaf_provider and writes the required nginx configurations.
+# It creates the initial folders and uploads some example files to the csaf_provider with the help of `uploadToProvider.sh`
+
+set -e
+
+chgrp -R www-data  /var/www
+chmod -R g+w  /var/www
+
+NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
+
+cp /usr/share/doc/fcgiwrap/examples/nginx.conf /etc/nginx/fcgiwrap.conf
+
+echo '
+# Include this file on your nginx.conf to support debian cgi-bin scripts using
+# fcgiwrap
+location /cgi-bin/ {
+  # Disable gzip (it makes scripts feel slower since they have to complete
+  # before getting gzipped)
+  gzip off;
+
+  # Set the root to /usr/lib (inside this location this means that we are
+  # giving access to the files under /usr/lib/cgi-bin)
+  root  /usr/lib;
+
+  # Fastcgi socket
+  fastcgi_pass  unix:/var/run/fcgiwrap.socket;
+
+  # Fastcgi parameters, include the standard ones
+  include /etc/nginx/fastcgi_params;
+
+  fastcgi_split_path_info ^(.+\.go)(.*)$;
+
+  # Adjust non standard parameters (SCRIPT_FILENAME)
+  fastcgi_param SCRIPT_FILENAME  /usr/lib$fastcgi_script_name;
+
+  fastcgi_param PATH_INFO $fastcgi_path_info;
+  fastcgi_param CSAF_CONFIG /usr/lib/csaf/config.toml;
+
+  fastcgi_param SSL_CLIENT_VERIFY $ssl_client_verify;
+  fastcgi_param SSL_CLIENT_S_DN $ssl_client_s_dn;
+  fastcgi_param SSL_CLIENT_I_DN $ssl_client_i_dn;
+}
+' > /etc/nginx/fcgiwrap.conf
+
+sed -i "/^server {/a        include fcgiwrap.conf;" $NGINX_CONFIG_PATH
+
+echo "
+        # For atomic directory switches
+        disable_symlinks off;
+
+        # directory listings
+        autoindex on;
+" > locationConfig.txt
+sed -i "/^\s*location \/ {/r locationConfig.txt" $NGINX_CONFIG_PATH # Insert config inside location{}
+
+systemctl reload nginx
+
+# assuming that we are in a checked out version in the docs/scripts directory
+# and we want to build the version that is currently checked out
+pushd ../..
+
+export PATH=$PATH:/usr/local/go/bin
+make build_linux
+# Place the binary under the corresponding path.
+mkdir -p /usr/lib/cgi-bin/
+cp bin-linux-amd64/csaf_provider /usr/lib/cgi-bin/csaf_provider.go
+
+mkdir -p /usr/lib/csaf/
+cp docs/test-keys/*.asc /usr/lib/csaf/
+# Configuration file
+echo '
+# upload_signature = true
+# key = "/usr/lib/csaf/public.asc"
+key = "/usr/lib/csaf/private.asc"
+#tlps = ["green", "red"]
+canonical_url_prefix = "https://localhost:8443"
+#no_passphrase = true
+' > /usr/lib/csaf/config.toml
+
+# Create the Folders
+curl https://localhost:8443/cgi-bin/csaf_provider.go/create --cert-type p12 --cert ~/devca1/testclient1.p12 --insecure
+
+popd
+
+# Upload files
+./uploadToProvider.sh

docs/scripts/uploadToProvider.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+#
+# Desc: Call ./downloadExamples.sh and then try csaf_uploader.
+#
+# This file is Free Software under the MIT License
+# without warranty, see README.md and LICENSES/MIT.txt for details.
+#
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
+# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
+
+set -e
+
+
+# assumes that the following script only downloads file with filenames
+# following https://docs.oasis-open.org/csaf/csaf/v2.0/cs01/csaf-v2.0-cs01.html#51-filename
+# which are save to process further
+./downloadExamples.sh
+
+TLPs=("white" "green" "amber" "red")
+COUNTER=0
+for f in $(ls csaf_examples);
+    do
+        ../../bin-linux-amd64/csaf_uploader -a upload -t ${TLPs[$COUNTER]} \
+        -u https://localhost:8443/cgi-bin/csaf_provider.go --insecure -P security123 \
+        --client-cert ~/devca1/testclient1.crt --client-key ~/devca1/testclient1-key.pem \
+        ./csaf_examples/"$f";
+        let COUNTER++
+    done;