SystemSh0cker/gocsaf
1
0
Fork 0
mirror of https://github.com/gocsaf/csaf.git synced 2025-12-22 05:40:11 +01:00
Code Issues Releases Activity Actions

Merge branch 'main' into sha-handling

Browse source
This commit is contained in:
koplas 2024-11-27 12:39:14 +01:00
commit a5f4b10c4e
No known key found for this signature in database
67 changed files with 236 additions and 215 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/Development.md
View file

@ -3,7 +3,7 @@
## Supported Go versions
We support the latest version and the one before
the latest version of Go (currently 1.22 and 1.21).
the latest version of Go (currently 1.22 and 1.23).
## Generated files

18
docs/csaf_aggregator.md
View file

@ -16,6 +16,7 @@ Help Options:
```
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/aggregator.toml
~/.csaf_aggregator.toml
@ -25,6 +26,7 @@ csaf_aggregator.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Usage example for a single run, to test if the config is good:
```bash
./csaf_aggregator -c docs/examples/aggregator.toml
```
@ -62,7 +64,6 @@ SHELL=/bin/bash
30 0-23 * * * $HOME/bin/csaf_aggregator --config /etc/csaf_aggregator.toml --interim >> /var/log/csaf_aggregator/interim.log 2>&1
```
#### serve via web server
Serve the paths where the aggregator writes its `html/` output
@ -78,7 +79,6 @@ a template. For the aggregator the difference is that you can leave out
the cgi-bin part, potentially commend out the TLS client parts and
adjust the `root` path accordingly.
### config options
The config file is written in [TOML](https://toml.io/en/v1.0.0).
@ -118,10 +118,12 @@ Next we have two TOML _tables_:
aggregator // basic infos for the aggregator object
remote_validator // config for optional remote validation checker
```
[See the provider config](csaf_provider.md#provider-options) about
how to configure `remote_validator`.
At last there is the TOML _array of tables_:
```
providers // each entry to be mirrored or listed
```
@ -148,6 +150,9 @@ header
Where valid `name` and `domain` settings are required.
If no user agent is specified with `header = "user-agent:custom-agent/1.0"`
then the default agent in the form of `csaf_distribution/VERSION` is sent.
If you want an entry to be listed instead of mirrored
in a `aggregator.category == "aggregator"` instance,
set `category` to `lister` in the entry.
@ -165,15 +170,16 @@ To offer an easy way of assorting CSAF documents by criteria like
document category, languages or values of the branch category within
the product tree, ROLIE category values can be configured in `categories`.
This can either
be done using an array of strings taken literally or, by prepending `"expr:"`.
The latter is evaluated as JSONPath and the result will be added into the
be done using an array of strings taken literally or, by prepending `"expr:"`.
The latter is evaluated as JSONPath and the result will be added into the
categories document. For a more detailed explanation and examples,
[refer to the provider config](csaf_provider.md#provider-options).
#### Example config file
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/examples/aggregator.toml) -->
<!-- The below code snippet is automatically added from ../docs/examples/aggregator.toml -->
```toml
workers = 2
folder = "/var/csaf_aggregator"
@ -233,8 +239,8 @@ insecure = true
category = "lister"
# ignore_pattern = [".*white.*", ".*red.*"]
```
<!-- MARKDOWN-AUTO-DOCS:END -->
<!-- MARKDOWN-AUTO-DOCS:END -->
#### Publish others' advisories

13
docs/csaf_checker.md
View file

@ -30,9 +30,12 @@ Help Options:
Will check all given _domains_, by trying each as a CSAF provider.
If no user agent is specified with `--header=user-agent:custom-agent/1.0` then the default agent in the form of `csaf_distribution/VERSION` is sent.
If a _domain_ starts with `https://` it is instead considered a direct URL to the `provider-metadata.json` and checking proceeds from there.
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/checker.toml
~/.csaf_checker.toml
@ -41,6 +44,7 @@ csaf_checker.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Supported options in config files:
```
output = ""
format = "json"
@ -58,9 +62,10 @@ validator_preset = ["mandatory"]
```
Usage example:
` ./csaf_checker example.com -f html --rate=5.3 -H apikey:SECRET -o check-results.html`
`./csaf_checker example.com -f html --rate=5.3 -H apikey:SECRET -o check-results.html`
Each performed check has a return type of either 0,1 or 2:
```
type 0: success
type 1: warning
@ -70,16 +75,16 @@ type 2: error
The checker result is a success if no checks resulted in type 2, and a failure otherwise.
The option `timerange` allows to only check advisories from a given time
interval. It can only be given once. See the
interval. It can only be given once. See the
[downloader documentation](csaf_downloader.md#timerange-option) for details.
You can ignore certain advisories while checking by specifying a list
of regular expressions[^1] to match their URLs by using the `ignorepattern`
option.
E.g. `-i='.*white.*' -i='*.red.*'` will ignore files which URLs contain
the sub strings **white** or **red**.
In the config file this has to be noted as:
```
ignorepattern = [".*white.*", ".*red.*"]
```
@ -88,7 +93,7 @@ ignorepattern = [".*white.*", ".*red.*"]
The `role` given in the `provider-metadata.json` is not
yet considered to change the overall result,
see https://github.com/csaf-poc/csaf_distribution/issues/221 .
see <https://github.com/gocsaf/csaf/issues/221> .
If a provider hosts one or more advisories with a TLP level of AMBER or RED, then these advisories must be access protected.
To check these advisories, authorization can be given via custom headers or certificates.

20
docs/csaf_downloader.md
View file

@ -1,4 +1,5 @@
## csaf_downloader
A tool to download CSAF documents from CSAF providers.
### Usage
@ -21,6 +22,7 @@ Application Options:
-f, --folder=FOLDER Download into a given subFOLDER
-i, --ignore_pattern=PATTERN Do not download files if their URLs match any of the given PATTERNs
-H, --header= One or more extra HTTP header fields
--enumerate_pmd_only If this flag is set to true, the downloader will only enumerate valid provider metadata files, but not download documents
--validator=URL URL to validate documents remotely
--validator_cache=FILE FILE to cache remote validations
--validator_preset=PRESETS One or more PRESETS to validate remotely (default: [mandatory])
@ -29,8 +31,8 @@ Application Options:
--forward_header= One or more extra HTTP header fields used by forwarding
--forward_queue=LENGTH Maximal queue LENGTH before forwarder (default: 5)
--forward_insecure Do not check TLS certificates from forward endpoint
--logfile=FILE FILE to log downloading to (default: downloader.log)
--loglevel=LEVEL[debug|info|warn|error] LEVEL of logging details (default: info)
--log_file=FILE FILE to log downloading to (default: downloader.log)
--log_level=LEVEL[debug|info|warn|error] LEVEL of logging details (default: info)
-c, --config=TOML-FILE Path to config TOML file
--preferred_hash=HASH[sha256|sha512] HASH to prefer
@ -40,6 +42,8 @@ Help Options:
Will download all CSAF documents for the given _domains_, by trying each as a CSAF provider.
If no user agent is specified with `--header=user-agent:custom-agent/1.0` then the default agent in the form of `csaf_distribution/VERSION` is sent.
If a _domain_ starts with `https://` it is instead considered a direct URL to the `provider-metadata.json` and downloading procedes from there.
Increasing the number of workers opens more connections to the web servers
@ -48,6 +52,7 @@ However, since this also increases the load on the servers, their administrators
have taken countermeasures to limit this.
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/downloader.toml
~/.csaf_downloader.toml
@ -57,6 +62,7 @@ csaf_downloader.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Supported options in config files:
```
# directory # not set by default
insecure = false
@ -91,6 +97,7 @@ option.
E.g. `-i='.*white.*' -i='*.red.*'` will ignore files which URLs contain
the sub strings **white** or **red**.
In the config file this has to be noted as:
```
ignorepattern = [".*white.*", ".*red.*"]
```
@ -107,16 +114,18 @@ into a given intervall. There are three possible notations:
and 'y' for years are recognized. In these cases only integer
values are accepted without any fractions.
Some examples:
- `"3h"` means downloading the advisories that have changed in the last three hours.
- `"30m"` .. changed within the last thirty minutes.
- `"30m"` .. changed within the last thirty minutes.
- `"3M2m"` .. changed within the last three months and two minutes.
- `"2y"` .. changed within the last two years.
- `"2y"` .. changed within the last two years.
2. Absolute. If the given string is an RFC 3339 date timestamp
the time interval between this date and now is used.
E.g. `"2006-01-02"` means that all files between 2006 January 2nd and now going to being
downloaded.
Accepted patterns are:
- `"2006-01-02T15:04:05Z"`
- `"2006-01-02T15:04:05+07:00"`
- `"2006-01-02T15:04:05-07:00"`
@ -135,6 +144,7 @@ into a given intervall. There are three possible notations:
All interval boundaries are inclusive.
#### Forwarding
The downloader is able to forward downloaded advisories and their checksums,
OpenPGP signatures and validation results to an HTTP endpoint.
The details of the implemented API are described [here](https://github.com/mfd2007/csaf_upload_interface).
@ -150,7 +160,7 @@ key protection mechanism based on RFC 1423, see
Thus it considered experimental and most likely to be removed
in a future release. Please only use this option, if you fully understand
the security implications!
Note that for fully automated processes, it usually not make sense
Note that for fully automated processes, it usually does not make sense
to protect the client certificate's private key with a passphrase.
Because the passphrase has to be accessible to the process anyway to run
unattented. In this situation the processing environment should be secured

6
docs/csaf_provider.md
View file

@ -4,7 +4,7 @@ The [setup docs](../README.md#setup-trusted-provider)
explain how to wire this up with nginx and where the config file lives.
When installed, two endpoints are offered,
and you should use the [csaf_uploader](../docs/csaf_uploader)
and you should use the [csaf_uploader](../docs/csaf_uploader.md)
to access them:
### /api/create
@ -141,5 +141,5 @@ contact_details = "Example Company can be reached at contact_us@example.com, or
There is an experimental upload interface which works with a web browser.
It is disabled by default, as there are known issues, notably:
* https://github.com/csaf-poc/csaf_distribution/issues/43
* https://github.com/csaf-poc/csaf_distribution/issues/256
* https://github.com/gocsaf/csaf/issues/43
* https://github.com/gocsaf/csaf/issues/256

2
docs/development-ca.md
View file

@ -55,7 +55,7 @@ signing_key
encryption_key
non_repudiation
dns_name = "*.local"
dns_name = "*.test"
dns_name = "localhost"
serial = 010

2
docs/provider-setup.md
View file

@ -115,7 +115,7 @@ sudo chmod g+r,o-rwx /etc/csaf/config.toml
Here is a minimal example configuration,
which you need to customize for a production setup,
see the [options of `csaf_provider`](https://github.com/csaf-poc/csaf_distribution/blob/main/docs/csaf_provider.md).
see the [options of `csaf_provider`](https://github.com/gocsaf/csaf/blob/main/docs/csaf_provider.md).
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/scripts/setupProviderForITest.sh&lines=94-101) -->
<!-- The below code snippet is automatically added from ../docs/scripts/setupProviderForITest.sh -->

12
docs/scripts/Readme.md
View file

@ -1,7 +1,7 @@
Scripts for assisting the Integration tests.
They were written on Ubuntu 20.04 TLS amd64 and also tested with 24.04 TLS.
- `prepareUbuntuInstanceForITests.sh` installs the required packages for the csaf_distribution integration tests on a naked ubuntu LTS amd64.
- `prepareUbuntuInstanceForITests.sh` installs the required packages for the csaf integration tests on a naked Ubuntu LTS amd64.
- `TLSConfigsForITest.sh` generates a root CA and webserver cert by running `createRootCAForITest.sh` and `createWebserverCertForITest.sh`
and configures nginx for serving TLS connections.
@ -12,13 +12,13 @@ and configures nginx for serving TLS connections.
As creating the folders needs to authenticate with the csaf_provider, the configurations of TLS server and Client certificate authentication should be set. So it is recommended to call the scripts in this order: `TLSConfigsForITest.sh`, `TLSClientConfigsForITest.sh`, `setupProviderForITest.sh`
Calling example (as root):
Calling example (as user with sudo privileges):
``` bash
curl --fail -O https://raw.githubusercontent.com/csaf-poc/csaf_distribution/main/docs/scripts/prepareUbuntuInstanceForITests.sh
bash prepareUbuntuInstanceForITests.sh
curl --fail -O https://raw.githubusercontent.com/gocsaf/csaf/main/docs/scripts/prepareUbuntuInstanceForITests.sh
sudo bash prepareUbuntuInstanceForITests.sh
git clone https://github.com/csaf-poc/csaf_distribution.git # --branch <name>
pushd csaf_distribution/docs/scripts/
git clone https://github.com/gocsaf/csaf.git # --branch <name>
pushd csaf/docs/scripts/
export FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)"
source ./TLSConfigsForITest.sh

2
docs/scripts/TLSClientConfigsForITest.sh
View file

@ -18,7 +18,7 @@ set -e
NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
cd ~/csaf_distribution/docs/scripts/
cd ~/csaf/docs/scripts/
source ./createCCForITest.sh
echo '

2
docs/scripts/TLSConfigsForITest.sh
View file

@ -17,7 +17,7 @@ set -e
NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
cd ~/csaf_distribution/docs/scripts/
cd ~/csaf/docs/scripts/
## Create Root CA
./createRootCAForITest.sh

4
docs/scripts/prepareUbuntuInstanceForITests.sh
View file

@ -2,11 +2,11 @@
set -e
# This script prepares a naked Ubuntu LTS amd64
# for the csaf_distribution integration tests
# for the csaf integration tests
# by installing the required packages.
apt update
apt install -y make bash curl gnupg sed tar git nginx fcgiwrap gnutls-bin
apt install -y make bash curl gnupg sed tar git nginx fcgiwrap gnutls-bin zip
# Install Go from binary distribution
latest_go="$(curl https://go.dev/VERSION\?m=text| head -1).linux-amd64.tar.gz"

2
docs/scripts/setupProviderForITest.sh
View file

@ -17,7 +17,7 @@ sudo chgrp -R www-data /var/www
sudo chmod -R g+ws /var/www
export NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
export DNS_NAME=csaf.data.security.localhost
export DNS_NAME=csaf.data.security.test
sudo cp /usr/share/doc/fcgiwrap/examples/nginx.conf /etc/nginx/fcgiwrap.conf

2
docs/scripts/setupValidationService.sh
View file

@ -21,7 +21,7 @@ echo '
remote_validator= { "url" = "http://localhost:8082", "presets" = ["mandatory"], "cache" = "/var/lib/csaf/validations.db" }
' | sudo tee --append /etc/csaf/config.toml
npm install pm2 -g
sudo npm install pm2 -g
pushd ~
git clone https://github.com/secvisogram/csaf-validator-service.git

2
docs/scripts/testAggregator.sh
View file

@ -29,6 +29,6 @@ popd
echo
echo '=== run aggregator'
cd ~/csaf_distribution/
cd ~/csaf/
sudo cp docs/examples/aggregator.toml /etc/csaf
sudo ./bin-linux-amd64/csaf_aggregator -c /etc/csaf/aggregator.toml

2
docs/scripts/testChecker.sh
View file

@ -11,7 +11,7 @@
set -e # to exit if a command in the script fails
echo '==== run checker (twice)'
cd ~/csaf_distribution
cd ~/csaf
./bin-linux-amd64/csaf_checker -f html -o ../checker-results.html --insecure \
--client_cert ~/devca1/testclient1.crt \

2
docs/scripts/testDownloader.sh
View file

@ -10,7 +10,7 @@
set -e # to exit if a command in the script fails
cd ~/csaf_distribution
cd ~/csaf
echo
echo '==== run downloader (1)'