SystemSh0cker/gocsaf
1
0
Fork 0
mirror of https://github.com/gocsaf/csaf.git synced 2025-12-22 05:40:11 +01:00
Code Issues Releases Activity Actions

Merge branch 'main' into emulate-validator-service-uris
Some checks failed
Go / build (push) Has been cancelled
Details

Browse source
This commit is contained in:
Bernhard Reiter 2025-06-12 11:45:32 +02:00
commit 143cef1227
No known key found for this signature in database
GPG key ID: 2B7BA3BF9BC3A554
171 changed files with 6486 additions and 1393 deletions
Download patch file Download diff file Expand all files Collapse all files

11
.github/workflows/go.yml vendored
View file

@ -12,12 +12,12 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v2
uses: actions/setup-go@v4
with:
go-version: 1.21.0
go-version: "stable"
- name: Build
run: go build -v ./cmd/...
@ -30,11 +30,8 @@ jobs:
with:
gofmt-flags: "-l -d"
- name: golint
uses: Jerome1337/golint-action@v1.0.2
- name: Revive Action
uses: morphy2k/revive-action@v2.5.1
uses: morphy2k/revive-action@v2.7.4
- name: Tests
run: go test -v ./...

26
.github/workflows/go_legacy.yml vendored Normal file
View file

@ -0,0 +1,26 @@
name: Go
on:
push:
paths:
- "**.go"
pull_request:
paths:
- "**.go"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 'oldstable'
- name: Build
run: go build -v ./cmd/...
- name: Tests
run: go test -v ./...

10
.github/workflows/itest.yml vendored
View file

@ -7,9 +7,9 @@ jobs:
steps:
- name: Set up Go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: 1.21.0
go-version: '^1.23.6'
- name: Set up Node.js
uses: actions/setup-node@v3
@ -17,7 +17,7 @@ jobs:
node-version: 16
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute the scripts
run: |
@ -25,7 +25,7 @@ jobs:
sudo apt install -y make nginx fcgiwrap gnutls-bin
cp -r $GITHUB_WORKSPACE ~
cd ~
cd csaf_distribution/docs/scripts/
cd csaf/docs/scripts/
# keep in sync with docs/scripts/Readme.md
export FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)"
source ./TLSConfigsForITest.sh
@ -36,7 +36,7 @@ jobs:
shell: bash
- name: Upload test results
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: checker-results
path: |

16
.github/workflows/release.yml vendored
View file

@ -7,22 +7,24 @@ on:
jobs:
releases-matrix:
name: Release Go binaries
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
permissions:
contents: write
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: '^1.21.0'
go-version: '^1.23.6'
- name: Build
run: make dist
- name: Upload release assets
uses: softprops/action-gh-release@v1
uses: softprops/action-gh-release@v2
with:
files: |
dist/csaf_distribution-*.zip
dist/csaf_distribution-*.tar.gz
dist/csaf-*.zip
dist/csaf-*.tar.gz

73
LICENSE-Apache-2.0.txt Normal file
View file

@ -0,0 +1,73 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

73
LICENSES/Apache-2.0.txt Normal file
View file

@ -0,0 +1,73 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

11
LICENSES/BSD-3-Clause.txt Normal file
View file

@ -0,0 +1,11 @@
Copyright (c) <year> <owner>.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

51
LICENSES/LicenseRef-Go119-BSD-Patentgrant.txt
View file

@ -1,51 +0,0 @@
Copyright (c) 2009 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Additional IP Rights Grant (Patents)
"This implementation" means the copyrightable works distributed by
Google as part of the Go project.
Google hereby grants to You a perpetual, worldwide, non-exclusive,
no-charge, royalty-free, irrevocable (except as stated in this section)
patent license to make, have made, use, offer to sell, sell, import,
transfer and otherwise run, modify and propagate the contents of this
implementation of Go, where such license applies only to those patent
claims, both currently owned or controlled by Google and acquired in
the future, licensable by Google that are necessarily infringed by this
implementation of Go. This grant does not include claims that would be
infringed only as a consequence of further modification of this
implementation. If you or your agent or exclusive licensee institute or
order or agree to the institution of patent litigation against any
entity (including a cross-claim or counterclaim in a lawsuit) alleging
that this implementation of Go or any code incorporated within this
implementation of Go constitutes direct or contributory patent
infringement, or inducement of patent infringement, then any patent
rights granted to you under this License for this implementation of Go
shall terminate as of the date such litigation is filed.

22
Makefile
View file

@ -1,12 +1,12 @@
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
#
# Makefile to build csaf_distribution components
# Makefile to build csaf components
SHELL = /bin/bash
BUILD = go build
@ -41,7 +41,7 @@ tag_checked_out:
# into a semver version. For this we increase the PATCH number, so that
# any commit after a tag is considered newer than the semver from the tag
# without an optional 'v'
# Note we need `--tags` because github release only creates lightweight tags
# Note we need `--tags` because github releases only create lightweight tags
# (see feature request https://github.com/github/feedback/discussions/4924).
# We use `--always` in case of being run as github action with shallow clone.
# In this case we might in some situations see an error like
@ -50,16 +50,16 @@ tag_checked_out:
GITDESC := $(shell git describe --tags --always)
GITDESCPATCH := $(shell echo '$(GITDESC)' | sed -E 's/v?[0-9]+\.[0-9]+\.([0-9]+)[-+]?.*/\1/')
SEMVERPATCH := $(shell echo $$(( $(GITDESCPATCH) + 1 )))
# Hint: The regexp in the next line only matches if there is a hyphen (`-`)
# followed by a number, by which we assume that git describe
# has added a string after the tag
SEMVER := $(shell echo '$(GITDESC)' | sed -E 's/v?([0-9]+\.[0-9]+\.)([0-9]+)(-[1-9].*)/\1$(SEMVERPATCH)\3/' )
# Hint: The second regexp in the next line only matches
# if there is a hyphen (`-`) followed by a number,
# by which we assume that git describe has added a string after the tag
SEMVER := $(shell echo '$(GITDESC)' | sed -E -e 's/^v//' -e 's/([0-9]+\.[0-9]+\.)([0-9]+)(-[1-9].*)/\1$(SEMVERPATCH)\3/' )
testsemver:
@echo from \'$(GITDESC)\' transformed to \'$(SEMVER)\'
# Set -ldflags parameter to pass the semversion.
LDFLAGS = -ldflags "-X github.com/csaf-poc/csaf_distribution/v3/util.SemVersion=$(SEMVER)"
LDFLAGS = -ldflags "-X github.com/gocsaf/csaf/v3/util.SemVersion=$(SEMVER)"
# Build binaries and place them under bin-$(GOOS)-$(GOARCH)
# Using 'Target-specific Variable Values' to specify the build target system
@ -78,7 +78,7 @@ build_linux build_win build_mac_amd64 build_mac_arm64:
env GOARCH=$(GOARCH) GOOS=$(GOOS) $(BUILD) -o $(BINDIR) $(LDFLAGS) -v ./cmd/...
DISTDIR := csaf_distribution-$(SEMVER)
DISTDIR := csaf-$(SEMVER)
dist: build_linux build_win build_mac_amd64 build_mac_arm64
mkdir -p dist
mkdir -p dist/$(DISTDIR)-windows-amd64/bin-windows-amd64

37
README.md
View file

@ -1,7 +1,28 @@
# csaf_distribution
<!--
This file is Free Software under the Apache-2.0 License
without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
An implementation of a
[CSAF 2.0](https://docs.oasis-open.org/csaf/csaf/v2.0/csaf-v2.0.html)
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: 2024 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
Software-Engineering: 2024 Intevation GmbH <https://intevation.de>
-->
> [!IMPORTANT]
> To avoid future breakage, if you still use `csaf-poc`:
> 1. Adjust your HTML links.
> 2. Adjust your go module paths, see [#579](https://github.com/gocsaf/csaf/issues/579#issuecomment-2497244379).
>
> (This repository was moved here on 2024-10-28. The old one is deprecated
> and redirection will be switched off a few months later.)
# csaf
Implements a [CSAF](https://oasis-open.github.io/csaf-documentation/)
([specification v2.0](https://docs.oasis-open.org/csaf/csaf/v2.0/os/csaf-v2.0-os.html)
and its [errata](https://docs.oasis-open.org/csaf/csaf/v2.0/csaf-v2.0.html))
trusted provider, checker, aggregator and downloader.
Includes an uploader command line tool for the trusted provider.
@ -31,10 +52,10 @@ is a CSAF Aggregator, to list or mirror providers.
## Other stuff
### [examples](./examples/README.md)
are small examples of how to use `github.com/csaf-poc/csaf_distribution`
are small examples of how to use `github.com/gocsaf/csaf`
as an API. Currently this is a work in progress, as usage of this repository
as a library to access is _not officially supported_, e.g.
see https://github.com/csaf-poc/csaf_distribution/issues/367 .
see https://github.com/gocsaf/csaf/issues/367 .
## Setup
Binaries for the server side are only available and tested
@ -58,9 +79,9 @@ Download the binaries from the most recent release assets on Github.
### Build from sources
- A recent version of **Go** (1.20+) should be installed. [Go installation](https://go.dev/doc/install)
- A recent version of **Go** (1.23+) should be installed. [Go installation](https://go.dev/doc/install)
- Clone the repository `git clone https://github.com/csaf-poc/csaf_distribution.git `
- Clone the repository `git clone https://github.com/gocsaf/csaf.git `
- Build Go components Makefile supplies the following targets:
- Build for GNU/Linux system: `make build_linux`
@ -89,7 +110,7 @@ For further details of the development process consult our [development page](./
## License
- `csaf_distribution` is licensed as Free Software under MIT License.
- `csaf` is licensed as Free Software under the terms of the [Apache License, Version 2.0](./LICENSES/Apache-2.0.txt).
- See the specific source files
for details, the license itself can be found in the directory `LICENSES/`.

18
cmd/csaf_aggregator/client.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -10,23 +10,25 @@ package main
import (
"errors"
"fmt"
"io"
"net/http"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
var errNotFound = errors.New("not found")
func downloadJSON(c util.Client, url string, found func(io.Reader) error) error {
res, err := c.Get(url)
if err != nil || res.StatusCode != http.StatusOK ||
if err != nil {
return fmt.Errorf("not found: %w", err)
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK ||
res.Header.Get("Content-Type") != "application/json" {
// ignore this as it is expected.
return errNotFound
}
return func() error {
defer res.Body.Close()
return found(res.Body)
}()
}

67
cmd/csaf_aggregator/client_test.go Normal file
View file

@ -0,0 +1,67 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
//
// SPDX-License-Identifier: MIT
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
package main
import (
"io"
"net/http"
"net/http/httptest"
"testing"
"github.com/gocsaf/csaf/v3/util"
)
func Test_downloadJSON(t *testing.T) {
tests := []struct {
name string
statusCode int
contentType string
wantErr error
}{
{
name: "status ok, application/json",
statusCode: http.StatusOK,
contentType: "application/json",
wantErr: nil,
},
{
name: "status found, application/json",
statusCode: http.StatusFound,
contentType: "application/json",
wantErr: errNotFound,
},
{
name: "status ok, application/xml",
statusCode: http.StatusOK,
contentType: "application/xml",
wantErr: errNotFound,
},
}
t.Parallel()
for _, testToRun := range tests {
test := testToRun
t.Run(test.name, func(tt *testing.T) {
tt.Parallel()
found := func(_ io.Reader) error {
return nil
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Add("Content-Type", test.contentType)
w.WriteHeader(test.statusCode)
}))
defer server.Close()
hClient := http.Client{}
client := util.Client(&hClient)
if gotErr := downloadJSON(client, server.URL, found); gotErr != test.wantErr {
t.Errorf("downloadJSON: Expected %q but got %q.", test.wantErr, gotErr)
}
})
}
}

60
cmd/csaf_aggregator/config.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -12,7 +12,7 @@ import (
"crypto/tls"
"errors"
"fmt"
"log"
"log/slog"
"net/http"
"os"
"runtime"
@ -20,12 +20,12 @@ import (
"time"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/internal/certs"
"github.com/csaf-poc/csaf_distribution/v3/internal/filter"
"github.com/csaf-poc/csaf_distribution/v3/internal/models"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/internal/certs"
"github.com/gocsaf/csaf/v3/internal/filter"
"github.com/gocsaf/csaf/v3/internal/models"
"github.com/gocsaf/csaf/v3/internal/options"
"github.com/gocsaf/csaf/v3/util"
"golang.org/x/time/rate"
)
@ -178,9 +178,11 @@ func (p *provider) ageAccept(c *config) func(time.Time) bool {
}
if c.Verbose {
log.Printf(
"Setting up filter to accept advisories within time range %s to %s\n",
r[0].Format(time.RFC3339), r[1].Format(time.RFC3339))
slog.Debug(
"Setting up filter to accept advisories within time range",
"from", r[0].Format(time.RFC3339),
"to", r[1].Format(time.RFC3339),
)
}
return r.Contains
}
@ -262,8 +264,14 @@ func (c *config) privateOpenPGPKey() (*crypto.Key, error) {
return c.key, c.keyErr
}
func (c *config) httpClient(p *provider) util.Client {
// httpLog does structured logging in a [util.LoggingClient].
func httpLog(method, url string) {
slog.Debug("http",
"method", method,
"url", url)
}
func (c *config) httpClient(p *provider) util.Client {
hClient := http.Client{}
var tlsConfig tls.Config
@ -282,6 +290,7 @@ func (c *config) httpClient(p *provider) util.Client {
hClient.Transport = &http.Transport{
TLSClientConfig: &tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
client := util.Client(&hClient)
@ -299,10 +308,18 @@ func (c *config) httpClient(p *provider) util.Client {
Client: client,
Header: c.ExtraHeader,
}
default:
client = &util.HeaderClient{
Client: client,
Header: http.Header{},
}
}
if c.Verbose {
client = &util.LoggingClient{Client: client}
client = &util.LoggingClient{
Client: client,
Log: httpLog,
}
}
if p.Rate == nil && c.Rate == nil {
@ -323,7 +340,6 @@ func (c *config) httpClient(p *provider) util.Client {
}
func (c *config) checkProviders() error {
if !c.AllowSingleProvider && len(c.Providers) < 2 {
return errors.New("need at least two providers")
}
@ -393,6 +409,17 @@ func (c *config) setDefaults() {
}
}
// prepareLogging sets up the structured logging.
func (c *config) prepareLogging() error {
ho := slog.HandlerOptions{
Level: slog.LevelDebug,
}
handler := slog.NewTextHandler(os.Stdout, &ho)
logger := slog.New(handler)
slog.SetDefault(logger)
return nil
}
// compileIgnorePatterns compiles the configured patterns to be ignored.
func (p *provider) compileIgnorePatterns() error {
pm, err := filter.NewPatternMatcher(p.IgnorePattern)
@ -452,7 +479,6 @@ func (c *config) prepareCertificates() error {
// prepare prepares internal state of a loaded configuration.
func (c *config) prepare() error {
if len(c.Providers) == 0 {
return errors.New("no providers given in configuration")
}

6
cmd/csaf_aggregator/files.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

48
cmd/csaf_aggregator/full.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -11,15 +11,15 @@ package main
import (
"errors"
"fmt"
"log"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
type fullJob struct {
@ -29,11 +29,13 @@ type fullJob struct {
err error
}
// setupProviderFull fetches the provider-metadate.json for a specific provider.
// setupProviderFull fetches the provider-metadata.json for a specific provider.
func (w *worker) setupProviderFull(provider *provider) error {
log.Printf("worker #%d: %s (%s)\n",
w.num, provider.Name, provider.Domain)
w.log.Info("Setting up provider",
"provider", slog.GroupValue(
slog.String("name", provider.Name),
slog.String("domain", provider.Domain),
))
w.dir = ""
w.provider = provider
@ -55,7 +57,7 @@ func (w *worker) setupProviderFull(provider *provider) error {
"provider-metadata.json has %d validation issues", len(errors))
}
log.Printf("provider-metadata: %s\n", w.loc)
w.log.Info("Using provider-metadata", "url", w.loc)
return nil
}
@ -79,7 +81,7 @@ func (w *worker) fullWork(wg *sync.WaitGroup, jobs <-chan *fullJob) {
func (p *processor) full() error {
if p.cfg.runAsMirror() {
log.Println("Running in aggregator mode")
p.log.Info("Running in aggregator mode")
// check if we need to setup a remote validator
if p.cfg.RemoteValidatorOptions != nil {
@ -96,16 +98,18 @@ func (p *processor) full() error {
}()
}
} else {
log.Println("Running in lister mode")
p.log.Info("Running in lister mode")
}
queue := make(chan *fullJob)
var wg sync.WaitGroup
log.Printf("Starting %d workers.\n", p.cfg.Workers)
p.log.Info("Starting workers...", "num", p.cfg.Workers)
for i := 1; i <= p.cfg.Workers; i++ {
wg.Add(1)
w := newWorker(i, p)
go w.fullWork(&wg, queue)
}
@ -135,12 +139,22 @@ func (p *processor) full() error {
for i := range jobs {
j := &jobs[i]
if j.err != nil {
log.Printf("error: '%s' failed: %v\n", j.provider.Name, j.err)
p.log.Error("Job execution failed",
slog.Group("job",
slog.Group("provider"),
"name", j.provider.Name,
),
"err", j.err,
)
continue
}
if j.aggregatorProvider == nil {
log.Printf(
"error: '%s' does not produce any result.\n", j.provider.Name)
p.log.Error("Job did not produce any result",
slog.Group("job",
slog.Group("provider"),
"name", j.provider.Name,
),
)
continue
}

77
cmd/csaf_aggregator/indices.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -12,7 +12,6 @@ import (
"bufio"
"encoding/csv"
"fmt"
"log"
"os"
"path/filepath"
"sort"
@ -20,8 +19,8 @@ import (
"strings"
"time"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
const (
@ -184,19 +183,26 @@ func (w *worker) writeROLIENoSummaries(label string) error {
fname := "csaf-feed-tlp-" + labelFolder + ".json"
feedURL := w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/" + labelFolder + "/" + fname
feedURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
feedURL = feedURL.JoinPath(labelFolder, fname)
links := []csaf.Link{{
Rel: "self",
HRef: feedURL,
HRef: feedURL.String(),
}}
if w.provider.serviceDocument(w.processor.cfg) {
serviceURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
serviceURL = serviceURL.JoinPath("service.json")
links = append(links, csaf.Link{
Rel: "service",
HRef: w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/service.json",
HRef: serviceURL.String(),
})
}
@ -224,8 +230,11 @@ func (w *worker) writeROLIE(label string, summaries []summary) error {
fname := "csaf-feed-tlp-" + labelFolder + ".json"
feedURL := w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/" + labelFolder + "/" + fname
feedURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
feedURL = feedURL.JoinPath(labelFolder, fname)
entries := make([]*csaf.Entry, len(summaries))
@ -237,10 +246,13 @@ func (w *worker) writeROLIE(label string, summaries []summary) error {
for i := range summaries {
s := &summaries[i]
csafURL := w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/" + label + "/" +
strconv.Itoa(s.summary.InitialReleaseDate.Year()) + "/" +
s.filename
csafURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
csafURLString := csafURL.JoinPath(label,
strconv.Itoa(s.summary.InitialReleaseDate.Year()),
s.filename).String()
entries[i] = &csaf.Entry{
ID: s.summary.ID,
@ -248,15 +260,15 @@ func (w *worker) writeROLIE(label string, summaries []summary) error {
Published: csaf.TimeStamp(s.summary.InitialReleaseDate),
Updated: csaf.TimeStamp(s.summary.CurrentReleaseDate),
Link: []csaf.Link{
{Rel: "self", HRef: csafURL},
{Rel: "hash", HRef: csafURL + ".sha256"},
{Rel: "hash", HRef: csafURL + ".sha512"},
{Rel: "signature", HRef: csafURL + ".asc"},
{Rel: "self", HRef: csafURLString},
{Rel: "hash", HRef: csafURLString + ".sha256"},
{Rel: "hash", HRef: csafURLString + ".sha512"},
{Rel: "signature", HRef: csafURLString + ".asc"},
},
Format: format,
Content: csaf.Content{
Type: "application/json",
Src: csafURL,
Src: csafURLString,
},
}
if s.summary.Summary != "" {
@ -268,14 +280,18 @@ func (w *worker) writeROLIE(label string, summaries []summary) error {
links := []csaf.Link{{
Rel: "self",
HRef: feedURL,
HRef: feedURL.String(),
}}
if w.provider.serviceDocument(w.processor.cfg) {
serviceURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
serviceURL = serviceURL.JoinPath("service.json")
links = append(links, csaf.Link{
Rel: "service",
HRef: w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/service.json",
HRef: serviceURL.String(),
})
}
@ -345,12 +361,15 @@ func (w *worker) writeService() error {
for _, ts := range labels {
feedName := "csaf-feed-tlp-" + ts + ".json"
href := w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" +
w.provider.Name + "/" + ts + "/" + feedName
hrefURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
hrefURL = hrefURL.JoinPath(ts, feedName)
collection := csaf.ROLIEServiceWorkspaceCollection{
Title: "CSAF feed (TLP:" + strings.ToUpper(ts) + ")",
HRef: href,
HRef: hrefURL.String(),
Categories: categories,
}
collections = append(collections, collection)
@ -377,7 +396,7 @@ func (w *worker) writeIndices() error {
}
for label, summaries := range w.summaries {
log.Printf("%s: %d\n", label, len(summaries))
w.log.Debug("Writing indices", "label", label, "summaries.num", len(summaries))
if err := w.writeInterims(label, summaries); err != nil {
return err
}

20
cmd/csaf_aggregator/interim.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -17,7 +17,6 @@ import (
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
@ -25,8 +24,8 @@ import (
"sync"
"time"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
type interimJob struct {
@ -102,12 +101,12 @@ func (w *worker) checkInterims(
// XXX: Should we return an error here?
for _, e := range errors {
log.Printf("validation error: %s: %v\n", url, e)
w.log.Error("validation error", "url", url, "err", e)
}
// We need to write the changed content.
// This will start the transcation if not already started.
// This will start the transaction if not already started.
dst, err := tx.Dst()
if err != nil {
return nil, err
@ -159,8 +158,7 @@ func (w *worker) checkInterims(
// setupProviderInterim prepares the worker for a specific provider.
func (w *worker) setupProviderInterim(provider *provider) {
log.Printf("worker #%d: %s (%s)\n",
w.num, provider.Name, provider.Domain)
w.log.Info("Setting up worker", provider.Name, provider.Domain)
w.dir = ""
w.provider = provider
@ -262,7 +260,7 @@ func (p *processor) interim() error {
queue := make(chan *interimJob)
var wg sync.WaitGroup
log.Printf("Starting %d workers.\n", p.cfg.Workers)
p.log.Info("Starting workers...", "num", p.cfg.Workers)
for i := 1; i <= p.cfg.Workers; i++ {
wg.Add(1)
w := newWorker(i, p)

13
cmd/csaf_aggregator/lazytransaction.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -9,11 +9,11 @@
package main
import (
"log"
"log/slog"
"os"
"path/filepath"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
type lazyTransaction struct {
@ -85,7 +85,8 @@ func (lt *lazyTransaction) commit() error {
os.RemoveAll(lt.dst)
return err
}
log.Printf("Move %q -> %q\n", symlink, lt.src)
slog.Debug("Moving directory", "from", symlink, "to", lt.src)
if err := os.Rename(symlink, lt.src); err != nil {
os.RemoveAll(lt.dst)
return err

10
cmd/csaf_aggregator/lister.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -11,8 +11,8 @@ package main
import (
"fmt"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// mirrorAllowed checks if mirroring is allowed.

19
cmd/csaf_aggregator/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -11,10 +11,12 @@ package main
import (
"fmt"
"log/slog"
"os"
"path/filepath"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/options"
"github.com/gofrs/flock"
)
@ -44,8 +46,9 @@ func lock(lockFile *string, fn func() error) error {
func main() {
_, cfg, err := parseArgsConfig()
options.ErrorCheck(err)
options.ErrorCheck(cfg.prepare())
p := processor{cfg: cfg}
options.ErrorCheck(lock(cfg.LockFile, p.process))
cfg.prepareLogging()
options.ErrorCheckStructured(err)
options.ErrorCheckStructured(cfg.prepare())
p := processor{cfg: cfg, log: slog.Default()}
options.ErrorCheckStructured(lock(cfg.LockFile, p.process))
}

92
cmd/csaf_aggregator/mirror.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -16,7 +16,7 @@ import (
"encoding/json"
"fmt"
"io"
"log"
"log/slog"
"net/http"
"net/url"
"os"
@ -30,8 +30,8 @@ import (
"github.com/ProtonMail/gopenpgp/v2/constants"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// mirrorAllowed checks if mirroring is allowed.
@ -47,7 +47,7 @@ func (w *worker) mirror() (*csaf.AggregatorCSAFProvider, error) {
if err != nil && w.dir != "" {
// If something goes wrong remove the debris.
if err := os.RemoveAll(w.dir); err != nil {
log.Printf("error: %v\n", err)
w.log.Error("Could not remove directory", "path", w.dir, "err", err)
}
}
return result, err
@ -103,9 +103,13 @@ func (w *worker) mirrorInternal() (*csaf.AggregatorCSAFProvider, error) {
}
// Add us as a mirror.
mirror, err := w.getProviderBaseURL()
if err != nil {
return nil, err
}
mirrorURL := csaf.ProviderURL(
fmt.Sprintf("%s/.well-known/csaf-aggregator/%s/provider-metadata.json",
w.processor.cfg.Domain, w.provider.Name))
mirror.JoinPath("provider-metadata.json").String(),
)
acp.Mirrors = []csaf.ProviderURL{
mirrorURL,
@ -128,8 +132,12 @@ func (w *worker) writeProviderMetadata() error {
fname := filepath.Join(w.dir, "provider-metadata.json")
prefixURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
pm := csaf.NewProviderMetadataPrefix(
w.processor.cfg.Domain+"/.well-known/csaf-aggregator/"+w.provider.Name,
prefixURL.String(),
w.labelsFromSummaries())
// Fill in directory URLs if needed.
@ -139,9 +147,8 @@ func (w *worker) writeProviderMetadata() error {
labels = append(labels, label)
}
sort.Strings(labels)
prefix := w.processor.cfg.Domain + "/.well-known/csaf-aggregator/" + w.provider.Name + "/"
for _, label := range labels {
pm.AddDirectoryDistribution(prefix + label)
pm.AddDirectoryDistribution(prefixURL.JoinPath(label).String())
}
}
@ -166,7 +173,7 @@ func (w *worker) writeProviderMetadata() error {
{Expr: `$.public_openpgp_keys`, Action: util.ReMarshalMatcher(&pm.PGPKeys)},
}, w.metadataProvider); err != nil {
// only log the errors
log.Printf("extracting data from orignal provider failed: %v\n", err)
w.log.Error("Extracting data from original provider failed", "err", err)
}
// We are mirroring the remote public keys, too.
@ -188,19 +195,22 @@ func (w *worker) mirrorPGPKeys(pm *csaf.ProviderMetadata) error {
return err
}
keyURL, err := w.getProviderBaseURL()
if err != nil {
return err
}
localKeyURL := func(fingerprint string) string {
return fmt.Sprintf("%s/.well-known/csaf-aggregator/%s/openpgp/%s.asc",
w.processor.cfg.Domain, w.provider.Name, fingerprint)
return keyURL.JoinPath("openpgp", (fingerprint + ".asc")).String()
}
for i := range pm.PGPKeys {
pgpKey := &pm.PGPKeys[i]
if pgpKey.URL == nil {
log.Printf("ignoring PGP key without URL: %s\n", pgpKey.Fingerprint)
w.log.Warn("Ignoring PGP key without URL", "fingerprint", pgpKey.Fingerprint)
continue
}
if _, err := hex.DecodeString(string(pgpKey.Fingerprint)); err != nil {
log.Printf("ignoring PGP with invalid fingerprint: %s\n", *pgpKey.URL)
w.log.Warn("Ignoring PGP key with invalid fingerprint", "url", *pgpKey.URL)
continue
}
@ -240,8 +250,8 @@ func (w *worker) mirrorPGPKeys(pm *csaf.ProviderMetadata) error {
}
// replace the URL
url := localKeyURL(fingerprint)
pgpKey.URL = &url
u := localKeyURL(fingerprint)
pgpKey.URL = &u
}
// If we have public key configured copy it into the new folder
@ -308,7 +318,7 @@ func (w *worker) createAggregatorProvider() (*csaf.AggregatorCSAFProvider, error
var (
lastUpdated = csaf.TimeStamp(lastUpdatedT)
role = csaf.MetadataRole(roleS)
url = csaf.ProviderURL(urlS)
providerURL = csaf.ProviderURL(urlS)
)
return &csaf.AggregatorCSAFProvider{
@ -316,7 +326,7 @@ func (w *worker) createAggregatorProvider() (*csaf.AggregatorCSAFProvider, error
LastUpdated: &lastUpdated,
Publisher: &pub,
Role: &role,
URL: &url,
URL: &providerURL,
},
}, nil
}
@ -344,7 +354,7 @@ func (w *worker) doMirrorTransaction() error {
// Check if there is a sysmlink already.
target := filepath.Join(w.processor.cfg.Folder, w.provider.Name)
log.Printf("target: '%s'\n", target)
w.log.Debug("Checking for path existance", "path", target)
exists, err := util.PathExists(target)
if err != nil {
@ -359,7 +369,7 @@ func (w *worker) doMirrorTransaction() error {
}
}
log.Printf("sym link: %s -> %s\n", w.dir, target)
w.log.Debug("Creating sym link", "from", w.dir, "to", target)
// Create a new symlink
if err := os.Symlink(w.dir, target); err != nil {
@ -368,7 +378,7 @@ func (w *worker) doMirrorTransaction() error {
}
// Move the symlink
log.Printf("Move: %s -> %s\n", target, webTarget)
w.log.Debug("Moving sym link", "from", target, "to", webTarget)
if err := os.Rename(target, webTarget); err != nil {
os.RemoveAll(w.dir)
return err
@ -462,8 +472,9 @@ func (w *worker) extractCategories(label string, advisory any) error {
expr := cat[len(exprPrefix):]
// Compile first to check that the expression is okay.
if _, err := w.expr.Compile(expr); err != nil {
fmt.Printf("Compiling category expression %q failed: %v\n",
expr, err)
slog.Error("Compiling category expression failed",
"expr", expr,
"err", err)
continue
}
// Ignore errors here as they result from not matching.
@ -499,14 +510,14 @@ func (w *worker) mirrorFiles(tlpLabel csaf.TLPLabel, files []csaf.AdvisoryFile)
u, err := url.Parse(file.URL())
if err != nil {
log.Printf("error: %s\n", err)
w.log.Error("Could not parse advisory file URL", "err", err)
continue
}
// Should we ignore this advisory?
if w.provider.ignoreURL(file.URL(), w.processor.cfg) {
if w.processor.cfg.Verbose {
log.Printf("Ignoring %s: %q\n", w.provider.Name, file.URL())
w.log.Info("Ignoring advisory", slog.Group("provider", "name", w.provider.Name), "file", file)
}
continue
}
@ -514,7 +525,7 @@ func (w *worker) mirrorFiles(tlpLabel csaf.TLPLabel, files []csaf.AdvisoryFile)
// Ignore not conforming filenames.
filename := filepath.Base(u.Path)
if !util.ConformingFileName(filename) {
log.Printf("Not conforming filename %q. Ignoring.\n", filename)
w.log.Warn("Ignoring advisory because of non-conforming filename", "filename", filename)
continue
}
@ -531,19 +542,18 @@ func (w *worker) mirrorFiles(tlpLabel csaf.TLPLabel, files []csaf.AdvisoryFile)
}
if err := downloadJSON(w.client, file.URL(), download); err != nil {
log.Printf("error: %v\n", err)
w.log.Error("Error while downloading JSON", "err", err)
continue
}
// Check against CSAF schema.
errors, err := csaf.ValidateCSAF(advisory)
if err != nil {
log.Printf("error: %s: %v", file, err)
w.log.Error("Error while validating CSAF schema", "err", err)
continue
}
if len(errors) > 0 {
log.Printf("CSAF file %s has %d validation errors.\n",
file, len(errors))
w.log.Error("CSAF file has validation errors", "num.errors", len(errors), "file", file)
continue
}
@ -551,29 +561,27 @@ func (w *worker) mirrorFiles(tlpLabel csaf.TLPLabel, files []csaf.AdvisoryFile)
if rmv := w.processor.remoteValidator; rmv != nil {
rvr, err := rmv.Validate(advisory)
if err != nil {
log.Printf("Calling remote validator failed: %s\n", err)
w.log.Error("Calling remote validator failed", "err", err)
continue
}
if !rvr.Valid {
log.Printf(
"CSAF file %s does not validate remotely.\n", file)
w.log.Error("CSAF file does not validate remotely", "file", file.URL())
continue
}
}
sum, err := csaf.NewAdvisorySummary(w.expr, advisory)
if err != nil {
log.Printf("error: %s: %v\n", file, err)
w.log.Error("Error while creating new advisory", "file", file, "err", err)
continue
}
if util.CleanFileName(sum.ID) != filename {
log.Printf("ID %q does not match filename %s",
sum.ID, filename)
w.log.Error("ID mismatch", "id", sum.ID, "filename", filename)
}
if err := w.extractCategories(label, advisory); err != nil {
log.Printf("error: %s: %v\n", file, err)
w.log.Error("Could not extract categories", "file", file, "err", err)
continue
}
@ -591,12 +599,10 @@ func (w *worker) mirrorFiles(tlpLabel csaf.TLPLabel, files []csaf.AdvisoryFile)
if err := os.MkdirAll(yearDir, 0755); err != nil {
return err
}
//log.Printf("created %s\n", yearDir)
yearDirs[year] = yearDir
}
fname := filepath.Join(yearDir, filename)
//log.Printf("write: %s\n", fname)
data := content.Bytes()
if err := writeFileHashes(
fname, filename,
@ -624,7 +630,7 @@ func (w *worker) downloadSignatureOrSign(url, fname string, data []byte) error {
if err != nil {
if err != errNotFound {
log.Printf("error: %s: %v\n", url, err)
w.log.Error("Could not find signature URL", "url", url, "err", err)
}
// Sign it our self.
if sig, err = w.sign(data); err != nil {

67
cmd/csaf_aggregator/processor.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -10,14 +10,15 @@ package main
import (
"fmt"
"log"
"log/slog"
"net/url"
"os"
"path/filepath"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/ProtonMail/gopenpgp/v2/crypto"
)
type processor struct {
@ -26,6 +27,9 @@ type processor struct {
// remoteValidator is a globally configured remote validator.
remoteValidator csaf.RemoteValidator
// log is the structured logger for the whole processor.
log *slog.Logger
}
type summary struct {
@ -48,6 +52,7 @@ type worker struct {
dir string // Directory to store data to.
summaries map[string][]summary // the summaries of the advisories.
categories map[string]util.Set[string] // the categories per label.
log *slog.Logger // the structured logger, supplied with the worker number.
}
func newWorker(num int, processor *processor) *worker {
@ -55,6 +60,7 @@ func newWorker(num int, processor *processor) *worker {
num: num,
processor: processor,
expr: util.NewPathEval(),
log: processor.log.With(slog.Int("worker", num)),
}
}
@ -84,16 +90,21 @@ func (w *worker) locateProviderMetadata(domain string) error {
lpmd := loader.Load(domain)
if w.processor.cfg.Verbose {
for i := range lpmd.Messages {
log.Printf(
"Loading provider-metadata.json of %q: %s\n",
domain, lpmd.Messages[i].Message)
}
}
if !lpmd.Valid() {
for i := range lpmd.Messages {
w.log.Error(
"Loading provider-metadata.json",
"domain", domain,
"message", lpmd.Messages[i].Message)
}
return fmt.Errorf("no valid provider-metadata.json found for '%s'", domain)
} else if w.processor.cfg.Verbose {
for i := range lpmd.Messages {
w.log.Debug(
"Loading provider-metadata.json",
"domain", domain,
"message", lpmd.Messages[i].Message)
}
}
w.metadataProvider = lpmd.Document
@ -102,6 +113,18 @@ func (w *worker) locateProviderMetadata(domain string) error {
return nil
}
// getProviderBaseURL returns the base URL for the provider.
func (w *worker) getProviderBaseURL() (*url.URL, error) {
baseURL, err := url.Parse(w.processor.cfg.Domain)
if err != nil {
return nil, err
}
baseURL = baseURL.JoinPath(".well-known",
"csaf-aggregator",
w.provider.Name)
return baseURL, nil
}
// removeOrphans removes the directories that are not in the providers list.
func (p *processor) removeOrphans() error {
@ -141,7 +164,7 @@ func (p *processor) removeOrphans() error {
fi, err := entry.Info()
if err != nil {
log.Printf("error: %v\n", err)
p.log.Error("Could not retrieve file info", "err", err)
continue
}
@ -153,13 +176,13 @@ func (p *processor) removeOrphans() error {
d := filepath.Join(path, entry.Name())
r, err := filepath.EvalSymlinks(d)
if err != nil {
log.Printf("error: %v\n", err)
p.log.Error("Could not evaluate symlink", "err", err)
continue
}
fd, err := os.Stat(r)
if err != nil {
log.Printf("error: %v\n", err)
p.log.Error("Could not retrieve file stats", "err", err)
continue
}
@ -169,18 +192,18 @@ func (p *processor) removeOrphans() error {
}
// Remove the link.
log.Printf("removing link %s -> %s\n", d, r)
p.log.Info("Removing link", "path", fmt.Sprintf("%s -> %s", d, r))
if err := os.Remove(d); err != nil {
log.Printf("error: %v\n", err)
p.log.Error("Could not remove symlink", "err", err)
continue
}
// Only remove directories which are in our folder.
if rel, err := filepath.Rel(prefix, r); err == nil &&
rel == filepath.Base(r) {
log.Printf("removing directory %s\n", r)
p.log.Info("Remove directory", "path", r)
if err := os.RemoveAll(r); err != nil {
log.Printf("error: %v\n", err)
p.log.Error("Could not remove directory", "err", err)
}
}
}

14
cmd/csaf_checker/config.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -13,10 +13,10 @@ import (
"fmt"
"net/http"
"github.com/csaf-poc/csaf_distribution/v3/internal/certs"
"github.com/csaf-poc/csaf_distribution/v3/internal/filter"
"github.com/csaf-poc/csaf_distribution/v3/internal/models"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/certs"
"github.com/gocsaf/csaf/v3/internal/filter"
"github.com/gocsaf/csaf/v3/internal/models"
"github.com/gocsaf/csaf/v3/internal/options"
)
type outputFormat string

8
cmd/csaf_checker/links.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -15,7 +15,7 @@ import (
"github.com/PuerkitoBio/goquery"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
type (

86
cmd/csaf_checker/links_test.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -10,8 +10,12 @@ package main
import (
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gocsaf/csaf/v3/util"
)
const page0 = `<html>
@ -31,7 +35,6 @@ const page0 = `<html>
</html>`
func TestLinksOnPage(t *testing.T) {
var links []string
err := linksOnPage(
@ -58,3 +61,78 @@ func TestLinksOnPage(t *testing.T) {
}
}
}
func Test_listed(t *testing.T) {
tests := []struct {
name string
badDirs util.Set[string]
path string
want bool
}{
{
name: "listed path",
badDirs: util.Set[string]{},
path: "/white/avendor-advisory-0004.json",
want: true,
},
{
name: "badDirs contains path",
badDirs: util.Set[string]{"/white/": {}},
path: "/white/avendor-advisory-0004.json",
want: false,
},
{
name: "not found",
badDirs: util.Set[string]{},
path: "/not-found/resource.json",
want: false,
},
{
name: "badDirs does not contain path",
badDirs: util.Set[string]{"/bad-dir/": {}},
path: "/white/avendor-advisory-0004.json",
want: true,
},
{
name: "unlisted path",
badDirs: util.Set[string]{},
path: "/white/avendor-advisory-0004-not-listed.json",
want: false,
},
}
t.Parallel()
for _, testToRun := range tests {
test := testToRun
t.Run(test.name, func(tt *testing.T) {
tt.Parallel()
serverURL := ""
fs := http.FileServer(http.Dir("../../testdata/simple-directory-provider"))
server := httptest.NewTLSServer(fs)
defer server.Close()
serverURL = server.URL
hClient := server.Client()
client := util.Client(hClient)
pgs := pages{}
cfg := config{RemoteValidator: "", RemoteValidatorCache: ""}
p, err := newProcessor(&cfg)
if err != nil {
t.Error(err)
}
p.client = client
badDirs := util.Set[string]{}
for dir := range test.badDirs {
badDirs.Add(serverURL + dir)
}
got, _ := pgs.listed(serverURL+test.path, p, badDirs)
if got != test.want {
t.Errorf("%q: Expected %t but got %t.", test.name, test.want, got)
}
})
}
}

8
cmd/csaf_checker/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -12,7 +12,7 @@ package main
import (
"log"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/options"
)
// run uses a processor to check all the given domains or direct urls

305
cmd/csaf_checker/processor.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -32,9 +32,8 @@ import (
"github.com/ProtonMail/gopenpgp/v2/crypto"
"golang.org/x/time/rate"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/internal/models"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// topicMessages stores the collected topicMessages for a specific topic.
@ -54,6 +53,8 @@ type processor struct {
pmd any
keys *crypto.KeyRing
labelChecker labelChecker
timesChanges map[string]time.Time
timesAdv map[string]time.Time
invalidAdvisories topicMessages
badFilenames topicMessages
@ -84,10 +85,8 @@ type reporter interface {
report(*processor, *Domain)
}
var (
// errContinue indicates that the current check should continue.
errContinue = errors.New("continue")
)
var errContinue = errors.New("continue")
type whereType byte
@ -139,7 +138,7 @@ func (m *topicMessages) info(format string, args ...any) {
m.add(InfoType, format, args...)
}
// use signals that we going to use this topic.
// use signals that we're going to use this topic.
func (m *topicMessages) use() {
if *m == nil {
*m = []Message{}
@ -165,9 +164,8 @@ func (m *topicMessages) hasErrors() bool {
return false
}
// newProcessor returns an initilaized processor.
// newProcessor returns an initialized processor.
func newProcessor(cfg *config) (*processor, error) {
var validator csaf.RemoteValidator
if cfg.RemoteValidator != "" {
@ -192,6 +190,9 @@ func newProcessor(cfg *config) (*processor, error) {
advisories: map[csaf.TLPLabel]util.Set[string]{},
whiteAdvisories: map[identifier]bool{},
},
timesAdv: map[string]time.Time{},
timesChanges: map[string]time.Time{},
noneTLS: util.Set[string]{},
}, nil
}
@ -203,17 +204,17 @@ func (p *processor) close() {
}
}
// clean clears the fields values of the given processor.
func (p *processor) clean() {
// reset clears the fields values of the given processor.
func (p *processor) reset() {
p.redirects = nil
p.noneTLS = nil
for k := range p.alreadyChecked {
delete(p.alreadyChecked, k)
}
p.pmdURL = ""
p.pmd256 = nil
p.pmd = nil
p.keys = nil
clear(p.alreadyChecked)
clear(p.noneTLS)
clear(p.timesAdv)
clear(p.timesChanges)
p.invalidAdvisories.reset()
p.badFilenames.reset()
@ -240,7 +241,6 @@ func (p *processor) clean() {
// Then it calls the report method on each report from the given "reporters" parameter for each domain.
// It returns a pointer to the report and nil, otherwise an error.
func (p *processor) run(domains []string) (*Report, error) {
report := Report{
Date: ReportTime{Time: time.Now().UTC()},
Version: util.SemVersion,
@ -248,6 +248,8 @@ func (p *processor) run(domains []string) (*Report, error) {
}
for _, d := range domains {
p.reset()
if !p.checkProviderMetadata(d) {
// We cannot build a report if the provider metadata cannot be parsed.
log.Printf("Could not parse the Provider-Metadata.json of: %s\n", d)
@ -288,7 +290,6 @@ func (p *processor) run(domains []string) (*Report, error) {
domain.Passed = rules.eval(p)
report.Domains = append(report.Domains, domain)
p.clean()
}
return &report, nil
@ -296,7 +297,6 @@ func (p *processor) run(domains []string) (*Report, error) {
// fillMeta fills the report with extra informations from provider metadata.
func (p *processor) fillMeta(domain *Domain) error {
if p.pmd == nil {
return nil
}
@ -322,7 +322,6 @@ func (p *processor) fillMeta(domain *Domain) error {
// domainChecks compiles a list of checks which should be performed
// for a given domain.
func (p *processor) domainChecks(domain string) []func(*processor, string) error {
// If we have a direct domain url we dont need to
// perform certain checks.
direct := strings.HasPrefix(domain, "https://")
@ -377,9 +376,6 @@ func (p *processor) checkDomain(domain string) error {
// checkTLS parses the given URL to check its schema, as a result it sets
// the value of "noneTLS" field if it is not HTTPS.
func (p *processor) checkTLS(u string) {
if p.noneTLS == nil {
p.noneTLS = util.Set[string]{}
}
if x, err := url.Parse(u); err == nil && x.Scheme != "https" {
p.noneTLS.Add(u)
}
@ -392,7 +388,6 @@ func (p *processor) markChecked(s string, mask whereType) bool {
}
func (p *processor) checkRedirect(r *http.Request, via []*http.Request) error {
url := r.URL.String()
p.checkTLS(url)
if p.redirects == nil {
@ -430,17 +425,16 @@ func (p *processor) fullClient() util.Client {
hClient.Transport = &http.Transport{
TLSClientConfig: &tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
client := util.Client(&hClient)
// Add extra headers.
if len(p.cfg.ExtraHeader) > 0 {
client = &util.HeaderClient{
Client: client,
Header: p.cfg.ExtraHeader,
}
}
// Add optional URL logging.
if p.cfg.Verbose {
@ -462,6 +456,7 @@ func (p *processor) basicClient() *http.Client {
if p.cfg.Insecure {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
Proxy: http.ProxyFromEnvironment,
}
return &http.Client{Transport: tr}
}
@ -494,7 +489,6 @@ func (p *processor) usedAuthorizedClient() bool {
// rolieFeedEntries loads the references to the advisory files for a given feed.
func (p *processor) rolieFeedEntries(feed string) ([]csaf.AdvisoryFile, error) {
client := p.httpClient()
res, err := client.Get(feed)
p.badDirListings.use()
@ -545,10 +539,9 @@ func (p *processor) rolieFeedEntries(feed string) ([]csaf.AdvisoryFile, error) {
var files []csaf.AdvisoryFile
rfeed.Entries(func(entry *csaf.Entry) {
// Filter if we have date checking.
if accept := p.cfg.Range; accept != nil {
if pub := time.Time(entry.Published); !pub.IsZero() && !accept.Contains(pub) {
if t := time.Time(entry.Updated); !t.IsZero() && !accept.Contains(t) {
return
}
}
@ -594,11 +587,17 @@ func (p *processor) rolieFeedEntries(feed string) ([]csaf.AdvisoryFile, error) {
var file csaf.AdvisoryFile
if sha256 != "" || sha512 != "" || sign != "" {
file = csaf.HashedAdvisoryFile{url, sha256, sha512, sign}
} else {
file = csaf.PlainAdvisoryFile(url)
switch {
case sha256 == "" && sha512 != "":
p.badROLIEFeed.info("%s has no sha256 hash file listed", url)
case sha256 != "" && sha512 == "":
p.badROLIEFeed.info("%s has no sha512 hash file listed", url)
case sha256 == "" && sha512 == "":
p.badROLIEFeed.error("No hash listed on ROLIE feed %s", url)
case sign == "":
p.badROLIEFeed.error("No signature listed on ROLIE feed %s", url)
}
file = csaf.PlainAdvisoryFile{Path: url, SHA256: sha256, SHA512: sha512, Sign: sign}
files = append(files, file)
})
@ -620,6 +619,8 @@ func makeAbsolute(base *url.URL) func(*url.URL) *url.URL {
var yearFromURL = regexp.MustCompile(`.*/(\d{4})/[^/]+$`)
// integrity checks several csaf.AdvisoryFiles for formal
// mistakes, from conforming filenames to invalid advisories.
func (p *processor) integrity(
files []csaf.AdvisoryFile,
base string,
@ -667,11 +668,6 @@ func (p *processor) integrity(
var folderYear *int
if m := yearFromURL.FindStringSubmatch(u); m != nil {
year, _ := strconv.Atoi(m[1])
// Check if the year is in the accepted time interval.
if accept := p.cfg.Range; accept != nil &&
!accept.Intersects(models.Year(year)) {
continue
}
folderYear = &year
}
@ -686,9 +682,9 @@ func (p *processor) integrity(
continue
}
// Warn if we do not get JSON.
// Error if we do not get JSON.
if ct := res.Header.Get("Content-Type"); ct != "application/json" {
lg(WarnType,
lg(ErrorType,
"The content type of %s should be 'application/json' but is '%s'",
u, ct)
}
@ -740,32 +736,41 @@ func (p *processor) integrity(
// Check if file is in the right folder.
p.badFolders.use()
if date, err := p.expr.Eval(
`$.document.tracking.initial_release_date`, doc); err != nil {
p.badFolders.error(
"Extracting 'initial_release_date' from %s failed: %v", u, err)
} else if text, ok := date.(string); !ok {
p.badFolders.error("'initial_release_date' is not a string in %s", u)
} else if d, err := time.Parse(time.RFC3339, text); err != nil {
p.badFolders.error(
"Parsing 'initial_release_date' as RFC3339 failed in %s: %v", u, err)
} else if folderYear == nil {
switch date, fault := p.extractTime(doc, `initial_release_date`, u); {
case fault != "":
p.badFolders.error(fault)
case folderYear == nil:
p.badFolders.error("No year folder found in %s", u)
} else if d.UTC().Year() != *folderYear {
p.badFolders.error("%s should be in folder %d", u, d.UTC().Year())
case date.UTC().Year() != *folderYear:
p.badFolders.error("%s should be in folder %d", u, date.UTC().Year())
}
current, fault := p.extractTime(doc, `current_release_date`, u)
if fault != "" {
p.badChanges.error(fault)
} else {
p.timesAdv[f.URL()] = current
}
// Check hashes
p.badIntegrities.use()
for _, x := range []struct {
type hash struct {
ext string
url func() string
hash []byte
}{
{"SHA256", f.SHA256URL, s256.Sum(nil)},
{"SHA512", f.SHA512URL, s512.Sum(nil)},
} {
}
hashes := []hash{}
if f.SHA256URL() != "" {
hashes = append(hashes, hash{"SHA256", f.SHA256URL, s256.Sum(nil)})
}
if f.SHA512URL() != "" {
hashes = append(hashes, hash{"SHA512", f.SHA512URL, s512.Sum(nil)})
}
couldFetchHash := false
hashFetchErrors := []string{}
for _, x := range hashes {
hu, err := url.Parse(x.url())
if err != nil {
lg(ErrorType, "Bad URL %s: %v", x.url(), err)
@ -776,14 +781,15 @@ func (p *processor) integrity(
p.checkTLS(hashFile)
if res, err = client.Get(hashFile); err != nil {
p.badIntegrities.error("Fetching %s failed: %v.", hashFile, err)
hashFetchErrors = append(hashFetchErrors, fmt.Sprintf("Fetching %s failed: %v.", hashFile, err))
continue
}
if res.StatusCode != http.StatusOK {
p.badIntegrities.error("Fetching %s failed: Status code %d (%s)",
hashFile, res.StatusCode, res.Status)
hashFetchErrors = append(hashFetchErrors, fmt.Sprintf("Fetching %s failed: Status code %d (%s)",
hashFile, res.StatusCode, res.Status))
continue
}
couldFetchHash = true
h, err := func() ([]byte, error) {
defer res.Body.Close()
return util.HashFromReader(res.Body)
@ -801,6 +807,19 @@ func (p *processor) integrity(
x.ext, u, hashFile)
}
}
msgType := ErrorType
// Log only as warning, if the other hash could be fetched
if couldFetchHash {
msgType = WarnType
}
if f.IsDirectory() {
msgType = InfoType
}
for _, fetchError := range hashFetchErrors {
p.badIntegrities.add(msgType, fetchError)
}
// Check signature
su, err := url.Parse(f.SignURL())
if err != nil {
@ -846,9 +865,48 @@ func (p *processor) integrity(
}
}
// If we tested an existing changes.csv
if len(p.timesAdv) > 0 && p.badChanges.used() {
// Iterate over all files again
for _, f := range files {
// If there was no previous error when extracting times from advisories and we have a valid time
if timeAdv, ok := p.timesAdv[f.URL()]; ok {
// If there was no previous error when extracting times from changes and the file was listed in changes.csv
if timeCha, ok := p.timesChanges[f.URL()]; ok {
// check if the time matches
if !timeAdv.Equal(timeCha) {
// if not, give an error and remove the pair so it isn't reported multiple times should integrity be called again
p.badChanges.error("Current release date in changes.csv and %s is not identical.", f.URL())
delete(p.timesAdv, f.URL())
delete(p.timesChanges, f.URL())
}
}
}
}
}
return nil
}
// extractTime extracts a time.Time value from a json document and returns it and an empty string or zero time alongside
// a string representing the error message that prevented obtaining the proper time value.
func (p *processor) extractTime(doc any, value string, u any) (time.Time, string) {
filter := "$.document.tracking." + value
date, err := p.expr.Eval(filter, doc)
if err != nil {
return time.Time{}, fmt.Sprintf("Extracting '%s' from %s failed: %v", value, u, err)
}
text, ok := date.(string)
if !ok {
return time.Time{}, fmt.Sprintf("'%s' is not a string in %s", value, u)
}
d, err := time.Parse(time.RFC3339, text)
if err != nil {
return time.Time{}, fmt.Sprintf("Parsing '%s' as RFC3339 failed in %s: %v", value, u, err)
}
return d, ""
}
// checkIndex fetches the "index.txt" and calls "checkTLS" method for HTTPS checks.
// It extracts the file names from the file and passes them to "integrity" function.
// It returns error if fetching/reading the file(s) fails, otherwise nil.
@ -893,7 +951,8 @@ func (p *processor) checkIndex(base string, mask whereType) error {
p.badIntegrities.error("index.txt contains invalid URL %q in line %d", u, line)
continue
}
files = append(files, csaf.PlainAdvisoryFile(u))
files = append(files, csaf.DirectoryAdvisoryFile{Path: u})
}
return files, scanner.Err()
}()
@ -916,7 +975,6 @@ func (p *processor) checkIndex(base string, mask whereType) error {
// of the fields' values and if they are sorted properly. Then it passes the files to the
// "integrity" functions. It returns error if some test fails, otherwise nil.
func (p *processor) checkChanges(base string, mask whereType) error {
bu, err := url.Parse(base)
if err != nil {
return err
@ -975,9 +1033,11 @@ func (p *processor) checkChanges(base string, mask whereType) error {
continue
}
path := r[pathColumn]
times, files =
append(times, t),
append(files, csaf.PlainAdvisoryFile(path))
append(files, csaf.DirectoryAdvisoryFile{Path: path})
p.timesChanges[path] = t
}
return times, files, nil
}()
@ -1149,7 +1209,6 @@ func (p *processor) checkMissing(string) error {
// checkInvalid goes over all found adivisories URLs and checks
// if file name conforms to standard.
func (p *processor) checkInvalid(string) error {
p.badDirListings.use()
var invalids []string
@ -1171,7 +1230,6 @@ func (p *processor) checkInvalid(string) error {
// checkListing goes over all found adivisories URLs and checks
// if their parent directory is listable.
func (p *processor) checkListing(string) error {
p.badDirListings.use()
pgs := pages{}
@ -1206,7 +1264,6 @@ func (p *processor) checkListing(string) error {
// checkWhitePermissions checks if the TLP:WHITE advisories are
// available with unprotected access.
func (p *processor) checkWhitePermissions(string) error {
var ids []string
for id, open := range p.labelChecker.whiteAdvisories {
if !open {
@ -1232,7 +1289,6 @@ func (p *processor) checkWhitePermissions(string) error {
// According to the result, the respective error messages added to
// badProviderMetadata.
func (p *processor) checkProviderMetadata(domain string) bool {
p.badProviderMetadata.use()
client := p.httpClient()
@ -1279,7 +1335,6 @@ func (p *processor) checkSecurity(domain string, legacy bool) (int, string) {
// checkSecurityFolder checks the security.txt in a given folder.
func (p *processor) checkSecurityFolder(folder string) string {
client := p.httpClient()
path := folder + "security.txt"
res, err := client.Get(path)
@ -1344,50 +1399,52 @@ func (p *processor) checkSecurityFolder(folder string) string {
// checkDNS checks if the "csaf.data.security.domain.tld" DNS record is available
// and serves the "provider-metadata.json".
// It returns an empty string if all checks are passed, otherwise the errormessage.
func (p *processor) checkDNS(domain string) string {
func (p *processor) checkDNS(domain string) {
p.badDNSPath.use()
client := p.httpClient()
path := "https://csaf.data.security." + domain
res, err := client.Get(path)
if err != nil {
return fmt.Sprintf("Fetching %s failed: %v", path, err)
p.badDNSPath.add(ErrorType,
fmt.Sprintf("Fetching %s failed: %v", path, err))
return
}
if res.StatusCode != http.StatusOK {
return fmt.Sprintf("Fetching %s failed. Status code %d (%s)",
path, res.StatusCode, res.Status)
p.badDNSPath.add(ErrorType, fmt.Sprintf("Fetching %s failed. Status code %d (%s)",
path, res.StatusCode, res.Status))
}
hash := sha256.New()
defer res.Body.Close()
content, err := io.ReadAll(res.Body)
if err != nil {
return fmt.Sprintf("Error while reading the response from %s", path)
p.badDNSPath.add(ErrorType,
fmt.Sprintf("Error while reading the response from %s", path))
}
hash.Write(content)
if !bytes.Equal(hash.Sum(nil), p.pmd256) {
return fmt.Sprintf("%s does not serve the same provider-metadata.json as previously found", path)
p.badDNSPath.add(ErrorType,
fmt.Sprintf("%s does not serve the same provider-metadata.json as previously found",
path))
}
return ""
}
// checkWellknownMetadataReporter checks if the provider-metadata.json file is
// available under the /.well-known/csaf/ directory. Returns the errormessage if
// an error was encountered, or an empty string otherwise
func (p *processor) checkWellknown(domain string) string {
// checkWellknown checks if the provider-metadata.json file is
// available under the /.well-known/csaf/ directory.
func (p *processor) checkWellknown(domain string) {
p.badWellknownMetadata.use()
client := p.httpClient()
path := "https://" + domain + "/.well-known/csaf/provider-metadata.json"
res, err := client.Get(path)
if err != nil {
return fmt.Sprintf("Fetching %s failed: %v", path, err)
p.badWellknownMetadata.add(ErrorType,
fmt.Sprintf("Fetching %s failed: %v", path, err))
}
if res.StatusCode != http.StatusOK {
return fmt.Sprintf("Fetching %s failed. Status code %d (%s)",
path, res.StatusCode, res.Status)
p.badWellknownMetadata.add(ErrorType, fmt.Sprintf("Fetching %s failed. Status code %d (%s)",
path, res.StatusCode, res.Status))
}
return ""
}
// checkWellknownSecurityDNS
@ -1399,66 +1456,49 @@ func (p *processor) checkWellknown(domain string) string {
// 4. Finally it checks if the "csaf.data.security.domain.tld" DNS record
// is available and serves the "provider-metadata.json".
//
// /
// If all three checks fail, errors are given,
// otherwise warnings for all failed checks.
// The function returns nil, unless errors outside the checks were found.
// In that case, errors are returned.
// For the security.txt checks, it first checks the default location.
// Should this lookup fail, a warning is will be given and a lookup
// for the legacy location will be made. If this fails as well, then an
// error is given.
func (p *processor) checkWellknownSecurityDNS(domain string) error {
warningsW := p.checkWellknown(domain)
p.checkWellknown(domain)
// Security check for well known (default) and legacy location
warningsS, sDMessage := p.checkSecurity(domain, false)
warnings, sDMessage := p.checkSecurity(domain, false)
// if the security.txt under .well-known was not okay
// check for a security.txt within its legacy location
sLMessage := ""
if warningsS == 1 {
warningsS, sLMessage = p.checkSecurity(domain, true)
if warnings == 1 {
warnings, sLMessage = p.checkSecurity(domain, true)
}
warningsD := p.checkDNS(domain)
p.badWellknownMetadata.use()
p.badSecurity.use()
p.badDNSPath.use()
var kind MessageType
if warningsS != 1 || warningsD == "" || warningsW == "" {
kind = WarnType
} else {
kind = ErrorType
// Report about Securitytxt:
// Only report about default location if it was succesful (0).
// Report default and legacy as errors if neither was succesful (1).
// Warn about missing security in the default position if not found
// but found in the legacy location, and inform about finding it there (2).
switch warnings {
case 0:
p.badSecurity.add(InfoType, sDMessage)
case 1:
p.badSecurity.add(ErrorType, sDMessage)
p.badSecurity.add(ErrorType, sLMessage)
case 2:
p.badSecurity.add(WarnType, sDMessage)
p.badSecurity.add(InfoType, sLMessage)
}
// Info, Warning or Error depending on kind and warningS
kindSD := kind
if warningsS == 0 {
kindSD = InfoType
}
kindSL := kind
if warningsS == 2 {
kindSL = InfoType
}
p.checkDNS(domain)
if warningsW != "" {
p.badWellknownMetadata.add(kind, warningsW)
}
p.badSecurity.add(kindSD, sDMessage)
// only if the well-known security.txt was not successful:
// report about the legacy location
if warningsS != 0 {
p.badSecurity.add(kindSL, sLMessage)
}
if warningsD != "" {
p.badDNSPath.add(kind, warningsD)
}
return nil
}
// checkPGPKeys checks if the OpenPGP keys are available and valid, fetches
// the the remotely keys and compares the fingerprints.
// As a result of these a respective error messages are passed to badPGP method
// in case of errors. It returns nil if all checks are passed.
// the remote pubkeys and compares the fingerprints.
// As a result of these checks respective error messages are passed
// to badPGP methods. It returns nil if all checks are passed.
func (p *processor) checkPGPKeys(_ string) error {
p.badPGPs.use()
src, err := p.expr.Eval("$.public_openpgp_keys", p.pmd)
@ -1517,14 +1557,13 @@ func (p *processor) checkPGPKeys(_ string) error {
defer res.Body.Close()
return crypto.NewKeyFromArmoredReader(res.Body)
}()
if err != nil {
p.badPGPs.error("Reading public OpenPGP key %s failed: %v", u, err)
continue
}
if !strings.EqualFold(ckey.GetFingerprint(), string(key.Fingerprint)) {
p.badPGPs.error("Fingerprint of public OpenPGP key %s does not match remotely loaded.", u)
p.badPGPs.error("Given Fingerprint (%q) of public OpenPGP key %q does not match remotely loaded (%q).", string(key.Fingerprint), u, ckey.GetFingerprint())
continue
}
if p.keys == nil {

259
cmd/csaf_checker/processor_test.go Normal file
View file

@ -0,0 +1,259 @@
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
package main
import (
"bytes"
"encoding/json"
"net/http/httptest"
"os"
"reflect"
"slices"
"strings"
"testing"
"text/template"
"github.com/gocsaf/csaf/v3/internal/testutil"
"github.com/gocsaf/csaf/v3/util"
)
func getRequirementTestData(t *testing.T, params testutil.ProviderParams, directoryProvider bool) []Requirement {
path := "../../testdata/processor-requirements/"
if params.EnableSha256 {
path += "sha256-"
}
if params.EnableSha512 {
path += "sha512-"
}
if params.ForbidSha256 {
path += "forbid-sha256-"
}
if params.ForbidSha512 {
path += "forbid-sha512-"
}
if directoryProvider {
path += "directory"
} else {
path += "rolie"
}
path += ".json"
content, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
tmplt, err := template.New("base").Parse(string(content))
if err != nil {
t.Fatal(err)
}
var output bytes.Buffer
err = tmplt.Execute(&output, params)
if err != nil {
t.Fatal(err)
}
var requirement []Requirement
err = json.Unmarshal(output.Bytes(), &requirement)
if err != nil {
t.Fatal(err)
}
return requirement
}
func TestContentTypeReport(t *testing.T) {
serverURL := ""
params := testutil.ProviderParams{
URL: "",
EnableSha256: true,
EnableSha512: true,
ForbidSha256: true,
ForbidSha512: true,
JSONContentType: "application/json; charset=utf-8",
}
server := httptest.NewTLSServer(testutil.ProviderHandler(&params, false))
defer server.Close()
serverURL = server.URL
params.URL = server.URL
hClient := server.Client()
client := util.Client(hClient)
cfg := config{}
err := cfg.prepare()
if err != nil {
t.Fatalf("SHA marking config failed: %v", err)
}
p, err := newProcessor(&cfg)
if err != nil {
t.Fatalf("could not init downloader: %v", err)
}
p.client = client
report, err := p.run([]string{serverURL + "/provider-metadata.json"})
if err != nil {
t.Errorf("Content-Type-Report: Expected no error, got: %v", err)
}
got := report.Domains[0].Requirements
idx := slices.IndexFunc(got, func(e *Requirement) bool {
return e.Num == 7
})
if idx == -1 {
t.Error("Content-Type-Report: Could not find requirement")
} else {
message := got[idx].Messages[0]
if message.Type != ErrorType || !strings.Contains(message.Text, "should be 'application/json'") {
t.Errorf("Content-Type-Report: Content Type Error, got %v", message)
}
}
p.close()
}
func TestShaMarking(t *testing.T) {
tests := []struct {
name string
directoryProvider bool
enableSha256 bool
enableSha512 bool
forbidSha256 bool
forbidSha512 bool
}{
{
name: "deliver sha256 and sha512",
directoryProvider: false,
enableSha256: true,
enableSha512: true,
},
{
name: "enable sha256 and sha512, forbid fetching",
directoryProvider: false,
enableSha256: true,
enableSha512: true,
forbidSha256: true,
forbidSha512: true,
},
{
name: "enable sha256 and sha512, forbid sha256",
directoryProvider: false,
enableSha256: true,
enableSha512: true,
forbidSha256: true,
forbidSha512: false,
},
{
name: "enable sha256 and sha512, forbid sha512",
directoryProvider: false,
enableSha256: true,
enableSha512: true,
forbidSha256: false,
forbidSha512: true,
},
{
name: "only deliver sha256",
directoryProvider: false,
enableSha256: true,
enableSha512: false,
},
{
name: "only deliver sha512",
directoryProvider: false,
enableSha256: false,
enableSha512: true,
},
{
name: "deliver sha256 and sha512, directory provider",
directoryProvider: true,
enableSha256: true,
enableSha512: true,
},
{
name: "only deliver sha256, directory provider",
directoryProvider: true,
enableSha256: true,
enableSha512: false,
},
{
name: "only deliver sha512, directory provider",
directoryProvider: true,
enableSha256: false,
enableSha512: true,
},
{
name: "no hash",
directoryProvider: false,
enableSha256: false,
enableSha512: false,
},
{
name: "no hash, directory provider",
directoryProvider: true,
enableSha256: false,
enableSha512: false,
},
}
t.Parallel()
for _, testToRun := range tests {
test := testToRun
t.Run(test.name, func(tt *testing.T) {
tt.Parallel()
serverURL := ""
params := testutil.ProviderParams{
URL: "",
EnableSha256: test.enableSha256,
EnableSha512: test.enableSha512,
ForbidSha256: test.forbidSha256,
ForbidSha512: test.forbidSha512,
}
server := httptest.NewTLSServer(testutil.ProviderHandler(&params, test.directoryProvider))
defer server.Close()
serverURL = server.URL
params.URL = server.URL
hClient := server.Client()
client := util.Client(hClient)
cfg := config{}
err := cfg.prepare()
if err != nil {
t.Fatalf("SHA marking config failed: %v", err)
}
p, err := newProcessor(&cfg)
if err != nil {
t.Fatalf("could not init downloader: %v", err)
}
p.client = client
report, err := p.run([]string{serverURL + "/provider-metadata.json"})
if err != nil {
t.Errorf("SHA marking %v: Expected no error, got: %v", test.name, err)
}
expected := getRequirementTestData(t,
testutil.ProviderParams{
URL: serverURL,
EnableSha256: test.enableSha256,
EnableSha512: test.enableSha512,
ForbidSha256: test.forbidSha256,
ForbidSha512: test.forbidSha512,
},
test.directoryProvider)
for i, got := range report.Domains[0].Requirements {
if !reflect.DeepEqual(expected[i], *got) {
t.Errorf("SHA marking %v: Expected %v, got %v", test.name, expected[i], *got)
}
}
p.close()
})
}
}

10
cmd/csaf_checker/report.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -18,8 +18,8 @@ import (
"os"
"time"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/internal/models"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/internal/models"
)
// MessageType is the kind of the message.

10
cmd/csaf_checker/reporters.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -13,7 +13,7 @@ import (
"sort"
"strings"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
type (
@ -178,7 +178,7 @@ func (r *tlpAmberRedReporter) report(p *processor, domain *Domain) {
return
}
if len(p.badAmberRedPermissions) == 0 {
req.message(InfoType, "All tested advisories labeled TLP:WHITE or TLP:RED were access-protected.")
req.message(InfoType, "All tested advisories labeled TLP:AMBER or TLP:RED were access-protected.")
return
}
req.Messages = p.badAmberRedPermissions

10
cmd/csaf_checker/roliecheck.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -15,8 +15,8 @@ import (
"sort"
"strings"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// identifier consist of document/tracking/id and document/publisher/namespace,

8
cmd/csaf_checker/rules.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -12,7 +12,7 @@ import (
"fmt"
"sort"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/gocsaf/csaf/v3/csaf"
)
type ruleCondition int

29
cmd/csaf_downloader/config.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -13,17 +13,16 @@ import (
"fmt"
"io"
"log"
"log/slog"
"net/http"
"os"
"path/filepath"
"time"
"golang.org/x/exp/slog"
"github.com/csaf-poc/csaf_distribution/v3/internal/certs"
"github.com/csaf-poc/csaf_distribution/v3/internal/filter"
"github.com/csaf-poc/csaf_distribution/v3/internal/models"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/certs"
"github.com/gocsaf/csaf/v3/internal/filter"
"github.com/gocsaf/csaf/v3/internal/models"
"github.com/gocsaf/csaf/v3/internal/options"
)
const (
@ -42,6 +41,13 @@ const (
validationUnsafe = validationMode("unsafe")
)
type hashAlgorithm string
const (
algSha256 = hashAlgorithm("sha256")
algSha512 = hashAlgorithm("sha512")
)
type config struct {
Directory string `short:"d" long:"directory" description:"DIRectory to store the downloaded files in" value-name:"DIR" toml:"directory"`
Insecure bool `long:"insecure" description:"Do not check TLS certificates from provider" toml:"insecure"`
@ -58,6 +64,8 @@ type config struct {
IgnorePattern []string `long:"ignore_pattern" short:"i" description:"Do not download files if their URLs match any of the given PATTERNs" value-name:"PATTERN" toml:"ignore_pattern"`
ExtraHeader http.Header `long:"header" short:"H" description:"One or more extra HTTP header fields" toml:"header"`
EnumeratePMDOnly bool `long:"enumerate_pmd_only" description:"If this flag is set to true, the downloader will only enumerate valid provider metadata files, but not download documents" toml:"enumerate_pmd_only"`
RemoteValidator string `long:"validator" description:"URL to validate documents remotely" value-name:"URL" toml:"validator"`
RemoteValidatorCache string `long:"validator_cache" description:"FILE to cache remote validations" value-name:"FILE" toml:"validator_cache"`
RemoteValidatorPresets []string `long:"validator_preset" description:"One or more PRESETS to validate remotely" value-name:"PRESETS" toml:"validator_preset"`
@ -78,6 +86,9 @@ type config struct {
clientCerts []tls.Certificate
ignorePattern filter.PatternMatcher
//lint:ignore SA5008 We are using choice or than once: sha256, sha512
PreferredHash hashAlgorithm `long:"preferred_hash" choice:"sha256" choice:"sha512" value-name:"HASH" description:"HASH to prefer" toml:"preferred_hash"`
}
// configPaths are the potential file locations of the config file.

395
cmd/csaf_downloader/downloader.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022, 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022, 2023 Intevation GmbH <https://intevation.de>
@ -19,29 +19,36 @@ import (
"fmt"
"hash"
"io"
"log/slog"
"net/http"
"net/url"
"os"
"path"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"time"
"golang.org/x/exp/slog"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"golang.org/x/time/rate"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
type hashFetchInfo struct {
url string
preferred bool
warn bool
hashType hashAlgorithm
}
type downloader struct {
cfg *config
client *util.Client // Used for testing
keys *crypto.KeyRing
eval *util.PathEval
validator csaf.RemoteValidator
forwarder *forwarder
mkdirMu sync.Mutex
@ -55,7 +62,6 @@ type downloader struct {
const failedValidationDir = "failed_validation"
func newDownloader(cfg *config) (*downloader, error) {
var validator csaf.RemoteValidator
if cfg.RemoteValidator != "" {
@ -74,7 +80,6 @@ func newDownloader(cfg *config) (*downloader, error) {
return &downloader{
cfg: cfg,
eval: util.NewPathEval(),
validator: validator,
}, nil
}
@ -106,7 +111,6 @@ func logRedirect(req *http.Request, via []*http.Request) error {
}
func (d *downloader) httpClient() util.Client {
hClient := http.Client{}
if d.cfg.verbose() {
@ -124,17 +128,21 @@ func (d *downloader) httpClient() util.Client {
hClient.Transport = &http.Transport{
TLSClientConfig: &tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
client := util.Client(&hClient)
// Overwrite for testing purposes
if d.client != nil {
client = *d.client
}
// Add extra headers.
if len(d.cfg.ExtraHeader) > 0 {
client = &util.HeaderClient{
Client: client,
Header: d.cfg.ExtraHeader,
}
}
// Add optional URL logging.
if d.cfg.verbose() {
@ -165,6 +173,36 @@ func httpLog(who string) func(string, string) {
}
}
func (d *downloader) enumerate(domain string) error {
client := d.httpClient()
loader := csaf.NewProviderMetadataLoader(client)
lpmd := loader.Enumerate(domain)
docs := []any{}
for _, pmd := range lpmd {
if d.cfg.verbose() {
for i := range pmd.Messages {
slog.Debug("Enumerating provider-metadata.json",
"domain", domain,
"message", pmd.Messages[i].Message)
}
}
docs = append(docs, pmd.Document)
}
// print the results
doc, err := json.MarshalIndent(docs, "", " ")
if err != nil {
slog.Error("Couldn't marshal PMD document json")
}
fmt.Println(string(doc))
return nil
}
func (d *downloader) download(ctx context.Context, domain string) error {
client := d.httpClient()
@ -172,7 +210,14 @@ func (d *downloader) download(ctx context.Context, domain string) error {
lpmd := loader.Load(domain)
if d.cfg.verbose() {
if !lpmd.Valid() {
for i := range lpmd.Messages {
slog.Error("Loading provider-metadata.json",
"domain", domain,
"message", lpmd.Messages[i].Message)
}
return fmt.Errorf("no valid provider-metadata.json found for '%s'", domain)
} else if d.cfg.verbose() {
for i := range lpmd.Messages {
slog.Debug("Loading provider-metadata.json",
"domain", domain,
@ -180,26 +225,25 @@ func (d *downloader) download(ctx context.Context, domain string) error {
}
}
if !lpmd.Valid() {
return fmt.Errorf("no valid provider-metadata.json found for '%s'", domain)
}
base, err := url.Parse(lpmd.URL)
if err != nil {
return fmt.Errorf("invalid URL '%s': %v", lpmd.URL, err)
}
expr := util.NewPathEval()
if err := d.loadOpenPGPKeys(
client,
lpmd.Document,
base,
expr,
); err != nil {
return err
}
afp := csaf.NewAdvisoryFileProcessor(
client,
d.eval,
expr,
lpmd.Document,
base)
@ -220,7 +264,6 @@ func (d *downloader) downloadFiles(
label csaf.TLPLabel,
files []csaf.AdvisoryFile,
) error {
var (
advisoryCh = make(chan csaf.AdvisoryFile)
errorCh = make(chan error)
@ -268,9 +311,9 @@ func (d *downloader) loadOpenPGPKeys(
client util.Client,
doc any,
base *url.URL,
expr *util.PathEval,
) error {
src, err := d.eval.Eval("$.public_openpgp_keys", doc)
src, err := expr.Eval("$.public_openpgp_keys", doc)
if err != nil {
// no keys.
return nil
@ -323,7 +366,6 @@ func (d *downloader) loadOpenPGPKeys(
defer res.Body.Close()
return crypto.NewKeyFromArmoredReader(res.Body)
}()
if err != nil {
slog.Warn(
"Reading public OpenPGP key failed",
@ -335,7 +377,7 @@ func (d *downloader) loadOpenPGPKeys(
if !strings.EqualFold(ckey.GetFingerprint(), string(key.Fingerprint)) {
slog.Warn(
"Fingerprint of public OpenPGP key does not match remotely loaded",
"url", u)
"url", u, "fingerprint", key.Fingerprint, "remote-fingerprint", ckey.GetFingerprint())
continue
}
if d.keys == nil {
@ -375,80 +417,74 @@ func (d *downloader) logValidationIssues(url string, errors []string, err error)
}
}
func (d *downloader) downloadWorker(
ctx context.Context,
wg *sync.WaitGroup,
label csaf.TLPLabel,
files <-chan csaf.AdvisoryFile,
errorCh chan<- error,
) {
defer wg.Done()
var (
client = d.httpClient()
// downloadContext stores the common context of a downloader.
type downloadContext struct {
d *downloader
client util.Client
data bytes.Buffer
lastDir string
initialReleaseDate time.Time
dateExtract = util.TimeMatcher(&initialReleaseDate, time.RFC3339)
lower = strings.ToLower(string(label))
stats = stats{}
)
// Add collected stats back to total.
defer d.addStats(&stats)
nextAdvisory:
for {
var file csaf.AdvisoryFile
var ok bool
select {
case file, ok = <-files:
if !ok {
return
}
case <-ctx.Done():
return
dateExtract func(any) error
lower string
stats stats
expr *util.PathEval
}
func newDownloadContext(d *downloader, label csaf.TLPLabel) *downloadContext {
dc := &downloadContext{
d: d,
client: d.httpClient(),
lower: strings.ToLower(string(label)),
expr: util.NewPathEval(),
}
dc.dateExtract = util.TimeMatcher(&dc.initialReleaseDate, time.RFC3339)
return dc
}
func (dc *downloadContext) downloadAdvisory(
file csaf.AdvisoryFile,
errorCh chan<- error,
) error {
u, err := url.Parse(file.URL())
if err != nil {
stats.downloadFailed++
dc.stats.downloadFailed++
slog.Warn("Ignoring invalid URL",
"url", file.URL(),
"error", err)
continue
return nil
}
if d.cfg.ignoreURL(file.URL()) {
if dc.d.cfg.ignoreURL(file.URL()) {
slog.Debug("Ignoring URL", "url", file.URL())
continue
return nil
}
// Ignore not conforming filenames.
filename := filepath.Base(u.Path)
if !util.ConformingFileName(filename) {
stats.filenameFailed++
dc.stats.filenameFailed++
slog.Warn("Ignoring none conforming filename",
"filename", filename)
continue
return nil
}
resp, err := client.Get(file.URL())
resp, err := dc.client.Get(file.URL())
if err != nil {
stats.downloadFailed++
dc.stats.downloadFailed++
slog.Warn("Cannot GET",
"url", file.URL(),
"error", err)
continue
return nil
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
stats.downloadFailed++
dc.stats.downloadFailed++
slog.Warn("Cannot load",
"url", file.URL(),
"status", resp.Status,
"status_code", resp.StatusCode)
continue
return nil
}
// Warn if we do not get JSON.
@ -466,50 +502,66 @@ nextAdvisory:
signData []byte
)
// Only hash when we have a remote counter part we can compare it with.
if remoteSHA256, s256Data, err = loadHash(client, file.SHA256URL()); err != nil {
slog.Warn("Cannot fetch SHA256",
"url", file.SHA256URL(),
"error", err)
hashToFetch := []hashFetchInfo{}
if file.SHA512URL() != "" {
hashToFetch = append(hashToFetch, hashFetchInfo{
url: file.SHA512URL(),
warn: true,
hashType: algSha512,
preferred: strings.EqualFold(string(dc.d.cfg.PreferredHash), string(algSha512)),
})
} else {
slog.Info("SHA512 not present")
}
if file.SHA256URL() != "" {
hashToFetch = append(hashToFetch, hashFetchInfo{
url: file.SHA256URL(),
warn: true,
hashType: algSha256,
preferred: strings.EqualFold(string(dc.d.cfg.PreferredHash), string(algSha256)),
})
} else {
slog.Info("SHA256 not present")
}
if file.IsDirectory() {
for i := range hashToFetch {
hashToFetch[i].warn = false
}
}
remoteSHA256, s256Data, remoteSHA512, s512Data = loadHashes(dc.client, hashToFetch)
if remoteSHA512 != nil {
s512 = sha512.New()
writers = append(writers, s512)
}
if remoteSHA256 != nil {
s256 = sha256.New()
writers = append(writers, s256)
}
if remoteSHA512, s512Data, err = loadHash(client, file.SHA512URL()); err != nil {
slog.Warn("Cannot fetch SHA512",
"url", file.SHA512URL(),
"error", err)
} else {
s512 = sha512.New()
writers = append(writers, s512)
}
// Remember the data as we need to store it to file later.
data.Reset()
writers = append(writers, &data)
dc.data.Reset()
writers = append(writers, &dc.data)
// Download the advisory and hash it.
hasher := io.MultiWriter(writers...)
var doc any
if err := func() error {
defer resp.Body.Close()
tee := io.TeeReader(resp.Body, hasher)
return json.NewDecoder(tee).Decode(&doc)
}(); err != nil {
stats.downloadFailed++
if err := json.NewDecoder(tee).Decode(&doc); err != nil {
dc.stats.downloadFailed++
slog.Warn("Downloading failed",
"url", file.URL(),
"error", err)
continue
return nil
}
// Compare the checksums.
s256Check := func() error {
if s256 != nil && !bytes.Equal(s256.Sum(nil), remoteSHA256) {
stats.sha256Failed++
dc.stats.sha256Failed++
return fmt.Errorf("SHA256 checksum of %s does not match", file.URL())
}
return nil
@ -517,7 +569,7 @@ nextAdvisory:
s512Check := func() error {
if s512 != nil && !bytes.Equal(s512.Sum(nil), remoteSHA512) {
stats.sha512Failed++
dc.stats.sha512Failed++
return fmt.Errorf("SHA512 checksum of %s does not match", file.URL())
}
return nil
@ -526,20 +578,20 @@ nextAdvisory:
// Validate OpenPGP signature.
keysCheck := func() error {
// Only check signature if we have loaded keys.
if d.keys == nil {
if dc.d.keys == nil {
return nil
}
var sign *crypto.PGPSignature
sign, signData, err = loadSignature(client, file.SignURL())
sign, signData, err = loadSignature(dc.client, file.SignURL())
if err != nil {
slog.Warn("Downloading signature failed",
"url", file.SignURL(),
"error", err)
}
if sign != nil {
if err := d.checkSignature(data.Bytes(), sign); err != nil {
if !d.cfg.IgnoreSignatureCheck {
stats.signatureFailed++
if err := dc.d.checkSignature(dc.data.Bytes(), sign); err != nil {
if !dc.d.cfg.IgnoreSignatureCheck {
dc.stats.signatureFailed++
return fmt.Errorf("cannot verify signature for %s: %v", file.URL(), err)
}
}
@ -550,8 +602,8 @@ nextAdvisory:
// Validate against CSAF schema.
schemaCheck := func() error {
if errors, err := csaf.ValidateCSAF(doc); err != nil || len(errors) > 0 {
stats.schemaFailed++
d.logValidationIssues(file.URL(), errors, err)
dc.stats.schemaFailed++
dc.d.logValidationIssues(file.URL(), errors, err)
return fmt.Errorf("schema validation for %q failed", file.URL())
}
return nil
@ -559,8 +611,8 @@ nextAdvisory:
// Validate if filename is conforming.
filenameCheck := func() error {
if err := util.IDMatchesFilename(d.eval, doc, filename); err != nil {
stats.filenameFailed++
if err := util.IDMatchesFilename(dc.expr, doc, filename); err != nil {
dc.stats.filenameFailed++
return fmt.Errorf("filename not conforming %s: %s", file.URL(), err)
}
return nil
@ -568,10 +620,10 @@ nextAdvisory:
// Validate against remote validator.
remoteValidatorCheck := func() error {
if d.validator == nil {
if dc.d.validator == nil {
return nil
}
rvr, err := d.validator.Validate(doc)
rvr, err := dc.d.validator.Validate(doc)
if err != nil {
errorCh <- fmt.Errorf(
"calling remote validator on %q failed: %w",
@ -579,7 +631,7 @@ nextAdvisory:
return nil
}
if !rvr.Valid {
stats.remoteFailed++
dc.stats.remoteFailed++
return fmt.Errorf("remote validation of %q failed", file.URL())
}
return nil
@ -598,71 +650,71 @@ nextAdvisory:
if err := check(); err != nil {
slog.Error("Validation check failed", "error", err)
valStatus.update(invalidValidationStatus)
if d.cfg.ValidationMode == validationStrict {
continue nextAdvisory
if dc.d.cfg.ValidationMode == validationStrict {
return nil
}
}
}
valStatus.update(validValidationStatus)
// Send to forwarder
if d.forwarder != nil {
d.forwarder.forward(
filename, data.String(),
if dc.d.forwarder != nil {
dc.d.forwarder.forward(
filename, dc.data.String(),
valStatus,
string(s256Data),
string(s512Data))
}
if d.cfg.NoStore {
if dc.d.cfg.NoStore {
// Do not write locally.
if valStatus == validValidationStatus {
stats.succeeded++
dc.stats.succeeded++
}
continue
return nil
}
if err := d.eval.Extract(
`$.document.tracking.initial_release_date`, dateExtract, false, doc,
if err := dc.expr.Extract(
`$.document.tracking.initial_release_date`, dc.dateExtract, false, doc,
); err != nil {
slog.Warn("Cannot extract initial_release_date from advisory",
"url", file.URL())
initialReleaseDate = time.Now()
dc.initialReleaseDate = time.Now()
}
initialReleaseDate = initialReleaseDate.UTC()
dc.initialReleaseDate = dc.initialReleaseDate.UTC()
// Advisories that failed validation are stored in a special folder.
var newDir string
if valStatus != validValidationStatus {
newDir = path.Join(d.cfg.Directory, failedValidationDir)
newDir = path.Join(dc.d.cfg.Directory, failedValidationDir)
} else {
newDir = d.cfg.Directory
newDir = dc.d.cfg.Directory
}
// Do we have a configured destination folder?
if d.cfg.Folder != "" {
newDir = path.Join(newDir, d.cfg.Folder)
if dc.d.cfg.Folder != "" {
newDir = path.Join(newDir, dc.d.cfg.Folder)
} else {
newDir = path.Join(newDir, lower, strconv.Itoa(initialReleaseDate.Year()))
newDir = path.Join(newDir, dc.lower, strconv.Itoa(dc.initialReleaseDate.Year()))
}
if newDir != lastDir {
if err := d.mkdirAll(newDir, 0755); err != nil {
if newDir != dc.lastDir {
if err := dc.d.mkdirAll(newDir, 0755); err != nil {
errorCh <- err
continue
return nil
}
lastDir = newDir
dc.lastDir = newDir
}
// Write advisory to file
path := filepath.Join(lastDir, filename)
path := filepath.Join(dc.lastDir, filename)
// Write data to disk.
for _, x := range []struct {
p string
d []byte
}{
{path, data.Bytes()},
{path, dc.data.Bytes()},
{path + ".sha256", s256Data},
{path + ".sha512", s512Data},
{path + ".asc", signData},
@ -670,13 +722,45 @@ nextAdvisory:
if x.d != nil {
if err := os.WriteFile(x.p, x.d, 0644); err != nil {
errorCh <- err
continue nextAdvisory
return nil
}
}
}
stats.succeeded++
dc.stats.succeeded++
slog.Info("Written advisory", "path", path)
return nil
}
func (d *downloader) downloadWorker(
ctx context.Context,
wg *sync.WaitGroup,
label csaf.TLPLabel,
files <-chan csaf.AdvisoryFile,
errorCh chan<- error,
) {
defer wg.Done()
dc := newDownloadContext(d, label)
// Add collected stats back to total.
defer d.addStats(&dc.stats)
for {
var file csaf.AdvisoryFile
var ok bool
select {
case file, ok = <-files:
if !ok {
return
}
case <-ctx.Done():
return
}
if err := dc.downloadAdvisory(file, errorCh); err != nil {
slog.Error("download terminated", "error", err)
return
}
}
}
@ -697,11 +781,11 @@ func loadSignature(client util.Client, p string) (*crypto.PGPSignature, []byte,
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, nil, fmt.Errorf(
"fetching signature from '%s' failed: %s (%d)", p, resp.Status, resp.StatusCode)
}
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil {
return nil, nil, err
@ -713,16 +797,60 @@ func loadSignature(client util.Client, p string) (*crypto.PGPSignature, []byte,
return sign, data, nil
}
func loadHashes(client util.Client, hashes []hashFetchInfo) ([]byte, []byte, []byte, []byte) {
var remoteSha256, remoteSha512, sha256Data, sha512Data []byte
// Load preferred hashes first
slices.SortStableFunc(hashes, func(a, b hashFetchInfo) int {
if a.preferred == b.preferred {
return 0
}
if a.preferred && !b.preferred {
return -1
}
return 1
})
for _, h := range hashes {
if remote, data, err := loadHash(client, h.url); err != nil {
if h.warn {
slog.Warn("Cannot fetch hash",
"hash", h.hashType,
"url", h.url,
"error", err)
} else {
slog.Info("Hash not present", "hash", h.hashType, "file", h.url)
}
} else {
switch h.hashType {
case algSha512:
{
remoteSha512 = remote
sha512Data = data
}
case algSha256:
{
remoteSha256 = remote
sha256Data = data
}
}
if h.preferred {
break
}
}
}
return remoteSha256, sha256Data, remoteSha512, sha512Data
}
func loadHash(client util.Client, p string) ([]byte, []byte, error) {
resp, err := client.Get(p)
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, nil, fmt.Errorf(
"fetching hash from '%s' failed: %s (%d)", p, resp.Status, resp.StatusCode)
}
defer resp.Body.Close()
var data bytes.Buffer
tee := io.TeeReader(resp.Body, &data)
hash, err := util.HashFromReader(tee)
@ -742,3 +870,14 @@ func (d *downloader) run(ctx context.Context, domains []string) error {
}
return nil
}
// runEnumerate performs the enumeration of PMDs for all the given domains.
func (d *downloader) runEnumerate(domains []string) error {
defer d.stats.log()
for _, domain := range domains {
if err := d.enumerate(domain); err != nil {
return err
}
}
return nil
}

160
cmd/csaf_downloader/downloader_test.go Normal file
View file

@ -0,0 +1,160 @@
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
package main
import (
"context"
"errors"
"log/slog"
"net/http/httptest"
"os"
"testing"
"github.com/gocsaf/csaf/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/testutil"
"github.com/gocsaf/csaf/v3/util"
)
func checkIfFileExists(path string, t *testing.T) bool {
if _, err := os.Stat(path); err == nil {
return true
} else if !errors.Is(err, os.ErrNotExist) {
t.Fatalf("Failed to check if file exists: %v", err)
}
return false
}
func TestShaMarking(t *testing.T) {
tests := []struct {
name string
directoryProvider bool
wantSha256 bool
wantSha512 bool
enableSha256 bool
enableSha512 bool
preferredHash hashAlgorithm
}{
{
name: "want sha256 and sha512",
directoryProvider: false,
wantSha256: true,
wantSha512: true,
enableSha256: true,
enableSha512: true,
},
{
name: "only want sha256",
directoryProvider: false,
wantSha256: true,
wantSha512: false,
enableSha256: true,
enableSha512: true,
preferredHash: algSha256,
},
{
name: "only want sha512",
directoryProvider: false,
wantSha256: false,
wantSha512: true,
enableSha256: true,
enableSha512: true,
preferredHash: algSha512,
},
{
name: "only want sha512",
directoryProvider: false,
wantSha256: false,
wantSha512: true,
enableSha256: true,
enableSha512: true,
preferredHash: algSha512,
},
{
name: "only deliver sha256",
directoryProvider: false,
wantSha256: true,
wantSha512: false,
enableSha256: true,
enableSha512: false,
preferredHash: algSha512,
},
{
name: "only want sha256, directory provider",
directoryProvider: true,
wantSha256: true,
wantSha512: false,
enableSha256: true,
enableSha512: true,
preferredHash: algSha256,
},
{
name: "only want sha512, directory provider",
directoryProvider: true,
wantSha256: false,
wantSha512: true,
enableSha256: true,
enableSha512: true,
preferredHash: algSha512,
},
}
t.Parallel()
for _, testToRun := range tests {
test := testToRun
t.Run(test.name, func(tt *testing.T) {
tt.Parallel()
serverURL := ""
params := testutil.ProviderParams{
URL: "",
EnableSha256: test.enableSha256,
EnableSha512: test.enableSha512,
}
server := httptest.NewTLSServer(testutil.ProviderHandler(&params, test.directoryProvider))
defer server.Close()
serverURL = server.URL
params.URL = server.URL
hClient := server.Client()
client := util.Client(hClient)
tempDir := t.TempDir()
cfg := config{LogLevel: &options.LogLevel{Level: slog.LevelDebug}, Directory: tempDir, PreferredHash: test.preferredHash}
err := cfg.prepare()
if err != nil {
t.Fatalf("SHA marking config failed: %v", err)
}
d, err := newDownloader(&cfg)
if err != nil {
t.Fatalf("could not init downloader: %v", err)
}
d.client = &client
ctx := context.Background()
err = d.run(ctx, []string{serverURL + "/provider-metadata.json"})
if err != nil {
t.Errorf("SHA marking %v: Expected no error, got: %v", test.name, err)
}
d.close()
// Check for downloaded hashes
sha256Exists := checkIfFileExists(tempDir+"/white/2020/avendor-advisory-0004.json.sha256", t)
sha512Exists := checkIfFileExists(tempDir+"/white/2020/avendor-advisory-0004.json.sha512", t)
if sha256Exists != test.wantSha256 {
t.Errorf("%v: expected sha256 hash present to be %v, got: %v", test.name, test.wantSha256, sha256Exists)
}
if sha512Exists != test.wantSha512 {
t.Errorf("%v: expected sha512 hash present to be %v, got: %v", test.name, test.wantSha512, sha512Exists)
}
})
}
}

22
cmd/csaf_downloader/forwarder.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -12,16 +12,15 @@ import (
"bytes"
"crypto/tls"
"io"
"log/slog"
"mime/multipart"
"net/http"
"os"
"path/filepath"
"strings"
"golang.org/x/exp/slog"
"github.com/csaf-poc/csaf_distribution/v3/internal/misc"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/internal/misc"
"github.com/gocsaf/csaf/v3/util"
)
// failedForwardDir is the name of the special sub folder
@ -107,17 +106,16 @@ func (f *forwarder) httpClient() util.Client {
hClient.Transport = &http.Transport{
TLSClientConfig: &tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
client := util.Client(&hClient)
// Add extra headers.
if len(f.cfg.ForwardHeader) > 0 {
client = &util.HeaderClient{
Client: client,
Header: f.cfg.ForwardHeader,
}
}
// Add optional URL logging.
if f.cfg.verbose() {
@ -226,12 +224,12 @@ func (f *forwarder) storeFailed(filename, doc, sha256, sha512 string) {
// limitedString reads max bytes from reader and returns it as a string.
// Longer strings are indicated by "..." as a suffix.
func limitedString(r io.Reader, max int) (string, error) {
func limitedString(r io.Reader, maxLength int) (string, error) {
var msg strings.Builder
if _, err := io.Copy(&msg, io.LimitReader(r, int64(max))); err != nil {
if _, err := io.Copy(&msg, io.LimitReader(r, int64(maxLength))); err != nil {
return "", err
}
if msg.Len() >= max {
if msg.Len() >= maxLength {
msg.WriteString("...")
}
return msg.String(), nil

13
cmd/csaf_downloader/forwarder_test.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -14,6 +14,7 @@ import (
"encoding/json"
"errors"
"io"
"log/slog"
"mime"
"mime/multipart"
"net/http"
@ -22,10 +23,8 @@ import (
"strings"
"testing"
"golang.org/x/exp/slog"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/internal/options"
"github.com/gocsaf/csaf/v3/util"
)
func TestValidationStatusUpdate(t *testing.T) {

16
cmd/csaf_downloader/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -11,12 +11,11 @@ package main
import (
"context"
"log/slog"
"os"
"os/signal"
"golang.org/x/exp/slog"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/options"
)
func run(cfg *config, domains []string) error {
@ -41,6 +40,11 @@ func run(cfg *config, domains []string) error {
d.forwarder = f
}
// If the enumerate-only flag is set, enumerate found PMDs,
// else use the normal load method
if cfg.EnumeratePMDOnly {
return d.runEnumerate(domains)
}
return d.run(ctx, domains)
}

8
cmd/csaf_downloader/stats.go
View file

@ -1,14 +1,14 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
package main
import "golang.org/x/exp/slog"
import "log/slog"
// stats contains counters of the downloads.
type stats struct {

9
cmd/csaf_downloader/stats_test.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -11,9 +11,8 @@ package main
import (
"bytes"
"encoding/json"
"log/slog"
"testing"
"golang.org/x/exp/slog"
)
func TestStatsAdd(t *testing.T) {

10
cmd/csaf_provider/actions.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -26,8 +26,8 @@ import (
"github.com/ProtonMail/gopenpgp/v2/constants"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
const dateFormat = time.RFC3339

8
cmd/csaf_provider/config.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -18,7 +18,7 @@ import (
"github.com/ProtonMail/gopenpgp/v2/crypto"
"golang.org/x/crypto/bcrypt"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/gocsaf/csaf/v3/csaf"
)
const (

8
cmd/csaf_provider/controller.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -174,7 +174,7 @@ func (c *controller) web(
// writeJSON sets the header for the response and writes the JSON encoding of the given "content".
// It logs out an error message in case of an error.
func writeJSON(rw http.ResponseWriter, content any, code int) {
rw.Header().Set("Content-type", "application/json; charset=utf-8")
rw.Header().Set("Content-type", "application/json")
rw.Header().Set("X-Content-Type-Options", "nosniff")
rw.WriteHeader(code)
if err := json.NewEncoder(rw).Encode(content); err != nil {

10
cmd/csaf_provider/create.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -22,8 +22,8 @@ import (
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// ensureFolders initializes the paths and call functions to create

8
cmd/csaf_provider/files.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -13,7 +13,7 @@ import (
"crypto/sha512"
"os"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
func writeHashedFile(fname, name string, data []byte, armored string) error {

8
cmd/csaf_provider/indices.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -18,7 +18,7 @@ import (
"sort"
"time"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
func updateIndex(dir, fname string) error {

10
cmd/csaf_provider/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -18,7 +18,7 @@ import (
"github.com/jessevdk/go-flags"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
type options struct {
@ -48,7 +48,7 @@ func main() {
cfg, err := loadConfig()
if err != nil {
cgi.Serve(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
cgi.Serve(http.HandlerFunc(func(rw http.ResponseWriter, _ *http.Request) {
http.Error(rw, "Something went wrong. Check server logs for more details",
http.StatusInternalServerError)
}))

6
cmd/csaf_provider/mux.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>

10
cmd/csaf_provider/rolie.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -15,8 +15,8 @@ import (
"strings"
"time"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
// mergeCategories merges the given categories into the old ones.

6
cmd/csaf_provider/tmpl/create.html
View file

@ -1,8 +1,8 @@
<!--
This file is Free Software under the MIT License
without warranty, see README.md and LICENSES/MIT.txt for details.
This file is Free Software under the Apache-2.0 License
without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
SPDX-License-Identifier: MIT
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
Software-Engineering: 2021 Intevation GmbH <https://intevation.de>

6
cmd/csaf_provider/tmpl/index.html
View file

@ -1,8 +1,8 @@
<!--
This file is Free Software under the MIT License
without warranty, see README.md and LICENSES/MIT.txt for details.
This file is Free Software under the Apache-2.0 License
without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
SPDX-License-Identifier: MIT
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
Software-Engineering: 2021 Intevation GmbH <https://intevation.de>

6
cmd/csaf_provider/tmpl/upload.html
View file

@ -1,8 +1,8 @@
<!--
This file is Free Software under the MIT License
without warranty, see README.md and LICENSES/MIT.txt for details.
This file is Free Software under the Apache-2.0 License
without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
SPDX-License-Identifier: MIT
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
Software-Engineering: 2021 Intevation GmbH <https://intevation.de>

10
cmd/csaf_provider/transaction.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -12,8 +12,8 @@ import (
"os"
"path/filepath"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
func doTransaction(

10
cmd/csaf_uploader/config.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -18,8 +18,8 @@ import (
"golang.org/x/crypto/bcrypt"
"golang.org/x/term"
"github.com/csaf-poc/csaf_distribution/v3/internal/certs"
"github.com/csaf-poc/csaf_distribution/v3/internal/options"
"github.com/gocsaf/csaf/v3/internal/certs"
"github.com/gocsaf/csaf/v3/internal/options"
)
const (

8
cmd/csaf_uploader/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -9,7 +9,7 @@
// Implements a command line tool that uploads csaf documents to csaf_provider.
package main
import "github.com/csaf-poc/csaf_distribution/v3/internal/options"
import "github.com/gocsaf/csaf/v3/internal/options"
func main() {
args, cfg, err := parseArgsConfig()

13
cmd/csaf_uploader/processor.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022, 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022, 2023 Intevation GmbH <https://intevation.de>
@ -26,9 +26,9 @@ import (
"github.com/ProtonMail/gopenpgp/v2/constants"
"github.com/ProtonMail/gopenpgp/v2/crypto"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/internal/misc"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/internal/misc"
"github.com/gocsaf/csaf/v3/util"
)
type processor struct {
@ -51,6 +51,7 @@ func (p *processor) httpClient() *http.Client {
client.Transport = &http.Transport{
TLSClientConfig: &tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
return &client

30
cmd/csaf_validator/main.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -18,8 +18,15 @@ import (
"github.com/jessevdk/go-flags"
"github.com/csaf-poc/csaf_distribution/v3/csaf"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/csaf"
"github.com/gocsaf/csaf/v3/util"
)
const (
exitCodeSchemaInvalid = 2 << iota
exitCodeNoRemoteValidator
exitCodeFailedRemoteValidation
exitCodeAllValid = 0
)
type options struct {
@ -53,6 +60,7 @@ func main() {
// run validates the given files.
func run(opts *options, files []string) error {
exitCode := exitCodeAllValid
var validator csaf.RemoteValidator
eval := util.NewPathEval()
@ -69,6 +77,9 @@ func run(opts *options, files []string) error {
"preparing remote validator failed: %w", err)
}
defer validator.Close()
} else {
exitCode |= exitCodeNoRemoteValidator
log.Printf("warn: no remote validator specified")
}
// Select amount level of output for remote validation.
@ -96,7 +107,7 @@ func run(opts *options, files []string) error {
log.Printf("error: loading %q as JSON failed: %v\n", file, err)
continue
}
// Validate agsinst Schema.
// Validate against Schema.
validationErrs, err := csaf.ValidateCSAF(doc)
if err != nil {
log.Printf("error: validating %q against schema failed: %v\n",
@ -104,6 +115,7 @@ func run(opts *options, files []string) error {
}
if len(validationErrs) > 0 {
exitCode |= exitCodeSchemaInvalid
fmt.Printf("schema validation errors of %q\n", file)
for _, vErr := range validationErrs {
fmt.Printf(" * %s\n", vErr)
@ -112,10 +124,9 @@ func run(opts *options, files []string) error {
fmt.Printf("%q passes the schema validation.\n", file)
}
// Check filename agains ID
// Check filename against ID
if err := util.IDMatchesFilename(eval, doc, filepath.Base(file)); err != nil {
log.Printf("%s: %s.\n", file, err)
continue
}
// Validate against remote validator.
@ -130,12 +141,15 @@ func run(opts *options, files []string) error {
if rvr.Valid {
passes = "passes"
} else {
exitCode |= exitCodeFailedRemoteValidation
passes = "does not pass"
}
fmt.Printf("%q %s remote validation.\n", file, passes)
}
}
// Exit code is based on validation results
os.Exit(exitCode)
return nil
}

150
csaf/advisories.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -9,80 +9,96 @@
package csaf
import (
"context"
"encoding/csv"
"fmt"
"io"
"log"
"log/slog"
"net/http"
"net/url"
"strings"
"time"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
// AdvisoryFile constructs the urls of a remote file.
type AdvisoryFile interface {
slog.LogValuer
URL() string
SHA256URL() string
SHA512URL() string
SignURL() string
IsDirectory() bool
}
// PlainAdvisoryFile is a simple implementation of checkFile.
// PlainAdvisoryFile contains all relevant urls of a remote file.
type PlainAdvisoryFile struct {
Path string
SHA256 string
SHA512 string
Sign string
}
// URL returns the URL of this advisory.
func (paf PlainAdvisoryFile) URL() string { return paf.Path }
// SHA256URL returns the URL of SHA256 hash file of this advisory.
func (paf PlainAdvisoryFile) SHA256URL() string { return paf.SHA256 }
// SHA512URL returns the URL of SHA512 hash file of this advisory.
func (paf PlainAdvisoryFile) SHA512URL() string { return paf.SHA512 }
// SignURL returns the URL of signature file of this advisory.
func (paf PlainAdvisoryFile) SignURL() string { return paf.Sign }
// IsDirectory returns true, if was fetched via directory feeds.
func (paf PlainAdvisoryFile) IsDirectory() bool { return false }
// LogValue implements [slog.LogValuer]
func (paf PlainAdvisoryFile) LogValue() slog.Value {
return slog.GroupValue(slog.String("url", paf.URL()))
}
// DirectoryAdvisoryFile only contains the base file path.
// The hash and signature files are directly constructed by extending
// the file name.
type PlainAdvisoryFile string
// URL returns the URL of this advisory.
func (paf PlainAdvisoryFile) URL() string { return string(paf) }
// SHA256URL returns the URL of SHA256 hash file of this advisory.
func (paf PlainAdvisoryFile) SHA256URL() string { return string(paf) + ".sha256" }
// SHA512URL returns the URL of SHA512 hash file of this advisory.
func (paf PlainAdvisoryFile) SHA512URL() string { return string(paf) + ".sha512" }
// SignURL returns the URL of signature file of this advisory.
func (paf PlainAdvisoryFile) SignURL() string { return string(paf) + ".asc" }
// HashedAdvisoryFile is a more involed version of checkFile.
// Here each component can be given explicitly.
// If a component is not given it is constructed by
// extending the first component.
type HashedAdvisoryFile [4]string
func (haf HashedAdvisoryFile) name(i int, ext string) string {
if haf[i] != "" {
return haf[i]
}
return haf[0] + ext
type DirectoryAdvisoryFile struct {
Path string
}
// URL returns the URL of this advisory.
func (haf HashedAdvisoryFile) URL() string { return haf[0] }
func (daf DirectoryAdvisoryFile) URL() string { return daf.Path }
// SHA256URL returns the URL of SHA256 hash file of this advisory.
func (haf HashedAdvisoryFile) SHA256URL() string { return haf.name(1, ".sha256") }
func (daf DirectoryAdvisoryFile) SHA256URL() string { return daf.Path + ".sha256" }
// SHA512URL returns the URL of SHA512 hash file of this advisory.
func (haf HashedAdvisoryFile) SHA512URL() string { return haf.name(2, ".sha512") }
func (daf DirectoryAdvisoryFile) SHA512URL() string { return daf.Path + ".sha512" }
// SignURL returns the URL of signature file of this advisory.
func (haf HashedAdvisoryFile) SignURL() string { return haf.name(3, ".asc") }
func (daf DirectoryAdvisoryFile) SignURL() string { return daf.Path + ".asc" }
// IsDirectory returns true, if was fetched via directory feeds.
func (daf DirectoryAdvisoryFile) IsDirectory() bool { return true }
// LogValue implements [slog.LogValuer]
func (daf DirectoryAdvisoryFile) LogValue() slog.Value {
return slog.GroupValue(slog.String("url", daf.URL()))
}
// AdvisoryFileProcessor implements the extraction of
// advisory file names from a given provider metadata.
type AdvisoryFileProcessor struct {
AgeAccept func(time.Time) bool
Log func(format string, args ...any)
Log func(loglevel slog.Level, format string, args ...any)
client util.Client
expr *util.PathEval
doc any
base *url.URL
}
// NewAdvisoryFileProcessor constructs an filename extractor
// NewAdvisoryFileProcessor constructs a filename extractor
// for a given metadata document.
func NewAdvisoryFileProcessor(
client util.Client,
@ -108,15 +124,15 @@ func empty(arr []string) bool {
return true
}
// Process extracts the adivisory filenames and passes them with
// Process extracts the advisory filenames and passes them with
// the corresponding label to fn.
func (afp *AdvisoryFileProcessor) Process(
fn func(TLPLabel, []AdvisoryFile) error,
) error {
lg := afp.Log
if lg == nil {
lg = func(format string, args ...any) {
log.Printf("AdvisoryFileProcessor.Process: "+format, args...)
lg = func(loglevel slog.Level, format string, args ...any) {
slog.Log(context.Background(), loglevel, "AdvisoryFileProcessor.Process: "+format, args...)
}
}
@ -124,7 +140,7 @@ func (afp *AdvisoryFileProcessor) Process(
rolie, err := afp.expr.Eval(
"$.distributions[*].rolie.feeds", afp.doc)
if err != nil {
lg("rolie check failed: %v\n", err)
lg(slog.LevelError, "rolie check failed", "err", err)
return err
}
@ -136,7 +152,7 @@ func (afp *AdvisoryFileProcessor) Process(
if err := util.ReMarshalJSON(&feeds, rolie); err != nil {
return err
}
lg("Found %d ROLIE feed(s).\n", len(feeds))
lg(slog.LevelInfo, "Found ROLIE feed(s)", "length", len(feeds))
for _, feed := range feeds {
if err := afp.processROLIE(feed, fn); err != nil {
@ -152,12 +168,12 @@ func (afp *AdvisoryFileProcessor) Process(
var dirURLs []string
if err != nil {
lg("extracting directory URLs failed: %v\n", err)
lg(slog.LevelError, "extracting directory URLs failed", "err", err)
} else {
var ok bool
dirURLs, ok = util.AsStrings(directoryURLs)
if !ok {
lg("directory_urls are not strings.\n")
lg(slog.LevelError, "directory_urls are not strings")
}
}
@ -193,9 +209,8 @@ func (afp *AdvisoryFileProcessor) Process(
// prefixed by baseURL/.
func (afp *AdvisoryFileProcessor) loadChanges(
baseURL string,
lg func(string, ...any),
lg func(slog.Level, string, ...any),
) ([]AdvisoryFile, error) {
base, err := url.Parse(baseURL)
if err != nil {
return nil, err
@ -228,12 +243,12 @@ func (afp *AdvisoryFileProcessor) loadChanges(
return nil, err
}
if len(r) < 2 {
lg("%q has not enough columns in line %d", line)
lg(slog.LevelError, "Not enough columns", "line", line)
continue
}
t, err := time.Parse(time.RFC3339, r[timeColumn])
if err != nil {
lg("%q has an invalid time stamp in line %d: %v", changesURL, line, err)
lg(slog.LevelError, "Invalid time stamp in line", "url", changesURL, "line", line, "err", err)
continue
}
// Apply date range filtering.
@ -242,11 +257,12 @@ func (afp *AdvisoryFileProcessor) loadChanges(
}
path := r[pathColumn]
if _, err := url.Parse(path); err != nil {
lg("%q contains an invalid URL %q in line %d", changesURL, path, line)
lg(slog.LevelError, "Contains an invalid URL", "url", changesURL, "path", path, "line", line)
continue
}
files = append(files,
PlainAdvisoryFile(base.JoinPath(path).String()))
DirectoryAdvisoryFile{Path: base.JoinPath(path).String()})
}
return files, nil
}
@ -262,31 +278,31 @@ func (afp *AdvisoryFileProcessor) processROLIE(
}
up, err := url.Parse(string(*feed.URL))
if err != nil {
log.Printf("Invalid URL %s in feed: %v.", *feed.URL, err)
slog.Error("Invalid URL in feed", "feed", *feed.URL, "err", err)
continue
}
feedURL := afp.base.ResolveReference(up)
log.Printf("Feed URL: %s\n", feedURL)
slog.Info("Got feed URL", "feed", feedURL)
fb, err := util.BaseURL(feedURL)
if err != nil {
log.Printf("error: Invalid feed base URL '%s': %v\n", fb, err)
slog.Error("Invalid feed base URL", "url", fb, "err", err)
continue
}
feedBaseURL, err := url.Parse(fb)
if err != nil {
log.Printf("error: Cannot parse feed base URL '%s': %v\n", fb, err)
slog.Error("Cannot parse feed base URL", "url", fb, "err", err)
continue
}
res, err := afp.client.Get(feedURL.String())
if err != nil {
log.Printf("error: Cannot get feed '%s'\n", err)
slog.Error("Cannot get feed", "err", err)
continue
}
if res.StatusCode != http.StatusOK {
log.Printf("error: Fetching %s failed. Status code %d (%s)",
feedURL, res.StatusCode, res.Status)
slog.Error("Fetching failed",
"url", feedURL, "status_code", res.StatusCode, "status", res.Status)
continue
}
rfeed, err := func() (*ROLIEFeed, error) {
@ -294,7 +310,7 @@ func (afp *AdvisoryFileProcessor) processROLIE(
return LoadROLIEFeed(res.Body)
}()
if err != nil {
log.Printf("Loading ROLIE feed failed: %v.", err)
slog.Error("Loading ROLIE feed failed", "err", err)
continue
}
@ -306,17 +322,16 @@ func (afp *AdvisoryFileProcessor) processROLIE(
}
p, err := url.Parse(u)
if err != nil {
log.Printf("error: Invalid URL '%s': %v", u, err)
slog.Error("Invalid URL", "url", u, "err", err)
return ""
}
return feedBaseURL.ResolveReference(p).String()
}
rfeed.Entries(func(entry *Entry) {
// Filter if we have date checking.
if afp.AgeAccept != nil {
if pub := time.Time(entry.Published); !pub.IsZero() && !afp.AgeAccept(pub) {
if t := time.Time(entry.Updated); !t.IsZero() && !afp.AgeAccept(t) {
return
}
}
@ -347,10 +362,15 @@ func (afp *AdvisoryFileProcessor) processROLIE(
var file AdvisoryFile
if sha256 != "" || sha512 != "" || sign != "" {
file = HashedAdvisoryFile{self, sha256, sha512, sign}
} else {
file = PlainAdvisoryFile(self)
switch {
case sha256 == "" && sha512 == "":
slog.Error("No hash listed on ROLIE feed", "file", self)
return
case sign == "":
slog.Error("No signature listed on ROLIE feed", "file", self)
return
default:
file = PlainAdvisoryFile{self, sha256, sha512, sign}
}
files = append(files, file)

6
csaf/advisory.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>

9
csaf/cvss20enums.go
View file

@ -1,10 +1,5 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
//
// SPDX-License-Identifier: MIT
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
// SPDX-License-Identifier: BSD-3-Clause
// SPDX-FileCopyrightText: 2017 FIRST.ORG, INC.
//
// THIS FILE IS MACHINE GENERATED. EDIT WITH CARE!

9
csaf/cvss3enums.go
View file

@ -1,10 +1,5 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
//
// SPDX-License-Identifier: MIT
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
// SPDX-License-Identifier: BSD-3-Clause
// SPDX-FileCopyrightText: 2017 FIRST.ORG, INC.
//
// THIS FILE IS MACHINE GENERATED. EDIT WITH CARE!

6
csaf/doc.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>

38
csaf/generate_cvss_enums.go
View file

@ -1,9 +1,9 @@
//go:build ignore
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -14,21 +14,21 @@ import (
"bytes"
"encoding/json"
"flag"
"fmt"
"go/format"
"log"
"os"
"slices"
"regexp"
"sort"
"strings"
"text/template"
)
const tmplText = `// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
//
// SPDX-License-Identifier: MIT
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
// We from Intevation consider the source code parts in the following
// template file as too insignificant to be a piece of work that gains
// "copyrights" protection in the European Union. So the license(s)
// of the output files are fully determined by the input file.
const tmplText = `// {{ $.License }}
//
// THIS FILE IS MACHINE GENERATED. EDIT WITH CARE!
@ -69,6 +69,7 @@ type definition struct {
}
type schema struct {
License []string `json:"license"`
Definitions map[string]*definition `json:"definitions"`
}
@ -135,11 +136,24 @@ func main() {
defs = append(defs, k)
}
}
slices.Sort(defs)
sort.Strings(defs)
license := "determine license(s) from input file and replace this line"
pattern := regexp.MustCompile(`Copyright \(c\) (\d+), FIRST.ORG, INC.`)
for _, line := range s.License {
if m := pattern.FindStringSubmatch(line); m != nil {
license = fmt.Sprintf(
"SPDX-License-Identifier: BSD-3-Clause\n"+
"// SPDX-FileCopyrightText: %s FIRST.ORG, INC.", m[1])
break
}
}
var source bytes.Buffer
check(tmpl.Execute(&source, map[string]any{
"License": license,
"Prefix": *prefix,
"Definitions": s.Definitions,
"Keys": defs,

8
csaf/models.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -17,7 +17,7 @@ import (
"strings"
"time"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
// TLPLabel is the traffic light policy of the CSAF.

90
csaf/providermetaloader.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2023 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2023 Intevation GmbH <https://intevation.de>
@ -14,10 +14,11 @@ import (
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"strings"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
// ProviderMetadataLoader helps load provider-metadata.json from
@ -45,7 +46,7 @@ const (
// WellknownSecurityMismatch indicates that the PMDs found under wellknown and
// in the security do not match.
WellknownSecurityMismatch
// IgnoreProviderMetadata indicates that a extra PMD was ignored.
// IgnoreProviderMetadata indicates that an extra PMD was ignored.
IgnoreProviderMetadata
)
@ -108,8 +109,51 @@ func NewProviderMetadataLoader(client util.Client) *ProviderMetadataLoader {
}
}
// Load loads a provider metadata for a given path.
// If the domain starts with `https://` it only attemps to load
// Enumerate lists all PMD files that can be found under the given domain.
// As specified in CSAF 2.0, it looks for PMDs using the well-known URL and
// the security.txt, and if no PMDs have been found, it also checks the DNS-URL.
func (pmdl *ProviderMetadataLoader) Enumerate(domain string) []*LoadedProviderMetadata {
// Our array of PMDs to be found
var resPMDs []*LoadedProviderMetadata
// Check direct path
if strings.HasPrefix(domain, "https://") {
return []*LoadedProviderMetadata{pmdl.loadFromURL(domain)}
}
// First try the well-known path.
wellknownURL := "https://" + domain + "/.well-known/csaf/provider-metadata.json"
wellknownResult := pmdl.loadFromURL(wellknownURL)
// Validate the candidate and add to the result array
if wellknownResult.Valid() {
slog.Debug("Found well known provider-metadata.json")
resPMDs = append(resPMDs, wellknownResult)
}
// Next load the PMDs from security.txt
secResults := pmdl.loadFromSecurity(domain)
slog.Info("Found provider metadata results in security.txt", "num", len(secResults))
for _, result := range secResults {
if result.Valid() {
resPMDs = append(resPMDs, result)
}
}
// According to the spec, only if no PMDs have been found, the should DNS URL be used
if len(resPMDs) > 0 {
return resPMDs
}
dnsURL := "https://csaf.data.security." + domain
return []*LoadedProviderMetadata{pmdl.loadFromURL(dnsURL)}
}
// Load loads one valid provider metadata for a given path.
// If the domain starts with `https://` it only attempts to load
// the data from that URL.
func (pmdl *ProviderMetadataLoader) Load(domain string) *LoadedProviderMetadata {
@ -129,23 +173,12 @@ func (pmdl *ProviderMetadataLoader) Load(domain string) *LoadedProviderMetadata
// We have a candidate.
if wellknownResult.Valid() {
wellknownGood = wellknownResult
} else {
pmdl.messages.AppendUnique(wellknownResult.Messages)
}
// Next load the PMDs from security.txt
secResults := pmdl.loadFromSecurity(domain)
// Filter out the results which are valid.
var secGoods []*LoadedProviderMetadata
for _, result := range secResults {
if len(result.Messages) > 0 {
// If there where validation issues append them
// to the overall report
pmdl.messages.AppendUnique(pmdl.messages)
} else {
secGoods = append(secGoods, result)
}
}
secGoods := pmdl.loadFromSecurity(domain)
// Mention extra CSAF entries in security.txt.
ignoreExtras := func() {
@ -176,28 +209,31 @@ func (pmdl *ProviderMetadataLoader) Load(domain string) *LoadedProviderMetadata
}
}
// Take the good well-known.
wellknownGood.Messages.AppendUnique(pmdl.messages)
wellknownGood.Messages = pmdl.messages
return wellknownGood
}
// Don't have well-known. Take first good from security.txt.
ignoreExtras()
secGoods[0].Messages.AppendUnique(pmdl.messages)
secGoods[0].Messages = pmdl.messages
return secGoods[0]
}
// If we have a good well-known take it.
if wellknownGood != nil {
wellknownGood.Messages.AppendUnique(pmdl.messages)
wellknownGood.Messages = pmdl.messages
return wellknownGood
}
// Last resort: fall back to DNS.
dnsURL := "https://csaf.data.security." + domain
return pmdl.loadFromURL(dnsURL)
dnsURLResult := pmdl.loadFromURL(dnsURL)
pmdl.messages.AppendUnique(dnsURLResult.Messages) // keep order of messages consistent (i.e. last occurred message is last element)
dnsURLResult.Messages = pmdl.messages
return dnsURLResult
}
// loadFromSecurity loads the PMDs mentioned in the security.txt.
// loadFromSecurity loads the PMDs mentioned in the security.txt. Only valid PMDs are returned.
func (pmdl *ProviderMetadataLoader) loadFromSecurity(domain string) []*LoadedProviderMetadata {
// If .well-known fails try legacy location.
@ -316,7 +352,7 @@ func (pmdl *ProviderMetadataLoader) loadFromURL(path string) *LoadedProviderMeta
case len(errors) > 0:
result.Messages = []ProviderMetadataLoadMessage{{
Type: SchemaValidationFailed,
Message: fmt.Sprintf("%s: Validating against JSON schema failed: %v", path, err),
Message: fmt.Sprintf("%s: Validating against JSON schema failed", path),
}}
for _, msg := range errors {
result.Messages.Add(

6
csaf/remotevalidation.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

8
csaf/rolie.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -14,7 +14,7 @@ import (
"sort"
"time"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
// ROLIEServiceWorkspaceCollectionCategoriesCategory is a category in a ROLIE service collection.

2
csaf/schema/aggregator_json_schema.json
View file

@ -175,7 +175,7 @@
"type": "object",
"required": [
"metadata",
"mirror",
"mirrors",
"update_interval"
],
"properties": {

2
csaf/schema/cvss-v2.0.json.license Normal file
View file

@ -0,0 +1,2 @@
SPDX-License-Identifier: BSD-3-Clause
SPDX-FileCopyrightText: 2017 FIRST.ORG, INC.

2
csaf/schema/cvss-v3.0.json.license Normal file
View file

@ -0,0 +1,2 @@
SPDX-License-Identifier: BSD-3-Clause
SPDX-FileCopyrightText: 2017 FIRST.ORG, INC.

2
csaf/schema/cvss-v3.1.json.license Normal file
View file

@ -0,0 +1,2 @@
SPDX-License-Identifier: BSD-3-Clause
SPDX-FileCopyrightText: 2021 FIRST.ORG, INC.

8
csaf/summary.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>
@ -11,7 +11,7 @@ package csaf
import (
"time"
"github.com/csaf-poc/csaf_distribution/v3/util"
"github.com/gocsaf/csaf/v3/util"
)
const (

67
csaf/util.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -36,3 +36,64 @@ func ExtractProviderURL(r io.Reader, all bool) ([]string, error) {
}
return urls, nil
}
// CollectProductIdentificationHelpers returns a slice of all ProductIdentificationHelper
// for a given ProductID.
func (pt *ProductTree) CollectProductIdentificationHelpers(id ProductID) []*ProductIdentificationHelper {
var helpers []*ProductIdentificationHelper
pt.FindProductIdentificationHelpers(
id, func(helper *ProductIdentificationHelper) {
helpers = append(helpers, helper)
})
return helpers
}
// FindProductIdentificationHelpers calls visit on all ProductIdentificationHelper
// for a given ProductID by iterating over all full product names and branches
// recursively available in the ProductTree.
func (pt *ProductTree) FindProductIdentificationHelpers(
id ProductID,
visit func(*ProductIdentificationHelper),
) {
// Iterate over all full product names
if fpns := pt.FullProductNames; fpns != nil {
for _, fpn := range *fpns {
if fpn != nil &&
fpn.ProductID != nil && *fpn.ProductID == id &&
fpn.ProductIdentificationHelper != nil {
visit(fpn.ProductIdentificationHelper)
}
}
}
// Iterate over branches recursively
var recBranch func(b *Branch)
recBranch = func(b *Branch) {
if b == nil {
return
}
if fpn := b.Product; fpn != nil &&
fpn.ProductID != nil && *fpn.ProductID == id &&
fpn.ProductIdentificationHelper != nil {
visit(fpn.ProductIdentificationHelper)
}
for _, c := range b.Branches {
recBranch(c)
}
}
for _, b := range pt.Branches {
recBranch(b)
}
// Iterate over relationships
if rels := pt.RelationShips; rels != nil {
for _, rel := range *rels {
if rel != nil {
if fpn := rel.FullProductName; fpn != nil && fpn.ProductID != nil &&
*fpn.ProductID == id && fpn.ProductIdentificationHelper != nil {
visit(fpn.ProductIdentificationHelper)
}
}
}
}
}

182
csaf/util_test.go Normal file
View file

@ -0,0 +1,182 @@
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
package csaf
import (
"reflect"
"testing"
)
func TestProductTree_FindProductIdentificationHelpers(t *testing.T) {
type fields struct {
Branches Branches
FullProductNames *FullProductNames
RelationShips *Relationships
}
type args struct {
id ProductID
}
tests := []struct {
name string
fields fields
args args
want []*ProductIdentificationHelper
}{
{
name: "empty product tree",
args: args{
id: "CSAFPID-0001",
},
want: nil,
},
{
name: "product tree with matching full product names",
fields: fields{
FullProductNames: &FullProductNames{{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
}},
},
args: args{
id: "CSAFPID-0001",
},
want: []*ProductIdentificationHelper{{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
}},
},
{
name: "product tree with no matching full product names",
fields: fields{
FullProductNames: &FullProductNames{{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
}},
},
args: args{
id: "CSAFPID-0002",
},
want: nil,
},
{
name: "product tree with matching branches",
fields: fields{
Branches: Branches{{
Name: &[]string{"beta"}[0],
Product: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
},
Branches: Branches{{
Name: &[]string{"beta-2"}[0],
Product: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta-2:*:*:*:*:*:*"}[0],
},
},
}},
}},
},
args: args{
id: "CSAFPID-0001",
},
want: []*ProductIdentificationHelper{{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
}, {
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta-2:*:*:*:*:*:*"}[0],
}},
},
{
name: "product tree with no matching branches",
fields: fields{
Branches: Branches{{
Name: &[]string{"beta"}[0],
Product: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
},
Branches: Branches{{
Name: &[]string{"beta-2"}[0],
Product: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta-2:*:*:*:*:*:*"}[0],
},
},
}},
}},
},
args: args{
id: "CSAFPID-0002",
},
want: nil,
},
{
name: "product tree with matching relationships",
fields: fields{
RelationShips: &Relationships{{
FullProductName: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
},
}},
},
args: args{
id: "CSAFPID-0001",
},
want: []*ProductIdentificationHelper{{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
}},
},
{
name: "product tree with no matching relationships",
fields: fields{
RelationShips: &Relationships{{
FullProductName: &FullProductName{
ProductID: &[]ProductID{"CSAFPID-0001"}[0],
ProductIdentificationHelper: &ProductIdentificationHelper{
CPE: &[]CPE{"cpe:2.3:a:microsoft:internet_explorer:1.0.0:beta:*:*:*:*:*:*"}[0],
},
},
}},
},
args: args{
id: "CSAFPID-0002",
},
want: nil,
},
}
t.Parallel()
for _, testToRun := range tests {
test := testToRun
t.Run(test.name, func(tt *testing.T) {
tt.Parallel()
pt := &ProductTree{
Branches: test.fields.Branches,
FullProductNames: test.fields.FullProductNames,
RelationShips: test.fields.RelationShips,
}
if got := pt.CollectProductIdentificationHelpers(test.args.id); !reflect.DeepEqual(got, test.want) {
tt.Errorf("ProductTree.FindProductIdentificationHelpers() = %v, want %v",
got, test.want)
}
})
}
}

6
csaf/validation.go
View file

@ -1,7 +1,7 @@
// This file is Free Software under the MIT License
// without warranty, see README.md and LICENSES/MIT.txt for details.
// This file is Free Software under the Apache-2.0 License
// without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
//
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: Apache-2.0
//
// SPDX-FileCopyrightText: 2021 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
// Software-Engineering: 2021 Intevation GmbH <https://intevation.de>

2
docs/Development.md
View file

@ -3,7 +3,7 @@
## Supported Go versions
We support the latest version and the one before
the latest version of Go (currently 1.21 and 1.20).
the latest version of Go (currently 1.23 and 1.24).
## Generated files

22
docs/csaf_aggregator.md
View file

@ -16,6 +16,7 @@ Help Options:
```
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/aggregator.toml
~/.csaf_aggregator.toml
@ -25,6 +26,7 @@ csaf_aggregator.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Usage example for a single run, to test if the config is good:
```bash
./csaf_aggregator -c docs/examples/aggregator.toml
```
@ -62,7 +64,6 @@ SHELL=/bin/bash
30 0-23 * * * $HOME/bin/csaf_aggregator --config /etc/csaf_aggregator.toml --interim >> /var/log/csaf_aggregator/interim.log 2>&1
```
#### serve via web server
Serve the paths where the aggregator writes its `html/` output
@ -78,7 +79,6 @@ a template. For the aggregator the difference is that you can leave out
the cgi-bin part, potentially commend out the TLS client parts and
adjust the `root` path accordingly.
### config options
The config file is written in [TOML](https://toml.io/en/v1.0.0).
@ -118,10 +118,12 @@ Next we have two TOML _tables_:
aggregator // basic infos for the aggregator object
remote_validator // config for optional remote validation checker
```
[See the provider config](csaf_provider.md#provider-options) about
how to configure `remote_validator`.
At last there is the TOML _array of tables_:
```
providers // each entry to be mirrored or listed
```
@ -148,6 +150,9 @@ header
Where valid `name` and `domain` settings are required.
If no user agent is specified with `header = "user-agent:custom-agent/1.0"`
then the default agent in the form of `csaf_distribution/VERSION` is sent.
If you want an entry to be listed instead of mirrored
in a `aggregator.category == "aggregator"` instance,
set `category` to `lister` in the entry.
@ -170,14 +175,15 @@ The latter is evaluated as JSONPath and the result will be added into the
categories document. For a more detailed explanation and examples,
[refer to the provider config](csaf_provider.md#provider-options).
#### Example config file
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/examples/aggregator.toml) -->
<!-- The below code snippet is automatically added from ../docs/examples/aggregator.toml -->
```toml
workers = 2
folder = "/var/csaf_aggregator"
lock_file = "/var/csaf_aggregator/run.lock"
lock_file = "/var/lock/csaf_aggregator/lock"
web = "/var/csaf_aggregator/html"
domain = "https://localhost:9443"
rate = 10.0
@ -187,6 +193,7 @@ insecure = true
#interim_years =
#passphrase =
#write_indices = false
#time_range =
# specification requires at least two providers (default),
# to override for testing, enable:
@ -208,6 +215,7 @@ insecure = true
create_service_document = true
# rate = 1.5
# insecure = true
# time_range =
[[providers]]
name = "local-dev-provider2"
@ -217,7 +225,7 @@ insecure = true
write_indices = true
client_cert = "./../devca1/testclient1.crt"
client_key = "./../devca1/testclient1-key.pem"
# client_passphrase =
# client_passphrase = # Limited and experimental, see downloader doc.
# header =
[[providers]]
@ -226,13 +234,13 @@ insecure = true
# rate = 1.8
# insecure = true
write_indices = true
# If aggregator.category == "aggregator", set for an entry that should
# If aggregator.category == "aggreator", set for an entry that should
# be listed in addition:
category = "lister"
# ignore_pattern = [".*white.*", ".*red.*"]
```
<!-- MARKDOWN-AUTO-DOCS:END -->
<!-- MARKDOWN-AUTO-DOCS:END -->
#### Publish others' advisories

9
docs/csaf_checker.md
View file

@ -30,9 +30,12 @@ Help Options:
Will check all given _domains_, by trying each as a CSAF provider.
If no user agent is specified with `--header=user-agent:custom-agent/1.0` then the default agent in the form of `csaf_distribution/VERSION` is sent.
If a _domain_ starts with `https://` it is instead considered a direct URL to the `provider-metadata.json` and checking proceeds from there.
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/checker.toml
~/.csaf_checker.toml
@ -41,6 +44,7 @@ csaf_checker.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Supported options in config files:
```
output = ""
format = "json"
@ -61,6 +65,7 @@ Usage example:
`./csaf_checker example.com -f html --rate=5.3 -H apikey:SECRET -o check-results.html`
Each performed check has a return type of either 0,1 or 2:
```
type 0: success
type 1: warning
@ -73,13 +78,13 @@ The option `timerange` allows to only check advisories from a given time
interval. It can only be given once. See the
[downloader documentation](csaf_downloader.md#timerange-option) for details.
You can ignore certain advisories while checking by specifying a list
of regular expressions[^1] to match their URLs by using the `ignorepattern`
option.
E.g. `-i='.*white.*' -i='*.red.*'` will ignore files which URLs contain
the sub strings **white** or **red**.
In the config file this has to be noted as:
```
ignorepattern = [".*white.*", ".*red.*"]
```
@ -88,7 +93,7 @@ ignorepattern = [".*white.*", ".*red.*"]
The `role` given in the `provider-metadata.json` is not
yet considered to change the overall result,
see https://github.com/csaf-poc/csaf_distribution/issues/221 .
see <https://github.com/gocsaf/csaf/issues/221> .
If a provider hosts one or more advisories with a TLP level of AMBER or RED, then these advisories must be access protected.
To check these advisories, authorization can be given via custom headers or certificates.

17
docs/csaf_downloader.md
View file

@ -1,4 +1,5 @@
## csaf_downloader
A tool to download CSAF documents from CSAF providers.
### Usage
@ -21,6 +22,7 @@ Application Options:
-f, --folder=FOLDER Download into a given subFOLDER
-i, --ignore_pattern=PATTERN Do not download files if their URLs match any of the given PATTERNs
-H, --header= One or more extra HTTP header fields
--enumerate_pmd_only If this flag is set to true, the downloader will only enumerate valid provider metadata files, but not download documents
--validator=URL URL to validate documents remotely
--validator_cache=FILE FILE to cache remote validations
--validator_preset=PRESETS One or more PRESETS to validate remotely (default: [mandatory])
@ -29,9 +31,10 @@ Application Options:
--forward_header= One or more extra HTTP header fields used by forwarding
--forward_queue=LENGTH Maximal queue LENGTH before forwarder (default: 5)
--forward_insecure Do not check TLS certificates from forward endpoint
--logfile=FILE FILE to log downloading to (default: downloader.log)
--loglevel=LEVEL[debug|info|warn|error] LEVEL of logging details (default: info)
--log_file=FILE FILE to log downloading to (default: downloader.log)
--log_level=LEVEL[debug|info|warn|error] LEVEL of logging details (default: info)
-c, --config=TOML-FILE Path to config TOML file
--preferred_hash=HASH[sha256|sha512] HASH to prefer
Help Options:
-h, --help Show this help message
@ -39,6 +42,8 @@ Help Options:
Will download all CSAF documents for the given _domains_, by trying each as a CSAF provider.
If no user agent is specified with `--header=user-agent:custom-agent/1.0` then the default agent in the form of `csaf_distribution/VERSION` is sent.
If a _domain_ starts with `https://` it is instead considered a direct URL to the `provider-metadata.json` and downloading procedes from there.
Increasing the number of workers opens more connections to the web servers
@ -47,6 +52,7 @@ However, since this also increases the load on the servers, their administrators
have taken countermeasures to limit this.
If no config file is explictly given the follwing places are searched for a config file:
```
~/.config/csaf/downloader.toml
~/.csaf_downloader.toml
@ -56,6 +62,7 @@ csaf_downloader.toml
with `~` expanding to `$HOME` on unixoid systems and `%HOMEPATH` on Windows systems.
Supported options in config files:
```
# directory # not set by default
insecure = false
@ -90,6 +97,7 @@ option.
E.g. `-i='.*white.*' -i='*.red.*'` will ignore files which URLs contain
the sub strings **white** or **red**.
In the config file this has to be noted as:
```
ignorepattern = [".*white.*", ".*red.*"]
```
@ -106,6 +114,7 @@ into a given intervall. There are three possible notations:
and 'y' for years are recognized. In these cases only integer
values are accepted without any fractions.
Some examples:
- `"3h"` means downloading the advisories that have changed in the last three hours.
- `"30m"` .. changed within the last thirty minutes.
- `"3M2m"` .. changed within the last three months and two minutes.
@ -116,6 +125,7 @@ into a given intervall. There are three possible notations:
E.g. `"2006-01-02"` means that all files between 2006 January 2nd and now going to being
downloaded.
Accepted patterns are:
- `"2006-01-02T15:04:05Z"`
- `"2006-01-02T15:04:05+07:00"`
- `"2006-01-02T15:04:05-07:00"`
@ -134,6 +144,7 @@ into a given intervall. There are three possible notations:
All interval boundaries are inclusive.
#### Forwarding
The downloader is able to forward downloaded advisories and their checksums,
OpenPGP signatures and validation results to an HTTP endpoint.
The details of the implemented API are described [here](https://github.com/mfd2007/csaf_upload_interface).
@ -149,7 +160,7 @@ key protection mechanism based on RFC 1423, see
Thus it considered experimental and most likely to be removed
in a future release. Please only use this option, if you fully understand
the security implications!
Note that for fully automated processes, it usually not make sense
Note that for fully automated processes, it usually does not make sense
to protect the client certificate's private key with a passphrase.
Because the passphrase has to be accessible to the process anyway to run
unattented. In this situation the processing environment should be secured

18
docs/csaf_provider.md
View file

@ -4,7 +4,7 @@ The [setup docs](../README.md#setup-trusted-provider)
explain how to wire this up with nginx and where the config file lives.
When installed, two endpoints are offered,
and you should use the [csaf_uploader](../docs/csaf_uploader)
and you should use the [csaf_uploader](../docs/csaf_uploader.md)
to access them:
### /api/create
@ -100,22 +100,12 @@ The following example file documents all available configuration options:
#tlps = ["csaf", "white", "amber", "green", "red"]
# Make the provider create a ROLIE service document.
#create_service_document = true
#create_service_document = false
# Make the provider create a ROLIE category document from a list of strings.
# If a list item starts with `expr:`
# the rest of the string is used as a JsonPath expression
# to extract a string from the incoming advisories.
# If the result of the expression is a string this string
# is used. If the result is an array each element of
# this array is tested if it is a string or an array.
# If this test fails the expression fails. If the
# test succeeds the rules are applied recursively to
# collect all strings in the result.
# Suggested expressions are:
# - vendor, product family and product names: "expr:$.product_tree..branches[?(@.category==\"vendor\" || @.category==\"product_family\" || @.category==\"product_name\")].name"
# - CVEs: "expr:$.vulnerabilities[*].cve"
# - CWEs: "expr:$.vulnerabilities[*].cwe.id"
# Strings not starting with `expr:` are taken verbatim.
# By default no category documents are created.
# This example provides an overview over the syntax,
@ -151,5 +141,5 @@ contact_details = "Example Company can be reached at contact_us@example.com, or
There is an experimental upload interface which works with a web browser.
It is disabled by default, as there are known issues, notably:
* https://github.com/csaf-poc/csaf_distribution/issues/43
* https://github.com/csaf-poc/csaf_distribution/issues/256
* https://github.com/gocsaf/csaf/issues/43
* https://github.com/gocsaf/csaf/issues/256

6
docs/csaf_uploader.md
View file

@ -43,6 +43,12 @@ E.g. uploading a csaf-document
which asks to enter a password interactively.
To upload an already signed document, use the `-x` option
```bash
# Note: The file CSAF-document-1.json.asc must exist
./csaf_uploader -x -a upload -I -t white -u https://localhost/cgi-bin/csaf_provider.go CSAF-document-1.json
```
By default csaf_uploader will try to load a config file
from the following places:

10
docs/csaf_validator.md
View file

@ -2,6 +2,16 @@
is a tool to validate local advisories files against the JSON Schema and an optional remote validator.
### Exit codes
If no fatal error occurs the program will exit with an exit code `n` with the following conditions:
- `n == 0`: all valid
- `(n & 1) > 0`: a general error occurred, all other flags are unset (see logs for more information)
- `(n & 2) > 0`: schema validation failed
- `(n & 4) > 0`: no remote validator configured
- `(n & 8) > 0`: failure in remote validation
### Usage
```

2
docs/development-ca.md
View file

@ -55,7 +55,7 @@ signing_key
encryption_key
non_repudiation
dns_name = "*.local"
dns_name = "*.test"
dns_name = "localhost"
serial = 010

3
docs/examples/aggregator.toml
View file

@ -5,6 +5,7 @@ web = "/var/csaf_aggregator/html"
domain = "https://localhost:9443"
rate = 10.0
insecure = true
#verbose = false
#openpgp_private_key =
#openpgp_public_key =
#interim_years =
@ -51,7 +52,7 @@ insecure = true
# rate = 1.8
# insecure = true
write_indices = true
# If aggregator.category == "aggreator", set for an entry that should
# If aggregator.category == "aggregator", set for an entry that should
# be listed in addition:
category = "lister"
# ignore_pattern = [".*white.*", ".*red.*"]

7
docs/provider-setup.md
View file

@ -78,6 +78,9 @@ server {
# directory listings
autoindex on;
# allow others web applications to get the static information
add_header Access-Control-Allow-Origin "*";
}
# enable CGI
@ -115,7 +118,7 @@ sudo chmod g+r,o-rwx /etc/csaf/config.toml
Here is a minimal example configuration,
which you need to customize for a production setup,
see the [options of `csaf_provider`](https://github.com/csaf-poc/csaf_distribution/blob/main/docs/csaf_provider.md).
see the [options of `csaf_provider`](https://github.com/gocsaf/csaf/blob/main/docs/csaf_provider.md).
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/scripts/setupProviderForITest.sh&lines=94-101) -->
<!-- The below code snippet is automatically added from ../docs/scripts/setupProviderForITest.sh -->
@ -155,7 +158,7 @@ Again replacing `{clientCert.crt}` and `{clientKey.pem}` accordingly.
To let nginx resolves the DNS record `csaf.data.security.domain.tld` to fulfill the [Requirement 10](https://docs.oasis-open.org/csaf/csaf/v2.0/cs01/csaf-v2.0-cs01.html#7110-requirement-10-dns-path) configure a new server block (virtual host) in a separated file under `/etc/nginx/available-sites/{DNSNAME}` like following:
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/scripts/DNSConfigForItest.sh&lines=18-35) -->
<!-- MARKDOWN-AUTO-DOCS:START (CODE:src=../docs/scripts/DNSConfigForItest.sh&lines=18-37) -->
<!-- The below code snippet is automatically added from ../docs/scripts/DNSConfigForItest.sh -->
```sh
server {

4
docs/proxy-provider-for-aggregator.md
View file

@ -5,7 +5,9 @@ calls it a *CSAF publisher*.
After manually downloading the advisories from such a publisher,
the tools here can be used to offer the CSAF files for automated downloading
as *CSAF aggregator*. (The construct is called *CSAF proxy provider*. See [Section 7.2.5](https://docs.oasis-open.org/csaf/csaf/v2.0/csaf-v2.0.html#725-role-csaf-aggregator) for more details.)
as *CSAF aggregator*. (The construct is called *CSAF proxy provider*.
See [Section 7.2.5](https://docs.oasis-open.org/csaf/csaf/v2.0/os/csaf-v2.0-os.html#725-role-csaf-aggregator)
for more details.)
There are three necessary steps, easiest is to use
one single virtual maschine (or container) per internal provider.

8
docs/scripts/DNSConfigForItest.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
#
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -28,6 +28,8 @@ echo "
location = / {
try_files /.well-known/csaf/provider-metadata.json =404;
# allow others web applications to get the static information
add_header Access-Control-Allow-Origin "*";
}
access_log /var/log/nginx/dns-domain_access.log;

15
docs/scripts/Readme.md
View file

@ -1,6 +1,7 @@
Scripts for assisting the Integration tests. They are written on Ubuntu 20.04 TLS amd64.
Scripts for assisting the Integration tests.
They were written on Ubuntu 20.04 TLS amd64 and also tested with 24.04 TLS.
- `prepareUbunutForITest.sh` installs the required packages for the csaf_distribution integration tests on a naked ubuntu 20.04 LTS amd64.
- `prepareUbuntuInstanceForITests.sh` installs the required packages for the csaf integration tests on a naked Ubuntu LTS amd64.
- `TLSConfigsForITest.sh` generates a root CA and webserver cert by running `createRootCAForITest.sh` and `createWebserverCertForITest.sh`
and configures nginx for serving TLS connections.
@ -11,13 +12,13 @@ and configures nginx for serving TLS connections.
As creating the folders needs to authenticate with the csaf_provider, the configurations of TLS server and Client certificate authentication should be set. So it is recommended to call the scripts in this order: `TLSConfigsForITest.sh`, `TLSClientConfigsForITest.sh`, `setupProviderForITest.sh`
Calling example (as root):
Calling example (as user with sudo privileges):
``` bash
curl --fail -O https://raw.githubusercontent.com/csaf-poc/csaf_distribution/main/docs/scripts/prepareUbuntuInstanceForITests.sh
bash prepareUbuntuInstanceForITests.sh
curl --fail -O https://raw.githubusercontent.com/gocsaf/csaf/main/docs/scripts/prepareUbuntuInstanceForITests.sh
sudo bash prepareUbuntuInstanceForITests.sh
git clone https://github.com/csaf-poc/csaf_distribution.git # --branch <name>
pushd csaf_distribution/docs/scripts/
git clone https://github.com/gocsaf/csaf.git # --branch <name>
pushd csaf/docs/scripts/
export FOLDERNAME=devca1 ORGANAME="CSAF Tools Development (internal)"
source ./TLSConfigsForITest.sh

8
docs/scripts/TLSClientConfigsForITest.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -18,7 +18,7 @@ set -e
NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
cd ~/csaf_distribution/docs/scripts/
cd ~/csaf/docs/scripts/
source ./createCCForITest.sh
echo '

8
docs/scripts/TLSConfigsForITest.sh
View file

@ -1,7 +1,7 @@
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -17,7 +17,7 @@ set -e
NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
cd ~/csaf_distribution/docs/scripts/
cd ~/csaf/docs/scripts/
## Create Root CA
./createRootCAForITest.sh

6
docs/scripts/createCCForITest.sh
View file

@ -1,7 +1,7 @@
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

6
docs/scripts/createRootCAForITest.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

6
docs/scripts/createWebserverCertForITest.sh
View file

@ -1,7 +1,7 @@
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

6
docs/scripts/downloadExamples.sh
View file

@ -2,10 +2,10 @@
#
# Desc: Tries getting csaf 2.0 examples from api.github. Do not run too often!
#
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>

16
docs/scripts/prepareUbuntuInstanceForITests.sh
View file

@ -1,12 +1,12 @@
#!/usr/bin/env bash
set -e
# This script prepares a naked Ubuntu 20.04 LTS amd64
# for the csaf_distribution integration tests
# This script prepares a naked Ubuntu LTS amd64
# for the csaf integration tests
# by installing the required packages.
apt update
apt install -y make bash curl gnupg sed tar git nginx fcgiwrap gnutls-bin
apt install -y make bash curl gnupg sed tar git nginx fcgiwrap gnutls-bin zip
# Install Go from binary distribution
latest_go="$(curl https://go.dev/VERSION\?m=text| head -1).linux-amd64.tar.gz"
@ -14,19 +14,17 @@ curl -O https://dl.google.com/go/$latest_go
rm -rf /usr/local/go # be sure that we do not have an old installation
tar -C /usr/local -xzf $latest_go
# Install newer Node.js version from nodesource
# Install a current Node.js version from nodesource
# as needed for https://github.com/secvisogram/csaf-validator-service
# Instructions from
# https://github.com/nodesource/distributions/blob/master/README.md#debmanual
KEYRING=/usr/share/keyrings/nodesource.gpg
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | gpg --dearmor > "$KEYRING"
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor > "$KEYRING"
gpg --no-default-keyring --keyring "$KEYRING" --list-keys
chmod a+r /usr/share/keyrings/nodesource.gpg
VERSION=node_16.x
DISTRO="$(lsb_release -s -c)"
echo "deb [signed-by=$KEYRING] https://deb.nodesource.com/$VERSION $DISTRO main" | tee /etc/apt/sources.list.d/nodesource.list
echo "deb-src [signed-by=$KEYRING] https://deb.nodesource.com/$VERSION $DISTRO main" | tee -a /etc/apt/sources.list.d/nodesource.list
NODE_MAJOR=20
echo "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list
apt-get update
apt-get install -y nodejs

11
docs/scripts/setupProviderForITest.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
#
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -17,7 +17,7 @@ sudo chgrp -R www-data /var/www
sudo chmod -R g+ws /var/www
export NGINX_CONFIG_PATH=/etc/nginx/sites-available/default
export DNS_NAME=csaf.data.security.localhost
export DNS_NAME=csaf.data.security.test
sudo cp /usr/share/doc/fcgiwrap/examples/nginx.conf /etc/nginx/fcgiwrap.conf
@ -61,6 +61,9 @@ echo "
# directory listings
autoindex on;
# allow others web applications to get the static information
add_header Access-Control-Allow-Origin "*";
" > locationConfig.txt
sudo sed -i "/^\s*location \/ {/r locationConfig.txt" $NGINX_CONFIG_PATH # Insert config inside location{}
./DNSConfigForItest.sh

8
docs/scripts/setupValidationService.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
#
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -21,7 +21,7 @@ echo '
remote_validator= { "url" = "http://localhost:8082", "presets" = ["mandatory"], "cache" = "/var/lib/csaf/validations.db" }
' | sudo tee --append /etc/csaf/config.toml
npm install pm2 -g
sudo npm install pm2 -g
pushd ~
git clone https://github.com/secvisogram/csaf-validator-service.git

8
docs/scripts/testAggregator.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -29,6 +29,6 @@ popd
echo
echo '=== run aggregator'
cd ~/csaf_distribution/
cd ~/csaf/
sudo cp docs/examples/aggregator.toml /etc/csaf
sudo ./bin-linux-amd64/csaf_aggregator -c /etc/csaf/aggregator.toml

8
docs/scripts/testChecker.sh
View file

@ -1,9 +1,9 @@
#!/usr/bin/env bash
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
@ -11,7 +11,7 @@
set -e # to exit if a command in the script fails
echo '==== run checker (twice)'
cd ~/csaf_distribution
cd ~/csaf
./bin-linux-amd64/csaf_checker -f html -o ../checker-results.html --insecure \
--client_cert ~/devca1/testclient1.crt \

8
docs/scripts/testDownloader.sh
View file

@ -1,16 +1,16 @@
#!/usr/bin/env bash
# This file is Free Software under the MIT License
# without warranty, see README.md and LICENSES/MIT.txt for details.
# This file is Free Software under the Apache-2.0 License
# without warranty, see README.md and LICENSES/Apache-2.0.txt for details.
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
#
# SPDX-FileCopyrightText: 2022 German Federal Office for Information Security (BSI) <https://www.bsi.bund.de>
# Software-Engineering: 2022 Intevation GmbH <https://intevation.de>
set -e # to exit if a command in the script fails
cd ~/csaf_distribution
cd ~/csaf
echo
echo '==== run downloader (1)'

Some files were not shown because too many files have changed in this diff Show more