1
0
Fork 0
mirror of https://github.com/gocsaf/csaf.git synced 2025-12-22 11:55:40 +01:00

Add concurrent downloads to downloader. (#363)

* Add concurrent downloads to downloader.

* Moved to Go 1.20

* close files channel on producer side.

* Improve error handling

* New flag to ignore signature check results. Improve docs. Do not use number of CPUs to determine number of download workers.

* Set number of default workers in downloader to two.
This commit is contained in:
Sascha L. Teichmann 2023-05-02 10:10:12 +02:00 committed by GitHub
parent 91479c9912
commit f32fba683d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 175 additions and 87 deletions

View file

@ -9,9 +9,11 @@ csaf_downloader [OPTIONS] domain...
Application Options:
-d, --directory=DIR DIRectory to store the downloaded files in
--insecure Do not check TLS certificates from provider
--ignoresigcheck Ignore signature check results, just warn on mismatch
--version Display version of the binary
-v, --verbose Verbose output
-r, --rate= The average upper limit of https operations per second (defaults to unlimited)
-w, --worker=NUM NUMber of concurrent downloads (default: 2)
-H, --header= One or more extra HTTP header fields
--validator=URL URL to validate documents remotely
--validatorcache=FILE FILE to cache remote validations
@ -24,3 +26,8 @@ Help Options:
Will download all CSAF documents for the given _domains_, by trying each as a CSAF provider.
If a _domain_ starts with `https://` it is instead considered a direct URL to the `provider-metadata.json` and downloading procedes from there.
Increasing the number of workers opens more connections to the web servers
to download more advisories at once. This may improve the overall speed of the download.
However, since this also increases the load on the servers, their administrators could
have taken countermeasures to limit this.