Public data downloads
Accessing complete species metadata, analysis and FASTA downloads
In order to facilitate access to the Pathogenwatch public data sets, we have exported all the metadata and analysis CSVs, along with the assembled genome FASTAs, to a public "S3" bucket on DigitalOcean.
Species names contain characters that will need to be "URL encoded" for access. Examples of how to do this are given below.
<species name>__<tool name>.csv.gz
<species name>__fasta.zip
Click on the root bucket URL to view an XML text representation of all the available files.
Use
Ctrl-F/Cmd-F
to search the page with the name of the speciesCopy the root bucket URL into a new tab +
/
at the end of the URL and append the the contents of the Key
field (i.e. <Key>[file name]</Key>
and your browser should automatically download it (tested in Chrome)curl https://pathogenwatch-public.ams3.cdn.digitaloceanspaces.com | xq '.ListBucketResult.Contents[].Key'
jq
is a tool for parsing JSON files on the command line. It can also be easily installed on most systems.Substitute the name of the file you wish to download into the command below.
curl -O https://pathogenwatch-public.ams3.cdn.digitaloceanspaces.com/$( printf "Klebsiella pneumoniae__kleborate.csv.gz" | jq -sRr '@uri )'
The easiest tool for working with S3 buckets is the s3cmd tool. It supports browsing, downloading and syncing from S3 buckets in general.
s3cmd --host ams3.cdn.digitaloceanspaces.com --host-bucket "%(bucket)s.ams3.cdn.digitaloceanspaces.com" ls s3://pathogenwatch-public | sed -re 's,\s+, ,g' | cut -f 4- -d ' '
s3cmd --host ams3.cdn.digitaloceanspaces.com --host-bucket "%(bucket)s.ams3.cdn.digitaloceanspaces.com" get "s3://pathogenwatch-public/Klebsiella pneumoniae__kleborate.csv.gz"
This will download all the files into the current directory
s3cmd --host ams3.cdn.digitaloceanspaces.com --host-bucket "%(bucket)s.ams3.cdn.digitaloceanspaces.com" get s3://pathogenwatch-public/ --recursive
There are also libraries supporting the S3 API in most programming languages and computation platforms (i.e Nextflow).
Last modified 4mo ago