Signing Docker Images with Notary

This section describes the use of Unbound CORE ("CORE") with Docker Notary to enable Docker Content Trust ("DCT") to align with CIS Docker Benchmark and Docker Enterprise security best practices. DCT provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side verification for the Docker image integrity and the Docker image publisher identity. DCT works with The Update Framework ("TUF") to provide protection against man-in-the-middle, replay, and other attacks.

CORE can greatly help enterprise administrators to assert content integrity throughout development and production. In these stages, the signing keys are expected to spread through delegation to developers, sysadmins, and third-party ISVs. With CORE’s multi-party computation technology, DCT is extended to better protect cryptographic keys against malware or other forms of unauthorized use.

This document is intended for customers that want to protect the Notary root key with CORE.

The system architecture is shown in the following figure.

See Docker Notary Getting Started and Notary Advanced Usage for more information on some of the topics discussed here.

The following sections describe how to use CORE to protect the root key of a docker.io registry and notary service. This method can be used with any other registry, but we demonstrate the capability with docker.io since it is globally available and reflects a common usage scenario.

Instructions are provided for running on a CentOS/RedHat 7 platform.

Prerequisites

The following prerequisites are required to use the Unbound signing with CORE.

  • CORE (UKC) 2.0.1907 or newer
  • Client device, , with an OS listed in the System Requirements, and the following software:
    • CORE (UKC) client 2.0.1907 or newer
    • Notary client v0.6.1 (installation described below)
    • Docker CE and Docker CLIClosedCommand Line Interface (installation described below)
    • GOLANG 1.3 (installation described below)

Prepare the Client Environment

As part of the setup described here, it is required to install the Go language. Go is used to create a special notary client that supports PKCSClosedPublic-Key Cryptography Standards - Industry-standard cryptography specifications.#11, which is used instead of the default Notary client.

  1. Install the required packages.

    2. sudo yum install openssl wget git gcc
    sudo yum install epel-release -y
    sudo yum install jq -y

  2. Add the Docker CE repository.

    3. sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

  3. Install the Docker CLIClosedCommand Line Interface.

    4. sudo yum install -y docker-ce

  4. Install Go. See https://golang.org/dl/ for more information.

  5. Using Go, get the Notary client.

    go get github.com/theupdateframework/notary

  6. Using Go, create the Notary binary file with PKCSClosedPublic-Key Cryptography Standards - Industry-standard cryptography specifications.#11.

    go install -tags pkcs11 github.com/theupdateframework/notary/cmd/notary

    This command creates a binary file called notary in the folder /path/to/go/bin/.

  7. Copy the notary binary file to a location in your path, such as /usr/bin/.

    8. sudo cp notary /usr/bin/

  8. Test that the installation was successful by running this command:

    9. notary version

    Example response:

    notary
    Version:
    Git commit:
    Go version: go1.13.3

  9. Start Docker

    10. sudo service docker start

Configure CORE Client

  1. Follow the CORE Setup tasks.
  2. Connect the Notary client to the CORE client.

    3. sudo ln -s <path to libekmpkcs11.so>/libekmpkcs11.so /usr/lib64/libykcs11.so

    For the platform-dependent location of the libekmpkcs11.so, see Path to PKCS#11 Library.

    Notes
    1. After this step, PKCSClosedPublic-Key Cryptography Standards - Industry-standard cryptography specifications.#11 requests are redirected to the CORE server via libekmpkcs11.so.
    2. When listing keys in Notary, the location will display as yubikey, which is how Notary communicates with CORE.

  3. Configure OpenSSL on the CORE Client
    1. Configure OpenSSL to work with CORE by running this command:

      2. /etc/ekm/dy_openssl

    2. Check that CORE is configured to work with OpenSSL.

      3. openssl engine

      Verify that Dyadic Security engine support is listed in the response. For example:

      (rdrand) Intel RDRAND engine
      (dynamic) Dynamic engine loading support
      (dyadicsec) Dyadic Security engine support

Configure the OpenSSL Configuration File

Update the OpenSSL configuration file with the following changes in order to enable creation of repository certificates with OpenSSL.

  1. Download the openssl.cnf file from the Unbound GitHub repo.
  2. Move the file to this location:

    3. /root/.docker/certs/

    Warning
    It is recommended to make a backup of the current openssl.cnf file before overwriting it.

The new configuration file contains changes to these sections: CA_default, policy_match, v3_ca, repo_cert, and polsect.

Configure Notary in CORE Client

  1. Create a folder called ~/.notary.

    2. mkdir ~/.notary

  2. Create a file called config.json in that folder and edit it. Add the following contents.

    3. Notes
    1. The ca.pem file is created in the Create the Root Key section.
    2. The trust_pinning section should have an entry for every repository that you are working with.
    3. Update the contents with the relevant information, such as the username.

    {
      "trust_dir": "~/.docker/trust",
       "remote_server": {
        "url": "https://notary.docker.io"
      },
      "trust_pinning": {
        "ca": {
          "docker.io/{username}/nginx" : "/root/.docker/certs/ca.pem"
        },
        "disable_tofu": true
      }
    }

  3. Log into Docker Hub with your Docker username and password.

    4. sudo docker login -u <Docker username> -p <Docker password> docker.io

  4. A method is needed to examine the hash of the Docker image. You can use Docker manifest or skopeo.
    1. To use Docker manifest add the parameters experimental and debug to ~/.docker/config.json.

      2. {
        …
        "HttpHeaders": {
          "User-Agent": "Docker-Client/19.03.5 (linux)"
        },
        "experimental": "enabled",
        "debug": true
      }

      Note
      You must be running Docker CE to use manifest.

    2. To use skopeo, install the package.

      sudo yum install skopeo

  5. Export the following environment variables.

    6. export NOTARY_TARGETS_PASSPHRASE=<New password for target>
    export NOTARY_SNAPSHOT_PASSPHRASE=<New password for snapshot>
    export NOTARY_DELEGATION_PASSPHRASE='{"username": "docker", "password": "<CORE password for docker user>"}'
    export NOTARY_AUTH=$(echo "<username>:<password>" | base64)

Example of Using Notary

In this section, we do the following tasks for the example:

  1. Create a Self-Signed Certificate Authority
  2. Create the Root Key
  3. Initialize Repository on Notary
  4. Delegation and Image Signing
  5. Validate an Image

Create a Self-Signed Certificate Authority

You can create a self-signed certificate authority using CORE. In a production system, you might use your production CA.

On the CORE client:

  1. Create CA key.

    2. ucl generate -t ECC -n ca --user docker -w '<user-password>'

  2. Export the CA key.

    3. ucl export -u <CA UID> -o ~/.docker/certs/ca.key --obfuscate

  3. Self-sign the CA.

    4. openssl req -config ~/.docker/certs/openssl.cnf -key ~/.docker/certs/ca.key -new -x509 -days 7300 -sha256 -extensions v3_ca -out ~/.docker/certs/ca.pem -subj "/CN=Docker Notary CA"

  4. Import CA certificate

    5. ucl import --input ~/.docker/certs/ca.pem --password '<user-password>' --user docker --name ca

Create the Root Key

A key and certificate are required for this step. See here for more information. For example, to export the certificate and obfuscated CA key, you can use the following commands. We use the names ca.key and ca.pem since they are used to sign the root keys and the certificate is trusted by all clients. You can use the self-signed key and certificate created in the previous section.

The following procedure creates the root key in CORE. It is not exportable and therefore the key is never exposed. This key is used as the default root of trust for all your docker repositories.

On the CORE client:

  1. Create root key.

    2. ucl generate -t ECC -n root --user docker -w '<user-password>'

  2. Create root certificate request. You must use the subject CN=root to convey to the Notary that it is the root key.

    3. ucl csr -o ~/.docker/certs/root.csr --subject "CN=root" -n root --user docker --password '<user-password>' --format PEM

  3. Export root key.

    4. ucl export -u <ROOT UID> -o ~/.docker/certs/root.key --obfuscate

  4. Self-sign the root key.

    5. openssl x509 -req -sha256 -days 365 -in ~/.docker/certs/root.csr -signkey ~/.docker/certs/root.key -out ~/.docker/certs/root.crt

  5. Import the root certificate.

    6. ucl import --input ~/.docker/certs/root.crt --password '<user-password>' --user docker --name root

  6. List the keys.

    7. ucl list

    Example response:

    Partition 0 part1: 4 objects found
    Certificate      : UID=a7ef7a4669bc6dc0 Name="ca"
    Private ECC key : UID=b2be1a48d15b5650 Name="ca"
    Certificate     : UID=4d41e5b72ea4a9af Name="root"
    Private ECC key : UID=581085b99643923f Name="root"

  7. Check existing keys in Notary.

    8. notary key list

    You should see the root key under YubiKey.

    ROLE    GUN    KEY ID                          LOCATION
    ----    ---    ------                          --------
    root          e38202b6664ad57...5579707260b1b  yubikey
    ca             c666f638241127b...c8baedd3602b7  yubikey

    Note
    The location is listed as yubikey, which is how Notary communicates with CORE.

Initialize Repository on Notary

In this example, we use NGINX as the image that will get signed and validated.

  1. Get the image.

    2. docker pull nginx

  2. Tag the image.

    3. docker tag nginx:latest docker.io/{username}/nginx:latest

  3. Push the image to the local registry. The value for {username} is your user name on Docker Hub.

    4. docker push docker.io/{username}/nginx:latest

  4. Check existing image with Docker Trust.

    5. export DOCKER_CONTENT_TRUST=1
    docker trust inspect docker.io/{username}/nginx

    Example response:

    No signatures or cannot access docker.io/{username}/nginx

  5. Check existing image with Notary.

    6. notary list docker.io/{username}/nginx

    Example response:

    * fatal: notary.docker.io does not have trust data for docker.io/{username}/nginx

  6. Create repository certificate request.

    7. ucl csr -o ~/.docker/certs/nginx.csr --subject "CN=docker.io/{username}/nginx" -u <Root key UID> --user docker --password <user-password> --format PEM

  7. Create the following file:

    8. touch ~/.docker/certs/index.txt

  8. Create repository certificate.

    9. openssl ca -config ~/.docker/certs/openssl.cnf -create_serial -extensions repo_cert -days 375 -in ~/.docker/certs/nginx.csr -out ~/.docker/certs/nginx.pem

  9. Initialize repository on Notary.

    10. notary -D init docker.io/{username}/nginx --publish --rootcert ~/.docker/certs/nginx.pem

    Note
    If you defined NOTARY_DELEGATION_PASSPHRASE it uses that as the passphrase. If not, when asked for the password, it should be in the format:
    {"username": "docker", "password": "<docker password>"}

  10. Delegate snapshot key to Notary server.

    11. notary -D key rotate docker.io/{username}/nginx snapshot --server-managed

    We now have root and target keys for our repository and are ready to sign our first image.

Delegation and Image Signing

The following sections refer to a delegate, which is a different device from the client.

Prepare the Delegation Environment

Delegation can be used on any device. Run the following commands to prepare the device for usage.

  1. Add the Docker CE repository.

    2. sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

  2. Install the Docker CLIClosedCommand Line Interface.

    3. sudo yum install -y docker-ce

  3. To install Notary, use the following commands.

    4. Note
    We do not need to use the modified Notary client for PKCSClosedPublic-Key Cryptography Standards - Industry-standard cryptography specifications.#11. The default Notary client is sufficient.

    curl -L https://github.com/theupdateframework/notary/releases/download/v0.6.1/notary-Linux-amd64 -o notary
    chmod +x notary
    mv notary /usr/bin/

  4. Test that the installation was successful by running this command:

    5. notary version

    Example response:

    notary
    Version: 0.6.1
    Git commit: d6e1431f

  5. Start Docker

    6. sudo service docker start

  6. A method is needed to examine the hash of the Docker image. You can use Docker manifest or skopeo.
    1. To use Docker manifest add the parameters experimental and debug to ~/.docker/config.json. Docker manifest provides capabilities to examine the hash of the Docker image.

      2. {
        …
        "HttpHeaders": {
          "User-Agent": "Docker-Client/19.03.5 (linux)"
        },
        "experimental": "enabled",
        "debug": true
      }

      Note
      You must be running Docker CE to use manifest.

    2. To use skopeo, install the package.

      3. sudo yum install skopeo

Create Delegation Key

Delegations in Docker Content Trust (DCT) allow you to control who can and cannot sign an image tag. A delegation has a pair of private and public delegation keys. See here for more information.

Run the following steps on the device used for image signing using a delegated key.

  1. Create a folder called ~/.notary.

    2. mkdir ~/.notary

  2. Create a file called config.json in that folder and edit it. Add the following contents.

    3. Notes
    1. The ca.pem file is created in the Create the Root Key section.
    2. The trust_pinning section should have an entry for every repository that you are working with.

    {

      "trust_dir": "~/.docker/trust",

       "remote_server": {

        "url": "https://notary.docker.io"

      },

      "trust_pinning": {

        "ca": {

          "docker.io/{username}/nginx" : "/root/.docker/certs/ca.pem"

        },

        "disable_tofu": true

      }

    }

  3. Log into Docker Hub with your Docker username and password.

    4. sudo docker login -u <Docker username> -p <Docker password> docker.io

  4. Copy ca.pem from the Admin device to the delegation key owner device and put it in the following directory (create the folder if it does not exist).

    5. ~/.docker/certs/

  5. Check existing delegations.

    6. notary delegation list docker.io/{username}/nginx

    Example response:

    No delegations present in this repository

  6. The delegate (such as a developer) creates a delegation key and certificate on their own device. The public key is then given to the admin (target key owner).

    7. docker trust key generate <delegation-key-name> --dir ~/.docker/certs/

  7. Transfer the public key <delegation-key-name> from the delegate's device to the admin (target key owner) device. The key should be copied to:

    8. ~/.docker/certs/<delegation-key-name>

  8. The target key owner (i.e., the CORE client) adds delegation to the repository using the public key.

    9. notary -D delegation add docker.io/{username}/nginx targets/me ~/.docker/certs/<delegation-key-name> --all-paths -p

  9. Check existing delegations.

    10. notary delegation list docker.io/{username}/nginx

    Example response:

    ROLE          PATHS           KEY IDS                             THRESHOLD
    ----          -----           -------                             ---------
    targets/me    "" <all paths>  97eddf88e1f25d...dae4170f02c19a542  1

Sign an Image

The delegation key owner device can now sign an image on the developer's device.

  1. Get the latest Docker image.

    2. docker pull docker.io/{username}/nginx

  2. Check existing signatures with Notary.

    3. notary list docker.io/{username}/nginx

    Example response:

    No targets present in this repository.

  3. Get image manifest hash and size, which can be found using Docker manifest or by using skopeo.
    1. In Docker manifest, the hash is the digest field, excluding any prefix, such as sha256. The size is given in the size field.

      2. docker manifest inspect docker.io/{username}/nginx -v

      Example response:

      {
        "Ref": "docker.io/{username}/nginx:latest",
        "Descriptor": {
          "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
          "digest": "sha256:36b77d8bb27ffca25c7f6f53cadd059aca2747d46fb6ef34064e31727325784e",
          "size": 948,
          "platform": {
            "architecture": "amd64",
            "os": "linux"
          }
        },
      }

    2. In skopeo:

      • Get image manifest hash.

        • skopeo inspect --raw docker://docker.io/{username}/nginx:latest | sha256sum

      • Get image manifest size.

        • skopeo inspect --raw docker://docker.io/{username}/nginx:latest | wc -c

  4. Sign image using one of these methods:

    5. notary -D addhash -p docker.io/{username}/nginx latest "<MANIFEST_SIZE>" --sha256 "<MANIFEST_HASH>" -r targets/me

    Or:

    docker push docker.io/{username}/nginx:latest

  5. Check existing signatures with Docker.

    6. docker trust inspect docker.io/{username}/nginx

    Example response:

    [
      {
        "Name": "docker.io/{username}/nginx",
        "SignedTags": [],
        "Signers": [
          {
            "Name": "me",
            "Keys": [
              {
                "ID": "97eddf88e1f25d9b8cae81faccb79134b39d2484a1488dcdae4170f02c19a542"
              }
            ]
          }
        ],
        "AdministrativeKeys": [
          {
            "Name": "Root",
            "Keys": [
              {
                "ID": "38a49f18df670f9cfaad15c519da80cf2f1e5093550e02a11b167dfec8c47c3d"
              }
            ]
          },
          {
            "Name": "Repository",
            "Keys": [
              {
                "ID": "e80d2bb843ab0a79fe033c6100c7b6e5bd4c09ebb2517b882a5b6f3031c57bca"
              }
            ]
          }
        ]
      }
    ]

  6. Check existing signatures with Notary.

    7. notary list docker.io/{username}/nginx

    Example response:

    NAME      DIGEST                             SIZE (BYTES)    ROLE
    ----      ------                             ------------    ----
    latest    36b77d8bb27ffc...4e31727325784e    948             targets/me

Validate an Image

Validate using Docker manifest.

  1. Retrieve the hash and hash size in Notary.

    2. notary lookup docker.io/{username}/nginx latest

  2. Validate that the hash and size have the same values as those stored in Docker.

    3. For example:

    [root@client ~]# notary lookup docker.io/{username}/nginx latest
    latest sha256:36b77d8bb27ffca25c7f6f53cadd059aca2747d46fb6ef34064e31727325784e 948

    [root@client ~]# docker manifest inspect docker.io/{username}/nginx -v | jq .Descriptor.size
    948

    [root@client ~]# docker manifest inspect docker.io/{username}/nginx -v | jq .Descriptor.digest
    "sha256:36b77d8bb27ffca25c7f6f53cadd059aca2747d46fb6ef34064e31727325784e"

Validate using skopeo:

  1. Retrieve the hash and hash size in Docker and what is stored in Notary.

    2. notary lookup docker.io/{username}/nginx latest
    skopeo inspect --raw docker://docker.io/{username}/nginx:latest | wc -c
    skopeo inspect --raw docker://docker.io/{username}/nginx:latest | sha256sum

  2. Validate that they have the same values.

    3. For example:

    [root@client ~]# notary lookup docker.io/{username}/nginx latest
    latest sha256:36b77d8bb27ffca25c7f6f53cadd059aca2747d46fb6ef34064e31727325784e 948

    [root@client ~]# skopeo inspect --raw docker://docker.io/{username}/nginx:latest | wc -c
    948

    [root@client ~]# skopeo inspect --raw docker://docker.io/{username}/nginx:latest | sha256sum
    36b77d8bb27ffca25c7f6f53cadd059aca2747d46fb6ef34064e31727325784e -