close
Star Fork Issue

devsecops-20190809.3 - 371bd00

open

The Azure Kubernetes Workshop

Welcome to the Azure Kubernetes Workshop. In this lab, you’ll go through tasks that will help you master the basic and more advanced topics required to deploy an application to Kubernetes on Azure Kubernetes Service (AKS) and setup automated build, security scans, and deployments using Codefresh CI/CD and Aqua Security.

Some of the things you’ll be going through:

  • Kubernetes deployments, services and ingress
  • Deploying MongoDB using Helm
  • Azure Monitor for Containers, Horizontal Pod Autoscaler and the Cluster Autoscaler
  • Building CI/CD pipelines using Codefresh and Azure Container Registry
  • Setting up security scans using Aqua and enforcing policy

You can review the changelog for what has recently changed.

Prerequisites

Access the Azure Portal

Using the credentials provided by the lab environment screen, navigate to Azure Portal and authenticate with the provided username and password.

Hint If you already have an Azure account, you can use that, or to use the lab environment you’ll need to use a unique email address.

Lab environment credentials

Launch Cloud Shell

Azure Cloud Shell is a browser-based CLI tool integrated directly into the Azure portal. Cloud shell provides all of the tools you need to manage your Azure resources in a pre-configured, on-demand virtual machine.

You may launch Cloud Shell from the Azure Portal or open a new browser window and navigate to Azure Cloud Shell https://shell.azure.com.

Launch Cloud Shell

The first time you launch Cloud Shell you will be prompted to create a resource group, storage account, and Azure Files share. This is a one-time step and will be automatically attached for all sessions. A single file share can be used by both Bash and PowerShell in Cloud Shell.

Kubernetes basics

There is an assumption of some prior knowledge of Kubernetes and its concepts. If you are new to Kubernetes, the following documentation can take you quickly through the basic concepts required to understand how it works https://aka.ms/LearnAKS

If you are a more experienced Kubernetes developer or administrator, you may have a look at the best practices guide https://aka.ms/aks/bestpractices

Application overview

You will be deploying a simple microservice to Kubernetes that displays different colors. This application is easy to understand, will have a simple ingress, and will help us look for and deal with security vulnerabilities.

You can read more about the colors app here.

Tasks

Useful resources are provided to help you work through each task. To ensure you progress at a good pace ensure workload is divided between team members where possible. This may mean anticipating work that might be required in a later task.

Hint: If you get stuck, you can ask for help from the proctors. You may also choose to peek at the solutions.

Core tasks

You are expected to at least complete the Getting up and running section. This involves setting up a Kubernetes cluster, deploying the application containers from Docker Hub, setting up monitoring and scaling your application.

DevSecOps tasks

Once you’re done with getting up and running, you will move on to some DevSecOps challenges. Complete as many tasks as you can. You’ll set up a continuous integration and continuous delivery pipeline (CI/CD) for your application and use Helm for deploys. You will also scan your images for security issues.

Getting up and running

Deploy Kubernetes with Azure Kubernetes Service (AKS)

Azure has a managed Kubernetes service, AKS (Azure Kubernetes Service).

Deploy the latest Kubernetes version available in AKS

Hint Enable Kubernetes Role-based access control (RBAC), which provides fine-grained control over cluster resources when creating the cluster because you can’t enable it post cluster creation. RBAC enabled clusters by default have degraded Kubernetes Dashboard functionality. This is a good security practice because it avoids unintended privilege escalation.

Tasks

Deploy Kubernetes to Azure, using CLI or Azure portal using the latest Kubernetes version available in AKS

Create a Resource Group for your AKS cluster. When you joined the workshop, you should have been assigned a location for your AKS cluster. Use the command below, and make sure you specify your assigned region.

az group create --name akschallenge --location YOURREGION

Pick a random region

centralus
eastus
eastus2
westus
northcentralus
southcentralus

Create a new cluster using the latest version

az aks create --resource-group akschallenge --name akschallenge \
    --kubernetes-version 1.13.5 --generate-ssh-keys

Important: For the lab, you’ll need to use an alternate command to create the cluster with your existing Service Principal passing in the Application Id and the Application Secret Key.

az aks create --resource-group akschallenge --name akschallenge \
  --enable-addons monitoring,http_application_routing \
  --kubernetes-version 1.13.5 --generate-ssh-keys \
  --service-principal <APP_ID> --client-secret <APP_SECRET>

Ensure you and your colleagues can connect to the cluster using kubectl

Authenticate with Azure and obtain a kubeconfig file with credentials to access the cluster

az aks get-credentials --resource-group akschallenge --name akschallenge

Check cluster connectivity by listing the nodes in your cluster

kubectl get nodes

Resources

Create private highly available container registry

Instead of using the public Docker Hub registry, create your own private container registry using Azure Container Registry (ACR).

Tasks

Create an Azure Container Registry (ACR)

export ACR_NAME=akschallenge${RANDOM}$$; echo $ACR_NAME
az acr create --resource-group akschallenge --name ${ACR_NAME} --sku Standard
Grant AKS access to Azure Container Registry

Authorize the AKS cluster to connect to the Azure Container Registry using the AKS cluster’s Service Principal.

Follow the Azure docs to learn how to grant access using Azure Active Directory Service Principals.

export AKS_RESOURCE_GROUP=akschallenge
export AKS_CLUSTER_NAME=akschallenge

# Get the id of the service principal configured for AKS
CLIENT_ID=$(az aks show --resource-group $AKS_RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "servicePrincipalProfile.clientId" --output tsv)

# Get the ACR registry resource id
ACR_ID=$(az acr show --name $ACR_NAME --resource-group $AKS_RESOURCE_GROUP --query "id" --output tsv)

# Create role assignment
az role assignment create --assignee $CLIENT_ID --role acrpull --scope $ACR_ID

Resources

Install Helm

To simplify application management in Kubernetes we’re going to use Helm. Helm is a Kubernetes application package manager.

Install Helm on the AKS cluster

Tiller, the server-side component with which Helm communicates, needs to use a ServiceAccount to authenticate to your AKS cluster. For this lab, the ServiceAccount will have full cluster access:

Save the YAML below as helm-rbac.yaml or download it from helm-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

And create the ServiceAccount and RBAC roles using kubectl apply

kubectl apply -f helm-rbac.yaml

Use helm init to install Tiller

helm init --service-account tiller

Optional: To verify Helm has installed correctly use

helm version

This will show a server version which means Helm is up and running.

Resources

Deploy the colors application

We’re going to deploy and expose our color-coded application which requires an external endpoint exposed on port 80.

Environmental variables

The color application takes an environmental variable to define the color that will be displayed.

  • COLOR=[color code, hex, or name]

Tasks

Provision color-coded app

Deployment

Save the YAML below as blue.deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: colors
  name: colors
spec:
  replicas: 3
  selector:
    matchLabels:
      app: "colors"
  template:
    metadata:
      labels:
        app: colors
    spec:
      containers:
      - env:
        - name: COLOR
          value: '#44B3C2'
        image: containers101/color-coded:master
        imagePullPolicy: Always
        name: colors
        ports:
        - containerPort: 8080
          protocol: TCP

Deploy the capture order application using kubectl apply

kubectl apply -f blue.deployment.yaml

Verify the pods are up and running

kubectl get deployments

This should show the colors deployment with three pods available and up-to-date.

Service

Kubernetes uses services to expose and loadbalance deployments.

Save the yaml below as blue.svc.yaml.

apiVersion: v1
kind: Service
metadata:
  name: colors
spec:
  type: LoadBalancer
  selector:
    app: colors
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Now create the service with kubectl

kubectl apply -f blue.svc.yaml

Because we used LoadBalancer for our service type, AKS will provision an external ip address we can use to access our application. You retrieve this ip address with this command:

kubectl get service colors

Provisioning an external ip can take a minute or two but once it does loads under the EXTERNAL-IP column you can open that ip in your browser to see the application.

Watch Kubernetes keep your app running

The colors app is special because when you open it in your browser and click anywhere it causes the pod to die. This is great for demostrating how AKS handles failover between pods and restarts. The best way to see this in action is to open two browser windows side by side. One with your app, and one with our Azure cloud shell.

In your cloud shell use this command to watch the status of your running pods

watch ubectl get pods -l app=colors

Hint To get out of a watch, use ctrl-c

Then, with your browser, click anywhere and watch the pod die and be restarted in your cloud window. Click back on your browser to get back to your running application.

Everytime you kill a pod, Kubernetes will wait longer and longer between pod restarts.

Resources

Deploy the Aqua Cloud-native Security Platform

We will be deploying Aqua Cloud-native Security Platform (CSP) in your Azure AKS environment using kubectl commands and YAML files (though Aqua can also provide Helm charts for deploying Aqua CSP).

Deploy the Aqua Server, Database, and Gateway

Step 1

Create the aqua-security namespace

Namespaces are used to segment secrets and services for security purposes.

To create a namespace for aqua, run this command:

kubectl create ns aqua-security

You should see the following message in your console:

namespace/aqua-security created

Step 2

Create the aqua-registry secret

We need to create a secret for accessing Aqua’s private registry to pull Aqua’s product images.

This secret will be referenced in the YAML files used to deploy the Aqua CSP components.

This lab will use the following temporary credentials for the secret:

  • Username: mscodeaqua19@gmail.com
  • Password: aqua1234

To create the secret, run this command:

kubectl create secret docker-registry aqua-registry --docker-server=registry.aquasec.com --docker-username=mscodeaqua19@gmail.com --docker-password=aqua1234 --docker-email=no@email.com -n aqua-security

You should see the following message in your console:

 secret/aqua-registry created

Step 3

Create the aqua-db secret

We need to create a secret for connecting to the Aqua CSP database.

This secret will be referenced in the YAML files used to deploy the Aqua CSP components.

To create the secret, run this command:

kubectl create secret generic aqua-db --from-literal=password=password1 -n aqua-security

You should see the following message in your console:

secret/aqua-db created

Step 4

Create the aqua service account

We need to create a service account that will use the aqua-registry secret to pull images from the Aqua private registry.

Using an editor of your choice (e.g. vi, nano, etc.), create a file named aqua-sa.yaml, and insert the following YAML:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: aqua
imagePullSecrets:
- name: aqua-registry

Save the file, and run this command:

kubectl create -n aqua-security -f aqua-sa.yaml 

You should see the following message in your console:

serviceaccount/aqua created

Step 5

Create the aqua-web, aqua-gateway and aqua-db services and deployments

Using an editor of your choice (e.g. vi, nano, etc.), create a file named aqua-csp.yaml, and insert the following YAML:

apiVersion: v1
kind: Service
metadata:
  name: aqua-db
  labels:
    app: aqua-db
spec:
  ports:
    - port: 5432
  selector:
    app: aqua-db
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: aqua-db
spec:
  template:
    metadata:
      labels:
        app: aqua-db
      name: aqua-db
    spec:
      serviceAccount: aqua
      containers:
      - name: aqua-db
        image: registry.aquasec.com/database:4.2
        env:
          - name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                name: aqua-db
                key: password
        volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: postgres-db
        ports:
        - containerPort: 5432
      volumes:
        - name: postgres-db
          hostPath:
            path: /var/lib/aqua/db
---
apiVersion: v1
kind: Service
metadata:
  name: aqua-gateway
  labels:
    app: aqua-gateway
spec:
  ports:
    - port: 3622
  selector:
    app: aqua-gateway
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: aqua-gateway
spec:
  template:
    metadata:
      labels:
        app: aqua-gateway
      name: aqua-gateway
    spec:
      serviceAccount: aqua      
      containers:
      - name: aqua-gateway
        image: registry.aquasec.com/gateway:4.2
        env:
          - name: SCALOCK_GATEWAY_PUBLIC_IP
            value: aqua-gateway
          - name: AQUA_CONSOLE_SECURE_ADDRESS
            value: aqua-web:443
          - name: SCALOCK_DBUSER
            value: "postgres"
          - name: SCALOCK_DBPASSWORD
            valueFrom: 
              secretKeyRef:
                name: aqua-db
                key: password
          - name: SCALOCK_DBNAME
            value: "scalock"
          - name: SCALOCK_DBHOST
            value: aqua-db
          - name: SCALOCK_DBPORT
            value: "5432"
          - name: SCALOCK_AUDIT_DBUSER
            value: "postgres"
          - name: SCALOCK_AUDIT_DBPASSWORD
            valueFrom: 
              secretKeyRef:
                name: aqua-db
                key: password
          - name: SCALOCK_AUDIT_DBNAME
            value: "slk_audit"
          - name: SCALOCK_AUDIT_DBHOST
            value: aqua-db
          - name: SCALOCK_AUDIT_DBPORT
            value: "5432"
        ports:
        - containerPort: 3622
---
apiVersion: v1
kind: Service
metadata:
  name: aqua-web
  labels:
    app: aqua-web
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 8443
      name: aqua-web-ssl
    - port: 8080
      protocol: TCP
      targetPort: 8080
      name: aqua-web

  selector:
    app: aqua-web
  type: LoadBalancer    
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: aqua-web
spec:
  template:
    metadata:
      labels:
        app: aqua-web
      name: aqua-web
    spec:
      serviceAccount: aqua
      containers:
      - name: aqua-web
        image: registry.aquasec.com/console:4.2
        env:
          - name: SCALOCK_DBUSER
            value: "postgres"
          - name: SCALOCK_DBPASSWORD
            valueFrom:
              secretKeyRef:
                name: aqua-db
                key: password
          - name: SCALOCK_DBNAME
            value: "scalock"
          - name: SCALOCK_DBHOST
            value: aqua-db
          - name: SCALOCK_DBPORT
            value: "5432"
          - name: SCALOCK_AUDIT_DBUSER
            value: "postgres"
          - name: SCALOCK_AUDIT_DBPASSWORD
            valueFrom: 
              secretKeyRef:
                name: aqua-db
                key: password
          - name: SCALOCK_AUDIT_DBNAME
            value: "slk_audit"
          - name: SCALOCK_AUDIT_DBHOST
            value: aqua-db
          - name: SCALOCK_AUDIT_DBPORT
            value: "5432"
        volumeMounts:
          - mountPath: /var/run/docker.sock
            name: docker-socket-mount
        ports:
        - containerPort: 8080
      volumes:
        - name: docker-socket-mount
          hostPath:
            path: /var/run/docker.sock

Save the file, and run this command:

kubectl create -n aqua-security -f aqua-csp.yaml 

You should see the following messages in your console:

service/aqua-db created
deployment.extensions/aqua-db created
service/aqua-gateway created
deployment.extensions/aqua-gateway created
service/aqua-web created
deployment.extensions/aqua-web created

Step 6

Monitor the availability of the Aqua pods and services

Run the following command to monitor the availability of the pods, and wait until all three pods are running (use CTRL+C to stop watching the pods):

kubectl get po -n aqua-security -w

Your output should be similar to this:

NAME                            READY   STATUS    RESTARTS   AGE
aqua-db-6847fdf9c6-mchgk        1/1     Running   0          114s
aqua-gateway-79898d9bfd-c6r2s   1/1     Running   0          114s
aqua-web-5fcb74d849-jrb82       1/1     Running   0          114s

Run the following command to monitor the availability of the services, and wait until the aqua-web service has been assigned an EXTERNAL-IP (use CTRL+C to stop watching the services):

kubectl get svc -n aqua-security -w

Your output should be similar to this:

NAME           TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                        AGE
aqua-db        ClusterIP      10.0.192.130   <none>           5432/TCP                       6m16s
aqua-gateway   ClusterIP      10.0.161.24    <none>           3622/TCP                       6m16s
aqua-web       LoadBalancer   10.0.108.155   52.168.170.202   443:31122/TCP,8080:30975/TCP   6m16s

Step 7

Connect to the Aqua web console

Open your web browser and navigate to the IP address which we got from the Step 6 above and port 8080. For example:

http://52.168.170.202:8080

When you access the Aqua Server for the first time, you must enter and confirm the password for the administrator username. The password must be at least 8 characters long. This is what you should see:

Aqua Output

The next screen will ask you to enter your Aqua License Token.

Aqua Output

Copy and paste the following License Token into Aqua Web Console.

jbRuYprRugrO-AlLAnWI54wUTlMU4YdY-nfUzn9vRKtUaMyddJ2DngKEZ4764xesKyWgybNczqFetCzoczOoBhkcLHuGj38rEybZIaquNhgqTWmA1ajB_GaSh5aXs8wzDltaVi47IwRxM0PCYQALtE8FeTE4P7UFPW8QAJRA965tHxjIQGLtFRDTHZH0qrxv4AcID-fk4l0tZrKUjAzYQUH93AvPiR9dKDpOD4RDujOJbMjI76QkBTMXM6czmnv18llboJ2JzRCPPftVfOE3fHIQqPLY3R1dPYGpPHeHvSTNVJ-H0IL5lfurmNUnFQRCS65IRlPaFtBAANZNh9Js1y25e4b5DLOWd9eWLjw1lKbkRnnTDEoeA34ehTEBErIcxEgwoju-POqPkEUj9Noafgydcx82rWRyAC4zSO78T4nKJ6br2xbzA-w9xLTPcFR51w3bWLN-cz5d6EgAwe_51dfQ0PPvPEh2_XYxRaleP-KXY0Y-QRcJ6of1ldYNSkInTLymlb1fl2rXwMc6a6v3YDqHYBO8hA2_j7ryF9fSFKh35lF6uypdn99gYQ== 

Great! Now you are now logged in as Administrator in your very own Aqua CSP web console!

Step 8

Deploy the Aqua Enforcers

To complete the deployment, and to use the Aqua CSP platform to its fullest, we will need to deploy the Aqua Enforcer components to your AKS nodes. The enforcers will monitor for and protect your running containers from “bad things” happening, as we will demostrate later on in the session.

This step will deploy the Aqua Enforcer by using a Kubernetes DaemonSet, which will automatically deploy a single Aqua Enforcer container on each node in your cluster.

Create an Enforcer Group

An Enforcer Group is a set of zero or more Aqua Enforcers with the same configuration. They are environment specific. We need to create one that will work with your AKS environment.

  1. In the Aqua web console, in the navigation menu on the left, click on Enforcers.
  2. Click the Add Enforcer Group button.
  3. For Group Name, enter: AKSWorkshop
  4. For Orchestrator, select: Kubernetes
  5. For Service Account, enter: aqua
  6. For Namespace, enter: aqua-security
  7. For Installation Token, enter: aqua
  8. Under Security Settings, for Enforcement Mode, select: Enforce
  9. Check all options under Auditing, Container Protection, Host Protection and Advanced Settings.
  10. Click the Create Group button.

On the Install Command page, you can copy the YAML by scrolling down and clicking the Copy to Clipboard button below the YAML text field, or you can copy it from below:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: aqua-agent
  namespace: aqua-security
spec:
  template:
    metadata:
      labels:
        app: aqua-agent
      name: aqua-agent
    spec:
      serviceAccount: aqua
      hostPID: true
      containers:
      - name: aqua-agent
        image: registry.aquasec.com/enforcer:4.2
        securityContext:
          privileged: true
        env:
        - name: AQUA_TOKEN
          value: "aqua"
        - name: AQUA_SERVER
          value: aqua-gateway:3622
        - name: AQUA_LOGICAL_NAME
          value: ""
        - name: RESTART_CONTAINERS
          value: "no"
        volumeMounts:
        - mountPath: /var/run
          name: var-run
        - mountPath: /dev
          name: dev
        - mountPath: /host/sys
          name: sys
          readOnly: true
        - mountPath: /host/proc
          name: proc
          readOnly: true
        - mountPath: /host/etc
          name: etc
          readOnly: true
        - mountPath: /host/opt/aquasec
          name: aquasec
          readOnly: true
        - mountPath: /opt/aquasec/tmp
          name: aquasec-tmp
        - mountPath: /opt/aquasec/audit
          name: aquasec-audit
      volumes:
      - name: var-run
        hostPath:
          path: /var/run
      - name: dev
        hostPath:
          path: /dev
      - name: sys
        hostPath:
          path: /sys
      - name: proc
        hostPath:
          path: /proc
      - name: etc
        hostPath:
          path: /etc
      - name: aquasec
        hostPath:
          path: /opt/aquasec
      - name: aquasec-tmp
        hostPath:
          path: /opt/aquasec/tmp
      - name: aquasec-audit
        hostPath:
          path: /opt/aquasec/audit

At the bottom of the Install Command page, click the Close link.

Using an editor of your choice (e.g. vi, nano, etc.), create a file named aqua-enforcer.yaml, and insert the YAML content that you copied from above.

Run the following command:

kubectl create -n aqua-security -f aqua-enforcer.yaml

You should see the following message in your console:

daemonset.extensions/aqua-agent created

You can monitory the availability of the Enforcer (a.k.a. aqua-agent) with this command (use CTRL+C to stop watching the pods):

kubectl get po -n aqua-security -w

A short time after the pods are running, you can refresh the Enforcers page in the Aqua web console to see the new Enforcers:

Aqua Output

Congratulations. You have deployed Aqua CSP.

DevSecOps tasks

Continuous Integration and Continuous Delivery

Your development team are making an increasing number of modifications to your application code. It is no longer feasible to manually deploy updates.

You are required to create a robust DevOps pipeline supporting CI/CD to deploy code changes. Codefresh is a CI/CD platform that makes working with containers and Kubernetes/Helm very easy. You will use the Codefresh platform to setup pipelines for automated deployments. Codefresh offers free accounts in the cloud, which are fully functional and can be connected with any Git repository and any Kubernetes cluster.

Hint Make sure you tokenize the Docker image tags in your Kubernetes YAML configuration files instead of using latest. You’ll need to set those in the build pipeline to the Build ID.

Tasks

First you need to sign-up for a Codefresh account (using your git provider as an authentication mechanism)

Create a Codefresh account

You can create a free Codefresh account (free forever and fully functional) by signing up at https://g.codefresh.io/signup. You can use any of the supported providers shown such as Azure or Github.

Codefresh signup

You will be asked to confirm the permissions requests for your GIT provider. Codefresh does not commit anything in your GIT repositories (it only reads them to check out code) and the privileges needed are mostly for automatic triggers (i.e. starting a pipeline when a commit happens).

You will also be asked to provide a username for your Codefresh account. Once that is done you will see the main Codefresh User Interface.

Hint Codefresh supports multi-login accounts. You can signup/login with multiple git providers and if the email used is the same, you will always reach the same Codefresh account.

For more information see the create account documentation.

Fork the source repositories on GitHub

The repository of the application that you are going to use is located at https://github.com/todaywasawesome/color-coded/tree/devsecops.

Hint Make sure you’re using the devsecops branch.

Login into your Github account and then navigate to the repository URL. Click the Fork button on the top right of the page:

Fork git repository

Github will start the forking process, so that all files are available to your account as well.

Import repository to Azure Repos

Once this is finished, you will see a “copy” of the git repository in your own account.

Connect your Azure cluster to Codefresh

You will now connect your Kubernetes cluster in Codefresh. This way you will extend the pipeline with deployment capabilities in the next section.

Go to your account integrations by selecting Account Settings on the left sidebar and then clicking on Configure next to “Kubernetes”.

Cluster integrations

From the cluster screen, select “Azure AKS” from the drop-down and authenticate with your Azure account. You will get a new dialog with your subscription and cluster selection:

Add Azure cluster

Hint You can also add your Azure cluster as a custom cluster by selecting Azure Custom.

Once the integration is complete, Codefresh will scan the cluster to finds its nodes. To verify the connection go back to the main Codefresh screen (click on the left arrow on the top left of the screen) and then select Kubernetes from the left sidebar.

You will see a view of all your namespaces in the cluster.

View Azure K8s cluster

Codefresh is now connected to your cluster.

Hint Make sure to note down the name your cluster as it appears in the dashboard, as you will use it later on in the deployment pipeline.

Connect the Azure Container Registry in Codefresh

The built-in Docker registry that Codefresh offers in all accounts works fine on its own. However, you have already setup secrets in the cluster for the Azure Container registry so it would make sense to change the pipeline to also push images there as well.

In the Codefresh UI, select Account Settings on the left sidebar and then click on Configure next to Docker Registry.

Codefresh Integrations

Click the Add Registry dropdown and then select Other Registries.

Fill in the details for the Azure Container registry registry with the following information:

  • Registry name: myazureregistry (user defined - lowercase)
  • Username: Your service principal ID (called Application/Client Id in your workshop details)
  • Password: Your service principal password (called Application Secret Key in your workshop details)
  • Domain: akschallengeXXXXXXXX.azurecr.io

Where XXXXXXXX is the random number assigned to your Docker registry during creation. Use echo $ACR_NAME in your cloud shell to find the exact name.

Hint If you can’t find the url for the registry, goto https://portal.azure.com/

Integrating Azure Registry

Click the Test button to make sure that your credentials are correct and finally the Save button to apply your changes. Codefresh is now connected to your Azure Container registry and can pull images from it.

For more information see the documentation on external registries and custom registries.

Create our CI/CD pipeline

As a first step for Continuous Integration you will create a pipeline that automatically builds a Docker image for the sample application, everytime a commit happens.

Go back to your Codefresh account and select Projects from the left sidebar. Click New Project

Name it color-coded and add a tag devsecops. You can also pick an icon. Projects are a way to organize pipelines, secrets, permissions and other resources.

Create a new project

Then add a new pipeline called color-coded

You will see a repository browser. Make sure that your git provider is selected on the right drop-down menu.

Select sample repository

Click create. Codefresh will generate a new pipeline for us which is very helpful but let’s build it up one step at a time. Clear the code there and start with this:

Setup Pipeline Stages
version: '1.0'
stages:
  - prepare
  - build
  - deploy
steps:

This will setup our pipeline with 3 stages which we’ll use to organize our pipeline.

Add a Git clone step

Next add this code for the Git clone step. Notice that we’re using variables like $ which will be filled in automatically by the webhook.

Hint Codefresh pipelines use YAML which requires proper indenting.


  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: prepare
 
Add build step

The build step uses another built-in step.

Hint Codefresh uses container-based pipelines which means every single step in a Codefresh pipeine uses separate containers. By default, a shared volume is attached to every step so we can access data between steps. In the previous step, we did a git clone, that code is now available for our build step.


  build:
    title: "Building Docker Image"
    type: "build"
    image_name: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
    tag: "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
    dockerfile: "Dockerfile"
    stage: "build"  

Again, here we make liberal use of variables.

Push image to ACR

Every image we build will be automatically saved in Codefresh’s built in registry. We’re using ACR for our production registry and so we’ll want to promote this image there.


  push:
    title: "Pushing image to Azure registry"
    type: "push"
    stage: deploy
    image_name: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
    registry: "myazureregistry"
    candidate: "${{build}}"
    tags:
      - "${{CF_BRANCH_TAG_NORMALIZED}}"
      - "${{CF_REVISION}}"
      - "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"

For more details see the Push step documentation.

Create pull secret

To pull the image we built and pushed to ACR, we’ll need a pull secret. This allows the cluster to pull private images. While this only needs to be done once, you can add this to your pipeline for better portability.

Hint Don’t forget to replace the cluster name with the one you created.


  createpullsecret:
    title: "Allowing cluster to pull Docker images"
    image: codefresh/cli
    stage: "deploy"  
    commands:
    - codefresh generate image-pull-secret --cluster 'akschallenge@CloudLabs M12VC - XXX' --registry myazureregistry 

Because your cluster name will probably has spaces in it name, you should use single quotes around the cluster name. Example of full command:

codefresh generate image-pull-secret --cluster 'akschallenge@CloudLabs M12VC - 065' --registry myazureregistry

Test the pipeline

Save and click run on the pipeline. Make sure to select devsecops for your branch. This will build an image, storing it in the Codefresh registry and pushing it into ACR.

Hint Both names for cluster and registry correspond to the names you used in the Codefresh integrations page. Visit Integrations if you don’t remember them.

Use Helm to deploy our application

Earlier, we deployed our application using simple Kubernetes manifests. While these YAMLs are easy to use, they become difficult to work with when you need to swap in new values. With Helm, we can use templated manifests and pass in new values at run time. In this case we’re versioning our image using the commit SHA.

Hint Replace the KUBE-CONTEXT with the name of your AKS cluster you added earlier.

Hint The deploy step uses an open source image from the Codefresh steps directory to execute Helm deployments. It’s essentially a Docker image with everything it needs to deploy to Helm built in. Read more about the Helm step or browse the https://steps.codefresh.io/.

Hint Don’t forget to replace the cluster name with the one you created. You can find the image path by going to images in Codefresh, and clicking on the image you created.


  deploy:
    image: codefresh/cfstep-helm:2.12.0
    stage: deploy
    environment:
      - CHART_REF=deploy/helm/colors
      - RELEASE_NAME=color-coded
      - KUBE_CONTEXT=akschallenge@CloudLabs M12VC - XXX
      - CUSTOM_service.type=LoadBalancer
      - CUSTOM_deployment[0].track=release
      - CUSTOM_deployment[0].image.repository=akschallengeXXXXXXXX.azurecr.io/${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}
      - CUSTOM_deployment[0].image.tag="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.version="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.pullSecret=codefresh-generated-akschallengeXXXXXXXX.azurecr.io-myazureregistry-default

You can find more information in the Helm documentation page

Build it!

Now that your pipeline is setup, go ahead and click “Run”, be sure to select the devsecops branch. When you’re done your service should be fully deployed. You’ll be able to see it in both the Codefresh Kubernetes view, and the Helm Dashboard.

Here is the full pipeline:


version: '1.0'
stages:
  - prepare
  - build
  - deploy
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: prepare
  build:
    title: "Building Docker Image"
    type: "build"
    image_name: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
    tag: "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
    dockerfile: "Dockerfile"
    stage: "build"  
  push:
    title: "Pushing image to Azure registry"
    type: "push"
    stage: deploy
    image_name: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
    registry: "myazureregistry"
    candidate: "${{build}}"
    tags:
      - "${{CF_BRANCH_TAG_NORMALIZED}}"
      - "${{CF_REVISION}}"
      - "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
  createpullsecret:
    title: "Allowing cluster to pull Docker images"
    image: codefresh/cli
    stage: "deploy"  
    commands:
    - codefresh generate image-pull-secret --cluster 'akschallenge@CloudLabs M12VC - XXX' --registry myazureregistry 
  deploy:
    image: codefresh/cfstep-helm:2.12.0
    stage: deploy
    environment:
      - CHART_REF=deploy/helm/colors
      - RELEASE_NAME=color-coded
      - KUBE_CONTEXT=akschallenge@CloudLabs M12VC - XXX
      - CUSTOM_service.type=LoadBalancer
      - CUSTOM_deployment[0].track=release
      - CUSTOM_deployment[0].image.repository=akschallengeXXXXXXXX.azurecr.io/${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}
      - CUSTOM_deployment[0].image.tag="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.version="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.pullSecret=codefresh-generated-akschallengeXXXXXXXX.azurecr.io-myazureregistry-default



You should replace akschallengeXXXXXXXX with your own Azure registry name and akschallenge@CloudLabs M12VC - XXX with your own cluster name as shown in Codefresh integrations.

Hint This is a new release from the old service and deployment you started with. It will provision a new ip address and from here on in, we’ll only upgrade this service.

Navigate to one of those views to see the ip address of the service you just deployed and open it to make sure everything is running.

Resources

Using the Aqua Cloud-native Security Platform

We will now run through the following use cases with Aqua:

Image Assurance

  • Manually add an image to scan
  • Connect to your Azure container registry
  • Scan an image from your Azure container registry
  • Connect to your Codefresh registry
  • Update the default image assurance policy

Runtime Protection

  • Update the default runtime policy to
  • Block unregistered images
  • Block certain executables from running
  • Prevent drift from happening

Image Assurance

Image Assurance is a subsystem of Aqua CSP. It is responsible for the following:

  • Scanning your images for security issues (including known vulnerabilities, sensitive data, malware, configuration issues and open source software).
  • Comparing scan results to applicable image assurance policies to determine compliance and actions to be taken accordingly (including generating audit events, failing CI/CD pipelines, and marking images as non-compliant).
  • Reporting all security risks found with detailed and actionable information in CI/CD tools, the Aqua web console, and other systems.

What is scanned?

Aqua scans images for known vulnerabilities, sensitive data, malware, configuration issues and open source software. Vulnerabilties and malware are identified by Aqua’s CyberCenter which aggregates and correlates multiple feeds from the National Vulnerability Database, various vendor advisories, Whitesource for open source, and proprietary research.

Aqua’s scanners send this information to the Aqua CyberCenter:

  • Aqua image digest
  • Image operating system
  • List of layer digests in the image
  • List of packages installed in the image, including:
    • Package format (e.g., rpm, deb, apk)
    • Package name (e.g., nginx)
    • Package version
    • Package hash (if applicable)
  • List of non-package executables, including:
    • Name of software
    • Version
    • CPE
    • SHA1 hashes of file contents, for purposes of malware identification
  • List of programming-related files (e.g., php, js, jar):
    • SHA1 hashes of file contents

Manually add an image to scan

We will now add an image from the public Docker Hub registry:

  1. In the Aqua web console, in the navigation menu on the left, click on Images.
  2. In the upper right corner, click on the Add Images button.
  3. In the Search Term field, enter: jboss/wildfly
  4. Click the Search button.
  5. Select the jboss/wildfly repository.
  6. Check the 9.0.2.Final tag.
  7. Click the Add button.

This is what you should see as you perform the above steps:

Aqua Output

This will start the scan of the the jboss/wildfy image from Docker Hub. It will not take long to scan the image. Once finished scanning, feel free to view the results, and see how many vulnerabilities were detected by Aqua.

Aqua CyberCenter uses includes a vulnerability feed from WhiteSource, the market leader in open source software security. This is unique to Aqua in the cloud-native security marketplace, and allows us to find more vulnerabilities and report them more accurately.

To see vulnerabilities found by WhiteSource, on the Vulnerabilities tab of the jboss/wildfly:9.0.2.Final image scan results, you can search by WS-:

Aqua Output

Aqua also uses WhiteSource to identify open source software licenses, and allows you to create image assurance policies that determine compliance based on allowed or disallowed licenses.

To see licenses used by resources as identified by WhiteSource, go to the Resources tab of the jboss/wildfly:9.0.2.Final image scan results:

Aqua Output

Notice how the image is marked as Approved, even though it has many high vulnerabilities. This will change later when we apply image assurance policies.

Aqua Output

Connect to your Azure container registry

Now we will integrate with your Azure container registry so you can pull in other images that might be on there.

  1. In the Aqua web console, in the navigation menu on the left, click System.
  2. Click Integrations.
  3. Click the Add Registry button.
  4. In the Registry Name field, enter: ACR
  5. In the Registry Type dropdown list, select: Azure Container Registry
  6. In the Registry URL field, enter the registry domain created by you in this workshop (e.g. https://akschallengeXXXXXXXX.azurecr.io)
  7. In the Username field, enter the Application/Client ID provided to your for this workshop.
  8. In the Password field, enter the Application Secret Key provided to your for this workshop.
  9. Click the Test Connection button.
  10. Click the Save button.

Scan an image from your Azure container registry

We have now integrated with your Azure container registry, and you will be able to select it when adding images to Aqua. Let’s do that now.

  1. In the Aqua web console, in the navigation menu on the left, click Images.
  2. In the upper right corner, click on the Add Images button.
  3. In the Registry dropdown list, select: ACR
  4. In the Search Term field, enter: color-coded
  5. Click the Search button.
  6. Select the <username>\/color-coded repository.
  7. Check the Add All option.
  8. Click the Add button.

This will start a scan of the images of the repository of your ACR registry. When the scans are finished, you can review their results on the Images page.

Connect to your Codefresh registry

In order to scan images built in Codefresh and not yet pushed to ACR, we will generate an access token for Aqua.

To create the access token, in Codefresh, in the navigation menu on the left, click User Settings, and scroll down until you see the Codefresh Registry section, then click the Generate button:

Codefresh registry

In the Key Name field, enter aqua-access, click the Create button, click the Copy token to clipboard link, and click the OK button:

Registry token

Make sure you don’t replace the token on your clipboard, or temporarily paste the value to a text editor, because you’ll need it below. 00759a79c828322b6861856e9d59086a

Now, in Aqua, connect to your Codefresh registry:

  1. In the Aqua web console, in the navigation menu on the left, click System.
  2. Click Integrations.
  3. Click the Add Registry button.
  4. In the Registry Name field, enter: Codefresh
  5. In the Registry URL field, enter: https://r.cfcr.io
  6. In the Username field, enter your Codefresh username.
  7. In the Password field, enter the access token you generated in Codefresh and copied to your clipboard.
  8. Click the Test Connection button.
  9. Click the Save button.

Adding Codefresh registry to Aqua

Note: because the Codefresh registry API does not support search, the Image Search step of Test Connection will fail. This is OK, and you should save the registry connection anyway. Later, if you want to add images from this registry on the Images page, you’ll need to select the registry, and enter the fully qualified image name into the search field (e.g. r.cfcr.io/burbanski/burbanski/color-coded:devsecops-91fc4da).

Update the default image assurance policy

After Aqua scans an image, it compares the results to all applicable image assurance policies. When a control in an applicable image assurance policy fails, several actions can be taken, including marking the image as non-compliant. Initially, Aqua includes a default image assurance policy that has no controls, so all image scans will pass image assurance, even if they have vulnerabilities.

Let’s update the default image assurance policy now:

  1. In the Aqua web console, in the navigation menu on the left, click Policies.
  2. Click Assurance Policies.
  3. Click the Default policy of type Image.
  4. Under Controls, click Vulnerability Severity.
  5. In the Vulnerability Severity control configuration, click High.
  6. Note the actions that will be taken if any control of this policy fail, including marking failed images as non-compliant. This policy will mark any image that contains one or more high severity vulnerabilities as non-compliant.
  7. Click the Save button.

Go back to the Images page, and open the scan results for the jboss/wildfly:9.0.2.Final image that we scanned earlier. Notice how this image is now marked as non-compliant by the Default image assurnace policy, showing which control failed and what actions are needed to make the image compliant:

Aqua Output

Runtime Protection

Runtime policies (and image profiles, vulnerability shields and container firewall services) can be used to monitor and enforce controls at runtime, according to your organization’s security requirements.

Enforcement can prevent containers that use non-compliant and/or unregistered images from running at all; or enforcement can prevent specific things from happening inside of running containers (e.g. certain executables from running, certain volumes from being mounted, etc.).

Update the default runtime policy

Let’s update the default container runtime policy now:

  1. In the Aqua web console, in the navigation menu on the left, click Policies.
  2. Click Runtime Policies.
  3. Click the Aqua default runtime policy policy of type Container.
  4. Under Enforcement Mode, click Enforce.
  5. Notice that the default policy already contains some controls.
  6. Under Controls, click Executables Blacklist.
  7. In the Executables Blacklist control configuration, in the exectuable name field, enter /bin/date, and click the Add button.
  8. Under Controls, click Block Unregistered Images.
  9. Under Controls, click Block Non-compliant Images.
  10. Click the Save button.

Aqua Output

Block unregistered images

In your Azure cloud shell, we’re going to try to deploy an application.

Run this command to deploy nginx:

kubectl create deploy nginx --image=nginx:latest

This will try to deploy the nginx application, but it will fail due to our runtime policy.

Run this command to check the status of the pod:

kubectl get pods

Notice that the nginx pod has a status of RunContainerError.

Run this command to get additional details on the pod status (using the full name of the nginx pod output by previous command):

kubectl describe pod <name>

You should see this message under Events:

[Aqua Security] You do not have permission to execute this command. Unregistered image

Run this command to delete the nginx deployment:

kubectl delete deploy nginx

Now, let’s register the nginx image, so Aqua will allow it to be deployed:

  1. In the Aqua web console, in the navigation menu on the left, click on Images.
  2. In the upper right corner, click on the Add Images button.
  3. In the Registry dropdown list, select: Docker Hub
  4. In the Search Term field, enter: nginx
  5. Click the Search button.
  6. Select the nginx repository.
  7. Check the latest tag.
  8. Click the Add button.

When the nginx:latest image is done scanning, when can try to deploy it again.

Run these commands:

kubectl create deploy nginx --image=nginx:latest
kubectl get po

The nginx pod should now be running.

Block certain executables from running

Our default runtime policy blacklists the date executable. If we exec into the pod, we should not be able to execute the date command.

Run this command to exec into the nginx container (using the full name of the nginx pod output by previous command):

kubectl exec -it <name> bash

Notice that we are now in the nginx container as the root user (something else that can be prevented by Aqua). As such, there’s nothing we should not be able to do. However, if we try executing the date executable, we will get a Permission denied error message. This is because Aqua blocked that executable from running.

Prevent drift from happening

Our default runtime policy prevents drift. Drift prevention ensures that your containers remain immutable, and protects you from both malicious attacks and bad habits by not allowing executables to run that were not part of the original image and/or not allowing the container to run when image parameters have changed.

To simulate a drift, we will copy an allowed executable, like ls, to another name like list:

    cp /bin/ls /bin/list

When we try to run the list command, instead of getting a list of the contents of the current directory, we will get a Permission denied error message. This is because Aqua considered that new executable to be drift and blocked that executable from running.

As you may have gathered when navigating Aqua during the workshop, Aqua has many more capabilities than were explored here, to include image profiles, vulnerability shields, container firewall services, secrets management, infrastructure discovery with pen testing and CIS benchmark testing, workload monitoring and more.

Security scanning with Codefresh

As your application is growing and new libraries are being added, it is important to know about any security vulnerabilities before reaching production.

In this section you will add to our Codefresh pipeline and use the Aqua Security platform you configured in the previous section for security scanning.

Make sure that you have the following at hand:

  • Your Aqua account username
  • Your Aqua account password
  • The URL of your Aqua server installation
  • The username of an account user with the Scanner role
  • The password of the account user with the Scanner role

Tasks

Using the Aqua Codefresh plugin

To scan images as part of your CI/CD pipeline we’ve created a simple to use step, the Codefresh-Aqua plugin.

Navigate to the Codefresh pipeline you created earlier.

Expand the Environment variables and add the following variables (you can also delete the PORT one as it is not needed):

  • AQUA_HOST - the Aqua server from the previous section including port (http://example.com:80)
  • AQUA_USERNAME - Your Aqua Username
  • AQUA_PASSWORD - Your Aqua password

In the Workflow section make sure that the first option is selected - Inline YAML and add this step between the build and push steps.


  AquaSecurityScan:
    title: 'Aqua Private scan'
    image: codefresh/cfstep-aqua
    stage: test
    environment:
      - 'AQUA_HOST=${{AQUA_HOST}}'
      - 'AQUA_PASSWORD=${{AQUA_PASSWORD}}'
      - 'AQUA_USERNAME=${{AQUA_USERNAME}}'
      - IMAGE=${{build}}
      - TAG=latest

            

Save and run the pipeline to get your security results.

Here is the full pipeline:


version: '1.0'
stages:
  - prepare
  - build
  - test
  - push 
  - deploy
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: prepare
  build:
    title: "Building Docker Image"
    type: "build"
    image_name: "${{CF_ACCOUNT}}/${{CF_REPO_NAME}}"
    tag: ${{CF_REVISION}}
    dockerfile: "Dockerfile"
    stage: "build"  
  AquaSecurityScan:
    title: 'Aqua Private scan'
    image: codefresh/cfstep-aqua
    stage: test
    environment:
      - 'AQUA_HOST=${{AQUA_HOST}}'
      - 'AQUA_PASSWORD=${{AQUA_PASSWORD}}'
      - 'AQUA_USERNAME=${{AQUA_USERNAME}}'
      - IMAGE=${{CF_ACCOUNT}}/${{CF_REPO_NAME}}
      - TAG=${{CF_REVISION}}
      - REGISTRY=codefresh
  push:
    title: "Pushing image to Azure registry"
    type: "push"
    stage: push
    image_name: "${{CF_ACCOUNT}}/${{CF_REPO_NAME}}"
    registry: "myazureregistry"
    candidate: "${{build}}"
    tags:
      - "${{CF_BRANCH_TAG_NORMALIZED}}"
      - "${{CF_REVISION}}"
      - "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"    

  createpullsecret:
    title: "Allowing cluster to pull Docker images"
    image: codefresh/cli
    stage: "deploy"  
    commands:
    - codefresh generate image-pull-secret --cluster 'akschallenge@CloudLabs M12VC - XXX' --registry myazureregistry 
  deploy:
    image: codefresh/cfstep-helm:2.12.0
    stage: deploy
    environment:
      - CHART_REF=deploy/helm/colors
      - RELEASE_NAME=color-coded
      - KUBE_CONTEXT=akschallenge@CloudLabs M12VC - 065
      - CUSTOM_service.type=LoadBalancer
      - CUSTOM_deployment[0].track=release
      - CUSTOM_deployment[0].image.repository=akschallengeXXXXXX.azurecr.io/${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}
      - CUSTOM_deployment[0].image.tag="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.version="${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      - CUSTOM_deployment[0].image.pullSecret=codefresh-generated-akschallengeXXXXXX.azurecr.io-myazureregistry-default


            

You should replace akschallengeXXXXXXXX with your own Azure registry name and akschallenge@CloudLabs M12VC - XXX with your own cluster name as shown in Codefresh integrations.

Uh-oh! We found a vulnerability, let’s fix it!

This concludes the scanning tasks. You have successfully used the Codefresh CI/CD platform to scan Docker images with the Aqua Security solution. Browse through the platforms to see what the issue is.

To resolve it we’ll need to update the version of OpenSSL. Open up the Dockerfile in your repo and comment out the following lines:

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.1/main' >> /etc/apk/repositories
RUN apk add "openssh==6.7_p1-r6"

Resources

Contributors

The following people have contributed to this workshop, thanks!