securing-your-azure-devops-ecosystem-jenkins-and-kubernetes-aks

Securing your Azure DevOps Ecosystem, Jenkins and Kubernetes (AKS) – Part 2

The first part of the series about securing your Azure DevOps ecosystem covered Jenkins and the integration of CodeNotary into your Jenkins pipeline. That way you an easily notarize everything that gets produced in your very own pipeline. If you want to double check the blog post – please click here.

This second part covers the authentication process when your Jenkins pipeline deployed container into your Azure Kubernetes (AKS). That way you can make sure, that nothing unknown or unwanted runs in AKS. You can of course configure alerts as well.

If you want to learn more about Notarization and Authentication, please check this blog post first: https://hackernoon.com/the-day-we-started-to-protect-devops-with-blockchain-a9g6y33gt

But let’s start step by step.

Azure Kubernetes Service (AKS)

Azure Kubernetes Service is an important part of the Azure DevOps Ecosystem and allows a very easy and straightforward deployment of services running on Kubernetes. The AKS environment used for this blog post has been nicely described here: https://medium.com/@adilsonbna/building-my-own-azure-devops-ecosystem-ef92b8db9da5

Azure AKS Login

Azure AKS can be added to your local kubectl config in a very simple way. Check your resource group and the name of the AKS deployment in your Azure Console:

Azure Ecosystem AKS

# add AKS access to .kube/config
az aks get-credentials --resource-group DevopsEcosystem --name cniokubecluster

If you don’t have az installed, please check out the following page: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest

# list all Kubernetes services in the current namespace
kubectl get svc

Helm deployment

The easiest installation of CodeNotary for Kubernetes is using a helm chart (alternatives are described in the repository). If you haven’t deployed helm yet, you can follow this guide.

Create a file helm-rbac.yaml

# create a tiller service account - helm-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

Deploy the Helm (Tiller) service account into your AKS

kubectl apply -f helm-rbac.yaml
# install helm and deploy tiller
helm init --history-max 200 --service-account tiller --node-selectors "beta.kubernetes.io/os=linux"
# check installed helm charts
helm list

# check helm and tiller (server) version
helm version

Azure AKS Kube-Notary deployment

Now you can deploy Kube-Notary using helm by simply cloning the repository and issuing a helm install command.

# clone git repository
git clone https://github.com/vchain-us/kube-notary.git
cd kube-notary

# deploy Kube-Notary
helm install -n kube-notary helm/kube-notary --namespace=monitoring

# you can check the status of Kube-Notary at any time using
helm status kube-notary

What happened when deploying the Kube-Notary helm chart?

The following resources are getting deployed to mainly support one Pod that consistently checks all running container /images/blog if they have been signed (you need a free account for that) using CodeNotary:

  • ClusterRole
  • ConfigMap
  • Deployment
  • Pod(related)
  • Service
  • ServiceAccount
  • ClusterRoleBinding

Please keep in mind, that Kube-Notary checks by default if a container image has been notarized on CodeNotary.io. But most of the time you only care about what you notarized, someone in your team or organization.

Configure a list of trusted keys

Lists can be expressed by enclosing keys in { and } separated by ,. For example:

helm install -n kube-notary helm/kube-notary --set "trust.keys={0x123..., 0x123...}"

Configure a trusted organization

helm install -n kube-notary helm/kube-notary --set "trust.org=your.organization.com"

Note

If both trust.org and trust.keys are set, only trust.org will be used.

If none is set, the last signature by the signer with the highest available level will be used during the verification.

You can find more information in the GitHub repository.

Check the Kube-Notary service

Kube-Notary comes with all common ways to extract information by default.

  1. Log output – to be consumed by ELK, Fluent or any other log collector
  2. Prometheus output – Kube-Notary pod gets automatically detected and scraped by a Prometheus server (if configured to scrape pods)
  3. json results page (can be checked manually or automatic)
  4. realtime status dashboard

If you want to check the output on your local system, you can use kubectl to forward the endpoint:

# find Service
export SERVICE_NAME=service/$(kubectl get service --namespace monitoring -l "app.kubernetes.io/name=kube-notary,app.kubernetes.io/instance=kube-notary" -o jsonpath="{.items[0].metadata.name}")

  # Metrics endpoint
echo "Check the metrics endpoint at http://127.0.0.1:9581/metrics"
kubectl port-forward --namespace monitoring $SERVICE_NAME 9581

  # Results endpoint
echo "Check the verification results endpoint at http://127.0.0.1:9581/results"
kubectl port-forward --namespace monitoring $SERVICE_NAME 9581

  # Status page
echo "Status page at http://127.0.0.1:9581/status/"
kubectl port-forward --namespace monitoring $SERVICE_NAME 9581

  # Stream logs
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=kube-notary,app.kubernetes.io/instance=kube-notary" -o jsonpath="{.items[0].metadata.name}")

kubectl logs --namespace monitoring -f $POD_NAME

The status page might look like that, depending on the number of notarized container /images/blog.

  • Trusted: you notarized the image (or someone, depending on you Kube-Notary configuration)
  • Untrusted: you untrusted the image (or someone, depending on you Kube-Notary configuration)
  • Unsupported: you unsupported the image (or someone, depending on you Kube-Notary configuration) . That’s typically a good idea, when you release a newer image that should be used instead.
  • Unknown: Container image has not been notarized at all on CodeNotary.io

Kube-Notary status page

Notarizing (Trust, Untrust, Unsupport) Azure AKS container /images/blog

Notarizing the first /images/blog on CodeNotary is key for Kube-Notary to be useful. You can find a complete guideline how to work with CodeNotary here.

https://docs.codenotary.io/guide/quickhelp.html

When you start using Kube-Notary, you might want to notarize the status quo and move on more strict afterwards. Honestly if you don’t trust the container /images/blog that are currently running, that might be an issue anyway.

# Bulk sign all running 
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=kube-notary,app.kubernetes.io/instance=kube-notary" -o jsonpath="{.items[0].metadata.name}")

kubectl exec --namespace monitoring -t $POD_NAME sh /bin/bulk_sign > vcn_bulk_sign.sh

chmod +x vcn_bulk_sign.sh && ./vcn_bulk_sign.sh

The moment you start notarizing container /images/blog, you’ll notice the changes in your Status Dashboard.

Prometheus and Grafana

As already mentioned, the Kube-Notary pod comes with Prometheus annotations and will be scraped immediately after the deployment, depending on your Prometheus configuration.

You can simply import the Grafana dashboard and you should see a nice dashboard version including notarization history.

The dashboard can be found here: https://grafana.com/grafana/dashboards/10339

Grafana Azure AKS integration

Add your very own immutable Trust to any container image

You can finally add immutable trust to your DevOps pipeline. It doesn’t matter if you want to authenticate Sourcecode, binaries, configuration files or container /images/blog – CodeNotary vcn supports all of these objects.

Using CodeNotary and the Kubernetes extension Kube-Notary allows you and your team to immediately secure and protect your running environment. No need to bother with complex GPG or TUF Notary installation, configuration and annoying digital certificates. CodeNotary doesn’t change the objects, but calculates an unique checksum and stores that including your trust level in an immutable way.

Signup today – completely free and without any risk

CNIL
Metrics and Logs

(formerly, Opvizor Performance Analyzer)

VMware vSphere & Cloud
PERFORMANCE MONITORING, LOG ANALYSIS, LICENSE COMPLIANCE!

Monitor and Analyze Performance and Log files:
Performance monitoring for your systems and applications with log analysis (tamperproof using immudb) and license compliance (RedHat, Oracle, SAP and more) in one virtual appliance!

Subscribe to Our Newsletter

Get the latest product updates, company news, and special offers delivered right to your inbox.

Subscribe to our newsletter

Use Case - Tamper-resistant Clinical Trials

Goal:

Blockchain PoCs were unsuccessful due to complexity and lack of developers.

Still the goal of data immutability as well as client verification is a crucial. Furthermore, the system needs to be easy to use and operate (allowing backup, maintenance windows aso.).

Implementation:

immudb is running in different datacenters across the globe. All clinical trial information is stored in immudb either as transactions or the pdf documents as a whole.

Having that single source of truth with versioned, timestamped, and cryptographically verifiable records, enables a whole new way of transparency and trust.

Use Case - Finance

Goal:

Store the source data, the decision and the rule base for financial support from governments timestamped, verifiable.

A very important functionality is the ability to compare the historic decision (based on the past rulebase) with the rulebase at a different date. Fully cryptographic verifiable Time Travel queries are required to be able to achieve that comparison.

Implementation:

While the source data, rulebase and the documented decision are stored in verifiable Blobs in immudb, the transaction is stored using the relational layer of immudb.

That allows the use of immudb’s time travel capabilities to retrieve verified historic data and recalculate with the most recent rulebase.

Use Case - eCommerce and NFT marketplace

Goal:

No matter if it’s an eCommerce platform or NFT marketplace, the goals are similar:

  • High amount of transactions (potentially millions a second)
  • Ability to read and write multiple records within one transaction
  • prevent overwrite or updates on transactions
  • comply with regulations (PCI, GDPR, …)


Implementation:

immudb is typically scaled out using Hyperscaler (i. e. AWS, Google Cloud, Microsoft Azure) distributed across the Globe. Auditors are also distributed to track the verification proof over time. Additionally, the shop or marketplace applications store immudb cryptographic state information. That high level of integrity and tamper-evidence while maintaining a very high transaction speed is key for companies to chose immudb.

Use Case - IoT Sensor Data

Goal:

IoT sensor data received by devices collecting environment data needs to be stored locally in a cryptographically verifiable manner until the data is transferred to a central datacenter. The data integrity needs to be verifiable at any given point in time and while in transit.

Implementation:

immudb runs embedded on the IoT device itself and is consistently audited by external probes. The data transfer to audit is minimal and works even with minimum bandwidth and unreliable connections.

Whenever the IoT devices are connected to a high bandwidth, the data transfer happens to a data center (large immudb deployment) and the source and destination date integrity is fully verified.

Use Case - DevOps Evidence

Goal:

CI/CD and application build logs need to be stored auditable and tamper-evident.
A very high Performance is required as the system should not slow down any build process.
Scalability is key as billions of artifacts are expected within the next years.
Next to a possibility of integrity validation, data needs to be retrievable by pipeline job id or digital asset checksum.

Implementation:

As part of the CI/CD audit functionality, data is stored within immudb using the Key/Value functionality. Key is either the CI/CD job id (i. e. Jenkins or GitLab) or the checksum of the resulting build or container image.

White Paper — Registration

We will also send you the research paper
via email.

CodeNotary — Webinar

White Paper — Registration

Please let us know where we can send the whitepaper on CodeNotary Trusted Software Supply Chain. 

Become a partner

Start Your Trial

Please enter contact information to receive an email with the virtual appliance download instructions.

Start Free Trial

Please enter contact information to receive an email with the free trial details.