DigitalOcean Kubernetes Challenge - Deploying an internal container registry

Okay! Today, while scrolling down Twitter as usual I found that Tweet and I got excited about why not to try, since I was trying to find a real-world scenarios to work with Kubernetes.

So I went with the Kubernetes experience on DigitalOcean. Also, I wanted to share what I've done. As I'm still beginner with K8s I chose one of their Kubernetes Beginner Challenges to work on it and document it.

In this article, I’ll explain the steps to create a K8s cluster and deploy and internal container registry using Trow. So Let’s get started.

Provision your first Kubernetes Cluster in DigitialOcean

We will go through the creation using Terraform. Terraform is a solution from HashiCorp which allows managing Infrastructure As Code. You just need to write your desired state and Terraform manages to build the desired infrastructure, using a modular system of providers. There is a specific provider for DigitalOcean, which encapsulates the translation from the Terraform definition to the DigitalOcean API. You can find the comprehensive documentation Here.

First, we need to set up the actual DigitalOcean Terraform provider, it's very simple. You need to provide your previous DigitalOcean personal access token. You can either use a variable and the prompt (Don't commit your token!) or environment variables. The provider automatically uses the below two variables, if they exist.

DIGITALOCEAN_TOKEN and DIGITALOCEAN_ACCESS_TOKEN

Here's where you will generate a new token.

Then, you will export it as below in you shell.

export DIGITALOCEAN_TOKEN=TOKEN-ID

Let the fun began, and write down the Terraform code in kubs.tf:

# Deploy the actual Kubernetes cluster
resource "digitalocean_kubernetes_cluster" "kubernetes_cluster" {
  name    = "terraform-do-cluster"
  region  = "ams3"
  version = "1.19.15-do.0"

  tags = ["my-tag"]

  # This default node pool is mandatory
  node_pool {
    name       = "default-pool"
    size       = "s-1vcpu-2gb" # minimum size, list available options with `doctl compute size list`
    auto_scale = false
    node_count = 2
    tags       = ["node-pool-tag"]
    labels = {
      "olwimamdouh.hashnode.dev" = "up"
    }
  }

}

And create a new file provider.tf:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

Perform Terraform deployment

We're ready, let's create our cluster in the cloud. First, run terraform init to download the required providers

omamdouh@DO-testing:~/DO$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding digitalocean/digitalocean versions matching "~> 2.0"...
- Installing digitalocean/digitalocean v2.16.0...
- Installed digitalocean/digitalocean v2.16.0 (signed by a HashiCorp partner, key ID F82037E524B9C0E8)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next, run terraform plan to verify what resources are planned to be created:

omamdouh@DO-testing:~/DO$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_kubernetes_cluster.kubernetes_cluster will be created
  + resource "digitalocean_kubernetes_cluster" "kubernetes_cluster" {
      + cluster_subnet = (known after apply)
      + created_at     = (known after apply)
      + endpoint       = (known after apply)
      + ha             = false
      + id             = (known after apply)
      + ipv4_address   = (known after apply)
      + kube_config    = (sensitive value)
      + name           = "terraform-do-cluster"
      + region         = "ams3"
      + service_subnet = (known after apply)
      + status         = (known after apply)
      + surge_upgrade  = true
      + tags           = [
          + "my-tag",
        ]
      + updated_at     = (known after apply)
      + urn            = (known after apply)
      + version        = "1.19.15-do.0"
      + vpc_uuid       = (known after apply)

      + maintenance_policy {
          + day        = (known after apply)
          + duration   = (known after apply)
          + start_time = (known after apply)
        }

      + node_pool {
          + actual_node_count = (known after apply)
          + auto_scale        = false
          + id                = (known after apply)
          + labels            = {
              + "olwimamdouh.hashnode.dev" = "up"
            }
          + name              = "default-pool"
          + node_count        = 2
          + nodes             = (known after apply)
          + size              = "s-1vcpu-2gb"
          + tags              = [
              + "node-pool-tag",
            ]
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.

And finally, let’s apply the configuration through terraform apply

omamdouh@DO-testing:~/DO$ terraform apply --auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_kubernetes_cluster.kubernetes_cluster will be created
  + resource "digitalocean_kubernetes_cluster" "kubernetes_cluster" {
      + cluster_subnet = (known after apply)
      + created_at     = (known after apply)
      + endpoint       = (known after apply)
      + ha             = false
      + id             = (known after apply)
      + ipv4_address   = (known after apply)
      + kube_config    = (sensitive value)
      + name           = "terraform-do-cluster"
      + region         = "ams3"
      + service_subnet = (known after apply)
      + status         = (known after apply)
      + surge_upgrade  = true
      + tags           = [
          + "my-tag",
        ]
      + updated_at     = (known after apply)
      + urn            = (known after apply)
      + version        = "1.19.15-do.0"
      + vpc_uuid       = (known after apply)

      + maintenance_policy {
          + day        = (known after apply)
          + duration   = (known after apply)
          + start_time = (known after apply)
        }

      + node_pool {
          + actual_node_count = (known after apply)
          + auto_scale        = false
          + id                = (known after apply)
          + labels            = {
              + "olwimamdouh.hashnode.dev" = "up"
            }
          + name              = "default-pool"
          + node_count        = 2
          + nodes             = (known after apply)
          + size              = "s-1vcpu-2gb"
          + tags              = [
              + "node-pool-tag",
            ]
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.
digitalocean_kubernetes_cluster.kubernetes_cluster: Creating...
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [50s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m0s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [1m50s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m0s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [2m50s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m0s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [3m50s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m0s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [4m50s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [5m0s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [5m10s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [5m20s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [5m30s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Still creating... [5m40s elapsed]
digitalocean_kubernetes_cluster.kubernetes_cluster: Creation complete after 5m43s [id=2ebc5c7c-8d07-497d-8d99-47d4939709a6]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

You should see this in Kubernetes Dashboard while the provisioning:

Once finished, you will see the K8S cluster available:

Connect to the Kubernetes cluster

Next we will connect to our Kubernetes cluster. To achieve that, we need to download the kubeconfig file. We can use CURL to do this. First, let’s create an environment variable with our cluster ID.

export CLUSTER_ID=ID

Next, let’s go ahead and download the kubectl config file:

curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" "https://api.digitalocean.com/v2/kubernetes/clusters/$CLUSTER_ID/kubeconfig" > config

This will download a config file in your current repository. Now we need to set the KUBECONFIG environment variable to point to the download config file

export KUBECONFIG=$(pwd)/config

Let’s now see if everything works as expected:

omamdouh@DO-testing:~/DO$ kubectl cluster-info
Kubernetes control plane is running at https://2ebc5c7c-8d07-497d-8d99-47d4939709a6.k8s.ondigitalocean.com
CoreDNS is running at https://2ebc5c7c-8d07-497d-8d99-47d4939709a6.k8s.ondigitalocean.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

This gives back indeed information related to our DigitalOcean Kubernetes cluster. Let’s see our pods:

omamdouh@DO-testing:~/DO$ kubectl get pod -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   cilium-2tt28                       1/1     Running   0          17m
kube-system   cilium-operator-7fd9d7b9dc-vq6xd   1/1     Running   0          19m
kube-system   cilium-tvr5f                       1/1     Running   0          17m
kube-system   coredns-57877dc48d-kn26z           1/1     Running   0          19m
kube-system   coredns-57877dc48d-wcwk6           1/1     Running   0          19m
kube-system   csi-do-node-gb6g6                  2/2     Running   0          17m
kube-system   csi-do-node-x7dsp                  2/2     Running   0          17m
kube-system   do-node-agent-8qppg                1/1     Running   0          17m
kube-system   do-node-agent-l298p                1/1     Running   0          17m
kube-system   kube-proxy-gs5vl                   1/1     Running   0          17m
kube-system   kube-proxy-tc5dl                   1/1     Running   0          17m

Deploying Trow

Yay! So you've got your shiny new Kubernetes cluster up-and-running. Let's see how to deploy a registry running with TLS.

If you have kubectl running and pointing at your Kubernetes cluster, all you need to do is:

omamdouh@DO-testing:~/DO$ git clone git@github.com:ContainerSolutions/trow.git
omamdouh@DO-testing:~/DO$ cd trow
omamdouh@DO-testing:~/DO/trow/quick-install$ ./install.sh
Trow AutoInstaller for Kubernetes
=================================

This installer assumes kubectl is configured to point to the cluster you want to
install Trow on and that your user has cluster-admin rights.

This installer will perform the following steps:

  - Create a ServiceAccount and associated Roles for Trow
  - Create a Kubernetes Service and Deployment
  - Request and sign a TLS certificate for Trow from the cluster CA
  - Copy the public certificate to all nodes in the cluster
  - Copy the public certificate to this machine (optional)
  - Register a ValidatingAdmissionWebhook (optional)

.... omitting 
Copying certs to nodes
job.batch/copy-certs-25fd327b-5761-423f-8c83-1f0d4124dc3a created
job.batch/copy-certs-6b32ae91-8feb-45a3-88bd-60fb14175d07 created
.... omitting
64.227.74.13 trow.kube-public # added for trow registry
Successfully configured localhost
Do you want to configure Trow as a validation webhook (NB this will stop external images from being deployed to the cluster)? (y/n) y
Setting up trow as a validating webhook
WARNING: This will limit what images can run in your cluster
By default, only images in Trow and official Kubernetes images will be
allowed
validatingwebhookconfiguration.admissionregistration.k8s.io/trow-validator created

The Trow registry is now up and running in our cluster.

So, we will notice the newly created pods.

omamdouh@DO-testing:~/DO/trow/quick-install$ kubectl get pod -A
NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-public   copy-certs-25fd327b-5761-423f-8c83-1f0d4124dc3a-dmvnp   0/1     Completed   0          2m18s
kube-public   copy-certs-6b32ae91-8feb-45a3-88bd-60fb14175d07-dpcx2   0/1     Completed   0          2m18s
kube-public   trow-deploy-744888894b-lbnhk                            1/1     Running     0          3m6s
kube-system   cilium-2tt28                                            1/1     Running     0          44m
kube-system   cilium-operator-7fd9d7b9dc-vq6xd                        1/1     Running     0          47m
kube-system   cilium-tvr5f                                            1/1     Running     0          44m
kube-system   coredns-57877dc48d-kn26z                                1/1     Running     0          47m
kube-system   coredns-57877dc48d-wcwk6                                1/1     Running     0          47m
kube-system   csi-do-node-gb6g6                                       2/2     Running     0          44m
kube-system   csi-do-node-x7dsp                                       2/2     Running     0          44m
kube-system   do-node-agent-8qppg                                     1/1     Running     0          44m
kube-system   do-node-agent-l298p                                     1/1     Running     0          44m
kube-system   kube-proxy-gs5vl                                        1/1     Running     0          44m
kube-system   kube-proxy-tc5dl                                        1/1     Running     0          44m

The script has set up the domain trow.kube-public to point at your cluster. We can now tag and push our local image:

omamdouh@DO-testing:~/DO/trow/quick-install$ sudo docker pull nginx:latest
omamdouh@DO-testing:~/DO/trow/quick-install$ sudo docker tag nginx:latest trow.kube-public:31000/nginx:test
omamdouh@DO-testing:~/DO/trow/quick-install$ sudo docker push trow.kube-public:31000/nginx:test

And finally run it inside Kubernetes:

omamdouh@DO-testing:~/DO/trow/quick-install$ kubectl run trow-test --image=trow.kube-public:31000/nginx:test
pod/trow-test created
omamdouh@DO-testing:~/DO/trow/quick-install$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
trow-test   1/1     Running   0          14s

Problem solved! We've got our image up and running in a few moments without any external services or extra cost.