Daniel McDonough

Using K3S: Overcoming CGNAT with Cloudflare Tunnel(Terraform)

 Personal Projects/Cloud Engineering

I purchased three mini computers on Black Friday to use for K3S learning and self hosting. Naturally, I want to access the cluster from the internet but there are some issues. The house only has copper run to it so the only internet options are cellular and Starlink. In both cases, port forwarding is next to impossible due to firmware restrictions and CGNAT. I will be explaining how I accessed the HTTP services via Cloudflare Tunnel.

K3S

K3S is a lightweight version of Kubernetes, tailored for simplicity and a smaller resource footprint, making it ideal for environments with limited resources. Unlike the full-fledged Kubernetes, K3S is designed to run efficiently on as few as three nodes, which is perfect for compact setups like mini computers. This minimalistic approach doesn’t compromise on functionality, maintaining high compatibility with standard Kubernetes features. It’s an excellent choice for edge computing, development, testing, or small-scale production environments. If you’re exploring efficient and manageable orchestration solutions, K3S could be your go-to option. For an in-depth understanding, see the K3s documentation.

CGNAT

There’s NAT for a home router and theres NAT as scale, Carrier Grade NAT. To extend the life of IPv4, CGNAT is used by some ISPs to reduce the amount of used IPv4 addresses by having multiple devices(customers) share an address. This makes port forwarding for residental customers impossible.

The Solutions

Since there’s no way to directly connect to devices behind a CGNAT, the only real solution is to forward that traffic over a virtual network. There are commercial services for this offering port-to-port forwards targeting gaming and self hosting, DIY VPS setups, and enterprise offerings like Cloudflare Tunnel. I’m choosing Cloudflare Tunnel in this case because it’s free, has great reliability, and can be automated with Terraform and Ansible.

I use OpenVPN and Caddy as the current fix which works but is tied into other infrastructure I plan on refactoring.

Another solution out of this scope is using IPv6 if the ISP supports changing network settings. Starlink’s router bypass feature allows for one to request and allocate a public IPv6 block to clients and route the traffic directly to them. IPv6 has a truly massive supply of addresses with ISPs and datacenters handing out /64 blocks(~18,446,744,073,709,551,616 IPs) to each customer as it’s the minimum subnet allocation.

Requirements

  • A k3s or kubernetes cluster
  • A pod with HTTP or use the example below
  • kubectl access to said cluster
  • Cloudflare account
  • Terraform(optional)
    • A Cloudflare API key with:
      • Zone - DNS - Edit
      • Account - Cloudflare Tunnel - Edit
      • Account - Access: Apps and Policies - Edit
    • Your choice of secret management(I use KeepassXC for personal use)

It is easy enough to install cloudflared on a Linux sever and achieve the same goal. Cloudflare has good instructions for this as well.

Create a Cloudflare Account

Login to Cloudflare at https://one.dash.cloudflare.com. Note: Cloudflare ZeroTrust isn’t in the same dashboard as standard Cloudflare.

Generate an API key

This is done on the standard Cloudflare dashboard at https://dash.cloudflare.com/profile/api-tokens Create a token with:

  • Permissions
    • Zone - DNS - Edit
    • Account - Cloudflare Tunnel - Edit
    • Account - Access: Apps and Policies - Edit
  • Account Resources
    • Include: Select your account
  • Zone Resources
    • Include: All Zones
    • or
    • Include: Specific zone - Your desired zone

Use Terraform to Deploy

To deploy via Terraform you need to add the Kubernetes and Cloudflare providers:

providers.tf

terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = ">= 4.9.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = ">= 2.24.0"
    }
    random = {
      source = "hashicorp/random"
    }
  }
  required_version = ">= 0.13"
}

You also need to add the relevant variables to variables.tf:

variables.tf


# Cloudflare variables
variable "cloudflare_zone" {
  description = "Domain used to expose the GCP VM instance to the Internet"
  type        = string
}

variable "cloudflare_zone_id" {
  description = "Zone ID for your domain"
  type        = string
}

variable "cloudflare_account_id" {
  description = "Account ID for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "cloudflare_email" {
  description = "Email address for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "cloudflare_token" {
  description = "Cloudflare API token created at https://dash.cloudflare.com/profile/api-tokens"
  type        = string
  sensitive   = true
}

And then add the variable values to the Terraform tfvars.tf file, an environment variable, or your secrets setup:

terraform.tfvars

cloudflare_zone           = "example.com"
cloudflare_zone_id        = "023e105f4ecef8ad9ca31a8372d0c353"
cloudflare_account_id     = "372e67954025e0ba6aaa6d586b9e0b59"
cloudflare_email          = "[email protected]"
cloudflare_token          = "your_cloudflare_token_here"

Create a Cloudflare Terraform config:

cloudflare-tunnel.tf

# Generates a 64-character secret for the tunnel.
# Using `random_password` means the result is treated as sensitive and, thus,
# not displayed in console output. Refer to: https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password
resource "random_password" "tunnel_secret" {
  length = 64
}

# Creates a new locally-managed tunnel for the GCP VM.
resource "cloudflare_tunnel" "auto_tunnel" {
  account_id = var.cloudflare_account_id
  name       = "Terraform k3s tunnel"
  secret     = base64sha256(random_password.tunnel_secret.result)
}

# Creates the CNAME record that routes http_app.${var.cloudflare_zone} to the tunnel.
# Change the name to the desired subdomain

resource "cloudflare_record" "http_app" {
  zone_id = var.cloudflare_zone_id
  name    = "http_app"
  value   = "${cloudflare_tunnel.auto_tunnel.cname}"
  type    = "CNAME"
  proxied = true
}

# Creates the configuration for the tunnel.
# Change the service to the desired pod name
# NOTE: To add more services, append ingress rules in the config statement. Do not create new tunnel config resources. They will overwrite one another.

resource "cloudflare_tunnel_config" "auto_tunnel" {
  tunnel_id = cloudflare_tunnel.auto_tunnel.id
  account_id = var.cloudflare_account_id
  config {
   ingress_rule {
     hostname = "${cloudflare_record.http_app.hostname}"
     service  = "http://httpbin"
   }
   ingress_rule {
     service  = "http_status:404"
   }
  }
}


# Creates the cloudflared secret in Kubernetes that allows cloudflared to authenticate to the tunnel.

resource "kubernetes_secret" "cloudflared_tunnel_secret" {
  metadata {
    name = "tunnel-secret"
  }

  data = {
    token = cloudflare_tunnel.auto-tunnel.tunnel_token
  }
}

# Creates the kubernetes deployment for cloudflared
# This creates the pods with 2 replicas and labels them with app=cloudflared


resource "kubernetes_manifest" "cloudflared_deployment" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name      = "cloudflared-deployment"
      namespace = "default"
      labels = {
        app = "cloudflared"
      }
    }
    spec = {
      replicas = 2
      selector = {
        matchLabels = {
          pod = "cloudflared"
        }
      }
      template = {
        metadata = {
          labels = {
            pod = "cloudflared"
          }
        }
        spec = {
          containers = [
            {
              name  = "cloudflared"
              image = "cloudflare/cloudflared:latest"
              command = [
                "cloudflared",
                "tunnel",
                "--metrics",
                "0.0.0.0:2000",
                "run"
              ]
              args = [
                "--token",
                "$(TOKEN)"
              ]
              env = [
                {
                  name = "TOKEN"
                  valueFrom = {
                    secretKeyRef = {
                      name = "tunnel-secret"
                      key  = "token"
                    }
                  }
                }
              ]
              livenessProbe = {
                httpGet = {
                  path = "/ready"
                  port = 2000
                }
                failureThreshold    = 1
                initialDelaySeconds = 10
                periodSeconds       = 10
              }
            }
          ]
        }
      }
    }
  }
}

Create an Example HTTP Endpoint(Optional)

Add these resources to the above Terraform file if you don’t have an HTTP endpoint and want an example. Be sure the service above is set to: service = "http://httpbin"

resource "kubernetes_deployment" "httpbin_deployment" {
  metadata {
    name = "httpbin-deployment"
  }

  spec {
    replicas = 2

    selector {
      match_labels = {
        app = "httpbin"
      }
    }

    template {
      metadata {
        labels = {
          app = "httpbin"
        }
      }

      spec {
        container {
          image = "kennethreitz/httpbin:latest"
          name  = "httpbin"

          port {
            container_port = 80
          }
        }
      }
    }
  }
}


resource "kubernetes_service" "web_service" {
  metadata {
    name = "web-service"
  }

  spec {
    selector = {
      app = "httpbin"
    }

    port {
      port        = 80
      protocol    = "TCP"
    }
  }
}

Deploy

Run terraform init to download the providers.

Run terraform plan -OUT PLAN and read through the changes, making sure everything looks correct.

Run terrafoam apply PLAN to apply the plan.

Check the Deployment

  • Check the tunnel dashboard for the presence of the tunnel and the public hostname.

  • Check on the deployment: kubectl get deployment cloudflared-deployment

  • Check on the pods: kubectl get pods -l pod=cloudflared

  • Check on the logs: kubectl logs -l pod=cloudflared

    • Example: 2023-12-12T04:09:54Z INF Registered tunnel connection connIndex=3 connection=in534653-e39b-4a3b-ad86-ien643 event=0 ip=your-public-ip location=clt01 protocol=quic

Check for Success

Try to access your resource. In my case it’s https://testing.daniel-mcdonough.com

Also check the Tunnel dashboard. It should show “HEALTHY” if cloudflared is connected.

Tunnel status

Troubleshooting

  • If Terraform is failing with a permissions issue, make sure the Cloudflare token for Terraform has the proper privileges.

  • If Terraform is able to plan but is failing on the deploy on Cloudflare, make sure your account ID and zone ID are correct. An incorrect ID can throw authorization errors.

  • If the tunnel is not healthy, check the cloudflared pod logs: kubectl logs -l pod=cloudflared. If the secret is causing the error, try modifying or deleting and recreating the Kubernetes secret.

  • If the tunnel is healthy, make sure you have the correct pod name and port in your public hostname config on the tunnel dashboard.

https://docs.k3s.io/cluster-access

https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/

https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/tunnel

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret

updatedupdated2023-12-172023-12-17

See Also: