If you want to deploy with Terraform, look here.
I purchased three mini computers on Black Friday to use for K3S learning and self hosting. Naturaly, I want to access the cluster from the internet but there are some issues. The house only has copper run to it so the only internet options are cellular and Starlink. In both cases, port forwarding is next to impossible due to firmware restrictions and CGNAT. I will be explaining how I accessed the HTTP services via Cloudflare Tunnel.
K3S
K3S is a lightweight version of Kubernetes, tailored for simplicity and a smaller resource footprint, making it ideal for environments with limited resources. Unlike the full-fledged Kubernetes, K3S is designed to run efficiently on as few as three nodes, which is perfect for compact setups like mini computers. This minimalistic approach doesn’t compromise on functionality, maintaining high compatibility with standard Kubernetes features. It’s an excellent choice for edge computing, development, testing, or small-scale production environments. If you’re exploring efficient and manageable orchestration solutions, K3S could be your go-to option. For an in-depth understanding, see the K3s documentation.
CGNAT
There’s NAT for a home router and theres NAT as scale, Carrier Grade NAT. To extend the life of IPv4, CGNAT is used by some ISPs to reduce the amount of used IPv4 addresses by having multiple devices(customers) share an address. This makes port forwarding for residental customers impossible.
The Solutions
Since there’s no way to directly connect to devices behind a CGNAT, the only real solution is to forward that traffic over a virtual network. There are commercial services for this offering port-to-port forwards targeting gaming and self hosting, DIY VPS setups, and enterprise offerings like Cloudflare Tunnel. I’m choosing Cloudflare Tunnel in this case because it’s free, has great reliability, and can be automated with Terraform and Ansible.
I use OpenVPN and Caddy as the current fix which works but is tied into other infrastructure I plan on refactoring.
Another solution out of this scope is using IPv6 if the ISP supports changing network settings. Starlink’s router bypass feature allows for one to request and allocate a public IPv6 block to clients and route the traffic directly to them. IPv6 has a truly massive supply of addresses with ISPs and datacenters handing out /64 blocks(~18,446,744,073,709,551,616 IPs) to each customer as it’s the minimum subnet allocation.
Requirements
- A k3s or kubernetes cluster
- A pod with HTTP or use the example below
- kubectl access to said cluster
- Cloudflare account
- Your choice of secret management(I use KeepassXC for personal use)
It’s easy enough to install cloudflared on a Linux sever and achieve the same goal. Cloudflare has good instructions for this as well.
Create a Cloudflare Account
Login to Cloudflare at https://one.dash.cloudflare.com.
Note: Cloudflare ZeroTrust isn’t in the same dashboard as standard Cloudflare.
Create a Tunnel and Generate the Cloudflared Token(non-Terraform)
In the menu on the left, go to Access. You should see the Tunnels option. Create a tunnel and name it.
You should now see install instructions. The install command has your cloudflared token. Copy the token.
Create a Kubernetes Secret
Create a secret using kubectl and the secret copied above. If you don’t have kubectl configured for your cluster, look here for instructions.
kubectl create secret generic tunnel-secret --from-literal=token=YOURTOKENHERE
Create Public Hostnames
Now that you’ve created a tunnel, you can to create hostnames to access your resources. Go access the tunnel page and select Public Hostname and add a public hostname.
The entries are fairly self-explanatory. The service is the internal protocol and url as if accessed from inside the cluster(which it’s doing). For my personal website, the ingress is named personalsite and accessed via port 3000.
You can also access external resources.
With a public hostname created pointing to your Kubernetes resources, we can deploy cloudflared to your cluster.
Create the Kubernetes Deployment Manifest
Modify the manifest to suit your needs.
cloudflared-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: cloudflared
name: cloudflared-deployment
namespace: default
spec:
replicas: 2
selector:
matchLabels:
pod: cloudflared
template:
metadata:
labels:
pod: cloudflared
spec:
containers:
- name: cloudflared
image: cloudflare/cloudflared:latest
command:
- cloudflared
- tunnel
- --metrics
- 0.0.0.0:2000
- run
args:
- --token
- $(TOKEN)
env:
- name: TOKEN
valueFrom:
secretKeyRef:
name: tunnel-secret
key: token
livenessProbe:
httpGet:
path: /ready
port: 2000
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
Deploy the Manifest
kubectl apply -f cloudflared-deployment.yml
Check on the deployment: kubectl get deployment cloudflared-deployment
Check on the pods: kubectl get pods -l pod=cloudflared
Check on the logs: kubectl logs -l pod=cloudflared
2023-12-12T04:09:54Z INF Registered tunnel connection connIndex=3 connection=in534653-e39b-4a3b-ad86-ien643 event=0 ip=your-public-ip location=clt01 protocol=quic
Check for Success
Try to access your resource. In my case it’s https://testing.daniel-mcdonough.com
Also check the Tunnel dashboard. It should show “HEALTHY” if cloudflared is connected.
Troubleshooting
-
If the tunnel is not healthy, check the cloudflared pod logs:
kubectl logs -l pod=cloudflared. If the secret is causing the error, try modifying or recreating the Kubernetes secret. -
If the tunnel is healthy, make sure you have the correct pod name and port in your public hostname config on the tunnel dashboard.
Links
https://docs.k3s.io/cluster-access
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/
https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/tunnel