Running a Native AI Stack on Your Own Metal

At sevenseven.tech, we don't run everything on bare metal, and we don't run everything in the cloud. We run whichever one fits the job. For this stack, the job was raw inference speed, and the answer was our own workstation.

In our previous article, we clustered older GPUs into a budget AI server. This time we swapped "budget" for "uncompromising power" and "Docker Compose" for "Kubernetes". We deployed a high-performance, self-hosted AI stack on a single workstation that blurs the line between a gaming rig and an enterprise server. Instead of a container script, we used MicroK8s on Ubuntu to orchestrate a suite of AI tools: Ollama, OpenWebUI, and Stable Diffusion.

Here is how we built a single-node AI cluster that screams performance.

The Hardware: The "Overkill" Node

If the previous build was a Frankenstein monster, this one is precision-engineered. We needed a machine capable of handling massive context windows, heavy parallel processing, and instant inference.

The RTX 4090 is the star of the show for inference speed. The Threadripper and 256 GB of RAM let us handle massive concurrent tasks, embedding generation, and even CPU-offloaded inference for 120B+ parameter models if the VRAM fills up.

The Stack: Why Kubernetes?

You might ask: Why use Kubernetes for a single machine?

Docker Compose is great, but Kubernetes (via MicroK8s) gives us:

1. Rolling Updates: We can update the UI without dropping active connections.
2. Resource Management: We strictly allocate GPU slices and CPU cores to specific services.
3. Scalability: If we add a second node later, the cluster simply expands.

We use Helm to manage the deployment, keeping our configuration version-controlled and reproducible.

Ingress Configuration

In a Kubernetes environment, an Ingress controller manages external access. Below is our production config. Note the proxy-body-size and timeout annotations. These are crucial for AI workloads where users might upload large PDF manuals for analysis, or wait for long "thought" processes from the model.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: open-webui-ingress
  namespace: open-webui
  annotations:
    # Automate SSL with Let's Encrypt
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # Allow large file uploads for RAG (100MB)
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    # Prevent timeouts during long model inference
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - your-domain.example   # your host, with a Let's Encrypt cert
    secretName: open-webui-tls
  rules:
  - host: your-domain.example
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: open-webui
            port:
              number: 80

The Deployment Strategy

Managing a K8s cluster can be complex, so we automated the maintenance. We wrote a custom Bash script that interacts with Helm to handle upgrades safely.

The script performs a few critical safety checks:

1. Stuck Operation Detection: It checks if a previous Helm operation failed and attempts a rollback.
2. Atomic Upgrades: It updates the OpenWebUI Helm chart with the --atomic flag, so if anything fails, the cluster reverts to the last working state.
3. Targeted Restarts: It gracefully rolls out restarts to the specific deployments (Ollama, Pipelines, and Automatic1111 for image generation).

Here is a snippet of the update logic:

### Helm Upgrades ###
echo "Upgrading open-webui..."
microk8s helm3 upgrade --timeout 30m0s \
  --namespace open-webui \
  --values values.yaml \
  --atomic \
  --wait \
  open-webui \
  open-webui/open-webui

### Deployment Management ###
echo -e "\n=== Restarting Deployments ==="

restart_deployment() {
  local deployment=$1
  echo "Restarting $deployment..."
  microk8s kubectl rollout restart deployment "$deployment" -n open-webui
  # Wait for the rollout to actually finish before moving on
  microk8s kubectl rollout status deployment "$deployment" -n open-webui --timeout=10m
}

# We restart the full stack to ensure new configs apply
restart_deployment automatic1111
restart_deployment open-webui-pipelines
restart_deployment open-webui-ollama

The Results: Pure Speed

The performance difference between the older GPU cluster and this beast is night and day. We tested the setup using Qwen 3 Coder (30B) with a massive 65k context window.

Architecture Diagram

Here is the single-node stack as it runs in production. Note the atomic Helm upgrade path that keeps the whole thing self-healing.

Figure 1: The single-node K8s AI stack. Ingress handles TLS and large uploads; OpenWebUI talks to Ollama, Pipelines, and Automatic1111; the 4090 does the heavy lifting; the Helm script keeps upgrades atomic.

Why does this matter?

The generation speed (~27 t/s) is faster than you can read, making the chat feel alive. But the real game-changer is the 1,146 t/s prompt processing. This means you can paste entire codebases or massive documents into the chat, and the system reads and understands them almost instantly before it even starts answering.

Not Just a Chatbot: A Developer's Engine

Because this stack runs a compliant API, we don't just use it via the browser. This server powers our entire development workflow:

1. VS Code Integration: We use plugins like Cline and Roo Code, pointing them at our local URL. The AI edits files, refactors legacy code, and writes tests directly in the IDE.
2. Command Line: We use tools like Mistral Vibe to pipe terminal output directly into the AI for analysis.

It functions exactly like the OpenAI API, but private, free of monthly fees, and running on our own metal.

Enterprise-Grade AI on Your Own Hardware

Building a reliable AI infrastructure goes beyond just buying a fast GPU. It requires orchestration, security, and lifecycle management.

Whether you are looking to cluster low-end hardware to save costs, or deploy a high-availability Kubernetes stack on high-end workstations, we have the expertise to make it happen. We help businesses deploy indestructible stacks that keep your data private and your uptime high.

Want to run enterprise-grade AI on your own hardware?

Contact us at support@sevenseven.tech or fill out our contact form.

Update: qwen3.8:27b, Same Machine

Update: September 2026. A few days after qwen3.8:27b landed, we pointed the same workstation at it and ran the same workload. The box has not changed - no new GPU, no new RAM. Only the model did.

Generation speed more than doubled, prompt processing nearly doubled, and the usable context jumped from 65k to 90k - on identical hardware. That is the real story behind this article: you do not need to chase the next rig to get the next level of performance. Sometimes the machine you already own is just waiting for the model to catch up.