Why I Built My Own AI Infrastructure Stack

Last month, my API bill for AI services crossed $4,800. That's not enterprise-scale — that's one person developing products across three companies. If I'd stayed on the cloud path, that number would be $15,000/month by end of year. Instead, I built an infrastructure stack that handles the same workload for the cost of electricity.

Here's the architecture, the tradeoffs, and why self-hosted AI is not just cheaper — it's better for the kind of work I do.

The Fleet

Five machines, six GPUs, 22 models, zero cloud dependencies. Total capital cost: approximately $12,000. That's less than three months of equivalent cloud API usage. The machines run 24/7 and have been for months.

The Models We Actually Use

I don't run models for benchmarks. I run models because they have specific jobs. Here's what's actually in production:

Code generation: deepseek-v4, qwen3-coder:30b, codestral:22b. The coder models are routed based on language and complexity — Python goes to the 30B model on thirteen; Rust and C++ go to codestral on archie; general code review goes to deepseek-v4.

Content synthesis and research: gemma4:26b, deepseek-r1:32b. These handle the heavy lifting — distilling 200-page technical documents, generating blog post drafts (yes, this article started as a local model output), and running multi-step research queries through the RAG pipeline.

Fast iteration and summarization: qwen3:8b, qwen3.5:4b. When I need a quick summary or a first draft of an email response, these fire in under two seconds. The quality isn't as high, but the speed matters for rapid iteration.

Embeddings and RAG: all-MiniLM-L6-v2 running locally via FastEmbed. 384-dimensional embeddings, sub-10ms retrieval on my document corpus. No API calls ever.

The RAG Pipeline That Actually Works

The Achilles heel of most self-hosted AI setups is retrieval. You can have the best LLM in the world, but if your retrieval pipeline returns garbage, the model will hallucinate confidently about that garbage.

My RAG pipeline has three layers:

Layer 1 — ChromaDB Vector Store: 32K+ chunked documents indexed for semantic search. When a query comes in, it retrieves the top 5 most semantically relevant chunks. This is the recall layer — it should return everything that might be relevant.

Layer 2 — Keyword Reranking: The initial results are re-ranked using BM25 (yes, the 1990s algorithm) over exact keyword matches. This catches things that semantic search misses — model numbers, error codes, specific function names. BM25 + vector search together outperform either alone, and both run locally.

Layer 3 — LLM-as-Judge Filtering: The top 3 candidates from the reranked results are evaluated by a lightweight model (qwen3:4b) that scores each chunk for relevance to the original query. Anything below threshold gets dropped. The survivors become the context window for the main model.

This three-layer approach is what makes the difference between a RAG system that works sometimes and one that works reliably. Total latency from query to context-assembled: 400-800ms. Total cost: zero dollars per query.

Why Not the Cloud?

I've used every major AI API provider. They're great for prototyping. They're terrible for production when you need:

1. Predictable costs. Cloud AI APIs charge per token with pricing that changes monthly. When you're running autonomous agents that can consume millions of tokens in a single long-context session, the costs become genuinely unpredictable.

2. Full control over the model. Cloud providers decide which models are available, when they're updated, and what system prompts they accept. When your application depends on a specific model's behavior, a silent update can break things. With local models, the checkpoint is frozen until you choose to change it.

3. Privacy for sensitive work. Some of the documents my agents process contain proprietary engineering designs, financial projections, and legal contracts. Sending those to a third-party API means someone else's server reads them. End of story.

4. Offline capability. My autonomous agents run 24/7. They don't stop when the internet goes down or the API endpoint changes its authentication scheme. Local models run even when Comcast fails.

5. Latency that doesn't depend on network conditions. Sub-100ms first-token latency on a local GPU versus 500-2000ms on a cloud API. When you're iterating on code or running autonomous agents that make dozens of sequential calls, that difference compounds to minutes saved per session.

The Stack: Software That Powers It

The Orchestration Layer

Five machines running models independently doesn't help unless you can route work intelligently. The orchestration layer (avcs-hal transport + Odysseus multi-model routing) handles this:

• Model discovery: Each machine advertises its available models and current load via Ollama's API. The orchestrator maintains a live inventory of what's available where.
• Intelligent routing: Queries are routed to the best available model based on task type (code vs. language vs. analysis), model capabilities, and current GPU load. A quick summarization request goes to a small model on a lightly-loaded GPU. A complex code generation request goes to the 30B model on the machine with the most free VRAM.
• Fallback handling: If a primary model is overloaded, the request cascades to the next-best available option. The orchestrator handles load shedding and retry logic transparently.
• Token governance: The efficiency loop monitors token consumption across all models and applies budget constraints to prevent any single agent from consuming excessive resources.

What I'd Do Differently

If I were starting today, I'd make exactly three changes:

1. Start with unified AMD ROCm. The CUDA/ROCm split between NVIDIA and AMD GPUs creates annoying compatibility issues. Some models only run well on one platform. If I'd committed to AMD from the start, I'd have fewer headaches — but the NVIDIA ecosystem was more mature when I started building.

2. Invest in faster networking earlier. Moving model weights between machines for distributed inference requires real bandwidth. My 1GbE LAN is the current bottleneck for multi-GPU inference. 10GbE switches are on the roadmap.

3. Dockerize everything from day one. Running models directly on the host was simpler to set up, but Docker Compose would have made updates, rollbacks, and environment isolation much cleaner. Odysseus now has Docker support; the model serving layer is next.

The Bottom Line

Self-hosted AI infrastructure is not for everyone. If you run one model occasionally and don't care about latency or privacy, use the cloud. But if you're building products that depend on AI, if you process sensitive documents, if you run autonomous agents that need sub-second response times, and if your API bill is approaching five figures per month — build your own stack.

The hardware costs less than you think. The software is free and improving daily. The only real cost is the time to set it up. And once it's running, you'll wonder why you ever paid someone else to run your models.

If you need help designing or deploying a self-hosted AI infrastructure stack, I do consulting. Reach out.