What a million tokens costs on my own hardware

I run a local AI lab: an NVIDIA DGX Spark, a Mac Studio, and a few Mac minis. I am also a CFA who builds financial models for a living. So I did the obvious thing and built the amortization model. The punchline: most of my tokens still go to frontier APIs, and the math says that is the right call.

The model

The cost of a locally generated token has three inputs people usually skip: what the hardware cost, how long it lives, and what fraction of the time it does useful work. Everything else is arithmetic.

cost per hour   = straight-line depreciation + blended power
tokens per hour = throughput (tok/s) x 3600 x utilization
cost per 1M     = cost per hour / tokens per hour x 1,000,000

The scripts are open source: local-inference-economics benchmarks any OpenAI-compatible endpoint and turns the result into a fully loaded cost per million tokens, plus the breakeven utilization against current API prices.

An illustrative rig

To show the shape of the answer before my measured numbers land, here is a hypothetical box: $4,000 purchase price, three-year life with $400 residual value, 240W at load and 40W idle, electricity at $0.14/kWh, running a model at 40 tok/s with 30% utilization.

That works out to about $3.60 per million generated tokens. Amortization is roughly 85 to 90 percent of that; electricity is nearly a rounding error. The hardware you already bought, not the power bill, is the cost driver.

The comparison that actually matters

Read that table carefully, because it cuts both ways. Against large frontier-model pricing, my rig only needs to be busy 7 percent of the time to pay for itself. But that comparison is mostly fiction: the local model is not producing frontier-quality tokens, so it is not a substitute for the work I send to those models. Against the honest comparison, hosted open models at 8B-class pricing, self-hosting never breaks even. Someone else's amortized GPU at scale beats mine at any utilization I can sustain.

Tokens are the wrong unit anyway

This is the part the per-token spreadsheet cannot capture, and it is why most of my usage is frontier models. The unit that matters is cost per finished task, not cost per token. A frontier model that produces the right valuation note in one pass is cheaper than a local model that needs three attempts and a human review, at almost any token price. In FP&A terms: the local rig has lower unit costs and dramatically lower yield.

So why keep the lab?

Because some workloads are not priced in tokens. Audio processing where the data should not leave the house. Always-on background automation where a fixed cost ceiling beats a variable bill. Experiments where I want to break things without watching a meter. And honestly, because understanding the stack from the metal up makes me better at judging what the API bill buys.

The finance framing holds: the lab is capex I justify by option value and learning, not by token arbitrage. Anyone telling you self-hosting is simply cheaper has not done the utilization math. Anyone telling you it is pointless has not priced privacy or the value of owning your own floor.