The Ledger: Observational Telemetry
Hardware realities, token economics, and quantization trade-offs gathered from community observations and public documentation. Point-in-time estimates—not an audited commercial guarantee.
Important Telemetry Notice: The figures, token costs, context metrics, and benchmarks compiled in The Ledger are rough, point-in-time community estimates. They do not represent audited warranties, vendor guarantees, or professional recommendations. Infrastructure pricing and model metrics fluctuate constantly and may contain unintentional inaccuracies or become outdated. Never make commercial, architectural, or investment commitments based on these numbers without direct vendor verification.
Table I: Model Economics & Speed
Normalized to hosted API endpoints under standard production concurrency.
| Model | Organization | Architecture | Active / Total | Context | Input (per 1M) | Output (per 1M) | Speed | TTFT | Status |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V3 | DeepSeek | MoE (Mixture-of-Experts) | 37B / 671B | 128K | $0.14 | $0.28 | 64 t/s | 380ms | Open Weights |
| Llama 3.3 70B | Meta AI | Dense | 70B / 70B | 128K | $0.18 | $0.40 | 82 t/s | 290ms | Open Weights |
| Claude 3.5 Sonnet | Anthropic | MoE (Mixture-of-Experts) | Undisclosed / ~200B | 200K | $3.00 | $15.00 | 75 t/s | 420ms | Frontier |
| GPT-4o | OpenAI | MoE (Mixture-of-Experts) | Undisclosed / ~220B | 128K | $2.50 | $10.00 | 95 t/s | 240ms | Frontier |
| Gemini 1.5 Pro | Google DeepMind | MoE (Mixture-of-Experts) | Undisclosed / ~350B | 2,000K | $1.25 | $5.00 | 68 t/s | 510ms | Frontier |
| Mistral Large 2 | Mistral AI | Dense | 123B / 123B | 128K | $2.00 | $6.00 | 52 t/s | 360ms | Production |
Table II: Quantization & Memory Footprint
Evaluated on 70B parameter models running on NVIDIA H100 SXM5 80GB.
| Format | Bits / Weight | VRAM (70B Model) | Perplexity Delta | Memory Bandwidth Speedup | Ideal Use Case |
|---|---|---|---|---|---|
| BF16 / FP16 (Baseline) | 16 bits | 140 GB | 0.00 (Reference) | 1.0x | Frontier Pretraining & Fine-Tuning |
| FP8 (E4M3 / E5M2) | 8 bits | 72 GB | +0.03 (<0.1% loss) | 1.85x | Modern Enterprise Production Serving |
| INT4 (AWQ / GPTQ) | 4 bits | 38 GB | +0.18 (Noticeable on coding) | 2.70x | Edge Devices, Local Mac & Workstations |
| BitNet b1.58 (Ternary) | 1.58 bits | 18 GB | Parity at equal FLOP budget | 4.20x | Next-Gen Native Integer Silicon |
💡 Key Engineering Finding
The industry consensus has settled on FP8 (8-bit floating point) for production inference. It halves GPU memory usage and delivers a 1.85x speedup in throughput while preserving mathematical accuracy within 0.1% of raw 16-bit representations.