Controlling AI spend across 40+ providers

About this session

AI infrastructure spend is the fastest-growing line item in many engineering budgets — and the hardest to predict. Unlike compute or storage, LLM spend is driven by usage patterns that vary dramatically by team, by application, and by time of day. A FinOps team accustomed to reserved instance planning finds that LLM cost management requires entirely different tools and mental models.

This session is designed for platform engineering leads and FinOps practitioners who need to bring order to multi-provider AI spend without slowing down the teams using AI to do their work. It covers both the technical configuration and the organizational processes that make cost management sustainable.

What you will learn

• How ManyLayers Gateway’s real-time cost tracking works: token counting, per-provider pricing tables, cost attribution by team, and the reconciliation process at month-end
• Configuring budget policies at team, department, and organization level — including soft caps, hard caps, alert thresholds, and the escalation workflow when a team needs more budget
• Using model allowlists to prevent teams from accessing expensive frontier models when cost-optimized alternatives are sufficient for their use case
• How semantic caching reduces provider spend for repetitive workloads — cache hit rate benchmarks by workload type and how to tune the similarity threshold for your traffic
• Fallback routing as a cost optimization tool: routing to cheaper providers during peak hours or when a premium provider’s latency exceeds SLA thresholds
• Exporting cost data to your data warehouse and integrating with existing FinOps dashboards (Grafana, Tableau, or Looker)

Session agenda

0:00 – 8:00 — The multi-provider cost problem: how spending is distributed across providers, models, and teams in a typical enterprise AI deployment. Why cost visibility is the prerequisite to cost control, and what the data looks like before and after a gateway is in place.

8:00 – 20:00 — Live demo — cost analytics: walkthrough of the ManyLayers Analytics dashboard. Team spend by model, daily cost trend, P95 cost-per-request distribution, and how to drill into individual requests to understand cost drivers. Exporting data as CSV and via the analytics API.

20:00 – 32:00 — Live demo — budget enforcement: configuring a budget policy for a fictional engineering team. Setting hard cap and alert threshold. Demonstrating what happens when the budget is exhausted — the error response a client application receives and the audit log entry that records the enforcement event. Configuring a budget increase workflow via Slack webhook.

32:00 – 40:00 — Semantic caching and cost optimization routing: enabling semantic caching on a high-volume alias and comparing cost per 1,000 requests before and after. Configuring a cost-priority routing strategy that prefers cheaper models and falls back to premium models only when the cheaper model returns an error.

40:00 – 44:00 — Q&A: questions on chargeback to internal cost centers, integration with AWS Cost Explorer and Azure Cost Management, and handling teams that have contractual commitments to specific providers.