Rolling out per-team LLM budgets

Giving every team in your organization access to large language models is a productivity win — until your monthly provider bill arrives and no one can explain where half of it went. Per-team budgets solve this problem by making spend visible, predictable, and enforceable at the gateway layer, without requiring any changes to the applications your teams have already built.

The problem with shared API keys

Most organizations start with a handful of shared API keys distributed across Slack or a secrets manager. Teams use whichever key they find first. Usage is aggregated under a single billing account with no way to attribute cost to a project, a team, or an individual. When the bill spikes, the investigation is a manual grep through logs — if you have logs at all.

Shared keys also create a ceiling problem: you either set a global spend limit (which affects everyone when one team goes over) or you set no limit (which means any team can exhaust your budget). Neither is workable for a mature organization.

How ManyLayers Gateway enforces budgets

ManyLayers Gateway centralizes all provider credentials. Your teams’ applications point at the Gateway endpoint and authenticate with team-specific tokens. The Gateway resolves the incoming token to a team identity, tracks cumulative token spend in real time, and enforces the budget policy assigned to that team.

Budget enforcement happens before the request is forwarded to a provider. When a team’s hard cap is reached, subsequent requests are rejected with a 429 status and a structured error body explaining the policy. The provider never sees the request, so you don’t incur charges.

Setting up budget policies

Navigate to Platform Settings → Teams → [Team Name] → Budget. The policy has four key fields:

{
  "monthly_budget_usd": 1500,
  "alert_threshold": 0.75,
  "hard_cap": true,
  "rollover": false
}

monthly_budget_usd is the nominal spend ceiling for the calendar month. ManyLayers calculates cost using the provider’s published per-token pricing at request time. If a provider changes pricing mid-month, the Gateway picks up the new rates at the next configuration refresh (default: 60 seconds).

alert_threshold triggers a webhook and an in-platform notification when the team crosses the specified fraction of their budget. Set this to 0.75 to warn at 75%. Teams receive the alert in their Workspace notification center; platform admins receive it via the configured alert webhook.

hard_cap determines whether requests are blocked once the budget is exhausted. Set to true for teams with unpredictable or experimental usage. Set to false (soft cap) for production workloads where availability matters more than strict cost control — the team receives an alert but requests continue.

rollover allows unused budget to carry forward into the next month. Useful for teams with seasonal usage patterns.

Model-level restrictions

Budgets work best alongside model allowlists. A team might have a $1,000/month budget but should only be allowed to use gpt-4o-mini and claude-haiku — not the flagship models that would exhaust the budget in a few hundred requests.

{
  "allowed_models": ["openai/gpt-4o-mini", "anthropic/claude-haiku-3-5"],
  "denied_models": ["openai/o1", "anthropic/claude-opus-4-5"]
}

The Gateway enforces model restrictions before budget checks. A request for a denied model returns a 403 regardless of remaining budget.

Aggregated spend visibility

Platform admins see a unified dashboard at Analytics → Cost showing spend by team, by model, and by time period. Each row links to the raw request log for that team so you can drill into which applications or users are driving spend.

Individual teams see their own usage in Workspace → Usage. They can see how much of their monthly budget they’ve consumed, which models they’ve used most, and a daily spend trend. Transparency reduces the “surprise bill” dynamic — teams self-regulate when they can see their own consumption.

Alerting and escalation

Configure alert destinations in Platform Settings → Alerts. ManyLayers supports:

• Webhook — POST a structured JSON payload to any URL (useful for Slack, PagerDuty, or a custom handler)
• Email — send to one or more addresses
• Audit log entry — always written, regardless of other destinations

For teams with hard caps, configure a second alert at 90% so team leads have time to request a budget increase before requests start failing.

Practical rollout sequence

1. Audit first. Before setting caps, run the gateway for two to four weeks in passthrough mode (no hard caps) to establish baseline spend per team. Use the Analytics dashboard to understand actual usage patterns.
2. Set generous initial caps. Start at 150% of the observed baseline. This prevents false positives while still catching runaway usage.
3. Enable soft caps before hard caps. Give teams two weeks on soft caps — where they receive alerts but requests are not blocked — so they can observe their budget relationship before enforcement starts.
4. Tighten incrementally. After two billing cycles, review actuals versus caps and tighten caps to 120% of average. Refine allowlists based on which models teams actually use.

Per-team budgets are not primarily a cost-cutting tool. They are a visibility and accountability mechanism. When spend is attributed to teams and teams can see it, cost conversations move from reactive (“why was this month’s bill $40,000?”) to proactive (“we need to increase our budget for Q3 because we’re launching a new product”).