Solutions / Model Serving

Serve open-weight models on your own GPUs.

Deploy Llama, Mistral, Qwen, and fine-tuned models to Kubernetes or dstack. They appear instantly as routing targets in Gateway — same API, same guardrails, zero external dependencies.

Talk to us Deployment options

The problem

Proprietary model APIs create cost and compliance ceilings.

High-volume workloads on commercial APIs carry per-token costs that scale linearly. Sensitive industries cannot send certain data to external endpoints at all. Self-hosting has historically required a dedicated MLOps team and a fragmented toolchain for serving, routing, and monitoring.

The outcome

Open-weight models served through one API, fully on your terms.

ManyLayers Deploy removes the MLOps ceiling. Models deploy from a curated library, register in Gateway automatically, and serve through the same OpenAI-compatible surface — with the same governance layer applied whether the request routes on-prem or to a cloud provider.

Capabilities

The full model-serving stack, not just inference.

Kubernetes & dstack

Helm chart for one-command Kubernetes deployment. Native dstack backend for cloud GPU orchestration with spot-instance cost optimisation across AWS, GCP, Azure, and more.

Curated Model Library

Browse and deploy Llama, Mistral, Qwen, Phi, Gemma, and other open-weight models directly from the admin UI. Model versioning and rollback included.

Fine-Tuning Jobs

LoRA and QLoRA fine-tuning on your proprietary datasets. GPU-scheduled job queue with progress tracking. Promote fine-tuned checkpoints to production with one click.

Unified API Surface

On-prem models served at the same OpenAI-compatible /v1/ endpoints as cloud providers. Fallback chains, budgets, and guardrails apply identically to self-hosted models.

Air-Gapped Operation

No outbound network calls required after initial image pull. License file activation without internet access. All inference, caching, and logs remain on-premises.

GPU Observability

Prometheus metrics for GPU utilisation, request throughput, and latency per model. Pre-built Grafana dashboards included. Structured log export to your existing stack.

Deployment flow

From cluster to serving traffic in five steps.

Select model from the curated library in admin UI ↓

Choose target GPU node pool and replica count ↓

Deploy — model starts serving within minutes ↓

Model registers automatically as a Gateway routing target ↓

Apply guardrails, budgets, and rate limits identically

Binary installs Gateway + model plane together

Outbound calls required in air-gapped mode

<1 hr

Typical cluster-to-serving deployment time

Serve open-weight models with complete data sovereignty.

Talk to us View deploy docs