Self-hosting open-weight models with minimal operational footprint

The economics of running open-weight models on your own hardware have crossed an inflection point. A single H100 node can serve Llama 3.1 70B at throughputs that would cost four to five times as much per token from a hosted API. The engineering challenge is no longer “can we afford it” — it is “can we operate it reliably without a dedicated MLOps team.”

ManyLayers Deploy is the answer to that second question. It handles model packaging, runtime configuration, health checking, autoscaling, and OpenAI-compatible API serving — all from a deployment that runs on standard Kubernetes or on dstack-managed bare-metal GPU nodes.

The traditional self-hosted inference stack problem

Traditional self-hosted inference stacks require assembling and operating several components: a container registry, a model server (vLLM, TGI, TensorRT-LLM), a load balancer, a metrics exporter, and a control plane to tie them together. Each component has its own upgrade cadence, configuration format, and failure mode.

Teams end up maintaining what amounts to a small internal platform just to run one or two models. When the model server releases a new version with a performance improvement, someone has to test it, stage it, and roll it out carefully. When a model weight update arrives, the update process is manual and fragile. When a GPU node fails, routing traffic to healthy nodes requires manual intervention.

ManyLayers Deploy collapses this into a single deployable with no external runtime dependencies. The deployment includes an embedded model registry, a multi-engine runtime adapter, and an OpenAI-compatible HTTP server — so you operate one thing instead of five.

No sprawling microservices to operate

ManyLayers Deploy includes:

• An embedded model registry that pulls weights from your configured object storage (S3-compatible, GCS, or local NFS)
• A multi-engine runtime adapter that can drive vLLM, llama.cpp, or TensorRT-LLM without exposing their individual APIs
• An OpenAI-compatible HTTP server (/v1/chat/completions, /v1/completions, /v1/embeddings)
• A Prometheus metrics endpoint and structured JSON access log
• A control-plane API for listing deployments, scaling replicas, and triggering model swaps

Deploying a new model is a single API call or a YAML manifest applied with kubectl. The binary handles pulling weights, warming the runtime, running a readiness check, and shifting traffic only when the new replica passes health checks.

Deployment targets

Kubernetes is the primary target for organizations that already run workloads on k8s. ManyLayers provides a Helm chart and a CRD (ModelDeployment) that maps cleanly onto Kubernetes primitives. GPU resource requests, node selectors, tolerations, and pod disruption budgets all work as you’d expect.

apiVersion: manylayers.io/v1
kind: ModelDeployment
metadata:
  name: llama-3-1-70b
spec:
  model: meta/llama-3.1-70b-instruct
  replicas: 2
  resources:
    gpuType: nvidia-h100
    gpuCount: 4
  runtime: vllm
  quantization: fp8

dstack is the target for teams running bare-metal GPU clusters or cloud spot instances outside Kubernetes. dstack manages GPU fleet allocation and job scheduling; ManyLayers Deploy sits on top to provide the application-layer API and model lifecycle management. This combination is particularly common in air-gapped environments where a full Kubernetes control plane would be operational overhead.

Air-gapped operation

Air-gapped deployments — networks with no outbound internet access — are a first-class use case. ManyLayers Deploy can be configured to pull model weights from an internal artifact store and never make outbound calls. License validation is offline. The model library metadata (which models are available, their checksums, recommended configurations) can be pre-seeded from a portable bundle and updated via an internal registry. No model vendor telemetry, no external calls.

This is critical for regulated industries — healthcare, defense, financial services — where data residency and network isolation requirements prohibit any outbound traffic from the inference layer.

The OpenAI-compatible API matters

The single most important architectural decision in ManyLayers Deploy is the choice to surface all model endpoints as a drop-in replacement for the OpenAI API. This means:

• Existing application code that calls openai.chat.completions.create(...) works unchanged — just point the base URL at your ManyLayers Gateway endpoint.
• SDKs in Python, TypeScript, Go, and every other language with an OpenAI client library work without modification.
• When your self-hosted model needs to fall back to a cloud provider (for a model you haven’t deployed locally, or during a maintenance window), the Gateway routes the request transparently with no changes on the client side.

Performance considerations

Running inference efficiently on your own hardware requires tuning that most teams don’t have time to do from scratch. ManyLayers Deploy applies sensible defaults based on the model family, GPU type, and available VRAM, including:

• Continuous batching with a tuned max_batch_size per model
• KV-cache sizing calibrated to the VRAM budget
• Quantization selection (fp8 for H100, GPTQ/AWQ for smaller GPUs)
• Tensor parallelism configuration for multi-GPU serving

These defaults get you 80–90% of optimal performance out of the box. For workloads with strict SLA requirements, the configuration is fully exposed and documented for further tuning.

The model library

ManyLayers maintains a curated model library — a catalog of tested model configurations with recommended runtime settings, hardware profiles, and benchmark results on representative workloads. Browse it in Deploy → Model Library. Each entry includes estimated throughput on common GPU types, quantization options, and minimum VRAM requirements.

Self-hosting open-weight models is now operationally tractable. The models are capable, the economics are compelling, and with the right operational layer, the maintenance burden is manageable by a small team. That’s the problem ManyLayers Deploy is built to solve.