Government, legal, medical, and industrial clients require AI on their own infrastructure. Here's what on-premises AI deployment actually involves.
The default assumption in most AI projects is cloud deployment. You call an API, the inference happens somewhere on AWS or Azure, and results come back. It's fast, cheap, and easy to get started.
But a significant portion of the clients we work with cannot use this model. Their requirements — legal, regulatory, or contractual — mandate that data and inference stay on their infrastructure. This post covers what on-premises AI deployment actually involves, when it's necessary, and the trade-offs you accept when you choose it.
Government and defense: Classified data cannot leave government-controlled infrastructure. AI systems that process classified documents, communications, or imagery must run on air-gapped or government-certified networks.
Legal and financial services: Attorney-client privilege and financial confidentiality requirements may prohibit sending documents to third-party cloud services. Several EU financial regulations have strict data residency requirements.
Healthcare: HIPAA compliance, combined with organizational risk management policies, leads many healthcare organizations to require that patient data never leaves their infrastructure.
Industrial and manufacturing: OT (operational technology) systems in factories and industrial environments are often air-gapped from the internet for security reasons. AI systems that integrate with these systems must run locally.
IP-sensitive R&D: Companies doing sensitive research (semiconductors, pharmaceuticals, aerospace) may not be comfortable sending their technical data to cloud providers.
In each of these cases, "we'll use the OpenAI API" is not an option. The system must run on their hardware.
A self-contained on-premises AI system typically includes:
You need a way to run model inference without cloud APIs. Options:
Self-hosted open-source models: Llama 3, Mistral, Falcon, and other open weights models can be deployed on your own hardware. They require GPU servers but eliminate API dependency.
vLLM: High-throughput inference server for LLMs. Supports PagedAttention for efficient memory management. Production-grade.
Ollama: Simpler inference server, easier to set up, good for smaller deployments.
TensorRT-LLM: NVIDIA's inference optimization library. Extracts maximum performance from NVIDIA GPUs.
For computer vision and specialized ML models, ONNX Runtime and TensorFlow Serving are standard choices.
Models are large (7B parameter models are ~14GB in FP16; 70B models are ~140GB). You need:
MLflow or custom model registries are common choices. Docker containers with model artifacts are a practical packaging approach.
On-premises RAG and data-driven AI systems need local versions of infrastructure that's often taken for granted in the cloud:
Without cloud observability services (CloudWatch, DataDog's cloud tier), you need self-hosted alternatives:
This is often the most underestimated component of on-premises AI. Without observability, you have no visibility into what the system is doing.
Rough guidance for model inference hardware:
| Model Size | Minimum GPU RAM | Recommended | |---|---|---| | 7B parameters | 16GB (single GPU) | 24GB | | 13B parameters | 28GB | 2× 24GB | | 34B parameters | 70GB | 2× 40GB | | 70B parameters | 140GB | 4× 40GB |
For production workloads with concurrent requests, add 50–100% to these numbers.
CPU-only inference is possible with quantized models (using llama.cpp) but is 5–20x slower than GPU inference. Appropriate for low-throughput use cases.
On-premises deployment is the right call for the use cases described above. But it comes with real trade-offs:
Infrastructure cost: A server with 4× H100 GPUs costs $200,000–$300,000. Cloud inference at comparable throughput might cost $5,000–$20,000/month. The payback period depends on your workload.
Operational burden: You are now running and maintaining GPU servers. This requires GPU-savvy DevOps and a maintenance plan.
Model currency: Open source models lag behind frontier models (GPT-4, Claude, Gemini) in capability. If you need the best available capability, on-premises may mean accepting a capability deficit.
Update cycles: Updating models on-premises requires downloading large model files and re-deploying. Cloud API users get improvements automatically.
Scaling: Cloud infrastructure scales on demand. On-premises infrastructure requires hardware procurement (weeks or months lead time).
The teams that successfully deploy AI on-premises share a few practices:
Start with the right hardware — underpowered hardware creates a performance ceiling you'll be stuck with. Size for your peak load, not average load.
Use Docker and Kubernetes — containerized deployment makes model updates and scaling much simpler. Don't deploy directly on bare metal.
Invest in monitoring from day one — on-premises systems are harder to debug than cloud systems. You need visibility into what's happening.
Plan for model updates — open-source models improve rapidly. Build a pipeline for evaluating, downloading, and deploying new model versions.
Document the architecture — the engineer who built it won't be there forever. Thorough documentation protects the investment.
Several of our Co-Build projects are deployed fully on-premises for clients with strict data sovereignty requirements, including manufacturing, legal, and government clients. If you're evaluating on-premises AI, get in touch to discuss whether it's the right architecture for your situation.