We size, deploy and operate GPU servers so you can run large language models (LLMs) and AI workloads in your own data center.
GPU server selection, procurement, deployment and operations for AI workloads such as LLM inference, RAG applications, fine-tuning and computer vision. We start from model size and cover VRAM, server architecture, networking, power and cooling and the software layer in one plan.
Every prompt and document sent to a public AI service leaves your organization. For workloads involving customer data, contracts, source code or trade secrets, that may be unacceptable for privacy and compliance. With an LLM running on your own GPU server, data stays inside.
In return, hardware investment and operations are your responsibility. If usage is periodic or you are still piloting, starting with GPU Cloud may make more sense.
The first step in sizing an LLM server is fitting the model into GPU memory (VRAM). Rough math for model weights: parameter count × bytes per parameter. 2 bytes in FP16/BF16, 1 byte with 8-bit quantization, about 0.5 bytes with 4-bit.
| Model size | FP16 / BF16 | 8-bit | 4-bit |
|---|---|---|---|
| 7–8B | ~16 GB | ~8 GB | ~4–5 GB |
| 13–14B | ~28 GB | ~14 GB | ~7–8 GB |
| 32–34B | ~64–68 GB | ~32–34 GB | ~16–18 GB |
| 70B | ~140 GB | ~70 GB | ~35–40 GB |
The table covers model weights only. More concurrent users and longer context require extra memory for the KV cache; in practice leave at least 20–30% headroom. For example, a 70B model in FP16 does not fit on a single 80 GB GPU and is split across several GPUs.
The right GPU depends on model size, concurrent users and whether you run inference or training. Common NVIDIA data center GPUs:
Availability, lead times and pricing change quickly, so the GPU model is finalized together at the proposal stage.
Organizations that want to use LLMs without sending data outside, teams building an internal assistant (RAG) over documents and knowledge bases, R&D units training or fine-tuning their own models, and manufacturing and security companies running computer vision projects.
A hypothetical law firm wants to search and summarize its contract archive with an LLM, but documents must not leave the firm. A 4-bit quantized 70B model runs with vLLM on a server with two 48 GB GPUs; documents are retrieved through RAG according to user permissions.
It depends on model size, concurrent users and workload type. L4 or L40S for small and mid-size models, H100 or H200 for large models and high throughput, and Blackwell platforms for next-generation and the largest workloads.
For weights alone, about 140 GB in FP16, 70 GB in 8-bit and 35–40 GB in 4-bit. KV cache for concurrent users and context length comes on top, so leave at least 20–30% headroom.
If utilization is sustained and high and data must stay inside, your own server can be more economical long term. For pilots, periodic training or uncertain demand, GPU Cloud lets you start without investment.
Because data stays in your infrastructure, it reduces cross-border transfer and third-party processing risk. Access rights, logging, retention and deletion policies still need to be designed alongside technical safeguards.
Open-weight families such as Llama, Qwen, Mistral and Gemma are widely used. License terms differ per model; check commercial use and user limits before deployment.
For Q&A over company documents, RAG is usually enough and needs no retraining when documents change. Fine-tuning is used to change a model's style, format or behavior on a specific task.
It depends on GPU count and model; a high-performance 8-GPU server can draw around 10 kW. Plan rack power capacity, cooling and UPS before deployment.
