Overview

Setting up an AI server means preparing a GPU-equipped machine with the correct drivers, toolkits, and frameworks so you can run model inference or training locally. The process involves choosing appropriate hardware, installing a Linux or Windows server OS, configuring NVIDIA GPU drivers and CUDA, installing Python and an AI framework such as PyTorch, and finally loading a model for inference. This tutorial covers each stage in order, with practical commands and decision points so you can go from a freshly provisioned server to a working AI workload.

What Hardware Do You Need for an AI Server?

The single most important component is the GPU, but memory, storage, and network all shape what workloads your server can handle. A well-balanced configuration prevents bottlenecks that waste GPU cycles.

Component	Minimum for Inference	Recommended for Training
GPU	NVIDIA T4 (16 GB) or RTX 3090 (24 GB)	A100 (40/80 GB) or H100 (80 GB)
System RAM	32 GB	64–128 GB
Storage	500 GB NVMe SSD	1–4 TB NVMe RAID
Network	1 Gbps	10–25 Gbps
CPU	8-core modern x86_64	16–32 core, high single-thread

Why GPU selection matters: AI model size is measured in billions of parameters, and each parameter requires memory for loading and compute for inference. A 7-billion-parameter model in FP16 precision needs roughly 14 GB of VRAM just for weights, leaving headroom for activations and KV cache. If your GPU cannot hold the full model, you must split it across devices or use quantization, both of which add complexity. Choosing a GPU with sufficient VRAM upfront is the single biggest simplification you can make.

GPU VRAM vs. Model Size Reference

Model Size	FP32 VRAM	FP16 VRAM	INT8 Quantized
7B parameters	~28 GB	~14 GB	~7 GB
13B parameters	~52 GB	~26 GB	~13 GB
70B parameters	~280 GB	~140 GB	~70 GB

If your target model exceeds single-GPU VRAM, you will need multi-GPU inference with tensor parallelism or a quantized deployment strategy.

Which Operating System Should You Choose?

Linux (Ubuntu 22.04 LTS or Rocky Linux 9) is the standard choice for AI workloads because NVIDIA's driver and CUDA ecosystem has the deepest support there. Windows Server works for inference and development but introduces extra friction for multi-GPU training and containerized deployments.

Choose Linux if:

• You are running production inference or distributed training
• You need Docker or Kubernetes orchestration
• You want the widest compatibility with AI frameworks and tools

Choose Windows if:

• You are prototyping with tools like Ollama or LM Studio that have polished Windows installers
• Your team is more comfortable with Windows administration
• You need specific Windows-only software alongside AI workloads

For most AI server setups, Ubuntu 22.04 LTS is the safest default. It has long-term support, mature NVIDIA driver packages, and first-class support from every major AI framework.

How to Install and Verify NVIDIA GPU Drivers

Once your operating system is running, the first software layer is the NVIDIA driver. Without it, the GPU is invisible to your AI frameworks.

Step 1: Update System Packages

sudo apt update && sudo apt upgrade -y

Step 2: Install the NVIDIA Driver

For Ubuntu, the recommended approach is to use the NVIDIA driver metapackage:

sudo apt install -y nvidia-driver-535
sudo reboot

After reboot, verify the driver is loaded:

nvidia-smi

The output should display your GPU model, driver version, CUDA version, and current memory usage. If nvidia-smi returns an error, the driver is not correctly installed.

Step 3: Install the CUDA Toolkit

The CUDA toolkit provides the compiler and libraries that AI frameworks use to run computations on the GPU:

sudo apt install -y nvidia-cuda-toolkit
nvcc --version

For the latest CUDA versions, NVIDIA recommends installing from their official repository rather than the Ubuntu package. Follow the instructions on NVIDIA's CUDA download page for your specific version requirement.

Common Driver Troubleshooting

If you encounter a black screen or lose desktop access after driver installation on a Windows server, it may be because uninstalling or modifying system components like .NET Framework transitions the system to Core mode. A recovery guide for this scenario is available in the RakSmart knowledge base for Windows Server 2012 desktop access issues.

How to Set Up Python and AI Frameworks

With GPU drivers confirmed, the next layer is Python and your chosen AI framework. PyTorch is the most widely used framework for both inference and training.

Install Python and pip

sudo apt install -y python3 python3-pip python3-venv
python3 --version

Create an Isolated Environment

Always use a virtual environment to avoid dependency conflicts:

python3 -m venv ~/ai-env
source ~/ai-env/bin/activate

Install PyTorch with CUDA Support

Install the CUDA-enabled version of PyTorch (check pytorch.org for the latest command for your CUDA version):

pip install torch torchvision torchaudio --index-url

Verify GPU Access from PyTorch

import torch
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))

Both lines should return True and your GPU model name. If torch.cuda.is_available() returns False, revisit your driver and CUDA installation.

How to Deploy Your First AI Model for Inference

With the framework installed, you can load and run a model. Here is a minimal example using a small Hugging Face model.

Install Transformers

pip install transformers accelerate

Run a Text Generation Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
 model_name,
 torch_dtype=torch.float16,
 device_map="auto"
)

inputs = tokenizer("What is the capital of France?", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This example uses device_map="auto" to automatically place the model on the available GPU. For larger models, you can quantize with bitsandbytes or use vLLM for a production-grade inference server.

Setting Up a Persistent Inference Server

For production use, wrap your model in an API server. Tools like vLLM, TGI (Text Generation Inference), or a simple FastAPI wrapper provide HTTP endpoints:

pip install vllm
vllm serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000

This starts an OpenAI-compatible API server on port 8000, ready to accept inference requests from your applications.

Accessing and Managing Your AI Server Remotely

Once your AI server is running, you will typically manage it remotely. For Linux servers, SSH is the standard access method. For Windows servers, Remote Desktop Connection provides a graphical interface.

If you are deploying on a managed server, you can obtain login credentials from your provider's console and connect using your system's remote desktop or SSH client. The RakSmart guide on hosting servers walks through the console navigation for obtaining server access details.

Pre-Deployment Checklist

Before running production AI workloads, verify each item:

• GPU driver installed and nvidia-smi returns correct GPU info
• CUDA toolkit version matches your AI framework's requirements
• Python virtual environment created and activated
• AI framework (PyTorch, TensorFlow, etc.) installed with GPU support
• torch.cuda.is_available() returns True
• Model loads successfully and inference completes without errors
• Sufficient disk space for model weights (check model card for size)
• Firewall configured to expose only necessary ports (API port, SSH)
• Monitoring in place for GPU temperature, memory usage, and utilization

How to Choose Between Self-Hosted AI and Cloud AI Services

If you are deciding whether to set up your own AI server or use a cloud AI platform, the right choice depends on your workload volume, budget, and technical requirements.

Factor	Self-Hosted AI Server	Cloud AI Platform
Upfront cost	High (hardware purchase or dedicated server rental)	Low (pay-per-use)
Ongoing cost	Predictable monthly fee	Scales with usage
Latency	Low (local GPU access)	Variable (network round-trip)
Data privacy	Full control	Depends on provider policy
Scaling	Requires hardware procurement	Near-instant
Best for	Consistent, high-volume workloads	Sporadic or experimental workloads

If your AI workload runs consistently and requires low latency or data privacy, a dedicated GPU server is often more cost-effective over 12–24 months than equivalent cloud GPU hours. Providers like RAKsmart offer bare-metal GPU servers in Silicon Valley that can be provisioned for continuous AI inference or training workloads, giving you direct hardware access without the hypervisor overhead of shared cloud instances.

Frequently Asked Questions

How much does it cost to set up an AI server for inference?

Costs vary widely depending on GPU choice. A single NVIDIA T4 server for light inference workloads may cost $100–200 per month from a dedicated hosting provider. An A100-based server for larger models typically runs $500–1,500 per month. Self-purchasing a workstation GPU like the RTX 4090 costs around $1,600–2,000 upfront but requires additional investment in the host system, power, and cooling.

Can I set up an AI server on Windows instead of Linux?

Yes. Windows Server supports NVIDIA drivers, CUDA, PyTorch, and most AI frameworks. Windows is suitable for prototyping, small-scale inference, and development workflows. Linux remains preferred for production deployments, multi-GPU training, and container-based orchestration because of broader tooling support and lower system overhead.

What is the difference between CUDA and cuDNN, and do I need both?

CUDA is NVIDIA's parallel computing platform that lets software use GPU cores. cuDNN is a library of optimized deep learning primitives built on top of CUDA. Most AI frameworks bundle cuDNN with their installation, so you typically install CUDA and then let pip handle cuDNN through PyTorch or TensorFlow. You do not need to install cuDNN separately in most cases.

How do I know which GPU driver version to install?

Check your AI framework's documentation for the recommended CUDA version, then match your NVIDIA driver to that CUDA version. NVIDIA's driver download page lists compatible CUDA versions for each driver release. A mismatch between driver and CUDA is the most common cause of "CUDA not available" errors in PyTorch.

Can I run AI models without a dedicated GPU?

Yes, but with significant performance limitations. CPU-only inference works for small models (under 3 billion parameters) using optimized runtimes like ONNX Runtime or llama.cpp. For anything larger, a GPU is practically required for acceptable response times. If you are experimenting on a budget, quantized models on a CPU can still be useful for learning and development.

Conclusion

Setting up an AI server follows a clear progression: provision hardware with adequate GPU VRAM, install a compatible operating system, configure NVIDIA drivers and CUDA, set up a Python environment with your chosen AI framework, and deploy your model for inference. The key to a smooth setup is matching each layer to the next—your CUDA version must align with your driver, and your framework must be installed with the correct CUDA backend.

Once your foundational stack is working, you can layer on production features like API servers, load balancing, monitoring, and automated scaling. If you are looking for a GPU-equipped server to get started, exploring dedicated server options with NVIDIA GPUs provides the direct hardware access and predictable costs that AI workloads demand.