Running a self-hosted LLM in Kubernetes with vLLM

Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some... Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some teams choose for reasons such as cost predictability at high request volumes, more control over latency, and controlling data-residency to meet contractual or regulatory requirements. A hybrid approach (running open source models locally for high-volume or sensitive workloads while using managed API calls for tasks that benefit from them) is a practical middle ground that combines both. This article documents what it took to set up that kind of self-hosted inference stack in a Kubernetes lab environment. In this example, we’ve chosen to use vLLM for inference and LINSTOR® for persistent storage. Background on vLLM vLLM is a high-performance, open source inference engine for large language models. It is designed for serving many concurrent requests efficiently in a cluster environment. An important characteristic of vLLM for this use case is that it exposes an OpenAI-compatible REST API. Anything that already talks to the OpenAI API (LangChain, LlamaIndex, or your own code that calls the OpenAI SDK) can be pointed at a self-hosted vLLM instance with nothing more than a URL change. That compatibility is what makes the hybrid architecture described earlier a viable alternative. Overview of the setup Instructions in this article use a Kubernetes cluster with LINSTOR providing persistent storage through the LINSTOR Container Storage Interface (CSI) driver. Kubernetes is the orchestration layer for the entire stack, and the storage integration follows the standard CSI used across th

Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some... Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some teams choose for reasons such as cost predictability at high request volumes, more control over latency, and controlling data-residency to meet contractual or regulatory requirements. A hybrid approach (running open source models locally for high-volume or sensitive workloads while using managed API calls for tasks that benefit from them) is a practical middle ground that combines both. This article documents what it took to set up that kind of self-hosted inference stack in a Kubernetes lab environment. In this example, we’ve chosen to use vLLM for inference and LINSTOR® for persistent storage. Background on vLLM vLLM is a high-performance, open source inference engine for large language models. It is designed for serving many concurrent requests efficiently in a cluster environment. An important characteristic of vLLM for this use case is that it exposes an OpenAI-compatible REST API. Anything that already talks to the OpenAI API (LangChain, LlamaIndex, or your own code that calls the OpenAI SDK) can be pointed at a self-hosted vLLM instance with nothing more than a URL change. That compatibility is what makes the hybrid architecture described earlier a viable alternative. Overview of the setup Instructions in this article use a Kubernetes cluster with LINSTOR providing persistent storage through the LINSTOR Container Storage Interface (CSI) driver. Kubernetes is the orchestration layer for the entire stack, and the storage integration follows the standard CSI used across the cloud native ecosystem. LINSTOR is an open source software-defined storage solution built on DRBD® that provides replicated block storage across nodes. Replicated storage is a good fit for storing large model weight files that need to survive pod restarts and node failures. The model used in this blog is meta-llama/Llama-3.2-1B-Instruct , a small but capable model from Meta that has been fine-tuned to follow user instructions. At 1B parameters, it is lightweight enough to run on CPU (important for a lab without dedicated GPU nodes) while still being useful for testing the setup. Prerequisites Before deploying anything in Kubernetes, you need access to the model itself. Meta Llama models are gated on Hugging Face, meaning you need to request access before you can download them. Create an account at huggingface.co . Navigate to the Llama-3.2-1B-Instruct model page and submit an access request. After approval, go to your Hugging Face account settings and create an access token with read permissions. Keep that token nearby because you will need it in the next section. You will also need LINSTOR deployed into Kubernetes along with a StorageClass for Kubernetes workloads to request PersistentVolumeClaims . The open source Piraeus Operator deploys LINSTOR and the LINSTOR CSI driver into Kubernetes and documents the setup. In the next section you will create a PersistentVolumeClaim from a LINSTOR StorageClass named, linstor-csi-lvm-thin-r2. Deployment The deployment consists of three Kubernetes resources: a PersistentVolumeClaim for model storage, a Secret for the Hugging Face token, and a Deployment and Service to run the inference server. Creating the PVC and secret The PVC uses the linstor-csi-lvm-thin-r2 storage class, which provisions a thin-provisioned LVM volume with two replicas across the cluster. This provides both redundancy and efficient use of disk space, which is important when model weights can easily consume tens of gigabytes. The mountPath of the container is /root/.cache/huggingface . This is where the vLLM container caches downloaded model weights. By backing this path with a persistent volume, the model is downloaded from Hugging Face only once, and later pod restarts skip the download entirely because they will persist on the LINSTOR-provided persistent storage. Enter the following command to create the PVC and the Hugging Face token secret: cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vllm-models spec: storageClassName: linstor-csi-lvm-thin-r2 accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 50Gi --- apiVersion: v1 kind: Secret metadata: name: hf-token-secret type: Opaque stringData: token: "REPLACE_WITH_YOUR_TOKEN" EOF ❗ IMPORTANT: Replace REPLACE_WITH_YOUR_TOKEN with your actual Hugging Face access token and change the StorageClass name if yours is different, before applying this configuration. Deploying vLLM Enter the following command to deploy the vLLM inference server and its service, adapted from the vLLM Kubernetes deployment documentation : VLLM_IMAGE = public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: vllm-server spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: vllm template: metadata: labels: app.kubernetes.io/name: vllm spec: containers: - name: vllm image: $VLLM_IMAGE command: ["/bin/sh", "-c"] args: [ "vllm serve meta-llama/Llama-3.2-1B-Instruct --gpu-memory-utilization 0.80" ] env: - name: HF_TOKEN valueFrom: secretKeyRef: name: hf-token-secret key: token ports: - containerPort: 8000 volumeMounts: - name: llama-storage mountPath: /root/.cache/huggingface volumes: - name: llama-storage persistentVolumeClaim: claimName: vllm-models --- apiVersion: v1 kind: Service metadata: name: vllm-server spec: selector: app.kubernetes.io/name: vllm ports: - protocol: TCP port: 8000 targetPort: 8000 type: ClusterIP EOF 📝

Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some... Running large language model (LLM) workloads in-house is one of several patterns teams adopt alongside managed API services. Managed API services are convenient and well suited to many workloads. Self-hosting is a complementary option that some teams choose for reasons such as cost predictability at high request volumes, more control over latency, and controlling data-residency to meet contractual or regulatory requirements. A hybrid approach (running open source models locally for high-volume or sensitive workloads while using managed API calls for tasks that benefit from them) is a practical middle ground that combines both. This article documents what it took to set up that kind of self-hosted inference stack in a Kubernetes lab environment. In this example, we’ve chosen to use vLLM for inference and LINSTOR® for persistent storage. Background on vLLM vLLM is a high-performance, open source inference engine for large language models. It is designed for serving many concurrent requests efficiently in a cluster environment. An important characteristic of vLLM for this use case is that it exposes an OpenAI-compatible REST API. Anything that already talks to the OpenAI API (LangChain, LlamaIndex, or your own code that calls the OpenAI SDK) can be pointed at a self-hosted vLLM instance with nothing more than a URL change. That compatibility is what makes the hybrid architecture described earlier a viable alternative. Overview of the setup Instructions in this article use a Kubernetes cluster with LINSTOR providing persistent storage through the LINSTOR Container Storage Interface (CSI) driver. Kubernetes is the orchestration layer for the entire stack, and the storage integration follows the standard CSI used across the cloud native ecosystem. LINSTOR is an open source software-defined storage solution built on DRBD® that provides replicated block storage across nodes. Replicated storage is a good fit for storing large model weight files that need to survive pod restarts and node failures. The model used in this blog is meta-llama/Llama-3.2-1B-Instruct , a small but capable model from Meta that has been fine-tuned to follow user instructions. At 1B parameters, it is lightweight enough to run on CPU (important for a lab without dedicated GPU nodes) while still being useful for testing the setup. Prerequisites Before deploying anything in Kubernetes, you need access to the model itself. Meta Llama models are gated on Hugging Face, meaning you need to request access before you can download them. Create an account at huggingface.co . Navigate to the Llama-3.2-1B-Instruct model page and submit an access request. After approval, go to your Hugging Face account settings and create an access token with read permissions. Keep that token nearby because you will need it in the next section. You will also need LINSTOR deployed into Kubernetes along with a StorageClass for Kubernetes workloads to request PersistentVolumeClaims . The open source Piraeus Operator deploys LINSTOR and the LINSTOR CSI driver into Kubernetes and documents the setup. In the next section you will create a PersistentVolumeClaim from a LINSTOR StorageClass named, linstor-csi-lvm-thin-r2. Deployment The deployment consists of three Kubernetes resources: a PersistentVolumeClaim for model storage, a Secret for the Hugging Face token, and a Deployment and Service to run the inference server. Creating the PVC and secret The PVC uses the linstor-csi-lvm-thin-r2 storage class, which provisions a thin-provisioned LVM volume with two replicas across the cluster. This provides both redundancy and efficient use of disk space, which is important when model weights can easily consume tens of gigabytes. The mountPath of the container is /root/.cache/huggingface . This is where the vLLM container caches downloaded model weights. By backing this path with a persistent volume, the model is downloaded from Hugging Face only once, and later pod restarts skip the download entirely because they will persist on the LINSTOR-provided persistent storage. Enter the following command to create the PVC and the Hugging Face token secret: cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vllm-models spec: storageClassName: linstor-csi-lvm-thin-r2 accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 50Gi --- apiVersion: v1 kind: Secret metadata: name: hf-token-secret type: Opaque stringData: token: "REPLACE_WITH_YOUR_TOKEN" EOF ❗ IMPORTANT: Replace REPLACE_WITH_YOUR_TOKEN with your actual Hugging Face access token and change the StorageClass name if yours is different, before applying this configuration. Deploying vLLM Enter the following command to deploy the vLLM inference server and its service, adapted from the vLLM Kubernetes deployment documentation : VLLM_IMAGE = public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: vllm-server spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: vllm template: metadata: labels: app.kubernetes.io/name: vllm spec: containers: - name: vllm image: $VLLM_IMAGE command: ["/bin/sh", "-c"] args: [ "vllm serve meta-llama/Llama-3.2-1B-Instruct --gpu-memory-utilization 0.80" ] env: - name: HF_TOKEN valueFrom: secretKeyRef: name: hf-token-secret key: token ports: - containerPort: 8000 volumeMounts: - name: llama-storage mountPath: /root/.cache/huggingface volumes: - name: llama-storage persistentVolumeClaim: claimName: vllm-models --- apiVersion: v1 kind: Service metadata: name: vllm-server spec: selector: app.kubernetes.io/name: vllm ports: - protocol: TCP port: 8000 targetPort: 8000 type: ClusterIP EOF 📝