Building a Custom Metrics Exporter for Kubernetes

Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap. This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it. What a metrics exporter actually does An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules. In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code. The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes. Choosing what to measure Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types: Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down. Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges. Histograms record the distribution of obse

Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap. This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it. What a metrics exporter actually does An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules. In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code. The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes. Choosing what to measure Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types: Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down. Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges. Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages. Once you know which type fits your signal, choose a name that follows the convention _ _ in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later. Setting up the project The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency: mkdir my-exporter && cd my-exporter go mod init example.com/my-exporter go get github.com/prometheus/client_golang/prometheus go get github.com/prometheus/client_golang/prometheus/promhttp Registering metrics Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded: package main import ( "log" "net/http" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" ) var ( jobsProcessed = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "worker_jobs_processed_total", Help: "Total number of jobs processed, partitioned by status.", }, []string{"status"}, ) queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{ Name: "worker_queue_depth", Help: "Current number of jobs waiting in the queue.", }) jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "worker_job_duration_seconds", Help: "Time spent processing a single job.", Buckets: prometheus.DefBuckets, }) ) func init() { prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration) } prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself. Collecting real values With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker: import ( "math/rand" "time" ) func collectMetrics() { for { // Replace these with real reads from your application. depth := float64(rand.Intn(50)) queueDepth.Set(depth) start := time.Now() time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond) jobDuration.Observe(time.Since(start).Seconds()) jobsProcessed.WithLabelValues("success").Inc() time.Sleep(5 * time.Second) } } The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom. Exposing the endpoint Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route: func main() { go collectMetrics() http.Handle("/metrics", promhttp.Handler()) http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) log.Println("Listening on :8080") if err := http.ListenAndServe(":8080", nil); err != nil { log.Fatalf("server error: %v", err) } } Verify the output locally before building the image: go run . curl http://localhost:8080/metrics | grep worker_ You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter

Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap. This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it. What a metrics exporter actually does An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules. In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code. The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes. Choosing what to measure Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types: Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down. Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges. Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages. Once you know which type fits your signal, choose a name that follows the convention _ _ in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later. Setting up the project The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency: mkdir my-exporter && cd my-exporter go mod init example.com/my-exporter go get github.com/prometheus/client_golang/prometheus go get github.com/prometheus/client_golang/prometheus/promhttp Registering metrics Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded: package main import ( "log" "net/http" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" ) var ( jobsProcessed = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "worker_jobs_processed_total", Help: "Total number of jobs processed, partitioned by status.", }, []string{"status"}, ) queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{ Name: "worker_queue_depth", Help: "Current number of jobs waiting in the queue.", }) jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "worker_job_duration_seconds", Help: "Time spent processing a single job.", Buckets: prometheus.DefBuckets, }) ) func init() { prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration) } prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself. Collecting real values With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker: import ( "math/rand" "time" ) func collectMetrics() { for { // Replace these with real reads from your application. depth := float64(rand.Intn(50)) queueDepth.Set(depth) start := time.Now() time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond) jobDuration.Observe(time.Since(start).Seconds()) jobsProcessed.WithLabelValues("success").Inc() time.Sleep(5 * time.Second) } } The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom. Exposing the endpoint Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route: func main() { go collectMetrics() http.Handle("/metrics", promhttp.Handler()) http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) log.Println("Listening on :8080") if err := http.ListenAndServe(":8080", nil); err != nil { log.Fatalf("server error: %v", err) } } Verify the output locally before building the image: go run . curl http://localhost:8080/metrics | grep worker_ You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter