Implementing Resource Quotas for CI/CD Runners in Kubernetes
This guide details how to enforce strict resource boundaries (CPU, Memory) on CI/CD runner deployments within a Kubernetes cluster using Namespaces and LimitRanges, preventing noisy neighbor problems.
Direct Diagnostic & Remediation Summary
- 1. Create Namespace: Ensure the dedicated namespace exists for runners.
kubectl create namespace ci-cd-runners. - 2. Apply ResourceQuota: Define the total resource ceiling for the namespace.
kubectl apply -f quota.yaml. - 3. Apply LimitRange: Set default requests/limits on all new Pods within that namespace.
kubectl apply -f limitrange.yaml.
Managing shared compute resources, especially from ephemeral workloads like CI/CD runners, is critical for cluster stability. Over-utilization by one project can starve others. This tutorial guides you through implementing Namespace-level quotas and enforcing Pod-level limits using Kubernetes mechanisms.
Executive Overview / Summary
This guide walks through the process of resource governance in Kubernetes. We will establish a dedicated namespace for CI/CD runners, then apply ResourceQuota to cap total cluster consumption within that scope, and use LimitRange to mandate default CPU/Memory requests and limits on every Pod deployed there.
Problem Overview & Error Symptoms
Without proper resource governance, CI/CD runners can lead to "noisy neighbor" syndrome. Symptoms include:
- Intermittent build failures due to CPU throttling or OOMKilled (Out Of Memory Killed) errors, even if overall cluster capacity seems available.
- Cluster instability when a single runaway job consumes excessive resources, impacting unrelated workloads.
- Inability to predict resource consumption for specific teams or projects running on shared infrastructure.
The root cause is the lack of explicit resource constraints (Requests and Limits) applied at the namespace or Pod level. Kubernetes scheduling relies on these values to ensure Quality of Service (QoS). If limits are missing, the scheduler might overcommit resources, leading to unpredictable behavior when nodes become saturated. ResourceQuota enforces ceiling checks, while LimitRange ensures every new workload adheres to a minimum standard.
Define the Dedicated Namespace and Quota
First, create an isolated namespace for all CI/CD runners. Then, apply a ResourceQuota object to set hard limits on total CPU, Memory, and potentially Service Accounts within this scope.
# 1. Create the dedicated namespace
kubectl create namespace ci-cd-runners
Next, define the quota YAML (e.g., quota.yaml):
apiVersion: v1
kind: ResourceQuota
metadata:
name: ci-cd-quota
namespace: ci-cd-runners
spec:
hard:
requests.cpu: "4"
requests.memory: 8Gi
limits.cpu: "10"
limits.memory: 20Gi
Apply the quota:
kubectl apply -f quota.yaml
Enforce Default Limits with LimitRange
The ResourceQuota only sets the ceiling. We must use LimitRange to ensure that any new Pod deployed into this namespace automatically requests and limits resources, preventing rogue deployments.
Define the limit range YAML (e.g., limitrange.yaml):
apiVersion: v1
kind: LimitRange
metadata:
name: ci-cd-limits
namespace: ci-cd-runners
spec:
limitsRecommendations:
- type: Container
default:
cpu: "200m"
memory: "256Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
warning:
containerName: "*"
Apply the limit range:
kubectl apply -f limitrange.yaml
Testing and Validation Deployment
To confirm the enforcement, deploy a test workload (e.g., a simple Job or Deployment) into this namespace. This Pod should automatically inherit the default requests/limits defined in LimitRange.
# Example deployment manifest for testing
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-runner
namespace: ci-cd-runners
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: runner-container
image: busybox
command: ["sh", "-c", "echo 'Test running...' && sleep 3"]
Deploy the test:
kubectl apply -f test-deployment.yaml
Check the resulting Pod specifications to confirm the default requests/limits were injected by LimitRange.
kubectl get pod test-runner -n ci-cd-runners -o yaml | grep -A 3 'resources:'
Common Pitfalls & Advanced Diagnostics
If resources are not being limited, check the following:
- Manifest Overrides: If a user explicitly defines resources: in their Pod spec (e.g., setting CPU to 0), this explicit definition will override the defaults set by LimitRange, effectively bypassing the safeguard unless the LimitRange is configured as mandatory.
- Quota Enforcement Check: To see if the quota is being approached or exceeded, use:
kubectl describe resourcequota ci-cd-quota -n ci-cd-runners
Check the Usage section for current consumption vs. hard limits. - Missing Namespaces: Ensure all components (Quota, LimitRange) are applied to the exact target namespace (ci-cd-runners).
If you see warnings about resource exhaustion, it means a job has hit the ResourceQuota ceiling.
Verification, Results & Long-Term Stability Validation
- Verify Quota Enforcement: Attempt to deploy a Pod that requires more total resources than the defined ResourceQuota (e.g., if quota is 10 CPU, try deploying services summing up to 12 CPU). The deployment should fail with an error indicating resource quota exceeded.
- Monitor Metrics: Use Prometheus/Grafana dashboards monitoring the kube-state-metrics endpoint to observe historical usage against the defined quotas. Look for sustained utilization near the limits.
- Preventative Maintenance: Periodically audit your CI/CD runner images and job definitions. As workloads evolve, update the default requests/limits in LimitRange to match modern baseline requirements (e.g., if all new jobs need 500m CPU, update the defaultRequest).
kubectl create namespace ci-cd-runners
kubectl apply -f quota.yaml
kubectl apply -f limitrange.yaml
kubectl get pod test-runner -n ci-cd-runners -o yaml | grep -A 3 'resources:'
Frequently Asked Questions
What happens if a user manually overrides the resource limits in their Pod spec?
If the explicit definition is present, it usually takes precedence over <strong class="terminal-content">LimitRange</strong> defaults. This requires developers to be aware of and respect the namespace's governance rules.
Can ResourceQuota stop me from deploying any Pods?
No. The quota only stops you when the sum of all resource requests (or limits, depending on how it's set) exceeds the defined hard limit. You will likely hit a specific ceiling first, not an absolute zero.
How do I check if my nodes have enough capacity left?
Check node status using <strong class="terminal-content">kubectl get nodes</strong>. For detailed resource pressure metrics across all resources (CPU/Memory), use the <strong class="terminal-content">kubectl top nodes</strong> command.
Need help implementing this solution?
Our cybersecurity and infrastructure engineers can audit, harden, or configure your systems directly.
Community Discussion & Confirmations (0)
Technical questions, hardware findings, and verified fix reports.
No reader questions posted yet. Have you tried this solution or experienced an edge case? Post below.