Performance Optimization for Persistent Inference

Quantization: The Single Biggest Lever

Quantization is the most impactful optimization available for self-hosted AI. By reducing model weights from 16-bit floating point to 4-bit integers, you can run models with roughly one-quarter the VRAM requirement at a small accuracy cost (typically 1-3% on benchmarks). For an always-on AI server running on consumer or edge hardware, quantization is not optional — it is the difference between running a 13B model and being stuck at 7B.

The GGUF format (used by llama.cpp and Ollama) supports numerous quantization levels. The practical sweet spots are Q4_K_M (good quality, 4.5 bits-per-weight effective) for general use and IQ4_XS (slightly more aggressive, 4.25 bits-per-weight) when VRAM is extremely tight. For persistent Jetson AI deployments, IQ4_XS often makes the difference between running and not running a desired model.

On 24/7 AI hardware with ample VRAM (24GB+), Q5_K_M or Q6_K offer better quality while still running faster than FP16 due to reduced memory bandwidth requirements. Test your specific workload — different quantization levels affect coding, reasoning, and creative tasks differently.

KV Cache Tuning

The Key-Value (KV) cache stores attention states for processed tokens, enabling efficient generation without recomputing. Its size directly impacts both maximum context length and VRAM consumption. For long-running inference workloads, tuning KV cache parameters is essential:

• Context Length: Set to the maximum your workflow needs, not the model's theoretical maximum. Every token of context window costs VRAM. If your use case never exceeds 8K tokens, do not allocate 32K.
• Cache Quantization: vLLM and llama.cpp both support quantizing the KV cache (FP8 or INT8), reducing its VRAM footprint by roughly 50% with negligible accuracy loss.
• Prefix Caching: For serving applications with repeated system prompts, enable prefix caching to share KV cache entries across requests. Dramatically reduces time-to-first-token for common prompt patterns.

Batching & Throughput

Continuous batching (also called inflight batching) allows the model server to process multiple requests simultaneously, interleaving tokens from different sequences. This is the key throughput optimization for production long-running private AI serving. Frameworks like vLLM implement this automatically; Ollama supports it in newer versions.

Key batching parameters to tune:

• Max Batch Size: Start at the number of concurrent users you expect, monitor GPU utilization. Increase until VRAM is fully utilized or latency exceeds your SLA.
• Max Queue Delay: How long the server waits for additional requests before processing a batch. Lower = lower latency, higher = better throughput. For interactive applications, keep this under 200ms.

GPU Memory Management

Efficient VRAM usage distinguishes a well-tuned always-on AI server from one that constantly swaps to system RAM (killing performance). Strategies include:

• Model Offloading: With llama.cpp, you can split layers between GPU and CPU. Put as many layers as possible on GPU, offload the rest to RAM. Acceptable for batch processing; unacceptable for interactive serving.
• Flash Attention: Reduces VRAM usage for attention computation from O(n²) to O(n) for long sequences. Supported by vLLM, llama.cpp, and most modern serving frameworks.
• LoRA Adapters: Serve one base model with multiple task-specific LoRA adapters loaded simultaneously. Each adapter is kilobytes to megabytes, versus gigabytes for a separate model. Ideal for multi-task dedicated AI hardware.

# Monitor GPU memory in real-time
watch -n 1 nvidia-smi

# For Jetson:
sudo tegrastats

Power Efficiency for 24/7 Operation

When your inference node runs continuously, power consumption translates directly to operating cost. A 300W GPU running 24/7 at $0.12/kWh costs approximately $315/year in electricity alone. Multiply by a cluster of nodes and the economics demand attention. The persistent Jetson AI platform excels here — a 15-60W node costs $16-63/year to operate continuously.

Power optimization techniques:

• Power Limit: Use nvidia-smi -pl to cap GPU power draw. A 3090 limited to 250W loses roughly 10% throughput while saving 28% power — a worthwhile trade-off for reliable always-on computing.
• Clock Locking: Lock GPU and memory clocks at a fixed frequency to prevent thermal throttling cycles that cause latency spikes.
• Idle Model Unloading: If your workload has predictable quiet periods, unload models from VRAM and power-gate unused GPU components. The Linux uptime guide includes power management best practices applicable here.
• Thermal Monitoring: Sustained high temperatures not only increase power consumption (resistance rises with temperature) but accelerate component degradation. The STH always-on lab provides thermal benchmarks for various chassis and cooling configurations.

Benchmarking Your Stack

Systematic benchmarking is essential for tuning and capacity planning. Key metrics to track for an always-on AI server:

• Time to First Token (TTFT): How long before the first output token appears. Target under 500ms for interactive use.
• Tokens Per Second (TPS): Generation speed. 30+ TPS is comfortable for reading; below 10 TPS feels sluggish.
• Requests Per Second: Throughput under concurrent load. Depends heavily on batching configuration.
• VRAM Headroom: Always leave 1-2GB VRAM free for CUDA context and unexpected spikes.

For ClawBox 24/7 operation and similar compact deployments, the priority is maximizing TPS per watt rather than raw throughput. A Jetson Orin achieving 15 TPS at 30W is more impressive than a 4090 achieving 150 TPS at 400W when power budget is the constraint.