Local AI Model Hardware Limitations: Memory, Bandwidth, Context, and Storage

1. Memory capacity (VRAM vs. RAM vs. Apple Silicon working set)

To run at usable speeds, a model's weights, attention cache, and runtime buffers must reside in memory the GPU can access directly. In Hugging Face's quantization documentation, an 8-billion-parameter model takes about 16 GB for weights alone in bf16 (2 bytes per parameter), about 8 GB at 8-bit precision, and roughly 4–5 GB at 4-bit precision (Hugging Face Transformers Quantization Overview). At the 70-billion-parameter scale, 16-bit (bf16/fp16) weights alone come to 140 GB decimal (~130.4 GiB binary at 2 bytes per parameter), exceeding a single 80 GB-class data-center GPU unless you quantize the weights or shard across multiple devices.

Sources: NVIDIA GeForce RTX 4090 Specs, Apple Support MacBook Pro Tech Specs, Apple Developer `recommendedMaxWorkingSetSize`, Ollama Hardware Support, Ollama FAQ, Apple Activity Monitor Guide.

One common hardware mistake on Mac is treating total unified memory as 100% dedicated VRAM. Apple's Metal API exposes recommendedMaxWorkingSetSize, an approximation of how much GPU memory an application can allocate before hurting runtime performance (Apple Developer Documentation). Because macOS reserves room for the operating system and active apps, a 24 GB Mac exposes noticeably less than 24 GiB to Metal compute backends, and LM Studio notes that while 8 GB Macs can run small models with modest context, 16 GB+ RAM is recommended (LM Studio System Requirements).

When a model does not fit cleanly into GPU memory, tools such as Ollama split layers across the GPU and system RAM. You can inspect that split directly while a model is loaded using ollama ps: 100% GPU means the full model is GPU-resident, whereas outputs such as 48%/52% CPU/GPU indicate partial CPU offloading and much slower responses (Ollama FAQ).

2. Memory bandwidth: the real limit on token speed

Even after a model fits inside VRAM or unified memory, memory bandwidth dictates how quickly it generates replies. During autoregressive decoding, a language model produces one output token at a time, and latency is dominated by transferring weights, KV tensors, and activations from memory rather than raw compute speed (NVIDIA Technical Blog: Inference Optimization). In standard dense models, generating each token requires reading all model weights from memory.

As an arithmetic upper bound (assuming 100% memory-bus utilization and ignoring compute overhead, quantization unpacking, and KV-cache reads):

Theoretical ceiling (tokens/s) ≈ Memory Bandwidth (GB/s) ÷ Active Model Footprint (GB)

Apple's M4 family illustrates why two Macs with similar CPU speeds perform very differently on local AI: Apple lists 120 GB/s memory bandwidth on M4, 273 GB/s on M4 Pro, and 410 GB/s to 546 GB/s on M4 Max (Apple Newsroom M4 Pro and M4 Max, Apple Support MacBook Pro Tech Specs). Dividing those bandwidth figures by a ~20 GB (decimal) quantized model yields theoretical bus ceilings of ~6.0 tokens/sec on base M4, ~13.7 tokens/sec on M4 Pro, and ~27.3 tokens/sec on a 546 GB/s M4 Max, before real-world driver, attention, and KV-cache overhead lowers sustained throughput further.

• CPU-only RAM bottleneck: Standard desktop or laptop system DDR memory typically has far lower bandwidth than GPU GDDR6X/HBM or high-tier unified memory, which is why CPU fallback feels dramatically slower even when a computer has 64 GB of system RAM.
• Dense vs. Mixture-of-Experts (MoE): An MoE model still requires enough RAM/VRAM and disk storage for all of its parameters, but because only a subset of expert parameters activates per token, its bandwidth demand per generated token is lower than an equally large dense model.

3. Context length and KV cache growth

Hardware tables often measure only the model file on disk and ignore the key-value (KV) cache, the working memory required to retain conversation history, system prompts, and pasted files without recomputing every earlier token (Hugging Face KV Cache Strategies).

For standard transformer architectures, KV-cache memory scales linearly with sequence length and batch size, governed by the number of layers, KV heads, head dimension, and cache precision (NVIDIA Technical Blog: Inference Optimization):

KV Cache Bytes = 2 × n_layers × n_kv_heads × d_head × bytes per element × context length × batch size

Calculating directly from Grouped-Query Attention (GQA) dimensions for a 32-layer model with 8 KV heads, head dimension 128, and bf16 (2-byte) cache entries gives 131,072 bytes (128 KiB) per token, adding roughly 0.50 GiB at 4K tokens, 1.00 GiB at 8K tokens, 4.00 GiB at 32K tokens, and 8.00 GiB at 64K tokens for a single request (using binary 1K = 1,024 tokens; at a decimal 64,000-token setting, the same model uses ~7.81 GiB). Serving multiple concurrent requests multiplies that allocation further; Ollama explicitly notes that parallel request processing scales context memory by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH (Ollama FAQ).

You can also curb context memory pressure with Flash Attention and KV-cache quantization. Ollama enables Flash Attention automatically on supported backends (or forces it with OLLAMA_FLASH_ATTENTION=1) and allows KV-cache quantization via OLLAMA_KV_CACHE_TYPE: q8_0 uses roughly half the memory of f16 with minimal precision loss, while q4_0 uses roughly one quarter with a larger potential quality trade-off (Ollama FAQ).

Hardware limits also trigger software context defaults. Ollama documents that it selects default context lengths from reported available VRAM (Ollama Context Length):

• Below 24 GiB VRAM: defaults to a 4k context window.
• 24–48 GiB VRAM: defaults to a 32k context window.
• 48 GiB VRAM or more: defaults to a 256k context window.

Note two important nuances:

1. Unified-memory threshold caveat on 24 GB Macs: Because Ollama picks default context windows from reported available VRAM (Ollama Context Length) while macOS manages GPU headroom under Metal's recommendedMaxWorkingSetSize (Apple Developer Documentation), a 24 GB unified-memory Mac may report less than 24 GiB of available GPU memory and drop into the 4k default tier, though Ollama's documentation does not explicitly confirm how Mac working-set rounding maps to these tiers. Check your actual allocation with ollama ps and override it with OLLAMA_CONTEXT_LENGTH only if the full model remains GPU-resident (Ollama Context Length).
2. Architecture caps: Not every architecture scales linearly across every layer: Hugging Face documents that models using sliding-window attention or chunked attention stop growing cache memory on those sliding layers once they reach their window size (Hugging Face KV Cache Strategies).

4. GPU compute backends, drivers, thermal limits, and power

Memory gets a model loaded, but compute compatibility, driver support, and power delivery determine whether hardware acceleration works at all:

• NVIDIA CUDA: Ollama supports NVIDIA GPUs with CUDA compute capability 5.0+ and driver version 550+ (or 570+ for compute capability 5.0–6.2 cards), with high-end desktop cards like the GeForce RTX 4090 offering 24 GB of GDDR6X memory alongside desktop thermal and PSU sizing demands (Ollama Hardware Support, NVIDIA RTX 4090 Specs).
• AMD and Intel GPUs: Ollama supports listed AMD Radeon RX, Radeon AI PRO, Radeon PRO, Ryzen AI, and Instinct GPUs on Linux via ROCm v7, and listed Radeon RX and PRO cards on Windows via ROCm v7 / HIP7-capable drivers. Additional GPU support on Windows and Linux, including extra AMD GPUs and Intel GPUs on Linux (with vendor or Mesa Vulkan driver setup), is provided via Vulkan, which is enabled by default when the backend is installed (Ollama Hardware Support).
• CPUs and macOS restrictions: Ollama supports Metal GPU acceleration on Apple M-series Macs running macOS 14 Sonoma or newer, while x86 Intel Macs are CPU-only (Ollama macOS Docs). LM Studio similarly lists Apple Silicon (M1/M2/M3/M4) on macOS 14.0+ as supported, notes Intel Macs are not supported, and requires AVX2 on x64 Windows/Linux CPUs (LM Studio System Requirements).
• Container virtualization limitation: GPU passthrough is not available inside Docker Desktop on macOS, meaning containerized local models on a Mac fall back to CPU unless the runtime executes natively on macOS (Ollama FAQ).
• Long prompts and sustained thermals: Generating output tokens is usually bandwidth-bound, whereas reading a long codebase or PDF (prompt prefill) is heavily compute-bound. Thin laptops without active cooling or desktop power headroom can throttle clocks during sustained prefill loops.

5. SSD storage, swap pressure, and local cache bloat

A common misconception is that a large or fast internal NVMe SSD can substitute for missing RAM or VRAM. While fast storage speeds up initially loading a model file into memory, running an active model out of virtual memory swap cripples generation speed because storage latency and bandwidth are orders of magnitude slower than GPU memory or unified RAM.

On macOS, Activity Monitor tracks this boundary directly under Memory: when Memory Pressure rises and Swap Used climbs on your startup disk, macOS is paging inactive data between RAM and SSD to stay afloat (Apple Activity Monitor Guide).

Where SSD capacity is a real hardware limitation is model hoarding and duplicate caches:

• Ollama notes that locally stored models can occupy tens to hundreds of gigabytes under ~/.ollama/models on macOS, though you can relocate them to another drive via OLLAMA_MODELS (Ollama macOS Docs, Ollama FAQ).
• Hugging Face libraries store downloaded checkpoints and shared revision blobs in ~/.cache/huggingface/hub unless redirected via HF_HOME or HF_HUB_CACHE (Hugging Face Manage Cache).
• Under macOS 27, Apple states its built-in Apple Intelligence models require up to 14 GB of local storage on Macs with M3 or later and at least 12 GB unified memory (or up to 8 GB on other eligible models), and notes that storage requirements may temporarily exceed those limits during and shortly after a software update (Apple Support: Apple Intelligence).

How to check your machine's real hardware limits before downloading

Run quick built-in checks before pulling a multi-gigabyte checkpoint:

1. Preview memory before loading (LM Studio CLI): Run lms load --estimate-only <model_key> (optionally adding --context-length and --gpu) to check estimated GPU and total memory before committing RAM (LM Studio `lms load`).
2. Verify 100% GPU residency (Ollama): After loading a model, run ollama ps and confirm the PROCESSOR column shows 100% GPU rather than splitting across CPU/GPU, and confirm CONTEXT matches your task (Ollama Context Length).
3. Check memory pressure and swap on Mac: Open Activity Monitor's Memory tab while generating text and watch Memory Pressure and Swap Used (Apple Activity Monitor Guide).
4. Audit disk space used by downloaded models: On macOS, inspect ~/.ollama/models and ~/.cache/huggingface/hub before pulling additional variants. If you want a visual breakdown across those scattered model directories alongside Xcode, Docker, and package caches on Mac, Diskfern Developer Storage scans readable folders so you can inspect large checkpoints before removing unused files yourself (note that disk tools only free SSD space, they do not expand RAM, VRAM, or token-generation speed).