Benchmarking llama.cpp on legacy hardware

Marcus Groeber

Inspired by the benchmarking results for llama.cpp on Apple Silicon published here, I have used the llama-bench tool to produce comparable results on some older and less powerful devices, as well as newer ones.

This gives an idea of the rate of progress in LLM performance over a longer time period (even going back to a time before anyone thought of running DNNs of the size that we take for granted today on consumer hardware). The key metric is “tokens per second” for prompt processing (pp512) and next-token generation (tg128). Prompt processing usually benefits more from better parallelization.

These numbers are not so much intended as a scientifically accurate study, but more as a few quick orders-of-magnitude estimates across different platforms. It should be noted that most of these numbers are for CPU inference, even if a GPU was available.

Most tests run on Windows 10/11, binaries compiled with MSVC 2022.

Typical command line:

llama-bench.exe -m models\llama2\llama-2-7b.Q4_0.gguf -p 512 -n 128 -t 4

Dell Latitude E6420

  • Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
  • 8 GB RAM
  • Release date: 2011
  • SIMD extensions active: AVX

Build 80f19b4

modelsizeparamsbackendthreadstestt/s
llama 7B Q4_03.56 GiB6.74 BCPU4pp5121.98 ± 0.02
llama 7B Q4_03.56 GiB6.74 BCPU4tg1281.90 ± 0.10

ZOTAC ZBOX ID89 Plus

  • Intel(R) Core(TM) i5-3470T CPU @ 2.90GHz
  • 8 GB RAM
  • Release date: 2013
  • SIMD extensions active: AVX

Build e3af5563 (6878)

modelsizeparamsbackendthreadstestt/s
llama 7B Q4_03.56 GiB6.74 BCPU4pp5122.31 ± 0.02
llama 7B Q4_03.56 GiB6.74 BCPU4tg1282.46 ± 0.01

Lenovo Yoga 730-15IKB

This is an interesting machine, because it contains two GPUs, an NVIDIA GeForce 1050 and an Intel UHD Graphics 620, together with a fairly powerful (for the time) Intel Core i7 CPU. This allows for comparative benchmarking of all three backends on the same machine, giving some indication of the relative performance of each for AI workloads.

Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

  • 16 GB RAM
  • Release date: 2018
  • CPU backend. Compile options (for cross-compiling): -DGGML_AVX2=1 -DGGML_NATIVE=0

Build e3af5563 (6878)

modelsizeparamsbackendthreadstestt/s
llama 7B Q4_03.56 GiB6.74 BCPU4pp51213.40 ± 0.08
llama 7B Q4_03.56 GiB6.74 BCPU4tg1285.64 ± 0.14
llama 7B Q4_03.56 GiB6.74 BCPU8pp51216.29 ± 0.29
llama 7B Q4_03.56 GiB6.74 BCPU8tg1286.37 ± 0.08

NVIDIA GeForce GTX 1050, compute capability 6.1

  • VRAM: 4095 MiB
  • CUDA 12.9.2

Build a4ce2595c (10013)

  • Build 6878 was crashing for me
modelsizeparamsbackendngltestt/s
llama 7B Q4_03.56 GiB6.74 BCUDA-1pp512207.42 ± 0.88
llama 7B Q4_03.56 GiB6.74 BCUDA-1tg12816.56 ± 0.05

Intel(R) UHD Graphics 620 (OpenCL 2.1 NEO)

  • OpenCL backend, set GGML_OPENCL_DEVICE=1
  • OpenCL driver: 26.20.100.6913
  • global mem size: 6479 MB (shared GPU memory)

build: d67c0b410 (10122) with patches

  • Requires commenting out mul_mv_f16_f32_l4 kernel because of LLVM error "Error: Attempt to redefine function: ; Function Attrs: convergent nounwind readonly declare spir_func <4 x float> @_Z17__spirv_ImageReadPU3AS126opencl.image1d_buffer_ro_ti(%opencl.image1d_buffer_ro_t addrspace(1), i32) #5 => <4 x half> (%opencl.image1d_buffer_ro_t addrspace(1), i32)"
  • ggml_cl_glu must pass nullptr to enqueue_ndrange_kernel for local workgroup size, as default value 512 is bigger than max workgroup size of device (256).
modelsizeparamsbackendngltestt/s
llama 7B Q4_03.56 GiB6.74 BOpenCL-1pp5129.49 ± 0.04
llama 7B Q4_03.56 GiB6.74 BOpenCL-1tg1284.01 ± 0.05

Samsung S22 Ultra

  • Android 12 smartphone with Exynos 2200 S5E9925 CPU
  • 8 GB RAM (so smaller quantization used)
  • Release date: 2022

Build 80f19b4

modelsizeparamsbackendthreadstestt/s
llama 7B Q3_K - Small2.75 GiB6.74 BCPU8pp5122.53 ± 0.04
llama 7B Q3_K - Small2.75 GiB6.74 BCPU8tg1281.73 ± 0.53

Lenovo Yoga 7 2-in-1 Gen 10 (16AKP10)

  • AMD Ryzen AI 7 350 w/ Radeon 860M (GPU not used)
  • 32 GB RAM
  • Compile options (for cross-compiling): -DGGML_AVX512=1 -DGGML_NATIVE=0
  • Release date: 2025

Build e3af5563 (6878)

modelsizeparamsbackendthreadstestt/s
llama 7B Q4_03.56 GiB6.74 BCPU4pp51231.37 ± 0.16
llama 7B Q4_03.56 GiB6.74 BCPU4tg12810.49 ± 0.08
llama 7B Q4_03.56 GiB6.74 BCPU8pp51235.84 ± 0.25
llama 7B Q4_03.56 GiB6.74 BCPU8tg12810.27 ± 0.02
llama 7B Q4_03.56 GiB6.74 BCPU16pp51236.16 ± 0.16
llama 7B Q4_03.56 GiB6.74 BCPU16tg12810.06 ± 0.10