Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...
Tags
llama.cpp
moe
Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...
llm-inference
Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...
epyc
Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...
benchmarking
Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...