Finding the Wall: How to Actually Test LLM Inference Performance
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...
A repeatable method for finding a machine’s real inference limits — demonstrated on a 64-core EPYC server running 750B-parameter MoE models, from raw memory ...