OI Performance Benchmark Technical Review
To evaluate the inference capabilities of a large language model (LLM), we focus on two key metrics: latency and throughput. Latency Latency measures the time it takes for an LLM to generate a response to a user’s prompt. It is a critical indicator of a language model’s speed and significantly impacts a user’s perception of […]
