Online Serving Benchmark Summary
- model:
gpt2
- dtype:
fp16
- n:
200
- scales:
[0.4, 0.8, 1.6]
- sampling: temperature=0.7, top_p=0.9, top_k=50
- wall_s:
223.29104198999994
- run_start_time:
2025-12-30T10:50:49+08:00
- run_end_time:
2025-12-30T10:54:33+08:00
- versions:
git_rev=0cd08cb, rosellm=0.1.0, vllm=0.7.2, sglang=0.4.6, tensorrt_llm=1.1.0, torch=2.6.0, transformers=4.51.3, python=3.11.11
| scale |
backend |
TTFT p50/p90/p99 (ms) |
TPOT p50/p90/p99 (ms) |
ITL p50/p90/p99 (ms) |
E2E p50/p90/p99 (ms) |
| 0.40 |
roseinfer |
9.48/15.80/33.33 |
1.19/1.45/1.88 |
1.13/1.39/2.42 |
154.50/188.30/245.44 |
| 0.40 |
SGLang |
7.80/9.39/14.89 |
1.10/1.21/1.35 |
1.07/1.28/2.85 |
144.13/156.65/170.35 |
| 0.40 |
TensorRT-LLM |
5.82/6.30/8.03 |
1.40/1.44/1.90 |
1.40/1.53/2.62 |
182.62/187.42/193.03 |
| 0.40 |
vLLM |
9.20/10.15/14.54 |
1.59/1.83/1.99 |
1.53/1.87/3.41 |
201.93/234.64/253.63 |
| 0.80 |
roseinfer |
5.24/5.96/6.69 |
1.12/1.19/1.24 |
1.10/1.27/1.63 |
145.27/155.80/161.87 |
| 0.80 |
SGLang |
8.63/9.83/14.13 |
1.07/1.15/1.27 |
1.06/1.21/2.02 |
142.70/149.64/160.67 |
| 0.80 |
TensorRT-LLM |
5.71/6.28/6.90 |
1.39/1.41/1.51 |
1.38/1.50/2.02 |
181.14/184.33/188.56 |
| 0.80 |
vLLM |
9.19/10.36/10.99 |
1.45/1.66/1.78 |
1.42/1.69/2.38 |
186.06/211.76/232.81 |
| 1.60 |
roseinfer |
5.38/5.92/6.57 |
1.11/1.17/1.25 |
1.09/1.23/1.55 |
144.00/153.93/163.60 |
| 1.60 |
SGLang |
8.87/9.98/14.93 |
1.06/1.16/1.34 |
1.05/1.20/2.02 |
142.46/151.47/166.25 |
| 1.60 |
TensorRT-LLM |
5.88/6.43/7.02 |
1.38/1.41/1.53 |
1.38/1.50/1.86 |
180.76/184.17/199.04 |
| 1.60 |
vLLM |
9.44/10.73/11.37 |
1.37/1.56/1.78 |
1.38/1.60/2.02 |
182.80/201.53/230.43 |