Online Serving Benchmark Summary

• model: gpt2
• dtype: fp16
• n: 200
• scales: [0.4, 0.8, 1.6]
• sampling: temperature=0.7, top_p=0.9, top_k=50
• wall_s: 225.87478081299923
• run_start_time: 2025-12-31T11:19:41+08:00
• run_end_time: 2026-01-01T08:11:17+08:00
• versions: git_rev=a547c7c, rosellm=0.1.0, vllm=0.7.2, sglang=0.4.6, tensorrt_llm=1.1.0, torch=2.6.0, transformers=4.51.3, python=3.11.11

scale	backend	TTFT p50/p90/p99 (ms)	TPOT p50/p90/p99 (ms)	ITL p50/p90/p99 (ms)	E2E p50/p90/p99 (ms)
0.40	roseinfer (+chunk1024, +batch16, +warmup cg16, +prefill meta, +fast plan)	8.06/11.51/14.80	1.33/1.44/1.58	1.27/1.52/2.55	172.45/186.81/207.18
0.40	roseinfer (+chunk1024, +chunk bucket, +batch16, +warmup cg16, +prefill meta, +fast plan)	8.11/11.64/17.40	1.18/1.35/1.52	1.13/1.38/2.36	154.02/169.88/203.97
0.40	roseinfer (in-proc, +chunk1024, +batch16, +prefill meta, +fast plan)	8.35/12.24/30.55	1.35/1.48/1.96	1.28/1.53/2.79	174.09/192.99/257.87
0.40	SGLang	7.80/9.39/14.89	1.10/1.21/1.35	1.07/1.28/2.85	144.13/156.65/170.35
0.40	TensorRT-LLM	5.82/6.30/8.03	1.40/1.44/1.90	1.40/1.53/2.62	182.62/187.42/193.03
0.40	vLLM	9.20/10.15/14.54	1.59/1.83/1.99	1.53/1.87/3.41	201.93/234.64/253.63
0.80	roseinfer (+chunk1024, +batch16, +warmup cg16, +prefill meta, +fast plan)	5.16/6.31/7.28	1.27/1.35/1.36	1.24/1.42/1.77	162.43/175.53/179.23
0.80	roseinfer (+chunk1024, +chunk bucket, +batch16, +warmup cg16, +prefill meta, +fast plan)	5.19/5.96/7.05	1.12/1.19/1.24	1.10/1.27/1.62	145.69/155.80/160.19
0.80	roseinfer (in-proc, +chunk1024, +batch16, +prefill meta, +fast plan)	4.00/5.15/5.45	1.28/1.36/1.38	1.25/1.42/2.14	162.59/175.32/179.33
0.80	SGLang	8.63/9.83/14.13	1.07/1.15/1.27	1.06/1.21/2.02	142.70/149.64/160.67
0.80	TensorRT-LLM	5.71/6.28/6.90	1.39/1.41/1.51	1.38/1.50/2.02	181.14/184.33/188.56
0.80	vLLM	9.19/10.36/10.99	1.45/1.66/1.78	1.42/1.69/2.38	186.06/211.76/232.81
1.60	roseinfer (+chunk1024, +batch16, +warmup cg16, +prefill meta, +fast plan)	5.29/6.20/6.63	1.26/1.33/1.44	1.23/1.40/1.70	161.71/173.39/188.51
1.60	roseinfer (+chunk1024, +chunk bucket, +batch16, +warmup cg16, +prefill meta, +fast plan)	5.21/5.89/6.38	1.11/1.18/1.25	1.09/1.24/1.53	144.36/154.56/163.51
1.60	roseinfer (in-proc, +chunk1024, +batch16, +prefill meta, +fast plan)	4.03/4.60/5.61	1.28/1.35/1.44	1.24/1.39/2.01	161.72/173.67/185.27
1.60	SGLang	8.87/9.98/14.93	1.06/1.16/1.34	1.05/1.20/2.02	142.46/151.47/166.25
1.60	TensorRT-LLM	5.88/6.43/7.02	1.38/1.41/1.53	1.38/1.50/1.86	180.76/184.17/199.04
1.60	vLLM	9.44/10.73/11.37	1.37/1.56/1.78	1.38/1.60/2.02	182.80/201.53/230.43