INFO 09-09 06:24:07 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 09-09 06:24:07 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 09-09 06:24:07 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 09-09 06:24:07 [__init__.py:238] Platform plugin ascend is activated
INFO 09-09 06:24:07 [platform.py:62] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet.
2026-09-09 06:24:18,650 - 2145768 - ServiceProfiler - INFO - VLLM_USE_V1 not set, auto-detected via vLLM 0.23.0+empty: default 1
INFO 09-09 06:24:19 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 09-09 06:24:19 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0xfffb77642520>, trust_remote_code=False, seed=0, num_prompts=256, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='openai-chat', base_url=None, host='127.0.0.1', port=8331, endpoint='/v1/chat/completions', header=None, max_concurrency=64, model='qwen14b', input_len=None, output_len=None, tokenizer='/data/shared_models/Qwen--Qwen2.5-Coder-14B-Instruct', tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, disable_tqdm=False, num_warmups=8, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/vllm-workspace/vllm-ascend-split-batch-hust/docs/evidence/w2b-fi-sampling/results', result_filename='serve_api1_off_r1.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot,itl', metric_percentiles='50,95,99', goodput=None, request_id_prefix='bench-56584081-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 09-09 06:24:20 [utils.py:90] Sampling input_len from [1024, 1024] and output_len from [256, 256]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Warming up with 8 requests...

  0%|          | 0/8 [00:00<?, ?it/s]
 12%|█▎        | 1/8 [00:24<02:51, 24.44s/it]
100%|██████████| 8/8 [00:24<00:00,  3.06s/it]
Warmup run completed.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 64

  0%|          | 0/256 [00:00<?, ?it/s]
  0%|          | 1/256 [00:29<2:04:58, 29.41s/it]
  1%|          | 3/256 [00:29<32:27,  7.70s/it]  
  2%|▏         | 5/256 [00:29<15:45,  3.77s/it]
  3%|▎         | 7/256 [00:29<09:07,  2.20s/it]
  4%|▎         | 9/256 [00:30<05:50,  1.42s/it]
  4%|▍         | 11/256 [00:30<03:56,  1.04it/s]
  5%|▌         | 13/256 [00:30<02:46,  1.46it/s]
  6%|▌         | 15/256 [00:30<02:01,  1.98it/s]
  7%|▋         | 17/256 [00:31<01:32,  2.59it/s]
  7%|▋         | 19/256 [00:31<01:12,  3.27it/s]
  8%|▊         | 21/256 [00:31<00:58,  3.99it/s]
  9%|▉         | 23/256 [00:31<00:49,  4.69it/s]
 10%|▉         | 25/256 [00:32<00:43,  5.34it/s]
 11%|█         | 27/256 [00:32<00:38,  5.92it/s]
 11%|█▏        | 29/256 [00:32<00:35,  6.42it/s]
 12%|█▏        | 30/256 [00:32<00:39,  5.77it/s]
 12%|█▎        | 32/256 [00:33<00:35,  6.36it/s]
 13%|█▎        | 34/256 [00:33<00:32,  6.78it/s]
 14%|█▍        | 36/256 [00:33<00:30,  7.10it/s]
 15%|█▍        | 38/256 [00:33<00:29,  7.27it/s]
 16%|█▌        | 40/256 [00:34<00:28,  7.50it/s]
 16%|█▋        | 42/256 [00:34<00:28,  7.61it/s]
 17%|█▋        | 44/256 [00:34<00:27,  7.73it/s]
 18%|█▊        | 46/256 [00:34<00:27,  7.77it/s]
 19%|█▉        | 48/256 [00:35<00:26,  7.84it/s]
 20%|█▉        | 50/256 [00:35<00:26,  7.85it/s]
 20%|█▉        | 51/256 [00:35<00:30,  6.70it/s]
 21%|██        | 53/256 [00:35<00:28,  7.05it/s]
 21%|██▏       | 55/256 [00:36<00:27,  7.30it/s]
 22%|██▏       | 57/256 [00:36<00:26,  7.49it/s]
 23%|██▎       | 59/256 [00:36<00:25,  7.62it/s]
 24%|██▍       | 61/256 [00:36<00:25,  7.73it/s]
 25%|██▍       | 63/256 [00:37<00:24,  7.78it/s]
 25%|██▌       | 65/256 [00:59<11:01,  3.46s/it]
 26%|██▌       | 67/256 [00:59<07:41,  2.44s/it]
 27%|██▋       | 68/256 [00:59<06:21,  2.03s/it]
 27%|██▋       | 70/256 [00:59<04:14,  1.37s/it]
 28%|██▊       | 72/256 [00:59<02:54,  1.05it/s]
 29%|██▉       | 74/256 [01:00<02:05,  1.45it/s]
 30%|██▉       | 76/256 [01:00<01:32,  1.95it/s]
 30%|███       | 78/256 [01:00<01:10,  2.54it/s]
 31%|███▏      | 80/256 [01:00<00:54,  3.20it/s]
 32%|███▏      | 82/256 [01:01<00:44,  3.90it/s]
 33%|███▎      | 84/256 [01:01<00:37,  4.60it/s]
 33%|███▎      | 85/256 [01:01<00:38,  4.47it/s]
 34%|███▍      | 87/256 [01:01<00:32,  5.23it/s]
 35%|███▍      | 89/256 [01:02<00:28,  5.85it/s]
 36%|███▌      | 91/256 [01:02<00:25,  6.39it/s]
 36%|███▋      | 93/256 [01:02<00:24,  6.78it/s]
 37%|███▋      | 95/256 [01:03<00:22,  7.11it/s]
 38%|███▊      | 97/256 [01:03<00:21,  7.34it/s]
 39%|███▊      | 99/256 [01:03<00:20,  7.49it/s]
 39%|███▉      | 101/256 [01:03<00:20,  7.61it/s]
 40%|███▉      | 102/256 [01:04<00:23,  6.52it/s]
 41%|████      | 104/256 [01:04<00:22,  6.90it/s]
 41%|████▏     | 106/256 [01:04<00:20,  7.18it/s]
 42%|████▏     | 108/256 [01:04<00:19,  7.44it/s]
 43%|████▎     | 110/256 [01:05<00:19,  7.56it/s]
 44%|████▍     | 112/256 [01:05<00:18,  7.68it/s]
 45%|████▍     | 114/256 [01:05<00:18,  7.75it/s]
 45%|████▌     | 116/256 [01:05<00:17,  7.78it/s]
 46%|████▌     | 118/256 [01:06<00:17,  7.83it/s]
 46%|████▋     | 119/256 [01:06<00:20,  6.66it/s]
 47%|████▋     | 121/256 [01:06<00:19,  7.03it/s]
 48%|████▊     | 123/256 [01:06<00:18,  7.27it/s]
 49%|████▉     | 125/256 [01:07<00:17,  7.49it/s]
 50%|████▉     | 127/256 [01:07<00:16,  7.61it/s]
 50%|█████     | 129/256 [01:29<07:19,  3.46s/it]
 51%|█████     | 131/256 [01:29<05:03,  2.43s/it]
 52%|█████▏    | 133/256 [01:29<03:32,  1.73s/it]
 53%|█████▎    | 135/256 [01:29<02:28,  1.23s/it]
 54%|█████▎    | 137/256 [01:29<01:44,  1.13it/s]
 54%|█████▍    | 139/256 [01:30<01:16,  1.53it/s]
 55%|█████▌    | 141/256 [01:30<00:57,  2.02it/s]
 56%|█████▌    | 143/256 [01:30<00:43,  2.60it/s]
 57%|█████▋    | 145/256 [01:30<00:34,  3.25it/s]
 57%|█████▋    | 147/256 [01:31<00:27,  3.94it/s]
 58%|█████▊    | 149/256 [01:31<00:23,  4.65it/s]
 59%|█████▊    | 150/256 [01:31<00:23,  4.50it/s]
 59%|█████▉    | 152/256 [01:31<00:19,  5.24it/s]
 60%|██████    | 154/256 [01:32<00:17,  5.87it/s]
 61%|██████    | 156/256 [01:32<00:15,  6.40it/s]
 62%|██████▏   | 158/256 [01:32<00:14,  6.80it/s]
 62%|██████▎   | 160/256 [01:32<00:13,  7.12it/s]
 63%|██████▎   | 162/256 [01:33<00:12,  7.34it/s]
 64%|██████▍   | 164/256 [01:33<00:12,  7.49it/s]
 65%|██████▍   | 166/256 [01:33<00:11,  7.60it/s]
 65%|██████▌   | 167/256 [01:33<00:13,  6.52it/s]
 66%|██████▌   | 169/256 [01:34<00:12,  6.86it/s]
 67%|██████▋   | 171/256 [01:34<00:11,  7.22it/s]
 68%|██████▊   | 173/256 [01:34<00:11,  7.43it/s]
 68%|██████▊   | 175/256 [01:34<00:10,  7.57it/s]
 69%|██████▉   | 177/256 [01:35<00:10,  7.68it/s]
 70%|██████▉   | 179/256 [01:35<00:09,  7.76it/s]
 71%|███████   | 181/256 [01:35<00:09,  7.77it/s]
 71%|███████▏  | 183/256 [01:36<00:09,  7.84it/s]
 72%|███████▏  | 184/256 [01:36<00:10,  6.66it/s]
 73%|███████▎  | 186/256 [01:36<00:09,  7.02it/s]
 73%|███████▎  | 188/256 [01:36<00:09,  7.27it/s]
 74%|███████▍  | 190/256 [01:37<00:08,  7.49it/s]
 75%|███████▌  | 192/256 [01:37<00:08,  7.60it/s]
 75%|███████▌  | 193/256 [01:59<04:16,  4.07s/it]
 76%|███████▌  | 195/256 [01:59<02:45,  2.71s/it]
 77%|███████▋  | 197/256 [01:59<01:49,  1.85s/it]
 78%|███████▊  | 199/256 [01:59<01:12,  1.28s/it]
 79%|███████▉  | 202/256 [01:59<00:42,  1.28it/s]
 80%|████████  | 206/256 [01:59<00:23,  2.16it/s]
 82%|████████▏ | 210/256 [02:00<00:13,  3.29it/s]
 84%|████████▎ | 214/256 [02:00<00:08,  4.69it/s]
 85%|████████▍ | 217/256 [02:00<00:06,  5.80it/s]
 86%|████████▋ | 221/256 [02:00<00:04,  7.70it/s]
 88%|████████▊ | 225/256 [02:00<00:03,  9.68it/s]
 89%|████████▉ | 229/256 [02:01<00:02, 11.64it/s]
 91%|█████████ | 232/256 [02:01<00:01, 12.41it/s]
 92%|█████████▏| 236/256 [02:01<00:01, 14.29it/s]
 94%|█████████▍| 240/256 [02:01<00:01, 15.85it/s]
 95%|█████████▌| 244/256 [02:01<00:00, 17.14it/s]
 97%|█████████▋| 248/256 [02:02<00:00, 18.19it/s]
 98%|█████████▊| 251/256 [02:02<00:00, 17.44it/s]
100%|██████████| 256/256 [02:02<00:00,  2.09it/s]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  122.32    
Total input tokens:                      269567    
Total generated tokens:                  65536     
Request throughput (req/s):              2.09      
Output token throughput (tok/s):         535.77    
Peak output token throughput (tok/s):    704.00    
Peak concurrent requests:                74.00     
Total token throughput (tok/s):          2739.52   
---------------Time to First Token----------------
Mean TTFT (ms):                          1725.81   
Median TTFT (ms):                        1008.39   
P50 TTFT (ms):                           1008.39   
P95 TTFT (ms):                           6780.81   
P99 TTFT (ms):                           8043.67   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          111.66    
Median TPOT (ms):                        113.48    
P50 TPOT (ms):                           113.48    
P95 TPOT (ms):                           114.67    
P99 TPOT (ms):                           115.83    
---------------Inter-token Latency----------------
Mean ITL (ms):                           111.23    
Median ITL (ms):                         95.58     
P50 ITL (ms):                            95.58     
P95 ITL (ms):                            253.01    
P99 ITL (ms):                            255.28    
==================================================
