INFO 09-09 06:29:13 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 09-09 06:29:13 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 09-09 06:29:13 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 09-09 06:29:13 [__init__.py:238] Platform plugin ascend is activated
INFO 09-09 06:29:13 [platform.py:62] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet.
2026-09-09 06:29:25,242 - 2147587 - ServiceProfiler - INFO - VLLM_USE_V1 not set, auto-detected via vLLM 0.23.0+empty: default 1
INFO 09-09 06:29:26 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 09-09 06:29:26 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0xfffba21dec00>, trust_remote_code=False, seed=0, num_prompts=256, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='openai-chat', base_url=None, host='127.0.0.1', port=8331, endpoint='/v1/chat/completions', header=None, max_concurrency=64, model='qwen14b', input_len=None, output_len=None, tokenizer='/data/shared_models/Qwen--Qwen2.5-Coder-14B-Instruct', tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, disable_tqdm=False, num_warmups=8, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/vllm-workspace/vllm-ascend-split-batch-hust/docs/evidence/w2b-fi-sampling/results', result_filename='serve_api1_on_r1.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot,itl', metric_percentiles='50,95,99', goodput=None, request_id_prefix='bench-992ea03f-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 09-09 06:29:27 [utils.py:90] Sampling input_len from [1024, 1024] and output_len from [256, 256]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Warming up with 8 requests...

  0%|          | 0/8 [00:00<?, ?it/s]
 12%|█▎        | 1/8 [00:23<02:47, 23.89s/it]
100%|██████████| 8/8 [00:23<00:00,  2.99s/it]
Warmup run completed.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 64

  0%|          | 0/256 [00:00<?, ?it/s]
  0%|          | 1/256 [00:28<2:02:09, 28.74s/it]
  1%|          | 3/256 [00:28<31:34,  7.49s/it]  
  2%|▏         | 6/256 [00:29<12:13,  2.93s/it]
  3%|▎         | 8/256 [00:29<07:49,  1.89s/it]
  4%|▍         | 10/256 [00:29<05:15,  1.28s/it]
  5%|▍         | 12/256 [00:29<03:40,  1.11it/s]
  5%|▌         | 14/256 [00:30<02:38,  1.53it/s]
  6%|▋         | 16/256 [00:30<01:57,  2.05it/s]
  7%|▋         | 18/256 [00:30<01:29,  2.66it/s]
  8%|▊         | 20/256 [00:30<01:10,  3.34it/s]
  9%|▊         | 22/256 [00:31<00:57,  4.05it/s]
  9%|▉         | 24/256 [00:31<00:48,  4.74it/s]
 10%|█         | 26/256 [00:31<00:42,  5.41it/s]
 11%|█         | 28/256 [00:31<00:38,  5.97it/s]
 12%|█▏        | 30/256 [00:32<00:35,  6.44it/s]
 12%|█▏        | 31/256 [00:32<00:38,  5.82it/s]
 13%|█▎        | 33/256 [00:32<00:34,  6.38it/s]
 14%|█▎        | 35/256 [00:32<00:32,  6.79it/s]
 14%|█▍        | 37/256 [00:33<00:30,  7.11it/s]
 15%|█▌        | 39/256 [00:33<00:29,  7.35it/s]
 16%|█▌        | 41/256 [00:33<00:28,  7.51it/s]
 17%|█▋        | 43/256 [00:33<00:27,  7.66it/s]
 18%|█▊        | 45/256 [00:34<00:27,  7.72it/s]
 18%|█▊        | 47/256 [00:34<00:26,  7.79it/s]
 19%|█▉        | 49/256 [00:34<00:26,  7.84it/s]
 20%|█▉        | 51/256 [00:34<00:26,  7.81it/s]
 20%|██        | 52/256 [00:35<00:30,  6.70it/s]
 21%|██        | 54/256 [00:35<00:28,  7.07it/s]
 22%|██▏       | 56/256 [00:35<00:27,  7.33it/s]
 23%|██▎       | 58/256 [00:35<00:26,  7.52it/s]
 23%|██▎       | 60/256 [00:36<00:25,  7.67it/s]
 24%|██▍       | 62/256 [00:36<00:25,  7.75it/s]
 25%|██▌       | 64/256 [00:36<00:24,  7.80it/s]
 25%|██▌       | 65/256 [00:58<12:33,  3.95s/it]
 26%|██▌       | 67/256 [00:58<08:19,  2.64s/it]
 28%|██▊       | 71/256 [00:58<04:13,  1.37s/it]
 29%|██▊       | 73/256 [00:58<03:11,  1.05s/it]
 29%|██▉       | 75/256 [00:58<02:24,  1.25it/s]
 30%|███       | 77/256 [00:59<01:49,  1.64it/s]
 31%|███       | 79/256 [00:59<01:23,  2.11it/s]
 32%|███▏      | 81/256 [00:59<01:05,  2.68it/s]
 32%|███▏      | 83/256 [00:59<00:52,  3.32it/s]
 33%|███▎      | 84/256 [01:00<00:50,  3.42it/s]
 34%|███▎      | 86/256 [01:00<00:40,  4.20it/s]
 34%|███▍      | 88/256 [01:00<00:34,  4.93it/s]
 35%|███▌      | 90/256 [01:00<00:29,  5.61it/s]
 36%|███▌      | 92/256 [01:01<00:26,  6.16it/s]
 37%|███▋      | 94/256 [01:01<00:24,  6.62it/s]
 38%|███▊      | 96/256 [01:01<00:22,  6.96it/s]
 38%|███▊      | 98/256 [01:02<00:21,  7.22it/s]
 39%|███▉      | 100/256 [01:02<00:21,  7.42it/s]
 39%|███▉      | 101/256 [01:02<00:24,  6.41it/s]
 40%|████      | 103/256 [01:02<00:22,  6.85it/s]
 41%|████      | 105/256 [01:03<00:21,  7.15it/s]
 42%|████▏     | 107/256 [01:03<00:20,  7.40it/s]
 43%|████▎     | 109/256 [01:03<00:19,  7.54it/s]
 43%|████▎     | 111/256 [01:03<00:18,  7.66it/s]
 44%|████▍     | 113/256 [01:04<00:18,  7.74it/s]
 45%|████▍     | 115/256 [01:04<00:18,  7.78it/s]
 46%|████▌     | 117/256 [01:04<00:17,  7.84it/s]
 46%|████▌     | 118/256 [01:04<00:20,  6.67it/s]
 47%|████▋     | 120/256 [01:05<00:19,  7.04it/s]
 48%|████▊     | 122/256 [01:05<00:18,  7.31it/s]
 48%|████▊     | 124/256 [01:05<00:17,  7.52it/s]
 49%|████▉     | 126/256 [01:05<00:17,  7.63it/s]
 50%|█████     | 128/256 [01:06<00:16,  7.75it/s]
 50%|█████     | 129/256 [01:27<08:32,  4.04s/it]
 51%|█████     | 131/256 [01:28<05:37,  2.70s/it]
 52%|█████▏    | 133/256 [01:28<03:45,  1.83s/it]
 53%|█████▎    | 135/256 [01:28<02:33,  1.27s/it]
 54%|█████▎    | 137/256 [01:28<01:46,  1.11it/s]
 54%|█████▍    | 139/256 [01:28<01:17,  1.52it/s]
 55%|█████▌    | 141/256 [01:28<00:57,  2.01it/s]
 56%|█████▌    | 143/256 [01:29<00:43,  2.60it/s]
 56%|█████▋    | 144/256 [01:29<00:40,  2.77it/s]
 57%|█████▋    | 146/256 [01:29<00:31,  3.55it/s]
 58%|█████▊    | 148/256 [01:29<00:24,  4.33it/s]
 58%|█████▊    | 149/256 [01:30<00:25,  4.25it/s]
 59%|█████▉    | 151/256 [01:30<00:20,  5.05it/s]
 60%|█████▉    | 153/256 [01:30<00:17,  5.74it/s]
 61%|██████    | 155/256 [01:30<00:16,  6.31it/s]
 61%|██████▏   | 157/256 [01:31<00:14,  6.73it/s]
 62%|██████▏   | 159/256 [01:31<00:13,  7.07it/s]
 63%|██████▎   | 161/256 [01:31<00:12,  7.31it/s]
 64%|██████▎   | 163/256 [01:31<00:12,  7.47it/s]
 64%|██████▍   | 165/256 [01:32<00:11,  7.60it/s]
 65%|██████▍   | 166/256 [01:32<00:13,  6.53it/s]
 66%|██████▌   | 168/256 [01:32<00:12,  6.91it/s]
 66%|██████▋   | 170/256 [01:32<00:11,  7.21it/s]
 67%|██████▋   | 172/256 [01:33<00:11,  7.42it/s]
 68%|██████▊   | 174/256 [01:33<00:10,  7.57it/s]
 69%|██████▉   | 176/256 [01:33<00:10,  7.67it/s]
 70%|██████▉   | 178/256 [01:33<00:10,  7.75it/s]
 70%|███████   | 180/256 [01:34<00:09,  7.78it/s]
 71%|███████   | 182/256 [01:34<00:09,  7.84it/s]
 71%|███████▏  | 183/256 [01:34<00:10,  6.68it/s]
 72%|███████▏  | 185/256 [01:35<00:10,  7.05it/s]
 73%|███████▎  | 187/256 [01:35<00:09,  7.28it/s]
 74%|███████▍  | 189/256 [01:35<00:08,  7.51it/s]
 75%|███████▍  | 191/256 [01:35<00:08,  7.59it/s]
 75%|███████▌  | 193/256 [01:57<03:36,  3.43s/it]
 76%|███████▌  | 195/256 [01:57<02:27,  2.41s/it]
 77%|███████▋  | 197/256 [01:57<01:40,  1.71s/it]
 79%|███████▊  | 201/256 [01:58<00:51,  1.07it/s]
 80%|████████  | 205/256 [01:58<00:29,  1.72it/s]
 82%|████████▏ | 209/256 [01:58<00:18,  2.58it/s]
 83%|████████▎ | 213/256 [01:58<00:11,  3.68it/s]
 84%|████████▍ | 216/256 [01:58<00:08,  4.63it/s]
 86%|████████▌ | 220/256 [01:58<00:05,  6.27it/s]
 88%|████████▊ | 224/256 [01:59<00:03,  8.12it/s]
 89%|████████▉ | 228/256 [01:59<00:02, 10.05it/s]
 90%|█████████ | 231/256 [01:59<00:02, 11.05it/s]
 92%|█████████▏| 235/256 [01:59<00:01, 13.07it/s]
 93%|█████████▎| 239/256 [01:59<00:01, 14.85it/s]
 95%|█████████▍| 243/256 [02:00<00:00, 16.35it/s]
 96%|█████████▋| 247/256 [02:00<00:00, 17.60it/s]
 98%|█████████▊| 250/256 [02:00<00:00, 17.17it/s]
 99%|█████████▉| 254/256 [02:00<00:00, 18.02it/s]
100%|██████████| 256/256 [02:00<00:00,  2.12it/s]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  120.70    
Total input tokens:                      269567    
Total generated tokens:                  65536     
Request throughput (req/s):              2.12      
Output token throughput (tok/s):         542.94    
Peak output token throughput (tok/s):    704.00    
Peak concurrent requests:                76.00     
Total token throughput (tok/s):          2776.22   
---------------Time to First Token----------------
Mean TTFT (ms):                          1690.44   
Median TTFT (ms):                        1006.87   
P50 TTFT (ms):                           1006.87   
P95 TTFT (ms):                           6557.91   
P99 TTFT (ms):                           7869.01   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          110.19    
Median TPOT (ms):                        111.49    
P50 TPOT (ms):                           111.49    
P95 TPOT (ms):                           113.70    
P99 TPOT (ms):                           114.97    
---------------Inter-token Latency----------------
Mean ITL (ms):                           109.76    
Median ITL (ms):                         94.33     
P50 ITL (ms):                            94.33     
P95 ITL (ms):                            252.54    
P99 ITL (ms):                            255.12    
==================================================
