INFO 09-09 06:34:03 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 09-09 06:34:03 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 09-09 06:34:03 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 09-09 06:34:03 [__init__.py:238] Platform plugin ascend is activated
INFO 09-09 06:34:03 [platform.py:62] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet.
2026-09-09 06:34:14,326 - 2152344 - ServiceProfiler - INFO - VLLM_USE_V1 not set, auto-detected via vLLM 0.23.0+empty: default 1
INFO 09-09 06:34:15 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 09-09 06:34:15 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0xfffb71b26520>, trust_remote_code=False, seed=0, num_prompts=256, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='openai-chat', base_url=None, host='127.0.0.1', port=8331, endpoint='/v1/chat/completions', header=None, max_concurrency=64, model='qwen14b', input_len=None, output_len=None, tokenizer='/data/shared_models/Qwen--Qwen2.5-Coder-14B-Instruct', tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, disable_tqdm=False, num_warmups=8, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/vllm-workspace/vllm-ascend-split-batch-hust/docs/evidence/w2b-fi-sampling/results', result_filename='serve_api1_off_r2.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot,itl', metric_percentiles='50,95,99', goodput=None, request_id_prefix='bench-926aeb4a-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 09-09 06:34:16 [utils.py:90] Sampling input_len from [1024, 1024] and output_len from [256, 256]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Warming up with 8 requests...

  0%|          | 0/8 [00:00<?, ?it/s]
 12%|█▎        | 1/8 [00:23<02:47, 23.99s/it]
100%|██████████| 8/8 [00:23<00:00,  3.00s/it]
Warmup run completed.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 64

  0%|          | 0/256 [00:00<?, ?it/s]
  0%|          | 1/256 [00:28<2:03:10, 28.98s/it]
  1%|          | 3/256 [00:29<31:59,  7.59s/it]  
  2%|▏         | 6/256 [00:29<12:18,  2.95s/it]
  4%|▎         | 9/256 [00:29<06:43,  1.63s/it]
  4%|▍         | 11/256 [00:29<04:46,  1.17s/it]
  5%|▌         | 13/256 [00:30<03:26,  1.18it/s]
  6%|▌         | 15/256 [00:30<02:32,  1.59it/s]
  7%|▋         | 17/256 [00:30<01:54,  2.09it/s]
  7%|▋         | 19/256 [00:30<01:28,  2.69it/s]
  8%|▊         | 21/256 [00:31<01:10,  3.36it/s]
  9%|▉         | 23/256 [00:31<00:57,  4.06it/s]
 10%|▉         | 25/256 [00:31<00:48,  4.75it/s]
 11%|█         | 27/256 [00:31<00:42,  5.40it/s]
 11%|█▏        | 29/256 [00:32<00:37,  5.98it/s]
 12%|█▏        | 30/256 [00:32<00:41,  5.47it/s]
 12%|█▎        | 32/256 [00:32<00:36,  6.11it/s]
 13%|█▎        | 34/256 [00:32<00:33,  6.60it/s]
 14%|█▍        | 36/256 [00:33<00:31,  6.97it/s]
 15%|█▍        | 38/256 [00:33<00:30,  7.23it/s]
 16%|█▌        | 40/256 [00:33<00:29,  7.44it/s]
 16%|█▋        | 42/256 [00:33<00:28,  7.57it/s]
 17%|█▋        | 44/256 [00:34<00:27,  7.69it/s]
 18%|█▊        | 46/256 [00:34<00:27,  7.77it/s]
 19%|█▉        | 48/256 [00:34<00:26,  7.84it/s]
 20%|█▉        | 50/256 [00:34<00:26,  7.83it/s]
 20%|█▉        | 51/256 [00:35<00:30,  6.72it/s]
 21%|██        | 53/256 [00:35<00:28,  7.08it/s]
 21%|██▏       | 55/256 [00:35<00:27,  7.33it/s]
 22%|██▏       | 57/256 [00:35<00:26,  7.52it/s]
 23%|██▎       | 59/256 [00:36<00:25,  7.65it/s]
 24%|██▍       | 61/256 [00:36<00:25,  7.77it/s]
 25%|██▍       | 63/256 [00:36<00:24,  7.81it/s]
 25%|██▌       | 65/256 [00:58<10:44,  3.38s/it]
 26%|██▌       | 67/256 [00:58<07:30,  2.38s/it]
 27%|██▋       | 69/256 [00:58<05:14,  1.68s/it]
 28%|██▊       | 71/256 [00:58<03:40,  1.19s/it]
 29%|██▊       | 73/256 [00:58<02:37,  1.17it/s]
 29%|██▉       | 75/256 [00:59<01:55,  1.57it/s]
 30%|███       | 77/256 [00:59<01:26,  2.06it/s]
 31%|███       | 79/256 [00:59<01:06,  2.66it/s]
 31%|███▏      | 80/256 [00:59<01:02,  2.82it/s]
 32%|███▏      | 82/256 [01:00<00:48,  3.59it/s]
 33%|███▎      | 84/256 [01:00<00:39,  4.36it/s]
 33%|███▎      | 85/256 [01:00<00:39,  4.28it/s]
 34%|███▍      | 87/256 [01:00<00:33,  5.09it/s]
 35%|███▍      | 89/256 [01:01<00:28,  5.77it/s]
 36%|███▌      | 91/256 [01:01<00:26,  6.33it/s]
 36%|███▋      | 93/256 [01:01<00:24,  6.74it/s]
 37%|███▋      | 95/256 [01:01<00:22,  7.07it/s]
 38%|███▊      | 97/256 [01:02<00:21,  7.31it/s]
 39%|███▊      | 99/256 [01:02<00:20,  7.50it/s]
 39%|███▉      | 101/256 [01:02<00:20,  7.61it/s]
 40%|███▉      | 102/256 [01:02<00:23,  6.54it/s]
 41%|████      | 104/256 [01:03<00:21,  6.93it/s]
 41%|████▏     | 106/256 [01:03<00:20,  7.22it/s]
 42%|████▏     | 108/256 [01:03<00:19,  7.43it/s]
 43%|████▎     | 110/256 [01:04<00:19,  7.58it/s]
 44%|████▍     | 112/256 [01:04<00:18,  7.70it/s]
 45%|████▍     | 114/256 [01:04<00:18,  7.78it/s]
 45%|████▌     | 116/256 [01:04<00:17,  7.81it/s]
 46%|████▌     | 118/256 [01:05<00:17,  7.86it/s]
 46%|████▋     | 119/256 [01:05<00:20,  6.69it/s]
 47%|████▋     | 121/256 [01:05<00:19,  7.05it/s]
 48%|████▊     | 123/256 [01:05<00:18,  7.31it/s]
 49%|████▉     | 125/256 [01:06<00:17,  7.52it/s]
 50%|████▉     | 127/256 [01:06<00:16,  7.63it/s]
 50%|█████     | 129/256 [01:27<07:13,  3.41s/it]
 51%|█████     | 131/256 [01:28<04:59,  2.40s/it]
 52%|█████▏    | 133/256 [01:28<03:29,  1.71s/it]
 53%|█████▎    | 135/256 [01:28<02:26,  1.21s/it]
 54%|█████▎    | 137/256 [01:28<01:43,  1.15it/s]
 54%|█████▍    | 139/256 [01:28<01:15,  1.55it/s]
 55%|█████▌    | 141/256 [01:29<00:56,  2.04it/s]
 56%|█████▌    | 143/256 [01:29<00:43,  2.63it/s]
 57%|█████▋    | 145/256 [01:29<00:33,  3.28it/s]
 57%|█████▋    | 147/256 [01:29<00:27,  3.97it/s]
 58%|█████▊    | 149/256 [01:30<00:22,  4.68it/s]
 59%|█████▊    | 150/256 [01:30<00:23,  4.53it/s]
 59%|█████▉    | 152/256 [01:30<00:19,  5.28it/s]
 60%|██████    | 154/256 [01:30<00:17,  5.90it/s]
 61%|██████    | 156/256 [01:31<00:15,  6.43it/s]
 62%|██████▏   | 158/256 [01:31<00:14,  6.80it/s]
 62%|██████▎   | 160/256 [01:31<00:13,  7.12it/s]
 63%|██████▎   | 162/256 [01:31<00:12,  7.35it/s]
 64%|██████▍   | 164/256 [01:32<00:12,  7.51it/s]
 65%|██████▍   | 166/256 [01:32<00:11,  7.63it/s]
 65%|██████▌   | 167/256 [01:32<00:13,  6.55it/s]
 66%|██████▌   | 169/256 [01:32<00:12,  6.92it/s]
 67%|██████▋   | 171/256 [01:33<00:11,  7.22it/s]
 68%|██████▊   | 173/256 [01:33<00:11,  7.43it/s]
 68%|██████▊   | 175/256 [01:33<00:10,  7.59it/s]
 69%|██████▉   | 177/256 [01:33<00:10,  7.70it/s]
 70%|██████▉   | 179/256 [01:34<00:09,  7.77it/s]
 71%|███████   | 181/256 [01:34<00:09,  7.77it/s]
 71%|███████▏  | 183/256 [01:34<00:09,  7.88it/s]
 72%|███████▏  | 184/256 [01:34<00:10,  6.70it/s]
 73%|███████▎  | 186/256 [01:35<00:09,  7.05it/s]
 73%|███████▎  | 188/256 [01:35<00:09,  7.31it/s]
 74%|███████▍  | 190/256 [01:35<00:08,  7.52it/s]
 75%|███████▌  | 192/256 [01:35<00:08,  7.63it/s]
 75%|███████▌  | 193/256 [01:57<04:13,  4.03s/it]
 76%|███████▌  | 195/256 [01:57<02:43,  2.68s/it]
 77%|███████▋  | 197/256 [01:57<01:48,  1.83s/it]
 78%|███████▊  | 200/256 [01:58<01:01,  1.10s/it]
 80%|███████▉  | 204/256 [01:58<00:33,  1.57it/s]
 81%|████████▏ | 208/256 [01:58<00:19,  2.44it/s]
 83%|████████▎ | 212/256 [01:58<00:12,  3.55it/s]
 84%|████████▍ | 215/256 [01:58<00:09,  4.52it/s]
 86%|████████▌ | 219/256 [01:58<00:05,  6.18it/s]
 87%|████████▋ | 223/256 [01:59<00:04,  8.02it/s]
 89%|████████▊ | 227/256 [01:59<00:02,  9.94it/s]
 90%|█████████ | 231/256 [01:59<00:02, 11.89it/s]
 91%|█████████▏| 234/256 [01:59<00:01, 12.66it/s]
 93%|█████████▎| 238/256 [01:59<00:01, 14.52it/s]
 95%|█████████▍| 242/256 [02:00<00:00, 16.12it/s]
 96%|█████████▌| 246/256 [02:00<00:00, 17.41it/s]
 97%|█████████▋| 249/256 [02:00<00:00, 16.98it/s]
 99%|█████████▉| 253/256 [02:00<00:00, 18.07it/s]
100%|██████████| 256/256 [02:00<00:00,  2.12it/s]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  120.72    
Total input tokens:                      269567    
Total generated tokens:                  65536     
Request throughput (req/s):              2.12      
Output token throughput (tok/s):         542.86    
Peak output token throughput (tok/s):    706.00    
Peak concurrent requests:                75.00     
Total token throughput (tok/s):          2775.81   
---------------Time to First Token----------------
Mean TTFT (ms):                          1720.17   
Median TTFT (ms):                        1005.79   
P50 TTFT (ms):                           1005.79   
P95 TTFT (ms):                           6756.81   
P99 TTFT (ms):                           8011.80   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          110.13    
Median TPOT (ms):                        111.87    
P50 TPOT (ms):                           111.87    
P95 TPOT (ms):                           113.08    
P99 TPOT (ms):                           114.50    
---------------Inter-token Latency----------------
Mean ITL (ms):                           109.70    
Median ITL (ms):                         93.97     
P50 ITL (ms):                            93.97     
P95 ITL (ms):                            252.16    
P99 ITL (ms):                            254.52    
==================================================
