INFO 09-09 06:38:42 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 09-09 06:38:42 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 09-09 06:38:42 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 09-09 06:38:42 [__init__.py:238] Platform plugin ascend is activated
INFO 09-09 06:38:42 [platform.py:62] Breakable cudagraph is force disabled on Ascend because DeepSeek V4 PIECEWISE cudagraph is not supported yet.
2026-09-09 06:38:53,424 - 2154086 - ServiceProfiler - INFO - VLLM_USE_V1 not set, auto-detected via vLLM 0.23.0+empty: default 1
INFO 09-09 06:38:54 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 09-09 06:38:54 [__init__.py:115] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0xfffb73dceca0>, trust_remote_code=False, seed=0, num_prompts=256, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='openai-chat', base_url=None, host='127.0.0.1', port=8331, endpoint='/v1/chat/completions', header=None, max_concurrency=64, model='qwen14b', input_len=None, output_len=None, tokenizer='/data/shared_models/Qwen--Qwen2.5-Coder-14B-Instruct', tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, disable_tqdm=False, num_warmups=8, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/vllm-workspace/vllm-ascend-split-batch-hust/docs/evidence/w2b-fi-sampling/results', result_filename='serve_api1_on_r2.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot,itl', metric_percentiles='50,95,99', goodput=None, request_id_prefix='bench-6c4d7ae4-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 09-09 06:38:55 [utils.py:90] Sampling input_len from [1024, 1024] and output_len from [256, 256]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Warming up with 8 requests...

  0%|          | 0/8 [00:00<?, ?it/s]
 12%|█▎        | 1/8 [00:24<02:52, 24.58s/it]
100%|██████████| 8/8 [00:24<00:00,  3.07s/it]
Warmup run completed.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 64

  0%|          | 0/256 [00:00<?, ?it/s]
  0%|          | 1/256 [00:29<2:05:48, 29.60s/it]
  1%|          | 2/256 [00:29<51:51, 12.25s/it]  
  2%|▏         | 4/256 [00:29<19:19,  4.60s/it]
  2%|▏         | 6/256 [00:29<10:12,  2.45s/it]
  3%|▎         | 8/256 [00:30<06:13,  1.51s/it]
  4%|▍         | 10/256 [00:30<04:06,  1.00s/it]
  5%|▍         | 12/256 [00:30<02:51,  1.42it/s]
  5%|▌         | 14/256 [00:30<02:04,  1.95it/s]
  6%|▋         | 16/256 [00:31<01:33,  2.57it/s]
  7%|▋         | 18/256 [00:31<01:13,  3.26it/s]
  8%|▊         | 20/256 [00:31<00:59,  3.98it/s]
  9%|▊         | 22/256 [00:31<00:49,  4.70it/s]
  9%|▉         | 24/256 [00:32<00:43,  5.36it/s]
 10%|█         | 26/256 [00:32<00:38,  5.96it/s]
 11%|█         | 28/256 [00:32<00:35,  6.42it/s]
 12%|█▏        | 30/256 [00:32<00:33,  6.85it/s]
 12%|█▏        | 31/256 [00:33<00:37,  6.06it/s]
 13%|█▎        | 33/256 [00:33<00:33,  6.59it/s]
 14%|█▎        | 35/256 [00:33<00:31,  6.97it/s]
 14%|█▍        | 37/256 [00:33<00:30,  7.25it/s]
 15%|█▌        | 39/256 [00:34<00:29,  7.41it/s]
 16%|█▌        | 41/256 [00:34<00:28,  7.61it/s]
 17%|█▋        | 43/256 [00:34<00:27,  7.73it/s]
 18%|█▊        | 45/256 [00:34<00:27,  7.78it/s]
 18%|█▊        | 47/256 [00:35<00:26,  7.83it/s]
 19%|█▉        | 49/256 [00:35<00:26,  7.88it/s]
 20%|█▉        | 51/256 [00:35<00:25,  7.90it/s]
 20%|██        | 52/256 [00:35<00:30,  6.73it/s]
 21%|██        | 54/256 [00:36<00:28,  7.10it/s]
 22%|██▏       | 56/256 [00:36<00:27,  7.36it/s]
 23%|██▎       | 58/256 [00:36<00:26,  7.55it/s]
 23%|██▎       | 60/256 [00:36<00:25,  7.69it/s]
 24%|██▍       | 62/256 [00:37<00:24,  7.78it/s]
 25%|██▌       | 64/256 [00:37<00:24,  7.82it/s]
 25%|██▌       | 65/256 [00:59<12:46,  4.01s/it]
 26%|██▌       | 67/256 [00:59<08:28,  2.69s/it]
 27%|██▋       | 70/256 [00:59<04:52,  1.57s/it]
 28%|██▊       | 72/256 [00:59<03:30,  1.14s/it]
 29%|██▉       | 74/256 [00:59<02:34,  1.18it/s]
 30%|██▉       | 76/256 [01:00<01:54,  1.57it/s]
 30%|███       | 78/256 [01:00<01:26,  2.06it/s]
 31%|███▏      | 80/256 [01:00<01:06,  2.64it/s]
 32%|███▏      | 81/256 [01:00<01:02,  2.80it/s]
 32%|███▏      | 83/256 [01:01<00:48,  3.57it/s]
 33%|███▎      | 84/256 [01:01<00:47,  3.63it/s]
 34%|███▎      | 86/256 [01:01<00:37,  4.50it/s]
 34%|███▍      | 88/256 [01:01<00:31,  5.26it/s]
 35%|███▌      | 90/256 [01:02<00:28,  5.92it/s]
 36%|███▌      | 92/256 [01:02<00:25,  6.41it/s]
 37%|███▋      | 94/256 [01:02<00:23,  6.85it/s]
 38%|███▊      | 96/256 [01:02<00:22,  7.15it/s]
 38%|███▊      | 98/256 [01:03<00:21,  7.35it/s]
 39%|███▉      | 100/256 [01:03<00:20,  7.54it/s]
 39%|███▉      | 101/256 [01:03<00:23,  6.50it/s]
 40%|████      | 103/256 [01:03<00:22,  6.90it/s]
 41%|████      | 105/256 [01:04<00:20,  7.21it/s]
 42%|████▏     | 107/256 [01:04<00:19,  7.45it/s]
 43%|████▎     | 109/256 [01:04<00:19,  7.58it/s]
 43%|████▎     | 111/256 [01:04<00:18,  7.66it/s]
 44%|████▍     | 113/256 [01:05<00:18,  7.78it/s]
 45%|████▍     | 115/256 [01:05<00:18,  7.82it/s]
 46%|████▌     | 117/256 [01:05<00:17,  7.87it/s]
 46%|████▌     | 118/256 [01:05<00:20,  6.70it/s]
 47%|████▋     | 120/256 [01:06<00:19,  7.07it/s]
 48%|████▊     | 122/256 [01:06<00:18,  7.33it/s]
 48%|████▊     | 124/256 [01:06<00:17,  7.53it/s]
 49%|████▉     | 126/256 [01:06<00:16,  7.66it/s]
 50%|█████     | 128/256 [01:07<00:16,  7.78it/s]
 50%|█████     | 129/256 [01:29<08:36,  4.06s/it]
 51%|█████     | 131/256 [01:29<05:39,  2.72s/it]
 52%|█████▏    | 132/256 [01:29<04:33,  2.20s/it]
 52%|█████▏    | 134/256 [01:29<02:55,  1.44s/it]
 53%|█████▎    | 136/256 [01:29<01:57,  1.02it/s]
 54%|█████▍    | 138/256 [01:30<01:22,  1.43it/s]
 55%|█████▍    | 140/256 [01:30<01:00,  1.93it/s]
 55%|█████▌    | 142/256 [01:30<00:45,  2.53it/s]
 56%|█████▋    | 144/256 [01:30<00:35,  3.20it/s]
 57%|█████▋    | 146/256 [01:31<00:28,  3.90it/s]
 58%|█████▊    | 148/256 [01:31<00:23,  4.62it/s]
 58%|█████▊    | 149/256 [01:31<00:23,  4.49it/s]
 59%|█████▉    | 151/256 [01:31<00:20,  5.25it/s]
 60%|█████▉    | 153/256 [01:32<00:17,  5.89it/s]
 61%|██████    | 155/256 [01:32<00:15,  6.42it/s]
 61%|██████▏   | 157/256 [01:32<00:14,  6.81it/s]
 62%|██████▏   | 159/256 [01:32<00:13,  7.14it/s]
 63%|██████▎   | 161/256 [01:33<00:12,  7.37it/s]
 64%|██████▎   | 163/256 [01:33<00:12,  7.52it/s]
 64%|██████▍   | 165/256 [01:33<00:11,  7.66it/s]
 65%|██████▍   | 166/256 [01:33<00:13,  6.56it/s]
 66%|██████▌   | 168/256 [01:34<00:12,  6.95it/s]
 66%|██████▋   | 170/256 [01:34<00:11,  7.24it/s]
 67%|██████▋   | 172/256 [01:34<00:11,  7.47it/s]
 68%|██████▊   | 174/256 [01:34<00:10,  7.59it/s]
 69%|██████▉   | 176/256 [01:35<00:10,  7.70it/s]
 70%|██████▉   | 178/256 [01:35<00:10,  7.77it/s]
 70%|███████   | 180/256 [01:35<00:09,  7.77it/s]
 71%|███████   | 182/256 [01:35<00:09,  7.90it/s]
 71%|███████▏  | 183/256 [01:36<00:10,  6.71it/s]
 72%|███████▏  | 185/256 [01:36<00:10,  7.07it/s]
 73%|███████▎  | 187/256 [01:36<00:09,  7.33it/s]
 74%|███████▍  | 189/256 [01:36<00:08,  7.53it/s]
 75%|███████▍  | 191/256 [01:37<00:08,  7.64it/s]
 75%|███████▌  | 193/256 [01:58<03:38,  3.46s/it]
 76%|███████▌  | 195/256 [01:59<02:28,  2.43s/it]
 77%|███████▋  | 197/256 [01:59<01:41,  1.72s/it]
 79%|███████▊  | 201/256 [01:59<00:51,  1.06it/s]
 80%|████████  | 205/256 [01:59<00:29,  1.70it/s]
 82%|████████▏ | 209/256 [01:59<00:18,  2.55it/s]
 83%|████████▎ | 213/256 [02:00<00:11,  3.65it/s]
 84%|████████▍ | 216/256 [02:00<00:08,  4.59it/s]
 86%|████████▌ | 220/256 [02:00<00:05,  6.21it/s]
 88%|████████▊ | 224/256 [02:00<00:03,  8.03it/s]
 89%|████████▉ | 228/256 [02:00<00:02,  9.95it/s]
 90%|█████████ | 231/256 [02:01<00:02, 10.95it/s]
 92%|█████████▏| 235/256 [02:01<00:01, 12.94it/s]
 93%|█████████▎| 239/256 [02:01<00:01, 14.71it/s]
 95%|█████████▍| 243/256 [02:01<00:00, 16.20it/s]
 96%|█████████▋| 247/256 [02:01<00:00, 17.41it/s]
 98%|█████████▊| 250/256 [02:02<00:00, 16.97it/s]
 99%|█████████▉| 254/256 [02:02<00:00, 17.83it/s]
100%|██████████| 256/256 [02:02<00:00,  2.09it/s]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  122.27    
Total input tokens:                      269567    
Total generated tokens:                  65536     
Request throughput (req/s):              2.09      
Output token throughput (tok/s):         536.01    
Peak output token throughput (tok/s):    704.00    
Peak concurrent requests:                74.00     
Total token throughput (tok/s):          2740.74   
---------------Time to First Token----------------
Mean TTFT (ms):                          1702.44   
Median TTFT (ms):                        1003.43   
P50 TTFT (ms):                           1003.43   
P95 TTFT (ms):                           6607.33   
P99 TTFT (ms):                           7908.95   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          111.66    
Median TPOT (ms):                        113.54    
P50 TPOT (ms):                           113.54    
P95 TPOT (ms):                           114.47    
P99 TPOT (ms):                           115.61    
---------------Inter-token Latency----------------
Mean ITL (ms):                           111.22    
Median ITL (ms):                         95.87     
P50 ITL (ms):                            95.87     
P95 ITL (ms):                            251.60    
P99 ITL (ms):                            254.24    
==================================================
