Generated from pytest-benchmark JSON output
main@eae64e4f
Write operation benchmarks across all providers.; Read operation benchmarks across all providers.; Benchmarks through the full Index API.
Parameters: n = entries to store, any_provider = storage backend
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_store_single_entry[sqlite]
Store single entry [sqlite storage backend] |
133.62 μs | 237.14 μs | 4.71 Kops/s |
|
test_store_single_entry[redis]
Store single entry [redis storage backend] |
161.49 μs | 11.61 μs | 6.02 Kops/s |
|
test_store_entries_bulk[sqlite-10]
Store entries bulk [sqlite storage backend, 10 entries to store] |
1.29 ms | 223.53 μs | 613.1 ops/s |
|
test_store_entries_bulk[sqlite-100]
Store entries bulk [sqlite storage backend, 100 entries to store] |
13.65 ms | 1.45 ms | 65.4 ops/s |
|
test_store_entries_bulk[sqlite-1k]
Store entries bulk [sqlite storage backend, 1000 entries to store] |
149.96 ms | 16.48 ms | 6.4 ops/s |
|
test_store_entries_bulk[redis-10]
Store entries bulk [redis storage backend, 10 entries to store] |
1.67 ms | 221.70 μs | 579.5 ops/s |
|
test_store_entries_bulk[redis-100]
Store entries bulk [redis storage backend, 100 entries to store] |
16.25 ms | 711.29 μs | 60.4 ops/s |
|
test_store_entries_bulk[redis-1k]
Store entries bulk [redis storage backend, 1000 entries to store] |
163.09 ms | 3.23 ms | 6.1 ops/s |
|
test_store_schema[sqlite]
Store schema [sqlite storage backend] |
115.27 μs | 45.83 μs | 5.83 Kops/s |
|
test_store_schema[redis]
Store schema [redis storage backend] |
150.85 μs | 17.35 μs | 6.37 Kops/s |
|
test_store_schema_versions[sqlite-10v]
Store schema versions [sqlite storage backend, 10 entries to store] |
1.16 ms | 367.57 μs | 597.0 ops/s |
|
test_store_schema_versions[sqlite-50v]
Store schema versions [sqlite storage backend, 50 entries to store] |
6.43 ms | 1.63 ms | 125.5 ops/s |
|
test_store_schema_versions[redis-10v]
Store schema versions [redis storage backend, 10 entries to store] |
1.53 ms | 72.01 μs | 648.0 ops/s |
|
test_store_schema_versions[redis-50v]
Store schema versions [redis storage backend, 50 entries to store] |
7.77 ms | 910.46 μs | 125.7 ops/s |
|
test_get_entry_by_name[sqlite]
Get entry by name [sqlite storage backend] |
8.62 μs | 120.0 ns | 112.52 Kops/s |
|
test_get_entry_by_name[redis]
Get entry by name [redis storage backend] |
49.31 ms | 904.43 μs | 20.1 ops/s |
|
test_get_entry_by_cid[sqlite]
Get entry by cid [sqlite storage backend] |
8.82 μs | 121.0 ns | 111.07 Kops/s |
|
test_get_entry_by_cid[redis]
Get entry by cid [redis storage backend] |
167.54 μs | 12.72 μs | 5.81 Kops/s |
|
test_iter_entries[sqlite-10]
Iter entries [sqlite storage backend, 10 entries to store] |
37.08 μs | 350.0 ns | 26.49 Kops/s |
|
test_iter_entries[sqlite-100]
Iter entries [sqlite storage backend, 100 entries to store] |
330.20 μs | 11.50 μs | 3.00 Kops/s |
|
test_iter_entries[sqlite-1k]
Iter entries [sqlite storage backend, 1000 entries to store] |
3.38 ms | 32.39 μs | 267.5 ops/s |
|
test_iter_entries[redis-10]
Iter entries [redis storage backend, 10 entries to store] |
191.95 ms | 14.34 ms | 5.2 ops/s |
|
test_iter_entries[redis-100]
Iter entries [redis storage backend, 100 entries to store] |
200.08 ms | 20.65 ms | 5.0 ops/s |
|
test_iter_entries[redis-1k]
Iter entries [redis storage backend, 1000 entries to store] |
192.94 ms | 16.13 ms | 5.0 ops/s |
|
test_get_schema_json[sqlite]
Get schema json [sqlite storage backend] |
4.55 μs | 50.0 ns | 215.87 Kops/s |
|
test_get_schema_json[redis]
Get schema json [redis storage backend] |
149.23 μs | 14.33 μs | 6.49 Kops/s |
|
test_find_latest_version[sqlite-5v]
Find latest version [sqlite storage backend, 5 entries to store] |
9.93 μs | 130.0 ns | 96.41 Kops/s |
|
test_find_latest_version[sqlite-20v]
Find latest version [sqlite storage backend, 20 entries to store] |
24.28 μs | 260.0 ns | 40.61 Kops/s |
|
test_find_latest_version[sqlite-50v]
Find latest version [sqlite storage backend, 50 entries to store] |
52.28 μs | 411.0 ns | 18.88 Kops/s |
|
test_find_latest_version[redis-5v]
Find latest version [redis storage backend, 5 entries to store] |
16.21 ms | 749.15 μs | 60.4 ops/s |
|
test_find_latest_version[redis-20v]
Find latest version [redis storage backend, 20 entries to store] |
19.23 ms | 2.50 ms | 50.2 ops/s |
|
test_find_latest_version[redis-50v]
Find latest version [redis storage backend, 50 entries to store] |
24.21 ms | 1.13 ms | 40.6 ops/s |
|
test_iter_schemas[sqlite]
Iter schemas [sqlite storage backend] |
16.54 μs | 100.0 ns | 58.03 Kops/s |
|
test_iter_schemas[redis]
Iter schemas [redis storage backend] |
19.25 ms | 2.46 ms | 49.8 ops/s |
|
test_index_insert_dataset
Index insert dataset |
302.97 μs | 198.02 μs | 2.32 Kops/s |
|
test_index_get_dataset
Index get dataset |
17.17 μs | 260.0 ns | 56.85 Kops/s |
|
test_index_list_datasets
Index list datasets |
8.21 μs | 120.0 ns | 119.10 Kops/s |
|
test_index_publish_schema
Index publish schema |
237.35 μs | 88.62 μs | 3.37 Kops/s |
Shard writing throughput benchmarks.; Shard reading and iteration benchmarks.; Full write-then-read round-trip benchmarks.
Parameters: n = samples per shard, batch_size = samples per batch
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_write_basic_shard[100]
Write basic shard [100 samples per shard] |
7.87 ms | 111.98 μs | 126.2 ops/s | 78.69 μs | 12.71 Kops/s |
|
test_write_basic_shard[1k]
Write basic shard [1000 samples per shard] |
74.78 ms | 381.74 μs | 13.4 ops/s | 74.78 μs | 13.37 Kops/s |
|
test_write_basic_shard[10k]
Write basic shard [10000 samples per shard] |
737.51 ms | 13.86 ms | 1.3 ops/s | 73.75 μs | 13.56 Kops/s |
|
test_write_numpy_shard[100]
Write numpy shard [100 samples per shard] |
31.74 ms | 714.13 μs | 31.5 ops/s | 317.40 μs | 3.15 Kops/s |
|
test_write_numpy_shard[1k]
Write numpy shard [1000 samples per shard] |
349.73 ms | 36.86 ms | 2.8 ops/s | 349.73 μs | 2.86 Kops/s |
|
test_write_large_numpy_shard
Write large numpy shard |
8.454 s | 247.47 ms | 0.1 ops/s | 845.36 ms | 1.2 ops/s |
|
test_write_with_manifest[100]
Write with manifest [100 samples per shard] |
13.67 ms | 184.39 μs | 72.9 ops/s | 136.66 μs | 7.32 Kops/s |
|
test_write_with_manifest[1k]
Write with manifest [1000 samples per shard] |
95.97 ms | 587.36 μs | 10.4 ops/s | 95.97 μs | 10.42 Kops/s |
|
test_write_multi_shard
Write multi shard |
740.42 ms | 7.74 ms | 1.3 ops/s | 74.04 μs | 13.51 Kops/s |
|
test_read_ordered[100]
Read ordered [100 samples per shard] |
7.39 ms | 77.20 μs | 135.3 ops/s | 73.93 μs | 13.53 Kops/s |
|
test_read_ordered[1k]
Read ordered [1000 samples per shard] |
70.38 ms | 546.35 μs | 14.2 ops/s | 70.38 μs | 14.21 Kops/s |
|
test_read_ordered[10k]
Read ordered [10000 samples per shard] |
685.73 ms | 5.38 ms | 1.5 ops/s | 68.57 μs | 14.58 Kops/s |
|
test_read_shuffled[100]
Read shuffled [100 samples per shard] |
7.09 ms | 82.82 μs | 139.7 ops/s | 70.94 μs | 14.10 Kops/s |
|
test_read_shuffled[1k]
Read shuffled [1000 samples per shard] |
69.76 ms | 964.39 μs | 14.3 ops/s | 69.76 μs | 14.33 Kops/s |
|
test_read_batched[batch32]
Read batched [32 samples per batch] |
67.11 ms | 466.71 μs | 14.9 ops/s | 67.11 μs | 14.90 Kops/s |
|
test_read_batched[batch128]
Read batched [128 samples per batch] |
66.72 ms | 411.85 μs | 15.0 ops/s | 66.72 μs | 14.99 Kops/s |
|
test_read_numpy_ordered[100]
Read numpy ordered [100 samples per shard] |
13.76 ms | 250.26 μs | 72.6 ops/s | 137.57 μs | 7.27 Kops/s |
|
test_read_numpy_ordered[1k]
Read numpy ordered [1000 samples per shard] |
135.78 ms | 1.54 ms | 7.4 ops/s | 135.78 μs | 7.37 Kops/s |
|
test_roundtrip_basic[100]
Roundtrip basic [100 samples per shard] |
14.99 ms | 180.50 μs | 66.7 ops/s | 149.90 μs | 6.67 Kops/s |
|
test_roundtrip_basic[1k]
Roundtrip basic [1000 samples per shard] |
145.28 ms | 7.72 ms | 6.8 ops/s | 145.28 μs | 6.88 Kops/s |
|
test_roundtrip_numpy[100]
Roundtrip numpy [100 samples per shard] |
39.47 ms | 367.51 μs | 25.2 ops/s | 394.74 μs | 2.53 Kops/s |
|
test_roundtrip_numpy[500]
Roundtrip numpy [500 samples per shard] |
194.09 ms | 1.14 ms | 5.1 ops/s | 388.17 μs | 2.58 Kops/s |
Benchmark different query predicate types on a medium dataset.; Benchmark iterating through query results to access sample locations.; Benchmark query performance at different scales.; Benchmark manifest loading from disk.; Benchmark manifest construction from samples.
Parameters: n_shards = number of shards (100 samples each), n = samples in manifest
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_query_simple_equality
Query simple equality |
8.41 ms | 132.92 μs | 118.2 ops/s | — | — |
|
test_query_numeric_range
Query numeric range |
8.87 ms | 101.38 μs | 112.4 ops/s | — | — |
|
test_query_combined
Query combined |
4.34 ms | 72.27 μs | 228.9 ops/s | — | — |
|
test_query_isin
Query isin |
14.59 ms | 311.79 μs | 68.3 ops/s | — | — |
|
test_query_no_results
Query no results |
1.87 ms | 30.19 μs | 521.7 ops/s | — | — |
|
test_query_all_results
Query all results |
30.76 ms | 600.36 μs | 32.4 ops/s | — | — |
|
test_iterate_equality_results
Iterate equality results |
8.42 ms | 93.11 μs | 116.7 ops/s | 42.11 μs | 23.75 Kops/s |
|
test_iterate_range_results
Iterate range results |
18.86 ms | 347.30 μs | 52.3 ops/s | 34.30 μs | 29.16 Kops/s |
|
test_iterate_large_result_set
Iterate large result set |
300.21 ms | 23.02 ms | 3.3 ops/s | 30.02 μs | 33.31 Kops/s |
|
test_query_small
Query small |
2.04 ms | 35.46 μs | 488.9 ops/s | — | — |
|
test_query_medium
Query medium |
18.26 ms | 162.07 μs | 54.7 ops/s | — | — |
|
test_query_large
Query large |
162.48 ms | 2.49 ms | 6.1 ops/s | — | — |
|
test_load_from_directory[2s]
Load from directory [2 number of shards (100 samples each)] |
3.28 ms | 27.19 μs | 303.4 ops/s | — | — |
|
test_load_from_directory[5s]
Load from directory [5 number of shards (100 samples each)] |
8.19 ms | 59.58 μs | 121.8 ops/s | — | — |
|
test_load_from_directory[10s]
Load from directory [10 number of shards (100 samples each)] |
16.38 ms | 101.06 μs | 61.0 ops/s | — | — |
|
test_load_from_directory[20s]
Load from directory [20 number of shards (100 samples each)] |
32.79 ms | 272.42 μs | 29.2 ops/s | — | — |
|
test_load_from_shard_urls[2s]
Load from shard urls [2 number of shards (100 samples each)] |
3.24 ms | 28.34 μs | 307.6 ops/s | — | — |
|
test_load_from_shard_urls[5s]
Load from shard urls [5 number of shards (100 samples each)] |
8.14 ms | 55.01 μs | 122.8 ops/s | — | — |
|
test_load_from_shard_urls[10s]
Load from shard urls [10 number of shards (100 samples each)] |
16.30 ms | 122.89 μs | 61.2 ops/s | — | — |
|
test_manifest_build[100]
Manifest build [100 samples in manifest] |
1.29 ms | 22.68 μs | 766.6 ops/s | 12.85 μs | 77.79 Kops/s |
|
test_manifest_build[1k]
Manifest build [1000 samples in manifest] |
7.68 ms | 115.10 μs | 113.8 ops/s | 7.68 μs | 130.25 Kops/s |
|
test_manifest_build[5k]
Manifest build [5000 samples in manifest] |
36.90 ms | 973.86 μs | 22.1 ops/s | 7.38 μs | 135.50 Kops/s |
|
test_manifest_write[100]
Manifest write [100 samples in manifest] |
3.62 ms | 68.19 μs | 274.4 ops/s | 36.22 μs | 27.61 Kops/s |
|
test_manifest_write[1k]
Manifest write [1000 samples in manifest] |
6.98 ms | 133.87 μs | 140.4 ops/s | 6.98 μs | 143.25 Kops/s |
S3 shard writing benchmarks via moto mock.
Parameters: n = samples per shard
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_s3_write_shards[100]
S3 write shards [100 samples per shard] |
19.73 ms | 614.78 μs | 49.4 ops/s | 197.32 μs | 5.07 Kops/s |
|
test_s3_write_shards[500]
S3 write shards [500 samples per shard] |
86.97 ms | 1.08 ms | 10.5 ops/s | 173.94 μs | 5.75 Kops/s |
|
test_s3_write_with_manifest
S3 write with manifest |
59.17 ms | 2.10 ms | 14.9 ops/s | 295.86 μs | 3.38 Kops/s |
|
test_s3_write_cache_local
S3 write cache local |
42.60 ms | 934.20 μs | 20.8 ops/s | 213.01 μs | 4.69 Kops/s |
|
test_s3_write_direct
S3 write direct |
36.27 ms | 213.47 μs | 27.5 ops/s | 181.36 μs | 5.51 Kops/s |
|
test_s3_write_numpy
S3 write numpy |
42.72 ms | 1.55 ms | 23.0 ops/s | 427.25 μs | 2.34 Kops/s |
Pure serialization/deserialization without disk I/O.
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_serialize_basic_sample
Serialize basic sample |
1.35 μs | 39.0 ns | 731.63 Kops/s |
|
test_deserialize_basic_sample
Deserialize basic sample |
1.82 μs | 60.0 ns | 536.76 Kops/s |
|
test_serialize_numpy_sample
Serialize numpy sample |
19.83 μs | 530.0 ns | 49.22 Kops/s |
|
test_deserialize_numpy_sample
Deserialize numpy sample |
5.77 μs | 101.0 ns | 169.16 Kops/s |
|
test_serialize_large_numpy
Serialize large numpy |
426.76 ms | 1.48 ms | 2.3 ops/s |
|
test_deserialize_large_numpy
Deserialize large numpy |
33.40 ms | 169.83 μs | 29.9 ops/s |
|
test_as_wds_basic
As wds basic |
4.56 μs | 70.0 ns | 215.81 Kops/s |
|
test_as_wds_numpy
As wds numpy |
24.40 μs | 761.0 ns | 39.26 Kops/s |