Generated from pytest-benchmark JSON output
main@dbe32010
Write operation benchmarks across all providers.; Read operation benchmarks across all providers.; Benchmarks through the full Index API.
Parameters: n = entries to store, any_provider = storage backend
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_store_single_entry[sqlite]
Store single entry [sqlite storage backend] |
128.98 μs | 265.18 μs | 4.42 Kops/s |
|
test_store_single_entry[redis]
Store single entry [redis storage backend] |
169.50 μs | 25.34 μs | 5.70 Kops/s |
|
test_store_entries_bulk[sqlite-10]
Store entries bulk [sqlite storage backend, 10 entries to store] |
1.38 ms | 156.12 μs | 586.4 ops/s |
|
test_store_entries_bulk[sqlite-100]
Store entries bulk [sqlite storage backend, 100 entries to store] |
15.33 ms | 1.21 ms | 59.6 ops/s |
|
test_store_entries_bulk[sqlite-1k]
Store entries bulk [sqlite storage backend, 1000 entries to store] |
150.08 ms | 2.36 ms | 6.7 ops/s |
|
test_store_entries_bulk[redis-10]
Store entries bulk [redis storage backend, 10 entries to store] |
1.71 ms | 41.25 μs | 581.5 ops/s |
|
test_store_entries_bulk[redis-100]
Store entries bulk [redis storage backend, 100 entries to store] |
17.19 ms | 2.52 ms | 56.2 ops/s |
|
test_store_entries_bulk[redis-1k]
Store entries bulk [redis storage backend, 1000 entries to store] |
177.32 ms | 8.51 ms | 5.6 ops/s |
|
test_store_schema[sqlite]
Store schema [sqlite storage backend] |
113.71 μs | 19.18 μs | 5.98 Kops/s |
|
test_store_schema[redis]
Store schema [redis storage backend] |
158.81 μs | 19.61 μs | 6.09 Kops/s |
|
test_store_schema_versions[sqlite-10v]
Store schema versions [sqlite storage backend, 10 entries to store] |
1.26 ms | 202.08 μs | 574.5 ops/s |
|
test_store_schema_versions[sqlite-50v]
Store schema versions [sqlite storage backend, 50 entries to store] |
6.88 ms | 388.37 μs | 118.9 ops/s |
|
test_store_schema_versions[redis-10v]
Store schema versions [redis storage backend, 10 entries to store] |
1.63 ms | 222.18 μs | 589.6 ops/s |
|
test_store_schema_versions[redis-50v]
Store schema versions [redis storage backend, 50 entries to store] |
8.13 ms | 204.27 μs | 122.0 ops/s |
|
test_get_entry_by_name[sqlite]
Get entry by name [sqlite storage backend] |
8.91 μs | 131.0 ns | 108.19 Kops/s |
|
test_get_entry_by_name[redis]
Get entry by name [redis storage backend] |
78.15 ms | 3.63 ms | 12.8 ops/s |
|
test_get_entry_by_cid[sqlite]
Get entry by cid [sqlite storage backend] |
8.93 μs | 121.0 ns | 106.86 Kops/s |
|
test_get_entry_by_cid[redis]
Get entry by cid [redis storage backend] |
192.40 μs | 25.00 μs | 5.28 Kops/s |
|
test_iter_entries[sqlite-10]
Iter entries [sqlite storage backend, 10 entries to store] |
37.30 μs | 330.0 ns | 25.67 Kops/s |
|
test_iter_entries[sqlite-100]
Iter entries [sqlite storage backend, 100 entries to store] |
330.17 μs | 12.31 μs | 2.99 Kops/s |
|
test_iter_entries[sqlite-1k]
Iter entries [sqlite storage backend, 1000 entries to store] |
3.39 ms | 75.27 μs | 241.2 ops/s |
|
test_iter_entries[redis-10]
Iter entries [redis storage backend, 10 entries to store] |
202.43 ms | 8.94 ms | 4.9 ops/s |
|
test_iter_entries[redis-100]
Iter entries [redis storage backend, 100 entries to store] |
207.20 ms | 8.86 ms | 4.8 ops/s |
|
test_iter_entries[redis-1k]
Iter entries [redis storage backend, 1000 entries to store] |
199.77 ms | 12.44 ms | 5.0 ops/s |
|
test_get_schema_json[sqlite]
Get schema json [sqlite storage backend] |
5.26 μs | 51.0 ns | 186.19 Kops/s |
|
test_get_schema_json[redis]
Get schema json [redis storage backend] |
160.45 μs | 20.17 μs | 5.99 Kops/s |
|
test_find_latest_version[sqlite-5v]
Find latest version [sqlite storage backend, 5 entries to store] |
10.55 μs | 130.0 ns | 92.63 Kops/s |
|
test_find_latest_version[sqlite-20v]
Find latest version [sqlite storage backend, 20 entries to store] |
24.99 μs | 310.0 ns | 39.43 Kops/s |
|
test_find_latest_version[sqlite-50v]
Find latest version [sqlite storage backend, 50 entries to store] |
53.65 μs | 400.0 ns | 18.37 Kops/s |
|
test_find_latest_version[redis-5v]
Find latest version [redis storage backend, 5 entries to store] |
17.22 ms | 1.14 ms | 57.5 ops/s |
|
test_find_latest_version[redis-20v]
Find latest version [redis storage backend, 20 entries to store] |
19.96 ms | 679.23 μs | 49.4 ops/s |
|
test_find_latest_version[redis-50v]
Find latest version [redis storage backend, 50 entries to store] |
24.40 ms | 1.31 ms | 40.2 ops/s |
|
test_iter_schemas[sqlite]
Iter schemas [sqlite storage backend] |
17.17 μs | 121.0 ns | 57.27 Kops/s |
|
test_iter_schemas[redis]
Iter schemas [redis storage backend] |
19.72 ms | 692.53 μs | 50.1 ops/s |
|
test_index_insert_dataset
Index insert dataset |
314.03 μs | 184.02 μs | 2.26 Kops/s |
|
test_index_get_dataset
Index get dataset |
18.09 μs | 220.0 ns | 53.91 Kops/s |
|
test_index_list_datasets
Index list datasets |
8.50 μs | 120.0 ns | 115.32 Kops/s |
|
test_index_publish_schema
Index publish schema |
238.27 μs | 85.58 μs | 3.32 Kops/s |
Shard writing throughput benchmarks.; Shard reading and iteration benchmarks.; Full write-then-read round-trip benchmarks.
Parameters: n = samples per shard, batch_size = samples per batch
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_write_basic_shard[100]
Write basic shard [100 samples per shard] |
7.93 ms | 122.36 μs | 125.6 ops/s | 79.31 μs | 12.61 Kops/s |
|
test_write_basic_shard[1k]
Write basic shard [1000 samples per shard] |
75.12 ms | 848.44 μs | 13.3 ops/s | 75.12 μs | 13.31 Kops/s |
|
test_write_basic_shard[10k]
Write basic shard [10000 samples per shard] |
751.96 ms | 15.94 ms | 1.3 ops/s | 75.20 μs | 13.30 Kops/s |
|
test_write_numpy_shard[100]
Write numpy shard [100 samples per shard] |
32.01 ms | 932.92 μs | 31.2 ops/s | 320.05 μs | 3.12 Kops/s |
|
test_write_numpy_shard[1k]
Write numpy shard [1000 samples per shard] |
358.27 ms | 30.41 ms | 2.8 ops/s | 358.27 μs | 2.79 Kops/s |
|
test_write_large_numpy_shard
Write large numpy shard |
8.200 s | 97.30 ms | 0.1 ops/s | 820.00 ms | 1.2 ops/s |
|
test_write_with_manifest[100]
Write with manifest [100 samples per shard] |
13.75 ms | 168.69 μs | 72.5 ops/s | 137.53 μs | 7.27 Kops/s |
|
test_write_with_manifest[1k]
Write with manifest [1000 samples per shard] |
96.41 ms | 873.14 μs | 10.4 ops/s | 96.41 μs | 10.37 Kops/s |
|
test_write_multi_shard
Write multi shard |
745.09 ms | 2.55 ms | 1.3 ops/s | 74.51 μs | 13.42 Kops/s |
|
test_read_ordered[100]
Read ordered [100 samples per shard] |
7.08 ms | 70.14 μs | 140.7 ops/s | 70.83 μs | 14.12 Kops/s |
|
test_read_ordered[1k]
Read ordered [1000 samples per shard] |
67.23 ms | 575.82 μs | 14.9 ops/s | 67.23 μs | 14.87 Kops/s |
|
test_read_ordered[10k]
Read ordered [10000 samples per shard] |
666.33 ms | 6.94 ms | 1.5 ops/s | 66.63 μs | 15.01 Kops/s |
|
test_read_shuffled[100]
Read shuffled [100 samples per shard] |
6.84 ms | 113.85 μs | 144.8 ops/s | 68.39 μs | 14.62 Kops/s |
|
test_read_shuffled[1k]
Read shuffled [1000 samples per shard] |
69.15 ms | 561.31 μs | 14.4 ops/s | 69.15 μs | 14.46 Kops/s |
|
test_read_batched[batch32]
Read batched [32 samples per batch] |
65.67 ms | 698.88 μs | 15.2 ops/s | 65.67 μs | 15.23 Kops/s |
|
test_read_batched[batch128]
Read batched [128 samples per batch] |
64.09 ms | 539.19 μs | 15.6 ops/s | 64.09 μs | 15.60 Kops/s |
|
test_read_numpy_ordered[100]
Read numpy ordered [100 samples per shard] |
14.71 ms | 175.29 μs | 67.9 ops/s | 147.09 μs | 6.80 Kops/s |
|
test_read_numpy_ordered[1k]
Read numpy ordered [1000 samples per shard] |
144.15 ms | 798.86 μs | 6.9 ops/s | 144.15 μs | 6.94 Kops/s |
|
test_roundtrip_basic[100]
Roundtrip basic [100 samples per shard] |
14.98 ms | 155.08 μs | 66.7 ops/s | 149.77 μs | 6.68 Kops/s |
|
test_roundtrip_basic[1k]
Roundtrip basic [1000 samples per shard] |
142.25 ms | 651.73 μs | 7.0 ops/s | 142.25 μs | 7.03 Kops/s |
|
test_roundtrip_numpy[100]
Roundtrip numpy [100 samples per shard] |
41.22 ms | 391.46 μs | 24.2 ops/s | 412.22 μs | 2.43 Kops/s |
|
test_roundtrip_numpy[500]
Roundtrip numpy [500 samples per shard] |
203.00 ms | 807.85 μs | 4.9 ops/s | 406.01 μs | 2.46 Kops/s |
Benchmark different query predicate types on a medium dataset.; Benchmark iterating through query results to access sample locations.; Benchmark query performance at different scales.; Benchmark manifest loading from disk.; Benchmark manifest construction from samples.
Parameters: n_shards = number of shards (100 samples each), n = samples in manifest
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_query_simple_equality
Query simple equality |
8.60 ms | 131.31 μs | 113.4 ops/s | — | — |
|
test_query_numeric_range
Query numeric range |
9.14 ms | 96.16 μs | 109.2 ops/s | — | — |
|
test_query_combined
Query combined |
4.40 ms | 45.64 μs | 226.5 ops/s | — | — |
|
test_query_isin
Query isin |
14.96 ms | 150.58 μs | 66.1 ops/s | — | — |
|
test_query_no_results
Query no results |
1.87 ms | 23.35 μs | 531.2 ops/s | — | — |
|
test_query_all_results
Query all results |
32.23 ms | 362.18 μs | 31.0 ops/s | — | — |
|
test_iterate_equality_results
Iterate equality results |
8.67 ms | 143.50 μs | 115.0 ops/s | 43.36 μs | 23.06 Kops/s |
|
test_iterate_range_results
Iterate range results |
19.48 ms | 208.66 μs | 51.1 ops/s | 35.41 μs | 28.24 Kops/s |
|
test_iterate_large_result_set
Iterate large result set |
296.10 ms | 1.87 ms | 3.4 ops/s | 29.61 μs | 33.77 Kops/s |
|
test_query_small
Query small |
2.12 ms | 33.07 μs | 470.1 ops/s | — | — |
|
test_query_medium
Query medium |
19.21 ms | 300.63 μs | 51.8 ops/s | — | — |
|
test_query_large
Query large |
168.20 ms | 1.52 ms | 5.9 ops/s | — | — |
|
test_load_from_directory[2s]
Load from directory [2 number of shards (100 samples each)] |
3.33 ms | 30.40 μs | 299.4 ops/s | — | — |
|
test_load_from_directory[5s]
Load from directory [5 number of shards (100 samples each)] |
8.28 ms | 54.50 μs | 120.0 ops/s | — | — |
|
test_load_from_directory[10s]
Load from directory [10 number of shards (100 samples each)] |
16.53 ms | 81.23 μs | 58.3 ops/s | — | — |
|
test_load_from_directory[20s]
Load from directory [20 number of shards (100 samples each)] |
33.12 ms | 158.33 μs | 29.7 ops/s | — | — |
|
test_load_from_shard_urls[2s]
Load from shard urls [2 number of shards (100 samples each)] |
3.28 ms | 29.54 μs | 304.1 ops/s | — | — |
|
test_load_from_shard_urls[5s]
Load from shard urls [5 number of shards (100 samples each)] |
8.22 ms | 47.16 μs | 121.2 ops/s | — | — |
|
test_load_from_shard_urls[10s]
Load from shard urls [10 number of shards (100 samples each)] |
16.53 ms | 266.97 μs | 59.8 ops/s | — | — |
|
test_manifest_build[100]
Manifest build [100 samples in manifest] |
1.30 ms | 22.07 μs | 766.0 ops/s | 12.98 μs | 77.06 Kops/s |
|
test_manifest_build[1k]
Manifest build [1000 samples in manifest] |
7.85 ms | 71.57 μs | 122.2 ops/s | 7.85 μs | 127.35 Kops/s |
|
test_manifest_build[5k]
Manifest build [5000 samples in manifest] |
36.73 ms | 537.09 μs | 23.9 ops/s | 7.35 μs | 136.12 Kops/s |
|
test_manifest_write[100]
Manifest write [100 samples in manifest] |
3.69 ms | 79.68 μs | 269.5 ops/s | 36.90 μs | 27.10 Kops/s |
|
test_manifest_write[1k]
Manifest write [1000 samples in manifest] |
6.95 ms | 125.89 μs | 143.0 ops/s | 6.95 μs | 143.78 Kops/s |
S3 shard writing benchmarks via moto mock.
Parameters: n = samples per shard
| Test | Median | IQR | OPS | Med/sample | Samples/s |
|---|---|---|---|---|---|
|
test_s3_write_shards[100]
S3 write shards [100 samples per shard] |
19.75 ms | 277.42 μs | 49.1 ops/s | 197.53 μs | 5.06 Kops/s |
|
test_s3_write_shards[500]
S3 write shards [500 samples per shard] |
86.06 ms | 1.22 ms | 10.8 ops/s | 172.13 μs | 5.81 Kops/s |
|
test_s3_write_with_manifest
S3 write with manifest |
58.40 ms | 1.42 ms | 15.4 ops/s | 292.02 μs | 3.42 Kops/s |
|
test_s3_write_cache_local
S3 write cache local |
42.90 ms | 758.95 μs | 21.2 ops/s | 214.50 μs | 4.66 Kops/s |
|
test_s3_write_direct
S3 write direct |
35.91 ms | 418.05 μs | 27.8 ops/s | 179.57 μs | 5.57 Kops/s |
|
test_s3_write_numpy
S3 write numpy |
43.81 ms | 1.76 ms | 22.7 ops/s | 438.09 μs | 2.28 Kops/s |
Pure serialization/deserialization without disk I/O.
| Test | Median | IQR | OPS |
|---|---|---|---|
|
test_serialize_basic_sample
Serialize basic sample |
1.34 μs | 40.0 ns | 731.59 Kops/s |
|
test_deserialize_basic_sample
Deserialize basic sample |
1.84 μs | 50.0 ns | 529.29 Kops/s |
|
test_serialize_numpy_sample
Serialize numpy sample |
19.48 μs | 341.0 ns | 50.37 Kops/s |
|
test_deserialize_numpy_sample
Deserialize numpy sample |
5.59 μs | 110.0 ns | 174.80 Kops/s |
|
test_serialize_large_numpy
Serialize large numpy |
350.95 ms | 1.76 ms | 2.9 ops/s |
|
test_deserialize_large_numpy
Deserialize large numpy |
25.19 ms | 283.32 μs | 39.6 ops/s |
|
test_as_wds_basic
As wds basic |
4.65 μs | 55.5 ns | 209.67 Kops/s |
|
test_as_wds_numpy
As wds numpy |
24.62 μs | 371.0 ns | 39.75 Kops/s |