ecod-rs benchmark

ecod-rs is a Rust implementation of the ECOD outlier detector (Li et al., 2022) with Python bindings. Here it is compared with ECOD from PyOD.

Data is random normal noise of several sizes (rows x features) and the classical datasets from ADBench. Each dataset is split 60/40 into train and test. Every measurement runs in a fresh single-threaded process, times are taken over 5 runs.

What is measured:

Summary

How many times ecod-rs is faster than pyod-ECOD: median over all 53 datasets, the range in brackets. Below 1 means slower.

phasevarianttimes faster than pyod-ECOD
ECODInductiveECOD
fit×2.7 (1.1 to 14.7)×2.7 (1.1 to 14.8)
score×5.9 (2.2 to 16.2)×7.5 (3.0 to 18.0)
pipeline×4.3 (1.6 to 14.5)×4.7 (1.8 to 17.0)
savedefault×2.2 (0.4 to 5.1)×2.2 (0.4 to 5.2)
savecompressed×1.6 (0.3 to 19.9)×1.6 (0.3 to 19.3)
loaddefault×3.5 (0.5 to 6.8)×3.5 (0.5 to 6.9)
loadcompressed×27.9 (9.7 to 108.6)×28.0 (9.3 to 107.8)

Speed

Median time in seconds. In brackets, how many times faster than pyod-ECOD. PyOD has no separate convert step.

datasetfit, sscore, spipeline, sconvert, s
ECODInductiveECODpyod-ECODECODInductiveECODpyod-ECODECODInductiveECODpyod-ECODECODInductiveECOD
1000x100.0003693 (×4.2)0.0003669 (×4.3)0.0015650.0002956 (×6.2)0.0002232 (×8.3)0.0018430.0006459 (×5.0)0.0005723 (×5.6)0.0032140.000020180.000019
10000x100.004552 (×1.9)0.004441 (×2.0)0.0088050.00378 (×3.8)0.002606 (×5.6)0.014490.008408 (×2.9)0.00689 (×3.5)0.024220.000017810.00002018
100000x100.06476 (×1.7)0.06797 (×1.6)0.10760.06099 (×3.2)0.04018 (×4.9)0.19650.1299 (×2.4)0.1067 (×2.9)0.3090.000020180.00002018
1000000x101.487 (×1.1)1.522 (×1.1)1.6141.232 (×2.2)0.9247 (×3.0)2.7612.698 (×1.6)2.482 (×1.8)4.3950.0000190.00002018
10000x1000.05109 (×1.7)0.05225 (×1.7)0.08710.04296 (×3.4)0.02694 (×5.4)0.14650.09342 (×2.5)0.07281 (×3.2)0.23280.000017810.000019
100000x1000.6725 (×1.9)0.7091 (×1.8)1.2640.6373 (×3.4)0.3999 (×5.5)2.1941.306 (×2.7)1.106 (×3.1)3.4840.000017810.00001781
ADBench datasets
datasetfit, sscore, spipeline, sconvert, s
ECODInductiveECODpyod-ECODECODInductiveECODpyod-ECODECODInductiveECODpyod-ECODECODInductiveECOD
1_ALOI0.06468 (×2.5)0.06308 (×2.6)0.16410.05063 (×5.9)0.03958 (×7.5)0.29640.1148 (×4.1)0.1011 (×4.6)0.46940.000017810.00001781
2_annthyroid0.001701 (×2.7)0.001687 (×2.7)0.004520.001243 (×5.6)0.001066 (×6.6)0.0070040.002911 (×3.8)0.002632 (×4.2)0.011120.0000190.00001781
3_backdoor0.8765 (×3.1)0.8846 (×3.1)2.6990.6949 (×6.9)0.524 (×9.1)4.7611.57 (×4.7)1.41 (×5.3)7.4480.0000190.00001781
4_breastw0.0002018 (×7.2)0.0002054 (×7.1)0.0014630.0001627 (×9.8)0.0001437 (×11.1)0.0015950.0003253 (×8.5)0.000298 (×9.3)0.0027640.0000190.000019
5_campaign0.1066 (×3.0)0.1035 (×3.1)0.32040.07728 (×7.6)0.06353 (×9.2)0.58350.1845 (×5.0)0.1684 (×5.5)0.9180.0000190.000019
6_cardio0.001247 (×3.2)0.00128 (×3.1)0.0040290.0009736 (×6.3)0.0008086 (×7.6)0.0061290.002193 (×4.5)0.001995 (×4.9)0.0097920.0000190.000019
7_Cardiotocography0.001479 (×3.0)0.00147 (×3.0)0.0044450.001117 (×6.1)0.0009558 (×7.2)0.0068610.002546 (×4.3)0.002313 (×4.7)0.010850.000017810.000019
8_celeba0.3787 (×2.5)0.3651 (×2.5)0.93060.281 (×5.9)0.2176 (×7.7)1.6670.6578 (×4.0)0.5837 (×4.5)2.5990.000017810.000019
9_census8.53 (×2.9)8.626 (×2.9)25.056.4 (×6.8)4.776 (×9.1)43.3414.71 (×4.6)13.16 (×5.2)68.20.0000190.00001781
10_cover0.2011 (×1.8)0.2103 (×1.7)0.35570.1639 (×4.0)0.1309 (×5.0)0.65170.3656 (×2.8)0.343 (×3.0)1.0180.0000190.00001781
11_donors0.3373 (×2.4)0.3176 (×2.5)0.79340.2432 (×7.0)0.1921 (×8.9)1.7130.5791 (×4.3)0.5101 (×4.9)2.5130.000020180.000019
12_fault0.001851 (×2.4)0.001837 (×2.5)0.0045330.001411 (×5.4)0.001154 (×6.7)0.0076780.003235 (×3.7)0.002829 (×4.2)0.01190.0000190.00001781
13_fraud0.789 (×1.6)0.8157 (×1.5)1.2520.7288 (×3.0)0.4719 (×4.6)2.1631.518 (×2.2)1.293 (×2.6)3.410.0000190.000019
14_glass0.00006886 (×14.4)0.00006768 (×14.7)0.00099380.00005937 (×14.6)0.00005105 (×17.0)0.00086790.0001104 (×14.1)0.0000938 (×16.6)0.0015590.0000190.000019
15_Hepatitis0.00006886 (×14.5)0.00006768 (×14.8)0.00099850.0000558 (×15.4)0.00005105 (×16.9)0.0008620.0001069 (×14.4)0.00009024 (×17.0)0.0015380.0000190.000019
16_http0.1454 (×1.8)0.1321 (×2.0)0.25970.105 (×4.5)0.08176 (×5.7)0.46870.2491 (×3.0)0.2131 (×3.5)0.74520.000017810.000019
17_InternetAds0.09395 (×3.9)0.09503 (×3.8)0.36570.06351 (×10.0)0.05435 (×11.7)0.63770.1572 (×6.5)0.1493 (×6.8)1.0210.0000190.00001662
18_Ionosphere0.0003526 (×4.1)0.000355 (×4.1)0.0014490.0003004 (×5.8)0.0002125 (×8.2)0.0017410.0006316 (×4.5)0.0005426 (×5.2)0.0028270.0000190.00001781
19_landsat0.00829 (×2.2)0.008426 (×2.1)0.018010.006918 (×5.0)0.005187 (×6.7)0.034780.01515 (×3.8)0.01332 (×4.3)0.057460.000017810.00001781
20_letter0.001471 (×3.1)0.001476 (×3.1)0.0046220.001159 (×6.1)0.0009807 (×7.2)0.0070350.002586 (×4.2)0.002344 (×4.7)0.010960.000017810.000019
21_Lymphography0.00009142 (×12.0)0.00009261 (×11.9)0.0010980.00007955 (×12.9)0.00005937 (×17.3)0.0010280.0001603 (×11.3)0.0001437 (×12.6)0.0018110.0000190.00001781
22_magic.gamma0.009471 (×1.8)0.009621 (×1.8)0.016920.007986 (×3.8)0.005573 (×5.4)0.03010.0173 (×3.0)0.01483 (×3.5)0.051710.000020180.000019
23_mammography0.002595 (×3.0)0.002578 (×3.1)0.0078880.002008 (×6.1)0.001684 (×7.3)0.012250.004557 (×4.7)0.004063 (×5.3)0.021340.000017810.000019
24_mnist0.02664 (×2.5)0.02668 (×2.5)0.067830.01967 (×6.3)0.01624 (×7.6)0.12420.04571 (×4.3)0.04212 (×4.7)0.19790.000017810.000019
25_musk0.01873 (×2.0)0.01866 (×2.0)0.037180.01415 (×5.1)0.0104 (×7.0)0.072650.0324 (×3.7)0.02859 (×4.2)0.12120.000020180.000019
26_optdigits0.01161 (×2.5)0.01133 (×2.5)0.028460.008739 (×6.1)0.007048 (×7.6)0.053360.01915 (×4.8)0.01783 (×5.1)0.091060.0000190.00002018
27_PageBlocks0.002075 (×2.7)0.002106 (×2.7)0.0056050.001679 (×5.3)0.00125 (×7.1)0.008930.003716 (×3.8)0.003219 (×4.4)0.014110.000017810.000019
28_pendigits0.004486 (×2.2)0.004403 (×2.3)0.0099450.003571 (×4.5)0.00263 (×6.1)0.015970.007976 (×3.5)0.00685 (×4.1)0.028080.000017810.00001781
29_Pima0.0002244 (×6.0)0.0002244 (×6.0)0.0013370.0001852 (×8.0)0.0001449 (×10.3)0.0014880.0003989 (×6.3)0.0003265 (×7.7)0.0025120.0000190.00001781
30_satellite0.008328 (×2.1)0.008423 (×2.1)0.017680.00688 (×5.0)0.005191 (×6.7)0.034590.01509 (×3.9)0.01337 (×4.4)0.058320.0000190.00001781
31_satimage-20.007395 (×2.1)0.007499 (×2.1)0.015820.006132 (×4.7)0.004658 (×6.2)0.028860.01341 (×3.8)0.01189 (×4.3)0.051520.0000190.000019
32_shuttle0.0186 (×2.5)0.01836 (×2.5)0.046420.01394 (×6.3)0.01159 (×7.6)0.08830.03242 (×4.5)0.02939 (×4.9)0.14480.0000190.000019
33_skin0.04873 (×1.8)0.05115 (×1.7)0.088430.03542 (×4.2)0.03268 (×4.6)0.150.08413 (×2.9)0.08313 (×3.0)0.2480.000017810.000019
34_smtp0.01472 (×2.1)0.01545 (×2.0)0.03070.01194 (×4.8)0.009622 (×6.0)0.057350.02599 (×3.7)0.02449 (×3.9)0.095160.0000190.000019
35_SpamBase0.00837 (×3.1)0.00837 (×3.1)0.025930.006106 (×7.8)0.005109 (×9.3)0.047550.0147 (×5.6)0.01324 (×6.2)0.082530.000017810.00002018
36_speech0.05769 (×1.9)0.05859 (×1.9)0.11060.04782 (×4.5)0.032 (×6.7)0.21510.1058 (×3.2)0.08996 (×3.7)0.33510.0000190.000019
37_Stamps0.0001342 (×8.1)0.0001164 (×9.4)0.0010910.0001009 (×10.3)0.00007005 (×14.8)0.0010390.0002054 (×8.8)0.0001781 (×10.1)0.0018040.000020180.000019
38_thyroid0.0008335 (×3.5)0.0008608 (×3.3)0.0028790.0006328 (×6.5)0.0005272 (×7.8)0.0041010.001428 (×4.6)0.001308 (×5.0)0.0065870.0000190.00001781
39_vertebral0.00006886 (×14.7)0.00006886 (×14.7)0.0010130.00005818 (×15.3)0.00005105 (×17.4)0.00088930.0001104 (×14.5)0.00009736 (×16.5)0.0016030.0000190.000019
40_vowels0.0006471 (×3.2)0.0006316 (×3.2)0.0020520.0005153 (×5.2)0.0003823 (×7.0)0.0026770.001126 (×3.9)0.0009534 (×4.6)0.0043850.000020180.00001662
41_Waveform0.002802 (×2.2)0.00282 (×2.2)0.0062440.002169 (×4.6)0.001644 (×6.0)0.0099320.004863 (×3.1)0.004274 (×3.6)0.015190.000017810.00001781
42_WBC0.00007718 (×13.5)0.00007718 (×13.5)0.0010440.00005937 (×16.2)0.00005343 (×18.0)0.00096290.0001164 (×14.0)0.0001033 (×15.8)0.0016330.000017810.00001781
43_WDBC0.0003562 (×4.2)0.0003574 (×4.2)0.0015110.0003229 (×5.7)0.0002042 (×9.1)0.001850.0006483 (×4.7)0.0005378 (×5.6)0.0030230.0000190.000019
44_Wilt0.000995 (×3.0)0.001008 (×3.0)0.0030010.0008299 (×5.2)0.0005806 (×7.4)0.0043220.001804 (×3.9)0.001522 (×4.6)0.0069860.000017810.000019
45_wine0.00007124 (×14.2)0.00007124 (×14.2)0.0010080.00006293 (×14.6)0.00005343 (×17.2)0.0009190.0001164 (×14.1)0.00009855 (×16.6)0.001640.000017810.000019
46_WPBC0.0002161 (×5.9)0.0002173 (×5.9)0.0012780.00019 (×7.0)0.0001187 (×11.2)0.0013310.0003716 (×6.1)0.0003123 (×7.2)0.0022630.000020180.000019
47_yeast0.0004037 (×4.9)0.0003966 (×4.9)0.0019590.0003146 (×8.0)0.0002565 (×9.8)0.0025060.0006863 (×6.0)0.0006162 (×6.6)0.004090.0000190.000019

Memory

How much the process memory peak grew during the run, in MB.

datasetfit, MBscore, MB
ECODInductiveECODpyod-ECODECODInductiveECODpyod-ECOD
1000x100.035160.0351650.020.078120.011720.3555
10000x100.60550.593852.70.77730.12113.816
100000x105.9345.92273.366.4881.22745.44
1000000x1059.2359.32278.877.5513.66407.4
10000x1004.7154.71573.386.6840.121137.7
100000x10047.1647.15279.674.981.23412.9
ADBench datasets
datasetfit, MBscore, MB
ECODInductiveECODpyod-ECODECODInductiveECODpyod-ECOD
1_ALOI6.3486.35280.510.280.601661.04
2_annthyroid00.0117249.940.082030.082032.602
3_backdoor85.5385.53476.8141.11.164770.7
4_breastw0.019530.0312549.830.0039060.0039060
5_campaign11.711.69108.319.540.5116.8
6_cardio0.01172049.830.015620.015622.449
7_Cardiotocography0.011720.0117250.110.019530.019532.656
8_celeba36.1736.17229.558.062.477327.5
9_census687.5687.5347611383.666164
10_cover13.0913.09111.622.273.5130.7
11_donors35.1235.13192.640.277.582283.4
12_fault0.01172049.970.019530.019533.023
13_fraud40.2740.27240.159.773.484338.9
14_glass0049.77000
15_Hepatitis0049.79000
16_http7.7937.79383.3613.886.94975.1
17_InternetAds14.0514.06120.423.410.01953116.4
18_Ionosphere0049.65000.4531
19_landsat0.01172052.240.074220.074228.84
20_letter00.0117250.340.18750.015622.969
21_Lymphography0.01172049.78000
22_magic.gamma00.0117251.980.22660.22668.57
23_mammography0050.980.13280.13283.07
24_mnist3.4453.45767.745.8090.0859428.8
25_musk2.3122.32862.063.8790.0312519.22
26_optdigits1.5041.52582.5510.0585912.44
27_PageBlocks0.01172050.010.058590.058593.094
28_pendigits00.0117251.270.078120.078124.195
29_Pima0049.760.0039060.0039060.03516
30_satellite0.01172053.821.0080.0742210.32
31_satimage-20052.090.066410.066417.969
32_shuttle0.96480.964858.753.4410.597720.03
33_skin4.4844.48865.535.9882.99632.3
34_smtp1.321.30555.962.3241.1612.59
35_SpamBase0052.290.046880.046888.93
36_speech6.756.76284.1611.270.0390655.93
37_Stamps00.0117249.64000.0625
38_thyroid0.01562049.740.039060.039060.9727
39_vertebral0.011720.0117249.71000
40_vowels0049.610.011720.011720.7891
41_Waveform00.0117251.010.035160.035162.758
42_WBC00.0117249.72000
43_WDBC00.0117249.64000.4688
44_Wilt0049.660.054690.054691.039
45_wine00.0117249.72000
46_WPBC00.0117249.73000
47_yeast0.01172049.750.011720.011720.5039

Saving and loading

datasetvariantsave, sload, sfile, MB
ECODInductiveECODpyod-ECODECODInductiveECODpyod-ECODECODInductiveECODpyod-ECOD
1000x10default0.0001615 (×3.5)0.0001769 (×3.2)0.0005580.00008667 (×4.3)0.00009617 (×3.8)0.00036930.043960.043970.2389
1000x10compressed0.001912 (×2.7)0.001898 (×2.7)0.0051090.00008786 (×18.2)0.00008786 (×18.2)0.0016020.043940.043950.09782
10000x10default0.0008097 (×2.0)0.0009475 (×1.7)0.0015930.0004856 (×1.4)0.0005283 (×1.3)0.00068860.43560.43562.381
10000x10compressed0.02222 (×3.5)0.01913 (×4.1)0.077910.0005984 (×27.2)0.0005937 (×27.4)0.016250.42790.42721.633
100000x10default0.01358 (×0.8)0.01334 (×0.8)0.011310.004849 (×2.0)0.004918 (×1.9)0.0095044.1794.17923.8
100000x10compressed0.6898 (×1.5)0.6837 (×1.5)1.0590.007158 (×26.5)0.007191 (×26.4)0.18993.7313.73121.61
1000000x10default0.1623 (×0.8)0.1617 (×0.8)0.13160.09452 (×0.5)0.09546 (×0.5)0.0498540.6140.61238
1000000x10compressed12.03 (×0.9)12.08 (×0.9)11.110.1057 (×17.4)0.1097 (×16.8)1.83830.6430.64223
10000x100default0.01265 (×0.9)0.01279 (×0.9)0.011340.005008 (×1.6)0.004722 (×1.7)0.0078174.3234.32322.98
10000x100compressed0.5782 (×1.3)0.5471 (×1.3)0.7380.005744 (×27.9)0.005576 (×28.7)0.16014.2814.28115.65
100000x100default0.1676 (×0.7)0.1698 (×0.6)0.11010.09749 (×0.5)0.09757 (×0.5)0.0476141.7141.71229.8
100000x100compressed10.49 (×1.0)10.57 (×1.0)10.240.1206 (×14.3)0.1199 (×14.4)1.72537.3137.31211.4

Quality

ROC AUC on the ADBench test part.

datasetROC AUC
ECODInductiveECODpyod-ECOD
1_ALOI0.52640.52620.5264
2_annthyroid0.7570.75420.757
3_backdoor0.84820.84860.8482
4_breastw0.99290.9920.9929
5_campaign0.76530.76520.7653
6_cardio0.92910.9270.9291
7_Cardiotocography0.78360.78530.7836
8_celeba0.74850.74860.7485
9_census0.66030.66080.6603
10_cover0.92380.92340.9238
11_donors0.88840.88790.8884
12_fault0.49750.50010.4975
13_fraud0.94640.94640.9464
14_glass0.78710.78710.7871
15_Hepatitis0.77140.78860.7714
16_http0.97870.97860.9787
17_InternetAds0.68330.6850.6833
18_Ionosphere0.73360.73540.7336
19_landsat0.38260.38530.3826
20_letter0.52060.51380.5206
21_Lymphography111
22_magic.gamma0.6350.6340.635
23_mammography0.92880.92950.9288
24_mnist0.74220.73910.7422
25_musk0.94920.94450.9492
26_optdigits0.59380.59850.5938
27_PageBlocks0.91160.90930.9116
28_pendigits0.92870.93010.9287
29_Pima0.60840.60420.6084
30_satellite0.59450.59660.5945
31_satimage-20.96360.96360.9636
32_shuttle0.99370.99380.9937
33_skin0.48850.48920.4885
34_smtp0.95360.95350.9536
35_SpamBase0.6730.67930.673
36_speech0.44810.44930.4481
37_Stamps0.87820.87760.8782
38_thyroid0.97750.97720.9775
39_vertebral0.28270.27680.2827
40_vowels0.56430.55070.5643
41_Waveform0.62880.62810.6288
42_WBC0.99230.99230.9923
43_WDBC0.95980.9510.9598
44_Wilt0.40180.39790.4018
45_wine0.430.40.43
46_WPBC0.440.42380.44
47_yeast0.43350.43270.4335