Performance Benchmark and Calibration¶
Performance Evaluation Protocol¶
We use official models for evaluation if available. Otherwise, we use the following settings to train and evaluate different models for simplicity and consistency:
Metric Type |
Train |
Test |
|---|---|---|
FR |
KADID-10k |
CSIQ, LIVE, TID2008, TID2013 |
NR |
KonIQ-10k |
LIVEC, KonIQ-10k (official split), TID2013, SPAQ |
Aesthetic IQA |
AVA |
AVA (official split) |
Results are calculated with:
PLCC without any correction. Although test time value correction is common in IQA papers, we want to use the original value in our benchmark.
Full image single input. We use multi-patch testing only when it is necessary for the model to work.
Basically, we use the largest existing datasets for training, and cross dataset evaluation performance for fair comparison. The following models do not provide official weights, and are retrained by our scripts:
NR:
cnniqa,dbcnn,hyperiqaAesthetic IQA:
nima,nima-vgg16-ava
Performance on FR benchmarks¶
Metric name |
csiq(PLCC/SRCC/KRCC) |
live(PLCC/SRCC/KRCC) |
tid2008(PLCC/SRCC/KRCC) |
tid2013(PLCC/SRCC/KRCC) |
|---|---|---|---|---|
cw_ssim |
0.6078/0.7585/0.556 |
0.6287/0.7856/0.5919 |
0.5971/0.6475/0.4626 |
0.5819/0.6534/0.4716 |
ckdn |
0.6258/0.624/0.4365 |
0.6728/0.6684/0.4826 |
0.6237/0.5381/0.3813 |
0.6805/0.5944/0.4272 |
ssim |
0.765/0.8367/0.6323 |
0.8253/0.9103/0.7309 |
0.6003/0.6242/0.4521 |
0.6558/0.6269/0.455 |
ms_ssim |
0.7718/0.9131/0.7385 |
0.7668/0.9508/0.8034 |
0.7897/0.8542/0.6568 |
0.7819/0.7859/0.6048 |
psnr |
0.7857/0.8087/0.5989 |
0.8435/0.873/0.6801 |
0.489/0.5245/0.3696 |
0.6601/0.6869/0.4958 |
fsim |
0.8207/0.9309/0.7683 |
0.8598/0.9646/0.8366 |
0.8341/0.884/0.6991 |
0.8322/0.8509/0.6665 |
stlpips |
0.823/0.8952/0.7094 |
0.813/0.8826/0.6931 |
0.624/0.6404/0.454 |
0.7147/0.7365/0.5387 |
ahiq |
0.8374/0.8288/0.6222 |
0.8368/0.8971/0.7104 |
0.6215/0.6153/0.4298 |
0.6796/0.6479/0.462 |
pieapp |
0.838/0.8968/0.7109 |
0.8577/0.9182/0.7491 |
0.6443/0.7971/0.6089 |
0.7195/0.8438/0.6571 |
lpips |
0.9005/0.9233/0.7499 |
0.8436/0.9235/0.7507 |
0.711/0.7151/0.5221 |
0.753/0.7445/0.5477 |
lpips+ |
0.9041/0.9285/0.7576 |
0.8456/0.9248/0.7546 |
0.7318/0.7379/0.5424 |
0.7656/0.7623/0.5639 |
lpips-vgg |
0.9043/0.883/0.6968 |
0.9336/0.9319/0.7646 |
0.6974/0.6536/0.4822 |
0.7324/0.6696/0.497 |
wadiqam_fr |
0.9087/0.922/0.7461 |
0.9163/0.9308/0.7584 |
0.8221/0.8222/0.6245 |
0.8424/0.8264/0.628 |
lpips-vgg+ |
0.9169/0.894/0.7128 |
0.9499/0.9503/0.7983 |
0.7406/0.6869/0.5113 |
0.7606/0.6913/0.5152 |
vif |
0.9219/0.9194/0.7532 |
0.9411/0.9636/0.8282 |
0.7769/0.7491/0.5861 |
0.7336/0.677/0.5148 |
dists |
0.9324/0.9296/0.7644 |
0.9115/0.9477/0.7927 |
0.7032/0.6648/0.4861 |
0.7538/0.7077/0.5212 |
topiq_fr |
0.9589/0.9674/0.8378 |
0.9542/0.9759/0.8617 |
0.9045/0.9226/0.7555 |
0.9158/0.9165/0.7441 |
Performance on NR benchmarks¶
Metric name |
livec(PLCC/SRCC/KRCC) |
koniq10k(PLCC/SRCC/KRCC) |
tid2013(PLCC/SRCC/KRCC) |
flive(PLCC/SRCC/KRCC) |
spaq(PLCC/SRCC/KRCC) |
koniq10k-1024(PLCC/SRCC/KRCC) |
|---|---|---|---|---|---|---|
brisque |
0.3509/0.3128/0.2113 |
0.2107/0.23/0.1548 |
0.4317/0.3672/0.2574 |
0.2661/0.237/0.16 |
0.5333/0.5259/0.366 |
|
nrqm |
0.4122/0.3012/0.2013 |
0.4756/0.3715/0.2517 |
0.4795/0.349/0.2413 |
0.5394/0.4935/0.3432 |
0.6784/0.6507/0.45 |
|
niqe |
0.48/0.4505/0.3069 |
0.3155/0.3769/0.2555 |
0.3669/0.3121/0.2124 |
0.21/0.2626/0.174 |
0.6692/0.6929/0.4928 |
|
ilniqe |
0.4938/0.4381/0.2972 |
0.5232/0.551/0.3862 |
0.5156/0.487/0.3442 |
0.4084/0.373/0.2532 |
0.6596/0.719/0.5137 |
|
nima |
0.4993/0.5071/0.348 |
0.7156/0.6662/0.4816 |
0.3324/0.321/0.2159 |
0.1655/0.1696/0.1123 |
0.5153/0.5201/0.3558 |
|
pi |
0.5201/0.4615/0.3139 |
0.4688/0.4573/0.3132 |
0.4627/0.3479/0.2398 |
0.3403/0.3428/0.2293 |
0.7353/0.7307/0.5227 |
|
cnniqa |
0.6372/0.6089/0.4257 |
0.7934/0.7551/0.558 |
0.398/0.1769/0.117 |
0.4546/0.4383/0.3034 |
0.7272/0.7397/0.5263 |
|
wadiqam_nr |
0.6631/0.6675/0.4721 |
0.83/0.8046/0.6129 |
0.3517/0.1544/0.1002 |
0.5631/0.5147/0.3588 |
||
clipiqa |
0.6883/0.6955/0.5065 |
0.7211/0.6572/0.4736 |
0.6471/0.5786/0.4107 |
0.5028/0.4674/0.3231 |
||
tres-flive |
0.7213/0.7336/0.5373 |
0.7507/0.7068/0.516 |
0.6411/0.6317/0.4518 |
0.6137/0.7269/0.533 |
||
paq2piq |
0.7542/0.7188/0.5302 |
0.7062/0.643/0.4622 |
0.5776/0.4011/0.2838 |
0.6906/0.6457/0.4637 |
0.775/0.8289/0.6207 |
|
clipiqa+_vitL14_512 |
0.7679/0.7729/0.5733 |
0.8747/0.861/0.6721 |
0.6063/0.5259/0.3709 |
0.5393/0.5436/0.3776 |
||
dbcnn |
0.774/0.7562/0.5563 |
0.9197/0.9034/0.7338 |
0.5141/0.3855/0.2691 |
0.6151/0.5764/0.4053 |
0.8549/0.8473/0.639 |
|
hyperiqa |
0.7779/0.7546/0.5562 |
0.9233/0.904/0.7336 |
0.5627/0.4537/0.3177 |
0.3656/0.3719/0.249 |
||
musiq-paq2piq |
0.8014/0.7672/0.5743 |
0.7655/0.7084/0.5146 |
0.7828/0.7436/0.5538 |
0.8112/0.8436/0.6412 |
||
tres-koniq |
0.8118/0.7771/0.5808 |
0.513/0.4919/0.3391 |
0.8624/0.8619/0.66 |
|||
musiq-spaq |
0.8134/0.789/0.5937 |
0.7528/0.6799/0.4927 |
0.6039/0.5627/0.3941 |
|||
clipiqa+_rn50_512 |
0.8181/0.818/0.6231 |
0.9012/0.8847/0.7033 |
0.6577/0.5949/0.4241 |
0.5766/0.5654/0.3961 |
||
topiq_nr |
0.8261/0.8106/0.6165 |
0.9436/0.9299/0.7727 |
0.5625/0.4452/0.3143 |
0.6289/0.5819/0.4105 |
0.8744/0.8704/0.6716 |
|
maniqa |
0.8262/0.8399/0.6491 |
0.9133/0.8934/0.7198 |
0.457/0.4515/0.318 |
0.4416/0.4489/0.3052 |
0.814/0.8166/0.6036 |
|
musiq-koniq |
0.8295/0.7889/0.5986 |
0.8958/0.8654/0.6817 |
0.6814/0.575/0.4131 |
0.5128/0.4978/0.3437 |
0.8626/0.8676/0.6649 |
|
clipiqa+ |
0.8312/0.8045/0.6109 |
0.8454/0.8026/0.6123 |
0.701/0.6318/0.4541 |
0.5973/0.5746/0.4044 |
0.909/0.8954/0.7181 |
|
qalign_4bit |
0.8935/0.8833/0.7024 |
0.8578/0.8385/0.6456 |
0.9452/0.9325/0.7778 |
|||
qalign |
0.8942/0.8814/0.6993 |
0.8529/0.8313/0.6368 |
0.9506/0.941/0.7924 |
|||
qalign_8bit |
0.8963/0.8819/0.7009 |
0.8527/0.8315/0.637 |
0.384/0.4147/0.283 |
0.9358/0.9324/0.7662 |
0.9497/0.9395/0.7898 |
Performance on image aesthetic benchmarks¶
Metric name |
ava(PLCC/SRCC/KRCC) |
|---|---|
nima-vgg16-ava |
0.6624/0.657/0.4719 |
nima |
0.7172/0.7126/0.5213 |
clipiqa |
0.3576/0.3383/0.2301 |
laion_aes |
0.666/0.6653/0.4788 |
topiq_iaa_res50 |
0.737/0.7359/0.5423 |
topiq_iaa |
0.7902/0.791/0.5969 |
qalign |
0.8192/0.8223/0.6307 |
Results Calibration¶
Method |
I03.bmp |
I04.bmp |
I06.bmp |
I08.bmp |
I19.bmp |
|---|---|---|---|---|---|
brisque |
94.6421 |
-0.1076 |
0.9929 |
5.3583 |
72.2617 |
brisque(ours) |
94.6443 |
-0.1068 |
1.0732 |
5.1447 |
66.8415 |
brisque_matlab |
70.68 |
-2.36 |
15.8 |
23.28 |
60.97 |
brisque_matlab(ours) |
70.6808 |
-2.369 |
15.8031 |
23.0786 |
60.8247 |
ckdn |
0.2833 |
0.5767 |
0.6367 |
0.658 |
0.5999 |
ckdn(ours) |
0.284 |
0.565 |
0.6263 |
0.6414 |
0.5935 |
cw_ssim |
0.2763 |
0.9996 |
1.0 |
0.9068 |
0.8658 |
cw_ssim(ours) |
0.2763 |
0.9996 |
1.0 |
0.9068 |
0.8658 |
dists |
0.4742 |
0.1424 |
0.0683 |
0.0287 |
0.3123 |
dists(ours) |
0.4742 |
0.1424 |
0.0683 |
0.0287 |
0.3123 |
entropy |
6.9511 |
6.9661 |
7.5309 |
7.5566 |
5.7629 |
entropy(ours) |
6.9511 |
6.9661 |
7.5309 |
7.5565 |
5.7629 |
fsim |
0.689 |
0.9702 |
0.9927 |
0.9575 |
0.822 |
fsim(ours) |
0.6891 |
0.9702 |
0.9927 |
0.9575 |
0.822 |
gmsd |
0.2203 |
0.0005 |
0.0004 |
0.1346 |
0.205 |
gmsd(ours) |
0.2203 |
0.0005 |
0.0004 |
0.1346 |
0.205 |
ilniqe |
113.4801 |
23.9968 |
19.975 |
22.4493 |
56.6721 |
ilniqe(ours) |
115.6107 |
24.0636 |
19.7494 |
22.3251 |
54.7628 |
laion_aes |
3.6420 |
5.5836 |
5.0716 |
4.6458 |
3.0889 |
laion_aes(ours) |
3.7204 |
5.5917 |
5.0756 |
4.6551 |
3.0973 |
lpips |
0.7237 |
0.2572 |
0.0508 |
0.052 |
0.4253 |
lpips(ours) |
0.7236 |
0.2572 |
0.0508 |
0.0521 |
0.4253 |
mad |
195.2796 |
80.8379 |
30.3918 |
84.3542 |
202.2371 |
mad(ours) |
195.2796 |
80.8379 |
30.3918 |
84.3542 |
202.2371 |
ms_ssim |
0.6733 |
0.9996 |
0.9998 |
0.9566 |
0.8462 |
ms_ssim(ours) |
0.6707 |
0.9996 |
0.9998 |
0.9567 |
0.8418 |
musiq |
12.494 |
75.332 |
73.429 |
75.188 |
36.938 |
musiq(ours) |
12.4785 |
75.7772 |
73.7454 |
75.4609 |
38.0257 |
musiq-ava |
3.398 |
5.648 |
4.635 |
5.186 |
4.128 |
musiq-ava(ours) |
3.4085 |
5.6933 |
4.697 |
5.1964 |
4.1957 |
musiq-koniq |
12.494 |
75.332 |
73.429 |
75.188 |
36.938 |
musiq-koniq(ours) |
12.4773 |
75.7764 |
73.7459 |
75.4604 |
38.0248 |
musiq-paq2piq |
46.035 |
72.66 |
73.625 |
74.361 |
69.006 |
musiq-paq2piq(ours) |
46.0196 |
72.6655 |
73.7657 |
74.388 |
69.7233 |
musiq-spaq |
17.685 |
70.492 |
78.74 |
79.015 |
49.105 |
musiq-spaq(ours) |
17.6808 |
70.6541 |
79.0371 |
79.3199 |
50.455 |
niqe |
15.7536 |
3.6549 |
3.2355 |
3.184 |
8.6352 |
niqe(ours) |
15.6538 |
3.6549 |
3.2342 |
3.1922 |
9.0722 |
niqe_matlab |
7.2 |
2.99 |
3.17 |
3.71 |
7.69 |
niqe_matlab(ours) |
7.1707 |
2.9907 |
3.1551 |
3.6781 |
7.4782 |
nlpd |
0.5616 |
0.0195 |
0.0159 |
0.3028 |
0.4326 |
nlpd(ours) |
0.5616 |
0.0139 |
0.011 |
0.3033 |
0.4335 |
nrqm |
1.3894 |
8.9394 |
8.9735 |
6.829 |
6.312 |
nrqm(ours) |
1.3932 |
8.9419 |
8.9721 |
6.8309 |
6.3031 |
paq2piq |
44.134 |
73.6015 |
74.3297 |
76.8748 |
70.9153 |
paq2piq(ours) |
44.1328 |
73.6013 |
74.3302 |
76.8752 |
70.9178 |
pi |
11.9235 |
3.072 |
2.618 |
2.8074 |
6.7713 |
pi(ours) |
11.9287 |
3.073 |
2.6357 |
2.7979 |
6.9546 |
piqe |
100.0 |
21.62 |
35.86 |
41.15 |
76.95 |
piqe(ours) |
100.0 |
21.6243 |
35.8646 |
41.1471 |
76.9485 |
psnr |
21.11 |
20.99 |
27.01 |
23.3 |
21.62 |
psnr(ours) |
21.1136 |
20.9872 |
27.0139 |
23.3002 |
21.6186 |
ssim |
0.6993 |
0.9978 |
0.9989 |
0.9669 |
0.6519 |
ssim(ours) |
0.6997 |
0.9978 |
0.9989 |
0.9671 |
0.6522 |
vif |
0.0172 |
0.9891 |
0.9924 |
0.9103 |
0.1745 |
vif(ours) |
0.0172 |
0.9891 |
0.9924 |
0.9104 |
0.175 |
vsi |
0.9139 |
0.962 |
0.9922 |
0.9571 |
0.9262 |
vsi(ours) |
0.9244 |
0.9497 |
0.9877 |
0.9541 |
0.9348 |
Efficiency Benchmark¶
Method |
cpu_xe6150 |
cuda_v100 |
Peak GPU Mem (GB) |
cpu_m1pro |
cpu_m2 |
|---|---|---|---|---|---|
nima-vgg16-ava |
0.0953 |
0.024 |
0.1253 |
0.0763 |
0.1021 |
paq2piq |
0.2343 |
0.0325 |
0.1888 |
0.2388 |
0.2706 |
piqe |
0.0795 |
0.0325 |
0.0565 |
0.1115 |
0.0989 |
unique |
2.2314 |
0.0327 |
0.2284 |
0.3453 |
0.4172 |
brisque_matlab |
0.1039 |
0.0354 |
0.1113 |
0.125 |
0.1262 |
entropy |
0.239 |
0.0358 |
2.2474 |
0.1457 |
0.1691 |
brisque |
0.1184 |
0.038 |
0.1075 |
0.1329 |
0.1292 |
niqe_matlab |
0.2105 |
0.0404 |
0.1214 |
0.2172 |
0.2247 |
niqe |
0.2113 |
0.0405 |
0.1214 |
0.2154 |
0.2229 |
wadiqam_nr |
0.4299 |
0.0413 |
0.4448 |
0.7347 |
0.7452 |
psnr |
0.0416 |
0.0443 |
0.0548 |
0.0212 |
0.0253 |
gmsd |
0.052 |
0.045 |
0.0626 |
0.0438 |
0.0569 |
hyperiqa |
1.2693 |
0.0451 |
0.4696 |
3.1934 |
2.8124 |
arniqa-tid |
0.7548 |
0.0457 |
0.3258 |
0.5111 |
0.6284 |
laion_aes |
0.456 |
0.0458 |
1.636 |
0.2491 |
0.3968 |
arniqa-live |
0.7568 |
0.0459 |
0.3258 |
0.5636 |
0.6182 |
psnry |
0.0436 |
0.0461 |
0.0547 |
0.0719 |
0.0449 |
topiq_iaa |
0.4252 |
0.0469 |
0.5821 |
0.3102 |
0.3741 |
arniqa-spaq |
0.7576 |
0.0483 |
0.3258 |
0.5064 |
0.6152 |
arniqa-flive |
0.7425 |
0.0487 |
0.3258 |
0.504 |
0.6369 |
arniqa-kadid |
0.7424 |
0.0492 |
0.3258 |
0.5122 |
0.634 |
liqe_mix |
0.3701 |
0.0502 |
0.9604 |
0.2168 |
0.3479 |
vsi |
0.0989 |
0.0528 |
0.1641 |
0.0927 |
0.1175 |
clipiqa+ |
0.8577 |
0.0535 |
0.6346 |
0.5471 |
0.6999 |
clipiqa+_rn50_512 |
0.8618 |
0.0538 |
0.6347 |
0.5471 |
0.7182 |
topiq_iaa_res50 |
0.2462 |
0.0554 |
0.3846 |
0.2011 |
0.2434 |
nima-koniq |
0.1658 |
0.057 |
0.2453 |
0.1194 |
0.1597 |
clipiqa |
0.9891 |
0.058 |
0.6344 |
0.6273 |
0.7961 |
liqe |
0.3716 |
0.0581 |
0.9604 |
0.1945 |
0.2804 |
topiq_nr-spaq |
1.1112 |
0.0589 |
0.6739 |
0.7977 |
0.9848 |
musiq-ava |
0.6021 |
0.0594 |
0.405 |
0.7081 |
0.927 |
topiq_nr-flive |
1.109 |
0.0594 |
0.6739 |
0.7931 |
0.9825 |
topiq_nr |
1.1161 |
0.0595 |
0.6739 |
0.8416 |
1.003 |
arniqa-clive |
0.7379 |
0.061 |
0.3258 |
0.4874 |
0.6417 |
arniqa |
0.7594 |
0.0617 |
0.3258 |
0.4851 |
0.6397 |
ckdn |
0.0854 |
0.0623 |
0.1055 |
0.0834 |
0.0799 |
dbcnn |
1.5287 |
0.0639 |
0.5854 |
1.4413 |
1.8485 |
musiq-spaq |
0.6012 |
0.0639 |
0.405 |
0.7371 |
0.8169 |
musiq |
0.6073 |
0.0654 |
0.405 |
0.7317 |
0.8374 |
nima |
0.1657 |
0.067 |
0.2453 |
0.1516 |
0.134 |
lpips |
0.2048 |
0.0718 |
0.2934 |
0.1742 |
0.2203 |
nlpd |
0.1339 |
0.0785 |
0.0783 |
0.1187 |
0.1103 |
nima-spaq |
0.1651 |
0.0793 |
0.2453 |
0.1538 |
0.1555 |
musiq-paq2piq |
0.5992 |
0.0804 |
0.405 |
0.7071 |
0.8245 |
lpips+ |
0.2761 |
0.081 |
0.4736 |
0.2498 |
0.251 |
fsim |
0.1167 |
0.0827 |
0.1065 |
0.1155 |
0.1396 |
wadiqam_fr |
0.8664 |
0.0829 |
0.4742 |
1.4219 |
1.502 |
arniqa-csiq |
0.749 |
0.09 |
0.3258 |
0.5203 |
0.6121 |
msswd |
0.0934 |
0.3477 |
|||
vif |
0.6349 |
0.0973 |
0.2526 |
1.5798 |
1.5738 |
topiq_fr-pipal |
1.7137 |
0.1152 |
1.0247 |
1.274 |
1.5432 |
ssimc |
1.4399 |
0.1179 |
0.3595 |
0.9434 |
1.2198 |
dists |
2.9309 |
0.1248 |
1.528 |
2.9516 |
3.7126 |
lpips-vgg+ |
3.878 |
0.1277 |
4.4663 |
2.6661 |
3.4065 |
deepdc |
0.1301 |
1.0967 |
|||
ms_ssim |
0.659 |
0.137 |
0.138 |
0.5294 |
0.5486 |
ilniqe |
2.7671 |
0.1385 |
1.5472 |
2.381 |
2.5457 |
lpips-vgg |
3.1749 |
0.1431 |
2.4982 |
2.514 |
3.1522 |
ssim |
0.5074 |
0.1579 |
0.138 |
0.35 |
0.4453 |
uranker |
2.2685 |
0.166 |
0.5723 |
2.6373 |
2.7708 |
stlpips |
1.3045 |
0.1799 |
0.9079 |
1.4463 |
1.4641 |
pieapp |
7.4984 |
0.1994 |
3.5438 |
8.4581 |
8.3669 |
topiq_fr |
1.7425 |
0.2113 |
1.0247 |
1.248 |
1.6452 |
qalign |
0.2204 |
15.4313 |
|||
stlpips-vgg |
4.1181 |
0.2231 |
2.5082 |
3.5801 |
4.5599 |
tres-flive |
28.6947 |
0.2246 |
2.0755 |
11.895 |
10.8521 |
tres |
29.2183 |
0.2273 |
2.0755 |
11.6795 |
10.8187 |
cnniqa |
0.2038 |
0.4322 |
0.4052 |
0.2049 |
0.1816 |
clipiqa+_vitL14_512 |
15.1827 |
0.5016 |
1.9617 |
9.4805 |
9.9749 |
maniqa |
13.0507 |
0.5424 |
2.6567 |
8.7182 |
10.4081 |
maniqa-pipal |
13.5027 |
0.5571 |
2.6567 |
8.6304 |
10.5159 |
qalign_8bit |
0.5595 |
8.1834 |
|||
qalign_4bit |
0.5684 |
5.0484 |
|||
maniqa-kadid |
13.6595 |
0.579 |
2.6567 |
8.5639 |
10.3526 |
nrqm |
17.0939 |
0.6223 |
2.2224 |
12.0018 |
13.265 |
pi |
16.9402 |
0.7532 |
2.1981 |
14.4017 |
13.9593 |
ahiq |
147.5412 |
0.7961 |
4.4696 |
23.8511 |
23.6154 |
mad |
14.4129 |
0.9686 |
3.3962 |
14.2657 |
14.4588 |
cw_ssim |
0.8336 |
0.9904 |
0.5117 |
1.0554 |
1.1038 |
compare2score |
1.7029 |
15.5104 |