Under development – this currently contains examples of the earlier MAD synthesis, but we have yet to reproduce it using plenoptic.
Reproducing Wang and Simoncelli, 2008 (MAD Competition)
Goal here is to reproduce original MAD Competition results, as generated using the matlab code originally provided by Zhou Wang and then modified by the authors. MAD Competition is a synthesis method for efficiently computing two models, by generating sets of images that minimize/maximize one model’s loss while holding the other’s constant. For more details, see the 07_MAD_Competition and 08_Simple_MAD notebooks.
[1]:
import imageio
import torch
import scipy.io as sio
import pyrtools as pt
from scipy.io import loadmat
import numpy as np
import matplotlib.pyplot as plt
import plenoptic as po
import os.path as op
%matplotlib inline
%load_ext autoreload
%autoreload 2
SSIM
Before we discuss MAD Competition, let’s look a little at SSIM, since that’s the metric used in the original paper, and which we’ll be using here. Important to remember that SSIM is a similarity metric, so higher is better, and thus a value of 1 means the images are identical and it’s bounded between 0 and 1.
We have tests to show that this matches the output of the MATLAB code, won’t show here.
[2]:
img1 = po.data.einstein()
img2 = po.data.curie()
noisy = po.tools.add_noise(img1, [2,4,8])
We can see that increasing the noise level decreases the SSIM value, but not linearly
[3]:
po.metric.ssim(img1, noisy)
/home/billbrod/Documents/plenoptic/src/plenoptic/metric/perceptual_distance.py:42: UserWarning: Image range falls outside [0, 1]. img1: tensor([0.0039, 1.0000]), img2: tensor([-12.3002, 11.9818]). Continuing anyway...
warnings.warn("Image range falls outside [0, 1]."
/home/billbrod/micromamba/envs/plenoptic/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[3]:
tensor([[0.0026],
[0.0016],
[0.0004]])
And that our noise level does match the MSE
[4]:
po.metric.mse(img1, noisy)
[4]:
tensor([[2.0000],
[4.0000],
[8.0000]])
MAD Competition
The following figure shows the results of MAD Competition synthesis using the original MATLAB code. It shows the original image in the top left. We then added some Gaussian noise (with a specified standard error) to get the image right below it. The four images to the right of that are the MAD-synthesized images. The first two have the same mean-squared error (MSE) as the first image (and each other), but the best and worst SSIM value (SSIM is a similarity metric, so higher is better), while the second two have the same SSIM as the first image, but the best and worst MSE. By comparing these images, we can get a sense for what MSE and SSIM consider important for image quality.
[5]:
# We need to download some additional data for this portion of the notebook. In order to do so,
# we use an optional dependency, pooch. If the following raises an ImportError or ModuleNotFoundError
# then install pooch in your plenoptic environment and restart your kernel.
fig, results = po.tools.external.plot_MAD_results('samp6', [128], vrange='row1', zoom=3)
There’s lots of info here, on the outputs of the MATLAB synthesis. We will later add stuff to investigate this using plenoptic.
[6]:
results
[6]:
{'L128': {'FIX_MSE': 127.99999999999999,
'FIX_SSIM': 0.8183184633106257,
'mse_fixmse_maxssim': array([128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128.]),
'ssim_fixmse_maxssim': array([0.82669306, 0.83641599, 0.84768936, 0.86021352, 0.87332037,
0.8861153 , 0.89794336, 0.90864194, 0.91828312, 0.9270046 ,
0.93495769, 0.94226293, 0.94896492, 0.95506452, 0.9605342 ,
0.96533558, 0.96945409, 0.97290935, 0.97575609, 0.97807185,
0.97994388, 0.98145704, 0.98268627, 0.98369369, 0.98452855,
0.98522884, 0.98582351, 0.98633451, 0.98677852, 0.98716831,
0.98751374, 0.98782249, 0.98810062, 0.98835294, 0.98858334,
0.98879493, 0.98899028, 0.98917148, 0.98934029, 0.98949816,
0.9896463 , 0.98978576, 0.98991742, 0.99004203, 0.99016023,
0.99027259, 0.99037961, 0.9904817 , 0.99057924, 0.99067257,
0.99076198, 0.99084774, 0.99093007, 0.9910092 , 0.99108531,
0.99115858, 0.99122917, 0.99129722, 0.99136287, 0.99142623,
0.99148744, 0.99154658, 0.99160377, 0.99165909, 0.99171263,
0.99176448, 0.9918147 , 0.99186338, 0.99191058, 0.99195637,
0.99200081, 0.99204396, 0.99208587, 0.99212661, 0.99216622,
0.99220476, 0.99224226, 0.99227877, 0.99231435, 0.99234902,
0.99238284, 0.99241583, 0.99244804, 0.99247949, 0.99251023,
0.99254029, 0.99256969, 0.99259847, 0.99262665, 0.99265426,
0.99268134, 0.99270789, 0.99273395, 0.99275955, 0.99278469,
0.99280941, 0.99283372, 0.99285765, 0.9928812 , 0.99290441]),
'mse_fixmse_minssim': array([128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128.]),
'ssim_fixmse_minssim': array([0.81069721, 0.80415136, 0.79827382, 0.7927989 , 0.78758476,
0.7825861 , 0.77782948, 0.77338629, 0.76933555, 0.7657286 ,
0.76258054, 0.75987991, 0.75759382, 0.75567185, 0.75405628,
0.7526936 , 0.75154034, 0.75056265, 0.74973342, 0.74902986,
0.74843227, 0.74792359, 0.7474892 , 0.74711674, 0.74679588,
0.74651809, 0.74627635, 0.74606493, 0.74587912, 0.74571508,
0.74556965, 0.74544023, 0.74532467, 0.74522116, 0.74512817,
0.74504441, 0.74496878, 0.74490032, 0.74483822, 0.74478177,
0.74473035, 0.74468342, 0.74464051, 0.74460123, 0.7445652 ,
0.74453212, 0.74450172, 0.74447375, 0.74444799, 0.74442426,
0.74440239, 0.74438221, 0.7443636 , 0.74434643, 0.74433057,
0.74431594, 0.74430244, 0.74428997, 0.74427846, 0.74426784,
0.74425803, 0.74424898, 0.74424062, 0.74423291, 0.74422579,
0.74421923, 0.74421317, 0.74420758, 0.74420242, 0.74419767,
0.74419328, 0.74418924, 0.74418551, 0.74418208, 0.74417891,
0.74417599, 0.74417331, 0.74417083, 0.74416855, 0.74416646,
0.74416453, 0.74416275, 0.74416112, 0.74415963, 0.74415825,
0.74415699, 0.74415583, 0.74415477, 0.7441538 , 0.74415291,
0.7441521 , 0.74415135, 0.74415068, 0.74415006, 0.7441495 ,
0.74414899, 0.74414853, 0.74414811, 0.74414773, 0.74414739]),
'maxssim': 0.99290440833461,
'minssim': 0.7441473913547447,
'mse_fixssim_minmse': array([127.62569966, 127.25907758, 126.89830629, 126.5432421 ,
126.19375253, 125.84970824, 125.51098323, 125.17745477,
124.84900331, 124.5255124 , 124.2068686 , 123.89296145,
123.58368333, 123.27892946, 122.97859775, 122.68258882,
122.39080584, 122.10315454, 121.81954311, 121.53988213,
121.26408454, 120.99206555, 120.72374258, 120.45903523,
120.19786522, 119.94015629, 119.68583422, 119.43482671,
119.18706338, 118.94247568, 118.70099686, 118.46256195,
118.22710767, 117.99457239, 117.76489611, 117.53802042,
117.31388843, 117.09244474, 116.87363542, 116.65740794,
116.44371116, 116.23249528, 116.02371179, 115.81731348,
115.61325436, 115.41148964, 115.21197571, 115.01467011,
114.81953145, 114.62651948, 114.43559495, 114.24671966,
114.0598564 , 113.87496892, 113.69202191, 113.510981 ,
113.33181268, 113.15448433, 112.97896417, 112.80522123,
112.63322535, 112.46294714, 112.29435797, 112.12742994,
111.96213588, 111.79844929, 111.63634436, 111.47579594,
111.31677951, 111.15927119, 111.00324769, 110.8486863 ,
110.6955649 , 110.54386192, 110.39355633, 110.24462762,
110.09705579, 109.95082136, 109.8059053 , 109.66228908,
109.5199546 , 109.37888422, 109.23906072, 109.10046731,
108.96308761, 108.82690563, 108.69190575, 108.55807275,
108.42539176, 108.29384826, 108.16342808, 108.03411739,
107.90590266, 107.77877069, 107.6527086 , 107.52770378,
107.40374392, 107.280817 , 107.15891125, 107.03801519]),
'ssim_fixssim_minmse': array([0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846]),
'mse_fixssim_maxmse': array([ 131.81989257, 136.34348117, 141.63201819, 147.93891278,
155.38338115, 164.2072263 , 174.58969259, 180.77038681,
187.35687873, 194.48181227, 198.57370304, 202.66885863,
206.95343129, 211.44231618, 216.13691245, 221.02634466,
226.11581241, 231.45406809, 237.02262995, 242.83028251,
248.81360619, 255.03345526, 261.46183743, 268.20589445,
271.92855589, 275.50581231, 279.16591822, 282.90724991,
286.73714745, 290.64830841, 294.64536986, 298.72977854,
302.9070491 , 307.17661949, 311.52367919, 315.9615864 ,
320.49015071, 325.12141654, 329.86378322, 334.71438487,
339.66399361, 344.73644098, 349.93210914, 355.25219363,
360.69572122, 366.26837608, 371.96846258, 377.80329199,
383.77938626, 389.88638347, 396.14353339, 402.54835547,
409.12344679, 415.86473831, 422.7769448 , 429.86538679,
437.11955142, 444.55628708, 452.18579893, 460.02165701,
468.0649593 , 476.29650757, 484.7347087 , 493.40456576,
502.30552879, 511.44021594, 520.83114704, 530.48547572,
540.40541267, 550.59925317, 561.07367344, 571.86215993,
582.96345493, 594.39243813, 606.16232589, 618.28830266,
630.7534013 , 643.57218989, 656.77008653, 670.38608448,
684.42229927, 698.90648368, 713.75502045, 729.09525608,
744.89191175, 761.03686543, 777.65684378, 794.83179371,
812.49479186, 830.66898008, 849.46972268, 868.84766347,
888.75751215, 909.25712128, 930.36668338, 952.1656941 ,
974.65873922, 997.87604088, 1021.77859313, 1046.42121422]),
'ssim_fixssim_maxmse': array([0.81838315, 0.81837394, 0.81840433, 0.81841667, 0.8184811 ,
0.81854265, 0.81865555, 0.8185667 , 0.81860168, 0.8186602 ,
0.81849321, 0.81849264, 0.81849454, 0.81849667, 0.81850134,
0.81850698, 0.81852163, 0.8185241 , 0.81853312, 0.81854646,
0.81856169, 0.81858209, 0.81861676, 0.8186252 , 0.8184758 ,
0.81848028, 0.81848157, 0.81848229, 0.8184804 , 0.81848067,
0.81848019, 0.81847912, 0.81847739, 0.81847648, 0.81848131,
0.81848611, 0.81848742, 0.81848633, 0.81848348, 0.81848151,
0.81848324, 0.81848096, 0.81847737, 0.81847369, 0.81847083,
0.81846829, 0.81846601, 0.81846364, 0.81846065, 0.81845897,
0.81845727, 0.8184584 , 0.81845481, 0.81845096, 0.81844714,
0.81844336, 0.81844286, 0.81844313, 0.8184418 , 0.81843787,
0.81843385, 0.81843244, 0.81843006, 0.81842643, 0.81842301,
0.81842012, 0.8184164 , 0.81841261, 0.81841113, 0.81841123,
0.81841167, 0.81840845, 0.81840571, 0.81840289, 0.81840006,
0.81839648, 0.81839794, 0.81840175, 0.81840656, 0.81840632,
0.8184053 , 0.8184015 , 0.81841222, 0.8184118 , 0.81841594,
0.81844041, 0.81845594, 0.81846008, 0.81847317, 0.81848535,
0.81848382, 0.81848619, 0.81849768, 0.81851117, 0.81852665,
0.81853106, 0.81853637, 0.81854015, 0.81854653, 0.81855083]),
'minmse': 107.03801518643529,
'maxmse': 1046.4212142210524,
'noise_level': 128,
'original_image': 'samp6'}}