Tips and Tricks
Why does synthesis take so long?
Synthesis can take a while to run, especially if you are trying to synthesize a large image or using a complicated model. The following might help:
Reducing the amount of time your model’s forward pass takes is the easiest way to reduce the overall duration of synthesis, as the forward pass is called many, many times over synthesis. Try using python’s built-in profiling tools to check which part of your model’s forward pass is taking the longest, and try to make those parts more efficient. Jupyter also has nice profiling tools. For example, if you have for loops in your code, try and replace them with matrix operations and einsum.
If you have access to a GPU, use it! If your inputs are on the GPU before initializing the synthesis methods, the synthesis methods will also make use of the GPU. You can also move the
plenoptic
’s synthesis methods and models over to the GPU after initialization using the.to()
method.
Optimization is hard
You should double-check whether synthesis has successfully completed before interpreting the outputs or using them in any experiments. This is not necessary for eigendistortions (see its notebook for more details on why), but is necessary for all the iterative optimization methods.
For metamers, this means double-checking that the difference between the model representation of the metamer and the target image is small enough. If your model’s representation is multi-scale, trying coarse-to-fine optimization may help (see notebook for details).
For MAD competition, this means double-checking that the reference metric is constant and that the optimized metric has converged at a lower or higher value (depending on the value of
synthesis_target
); useplenoptic.synthesize.mad_competition.plot_synthesis_status()
to visualize these values. You will likely need to spend time trying out different values for themetric_tradeoff_lambda
argument set during initialization to achieve this.For geodesics, check that your geodesic’s path energy is small enough and that the deviation from a straight line in representational space is minimal (use
plenoptic.synthesize.geodesic.plot_deviation_from_line()
)
For all of the above, if synthesis has not found a good solution, you may need
to run synthesis longer, use a learning-rate scheduler, change the learning
rate, or try different optimizers. Each method’s objective_function
method
captures the value that we are trying to minimize, but may contain other values
(such as the penalty on allowed range values).
Additionally, it may be helpful to visualize the progression of synthesis, using
each synthesis method’s animate
or plot_synthesis_status
helper
functions (e.g., plenoptic.synthesize.metamer.plot_synthesis_status()
).
None of the existing synthesis methods meet my needs
plenoptic
provides four synthesis methods, but you may find you wish to do
something slightly outside the capabilities of the existing methods. There are
generally two ways to do this: by tweaking your model or by extending one of the
methods.
See the Portilla-Simoncelli texture model notebook for examples on how to get different metamer results by tweaking your model or extending the
Metamer
class.The coarse-to-fine optimization, discussed in the metamer notebook, is an example of changing optimization by extending the
Metamer
class.The Synthesis extensions notebook contains a discussion focused on this as well.
If you extend a method successfully or would like help making it work, please let us know by posting a discussion!