plenoptic.synthesize package
Submodules
plenoptic.synthesize.autodiff module
- plenoptic.synthesize.autodiff.jacobian(y, x)[source]
Explicitly compute the full Jacobian matrix. N.B. This is only recommended for small input sizes (e.g. <100x100 image)
- Parameters:
y (
Tensor
) – Model output with gradient attachedx (
Tensor
) – Tensor with gradient function model input with gradient attached
- Returns:
Jacobian matrix with
torch.Size([len(y), len(x)])
- Return type:
J
- plenoptic.synthesize.autodiff.jacobian_vector_product(y, x, V, dummy_vec=None)[source]
Compute Jacobian Vector Product: \(\text{jvp} = (\partial y/\partial x) v\)
Forward Mode Auto-Differentiation (
Rop
in Theano). PyTorch does not natively support this operation; this function essentially calls backward mode autodiff twice, as described in [1].See
vector_jacobian_product()
docstring on why we and pass arguments forretain_graph
andcreate_graph
.- Parameters:
y (
Tensor
) – Model output with gradient attached, shape is torch.Size([m, 1])x (
Tensor
) – Model input with gradient attached, shape is torch.Size([n, 1]), i.e. same dim as input tensorV (
Tensor
) – Directions in which to compute product, shape is torch.Size([n, k]) where k is number of vectors to computedummy_vec (
Tensor
) – Vector with which to do jvp trick [1]. If argument exists, then use some pre-allocated, cached vector, otherwise create a new one and move to device in this method.
- Returns:
Jacobian-vector product, torch.Size([n, k])
- Return type:
Jv
Notes
- plenoptic.synthesize.autodiff.vector_jacobian_product(y, x, U, retain_graph=True, create_graph=True, detach=False)[source]
Compute vector Jacobian product: \(\text{vjp} = u^T(\partial y/\partial x)\)
Backward Mode Auto-Differentiation (Lop in Theano)
Note on efficiency: When this function is used in the context of power iteration for computing eigenvectors, the vector output will be repeatedly fed back into
vector_jacobian_product()
andjacobian_vector_product()
. To prevent the accumulation of gradient history in this vector (especially on GPU), we need to ensure the computation graph is not kept in memory after each iteration. We can do this by detaching the output, as well as carefully specifying where/when to retain the created graph.- Parameters:
y (
Tensor
) – Output with gradient attached,torch.Size([m, 1])
.x (
Tensor
) – Input with gradient attached,torch.Size([n, 1])
.U (
Tensor
) – Direction, shape istorch.Size([m, k])
, i.e. same dim as output tensor.retain_graph (
bool
) – Whether or not to keep graph after doing onevector_jacobian_product()
. Must be set to True if k>1.create_graph (
bool
) – Whether or not to create computational graph. Usually should be set to True unless you’re reusing the graph like in the second step ofjacobian_vector_product()
.detach (
bool
) – As withcreate_graph
, only necessary to be True when reusing the output like we do in the 2nd step ofjacobian_vector_product()
.
- Returns:
vector-Jacobian product,
torch.Size([m, k])
.- Return type:
vJ
plenoptic.synthesize.eigendistortion module
- class plenoptic.synthesize.eigendistortion.Eigendistortion(image, model)[source]
Bases:
Synthesis
Synthesis object to compute eigendistortions induced by a model on a given input image.
- Parameters:
image (
Tensor
) – Image, torch.Size(batch=1, channel, height, width). We currently do not support batches of images, as each image requires its own optimization.model (
Module
) – Torch model with defined forward and backward operations.
- batch_size
- Type:
int
- n_channels
- Type:
int
- im_height
- Type:
int
- im_width
- Type:
int
- jacobian
Is only set when
synthesize()
is run withmethod='exact'
. Default toNone
.- Type:
Tensor
- eigendistortions
Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue, with Size((n_distortions, n_channels, im_height, im_width)).
- Type:
Tensor
- eigenvalues
Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.
- Type:
Tensor
- eigenindex
Index of each eigenvector/eigenvalue.
- Type:
listlike
Notes
This is a method for comparing image representations in terms of their ability to explain perceptual sensitivity in humans. It estimates eigenvectors of the FIM. A model, \(y = f(x)\), is a deterministic (and differentiable) mapping from the input pixels \(x \in \mathbb{R}^n\) to a mean output response vector \(y\in \mathbb{ R}^m\), where we assume additive white Gaussian noise in the response space. The Jacobian matrix at x is:
\(J(x) = J = dydx\), \(J\in\mathbb{R}^{m \times n}\) (ie. output_dim x input_dim)
is the matrix of all first-order partial derivatives of the vector-valued function f. The Fisher Information Matrix (FIM) at x, under white Gaussian noise in the response space, is:
\(F = J^T J\)
It is a quadratic approximation of the discriminability of distortions relative to \(x\).
References
[1]Berardino, A., Laparra, V., Ballé, J. and Simoncelli, E., 2017. Eigen-distortions of hierarchical representations. In Advances in neural information processing systems (pp. 3530-3539). http://www.cns.nyu.edu/pub/lcv/berardino17c-final.pdf http://www.cns.nyu.edu/~lcv/eigendistortions/
- Attributes:
eigendistortions
Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue.
eigenindex
Index of each eigenvector/eigenvalue.
eigenvalues
Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.
- image
jacobian
Is only set when
synthesize()
is run withmethod='exact'
.- model
Methods
Calls autodiff.jacobian and returns jacobian.
load
(file_path[, map_location])Load all relevant stuff from a .pt file.
save
(file_path)Save all relevant variables in .pt file.
synthesize
([method, k, max_iter, p, q, ...])Compute eigendistortions of Fisher Information Matrix with given input image.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
- compute_jacobian()[source]
Calls autodiff.jacobian and returns jacobian. Will throw error if input too big.
- Returns:
Jacobian of representation wrt input.
- Return type:
J
- property eigendistortions
Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue.
- property eigenindex
Index of each eigenvector/eigenvalue.
- property eigenvalues
Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.
- property image
- property jacobian
Is only set when
synthesize()
is run withmethod='exact'
. Default toNone
.
- load(file_path, map_location=None, **pickle_load_args)[source]
Load all relevant stuff from a .pt file.
This should be called by an initialized
Eigendistortion
object – we will ensure thatimage
andmodel
are identical.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (str) – The path to load the synthesis object from
map_location (str, optional) – map_location argument to pass to
torch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
Examples
>>> geo = po.synth.Geodesic(img_a, img_b, model) >>> geo.synthesize(max_iter=10, store_progress=True) >>> geo.save('geo.pt') >>> geo_copy = po.synth.Geodesic(img_a, img_b, model) >>> geo_copy.load('geo.pt')
Note that you must create a new instance of the Synthesis object and then load.
- property model
- save(file_path)[source]
Save all relevant variables in .pt file.
See
load
docstring for an example of use.- Parameters:
file_path (str) – The path to save the Eigendistortion object to
- synthesize(method='power', k=1, max_iter=1000, p=5, q=2, stop_criterion=1e-07)[source]
Compute eigendistortions of Fisher Information Matrix with given input image.
- Parameters:
method (
Literal
['exact'
,'power'
,'randomized_svd'
]) – Eigensolver method. ‘exact’ tries to do eigendecomposition directly ( not recommended for very large inputs). ‘power’ (default) uses the power method to compute first and last eigendistortions, with maximum number of iterations dictated by n_steps. ‘randomized_svd’ uses randomized SVD to approximate the top k eigendistortions and their corresponding eigenvalues.k (
int
) – How many vectors to return using block power method or svd.max_iter (
int
) – Maximum number of steps to run formethod='power'
in eigenvalue computation. Ignored for other methods.p (
int
) – Oversampling parameter for randomized SVD. k+p vectors will be sampled, and k will be returned. See docstring of_synthesize_randomized_svd
for more details including algorithm reference.q (
int
) – Matrix power parameter for randomized SVD. This is an effective trick for the algorithm to converge to the correct eigenvectors when the eigenspectrum does not decay quickly. See_synthesize_randomized_svd
for more details including algorithm reference.stop_criterion (
float
) – Used ifmethod='power'
to check for convergence. If the L2-norm of the eigenvalues has changed by less than this value from one iteration to the next, we terminate synthesis.
- to(*args, **kwargs)[source]
Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.See below for examples.
Note
This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- device (
- plenoptic.synthesize.eigendistortion.display_eigendistortion(eigendistortion, eigenindex=0, alpha=5.0, process_image=<function <lambda>>, ax=None, plot_complex='rectangular', **kwargs)[source]
Displays specified eigendistortion added to the image.
If image or eigendistortions have 3 channels, then it is assumed to be a color image and it is converted to grayscale. This is merely for display convenience and may change in the future.
- Parameters:
eigendistortion (
Eigendistortion
) – Eigendistortion object whose synthesized eigendistortion we want to displayeigenindex (
int
) – Index of eigendistortion to plot. E.g. If there are 10 eigenvectors, 0 will index the first one, and -1 or 9 will index the last one.alpha (
float
) – Amount by which to scale eigendistortion for image + (alpha * eigendistortion) for display.process_image (
Callable
[[Tensor
],Tensor
]) – A function to process the image+alpha*distortion before clamping between 0,1. E.g. multiplying by the stdev ImageNet then adding the mean of ImageNet to undo image preprocessing.ax (
Optional
[axis
]) – Axis handle on which to plot.plot_complex (
str
) – Parameter forplenoptic.imshow()
determining how to handle complex values. Defaults to ‘rectangular’, which plots real and complex components as separate images. Can also be ‘polar’ or ‘logpolar’; see that method’s docstring for details.kwargs – Additional arguments for
po.imshow()
.
- Returns:
matplotlib Figure handle returned by plenoptic.imshow()
- Return type:
fig
- plenoptic.synthesize.eigendistortion.fisher_info_matrix_eigenvalue(y, x, v, dummy_vec=None)[source]
Compute the eigenvalues of the Fisher Information Matrix corresponding to eigenvectors in v \(\lambda= v^T F v\)
- Return type:
Tensor
- plenoptic.synthesize.eigendistortion.fisher_info_matrix_vector_product(y, x, v, dummy_vec)[source]
Compute Fisher Information Matrix Vector Product: \(Fv\)
- Parameters:
y (
Tensor
) – Output tensor with gradient attachedx (
Tensor
) – Input tensor with gradient attachedv (
Tensor
) – The vectors with which to compute Fisher vector productsdummy_vec (
Tensor
) – Dummy vector for Jacobian vector product trick
- Returns:
Vector, Fisher vector product
- Return type:
Fv
Notes
Under white Gaussian noise assumption, \(F\) is matrix multiplication of Jacobian transpose and Jacobian: \(F = J^T J\). Hence: \(Fv = J^T (Jv)\)
plenoptic.synthesize.geodesic module
- class plenoptic.synthesize.geodesic.Geodesic(image_a, image_b, model, n_steps=10, initial_sequence='straight', range_penalty_lambda=0.1, allowed_range=(0, 1))[source]
Bases:
OptimizedSynthesis
Synthesize an approximate geodesic between two images according to a model.
This method can be used to visualize and refine the invariances of a model’s representation as described in [1].
NOTE: This synthesis method is still under construction. It will run, but it might not find the most informative geodesic.
- Parameters:
image_a (
Tensor
) – Start and stop anchor points of the geodesic, of shape (1, channel, height, width).image_b (
Tensor
) – Start and stop anchor points of the geodesic, of shape (1, channel, height, width).model (
Module
) – an analysis model that computes representations on signals like image_a.n_steps (
int
) – the number of steps (i.e., transitions) in the trajectory between the two anchor points.initial_sequence (
Literal
['straight'
,'bridge'
]) – initialize the geodesic with pixel linear interpolation ('straight'
), or with a brownian bridge between the two anchors ('bridge'
).range_penalty_lambda (
float
) – strength of the regularizer that enforces the allowed_range. Must be non-negative.allowed_range (
Tuple
[float
,float
]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.
- geodesic
the synthesized sequence of images between the two anchor points that minimizes representation path energy, of shape
(n_steps+1, channel, height, width)
. It starts with image_a and ends with image_b.- Type:
Tensor
- pixelfade
the straight interpolation between the two anchor points, used as reference
- Type:
Tensor
- losses
A list of our loss over iterations.
- Type:
Tensor
- gradient_norm
A list of the gradient’s L2 norm over iterations.
- Type:
list
- pixel_change_norm
A list containing the L2 norm of the pixel change over iterations (
pixel_change_norm[i]
is the pixel change norm ingeodesic
between iterationsi
andi-1
).- Type:
list
- step_energy
step lengths in representation space, stored along the optimization process.
- Type:
Tensor
- dev_from_line
deviation of the representation to the straight line interpolation, measures distance from straight line and distance along straight line, stored along the optimization process
- Type:
Tensor
Notes
Manifold prior hypothesis: natural images form a manifold 𝑀ˣ embedded in signal space (ℝⁿ), a model warps this manifold to another manifold 𝑀ʸ embedded in representation space (ℝᵐ), and thereby induces a different local metric.
This method computes an approximate geodesics by solving an optimization problem: it minimizes the path energy (aka. action functional), which has the same minimum as minimizing path length and by Cauchy-Schwarz, reaches it with constant-speed minimizing geodesic
Caveat: depending on the geometry of the manifold, geodesics between two anchor points may not be unique and may depend on the initialization.
References
[1]Geodesics of learned representations O J Hénaff and E P Simoncelli Published in Int’l Conf on Learning Representations (ICLR), May 2016. http://www.cns.nyu.edu/~lcv/pubs/makeAbs.php?loc=Henaff16b
- Attributes:
- allowed_range
dev_from_line
Deviation of representation each from of
self.geodesic
from a straight line.- geodesic
gradient_norm
Synthesis gradient’s L2 norm over iterations.
- image_a
- image_b
losses
Synthesis loss over iterations.
- model
- optimizer
pixel_change_norm
L2 norm change in pixel values over iterations.
- range_penalty_lambda
step_energy
Squared L2 norm of transition between geodesic frames in representation space.
- store_progress
Methods
calculate_jerkiness
([geodesic])Compute the alignment of representation's acceleration to model local curvature.
load
(file_path[, map_location])Load all relevant stuff from a .pt file.
objective_function
([geodesic])Compute geodesic synthesis loss.
save
(file_path)Save all relevant variables in .pt file.
synthesize
([max_iter, optimizer, ...])Synthesize a geodesic via optimization.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
- calculate_jerkiness(geodesic=None)[source]
Compute the alignment of representation’s acceleration to model local curvature.
This is the first order optimality condition for a geodesic, and can be used to assess the validity of the solution obtained by optimization.
- Parameters:
geodesic (
Optional
[Tensor
]) – Geodesic to check. If None, we useself.geodesic
. Must have a gradient attached.- Return type:
jerkiness
- property dev_from_line
Deviation of representation each from of
self.geodesic
from a straight line.Has shape
(np.ceil(synth_iter/store_progress), n_steps+1, 2)
, wheresynth_iter
is the number of iterations of synthesis that have happened. For final dimension, the first element is the Euclidean distance along the straight line and the second is the Euclidean distance to the line.
- property geodesic
- property image_a
- property image_b
- load(file_path, map_location=None, **pickle_load_args)[source]
Load all relevant stuff from a .pt file.
This should be called by an initialized
Geodesic
object – we will ensure thatimage_a
,image_b
,model
,n_steps
,initial_sequence
,range_penalty_lambda
,allowed_range
, andpixelfade
are all identical.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (str) – The path to load the synthesis object from
map_location (str, optional) – map_location argument to pass to
torch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
Examples
>>> geo = po.synth.Geodesic(img_a, img_b, model) >>> geo.synthesize(max_iter=10, store_progress=True) >>> geo.save('geo.pt') >>> geo_copy = po.synth.Geodesic(img_a, img_b, model) >>> geo_copy.load('geo.pt')
Note that you must create a new instance of the Synthesis object and then load.
- property model
- objective_function(geodesic=None)[source]
Compute geodesic synthesis loss.
This is the path energy (i.e., squared L2 norm of each step) of the geodesic’s model representation, with the weighted range penalty.
Additionally, caches:
self._geodesic_representation = self.model(geodesic)
self._most_recent_step_energy = self._calculate_step_energy(self._geodesic_representation)
These are cached because we might store them (if
self.store_progress is True
) and don’t want to recalcualte them- Parameters:
geodesic (
Optional
[Tensor
]) – Geodesic to check. If None, we useself.geodesic
.- Return type:
loss
- save(file_path)[source]
Save all relevant variables in .pt file.
See
load
docstring for an example of use.- Parameters:
file_path (str) – The path to save the Geodesic object to
- property step_energy
Squared L2 norm of transition between geodesic frames in representation space.
Has shape
(np.ceil(synth_iter/store_progress), n_steps)
, wheresynth_iter
is the number of iterations of synthesis that have happened.
- synthesize(max_iter=1000, optimizer=None, store_progress=False, stop_criterion=None, stop_iters_to_check=50)[source]
Synthesize a geodesic via optimization.
- Parameters:
max_iter (
int
) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).optimizer (
Optional
[Optimizer
]) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.001, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.store_progress (
Union
[bool
,int
]) – Whether we should store the step energy and deviation of the representation from a straight line. If False, we don’t save anything. If True, we save every iteration. If an int, we save everystore_progress
iterations (note then that 0 is the same as False and 1 the same as True).stop_criterion (
Optional
[float
]) – If pixel_change_norm (i.e., the norm of the difference inself.geodesic
from one iteration to the next) over the paststop_iters_to_check
has been less thanstop_criterion
, we terminate synthesis. If None, we pick a default value based on the norm ofself.pixelfade
.stop_iters_to_check (
int
) – How many iterations back to check in order to see if pixel_change_norm has stopped decreasing (forstop_criterion
).
- to(*args, **kwargs)[source]
Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.See below for examples.
Note
This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- device (
- plenoptic.synthesize.geodesic.plot_deviation_from_line(geodesic, natural_video=None, ax=None)[source]
Visual diagnostic of geodesic linearity in representation space.
This plot illustrates the deviation from the straight line connecting the representations of a pair of images, for different paths in representation space.
- Parameters:
geodesic (
Geodesic
) – Geodesic object to visualize.natural_video (
Optional
[Tensor
]) – Natural video that bridges the anchor points, for comparison.ax (
Optional
[Axes
]) – If not None, the axis to plot this representation on. If None, we callplt.gca()
- Returns:
Axes containing the plot
- Return type:
ax
Notes
Axes are in the same units, normalized by the distance separating the end point representations.
Knots along each curve indicate samples used to compute the path.
When the representation is non-linear it may not be feasible for the geodesic to be straight (for example if the representation is normalized, all paths are constrained to live on a hypershpere). Nevertheless, if the representation is able to linearize the transformation between the anchor images, then we expect that both the ground truth natural video sequence and the geodesic will deviate from straight line similarly. By contrast the pixel-based interpolation will deviate significantly more from a straight line.
- plenoptic.synthesize.geodesic.plot_loss(geodesic, ax=None, **kwargs)[source]
Plot synthesis loss.
- Parameters:
geodesic (
Geodesic
) – Geodesic object whose synthesis loss we want to plot.ax (
Optional
[Axes
]) – If not None, the axis to plot this representation on. If None, we callplt.gca()
kwargs – passed to plt.semilogy
- Returns:
Axes containing the plot.
- Return type:
ax
plenoptic.synthesize.mad_competition module
Run MAD Competition.
- class plenoptic.synthesize.mad_competition.MADCompetition(image, optimized_metric, reference_metric, minmax, initial_noise=0.1, metric_tradeoff_lambda=None, range_penalty_lambda=0.1, allowed_range=(0, 1))[source]
Bases:
OptimizedSynthesis
Synthesize a single maximally-differentiating image for two metrics.
Following the basic idea in [1], this class synthesizes a maximally-differentiating image for two given metrics, based on a given image. We start by adding noise to this image and then iteratively adjusting its pixels so as to either minimize or maximize
optimized_metric
while holding the value ofreference_metric
constant.MADCompetiton accepts two metrics as its input. These should be callables that take two images and return a single number, and that number should be 0 if and only if the two images are identical (thus, the larger the number, the more different the two images).
Note that a full set of images MAD Competition images consists of two pairs: a maximal and a minimal image for each metric. A single instantiation of
MADCompetition
will generate one of these four images.- Parameters:
image (
Tensor
) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.optimized_metric (
Union
[Module
,Callable
[[Tensor
,Tensor
],Tensor
]]) – The metric whose value you wish to minimize or maximize, which takes two tensors and returns a scalar. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the MADCompetition object (i.e., it must be one of our built-in functions or defined using a def statement)reference_metric (
Union
[Module
,Callable
[[Tensor
,Tensor
],Tensor
]]) – The metric whose value you wish to keep fixed, which takes two tensors and returns a scalar. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the MADCompetition object (i.e., it must be one of our built-in functions or defined using a def statement)minmax (
Literal
['min'
,'max'
]) – Whether you wish to minimize or maximizeoptimized_metric
.initial_noise (
float
) – Standard deviation of the Gaussian noise used to initializemad_image
fromimage
.metric_tradeoff_lambda (
Optional
[float
]) – Lambda to multiply byreference_metric
loss and add tooptimized_metric
loss. IfNone
, we pick a value so the two initial losses are approximately equal in magnitude.range_penalty_lambda (
float
) – Lambda to multiply by range penalty and add to loss.allowable_range – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.
- mad_image
The Maximally-Differentiating Image. This may be unfinished depending on how many iterations we’ve run for.
- Type:
torch.Tensor
- initial_image
The initial
mad_image
, which we obtain by adding Gaussian noise toimage
.- Type:
torch.Tensor
- losses
A list of the objective function’s loss over iterations.
- Type:
list
- gradient_norm
A list of the gradient’s L2 norm over iterations.
- Type:
list
- pixel_change_norm
A list containing the L2 norm of the pixel change over iterations (
pixel_change_norm[i]
is the pixel change norm inmad_image
between iterationsi
andi-1
).- Type:
list
- optimized_metric_loss
A list of the
optimized_metric
loss over iterations.- Type:
list
- reference_metric_loss
A list of the
reference_metric
loss over iterations.- Type:
list
- saved_mad_image
Saved
self.mad_image
for later examination.- Type:
torch.Tensor
References
[1]Wang, Z., & Simoncelli, E. P. (2008). Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual discriminability. Journal of Vision, 8(12), 1–13. http://dx.doi.org/10.1167/8.12.8
- Attributes:
- allowed_range
gradient_norm
Synthesis gradient’s L2 norm over iterations.
- image
- initial_image
losses
Synthesis loss over iterations.
- mad_image
- metric_tradeoff_lambda
- minmax
- optimized_metric
- optimized_metric_loss
- optimizer
pixel_change_norm
L2 norm change in pixel values over iterations.
- range_penalty_lambda
- reference_metric
- reference_metric_loss
- saved_mad_image
- store_progress
Methods
load
(file_path[, map_location])Load all relevant stuff from a .pt file.
objective_function
([mad_image, image])Compute the MADCompetition synthesis loss.
save
(file_path)Save all relevant variables in .pt file.
synthesize
([max_iter, optimizer, scheduler, ...])Synthesize a MAD image.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
- property image
- property initial_image
- load(file_path, map_location=None, **pickle_load_args)[source]
Load all relevant stuff from a .pt file.
This should be called by an initialized
MADCompetition
object – we will ensure thatimage
,metric_tradeoff_lambda
,range_penalty_lambda
,allowed_range
,minmax
are all identical, and thatreference_metric
andoptimize_metric
return identical values.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (str) – The path to load the synthesis object from
map_location (str, optional) – map_location argument to pass to
torch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
Examples
>>> mad = po.synth.MADCompetition(img, model) >>> mad.synthesize(max_iter=10, store_progress=True) >>> mad.save('mad.pt') >>> mad_copy = po.synth.MADCompetition(img, model) >>> mad_copy.load('mad.pt')
Note that you must create a new instance of the Synthesis object and then load.
- property mad_image
- property metric_tradeoff_lambda
- property minmax
- objective_function(mad_image=None, image=None)[source]
Compute the MADCompetition synthesis loss.
This computes:
\[\begin{split}t L_1(x, \hat{x}) &+ \lambda_1 [L_2(x, x+\epsilon) - L_2(x, \hat{x})]^2 \\ &+ \lambda_2 \mathcal{B}(\hat{x})\end{split}\]where \(t\) is 1 if
self.minmax
is'min'
and -1 if it’s'max'
, \(L_1\) isself.optimized_metric
, \(L_2\) isself.reference_metric
, \(x\) isself.image
, \(\hat{x}\) isself.mad_image
, \(\epsilon\) is the initial noise, \(\mathcal{B}\) is the quadratic bound penalty, \(\lambda_1\) isself.metric_tradeoff_lambda
and \(\lambda_2\) isself.range_penalty_lambda
.- Parameters:
mad_image (
Optional
[Tensor
]) – Proposedmad_image
, \(\hat{x}\) in the above equation. If None, useself.mad_image
.image (
Optional
[Tensor
]) – Proposedimage
, \(x\) in the above equation. If None, useself.image
.
- Return type:
loss
- property optimized_metric
- property optimized_metric_loss
- property reference_metric
- property reference_metric_loss
- save(file_path)[source]
Save all relevant variables in .pt file.
Note that if store_progress is True, this will probably be very large.
See
load
docstring for an example of use.- Parameters:
file_path (str) – The path to save the MADCompetition object to
- property saved_mad_image
- synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50)[source]
Synthesize a MAD image.
Update the pixels of
initial_image
to maximize or minimize (depending on the value ofminmax
) the value ofoptimized_metric(image, mad_image)
while keeping the value ofreference_metric(image, mad_image)
constant.We run this until either we reach
max_iter
or the change over the paststop_iters_to_check
iterations is less thanstop_criterion
, whichever comes first- Parameters:
max_iter (
int
) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).optimizer (
Optional
[Optimizer
]) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.scheduler (
Optional
[_LRScheduler
]) – The learning rate scheduler to use. If None, we don’t use one.store_progress (
Union
[bool
,int
]) – Whether we should store the representation of the MAD image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save everystore_progress
iterations (note then that 0 is the same as False and 1 the same as True).stop_criterion (
float
) – If the loss over the paststop_iters_to_check
has changed less thanstop_criterion
, we terminate synthesis.stop_iters_to_check (
int
) – How many iterations back to check in order to see if the loss has stopped decreasing (forstop_criterion
).
- to(*args, **kwargs)[source]
Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.See below for examples.
Note
This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- device (
- plenoptic.synthesize.mad_competition.animate(mad, framerate=10, batch_idx=0, channel_idx=None, zoom=None, fig=None, axes_idx={}, figsize=None, included_plots=['display_mad_image', 'plot_loss', 'plot_pixel_values'], width_ratios={})[source]
Animate synthesis progress.
This is essentially the figure produced by
mad.plot_synthesis_status
animated over time, for each stored iteration.We return the matplotlib FuncAnimation object. In order to view it in a Jupyter notebook, use the
plenoptic.tools.display.convert_anim_to_html(anim)
function. In order to save, useanim.save(filename)
(note for this that you’ll need the appropriate writer installed and on your path, e.g., ffmpeg, imagemagick, etc). Either of these will probably take a reasonably long amount of time.- Parameters:
mad (
MADCompetition
) – MADCompetition object whose synthesis we want to animate.framerate (
int
) – How many frames a second to display.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).zoom (
Optional
[float
]) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.fig (
Optional
[Figure
]) – If None, create the figure from scratch. Else, should be an empty figure with enough axes (the expected use here is have same-size movies with different plots).axes_idx (
Dict
[str
,int
]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys:'mad_image', 'loss', 'pixel_values', 'misc'
. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in'misc'
.figsize (
Optional
[Tuple
[float
]]) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)width_ratios (
Dict
[str
,float
]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: [‘display_mad_image’, ‘plot_loss’, ‘plot_pixel_values’] and floats specifying their relative width. Any not included will be assumed to be 1.
- Returns:
The animation object. In order to view, must convert to HTML or save.
- Return type:
anim
Notes
By default, we use the ffmpeg backend, which requires that you have ffmpeg installed and on your path (https://ffmpeg.org/download.html). To use a different, use the matplotlib rcParams: matplotlib.rcParams[‘animation.writer’] = writer, see https://matplotlib.org/stable/api/animation_api.html#writer-classes for more details.
For displaying in a jupyter notebook, ffmpeg appears to be required.
- plenoptic.synthesize.mad_competition.display_mad_image(mad, batch_idx=0, channel_idx=None, zoom=None, iteration=None, ax=None, title='MADCompetition', **kwargs)[source]
Display MAD image.
You can specify what iteration to view by using the
iteration
arg. The default,None
, shows the final one.We use
plenoptic.imshow
to display the synthesized image and attempt to automatically find the most reasonable zoom value. You can override this value using the zoom arg, but remember thatplenoptic.imshow
is opinionated about the size of the resulting image and will throw an Exception if the axis created is not big enough for the selected zoom.- Parameters:
mad (
MADCompetition
) – MADCompetition object whose MAD image we want to display.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we assume image is RGB(A) and show all channels.zoom (
Optional
[float
]) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.title (
str
) – Title of the axis.kwargs – Passed to
plenoptic.imshow
- Returns:
The matplotlib axes containing the plot.
- Return type:
ax
- plenoptic.synthesize.mad_competition.display_mad_image_all(mad_metric1_min, mad_metric2_min, mad_metric1_max, mad_metric2_max, metric1_name=None, metric2_name=None, zoom=1, **kwargs)[source]
Display all MAD Competition images.
To generate a full set of MAD Competition images, you need four instances: one for minimizing and maximizing each metric. This helper function creates a figure to display the full set of images.
In addition to the four MAD Competition images, this also plots the initial image from mad_metric1_min, for comparison.
Note that all four MADCompetition instances must have the same image.
- Parameters:
mad_metric1_min (
MADCompetition
) – MADCompetition object that minimized the first metric.mad_metric2_min (
MADCompetition
) – MADCompetition object that minimized the second metric.mad_metric1_max (
MADCompetition
) – MADCompetition object that maximized the first metric.mad_metric2_max (
MADCompetition
) – MADCompetition object that maximized the second metric.metric1_name (
Optional
[str
]) – Name of the first metric. If None, we use the name of the optimized_metric function from mad_metric1_min.metric2_name (
Optional
[str
]) – Name of the second metric. If None, we use the name of the optimized_metric function from mad_metric2_min.zoom (
Union
[int
,float
]) – Ratio of display pixels to image pixels. See plenoptic.imshow for details.kwargs – Passed to plenoptic.imshow.
- Returns:
Figure containing the images.
- Return type:
fig
- plenoptic.synthesize.mad_competition.plot_loss(mad, iteration=None, axes=None, **kwargs)[source]
Plot metric losses.
Plots
mad.optimized_metric_loss
andmad.reference_metric_loss
on two separate axes, over all iterations. Also plots a red dot atiteration
, to highlight the loss there. Ifiteration=None
, then the dot will be at the final iteration.- Parameters:
mad (
MADCompetition
) – MADCompetition object whose loss we want to plot.iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.axes (
Union
[List
[Axes
],Axes
,None
]) – Pre-existing axes for plot. If a list of axes, must be the two axes to use for this plot. If a single axis, we’ll split it in half horizontally. If None, we callplt.gca()
.kwargs – passed to plt.plot
- Returns:
The matplotlib axes containing the plot.
- Return type:
axes
Notes
We plot
abs(mad.losses)
because if we’re maximizing the synthesis metric, we minimized its negative. By plotting the absolute value, we get them all on the same scale.
- plenoptic.synthesize.mad_competition.plot_loss_all(mad_metric1_min, mad_metric2_min, mad_metric1_max, mad_metric2_max, metric1_name=None, metric2_name=None, metric1_kwargs={'c': 'C0'}, metric2_kwargs={'c': 'C1'}, min_kwargs={'linestyle': '--'}, max_kwargs={'linestyle': '-'}, figsize=(10, 5))[source]
Plot loss for full set of MAD Competiton instances.
To generate a full set of MAD Competition images, you need four instances: one for minimizing and maximizing each metric. This helper function creates a two-axis figure to display the loss for this full set.
Note that all four MADCompetition instances must have the same image.
- Parameters:
mad_metric1_min (
MADCompetition
) – MADCompetition object that minimized the first metric.mad_metric2_min (
MADCompetition
) – MADCompetition object that minimized the second metric.mad_metric1_max (
MADCompetition
) – MADCompetition object that maximized the first metric.mad_metric2_max (
MADCompetition
) – MADCompetition object that maximized the second metric.metric1_name (
Optional
[str
]) – Name of the first metric. If None, we use the name of the optimized_metric function from mad_metric1_min.metric2_name (
Optional
[str
]) – Name of the second metric. If None, we use the name of the optimized_metric function from mad_metric2_min.metric1_kwargs (
Dict
) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where the first metric was being optimized.metric2_kwargs (
Dict
) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where the second metric was being optimized.min_kwargs (
Dict
) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where optimized_metric was being minimized.max_kwargs (
Dict
) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where optimized_metric was being maximized.figsize – Size of the figure we create.
- Returns:
Figure containing the plot.
- Return type:
fig
- plenoptic.synthesize.mad_competition.plot_pixel_values(mad, batch_idx=0, channel_idx=None, iteration=None, ylim=False, ax=None, **kwargs)[source]
Plot histogram of pixel values of reference and MAD images.
As a way to check the distributions of pixel intensities and see if there’s any values outside the allowed range
- Parameters:
mad (
MADCompetition
) – MADCompetition object with the images whose pixel values we want to compare.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) images).iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ylim (
Union
[Tuple
[float
],Literal
[False
]]) – if tuple, the ylimit to set for this axis. If False, we leave it untouchedax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.kwargs – passed to plt.hist
- Returns:
Creates axes.
- Return type:
ax
- plenoptic.synthesize.mad_competition.plot_synthesis_status(mad, batch_idx=0, channel_idx=None, iteration=None, vrange='indep1', zoom=None, fig=None, axes_idx={}, figsize=None, included_plots=['display_mad_image', 'plot_loss', 'plot_pixel_values'], width_ratios={})[source]
Make a plot showing synthesis status.
We create several subplots to analyze this. By default, we create two subplots on a new figure: the first one contains the MAD image and the second contains the loss.
There is an optional additional plot: pixel_values, a histogram of pixel values of the synthesized and target images.
All of these (including the default plots) can be toggled using their corresponding boolean flags, and can be created separately using the method with the name plot_{flag}.
- Parameters:
mad (
MADCompetition
) – MADCompetition object whose status we want to plot.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.vrange (
Union
[Tuple
[float
],str
]) – The vrange option to pass todisplay_mad_image()
. See docstring ofimshow
for possible values.zoom (
Optional
[float
]) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.fig (
Optional
[Figure
]) – if None, we create a new figure. otherwise we assume this is an empty figure that has the appropriate size and number of subplotsaxes_idx (
Dict
[str
,int
]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys:'mad_image', 'loss', 'pixel_values', 'misc'
. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in'misc'
.figsize (
Optional
[Tuple
[float
]]) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)included_plots (
List
[str
]) – Which plots to include. Must be some subset of'display_mad_image', 'plot_loss', 'plot_pixel_values'
.width_ratios (
Dict
[str
,float
]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: [‘display_mad_image’, ‘plot_loss’, ‘plot_pixel_values’] and floats specifying their relative width. Any not included will be assumed to be 1.
- Return type:
Tuple
[Figure
,Dict
[str
,int
]]- Returns:
fig – The figure containing this plot
axes_idx – Dictionary giving index of each plot.
plenoptic.synthesize.metamer module
Synthesize model metamers.
- class plenoptic.synthesize.metamer.Metamer(image, model, loss_function=<function mse>, range_penalty_lambda=0.1, allowed_range=(0, 1), initial_image=None)[source]
Bases:
OptimizedSynthesis
Synthesize metamers for image-computable differentiable models.
Following the basic idea in [1], this class creates a metamer for a given model on a given image. We start with
initial_image
and iteratively adjust the pixel values so as to match the representation of themetamer
andimage
.All
saved_
attributes are initialized as empty lists and will be non-empty if thestore_progress
arg tosynthesize()
is notFalse
. They will be appended to on every iteration ifstore_progress=True
or everystore_progress
iterations if it’s anint
.- Parameters:
image (
Tensor
) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.model (
Module
) – A visual model, see Metamer notebook for more detailsloss_function (
Callable
[[Tensor
,Tensor
],Tensor
]) – the loss function to use to compare the representations of the models in order to determine their loss. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the Metamer object (i.e., it must be one of our built-in functions or defined using a def statement)range_penalty_lambda (
float
) – strength of the regularizer that enforces the allowed_range. Must be non-negative.allowed_range (
Tuple
[float
,float
]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.initial_image (
Optional
[Tensor
]) – 4d Tensor to initialize our metamer with. If None, will draw a sample of uniform noise withinallowed_range
.
- target_representation
Whatever is returned by
model(image)
, this is what we match in order to create a metamer- Type:
torch.Tensor
- metamer
The metamer. This may be unfinished depending on how many iterations we’ve run for.
- Type:
torch.Tensor
- losses
A list of our loss over iterations.
- Type:
list
- gradient_norm
A list of the gradient’s L2 norm over iterations.
- Type:
list
- pixel_change_norm
A list containing the L2 norm of the pixel change over iterations (
pixel_change_norm[i]
is the pixel change norm inmetamer
between iterationsi
andi-1
).- Type:
list
- saved_metamer
Saved
self.metamer
for later examination.- Type:
torch.Tensor
References
[1]J Portilla and E P Simoncelli. A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients. Int’l Journal of Computer Vision. 40(1):49-71, October, 2000. http://www.cns.nyu.edu/~eero/ABSTRACTS/portilla99-abstract.html http://www.cns.nyu.edu/~lcv/texture/
- Attributes:
- allowed_range
gradient_norm
Synthesis gradient’s L2 norm over iterations.
- image
losses
Synthesis loss over iterations.
- metamer
- model
- optimizer
pixel_change_norm
L2 norm change in pixel values over iterations.
- range_penalty_lambda
- saved_metamer
- store_progress
target_representation
Model representation of
image
, the goal of synthesis is formodel(metamer)
to match this value.
Methods
load
(file_path[, map_location])Load all relevant stuff from a .pt file.
objective_function
([metamer_representation, ...])Compute the metamer synthesis loss.
save
(file_path)Save all relevant variables in .pt file.
synthesize
([max_iter, optimizer, scheduler, ...])Synthesize a metamer.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
- property image
- load(file_path, map_location=None, **pickle_load_args)[source]
Load all relevant stuff from a .pt file.
This should be called by an initialized
Metamer
object – we will ensure thatimage
,target_representation
(and thusmodel
), andloss_function
are all identical.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (str) – The path to load the synthesis object from
map_location (str, optional) – map_location argument to pass to
torch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
Examples
>>> metamer = po.synth.Metamer(img, model) >>> metamer.synthesize(max_iter=10, store_progress=True) >>> metamer.save('metamers.pt') >>> metamer_copy = po.synth.Metamer(img, model) >>> metamer_copy.load('metamers.pt')
Note that you must create a new instance of the Synthesis object and then load.
- property metamer
- property model
- objective_function(metamer_representation=None, target_representation=None)[source]
Compute the metamer synthesis loss.
This calls self.loss_function on
metamer_representation
andtarget_representation
and then adds the weighted range penalty.- Parameters:
metamer_representation (
Optional
[Tensor
]) – Model response tometamer
. If None, we useself.model(self.metamer)
target_representation (
Optional
[Tensor
]) – Model response toimage
. If None, we useself.target_representation
.
- Return type:
loss
- save(file_path)[source]
Save all relevant variables in .pt file.
Note that if store_progress is True, this will probably be very large.
See
load
docstring for an example of use.- Parameters:
file_path (str) – The path to save the metamer object to
- property saved_metamer
- synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50)[source]
Synthesize a metamer.
Update the pixels of
initial_image
until its representation matches that ofimage
.We run this until either we reach
max_iter
or the change over the paststop_iters_to_check
iterations is less thanstop_criterion
, whichever comes first- Parameters:
max_iter (
int
) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).optimizer (
Optional
[Optimizer
]) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.scheduler (
Optional
[_LRScheduler
]) – The learning rate scheduler to use. If None, we don’t use one.store_progress (
Union
[bool
,int
]) – Whether we should store the metamer image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save everystore_progress
iterations (note then that 0 is the same as False and 1 the same as True).stop_criterion (
float
) – If the loss over the paststop_iters_to_check
has changed less thanstop_criterion
, we terminate synthesis.stop_iters_to_check (
int
) – How many iterations back to check in order to see if the loss has stopped decreasing (forstop_criterion
).
- property target_representation
Model representation of
image
, the goal of synthesis is formodel(metamer)
to match this value.
- to(*args, **kwargs)[source]
Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.See below for examples.
Note
This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- device (
- class plenoptic.synthesize.metamer.MetamerCTF(image, model, loss_function=<function mse>, range_penalty_lambda=0.1, allowed_range=(0, 1), initial_image=None, coarse_to_fine='together')[source]
Bases:
Metamer
Synthesize model metamers with coarse-to-fine synthesis.
This is a special case of
Metamer
, which uses the coarse-to-fine synthesis procedure described in [1]_: we start by updating metamer with respect to only a subset of the model’s representation (generally, that which corresponds to the lowest spatial frequencies), and changing which subset we consider over the course of synthesis. This is similar to optimizing with a blurred version of the objective function and gradually adding in finer details. It improves synthesis performance for some models.- Parameters:
image (
Tensor
) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.model (
Module
) – A visual model, see Metamer notebook for more detailsloss_function (
Callable
[[Tensor
,Tensor
],Tensor
]) – the loss function to use to compare the representations of the models in order to determine their loss. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the Metamer object (i.e., it must be one of our built-in functions or defined using a def statement)range_penalty_lambda (
float
) – strength of the regularizer that enforces the allowed_range. Must be non-negative.allowed_range (
Tuple
[float
,float
]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.initial_image (
Optional
[Tensor
]) – 4d Tensor to initialize our metamer with. If None, will draw a sample of uniform noise withinallowed_range
.coarse_to_fine (
Literal
['together'
,'separate'
]) –‘together’: start with the coarsest scale, then gradually add each finer scale.
’separate’: compute the gradient with respect to each scale separately (ignoring the others), then with respect to all of them at the end.
(see
Metamer
tutorial for more details).
- target_representation
Whatever is returned by
model(image)
, this is what we match in order to create a metamer- Type:
torch.Tensor
- metamer
The metamer. This may be unfinished depending on how many iterations we’ve run for.
- Type:
torch.Tensor
- losses
A list of our loss over iterations.
- Type:
list
- gradient_norm
A list of the gradient’s L2 norm over iterations.
- Type:
list
- pixel_change_norm
A list containing the L2 norm of the pixel change over iterations (
pixel_change_norm[i]
is the pixel change norm inmetamer
between iterationsi
andi-1
).- Type:
list
- saved_metamer
Saved
self.metamer
for later examination.- Type:
torch.Tensor
- scales
The list of scales in optimization order (i.e., from coarse to fine). Will be modified during the course of optimization.
- Type:
list or None
- scales_loss
The scale-specific loss at each iteration
- Type:
list or None
- scales_timing
Keys are the values found in
scales
, values are lists, specifying the iteration where we started and stopped optimizing this scale.- Type:
dict or None
- scales_finished
List of scales that we’ve finished optimizing.
- Type:
list or None
- Attributes:
- allowed_range
- coarse_to_fine
gradient_norm
Synthesis gradient’s L2 norm over iterations.
- image
losses
Synthesis loss over iterations.
- metamer
- model
- optimizer
pixel_change_norm
L2 norm change in pixel values over iterations.
- range_penalty_lambda
- saved_metamer
- scales
- scales_finished
- scales_loss
- scales_timing
- store_progress
target_representation
Model representation of
image
, the goal of synthesis is formodel(metamer)
to match this value.
Methods
load
(file_path[, map_location])Load all relevant stuff from a .pt file.
objective_function
([metamer_representation, ...])Compute the metamer synthesis loss.
save
(file_path)Save all relevant variables in .pt file.
synthesize
([max_iter, optimizer, scheduler, ...])Synthesize a metamer.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
- property coarse_to_fine
- load(file_path, map_location=None, **pickle_load_args)[source]
Load all relevant stuff from a .pt file.
This should be called by an initialized
Metamer
object – we will ensure thatimage
,target_representation
(and thusmodel
), andloss_function
are all identical.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (str) – The path to load the synthesis object from
map_location (str, optional) – map_location argument to pass to
torch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
Examples
>>> metamer = po.synth.Metamer(img, model) >>> metamer.synthesize(max_iter=10, store_progress=True) >>> metamer.save('metamers.pt') >>> metamer_copy = po.synth.Metamer(img, model) >>> metamer_copy.load('metamers.pt')
Note that you must create a new instance of the Synthesis object and then load.
- property scales
- property scales_finished
- property scales_loss
- property scales_timing
- synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50, change_scale_criterion=0.01, ctf_iters_to_check=50)[source]
Synthesize a metamer.
Update the pixels of
initial_image
until its representation matches that ofimage
.We run this until either we reach
max_iter
or the change over the paststop_iters_to_check
iterations is less thanstop_criterion
, whichever comes first- Parameters:
max_iter (
int
) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).optimizer (
Optional
[Optimizer
]) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.scheduler (
Optional
[_LRScheduler
]) – The learning rate scheduler to use. If None, we don’t use one.store_progress (
Union
[bool
,int
]) – Whether we should store the metamer image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save everystore_progress
iterations (note then that 0 is the same as False and 1 the same as True).stop_criterion (
float
) – If the loss over the paststop_iters_to_check
has changed less thanstop_criterion
, we terminate synthesis.stop_iters_to_check (
int
) – How many iterations back to check in order to see if the loss has stopped decreasing (forstop_criterion
).change_scale_criterion (
Optional
[float
]) – Scale-specific analogue ofchange_scale_criterion
: we consider a given scale finished (and move onto the next) if the loss has changed less than this in the pastctf_iters_to_check
iterations. IfNone
, we’ll change scales as soon as we’ve spentctf_iters_to_check
on a given scalectf_iters_to_check (
int
) – Scale-specific analogue ofstop_iters_to_check
: how many iterations back in order to check in order to see if we should switch scales.
- plenoptic.synthesize.metamer.animate(metamer, framerate=10, batch_idx=0, channel_idx=None, ylim=None, vrange=(0, 1), zoom=None, plot_representation_error_as_rgb=False, fig=None, axes_idx={}, figsize=None, included_plots=['display_metamer', 'plot_loss', 'plot_representation_error'], width_ratios={})[source]
Animate synthesis progress.
This is essentially the figure produced by
metamer.plot_synthesis_status
animated over time, for each stored iteration.We return the matplotlib FuncAnimation object. In order to view it in a Jupyter notebook, use the
plenoptic.tools.display.convert_anim_to_html(anim)
function. In order to save, useanim.save(filename)
(note for this that you’ll need the appropriate writer installed and on your path, e.g., ffmpeg, imagemagick, etc). Either of these will probably take a reasonably long amount of time.- Parameters:
metamer (
Metamer
) – Metamer object whose synthesis we want to animate.framerate (
int
) – How many frames a second to display.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).ylim (
Union
[str
,None
,Tuple
[float
,float
],Literal
[False
]]) –The y-limits of the representation_error plot:
If a tuple, then this is the ylim of all plots
If None, then all plots have the same limits, all symmetric about 0 with a limit of
np.abs(representation_error).max()
(for the initial representation_error)If False, don’t modify limits.
If a string, must be ‘rescale’ or of the form ‘rescaleN’, where N can be any integer. If ‘rescaleN’, we rescale the limits every N frames (we rescale as if ylim = None). If ‘rescale’, then we do this 10 times over the course of the animation
vrange (
Union
[Tuple
[float
,float
],str
]) – The vrange option to pass todisplay_metamer()
. See docstring ofimshow
for possible values.zoom (
Optional
[float
]) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.plot_representation_error_as_rgb (
bool
) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the representation doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details. since plot_synthesis_status normally sets it up for usfig (
Optional
[Figure
]) – If None, create the figure from scratch. Else, should be an empty figure with enough axes (the expected use here is have same-size movies with different plots).axes_idx (
Dict
[str
,int
]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys:'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values', 'misc'
. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in'misc'
.figsize (
Optional
[Tuple
[float
,float
]]) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)included_plots (
List
[str
]) – Which plots to include. Must be some subset of'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'
.width_ratios (
Dict
[str
,float
]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys:'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'
and floats specifying their relative width. Any not included will be assumed to be 1.
- Returns:
The animation object. In order to view, must convert to HTML or save.
- Return type:
anim
Notes
By default, we use the ffmpeg backend, which requires that you have ffmpeg installed and on your path (https://ffmpeg.org/download.html). To use a different, use the matplotlib rcParams: matplotlib.rcParams[‘animation.writer’] = writer, see https://matplotlib.org/stable/api/animation_api.html#writer-classes for more details.
For displaying in a jupyter notebook, ffmpeg appears to be required.
- plenoptic.synthesize.metamer.display_metamer(metamer, batch_idx=0, channel_idx=None, zoom=None, iteration=None, ax=None, **kwargs)[source]
Display metamer.
You can specify what iteration to view by using the
iteration
arg. The default,None
, shows the final one.We use
plenoptic.imshow
to display the metamer and attempt to automatically find the most reasonable zoom value. You can override this value using the zoom arg, but remember thatplenoptic.imshow
is opinionated about the size of the resulting image and will throw an Exception if the axis created is not big enough for the selected zoom.- Parameters:
metamer (
Metamer
) – Metamer object whose synthesized metamer we want to display.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we assume image is RGB(A) and show all channels.iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.zoom (
Optional
[float
]) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.kwargs – Passed to
plenoptic.imshow
- Returns:
The matplotlib axes containing the plot.
- Return type:
ax
- plenoptic.synthesize.metamer.plot_loss(metamer, iteration=None, ax=None, **kwargs)[source]
Plot synthesis loss with log-scaled y axis.
Plots
metamer.losses
over all iterations. Also plots a red dot atiteration
, to highlight the loss there. Ifiteration=None
, then the dot will be at the final iteration.- Parameters:
metamer (
Metamer
) – Metamer object whose loss we want to plot.iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.kwargs – passed to plt.semilogy
- Returns:
The matplotlib axes containing the plot.
- Return type:
ax
- plenoptic.synthesize.metamer.plot_pixel_values(metamer, batch_idx=0, channel_idx=None, iteration=None, ylim=False, ax=None, **kwargs)[source]
Plot histogram of pixel values of target image and its metamer.
As a way to check the distributions of pixel intensities and see if there’s any values outside the allowed range
- Parameters:
metamer (
Metamer
) – Metamer object with the images whose pixel values we want to compare.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) images).iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ylim (
Union
[Tuple
[float
,float
],Literal
[False
]]) – if tuple, the ylimit to set for this axis. If False, we leave it untouchedax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.kwargs – passed to plt.hist
- Returns:
Created axes.
- Return type:
ax
- plenoptic.synthesize.metamer.plot_representation_error(metamer, batch_idx=0, iteration=None, ylim=None, ax=None, as_rgb=False, **kwargs)[source]
Plot distance ratio showing how close we are to convergence.
We plot
_representation_error(metamer, iteration)
. For more details, seeplenoptic.tools.display.plot_representation
.- Parameters:
metamer (
Metamer
) – Metamer object whose synthesized metamer we want to display.batch_idx (
int
) – Which index to take from the batch dimensioniteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ylim (
Union
[Tuple
[float
,float
],None
,Literal
[False
]]) – Ifylim
isNone
, we sets the axes’ y-limits to be(-y_max, y_max)
, wherey_max=np.abs(data).max()
. If it’sFalse
, we do nothing. If a tuple, we use that range.ax (
Optional
[Axes
]) – Pre-existing axes for plot. If None, we callplt.gca()
.as_rgb (bool, optional) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the response doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details.
kwargs – Passed to
metamer.model.forward
- Returns:
List of created axes
- Return type:
axes
- plenoptic.synthesize.metamer.plot_synthesis_status(metamer, batch_idx=0, channel_idx=None, iteration=None, ylim=None, vrange='indep1', zoom=None, plot_representation_error_as_rgb=False, fig=None, axes_idx={}, figsize=None, included_plots=['display_metamer', 'plot_loss', 'plot_representation_error'], width_ratios={})[source]
Make a plot showing synthesis status.
We create several subplots to analyze this. By default, we create three subplots on a new figure: the first one contains the synthesized metamer, the second contains the loss, and the third contains the representation error.
There is an optional additional plot:
plot_pixel_values
, a histogram of pixel values of the metamer and target image.The plots to include are specified by including their name in the
included_plots
list. All plots can be created separately using the method with the same name.- Parameters:
metamer (
Metamer
) – Metamer object whose status we want to plot.batch_idx (
int
) – Which index to take from the batch dimensionchannel_idx (
Optional
[int
]) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).iteration (
Optional
[int
]) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.ylim (
Union
[Tuple
[float
,float
],None
,Literal
[False
]]) – The ylimit to use for the representation_error plot. We pass this value directly toplot_representation_error
vrange (
Union
[Tuple
[float
,float
],str
]) – The vrange option to pass todisplay_metamer()
. See docstring ofimshow
for possible values.zoom (
Optional
[float
]) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.plot_representation_error_as_rgb (bool, optional) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the response doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details.
fig (
Optional
[Figure
]) – if None, we create a new figure. otherwise we assume this is an empty figure that has the appropriate size and number of subplotsaxes_idx (
Dict
[str
,int
]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys:'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values', 'misc'
. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in'misc'
.figsize (
Optional
[Tuple
[float
,float
]]) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)included_plots (
List
[str
]) – Which plots to include. Must be some subset of'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'
.width_ratios (
Dict
[str
,float
]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys:'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'
and floats specifying their relative width. Any not included will be assumed to be 1.
- Return type:
Tuple
[Figure
,Dict
[str
,int
]]- Returns:
fig – The figure containing this plot
axes_idx – Dictionary giving index of each plot.
plenoptic.synthesize.simple_metamer module
Simple Metamer Class
- class plenoptic.synthesize.simple_metamer.SimpleMetamer(image, model)[source]
Bases:
Synthesis
Simple version of metamer synthesis.
This doesn’t have any of the bells and whistles of the full Metamer class, but does perform basic metamer synthesis: given a target image and a model, synthesize a new image (initialized with uniform noise) that has the same model output.
This is meant as a demonstration of the basic logic of synthesis.
- Parameters:
image (
Tensor
) – A 4d tensor, this is the image whose model representation we wish to match.model (
Module
) – The visual model whose representation we wish to match.
Methods
load
(file_path[, map_location])Load all relevant attributes from a .pt file.
save
(file_path)Save all relevant (non-model) variables in .pt file.
synthesize
([max_iter, optimizer])Synthesize a simple metamer.
to
(*args, **kwargs)Move and/or cast the parameters and buffers.
- load(file_path, map_location=None)[source]
Load all relevant attributes from a .pt file.
Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (
str
) – The path to load the synthesis object from
- save(file_path)[source]
Save all relevant (non-model) variables in .pt file.
- Parameters:
file_path (
str
) – The path to save the SimpleMetamer object to.
- synthesize(max_iter=100, optimizer=None)[source]
Synthesize a simple metamer.
If called multiple times, will continue where we left off.
- Parameters:
max_iter (
int
) – Number of iterations to run synthesis for.optimizer (
Optional
[Optimizer
]) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, we reuse the previous optimizer.
- Returns:
The synthesized metamer
- Return type:
metamer
- to(*args, **kwargs)[source]
Move and/or cast the parameters and buffers.
This can be called as .. function:: to(device=None, dtype=None, non_blocking=False) .. function:: to(dtype, non_blocking=False) .. function:: to(tensor, non_blocking=False) Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. When calling this method to move tensors to a CUDA device, items inattrs
that start with “saved_” will not be moved. .. note:This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- attrs (
list
): list of strs containing the attributes of this object to move to the specified device/dtype
- device (
- Returns:
Module: self
plenoptic.synthesize.synthesis module
abstract synthesis super-class.
- class plenoptic.synthesize.synthesis.OptimizedSynthesis(range_penalty_lambda=0.1, allowed_range=(0, 1))[source]
Bases:
Synthesis
Abstract super-class for synthesis objects that use optimization.
The primary difference between this and the generic Synthesis class is that these will use an optimizer object to iteratively update their output.
- Attributes:
- allowed_range
gradient_norm
Synthesis gradient’s L2 norm over iterations.
losses
Synthesis loss over iterations.
- optimizer
pixel_change_norm
L2 norm change in pixel values over iterations.
- range_penalty_lambda
- store_progress
Methods
load
(file_path[, map_location, ...])Load all relevant attributes from a .pt file.
How good is the current synthesized object.
save
(file_path[, attrs])Save all relevant (non-model) variables in .pt file.
synthesize
()Synthesize something.
to
(*args[, attrs])Moves and/or casts the parameters and buffers.
- property allowed_range
- property gradient_norm
Synthesis gradient’s L2 norm over iterations.
- property losses
Synthesis loss over iterations.
- abstract objective_function()[source]
How good is the current synthesized object.
See
plenoptic.tools.optim
for some examples.
- property optimizer
- property pixel_change_norm
L2 norm change in pixel values over iterations.
- property range_penalty_lambda
- property store_progress
- class plenoptic.synthesize.synthesis.Synthesis[source]
Bases:
ABC
Abstract super-class for synthesis objects.
All synthesis objects share a variety of similarities and thus need to have similar methods. Some of these can be implemented here and simply inherited, some of them will need to be different for each sub-class and thus are marked as abstract methods here
Methods
load
(file_path[, map_location, ...])Load all relevant attributes from a .pt file.
save
(file_path[, attrs])Save all relevant (non-model) variables in .pt file.
Synthesize something.
to
(*args[, attrs])Moves and/or casts the parameters and buffers.
- load(file_path, map_location=None, check_attributes=[], check_loss_functions=[], **pickle_load_args)[source]
Load all relevant attributes from a .pt file.
This should be called by an initialized
Synthesis
object – we will ensure that the attributes in thecheck_attributes
arg all match in the current and loaded object.Note this operates in place and so doesn’t return anything.
- Parameters:
file_path (
str
) – The path to load the synthesis object frommap_location (
Optional
[str
]) – map_location argument to pass totorch.load
. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass totorch.device
check_attributes (
List
[str
]) – List of strings we ensure are identical in the currentSynthesis
object and the loaded one. Checking the model is generally not recommended, since it can be hard to do (checking callable objects is hard in Python) – instead, checking thebase_representation
should ensure the model hasn’t functinoally changed.check_loss_functions (
List
[str
]) – Names of attributes that are loss functions and so must be checked specially – loss functions are callables, and it’s very difficult to check python callables for equality so, to get around that, we instead call the two versions on the same pair of tensors, and compare the outputs.pickle_load_args – any additional kwargs will be added to
pickle_module.load
viatorch.load
, see that function’s docstring for details.
- save(file_path, attrs=None)[source]
Save all relevant (non-model) variables in .pt file.
If you leave attrs as None, we grab vars(self) and exclude ‘model’. This is probably correct, but the option is provided to override it just in case
- Parameters:
file_path (str) – The path to save the synthesis object to
attrs (list or None, optional) – List of strs containing the names of the attributes of this object to save. See above for behavior if attrs is None.
- abstract to(*args, attrs=[], **kwargs)[source]
Moves and/or casts the parameters and buffers. Similar to
save
, this is an abstract method only because you need to define the attributes to call to on.This can be called as .. function:: to(device=None, dtype=None, non_blocking=False) .. function:: to(dtype, non_blocking=False) .. function:: to(tensor, non_blocking=False) Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. When calling this method to move tensors to a CUDA device, items inattrs
that start with “saved_” will not be moved. .. note:This method modifies the module in-place.
- Args:
- device (
torch.device
): the desired device of the parameters and buffers in this module
- dtype (
torch.dtype
): the desired floating point type of the floating point parameters and buffers in this module
- tensor (torch.Tensor): Tensor whose dtype and device are the desired
dtype and device for all parameters and buffers in this module
- attrs (
list
): list of strs containing the attributes of this object to move to the specified device/dtype
- device (