WeightNorm< InputDataType, OutputDataType, CustomLayers > Class Template Reference

Declaration of the WeightNorm layer class. More...

Public Member Functions
	WeightNorm (LayerTypes< CustomLayers...> layer=LayerTypes< CustomLayers...>())
	Create the WeightNorm layer object. More...
	~WeightNorm ()
	Destructor to release allocated memory. More...
template<typename eT >
void	Backward (const arma::Mat< eT > &input, const arma::Mat< eT > &gy, arma::Mat< eT > &g)
	Backward pass through the layer. More...
OutputDataType const &	Delta () const
	Get the delta. More...
OutputDataType &	Delta ()
	Modify the delta. More...
template<typename eT >
void	Forward (const arma::Mat< eT > &input, arma::Mat< eT > &output)
	Forward pass of the WeightNorm layer. More...
template<typename eT >
void	Gradient (const arma::Mat< eT > &input, const arma::Mat< eT > &error, arma::Mat< eT > &gradient)
	Calculate the gradient using the output delta, input activations and the weights of the wrapped layer. More...
OutputDataType const &	Gradient () const
	Get the gradient. More...
OutputDataType &	Gradient ()
	Modify the gradient. More...
LayerTypes< CustomLayers...> const &	Layer ()
	Get the wrapped layer. More...
OutputDataType const &	OutputParameter () const
	Get the output parameter. More...
OutputDataType &	OutputParameter ()
	Modify the output parameter. More...
OutputDataType const &	Parameters () const
	Get the parameters. More...
OutputDataType &	Parameters ()
	Modify the parameters. More...
void	Reset ()
	Reset the layer parameters. More...
template<typename Archive >
void	serialize (Archive &ar, const unsigned int)
	Serialize the layer. More...

Detailed Description

template<typename InputDataType, typename OutputDataType, typename... CustomLayers>
class mlpack::ann::WeightNorm< InputDataType, OutputDataType, CustomLayers >

Declaration of the WeightNorm layer class.

The layer reparameterizes the weight vectors in a neural network, decoupling the length of those weight vectors from their direction. This reparameterization does not introduce any dependencies between the examples in a mini-batch.

This class will be a wrapper around existing layers. It will just modify the calculation and updation of weights of the layer.

For more information, refer to the following paper,

@inproceedings{Salimans2016WeightNorm,
  title = {Weight Normalization: A Simple Reparameterization to Accelerate
           Training of Deep Neural Networks},
  author = {Tim Salimans, Diederik P. Kingma},
  booktitle = {Neural Information Processing Systems 2016},
  year = {2016},
  url  = {https://arxiv.org/abs/1602.07868},
}

Template Parameters

InputDataType	Type of the input data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
OutputDataType	Type of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
CustomLayers	Additional custom layers that can be added.

Definition at line 208 of file layer_types.hpp.

Constructor & Destructor Documentation

WeightNorm

(

LayerTypes< CustomLayers...>

layer = LayerTypes< CustomLayers...>()

)

Create the WeightNorm layer object.

Parameters

layer

The layer whose weights are needed to be normalized.

~WeightNorm

(

)

Destructor to release allocated memory.

Member Function Documentation

void Backward	(	const arma::Mat< eT > &	input,
		const arma::Mat< eT > &	gy,
		arma::Mat< eT > &	g
	)

Backward pass through the layer.

This function calls the Backward() function of the wrapped layer.

Parameters

input	The input activations.
gy	The backpropagated error.
g	The calculated gradient.

OutputDataType const& Delta ( ) const

inline

Get the delta.

Definition at line 118 of file weight_norm.hpp.

OutputDataType& Delta ( )

inline

Modify the delta.

Definition at line 120 of file weight_norm.hpp.

void Forward	(	const arma::Mat< eT > &	input,
		arma::Mat< eT > &	output
	)

Forward pass of the WeightNorm layer.

Calculates the weights of the wrapped layer from the parameter vector v and the scalar parameter g. It then calulates the output of the wrapped layer from the calculated weights.

Parameters

input	Input data for the layer.
output	Resulting output activations.

void Gradient	(	const arma::Mat< eT > &	input,
		const arma::Mat< eT > &	error,
		arma::Mat< eT > &	gradient
	)

Calculate the gradient using the output delta, input activations and the weights of the wrapped layer.

Parameters

input	The input activations.
error	The calculated error.
gradient	The calculated gradient.

OutputDataType const& Gradient ( ) const

inline

Get the gradient.

Definition at line 123 of file weight_norm.hpp.

OutputDataType& Gradient ( )

inline

Modify the gradient.

Definition at line 125 of file weight_norm.hpp.

LayerTypes<CustomLayers...> const& Layer ( )

inline

Get the wrapped layer.

Definition at line 138 of file weight_norm.hpp.

OutputDataType const& OutputParameter ( ) const

inline

Get the output parameter.

Definition at line 128 of file weight_norm.hpp.

OutputDataType& OutputParameter ( )

inline

Modify the output parameter.

Definition at line 130 of file weight_norm.hpp.

OutputDataType const& Parameters ( ) const

inline

Get the parameters.

Definition at line 133 of file weight_norm.hpp.

OutputDataType& Parameters ( )

inline

Modify the parameters.

Definition at line 135 of file weight_norm.hpp.

void Reset

(

)

Reset the layer parameters.

void serialize	(	Archive &	ar,
		const unsigned	int
	)

Serialize the layer.

---

The documentation for this class was generated from the following files:

• src/mlpack/methods/ann/layer/layer_types.hpp
• src/mlpack/methods/ann/layer/weight_norm.hpp