Recurrent Neural Networks cheatsheet

Overview

Architecture of a traditional RNN ― Recurrent neural networks, also known as RNNs, are a class of neural networks that allow previous outputs to be used as inputs while having hidden states. They are typically as follows:

For each timestep $t$ , the activation $a^{< t >}$ and the output $y^{< t >}$ are expressed as follows:

a^{< t >} = g_{1} (W_{a a} a^{< t - 1 >} + W_{a x} x^{< t >} + b_{a}) and y^{< t >} = g_{2} (W_{y a} a^{< t >} + b_{y})

where $W_{a x}, W_{a a}, W_{y a}, b_{a}, b_{y}$ are coefficients that are shared temporally and $g_{1}, g_{2}$ activation functions.

The pros and cons of a typical RNN architecture are summed up in the table below:

Advantages	Drawbacks
• Possibility of processing input of any length • Model size not increasing with size of input • Computation takes into account historical information • Weights are shared across time	• Computation being slow • Difficulty of accessing information from a long time ago • Cannot consider any future input for the current state

Applications of RNNs ― RNN models are mostly used in the fields of natural language processing and speech recognition. The different applications are summed up in the table below:

Type of RNN	Illustration	Example
One-to-one $T_{x} = T_{y} = 1$		Traditional neural network
One-to-many $T_{x} = 1, T_{y} > 1$		Music generation
Many-to-one $T_{x} > 1, T_{y} = 1$		Sentiment classification
Many-to-many $T_{x} = T_{y}$		Name entity recognition
Many-to-many $T_{x} \neq T_{y}$		Machine translation

Loss function ― In the case of a recurrent neural network, the loss function $L$ of all time steps is defined based on the loss at every time step as follows:

L (ˆ y, y) = T_{y} \sum t = 1 L ({ˆ y}^{< t >}, y^{< t >})

Backpropagation through time ― Backpropagation is done at each point in time. At timestep $T$ , the derivative of the loss $L$ with respect to weight matrix $W$ is expressed as follows:

\frac{\partial L^{(T)}}{\partial W} = T \sum t = 1 {\frac{\partial^{(T)}}{\partial W} ∣ ∣ ∣}_{(t)}

Handling long term dependencies

Commonly used activation functions ― The most common activation functions used in RNN modules are described below:

Sigmoid	Tanh	RELU
$g (z) = \frac{1}{1 + e^{- z}}$	$g (z) = \frac{e^{z} - e^{- z}}{e^{z} + e^{- z}}$	$g (z) = max (0, z)$

Vanishing/exploding gradient ― The vanishing and exploding gradient phenomena are often encountered in the context of RNNs. The reason why they happen is that it is difficult to capture long term dependencies because of multiplicative gradient that can be exponentially decreasing/increasing with respect to the number of layers.

Gradient clipping ― It is a technique used to cope with the exploding gradient problem sometimes encountered when performing backpropagation. By capping the maximum value for the gradient, this phenomenon is controlled in practice.

Types of gates ― In order to remedy the vanishing gradient problem, specific gates are used in some types of RNNs and usually have a well-defined purpose. They are usually noted $Γ$ and are equal to:

Γ = σ (W x^{< t >} + U a^{< t - 1 >} + b)

where $W, U, b$ are coefficients specific to the gate and $σ$ is the sigmoid function. The main ones are summed up in the table below:

Type of gate	Role	Used in
Update gate $Γ_{u}$	How much past should matter now?	GRU, LSTM
Relevance gate $Γ_{r}$	Drop previous information?	GRU, LSTM
Forget gate $Γ_{f}$	Erase a cell or not?	LSTM
Output gate $Γ_{o}$	How much to reveal of a cell?	LSTM

GRU/LSTM ― Gated Recurrent Unit (GRU) and Long Short-Term Memory units (LSTM) deal with the vanishing gradient problem encountered by traditional RNNs, with LSTM being a generalization of GRU. Below is a table summing up the characterizing equations of each architecture:

Characterization	Gated Recurrent Unit (GRU)	Long Short-Term Memory (LSTM)
${\tilde{c}}^{< t >}$	$tanh (W_{c} [Γ_{r} ⋆ a^{< t - 1 >}, x^{< t >}] + b_{c})$	$tanh (W_{c} [Γ_{r} ⋆ a^{< t - 1 >}, x^{< t >}] + b_{c})$
$c^{< t >}$	$Γ_{u} ⋆ {\tilde{c}}^{< t >} + (1 - Γ_{u}) ⋆ c^{< t - 1 >}$	$Γ_{u} ⋆ {\tilde{c}}^{< t >} + Γ_{f} ⋆ c^{< t - 1 >}$
$a^{< t >}$	$c^{< t >}$	$Γ_{o} ⋆ c^{< t >}$
Dependencies

Remark: the sign $⋆$ denotes the element-wise multiplication between two vectors.

Variants of RNNs ― The table below sums up the other commonly used RNN architectures:

Bidirectional (BRNN)	Deep (DRNN)

Learning word representation

In this section, we note $V$ the vocabulary and $| V |$ its size.

Motivation and notations

Representation techniques ― The two main ways of representing words are summed up in the table below:

1-hot representation	Word embedding

• Noted $o_{w}$ • Naive approach, no similarity information	• Noted $e_{w}$ • Takes into account words similarity

Embedding matrix ― For a given word $w$ , the embedding matrix $E$ is a matrix that maps its 1-hot representation $o_{w}$ to its embedding $e_{w}$ as follows:

e_{w} = E o_{w}

Remark: learning the embedding matrix can be done using target/context likelihood models.

Word embeddings

Word2vec ― Word2vec is a framework aimed at learning word embeddings by estimating the likelihood that a given word is surrounded by other words. Popular models include skip-gram, negative sampling and CBOW.

Skip-gram ― The skip-gram word2vec model is a supervised learning task that learns word embeddings by assessing the likelihood of any given target word $t$ happening with a context word $c$ . By noting $θ_{t}$ a parameter associated with $t$ , the probability $P (t | c)$ is given by:

P (t | c) = \frac{exp (θ_{t}^{T} e_{c})}{| V | \sum j = 1 exp (θ_{j}^{T} e_{c})}

Remark: summing over the whole vocabulary in the denominator of the softmax part makes this model computationally expensive. CBOW is another word2vec model using the surrounding words to predict a given word.

Negative sampling ― It is a set of binary classifiers using logistic regressions that aim at assessing how a given context and a given target words are likely to appear simultaneously, with the models being trained on sets of $k$ negative examples and 1 positive example. Given a context word $c$ and a target word $t$ , the prediction is expressed by:

P (y = 1 | c, t) = σ (θ_{t}^{T} e_{c})

Remark: this method is less computationally expensive than the skip-gram model.

GloVe ― The GloVe model, short for global vectors for word representation, is a word embedding technique that uses a co-occurence matrix $X$ where each $X_{i, j}$ denotes the number of times that a target $i$ occurred with a context $j$ . Its cost function $J$ is as follows:

J (θ) = \frac{1}{2} | V | \sum i, j = 1 f (X_{i j}) (θ_{i}^{T} e_{j} + b_{i} + b_{j}^{'} - log (X_{i j}))^{2}

where $f$ is a weighting function such that $X_{i, j} = 0 ⟹ f (X_{i, j}) = 0$ .
Given the symmetry that $e$ and $θ$ play in this model, the final word embedding $e_{w}^{(final)}$ is given by:

e_{w}^{(final)} = \frac{e_{w} + θ_{w}}{2}

Remark: the individual components of the learned word embeddings are not necessarily interpretable.

Comparing words

Cosine similarity ― The cosine similarity between words $w_{1}$ and $w_{2}$ is expressed as follows:

similarity = \frac{w_{1} \cdot w_{2}}{| | w_{1} | | | | w_{2} | |} = cos (θ)

Remark: $θ$ is the angle between words $w_{1}$ and $w_{2}$ .

$t$ -SNE ― $t$ -SNE ( $t$ -distributed Stochastic Neighbor Embedding) is a technique aimed at reducing high-dimensional embeddings into a lower dimensional space. In practice, it is commonly used to visualize word vectors in the 2D space.

Language model

Overview ― A language model aims at estimating the probability of a sentence $P (y)$ .

$n$ -gram model ― This model is a naive approach aiming at quantifying the probability that an expression appears in a corpus by counting its number of appearance in the training data.

Perplexity ― Language models are commonly assessed using the perplexity metric, also known as PP, which can be interpreted as the inverse probability of the dataset normalized by the number of words $T$ . The perplexity is such that the lower, the better and is defined as follows:

PP = T \prod t = 1 {⎛ ⎝ \frac{1}{_{j = 1}^{| V |}_{j}^{(t)} \cdot_{j}^{(t)}} ⎞ ⎠}^{\frac{1}{T}}

Remark: PP is commonly used in $t$ -SNE.

Machine translation

Overview ― A machine translation model is similar to a language model except it has an encoder network placed before. For this reason, it is sometimes referred as a conditional language model.

The goal is to find a sentence $y$ such that:

y = arg max y^{< 1 >}, . . ., y^{< T_{y} >} P (y^{< 1 >}, . . ., y^{< T_{y} >} | x)

Beam search ― It is a heuristic search algorithm used in machine translation and speech recognition to find the likeliest sentence $y$ given an input $x$ .

• Step 1: Find top $B$ likely words $y^{< 1 >}$
• Step 2: Compute conditional probabilities $y^{< k >} | x, y^{< 1 >}, . . ., y^{< k - 1 >}$
• Step 3: Keep top $B$ combinations $x, y^{< 1 >}, . . ., y^{< k >}$

Remark: if the beam width is set to 1, then this is equivalent to a naive greedy search.

Beam width ― The beam width $B$ is a parameter for beam search. Large values of $B$ yield to better result but with slower performance and increased memory. Small values of $B$ lead to worse results but is less computationally intensive. A standard value for $B$ is around 10.

Length normalization ― In order to improve numerical stability, beam search is usually applied on the following normalized objective, often called the normalized log-likelihood objective, defined as:

Objective = \frac{1}{T_{y}^{α}} T_{y} \sum t = 1 log [p (y^{< t >} | x, y^{< 1 >}, . . ., y^{< t - 1 >})]

Remark: the parameter $α$ can be seen as a softener, and its value is usually between 0.5 and 1.

Error analysis ― When obtaining a predicted translation $\hat{y}$ that is bad, one can wonder why we did not get a good translation $y^{*}$ by performing the following error analysis:

Case	$P (y^{*} \| x) > P (\hat{y} \| x)$	$P (y^{*} \| x) ⩽ P (\hat{y} \| x)$
Root cause	Beam search faulty	RNN faulty
Remedies	Increase beam width	• Try different architecture • Regularize • Get more data

Bleu score ― The bilingual evaluation understudy (bleu) score quantifies how good a machine translation is by computing a similarity score based on $n$ -gram precision. It is defined as follows:

bleu score = exp (\frac{1}{n} n \sum k = 1 p_{k})

where $p_{n}$ is the bleu score on $n$ -gram only defined as follows:

p_{n} = \frac{\sum n-gram \in ˆ y {count}_{clip} (n-gram)}{\sum n-gram \in ˆ y count (n-gram)}

Remark: a brevity penalty may be applied to short predicted translations to prevent an artificially inflated bleu score.

Attention

Attention model ― This model allows an RNN to pay attention to specific parts of the input that is considered as being important, which improves the performance of the resulting model in practice. By noting $α^{< t, t^{'} >}$ the amount of attention that the output $y^{< t >}$ should pay to the activation $a^{< t^{'} >}$ and $c^{< t >}$ the context at time $t$ , we have:

c^{< t >} = \sum t^{'} α^{< t, t >} a^{< t >} with \sum t^{'} α^{< t, t >} = 1

Remark: the attention scores are commonly used in image captioning and machine translation.

Attention weight ― The amount of attention that the output $y^{< t >}$ should pay to the activation $a^{< t^{'} >}$ is given by $α^{< t, t^{'} >}$ computed as follows:

α^{< t, t >} = \frac{exp (e^{< t, t >})}{T_{x} \sum t^{''} = 1 exp (e^{< t, t >})}

Remark: computation complexity is quadratic with respect to $T_{x}$ .

Source: Recurrent Neural Networks cheatsheet

Related Blogs:

Transforming Enterprises with
Data & AI Services & Solutions.

ThirdEye delivers Data and AI services & solutions for enterprises worldwide by
leveraging state-of-the-art Data & AI technologies.

Talk to ThirdEye

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Recurrent Neural Networks cheatsheet

Overview

Handling long term dependencies

Learning word representation

Motivation and notations

Word embeddings

Comparing words

Language model

Machine translation

Attention

Transforming Enterprises with
Data & AI Services & Solutions.

Primary Services

Pre-Built Applications

Data & AI Solutions

ThirdEye Exclusive

Insights

Talk To Us

Recurrent Neural Networks cheatsheet

Overview

Handling long term dependencies

Learning word representation

Motivation and notations

Word embeddings

Comparing words

Language model

Machine translation

Attention

Transforming Enterprises with Data & AI Services & Solutions.

Share This Article

Related Posts

Time Series Data Exploration with Wavelet Transform

Synthetic Time Series Data Generation

Stock Portfolio Balancing with Monte Carlo Simulation

Tabular Data Column Semantic Type Identification with Contrastive Deep Learning

Primary Services

Pre-Built Applications

Data & AI Solutions

ThirdEye Exclusive

Insights

Talk To Us

Transforming Enterprises with
Data & AI Services & Solutions.