从零开始学Pytorch（十一）之ModernRNN

RNN:

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ycy6d8iB-1584285348969)(https://cdn.kesci.com/upload/image/q5jjvcykud.png?imageView2/0/w/320/h/320)]

$$H_t=\varphi (X_t{W}_{xh}+H_{t-1}{W}_{hh}+b_h)$$

GRU:

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-r6S2i6jZ-1584285348971)(https://cdn.kesci.com/upload/image/q5jk0q9suq.png?imageView2/0/w/640/h/640)]

$$R_t=\sigma (X_t{W}_{xr}+H_{t-1}{W}_{hr}+b_r)$$
$$Z_t=\sigma (X_t{W}_{xz}+H_{t-1}{W}_{hz}+b_z)$$
$${\tilde{H}}_t=tanh(X_t{W}_{xh}+(R_t\odot H_{t-1})W_{hh}+b_h)$$
$$H_t=Z_t\odot H_{t-1}+(1-Z_t)\odot {\tilde{H}}_t$$
• 重置⻔有助于捕捉时间序列⾥短期的依赖关系；
• 更新⻔有助于捕捉时间序列⾥⻓期的依赖关系。

载入数据集

import os
os.listdir('/home/input')#数据集文件夹
import numpy as np
import torch
from torch import nn, optim
import torch.nn.functional as F
import sys
sys.path.append("../input/")
import d2l_jay4504 as d2l
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

(corpus_indices, char_to_idx, idx_to_char, vocab_size) = d2l.load_data_jay_lyrics()

  

初始化参数

num_inputs, num_hiddens, num_outputs = vocab_size, 256, vocab_size
print('will use', device)

def get_params():  
    def _one(shape):
        ts = torch.tensor(np.random.normal(0, 0.01, size=shape), device=device, dtype=torch.float32) #正态分布
        return torch.nn.Parameter(ts, requires_grad=True)
    def _three():
        return (_one((num_inputs, num_hiddens)),
                _one((num_hiddens, num_hiddens)),
                torch.nn.Parameter(torch.zeros(num_hiddens, device=device, dtype=torch.float32), requires_grad=True))
     
    W_xz, W_hz, b_z = _three()  # 更新门参数
    W_xr, W_hr, b_r = _three()  # 重置门参数
    W_xh, W_hh, b_h = _three()  # 候选隐藏状态参数
    
    # 输出层参数
    W_hq = _one((num_hiddens, num_outputs))
    b_q = torch.nn.Parameter(torch.zeros(num_outputs, device=device, dtype=torch.float32), requires_grad=True)
    return nn.ParameterList([W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q])

def init_gru_state(batch_size, num_hiddens, device):   #隐藏状态初始化
    return (torch.zeros((batch_size, num_hiddens), device=device), )

  

GRU模型

def gru(inputs, state, params):
    W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q = params
    H, = state
    outputs = []
    for X in inputs:
        Z = torch.sigmoid(torch.matmul(X, W_xz) + torch.matmul(H, W_hz) + b_z)
        R = torch.sigmoid(torch.matmul(X, W_xr) + torch.matmul(H, W_hr) + b_r)
        H_tilda = torch.tanh(torch.matmul(X, W_xh) + R * torch.matmul(H, W_hh) + b_h)
        H = Z * H + (1 - Z) * H_tilda
        Y = torch.matmul(H, W_hq) + b_q
        outputs.append(Y)
    return outputs, (H,)

  

训练模型

num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']
d2l.train_and_predict_rnn(gru, get_params, init_gru_state, num_hiddens,
                          vocab_size, device, corpus_indices, idx_to_char,
                          char_to_idx, False, num_epochs, num_steps, lr,
                          clipping_theta, batch_size, pred_period, pred_len,
                          prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-YiCbWe1Y-1584285348972)(https://imgkr.cn-bj.ufileos.com/2796f004-0af8-41ec-951d-38d45ca897e0.png)]

简洁实现

num_hiddens=256
num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']

lr = 1e-2 # 注意调整学习率
gru_layer = nn.GRU(input_size=vocab_size, hidden_size=num_hiddens)
model = d2l.RNNModel(gru_layer, vocab_size).to(device)
d2l.train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-s7GZt0zB-1584285348975)(https://imgkr.cn-bj.ufileos.com/31d8288d-7d11-43af-a65b-3c46692cd4b4.png)]

LSTM

长短期记忆long short-term memory:
遗忘门:控制上一时间步的记忆细胞
输入门:控制当前时间步的输入
输出门:控制从记忆细胞到隐藏状态
记忆细胞：⼀种特殊的隐藏状态的信息的流动

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-V5S3OxqQ-1584285348976)(https://cdn.kesci.com/upload/image/q5jk2bnnej.png?imageView2/0/w/640/h/640)]

$$I_t=\sigma (X_t{W}_{xi}+H_{t-1}{W}_{hi}+b_i)$$
$$F_t=\sigma (X_t{W}_{xf}+H_{t-1}{W}_{hf}+b_f)$$
$$O_t=\sigma (X_t{W}_{xo}+H_{t-1}{W}_{ho}+b_o)$$
$${\tilde{C}}_t=tanh(X_t{W}_{xc}+H_{t-1}{W}_{hc}+b_c)$$
$$C_t=F_t\odot C_{t-1}+I_t\odot {\tilde{C}}_t$$
$$H_t=O_t\odot tanh(C_t)$$

初始化参数

num_inputs, num_hiddens, num_outputs = vocab_size, 256, vocab_size
print('will use', device)

def get_params():
    def _one(shape):
        ts = torch.tensor(np.random.normal(0, 0.01, size=shape), device=device, dtype=torch.float32)
        return torch.nn.Parameter(ts, requires_grad=True)
    def _three():
        return (_one((num_inputs, num_hiddens)),
                _one((num_hiddens, num_hiddens)),
                torch.nn.Parameter(torch.zeros(num_hiddens, device=device, dtype=torch.float32), requires_grad=True))
    
    W_xi, W_hi, b_i = _three()  # 输入门参数
    W_xf, W_hf, b_f = _three()  # 遗忘门参数
    W_xo, W_ho, b_o = _three()  # 输出门参数
    W_xc, W_hc, b_c = _three()  # 候选记忆细胞参数
    
    # 输出层参数
    W_hq = _one((num_hiddens, num_outputs))
    b_q = torch.nn.Parameter(torch.zeros(num_outputs, device=device, dtype=torch.float32), requires_grad=True)
    return nn.ParameterList([W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c, W_hq, b_q])

def init_lstm_state(batch_size, num_hiddens, device):
    return (torch.zeros((batch_size, num_hiddens), device=device), 
            torch.zeros((batch_size, num_hiddens), device=device))

  

LSTM模型

def lstm(inputs, state, params):
    [W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c, W_hq, b_q] = params
    (H, C) = state
    outputs = []
    for X in inputs:
        I = torch.sigmoid(torch.matmul(X, W_xi) + torch.matmul(H, W_hi) + b_i)
        F = torch.sigmoid(torch.matmul(X, W_xf) + torch.matmul(H, W_hf) + b_f)
        O = torch.sigmoid(torch.matmul(X, W_xo) + torch.matmul(H, W_ho) + b_o)
        C_tilda = torch.tanh(torch.matmul(X, W_xc) + torch.matmul(H, W_hc) + b_c)
        C = F * C + I * C_tilda
        H = O * C.tanh()
        Y = torch.matmul(H, W_hq) + b_q
        outputs.append(Y)
    return outputs, (H, C)

  

训练模型

num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']

d2l.train_and_predict_rnn(lstm, get_params, init_lstm_state, num_hiddens,
                          vocab_size, device, corpus_indices, idx_to_char,
                          char_to_idx, False, num_epochs, num_steps, lr,
                          clipping_theta, batch_size, pred_period, pred_len,
                          prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-u6yjmgTB-1584285348977)(https://imgkr.cn-bj.ufileos.com/2b6c7757-175f-4de4-96db-c34984cd6795.png)]

简洁实现

num_hiddens=256
num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']

lr = 1e-2 # 注意调整学习率
lstm_layer = nn.LSTM(input_size=vocab_size, hidden_size=num_hiddens)
model = d2l.RNNModel(lstm_layer, vocab_size)
d2l.train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes)

  

深度循环神经网络

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lAOtlHvs-1584285348979)(https://cdn.kesci.com/upload/image/q5jk3z1hvz.png?imageView2/0/w/320/h/320)]

$${\mathbf{H}}_t^{(1)}=\varphi ({\mathbf{X}}_t{\mathbf{W}}_{xh}^{(1)}+{\mathbf{H}}_{t-1}^{(1)}{\mathbf{W}}_{hh}^{(1)}+{\mathbf{b}}_h^{(1)})$$
$${\mathbf{H}}_t^{(\ell )}=\varphi ({\mathbf{H}}_t^{(\ell -1)}{\mathbf{W}}_{xh}^{(\ell )}+{\mathbf{H}}_{t-1}^{(\ell )}{\mathbf{W}}_{hh}^{(\ell )}+{\mathbf{b}}_h^{(\ell )})$$
$${\mathbf{O}}_t={\mathbf{H}}_t^{(L)}{\mathbf{W}}_{hq}+{\mathbf{b}}_q$$

num_hiddens=256
num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']

lr = 1e-2 # 注意调整学习率

gru_layer = nn.LSTM(input_size=vocab_size, hidden_size=num_hiddens,num_layers=2)
model = d2l.RNNModel(gru_layer, vocab_size).to(device)
d2l.train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-LPDnzNUI-1584285348981)(https://imgkr.cn-bj.ufileos.com/524263d2-c8b0-4ad6-8d3e-69e970187cbb.png)]

gru_layer = nn.LSTM(input_size=vocab_size, hidden_size=num_hiddens,num_layers=6)
model = d2l.RNNModel(gru_layer, vocab_size).to(device)
d2l.train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cdLIhYUy-1584285348982)(https://imgkr.cn-bj.ufileos.com/43fd13df-1c5b-4dc3-aeba-3d21c7844b8b.png)]

双向循环神经网络

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-o1iBelTn-1584285348984)(https://cdn.kesci.com/upload/image/q5j8hmgyrz.png?imageView2/0/w/320/h/320)]

$$\begin{array}{rl}{\overrightarrow{\mathbf{H}}}_t& =\varphi ({\mathbf{X}}_t{\mathbf{W}}_{xh}^{(f)}+{\overrightarrow{\mathbf{H}}}_{t-1}{\mathbf{W}}_{hh}^{(f)}+{\mathbf{b}}_h^{(f)})\\ {\overleftarrow{\mathbf{H}}}_t& =\varphi ({\mathbf{X}}_t{\mathbf{W}}_{xh}^{(b)}+{\overleftarrow{\mathbf{H}}}_{t+1}{\mathbf{W}}_{hh}^{(b)}+{\mathbf{b}}_h^{(b)})\end{array}$$
$${\mathbf{H}}_t=({\overrightarrow{\mathbf{H}}}_t,{\overleftarrow{\mathbf{H}}}_t)$$
$${\mathbf{O}}_t={\mathbf{H}}_t{\mathbf{W}}_{hq}+{\mathbf{b}}_q$$

num_hiddens=128
num_epochs, num_steps, batch_size, lr, clipping_theta = 160, 35, 32, 1e-2, 1e-2
pred_period, pred_len, prefixes = 40, 50, ['分开', '不分开']

lr = 1e-2 # 注意调整学习率

gru_layer = nn.GRU(input_size=vocab_size, hidden_size=num_hiddens,bidirectional=True)
model = d2l.RNNModel(gru_layer, vocab_size).to(device)
d2l.train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes)

  

输出：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-GhF608ea-1584285348985)(https://imgkr.cn-bj.ufileos.com/4bcdb31e-b134-485d-bdd5-0eb4ec923061.png)]

文章来源: blog.csdn.net，作者：小小谢先生，版权归原作者所有，如需转载，请联系作者。

原文链接：blog.csdn.net/xiewenrui1996/article/details/104889329