长短期记忆网络（LSTM）技术详解：从RNN到现代序列建模

摘要

长短期记忆网络（Long Short-Term Memory, LSTM）是循环神经网络（RNN）的重要变体，专门设计用于解决传统RNN的梯度消失问题。本文深入解析LSTM的核心原理、网络架构、关键技术以及从RNN到现代序列建模的发展历程，帮助读者全面理解这一重要技术。

关键词： 长短期记忆网络、LSTM、循环神经网络、序列建模、梯度消失

文章目录

1. 引言

长短期记忆网络（LSTM）是Hochreiter和Schmidhuber在1997年提出的一种特殊的循环神经网络架构，专门用于解决传统RNN在处理长序列时的梯度消失问题。LSTM通过引入门控机制，能够有效地学习长期依赖关系，在自然语言处理、时间序列预测等领域取得了巨大成功。

1.1 LSTM的发展历程

  • 1997年: LSTM的原始论文发表
  • 2000s: LSTM在语音识别中的应用
  • 2010s: LSTM在机器翻译和文本生成中的突破
  • 2015年至今: LSTM与注意力机制、Transformer的结合

2. RNN的问题与LSTM的解决方案

2.1 传统RNN的问题

传统RNN在处理长序列时面临的主要问题

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.W_ih = nn.Linear(input_size, hidden_size)
        self.W_hh = nn.Linear(hidden_size, hidden_size)
        self.W_ho = nn.Linear(hidden_size, output_size)
        self.tanh = nn.Tanh()
    
    def forward(self, input_seq):
        batch_size = input_seq.size(0)
        seq_len = input_seq.size(1)
        
        # 初始化隐藏状态
        hidden = torch.zeros(batch_size, self.hidden_size)
        outputs = []
        
        for t in range(seq_len):
            # RNN前向传播
            input_t = input_seq[:, t, :]
            hidden = self.tanh(self.W_ih(input_t) + self.W_hh(hidden))
            output = self.W_ho(hidden)
            outputs.append(output)
        
        return torch.stack(outputs, dim=1)

# 演示梯度消失问题
def demonstrate_gradient_vanishing():
    rnn = SimpleRNN(input_size=10, hidden_size=20, output_size=1)
    
    # 创建长序列
    long_sequence = torch.randn(1, 50, 10)  # (batch, seq_len, input_size)
    target = torch.randn(1, 50, 1)
    
    # 前向传播
    output = rnn(long_sequence)
    loss = F.mse_loss(output, target)
    
    # 反向传播
    loss.backward()
    
    # 检查梯度
    print("RNN权重梯度:")
    for name, param in rnn.named_parameters():
        if param.grad is not None:
            grad_norm = param.grad.norm().item()
            print(f"{name}: {grad_norm:.6f}")

demonstrate_gradient_vanishing()

2.2 LSTM的核心思想

LSTM通过引入三个门控机制来解决梯度消失问题

  1. 遗忘门（Forget Gate）: 决定从细胞状态中丢弃什么信息
  2. 输入门（Input Gate）: 决定什么新信息被存储在细胞状态中
  3. 输出门（Output Gate）: 决定输出什么值

3. LSTM的数学原理

3.1 LSTM的前向传播

LSTM的数学公式如下

class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        
        # 输入门
        self.W_ii = nn.Linear(input_size, hidden_size)
        self.W_hi = nn.Linear(hidden_size, hidden_size)
        
        # 遗忘门
        self.W_if = nn.Linear(input_size, hidden_size)
        self.W_hf = nn.Linear(hidden_size, hidden_size)
        
        # 候选值
        self.W_ig = nn.Linear(input_size, hidden_size)
        self.W_hg = nn.Linear(hidden_size, hidden_size)
        
        # 输出门
        self.W_io = nn.Linear(input_size, hidden_size)
        self.W_ho = nn.Linear(hidden_size, hidden_size)
        
        self.sigmoid = nn.Sigmoid()
        self.tanh = nn.Tanh()
    
    def forward(self, input_t, hidden, cell):
        """
        LSTM前向传播
        input_t: 当前时间步的输入
        hidden: 前一时间步的隐藏状态
        cell: 前一时间步的细胞状态
        """
        # 输入门
        i_t = self.sigmoid(self.W_ii(input_t) + self.W_hi(hidden))
        
        # 遗忘门
        f_t = self.sigmoid(self.W_if(input_t) + self.W_hf(hidden))
        
        # 候选值
        g_t = self.tanh(self.W_ig(input_t) + self.W_hg(hidden))
        
        # 输出门
        o_t = self.sigmoid(self.W_io(input_t) + self.W_ho(hidden))
        
        # 更新细胞状态
        cell_t = f_t * cell + i_t * g_t
        
        # 更新隐藏状态
        hidden_t = o_t * self.tanh(cell_t)
        
        return hidden_t, cell_t

# 使用PyTorch内置LSTM
class PyTorchLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers=1):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
    
    def forward(self, x):
        output, (hidden, cell) = self.lstm(x)
        return output, hidden, cell

3.2 LSTM的数学公式

对于时间步
t t t
LSTM的计算过程如下：

遗忘门:
f t = σ ( W f ⋅ [ h t − 1 , x t ] + b f ) f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) ft​=σ(Wf​⋅[ht−1​,xt​]+bf​)
输入门:
i t = σ ( W i ⋅ [ h t − 1 , x t ] + b i ) i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) it​=σ(Wi​⋅[ht−1​,xt​]+bi​)
候选值:
C ~ t = tanh ⁡ ( W C ⋅ [ h t − 1 , x t ] + b C ) \tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C) C~t​=tanh(WC​⋅[ht−1​,xt​]+bC​)
细胞状态更新:
C t = f t ∗ C t − 1 + i t ∗ C ~ t C_t = f_t * C_{t-1} + i_t * \tilde{C}_t Ct​=ft​∗Ct−1​+it​∗C~t​
输出门:
o t = σ ( W o ⋅ [ h t − 1 , x t ] + b o ) o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) ot​=σ(Wo​⋅[ht−1​,xt​]+bo​)
隐藏状态:
h t = o t ∗ tanh ⁡ ( C t ) h_t = o_t * \tanh(C_t) ht​=ot​∗tanh(Ct​)
其中：

  • sigma 是sigmoid函数

  • 表示逐元素乘法

  • [ h t − 1 , x t ] 表示向量连接 [h_{t-1}, x_t] 表示向量连接 [ht−1​,xt​]表示向量连接

4. LSTM的变体

4.1 双向LSTM（BiLSTM）

双向LSTM同时考虑前向和后向的序列信息

class BidirectionalLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers=1):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size, 
            hidden_size, 
            num_layers, 
            batch_first=True,
            bidirectional=True  # 双向LSTM
        )
        self.output_size = hidden_size * 2  # 双向输出维度翻倍
    
    def forward(self, x):
        output, (hidden, cell) = self.lstm(x)
        return output, hidden, cell

# 使用示例
bi_lstm = BidirectionalLSTM(input_size=10, hidden_size=20)
x = torch.randn(2, 5, 10)  # (batch, seq_len, input_size)
output, hidden, cell = bi_lstm(x)
print(f"输入形状: {x.shape}")
print(f"输出形状: {output.shape}")
print(f"隐藏状态形状: {hidden.shape}")

4.2 多层LSTM

class MultiLayerLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, dropout=0.2):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size, 
            hidden_size, 
            num_layers,
            batch_first=True,
            dropout=dropout if num_layers > 1 else 0
        )
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):
        output, (hidden, cell) = self.lstm(x)
        output = self.dropout(output)
        return output, hidden, cell

# 使用示例
multi_lstm = MultiLayerLSTM(input_size=10, hidden_size=20, num_layers=3)
x = torch.randn(2, 5, 10)
output, hidden, cell = multi_lstm(x)
print(f"多层LSTM输出形状: {output.shape}")

4.3 注意力机制增强的LSTM

class AttentionLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, attention_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.attention = nn.Linear(hidden_size, attention_size)
        self.context_vector = nn.Linear(attention_size, 1)
    
    def forward(self, x):
        # LSTM编码
        lstm_out, _ = self.lstm(x)
        
        # 计算注意力权重
        attention_weights = self.attention(lstm_out)
        attention_weights = torch.tanh(attention_weights)
        attention_weights = self.context_vector(attention_weights)
        attention_weights = F.softmax(attention_weights, dim=1)
        
        # 加权求和
        context = torch.sum(lstm_out * attention_weights, dim=1)
        
        return context, attention_weights

# 使用示例
att_lstm = AttentionLSTM(input_size=10, hidden_size=20, attention_size=15)
x = torch.randn(2, 5, 10)
context, attention = att_lstm(x)
print(f"上下文向量形状: {context.shape}")
print(f"注意力权重形状: {attention.shape}")

5. LSTM的应用实例

5.1 文本分类

class LSTMTextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_size, hidden_size, num_classes, num_layers=2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, 
                           batch_first=True, dropout=0.2)
        self.classifier = nn.Linear(hidden_size, num_classes)
        self.dropout = nn.Dropout(0.3)
    
    def forward(self, x):
        # 词嵌入
        embedded = self.embedding(x)
        
        # LSTM编码
        lstm_out, (hidden, _) = self.lstm(embedded)
        
        # 使用最后一个时间步的输出
        last_output = lstm_out[:, -1, :]
        
        # 分类
        output = self.dropout(last_output)
        output = self.classifier(output)
        
        return output

# 训练
def train_text_classifier():
    # 模拟数据
    vocab_size = 10000
    embed_size = 128
    hidden_size = 64
    num_classes = 5
    
    model = LSTMTextClassifier(vocab_size, embed_size, hidden_size, num_classes)
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    
    # 模拟训练数据
    batch_size = 32
    seq_len = 50
    
    for epoch in range(10):
        # 生成随机数据
        x = torch.randint(0, vocab_size, (batch_size, seq_len))
        y = torch.randint(0, num_classes, (batch_size,))
        
        # 前向传播
        outputs = model(x)
        loss = criterion(outputs, y)
        
        # 反向传播
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if epoch % 2 == 0:
            print(f'Epoch {epoch}, Loss: {loss.item():.4f}')

train_text_classifier()

5.2 时间序列预测

class LSTMTimeseriesPredictor(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=2):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, 
                           batch_first=True, dropout=0.2)
        self.linear = nn.Linear(hidden_size, output_size)
        self.dropout = nn.Dropout(0.1)
    
    def forward(self, x):
        lstm_out, _ = self.lstm(x)
        # 使用最后一个时间步的输出
        last_output = lstm_out[:, -1, :]
        output = self.dropout(last_output)
        output = self.linear(output)
        return output

# 时间序列预测示例
def timeseries_prediction_example():
    # 生成正弦波时间序列
    def generate_sine_wave(length, frequency=0.1):
        t = np.linspace(0, length, length)
        return np.sin(2 * np.pi * frequency * t)
    
    # 创建训练数据
    seq_length = 20
    pred_length = 1
    
    data = generate_sine_wave(1000)
    X, y = [], []
    
    for i in range(len(data) - seq_length - pred_length + 1):
        X.append(data[i:i+seq_length])
        y.append(data[i+seq_length:i+seq_length+pred_length])
    
    X = torch.FloatTensor(X).unsqueeze(-1)  # (samples, seq_len, features)
    y = torch.FloatTensor(y)  # (samples, pred_length)
    
    # 创建模型
    model = LSTMTimeseriesPredictor(input_size=1, hidden_size=50, output_size=1)
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    
    # 训练
    for epoch in range(100):
        outputs = model(X)
        loss = criterion(outputs, y)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if epoch % 20 == 0:
            print(f'Epoch {epoch}, Loss: {loss.item():.6f}')
    
    # 预测
    model.eval()
    with torch.no_grad():
        test_input = X[0:1]  # 取第一个样本
        prediction = model(test_input)
        print(f'真实值: {y[0].item():.4f}')
        print(f'预测值: {prediction.item():.4f}')

timeseries_prediction_example()

5.3 序列到序列模型

class Seq2SeqLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=2):
        super().__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        # 编码器
        self.encoder = nn.LSTM(input_size, hidden_size, num_layers, 
                              batch_first=True, dropout=0.2)
        
        # 解码器
        self.decoder = nn.LSTM(output_size, hidden_size, num_layers,
                              batch_first=True, dropout=0.2)
        
        # 输出层
        self.output_layer = nn.Linear(hidden_size, output_size)
        self.dropout = nn.Dropout(0.1)
    
    def forward(self, encoder_input, decoder_input):
        # 编码
        encoder_output, (hidden, cell) = self.encoder(encoder_input)
        
        # 解码
        decoder_output, _ = self.decoder(decoder_input, (hidden, cell))
        
        # 输出
        output = self.dropout(decoder_output)
        output = self.output_layer(output)
        
        return output

# 序列到序列
def seq2seq_example():
    # 模拟数据：输入序列长度10，输出序列长度5
    batch_size = 16
    input_seq_len = 10
    output_seq_len = 5
    input_size = 3
    output_size = 2
    
    # 生成随机数据
    encoder_input = torch.randn(batch_size, input_seq_len, input_size)
    decoder_input = torch.randn(batch_size, output_seq_len, output_size)
    target = torch.randn(batch_size, output_seq_len, output_size)
    
    # 创建模型
    model = Seq2SeqLSTM(input_size, hidden_size=64, output_size=output_size)
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    
    # 训练
    for epoch in range(50):
        output = model(encoder_input, decoder_input)
        loss = criterion(output, target)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if epoch % 10 == 0:
            print(f'Epoch {epoch}, Loss: {loss.item():.6f}')

seq2seq_example()

6. LSTM的训练技巧

6.1 梯度裁剪

def train_with_gradient_clipping(model, train_loader, optimizer, criterion, max_grad_norm=1.0):
    model.train()
    total_loss = 0
    
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        
        # 梯度裁剪
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)
        
        optimizer.step()
        total_loss += loss.item()
    
    return total_loss / len(train_loader)

6.2 学习率调度

import torch.optim as optim

def train_with_scheduler():
    model = MultiLayerLSTM(input_size=10, hidden_size=20, num_layers=2)
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    
    # 学习率调度器
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, mode='min', factor=0.5, patience=10, verbose=True
    )
    
    # 训练循环
    for epoch in range(100):
        # 模拟训练损失
        train_loss = 1.0 / (epoch + 1) + 0.1 * torch.randn(1).item()
        
        # 更新学习率
        scheduler.step(train_loss)
        
        if epoch % 20 == 0:
            print(f'Epoch {epoch}, LR: {optimizer.param_groups[0]["lr"]:.6f}, Loss: {train_loss:.4f}')

train_with_scheduler()

6.3 正则化技术

class RegularizedLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, dropout=0.2, weight_decay=1e-4):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, 
                           batch_first=True, dropout=dropout)
        self.dropout = nn.Dropout(dropout)
        self.weight_decay = weight_decay
    
    def forward(self, x):
        lstm_out, _ = self.lstm(x)
        output = self.dropout(lstm_out)
        return output
    
    def l2_regularization(self):
        """计算L2正则化项"""
        l2_reg = torch.tensor(0.)
        for param in self.parameters():
            l2_reg += torch.norm(param)
        return self.weight_decay * l2_reg

# 使用正则化训练
def train_with_regularization():
    model = RegularizedLSTM(input_size=10, hidden_size=20, num_layers=2)
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    criterion = nn.MSELoss()
    
    for epoch in range(50):
        # 模拟数据
        x = torch.randn(16, 20, 10)
        y = torch.randn(16, 20, 20)
        
        optimizer.zero_grad()
        output = model(x)
        
        # 计算损失（包含正则化项）
        loss = criterion(output, y) + model.l2_regularization()
        
        loss.backward()
        optimizer.step()
        
        if epoch % 10 == 0:
            print(f'Epoch {epoch}, Loss: {loss.item():.6f}')

train_with_regularization()

7. LSTM的优化与改进

7.1 门控循环单元（GRU）

GRU是LSTM的简化版本，只有两个门

class GRU(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers=1):
        super().__init__()
        self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
    
    def forward(self, x):
        output, hidden = self.gru(x)
        return output, hidden

# 比较LSTM和GRU
def compare_lstm_gru():
    input_size = 10
    hidden_size = 20
    seq_len = 15
    batch_size = 8
    
    lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
    gru = nn.GRU(input_size, hidden_size, batch_first=True)
    
    x = torch.randn(batch_size, seq_len, input_size)
    
    # LSTM前向传播
    lstm_out, (lstm_hidden, lstm_cell) = lstm(x)
    
    # GRU前向传播
    gru_out, gru_hidden = gru(x)
    
    print(f"LSTM输出形状: {lstm_out.shape}")
    print(f"GRU输出形状: {gru_out.shape}")
    print(f"LSTM参数量: {sum(p.numel() for p in lstm.parameters())}")
    print(f"GRU参数量: {sum(p.numel() for p in gru.parameters())}")

compare_lstm_gru()

7.2 注意力机制

class AttentionLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, attention_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.attention = nn.Linear(hidden_size, attention_size)
        self.context_vector = nn.Linear(attention_size, 1)
    
    def forward(self, x):
        # LSTM编码
        lstm_out, _ = self.lstm(x)
        
        # 计算注意力权重
        attention_weights = self.attention(lstm_out)
        attention_weights = torch.tanh(attention_weights)
        attention_weights = self.context_vector(attention_weights)
        attention_weights = F.softmax(attention_weights, dim=1)
        
        # 加权求和
        context = torch.sum(lstm_out * attention_weights, dim=1)
        
        return context, attention_weights

8. 相关论文与研究方向

8.1 经典论文

  1. “Long Short-Term Memory” (1997) - Hochreiter & Schmidhuber

    • LSTM的原始论文
    • 提出了门控机制解决梯度消失问题

  2. “Learning to Forget: Continual Prediction with LSTM” (2000) - Gers et al.

    • 改进了LSTM的遗忘门机制
    • 提出了窥视孔连接

  3. “Bidirectional LSTM Networks for Improved Phoneme Classification” (2005) - Graves & Schmidhuber

    • 提出了双向LSTM
    • 在语音识别中取得突破

8.2 现代发展

  1. “Sequence to Sequence Learning with Neural Networks” (2014) - Sutskever et al.

    • 将LSTM应用于序列到序列学习
    • 开启了神经机器翻译的时代

  2. “Neural Machine Translation by Jointly Learning to Align and Translate” (2014) - Bahdanau et al.

    • 提出了注意力机制
    • 显著改善了序列到序列模型

  3. “Attention Is All You Need” (2017) - Vaswani et al.

    • Transformer架构的提出
    • 对LSTM产生了重要影响

参考文献

  1. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.

  2. Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural computation, 12(10), 2451-2471.

  3. Graves, A., & Schmidhuber, J. (2005). Bidirectional LSTM networks for improved phoneme classification. International conference on artificial neural networks, 799-804.

  4. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 3104-3112.

  5. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

# 机器学习 # 人工智能
Logo
码道开发者社区

码道开发者社区，聚焦华为云码道 CodeArts 代码智能体，沉淀 Agent、Skill、鸿蒙开发实战内容，供开发者查阅资料、交流技术、分享工程实践

更多推荐

cover

AI 已经能直接改设计稿了:Windows部署Penpot，接入Codex MCP实测

avatar 码道开发者社区

【AI 业务流架构师】AI 业务流架构师方法论：Agentic AI 范式转移与业务编排思维

2026 年，Anthropic CEO Dario Amodei 在 Code with Claude 大会上的一个判断广为流传：一个人运营一家十亿美元级公司的概率是 70%–80%。这句话听起来像科幻小说的设定，但它背后是一个正在发生的结构性变化——AI 把"执行"的成本压到了历史低点，同时把"编排"的价值推到了前台。这意味着我们需要一套新的思考框架，来回答两个问题：一个人如何拥有一个公司的产

avatar 码道开发者社区
cover

Codex CLI实战 从一句模糊需求到可执行可测试可验收的开发任务把“帮我优化一下”变成 Codex 能执行、团队能复核、测试能证明的工程任务

avatar 码道开发者社区