Transformer技术详解：从注意力机制到现代大语言模型

摘要

Transformer是一种基于注意力机制的神经网络架构，彻底改变了自然语言处理和深度学习领域。本文深入解析Transformer的核心原理、注意力机制、网络架构以及从基础理论到现代大语言模型的发展历程，帮助读者全面理解这一革命性技术。

关键词： Transformer、注意力机制、自注意力、多头注意力、大语言模型

文章目录

1. 引言

Transformer是Google在2017年提出的一种革命性的神经网络架构，完全基于注意力机制，摒弃了传统的循环和卷积结构。这一架构不仅在机器翻译任务上取得了突破性成果，更成为了现代大语言模型（如GPT、BERT、T5等）的基础，彻底改变了自然语言处理领域。

1.1 Transformer的发展历程

  • 2017年: Transformer架构首次提出
  • 2018年: BERT和GPT的发布
  • 2019年: T5和GPT-2的突破
  • 2020年至今: GPT-3、ChatGPT等大语言模型的兴起

2. 注意力机制基础

2.1 注意力机制的数学原理

注意力机制的核心思想是让模型能够关注输入序列中的不同部分。对于查询（Query）、键（Key）和值（Value），注意力计算如下：

Attention ( Q , K , V ) = softmax ( Q K T d k ) V \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V Attention(Q,K,V)=softmax(dk​ ​QKT​)V

其中 d k d_k dk​ 是键的维度。

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import math

class Attention(nn.Module):
    def __init__(self, d_model, d_k, d_v):
        super().__init__()
        self.d_model = d_model
        self.d_k = d_k
        self.d_v = d_v
        
        # 线性变换层
        self.W_q = nn.Linear(d_model, d_k)
        self.W_k = nn.Linear(d_model, d_k)
        self.W_v = nn.Linear(d_model, d_v)
        
    def forward(self, query, key, value, mask=None):
        """
        query: (batch_size, seq_len, d_model)
        key: (batch_size, seq_len, d_model)
        value: (batch_size, seq_len, d_model)
        """
        batch_size, seq_len, _ = query.size()
        
        # 线性变换
        Q = self.W_q(query)  # (batch_size, seq_len, d_k)
        K = self.W_k(key)    # (batch_size, seq_len, d_k)
        V = self.W_v(value)  # (batch_size, seq_len, d_v)
        
        # 计算注意力分数
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        
        # 应用掩码（如果有）
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        
        # 计算注意力权重
        attention_weights = F.softmax(scores, dim=-1)
        
        # 计算加权和
        output = torch.matmul(attention_weights, V)
        
        return output, attention_weights

# 使用示例
d_model = 512
d_k = 64
d_v = 64
seq_len = 10
batch_size = 2

attention = Attention(d_model, d_k, d_v)

# 创建输入
query = torch.randn(batch_size, seq_len, d_model)
key = torch.randn(batch_size, seq_len, d_model)
value = torch.randn(batch_size, seq_len, d_model)

# 计算注意力
output, weights = attention(query, key, value)

print(f"输入形状: {query.shape}")
print(f"输出形状: {output.shape}")
print(f"注意力权重形状: {weights.shape}")

2.2 自注意力机制

自注意力是注意力机制的特殊情况，其中查询、键和值都来自同一个输入序列：

class SelfAttention(nn.Module):
    def __init__(self, d_model, d_k, d_v):
        super().__init__()
        self.attention = Attention(d_model, d_k, d_v)
        
    def forward(self, x, mask=None):
        """
        x: (batch_size, seq_len, d_model)
        """
        return self.attention(x, x, x, mask)

# 使用示例
self_attention = SelfAttention(d_model, d_k, d_v)
x = torch.randn(batch_size, seq_len, d_model)

output, weights = self_attention(x)
print(f"自注意力输出形状: {output.shape}")

3. 多头注意力机制

3.1 多头注意力的实现

多头注意力允许模型同时关注不同位置的不同表示子空间：

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        assert d_model % num_heads == 0
        
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        # 线性变换层
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        
    def forward(self, query, key, value, mask=None):
        batch_size, seq_len, d_model = query.size()
        
        # 线性变换并重塑为多头
        Q = self.W_q(query).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(key).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(value).view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
        
        # 计算注意力
        attention_output, attention_weights = self._scaled_dot_product_attention(Q, K, V, mask)
        
        # 连接多头
        attention_output = attention_output.transpose(1, 2).contiguous().view(
            batch_size, seq_len, d_model
        )
        
        # 最终线性变换
        output = self.W_o(attention_output)
        
        return output, attention_weights
    
    def _scaled_dot_product_attention(self, Q, K, V, mask=None):
        """缩放点积注意力"""
        d_k = Q.size(-1)
        
        # 计算注意力分数
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
        
        # 应用掩码
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        
        # 计算注意力权重
        attention_weights = F.softmax(scores, dim=-1)
        
        # 计算加权和
        output = torch.matmul(attention_weights, V)
        
        return output, attention_weights

# 使用示例
d_model = 512
num_heads = 8
seq_len = 10
batch_size = 2

multi_head_attention = MultiHeadAttention(d_model, num_heads)

# 创建输入
x = torch.randn(batch_size, seq_len, d_model)

# 计算多头注意力
output, weights = multi_head_attention(x, x, x)

print(f"多头注意力输出形状: {output.shape}")
print(f"注意力权重形状: {weights.shape}")

3.2 位置编码

由于Transformer没有循环结构，需要位置编码来提供序列位置信息：

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        
        # 创建位置编码矩阵
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        
        # 计算分母
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * 
                           (-math.log(10000.0) / d_model))
        
        # 计算位置编码
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        
        # 注册为缓冲区
        self.register_buffer('pe', pe.unsqueeze(0))
    
    def forward(self, x):
        """
        x: (batch_size, seq_len, d_model)
        """
        return x + self.pe[:, :x.size(1)]

# 使用示例
pos_encoding = PositionalEncoding(d_model)
x = torch.randn(batch_size, seq_len, d_model)

# 添加位置编码
x_with_pos = pos_encoding(x)
print(f"添加位置编码后的形状: {x_with_pos.shape}")

4. Transformer编码器

4.1 前馈网络

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff, dropout=0.1):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, x):
        return self.linear2(self.dropout(F.relu(self.linear1(x))))

# 使用示例
d_ff = 2048
feed_forward = FeedForward(d_model, d_ff)
x = torch.randn(batch_size, seq_len, d_model)

output = feed_forward(x)
print(f"前馈网络输出形状: {output.shape}")

4.2 编码器层

class EncoderLayer(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attention = MultiHeadAttention(d_model, num_heads)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)
        
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, x, mask=None):
        # 自注意力 + 残差连接 + 层归一化
        attn_output, _ = self.self_attention(x, x, x, mask)
        x = self.norm1(x + self.dropout(attn_output))
        
        # 前馈网络 + 残差连接 + 层归一化
        ff_output = self.feed_forward(x)
        x = self.norm2(x + self.dropout(ff_output))
        
        return x

# 使用示例
encoder_layer = EncoderLayer(d_model, num_heads, d_ff)
x = torch.randn(batch_size, seq_len, d_model)

output = encoder_layer(x)
print(f"编码器层输出形状: {output.shape}")

4.3 完整编码器

class TransformerEncoder(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, num_layers, dropout=0.1):
        super().__init__()
        self.layers = nn.ModuleList([
            EncoderLayer(d_model, num_heads, d_ff, dropout)
            for _ in range(num_layers)
        ])
        
    def forward(self, x, mask=None):
        for layer in self.layers:
            x = layer(x, mask)
        return x

# 使用示例
num_layers = 6
encoder = TransformerEncoder(d_model, num_heads, d_ff, num_layers)
x = torch.randn(batch_size, seq_len, d_model)

output = encoder(x)
print(f"编码器输出形状: {output.shape}")

5. Transformer解码器

5.1 掩码多头注意力

class MaskedMultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.multi_head_attention = MultiHeadAttention(d_model, num_heads)
        
    def forward(self, x, mask=None):
        # 创建因果掩码（下三角矩阵）
        if mask is None:
            seq_len = x.size(1)
            mask = torch.tril(torch.ones(seq_len, seq_len))
        
        return self.multi_head_attention(x, x, x, mask)

# 使用示例
masked_attention = MaskedMultiHeadAttention(d_model, num_heads)
x = torch.randn(batch_size, seq_len, d_model)

output, weights = masked_attention(x)
print(f"掩码多头注意力输出形状: {output.shape}")

5.2 解码器层

class DecoderLayer(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attention = MaskedMultiHeadAttention(d_model, num_heads)
        self.cross_attention = MultiHeadAttention(d_model, num_heads)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)
        
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.norm3 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, x, encoder_output, src_mask=None, tgt_mask=None):
        # 掩码自注意力
        attn_output, _ = self.self_attention(x, tgt_mask)
        x = self.norm1(x + self.dropout(attn_output))
        
        # 交叉注意力
        cross_attn_output, _ = self.cross_attention(x, encoder_output, encoder_output, src_mask)
        x = self.norm2(x + self.dropout(cross_attn_output))
        
        # 前馈网络
        ff_output = self.feed_forward(x)
        x = self.norm3(x + self.dropout(ff_output))
        
        return x

# 使用
decoder_layer = DecoderLayer(d_model, num_heads, d_ff)
x = torch.randn(batch_size, seq_len, d_model)
encoder_output = torch.randn(batch_size, seq_len, d_model)

output = decoder_layer(x, encoder_output)
print(f"解码器层输出形状: {output.shape}")

5.3 完整解码器

class TransformerDecoder(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, num_layers, dropout=0.1):
        super().__init__()
        self.layers = nn.ModuleList([
            DecoderLayer(d_model, num_heads, d_ff, dropout)
            for _ in range(num_layers)
        ])
        
    def forward(self, x, encoder_output, src_mask=None, tgt_mask=None):
        for layer in self.layers:
            x = layer(x, encoder_output, src_mask, tgt_mask)
        return x

# 使用示例
decoder = TransformerDecoder(d_model, num_heads, d_ff, num_layers)
x = torch.randn(batch_size, seq_len, d_model)
encoder_output = torch.randn(batch_size, seq_len, d_model)

output = decoder(x, encoder_output)
print(f"解码器输出形状: {output.shape}")

6. 完整Transformer模型

6.1 Transformer架构

class Transformer(nn.Module):
    def __init__(self, src_vocab_size, tgt_vocab_size, d_model, num_heads, d_ff, 
                 num_layers, max_len=5000, dropout=0.1):
        super().__init__()
        self.d_model = d_model
        
        # 嵌入层
        self.src_embedding = nn.Embedding(src_vocab_size, d_model)
        self.tgt_embedding = nn.Embedding(tgt_vocab_size, d_model)
        self.pos_encoding = PositionalEncoding(d_model, max_len)
        
        # 编码器和解码器
        self.encoder = TransformerEncoder(d_model, num_heads, d_ff, num_layers, dropout)
        self.decoder = TransformerDecoder(d_model, num_heads, d_ff, num_layers, dropout)
        
        # 输出层
        self.output_projection = nn.Linear(d_model, tgt_vocab_size)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, src, tgt, src_mask=None, tgt_mask=None):
        # 源序列嵌入
        src_embedded = self.dropout(self.pos_encoding(self.src_embedding(src) * math.sqrt(self.d_model)))
        
        # 目标序列嵌入
        tgt_embedded = self.dropout(self.pos_encoding(self.tgt_embedding(tgt) * math.sqrt(self.d_model)))
        
        # 编码
        encoder_output = self.encoder(src_embedded, src_mask)
        
        # 解码
        decoder_output = self.decoder(tgt_embedded, encoder_output, src_mask, tgt_mask)
        
        # 输出投影
        output = self.output_projection(decoder_output)
        
        return output
    
    def generate_square_subsequent_mask(self, sz):
        """生成因果掩码"""
        mask = torch.triu(torch.ones(sz, sz)) == 1
        mask = mask.transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

# 使用
src_vocab_size = 1000
tgt_vocab_size = 1000
max_len = 100

transformer = Transformer(
    src_vocab_size=src_vocab_size,
    tgt_vocab_size=tgt_vocab_size,
    d_model=d_model,
    num_heads=num_heads,
    d_ff=d_ff,
    num_layers=num_layers,
    max_len=max_len
)

# 创建输入
src = torch.randint(0, src_vocab_size, (batch_size, seq_len))
tgt = torch.randint(0, tgt_vocab_size, (batch_size, seq_len))

# 生成掩码
tgt_mask = transformer.generate_square_subsequent_mask(seq_len)

# 前向传播
output = transformer(src, tgt, tgt_mask=tgt_mask)
print(f"Transformer输出形状: {output.shape}")

6.2 训练和推理

class TransformerTrainer:
    def __init__(self, model, learning_rate=0.0001):
        self.model = model
        self.optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
        self.criterion = nn.CrossEntropyLoss(ignore_index=0)  # 忽略填充token
        
    def train_step(self, src, tgt):
        """训练一步"""
        self.model.train()
        self.optimizer.zero_grad()
        
        # 准备输入和目标
        tgt_input = tgt[:, :-1]  # 输入序列
        tgt_output = tgt[:, 1:]  # 目标序列
        
        # 生成掩码
        tgt_mask = self.model.generate_square_subsequent_mask(tgt_input.size(1))
        
        # 前向传播
        output = self.model(src, tgt_input, tgt_mask=tgt_mask)
        
        # 计算损失
        loss = self.criterion(output.reshape(-1, output.size(-1)), tgt_output.reshape(-1))
        
        # 反向传播
        loss.backward()
        self.optimizer.step()
        
        return loss.item()
    
    def generate(self, src, max_length=50, start_token=1, end_token=2):
        """生成序列"""
        self.model.eval()
        batch_size = src.size(0)
        device = src.device
        
        # 初始化目标序列
        tgt = torch.full((batch_size, 1), start_token, dtype=torch.long, device=device)
        
        with torch.no_grad():
            for _ in range(max_length):
                # 生成掩码
                tgt_mask = self.model.generate_square_subsequent_mask(tgt.size(1))
                
                # 前向传播
                output = self.model(src, tgt, tgt_mask=tgt_mask)
                
                # 获取下一个token
                next_token = output[:, -1, :].argmax(dim=-1, keepdim=True)
                tgt = torch.cat([tgt, next_token], dim=1)
                
                # 检查是否结束
                if (next_token == end_token).all():
                    break
        
        return tgt

# 使用
trainer = TransformerTrainer(transformer)

# 模拟训练数据
src = torch.randint(0, src_vocab_size, (batch_size, seq_len))
tgt = torch.randint(0, tgt_vocab_size, (batch_size, seq_len))

# 训练一步
loss = trainer.train_step(src, tgt)
print(f"训练损失: {loss:.4f}")

# 生成序列
generated = trainer.generate(src, max_length=20)
print(f"生成序列形状: {generated.shape}")

7. Transformer变体

7.1 BERT（双向编码器）

class BERT(nn.Module):
    def __init__(self, vocab_size, d_model, num_heads, d_ff, num_layers, max_len=512):
        super().__init__()
        self.d_model = d_model
        
        # 嵌入层
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(max_len, d_model)
        self.segment_embedding = nn.Embedding(2, d_model)  # 句子A和句子B
        
        # 编码器
        self.encoder = TransformerEncoder(d_model, num_heads, d_ff, num_layers)
        
        # 分类头
        self.classifier = nn.Linear(d_model, 2)  # 二分类任务
        
    def forward(self, input_ids, segment_ids=None, attention_mask=None):
        seq_len = input_ids.size(1)
        
        # 位置编码
        position_ids = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)
        
        # 嵌入
        token_emb = self.token_embedding(input_ids)
        pos_emb = self.position_embedding(position_ids)
        
        if segment_ids is not None:
            seg_emb = self.segment_embedding(segment_ids)
            embeddings = token_emb + pos_emb + seg_emb
        else:
            embeddings = token_emb + pos_emb
        
        # 编码
        encoded = self.encoder(embeddings, attention_mask)
        
        # 分类（使用[CLS] token）
        cls_output = encoded[:, 0, :]
        logits = self.classifier(cls_output)
        
        return logits, encoded

# 使用
vocab_size = 10000
bert = BERT(vocab_size, d_model, num_heads, d_ff, num_layers)

# 创建输入
input_ids = torch.randint(0, vocab_size, (batch_size, seq_len))
segment_ids = torch.randint(0, 2, (batch_size, seq_len))

# 前向传播
logits, encoded = bert(input_ids, segment_ids)
print(f"BERT分类输出形状: {logits.shape}")
print(f"BERT编码输出形状: {encoded.shape}")

7.2 GPT（生成式预训练）

class GPT(nn.Module):
    def __init__(self, vocab_size, d_model, num_heads, d_ff, num_layers, max_len=1024):
        super().__init__()
        self.d_model = d_model
        
        # 嵌入层
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(max_len, d_model)
        
        # 解码器层（只使用掩码自注意力）
        self.layers = nn.ModuleList([
            DecoderLayer(d_model, num_heads, d_ff)
            for _ in range(num_layers)
        ])
        
        # 输出层
        self.ln_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)
        
    def forward(self, input_ids, attention_mask=None):
        seq_len = input_ids.size(1)
        
        # 位置编码
        position_ids = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)
        
        # 嵌入
        token_emb = self.token_embedding(input_ids)
        pos_emb = self.position_embedding(position_ids)
        embeddings = token_emb + pos_emb
        
        # 生成因果掩码
        if attention_mask is None:
            attention_mask = torch.tril(torch.ones(seq_len, seq_len, device=input_ids.device))
        
        # 通过解码器层
        x = embeddings
        for layer in self.layers:
            x = layer(x, None, None, attention_mask)
        
        # 输出投影
        x = self.ln_f(x)
        logits = self.head(x)
        
        return logits
    
    def generate(self, input_ids, max_length=50, temperature=1.0, top_k=None):
        """生成文本"""
        self.eval()
        device = input_ids.device
        
        with torch.no_grad():
            for _ in range(max_length):
                # 前向传播
                logits = self.forward(input_ids)
                
                # 获取最后一个token的logits
                next_token_logits = logits[:, -1, :] / temperature
                
                # Top-k采样
                if top_k is not None:
                    top_k_logits, top_k_indices = torch.topk(next_token_logits, top_k)
                    next_token_logits = torch.full_like(next_token_logits, float('-inf'))
                    next_token_logits.scatter_(1, top_k_indices, top_k_logits)
                
                # 采样
                probs = F.softmax(next_token_logits, dim=-1)
                next_token = torch.multinomial(probs, num_samples=1)
                
                # 添加到序列
                input_ids = torch.cat([input_ids, next_token], dim=1)
        
        return input_ids

# 使用
gpt = GPT(vocab_size, d_model, num_heads, d_ff, num_layers)

# 创建输入
input_ids = torch.randint(0, vocab_size, (batch_size, 10))

# 前向传播
logits = gpt(input_ids)
print(f"GPT输出形状: {logits.shape}")

# 生成文本
generated = gpt.generate(input_ids, max_length=20)
print(f"生成序列形状: {generated.shape}")

8. 现代大语言模型

8.1 模型缩放

class ModelScaler:
    def __init__(self):
        self.scaling_laws = {
            "parameters": [1e6, 1e7, 1e8, 1e9, 1e10, 1e11, 1e12],
            "layers": [6, 12, 24, 48, 96, 192, 384],
            "heads": [8, 16, 32, 64, 128, 256, 512],
            "d_model": [512, 768, 1024, 2048, 4096, 8192, 16384]
        }
    
    def get_model_config(self, scale="small"):
        """获取不同规模的模型配置"""
        scales = {
            "tiny": 0,
            "small": 1,
            "medium": 2,
            "large": 3,
            "xl": 4,
            "xxl": 5,
            "xxxl": 6
        }
        
        idx = scales[scale]
        return {
            "d_model": self.scaling_laws["d_model"][idx],
            "num_heads": self.scaling_laws["heads"][idx],
            "num_layers": self.scaling_laws["layers"][idx],
            "d_ff": self.scaling_laws["d_model"][idx] * 4
        }

# 使用示例
scaler = ModelScaler()

# 获取不同规模的配置
small_config = scaler.get_model_config("small")
large_config = scaler.get_model_config("large")

print("小模型配置:", small_config)
print("大模型配置:", large_config)

8.2 训练优化技术

class TrainingOptimizer:
    def __init__(self, model, learning_rate=1e-4):
        self.model = model
        self.optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
        self.scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(self.optimizer, T_max=1000)
        
    def gradient_accumulation(self, loss, accumulation_steps=4):
        """梯度累积"""
        loss = loss / accumulation_steps
        loss.backward()
        
        if (self.step + 1) % accumulation_steps == 0:
            self.optimizer.step()
            self.scheduler.step()
            self.optimizer.zero_grad()
        
        self.step += 1
    
    def gradient_clipping(self, max_norm=1.0):
        """梯度裁剪"""
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm)
    
    def mixed_precision_training(self, loss, scaler):
        """混合精度训练"""
        scaler.scale(loss).backward()
        scaler.step(self.optimizer)
        scaler.update()
        self.optimizer.zero_grad()

# 使用
optimizer = TrainingOptimizer(transformer)

# 模拟训练循环
for epoch in range(10):
    for batch_idx in range(100):
        # 模拟损失
        loss = torch.randn(1, requires_grad=True)
        
        # 梯度累积
        optimizer.gradient_accumulation(loss, accumulation_steps=4)
        
        # 梯度裁剪
        optimizer.gradient_clipping(max_norm=1.0)
        
        if batch_idx % 10 == 0:
            print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")

9. 相关论文与研究方向

9.1 经典论文

  1. “Attention Is All You Need” (2017) - Vaswani et al.

    • Transformer的原始论文
    • 提出了基于注意力机制的架构

  2. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2018) - Devlin et al.

    • BERT的论文
    • 提出了双向编码器预训练

  3. “Language Models are Unsupervised Multitask Learners” (2019) - Radford et al.

    • GPT-2的论文
    • 展示了大规模语言模型的能力

9.2 现代发展

  1. “Training Compute-Optimal Large Language Models” (2022) - Hoffmann et al.

    • Chinchilla论文
    • 提出了计算最优的模型缩放定律

  2. “PaLM: Scaling Language Modeling with Pathways” (2022) - Chowdhery et al.

    • PaLM论文
    • 展示了大规模模型的能力

  3. “GPT-4 Technical Report” (2023) - OpenAI

    • GPT-4技术报告
    • 展示了多模态大语言模型

参考文献

  1. Vaswani, A., et al. (2017). Attention is all you need. Advances in neural information processing systems, 30, 5998-6008.

  2. Devlin, J., et al. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

  3. Radford, A., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.

  4. Hoffmann, J., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

  5. Chowdhery, A., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

# 机器学习 # transformer # 人工智能
Logo
码道开发者社区

码道开发者社区，聚焦华为云码道 CodeArts 代码智能体，沉淀 Agent、Skill、鸿蒙开发实战内容，供开发者查阅资料、交流技术、分享工程实践

更多推荐

cover

AI 已经能直接改设计稿了:Windows部署Penpot，接入Codex MCP实测

avatar 码道开发者社区

【AI 业务流架构师】AI 业务流架构师方法论：Agentic AI 范式转移与业务编排思维

2026 年，Anthropic CEO Dario Amodei 在 Code with Claude 大会上的一个判断广为流传：一个人运营一家十亿美元级公司的概率是 70%–80%。这句话听起来像科幻小说的设定，但它背后是一个正在发生的结构性变化——AI 把"执行"的成本压到了历史低点，同时把"编排"的价值推到了前台。这意味着我们需要一套新的思考框架，来回答两个问题：一个人如何拥有一个公司的产

avatar 码道开发者社区
cover

Codex CLI实战 从一句模糊需求到可执行可测试可验收的开发任务把“帮我优化一下”变成 Codex 能执行、团队能复核、测试能证明的工程任务

avatar 码道开发者社区