we encounter words like “mask“ “embedding“ alot when reading ai papers,what are these mean?

interesting quanta

706人浏览 · 2025-09-12 16:33:55

interesting quanta · 2025-09-12 16:33:55 发布

🔍 1. Mask – Hiding Information on Purpose

🎯 What is a mask?

A mask is a binary tensor (0s and 1s) used to hide or reveal parts of your data.

🧠 Why do we mask?

Use Case	Purpose
Padding	Ignore padded tokens in variable-length sequences
Future tokens	Prevent the model from “cheating” by seeing future words
Missing data	Ignore corrupted or missing inputs

🧪 Example: Padding Mask in PyTorch

import torch

# Suppose we have padded sequences
seq = torch.tensor([
    [1, 2, 3, 0, 0],  # "i love ai"
    [4, 5, 0, 0, 0]   # "you rock"
])

# Create a mask: 1 = real token, 0 = pad
mask = (seq != 0).long()

print(mask)
# tensor([[1, 1, 1, 0, 0],
#         [1, 1, 0, 0, 0]])

🧠 Attention Mask (Transformer)

In transformers, we use a lower triangular mask to block future tokens:

def generate_square_subsequent_mask(sz):
    mask = torch.triu(torch.ones(sz, sz) == 1).transpose(0, 1)
    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
    return mask

print(generate_square_subsequent_mask(5))

Output:

tensor([[0., -inf, -inf, -inf, -inf],
        [0., 0., -inf, -inf, -inf],
        [0., 0., 0., -inf, -inf],
        [0., 0., 0., 0., -inf],
        [0., 0., 0., 0., 0.]])

🔍 2. Embedding – Turning Words into Vectors

🎯 What is an embedding?

An embedding is a dense, low-dimensional vector that represents a discrete symbol (like a word, user, product, etc.) in a continuous space.

🧠 Why not use one-hot vectors?

One-hot	Embedding
Sparse, huge	Dense, compact
No similarity	Similar words = close vectors
1M vocab = 1M dims	1M vocab = 300 dims

🧪 Example: Word Embedding in PyTorch

import torch.nn as nn

vocab_size = 10000
embed_dim = 300

embedding = nn.Embedding(vocab_size, embed_dim)

# Input: word indices
input_ids = torch.tensor([1, 5, 999])  # 3 words
embedded = embedding(input_ids)        # shape: [3, 300]

🧠 Visual: Word Embeddings in 2D (PCA)

Imagine these words in 2D space:

king     → (0.2, 0.8)
queen    → (0.3, 0.9)
man      → (0.1, 0.5)
woman    → (0.2, 0.6)

You can do arithmetic:

king - man + woman ≈ queen

This is semantic arithmetic — possible because embeddings capture meaning.

🔍 3. Masking + Embedding Together

In transformers, embedding + masking is everywhere.

🧪 Full Transformer Input Pipeline

import torch.nn as nn

class SimpleTransformerBlock(nn.Module):
    def __init__(self, vocab_size, d_model, max_len=512):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)
        self.pos_embed = nn.Embedding(max_len, d_model)
        self.transformer = nn.TransformerEncoderLayer(d_model=d_model, nhead=8)

    def forward(self, input_ids, mask=None):
        seq_len = input_ids.size(1)
        pos = torch.arange(seq_len).unsqueeze(0).to(input_ids.device)

        x = self.embed(input_ids) + self.pos_embed(pos)
        x = x.transpose(0, 1)  # Transformer expects (seq_len, batch, dim)
        out = self.transformer(x, src_key_padding_mask=mask)
        return out

🔍 4. Types of Embeddings You’ll See

Type	What it embeds	Example
Word	Tokens	“cat” → [0.2, -0.1, …]
Position	Position in sentence	Position 3 → [0.1, 0.4, …]
Segment	Sentence A vs B	“A” → [1, 0], “B” → [0, 1]
Token Type	For BERT	Same as segment
User/Item	Recommender systems	User #123 → [0.3, -0.2, …]
Image patch	Vision Transformer	16x16 patch → vector

🔍 5. Interactive Analogy

Concept	Real-world analogy
Embedding	A name tag that summarizes a person’s personality in 300 numbers
Mask	A sticky note over part of a page saying “don’t read this”

🔍 6. Advanced Tips

Pretrained embeddings: Word2Vec, GloVe, FastText
Contextual embeddings: ELMo, BERT, GPT (same word = different vector per context)
Learnable vs frozen: Sometimes you freeze embeddings (no training), sometimes you fine-tune them