# Resume Deep Dive: From First Principles This is the technical knowledge base behind my resume. Every concept, every architecture, every technique—explained from first principles. If you're a junior engineer looking to understand modern ML systems, or if you're preparing for technical interviews and want to go deep, this is for you. The goal isn't just to explain what these techniques do, but why they exist, what problems they solve, and how they connect to production realities. We'll build from probability theory to production-scale distributed training, from signal processing to state-of-the-art speech recognition. > **Disclaimer**: This post was written with the help of LLMs, with content carefully curated based on papers I've read and systems I've built. If you notice any errors, please send a note to [moghan.anand@gmail.com](mailto:moghan.anand@gmail.com). ## Foundations of Machine Learning Before diving into neural networks and transformers, we need to understand the probabilistic foundations that everything rests on. ### Probability and Bayes' Theorem **References**: [1, 2] Every classifier, every generative model, every alignment objective ultimately comes back to probability. Let's start with the basics. A **sample space** is the set of all possible outcomes. For a coin flip, it's \{H, T\}. For a speech frame, it's all possible acoustic feature vectors in $\mathbb{R}^d$. An **event** is a subset of the sample space—"the coin lands heads" or "the frame contains a vowel sound." **Probability axioms** (Kolmogorov): 1. $P(A) \geq 0$ for any event $A$ 2. $P(\Omega) = 1$ where $\Omega$ is the sample space 3. For disjoint events $A_1, A_2, ...$: $P(A_1 \cup A_2 \cup ...) = P(A_1) + P(A_2) + ...$ From these axioms, we derive **conditional probability**: $P(A|B) = \frac{P(A \cap B)}{P(B)}$ **Bayes' Theorem** follows directly from the definition of conditional probability: $$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$ This isn't just a formula—it's the foundation of how we reason under uncertainty. In speech recognition, this becomes: $$ P(\text{words}|\text{audio}) = \frac{P(\text{audio}|\text{words}) \cdot P(\text{words})}{P(\text{audio})} $$ The terms have names: - **Prior** $P(\text{words})$: Our belief before seeing data (language model) - **Likelihood** $P(\text{audio}|\text{words})$: How likely this audio given these words (acoustic model) - **Posterior** $P(\text{words}|\text{audio})$: Our updated belief after seeing data - **Evidence** $P(\text{audio})$: Normalizing constant (often ignored in practice) **Maximum Likelihood Estimation (MLE)** finds parameters that maximize $P(\text{data}|\theta)$. For a Gaussian distribution with unknown mean $\mu$ and known variance $\sigma^2$: $$ \mu_{\text{MLE}} = \arg\max_\mu \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right) $$ Taking the log (monotonic transformation preserves the argmax): $$ \mu_{\text{MLE}} = \arg\max_\mu \sum_{i=1}^n \left[-\frac{(x_i - \mu)^2}{2\sigma^2}\right] = \frac{1}{n}\sum_{i=1}^n x_i $$ The MLE for the mean is just the sample average—intuitive and elegant. **Maximum A Posteriori (MAP)** adds a prior: $\theta_{\text{MAP}} = \arg\max_\theta P(\theta|\text{data}) = \arg\max_\theta P(\text{data}|\theta)P(\theta)$ With a Gaussian prior $P(\theta) \propto \exp(-\lambda\theta^2)$, MAP becomes: $$ \theta_{\text{MAP}} = \arg\max_\theta \left[\log P(\text{data}|\theta) - \lambda\theta^2\right] $$ This is exactly L2 regularization! Adding a prior is equivalent to regularization—a fundamental connection between Bayesian inference and modern ML. ### The Bayes Classifier and Generative vs. Discriminative Models **References**: [3, 4] The **Bayes optimal classifier** minimizes expected 0-1 loss by assigning each input to the class with highest posterior probability: $$ y^* = \arg\max_y P(y|x) = \arg\max_y P(x|y)P(y) $$ No classifier can do better in expectation—this is the theoretical ceiling. **Generative models** learn $P(x|y)$ and $P(y)$ separately, then apply Bayes' rule. Example: Naive Bayes assumes feature independence: $$ P(x|y) = \prod_{i=1}^d P(x_i|y) $$ This "naive" assumption is often wrong, but the model still works surprisingly well. Why? Because we only care about the decision boundary, not the exact probabilities. Even if $P(x|y)$ is misspecified, the argmax can still be correct. **Discriminative models** directly learn $P(y|x)$, bypassing the need to model the data distribution. Logistic regression is the canonical example: $$ P(y=1|x) = \sigma(w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}} $$ **The trade-off**: With finite data, generative models can be more data-efficient (stronger assumptions = more inductive bias) but suffer when assumptions are wrong. Discriminative models are more flexible but need more data. This tension echoes throughout the field. GMM-HMM acoustic models (generative) dominated speech recognition for decades. Then neural networks (discriminative) took over. Now we're seeing a return to generative models with diffusion and autoregressive transformers—but at massive scale where data efficiency matters less. ### Linear Regression: The Simplest Learning Machine **References**: [5, 6] The linear model: $y = w^T x + b$. Simple, but it teaches us everything we need to know about learning. **Mean Squared Error (MSE)** isn't arbitrary—it's the MLE solution for Gaussian noise. If we assume $y = w^T x + \epsilon$ where $\epsilon \sim \mathcal{N}(0, \sigma^2)$, then: $$ P(y|x, w) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y - w^T x)^2}{2\sigma^2}\right) $$ Maximizing log-likelihood: $$ \max_w \sum_{i=1}^n \log P(y_i|x_i, w) = \max_w \sum_{i=1}^n -\frac{(y_i - w^T x_i)^2}{2\sigma^2} = \min_w \sum_{i=1}^n (y_i - w^T x_i)^2 $$ MSE is the MLE solution for Gaussian noise—not a random choice. **Normal equations** give the closed-form solution by setting $\nabla_w L = 0$: $$ w^* = (X^T X)^{-1} X^T y $$ This works when $X^T X$ is invertible and $n$ is small. For millions of examples, we need iterative optimization. **Gradient descent**: $w \leftarrow w - \alpha \nabla_w L$ For MSE, the gradient is: $$ \nabla_w L = \nabla_w \frac{1}{n}\sum_{i=1}^n (y_i - w^T x_i)^2 = -\frac{2}{n}\sum_{i=1}^n (y_i - w^T x_i)x_i $$ Update rule: $w \leftarrow w + \frac{\alpha}{n}\sum_{i=1}^n (y_i - w^T x_i)x_i$ **Variants**: - **Batch GD**: Use all $n$ examples per update (slow but stable) - **Stochastic GD (SGD)**: Use one example per update (fast but noisy) - **Mini-batch GD**: Use $b$ examples per update (best of both worlds) Learning rate $\alpha$ is critical: too large → divergence, too small → slow convergence. ### Adam Optimizer: The Modern Standard **References**: [5a, 5b] While SGD with momentum works, it requires careful learning rate tuning. Adam (Adaptive Moment Estimation) adapts the learning rate per parameter automatically—it's the default optimizer for most deep learning. **The core idea**: Combine momentum (first moment) with adaptive learning rates (second moment). Each parameter gets its own learning rate based on the history of gradients. **Adam update rule**: $$ \begin{aligned} m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t \quad \text{(first moment: momentum)} \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \quad \text{(second moment: variance)} \\ \hat{m}_t &= \frac{m_t}{1 - \beta_1^t} \quad \text{(bias correction)} \\ \hat{v}_t &= \frac{v_t}{1 - \beta_2^t} \quad \text{(bias correction)} \\ \theta_t &= \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \end{aligned} $$ **Breaking it down**: 1. **First moment $m_t$**: Exponential moving average of gradients (like momentum). $\beta_1 = 0.9$ is standard—90% of previous momentum, 10% current gradient. 2. **Second moment $v_t$**: Exponential moving average of squared gradients (measures variance). $\beta_2 = 0.999$ is standard—very slow decay, tracks long-term gradient magnitude. 3. **Bias correction**: At initialization, $m_0 = 0$ and $v_0 = 0$. Without correction, early updates are biased toward zero. Dividing by $(1 - \beta^t)$ corrects this—as $t \to \infty$, $(1 - \beta^t) \to 1$ and correction disappears. 4. **Adaptive learning rate**: Each parameter's effective learning rate is $\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}$. Parameters with large gradients get smaller learning rates (prevents overshooting). Parameters with small gradients get larger learning rates (speeds up convergence). **Why bias correction matters**: Without it, the first few updates are tiny because $m_t$ and $v_t$ start at zero. With correction: $ \hat{m}_1 = \frac{(1-\beta_1)g_1}{1-\beta_1} = g_1 $ The first update uses the full gradient—no artificial dampening. **Hyperparameters**: - $\alpha = 0.001$ (learning rate): Standard default, often works without tuning - $\beta_1 = 0.9$ (momentum decay): Controls how much history to keep for gradient direction - $\beta_2 = 0.999$ (variance decay): Controls how much history to keep for gradient magnitude - $\epsilon = 10^{-8}$ (numerical stability): Prevents division by zero **Adam vs. SGD with momentum**: | Optimizer | Pros | Cons | |-----------|------|------| | SGD + momentum | Better generalization on some tasks, simpler | Requires learning rate tuning, slower convergence | | Adam | Fast convergence, works out-of-the-box, per-parameter learning rates | Can overfit on small datasets, sometimes worse final performance | **Production reality**: Adam is the default for training transformers, speech models, and most modern architectures. For computer vision (ResNets, ViTs), SGD with momentum sometimes generalizes better, but Adam converges faster. The trade-off: Adam gets you to 95% accuracy quickly, SGD might get you to 96% with careful tuning. **Variants**: - **AdamW**: Adam with decoupled weight decay (better regularization) - **AdaGrad**: Accumulates all past squared gradients (learning rate decays too aggressively) - **RMSProp**: Like Adam but without momentum (used in some RL settings) **Why Adam works for speech**: Speech models have millions of parameters with vastly different gradient magnitudes. Mel filterbank features have different scales than attention weights. Adam's per-parameter learning rates handle this heterogeneity automatically—no manual feature scaling needed. **Bias-Variance Trade-off**: Generalization error decomposes as: $$ \mathbb{E}[(y - \hat{f}(x))^2] = \text{Bias}[\hat{f}(x)]^2 + \text{Var}[\hat{f}(x)] + \sigma^2 $$ - **Bias**: Error from wrong assumptions (underfitting) - **Variance**: Error from sensitivity to training data (overfitting) - **Irreducible error**: Noise in the data Simple models have high bias, low variance. Complex models have low bias, high variance. The art is finding the sweet spot. **Regularization** controls this trade-off: - **L2 (Ridge)**: $L = \text{MSE} + \lambda \|w\|^2$ (equivalent to Gaussian prior) - **L1 (Lasso)**: $L = \text{MSE} + \lambda \|w\|_1$ (induces sparsity) L1 produces sparse solutions because the constraint region is a diamond—the optimal point often lies on an axis where some weights are exactly zero. ### Logistic Regression and Classification **References**: [7, 8] For classification, we need outputs in $(0,1)$ to represent probabilities. The **sigmoid function** does this: $$ \sigma(z) = \frac{1}{1 + e^{-z}} $$ Properties: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ (clean gradient), $\sigma(0) = 0.5$, $\sigma(\infty) = 1$, $\sigma(-\infty) = 0$. **Binary cross-entropy loss** comes from the negative log-likelihood of a Bernoulli distribution: $$ L = -\frac{1}{n}\sum_{i=1}^n [y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)] $$ Why not MSE for classification? MSE has flat gradients when predictions are wrong (sigmoid saturation), making training unstable. Cross-entropy has strong gradients when wrong—exactly what we want. **Cross-Entropy from Information Theory**: The KL Divergence Perspective Cross-entropy isn't arbitrary—it comes from information theory. The **KL divergence** measures how different one probability distribution is from another: $ D_{\text{KL}}(P \| Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)} = \sum_x P(x) \log P(x) - \sum_x P(x) \log Q(x) $ The first term is the **entropy** of $P$ (constant for fixed true distribution). The second term is the **cross-entropy** between $P$ and $Q$: $ H(P, Q) = -\sum_x P(x) \log Q(x) $ **Minimizing cross-entropy = minimizing KL divergence**: Since entropy $H(P)$ is constant, minimizing $H(P, Q)$ is equivalent to minimizing $D_{\text{KL}}(P \| Q)$. We're making our predicted distribution $Q$ as close as possible to the true distribution $P$. **For classification**: $P$ is the true label distribution (one-hot: $P(y=k) = 1$ for correct class, $0$ otherwise). $Q$ is the model's predicted distribution (softmax output). Cross-entropy loss: $ L = -\sum_k P(y=k) \log Q(y=k) = -\log Q(y=\text{true}) $ This is exactly the negative log-likelihood—we're maximizing the probability assigned to the correct class. **Intuition**: Cross-entropy penalizes confident wrong predictions heavily. If the model predicts $Q(y=\text{wrong}) = 0.99$, the loss is $-\log(0.01) \approx 4.6$. If it predicts $Q(y=\text{correct}) = 0.99$, the loss is $-\log(0.99) \approx 0.01$. The gradient pushes the model toward the correct answer with force proportional to how wrong it is. **Multiclass: Softmax** extends to $K$ classes: $$ P(y=k|x) = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}} $$ where $z = Wx + b$ are the logits. The gradient of cross-entropy with softmax is remarkably clean: $$ \frac{\partial L}{\partial z_k} = \hat{y}_k - y_k $$ This is why softmax + cross-entropy is the standard for classification—the math just works. **Softmax Temperature**: Controlling Confidence and Exploration The standard softmax can be generalized with a temperature parameter $T$: $ P(y=k|x) = \frac{e^{z_k/T}}{\sum_{j=1}^K e^{z_j/T}} $ **Temperature effects**: - **$T = 1$**: Standard softmax (default) - **$T \to 0$**: Approaches argmax (one-hot, deterministic). High confidence, low diversity. - **$T \to \infty$**: Approaches uniform distribution. Low confidence, high diversity. **High temperature ($T > 1$)**: "Softens" the distribution—makes probabilities more uniform: ``` Logits: [2.0, 1.0, 0.5] T=1.0: [0.659, 0.242, 0.099] (confident) T=2.0: [0.506, 0.307, 0.187] (less confident) T=5.0: [0.391, 0.331, 0.278] (nearly uniform) ``` **Low temperature ($T < 1$)**: "Sharpens" the distribution—makes the model more confident: ``` Logits: [2.0, 1.0, 0.5] T=1.0: [0.659, 0.242, 0.099] T=0.5: [0.842, 0.114, 0.044] (more confident) T=0.1: [0.997, 0.002, 0.001] (nearly one-hot) ``` **Why temperature matters**: 1. **Knowledge distillation**: Train a small "student" model to match a large "teacher" model's soft predictions (high temperature). The soft targets contain more information than hard labels. 2. **Sampling diversity**: For text generation, high temperature increases diversity (creative writing). Low temperature increases quality (factual answers). 3. **Alignment (DPO)**: The $\beta$ parameter in DPO is effectively an inverse temperature—it controls how much the policy can deviate from the reference model. 4. **Calibration**: Models are often overconfident. Temperature scaling (learned on a validation set) can calibrate probabilities to match true frequencies. **DPO connection**: In Direct Preference Optimization, the $\beta$ parameter plays the role of inverse temperature: $ P(y_w \succ y_l | x) = \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right) $ Higher $\beta$ → sharper preference distribution → policy stays closer to reference. Lower $\beta$ → softer preferences → policy can deviate more. **Production use**: For ASR beam search, we use temperature to control hypothesis diversity. Low temperature (0.5-0.8) for final decoding (want best hypothesis). High temperature (1.2-1.5) for generating diverse N-best lists for rescoring. **Decision boundaries**: Logistic regression produces linear decision boundaries in feature space. For $P(y=1|x) = \sigma(w^T x + b)$, the boundary is where $w^T x + b = 0$—a hyperplane. This limitation motivates nonlinear models. We need to transform the feature space. ### The Perceptron: Where Neural Networks Begin **References**: [9, 10] The perceptron (1958) is the ancestor of every neural network. It's a binary classifier: $\hat{y} = \text{sign}(w^T x + b)$. **Perceptron learning algorithm**: If a point is misclassified, update: $$ w \leftarrow w + y \cdot x $$ This moves the hyperplane toward the misclassified point. The **Perceptron Convergence Theorem** guarantees convergence for linearly separable data in finite steps. **The XOR problem**: No single hyperplane can separate XOR: ``` (0,0) → 0 (0,1) → 1 (1,0) → 1 (1,1) → 0 ``` This killed early neural network research (the "AI winter" of the 1970s). Minsky and Papert's book "Perceptrons" (1969) proved this formally. **Why this matters**: XOR requires a nonlinear decision boundary. A two-layer network can solve it: ``` Hidden layer: h1 = σ(x1 + x2), h2 = σ(x1 + x2 - 1.5) Output: y = σ(h1 - h2) ``` This is the conceptual leap to multi-layer perceptrons (MLPs). Depth enables nonlinearity. ### Multi-Layer Perceptron and Backpropagation **References**: [11, 12, 13] An $L$-layer network is a composition of linear transforms and nonlinearities: $$ h^{(l)} = \sigma(W^{(l)} h^{(l-1)} + b^{(l)}) $$ Each layer learns a new representation. The first layer might learn edges, the second layer combines edges into shapes, the third layer combines shapes into objects. **Activation functions**: - **ReLU**: $f(x) = \max(0, x)$ — simple, no saturation, but can "die" (gradient = 0 for $x < 0$) - **Sigmoid**: $f(x) = \frac{1}{1+e^{-x}}$ — saturates (vanishing gradients), rarely used in hidden layers - **Tanh**: $f(x) = \tanh(x)$ — zero-centered (better than sigmoid), still saturates - **GELU**: $f(x) = x \cdot \Phi(x)$ — smooth approximation of ReLU, used in BERT/GPT **Backpropagation** is just the chain rule applied systematically. For a 2-layer network: $$ \frac{\partial L}{\partial W^{(1)}} = \frac{\partial L}{\partial h^{(2)}} \cdot \frac{\partial h^{(2)}}{\partial h^{(1)}} \cdot \frac{\partial h^{(1)}}{\partial W^{(1)}} $$ The key insight: gradients flow backward through the network. We compute them layer by layer, reusing intermediate results. **The Calculus of Backpropagation: Linear Layer Gradient Derivation** Consider a single linear layer: $Y = XW + b$ where: - $X \in \mathbb{R}^{B \times D_{\text{in}}}$ (batch of inputs) - $W \in \mathbb{R}^{D_{\text{in}} \times D_{\text{out}}}$ (weight matrix) - $b \in \mathbb{R}^{D_{\text{out}}}$ (bias vector) - $Y \in \mathbb{R}^{B \times D_{\text{out}}}$ (output) Given $\frac{\partial L}{\partial Y} \in \mathbb{R}^{B \times D_{\text{out}}}$ (gradient from the next layer), we need: 1. $\frac{\partial L}{\partial W}$ to update weights 2. $\frac{\partial L}{\partial X}$ to pass gradients to previous layer 3. $\frac{\partial L}{\partial b}$ to update bias **Gradient with respect to weights**: $$ \frac{\partial L}{\partial W_{ij}} = \sum_{b=1}^B \frac{\partial L}{\partial Y_{bj}} \cdot \frac{\partial Y_{bj}}{\partial W_{ij}} $$ Since $Y_{bj} = \sum_k X_{bk} W_{kj} + b_j$, we have $\frac{\partial Y_{bj}}{\partial W_{ij}} = X_{bi}$. Therefore: $$ \frac{\partial L}{\partial W_{ij}} = \sum_{b=1}^B \frac{\partial L}{\partial Y_{bj}} \cdot X_{bi} $$ In matrix form: $$ \frac{\partial L}{\partial W} = X^T \frac{\partial L}{\partial Y} $$ **Dimension check**: $(D_{\text{in}} \times B) \times (B \times D_{\text{out}}) = (D_{\text{in}} \times D_{\text{out}})$ ✓ **Gradient with respect to input**: $$ \frac{\partial L}{\partial X_{bi}} = \sum_{j=1}^{D_{\text{out}}} \frac{\partial L}{\partial Y_{bj}} \cdot \frac{\partial Y_{bj}}{\partial X_{bi}} $$ Since $\frac{\partial Y_{bj}}{\partial X_{bi}} = W_{ij}$: $$ \frac{\partial L}{\partial X_{bi}} = \sum_{j=1}^{D_{\text{out}}} \frac{\partial L}{\partial Y_{bj}} \cdot W_{ij} $$ In matrix form: $$ \frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} W^T $$ **Dimension check**: $(B \times D_{\text{out}}) \times (D_{\text{out}} \times D_{\text{in}}) = (B \times D_{\text{in}})$ ✓ **Gradient with respect to bias**: $$ \frac{\partial L}{\partial b_j} = \sum_{b=1}^B \frac{\partial L}{\partial Y_{bj}} $$ In matrix form (sum over batch dimension): $$ \frac{\partial L}{\partial b} = \sum_{b=1}^B \frac{\partial L}{\partial Y}_{b,:} $$ **Vectorized backward pass implementation**: ```python import numpy as np def linear_backward(grad_output, X, W): """ Backward pass for linear layer Y = XW + b. Args: grad_output: Gradient from next layer, dL/dY (Batch, Out_Dim) X: Input from forward pass (Batch, In_Dim) W: Weight matrix (In_Dim, Out_Dim) Returns: grad_W: Gradient w.r.t. weights (In_Dim, Out_Dim) grad_X: Gradient w.r.t. input (Batch, In_Dim) grad_b: Gradient w.r.t. bias (Out_Dim,) Matrix transpose operations for dimension matching: - grad_W: X.T @ grad_output (In_Dim, Batch) @ (Batch, Out_Dim) = (In_Dim, Out_Dim) ✓ - grad_X: grad_output @ W.T (Batch, Out_Dim) @ (Out_Dim, In_Dim) = (Batch, In_Dim) ✓ - grad_b: sum over batch dimension (Batch, Out_Dim) -> (Out_Dim,) ✓ Inference vs. Training: - Training: Compute and store X, W for backward pass - Inference: No backward pass needed, discard intermediate activations """ # Gradient w.r.t. weights: X^T @ grad_output # Shape: (In_Dim, Batch) @ (Batch, Out_Dim) = (In_Dim, Out_Dim) grad_W = np.dot(X.T, grad_output) # Gradient w.r.t. input (for previous layers): grad_output @ W^T # Shape: (Batch, Out_Dim) @ (Out_Dim, In_Dim) = (Batch, In_Dim) grad_X = np.dot(grad_output, W.T) # Gradient w.r.t. bias: sum over batch dimension # Shape: (Batch, Out_Dim) -> (Out_Dim,) grad_b = np.sum(grad_output, axis=0) return grad_W, grad_X, grad_b def linear_forward_backward_example(): """ Complete example: forward and backward pass for a linear layer. """ # Setup batch_size, in_dim, out_dim = 32, 128, 64 X = np.random.randn(batch_size, in_dim) W = np.random.randn(in_dim, out_dim) * 0.01 b = np.zeros(out_dim) # Forward pass Y = np.dot(X, W) + b # (32, 64) # Assume some loss and gradient from next layer grad_output = np.random.randn(batch_size, out_dim) # Backward pass grad_W, grad_X, grad_b = linear_backward(grad_output, X, W) # Gradient descent update learning_rate = 0.01 W -= learning_rate * grad_W b -= learning_rate * grad_b return grad_W, grad_X, grad_b ``` **Why matrix transposes are necessary**: The chain rule requires matching dimensions. For $\frac{\partial L}{\partial W}$, we need to contract the batch dimension—this requires $X^T$ on the left. For $\frac{\partial L}{\partial X}$, we need to contract the output dimension—this requires $W^T$ on the right. **Computational complexity**: - Forward pass: $O(B \cdot D_{\text{in}} \cdot D_{\text{out}})$ for matrix multiplication - Backward pass: $O(B \cdot D_{\text{in}} \cdot D_{\text{out}})$ for $\frac{\partial L}{\partial W}$ and $\frac{\partial L}{\partial X}$ - Memory: Must store $X$ and $W$ from forward pass for backward pass **Production optimization**: In frameworks like PyTorch, these operations are fused and optimized with CUDA kernels. The naive NumPy implementation above is for understanding—production code uses cuBLAS for matrix multiplications and custom kernels for element-wise operations. **Vanishing and exploding gradients**: Gradients multiply through layers. If weights are $< 1$, gradients shrink exponentially (vanishing). If $> 1$, they grow exponentially (exploding). For an $L$-layer network, the gradient magnitude scales as $\prod_{l=1}^L \|W^{(l)}\| \cdot \prod_{l=1}^L |\sigma'(z^{(l)})|$. With sigmoid activation, $|\sigma'| \leq 0.25$, so gradients vanish rapidly. Solutions: - **Careful initialization** (Xavier, Kaiming): Initialize weights to preserve gradient magnitude - **Batch normalization**: Normalize activations to prevent saturation - **Residual connections**: Add skip connections so gradients can flow directly - **Better activations**: ReLU doesn't saturate for positive inputs **Universal Approximation Theorem**: A single hidden layer with enough neurons can approximate any continuous function. But depth is better than width in practice—exponentially more parameter-efficient and learns compositional features. ## Speech Signal Processing Before we can build speech models, we need to understand what we're modeling. Sound is physics—a pressure wave propagating through air. How we represent it computationally is a deliberate set of compromises. ### What Is Sound? The Waveform **References**: [14, 15] Sound is a continuous pressure wave. A microphone converts this to an electrical analog signal. To process it digitally, we sample at regular intervals. **Sampling theorem (Nyquist)**: To reconstruct a signal with maximum frequency $f_{\max}$, you must sample at $\geq 2f_{\max}$. For speech, useful content is below ~8kHz, so 16kHz sampling is standard. Telephone systems use 8kHz (sufficient for intelligibility). Music uses 44.1kHz or 48kHz (human hearing extends to ~20kHz). **Aliasing**: If you undersample, high frequencies fold back into low frequencies, creating artifacts. This is why anti-aliasing filters are applied before sampling. **Quantization**: Each sample is rounded to the nearest representable value. 16-bit audio has 65,536 levels. Quantization introduces noise, but 16-bit is sufficient for speech (96 dB dynamic range). **Pre-emphasis**: Before frequency analysis, apply a high-pass filter to boost high frequencies: $$ y[n] = x[n] - \alpha x[n-1] $$ where $\alpha \approx 0.97$ is standard. Why? Speech has a natural spectral tilt—low frequencies (fundamental, first formant) are much stronger than high frequencies (fricatives, consonant bursts). Pre-emphasis flattens the spectrum, making high-frequency content more visible to the model and improving numerical stability in subsequent processing. **Time-domain limitations**: A raw waveform tells you amplitude at each moment, but not what frequencies are present. A 440Hz tone and a 880Hz tone might have similar waveforms but sound completely different. This motivates frequency analysis. ### Fourier Transform and STFT **References**: [16, 17, 18] The **Discrete Fourier Transform (DFT)** decomposes a signal into sinusoidal components: $$ X[k] = \sum_{n=0}^{N-1} x[n] e^{-j2\pi kn/N} $$ Each frequency bin $k$ represents a sinusoid at frequency $f_k = k \cdot f_s / N$ where $f_s$ is the sampling rate. The output is complex—magnitude gives amplitude, phase gives timing. **Fast Fourier Transform (FFT)**: The naive DFT is $O(N^2)$. FFT uses divide-and-conquer to achieve $O(N \log N)$—a massive speedup that makes real-time audio processing feasible. **The problem with global FFT**: Speech is non-stationary—its statistical properties change over time. A global FFT loses temporal information. You can't tell when a phoneme occurred. **Short-Time Fourier Transform (STFT)**: Apply FFT to overlapping short frames: 1. Split signal into frames (typically 25ms) 2. Apply window function to each frame 3. Compute FFT of each frame 4. Result: 2D time-frequency representation **Windowing**: Abrupt frame boundaries create spectral leakage (energy spreads to adjacent frequencies). The **Hann window** tapers the edges: $$ w[n] = 0.5 \left(1 - \cos\left(\frac{2\pi n}{N-1}\right)\right) $$ **Frame parameters**: - **Frame length**: 25ms (400 samples at 16kHz) — long enough for frequency resolution - **Hop size**: 10ms (160 samples) — 15ms overlap for smooth transitions **The spectrogram**: $|STFT|^2$ gives a time-frequency heat map. Vowels show clear harmonic structure (horizontal bands). Fricatives show broadband noise. This is now an image—CNNs can process it. ### Mel Filterbanks and Log-Mel Spectrograms **References**: [19, 20] The raw spectrogram treats all frequencies equally. Human hearing does not. We're more sensitive to differences at low frequencies than high. **The Mel scale** is a perceptual frequency scale: $$ m = 2595 \log_{10}\left(1 + \frac{f}{700}\right) $$ This maps Hz to Mels (named after "melody"). The constants are empirically derived to match human perception. **Mel filterbank construction**: 1. Define $M$ triangular filters (typically 40-80) 2. Space them equally on the Mel scale 3. Each filter is a weighted average of spectrogram bins in its frequency range The result: $M$ energy values per frame, where $M \ll$ FFT bins. This compresses the frequency dimension while preserving perceptually relevant information. **Log compression**: Apply $\log(\max(\epsilon, E))$ to filterbank energies. Why? 1. **Perceptual**: Loudness perception is logarithmic (Weber's law) 2. **Dynamic range**: Compresses large values, expands small values 3. **Noise**: Converts multiplicative noise to additive (easier to handle) **MFCCs vs. Log-Mel**: Mel-Frequency Cepstral Coefficients (MFCCs) apply DCT to log-mel features to decorrelate them. This was standard for decades (GMM-HMM systems). Modern deep learning uses log-mel directly—neural networks can learn the decorrelation. **Production config**: 80 mel filterbanks, 25ms frames, 10ms hop → feature matrix of shape $[T \times 80]$ for a $T$-frame utterance. This feeds into the neural encoder. ### Front-End System Design: Far-Field Speech Processing **References**: [21a, 21b] The signal processing above assumes clean, close-talk audio. Production voice assistants (Alexa, Google Home) face far-field challenges: reverberation, background noise, and echo from the device's own speaker. This requires sophisticated front-end processing before the ASR model. **Acoustic Echo Cancellation (AEC)**: The Problem of Self-Interference When a device plays audio (music, TTS response), its microphones pick up that playback. Without cancellation, the ASR model tries to transcribe the device's own output—catastrophic for barge-in scenarios where users interrupt the device. **Linear AEC formulation**: Model the echo path as a linear filter: $$ y(n) = x(n) + h * s(n) + v(n) $$ where: - $y(n)$: microphone signal - $x(n)$: desired speech - $s(n)$: reference signal (what the speaker is playing) - $h$: room impulse response (echo path) - $v(n)$: background noise **Adaptive filtering**: Use LMS (Least Mean Squares) or NLMS (Normalized LMS) to estimate $\hat{h}$: $$ \hat{h}(n+1) = \hat{h}(n) + \mu \cdot e(n) \cdot s(n) $$ where $e(n) = y(n) - \hat{h}(n) * s(n)$ is the error signal and $\mu$ is the step size. **Why linear AEC fails**: Real echo paths are nonlinear (speaker distortion, microphone saturation). Modern systems use: - **Nonlinear AEC**: Neural networks to model nonlinear echo paths - **Residual echo suppression**: Post-filter to remove remaining echo after linear cancellation **Production reality**: AEC must adapt in real-time (<10ms latency) while the user is speaking. The echo path changes when someone moves furniture or the device is relocated. Convergence time vs. tracking speed is the critical trade-off. Too slow → residual echo. Too fast → unstable, removes desired speech. **Beamforming**: Spatial Filtering with Microphone Arrays Far-field devices use microphone arrays (typically 6-8 mics in a circular arrangement) to spatially filter audio—enhance speech from a target direction while suppressing noise from other directions. **Delay-and-Sum Beamforming**: The simplest approach: $$ y(n) = \frac{1}{M} \sum_{m=1}^M x_m(n - \tau_m) $$ where $\tau_m$ is the delay to align microphone $m$ with the target direction. Delays are computed from geometry: $$ \tau_m = \frac{d_m \cos(\theta)}{c} $$ where $d_m$ is the distance from mic $m$ to array center, $\theta$ is the target angle, and $c$ is the speed of sound (343 m/s). **Why delay-and-sum is insufficient**: It provides only modest noise suppression (~3-6 dB) and doesn't adapt to the noise field. **Minimum Variance Distortionless Response (MVDR)**: Optimal beamforming MVDR minimizes output power while maintaining unit gain in the target direction: $$ \min_w w^H R_{nn} w \quad \text{subject to} \quad w^H d = 1 $$ where: - $w$: beamformer weights (complex, one per mic) - $R_{nn}$: noise covariance matrix - $d$: steering vector (target direction) **Closed-form solution**: $$ w_{\text{MVDR}} = \frac{R_{nn}^{-1} d}{d^H R_{nn}^{-1} d} $$ **Estimating $R_{nn}$**: The noise covariance is estimated during non-speech periods (VAD-based) or using spatial covariance matrix subtraction: $$ R_{nn} = R_{yy} - R_{xx} $$ where $R_{yy}$ is the observed covariance and $R_{xx}$ is the speech covariance (estimated from target direction). **Generalized Sidelobe Canceller (GSC)**: Equivalent formulation that's more computationally efficient: $$ y = w_{\text{fixed}}^H x - w_{\text{adaptive}}^H B^H x $$ where $w_{\text{fixed}}$ is a fixed beamformer (delay-and-sum) and $B$ is a blocking matrix that nulls the target direction. The adaptive filter $w_{\text{adaptive}}$ removes residual noise. **Production reality**: MVDR requires accurate $R_{nn}$ estimation. In practice: - **Overestimation** of $R_{nn}$ → speech distortion (too aggressive) - **Underestimation** of $R_{nn}$ → insufficient noise suppression - **Non-stationary noise** (TV, music) requires continuous adaptation - **Multiple speakers**: MVDR assumes single target direction. Multi-speaker scenarios need speaker tracking or blind source separation **The front-end pipeline**: 1. **AEC**: Remove device's own playback 2. **Beamforming**: Spatially filter toward target speaker 3. **Noise suppression**: Spectral subtraction or neural post-filter 4. **Dereverberation**: Remove room reflections (optional, expensive) 5. **Feature extraction**: STFT → Mel filterbanks → ASR model **System design trade-offs**: - **Latency budget**: Each stage adds latency. AEC (5-10ms) + Beamforming (frame-based, 10ms) + ASR (streaming, 50-200ms) = total system latency - **Compute budget**: MVDR requires matrix inversion per frame. On-device deployment needs fixed-point arithmetic or neural beamformers - **Robustness vs. quality**: Aggressive front-end processing improves SNR but risks speech distortion. Conservative processing preserves speech but passes more noise to ASR **Why this matters for ML engineers**: The ASR model sees the output of this pipeline, not raw audio. Front-end artifacts (musical noise from spectral subtraction, speech distortion from aggressive beamforming) become the model's training distribution. Mismatch between training (clean speech) and production (front-end processed) causes significant WER degradation. Modern approaches: - **Joint training**: Train ASR on front-end processed audio - **Multi-condition training**: Mix clean, noisy, and processed audio - **Neural front-ends**: Replace classical DSP with learned front-ends (e.g., TasNet, Conv-TasNet) ### Voice Activity Detection: When Does Speech Start and Stop? **References**: [21c, 21d] Voice Activity Detection (VAD) determines which audio frames contain speech vs. silence, noise, or music. For voice assistants, VAD is critical—it determines when the user starts speaking (wake word detection) and when they stop (end-of-speech detection for query submission). **Why VAD matters**: 1. **Computational efficiency**: Don't run ASR on silence—wastes compute and battery 2. **User experience**: End-of-speech detection determines when to submit the query. Too early → truncated queries. Too late → high latency. 3. **Training data quality**: Filter out non-speech frames from training data **Classical VAD: Energy-based thresholding** The simplest approach: compute frame energy and threshold: $$ E[n] = \sum_{i=0}^{N-1} |x[n+i]|^2 $$ If $E[n] > \tau$, classify as speech. Else, silence. **Problems**: Fails in noisy environments (background noise has high energy), fails for quiet speech (whispers have low energy). **Improved classical VAD: Multi-feature** Combine multiple features: - **Energy**: As above - **Zero-crossing rate**: Speech has lower ZCR than noise - **Spectral flatness**: Speech has harmonic structure (low flatness), noise is flat - **Pitch**: Voiced speech has clear pitch, unvoiced and noise don't Use a decision tree or simple classifier to combine features. **Neural VAD: The modern standard** Train a small neural network (LSTM or lightweight CNN) to classify frames as speech/non-speech: **Architecture**: - Input: Log-mel spectrogram (e.g., 40 filterbanks, 25ms frames) - Encoder: 2-3 layer LSTM or 1D CNN - Output: Binary classification (speech/non-speech) per frame **Training data**: Labeled speech/non-speech segments from diverse conditions (clean, noisy, music, TV). **WebRTC VAD**: Open-source neural VAD used in Chrome, widely deployed. Lightweight (runs on CPU), robust to noise. **Production VAD pipeline for Alexa**: 1. **Wake word detection**: Specialized keyword spotting model detects "Alexa" 2. **Start-of-speech**: VAD confirms speech starts after wake word (filters false wakes) 3. **Streaming ASR**: RNN-T processes speech frames in real-time 4. **End-of-speech (EOS) detection**: VAD detects silence after speech → submit query **EOS detection challenges**: - **Natural pauses**: Users pause mid-sentence. Don't submit query during pauses. - **Latency**: Must detect EOS quickly (<500ms) for good UX - **False positives**: Submitting too early truncates queries ("Play... [pause] ...Bohemian Rhapsody" → only "Play" is submitted) **Solution: Adaptive EOS timeout** Use a dynamic timeout based on ASR confidence and linguistic context: - High confidence + complete sentence → short timeout (300ms) - Low confidence or incomplete sentence → long timeout (800ms) - Detected pause mid-sentence → very long timeout (1500ms) **Production reality**: VAD is one of the most tuned components in production voice assistants. The EOS timeout directly impacts user experience—too short causes frustration (truncated queries), too long causes perceived latency. A/B testing shows even 100ms changes in EOS timeout significantly affect user satisfaction metrics. ## Automatic Speech Recognition ASR is the problem of mapping variable-length audio to variable-length text. The classical approach decomposed this into probabilistic components. Modern approaches are end-to-end neural networks. ### The ASR Problem and Classic HMM Framework **References**: [21, 22, 23] The ASR objective: $W^* = \arg\max_W P(W|X)$ where $X$ is audio and $W$ is words. $$ \begin{align} \alpha_1(j) &= \pi_j b_j(o_1) \\ \alpha_t(j) &= \left[\sum_{i=1}^N \alpha_{t-1}(i) a_{ij}\right] b_j(o_t) \end{align} $$ Final probability: $P(O|\lambda) = \sum_{j=1}^N \alpha_T(j)$ By Bayes' rule: $$ W^* = \arg\max_W P(X|W) P(W) $$ Three components: - **Acoustic model** $P(X|W)$: How likely is this audio given these words? - **Language model** $P(W)$: How likely is this word sequence? - **Decoder**: Search for the best $W$ **Hidden Markov Models (HMMs)**: A sequence of hidden states (phoneme subunits) that emit observable features (MFCCs). Parameters: - **Transition probabilities** $A$: $P(\text{state}_t | \text{state}_{t-1})$ - **Emission probabilities** $B$: $P(\text{observation}_t | \text{state}_t)$ - **Initial probabilities** $\pi$: $P(\text{state}_1)$ **The three HMM problems**: 1. **Evaluation**: Given HMM and observations, compute $P(X|\text{model})$ → Forward algorithm 2. **Decoding**: Given observations, find most likely state sequence → Viterbi algorithm 3. **Training**: Given observations, find best parameters → Baum-Welch (EM) **Forward algorithm**: Computes the probability of the observation sequence by summing over all possible state sequences: $$ \alpha_t(j) = P(o_1, o_2, ..., o_t, q_t = j | \lambda) $$ Recursion: **Backward algorithm**: Computes the probability of future observations given current state: $$ \beta_t(i) = P(o_{t+1}, o_{t+2}, ..., o_T | q_t = i, \lambda) $$ Recursion: $$ \begin{align} \beta_T(i) &= 1 \\ \beta_t(i) &= \sum_{j=1}^N a_{ij} b_j(o_{t+1}) \beta_{t+1}(j) \end{align} $$ **Baum-Welch (EM) training**: Uses forward-backward to compute expected state occupancies: $$ \gamma_t(i) = P(q_t = i | O, \lambda) = \frac{\alpha_t(i)\beta_t(i)}{\sum_{j=1}^N \alpha_t(j)\beta_t(j)} $$ And expected transitions: $$ \xi_t(i,j) = P(q_t = i, q_{t+1} = j | O, \lambda) = \frac{\alpha_t(i) a_{ij} b_j(o_{t+1}) \beta_{t+1}(j)}{P(O|\lambda)} $$ Update rules: $$ \begin{align} \bar{a}_{ij} &= \frac{\sum_{t=1}^{T-1} \xi_t(i,j)}{\sum_{t=1}^{T-1} \gamma_t(i)} \\ \bar{b}_j(k) &= \frac{\sum_{t=1, o_t=v_k}^T \gamma_t(j)}{\sum_{t=1}^T \gamma_t(j)} \end{align} $$ **Viterbi algorithm**: Dynamic programming to find the most likely path: $$ \delta_t(j) = \max_{i} [\delta_{t-1}(i) \cdot a_{ij}] \cdot b_j(o_t) $$ where $\delta_t(j)$ is the probability of the best path ending in state $j$ at time $t$. Backpointers $\psi_t(j)$ store the best previous state for path reconstruction. **GMM acoustic model**: Model $P(x|\text{state})$ as a Gaussian Mixture Model: $$ P(x|\text{state}) = \sum_{k=1}^K w_k \mathcal{N}(x; \mu_k, \Sigma_k) $$ Each phoneme state has its own GMM. Mixtures capture the variance within a phoneme. **Context-dependent phonemes (triphones)**: The same phoneme sounds different depending on context. /t/ in "stop" vs. "top" is acoustically different. Solution: model triphones (left-context, phoneme, right-context). This explodes the state space—managed with decision tree clustering. **HCLG Decoding Graphs**: The Weighted Finite-State Transducer (WFST) Framework Production ASR systems use WFST-based decoding, where the search space is represented as a composition of four FSTs: $$ \text{HCLG} = H \circ C \circ L \circ G $$ Each component encodes a different knowledge source: **H (HMM structure)**: Maps HMM states to context-dependent phones. Encodes the topology of each HMM (typically 3-state left-to-right). Input: HMM states, Output: context-dependent phones (triphones). **C (Context dependency)**: Maps context-dependent phones to context-independent phones. Handles the explosion of triphone models by clustering similar contexts. Input: triphones (e.g., a-b+c), Output: monophones (e.g., b). **L (Lexicon)**: Maps phones to words. Encodes pronunciation dictionary—how words are pronounced as phone sequences. Handles multiple pronunciations per word. Input: phones, Output: words. **G (Grammar/Language Model)**: Encodes the language model. Can be an n-gram LM (represented as FST) or neural LM (applied during rescoring). Input: words, Output: words (with LM scores on arcs). **Why composition?**: Composing these FSTs offline creates a single optimized search graph. At runtime, Viterbi search operates on this precompiled HCLG graph, making decoding efficient. The composition operation: $$ (T_1 \circ T_2)(x, z) = \bigoplus_{y} T_1(x, y) \otimes T_2(y, z) $$ where $\oplus$ is the semiring addition (e.g., log-add for probabilities) and $\otimes$ is multiplication. **Optimization techniques**: - **Determinization**: Removes redundant paths with same input/output - **Minimization**: Reduces number of states while preserving language - **Weight pushing**: Moves weights toward initial state for better pruning - **On-the-fly composition**: For very large LMs, compose H∘C∘L offline, compose with G during decoding **Why GMM-HMMs worked and why they were replaced**: GMM-HMMs achieved good performance with limited data and compute. But GMMs are poor feature learners—they can't learn hierarchical representations. DNNs replaced GMMs as the emission model (hybrid DNN-HMM), then end-to-end models removed the HMM entirely. ### Connectionist Temporal Classification (CTC) **References**: [24, 25] CTC was the first successful end-to-end approach. The key innovation: eliminate the need for frame-level alignment labels. **The alignment problem**: Supervised training requires aligning each audio frame to a label. Getting this alignment is expensive (forced alignment using existing HMM systems). CTC bypasses this. **CTC's key insight**: Introduce a blank symbol $\epsilon$ and allow repetitions. A many-to-one mapping (collapsing) converts any raw output to a final transcript: ``` 'hh_ee_ll_ll_oo' → 'hello' '_h_e_l_l_o_' → 'hello' ``` **CTC loss**: Sum over all valid alignments that collapse to the target: $$ P(y|x) = \sum_{\pi: \mathcal{B}(\pi) = y} P(\pi|x) $$ where $\mathcal{B}$ is the collapsing function. This sum is exponential in length, but the forward-backward algorithm computes it in $O(T \cdot |y|)$ time. **CTC Forward-Backward Algorithm**: Efficient computation of the CTC loss The naive approach of enumerating all alignments is $O(|\mathcal{L}|^T)$ where $\mathcal{L}$ is the label set and $T$ is the sequence length—intractable. The forward-backward algorithm exploits dynamic programming. **Label sequence preparation**: For target $y = (y_1, ..., y_U)$, create extended label sequence $z$ by inserting blanks: $$ z = (\epsilon, y_1, \epsilon, y_2, \epsilon, ..., \epsilon, y_U, \epsilon) $$ Length: $|z| = 2U + 1$ **Forward variables** $\alpha_t(s)$: Probability of all paths that output $z_{1:s}$ after $t$ timesteps: $$ \alpha_t(s) = P(\pi_{1:t}: \mathcal{B}(\pi_{1:t}) = z_{1:s} | x) $$ Initialization: $$ \begin{align} \alpha_1(1) &= p(\epsilon | x_1) \\ \alpha_1(2) &= p(y_1 | x_1) \\ \alpha_1(s) &= 0 \text{ for } s > 2 \end{align} $$ Recursion (two cases for each position): $$ \alpha_t(s) = \begin{cases} (\alpha_{t-1}(s) + \alpha_{t-1}(s-1)) \cdot p(z_s | x_t) & \text{if } z_s = \epsilon \text{ or } z_s = z_{s-2} \\ (\alpha_{t-1}(s) + \alpha_{t-1}(s-1) + \alpha_{t-1}(s-2)) \cdot p(z_s | x_t) & \text{otherwise} \end{cases} $$ The first case handles blanks and repeated labels (can only come from same label or blank). The second case handles new labels (can come from same label, blank, or previous label). **Backward variables** $\beta_t(s)$: Probability of all paths that complete the sequence from position $s$ at time $t$: $$ \beta_t(s) = P(\pi_{t+1:T}: \mathcal{B}(\pi_{1:T}) = z | x, \pi_t = z_s) $$ Initialization: $$ \begin{align} \beta_T(|z|) &= 1 \\ \beta_T(|z|-1) &= 1 \\ \beta_T(s) &= 0 \text{ for } s < |z|-1 \end{align} $$ Recursion (symmetric to forward): $$ \beta_t(s) = \begin{cases} (\beta_{t+1}(s) + \beta_{t+1}(s+1)) \cdot p(z_s | x_t) & \text{if } z_s = \epsilon \text{ or } z_s = z_{s+2} \\ (\beta_{t+1}(s) + \beta_{t+1}(s+1) + \beta_{t+1}(s+2)) \cdot p(z_s | x_t) & \text{otherwise} \end{cases} $$ **Total probability**: $$ P(y|x) = \sum_{s=1}^{|z|} \alpha_T(s) = \alpha_T(|z|) + \alpha_T(|z|-1) $$ **Gradient computation**: For training, we need $\frac{\partial P(y|x)}{\partial p(k|x_t)}$. Using forward-backward: $$ \frac{\partial \log P(y|x)}{\partial \log p(k|x_t)} = \frac{1}{P(y|x)} \sum_{s: z_s = k} \alpha_t(s) \beta_t(s) $$ This gives the gradient for each output at each timestep—the sum of forward-backward probabilities for all positions in $z$ that match label $k$. **Complexity**: $O(T \cdot |z|) = O(T \cdot U)$ where $T$ is audio length and $U$ is label length. Much better than $O(|\mathcal{L}|^T)$ naive enumeration. **CTC assumptions and limitations**: CTC assumes conditional independence of outputs given the encoder—each frame's output is independent of previous output tokens. This means CTC can't learn output-to-output dependencies (no language model). The encoder can still capture context through its receptive field. **CTC in production**: Used as an auxiliary loss in many systems including RNN-T. Also used standalone for streaming ASR. Critical for Wav2Vec 2.0 fine-tuning. ### RNN-T: The Production Standard **References**: [26, 27, 28] RNN-T (Recurrent Neural Network Transducer) extends CTC by adding an autoregressive prediction network. It's the de facto standard for production streaming ASR. **Three components**: 1. **Encoder** (Transcription network): Processes audio frames → high-level representations $h_t^{\text{enc}}$ 2. **Prediction network** (Language model): Autoregressive model over output tokens → $h_u^{\text{pred}}$ 3. **Joiner network**: Combines encoder and prediction network → distribution over output tokens The key insight: encoder runs at frame rate (audio time), prediction network runs at label rate (output time)—they operate on different time axes. **Mathematical formulation**: At each step $(t, u)$ in the lattice: $$ \begin{align} h_t^{\text{enc}} &= \text{Encoder}(x_1, ..., x_t) \\ h_u^{\text{pred}} &= \text{Prediction}(y_1, ..., y_{u-1}) \\ z_{t,u} &= \text{Joiner}(h_t^{\text{enc}}, h_u^{\text{pred}}) \\ P(k | t, u) &= \text{Softmax}(z_{t,u})_k \end{align} $$ **The output lattice**: A $(t, u)$ lattice where $t \in [1, T]$ is frame index and $u \in [0, U]$ is output token index. At each cell $(t, u)$, the model decides: - **Emit a non-blank token** $y_u$: move right in $u$ (stay at same $t$) - **Emit blank** $\epsilon$: move down in $t$ (stay at same $u$) This creates a 2D grid of possible alignment paths. Any path from $(1, 0)$ to $(T, U)$ represents a valid alignment. **RNN-T Forward Algorithm**: Computes the probability of all paths that produce the target sequence Define forward variable $\alpha_{t,u}$: probability of all paths from $(1, 0)$ to $(t, u)$ that output $y_{1:u}$: $$ \alpha_{t,u} = P(\text{paths to } (t,u) \text{ outputting } y_{1:u} | x_{1:t}) $$ Initialization: $$ \begin{align} \alpha_{1,0} &= P(\epsilon | 1, 0) \\ \alpha_{t,0} &= \alpha_{t-1,0} \cdot P(\epsilon | t, 0) \quad \text{for } t > 1 \\ \alpha_{1,u} &= \alpha_{1,u-1} \cdot P(y_u | 1, u-1) \quad \text{for } u > 0 \end{align} $$ Recursion (two ways to reach $(t, u)$): $$ \alpha_{t,u} = \alpha_{t-1,u} \cdot P(\epsilon | t-1, u) + \alpha_{t,u-1} \cdot P(y_u | t, u-1) $$ The first term: came from $(t-1, u)$ by emitting blank. The second term: came from $(t, u-1)$ by emitting $y_u$. **RNN-T Backward Algorithm**: Computes the probability of completing the sequence from $(t, u)$ Define backward variable $\beta_{t,u}$: probability of all paths from $(t, u)$ to $(T, U)$ that output $y_{u+1:U}$: $$ \beta_{t,u} = P(\text{paths from } (t,u) \text{ to } (T,U) \text{ outputting } y_{u+1:U} | x_{t:T}) $$ Initialization: $$ \beta_{T,U} = 1 $$ Recursion (two ways to leave $(t, u)$): $$ \beta_{t,u} = P(\epsilon | t, u) \cdot \beta_{t+1,u} + P(y_{u+1} | t, u) \cdot \beta_{t,u+1} $$ **Total probability**: $$ P(y|x) = \alpha_{T,U} $$ **RNN-T loss**: Negative log-likelihood: $$ \mathcal{L} = -\log P(y|x) = -\log \alpha_{T,U} $$ **Gradient computation**: For training, we need gradients with respect to the joiner outputs. The gradient for output $k$ at position $(t, u)$ is: $$ \frac{\partial \log P(y|x)}{\partial \log P(k|t,u)} = \frac{1}{P(y|x)} \sum_{\text{paths through } (t,u) \text{ emitting } k} \alpha_{t,u} \cdot \beta_{t',u'} $$ where $(t', u')$ is the next position after emitting $k$ from $(t, u)$. **Complexity**: $O(T \cdot U)$ for forward-backward, where $T$ is audio length and $U$ is label length. The lattice size is manageable in practice. **Streaming property**: The encoder processes frames left-to-right (or with limited future context), so RNN-T can operate in streaming mode—emit tokens as audio arrives. This is critical for Alexa and other voice assistants. **Prediction network as LM**: Unlike CTC, output tokens are conditioned on previous outputs through the prediction network. This is a built-in language model—a key quality advantage. The prediction network is typically a small LSTM or Transformer decoder. **Joiner network**: Typically a simple feedforward network: $$ z_{t,u} = \text{Linear}(\text{ReLU}(\text{Linear}(h_t^{\text{enc}} + h_u^{\text{pred}}))) $$ Or with projection: $$ z_{t,u} = W_{\text{out}} \tanh(W_{\text{enc}} h_t^{\text{enc}} + W_{\text{pred}} h_u^{\text{pred}}) $$ **Comparison to CTC**: - **RNN-T**: Better quality (has LM component), flexible alignment, standard for production streaming ASR - **CTC**: Simpler, faster to train, easier to combine with external LMs, no prediction network overhead Both are end-to-end—no explicit alignment labels needed. **Decoding strategies**: - **Greedy**: At each step, emit argmax token. Fast but suboptimal. - **Beam search**: Maintain top-$k$ hypotheses, expand each by emitting or not emitting. Standard for production. - **Prefix beam search**: Merge hypotheses with same prefix to reduce search space. **Production optimizations**: - **Stateless prediction network**: Share prediction network states across beam hypotheses - **Joint network caching**: Cache joiner outputs for common (encoder, prediction) pairs - **Pruning**: Prune low-probability paths early to reduce lattice size ### Multilingual ASR **References**: [29, 30, 31] Training a separate model per language is expensive and fails for low-resource languages. A multilingual model shares acoustic representations across languages. **Why multilingual works**: Phonetic universals—many sounds appear across languages. /a/, /i/, /u/ are nearly universal vowels. Plosives, fricatives, nasals exist in most languages. Transfer learning leverages these shared patterns. **Architecture**: - **Shared encoder**: Most parameters (captures universal acoustic patterns) - **Language-specific output layers**: Or language ID conditioning - **Token space**: Character-level or BPE with joint vocabulary **Trade-off**: More sharing = better transfer for low-resource but potential negative transfer for high-resource. Finding the right balance is an art. **Whisper**: OpenAI's multilingual ASR model is the canonical modern example: - Encoder-decoder Transformer - Log-mel features (80 filterbanks) - Trained on 680k hours of weakly supervised web data - Language tokens prepended to decoder - Multitask training: transcription + translation + language ID **AMuSE (ICASSP 2025)**: My work on Attentive Multilingual Speech Encoding for Zero-Prior ASR. The key idea: leverage attention to handle languages with zero prior training data by extracting transferable acoustic representations. **Zero-Prior generalization**: Traditional multilingual models require training data in every target language. AMuSE achieves zero-shot transfer—train on languages A, B, C and generalize to language D without any D training data. This is critical for low-resource languages where labeled data doesn't exist. **Mixed-Conditioning Masking**: The core technical innovation. During training, randomly mask language ID tokens with probability $p_{\text{mask}}$: $$ \text{lang\_token} = \begin{cases} \text{} & \text{with probability } 1 - p_{\text{mask}} \\ \text{} & \text{with probability } p_{\text{mask}} \end{cases} $$ This forces the encoder to learn language-agnostic acoustic representations—it can't rely on the language ID token. When the language ID is masked, the model must infer phonetic content purely from acoustics. **Why this enables zero-shot**: The encoder learns universal phonetic features (formants, voicing, manner of articulation) that transfer across languages. At test time, even for unseen languages, these universal features are present—the model can decode them without language-specific training. **Attention's role**: Multi-head attention learns to focus on different phonetic aspects—one head may attend to vowel formants, another to consonant bursts. This decomposition of acoustic features is what enables cross-lingual transfer. **Production reality**: Zero-shot ASR is never as good as supervised training on the target language. But for languages with <100 hours of labeled data, zero-shot transfer from high-resource languages (English, Mandarin, Spanish) outperforms training from scratch. The trade-off: 10-20% higher WER vs. no model at all. ### Word Error Rate (WER): The Primary ASR Metric **References**: [23a] **Word Error Rate (WER)** is the standard metric for ASR performance—it measures the edit distance between the hypothesis (model output) and reference (ground truth): $ \text{WER} = \frac{S + D + I}{N} \times 100\% $ where: - **S** = Substitutions (words replaced) - **D** = Deletions (words missing) - **I** = Insertions (extra words) - **N** = Total words in reference ## Neural Architecture Fundamentals The evolution from RNNs to Transformers to Conformers represents the architectural journey of modern deep learning. Each innovation solved a specific limitation of its predecessor. ### Recurrent Neural Networks and LSTMs **References**: [32, 33, 34] Standard MLPs process fixed-size inputs. Speech is variable-length. RNNs process sequences by maintaining a hidden state: $$ h_t = f(W x_t + U h_{t-1} + b) $$ The same weights apply at every timestep—parameter sharing across time. **Backpropagation Through Time (BPTT)**: Unroll the RNN and apply backprop. Gradients involve $\frac{\partial h_t}{\partial h_0} = \prod_{i=1}^t \frac{\partial h_i}{\partial h_{i-1}}$—a product of $t$ Jacobians. If the spectral radius of $U < 1$: vanishing gradients. If $> 1$: exploding gradients. Vanilla RNNs can't learn long-range dependencies. **LSTM: The engineering solution**: Add a cell state $c$ that flows through time with only additive interactions (no multiplication!): $$ \begin{align} f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \quad \text{(forget gate)} \\ i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \quad \text{(input gate)} \\ \tilde{c}_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) \quad \text{(candidate values)} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \quad \text{(cell state update)} \\ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \quad \text{(output gate)} \\ h_t &= o_t \odot \tanh(c_t) \quad \text{(hidden state)} \end{align} $$ The cell state is like a conveyor belt—information can pass through many timesteps unchanged if the forget gate stays open. Gradients flow much better. **Bidirectional LSTMs**: For non-streaming tasks, process the sequence both forward and backward: $$ h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t] $$ This gives each position access to full context. Used in my thesis for speaker and language recognition. **GRU (Gated Recurrent Unit)**: Simpler alternative with two gates instead of three: $$ \begin{align} r_t &= \sigma(W_r x_t + U_r h_{t-1}) \quad \text{(reset gate)} \\ z_t &= \sigma(W_z x_t + U_z h_{t-1}) \quad \text{(update gate)} \\ \tilde{h}_t &= \tanh(W_h x_t + U_h (r_t \odot h_{t-1})) \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \end{align} $$ Fewer parameters than LSTM, similar performance. My thesis used hierarchical GRUs for language ID. ### Attention Mechanisms **References**: [35, 36, 37] Attention is the most important idea in modern deep learning. It allows selective focus on relevant parts of the input. **The bottleneck problem**: Encoder-decoder with RNNs passes all information through a single fixed-size vector $h_T$. For long sequences, this is a severe bottleneck. **Bahdanau Attention (2015)**: For each decoder timestep $s$, compute alignment scores: $$ e_{st} = v^T \tanh(W_s h_s^{\text{dec}} + W_h h_t^{\text{enc}}) $$ Normalize with softmax: $\alpha_{st} = \frac{\exp(e_{st})}{\sum_{t'} \exp(e_{st'})}$ Compute context vector: $c_s = \sum_t \alpha_{st} h_t^{\text{enc}}$ The decoder "queries" the encoder to find relevant input frames. This is **cross-attention**—$Q$ comes from the decoder, $K$ and $V$ come from the encoder. **Scaled Dot-Product Attention (Transformers)**: Reformulate as Q, K, V matrices: $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V $$ - **Query (Q)**: What we're looking for - **Key (K)**: What we're looking at - **Value (V)**: What we retrieve The $\sqrt{d_k}$ scaling prevents softmax saturation at large dimensions. Without scaling, for large $d_k$, the dot products grow large in magnitude, pushing softmax into regions with vanishing gradients. **Self-Attention vs. Cross-Attention**: - **Self-Attention**: $Q$, $K$, $V$ all derived from the same input $X$: $$ Q = XW_Q, \quad K = XW_K, \quad V = XW_V $$ Each position attends to all positions in the same sequence. Used in encoder blocks (BERT, Conformer encoder). - **Cross-Attention**: $Q$ from one sequence (decoder), $K$ and $V$ from another (encoder): $$ Q = X_{\text{dec}}W_Q, \quad K = X_{\text{enc}}W_K, \quad V = X_{\text{enc}}W_V $$ Decoder attends to encoder outputs. Used in encoder-decoder models (Transformer, Whisper). **Multi-Head Attention**: Run $h$ parallel attention heads with different learned projections: $$ \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W^O $$ where each head is: $$ \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) $$ Each head can attend to different aspects—syntax, semantics, position. This is more expressive than single-head attention. **Vectorized implementation**: NumPy code for scaled dot-product attention: ```python import numpy as np def scaled_dot_product_attention(Q, K, V, mask=None): """ Scaled Dot-Product Attention. Args: Q: Query matrix (Batch, Seq_Q, D_k) K: Key matrix (Batch, Seq_K, D_k) V: Value matrix (Batch, Seq_K, D_v) mask: Optional mask (Batch, Seq_Q, Seq_K) - True for positions to mask Returns: Output: (Batch, Seq_Q, D_v) Attention weights: (Batch, Seq_Q, Seq_K) Inference vs. Training: - Training: Compute full attention over all positions - Inference (autoregressive): Use causal mask + KV-cache to avoid recomputation """ d_k = Q.shape[-1] # Compute attention scores: (Batch, Seq_Q, Seq_K) scores = np.matmul(Q, K.swapaxes(-2, -1)) / np.sqrt(d_k) # Apply mask (set masked positions to -inf before softmax) if mask is not None: scores = np.where(mask, -1e9, scores) # Softmax over key dimension: (Batch, Seq_Q, Seq_K) # Numerical stability: subtract max before exp scores_max = np.max(scores, axis=-1, keepdims=True) exp_scores = np.exp(scores - scores_max) attention_weights = exp_scores / np.sum(exp_scores, axis=-1, keepdims=True) # Weighted sum of values: (Batch, Seq_Q, D_v) output = np.matmul(attention_weights, V) return output, attention_weights def multi_head_attention_forward(X, W_q, W_k, W_v, W_o, num_heads): """ Multi-Head Self-Attention forward pass. Args: X: Input (Batch, Seq, Dim) W_q, W_k, W_v: Weight matrices (Dim, Dim) W_o: Output projection (Dim, Dim) num_heads: Number of attention heads Returns: Output: (Batch, Seq, Dim) Dimension matching: - Split Dim into (num_heads, D_k) where D_k = Dim // num_heads - Each head operates on D_k dimensions independently - Concatenate heads and project with W_o """ batch_size, seq_len, dim = X.shape d_k = dim // num_heads # Linear projections: (Batch, Seq, Dim) Q = np.dot(X, W_q) K = np.dot(X, W_k) V = np.dot(X, W_v) # Reshape for multi-head: (Batch, Seq, Heads, D_k) -> (Batch, Heads, Seq, D_k) Q = Q.reshape(batch_size, seq_len, num_heads, d_k).transpose(0, 2, 1, 3) K = K.reshape(batch_size, seq_len, num_heads, d_k).transpose(0, 2, 1, 3) V = V.reshape(batch_size, seq_len, num_heads, d_k).transpose(0, 2, 1, 3) # Apply attention per head: (Batch, Heads, Seq, D_k) d_k_sqrt = np.sqrt(d_k) scores = np.matmul(Q, K.swapaxes(-2, -1)) / d_k_sqrt weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True)) weights = weights / np.sum(weights, axis=-1, keepdims=True) attn_output = np.matmul(weights, V) # Concatenate heads: (Batch, Heads, Seq, D_k) -> (Batch, Seq, Dim) attn_output = attn_output.transpose(0, 2, 1, 3).reshape(batch_size, seq_len, dim) # Output projection: (Batch, Seq, Dim) output = np.dot(attn_output, W_o) return output ``` **Complexity analysis**: For sequence length $n$ and dimension $d$: - Computing $QK^T$: $O(n^2 d)$ time, $O(n^2)$ space for attention matrix - Applying softmax: $O(n^2)$ - Computing attention output: $O(n^2 d)$ - Total: $O(n^2 d)$ time and space—quadratic in sequence length **Production Reality: My Master's Thesis Work** My thesis applied multi-head attention to speaker and language identification. The key insight: not all speech frames are equally informative. Silence, background noise, and speaker-independent phonemes contribute little to identity. **Frame-level attention for language ID**: Instead of pooling all frames equally (mean/max pooling), learn attention weights: $$ \alpha_t = \frac{\exp(w^T \tanh(W h_t))}{\sum_{t'} \exp(w^T \tanh(W h_{t'}))} $$ where $h_t$ is the frame-level encoder output. The utterance embedding is: $$ e = \sum_t \alpha_t h_t $$ **Multi-head attention for speaker ID**: Different heads focus on different acoustic aspects: - **Head 1**: High attention on vowels (formant structure is speaker-specific) - **Head 2**: High attention on fricatives (spectral characteristics vary by speaker) - **Head 3**: High attention on prosody (pitch contours, speaking rate) This decomposition emerged without supervision—the model learned to specialize heads based on the classification objective. **Results**: On NIST LRE 2017 (language recognition), attention-based pooling improved accuracy by 12% over mean pooling. On multi-speaker corpora (VAST), where the target speaker is intermixed with others, multi-head attention improved by 38%—the model learned to suppress non-target speakers. **Why attention works without supervision**: Visualizing attention weights on speech shows they align with phonetically meaningful regions. The model learns to suppress silence and noise without being told to. This emergent behavior is powerful—the inductive bias of attention (weighted sum with learned weights) is sufficient to discover relevant structure. ### The Transformer **References**: [38, 39, 40] The Transformer (2017) removed recurrence entirely, relying solely on attention and feed-forward networks. **Encoder architecture**: $N$ stacked blocks, each containing: 1. Multi-Head Self-Attention 2. Feed-Forward Network (2-layer MLP) 3. Layer normalization and residual connections **Positional encoding**: Self-attention is permutation-invariant—it treats inputs as a set, not a sequence. We must inject position information. Sinusoidal positional encodings: $$ \begin{align} PE_{(pos, 2i)} &= \sin\left(\frac{pos}{10000^{2i/d}}\right) \\ PE_{(pos, 2i+1)} &= \cos\left(\frac{pos}{10000^{2i/d}}\right) \end{align} $$ Why sinusoids? They allow the model to learn relative position offsets, and they generalize to sequence lengths not seen in training. ### RoPE: Rotary Position Embedding **References**: [40a, 40b] While sinusoidal encodings work, they have limitations: they're added to embeddings (not integrated into attention), and they don't explicitly encode relative positions. **RoPE (Rotary Position Embedding)** solves both issues and is now standard in modern models (LLaMA, GPT-NeoX, Conformer variants). **The core idea**: Rotate query and key vectors by an angle proportional to their position. This encodes absolute position while making attention scores depend only on relative position. **2D rotation matrix**: $ R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix} $ **RoPE for position $m$**: Rotate each pair of dimensions by $\theta_i = m / 10000^{2i/d}$: $ \text{RoPE}(x, m) = \begin{pmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \\ \vdots \end{pmatrix} \otimes \begin{pmatrix} \cos(m\theta_1) \\ \cos(m\theta_1) \\ \cos(m\theta_2) \\ \cos(m\theta_2) \\ \vdots \end{pmatrix} + \begin{pmatrix} -x_2 \\ x_1 \\ -x_4 \\ x_3 \\ \vdots \end{pmatrix} \otimes \begin{pmatrix} \sin(m\theta_1) \\ \sin(m\theta_1) \\ \sin(m\theta_2) \\ \sin(m\theta_2) \\ \vdots \end{pmatrix} $ **Why this works**: For query at position $m$ and key at position $n$: $ q_m^T k_n = (\text{RoPE}(q, m))^T (\text{RoPE}(k, n)) = q^T R^T(m\theta) R(n\theta) k = q^T R((n-m)\theta) k $ The attention score depends only on the relative position $(n-m)$, not absolute positions $m$ and $n$. This is exactly what we want for sequences—"how far apart are these tokens?" matters more than "what are their absolute positions?" **Advantages over sinusoidal PE**: 1. **Integrated into attention**: RoPE rotates Q and K directly, not added to embeddings. This makes position information part of the attention mechanism itself. 2. **Relative position encoding**: Attention scores naturally depend on relative distance, which is more meaningful for sequences. 3. **Better extrapolation**: RoPE generalizes better to longer sequences than seen in training. Sinusoidal PE degrades significantly beyond training length. 4. **No learned parameters**: Like sinusoidal PE, RoPE is deterministic—no extra parameters to learn. **Implementation efficiency**: RoPE can be implemented with element-wise operations (no matrix multiplications): ```python import numpy as np def apply_rope(x, position, theta_base=10000): """ Apply RoPE to input tensor. Args: x: Input tensor (Seq, Dim) - must have even dimension position: Position indices (Seq,) theta_base: Base for frequency calculation Returns: Rotated tensor (Seq, Dim) """ seq_len, dim = x.shape assert dim % 2 == 0, "Dimension must be even for RoPE" # Compute frequencies for each dimension pair # theta_i = 1 / (theta_base^(2i/dim)) i = np.arange(0, dim, 2) theta = 1.0 / (theta_base ** (i / dim)) # Compute angles: position * theta # Shape: (Seq, Dim/2) angles = position[:, None] * theta[None, :] # Compute cos and sin cos = np.cos(angles) sin = np.sin(angles) # Split x into pairs: (x1, x2), (x3, x4), ... x1 = x[:, 0::2] # Even indices x2 = x[:, 1::2] # Odd indices # Apply rotation: [x1*cos - x2*sin, x1*sin + x2*cos] x_rotated = np.empty_like(x) x_rotated[:, 0::2] = x1 * cos - x2 * sin x_rotated[:, 1::2] = x1 * sin + x2 * cos return x_rotated ``` **Production use in Conformer**: Modern Conformers (including Nova Sonic) use RoPE instead of sinusoidal PE. For speech, relative position is critical—"this phoneme is 50ms after that phoneme" is more meaningful than "this phoneme is at absolute position 237." **Extrapolation to longer sequences**: RoPE can handle sequences 2-4× longer than training length with minimal degradation. This is critical for speech—training on 10-second clips but deploying on 60-second clips. Sinusoidal PE would fail catastrophically. **My work**: Nova Sonic uses RoPE in the Conformer encoder. We train on 10-second audio clips but deploy on arbitrary-length audio (up to 60 seconds for podcasts). RoPE's extrapolation properties make this possible without quality degradation. **Feed-Forward Networks**: Each block contains a 2-layer MLP applied position-wise: $$ \text{FFN}(x) = \text{GELU}(xW_1 + b_1)W_2 + b_2 $$ The FFN dimension is typically $4 \times$ the model dimension. Interpretability research shows FFN layers act as key-value memories—they store factual knowledge. **Layer Normalization**: Normalize across features (not batch): $$ \text{LN}(x) = \gamma \frac{x - \mu}{\sigma} + \beta $$ where $\mu, \sigma$ are computed per-example per-layer. Modern models use pre-norm (more stable training). ### Batch Normalization vs. Layer Normalization: Why LN for Sequences **References**: [106a, 106b] Normalization is critical for training deep networks—it stabilizes gradients and allows higher learning rates. But the choice between Batch Norm and Layer Norm matters, especially for speech and NLP. **Batch Normalization** (BatchNorm): $ \text{BN}(x) = \gamma \frac{x - \mu_{\text{batch}}}{\sigma_{\text{batch}}} + \beta $ where $\mu_{\text{batch}}$ and $\sigma_{\text{batch}}$ are computed across the batch dimension for each feature: $ \mu_j = \frac{1}{B}\sum_{i=1}^B x_{ij}, \quad \sigma_j^2 = \frac{1}{B}\sum_{i=1}^B (x_{ij} - \mu_j)^2 $ **Why BatchNorm works**: Reduces internal covariate shift—the distribution of layer inputs changes during training as parameters update. By normalizing, each layer sees a stable input distribution, enabling faster convergence. **BatchNorm's fatal flaw for sequences**: It normalizes across the batch dimension. For variable-length sequences (speech, text), this creates problems: 1. **Batch size dependency**: Statistics depend on batch composition. Different batch sizes give different outputs—breaks at inference time with batch size 1. 2. **Variable-length sequences**: For speech, utterances have different lengths. Padding creates artificial statistics. Short utterances in a batch with long utterances get contaminated by padding values. 3. **Temporal dependencies**: BatchNorm mixes information across examples in the batch. For sequences, this leaks information across time—the model can "cheat" by using batch statistics to infer temporal patterns. **Layer Normalization** (LayerNorm): $ \text{LN}(x) = \gamma \frac{x - \mu_{\text{layer}}}{\sigma_{\text{layer}}} + \beta $ where $\mu_{\text{layer}}$ and $\sigma_{\text{layer}}$ are computed across the feature dimension for each example: $ \mu_i = \frac{1}{D}\sum_{j=1}^D x_{ij}, \quad \sigma_i^2 = \frac{1}{D}\sum_{j=1}^D (x_{ij} - \mu_i)^2 $ **Why LayerNorm for sequences**: 1. **Batch-independent**: Statistics computed per-example. Inference with batch size 1 is identical to training with batch size 32. 2. **Variable-length friendly**: Each example normalized independently. No padding contamination. 3. **No information leakage**: No cross-example dependencies. The model can't cheat. **The trade-off**: BatchNorm often trains faster and generalizes better for CNNs on images (fixed-size inputs, large batches). LayerNorm is essential for RNNs, Transformers, and any variable-length sequence model. **Dimension comparison**: For input $X \in \mathbb{R}^{B \times T \times D}$ (batch, time, features): - **BatchNorm**: Normalize across $B$ (batch dimension), separate stats for each $(t, d)$ - **LayerNorm**: Normalize across $D$ (feature dimension), separate stats for each $(b, t)$ **Production reality**: All modern speech models (Conformer, RNN-T, Whisper) use LayerNorm. All modern LLMs (GPT, BERT, LLaMA) use LayerNorm. BatchNorm is relegated to computer vision (ResNets, EfficientNets) where inputs are fixed-size images. **My work**: Nova Sonic uses LayerNorm throughout—in the Conformer encoder, RNN-T prediction network, and joiner. We tried BatchNorm early in development and saw catastrophic failures on variable-length utterances. LayerNorm solved it immediately. ### Residual Connections: The Identity Shortcut **References**: [106c, 106d] **Residual connections** (skip connections) are one of the most important innovations in deep learning. Introduced in ResNet (2015), they're now ubiquitous—every modern architecture uses them. **The core idea**: Instead of learning $F(x)$, learn the residual $F(x) - x$: $ y = F(x) + x $ where $F(x)$ is a stack of layers (conv, attention, FFN, etc.) and $x$ is the input. The $+x$ is the "identity shortcut." **Why residual connections solve vanishing gradients**: During backpropagation, gradients flow through the residual connection: $ \frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x} = \frac{\partial L}{\partial y} \cdot \left(\frac{\partial F(x)}{\partial x} + 1\right) $ The $+1$ term guarantees that gradients can flow directly backward, even if $\frac{\partial F(x)}{\partial x} \approx 0$. This is the key to training 100+ layer networks. **Without residual connections**: Gradients multiply through layers: $ \frac{\partial L}{\partial x_0} = \frac{\partial L}{\partial x_L} \cdot \prod_{l=1}^L \frac{\partial x_l}{\partial x_{l-1}} $ If any $\frac{\partial x_l}{\partial x_{l-1}} < 1$, gradients vanish exponentially. For 50 layers with $\frac{\partial x_l}{\partial x_{l-1}} = 0.9$, gradients shrink by $0.9^{50} \approx 0.005$—effectively zero. **With residual connections**: Gradients have a direct path: $ \frac{\partial L}{\partial x_0} = \frac{\partial L}{\partial x_L} \cdot \left(1 + \sum_{\text{paths}} \prod_{\text{layers in path}} \frac{\partial F}{\partial x}\right) $ The $1$ term ensures gradients never vanish completely—there's always a gradient highway from output to input. **Initialization benefit**: At initialization, if $F(x) \approx 0$, then $y \approx x$—the network starts as an identity function. This is a much better starting point than random initialization. The network can gradually learn to deviate from identity as needed. **Pre-activation vs. Post-activation residuals**: **Post-activation** (original ResNet): $ y = F(\text{ReLU}(\text{BN}(x))) + x $ **Pre-activation** (ResNet v2, modern standard): $ y = F(x) + x, \quad \text{where } F(x) = \text{Conv}(\text{ReLU}(\text{BN}(x))) $ Pre-activation is more stable—the identity path is completely clean, no nonlinearities in the way. **Residual connections in Transformers**: Every Transformer block has two residual connections: ``` # Attention block x = x + MultiHeadAttention(LayerNorm(x)) # FFN block x = x + FFN(LayerNorm(x)) ``` This is **pre-norm** (LayerNorm before the sublayer). The alternative is **post-norm** (LayerNorm after addition), but pre-norm is more stable for deep models. **Why residual connections enable depth**: - **ResNet-50**: 50 layers, 25M parameters, 76% ImageNet accuracy - **ResNet-152**: 152 layers, 60M parameters, 78% ImageNet accuracy - **Without residuals**: 20+ layers fail to train (gradients vanish) **Production reality**: Every modern architecture uses residual connections: - **Vision**: ResNet, EfficientNet, Vision Transformers - **NLP**: BERT, GPT, T5 (every Transformer block) - **Speech**: Conformer, Wav2Vec 2.0, Whisper (every encoder block) **My work**: Nova Sonic's Conformer encoder has 12 blocks, each with residual connections around attention and FFN. Without residuals, we couldn't train beyond 4-6 blocks—gradients would vanish. With residuals, 12 blocks train stably and converge faster than 6 blocks without residuals. **Complexity analysis**: - Self-attention: $O(n^2 d)$ time and space - FFN: $O(nd^2)$ - For typical configs ($d=512$, $n=500$): attention dominates for long sequences **The memory bottleneck**: Standard attention materializes the full $n \times n$ attention matrix in GPU memory. For $n=2048$ and batch size 32, this is $32 \times 2048 \times 2048 \times 4 = 512$ MB just for attention scores—before keys, values, and gradients. This limits sequence length and batch size. ### Flash Attention: Making Transformers Memory-Efficient **References**: [100, 101] Flash Attention is now standard in every production Transformer implementation. It makes attention 2-4× faster and uses 10-20× less memory—without changing the output at all. **The key insight**: Don't materialize the full attention matrix. Compute attention in blocks, fusing operations to minimize memory reads/writes. **Standard attention memory access pattern**: 1. Load $Q$, $K$ from HBM (high-bandwidth memory) → compute $S = QK^T$ → write $S$ to HBM 2. Load $S$ from HBM → compute $P = \text{softmax}(S)$ → write $P$ to HBM 3. Load $P$, $V$ from HBM → compute $O = PV$ → write $O$ to HBM Each intermediate result ($S$, $P$) is written to and read from slow HBM. For $n=2048$, this is gigabytes of memory traffic. **Flash Attention approach**: Tile $Q$, $K$, $V$ into blocks that fit in fast SRAM (on-chip cache). Compute attention block-by-block, never materializing the full matrix. **Algorithm** (simplified): 1. Divide $Q$ into blocks $Q_1, ..., Q_T$ (each block fits in SRAM) 2. Divide $K$, $V$ into blocks $K_1, ..., K_T$ and $V_1, ..., V_T$ 3. For each $Q_i$: - Initialize output block $O_i = 0$, normalization $l_i = 0$ - For each $K_j$, $V_j$: - Load $Q_i$, $K_j$, $V_j$ into SRAM - Compute $S_{ij} = Q_i K_j^T$ (in SRAM) - Compute $P_{ij} = \text{softmax}(S_{ij})$ (in SRAM, with online softmax trick) - Update $O_i \leftarrow O_i + P_{ij} V_j$ (in SRAM) - Write $O_i$ to HBM **Online softmax trick**: Standard softmax requires two passes (max, then exp and sum). Flash Attention uses an online algorithm that computes softmax incrementally as blocks arrive, requiring only one pass. **Memory complexity**: - **Standard attention**: $O(n^2)$ memory for attention matrix - **Flash Attention**: $O(n)$ memory—only stores output and normalization statistics **Speed improvement**: 2-4× faster due to reduced memory traffic. On A100 GPUs, memory bandwidth (1.5 TB/s) is the bottleneck, not compute (312 TFLOPS). Flash Attention minimizes memory reads/writes. **Production impact**: Flash Attention enables: - Longer sequences (8k, 16k, 32k tokens) without OOM - Larger batch sizes (2-4× increase) - Training 100B+ models that were previously infeasible **Flash Attention 2 and 3**: Subsequent versions optimize further: - **Flash Attention 2**: Better parallelization across GPU cores, 2× faster than v1 - **Flash Attention 3**: Optimized for H100 GPUs with asynchronous memory operations **Why this matters for speech**: Speech models process long sequences—a 30-second utterance at 10ms frame rate is 3000 frames. Without Flash Attention, batch size is limited to 4-8. With Flash Attention, batch size can be 32-64, dramatically improving training throughput. **My work on Nova Sonic**: All Conformer training uses Flash Attention. This enables training on 30-second audio clips (the longest utterances in the training set) with batch size 32 per GPU, which would OOM without Flash Attention. ### Conformer: Convolution + Attention for Speech **References**: [41, 42] The Conformer is the dominant architecture for speech processing. It addresses a key limitation: pure Transformers don't have explicit local feature extraction. **Motivation**: Speech has both local structure (phoneme articulation, formants) and global structure (prosody, sentence context). Conformers explicitly model both. **Conformer block structure**: 1. Feed-Forward module (half-step, Macaron style) 2. Multi-Head Self-Attention module 3. Convolution module (depthwise separable) 4. Feed-Forward module (half-step) Each followed by LayerNorm and residual. **Depthwise Separable Convolution**: Standard conv applies kernel across all channels simultaneously. Depthwise: one filter per channel. Pointwise: $1 \times 1$ conv to mix channels. This reduces parameters by ~8-9×. Within the Conformer, depthwise conv is applied along the time axis to capture local temporal patterns. **Relative positional encodings**: Conformers use relative position encodings (e.g., RoPE) instead of absolute. These allow reasoning about relative distances between frames—better for variable-length sequences. **Conformer in production**: Powers Google's production ASR, Whisper's encoder, and Amazon's Nova Sonic (my direct work). Typical hyperparameters: - 16-18 blocks - 256-512 model dimension (streaming), larger for offline ## Speaker and Language Identification ASR cares about what was said. Speaker and language ID care about who said it and in what language—regardless of content. This requires extracting speaker/language-specific information while being invariant to content, noise, and channel. ### Task Definitions and Evaluation **References**: [43, 44] **Speaker Verification (SV)**: Given two utterances, are they from the same speaker? (1:1 comparison) **Speaker Identification (SID)**: Given an utterance, which of $N$ enrolled speakers is it? (1:N) **Language Identification (LID)**: Which language is being spoken? **Evaluation metrics**: - **Equal Error Rate (EER)**: False acceptance rate where false rejection rate equals it—lower is better - **Detection Cost Function (DCF)**: Weighted cost accounting for prior probabilities - **NIST SRE**: Standard benchmark (I competed in NIST SRE 2018 with LEAP system) ### i-vectors and PLDA **References**: [45, 46] **i-vectors**: Factor analysis model representing a speaker utterance as a single fixed-dimensional vector in "total variability space": $$ M = m + Tw $$ where $M$ is the GMM supervector, $m$ is the universal background model (UBM) mean, $T$ is the total variability matrix, and $w$ is the i-vector. **PLDA scoring**: Probabilistic Linear Discriminant Analysis decomposes i-vector variability into speaker subspace and residual. During scoring, PLDA computes the log-likelihood ratio of "same speaker" vs. "different speaker" hypotheses. **Why they're no longer state-of-the-art**: i-vectors use a single GMM-based Gaussian subspace—limited capacity. Neural embeddings (x-vectors) directly optimize discriminative objectives and scale with data. ### x-vectors and Neural Speaker Embeddings **References**: [47, 48] **x-vector architecture**: TDNN (Time Delay Neural Network, 1D dilated convolution) encoder → statistics pooling → segment-level classifier. **Statistics pooling**: The key innovation—aggregates variable-length frame-level representations into fixed-size segment-level representation by computing mean and standard deviation across time: $$ \mu = \frac{1}{T}\sum_{t=1}^T h_t, \quad \sigma = \sqrt{\frac{1}{T}\sum_{t=1}^T h_t^2 - \mu^2} $$ **Training objective**: Trained on speaker classification (softmax over thousands of speakers). At test time, the classifier is discarded—only the embedding extractor is used. **ECAPA-TDNN**: Current state-of-the-art—Emphasized Channel Attention, Propagation and Aggregation. Adds squeeze-excitation, multi-scale feature aggregation, and residual connections to x-vectors. ### Attention-Based Embeddings: My Thesis Work **References**: [49, 50] **Frame-level attention for LID**: Not all frames are equally informative—silence, noise bursts, and speaker-independent phonemes are less informative. Attention learns to upweight informative frames. **Hierarchical GRU for LID**: My architecture for NIST LRE 2017: 1. Process short-term frames with one GRU 2. Aggregate into segments with segment-level attention 3. Aggregate segments with utterance-level attention This hierarchical pooling captures multi-scale structure. **Multi-head attention for SID**: Different heads attend to different aspects—one head may focus on vowels, another on prosody. The 38% improvement on VAST (multi-speaker corpus) suggests attention is especially valuable when the target speaker is intermixed with others. **Why attention works without supervision**: Attention weights visualized on speech align with phonetically meaningful regions—the model learns to suppress silence and noise without being told to. This emergent behavior is powerful. ## Language Modeling Language models assign probabilities to word sequences. They're critical for ASR (prefer "recognize speech" over "wreck a nice beach") and for modern LLMs. ### N-gram Language Models **References**: [51, 52] A language model is a probability distribution over sequences: $P(w_1, w_2, ..., w_n)$. By the chain rule: $$ P(w_1...w_n) = \prod_{t=1}^n P(w_t | w_1...w_{t-1}) $$ **N-gram approximation**: Approximate the full history with the last $n-1$ words: $$ P(w_t | w_1...w_{t-1}) \approx P(w_t | w_{t-n+1}...w_{t-1}) $$ Count-based estimation from a text corpus. Simple, interpretable, fast inference. **Smoothing**: Most n-grams will never appear in training. Without smoothing, $P=0$ for any unseen n-gram. **Kneser-Ney smoothing** distributes probability mass from seen to unseen n-grams using continuation counts—the standard for production n-gram LMs. **Shallow fusion with ASR**: At decoding time, combine LM scores with acoustic model scores: $$ \text{score} = \log P(X|W) + \lambda \log P(W) $$ The $\lambda$ is tuned on a dev set. ### Byte Pair Encoding (BPE): Modern Tokenization **References**: [52a, 52b, 52c] Before neural language models, we need to address a fundamental question: what is a "word"? Traditional tokenization splits on whitespace, but this creates problems: - **Vocabulary explosion**: English has 170,000+ words. Rare words become "unknown" tokens. - **Morphology**: "run", "running", "runs" are treated as completely different despite shared meaning. - **Out-of-vocabulary (OOV)**: New words (slang, names, technical terms) can't be represented. **BPE solves this**: Learn a vocabulary of subword units that balance between character-level (no OOV, but long sequences) and word-level (short sequences, but OOV problems). **The BPE algorithm**: 1. **Initialize**: Start with character-level vocabulary (a-z, A-Z, 0-9, punctuation, etc.) 2. **Count pairs**: Count all adjacent symbol pairs in the training corpus. Example: "low" appears 5 times, "lower" appears 2 times → "l o w" (5 times), "l o w e r" (2 times). 3. **Merge most frequent**: Find the most frequent pair and merge it into a single symbol. If "e r" is most frequent, merge it → "er" becomes a new symbol. 4. **Repeat**: Continue merging until vocabulary reaches target size (typically 32k-50k tokens). **Example BPE training**: ``` Corpus: "low low low low low lower lower newest newest newest newest widest widest widest" Initial: l o w (5×), l o w e r (2×), n e w e s t (4×), w i d e s t (3×) Iteration 1: Most frequent pair is "e s" (7 times) → Merge: l o w (5×), l o w e r (2×), n e w es t (4×), w i d es t (3×) Iteration 2: Most frequent pair is "es t" (7 times) → Merge: l o w (5×), l o w e r (2×), n e w est (4×), w i d est (3×) Iteration 3: Most frequent pair is "l o" (7 times) → Merge: lo w (5×), lo w e r (2×), n e w est (4×), w i d est (3×) ... continue until vocabulary size reached ``` **Final vocabulary**: \{l, o, w, e, r, n, s, t, i, d, lo, est, low, ...\} **Encoding with BPE**: Given a trained BPE vocabulary, encode new text by greedily applying merges: ``` Input: "lowest" 1. Start: l o w e s t 2. Apply merge "lo": lo w e s t 3. Apply merge "est": lo w est 4. No more merges apply Output: ["lo", "w", "est"] ``` **Why BPE works**: 1. **No OOV**: Any word can be decomposed into subwords, down to characters if needed. "supercalifragilisticexpialidocious" → ["super", "cal", "if", "rag", "il", "istic", ...]. 2. **Morphology**: Common morphemes become tokens. "run" + "ning" → ["run", "ning"]. The model learns that "ning" is a suffix. 3. **Frequency-based**: Common words stay as single tokens ("the", "and"). Rare words split into subwords. This balances vocabulary size with sequence length. 4. **Language-agnostic**: Works for any language. For Chinese/Japanese, starts with characters. For English, starts with bytes (UTF-8). **BPE variants**: - **WordPiece** (BERT, Whisper): Similar to BPE but merges based on likelihood increase rather than frequency. Slightly better for language modeling. - **SentencePiece** (T5, LLaMA): Treats text as raw byte stream, no pre-tokenization. Handles any language uniformly, including those without spaces. - **Unigram LM** (SentencePiece): Starts with large vocabulary, iteratively removes tokens that minimize loss. More principled than greedy BPE. **BPE for ASR**: Traditional ASR used phoneme or character outputs. Modern ASR (Whisper, Nova Sonic) uses BPE: 1. **Multilingual**: BPE vocabulary covers 100+ languages with a single 50k token vocabulary. Characters would require separate models per language. 2. **Efficiency**: BPE sequences are 3-4× shorter than character sequences. Faster decoding, less memory. 3. **Implicit language model**: BPE tokens capture common word patterns. The model learns that "ing" follows verbs, "tion" ends nouns—implicit linguistic knowledge. **Production BPE pipeline**: 1. **Train BPE**: On large text corpus (Wikipedia, web crawl). Vocabulary size: 32k-50k. 2. **Encode training data**: Convert text transcripts to BPE token sequences. 3. **Train ASR model**: Model outputs BPE tokens, not characters or words. 4. **Decode**: Convert BPE tokens back to text by concatenating and removing special markers (e.g., "##" prefix in WordPiece). **My work on Nova Sonic**: We use SentencePiece with 50k vocabulary trained on multilingual text (100+ languages). The same vocabulary handles English, Spanish, Mandarin, Hindi, etc. This enables zero-shot cross-lingual transfer—the model learns universal phonetic patterns that transfer across languages. **The trade-off**: BPE introduces ambiguity—"lowest" could be ["low", "est"] or ["lo", "w", "est"] depending on training data. But this ambiguity is beneficial—it forces the model to learn robust representations that work for multiple segmentations. ### LSTM Language Models **References**: [53, 54] N-gram LMs can't capture long-range dependencies beyond $n$ words. Neural LMs maintain a distributed hidden state that can theoretically capture unlimited context. **Word embeddings**: Map discrete words to dense vectors in $\mathbb{R}^d$ (typically $d=256$-$1024$). Similar words end up close in embedding space—the crucial generalization mechanism. **LSTM LM architecture**: Embed input word → LSTM → project to vocabulary size → softmax. Trained with cross-entropy on next-word prediction. **Perplexity**: Standard evaluation metric: $$ \text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log P(w_i | w_1...w_{i-1})\right) $$ Lower perplexity = better model. But lower perplexity doesn't always mean lower Word Error Rate in ASR—the LM must complement the acoustic model. **Integration into ASR**: LSTM LMs can be used in lattice rescoring (rerank N-best hypotheses) or in shallow/deep fusion. Deep fusion concatenates LM hidden state with acoustic encoder state—more powerful but requires joint training. ### BERT-Based Language Models in ASR **References**: [55, 56] **BERT architecture**: Bidirectional Encoder Representations from Transformers. Stack of Transformer encoder blocks. Pre-trained with: - **Masked Language Modeling**: Randomly mask 15% of tokens, predict them - **Next Sentence Prediction**: Predict if sentence B follows sentence A The key innovation: truly bidirectional context—each token attends to all others simultaneously. **Why BERT for ASR?**: In production ASR, a first-pass hypothesis is available before LM rescoring. BERT can rescore the full hypothesis using bidirectional context—not constrained to left-to-right like standard LMs. Extremely powerful for error correction. **My migration work**: Migrated production systems from LSTM-based neural LMs to BERT-based architectures while maintaining production stability. This involves: 1. Different inference paradigm (autoregressive vs. masked) 2. Integration as a rescorer rather than decoder-integrated LM 3. Managing latency (BERT is heavier than LSTM LMs) ### GPT-Style Autoregressive Language Models **References**: [56a, 56b, 56c] While BERT is bidirectional (encoder-only), **GPT (Generative Pre-trained Transformer)** is autoregressive (decoder-only)—it generates text one token at a time, conditioning on all previous tokens. This is the architecture behind ChatGPT, LLaMA, and most modern LLMs. **The autoregressive objective**: Maximize the likelihood of the next token given all previous tokens: $ P(x_1, ..., x_T) = \prod_{t=1}^T P(x_t | x_1, ..., x_{t-1}) $ **Causal masking**: To prevent the model from "cheating" by looking at future tokens, we use a causal attention mask: $ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right) V $ where $M$ is a lower-triangular mask: $ M_{ij} = \begin{cases} 0 & \text{if } i \geq j \\ -\infty & \text{if } i < j \end{cases} $ This ensures token $i$ can only attend to tokens $1, ..., i$ (past and present), not future tokens. **Architecture differences: BERT vs. GPT**: | Aspect | BERT (Encoder) | GPT (Decoder) | |--------|----------------|---------------| | Attention | Bidirectional (full) | Causal (masked) | | Training | Masked LM (predict masked tokens) | Next-token prediction | | Use case | Understanding (classification, NER) | Generation (text completion, chat) | | Context | Sees full sequence | Sees only past tokens | **Why GPT for speech-to-speech (Nova Sonic)**: Nova Sonic is a speech LLM—it takes audio input and generates audio output. The decoder must be autoregressive because: 1. **Generation is sequential**: Can't generate all output tokens simultaneously. Must generate one at a time. 2. **Streaming**: For real-time applications, we need to start generating output before seeing the full input. 3. **Conditioning**: Each output token conditions on all previous output tokens—this is autoregressive by definition. **GPT architecture details**: ``` Input: [BOS, x1, x2, ..., xT] For each position t: 1. Embed token: e_t = Embed(x_t) 2. Add positional encoding: e_t = e_t + PE(t) 3. Pass through N decoder blocks: - Causal self-attention (masked) - Feed-forward network - Residual connections + LayerNorm 4. Project to vocabulary: logits_t = Linear(h_t) 5. Sample next token: x_{t+1} ~ softmax(logits_t / T) ``` **Training objective**: Cross-entropy loss on next-token prediction: $ L = -\frac{1}{T}\sum_{t=1}^T \log P(x_t | x_1, ..., x_{t-1}) $ **Inference (autoregressive decoding)**: ```python def generate(model, prompt, max_length=100, temperature=1.0): """ Autoregressive text generation. Args: model: GPT model prompt: Initial tokens (list of token IDs) max_length: Maximum sequence length temperature: Sampling temperature Returns: Generated sequence """ tokens = prompt.copy() for _ in range(max_length - len(prompt)): # Forward pass (only on current sequence) logits = model(tokens) # Shape: (Seq, Vocab) # Get logits for last position next_logits = logits[-1] / temperature # Sample next token probs = softmax(next_logits) next_token = sample(probs) # Append to sequence tokens.append(next_token) # Stop if EOS token if next_token == EOS_TOKEN: break return tokens ``` **The quadratic cost problem**: For sequence length $T$, generating $T$ tokens requires $O(T^2)$ attention operations: - Token 1: Attends to 1 token - Token 2: Attends to 2 tokens - Token T: Attends to T tokens - Total: $1 + 2 + ... + T = O(T^2)$ This is why KV-caching is essential for production inference (covered in the Production ML section). **GPT scaling laws**: Larger GPT models are consistently better. The relationship between model size, data size, and performance is predictable: $ L(N, D) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} $ where $N$ is model parameters, $D$ is training tokens, and $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$ (from Chinchilla paper). **Production GPT models**: - **GPT-3**: 175B parameters, trained on 300B tokens - **LLaMA 2**: 7B-70B parameters, trained on 2T tokens - **GPT-4**: Estimated 1T+ parameters (not public) **My work on Nova Sonic**: The decoder is a GPT-style autoregressive model that generates audio tokens (from a neural codec). It conditions on the encoder output (audio understanding) and generates output audio tokens one at a time. This enables speech-to-speech translation, voice conversion, and speech enhancement—all in a single unified model. ## Parameter-Efficient Fine-Tuning and LoRA Modern models have billions of parameters. Full fine-tuning is expensive—it stores a separate copy of all parameters per task, requires optimizer states (Adam stores 2× model size), and can cause catastrophic forgetting. ### Why Full Fine-Tuning is Expensive **References**: [57, 58] For a 7B model: - Weights: 14GB (fp16) - Adam states: 56GB (2× weights for momentum and variance) - Total: 70GB+ per GPU **Catastrophic forgetting**: Full fine-tuning on task-specific data can overwrite general representations learned during pre-training. **The goal of PEFT**: Achieve near-full-fine-tuning performance while updating only a small fraction of parameters (<1% in LoRA). ### The PEFT Landscape: Adapters, Prefix Tuning, and LoRA **References**: [61, 98, 99] Before LoRA, several PEFT methods existed, each with different trade-offs: **Adapter layers (Houlsby et al., 2019)**: Insert small bottleneck layers between Transformer blocks: $$ h' = h + f_{\text{adapter}}(h) $$ where $f_{\text{adapter}}(h) = W_{\text{up}} \cdot \text{ReLU}(W_{\text{down}} \cdot h)$ with $W_{\text{down}} \in \mathbb{R}^{d \times r}$ and $W_{\text{up}} \in \mathbb{R}^{r \times d}$, $r \ll d$. **Pros**: Simple, modular (can swap adapters per task) **Cons**: Adds inference latency (extra forward passes), typically requires $r=64$-$128$ for good performance (~3-5% of parameters) **Prefix tuning (Li & Liang, 2021)**: Prepend learnable prefix vectors to keys and values in each layer: $$ \text{Attention}(Q, [K_{\text{prefix}}; K], [V_{\text{prefix}}; V]) $$ **Pros**: No architectural changes, no inference latency **Cons**: Reduces effective sequence length, harder to optimize, requires careful initialization **Prompt tuning (Lester et al., 2021)**: Simpler version—only prepend to input embeddings, not every layer. Works well for very large models (100B+) but poorly for smaller models (<10B). **Why LoRA won**: LoRA combines the best properties: - No inference latency (merge weights at deployment) - Works well even for small models (1B-10B) - Extremely parameter-efficient (0.1-1% of parameters) - Easy to implement and stable to train ### LoRA: Low-Rank Adaptation **References**: [59, 60, 61] LoRA is the dominant PEFT method in production. I use LoRA for periodic ASR model updates at Amazon. **Core insight: Low-rank hypothesis**: The weight updates during fine-tuning lie in a low-dimensional subspace: $$ \Delta W \approx BA $$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with $r \ll \min(d, k)$. **Mathematical formulation**: For a pre-trained weight $W_0 \in \mathbb{R}^{d \times k}$: $$ h = (W_0 + \Delta W)x = W_0 x + BAx $$ During training: $W_0$ is frozen, $B$ and $A$ are trainable. At initialization: $A \sim \mathcal{N}(0, \sigma^2)$, $B = 0$ → $\Delta W = 0$ at initialization. **Scaling factor**: LoRA uses $\frac{\alpha}{r}$ applied to $BA$: $$ h = W_0 x + \frac{\alpha}{r} BAx $$ This separates learning rate sensitivity from rank choice. **Rank selection**: $r=4$ to $r=64$ are common. Higher $r$ = more parameters = more expressivity but more risk of overfitting. For ASR adaptation, $r=8$ to $r=32$ is typical. **Which matrices to adapt?**: Original LoRA applies to Q and V in attention. Later work shows adapting all linear layers (Q, K, V, O, FFN up, FFN down) often helps. **Inference: Zero overhead**: At deployment, merge: $W = W_0 + BA$. The adapter adds no inference latency—key advantage over adapter methods. **LoRA merge implementation**: ```python import numpy as np def lora_merge_weights(W_base, lora_A, lora_B, alpha, rank): """ Merge LoRA adapter into base weights for zero-overhead inference. Args: W_base: Base model weights (D_out, D_in) lora_A: LoRA A matrix (Rank, D_in) lora_B: LoRA B matrix (D_out, Rank) alpha: LoRA scaling factor (typically 16-32) rank: LoRA rank (typically 4-64) Returns: W_merged: Merged weights (D_out, D_in) Inference vs. Training trade-off: - Training: Keep W_base frozen, update lora_A and lora_B Forward: y = (W_base + (alpha/rank) * lora_B @ lora_A) @ x Memory: Store W_base + lora_A + lora_B - Inference: Merge once, discard adapters Forward: y = W_merged @ x Memory: Store only W_merged (same as base model) Latency: Identical to base model (no adapter overhead) Production use case: - Hot-fix ASR errors: Train LoRA adapter on error cases - Merge adapter into production model - Deploy merged model (no latency penalty) - Rollback: Keep base model, remove adapter """ # Compute LoRA delta: (D_out, Rank) @ (Rank, D_in) = (D_out, D_in) lora_delta = np.dot(lora_B, lora_A) # Scale by alpha/rank lora_delta_scaled = (alpha / rank) * lora_delta # Merge into base weights W_merged = W_base + lora_delta_scaled return W_merged def lora_inference_comparison(): """ Compare inference with and without merging. """ # Setup d_in, d_out, rank = 512, 512, 16 alpha = 32 # Base model and LoRA adapter W_base = np.random.randn(d_out, d_in) * 0.01 lora_A = np.random.randn(rank, d_in) * 0.01 lora_B = np.zeros((d_out, rank)) # Initialized to zero # Input x = np.random.randn(d_in) # Method 1: Compute with adapter (training/inference without merge) y_adapter = np.dot(W_base, x) + (alpha / rank) * np.dot(lora_B, np.dot(lora_A, x)) # Method 2: Merge and compute (production inference) W_merged = lora_merge_weights(W_base, lora_A, lora_B, alpha, rank) y_merged = np.dot(W_merged, x) # Verify equivalence assert np.allclose(y_adapter, y_merged), "Merge should be mathematically equivalent" print(f"Adapter method: 2 matrix multiplications (W_base @ x, lora_B @ lora_A @ x)") print(f"Merged method: 1 matrix multiplication (W_merged @ x)") print(f"Speedup: ~2x for inference (no adapter overhead)") return W_merged ``` **Why this matters for production**: 1. **Hot-fixing ASR errors**: Train LoRA adapter on specific error cases (e.g., "Alexa, turn on the lights" misrecognized). Merge adapter into production model. Deploy with zero latency penalty. 2. **A/B testing**: Deploy base model to 95% of traffic, base + adapter to 5%. If adapter improves metrics, merge and deploy to 100%. If not, discard adapter—base model unchanged. 3. **Rollback**: Keep base model weights. If adapter causes regression, remove it instantly—no need to retrain or redeploy base model. 4. **Multiple adapters**: Train separate adapters for different domains (smart home, music, shopping). Merge the relevant adapter based on user context. Or merge all adapters additively (works if adapters are orthogonal). **Mathematical guarantee**: The merge is exact—$W_{\text{base}} + \frac{\alpha}{r} BA$ is mathematically equivalent to computing $W_{\text{base}}x + \frac{\alpha}{r} BAx$ separately. No approximation, no quality loss. **Memory footprint**: - **Training**: $|W_{\text{base}}| + |A| + |B| = d_{\text{in}} \times d_{\text{out}} + r \times (d_{\text{in}} + d_{\text{out}})$ - **Inference (merged)**: $|W_{\text{merged}}| = d_{\text{in}} \times d_{\text{out}}$ (same as base model) For $d=512$, $r=16$: Training adds $16 \times 1024 = 16$K parameters (6% overhead). Inference adds zero. **Continual LoRA for ASR updates**: My production use case—periodically release a new LoRA adapter that hot-fixes specific ASR errors, while the base model is unchanged. Multiple adapters can be merged additively. ## Post-Training Alignment Pre-training optimizes next-token prediction on internet-scale data. This makes models knowledgeable but not necessarily helpful, harmless, or aligned with user intent. ### Why Pre-Trained Models Need Alignment **References**: [62, 63] **Pre-training objective mismatch**: A pre-trained model might complete "How do I bake a cake?" with "How do I bake a pie?" (next likely token) rather than actual instructions. **Alignment objectives**: We want models to: - Follow instructions (helpfulness) - Avoid harmful outputs (harmlessness) - Be accurate (honesty) These are human preference objectives that can't be easily specified as a loss function on text. **The Alexa+ alignment problem**: For Nova Sonic specifically—the foundation model is trained on general audio/speech. Alignment for Alexa+ means: - Correct transcription errors on Alexa-specific queries (hot-fixing) - Handle Alexa command styles (steering) - Avoid hallucinating device capabilities ### Supervised Fine-Tuning (SFT) **References**: [64, 65] Collect demonstration data: (prompt, ideal response) pairs from human annotators. Fine-tune the pre-trained model on this data with standard cross-entropy loss. **Quality vs. quantity**: InstructGPT showed that 13k high-quality curated examples outperformed 100k noisy ones. Data quality matters more than scale for SFT. **SFT for speech models**: For Nova Sonic, SFT data includes (audio, correct transcript) pairs for Alexa-specific domains—smart home commands, shopping queries, music requests. **SFT limitations**: Can only teach behaviors that annotators demonstrate. Subtle misalignments are hard to fix with SFT alone. This motivates preference learning. ### RLHF: Reinforcement Learning from Human Feedback **References**: [66, 67, 68] RLHF is the technique that made ChatGPT possible. The key insight: instead of just predicting what humans write, optimize for what humans prefer. **Stage 1: SFT** (as above) **Stage 2: Reward model training**: - Show annotators pairs of model outputs (A, B) for the same prompt - Ask which is better - Train a reward model $r_\phi(x, y)$ to predict human preferences Loss (Bradley-Terry model): $$ L = -\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) $$ where $y_w$ is preferred (winning) and $y_l$ is less preferred (losing). **Bradley-Terry model**: Assumes preferences are determined by latent reward values: $$ P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l)) $$ **Stage 3: PPO optimization**: $$ \text{objective}(\theta) = \mathbb{E}_{(x,y) \sim \pi_\theta} [r_\theta(x, y) - \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}] $$ The KL penalty ($\beta$ term) prevents the policy from deviating too far from the SFT model—without it, the model exploits the reward model by generating nonsensical outputs that happen to score high. **PPO clipped objective**: $$ L^{\text{CLIP}}(\theta) = \mathbb{E}[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)] $$ where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the probability ratio. **Key findings**: InstructGPT showed that a 1.3B parameter model trained with RLHF outperformed the 175B parameter GPT-3 on human evaluations. Alignment matters more than raw scale. **Reward hacking problem**: Models can exploit weaknesses in the reward model. If the reward model incorrectly assigns high scores to certain patterns, the policy will learn to generate those patterns even if they're not actually better. ### DPO: Direct Preference Optimization **References**: [69, 70] DPO was a breakthrough: skip the reward model and RL entirely. The optimal policy for RLHF can be derived in closed form: $$ \pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r(x,y)\right) $$ Rearranging: $$ r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x) $$ Substituting into the Bradley-Terry preference model: $$ \mathcal{L}_{\text{DPO}}(\pi_\theta) = -\mathbb{E}_{(x,y_w,y_l)} \left[\log \sigma \left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right] $$ This is just binary cross-entropy! No reward model, no RL—just supervised learning on preference pairs. **Advantages**: 1. **Simplicity**: Single training stage 2. **Stability**: No reward hacking—DPO transforms the alignment problem into a stable classification objective on a static preference dataset 3. **Efficiency**: Faster training, less memory 4. **Performance**: Matches or exceeds PPO on many benchmarks **Why DPO is more stable than PPO**: PPO suffers from non-stationary policy-gradient issues: 1. **Reward model is fixed**: As the policy improves, it generates outputs the reward model hasn't seen. The reward model's predictions become less reliable (distribution shift). 2. **Policy updates affect data distribution**: Each policy update changes the distribution of outputs, which changes the effective training objective. This creates instability. 3. **Hyperparameter sensitivity**: PPO requires careful tuning of KL penalty $\beta$, learning rate, clipping threshold $\epsilon$. Small changes can cause training collapse. DPO avoids these issues: 1. **Static dataset**: Preferences are collected once, never change. No distribution shift during training. 2. **Supervised learning**: The loss is a fixed classification objective (binary cross-entropy on preferences). No moving target. 3. **Implicit reward**: The reward is implicitly defined by the policy itself: $r(x,y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$. This reward automatically adapts as the policy improves—no separate reward model to exploit. 4. **Fewer hyperparameters**: Only $\beta$ (temperature) needs tuning. Learning rate follows standard supervised learning practices. **The classification perspective**: DPO is binary classification on preference pairs. Given $(x, y_w, y_l)$, predict which output is preferred: $$ P(y_w \succ y_l | x) = \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right) $$ This is a well-understood supervised learning problem—no RL instability, no reward hacking, no non-stationarity. **When DPO works best**: Simpler alignment tasks, well-defined preferences, limited compute budgets. DPO has been successfully applied from 7B to 70B+ models—the key factor is preference quality, not model size. **When PPO wins**: Complex alignment (safety-critical), tasks requiring fine-grained control, very large models where the implicit reward needs explicit shaping. **The Stability-Plasticity Trade-off: Role of the Reference Model** The reference model $\pi_{\text{ref}}$ in DPO serves a critical dual purpose—it provides both stability (preventing catastrophic forgetting) and plasticity (allowing adaptation to preferences). **The implicit reward in DPO**: $$ r_{\text{DPO}}(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} $$ This log-ratio acts as a dynamic, self-regulating reward: 1. **When the model improves** ($\pi_\theta(y|x) > \pi_{\text{ref}}(y|x)$): The reward is positive, encouraging this behavior. 2. **When the model degrades** ($\pi_\theta(y|x) < \pi_{\text{ref}}(y|x)$): The reward becomes negative. The $\pi_{\text{ref}}$ term in the denominator increases, raising the loss and pulling the model back toward the reference. **Why this prevents forgetting**: If the model starts generating outputs that deviate too far from the reference (e.g., forgetting base capabilities like grammar or factual knowledge), the denominator $\pi_{\text{ref}}(y|x)$ remains high while $\pi_\theta(y|x)$ drops. This creates a large negative reward, effectively acting as an implicit KL penalty: $$ \mathbb{E}_{y \sim \pi_\theta} \left[\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}\right] = D_{\text{KL}}(\pi_\theta || \pi_{\text{ref}}) $$ **The stability-plasticity balance**: - **Stability**: The reference model anchors the policy, preventing drift into regions where base capabilities degrade - **Plasticity**: The policy can still adapt to preferences—the log-ratio allows movement in directions that improve preference satisfaction while staying grounded **Temperature parameter $\beta$**: Controls the trade-off: - **High $\beta$** ($> 0.5$): Strong anchoring to reference, high stability, limited plasticity - **Low $\beta$** ($< 0.1$): Weak anchoring, high plasticity, risk of forgetting **Production intuition**: The reference model is like a "home base" in optimization space. The policy can explore to find better preference-aligned outputs, but the implicit KL penalty always pulls it back if it wanders too far. This is why DPO doesn't need explicit KL constraints like PPO—the constraint is baked into the reward formulation. ### Production Reality: Reward Model Hacking and Ambiguous Preferences **The reward hacking problem in depth**: Reward models are trained on a finite dataset of human preferences. They learn to approximate human judgment but inevitably have blind spots. During RL training, the policy explores the space of possible outputs and discovers inputs where the reward model gives high scores despite the output being poor quality. **Common reward hacking patterns**: 1. **Length exploitation**: Reward models often prefer longer responses regardless of quality. The policy learns to pad outputs with verbose but low-information content. 2. **Style mimicry**: Reward models may associate certain stylistic patterns (formal language, bullet points, specific phrases) with quality. The policy learns to generate these patterns even when inappropriate. 3. **Confidence gaming**: Reward models trained on preferences may reward confident-sounding responses over accurate but uncertain ones. The policy learns to be confidently wrong. 4. **Adversarial examples**: The policy finds specific token sequences that trigger high reward model scores through spurious correlations in the training data. **Quantifying reward hacking**: Monitor the KL divergence between policy and reference model: $$ D_{\text{KL}}(\pi_\theta || \pi_{\text{ref}}) = \mathbb{E}_{y \sim \pi_\theta} \left[\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}\right] $$ As KL increases, the policy moves into regions where the reward model is less reliable. The KL penalty $\beta$ in the RLHF objective controls this trade-off: - **Low $\beta$** ($< 0.01$): Policy drifts far from reference, high reward hacking risk - **High $\beta$** ($> 0.1$): Policy stays close to reference, limited improvement **Production mitigation strategies**: 1. **Ensemble reward models**: Train multiple reward models on different data splits. Use the minimum score across models to reduce exploitation of any single model's weaknesses. 2. **Adversarial reward model training**: Periodically generate outputs from the current policy, have humans label them, and retrain the reward model. This closes the distribution gap. 3. **Rule-based constraints**: Hard constraints on length, repetition, and format violations that override reward model scores. 4. **Human-in-the-loop validation**: Sample outputs from the policy, have humans verify quality before deployment. Reject updates that show reward hacking patterns. **Ambiguous preferences**: The fundamental challenge Human preferences are often inconsistent, context-dependent, and subjective. Two annotators may disagree on which response is better. The same annotator may give different preferences on different days. **Measuring preference ambiguity**: Inter-annotator agreement (Cohen's kappa): $$ \kappa = \frac{p_o - p_e}{1 - p_e} $$ where $p_o$ is observed agreement and $p_e$ is expected agreement by chance. Typical values: - $\kappa > 0.8$: Strong agreement (factual correctness, grammar) - $\kappa = 0.4$-$0.6$: Moderate agreement (helpfulness, tone) - $\kappa < 0.4$: Weak agreement (creativity, style preferences) **Why this matters**: The reward model is trained to predict a single "correct" preference, but the ground truth is a distribution. When preferences are ambiguous, the reward model overfits to noise in the training data. **Production approaches**: 1. **Confidence-weighted preferences**: Instead of binary preferences, collect confidence scores. Weight the loss by annotator confidence. 2. **Multi-objective reward models**: Train separate reward models for different aspects (helpfulness, harmlessness, honesty) and combine them with learned or hand-tuned weights. 3. **Uncertainty quantification**: Use ensemble disagreement or Bayesian neural networks to estimate reward model uncertainty. Penalize the policy for generating outputs with high reward uncertainty. **The DPO vs. PPO trade-off in production**: **Choose DPO when**: - **Preference data is clean and consistent**: DPO directly optimizes on preferences, so noisy data hurts more than PPO - **Compute budget is limited**: DPO is 2-3× faster than PPO (no reward model training, no value network) - **Stability is critical**: DPO has no hyperparameters for KL penalty tuning, no risk of reward hacking - **Task is well-defined**: Instruction following, summarization, translation—clear right/wrong answers **Choose PPO when**: - **Safety is paramount**: PPO allows fine-grained control through reward shaping. You can add hard constraints (toxicity classifiers, rule-based filters) as negative rewards - **Preferences are ambiguous**: PPO's reward model can be retrained iteratively as the policy improves. DPO is offline—stuck with initial preference data - **Complex multi-objective alignment**: PPO can combine multiple reward signals (helpfulness + harmlessness + factuality) with learned weights. DPO requires pre-combining preferences - **Very large models**: PPO's value network helps with credit assignment in long sequences. DPO's implicit reward can be noisy for 100B+ models **Real production example (Nova Sonic)**: For speech alignment, I use DPO because: 1. **Preferences are unambiguous**: Correct transcript vs. incorrect transcript is objective 2. **Data is clean**: Transcription errors are verified by human annotators 3. **Stability matters**: Hot-fixing production models requires predictable behavior 4. **Compute budget**: DPO allows rapid iteration (hours vs. days for PPO) But for general LLM alignment (helpfulness, harmlessness), PPO is often better because: 1. **Preferences are subjective**: "Helpful" depends on user intent, context, and personal preference 2. **Safety is critical**: Need to combine toxicity classifiers, PII detectors, and factuality checks as negative rewards 3. **Iterative improvement**: Reward model is retrained weekly as the policy improves **The hybrid approach**: Many production systems use DPO for initial alignment (fast, stable) followed by PPO for fine-tuning on safety-critical aspects (flexible, controllable). This combines the best of both worlds. **DPO for speech alignment**: For Nova Sonic, preference pairs are (audio, transcript_correct, transcript_wrong). The model is trained to prefer the correct transcript. This is my post-training alignment work—hot-fixing specific transcription errors. ## Continual Learning When a neural network is trained sequentially on tasks $T_1$ then $T_2$, performance on $T_1$ degrades severely. This is catastrophic forgetting. ### The Catastrophic Forgetting Problem **References**: [71, 72] **Why it happens**: All tasks share the same parameters. Gradient updates for $T_2$ move weights in directions optimal for $T_2$, which are likely destructive for $T_1$'s learned function. **The production context**: For Alexa ASR, the base model is trained on massive data. New hot-fixes and domain updates must be applied periodically without degrading general ASR performance. A naive fine-tune on new error cases will cause regression on existing capabilities. ### Elastic Weight Consolidation (EWC) **References**: [73, 74] **Core idea**: Some parameters matter more to previous tasks than others. EWC identifies these important parameters and penalizes large updates to them. **Fisher Information as importance measure**: $$ F_i = \mathbb{E}\left[\left(\frac{\partial \log p(y|x,\theta)}{\partial \theta_i}\right)^2\right] $$ High Fisher information means the parameter strongly influences predictions—changing it will hurt performance. **EWC loss**: $$ L_{\text{EWC}}(\theta) = L_B(\theta) + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta^*_{A,i})^2 $$ $L_B$ is the new task loss. The second term penalizes deviation from old task parameters $\theta^*_A$, weighted by their Fisher importance $F_i$. **Practical computation**: The full Fisher matrix is $d \times d$ where $d$ can be billions—intractable. In practice: 1. Diagonal Fisher approximation (one value per parameter) 2. Computed on a subset of old task data 3. Stored as a fixed constant after task A **EWC in production ASR**: Apply EWC when fine-tuning on new error cases—the Fisher Information from the original training data discourages changing parameters important for existing capabilities. ### Replay Methods **References**: [75, 76] **Experience replay**: Maintain a memory buffer of old task examples. During new task training, mix old and new examples in each mini-batch. The simplest effective approach—if you can store old data. **Coreset selection**: Choosing which old examples to store is non-trivial if storage is limited. Coreset selection algorithms (greedy k-center, herding) pick a small subset that best approximates the full data distribution. **Golden set / Regression testing**: My patent approach—maintain a curated "golden set" of examples that cover important capabilities. After each model update, evaluate on the golden set before deployment. Reject updates that cause regression beyond a threshold. ### My Patent: Continuous Learning for ML Models (US 12488798) **References**: [77] **The hot-fixing problem**: Production ASR systems receive defect reports—specific utterances or query types where the model fails. The team wants to fix these quickly without degrading existing performance. Full retraining is too expensive (weeks). Fine-tuning is too risky (regression). **The mixing strategy**: The patent's key contribution—a principled mixing strategy that combines hot-fix training data with a replay of golden set data. The mixing ratio is tuned to balance correction of new errors against preservation of existing capabilities. **Deployment validation pipeline**: Automated evaluation on golden set before each deployment. Metrics: - WER on general test sets - WER on specific error categories being fixed - Regression rate on the golden set Only deploy if all metrics are within acceptance bounds. **Connection to LoRA**: In the modern version of this system (my current work), LoRA adapters carry the hot-fix delta rather than modifying base model weights directly. This decouples the hot-fix from the base model and makes rollback trivial—just don't apply the adapter. ## Federated Learning In standard ML, all training data is centralized on a server. For voice assistants, users' audio is sensitive—it may contain personal conversations, medical information, financial data. Federated learning trains on data that never leaves the device. ### Why Federated Learning? **References**: [78, 79] **The privacy problem**: Regulations (GDPR, CCPA) restrict data centralization. Federated learning trains on data that never leaves the device. **The federated setting**: $N$ clients (devices, silos) each hold a local dataset $D_i$. A central server coordinates training. Goal: train a global model as if all data were centralized, without any raw data leaving clients. **Cross-device vs. cross-silo**: - **Cross-device**: Millions of mobile devices, each with small dataset, unreliable connection, heterogeneous hardware (Alexa devices) - **Cross-silo**: Small number of institutional participants (hospitals, enterprise partners), reliable connections, larger datasets per participant My gRPC system is cross-silo. ### Federated Averaging (FedAvg) **References**: [80, 81] **Algorithm**: 1. Server sends global model $w$ 2. Clients run $E$ local gradient steps on their data 3. Clients send updated local weights $w_i$ back to server 4. Server aggregates: $w_{\text{global}} = \sum_i \frac{n_i}{n} w_i$ where $n_i$ is client $i$'s data size 5. Repeat **Why weighted averaging works**: FedAvg is equivalent to gradient descent on the sum of local losses when clients run just one step. With multiple local steps, it diverges—this is the "client drift" problem. **Communication efficiency**: Communication is the bottleneck. Techniques: - Gradient compression (sparsification, quantization) - Local SGD (more local steps before communication) - Federated distillation (communicate predictions not weights) **Statistical heterogeneity (non-IID data)**: Each client's data distribution is different—Alexa devices in different countries hear different accents, commands, noise. FedAvg can converge to a model that performs well on average but poorly for some clients. ### My Federated Learning System **References**: [82] **gRPC-based architecture**: gRPC (Google Remote Procedure Call) is a high-performance RPC framework using Protocol Buffers for serialization. For FL: - Clients call `GetGlobalModel` and `SubmitUpdate` RPCs - Protocol Buffers enable efficient serialization of model weights **Convergence challenges in cross-silo**: In cross-silo FL with diverse clients (e.g., different country Alexa deployments), gradient directions conflict. My "novel convergence techniques" likely include: - **FedProx**: Adds proximal term to local objective to prevent client drift - **SCAFFOLD**: Variance reduction using control variates - Custom aggregation strategies that weight client updates by data quality **SSL in federated settings**: Self-Supervised Learning in federated settings—clients can pre-train on local unlabeled data using contrastive or masked objectives, then share only the learned representations or gradients for the SSL task. Particularly powerful for speech—unlabeled audio is abundant on-device, labeled data is scarce. ### Differential Privacy in Federated Learning **References**: [82a, 82b, 82c] Basic federated learning provides limited privacy—the server still sees model updates, which can leak information through gradient inversion attacks. **Differential Privacy (DP)** provides formal privacy guarantees by adding calibrated noise to gradients before sharing. **The formal definition**: A randomized mechanism $M$ is $(\epsilon, \delta)$-differentially private if for all datasets $D$ and $D'$ differing in one record, and all possible outputs $S$: $ P(M(D) \in S) \leq e^\epsilon \cdot P(M(D') \in S) + \delta $ **Intuition**: An adversary observing the output $M(D)$ cannot tell whether any individual's data was included in $D$. The parameters: - **$\epsilon$ (privacy budget)**: Lower = stronger privacy. $\epsilon < 1$ is strong, $\epsilon > 10$ is weak. - **$\delta$ (failure probability)**: Probability that privacy guarantee fails. Typically $\delta < 10^{-5}$. **Why this matters**: Even if an attacker has access to the model updates, they cannot infer whether a specific user's data was used in training. This is a mathematical guarantee, not just a heuristic. **The Gaussian Mechanism**: Add Gaussian noise calibrated to the sensitivity of the function: $ M(D) = f(D) + \mathcal{N}(0, \sigma^2) $ where $\sigma = \frac{\sqrt{2\log(1.25/\delta)} \cdot \Delta f}{\epsilon}$ and $\Delta f$ is the sensitivity (maximum change in $f$ when one record changes). **For gradient descent**: The gradient $g = \nabla L(D)$ has sensitivity $\Delta g$ (bounded by gradient clipping). Add noise: $ \tilde{g} = g + \mathcal{N}(0, \sigma^2 I) $ where $\sigma = \frac{C \sqrt{2\log(1.25/\delta)}}{B\epsilon}$, $C$ is the clipping threshold, and $B$ is the batch size. **DP-SGD (Differentially Private Stochastic Gradient Descent)**: ```python def dp_sgd_step(model, batch, epsilon, delta, C=1.0): """ One step of DP-SGD. Args: model: Neural network model batch: Training batch epsilon: Privacy budget per step delta: Failure probability C: Gradient clipping threshold Returns: Noisy gradient """ # Compute per-example gradients per_example_grads = [] for x, y in batch: grad = compute_gradient(model, x, y) per_example_grads.append(grad) # Clip each gradient to bound sensitivity clipped_grads = [] for grad in per_example_grads: grad_norm = np.linalg.norm(grad) if grad_norm > C: grad = grad * (C / grad_norm) clipped_grads.append(grad) # Average clipped gradients avg_grad = np.mean(clipped_grads, axis=0) # Add Gaussian noise sigma = C * np.sqrt(2 * np.log(1.25 / delta)) / (len(batch) * epsilon) noise = np.random.normal(0, sigma, size=avg_grad.shape) noisy_grad = avg_grad + noise return noisy_grad ``` **Privacy accounting**: Each gradient update consumes privacy budget. After $T$ steps, the total privacy is $(\epsilon_{\text{total}}, \delta)$ where: $ \epsilon_{\text{total}} = \epsilon \sqrt{2T\log(1/\delta)} $ This is the **composition theorem**—privacy degrades with more updates. For long training, $\epsilon_{\text{total}}$ can become large (weak privacy). **Advanced composition (RDP)**: Rényi Differential Privacy provides tighter bounds: $ \epsilon_{\text{total}} \approx \epsilon \sqrt{T} $ This is much better than naive composition ($\epsilon T$) but still grows with training time. **The privacy-utility trade-off**: | Privacy ($\epsilon$) | Noise level | Accuracy impact | |---------------------|-------------|-----------------| | $\epsilon = 0.1$ | Very high | -10% to -20% | | $\epsilon = 1.0$ | High | -3% to -5% | | $\epsilon = 10.0$ | Moderate | -1% to -2% | | No DP | None | Baseline | **Production DP-FL**: For federated learning with differential privacy: 1. **Client-side DP**: Each client adds noise to their local gradients before sending to server. Privacy guarantee: server cannot infer individual user's data. 2. **Server-side DP**: Server adds noise to aggregated gradients. Privacy guarantee: clients cannot infer other clients' data. 3. **Hybrid**: Both client and server add noise. Strongest privacy but highest accuracy cost. **My work on federated learning**: For cross-silo FL (enterprise partners, not individual users), we use $\epsilon = 10$ (moderate privacy) to balance privacy and accuracy. For cross-device FL (millions of mobile devices), we use $\epsilon = 1$ (strong privacy) because individual user data is more sensitive. **The gradient inversion attack**: Without DP, an attacker can reconstruct training data from gradients: 1. Initialize random data $\tilde{x}$ 2. Compute gradient $\tilde{g} = \nabla L(\tilde{x})$ 3. Minimize $\|\tilde{g} - g\|^2$ to find $\tilde{x}$ that produces gradient $g$ 4. Recover original training data $x \approx \tilde{x}$ This works surprisingly well for small batches (batch size 1-8). DP prevents this by adding noise that overwhelms the signal. **Production reality**: DP is expensive—it requires per-example gradients (no batch-level optimization), gradient clipping (limits learning rate), and noise addition (degrades accuracy). But for privacy-critical applications (healthcare, finance), it's the only way to provide formal guarantees. **My ICASSP 2021 paper**: Cross-silo Federated Training with Diversity Scaling and SSL. The key contribution: diversity scaling—weight client updates by their diversity to prevent dominant clients from overwhelming the global model. **The Non-IID problem in depth**: In cross-silo FL, each client (e.g., country-specific Alexa deployment) has vastly different data distributions: - **US English**: Rhotic accent, fast speech rate, smart home commands dominate - **UK English**: Non-rhotic accent, different vocabulary ("lift" vs. "elevator") - **Indian English**: Retroflex consonants, code-switching with Hindi/Tamil Standard FedAvg weights clients by data size: $w_{\text{global}} = \sum_i \frac{n_i}{n} w_i$. This means the US client (largest data) dominates the global model, causing poor performance for UK and Indian clients. **Diversity Scaling (FedAvg-DS)**: Weight clients by their contribution to model diversity, not just data size. **Measuring client diversity**: Compute the gradient diversity between client $i$ and the global model: $$ d_i = \|\nabla L_i(w_{\text{global}}) - \nabla L_{\text{global}}(w_{\text{global}})\|_2 $$ High $d_i$ means client $i$'s data distribution is very different from the global distribution—this client provides unique information. **Diversity-scaled aggregation**: $$ w_{\text{global}} = \sum_i \alpha_i w_i $$ where $\alpha_i = \frac{n_i \cdot d_i}{\sum_j n_j \cdot d_j}$ This balances data size (clients with more data get more weight) with diversity (clients with unique data get more weight). **Why this works**: Clients with rare accents or domains have high diversity—their gradients point in directions the global model hasn't explored. By upweighting them, we prevent the global model from overfitting to the dominant client's distribution. **Practical implementation challenges**: 1. **Computing gradient diversity is expensive**: Requires computing gradients on the global model for each client's data. Solution: Approximate using a small validation set. 2. **Diversity changes over training**: Early in training, all clients have high diversity. Late in training, only truly unique clients have high diversity. Solution: Recompute diversity weights every $K$ rounds. 3. **Privacy**: Gradient diversity reveals information about client data distributions. Solution: Add differential privacy noise to diversity scores. **Experimental results (ICASSP 2021)**: - **Baseline FedAvg**: WER = 12.3% (US), 18.7% (UK), 22.1% (India) - **FedAvg-DS**: WER = 12.5% (US), 15.2% (UK), 17.8% (India) Diversity scaling slightly hurts the dominant client (US) but significantly helps minority clients (UK, India). The global average WER improves from 17.7% to 15.2%. **Connection to fairness**: Diversity scaling is a fairness mechanism—it prevents the model from being biased toward the majority group. This is critical for production systems serving diverse user populations. ## Distributed Training A 7B parameter model in fp16: 14GB just for weights. Adam optimizer states: 28GB. Gradients: 14GB. Total: 56GB+ per GPU. A 100B model is impossible on a single GPU. ### Why Distributed Training? **References**: [83, 84] **Memory constraints**: Even if the model fits, training takes too long on one GPU. GPT-3 (175B) would take 355 GPU-years on a single V100. With 1024 A100s, training takes weeks. **The communication-compute trade-off**: Distributed training requires synchronizing gradients or activations across GPUs. Network bandwidth is orders of magnitude slower than GPU compute. All parallelism strategies are different ways to manage this trade-off. ### Data Parallelism and DDP **References**: [85, 86] **Naive data parallelism**: Copy the full model to each GPU. Each GPU processes a different batch. Synchronize gradients at each step: all-reduce (average gradients across GPUs), then update. **All-Reduce**: The fundamental collective communication operation. Ring all-reduce organizes GPUs in a ring, each GPU sends a chunk to its neighbor and receives from the other direction. After $N$ rounds, all GPUs have the full gradient sum. Bandwidth-optimal: communication volume = $\frac{2(N-1)}{N} \times \text{model\_size}$ per GPU. **PyTorch DDP**: DistributedDataParallel overlaps backward pass computation with gradient communication using gradient buckets. As soon as a bucket of gradients is ready, their all-reduce starts before the rest of backprop completes. This hides communication latency. **DDP scaling efficiency**: Linear scaling means 2× GPUs should give 2× throughput. In practice: 70-90% efficiency due to communication overhead, stragglers, and data loading. ### ZeRO: Zero Redundancy Optimizer **References**: [87, 88] ZeRO is DeepSpeed's flagship memory optimization. It eliminates redundancy in data-parallel training by partitioning optimizer states, gradients, and model weights across GPUs. **ZeRO Stage 1: Optimizer state partitioning**: Each GPU holds only $1/N$ of the optimizer states (momentum, variance for Adam). After computing gradients, each GPU updates only its shard of parameters. All-gather to broadcast updated parameters. Memory per parameter: $4$ bytes (fp16 weights) + $4$ bytes (fp16 gradients) + $\frac{12}{N}$ bytes (fp32 optimizer states divided across GPUs) = $8 + \frac{12}{N}$ bytes. For large $N$, this approaches 8 bytes/parameter vs. 20 bytes/parameter for standard data parallelism. **ZeRO Stage 2: + Gradient partitioning**: Each GPU also holds only $1/N$ of gradients. Reduce-scatter instead of all-reduce—each GPU accumulates only its gradient shard. **ZeRO Stage 3: + Parameter partitioning**: Each GPU holds only $1/N$ of model parameters. Before each layer's forward/backward pass, gather the full layer weights (all-gather), compute, then discard. Completely eliminates redundancy but increases communication to 3× DDP. **ZeRO-Offload and ZeRO-Infinity**: Move optimizer states and gradients to CPU memory (or NVMe for Infinity). This allows training much larger models but at reduced throughput due to CPU-GPU transfer bandwidth. ### System Bottlenecks: Communication, Compute, and the Bubble Problem **References**: [89a, 89b] Understanding distributed training requires understanding where time is actually spent. Modern GPUs can perform 300+ TFLOPS (A100) or 1000+ TFLOPS (H100), but network bandwidth is orders of magnitude slower. This creates fundamental bottlenecks. **The communication-compute overlap problem**: Ideal distributed training: communication happens entirely during computation, adding zero overhead. Reality: communication and computation partially overlap, with exposed communication time causing GPUs to idle. **Quantifying overlap**: Define the communication-to-computation ratio: $$ \rho = \frac{T_{\text{comm}}}{T_{\text{comp}}} $$ where $T_{\text{comm}}$ is time to communicate gradients and $T_{\text{comp}}$ is time to compute gradients. - **$\rho < 1$**: Communication can be fully hidden by computation (ideal) - **$\rho > 1$**: GPUs idle waiting for communication (bottleneck) **For data parallelism**: $T_{\text{comm}} = \frac{2M}{B \cdot N}$ where $M$ is model size, $B$ is bandwidth, and $N$ is number of GPUs (ring all-reduce). $T_{\text{comp}} = \frac{F}{C}$ where $F$ is FLOPs per batch and $C$ is compute throughput. **Example calculation** (100B model, 8 A100 GPUs): - Model size: $M = 200$ GB (fp16 weights + gradients) - NVLink bandwidth: $B = 600$ GB/s per GPU - Communication time: $T_{\text{comm}} = \frac{2 \times 200}{600 \times 8} = 0.083$ seconds - Computation time (batch size 32, seq len 2048): $T_{\text{comp}} \approx 0.5$ seconds - Ratio: $\rho = 0.17$ → communication is 17% of compute time, mostly hidden **Interconnect bandwidth: The critical bottleneck** **NVLink vs. PCIe vs. InfiniBand**: | Interconnect | Bandwidth (per GPU) | Latency | Use Case | |--------------|---------------------|---------|----------| | NVLink 3.0 (A100) | 600 GB/s | ~1 μs | Intra-node (8 GPUs) | | NVLink 4.0 (H100) | 900 GB/s | ~1 μs | Intra-node (8 GPUs) | | PCIe 4.0 | 32 GB/s | ~5 μs | CPU-GPU, older systems | | PCIe 5.0 | 64 GB/s | ~5 μs | CPU-GPU, newer systems | | InfiniBand HDR | 200 Gb/s = 25 GB/s | ~1 μs | Inter-node | | InfiniBand NDR | 400 Gb/s = 50 GB/s | ~1 μs | Inter-node (latest) | **Why this matters**: 1. **Intra-node (NVLink)**: 8 GPUs connected via NVLink can communicate at 600 GB/s. All-reduce of 200 GB takes ~0.08s. Communication is negligible. 2. **Inter-node (InfiniBand)**: 64 GPUs across 8 nodes. Each node has 8 GPUs. All-reduce now involves: - Intra-node all-reduce (fast, NVLink) - Inter-node all-reduce (slow, InfiniBand) - Communication time dominated by inter-node: $T_{\text{comm}} = \frac{2 \times 200}{25 \times 8} = 2$ seconds - Now $\rho = 4$ → communication takes 4× longer than computation! **Production implications**: - **Scaling efficiency drops at node boundaries**: 8 GPUs → 90% efficiency. 64 GPUs → 60% efficiency. 512 GPUs → 40% efficiency. - **Batch size must increase with scale**: To maintain $\rho < 1$, increase batch size (more compute per communication). But large batch sizes hurt convergence. - **Gradient accumulation**: Accumulate gradients over multiple micro-batches before all-reduce. Reduces communication frequency at the cost of staleness. **The all-reduce algorithm matters**: **Ring all-reduce** (standard): - Communication volume: $\frac{2(N-1)}{N} \times M$ per GPU - Bandwidth-optimal but latency-bound for small messages - Time: $\frac{2M}{B}$ (independent of $N$ for large $M$) **Tree all-reduce** (hierarchical): - Communication volume: $2 \log_2(N) \times M$ per GPU - Better for small messages (lower latency) - Time: $\frac{2M \log_2(N)}{B}$ (grows with $N$) **Bucket all-reduce** (PyTorch DDP): - Split gradients into buckets (~25 MB each) - Start all-reduce as soon as a bucket is ready - Overlaps communication with backward pass - Critical for hiding communication latency **The Bubble Problem in 3D Parallelism**: Pipeline parallelism introduces pipeline bubbles—periods where GPUs are idle waiting for data from the previous stage. **Naive pipeline** (GPipe): ``` GPU 0: [F1][F2][F3][F4] [B4][B3][B2][B1] GPU 1: [F1][F2][F3][F4] [B4][B3][B2][B1] GPU 2: [F1][F2][F3][F4][B4][B3][B2][B1] GPU 3: [F1][F2][F3][F4][B3][B2][B1] ``` where F = forward, B = backward. The bubble (idle time) is: $$ \text{Bubble} = \frac{(P-1)}{M} \times 100\% $$ where $P$ is number of pipeline stages and $M$ is number of micro-batches. **Example**: 4 stages, 4 micro-batches → 75% bubble! GPUs are idle 75% of the time. **Solution: Increase micro-batches**: With $M = 16$ micro-batches: $$ \text{Bubble} = \frac{3}{16} = 18.75\% $$ Much better, but requires 4× more memory for activations. **Interleaved pipeline (Megatron-LM)**: Instead of assigning consecutive layers to GPUs, interleave them: - GPU 0: layers 1, 5, 9, 13 - GPU 1: layers 2, 6, 10, 14 - GPU 2: layers 3, 7, 11, 15 - GPU 3: layers 4, 8, 12, 16 This reduces bubble to: $$ \text{Bubble} = \frac{(P-1)}{M + P - 1} \times 100\% $$ With $P=4$, $M=8$: Bubble = 27% (vs. 37.5% for naive pipeline). **1F1B schedule** (One-Forward-One-Backward): Instead of completing all forwards before backwards, interleave them: ``` GPU 0: [F1][F2][F3][F4][B1][F5][B2][F6][B3][F7][B4][F8][B5][B6][B7][B8] GPU 1: [F1][F2][F3][B1][F4][B2][F5][B3][F6][B4][F7][B5][F8][B6][B7][B8] GPU 2: [F1][F2][B1][F3][B2][F4][B3][F5][B4][F6][B5][F7][B6][F8][B7][B8] GPU 3: [F1][B1][F2][B2][F3][B3][F4][B4][F5][B5][F6][B6][F7][B7][F8][B8] ``` This reduces peak memory (don't need to store all forward activations) and maintains low bubble. **Production reality: The 3D parallelism configuration problem**: For a 100B model on 512 GPUs, how do you configure Data × Tensor × Pipeline parallelism? **Constraints**: 1. **Memory**: Model must fit in GPU memory with chosen parallelism 2. **Communication**: Minimize exposed communication time 3. **Bubble**: Keep pipeline bubble < 20% 4. **Batch size**: Large enough for convergence, small enough for memory **Example configuration** (100B model, 512 A100 GPUs, 64 nodes): - **Tensor Parallelism**: $T = 8$ (within node, NVLink) - Splits each layer across 8 GPUs - Communication: all-reduce per layer (fast, NVLink) - Memory: $\frac{M}{8}$ per GPU - **Pipeline Parallelism**: $P = 8$ (across nodes, InfiniBand) - 8 pipeline stages, each stage is 8 GPUs (tensor parallel) - Communication: point-to-point between stages (slow, InfiniBand) - Bubble: 18.75% with 32 micro-batches - **Data Parallelism**: $D = 8$ (across node groups) - 8 replicas of the full model - Communication: all-reduce of gradients (slow, InfiniBand) - Batch size: $8 \times 32 = 256$ global batch size **Total**: $T \times P \times D = 8 \times 8 \times 8 = 512$ GPUs. **Why this configuration**: - Tensor parallelism within nodes (fast NVLink) - Pipeline parallelism across nodes (minimizes inter-node communication) - Data parallelism for throughput (embarrassingly parallel) **Production gotchas**: 1. **Stragglers**: One slow GPU slows the entire pipeline. Requires careful load balancing and fault tolerance. 2. **Gradient accumulation**: With 32 micro-batches, gradients are stale by 32 steps. This can hurt convergence—requires careful learning rate tuning. 3. **Memory fragmentation**: ZeRO Stage 3 + Pipeline Parallelism can cause memory fragmentation. Requires careful memory management. 4. **Debugging**: Distributed training failures are hard to debug. One GPU OOM crashes all 512 GPUs. Requires extensive logging and checkpointing. ### Tensor Parallelism and Pipeline Parallelism **References**: [89, 90, 91] **Tensor Parallelism (Megatron)**: Split individual operations (matrix multiplications) across GPUs. For a linear layer $Y = XA$: partition $A$ column-wise across $N$ GPUs. Each GPU computes $XA_i$. Concatenate results. For attention: split attention heads across GPUs—each GPU computes a subset of heads. **Pipeline Parallelism**: Assign consecutive layers to consecutive GPUs. GPU 1 runs layers 1-4, GPU 2 runs layers 5-8, etc. The problem: GPUs are idle while data passes through other GPUs (pipeline bubble). Solution: micro-batching (split mini-batch into micro-batches, fill the pipeline). **Sequence Parallelism**: Extension of tensor parallelism for long sequences **References**: [102, 103] For speech models processing long audio (30+ seconds = 3000+ frames), even with tensor parallelism, a single sequence may not fit in GPU memory. Sequence parallelism splits the sequence dimension across GPUs. **The problem**: Tensor parallelism splits the model dimension (hidden size, attention heads) but each GPU still processes the full sequence. For a Conformer with $d=512$, sequence length $n=3000$, batch size $b=32$: Activation memory per layer: $b \times n \times d \times 2 = 32 \times 3000 \times 512 \times 2 = 98$ MB With 16 layers and gradients: $98 \times 16 \times 2 = 3.1$ GB just for activations. Add KV-cache, optimizer states → OOM. **Sequence parallelism solution**: Split the sequence dimension across GPUs in regions where it's safe. **Where sequence parallelism applies**: 1. **LayerNorm**: Each position is independent → split across sequence dimension 2. **Dropout**: Each position is independent → split across sequence dimension 3. **FFN**: Each position is independent → split across sequence dimension **Where it doesn't apply**: 1. **Self-attention**: Each position attends to all others → requires all-gather before attention, reduce-scatter after 2. **Convolution**: Requires local context → requires halo exchange (communicate boundary frames) **Algorithm** (for FFN layer with tensor + sequence parallelism): 1. Input $X \in \mathbb{R}^{b \times n \times d}$ is partitioned across $N$ GPUs: - Sequence dimension: each GPU has $n/N$ frames - Model dimension: each GPU has $d/N$ features 2. FFN forward: $Y = \text{GELU}(XW_1)W_2$ - $W_1$ is column-partitioned (tensor parallelism) - Each GPU computes on its $n/N$ frames (sequence parallelism) - All-reduce after $W_2$ to sum across tensor parallel GPUs 3. Memory: Each GPU stores $b \times (n/N) \times d$ activations → $N$× memory reduction **Communication pattern**: ``` Attention: All-gather(sequence) → Attention → Reduce-scatter(sequence) FFN: (No sequence communication needed) All-reduce(tensor) after output projection ``` **Memory savings**: For $N=8$ GPUs with sequence parallelism, activation memory drops from 3.1 GB to 0.39 GB per GPU—enables 8× longer sequences or 8× larger batch size. **Production trade-off**: Sequence parallelism adds all-gather/reduce-scatter communication before/after attention. For short sequences (<1000 frames), this overhead outweighs the memory benefit. For long sequences (>2000 frames), it's essential. **My work on Nova Sonic**: We use sequence parallelism for training on long-form audio (podcasts, meetings, lectures). Without it, we'd be limited to 10-second clips. With it, we can train on 60-second clips, which significantly improves model quality for long-form transcription. **3D Parallelism**: Combine all three: Data Parallelism × Tensor Parallelism × Pipeline Parallelism. This is how models like GPT-3 and Nova Sonic are trained. Each dimension reduces a different bottleneck. ### 4D Parallelism: Adding Sequence Parallelism **References**: [102, 103] For extremely long sequences (speech models processing 30+ second audio, document-level language models), even 3D parallelism isn't enough. 4D parallelism adds sequence parallelism as the fourth dimension. **The 4D configuration**: Data × Tensor × Pipeline × Sequence **Example** (100B speech model, 1024 GPUs, 30-second audio = 3000 frames): - **Data Parallelism**: $D = 4$ (4 replicas of full model) - **Tensor Parallelism**: $T = 8$ (split model dimension within node) - **Pipeline Parallelism**: $P = 8$ (split layers across nodes) - **Sequence Parallelism**: $S = 4$ (split sequence dimension) Total: $D \times T \times P \times S = 4 \times 8 \times 8 \times 4 = 1024$ GPUs **Memory savings**: Each GPU processes: - $\frac{1}{D}$ of the batch - $\frac{1}{T}$ of the model dimension - $\frac{1}{P}$ of the layers - $\frac{1}{S}$ of the sequence length Activation memory: $\frac{B \times N \times D}{D \times S \times T} = \frac{32 \times 3000 \times 512}{4 \times 4 \times 8} = 3$ MB per GPU (vs. 3 GB without parallelism) **The KV-Cache challenge in sequence parallelism**: In autoregressive models (RNN-T decoder, Transformer decoder), KV-cache grows with sequence length. When the sequence is split across GPUs, managing the cache becomes complex. **Problem**: At decoding step $t$, the model needs $K_{1:t}$ and $V_{1:t}$ (all previous keys and values). But with sequence parallelism: - GPU 0 has $K_{1:n/S}$, $V_{1:n/S}$ - GPU 1 has $K_{n/S+1:2n/S}$, $V_{n/S+1:2n/S}$ - ... Each GPU only has a shard of the cache! **Solution 1: All-gather KV-cache before attention** ```python # Each GPU has local KV shard K_local = cache_K[start_idx:end_idx] # (Batch, n/S, D) V_local = cache_V[start_idx:end_idx] # (Batch, n/S, D) # All-gather to get full KV on each GPU K_full = all_gather(K_local, dim=1) # (Batch, n, D) V_full = all_gather(V_local, dim=1) # (Batch, n, D) # Compute attention with full KV output = attention(Q, K_full, V_full) # Reduce-scatter output back to shards output_local = reduce_scatter(output, dim=1) # (Batch, n/S, D) ``` **Communication cost**: All-gather of KV-cache at every decoding step. For cache size $C = 2 \times L \times n \times d$ (keys + values, $L$ layers, $n$ tokens, $d$ dimension): $$ \text{Communication per step} = \frac{C \times (S-1)}{S} $$ For $S=4$, this is $0.75C$ per step—significant overhead. **Solution 2: Lookahead with local attention** Instead of all-gathering the full cache, use a sliding window approach: 1. Each GPU maintains its local cache shard 2. Attention is computed only over a local window (e.g., last 512 tokens) 3. All-gather only the window, not the full cache ```python # Each GPU has local KV shard K_local = cache_K[start_idx:end_idx] # (Batch, n/S, D) # Extract recent window (last W tokens from local shard) window_size = 512 K_window = K_local[:, -window_size:, :] # (Batch, W, D) # All-gather only the window K_window_full = all_gather(K_window, dim=1) # (Batch, W*S, D) # Attention over window only output = attention(Q, K_window_full, V_window_full) ``` **Trade-off**: Reduces communication by $\frac{n}{W}$ but limits attention to recent context. For speech, this is acceptable—phoneme context is local (50-100 frames). For language models with long-range dependencies, this hurts quality. **Solution 3: Hierarchical KV-cache (production approach)** Combine local and global cache: 1. **Local cache**: Recent $W$ tokens, all-gathered every step (low latency) 2. **Global cache**: Full history, all-gathered every $K$ steps (amortized cost) ```python # Local cache: recent tokens, updated every step K_local_recent = cache_K[:, -window_size:, :] K_recent_full = all_gather(K_local_recent, dim=1) # Global cache: full history, updated every K steps if step % K == 0: K_global_full = all_gather(cache_K, dim=1) # Attention over both output = attention(Q, [K_recent_full, K_global_full], [V_recent_full, V_global_full]) ``` **KV-Cache Fragmentation Challenge** When sequences are split across GPUs via sequence parallelism, the KV-cache becomes fragmented—each GPU holds non-contiguous chunks. This creates two problems: 1. **Memory inefficiency**: Traditional memory allocation assumes contiguous blocks. With fragmentation, you can't efficiently pack variable-length sequences into GPU memory. 2. **Attention computation**: Standard attention kernels expect contiguous KV tensors. Fragmented cache requires gathering before attention, adding overhead. **Paged Attention: The Solution** Paged Attention (vLLM) treats KV-cache like virtual memory in operating systems: 1. **Block-based storage**: Divide KV-cache into fixed-size blocks (e.g., 16 tokens per block) 2. **Non-contiguous allocation**: Blocks can be stored anywhere in GPU memory, tracked by a page table 3. **On-demand gathering**: Attention kernel reads blocks via indirection, no explicit gather needed **Memory savings example**: Traditional approach (contiguous): - Batch of 32 sequences, max length 2048, actual lengths vary (500-2000) - Must allocate $32 \times 2048 \times d$ for all sequences - Wasted memory: ~40% (padding for short sequences) Paged Attention: - Allocate only $\sum_i \lceil \frac{n_i}{16} \rceil \times 16 \times d$ (actual length rounded to block size) - Wasted memory: ~3% (only block-level padding) - 13× better memory utilization → 13× higher batch size **Implementation sketch**: ```python # Page table: maps logical position to physical block # page_table[seq_id][block_id] = physical_block_id def paged_attention(Q, page_table, block_storage): """ Attention with paged KV-cache. Args: Q: Query (Batch, Seq_Q, D) page_table: Mapping from logical to physical blocks block_storage: Physical KV blocks (Num_Blocks, Block_Size, D) Returns: Output: (Batch, Seq_Q, D) """ batch_size, seq_q, d = Q.shape output = [] for b in range(batch_size): # Gather K, V blocks for this sequence via page table seq_blocks = [block_storage[page_table[b][i]] for i in range(len(page_table[b]))] K_seq = concat(seq_blocks, dim=0) # (Seq_K, D) V_seq = concat(seq_blocks, dim=0) # (Seq_K, D) # Standard attention output_seq = attention(Q[b], K_seq, V_seq) output.append(output_seq) return stack(output, dim=0) ``` **Production reality**: Paged Attention is now standard in production LLM serving (vLLM, TensorRT-LLM, Text Generation Inference). It enables: - 2-3× higher throughput (more sequences per GPU) - Support for extremely long contexts (100k+ tokens) - Efficient multi-turn conversations (share KV-cache across turns) **Connection to sequence parallelism**: With Paged Attention, each GPU in sequence parallelism maintains its own page table for its shard. Cross-GPU attention requires: 1. Exchange page table metadata (lightweight) 2. Remote block reads via RDMA or all-gather (only needed blocks) 3. Compute attention with mixed local/remote blocks This is far more efficient than gathering the entire KV-cache. **Production reality**: 4D parallelism is only necessary for extreme-scale models with long sequences. The communication overhead is substantial—only worth it when memory is the absolute bottleneck. For most production systems, 3D parallelism (Data + Tensor + Pipeline) is sufficient. **My work on Nova Sonic**: We use 3D parallelism for training (Data × Tensor × Pipeline) and sequence parallelism only for inference on extremely long audio (>60 seconds). The KV-cache management uses hierarchical caching with a 512-frame local window and full cache refresh every 50 steps. **DeepSpeed vs. Megatron-LM**: - **DeepSpeed**: ZeRO, offloading, gradient checkpointing, mixed precision—strong memory optimization - **Megatron-LM**: Tensor Parallelism and Pipeline Parallelism—strong for model parallelism at scale - In practice, used together (Megatron-DeepSpeed) My resume mentions both—I use them for training large-scale speech models. **Gradient checkpointing**: Trade compute for memory—during the forward pass, don't store all intermediate activations. Recompute them during the backward pass when needed. Reduces activation memory from $O(\text{layers} \times \text{batch} \times \text{seq})$ to $O(\sqrt{\text{layers}})$. **Mixed Precision Training**: Train in float16 (or bfloat16) for speed and memory, maintain a float32 master copy for numerical stability. bf16 avoids the underflow problem (larger exponent range) at the cost of precision. Standard in all production training pipelines. ### Quantization: Compressing Models for Production Deployment **References**: [104, 105, 106] A 7B parameter model in float16 requires 14GB of memory. On mobile devices or edge hardware, this is impossible. Quantization reduces precision to make models deployable—the essential technique for production inference. **The core idea**: Represent weights and activations with fewer bits. Float32 (32 bits) → Float16 (16 bits) → Int8 (8 bits) → Int4 (4 bits). Each reduction halves memory and doubles throughput (on hardware with appropriate support). **Quantization formula**: Map floating-point values to integers: $ x_{\text{quant}} = \text{round}\left(\frac{x - z}{s}\right) $ where: - $s$ is the **scale factor** (controls the range) - $z$ is the **zero point** (handles asymmetry) **Dequantization** (for computation): $ x_{\text{float}} = s \cdot x_{\text{quant}} + z $ **Symmetric vs. Asymmetric Quantization**: **Symmetric** ($z = 0$): Range is $[-\alpha, \alpha]$ mapped to $[-127, 127]$ for int8. Scale: $s = \frac{\alpha}{127}$. **Asymmetric** ($z \neq 0$): Range is $[\alpha_{\min}, \alpha_{\max}]$ mapped to $[0, 255]$ for uint8. Scale: $s = \frac{\alpha_{\max} - \alpha_{\min}}{255}$, zero point: $z = -\text{round}(\frac{\alpha_{\min}}{s})$. **Why asymmetric is better for activations**: ReLU activations are non-negative—symmetric quantization wastes half the range on negative values that never occur. Asymmetric quantization uses the full range. **Post-Training Quantization (PTQ)**: Quantize a trained model without retraining. 1. **Weight quantization**: Quantize weights using their min/max values. No data needed. 2. **Activation quantization**: Requires calibration data to estimate activation ranges. Run the model on a small calibration set (100-1000 examples), record min/max activations per layer, compute scale factors. 3. **Quantize and deploy**: Convert model to int8, deploy with int8 inference kernels. **Quantization-Aware Training (QAT)**: Simulate quantization during training so the model learns to be robust to quantization noise. **Fake quantization**: During forward pass, quantize and immediately dequantize: $ x_{\text{fake\_quant}} = s \cdot \text{round}\left(\frac{x}{s}\right) $ This introduces quantization noise during training. Gradients flow through the round operation using the **straight-through estimator** (STE): treat round as identity for backprop. **Why QAT works better than PTQ**: The model learns to use the limited precision effectively. Weights and activations adapt to minimize quantization error. Typical accuracy: PTQ loses 1-3%, QAT loses <0.5%. **Per-Channel vs. Per-Tensor Quantization**: **Per-tensor**: One scale factor for the entire tensor. Simple but coarse—if one outlier has large magnitude, the scale factor is large, wasting precision on other values. **Per-channel**: One scale factor per output channel (for weights) or per activation channel. More fine-grained, better accuracy, minimal overhead. For a weight matrix $W \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}}}$, per-channel quantization uses $C_{\text{out}}$ scale factors—one per output channel. **Mixed Precision Quantization**: Not all layers are equally sensitive to quantization. Quantize most layers to int8, keep sensitive layers (first/last layer, attention) in float16. **Sensitivity analysis**: For each layer, quantize it to int8 and measure accuracy drop. Layers with large drops stay in float16. This is how production models achieve <1% accuracy loss with 4× memory reduction. **Quantization for Speech Models**: Speech models have unique quantization challenges: 1. **Mel filterbank inputs**: Values are log-scale, range is $[-80, 0]$ dB. Asymmetric quantization is essential. 2. **Attention weights**: Softmax outputs are in $(0, 1)$, but attention scores before softmax can have large range. Quantize after softmax for stability. 3. **RNN-T joiner**: The joiner combines encoder and prediction network outputs—quantization error compounds. Keep joiner in float16 for production. **Production quantization pipeline**: 1. **Train in float32/bf16**: Full precision training 2. **PTQ to int8**: Quick baseline, 2-3% WER degradation 3. **QAT to int8**: Recover most accuracy, <1% WER degradation 4. **Mixed precision**: Keep attention in float16, rest in int8 5. **Deploy**: int8 inference on CPU/mobile, 4× faster than float16 **Hardware support**: Modern CPUs (AVX-512 VNNI), GPUs (Tensor Cores), and mobile processors (ARM NEON) have int8 GEMM instructions that are 4-8× faster than float16. Quantization isn't just about memory—it's about throughput. **My work on Nova Sonic**: We use per-channel int8 quantization for the Conformer encoder (80% of compute) and keep the RNN-T joiner in float16 (20% of compute, sensitive to quantization). This achieves 3.5× speedup with <0.5% WER degradation. On-device deployment uses int8 throughout with QAT to maintain quality. ## Production ML System Design Research focuses on accuracy. Production focuses on latency, throughput, memory efficiency, and robustness. This section covers the engineering decisions that separate a paper from a product. ### KV-Caching: Making Autoregressive Models Fast **References**: [92, 93] Autoregressive models (GPT, LLaMA, Nova Sonic decoder) generate one token at a time, conditioning on all previous tokens. Naive implementation recomputes attention over the entire history at every step—quadratic in sequence length. **The recomputation problem**: At step $t$, computing attention requires: $$ \text{Attention}(Q_t, K_{1:t}, V_{1:t}) = \text{softmax}\left(\frac{Q_t K_{1:t}^T}{\sqrt{d_k}}\right) V_{1:t} $$ where $K_{1:t}$ and $V_{1:t}$ are keys and values for all previous tokens. Computing $K_{1:t}$ and $V_{1:t}$ requires running the encoder on all previous tokens—wasteful since they don't change. **KV-Cache solution**: Cache the computed keys and values from previous steps: ``` Step 1: Compute K_1, V_1 → Cache them Step 2: Compute K_2, V_2 → Append to cache → Attention(Q_2, [K_1, K_2], [V_1, V_2]) Step 3: Compute K_3, V_3 → Append to cache → Attention(Q_3, [K_1, K_2, K_3], [V_1, V_2, V_3]) ... ``` **Complexity reduction**: - **Without KV-cache**: $O(t^2 d)$ per step → $O(T^3 d)$ total for sequence length $T$ - **With KV-cache**: $O(t d)$ per step → $O(T^2 d)$ total For $T=2048$, this is a 2048× speedup per step! **Memory cost**: Store $K$ and $V$ for all previous tokens. For a model with $L$ layers, $h$ heads, and head dimension $d_h$: $$ \text{Memory} = 2 \times L \times T \times h \times d_h \times \text{sizeof(dtype)} $$ **Example** (LLaMA-7B, $L=32$, $h=32$, $d_h=128$, fp16): - Per token: $2 \times 32 \times 32 \times 128 \times 2 = 524$ KB - For $T=2048$: $524 \times 2048 = 1$ GB per sequence **Production implications**: 1. **Batch size limited by KV-cache memory**: With 40 GB GPU memory, 7B model weights take ~14 GB, leaving ~26 GB for KV-cache and activations. At 1 GB per sequence, batch size is limited to ~20-25. 2. **Variable-length sequences**: Different sequences in a batch have different lengths. Naive implementation pads all to max length—wasteful. Solution: **Paged Attention** (vLLM) stores KV-cache in non-contiguous memory blocks, eliminating padding waste. 3. **Multi-turn conversations**: In chatbots, the KV-cache grows with conversation length. After 10 turns (avg 100 tokens/turn), cache is 1000 tokens. Solution: **Sliding window attention** or **cache eviction policies** (evict least-recently-used tokens). **Production reality**: KV-caching is mandatory for production autoregressive models. Without it, latency is 100-1000× higher. But KV-cache memory is the primary bottleneck for serving throughput—more memory spent on cache = fewer concurrent requests. ### Telemetry and Drift Detection **References**: [94, 95] Production models degrade over time. User behavior changes, new slang emerges, device hardware evolves. Without monitoring, you won't know your model is failing until users complain. **Types of drift**: 1. **Covariate shift**: $P(X)$ changes but $P(Y|X)$ stays the same. Example: Users start saying "Hey Alexa" instead of "Alexa" (input distribution changes, but the task is the same). 2. **Concept drift**: $P(Y|X)$ changes. Example: "Call me an Uber" used to mean "insult me" but now means "request a ride" (same input, different meaning). 3. **Label drift**: $P(Y)$ changes. Example: More users ask about COVID-19 in 2020 than 2019 (output distribution changes). **Detecting covariate shift: KL divergence** Compare the distribution of production inputs to training inputs using KL divergence: $$ D_{\text{KL}}(P_{\text{prod}} \| P_{\text{train}}) = \sum_x P_{\text{prod}}(x) \log \frac{P_{\text{prod}}(x)}{P_{\text{train}}(x)} $$ For continuous features (e.g., mel spectrograms), discretize into bins or use kernel density estimation. **Practical implementation**: 1. **Feature-level KL**: Compute KL divergence for each feature dimension. High KL on specific features indicates drift in those aspects. 2. **Embedding-level KL**: Pass inputs through the encoder, compute KL on the embedding distribution. This captures semantic drift. 3. **Threshold-based alerting**: If $D_{\text{KL}} > \tau$, trigger an alert. Tune $\tau$ on historical data to balance false positives vs. detection latency. **Detecting concept drift: Label distribution shifts** Monitor the distribution of predicted labels over time: $$ P_{\text{week}_t}(y) \text{ vs. } P_{\text{week}_{t-1}}(y) $$ Sudden changes indicate concept drift. Example: If "play music" queries drop from 30% to 10% in one week, something changed (new feature, UI change, or model bug). **Detecting performance degradation: Online metrics** For ASR, track Word Error Rate (WER) on a held-out test set evaluated weekly. For classification, track accuracy, precision, recall. **Challenge**: Ground truth labels are expensive. For ASR, you need human transcriptions. Solution: - **Pseudo-labeling**: Use a high-confidence threshold (e.g., model confidence > 0.95) to automatically label production data. Evaluate on these pseudo-labels. - **Human-in-the-loop sampling**: Sample 1% of production data for human labeling. Extrapolate metrics to full production traffic. **Production reality**: Drift detection is reactive—you detect drift after it happens. Proactive approaches: - **Continual learning**: Retrain weekly on recent production data (with replay to prevent forgetting) - **A/B testing**: Deploy new models to 5% of traffic, monitor metrics, gradually ramp to 100% - **Canary deployments**: Deploy to one datacenter first, monitor for 24 hours, then roll out globally **My production experience**: For Nova Sonic, we monitor: 1. **WER by domain**: Smart home, music, shopping, general queries 2. **WER by accent**: US English, UK English, Indian English, etc. 3. **Latency p50, p90, p99**: Ensure no regression in user experience 4. **Confidence distribution**: Sudden drops in confidence indicate model uncertainty ### Latency-Accuracy Trade-off: Decision Matrix **References**: [96, 97] Production ASR systems face a fundamental trade-off: accuracy vs. latency. Users want perfect transcriptions instantly. Physics says you can't have both. **Streaming vs. Offline models**: | Model Type | Latency | Accuracy | Use Case | |------------|---------|----------|----------| | Streaming RNN-T | 50-200 ms | Good | Voice assistants, live captions | | Offline Conformer | 500-2000 ms | Excellent | Transcription services, post-call analytics | | Hybrid (streaming + rescore) | 200-500 ms | Very Good | Production voice assistants | **Why offline models are more accurate**: 1. **Bidirectional context**: Offline models see the entire utterance before predicting. Streaming models only see past and limited future context (lookahead). 2. **Larger models**: Offline models can be 2-5× larger (more parameters, more compute per frame) because latency isn't critical. 3. **Better language models**: Offline models can use large external LMs for rescoring. Streaming models must use lightweight on-device LMs. **Streaming RNN-T architecture constraints**: - **Encoder**: Causal or limited lookahead (e.g., 30 frames = 300 ms future context) - **Prediction network**: Autoregressive, must be fast (typically 1-2 layer LSTM) - **Joiner**: Lightweight (1-2 layer FFN) **Offline Conformer architecture freedom**: - **Encoder**: Bidirectional self-attention over full utterance - **Decoder**: Large Transformer decoder with external LM fusion - **Rescoring**: N-best hypotheses rescored with BERT-based LM **The decision matrix**: When to use which model? **Use Streaming RNN-T when**: - **Real-time interaction required**: Voice assistants, live captions, dictation - **Latency budget < 200 ms**: User experience degrades beyond this threshold - **On-device deployment**: Mobile devices, edge devices with limited compute - **Accuracy requirement: WER < 10%**: Good enough for most voice commands **Use Offline Conformer when**: - **Batch processing**: Transcribing recorded calls, meetings, podcasts - **Latency budget > 1 second**: Users don't expect instant results - **Cloud deployment**: Unlimited compute, can use large models - **Accuracy requirement: WER < 5%**: Critical applications (medical, legal transcription) **Use Hybrid (streaming + offline rescore) when**: - **Latency budget: 200-500 ms**: Show streaming results immediately, update with offline results - **Accuracy requirement: WER < 7%**: Better than streaming alone, faster than offline alone - **User experience**: Progressive enhancement—users see fast results that improve over time **Production example (Alexa)**: 1. **Streaming RNN-T**: Runs on-device or in the cloud, emits tokens as user speaks. Latency: 50-100 ms per token. 2. **Offline rescore**: After user stops speaking (detected by end-of-speech), rescore the streaming hypothesis with a large Conformer + BERT LM. Latency: +200-300 ms. 3. **User experience**: User sees streaming results in real-time (for feedback), but the final action is based on the rescored result (for accuracy). **The latency-accuracy Pareto frontier**: ``` Accuracy (WER) ^ | Offline Conformer (5%) | / | / Hybrid (7%) | / | / | / Streaming RNN-T (9%) | / | / +------------------------> Latency (ms) 0 100 200 500 1000 ``` **Production reality**: Most production systems use hybrid approaches. Pure streaming is too inaccurate for critical tasks. Pure offline is too slow for interactive tasks. The sweet spot is streaming for responsiveness + offline rescore for accuracy. **My work on Nova Sonic**: Streaming Conformer-RNN-T for real-time transcription, with optional offline rescoring for high-accuracy use cases. The model architecture is designed to be configurable—same weights, different inference modes depending on latency budget. ## Spectral Graph Theory: Connecting Signal Processing to Modern GNNs **References**: [98, 99] My B.Tech thesis at NIT Calicut focused on 2-channel perfect reconstruction wavelet filter banks on bipartite graphs—a bridge between classical signal processing and graph-structured data. This work connects directly to modern Graph Neural Networks (GNNs), showing a 10-year thread of mathematical consistency. **From Euclidean to Non-Euclidean Signal Processing** Classical signal processing assumes data lives on a regular grid (time series, images). But many real-world structures are irregular: social networks, molecular structures, sensor networks, knowledge graphs. How do we extend Fourier analysis and filtering to graphs? **The Graph Laplacian Matrix** For a graph $G = (V, E)$ with adjacency matrix $A$ and degree matrix $D$ (where $D_{ii} = \sum_j A_{ij}$): **Unnormalized Laplacian**: $$ L = D - A $$ **Normalized Laplacian** (the key to spectral methods): $$ L_{\text{norm}} = I - D^{-1/2} A D^{-1/2} $$ **Why the normalized Laplacian matters**: 1. **Eigenvalues in [0, 2]**: The normalized form has bounded eigenvalues, making it numerically stable and interpretable. 2. **Graph Fourier Transform**: The eigenvectors of $L_{\text{norm}}$ form an orthonormal basis for signals on the graph. Just as sine/cosine are eigenfunctions of the Laplacian operator in Euclidean space, these eigenvectors are the "frequencies" on the graph. 3. **Spectral filtering**: A filter $g(\lambda)$ applied to a graph signal $x$ becomes: $$ y = g(L_{\text{norm}}) x = U g(\Lambda) U^T x $$ where $L_{\text{norm}} = U \Lambda U^T$ is the eigendecomposition. **Connection to Graph Neural Networks** Modern GNNs are spectral filters in disguise: **Spectral Graph Convolution** (Bruna et al., 2013): $$ y = \sigma\left(\sum_{k=0}^K \theta_k L_{\text{norm}}^k x\right) $$ This is a polynomial filter on the graph—each power of $L_{\text{norm}}$ aggregates information from $k$-hop neighbors. **ChebNet** (Defferrard et al., 2016): Uses Chebyshev polynomials for efficient approximation: $$ g_\theta(\Lambda) = \sum_{k=0}^K \theta_k T_k(\tilde{\Lambda}) $$ where $T_k$ are Chebyshev polynomials and $\tilde{\Lambda} = \frac{2\Lambda}{\lambda_{\max}} - I$ normalizes eigenvalues to $[-1, 1]$. **Graph Convolutional Networks (GCN)** (Kipf & Welling, 2017): First-order approximation ($K=1$): $$ H^{(l+1)} = \sigma\left(\tilde{D}^{-1/2} \tilde{A} \tilde{D}^{-1/2} H^{(l)} W^{(l)}\right) $$ where $\tilde{A} = A + I$ (add self-loops) and $\tilde{D}$ is the degree matrix of $\tilde{A}$. This is exactly the normalized Laplacian applied to node features! **My B.Tech Thesis: Bipartite Graphs and Perfect Reconstruction** A **bipartite graph** has vertices partitioned into two sets $V_1$ and $V_2$ where all edges connect $V_1$ to $V_2$. Examples: sensor networks (sensors ↔ base stations), recommendation systems (users ↔ items), molecular graphs (atoms ↔ bonds). **Perfect reconstruction filter banks**: Design analysis and synthesis filters such that: $$ x = \text{Synthesis}(\text{Analysis}(x)) $$ No information is lost during decomposition and reconstruction. This is critical for: - **Compression**: Discard high-frequency components without losing essential structure - **Denoising**: Filter in spectral domain, reconstruct clean signal - **Multi-resolution analysis**: Analyze graph structure at multiple scales **The 10-year thread**: From wavelets on bipartite graphs (2014) to modern GNNs (2024), the mathematical foundation is the same—spectral analysis via the normalized Laplacian. My thesis work on perfect reconstruction ensures that graph-based representations preserve information, a principle that underlies modern graph representation learning. **Production relevance**: GNNs are now used in: - **Recommendation systems**: User-item bipartite graphs (Amazon, Netflix) - **Drug discovery**: Molecular graphs for property prediction - **Knowledge graphs**: Entity-relation graphs for reasoning (Alexa knowledge base) - **Traffic prediction**: Road networks as graphs The spectral perspective provides theoretical guarantees (e.g., why GCNs work, what information is preserved) that purely empirical approaches lack. ## Production Serving Infrastructure: Kubernetes and Model Deployment **References**: [107, 108, 109] Training a model is only half the battle. Deploying it to serve billions of requests with low latency, high availability, and efficient resource utilization is where production ML engineering happens. ### Kubernetes Core Concepts **Kubernetes** is the de facto standard for container orchestration in production ML systems. Key components: 1. **Pods**: Smallest deployable unit—contains one or more containers (model server, sidecar for logging/metrics) 2. **Deployments**: Manages pod replicas, handles rolling updates 3. **Services**: Stable endpoint for accessing pods (load balancing) 4. **Horizontal Pod Autoscaler (HPA)**: Automatically scales pod replicas based on CPU, memory, or custom metrics (GPU utilization, requests per second, inference latency) **Why Kubernetes**: Automatic scaling, self-healing (restart failed containers), load balancing, rolling updates with zero downtime, and resource management (CPU/memory/GPU allocation per pod). ### Model Deployment Strategies **Canary Deployment**: - Deploy new version to small percentage of traffic (5%) - Monitor metrics (latency, error rate, WER) - Gradually increase traffic (5% → 25% → 100%) - Rollback if metrics degrade **Blue-Green Deployment**: - Deploy new version (green) alongside old version (blue) - Switch 100% traffic to green after validation - Keep blue for quick rollback **A/B Testing**: - Route traffic based on user ID - Compare metrics between model versions - Winner becomes the new production model ### My Production Experience **Nova Sonic deployment**: - Kubernetes cluster with 200+ GPU nodes (A100, T4) - HPA based on GPU utilization (80%) and RPS (50 per pod) - Canary deployment with 5% → 25% → 100% rollout over 24 hours - Monitoring: Prometheus + Grafana for metrics, distributed tracing for latency debugging - Cost optimization: int8 quantization (3.5× cheaper), batch inference (2× throughput) ## Key Takeaways This post covered the complete technical stack behind my resume—from probability theory to production-scale distributed training. Here are the key insights: **Foundations matter**: Every modern technique rests on probabilistic foundations. Bayes' theorem, MLE/MAP, the bias-variance trade-off—these aren't just theory, they're the lens through which we understand why techniques work. **Speech is special**: Raw waveforms don't work for ML. We need frequency analysis (STFT), perceptual scaling (Mel filterbanks), and log compression. The log-mel spectrogram is the standard input representation for modern speech models. **End-to-end wins**: GMM-HMMs dominated for decades, but end-to-end neural approaches (CTC, RNN-T) are now standard. They eliminate the need for explicit alignment labels and learn better representations. **Attention is fundamental**: From Bahdanau attention to Transformers to Conformers—attention mechanisms enable selective focus on relevant information. Multi-head attention provides multiple representational subspaces. **Architecture evolution**: RNNs → LSTMs → Transformers → Conformers. Each innovation solved a specific limitation. Conformers combine convolution (local patterns) with attention (global context)—optimal for speech. **Embeddings for identity**: Speaker and language ID require extracting identity-specific information while being invariant to content. x-vectors with statistics pooling are the standard. Attention-based pooling (my thesis work) improves performance by focusing on discriminative frames. **PEFT is essential**: Full fine-tuning is expensive and causes catastrophic forgetting. LoRA achieves near-full-fine-tuning performance with <1% of parameters updated. Zero inference overhead makes it production-ready. **Alignment is critical**: Pre-training gives you a knowledgeable model, not a helpful one. RLHF and DPO align models with human preferences. For speech, this means hot-fixing transcription errors while preserving general capabilities. **Continual learning prevents regression**: Production models must be updated without breaking existing capabilities. EWC, replay methods, and golden set validation (my patent) enable safe updates. **Federated learning enables privacy**: Training on sensitive data without centralization. FedAvg is the foundation. Cross-silo FL with gRPC (my ICASSP 2021 work) enables enterprise-scale federated training. **Distributed training enables scale**: Data parallelism, ZeRO, tensor parallelism, pipeline parallelism—each technique addresses a different bottleneck. 3D parallelism combines them all for training 100B+ parameter models. **Production is different from research**: Research focuses on accuracy. Production focuses on latency, throughput, memory efficiency, and robustness. Techniques like gradient checkpointing, mixed precision, and careful hyperparameter tuning make the difference between a paper and a product. ## References [1] MIT 18.650 Statistics for Applications. [Course materials](https://ocw.mit.edu/courses/mathematics/18-650-statistics-for-applications-fall-2016/). [2] 3Blue1Brown. [Bayes Theorem](https://www.youtube.com/watch?v=HZGCoVF3YvM). [3] Ng, A. & Jordan, M. (2002). [On Discriminative vs Generative Classifiers](https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf). NIPS 2002. [4] Bishop, C. (2006). Pattern Recognition and Machine Learning. Springer. [5] CS229 Lecture Notes. [Linear Regression](http://cs229.stanford.edu/notes2020spring/cs229-notes1.pdf). [5a] Kingma, D. & Ba, J. (2015). [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980). ICLR 2015. [5b] Ruder, S. (2016). [An overview of gradient descent optimization algorithms](https://arxiv.org/abs/1609.04747). arXiv preprint. [6] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning. Springer. [7] StatQuest. [Logistic Regression](https://www.youtube.com/watch?v=yIYKR4sgzI8). [8] CS229 Lecture Notes. [Logistic Regression](http://cs229.stanford.edu/notes2020spring/cs229-notes1.pdf). [9] Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review. [10] Minsky, M. & Papert, S. (1969). Perceptrons. MIT Press. [11] Nielsen, M. (2015). [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/). [12] Karpathy, A. [Micrograd](https://github.com/karpathy/micrograd). [13] CS231n. [Backpropagation Notes](https://cs231n.github.io/optimization-2/). [14] Fayek, H. [Speech Processing for Machine Learning](https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html). [15] Smith, S. (1997). [The Scientist and Engineer's Guide to Digital Signal Processing](http://www.dspguide.com/). [16] 3Blue1Brown. [But what is the Fourier Transform?](https://www.youtube.com/watch?v=spUNpyF58BY) [17] Stanford EE261. [The Fourier Transform and its Applications](https://see.stanford.edu/Course/EE261). [18] Smith, J. O. [Mathematics of the DFT](https://ccrma.stanford.edu/~jos/mdft/). [19] Fayek, H. [Mel Filterbanks](https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html). [20] Huang, X., Acero, A., & Hon, H. (2001). Spoken Language Processing. Prentice Hall. [21] Benesty, J., et al. (2008). [Acoustic Echo Cancellation](https://link.springer.com/chapter/10.1007/978-3-540-78612-2_1). Springer Handbook of Speech Processing. [21a] Vary, P. & Martin, R. (2006). [Digital Speech Transmission: Enhancement, Coding and Error Concealment](https://www.wiley.com/en-us/Digital+Speech+Transmission%3A+Enhancement%2C+Coding+and+Error+Concealment-p-9780470870143). Wiley. [21b] Benesty, J., et al. (2008). [Microphone Array Signal Processing](https://link.springer.com/book/10.1007/978-3-540-78612-2). Springer. [21c] Sohn, J., et al. (1999). [A Statistical Model-Based Voice Activity Detection](https://ieeexplore.ieee.org/document/736356). IEEE Signal Processing Letters. [21d] WebRTC VAD. [Voice Activity Detection](https://webrtc.googlesource.com/src/+/refs/heads/main/common_audio/vad/). Open-source implementation. [22] Rabiner, L. (1989). [A Tutorial on Hidden Markov Models](https://www.cs.ubc.ca/~murphyk/Bayes/rabiner.pdf). Proceedings of the IEEE. [22] Jurafsky, D. & Martin, J. H. [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/). [23] Huang, X., Acero, A., & Hon, H. (2001). Spoken Language Processing. Prentice Hall. [23a] NIST. [Speech Recognition Scoring Toolkit](https://github.com/usnistgov/SCTK). SCTK documentation. [24] Graves, A., et al. (2006). [Connectionist Temporal Classification](https://www.cs.toronto.edu/~graves/icml_2006.pdf). ICML 2006. [25] Hannun, A. [Sequence Modeling with CTC](https://distill.pub/2017/ctc/). [26] Graves, A. (2012). [Sequence Transduction with Recurrent Neural Networks](https://arxiv.org/abs/1211.3711). arXiv preprint. [27] He, Y., et al. (2019). [Streaming End-to-End Speech Recognition For Mobile Devices](https://arxiv.org/abs/1811.06621). ICASSP 2019. [28] Hu, J., et al. (2020). [Exploring the Limits of Efficient Transformers for RNN-T](https://arxiv.org/abs/2109.07513). arXiv preprint. [29] Radford, A., et al. (2022). [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356). arXiv preprint. [30] Pratap, V., et al. (2023). [Scaling Speech Technology to 1000+ Languages](https://arxiv.org/abs/2305.13516). arXiv preprint. [31] Kannan, A., et al. (2019). [Large-Scale Multilingual Speech Recognition](https://arxiv.org/abs/1907.05446). Interspeech 2019. [32] Hochreiter, S. & Schmidhuber, J. (1997). [Long Short-Term Memory](https://www.bioinf.jku.at/publications/older/2604.pdf). Neural Computation. [33] Colah. [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/). [34] Karpathy, A. [The Unreasonable Effectiveness of RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). [35] Bahdanau, D., Cho, K., & Bengio, Y. (2015). [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473). ICLR 2015. [36] Vaswani, A., et al. (2017). [Attention Is All You Need](https://arxiv.org/abs/1706.03762). NeurIPS 2017. [37] Weng, L. [Attention Survey](https://lilianweng.github.io/posts/2018-06-24-attention/). [38] Vaswani, A., et al. (2017). [Attention Is All You Need](https://arxiv.org/abs/1706.03762). NeurIPS 2017. [39] Alammar, J. [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/). [40] Rush, A. [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html). [40a] Su, J., et al. (2021). [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864). arXiv preprint. [40b] Press, O., et al. (2021). [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409). ICLR 2022. [41] Gulati, A., et al. (2020). [Conformer: Convolution-augmented Transformer for Speech Recognition](https://arxiv.org/abs/2005.08100). Interspeech 2020. [42] NVIDIA NeMo. [Conformer Documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/models.html). [43] NIST. [Speaker Recognition Evaluation](https://www.nist.gov/itl/iad/mig/speaker-recognition). [44] Reynolds, D. (2002). [An Overview of Automatic Speaker Recognition](https://www.cse.iitk.ac.in/users/sigml/lec/Reynolds_Automatic_Speaker_Recognition.pdf). ICASSP 2002. [45] Dehak, N., et al. (2011). [Front-End Factor Analysis for Speaker Verification](https://ieeexplore.ieee.org/document/5545402). IEEE TASLP. [46] Prince, S. & Elder, J. (2007). [Probabilistic Linear Discriminant Analysis](https://www.robots.ox.ac.uk/~az/lectures/ml/prince_plda07.pdf). ICCV 2007. [47] Snyder, D., et al. (2018). [X-Vectors: Robust DNN Embeddings for Speaker Recognition](https://www.danielpovey.com/files/2018_icassp_xvectors.pdf). ICASSP 2018. [48] Desplanques, B., et al. (2020). [ECAPA-TDNN](https://arxiv.org/abs/2005.07143). Interspeech 2020. [49] Mohan, A. (2019). Attention-Based Speaker and Language Recognition. PhD Thesis. [50] NIST LRE. [Language Recognition Evaluation](https://www.nist.gov/itl/iad/mig/language-recognition). [51] Chen, S. & Goodman, J. (1999). [An Empirical Study of Smoothing Techniques for Language Modeling](https://dash.harvard.edu/bitstream/handle/1/25104739/tr-10-98.pdf). Computer Speech & Language. [52] Jurafsky, D. & Martin, J. H. [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/). [52a] Sennrich, R., et al. (2016). [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909). ACL 2016. [52b] Kudo, T. & Richardson, J. (2018). [SentencePiece: A simple and language independent approach to subword tokenization](https://arxiv.org/abs/1808.06226). EMNLP 2018. [52c] Hugging Face. [Tokenizers Documentation](https://huggingface.co/docs/tokenizers/index). Comprehensive tokenization guide. [53] Mikolov, T., et al. (2010). [Recurrent Neural Network Based Language Model](https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf). Interspeech 2010. [54] Sundermeyer, M., et al. (2012). [LSTM Neural Networks for Language Modeling](https://www.isca-speech.org/archive/interspeech_2012/i12_0194.html). Interspeech 2012. [55] Devlin, J., et al. (2019). [BERT: Pre-training of Deep Bidirectional Transformers](https://arxiv.org/abs/1810.04805). NAACL 2019. [56] Alammar, J. [The Illustrated BERT](https://jalammar.github.io/illustrated-bert/). [56a] Radford, A., et al. (2018). [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf). OpenAI. [56b] Radford, A., et al. (2019). [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). OpenAI. [56c] Brown, T., et al. (2020). [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165). NeurIPS 2020. [57] Hu, E., et al. (2021). [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685). ICLR 2022. [58] Ding, N., et al. (2022). [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods](https://arxiv.org/abs/2203.06904). arXiv preprint. [59] Hu, E., et al. (2021). [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685). ICLR 2022. [60] Raschka, S. (2023). [Parameter-Efficient LLM Fine-Tuning with LoRA](https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms). [61] Houlsby, N., et al. (2019). [Parameter-Efficient Transfer Learning for NLP](https://arxiv.org/abs/1902.00751). ICML 2019. [62] Ouyang, L., et al. (2022). [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155). NeurIPS 2022. [63] Bai, Y., et al. (2022). [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073). arXiv preprint. [64] Ouyang, L., et al. (2022). [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155). NeurIPS 2022. [65] Sanh, V., et al. (2021). [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207). ICLR 2022. [66] Christiano, P., et al. (2017). [Deep Reinforcement Learning from Human Preferences](https://arxiv.org/abs/1706.03741). NeurIPS 2017. [67] Ouyang, L., et al. (2022). [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155). NeurIPS 2022. [68] Bai, Y., et al. (2022). [Training a Helpful and Harmless Assistant with RLHF](https://arxiv.org/abs/2204.05862). arXiv preprint. [69] Rafailov, R., et al. (2023). [Direct Preference Optimization](https://arxiv.org/abs/2305.18290). NeurIPS 2023. [70] Weng, L. [RLHF Overview](https://lilianweng.github.io/posts/2023-05-02-rlhf/). [71] McCloskey, M. & Cohen, N. (1989). [Catastrophic Interference in Connectionist Networks](https://www.sciencedirect.com/science/article/pii/S0079742108605368). Psychology of Learning and Motivation. [72] French, R. (1999). [Catastrophic Forgetting in Connectionist Networks](https://www.sciencedirect.com/science/article/pii/S1364661399012942). Trends in Cognitive Sciences. [73] Kirkpatrick, J., et al. (2017). [Overcoming Catastrophic Forgetting in Neural Networks](https://arxiv.org/abs/1612.00796). PNAS. [74] Schwarz, J., et al. (2018). [Progress & Compress: A scalable framework for continual learning](https://arxiv.org/abs/1805.06370). ICML 2018. [75] Robins, A. (1995). [Catastrophic Forgetting, Rehearsal and Pseudorehearsal](https://www.tandfonline.com/doi/abs/10.1080/09540099550039318). Connection Science. [76] van de Ven, G., et al. (2022). [Three types of incremental learning](https://www.nature.com/articles/s42256-022-00568-3). Nature Machine Intelligence. [77] Mohan, A., et al. (2024). [Continuous Learning for Machine Learning Models](https://patents.google.com/patent/US12488798B1/en). US Patent 12488798. [78] McMahan, B., et al. (2017). [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/abs/1602.05629). AISTATS 2017. [79] Kairouz, P., et al. (2021). [Advances and Open Problems in Federated Learning](https://arxiv.org/abs/1912.04977). Foundations and Trends in Machine Learning. [80] McMahan, B., et al. (2017). [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/abs/1602.05629). AISTATS 2017. [81] Li, T., et al. (2020). [Federated Optimization in Heterogeneous Networks](https://arxiv.org/abs/1812.06127). MLSys 2020. [82] Mohan, A., et al. (2021). [Cross-silo Federated Training with Diversity Scaling and Self-Supervised Learning](https://ieeexplore.ieee.org/document/9414878). ICASSP 2021. [82a] Dwork, C. & Roth, A. (2014). [The Algorithmic Foundations of Differential Privacy](https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf). Foundations and Trends in Theoretical Computer Science. [82b] Abadi, M., et al. (2016). [Deep Learning with Differential Privacy](https://arxiv.org/abs/1607.00133). CCS 2016. [82c] McMahan, H., et al. (2018). [Learning Differentially Private Recurrent Language Models](https://arxiv.org/abs/1710.06963). ICLR 2018. [83] Shoeybi, M., et al. (2019). [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053). arXiv preprint. [84] Weng, L. [Large-Scale Training Survey](https://lilianweng.github.io/posts/2021-09-25-train-large/). [85] Li, S., et al. (2020). [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://arxiv.org/abs/2006.15704). VLDB 2020. [86] Sergeev, A. & Del Balso, M. (2018). [Horovod: fast and easy distributed deep learning in TensorFlow](https://arxiv.org/abs/1802.05799). arXiv preprint. [87] Rajbhandari, S., et al. (2020). [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054). SC 2020. [88] Ren, J., et al. (2021). [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840). USENIX ATC 2021. [89] Shoeybi, M., et al. (2019). [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053). arXiv preprint. [89a] Jia, Z., et al. (2018). [Beyond Data and Model Parallelism for Deep Neural Networks](https://arxiv.org/abs/1807.05358). MLSys 2019. [89b] Narayanan, D., et al. (2021). [Memory-Efficient Pipeline-Parallel DNN Training](https://arxiv.org/abs/2006.09503). ICML 2021. [90] Huang, Y., et al. (2019). [GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism](https://arxiv.org/abs/1811.06965). NeurIPS 2019. [91] Narayanan, D., et al. (2021). [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://arxiv.org/abs/2104.04473). SC 2021. [92] Pope, R., et al. (2022). [Efficiently Scaling Transformer Inference](https://arxiv.org/abs/2211.05102). MLSys 2023. [93] Kwon, W., et al. (2023). [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). SOSP 2023. [94] Gama, J., et al. (2014). [A Survey on Concept Drift Adaptation](https://dl.acm.org/doi/10.1145/2523813). ACM Computing Surveys. [95] Lu, J., et al. (2018). [Learning under Concept Drift: A Review](https://ieeexplore.ieee.org/document/8496795). IEEE TKDE. [96] He, Y., et al. (2019). [Streaming End-to-End Speech Recognition For Mobile Devices](https://arxiv.org/abs/1811.06621). ICASSP 2019. [97] Prabhavalkar, R., et al. (2017). [A Comparison of Sequence-to-Sequence Models for Speech Recognition](https://www.isca-speech.org/archive/interspeech_2017/prabhavalkar17_interspeech.html). Interspeech 2017. [98] Chung, F. (1997). [Spectral Graph Theory](https://mathweb.ucsd.edu/~fan/research/revised.html). American Mathematical Society. [99] Kipf, T. & Welling, M. (2017). [Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907). ICLR 2017. [102] Korthikanti, V., et al. (2022). [Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/abs/2205.05198). MLSys 2023. [103] Liu, Y., et al. (2023). [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/abs/2310.01889). arXiv preprint. [98] Pfeiffer, J., et al. (2020). [AdapterHub: A Framework for Adapting Transformers](https://arxiv.org/abs/2007.07779). EMNLP 2020. [99] Li, X. & Liang, P. (2021). [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190). ACL 2021. [100] Dao, T., et al. (2022). [FlashAttention: Fast and Memory-Efficient Exact Attention](https://arxiv.org/abs/2205.14135). NeurIPS 2022. [101] Dao, T. (2023). [FlashAttention-2: Faster Attention with Better Parallelism](https://arxiv.org/abs/2307.08691). ICLR 2024. [102] Korthikanti, V., et al. (2022). [Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/abs/2205.05198). MLSys 2023. [103] Liu, S., et al. (2023). [Sequence Parallelism: Long Sequence Training from System Perspective](https://arxiv.org/abs/2105.13120). ACL 2023. [104] Jacob, B., et al. (2018). [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://arxiv.org/abs/1712.05877). CVPR 2018. [105] Nagel, M., et al. (2021). [A White Paper on Neural Network Quantization](https://arxiv.org/abs/2106.08295). arXiv preprint. [106] Gholami, A., et al. (2021). [A Survey of Quantization Methods for Efficient Neural Network Inference](https://arxiv.org/abs/2103.13630). arXiv preprint. [106a] Ioffe, S. & Szegedy, C. (2015). [Batch Normalization: Accelerating Deep Network Training](https://arxiv.org/abs/1502.03167). ICML 2015. [106b] Ba, J., et al. (2016). [Layer Normalization](https://arxiv.org/abs/1607.06450). arXiv preprint. [106c] He, K., et al. (2016). [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385). CVPR 2016. [106d] He, K., et al. (2016). [Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027). ECCV 2016. [107] Kubernetes Documentation. [Kubernetes Concepts](https://kubernetes.io/docs/concepts/). Official Kubernetes documentation. [108] NVIDIA. [Triton Inference Server](https://github.com/triton-inference-server/server). Multi-framework inference serving. [109] Crankshaw, D., et al. (2017). [Clipper: A Low-Latency Online Prediction Serving System](https://arxiv.org/abs/1612.03079). NSDI 2017.