feat(implicit): add theoretical bridge and amortized estimation sections

Add NTK-style theoretical bridge connecting architectural conditioning
to varying-coefficient regression, plus a new "Amortized Estimation:
Context Encoders" subsection with estimation evolution figure.

Author: cnellington

content/07.implicit.md

@@ -12,10 +12,19 @@ In this section, we offer a systematic review of the mechanisms underlying impli
 
 The capacity for implicit adaptation does not originate from a single mechanism, but reflects a range of capabilities grounded in fundamental principles of neural network design. Unlike approaches that adjust parameters by directly mapping context to coefficients, implicit adaptation emerges from the way information is processed within a model, even when the global parameters remain fixed. To provide a basis for understanding more advanced forms of adaptation, such as in-context learning, this section reviews the architectural components that enable context-aware computation. We begin with simple context-as-input models and then discuss the more dynamic forms of conditioning enabled by attention mechanisms.
 
-#### Architectural Conditioning via Context Inputs
+#### Theoretical Bridge: Architectural Conditioning via Context Inputs
 
 In contrast to explicit parameter mapping, the simplest route to implicit adaptation is to feed context directly as part of the input. The simplest form of implicit adaptation appears in neural network models that directly incorporate context as part of their input. In models written as $y_i = g([x_i, c_i]; \Phi)$, context features $c_i$ are concatenated with the primary features $x_i$, and the mapping $g$ is determined by a single set of fixed global weights $\Phi$. Even though these parameters do not change during inference, the network’s nonlinear structure allows it to capture complex interactions. As a result, the relationship between $x_i$ and $y_i$ can vary depending on the specific value of $c_i$.
 
+<!-- Todo: Explore NTKs to make this explicit -->
+The connection is explicit for differentiable models $g$. Consider the model $P(Y | X, C)$ as a varying-coefficient regression model. An explicit estimator for regression parameters will solve for the regression parameter map $\beta_i = f(c_i)$ through
+$$\hat{f} = \text{argmin}_f \sum_i (y_i - x_i \cdot f(c_i))^2,$$
+while a differentiable model (e.g. a neural network) will solve 
+$$\hat{\Phi} = \text{argmin}_\Phi \sum_i (y_i - g([x_i, c_i]; \Phi).$$
+Under mild assumptions, these result in an identical solution for the intermediate regression parameters $\beta$. While the varying-coefficient model solves this explicitly, these can be obtained post-hoc from the differentiable model by differentiating with respect to $c_i$
+$$\beta_i = \frac{\delta}{\delta c} g([x_i, c_i]; \Phi).$$
+This is the first-order Taylor approximation of the model, a locally linear approximation [@doi:10.48550/arXiv.1602.04938] often used in post-hoc interpretation methods.
+
 This basic yet powerful principle is central to many conditional prediction tasks. For example, personalized recommendation systems often combine a user embedding (as context) with item features to predict ratings. Similarly, in multi-task learning frameworks, shared networks learn representations conditioned on task or environment identifiers, which allows a single model to solve multiple related problems [@doi:10.48550/arXiv.1706.05098].
 
 #### Interaction Effects and Attention Mechanisms
@@ -24,7 +33,7 @@ Modern architectures go beyond simple input concatenation by introducing interac
 
 Attention allows a model to assign varying degrees of importance to different parts of an input sequence, depending on the overall context. In the self-attention mechanism, each element in a sequence computes a set of query, key, and value vectors. The model then evaluates the relevance of each element to every other element, and these relevance scores determine a weighted sum of the value vectors. This process enables the model to focus on the most relevant contextual information for each step in computation. The ability to adapt processing dynamically in this way is not dictated by explicit parameter functions, but emerges from the network’s internal organization. By enabling dynamic, input-dependent weighting, attention supports context-aware computation without altering global parameters, thereby setting the stage for advanced on-the-fly adaptation such as in-context learning.
 
-### Amortized Inference and Meta-Learning
+### Amortized Inference, Meta-Learning, and Context Encoding
 
 Moving beyond fixed architectures that implicitly adapt, another family of methods deliberately trains models to become efficient learners. These approaches, broadly termed meta-learning or "learning to learn," distribute the cost of adaptation across a diverse training phase. As a result, models can make rapid, task-specific adjustments during inference. Rather than focusing on solving a single problem, these methods train models to learn the process of problem-solving itself. This perspective provides an important conceptual foundation for understanding the in-context learning capabilities of foundation models.
 
@@ -38,14 +47,26 @@ Meta-learning builds upon these ideas by training models on a broad distribution
 
 Gradient-based meta-learning frameworks such as Model-Agnostic Meta-Learning (MAML) illustrate this principle. In these frameworks, the model discovers a set of initial parameters that can be quickly adapted to a new task with only a small number of gradient updates [@doi:10.48550/arXiv.1703.03400]. Training proceeds in a nested loop: the inner loop simulates adaptation to individual tasks, while the outer loop updates the initial parameters to improve adaptability across tasks. As a result, the capacity for adaptation becomes encoded in the meta-learned parameters themselves. When confronted with a new task at inference, the model can rapidly achieve strong performance using just a few examples, without the need for a hand-crafted mapping from context to parameters. In this view, the capacity to adapt becomes encoded in the meta-learned parameters themselves, enabling rapid generalization from few examples without a hand-crafted map from context to coefficients and standing in clear contrast to explicit approaches.
 
+#### Amortized Estimation: Context Encoders
+
+In the previous chapter we looked at modeling frameworks with explicit context-adaptive components. The most common implementation is a context encoder, which is learned to map from a task's context to a set of task-specific parameters. The result is an amortized estimator, which predicts the parameters of the downstream model that would have been produced by a classical estimator if sufficient data was collected from that context.
+
+This is desirable because data collection and model estimation are often far more costly than model inference. Plotting the progression of estimation methods, classical methods require a sufficient amount of data for every task. Transfer learning and meta-learning are formulated to achieve similar performance with fewer samples, but still require explicit gradient-based estimators. Pushing this to its limit produces amortized estimators, such as context encoders, which only use a task's context to infer the task-specific data distribution and produce a in-context inferences. 
+
+![Evoluation of Statistical Estimation. Classical estimators (left) require sufficiently large context-specific datasets for single-instance learning. Meta-learning and transfer learning (middle) decrease the need for context-specific data collection, but still perform a few rounds of gradient descent on context-specific data. Finally, amortized estimators (right) only use a representation of the data context to produce new model parameters, handling data collection and classical estimation implicitly.](images/estimation_evolution.png){#fig:estimation-evolution width="80%"}  
+
+In this regime, formal estimators are no longer required after the context encoder is obtained. We assume the context encoder can internally sample the data distribution and perform the necessary estimation steps entirely within a forward pass, similar to how amortized inference infers trajectories for expensive inference procedures. The key question becomes a practical one: When we don't need data, how do we encode the context of our hypothetical data distribution? Ironically, context itself is often hard to parameterize. A clearer representation of data distribution may be a few representative samples from the distribution itself. This leads to the practical implementation of implicit in-context learning with LLMs, where a few samples provided as natural language serve as context for implicit amortized estimation. 
+
 ### In-Context Learning in Foundation Models
 
-The most powerful and, arguably, most enigmatic form of implicit adaptivity is in-context learning (ICL), an emergent capability of large-scale foundation models. This phenomenon has become a central focus of modern AI research, as it represents a significant shift in how models learn and adapt to new tasks. This section provides an expanded review of ICL, beginning with a description of the core phenomenon, then deconstructing the key factors that influence its performance, reviewing the leading hypotheses for its underlying mechanisms, and concluding with its current limitations and open questions.
+The most powerful and, arguably, most enigmatic form of implicit adaptivity is in-context learning (ICL), an emergent capability of large-scale foundation models. This phenomenon has become a central focus of modern AI research, as it represents a significant shift in how models learn and adapt to new tasks. This section provides an expanded review of ICL, beginning with a description of the core phenomenon and how it relates to context-adaptive inference, then deconstructing the key factors that influence its performance, reviewing the leading hypotheses for its underlying mechanisms, and concluding with its current limitations and open questions.
 
 #### The Phenomenon of Few-Shot In-Context Learning
 
 First systematically demonstrated in large language models such as GPT-3 [@doi:10.48550/arXiv.2005.14165], ICL is the ability of a model to perform a new task after being conditioned on just a few examples provided in its input prompt. Critically, this adaptation occurs entirely within a single forward pass, without any updates to the model's weights. For instance, a model can be prompted with a few English-to-French translation pairs and then successfully translate a new word, effectively learning the task on the fly. This capability supports a broad range of applications, including few-shot classification, following complex instructions, and even inducing and applying simple algorithms from examples. Subsequent work has shown that the ability to generalize from few in-context examples can itself be enhanced through meta-training. MetaICL explicitly trains models across diverse meta-tasks, teaching them to infer and adapt within context at test time without gradient updates, thereby strengthening the implicit adaptability of large language models [@doi:10.48550/arXiv.2110.15943].
 
+This behavior runs against the grain of other context-adaptive inference methods. 
+
 #### Deconstructing ICL: Key Influencing Factors
 
 The effectiveness of ICL is not guaranteed and depends heavily on several interacting factors, which have been the subject of extensive empirical investigation.