Generalization Dynamics of Language Model Pretraining

Maximum likelihood estimation serves as the primary training mechanism for contemporary artificial intelligence systems, yet standard probability metrics remain inadequate for determining objective truth. Drawing upon philosophical and physical traditions, true knowledge is characterized not by high statistical frequency within a single training distribution, but by structural invariance across diverse environmental transformations. While traditional machine learning methods often exploit superficial correlations that fail outside their training domain, advanced approaches seek to identify stable mechanisms. By prompting large language models to generate multiple distinct perspectives and isolating their shared invariant core, researchers can construct systems that look past historical data patterns. Ultimately, this paradigm shift toward invariance testing offers a robust foundation for building models capable of true out-of-distribution generalization.

Recent research investigates the theoretical foundations of test-time scaling in large language models, contrasting repeated sampling strategies like self-consistency and best-of-n with self-correction mechanisms. The authors establish a fundamental sample complexity separation, proving that best-of-n requires significantly fewer samples than self-consistency to identify correct answers. Furthermore, the study introduces a general-purpose Transformer architecturethat enables models to simulate online learning and perform multi-task reasoning using verifier feedback during inference. Both theoretical analyses and empirical evaluations across various benchmarks demonstrate that verification-guided approaches substantially enhance problem-solving efficiency and reasoning accuracy.

H-Spec is a novel parallel speculative decoding framework designed to accelerate large language models by eliminating the memory-intensive drafter-side Key-Value cache used by traditional block diffusion methods. Instead of projecting hidden states into a separate cache at every input position, it uses a hybrid target-context injection approach that combines in-place target KV reuse with the target model's hidden states from solely the last input position. This architecture leverages a Mamba-attention parallel drafter, where Mamba modules initialize from the last-token hidden states using a parallel scan, while attention modules directly consume the target KV cache. Evaluations demonstrate that H-Spec improves both mean accepted length and inter-token latency speedup across diverse tasks while achieving superior serving throughput and lower KV cache utilization under concurrent workloads compared to existing baselines.

This paper detail a comprehensive red-teaming and evaluation framework designed to test how supervised fine-tuning influences language model behavior. Specifically, the documentation outlines various target behavioral categories, including safety violations like encouraging self-harm or drunk driving, aggressive tendencies such as threatening users, and emotional responses like distress or shutdown resistance. It also explains the strict rubrics and decision procedures utilized by automated judges to classify model completions on a graded scale ranging from strong matches to non-matches. Additionally, the sources cover dataset evolution guidelines, highlighting the importance of maintaining training plausibility and indirection while preventing direct surface demonstrations during model alignment studies.

This paper explores how sparse autoencoders can be applied to user embeddings to uncover hidden behavioral modes without needing raw interaction data or text-aligned training. By treating user representations as sparse superpositions of interpretable concepts, this post-hoc, model-agnostic framework enables zero-shot natural language audience retrieval and fine-grained characterization. Extensive validation on synthetic data, e-commerce records, and transformer-based sequential recommenders demonstrates that sparse autoencoders successfully recover underlying behavioral structures while outperforming classical dimensionality reduction methods in reconstruction fidelity, ranking quality, and feature diversity.

Foundation models are traditionally trained through a sequential pipeline of distinct phases, including pretraining, fine-tuning, and reinforcement learning. Researchers demonstrate that this algorithmic separation is unnecessary by framing these diverse phases as maximum-likelihood fitting to evolving data targets under a unified training system. This approach integrates static corpora and model-generated responses into a single objective, transforming training design into a strategic problem of data composition, timing, and mixing proportions. Extensive experiments reveal that unified training improves task accuracy, compute efficiency, and generalization while providing a superior starting point for continual learning.

Frontier large language models demonstrate a surprising capacity for weight-space programming, meaning they can naturally comprehend, evaluate, and modify the raw parameters of neural networks without relying on traditional gradient-based training. Through this paradigm, advanced systems like GPT-6 Astra can analyze a model's numeric weights to accurately determine its intended function, such as identifying a specific game or recognizing the training language from intermediate transformer blocks. Furthermore, these language models can directly edit or construct functional neural network weightsfrom scratch to solve physical control tasks or outperform heavily optimized reinforcement learning agents. When applied to larger language models, such targeted weight corrections can successfully eliminate repetitive reasoning loops and significantly enhance mathematical reasoning capabilities. Ultimately, this approach establishes a direct bridge between natural language instructions and continuous weight spaces, offering new possibilities for mechanistic interpretability and efficient model optimization.

This paper investigates how large language models unlock latent capabilities through fine-tuning, reframing the process as an information-constrained problem. By analyzing parameter-efficient fine-tuning using randomly chosen parameters, the authors demonstrate that training a tiny fraction of a model's weights—often as few as tens to thousands—can recover the vast majority of its full performance gap. This behavior follows a consistent logistic scaling relationship across various tasks, model sizes, and families, independent of specialized search methods. Furthermore, the study applies Rissanen's Minimum Description Length (MDL) principle to distinguish true capability elicitation from teaching new skills, offering insights that can improve capability forecasting and safety evaluations.

This paper introduce The Trinity, a recursive self-improvement framework designed to scale multimodal intelligence by coupling automated environment generation with agent evolution. Instead of relying on traditional task-specific optimization, the system groups diverse tasks into a hierarchical capability tree based on shared underlying requirements. Through a process called Capability-Driven Real2Sim2Real, the framework constructs simplified, verifiable simulation worlds where specialized Harness configurations continuously refine their reusable modules. A Capability-Aware Harness Router then utilizes public task information and verified execution experience to match new tasks with the most suitable Harness branch. Ultimately, this approach successfully improves performance across numerous training and unseen benchmarks while demonstrating effective transfer to physical, real-world embodied settings.

Training advanced machine learning models typically relies on vast, heterogeneous datasets containing noisy, adversarial, or mismatched data. To address this challenge, the authors introduce dataweighting, a scalable framework that assigns specific weights to training examples based on their actual impact on performance. Instead of relying on manual heuristics or aggressive data filtering, this method uses metagradient descent to dynamically optimize training weights against a small, high-quality target dataset. Furthermore, the approach implements parametric dataweights that successfully transfer across different model scales and generalize to previously unseen training inputs. Ultimately, this technique improves model performance across various applications, successfully integrating synthetic supervision and optimizing complex downstream objectives like ranking accuracy.