Sebastian Raschka
ML research & education
AI researcher, author of "Machine Learning with PyTorch and Scikit-Learn". Staff ML at Lightning AI.
Recent activity
-
This video course, "Build Your Own Reasoning Model: A Full Video Course," covers the development of a reasoning model from scratch using Python and PyTorch, focusing on evaluation, inference-time scaling, reinforcement learning, and distillation. The course consists of 8 videos (11 hours of content) and accompanies the book "Build a Reasoning Model (From Scratch)". The instructor emphasizes the importance of understanding how reasoning models work and how they can be used to improve large language models (LLMs). The course is designed for developers and AI/ML enthusiasts who want to learn about reasoning models and how to implement them from scratch. AI summary
Read more → -
The article discusses the evolution of language models for text classification, from bag-of-words to Jev. Key points include: * Bag-of-words models, such as naive Bayes and logistic regression, are simple and effective for text classification but lose word order information. * Deep neural networks, including recurrent neural networks (RNNs) and convolutional neural networks (CNNs), can handle sequential data like text but are computationally expensive. * Transformers, introduced in 2017, have revolutionized NLP with their encoder-decoder architecture, allowing for parallel processing and attention mechanisms. * Encoder-style models like BERT and decoder-style models like GPT can be fine-tuned for text classification, achieving high accuracy with minimal effort. * Jev, a recently released model, aims to classify things quickly and efficiently, but its performance depends on the specific task and dataset. AI summary
Read more → -
Researchers have observed that GPT-6 Astra, a recent language model release from OpenAI, employs a technique called "looped transformers" or "recurrent depth" to improve its performance. This involves passing intermediate representations through the same transformer blocks multiple times, without adding new weights, which reduces the model's parameter count and memory requirements. However, this approach also adds computational cost and does not provide significant benefits over a conventional transformer with a larger number of distinct blocks. AI summary
Read more → -
Claude's text watermarking technique works by introducing a random seed that is derived from a secret key and the previous four tokens in the generated text. This seed is used to select the next token, making the text generation more deterministic in certain positions. By using a secret key, the watermarking process ensures that the same next token is selected for the same prompt, even with different random seeds. AI summary
Read more → -
A developer can build an AI text detector from scratch by fine-tuning a DistilBERT classifier on a labeled dataset, similar to Pangram models used by Substack's AI detection feature. The detector returns a 0-100 score indicating the likelihood of a text being AI-generated, and can be used to filter out spammy content or improve personal writing. By building a verifier-based LLM application, developers can study the limitations of AI detectors and explore alternative approaches to regular reasoning models. AI summary
Read more → -
Researchers have developed a way to control the effort mode of large language models (LLMs), allowing them to switch between low-effort, medium-effort, and high-effort reasoning modes. This is achieved through training using reinforcement learning with verifiable rewards (RLVR), which provides a reward signal for correct and incorrect responses. AI summary
Read more → -
To set up a production-ready coding agent with a fully local stack, developers can use open-source tools and open-weight Large Language Models (LLMs) hosted through an inference engine/runtime server, leveraging local models and harnesses for tasks such as code generation, file reading, command execution, and verification. This approach offers transparency, inspectability, and cost-effectiveness, with the added benefit of full control over the local setup. AI summary
Read more → -
Here's a summary of the article for a developer/AI-ML audience: Researchers have made significant progress in LLM architecture, exploring hybrid designs that combine regular attention layers with state space layers (e.g., Nemotron 3 and Mamba-3) to improve efficiency at long contexts. Other notable papers include work on MoE capacity allocation, activation behavior, and representation geometry, which have led to the development of more efficient and scalable models. AI summary
Read more → -
Recent developments in Large Language Model (LLM) architectures, such as KV sharing, multi-head compression (mHC), and compressed attention, aim to reduce long-context costs by minimizing memory traffic and attention costs, enabling the use of longer contexts in reasoning models and agent workflows. Architectures like Gemma 4, Laguna XS.2, ZAYA1-8B, and DeepSeek V4 incorporate these techniques, including KV sharing, per-layer embeddings, attention budgeting, and compressed attention. These innovations enable the creation of larger, more efficient LLMs that can handle longer contexts without excessive memory usage. AI summary
Read more → -
To understand LLM architectures, Sebastian Raschka follows a workflow that starts with official technical reports, then moves to inspecting the config file and reference implementation of the model, especially when working with open-weight models supported by the Python transformers library. This manual process involves analyzing code and config files to gain insights into the architecture details. The goal is to learn how these architectures work, and doing so manually is considered a valuable exercise. AI summary
Read more → -
A coding agent is a layer on top of a large language model (LLM) that uses tools, memory, and repository context to make the model work better in practice. This agent harness manages context, tool use, prompts, state, and control flow to enable more effective coding tasks. AI summary
Read more → -
Modern LLMs employ various attention variants, including Multi-Head Attention (MHA), which processes input sequences by running self-attention heads in parallel with learned projections, and sparse attention, which reduces the dimensionality of the attention matrix. Hybrid architectures, such as Olmo 2, combine multiple attention variants to achieve improved performance. AI summary
Read more → -
Arcee AI's Trinity Large, released on January 27, 2026, features a 400B param Mixture-of-Experts (MoE) with 13B active parameters, incorporating alternating local:global (sliding window) attention layers, QK-Norm, and no positional embeddings, similar to SmolLM3. The model's architecture is compared to z.AI's GLM-4.5, highlighting similarities and differences. AI summary
Read more → -
Inference-time scaling is a method to improve LLM reasoning by allocating more compute and time during inference, leading to better answer quality and accuracy, with various techniques including chain-of-thought prompting, self-consistency, and rejection sampling with a verifier. These methods can be categorized into four main approaches: chain-of-thought prompting, self-consistency, best-of-N ranking, and rejection sampling. Recent work has also explored combinations of these methods and their applications in recursive language models. AI summary
Read more → -
Here's a summary of the article in 2-3 plain sentences for a developer/AI-ML audience: The year 2025 saw significant progress in large language models (LLMs) with the release of DeepSeek's R1 paper, which introduced a new approach to developing reasoning-like behavior using reinforcement learning (RL) and a novel algorithm called GRPO. This development led to the creation of reasoning variants of major open-weight and proprietary LLMs, with many researchers focusing on post-training methods to improve LLM capabilities. The RLVR and GRPO approach has the potential to unlock capabilities through scaling compute during post-training, making it a promising direction for future LLM development. AI summary
Read more →