A configuration mismatch in GLM 5.3 on Workers AI triggered an SGLang bug that exposed KV cache pages across requests. Here is how we addressed the issue.
Firehose
Filtered to tagged “KV Cache” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
artificial intelligence 70continual learning 27AI 18open-weight models 15reinforcement learning 14agentic coding 12AI agents 11large language models 9tech 9AI safety 7benchmarks 7finance 7open-source 7robotics 7language models 6machine learning 6Reinforcement learning 6Diffusion models 5game development 5natural language processing 5Agentic AI 4anthropic 4computer vision 4cybersecurity 4data integration 4deep learning 4Recursive self-improvement 4Vercel 4video games 4AI infrastructure 3
In this episode, Philip Kiely and Ali Taha from Baseten discuss the complexities and innovations in inference engineering for large AI models. They cover topics including model deployment, speculative decoding, quantization, hardware optimi…
Inference engineeringSpeculative decodingQuantizationModel deploymentTool callingKV cacheTensor parallelismExpert parallelismGPU hardwareRubin GPUVideo diffusionAutoregressive modelsDiffusion modelsTraining-inference convergenceContinual learningOpen source modelsInference infrastructureModel optimizationLatency vs throughputMulti-modal models
Reiner Pope delivers a blackboard lecture on the mathematical and hardware principles behind training and serving large language models. He explains how batch size, sparsity, and various parallelism strategies (expert, pipeline) impact late…
LLM trainingLLM inferenceBatch sizeLatency optimizationCost analysisRoofline analysisMemory bandwidthCompute performanceKV CacheSparsityMixture of ExpertsExpert parallelismData center architectureScale up networkScale out networkPipeline parallelismMicrobatchingMemory capacityChinchilla scalingRL generationAPI pricingContext lengthCryptographic ciphersNeural network architectureReversible networks