Rishabh Misra

Rishabh MisraApr 30, 2026

Most teams aren’t building personalization in generative recommender systems. They’re building an expensive way to re-read user logs. A common pattern I see: → take a user’s months of interaction history → dump it into an LLM context window → expect high-quality personalization This breaks quickly. Latency blows up, costs scale with tokens, signal quality actually drops, and the model still misses what matters. The "Lost in the Middle" effect is very real in long-context personalization. So teams try: "Let’s just pass the last N interactions." This is the truncation trap because user intent isn’t just recent, it’s layered. → long-term preferences are stable and slow-changing → short-term intent is session-specific and fast-changing. In generative recsys, this is not a context window problem; it’s a memory routing problem. The architecture that works separates memory into two systems: [1] Long-term memory (offline layer) Compress history into a dense user summary: → Run an async summarization pipeline (e.g., nightly) → Maintain a dense user profile capturing stable preferences → Keep it small (on the order of a few hundred tokens) [2] Short-term memory (online retrieval layer) For immediate intent, use retrieval, not raw history. → Store interactions in a vector database (e.g., HNSW index) → At inference, retrieve only the top-K relevant items → Keep it tightly scoped to the current task Then at generation time, combine both: → Long-term summary (global preference) → Retrieved short-term context (local intent) The result is a system that preserves personalization quality, reduces latency significantly, and cuts token usage and cost by an order of magnitude. The key idea: what you choose not to send to the model is just as important as what you include. This is Part 5 of my Generative RecSys series. Next up: I'll break down some of the evaluation aspects for these complex systems. Curious: Where are you drawing the line between latency, cost, and context quality in your Generative RecSys stack?

Rishabh MisraMay 5, 2026

Most engineers still treat AI like a software wrapper. That’s exactly why they plateau. Over the past decade, I’ve built and reviewed production ML systems at big techs and startups. The pattern hasn’t changed - only the tools have. → The Plateau: Treating ML as a black-box API or prompting exercise → The Breakthrough: Treating ML as a systems engineering problem In the early era (2015–2020), the plateau was: using high-level libraries without understanding the math or data. Today, it’s: using LLM APIs without understanding the hardware. If you can’t reason about memory bandwidth, you’re not building for scale. A quick diagnostic for 2026: ① Why do LLMs become memory-bound during decoding - even on H100s? ② How does KV cache growth impact throughput under concurrency? ③ When does quantization hurt latency instead of helping? ④ How do you prevent long multimodal requests from blocking everything else? If these feel uncomfortable, you’re still operating at the application layer. When I designed LLM inference curriculum at Udacity, the goal wasn’t just to teach how to call models. It was to teach how they behave in production. The highest-leverage engineers in 2026 won’t write better prompts. They’ll build systems that serve large models reliably, efficiently, and at scale. I’ve been compiling the exact papers, mental models, and war stories I use to mentor senior engineers through this transition. We go deep on KV cache optimization, Batching strategies (continuous vs paged), Quantization trade-offs, Inference scheduling, and more. I’m opening a small, focused group for engineers who want to operate at this layer. If that’s you: comment "Inference" and I’ll share some resources + a short form to see if it’s a fit. I’m prioritizing people actively working on (or struggling with) production-scale systems.

Rishabh MisraMay 14, 2026

95% recommendation hit rate. 15% lower long-term retention. We called it the Accuracy Trap. And it almost broke the recommender system of an e-commerce startup I advised. Here's what happened, and the two metrics that saved us. A user clicks on a navy blue t-shirt. The model generates six more navy blue t-shirts. Hit rate? Near perfect. User experience? An echo chamber. Most recommender systems don't fail because the models are weak. They fail because they become boring prediction machines. Traditionally, engineers optimize for: Precision, Hit Rate, and NDCG. The SOTA systems also optimize for: Discovery, Long-term LTV, and Serendipity. The difference? User intent is exploratory, not just predictive. If your model only learns historical similarity, it slowly collapses. The good news: "Serendipity" is measurable. Two metrics I consider non-negotiable in production: ① Intra-List Diversity (ILD): Average cosine distance between recommended items. ILD → 0 means your feed is redundant. ② Novelty: Negative log probability of item popularity. Are you actually surfacing niche preferences, or just lazily recommending global bestsellers to everyone? One more layer: GenAI explanations. → If you're generating personalized recommendation explanations with LLMs, BLEU and ROUGE are no longer enough. → Semantic similarity matters more than token overlap. That's why BERTScore often outperforms both. The hardest lesson I learned building ML systems at Amazon and Twitter: * Your evaluation framework IS your product strategy * Models aggressively optimize whatever objective you give them - even when it actively hurts users. ───────────────────── This wraps up my Generative RecSys series. Past posts: → Semantic IDs: https://lnkd.in/g75sgtxd → Vision RecSys: https://lnkd.in/gg5efmzf → LLM Enhanced RecSys: https://lnkd.in/gBPUMSZa → Encode and Generate Paradigm: https://lnkd.in/g9MmWZaN → RecSys Context Management: https://lnkd.in/gs3nSKSc Follow along as I have interesting deep dives coming up. ───────────────────── Curious: What's one metric your team uses to keep your recommender from becoming a boring prediction machine?