Brij Kishore Pandey

Brij Kishore PandeySep 16, 2026

An LLM does not generate a sentence. It generates one token. Then another. Then another. But a lot happens before each token reaches your screen. Here is the simplified inference pipeline: Your prompt is split into tokens. Each token is converted into a numerical embedding. Transformer layers use attention and feed-forward networks to process the context. The model produces a probability distribution across its vocabulary. A decoding strategy selects the next token. That token is added to the context, and the cycle repeats. Tokens are converted back into text and streamed to you. The most interesting part is that inference has two very different phases: Prefill processes the input tokens largely in parallel. Decode generates tokens sequentially, making memory movement, KV-cache size, and bandwidth increasingly important. This is why: → Longer prompts increase prefill work → Longer conversations grow the KV cache → Quantization can reduce memory requirements → FlashAttention reduces expensive memory reads → Speculative decoding can improve generation speed → Temperature, Top-K, and Top-P change how the next token is selected The model may look like it is "writing." Underneath, it is repeatedly performing numerical operations, selecting a token, updating its context, and doing it all again. I mapped the complete flow in one diagram. Save it if you are learning how LLM inference actually works.

Brij Kishore PandeySep 17, 2026

Your AI agent gives the right answer. It also takes 15 model calls, retries the same failed tool three times, and costs more than the task is worth. Would you ship it? The engineering around the model is what decides that outcome. These are the concepts worth understanding. 𝟭. 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗹𝗼𝗼𝗽𝘀 Plan, act, observe, adjust. Without a clear stopping condition, a failed attempt turns into an endless loop. 𝟮. 𝗠𝗖𝗣 A standard way for agents to connect to tools and data. It does not decide which actions each agent is allowed to take, that part is still yours. 𝟯. 𝗦𝘂𝗯𝗮𝗴𝗲𝗻𝘁𝘀 𝗮𝗻𝗱 𝗺𝘂𝗹𝘁𝗶 𝗮𝗴𝗲𝗻𝘁 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 Delegate bounded tasks with clear inputs, context, and outputs. Each added agent also adds coordination overhead. 𝟰. 𝗔𝗜 𝗴𝗮𝘁𝗲𝘄𝗮𝘆 A shared layer for authentication, model routing, rate limits, and usage controls. Without it, every application ends up building the same plumbing on its own. 𝟱. 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗲𝗰𝗼𝗻𝗼𝗺𝗶𝗰𝘀 Cost per completed task, not cost per call. Token counts, caching, model choice, and repeated calls all show up on the bill. 𝟲. 𝗘𝘃𝗮𝗹𝘀 A definition of what a successful result looks like, tested against realistic tasks, edge cases, and failure modes before a change ships. 𝟳. 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 Checks on inputs, tool actions, and outputs. A model requesting an action should never be the only thing authorizing it. 𝟴. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 A complete trace of the run: model calls, tool responses, latency, errors, spend. When something fails, this is the evidence you need to explain why. 𝟵. 𝗔𝗴𝗲𝗻𝘁 𝗵𝗮𝗿𝗻𝗲𝘀𝘀 The runtime holding everything together: state, memory, tool execution, checks, recovery. These concepts connect in practice, not just in theory. A repeated tool call shows up in your traces. It drives up cost. An eval can expose the pattern. A limit in the harness can stop it. Each step is testable on its own. If you are learning this stack, take one small agent and follow a single task from request to completion. Then deliberately break a tool call and watch what the system does next. That single exercise teaches more than reading about any one of these nine concepts in isolation. Which of these nine layers do you think gets skipped most often when teams move agents into production too fast?

Brij Kishore PandeySep 18, 2026

Your coding agent can read the whole repo. It still cannot see what happens when that code hits production. I keep seeing the same pattern. Someone asks the agent to add retries. The patch looks fine. Tests pass. It gets merged. Then one dependency slows down. Retries stack. Queues grow. The connection pool dies. “Add retries” was never the hard part. The hard part is knowing what that change does to the system around it. If you are going to let an agent touch production paths, give it runtime evidence first: a trace that shows where the request actually waits metrics for latency and errors under load logs that tie a failure back to one request Then the prompt changes. Not: “add retries.” This: “Under load, retries are amplifying pressure on this dependency. Look at the trace and propose a bounded retry strategy.” Same agent. Much better job. You still review the diff. Dynatrace wrote this up well , including how agents can pull that telemetry through tools like dtctl. Worth reading if your team is past the “first generated commit” stage. Link in the comments. Which signal do you hand the agent first: traces, metrics, or logs? Thanks to Dynatrace for sponsoring this post.