LLM Engineering Roadmap: What You Learned and Where to Go Next

Forty lessons ago, a language model was a black box that somehow finished your sentences. Today you can explain how it is trained, ground it in your own data, adapt it, give it tools, and serve it in production. That is a real change in what you can build.

• Lesson 40 of 40
• Course complete

In the previous lesson, our fourth capstone deployed a tool-using agent, with guardrails and monitoring around it. That project quietly used almost everything in this course. Today we step back, connect the pieces, and decide where you go from here.

• Deployed a tool-using agent

We will build one mental map of the whole course. Then we cover the trade-offs and decision habits that matter most, be honest about what is still unsolved, and finish with a concrete roadmap. There is no new heavy math today, only perspective.

• Map, trade-offs, habits
• Limits, then roadmap

Here is the whole course in one picture. Foundations, then prompting, then retrieval, then adaptation, then agents, then production. Each layer sits on the one before it, and each is a bigger lever than the last, but also more expensive and harder to get right.

• Foundations to production

Foundations taught you that a model is trained to predict the next token, and that most of its behavior follows from that objective. You met scaling laws, sampling with temperature and top-p, context windows with rotary positions, the KV cache, and why hallucination is a natural side effect of fluent prediction.

• Next-token prediction
• Scaling, sampling, KV cache

Prompting showed how far instructions, examples, reasoning and structured output can take a frozen model. Retrieval went further: embeddings, vector indexes, chunking, reranking and hybrid search let the model answer from documents it was never trained on, with evidence you can inspect.

• Instructions and structure
• Ground answers in documents

Adaptation covered supervised fine-tuning, LoRA, QLoRA, reinforcement learning from human feedback, and DPO, for when behavior itself must change. Agents then added tool calling, the ReAct loop, memory, multi-agent designs, and the Model Context Protocol, so a model can act rather than only answer.

• LoRA, QLoRA, DPO
• Tools, loops, memory, MCP

Production tied it together. Latency and cost math, quantization, serving with continuous batching and PagedAttention, and guardrails against prompt injection are what separate a demo from a system people can rely on. Notice that this layer is mostly engineering discipline, not model magic.

• Latency, cost, quantization
• Serving, safety, monitoring

If you remember one thing about engineering with language models, remember that nothing is free. Every technique buys you something and charges you something else. Good engineers do not look for the best technique. They look for the cheapest one that clears the bar.

• No free lunch in LLM systems

Recall the decision from lesson twenty-six. If the model lacks knowledge, retrieve it. If it lacks the right behavior or style, fine-tune. If it just needs clearer instructions, prompt. Mixing these up is the classic mistake, like fine-tuning facts into weights that change next week.

• Knowledge: RAG
• Behavior: fine-tune
• Instructions: prompt

A larger model costs more per token and responds more slowly. Often a smaller model with good retrieval, clean context and a tight prompt matches it on your specific task. Whether that is true for you is not something to guess. It is something to measure on your own data.

• Model size vs pipeline design
• Small models, good context

Latency, cost and quality pull against one another. Longer reasoning chains improve answers but add tokens, which add time and money. Reranking improves retrieval but adds another model call. The question is never whether a step helps, but whether it helps enough to pay for itself.

• Pick two, tune the third

Agents add a similar trade. A fixed workflow is predictable and easy to test. A free-roaming agent is flexible but can wander, loop, or take actions you did not expect. A good rule is to give the model only as much autonomy as the task truly needs.

• More freedom, less predictability

The first habit is to build your evaluation before you build your system. Collect even fifty realistic questions with good reference answers. Without that, every change is an opinion, and you cannot tell an improvement from a lucky demo.

• Build the test set before the system

Here is an illustrative example with made-up numbers. On fifty test questions, a plain prompt answers thirty-one correctly, which is sixty-two percent. Adding retrieval lifts it to forty-one, eighty-two percent. A fine-tune might reach forty-three, eighty-six percent. Two more correct answers may not justify the training and maintenance burden.

• Illustrative numbers only
• 62% to 82% to 86%

The second habit is to start with the simplest thing that could work. Try a clear prompt first. Add retrieval when the failures are about missing knowledge. Fine-tune when failures are about behavior. Reach for agents last. Each rung of this ladder costs more to build, test and maintain.

• Start with a prompt
• Escalate only on evidence

The third habit is to measure cost early. Suppose, with invented prices, a request uses two thousand input tokens at one dollar per million, and three hundred output tokens at four dollars per million. That is two tenths of a cent plus about a tenth of a cent, so roughly point zero zero three two dollars per request.

• Tokens in, tokens out
• Multiply by traffic

Now multiply by traffic. At one hundred thousand requests a day, that tiny number becomes three hundred twenty dollars a day. Trimming the prompt, caching shared prefixes, or routing easy questions to a smaller model suddenly matters. Cost surprises almost always come from volume, not from any single call.

• 0.0032 dollars times 100,000
• 320 dollars per day

The common mistakes follow from skipping these habits. Teams tune prompts without a test set. They build multi-agent systems for problems a single call solves. And they never read their production logs, where real users show you exactly how the system fails. Looking at real failures is the highest-value hour in the week.

• No eval
• Over-engineering
• Ignoring failures in logs

Now for honesty. Hallucination is reduced by retrieval, citations, better training and verification, but it is not eliminated. A model optimized to produce plausible text can still produce plausible falsehoods. Design your system so that wrong answers are detectable and low-impact.

• Mitigated, not eliminated

Agents have a reliability problem rooted in simple arithmetic. If each step succeeds ninety-five percent of the time, ten steps in a row succeed about sixty percent of the time. Long chains of actions magnify small error rates, which is why checkpoints, retries and human approval remain important.

• Errors compound across steps

Evaluation is genuinely hard. Open-ended answers have no single correct string to compare. Using a model as judge is useful but can carry biases, and benchmarks can leak into training data. The practical answer is a mix: task-specific test sets, careful human review, and skepticism toward any single headline number.

• Open-ended answers
• Judges can be biased

Security is the limitation that deserves the most respect. When a model reads untrusted text, that text can contain instructions, and the model may follow them. Treat every tool as a permission, give agents least privilege, validate outputs, and never assume a prompt alone is a security boundary.

• Untrusted text is an attack surface

Beyond these, real open questions remain. How reliably can models reason over long tasks? Why exactly do they generalize? How do we make their behavior understandable and controllable? Nobody has final answers, and that is good news for you, because it means the frontier is still reachable.

• Reasoning, reliability, understanding

Your first advanced topic is multimodal models. These systems accept images, audio, or video alongside text, usually by turning each into tokens or embeddings a Transformer can process. Everything you learned about embeddings, attention, and evaluation carries over, with new data pipelines.

• Images, audio, video, text

Next is mixture of experts. Instead of every token passing through the same large feed-forward layer, a router sends each token to a small number of specialist sub-networks. The model holds many parameters but uses only some per token, trading memory and engineering complexity for compute efficiency.

• Route each token to a few experts

Then there is long context. Standard attention gets expensive as sequences grow, so researchers explore efficient attention, better positional schemes, and state-space models, which carry a compact running state instead of looking back at everything. Hybrids are an active area, and the trade-offs are still being mapped.

• Attention cost grows with length
• Alternatives: state-space models

Reinforcement learning for reasoning builds on what you saw with RLHF and reasoning models. When answers can be checked, such as math results or passing unit tests, a model can be rewarded for reaching correct outcomes through longer thinking. Read recent papers here with care, because details change quickly.

• Reward verifiable answers

Two more deserve a mention. Interpretability tries to understand what happens inside a network, finding features and circuits behind its behavior. Distributed training asks how to split data, parameters and computation across many accelerators. One explains models, the other makes giant ones possible.

• Look inside the model
• Train across many GPUs

To keep learning, get comfortable with papers. Read the abstract, then the figures, then the introduction and conclusion, and only then the method. Ask what is actually new, what was compared against, and whether the evaluation supports the claim. Re-implementing a small piece teaches more than rereading ten times.

• Abstract, figures, then method
• Check the evals

Next, get your hands dirty with open-weight models. Run one locally, quantize it, change the sampler, swap the tokenizer, fine-tune a small adapter. Break things on purpose and measure what happens. Everything you only understood in theory becomes real the first time you see your own model misbehave.

• Run, modify, break, measure

A portfolio beats a certificate. Build something for a real user, such as a Q and A bot over a community's documents, a code reviewer for your own repository, or a small fine-tuned model for a narrow domain. Publish your test set, your failures, and your cost numbers, because that shows engineering judgment.

• Solve a real problem
• Show your evals

The field moves fast, but not uniformly. Tools, model names and prices change constantly, while fundamentals such as attention, retrieval, evaluation and optimization age slowly. Learn the fundamentals deeply, hold specific tools loosely, and follow a few trusted sources instead of everything.

• Fundamentals age slowly
• Tools age fast

Let us recap. The map runs from foundations through prompting, retrieval, adaptation and agents to production. The habits are to evaluate first, use the simplest thing that works, and measure cost. The limits are hallucination, agent reliability, evaluation difficulty and security. And the roadmap is papers, open models, advanced topics and real projects.

• Map, habits, limits, roadmap

Thank you for sticking with all forty lessons. If any foundation felt shaky along the way, the Machine Learning from Zero to Advanced playlist is there to rebuild it, from gradient descent to the Transformer. Strong basics make every new paper easier to read.

• And back to the ML course

Then pick one small project and build it this week, with a test set.

• Subscribe for what's next
• Build something this week