5 — AI Integration, Model Serving & Agentic Systems

Hosted Model APIs or Self-Hosting A companion note to the opening slide of Integrating AI: Model APIs, Self-Hosting and Agents Md Ahbab Hamid Khan https://ahbab.dev/ August 24, 2026 Summary. This note fills in the machinery behind onechoice: call a model over somebody else’s API, or run the model on hardwareyou control. It starts with what a single call actually does, from text totokens to the first streamed character, because that shape sets every costand latency number that follows. It then turns the usual talking points intomeasurable axes, gives a break-even model with the algebra written out andone worked illustrative example, and works through the memory arithmetic youneed before you can size a GPU. It closes with a short map of the later partsof the deck and a glossary. No machine learning background is assumed. Everynumber here is invented to show the shape of the problem, and is labelled asillustrative. 1 What a model call really is A hosted model endpoint looks like any other web service. You send JSON, youget JSON back. That surface hides a machine with a very particular cost shape,and almost every later argument in this deck is a consequence of it. Figure 1shows the pat

1.3 Prefill and decode Serving one request has two phases with completely different physics.Prefill reads the whole prompt at once. Every token can be processed inparallel, so this phase saturates the arithmetic units of the GPU. It iscompute bound. Its cost grows roughly in proportion to prompt length, with anextra term from attention that only starts to bite at very long contexts.Prefill leaves behind the KV cache, the per token attention state thatthe model must keep in GPU memory for the rest of the request. Section 4shows how quickly that cache grows.Decode produces the answer one token at a time. Each new token requiresa pass that reads the model weights again, so this phase is limited by memorybandwidth, not arithmetic. It is strictly sequential within a single sequence,because token twelve depends on token eleven. You cannot make one answerfaster by adding more work in parallel; you can only make more answers happenat once.Two consequences follow, and they matter later. First, time to the firstvisible character tracks prompt length, while total wall clock time tracksoutput length. Second, output tokens are far more expensive in machine timethan input tokens, which is why pr

Table 1: Illustrative latency budget for one streamed request with a 3,000token prompt and a 300 token answer. Every number is invented to showproportions, not to describe any product. Stage Illustrative time What drives it Client network and TLS 25 ms Distance to the region, connection reuse. Gateway, auth, quota 10 ms Fixed overhead per call. Queue wait 40 ms Load. The first number that explodes under peak traffic. Tokenise 5 ms Prompt length. Prefill 180 ms Prompt length and available GPU com- pute. Time to first token 260 ms Sum of the rows above. This is what the user feels. Decode, 300 tokens at 22 ms each 6,600 ms Output length, memory bandwidth, batch size. Total wall clock 6,860 ms Dominated by output length, not by prompt length. 2.1 Unit cost against amortised hours A hosted API charges per million tokens, usually with a lower price for inputthan for output. The bill is variable and it starts at zero. A self-hosteddeployment charges per GPU hour, whether the card is decoding or idle at threein the morning. The bill is fixed and it starts high.These are not comparable until you convert one into the other. The bridge isthroughput: how many tokens per hour one replica susta