The BiOS inference engine: serving as a cache-management problem

A chat request is mostly new. An agent request is mostly old, and that difference decides where the cost goes.

An agent reading a codebase sends the same codebase on every step. A support workflow carries the same long standing instructions into every conversation. A document assistant re-reads the same document for the fortieth question about it. In each case the expensive part of the work was already done, often minutes ago, and a stack that treats every request as novel pays for it again.

Once you accept that framing, throughput becomes a memory-management question more than an arithmetic one[2], and attention performance is dominated by data movement rather than by the maths[3].

That is why kernels and cache tiers are first-class concerns in the design rather than optimisations bolted on later. Kernels sit behind one portable interface with a registry, so the engine can select the fastest implementation for whatever hardware it finds. Caching is layered, because the repetition happens at several different granularities at once.

A property follows that inverts the usual relationship between scale and bill: cost per unit of work falls as reuse rises. The more an agent revisits the same material, the cheaper each visit becomes. For a workload built on a large shared context, that is the difference between a pilot and a system somebody can afford to run.

Request lifecycle and cache ownership are modelled as an explicit state machine, with the safety of reuse checked when the engine is compiled rather than hoped for at runtime. Caching bugs are the worst class of bug in a serving system precisely because they produce plausible output, so this is the right place to spend a type system.

Every tier above is exact. A cached path and an uncached path produce the same output.

This sounds like a technical footnote and is actually the whole reason the caching is useful. Approximate caching forces a quality decision onto whoever configures the system, and a quality decision that saves money is one nobody can evaluate honestly under deadline. Exactness removes the decision: caching becomes a pure cost reduction, so it can be on by default rather than being a setting somebody has to justify in a review.

A single image runs across accelerator generations and selects the fastest kernel for whatever it finds at startup.

For a hosted service this is a convenience. For an on-premise customer it is a requirement, and it is the constraint most serving stacks quietly fail. An enterprise fleet is not uniform: it accumulated over several purchasing cycles, it contains more than one generation, and none of it is going to be replaced to suit a vendor's build matrix.

A stack that needs a different build per generation turns every hardware refresh into a software project, and turns any heterogeneity into a reason not to deploy. Detecting the hardware and choosing the right implementation is the difference between a product that can be installed where the customer actually lives and one that can only be installed where the vendor wishes they did.

Serving is where the traces come from, which makes it the first stage of the learning loop rather than the last stage of the pipeline.

Every answer this engine serves, and every human judgement attached to it, is the material the training engine consumes to produce the next version of the model. A model trained in one system and served by another has no path from a live answer back to itself. Owning both ends is what closes that path.

It also changes the economics in a direction the market is moving anyway. The price of general capability has fallen sharply while enterprise bills have risen[4], and spending on the model and platform layer is growing at more than sixty per cent a year[5]. That spend is going into volume rather than into advantage. A right-sized model, served on hardware the organisation already pays for, with the repeated parts of every request cached exactly, is a different cost structure, and it is only available to a vendor that controls both layers and the path between them.