A vast riveted steel grain hopper filling the frame, tapering to a single narrow spout releasing one thin trickle into a nearly empty tin pail, photoreal

Perplexity Open Sourced Its Mac Inference Engine. It Reads 24 Times Faster Than It Writes.

Lily, released 2 September under MIT, averages 4,156 prefill tokens per second and 170 decode tokens per second on an M5 Max. That 24x gap is not a defect, it is the shape of every model you will ever run on a laptop.

The local half of Perplexity's new hybrid setup classifies and routes, which is almost entirely a reading job. The workload was matched to what the hardware is good at, not the other way around.

The speedups came from giving things up. Lily runs one model on one chip family, and its four largest optimizations are all architecture-specific. You can read the code, you cannot point it at your model.

Perplexity shipped two things one day apart, and the second one explains the first.

On 1 September it turned on hybrid compute in the Mac app. Each Perplexity Computer task now gets split between frontier models in the cloud and a small model running on your machine. The local model is the gate. It looks at every subtask, decides whether it contains names, addresses, account numbers, credentials, payment cards or government IDs, and then either keeps the work on device, masks the sensitive fields, rewrites the request so the cloud model can continue without them, or refuses. Enterprise admins can set org-wide rules for what must stay local and audit when anything leaves a device.

On 2 September it open sourced Lily, the engine underneath that. Rust runtime, hand-written Metal kernels, no PyTorch and no MLX anywhere in the execution path. MIT license, dropped into Perplexity's pplx-garden repo. It wants macOS 15 or later and 24 GB of unified memory, with 32 GB recommended.

The blog post is framed as a speed story. Lily averages 1.23x MLX-LM on prefill and 1.35x on decode across ten prompt lengths and ten decode contexts, at 0.04% perplexity difference and the same top-ranked token 96.35% of the time. Those are respectable numbers for a runtime that has to produce identical output.

But the ratio inside the benchmark is more useful than the ratio against MLX.

4,156 in, 170 out

On a 40-core, 128 GB M5 Max, Lily averaged 4,156 prefill tokens per second and 170.0 decode tokens per second. At a 4K prompt it hits 5,749.9 prefill and 186.6 decode.

Reading is roughly twenty-four times faster than writing on the same chip, running the same model, in the same process.

This is not specific to Lily. It is what the hardware does. Prefill takes the whole prompt at once and pushes it through the weights in large matrix multiplies, so one pass over memory covers thousands of tokens. Decode produces one token at a time, and each token needs its own trip through the weights. Prefill is bounded by how fast the GPU can compute. Decode is bounded by how fast memory can feed it. On a laptop, memory bandwidth is the wall, and you hit it once per output token.

The practical version: a local model that reads 8,000 tokens of your files and emits a twenty-token routing decision finishes in about two seconds, and most of that is the read. The same model writing a 2,000 token answer takes twelve seconds of pure generation, and there is no prompt engineering that gets you out of it.

The gate is a reading job

Look back at what Perplexity gave the local model to do. Classify sensitivity. Detect account numbers. Decide a route. Rewrite a request with the protected fields removed.

Every one of those is short output over long input. The gate reads your financial file and writes a verdict. It reads the draft request and writes a slightly shorter draft request. Nothing in the privacy gate's job description asks a 35B model on a MacBook to produce paragraphs.

Meanwhile the cloud keeps frontier reasoning, web search and planning, which is where the long generations live.

That split is usually explained as a privacy decision, and the privacy story is real. But it is also the split that the silicon was going to force anyway. The tasks that stayed local are the tasks a laptop runs at 4,156 tokens per second. The tasks that went to the cloud are the ones it would have run at 170.

If you are designing your own local and cloud split, this is the line to cut on. Not "small tasks local, big tasks cloud." Not "cheap local, expensive cloud." Split on output length. Classification, extraction, routing, redaction and tool selection belong on the device, and they stay fast even when the input is enormous. Summarization into anything longer than a paragraph, drafting, code generation and multi-step reasoning traces belong upstream, because the device charges you per token produced and the bill is measured in seconds of waiting.

The speed came from giving things up

The other thing worth taking from Lily is how the 1.23x and 1.35x were earned, because the answer is not "better engineering."

Lily supports one model. Qwen3.6-35B-A3B, 35 billion parameters with about 3 billion active per token, 256 experts of which 8 plus one shared are selected, 10 full-attention layers using grouped-query attention with 16 query heads and 2 KV heads, and 30 Gated DeltaNet layers. The checkpoint is quantized to 4-bit groupwise affine, every group of 64 weights sharing a bfloat16 scale and bias, which takes roughly 70 GB of bfloat16 weights down to 19.4 GB.

Perplexity lists the four optimizations that moved the needle most. GPU-resident expert routing, worth 89%. Dequantization fused into the grouped matrix multiply, worth 77.4%. GQA packing, worth 23.8% at 32K context. Fixed-block attention, worth 40.2% at 128K.

Read those again as constraints rather than achievements. Expert routing on the GPU only matters because the model is a mixture of experts. Fusing dequantization into the GEMM only matters because of that specific 4-bit group layout. GQA packing only helps because of the 16-to-2 head ratio. Fixed-block attention pays off because 30 of the 40 layers are not full attention at all.

None of that lifts into a general runtime. It is a set of bets on one architecture, compiled against one GPU family, and the payoff is between 1.2x and 1.4x.

That is the honest price of specialization at this layer, and it is worth knowing before you decide to write your own kernels. A team that builds inference for a living, targeting a single model on a single chip, with no framework overhead in the path, beat the well-optimized general option by about a third on the metric that actually gates user experience.

What to do with it

If you run anything locally, benchmark prefill and decode separately. A single tokens-per-second number averaged across both hides the only ratio that determines whether a design works, and most published local-inference numbers are quoted in a way that flatters prefill.

If you are building a hybrid product, write down the expected output length of every subtask before you decide where it runs. That column predicts local latency better than input size, model size or task difficulty.

And if you were waiting for local inference to feel fast enough to move real work down from the cloud, it already is, for the half of the work that mostly reads. The other half is going to stay upstream until memory bandwidth on consumer machines changes, and that is a hardware roadmap problem, not a software one.

Perplexity did not open source a general engine. It open sourced a very sharp answer to one question, and published the measurements that show which question it was.

Sources