Inference
3 articles
-
InferencePerplexity's Mac Engine Reads 24x Faster Than It Writes
Perplexity turned on hybrid compute in its Mac app on 1 September, then open sourced Lily, the engine underneath it, on 2 September. Lily averages 4,156 prefill tokens per second and 170 decode tokens per second on an M5 Max. That 24x gap is the real design constraint, and it explains exactly which half of the work Perplexity kept on the device.
-
InferenceGPT-5.6 Rewrote Its Own GPU Kernels
OpenAI cut GPT-5.6 Luna 80 percent on July 30, three weeks after general availability, funded by Sol rewriting OpenAI's own production GPU kernels and its speculative-decoding draft model. The transferable part is not the discount, it is FpSan, the floating-point sanitizer built to verify kernels no human read line by line.
-
Open WeightsKimi K3 Is Free. Running It Takes Eight B300s.
Moonshot released the full 2.8-trillion-parameter Kimi K3 weights on July 26, a 1.56TB download that vLLM will only serve on a node of at least eight B300s. The software gate closed, since Moonshot upstreamed its attention kernel for day-zero vLLM and SGLang support. The hardware bill and a bespoke license are what stop you now.