Together AI announces its Together Compiler for inference optimization

Together Compiler, also referred to as TKCC, takes traffic shape, the model and hardware as inputs, then uses agents to generate optimizations for an inference engine.[3] Together AI says agents work in parallel and communicate through a message board to generate optimization ideas.[3] It uses fast engines with a few model layers for rapid iteration, alongside slower engines with more layers to check performance on realistic data distributions.[3] This setup pairs quick experimentation with a more representative performance check.

Together AI reports up to 2.6 times higher TPS across four named frontier open-source models in less than 24 hours, and 252 times higher agent experiment throughput than baseline agentic performance-optimization systems.[3] Its comparison puts total time to serving at 467 seconds for the status quo and 9.0 seconds with Together Compiler.[3] The company also reports 1,111 tokens per second on MiniMax M3.1 Flash running on an NVIDIA B200, or 1.8 times its baseline, and 1,205 tokens per second on DeepSeek v4.1 Flash running on an NVIDIA GB300, or 1.3 times baseline.[2] These are company-reported results, not a guarantee of the same gains for other workloads.

Together AI says the compiler is already running across many of its most popular serverless endpoints and serves trillions of tokens per day.[2] Customers can request access to a limited beta for their own dedicated deployments, while self-serve access is planned for later.[2] The company describes production traffic spanning thousands of customers and many trillions of tokens per day, with variation in prompt lengths and customer-request bursts.[3] It reports that prompt lengths rose 80% in the two weeks after a model release as users shifted toward agentic workloads.[3] Together AI also reports speedups of 1.5 to 6.4 times on tested scientific models beyond language-model inference, with bit-exact outputs on those evaluations.[1]