FreeToken Enables 753B Model Inference on Single Workstation GPUs

A team of researchers from UC Berkeley and UT Austin propose FreeToken. [1] The system continuously maps computation and model state onto whatever GPU, CPU, memory and interconnect bandwidth the machine actually has. [1] The result is a 35B model at interactive speed on an 8 GB laptop GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation card. [1]

FreeToken is available under an Apache-2.0 license on GitHub, published on PyPI, and shipped as a desktop app for Windows and Linux. [1] The command line interface targets Linux x86_64 systems with an NVIDIA GPU on driver r580+ and CUDA 13. [1] It exposes OpenAI- and Anthropic-compatible endpoints on port 1919. [1] Enterprises should treat it as an air-gapped or regulated-workload path rather than a datacenter replacement. [1]

FreeToken addresses a critical bottleneck in the open-weight ecosystem: the prohibitive hardware costs of serving massive models like GLM-5.2 and DeepSeek-V4-Flash. By enabling local execution on consumer-grade hardware, the system allows startups and solo developers to bypass cloud API expenses, while providing a secure deployment route for sectors requiring strict data sovereignty, such as healthcare and defense. What the sources don't address: How the inference latency of a 753B model running across system memory and CPU on a single workstation compares in practical benchmarks against standard datacenter deployments.