Bringing Language Models and Chat to Web Browsers with ML Compilation on TVM Unity

The mlc-ai/web-llm Github repository hosts a project that aims to bring language models and chat to web browsers with no server support. The project utilizes machine learning compilation (MLC) on TVM Unity, along with WebGPU, to enable native GPU executions on the browser. The project also offers support for client personal AI models with cost reduction, privacy protection, and enhanced personalization. Compression techniques, memory planning optimizations, and int4 quantization are used to make the models fit into memory. The TVM web runtime, Emscripten and TypeScript, and wasm port of SentencePiece tokenizer are also used.

The project is made possible thanks to the open-source ecosystem, including Apache TVM, Hugging Face, LLaMA, Alpaca, Vicuna, Dolly, WebAssembly, Emscripten, and WebGPU communities. The project is mainly done in Python, with a 600 loc JavaScript app for connecting things together. Comparison to Native GPU Runtime, Limitations, and Opportunities are also discussed in the repository.