jundot /omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
★ 23,653⑂ 2,134👁 113⑃ 2,570● 2,393▤ 138
+19
Daily
+325
Weekly
+789
Monthly
23,653
stars
2,134
forks
113
watchers
2,570
prs
2,393
issues
138
releases
README
oMLX LLM inference, optimized for your MacContinuous batching and tiered KV caching, managed directly from your menu bar. [email protected] · https://omlx.ai/me Install · Quickstart · Features · Models · CLI Configuration · Benchmarks · oMLX.ai English · 中文 · 한국어 · 日本語 --- Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar. oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it. Install macOS App Download the .dmg from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click. The macOS app also installs a lightweight ~/.omlx/bin/omlx CLI shim so terminal commands and Apple Shortcuts can control the app-managed server. Homebrew bash brew tap jundot/omlx https://github.com/jundot/omlx brew install jundot/omlx/omlx --with-custom-kernel With native custom kernels (needs full Xcode) No full Xcode? brew install jundot/omlx/omlx installs without the kernels Upgrade to the latest version brew update && brew upgrade omlx Run as a background service (auto-restarts on crash) omlx start Optional: MCP (Model Context Protocol) support, then restart to load it /opt/homebrew/opt/omlx/libexec/bin/pip install mcp omlx restart From Source bash git clone https://github.com/jundot/omlx.git cd omlx make install Editable install with the web UI and native custom kernels No full Xcode? make install-no-kernels installs without the kernels make mcp Optional: MCP support, restart omlx serve to load it Requires macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4/M5). Note on native custom kernels: GLM-5.2, MiniMax M3, and Qwen3.5 need them. Without them those families silently fall back to much slower generic paths -- for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels (measured 845 vs ~29 tok/s on an M3 Ultra), and the fallback also uses more memory (2137). make install builds them and checks that each one loads. That needs full Xcode with the Metal toolchain (xcodebuild -downloadComponent MetalToolchain); Command Line Tools alone do not provide it (xcrun: error: unable to find utility "metal"). Without Xcode, use the official DMG, which ships the kernels precompiled, or make install-no-kernels. A plain pip install -e . also skips them. make kernels rebuilds only the kernels from scratch. Homebrew builds them with --with-custom-kernel, which also needs full Xcode. To verify any install: bash python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_st
Top issues
#114m5 运行很慢27 comments · 2026-03-10#1835macOS 27: long-context inference 10x slower than macOS 26.6 (even after HOST_VM_INFO64 fix)16 comments · 2026-06-11#1550Qwen 3.6 MTP + batch of 2 = huge slowdown and corruption14 comments · 2026-05-30#42130.7.0 release: Memory Guard dynamic ceiling issue killing workflow13 comments · 2026-10-02#5223Prefill throttle still inflates per_token after the enforcer reclaims the pool (single request, tight ceiling): follow-up to #4587 / #460312 comments · 2026-11-13#5075Time To Full Decode Speed (TTFDS) and Stream Interrupt12 comments · 2026-11-04#2150Lightning MTP (PR #2113) regresses tgTPS by 8-12% on Qwen3.6-35B-A3B (MoE) vs. old depth-1 MTP, and is fully bypassed during continuous batching11 comments · 2026-07-09#3697Can mlx-lm/mlx-vlm be kept up-to-date?10 comments · 2026-09-16
Discussions
#5270Per-request LoRA adapters, to use AutoTrust JEV-27B for both chat and decisions1 comments · 2026-11-14#5256Separate models selection for audio processing (tts & stt) and reasoning in Ask Ai0 comments · 2026-11-14#5249Prefill Accelerator Mac Upgrade Help0 comments · 2026-11-13#5231Parallel requests for decision model0 comments · 2026-11-13#5220LeapQuant support0 comments · 2026-11-12#5217Cross-instance prefix cache transfer for load-balanced oMLX replicas0 comments · 2026-11-12#5215Try same-prompt prefill across two CUDA helper hosts0 comments · 2026-11-12#5214Support discrete CUDA VRAM monitoring for mixed clusters0 comments · 2026-11-12
Latest release
0.8.0rc1 v0.8.0rc1 2026-11-12oMLX 0.8.0rc1 This is a release candidate. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.3. The release candidate for oMLX 0.8.0 adds the Prefill Accelerator, which lets a PC with NVIDIA GPUs or a second Mac speed up prefill for a big model on your Mac, and makes DeepSeek V4.1 faster with many requests and on clusters. Download: macOS 26 / 27 | macOS 15 Sequoia If you downloaded build 3727 right after release, the Cluster tab may not find a DGX Spark or NVIDIA PC, so please download it again. Prefill Accelerator with NVIDIA
相关报道 (5)
Yahoo奇摩新聞Mac 本機 AI 編碼 agent 搭建指南:oMLX + OpenCode + Qwen 3.6 完整設定教學2026-05-18腾讯网Mac用户可以在oMLX中使用TurboQuant了,搭配Gemma-4-31B,谷歌全家桶实测很能打!2026-04-08cyberqoMLX 本地大語言模型伺服器 MacOS 平台地端 LLM 部署實作2026-03-09CSDN博客DFlash 2 发布:解码速度近自回归 3 倍,多模型平均接受长度显著提升!2026-08-20CSDNoMLX:为 Mac 优化的大语言模型推理工具,多特性助力高效开发与应用_IT界那些事儿2026-05-11
News & coverage (12)
Hacker NewsOMLX: Local LLM Server for Apple Silicon Macs▲ 4 · 0 comments · 2026-05-11Hacker NewsShow HN: oMLX – SSD-backed KV cache cuts coding agent TTFT from 90s to 1s on Mac▲ 4 · 1 comments · 2026-03-04Hacker NewsShow HN: oMLX – coding agents on local LLMs without the painful reprefill▲ 3 · 0 comments · 2026-02-24Hacker NewsOMLX – LLM Inference Server for Apple Silicon (Ollama for MLX)▲ 3 · 0 comments · 2026-02-13Hacker NewsOMLX v0.3.9 Stable Merges Native MTP (Multi-Token Prediction)▲ 2 · 0 comments · 2026-05-22Hacker NewsOMLX – Ollama for MLX (LLM Inference Server for Apple Silicon)▲ 2 · 0 comments · 2026-02-15Hacker NewsOMLX: Use a second Mac to speed up local LLM prefill 2.9x▲ 1 · 0 comments · 2026-11-07Hacker NewsShow HN: oMLX – Native Mac inference server that persists KV cache to SSD▲ 1 · 0 comments · 2026-02-19noteHow to use oMLX: Keep models resident on your Mac and stop recalculating conversation continuations2026-09-07noteTTL이라는 답을 찾아 아날로그로 속도를 측정한 이야기 (oMLX 편)2026-08-16