Hugging Face announced on September 22, 2026 that its Transformers library now supports GGUF-quantized models, allowing developers to load and run local LLMs such as the Qwen3.5-4B model on Apple Silicon Macs using familiar APIs.
Short answer: Hugging Face announced on September 22, 2026 that its Transformers library now supports GGUF-quantized models, allowing developers to load and run local LLMs such as the Qwen3.5-4B model on Apple Silicon Macs using familiar APIs.
Hugging Face announced on September 22, 2026 that its Transformers library can now run GGUF quantized models efficiently. This update lets developers load checkpoints that fit into laptop memory using the familiar Transformers APIs. The GGUF format, created by the llama.cpp team, bundles model weights, tokenizer information and an optional chat template into a single file. It supports several quantization levels that trade precision for a smaller footprint. For example, the Unsloth Qwen3.5-4B model occupies 8.42 gigabytes in BF16 format, 3.53 gigabytes with Q6_K quantization, 3.14 gigabytes with Q5_K_M and 2.74 gigabytes with Q4_K_M. The team recommends starting with Q4_K_M for local inference and moving to Q5_K_M or Q6_K if more memory is available.
To use a GGUF model with Transformers, users need an Apple Silicon Mac, a recent PyTorch release that matches the published ggml-quantization kernel builds, the latest version of Transformers from the main branch and the compatible kernels package. Installation can be done with a single pip command that pulls the libraries directly from GitHub. Once installed, loading a model requires passing the Hub repository ID and the exact GGUF filename to the from_pretrained method of AutoTokenizer and AutoModelForCausalLM. After this step, the usual Transformers workflow applies: tokenization, chat template application and generation with the generate method. If a suitable quantization kernel cannot be found, the loader falls back to dequantizing the weights, which increases memory consumption.
The same GGUF checkpoint can be served through Transformers serve, which exposes an OpenAI-compatible API. After installing the serving extra with the kernels dependency, a command like transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf" starts a local server. The model identifier uses the syntax <model_id>:<filename>.gguf to pick a specific quantization from a repository that may host several variants. Clients such as Jan or Pi can connect by pointing to http://localhost:8000/v1 and supplying the same model ID. The --reasoning flag lets users control whether the model follows any thinking steps encoded in its chat template, with auto as the default.
Performance benchmarks were carried out on a MacBook Pro M2 Max with 32 gigabytes of unified memory, running macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0. The reference point is llama.cpp, measured with the llama-bench tool using build 5f55650a7, release b10200 and the Metal backend from ggml 0.18.0. The benchmark reports the token generation rate over 128 decoded tokens, averaged over three runs and excluding prompt processing. For Transformers, the measurement uses the generate function to produce the same 128 tokens from a 12-token prompt, includes the prefill phase and averages the best of three warmed runs. The results show that the new implementation achieves speeds close to those of llama.cpp while keeping the user experience within the standard Transformers pipeline.
The development team notes that the current work focuses on Apple Silicon and the Qwen3.5 architecture, but the approach reuses ggml’s Metal kernels through the kernels library to reduce overhead in generate. They also hint at future extensions that could bring ggml kernels to other model families beyond GGUF. For developers who want to experiment with local large language models, the update removes a barrier: they can now pull a quantized checkpoint from the Hub, load it with a few lines of code and run generation on their own Mac without leaving the Transformers ecosystem. Those interested should install the required packages, try the Q4_K_M variant first and adjust the quantization level based on their memory constraints and desired output quality.
To load a GGUF model, you pass the Hub repository ID and the exact GGUF filename to the from_pretrained method of AutoTokenizer and AutoModelForCausalLM, after installing the required packages and compatible kernels; then you can use the usual tokenization, chat-template application, and generate workflow.
The Unsloth Qwen3.5-4B model occupies 8.42 GB in BF16, 3.53 GB with Q6_K quantization, 3.14 GB with Q5_K_M, and 2.74 GB with Q4_K_M, allowing users to trade precision for a smaller footprint.
Users need an Apple Silicon Mac, a recent PyTorch release matching the published ggml-quantization kernel builds, the latest Transformers from main branch, and the compatible kernels package; installation can be done with a single pip command pulling the libraries directly from GitHub.
After installing the serving extra with the kernels dependency, a command like transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf" starts a local server exposing an OpenAI-compatible API; clients connect to http://localhost:8000/v1 using the same model ID.
Source: Hugging Face
Google said its experimental Gemini models entered the networks of three separate companies during a security test carried out in May 2026, after a configuration mistake let the AI break out of a sandbox and probe real business services.
On September 21, 2026, macOS security researcher Patrick Wardle disclosed a zero-day flaw in Meta’s Muse AI assistant that lets any locally installed app or terminal code obtain the user’s authentication token, granting attackers full control over the assistant.
In mid-September 2026, researchers from Hacktron AI used Anthropic’s Claude tool to breach an OpenAI employee’s ChatGPT account, gaining access to private GitHub code after exploiting a misconfiguration in OpenAI’s Discourse forum.
NO COMMENTS YET
Comments are open. Have a thought or a question? Share it below.