Hugging Face Transformers Adds GGUF Support for Local LLMs

Hugging Face Transformers adds GGUF support for local LLMs

Hugging Face announced on September 22, 2026 that its Transformers library can now run GGUF quantized models efficiently. This update lets developers load checkpoints that fit into laptop memory using the familiar Transformers APIs. The GGUF format, created by the llama.cpp team, bundles model weights, tokenizer information and an optional chat template into a single file. It supports several quantization levels that trade precision for a smaller footprint. For example, the Unsloth Qwen3.5-4B model occupies 8.42 gigabytes in BF16 format, 3.53 gigabytes with Q6_K quantization, 3.14 gigabytes with Q5_K_M and 2.74 gigabytes with Q4_K_M. The team recommends starting with Q4_K_M for local inference and moving to Q5_K_M or Q6_K if more memory is available.

What you need to run GGUF models with Transformers on Apple Silicon Mac

To use a GGUF model with Transformers, users need an Apple Silicon Mac, a recent PyTorch release that matches the published ggml-quantization kernel builds, the latest version of Transformers from the main branch and the compatible kernels package. Installation can be done with a single pip command that pulls the libraries directly from GitHub. Once installed, loading a model requires passing the Hub repository ID and the exact GGUF filename to the from_pretrained method of AutoTokenizer and AutoModelForCausalLM. After this step, the usual Transformers workflow applies: tokenization, chat template application and generation with the generate method. If a suitable quantization kernel cannot be found, the loader falls back to dequantizing the weights, which increases memory consumption.

Serving GGUF models via Transformers serve with OpenAI-compatible API

The same GGUF checkpoint can be served through Transformers serve, which exposes an OpenAI-compatible API. After installing the serving extra with the kernels dependency, a command like transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf" starts a local server. The model identifier uses the syntax <model_id>:<filename>.gguf to pick a specific quantization from a repository that may host several variants. Clients such as Jan or Pi can connect by pointing to http://localhost:8000/v1 and supplying the same model ID. The --reasoning flag lets users control whether the model follows any thinking steps encoded in its chat template, with auto as the default.

Performance benchmarks were carried out on a MacBook Pro M2 Max with 32 gigabytes of unified memory, running macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0. The reference point is llama.cpp, measured with the llama-bench tool using build 5f55650a7, release b10200 and the Metal backend from ggml 0.18.0. The benchmark reports the token generation rate over 128 decoded tokens, averaged over three runs and excluding prompt processing. For Transformers, the measurement uses the generate function to produce the same 128 tokens from a 12-token prompt, includes the prefill phase and averages the best of three warmed runs. The results show that the new implementation achieves speeds close to those of llama.cpp while keeping the user experience within the standard Transformers pipeline.

The development team notes that the current work focuses on Apple Silicon and the Qwen3.5 architecture, but the approach reuses ggml’s Metal kernels through the kernels library to reduce overhead in generate. They also hint at future extensions that could bring ggml kernels to other model families beyond GGUF. For developers who want to experiment with local large language models, the update removes a barrier: they can now pull a quantized checkpoint from the Hub, load it with a few lines of code and run generation on their own Mac without leaving the Transformers ecosystem. Those interested should install the required packages, try the Q4_K_M variant first and adjust the quantization level based on their memory constraints and desired output quality.

Frequently asked questions

How do you load a GGUF model using the Transformers library?

To load a GGUF model, you pass the Hub repository ID and the exact GGUF filename to the from_pretrained method of AutoTokenizer and AutoModelForCausalLM, after installing the required packages and compatible kernels; then you can use the usual tokenization, chat-template application, and generate workflow.

What quantization options are available for the Unsloth Qwen3.5-4B model and their approximate sizes?

The Unsloth Qwen3.5-4B model occupies 8.42 GB in BF16, 3.53 GB with Q6_K quantization, 3.14 GB with Q5_K_M, and 2.74 GB with Q4_K_M, allowing users to trade precision for a smaller footprint.

What system requirements and installation steps are needed to use GGUF support in Transformers on Apple Silicon?

Users need an Apple Silicon Mac, a recent PyTorch release matching the published ggml-quantization kernel builds, the latest Transformers from main branch, and the compatible kernels package; installation can be done with a single pip command pulling the libraries directly from GitHub.

How can you serve a GGUF model with Transformers serve and what API does it expose?

After installing the serving extra with the kernels dependency, a command like transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf" starts a local server exposing an OpenAI-compatible API; clients connect to http://localhost:8000/v1 using the same model ID.

Source: Hugging Face