AI4Bharat and NVIDIA Release 2.8 Trillion-Token Indic Corpus

AI4Bharat NVIDIA release 2.8 trillion token Indian language corpus

On October 16, 2026, the AI community saw a big leap toward more inclusive language models when AI4Bharat and NVIDIA teamed up to roll out a 2.8-trillion-token dataset that covers all 22 languages listed in India’s constitution. Dubbed Nemotron-pretraining-Indic-v1, the release pulls together native-script text for every language and romanised versions for twelve of them, totalling 2,825.6 billion tokens. That tally splits into 2,601.0 billion native-script tokens and 224.6 billion romanised tokens, all spun from a carefully curated set of English source documents that span mathematics, code, question-answer pairs, and general web text.

The dataset isn’t just a static pile of files; it ships with a fully open, GPU-accelerated pipeline that mirrors the entire generation workflow. Using NVIDIA’s NeMo Data Designer and the Gemma-4-31B-it model, the pipeline translates full English documents into each target language, evaluates translation quality, fixes any low-scoring outputs, and then, for the twelve dual-script languages, turns a subset of the accepted translations into Latin script. Each step is guided by a rubric inspired by the Multidimensional Quality Metrics framework, and only outputs scoring at least 75 out of 100 on that rubric are kept. The pipeline also embeds provenance tags that link every synthetic record back to the original English source, making the data auditable and traceable.

Why Indian language AI lags: data shortage highlighted by Khapra

The initiative responds to a long-standing issue flagged by AI4Bharat’s director Mitesh Khapra, who noted that Indian language technology trails English because of a lack of data. By combining a massive volume of synthetic text with an open recipe for generating more, the release aims to bridge that gap. The dataset is built from documents drawn from fifteen Nemotron-CC-v2 subsets, ensuring that the content includes not just general prose but also specialised technical material such as equations, tables, and code snippets. This breadth is crucial for developers who want models that can explain technical concepts and solve problems in Indian languages.

The 22 languages represented are the full slate listed in India’s constitution: Assamese, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani, Kashmiri, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, Urdu, and Bodo. For the twelve languages that support both native script and romanisation-Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu-the pipeline generates both forms. The native-script records come from translating the English source, while the romanised records are created by converting a subset of the accepted native-script translations into Latin script and then verifying that the conversion preserves meaning, fluency, and formatting. Early tests comparing this two-step approach to direct romanisation from English showed fewer hallucinations and less residual English, underscoring the value of the pipeline’s design.

Demo shows practical use of new 2.8 trillion token corpus

Beyond the raw data, the release also includes a practical demonstration of the corpus’s utility. NVIDIA’s Nemotron 3.5 Super VL model was continued pretraining on ten billion additional Indic tokens drawn from a twelve-language subset of the new corpus, along with natural Indic text. The resulting model reportedly shows broader Indic capabilities, and the technical report that accompanies the release details the full training process. The developers plan to build on this corpus for future Nemotron iterations, indicating that the dataset is not a one-off but a foundation for ongoing research.

For builders and users of AI, the implications are significant. First, the sheer scale of 2.8 trillion tokens means that researchers can experiment with larger pretraining regimes without the usual data-collection bottlenecks. Second, the open pipeline lowers the barrier to creating custom synthetic data: teams can feed their own English documents-whether domain-specific or general-into the workflow, tweak prompts or quality thresholds, and produce high-quality Indic text that matches their particular needs. Third, the provenance tracking ensures that downstream users can audit the data, a feature that becomes increasingly important as models are deployed in sensitive contexts.

What developers should do next after corpus release

What should developers do next? The release is already hosted on Hugging Face, where the dataset and the pipeline scripts can be downloaded and run on any GPU-accelerated infrastructure. Teams that rely on English-centric models can start by fine-tuning their existing architectures on the native-script data to improve accuracy in Indian languages. Those who need romanised text, perhaps for applications that prefer Latin script for easier input, can use the pipeline’s romanisation step to generate the required data. Finally, the open recipe invites community contributions: researchers can suggest new prompts, improve the quality rubric, or add additional source documents, thereby expanding the dataset further.

In short, AI4Bharat and NVIDIA have delivered more than a new dataset; they have provided a reusable framework that democratises large-scale synthetic data generation for a linguistically diverse region. By making both the data and the process open, they are encouraging a wave of innovation that could bring advanced language models to the nearly 1.2 billion people who speak Indian languages, a demographic that has historically been underserved in the AI ecosystem.

Source: Hugging Face