Guilherme Penedo

• Oct 2025 We have just released FineWiki, an updated, better extracted version of Wikipedia in 300+ languages
• Oct 2025 I will be attending COLM 2025 in Montreal, Canada. Presenting as a spotlight: "FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language"

• 🏆 Open-R1: Olympic coding

  2025
  Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching et al.
  Datasets and models to produce DeepSeek-R1 like reasoning models excelling at competitive programming
  Blogpost Models & Datasets Bibtex
  ×

  @misc{penedo2025codeforces,
        title={CodeForces}, 
        author={Guilherme Penedo and Anton Lozhkov and Hynek Kydlíček and Loubna Ben Allal and Edward Beeching and Agustín Piqueres Lajarín and Quentin Gallouédec and Nathan Habib and Lewis Tunstall and Leandro von Werra},
        year={2025},
        publisher = {Hugging Face},
        journal = {Hugging Face repository},
        howpublished = {\url{https://huggingface.co/datasets/open-r1/codeforces}}
  }

  Copy to Clipboard

• 🥂 FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language

  2024
  Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan et al.
  Second Conference on Language Modeling (COLM) 2025 Spotlight
  An extension of FineWeb to over 1000 languages
  Arxiv Dataset Abstract Bibtex

  Large language models have demonstrated remarkable capabilities across diverse tasks, yet their performance heavily relies on the quality and scale of pre-training data. While recent efforts have focused on improving data quality for English, most languages lack comparable resources. We introduce FineWeb2, a comprehensive pipeline for processing web data that adapts to any language while maintaining consistent quality standards. Our approach combines language-agnostic filtering techniques with language-specific optimizations, enabling the creation of high-quality pre-training datasets across 95 languages. We demonstrate that models trained on FineWeb2 data consistently outperform those trained on existing multilingual datasets, with particularly strong improvements for low-resource languages. By open-sourcing both our pipeline and processed datasets, we aim to democratize access to high-quality pre-training data and accelerate multilingual AI development.

  ×

  @misc{penedo2025fineweb2pipelinescale,
        title={FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language}, 
        author={Guilherme Penedo and Hynek Kydlíček and Vinko Sabolčec and Bettina Messmer and Negar Foroutan and Amir Hossein Kargaran and Colin Raffel and Martin Jaggi and Leandro Von Werra and Thomas Wolf},
        year={2025},
        eprint={2506.20920},
        archivePrefix={arXiv},
        primaryClass={cs.CL},
        url={https://arxiv.org/abs/2506.20920}, 
  }

  Copy to Clipboard

• 🍷 The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

  2024
  Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell et al.
  Advances in Neural Information Processing Systems (NeurIPS) 2024 Spotlight
  A state-of-the-art 15-trillion+ token English dataset for LLM pre-training
  Arxiv Dataset Blogpost Abstract Bibtex

  The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.

  ×

  @inproceedings{NEURIPS2024_370df50c,
   author = {Penedo, Guilherme and Kydl\'{\'\i}\v{c}ek, Hynek and allal, Loubna Ben and Lozhkov, Anton and Mitchell, Margaret and Raffel, Colin and Von Werra, Leandro and Wolf, Thomas},
   booktitle = {Advances in Neural Information Processing Systems},
   doi = {10.52202/079017-0970},
   editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
   pages = {30811--30849},
   publisher = {Curran Associates, Inc.},
   title = {The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},
   url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf},
   volume = {37},
   year = {2024}
  }

  Copy to Clipboard

• 📀 The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

  2023
  Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli et al.
  Advances in Neural Information Processing Systems (NeurIPS) 2023
  Pre-training dataset used to train the Falcon LLMs
  Arxiv Dataset Abstract Bibtex

  Large language models are commonly trained on a mixture of filtered web data and curated high-quality'' corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation, and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 500 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

  ×

  @inproceedings{NEURIPS2023_fa3ed726,
   author = {Penedo, Guilherme and Malartic, Quentin and Hesslow, Daniel and Cojocaru, Ruxandra and Alobeidli, Hamza and Cappelli, Alessandro and Pannier, Baptiste and Almazrouei, Ebtesam and Launay, Julien},
   booktitle = {Advances in Neural Information Processing Systems},
   editor = {A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine},
   pages = {79155--79172},
   publisher = {Curran Associates, Inc.},
   title = {The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only},
   url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf},
   volume = {36},
   year = {2023}
  }

  Copy to Clipboard

See all Publications, Projects, and Blogposts →

No talks yet.