Journal of Artificial Intelligence, Applications and Innovations 3060-7124 1 Journal of Artificial Intelligence, Application and Inovations 2024 04 02 naab: A ready-to-use plug-and-play corpus for Farsi naab: A ready-to-use plug-and-play corpus for Farsi 1 8 10.61838/jaiai.1.2.1 EN Journal Article 2023 11 10 <p>The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce Naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. Naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word ناب (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.</p> https://www.journalaiai.com/index.php/aiai/article/download/7/7