Data quality is crucial. Filtering can remove unwanted data, improving training efficiency and ensuring desirable properties like high information content, desired languages, low toxicity, and minimal personally identifiable information. Consider trade-offs when using filters and understand the importance of data mixtures.
A tool for selecting data with a similar distribution to a target dataset
A Python framework for defining Taggers that identify non-language text, language ID, PII, toxic text, and “quality” text. Includes reimplementation of heuristics used by Gopher and C4 for non-natural language.
Frequently Used N-grams Language ID model, a character 4-gram model trained to recognize up to 1633 languages.
A tool to predict the language of text, used to filter out/in data from the desired languages
A pipeline for processing and improving quality of crowdsourced datasets
Pre-trained spoken language identification model trained on VoxLingua107, dataset of audio sourced from YouTube for 107 languages
A series of scripts to replicate the Pile dataset. Includes filtering and cleaning for: language, profanity, deduplication, and test set decontamination.
