In our increasingly interconnected world, language remains a persistent barrier. While machine translation has improved dramatically for major languages like English, Spanish, and Mandarin, quality translation still lags for thousands of smaller languages.

A new effort called No Language Left Behind (NLLB) aims to close this gap by developing high-quality translation systems covering over 200 languages. The project employs a human-centered approach to translation, collaborating with speakers of diverse languages to ensure cultural relevance. This responds directly to the needs illuminated in interviews with translators and language communities, who found existing systems unreliable.

Creating useful systems requires quality training data. But most languages lack large parallel corpora needed for machine translation. NLLB’s ingenious solution mines the web to extract potential translations from multilingual websites. Using specialized cross-lingual sentence embedding models, they align sentences conveying similar meanings in different languages with high accuracy. This novel mining process enabled building datasets for low-resource languages at unprecedented scale.

For evaluation, NLLB created the Flores benchmark spanning 200 languages. Careful protocols ensure accurate human translation and alignment. Flores helps quantify progress, with higher scores indicating better translation quality. NLLB models achieve state-of-the-art results on Flores, with many languages nearing usability. The project also conducted human studies confirming considerable gains over past systems.

Another key innovation was tailoring the training process to prevent overfitting on high-resource languages. Curriculum learning introduces low-resource languages slowly to focus model capacity. A mixture-of-experts architecture further reduced interference between languages. Together these advances allowed a single model to translate between all 200+ directions while maintaining specialization.

The NLLB project has committed to ethics and inclusion. Their partnership with Wikipedia is enabling content translation for many smaller wikipedias. With potential harms like cultural erasure in mind, the project also interviewed speakers from vulnerable language communities to guide decision making. Framing translation as a human right, the goal is to enhance global communication.

NLLB's ambitions embody AI's expanding potential to connect us. Machine translation was long considered nearly hopeless for low-resource languages. But the project upends these assumptions by proving quality translation possible for all tongues when equity becomes a design principle.

NLLB believes openness and collaboration will be central to fulfilling this vision. Models and datasets are fully public to catalyze innovation. The project also plans grants, support for adapting their tools, and partnerships with researchers invested in specific languages. Only through such collective effort can we ensure no language gets left out.

NLLB faces challenges in improving under-served language pairs, capturing informal dialects, and evaluation beyond text. But these are solvable given dedicated investment and commitment. Progress shows what's possible when innovation grounds itself in lived experience.

Past AI has often encoded the priorities of English-speaking elites. NLLB points toward an alternative future, designed for international users. When technology accommodates our differences rather than erasing them, new opportunities for connection emerge.

No Language Left Behind offers a roadmap for inclusive AI. Its translation advancements - enabled by novel datasets, training methods, and modeling architectures - carry lessons for ensuring equity more widely. Expanding access requires investigating deficiencies and confronting barriers directly.

The project's mission resonates far beyond translation, to all intelligent systems. For as long as exclusion persists, the work remains unfinished. These efforts bring us closer to AI that knows no language boundaries, only the poetry of voices united across divides.

source:

arxiv