Arabic RAG that works: embeddings, chunking, and reranking for real Arabic documents

You built a retrieval pipeline that works beautifully in English, pointed it at your Arabic contracts and policies, and the answers came back thin or plain wrong. A user searches for a word written with a hamza, and the document that spells it without one never surfaces. A dialectal question misses a formal Arabic answer sitting right there in the file. This is not your fault. Most retrieval stacks were tuned on English and quietly underperform on Arabic. The good news: the fixes are known, cheap, and measurable. This post walks an engineer through building an Arabic document assistant that actually retrieves.

Why English pipelines fail on Arabic

Arabic breaks three assumptions baked into most retrieval stacks. First, the same word can be written several valid ways: the letter alef appears as ا, أ, إ or آ, the hamza takes several forms, and ta marbuta (ة) is often typed as plain ha (ه). To a naive matcher these are different tokens. Second, diacritics (the small vowel marks, tashkeel) are optional, so the same word exists both with and without them, splitting matches in two. Third, Arabic is diglossic: your documents are usually in Modern Standard Arabic while real users type in dialect, so the query and the answer barely overlap on the surface. An English-tuned model treats all of this as noise. The result is low recall, and no amount of clever prompting downstream can recover a passage the retriever never returned.

Normalize the text before anything else

Normalization is the cheapest, highest-return step in an Arabic pipeline, and it is the one most teams skip. The goal is to collapse the harmless spelling variants so that a document and a query meet on the same surface form. The rule that matters most: apply the identical normalization function to your documents at index time and to every incoming query, character for character. If the two drift apart, you reintroduce the exact misses you were trying to remove. Keep the raw text too, so you can display the original to the user while searching over the normalized copy.

1. 1 Unify the alef and hamza
   Map أ, إ, آ to bare alef (ا), and normalize the standalone hamza forms (ؤ, ئ) to a consistent base. This alone fixes a large share of near-miss searches.
2. 2 Strip the diacritics
   Remove tashkeel (the fatha, damma, kasra, shadda, sukun and tanwin marks) unless your domain truly depends on them, such as Quranic or poetic text. Dropping them shrinks the vocabulary and merges duplicates.
3. 3 Fold the trailing letters
   Convert ta marbuta (ة) to ha (ه) and alif maqsura (ى) to ya (ي). These endings are typed inconsistently across writers and keyboards.
4. 4 Normalize the digits
   Convert Eastern Arabic numerals (٠١٢٣٤٥٦٧٨٩) to Western ones (0123456789) so a date or amount matches regardless of how it was typed.

Choose an embedding model tested on Arabic

An embedding model turns text into a vector so that similar meanings land close together, and this is the component that most often fails silently on Arabic. Do not pick by a general English score. A 2026 systematic study of Arabic RAG found that strong multilingual, contrastively trained models led the field: BGE-M3 and multilingual-E5-large scored highest on Arabic retrieval, ahead of several Arabic-only alternatives. On the broader MTEB multilingual leaderboard, Qwen3-Embedding sits near the top for non-English retrieval. There are also genuinely Arabic-centric options worth testing, such as the Swan family and GATE-AraBERT, which are trained to be dialect and culture aware. Whatever you shortlist, the vector store underneath matters too. If you have not settled that layer yet, this primer on vector databases covers the trade-offs.

Chunk on sentence boundaries, not fixed windows

How you split a document into passages decides what the retriever can even find. Fixed-size windows are the common default because they are simple, but they cut Arabic sentences mid-clause and strand the answer across two chunks. The same 2026 study compared strategies and found sentence-aware chunking clearly best, scoring 74.78 on average against 69.41 for fixed-size and 66.92 for semantic splitting. Split on real sentence boundaries, keep each chunk to a few sentences, and add a small overlap so a thought that spans a boundary is not lost. Because Arabic is written right to left, make sure your splitter and your storage handle the direction and any mixed Arabic-and-Latin runs cleanly rather than reordering characters.

Add a reranker and handle dialect at query time

Embeddings give you a fast, rough shortlist. A reranker then reads each candidate passage against the actual query and reorders them by true relevance, which is where a lot of Arabic accuracy is won. A cross-encoder such as bge-reranker-v2-m3 works well and is cheap to run over the top twenty or thirty results. For the dialect gap, do not force users to write formal Arabic. Instead, normalize both sides and lean on multilingual embeddings that already map dialectal and Modern Standard Arabic near each other, and if needed, expand the query with a formal paraphrase before searching. Your generation model still has to write a clean Arabic answer from the retrieved passages, so pair this with a model that handles the language well, and pick it by scoring candidates on your own questions rather than on a leaderboard.

Evaluate on your documents, and mind what feeds the index

A leaderboard tells you which model tends to win on average. It does not tell you whether it retrieves your policy clauses. Build a small evaluation set of real questions with the passages you know are the correct answers, then measure two things: recall (did the right passage appear in the top results at all) and precision after reranking (did it land near the top). Run your two or three shortlisted stacks against it, and have a native speaker read the retrieved chunks, because a number can look healthy while the text is subtly wrong. One more thing decides the ceiling: retrieval can only be as good as the text you indexed. If your documents are scanned PDFs, garbled OCR quietly poisons every downstream step, so start from clean extraction. Our note on why Arabic OCR fails covers how to get trustworthy text out of scans before it ever reaches the embedder.