Salam
Online editor

Standard library · Text

arabic

Normalising Arabic-script text: Arabic yeh and kaf, vowel marks, tatweel, the zero width non-joiner and digits, for search and comparison.

import arabic
On this page
  1. Constants and variables
  2. Functions

Arabic-script text does not arrive in one shape. One source writes «كتاب» with an Arabic kaf and another with a Persian one; one adds vowel marks, another a tatweel (ـــ), and some text, from a PDF for instance, carries the presentation forms of the letters. To a reader they are the same word; to a computer they are different strings, and the search fails.

The arabic package removes those differences. Normalize works from flags you choose, combined with +. For most purposes NormalizePersian or SearchKey is all you need.

Constants and variables

FoldLetters

const FoldLetters: i32 = 1
A flag: converts Arabic letters to their Persian counterparts, "ي" to "ی" and "ك" to "ک".
arabic-Normalize.salam
import arabic

func main:
    text := "كتابٌ عربيّة ـــ ٣٤ ﻻ"
    println arabic.NormalizePersian(text)
    println arabic.Normalize(text, arabic.StripHarakat)
    println arabic.Normalize(text, arabic.FoldLetters + arabic.FoldDigits)
    println arabic.Normalize("می‌خواهم", arabic.StripZWNJ)
    println arabic.Normalize("أحمد إيران", arabic.FoldAlef)
    println arabic.Normalize("مدرسة", arabic.FoldTehMarbuta)
    println arabic.Normalize("ﻻ ﻣﺮﺣﺒﺎ", arabic.FoldPresentation)
    println arabic.Normalize("ســلام", arabic.StripTatweel)
end
Outputکتابٌ عربیّة ٣٤ لا كتاب عربية ـــ ٣٤ ﻻ کتابٌ عربیّة ـــ 34 ﻻ میخواهم احمد ايران مدرسه لا مرحبا سلام

StripHarakat

const StripHarakat: i32 = 2
A flag: removes the vowel marks, fatha, kasra, damma, tanwin, shadda and the rest.

The example for this item is shown with the ones above: arabic-Normalize.salam

StripTatweel

const StripTatweel: i32 = 4
A flag: removes the tatweel (ـ).

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldDigits

const FoldDigits: i32 = 8
A flag: converts Persian and Arabic digits to English ones.

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldAlef

const FoldAlef: i32 = 16
A flag: converts the forms of alef (أ, إ, آ) to plain "ا".

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldTehMarbuta

const FoldTehMarbuta: i32 = 32
A flag: converts "ة" to "ه".

The example for this item is shown with the ones above: arabic-Normalize.salam

StripZWNJ

const StripZWNJ: i32 = 64
A flag: removes the zero width non-joiner, so that «می‌خواهم» and «میخواهم» become one string.

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldPresentation

const FoldPresentation: i32 = 128
A flag: converts the presentation forms of letters, such as «ﻻ» and «ﻣ», back to the ordinary letters.

The example for this item is shown with the ones above: arabic-Normalize.salam

Functions

IsArabicScript

func IsArabicScript(cp: int): bool
True when the character belongs to the Arabic script, which includes the Persian letters.
arabic-IsArabicScript.salam
import arabic

func main:
    println arabic.IsArabicScript(1588), arabic.IsArabicScript(65)
    println arabic.IsHaraka(1614), arabic.IsHaraka(1588)
    println arabic.IsTatweel(1600), arabic.IsTatweel(1588)
end
Outputtrue false true false true false

IsHaraka

func IsHaraka(cp: int): bool
True when the character is one of the vowel marks.

The example for this item is shown with the ones above: arabic-IsArabicScript.salam

IsTatweel

func IsTatweel(cp: int): bool
True when the character is a tatweel (ـ).

The example for this item is shown with the ones above: arabic-IsArabicScript.salam

Normalize

func Normalize(s: str, flags: int): str
Normalises the text with the flags you give. Any aspect whose flag you leave out is left untouched.

The example for this item is shown with the ones above: arabic-Normalize.salam

NormalizePersian

func NormalizePersian(s: str): str
The default normalisation for Persian text: it converts Arabic letters to Persian ones, removes tatweel and expands presentation forms, but keeps the vowel marks, the zero width non-joiner and the digits, so the text still reads as it was written.

The example for this item is shown with the ones above: arabic-Normalize.salam

SearchKey

func SearchKey(s: str): str
Builds a key for searching: it applies every flag, so that two spellings a reader sees as the same produce the same key. Do not display the key; it is only for comparing and indexing.
arabic-SearchKey.salam
import arabic

func main:
    println arabic.SearchKey("كتابٌ"), arabic.SearchKey("کتاب")
    println arabic.EqualLoose("مي‌خواهم", "میخواهم")
    println arabic.EqualLoose("سلام", "کلام")
end
Outputکتاب کتاب true false

EqualLoose

func EqualLoose(a: str, b: str): bool
Compares two strings after building a search key for each, so that «مي‌خواهم» and «میخواهم» count as equal.

The example for this item is shown with the ones above: arabic-SearchKey.salam