Salam
Online editor

Standard library · Text

unicode

What Unicode knows about a character: category, script, writing direction, case, decomposition and composition.

import unicode
On this page
  1. Constants and variables
  2. Functions

Unicode gives every character of every script in the world a number, called a code point: "ش" is 1588 and "A" is 65. For any code point, the unicode package says what kind of character it is: letter or digit, which script it belongs to, whether it is written right to left or left to right, and what its upper and lower case forms are.

Most of the time the str package is enough, and it uses this one underneath. This package is for when you are inspecting text character by character yourself, in a parser, an editor or a text cleaning tool. The data is generated from the official Unicode database, version 16.

To get the code points of a string, use str.CodePoints.

Constants and variables

ZWNJ

const ZWNJ: i32 = 8204
The code point of the zero width non-joiner (U+200C), which in Persian keeps two letters side by side without joining them, as in «می‌روم».
unicode-ZWNJ.salam
import unicode

func main:
    println unicode.ZWNJ, unicode.ZWJ, unicode.MaxCodePoint
    println "unicode", unicode.Version
end
Output8204 8205 1114111 unicode 16.0.0

ZWJ

const ZWJ: i32 = 8205
The code point of the zero width joiner (U+200D), which joins two characters; composed emoji such as the family are built with it.

The example for this item is shown with the ones above: unicode-ZWNJ.salam

MaxCodePoint

const MaxCodePoint: i32 = 1114111
The largest code point Unicode allows, 0x10FFFF.

The example for this item is shown with the ones above: unicode-ZWNJ.salam

Hangul syllable constants Hangul…

The numbers needed to compose and decompose Korean (Hangul) syllables arithmetically. Each syllable is built from a leading consonant, a vowel and sometimes a trailing consonant, and its code point can be computed directly.
NameValue
HangulSBase44032
HangulLBase4352
HangulVBase4449
HangulTBase4519
HangulLCount19
HangulVCount21
HangulTCount28
HangulNCount588
HangulSCount11172
unicode-Hangul.salam
import unicode

func main:
    lead := 18
    vowel := 0
    tail := 4
    syllable := unicode.HangulSBase + (lead * unicode.HangulVCount + vowel) * unicode.HangulTCount + tail
    println syllable
    println unicode.HangulNCount, unicode.HangulSCount
    println unicode.HangulLBase, unicode.HangulVBase, unicode.HangulTBase
    println unicode.HangulLCount
end
Output54620 588 11172 4352 4449 4519 19

Version

const Version: str = "16.0.0"
The version of Unicode the data in this package was generated from.

The example for this item is shown with the ones above: unicode-ZWNJ.salam

MaxFoldLen

const MaxFoldLen: i32 = 3
The largest number of code points a full case fold of one character can produce. A buffer of this size is enough to fold any character.
unicode-ToUpper.salam
import unicode

func main:
    println unicode.ToUpper(945), unicode.ToLower(913), unicode.ToTitle(454), unicode.ToFold(913)
    println unicode.ToUpper(1588)
    c := 223
    println unicode.HasFullFold(c), unicode.FullFoldLen(c)
    repeat unicode.FullFoldLen(c) in i:
        print unicode.FullFoldAt(c, i), ""
    end
    println ""
    println unicode.MaxFoldLen
end
Output913 945 453 945 1588 true 2 115 115 3

General categories Cat…

Every Unicode character has a "general category". Category returns one of these numbers and CategoryName returns its two-letter short name, such as Lo. Categories whose first letter is the same are numbered consecutively.
NameValue
CatLu0
CatLl1
CatLt2
CatLm3
CatLo4
CatMn5
CatMc6
CatMe7
CatNd8
CatNl9
CatNo10
CatPc11
CatPd12
CatPs13
CatPe14
CatPi15
CatPf16
CatPo17
CatSm18
CatSc19
CatSk20
CatSo21
CatZs22
CatZl23
CatZp24
CatCc25
CatCf26
CatCs27
CatCo28
CatCn29
unicode-Category.salam
import unicode

func main:
    each c in [65, 1588, 1776, 1614, 8204, 12288, 888]:
        println c, unicode.CategoryName(c), unicode.Category(c), unicode.IsAssigned(c)
    end
    println unicode.Category(1588) == unicode.CatLo
end
Output65 Lu 0 true 1588 Lo 4 true 1776 Nd 8 true 1614 Mn 5 true 8204 Cf 26 true 12288 Zs 22 true 888 Cn 29 false true

Grapheme break classes Gcb…

The classes GraphemeBreak returns. The str package uses them to find where one "visible letter" ends and the next begins; a letter with its marks, or a composed emoji, is one cluster.
NameValue
GcbOther0
GcbCR1
GcbLF2
GcbControl3
GcbExtend4
GcbZWJ5
GcbRegionalIndicator6
GcbPrepend7
GcbSpacingMark8
GcbL9
GcbV10
GcbT11
GcbLV12
GcbLVT13
GcbExtendedPictographic14
unicode-GraphemeBreak.salam
import unicode

func main:
    println unicode.GraphemeBreak(1614) == unicode.GcbExtend
    println unicode.GraphemeBreak(8205) == unicode.GcbZWJ
    println unicode.GraphemeBreak(128512) == unicode.GcbExtendedPictographic
    println unicode.GraphemeBreak(10), unicode.GraphemeBreak(65)
    println unicode.ConjunctBreak(2325) == unicode.IncbConsonant, unicode.ConjunctBreak(2381) == unicode.IncbLinker
end
Outputtrue true true 2 0 true true

Conjunct break classes Incb…

The classes ConjunctBreak returns. This rule keeps consonants joined by a virama in Indic scripts inside a single cluster.
NameValue
IncbNone0
IncbConsonant1
IncbExtend2
IncbLinker3

The example for this item is shown with the ones above: unicode-GraphemeBreak.salam

Bidirectional classes Bc…

The class BidiClass returns, which the bidirectional algorithm uses to decide where each character of mixed text appears. Arabic script letters are "Arabic", Latin letters are "left to right", and digits have classes of their own.
NameValue
BcL0
BcR1
BcAL2
BcEN3
BcES4
BcET5
BcAN6
BcCS7
BcNSM8
BcBN9
BcB10
BcS11
BcWS12
BcON13
BcLRE14
BcRLE15
BcLRO16
BcRLO17
BcPDF18
BcLRI19
BcRLI20
BcFSI21
BcPDI22
unicode-BidiClass.salam
import unicode

func main:
    println unicode.BidiClass(65) == unicode.BcL
    println unicode.BidiClass(1588) == unicode.BcAL
    println unicode.BidiClass(1488) == unicode.BcR
    println unicode.BidiClass(1776) == unicode.BcEN, unicode.BidiClass(1632) == unicode.BcAN
    println unicode.PairedBracketType(40) == unicode.PbtOpen, unicode.PairedBracketType(41) == unicode.PbtClose
    println unicode.PairedBracket(40), unicode.PairedBracket(65)
end
Outputtrue true true true true true true 41 65

Paired bracket types Pbt…

The result of PairedBracketType: whether the character is an opening bracket, a closing one, or not a bracket at all.
NameValue
PbtNone0
PbtOpen1
PbtClose2

The example for this item is shown with the ones above: unicode-BidiClass.salam

Script identifiers Script…

Every script Unicode knows has an identifier, and Script returns one of them. Persian and Arabic letters are both "Arabic script". "Common" characters such as digits and punctuation are used by several scripts, and "inherited" means the character takes the script of the character before it, as marks do.
NameValue
ScriptUnknown0
ScriptCommon1
ScriptLatin2
ScriptGreek3
ScriptCyrillic4
ScriptArmenian5
ScriptHebrew6
ScriptArabic7
ScriptSyriac8
ScriptThaana9
ScriptDevanagari10
ScriptBengali11
ScriptGurmukhi12
ScriptGujarati13
ScriptOriya14
ScriptTamil15
ScriptTelugu16
ScriptKannada17
ScriptMalayalam18
ScriptSinhala19
ScriptThai20
ScriptLao21
ScriptTibetan22
ScriptMyanmar23
ScriptGeorgian24
ScriptHangul25
ScriptEthiopic26
ScriptCherokee27
ScriptCanadianAboriginal28
ScriptOgham29
ScriptRunic30
ScriptKhmer31
ScriptMongolian32
ScriptHiragana33
ScriptKatakana34
ScriptBopomofo35
ScriptHan36
ScriptYi37
ScriptOldItalic38
ScriptGothic39
ScriptDeseret40
ScriptInherited41
ScriptTagalog42
ScriptHanunoo43
ScriptBuhid44
ScriptTagbanwa45
ScriptLimbu46
ScriptTaiLe47
ScriptLinearB48
ScriptUgaritic49
ScriptShavian50
ScriptOsmanya51
ScriptCypriot52
ScriptBraille53
ScriptBuginese54
ScriptCoptic55
ScriptNewTaiLue56
ScriptGlagolitic57
ScriptTifinagh58
ScriptSylotiNagri59
ScriptOldPersian60
ScriptKharoshthi61
ScriptBalinese62
ScriptCuneiform63
ScriptPhoenician64
ScriptPhagsPa65
ScriptNko66
ScriptSundanese67
ScriptLepcha68
ScriptOlChiki69
ScriptVai70
ScriptSaurashtra71
ScriptKayahLi72
ScriptRejang73
ScriptLycian74
ScriptCarian75
ScriptLydian76
ScriptCham77
ScriptTaiTham78
ScriptTaiViet79
ScriptAvestan80
ScriptEgyptianHieroglyphs81
ScriptSamaritan82
ScriptLisu83
ScriptBamum84
ScriptJavanese85
ScriptMeeteiMayek86
ScriptImperialAramaic87
ScriptOldSouthArabian88
ScriptInscriptionalParthian89
ScriptInscriptionalPahlavi90
ScriptOldTurkic91
ScriptKaithi92
ScriptBatak93
ScriptBrahmi94
ScriptMandaic95
ScriptChakma96
ScriptMeroiticCursive97
ScriptMeroiticHieroglyphs98
ScriptMiao99
ScriptSharada100
ScriptSoraSompeng101
ScriptTakri102
ScriptCaucasianAlbanian103
ScriptBassaVah104
ScriptDuployan105
ScriptElbasan106
ScriptGrantha107
ScriptPahawhHmong108
ScriptKhojki109
ScriptLinearA110
ScriptMahajani111
ScriptManichaean112
ScriptMendeKikakui113
ScriptModi114
ScriptMro115
ScriptOldNorthArabian116
ScriptNabataean117
ScriptPalmyrene118
ScriptPauCinHau119
ScriptOldPermic120
ScriptPsalterPahlavi121
ScriptSiddham122
ScriptKhudawadi123
ScriptTirhuta124
ScriptWarangCiti125
ScriptAhom126
ScriptAnatolianHieroglyphs127
ScriptHatran128
ScriptMultani129
ScriptOldHungarian130
ScriptSignWriting131
ScriptAdlam132
ScriptBhaiksuki133
ScriptMarchen134
ScriptNewa135
ScriptOsage136
ScriptTangut137
ScriptMasaramGondi138
ScriptNushu139
ScriptSoyombo140
ScriptZanabazarSquare141
ScriptDogra142
ScriptGunjalaGondi143
ScriptMakasar144
ScriptMedefaidrin145
ScriptHanifiRohingya146
ScriptSogdian147
ScriptOldSogdian148
ScriptElymaic149
ScriptNandinagari150
ScriptNyiakengPuachueHmong151
ScriptWancho152
ScriptChorasmian153
ScriptDivesAkuru154
ScriptKhitanSmallScript155
ScriptYezidi156
ScriptCyproMinoan157
ScriptOldUyghur158
ScriptTangsa159
ScriptToto160
ScriptVithkuqi161
ScriptKawi162
ScriptNagMundari163
ScriptGaray164
ScriptGurungKhema165
ScriptKiratRai166
ScriptOlOnal167
ScriptSunuwar168
ScriptTodhri169
ScriptTuluTigalari170
unicode-Script.salam
import unicode

func main:
    each c in [65, 1588, 945, 20013, 12354, 1776, 1614]:
        println c, unicode.ScriptName(c)
    end
    println unicode.Script(1588) == unicode.ScriptArabic, unicode.Script(65) == unicode.ScriptLatin
    println unicode.Script(66464) == unicode.ScriptOldPersian, unicode.ScriptName(66464)
end
Output65 Latin 1588 Arabic 945 Greek 20013 Han 12354 Hiragana 1776 Arabic 1614 Inherited true true true Old_Persian

Functions

Category

func Category(cp: int): int
Returns the number of the character's general category, one of the Cat constants above.

The example for this item is shown with the ones above: unicode-Category.salam

CategoryName

func CategoryName(cp: int): str
Returns the two-letter category name, such as Lo for a Persian letter and Nd for a digit.

The example for this item is shown with the ones above: unicode-Category.salam

IsAssigned

func IsAssigned(cp: int): bool
True when this code point has been assigned to a character in Unicode.

The example for this item is shown with the ones above: unicode-Category.salam

IsLetter

func IsLetter(cp: int): bool
True when the character is a letter, that is, in one of the L categories.
unicode-IsLetter.salam
import unicode

func main:
    println unicode.IsLetter(1588), unicode.IsAlphabetic(1614), unicode.IsLetter(1614)
    println unicode.IsUpper(915), unicode.IsLower(947), unicode.IsTitle(453)
    println unicode.IsMark(1614), unicode.IsDigit(1776), unicode.IsNumber(8544), unicode.IsDigit(8544)
    println unicode.IsAlnum(1776), unicode.IsPunct(1548), unicode.IsSymbol(65020)
    println unicode.IsSpace(12288), unicode.IsControl(10), unicode.IsFormat(8204), unicode.IsJoinControl(8204)
end
Outputtrue true false true true true true true true false true true true true true true true

IsAlphabetic

func IsAlphabetic(cp: int): bool
Like IsLetter, but wider: it also counts the marks that are part of reading a letter. Most of the time this is what "is it a letter?" means.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsUpper

func IsUpper(cp: int): bool
True when the character is an upper case letter.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsLower

func IsLower(cp: int): bool
True when the character is a lower case letter.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsTitle

func IsTitle(cp: int): bool
The few "title case" letters that are half upper and half lower case, such as "Dž".

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsMark

func IsMark(cp: int): bool
True when the character is a combining mark, such as an accent or an Arabic vowel sign.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsDigit

func IsDigit(cp: int): bool
Decimal digits only, in any script; Persian digits count.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsNumber

func IsNumber(cp: int): bool
Any numeric character, Roman numerals and fractions included; wider than IsDigit.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsAlnum

func IsAlnum(cp: int): bool
A letter or a digit.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsPunct

func IsPunct(cp: int): bool
Punctuation, such as a comma in any script.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsSymbol

func IsSymbol(cp: int): bool
Symbols, such as currency signs and mathematical operators.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsSpace

func IsSpace(cp: int): bool
Whitespace characters by Unicode's official White_Space property. The zero width non-joiner is not whitespace.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsControl

func IsControl(cp: int): bool
Control characters.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsFormat

func IsFormat(cp: int): bool
Invisible formatting characters, such as the zero width non-joiner and the direction marks.

The example for this item is shown with the ones above: unicode-IsLetter.salam

IsJoinControl

func IsJoinControl(cp: int): bool
Only the zero width non-joiner and the zero width joiner, the two characters that control whether letters join.

The example for this item is shown with the ones above: unicode-IsLetter.salam

ToUpper

func ToUpper(cp: int): int
Returns the upper case form of a character, or the character itself when it has none.

The example for this item is shown with the ones above: unicode-ToUpper.salam

ToLower

func ToLower(cp: int): int
Returns the lower case form of a character.

The example for this item is shown with the ones above: unicode-ToUpper.salam

ToTitle

func ToTitle(cp: int): int
Returns the title case form of a character.

The example for this item is shown with the ones above: unicode-ToUpper.salam

ToFold

func ToFold(cp: int): int
Returns the simple case folded form of a character, the form used for case insensitive comparison.

The example for this item is shown with the ones above: unicode-ToUpper.salam

HasFullFold

func HasFullFold(cp: int): bool
True when the full case fold of the character is more than one code point, as "ß" folds to "ss".

The example for this item is shown with the ones above: unicode-ToUpper.salam

FullFoldLen

func FullFoldLen(cp: int): int
The number of code points in the character's full case fold.

The example for this item is shown with the ones above: unicode-ToUpper.salam

FullFoldAt

func FullFoldAt(cp: int, k: int): int
Code point number k of the character's full case fold.

The example for this item is shown with the ones above: unicode-ToUpper.salam

CombiningClass

func CombiningClass(cp: int): int
Returns the combining class of a mark. During normalisation, the marks following a letter are sorted by this number so that the order they were typed in does not matter. For ordinary characters it is zero.
unicode-Decomp.salam
import unicode

func main:
    alef_madda := 1570
    println unicode.DecompLen(alef_madda), unicode.DecompAt(alef_madda, 0), unicode.DecompAt(alef_madda, 1)
    println unicode.Compose(1575, 1619), unicode.Compose(1575, 65)
    println unicode.CombiningClass(1614), unicode.CombiningClass(1616), unicode.CombiningClass(1575)
    lam_alef := 65271
    println unicode.DecompLen(lam_alef), unicode.CompatDecompLen(lam_alef)
    println unicode.CompatDecompAt(lam_alef, 0), unicode.CompatDecompAt(lam_alef, 1)
end
Output2 1575 1619 1570 -1 30 32 0 1 3 1604 1575

DecompLen

func DecompLen(cp: int): int
The number of code points in the character's canonical decomposition, or zero when it does not decompose.

The example for this item is shown with the ones above: unicode-Decomp.salam

DecompAt

func DecompAt(cp: int, k: int): int
Code point number k of the character's canonical decomposition.

The example for this item is shown with the ones above: unicode-Decomp.salam

CompatDecompLen

func CompatDecompLen(cp: int): int
The number of code points in the character's compatibility decomposition. Arabic presentation forms expand to ordinary letters here.

The example for this item is shown with the ones above: unicode-Decomp.salam

CompatDecompAt

func CompatDecompAt(cp: int, k: int): int
Code point number k of the compatibility decomposition.

The example for this item is shown with the ones above: unicode-Decomp.salam

Compose

func Compose(a: int, b: int): int
Combines two code points into one when they have a canonical composition, and returns minus one otherwise.

The example for this item is shown with the ones above: unicode-Decomp.salam

GraphemeBreak

func GraphemeBreak(cp: int): int
Returns the character's grapheme break class.

The example for this item is shown with the ones above: unicode-GraphemeBreak.salam

ConjunctBreak

func ConjunctBreak(cp: int): int
Returns the character's conjunct break class, used by Indic scripts.

The example for this item is shown with the ones above: unicode-GraphemeBreak.salam

BidiClass

func BidiClass(cp: int): int
Returns the character's bidirectional class.

The example for this item is shown with the ones above: unicode-BidiClass.salam

PairedBracketType

func PairedBracketType(cp: int): int
Says whether the character is an opening bracket, a closing one, or neither.

The example for this item is shown with the ones above: unicode-BidiClass.salam

PairedBracket

func PairedBracket(cp: int): int
Returns the partner of a bracket; the partner of "(" is ")". For a character that is not a bracket it returns the character itself.

The example for this item is shown with the ones above: unicode-BidiClass.salam

Script

func Script(cp: int): int
Returns the identifier of the character's script.

The example for this item is shown with the ones above: unicode-Script.salam

ScriptName

func ScriptName(cp: int): str
Returns the English name of the character's script, such as Arabic.

The example for this item is shown with the ones above: unicode-Script.salam