Standard library · Encoding and data

encoding/json

Building and reading JSON: from a plain string through automatic struct conversion to streaming.

import encoding.json

There are three ways to work with JSON here. For simple jobs, functions such as Member, Object and Get work directly on JSON text. For real programs, Marshal and Unmarshal convert a Salam struct to JSON and back. For data whose shape you do not know in advance, Decode returns a dynamic Value.

Set the key name of a field with @json "name" above it; @json "-" leaves the field out. Without it, the field's own name becomes the key.

Use this package for APIs, config files and anything another program will read. Its neighbours: encoding/yaml and encoding/toml for files people edit, encoding/msgpack when the data should be small and binary, and encoding/serde to read or write the same struct in several formats.

The things people get wrong:

  • GetInt returns an int, 32 bits, and clamps anything larger: 3000000000 comes back as 2147483647. For ids and timestamps, read the field with Unmarshal into an i64, or parse the text from Get.
  • Get, GetInt and Has look only at the top level of an object. For nested values use GetPath, such as GetPath(text, "user.address.city").
  • Strict or lenient. Unmarshal fails on a missing field, an unknown key or a type that does not match; UnmarshalLenient keeps the struct's defaults for missing fields and skips unknown keys, but a wrong type still fails.
  • Key order. Marshal writes fields in struct order. A decoded Value object keeps the order of the text and Encode writes its keys in that order; pass sort_keys to Encode when output must be alphabetical.

Guide: load a config file into a struct

A config file is the most common JSON a program reads. Describe it as structs, give every field the default you want, and let UnmarshalLenient fill in what the file has. Nested objects map to nested structs and arrays map to Vector fields.

encoding-json-guide-config.salam
import encoding.json
import encoding.encerr

struct Database:
    pub host: str = "localhost"
    pub port: int = 5432
    pub pool: int = 4
end

struct Config:
    pub name: str = "app"
    pub debug: bool = false
    pub db: Database = Database {}
    pub admins: Vector<str> = Vector {}
end

func main:
    text := `{
  "name": "shop",
  "db": {"host": "db.internal", "port": 6432},
  "admins": ["sara", "reza"]
}`
    mut cfg := Config {}
    mut e := encerr.NoError()
    if not json.UnmarshalLenient(text, cfg, e):
        println "bad config:", e.message
        ret 1
    end
    println "name:", cfg.name
    println "debug:", cfg.debug
    println "database:", cfg.db.host, cfg.db.port, "pool", cfg.db.pool
    println "admins:", cfg.admins.len(), cfg.admins.get(0), cfg.admins.get(1)
end
Outputname: shop debug: false database: db.internal 6432 pool 4 admins: 2 sara reza

The file left out debug and db.pool, so they keep their defaults. That is the reason for the lenient form here: plain Unmarshal is strict and fails with missing key 'pool' when a field is absent, and with unknown key when the file has a key the struct does not. Strict is right for messages between programs, where a missing field is a bug; lenient is right for files people edit by hand. To read a file from disk, pass the text of file.ReadAll from fs/file.

Guide: build an API response

Going the other way, Marshal turns a struct into compact JSON for the wire and MarshalIndent into readable JSON for logs and files. A Vector of structs becomes an array of objects, quotes inside strings are escaped, and the field order is the order of the struct.

encoding-json-guide-response.salam
import encoding.json

struct Item:
    pub sku: str = ""
    pub title: str = ""
    @json "price_cents"
    pub price: int = 0
    pub in_stock: bool = true
end

struct Page:
    pub page: int = 1
    pub total: int = 0
    pub items: Vector<Item> = Vector {}
    @json "-"
    pub cache_key: str = ""
end

func main:
    mut p := Page { page = 2, total = 14, cache_key = "products:2" }
    p.items.push(Item { sku = "BK-01", title = "Notebook", price = 1250 })
    p.items.push(Item { sku = "PN-07", title = "Pen \"fine\"", price = 300, in_stock = false })

    println json.Marshal(p)
    println json.MarshalIndent(p, "  ")
end
Output{"page":2,"total":14,"items":[{"sku":"BK-01","title":"Notebook","price_cents":1250,"in_stock":true},{"sku":"PN-07","title":"Pen \"fine\"","price_cents":300,"in_stock":false}]} { "page": 2, "total": 14, "items": [ { "sku": "BK-01", "title": "Notebook", "price_cents": 1250, "in_stock": true }, { "sku": "PN-07", "title": "Pen \"fine\"", "price_cents": 300, "in_stock": false } ] }

@json "price_cents" renames a field in the JSON without renaming it in the code, and @json "-" keeps internal fields such as cache_key out of the output. Use it for passwords and tokens too, so they can never leak into a response by accident.

Guide: explore JSON of unknown shape

When you do not know the shape ahead of time, for example a webhook from another service, Decode gives a tree of Value nodes. Ask each node what it is with IsString, IsInt, IsObject and the rest, then read it with the matching As function.

encoding-json-guide-dynamic.salam
import encoding.json
import encoding.value
import encoding.encerr
import str

func describe(v: value.Value): str:
    if value.IsString(v): ret "string " + value.AsString(v) end
    if value.IsInt(v): ret "int " + str.FromInt(value.AsInt(v)) end
    if value.IsFloat(v): ret "float " + str.FromFloat(value.AsFloat(v)) end
    if value.IsBool(v):
        if value.AsBool(v): ret "bool true" end
        ret "bool false"
    end
    if value.IsArray(v): ret "array of " + str.FromInt(value.ArrayLen(v)) end
    if value.IsObject(v): ret "object with " + str.FromInt(value.ObjectLen(v)) + " keys" end
    ret "null"
end

func main:
    mut e := encerr.NoError()
    doc := json.Decode(`{"id": 7, "name": "Lamp", "price": 19.5, "sale": true, "tags": ["home", "light"], "size": {"w": 20, "h": 45}, "note": null}`, e)
    if encerr.IsError(e):
        println "parse error:", e.message
        ret 1
    end
    println "keys:", value.ObjectLen(doc)
    keys := ["id", "name", "price", "sale", "tags", "size", "note", "color"]
    each k in keys:
        if not value.ObjectHas(doc, k):
            println k, "is missing"
            continue
        end
        println k, "=", describe(value.ObjectGet(doc, k))
    end
    size := value.ObjectGet(doc, "size")
    println "area:", value.AsInt(value.ObjectGet(size, "w")) * value.AsInt(value.ObjectGet(size, "h"))
end
Outputkeys: 7 id = int 7 name = string Lamp price = float 19.5 sale = bool true tags = array of 2 size = object with 2 keys note = null color is missing area: 900

Whole numbers come back as int and numbers with a fraction as float; IsNumber accepts both. A decoded object keeps its keys in the order of the text, but a sender is free to change that order, so walk a list of the keys you expect, as this program does, and use ObjectHas for the optional ones.

Guide: report bad input

Every decoding function takes an EncodingError from encoding/encerr. When it fails, message says what was wrong and position is the byte offset where the reader stopped, which is what you show the user. The text helpers like Get never fail; they return an empty string for a missing key, so check with Has or Valid first when that matters.

encoding-json-guide-errors.salam
import encoding.json
import encoding.encerr

struct Signup:
    pub email: str = ""
    pub age: int = 0
end

func load(text: str):
    mut s := Signup {}
    mut e := encerr.NoError()
    if json.Unmarshal(text, s, e):
        println "ok:", s.email, s.age
    else:
        println "error at", e.position, "->", e.message
    end
end

func load_lenient(text: str):
    mut s := Signup {}
    mut e := encerr.NoError()
    if json.UnmarshalLenient(text, s, e):
        println "lenient ok:", s.email, s.age
    end
end

func main:
    load(`{"email": "a@example.com", "age": 31}`)
    load(`{"email": "a@example.com", "age": "31"}`)
    load(`{"email": "a@example.com", "age": 31`)
    load(`{"email": "a@example.com", "age": 31, "plan": "pro"}`)
    load_lenient(`{"email": "a@example.com", "age": 31, "plan": "pro"}`)

    println "valid:", json.Valid(`[1, 2, 3]`), json.Valid(`[1, 2,]`)
    println "missing key gives [" + json.Get(`{"a": 1}`, "b") + "]"
end
Outputok: a@example.com 31 error at 34 -> value does not match the expected type error at 36 -> expected ',' or '}' error at 46 -> unknown key lenient ok: a@example.com 31 valid: true false missing key gives []

Guide: stream newline-delimited JSON

Logs and exports often hold one JSON value per line (NDJSON). A decoder reads them one at a time: More says whether another value is left, and Decode returns the next one. NewBufDecoder reads from a string and NewDecoder from an open file. Encoders write the same format, one value per line.

encoding-json-guide-ndjson.salam
import encoding.json
import encoding.value

func main:
    logs := `{"level": "info", "msg": "started", "ms": 3}
{"level": "warn", "msg": "slow query", "ms": 820}
{"level": "info", "msg": "request", "ms": 41}
{"level": "error", "msg": "timeout", "ms": 5000}
{"level": "info", "msg": "request", "ms": 37}
`
    mut counts := HashMap {} as HashMap<str, int>
    mut slow := 0
    mut d := json.NewBufDecoder(logs)
    until d.More():
        line := d.Decode()
        level := value.AsString(value.ObjectGet(line, "level"))
        counts.put(level, counts.get_or(level, 0) + 1)
        if value.AsInt(value.ObjectGet(line, "ms")) > 500:
            slow += 1
        end
    end
    println "info:", counts.get_or("info", 0), "warn:", counts.get_or("warn", 0), "error:", counts.get_or("error", 0)
    println "slower than 500 ms:", slow

    mut out := json.NewBufEncoder()
    out.Encode(value.ValueString("summary"))
    out.Encode(value.ValueInt(slow))
    println out.String()
    out.Close()
end
Outputinfo: 3 warn: 1 error: 1 slower than 500 ms: 2 "summary" 2

until d.More(): reads naturally as "while there is more", because until in Salam runs while its condition holds.

Which function for which job

JobUse
Read one field from a small JSON textGet, GetInt, GetPath
Build a small JSON text by handMember, MemberInt, Object, Array
Struct to JSONMarshal, MarshalIndent
JSON to struct, exactUnmarshal
JSON to struct, with defaults and extra keysUnmarshalLenient
Unknown shapeDecode, Encode
One value per lineNewDecoder, NewEncoder
Pretty print or shrink existing textIndent, Minify
A JSON Schema for a structSchema

Types

JSONOptions

struct JSONOptions
The options for Encode.
NameTypeDescription
indentstr = ""The indent unit; empty means one line.
sort_keysbool = falseWrite the keys in alphabetical order. Without it they are written in the order the object holds them: the order they were added or decoded.
ignore_nullbool = falseLeave out the keys whose value is null.
encoding-json-Value.salam
import encoding.json
import encoding.value
import encoding.encerr
import fs.file

func main:
    mut e := encerr.NoError()
    v := json.Decode(`{"city": "Tehran", "zones": [1, 2, 22]}`, e)
    println encerr.IsError(e), value.AsString(value.ObjectGet(v, "city"))
    println json.EncodeDefault(v)
    opts := json.JSONOptions { indent = "  ", sort_keys = true }
    println json.Encode(v, opts)
    _ := json.Decode("[1, 2", e)
    println encerr.IsError(e), e.position
    mut enc := json.NewBufEncoder()
    enc.Encode(value.ValueInt(1))
    enc.Encode(value.ValueString("two"))
    println enc.String()
    enc.Close()
    mut pretty := json.NewBufEncoderIndent("  ")
    pretty.Encode(v)
    println pretty.String()
    pretty.Close()
    out := file.Open("lines.ndjson", "wb")
    mut lines := json.NewEncoder(out)
    lines.Encode(value.ValueInt(10))
    lines.Encode(value.ValueInt(20))
    lines.Close()
    mut tabbed := json.NewEncoderIndent(file.Open("pretty.json", "wb"), "\t")
    tabbed.Encode(v)
    tabbed.Close()
    mut reader := json.NewDecoder(file.Open("lines.ndjson", "rb"))
    mut total := 0 as i64
    until reader.More():
        total += value.AsInt(reader.Decode())
    end
    println total
    mut two := json.NewBufDecoder("{\"a\": 1}\n{\"a\": 2}\n")
    until two.More():
        println json.EncodeDefault(two.Decode())
    end
end
Outputfalse Tehran {"city": "Tehran", "zones": [1, 2, 22]} { "city": "Tehran", "zones": [ 1, 2, 22 ] } true 5 1 "two" { "city": "Tehran", "zones": [ 1, 2, 22 ] } 30 {"a": 1} {"a": 2}

JSONEncoder

struct JSONEncoder
A streaming writer: it writes several JSON values one after another, each on its own line.
NameTypeDescription
dstFile*The destination file, when it writes to one.
useFilebool = falseWhether it writes to a file or into memory.
bufstr.StringBuilderThe text buffer, when it writes into memory.
indentstr = ""The indent unit; empty means one value per line.
countint = 0How many values have been written.

The example for this item is shown with the ones above: encoding-json-Value.salam

Encode

func Encode(v: value.Value): bool
Writes one value and a newline after it.

The example for this item is shown with the ones above: encoding-json-Value.salam

String

func String(): str
The text written into memory.

The example for this item is shown with the ones above: encoding-json-Value.salam

Close

func Close()
Closes the writer.

The example for this item is shown with the ones above: encoding-json-Value.salam

JSONDecoder

struct JSONDecoder
A streaming reader: it reads consecutive JSON values one at a time, as in a JSON Lines file.
NameTypeDescription
srcFile*The source file.
useFilebool = falseWhether it reads from a file.
bufstr = ""The text that was read.
posint = 0The current position in the text.
loadedbool = falseWhether the file has been read.
eofbool = falseIt has reached the end of the text.
badbool = falseAn error happened while reading.

The example for this item is shown with the ones above: encoding-json-Value.salam

More

func More(): bool
True when another value is left.

The example for this item is shown with the ones above: encoding-json-Value.salam

Decode

func Decode(): value.Value
Reads the next value.

The example for this item is shown with the ones above: encoding-json-Value.salam

Functions

Decode

func Decode(text: str, err &: encerr.EncodingError): value.Value
Turns JSON into a dynamic Value; on an error err is filled in.

The example for this item is shown with the ones above: encoding-json-Value.salam

Encode

func Encode(v: value.Value, opts: JSONOptions): str
Turns a Value into JSON with the given options.

The example for this item is shown with the ones above: encoding-json-Value.salam

EncodeDefault

func EncodeDefault(v: value.Value): str
Turns a Value into JSON with the default options.

The example for this item is shown with the ones above: encoding-json-Value.salam

SchemaEnvelope

func SchemaEnvelope<T>(v: T): str
Builds the JSON Schema of a value's type, along with the definitions of the nested types, inside an envelope to be combined later with SchemaMerge.
encoding-json-Marshal.salam
import encoding.json
import encoding.encerr

struct User:
    @json "name"
    pub name: str = ""
    @json "age"
    pub age: int = 0
    @json "tags"
    pub tags: Vector<str> = Vector {}
    @json "-"
    pub password: str = ""
end

func main:
    mut u := User { name = "Maryam", age = 28, password = "secret" }
    u.tags.push("admin")
    println json.Marshal(u)
    println json.MarshalIndent(u, "  ")
    mut loaded := User {}
    mut e := encerr.NoError()
    println json.Unmarshal(`{"name": "Reza", "age": 41, "tags": ["a", "b"]}`, loaded, e)
    println loaded.name, loaded.age, loaded.tags.len()
    println json.Unmarshal(`{"name": "Reza", "age": "forty"}`, loaded, e), e.message
    mut lenient := User {}
    println json.UnmarshalLenient(`{"name": "Sara", "extra": 1}`, lenient, e), lenient.name
    println json.Schema(u)
    envelope := json.SchemaEnvelope(u)
    println json.SchemaMerge(envelope) == json.Schema(u)
end
Output{"name":"Maryam","age":28,"tags":["admin"]} { "name": "Maryam", "age": 28, "tags": [ "admin" ] } true Reza 41 2 false value does not match the expected type true Sara {"$schema":"https://json-schema.org/draft/2020-12/schema","$defs":{"User":{"title":"User","type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer","format":"int32"},"tags":{"type":"array","items":{"type":"string"}}},"required":["name","age","tags"],"additionalProperties":false}},"$ref":"#/$defs/User"} true

Schema

func Schema<T>(v: T): str
The complete JSON Schema of a value's type, for documenting an API or validating at the other end.

The example for this item is shown with the ones above: encoding-json-Marshal.salam

SchemaMerge

func SchemaMerge(envelope: str): str
Merges several schema envelopes into one document with shared definitions.

The example for this item is shown with the ones above: encoding-json-Marshal.salam

NewEncoder

func NewEncoder(f: File*): JSONEncoder
A writer that writes into an open file.

The example for this item is shown with the ones above: encoding-json-Value.salam

NewEncoderIndent

func NewEncoderIndent(f: File*, unit: str): JSONEncoder
A file writer with indenting.

The example for this item is shown with the ones above: encoding-json-Value.salam

NewBufEncoder

func NewBufEncoder(): JSONEncoder
A writer that writes into memory; get the text with String.

The example for this item is shown with the ones above: encoding-json-Value.salam

NewBufEncoderIndent

func NewBufEncoderIndent(unit: str): JSONEncoder
A memory writer with indenting.

The example for this item is shown with the ones above: encoding-json-Value.salam

NewDecoder

func NewDecoder(f: File*): JSONDecoder
A reader that reads from an open file.

The example for this item is shown with the ones above: encoding-json-Value.salam

NewBufDecoder

func NewBufDecoder(s: str): JSONDecoder
A reader that reads from a string.

The example for this item is shown with the ones above: encoding-json-Value.salam

Valid

func Valid(text: str): bool
True when the text is valid JSON.
encoding-json-Build.salam
import encoding.json

func main:
    mut tags := Vector {} as Vector<str>
    tags.push(json.Str("hello"))
    tags.push(json.Str(`a "quoted" line`))
    mut members := Vector {} as Vector<str>
    members.push(json.Member("name", "Ali"))
    members.push(json.MemberInt("age", 30))
    members.push(json.MemberBool("admin", false))
    members.push(json.MemberFloat("score", 18.5))
    members.push(json.MemberRaw("tags", json.Array(tags)))
    text := json.Object(members)
    println text
    println json.Valid(text), json.Valid("{broken")
    println json.Get(text, "name"), json.GetInt(text, "age"), json.GetFloat(text, "score")
    println json.Has(text, "admin"), json.Has(text, "email"), json.Keys(text)
    println json.Escape("one\ttwo")
    pretty := json.Indent(text)
    println pretty
    println json.Minify(pretty) == json.Minify(text)
    println json.Indent4(`{"a":1}`)
    println json.IndentTabs(`{"a":1}`) == "{\n\t\"a\": 1\n}", json.IndentWith(`[1]`, "--")
    nested := `{"user":{"tags":["a","b"],"name":"Ali"}}`
    println json.GetPath(nested, "user.name"), json.GetPath(nested, "user.tags.1"), json.Items(json.GetPath(nested, "user.tags")).len()
    println json.Unquote(`"\u0633\u0644\u0627\u0645"`)
end
Output{"name": "Ali", "age": 30, "admin": false, "score": 18.5, "tags": ["hello", "a \"quoted\" line"]} true false Ali 30 18.5 true false ["name", "age", "admin", "score", "tags"] one\ttwo { "name": "Ali", "age": 30, "admin": false, "score": 18.5, "tags": [ "hello", "a \"quoted\" line" ] } true { "a": 1 } true [ --1 ] Ali b 2 سلام

Get

func Get(text: str, key: str): str
The value of a top-level key as text; strings come back without their quotes.

The example for this item is shown with the ones above: encoding-json-Build.salam

Escape

func Escape(s: str): str
Escapes the special characters of a string for JSON, without the surrounding quotes.

The example for this item is shown with the ones above: encoding-json-Build.salam

Minify

func Minify(text: str): str
Removes all the unnecessary whitespace.

The example for this item is shown with the ones above: encoding-json-Build.salam

Member

func Member(key: str, value: str): str
One key and string pair, such as "name": "Ali", for building an object.

The example for this item is shown with the ones above: encoding-json-Build.salam

GetInt

func GetInt(text: str, key: str): int
The value of a top-level key as an int. Values beyond 32 bits are clamped; use Unmarshal into an i64 for large numbers.

The example for this item is shown with the ones above: encoding-json-Build.salam

GetFloat

func GetFloat(text: str, key: str): f64
The decimal value of a top-level key.

The example for this item is shown with the ones above: encoding-json-Build.salam

Has

func Has(text: str, key: str): bool
True when the top-level key is there.

The example for this item is shown with the ones above: encoding-json-Build.salam

MemberInt

func MemberInt(key: str, value: i64): str
A key and whole number pair.

The example for this item is shown with the ones above: encoding-json-Build.salam

MemberBool

func MemberBool(key: str, value: bool): str
A key and boolean pair.

The example for this item is shown with the ones above: encoding-json-Build.salam

MemberFloat

func MemberFloat(key: str, value: f64): str
A key and decimal number pair.

The example for this item is shown with the ones above: encoding-json-Build.salam

MemberRaw

func MemberRaw(key: str, raw: str): str
A key paired with ready-made JSON, such as another array or object.

The example for this item is shown with the ones above: encoding-json-Build.salam

Str

func Str(value: str): str
Turns a string into a JSON value, with quotes and escaping.

The example for this item is shown with the ones above: encoding-json-Build.salam

Object

func Object(members: Vector<str>): str
Puts the pairs you built together into an object, {...}.

The example for this item is shown with the ones above: encoding-json-Build.salam

Array

func Array(items: Vector<str>): str
Puts values together into an array, [...].

The example for this item is shown with the ones above: encoding-json-Build.salam

IndentWith

func IndentWith(text: str, unit: str): str
Makes JSON readable with an indent unit of your choice.

The example for this item is shown with the ones above: encoding-json-Build.salam

IndentTabs

func IndentTabs(text: str): str
Makes JSON readable with tabs.

The example for this item is shown with the ones above: encoding-json-Build.salam

Indent

func Indent(text: str): str
Makes JSON readable with two spaces.

The example for this item is shown with the ones above: encoding-json-Build.salam

Indent4

func Indent4(text: str): str
Makes JSON readable with four spaces.

The example for this item is shown with the ones above: encoding-json-Build.salam

Keys

func Keys(text: str): Vector<str>
The keys of the top-level object, in the order of the text.

The example for this item is shown with the ones above: encoding-json-Build.salam

Items

func Items(text: str): Vector<str>
The raw text of each element of a top-level array, ready for Get or Items again.

The example for this item is shown with the ones above: encoding-json-Build.salam

Unquote

func Unquote(raw: str): str
Decodes a raw JSON string token, escapes included; anything else comes back unchanged.

The example for this item is shown with the ones above: encoding-json-Build.salam

GetPath

func GetPath(text: str, path: str): str
Follows a dotted path through objects and arrays, such as hits.hits.0._source; strings come back decoded, anything else as raw JSON, and a missing step gives an empty string.

The example for this item is shown with the ones above: encoding-json-Build.salam

Marshal

func Marshal<T>(v: T): str
Turns any Salam value, from a number or a string to a struct, a vector or a map, into compact JSON.

The example for this item is shown with the ones above: encoding-json-Marshal.salam

MarshalIndent

func MarshalIndent<T>(v: T, unit: str): str
Like Marshal, indented.

The example for this item is shown with the ones above: encoding-json-Marshal.salam

Unmarshal

func Unmarshal<T>(text: str, out &: T, err &: encerr.EncodingError): bool
Fills the struct out from JSON. Every field has to be present with a matching type and an unknown key is an error too; on failure err is filled in and it returns false.

The example for this item is shown with the ones above: encoding-json-Marshal.salam

UnmarshalLenient

func UnmarshalLenient<T>(text: str, out &: T, err &: encerr.EncodingError): bool
Like Unmarshal, but unknown keys are skipped and missing fields keep the value they already had. A value of the wrong type is still an error.

The example for this item is shown with the ones above: encoding-json-Marshal.salam