encoding/msgpack
The compact binary MessagePack format for dynamic values.
import encoding.msgpack
MessagePack holds the same data as JSON in a shorter binary form that is quicker to read, which suits messages between services, cache entries and RPC. This package turns a Value into a compress.ByteBuf with Marshal and reads one back with Unmarshal (or UnmarshalBytes for raw memory). Errors arrive in an EncodingError.
Pick it over encoding/json when both ends are programs and size or speed matters; keep JSON when people read the data or a browser consumes it. For a fixed binary layout of your own, such as a file header, write the fields directly with encoding/binary; for schemas shared across languages, encoding/protobuf.
How values are written
| Value | MessagePack |
|---|---|
| null, bool | nil, true/false |
| int | the smallest int format that fits, from one byte up to 8 |
| float | always float64 (9 bytes); float32 is read but never written |
| string | fixstr, str8, str16 or str32 by length |
| array, object | fixarray/array16/array32, fixmap/map16/map32 with string keys |
What trips people up
- No
binorexttypes. Data from another library that stores bytes asbin, or timestamps asext, fails withKindCustom. Strings stop at the first zero byte, so binary data cannot ride in a string either; Base64 it first. - Map keys must be strings. A map with integer keys, as some other libraries write, is read with every key turned into
"", so the entries overwrite each other. - Trailing bytes are ignored.
Unmarshalreads one value and does not report what follows it or how many bytes it used, so you cannot read several messages from one buffer. Frame each message yourself, for example with a 4-byte length in front. - Reset the error between calls.
Unmarshalreturns null at once if theEncodingErroryou pass already holds an error. - Integers above
i64(auint64over 9223372036854775807) wrap around to negative numbers.
Guide: how much smaller is it?
MessagePack drops the quotes, colons and commas of JSON, and stores numbers and lengths in as few bytes as it can. For records with short keys and small numbers it usually comes to a half or two thirds of compact JSON. Decode JSON into a Value once, and Marshal writes it as MessagePack.
import encoding.msgpack
import encoding.json
import encoding.value
import encoding.encerr
import compress
func main:
text := `[{"id": 1, "name": "tea", "price": 120, "stock": true},
{"id": 2, "name": "coffee", "price": 90, "stock": true},
{"id": 3, "name": "cocoa", "price": 75, "stock": false}]`
mut e := encerr.NoError()
rows := json.Decode(text, e)
compact := json.EncodeDefault(rows)
packed := msgpack.Marshal(rows)
println "JSON bytes: ", compact.len()
println "MessagePack bytes:", compress.BufLen(packed)
back := msgpack.Unmarshal(packed, e)
println "same data:", value.Equal(rows, back), "error:", encerr.IsError(e)
endThe keys are still written in full in every record, so long key names cost as much as in JSON. If size matters more than that, compress the buffer as well with compress.
Guide: what the bytes look like
Each value starts with a type byte. Small integers from 0 to 127 are a single byte, larger ones get a marker such as cc (one byte follows) or ce (four bytes follow). Short strings, arrays and maps carry their length in the first byte: a2 is a string of 2 bytes, 92 an array of 2 items, 81 a map with one pair.
import encoding.msgpack
import encoding.value
import compress
func hex(v: value.Value): str:
digits := "0123456789abcdef"
mut b := msgpack.Marshal(v)
mut out := ""
repeat compress.BufLen(b) in i:
x := compress.BufGet(b, i)
if i > 0:
out = out + " "
end
out = out + digits.substr(x / 16, 1) + digits.substr(x % 16, 1)
end
compress.BufFree(b)
ret out
end
func main:
println "5 ", hex(value.ValueInt(5))
println "-1 ", hex(value.ValueInt(-1))
println "200 ", hex(value.ValueInt(200))
println "70000 ", hex(value.ValueInt(70000))
println "true ", hex(value.ValueBool(true))
println "null ", hex(value.ValueNull())
println "1.5 ", hex(value.ValueFloat(1.5))
println "\"hi\" ", hex(value.ValueString("hi"))
mut arr := value.NewValueArray()
value.ArrayPush(arr, value.ValueInt(1))
value.ArrayPush(arr, value.ValueInt(2))
println "[1, 2] ", hex(arr)
mut obj := value.NewValueObject()
value.ObjectSet(obj, "a", value.ValueInt(1))
println "{\"a\": 1}", hex(obj)
endThe program reads the buffer byte by byte with compress.BufGet. Turning the buffer into a str first would not work: 70000 contains a zero byte, and a str ends at the first zero. Keep MessagePack data in its ByteBuf, or use BufToBase64 when you need text.
Guide: structs in a cache
A cache such as Redis stores bytes, and MessagePack bytes are quicker to read back than JSON. For a struct, go through JSON on the way in and out: json.Marshal and json.Decode give a Value to pack, and json.Unmarshal fills the struct from the unpacked Value. Here a HashMap stands in for the cache.
import encoding.msgpack
import encoding.json
import encoding.encerr
import compress
struct Session:
pub user: str = ""
pub role: str = ""
pub visits: int = 0
end
func save(cache &: HashMap<str, compress.ByteBuf>, key: str, s: Session):
mut e := encerr.NoError()
v := json.Decode(json.Marshal(s), e)
cache.put(key, msgpack.Marshal(v))
end
func load(cache: HashMap<str, compress.ByteBuf>, key: str, out &: Session): bool:
if not cache.has(key):
ret false
end
mut e := encerr.NoError()
v := msgpack.Unmarshal(cache.get(key), e)
if encerr.IsError(e):
ret false
end
ret json.Unmarshal(json.EncodeDefault(v), out, e)
end
func main:
mut cache := HashMap {} as HashMap<str, compress.ByteBuf>
save(cache, "session:9f2", Session { user = "sara", role = "admin", visits = 12 })
mut s := Session {}
if load(cache, "session:9f2", s):
println "found", s.user, s.role, s.visits
end
println "missing key loads:", load(cache, "session:000", s)
endFor a real cache, send b.data and b.len to the client library as a binary value, and decode what comes back with UnmarshalBytes.
Guide: broken, cut and foreign buffers
Network reads can stop halfway, and other programs may use parts of MessagePack this package does not read. Both Unmarshal and UnmarshalBytes report a buffer that ends in the middle of a value as KindUnexpectedEOF, so you can tell "wait for more bytes" apart from data that is simply wrong.
import encoding.msgpack
import encoding.value
import encoding.encerr
import compress
func main:
mut v := value.NewValueObject()
value.ObjectSet(v, "name", value.ValueString("salam"))
b := msgpack.Marshal(v)
mut e := encerr.NoError()
whole := msgpack.UnmarshalBytes(b.data, b.len, e)
println "whole buffer:", encerr.IsError(e), value.AsString(value.ObjectGet(whole, "name"))
mut cut := encerr.NoError()
_ := msgpack.UnmarshalBytes(b.data, b.len - 3, cut)
println "cut short:", encerr.IsError(cut), cut.kind == encerr.KindUnexpectedEOF, cut.message
mut blob := compress.NewByteBuf()
compress.BufPush(blob, 0xc4)
compress.BufPush(blob, 2)
compress.BufPush(blob, 0xca)
compress.BufPush(blob, 0xfe)
mut bin := encerr.NoError()
_ := msgpack.Unmarshal(blob, bin)
println "bin from another library:", bin.kind == encerr.KindCustom, bin.message
mut two := compress.BufClone(b)
compress.BufPush(two, 0x2a)
mut extra := encerr.NoError()
first := msgpack.Unmarshal(two, extra)
println "extra byte after the value:", encerr.IsError(extra), value.ObjectLen(first), "key read"
endThe third case is a bin value, type byte c4, as written by libraries that keep bytes apart from strings; it fails with KindCustom. Ask the sender to use strings, or Base64 the bytes, when it has to talk to Salam. The last case shows that one stray byte after a complete value is not noticed, which is why a stream of messages needs its own framing.
Types
MsgpackOptions
struct MsgpackOptions| Name | Type | Description |
|---|---|---|
use_str8 | bool = true | Use the str8 format for strings of 32 to 255 bytes; true by default. Set it false for very old readers that predate str8; those strings are then written as str16. |
import encoding.msgpack
import encoding.value
import encoding.encerr
import encoding
import compress
func main:
mut v := value.NewValueObject()
value.ObjectSet(v, "id", value.ValueInt(7))
value.ObjectSet(v, "ok", value.ValueBool(true))
b := msgpack.Marshal(v)
println compress.BufLen(b), compress.BufToBase64(b)
mut err := encerr.NoError()
back := msgpack.Unmarshal(b, err)
println encerr.IsError(err), value.Equal(v, back), value.AsInt(value.ObjectGet(back, "id"))
back2 := msgpack.UnmarshalBytes(b.data, b.len, err)
println value.Equal(v, back2)
o := msgpack.MsgpackOptions {}
println compress.BufLen(msgpack.MarshalWith(v, o))
println encoding.HexEncode(compress.BufToStrUnsafe(msgpack.Marshal(value.ValueInt(5))))
endFunctions
MarshalWith
func MarshalWith(v: value.Value, opts: MsgpackOptions): compress.ByteBufThe example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
Marshal
func Marshal(v: value.Value): compress.ByteBufcompress.BufFree when you are done.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
UnmarshalBytes
func UnmarshalBytes(p: void*, n: int, err &: encerr.EncodingError): value.Valuen bytes at p, for data that is not in a ByteBuf, such as a reply from a cache client. Bytes after the value are ignored.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
Unmarshal
func Unmarshal(b: compress.ByteBuf, err &: encerr.EncodingError): value.Valueerr is filled in when the data is cut short or uses bin or ext. Pass a fresh NoError() each time.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam