Standard library · Encoding and data

encoding/protobuf

Reading .proto files, the Protocol Buffers wire encoding, and generating Salam code from them.

import encoding.protobuf

On this page

Writing a message by hand
Reading a message and skipping unknown fields
Nested and repeated messages
Packed repeated numbers
Negative numbers: int64 or sint64
Generating the code from a .proto file
Which function for which field type
Types
Constants and variables
Functions

Protocol Buffers is Google's binary format for structured messages. The shape of the messages is defined in a .proto file, and this package reads it, generates Salam code for the messages and services, and also carries the low level tools for writing and reading fields.

In the bytes, every field comes as a tag, the field number and the wire type, followed by its value. Numbers are usually written as a varint: each byte holds 7 bits of the number and its top bit says whether another byte follows.

Use Protocol Buffers when you talk to a service that already speaks it, usually over grpc, or when several languages share one schema. Between Salam programs, or when no .proto file exists, encoding/msgpack needs no schema and encoding/json stays readable.

What trips people up

Generated readers do not accept packed repeated numbers. Proto3 writers such as protoc, Go and Java send a repeated int32 as one length-prefixed block by default. The generated Unmarshal reads one tag per element and does not check the wire type, so a packed field fails or gives wrong numbers. The generated Marshal writes the unpacked form, which every reader accepts. Decode packed fields by hand as in the packed guide, or mark the field [packed = false] on the other side.
The generator skips message, enum and map fields. Members of a oneof become ordinary fields, so nothing stops two of them being set. Write those parts by hand.
Varints are u64. ReadVarint returns a u64; cast to i64 or int yourself, and run sint fields through ZigZagDecode64.
Check ok once, at the end. Every reader clears the shared ok flag when the data runs out; nothing stops the program, so test it before you use the result.

The guide below works through the jobs you meet in practice: writing a message by hand, reading one back while skipping fields you do not know, nesting messages, packed repeated numbers, choosing between int64 and sint64, and generating the code from a .proto file instead of writing it. Every byte buffer here is a compress.ByteBuf from compress, so programs import both packages.

Writing a message by hand

A message is nothing more than its fields one after another. For each field call WriteTag with the field number from the .proto file and the wire type, then the writer for the value: WriteVarint for integers, enums and booleans, WriteLenPrefixedStr for strings and WriteLenPrefixedBytes for bytes and nested messages. The program below writes User { id: 42, name: "Sara", admin: true } and prints the bytes in hex.

encoding-protobuf-encode-user.salam

import encoding.protobuf
import compress

func hex(b: compress.ByteBuf): str:
    digits := "0123456789abcdef"
    mut out := ""
    repeat compress.BufLen(b) in i:
        v := compress.BufGet(b, i)
        if i > 0: out = out + " " end
        out = out + digits.substr(v / 16, 1) + digits.substr(v % 16, 1)
    end
    ret out
end

func main:
    mut b := compress.NewByteBuf()
    protobuf.WriteTag(b, 1, protobuf.WT_VARINT)
    protobuf.WriteVarint(b, 42 as u64)
    protobuf.WriteTag(b, 2, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(b, "Sara")
    protobuf.WriteTag(b, 3, protobuf.WT_VARINT)
    protobuf.WriteVarint(b, 1 as u64)
    println "bytes:", compress.BufLen(b)
    println hex(b)
    compress.BufFree(b)
end

Outputbytes: 10 08 2a 12 04 53 61 72 61 18 01

Read the output in pairs: 08 is field 1 with wire type 0, 2a is 42; 12 is field 2 with wire type 2, 04 the length and then the four letters; 18 01 is field 3 set to true. Proto3 leaves out fields that hold their default value (0, an empty string, false), so a sender usually checks the value before it writes the tag.

Reading a message and skipping unknown fields

Decoding is a loop: read a tag, look at the field number, read the value with the reader that matches, and repeat until the reader is empty. A field number you do not know must still be stepped over, which is what SkipField does with the wire type. That is how an old program keeps reading messages from a newer one that added fields. Every reader takes the same ok flag, so one check after the loop catches a message that was cut short.

encoding-protobuf-decode-loop.salam

import encoding.protobuf
import compress

struct User:
    pub id: int = 0
    pub name: str = ""
    pub admin: bool = false
end

func decodeUser(data: void*, n: int, ok &: bool): User:
    mut u := User {}
    mut r := compress.NewByteReader(data, n)
    ok = true
    until compress.RdLeft(r) > 0 and ok:
        mut number := 0
        mut wire := 0
        protobuf.ReadTag(r, number, wire, ok)
        if not ok: ret u end
        switch number:
            1:
                u.id = protobuf.ReadVarint(r, ok) as int
                break
            end
            2:
                u.name = protobuf.ReadLenPrefixedStr(r, ok)
                break
            end
            3:
                u.admin = protobuf.ReadVarint(r, ok) != (0 as u64)
                break
            end
            else:
                println "skipping unknown field", number
                protobuf.SkipField(r, wire, ok)
            end
        end
    end
    ret u
end

func main:
    mut b := compress.NewByteBuf()
    protobuf.WriteTag(b, 1, protobuf.WT_VARINT)
    protobuf.WriteVarint(b, 42 as u64)
    protobuf.WriteTag(b, 9, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(b, "added in v2")
    protobuf.WriteTag(b, 2, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(b, "Sara")
    mut ok := false
    u := decodeUser(b.data, b.len, ok)
    println ok, u.id, u.name, u.admin
    _ := decodeUser(b.data, b.len - 2, ok)
    println "truncated:", ok
    compress.BufFree(b)
end

Outputskipping unknown field 9 true 42 Sara false skipping unknown field 9 truncated: false

The second call reads the same bytes minus the last two: the string length promises four bytes that are not there, ReadLenPrefixedStr clears ok, and the loop stops. Always test ok before you trust the struct you filled.

Nested and repeated messages

A field whose type is another message is written as bytes: encode the inner message into its own buffer, then write it with wire type WT_LEN and WriteLenPrefixedBytes. A repeated field is the same tag written once per element. On the reading side ReadLenPrefixedBuf hands you the inner bytes as a new buffer, which you decode with the same loop and then free.

encoding-protobuf-nested.salam

import encoding.protobuf
import compress

func writeItem(order &: compress.ByteBuf, name: str, qty: int):
    mut item := compress.NewByteBuf()
    protobuf.WriteTag(item, 1, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(item, name)
    protobuf.WriteTag(item, 2, protobuf.WT_VARINT)
    protobuf.WriteVarint(item, qty as u64)
    protobuf.WriteTag(order, 2, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedBytes(order, item.data, item.len)
    compress.BufFree(item)
end

func printItem(b: compress.ByteBuf):
    mut r := compress.ReaderFromBuf(b)
    mut ok := true
    mut name := ""
    mut qty := 0
    until compress.RdLeft(r) > 0 and ok:
        mut number := 0
        mut wire := 0
        protobuf.ReadTag(r, number, wire, ok)
        if number == 1: name = protobuf.ReadLenPrefixedStr(r, ok)
        else number == 2: qty = protobuf.ReadVarint(r, ok) as int
        else: protobuf.SkipField(r, wire, ok) end
    end
    println "  item", name, "x", qty
end

func main:
    mut order := compress.NewByteBuf()
    protobuf.WriteTag(order, 1, protobuf.WT_VARINT)
    protobuf.WriteVarint(order, 1001 as u64)
    writeItem(order, "pen", 3)
    writeItem(order, "notebook", 1)
    println "order bytes:", compress.BufLen(order)

    mut r := compress.ReaderFromBuf(order)
    mut ok := true
    until compress.RdLeft(r) > 0 and ok:
        mut number := 0
        mut wire := 0
        protobuf.ReadTag(r, number, wire, ok)
        if number == 1:
            println "order", protobuf.ReadVarint(r, ok)
        else number == 2:
            mut sub := protobuf.ReadLenPrefixedBuf(r, ok)
            printItem(sub)
            compress.BufFree(sub)
        else:
            protobuf.SkipField(r, wire, ok)
        end
    end
    compress.BufFree(order)
end

Outputorder bytes: 26 order 1001 item pen x 3 item notebook x 1

Packed repeated numbers

A repeated field of numbers can travel in two shapes. Unpacked, each element has its own tag. Packed, the default in proto3, there is one tag with wire type WT_LEN and the varints follow one after another inside it. A correct reader accepts both, so look at the wire type: for WT_LEN, take the block with ReadLenPrefixedBuf and read varints until it is empty.

encoding-protobuf-packed.salam

import encoding.protobuf
import compress

func readScores(b: compress.ByteBuf): Vector<int>:
    mut out := Vector {} as Vector<int>
    mut r := compress.ReaderFromBuf(b)
    mut ok := true
    until compress.RdLeft(r) > 0 and ok:
        mut number := 0
        mut wire := 0
        protobuf.ReadTag(r, number, wire, ok)
        if number == 1 and wire == protobuf.WT_LEN:
            mut packed := protobuf.ReadLenPrefixedBuf(r, ok)
            mut pr := compress.ReaderFromBuf(packed)
            until compress.RdLeft(pr) > 0 and ok:
                out.push(protobuf.ReadVarint(pr, ok) as int)
            end
            compress.BufFree(packed)
        else number == 1 and wire == protobuf.WT_VARINT:
            out.push(protobuf.ReadVarint(r, ok) as int)
        else:
            protobuf.SkipField(r, wire, ok)
        end
    end
    ret out
end

func main:
    scores := [7, 150, 3]

    mut body := compress.NewByteBuf()
    each s in scores:
        protobuf.WriteVarint(body, s as u64)
    end
    mut packed := compress.NewByteBuf()
    protobuf.WriteTag(packed, 1, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedBytes(packed, body.data, body.len)

    mut unpacked := compress.NewByteBuf()
    each s in scores:
        protobuf.WriteTag(unpacked, 1, protobuf.WT_VARINT)
        protobuf.WriteVarint(unpacked, s as u64)
    end

    println "packed:  ", compress.BufLen(packed), "bytes ->", readScores(packed)
    println "unpacked:", compress.BufLen(unpacked), "bytes ->", readScores(unpacked)
    compress.BufFree(unpacked)
    compress.BufFree(packed)
    compress.BufFree(body)
end

Outputpacked: 6 bytes -> [7, 150, 3] unpacked: 7 bytes -> [7, 150, 3]

Packing saves one tag byte per element, which adds up in long lists of small numbers. This is also the code to use in place of the generated Unmarshal for a field that comes from protoc-generated code in another language.

Negative numbers: int64 or sint64

A plain int32 or int64 field writes a negative number as a huge unsigned value, which always takes ten bytes. The sint32 and sint64 types first map the number with ZigZag (0, -1, 1, -2 become 0, 1, 2, 3), so small negative numbers stay small. Use ZigZagEncode64 before WriteVarint and ZigZagDecode64 after ReadVarint, and pick sint64 in the .proto file for fields such as offsets or temperature deltas that are often below zero.

encoding-protobuf-zigzag.salam

import encoding.protobuf
import compress

func varintSize(n: u64): int:
    mut b := compress.NewByteBuf()
    protobuf.WriteVarint(b, n)
    size := compress.BufLen(b)
    compress.BufFree(b)
    ret size
end

func main:
    values := [5, -1, -64, 300, -300]
    each v in values:
        wide := v as i64
        plain := varintSize(wide as u64)
        zig := varintSize(protobuf.ZigZagEncode64(wide))
        println v, "takes", plain, "bytes as int64 and", zig, "as sint64"
    end
    println "zigzag of -1 is", protobuf.ZigZagEncode64(-1 as i64), "and back", protobuf.ZigZagDecode64(1 as u64)
end

Output5 takes 1 bytes as int64 and 1 as sint64 -1 takes 10 bytes as int64 and 1 as sint64 -64 takes 10 bytes as int64 and 1 as sint64 300 takes 2 bytes as int64 and 2 as sint64 -300 takes 10 bytes as int64 and 2 as sint64 zigzag of -1 is 1 and back -1

Generating the code from a .proto file

For more than a couple of messages, let the package write the code. GenerateSingle turns a .proto source into a Salam package with one struct per message and a Marshal and Unmarshal function for each; GenerateWithServices adds gRPC client stubs for the services (see grpc). The program below prints only the declarations of the generated code.

encoding-protobuf-codegen.salam

import encoding.protobuf

func main:
    proto := "syntax = \"proto3\";\nmessage Point {\n  sint32 x = 1;\n  sint32 y = 2;\n  string label = 3;\n}\n"
    mut ok := false
    code := protobuf.GenerateSingle(proto, ok)
    if not ok:
        println "the proto file did not parse"
        ret 1
    end
    lines := code.split("\n")
    each line in lines:
        if line.starts_with("pub ") or line.starts_with("package "):
            println line
        end
    end
    ret 0
end

Outputpackage generated_pb pub struct Point: pub func MarshalPoint(m: Point): compress.ByteBuf: pub func UnmarshalPoint(data: void*, n: int, ok &: bool): Point:

Save the text to a file such as point_pb.salam, add include pb "point_pb.salam" to your program, and call pb.MarshalPoint(p) to get a ByteBuf and pb.UnmarshalPoint(buf.data, buf.len, ok) to read one back. The generator covers scalar fields, strings, bytes and repeated scalars. It leaves out fields whose type is another message or an enum, gives map fields no code and turns oneof members into ordinary fields, so write those few by hand as in the nested example above, or keep the enum as an int32 in the .proto file. Repeated numbers are read only in the unpacked form; see the packed guide.

Which function for which field type

.proto type	Wire type	Write	Read
`int32 int64 uint32 uint64 bool enum`	`WT_VARINT`	`WriteVarint`	`ReadVarint`
`sint32 sint64`	`WT_VARINT`	`ZigZagEncode64`, then `WriteVarint`	`ReadVarint`, then `ZigZagDecode64`
`fixed32 sfixed32`	`WT_FIXED32`	`WriteFixed32`	`ReadFixed32`
`fixed64 sfixed64`	`WT_FIXED64`	`WriteFixed64`	`ReadFixed64`
`float`	`WT_FIXED32`	`WriteFloat`	`ReadFloat`
`double`	`WT_FIXED64`	`WriteDouble`	`ReadDouble`
`string`	`WT_LEN`	`WriteLenPrefixedStr`	`ReadLenPrefixedStr`
`bytes`, nested message	`WT_LEN`	`WriteLenPrefixedBytes`	`ReadLenPrefixedBuf`

Types

`PField`

struct PField

One message field.

Name	Type	Description
`name`	`str = ""`	The name.
`kind`	`int = 0`	The field kind; one of the `FK_` constants.
`typeName`	`str = ""`	The type name for a message and an enum. The parser does not resolve names, so an enum comes out as a message kind too.
`keyKind`	`int = 0`	The key kind of a `map`.
`mapValueKind`	`int = 0`	The value kind of a `map`.
`mapValueTypeName`	`str = ""`	The type name of the `map` value.
`number`	`int = 0`	The field number.
`repeated`	`bool = false`	The field is repeated.
`packedOpt`	`bool = true`	Repeated numbers are written packed.
`oneofIndex`	`int = -1`	The index of the `oneof` group; -1 when there is none.
`is_deprecated`	`bool = false`	The field is marked deprecated.

encoding-protobuf-Parse.salam

import encoding.protobuf

func main:
    src := "syntax = \"proto3\";\npackage shop;\n\nenum Status { OPEN = 0; DONE = 1; }\n\nmessage Order {\n  int64 id = 1;\n  repeated string items = 2;\n  map<string, int32> counts = 3;\n  Status status = 4;\n  oneof pay { string card = 5; bool cash = 6; }\n}\n\nservice Orders {\n  rpc Get (Order) returns (Order);\n  rpc Watch (Order) returns (stream Order);\n}\n"
    mut ok := false
    p := protobuf.ParseFile(src, ok)
    println ok, p.packageName, p.messages.len(), p.enums.len(), p.services.len(), p.imports.len()
    msg := p.messages[0]
    println msg.name, msg.fields.len(), msg.oneofs[0].name, msg.oneofs[0].fieldIndices
    each f in msg.fields:
        println f.number, f.name, protobuf.FieldKindName(f.kind), f.typeName, f.repeated, f.oneofIndex
    end
    m := msg.fields[2]
    println m.kind == protobuf.FK_MAP, protobuf.FieldKindName(m.keyKind), protobuf.FieldKindName(m.mapValueKind)
    e := p.enums[0]
    println e.name, e.values[1].name, e.values[1].number
    s := p.services[0]
    println s.name, s.methods[1].name, s.methods[1].inputType, s.methods[1].serverStream, s.methods[1].clientStream
    println protobuf.ScalarKindFromName("sint64") == protobuf.FK_SINT64, protobuf.ScalarKindFromName("Order")
    mut toks := protobuf.Tokenize("message A { int32 x = 1; }", ok)
    println ok, toks.toks.len(), toks.toks[0].lexeme, toks.toks[0].kind == protobuf.PTK_IDENT, toks.toks[2].kind == protobuf.PTK_LBRACE
    println toks.toks[6].ival, toks.toks[6].line
    _ := protobuf.ParseFile("message {", ok)
    println ok
end

Outputtrue shop 1 1 1 0 Order 6 pay [4, 5] 1 id int64 false -1 2 items string true -1 3 counts map false -1 4 status message Status false -1 5 card string false 0 6 cash bool false 0 true string int32 Status DONE 1 Orders Watch Order true false true 0 true 10 message true true 1 1 false

`POneof`

struct POneof

One oneof group: only one of its fields has a value.

Name	Type	Description
`name`	`str = ""`	The name.
`fieldIndices`	`Vector`	The indices of the member fields in the message.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PEnumValue`

struct PEnumValue

One enum value.

Name	Type	Description
`name`	`str = ""`	The name.
`number`	`int = 0`	The number.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PEnum`

struct PEnum

One enum.

Name	Type	Description
`name`	`str = ""`	The name.
`values`	`Vector`	The values.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PMessage`

struct PMessage

One message.

Name	Type	Description
`name`	`str = ""`	The name.
`fields`	`Vector`	The fields in the order they were defined.
`oneofs`	`Vector`	The `oneof` groups.
`nestedMessages`	`Vector`	The nested messages.
`nestedEnums`	`Vector`	The nested enums.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PMethod`

struct PMethod

One service method.

Name	Type	Description
`name`	`str = ""`	The name.
`inputType`	`str = ""`	The input type.
`outputType`	`str = ""`	The output type.
`clientStream`	`bool = false`	The input is a stream.
`serverStream`	`bool = false`	The output is a stream.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PService`

struct PService

One gRPC service.

Name	Type	Description
`name`	`str = ""`	The name.
`methods`	`Vector`	The methods.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PFile`

struct PFile

One parsed .proto file.

Name	Type	Description
`packageName`	`str = ""`	The package name.
`messages`	`Vector`	The messages.
`enums`	`Vector`	The top level enums.
`imports`	`Vector`	The imported files.
`services`	`Vector`	The services.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PTokenStream`

struct PTokenStream

The tokens of a text.

Name	Type	Description
`toks`	`Vector`	The tokens.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PToken`

struct PToken

One token.

Name	Type	Description
`kind`	`int`	The token kind.
`lexeme`	`str = ""`	The text of the token.
`ival`	`i64 = 0`	The integer value.
`fval`	`f64 = 0`	The float value.
`sval`	`str = ""`	The string value.
`line`	`int = 0`	The line number.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Constants and variables

Field kinds `FK_…`

The kind of a field: the plain protobuf types, message, enum and map.

Name	Value
`FK_DOUBLE`	`1`
`FK_FLOAT`	`2`
`FK_INT32`	`3`
`FK_INT64`	`4`
`FK_UINT32`	`5`
`FK_UINT64`	`6`
`FK_SINT32`	`7`
`FK_SINT64`	`8`
`FK_FIXED32`	`9`
`FK_FIXED64`	`10`
`FK_SFIXED32`	`11`
`FK_SFIXED64`	`12`
`FK_BOOL`	`13`
`FK_STRING`	`14`
`FK_BYTES`	`15`
`FK_MESSAGE`	`16`
`FK_ENUM`	`17`
`FK_MAP`	`18`

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Wire types `WT_…`

How a field value is written in the bytes: varint, 8 fixed bytes, length prefixed or 4 fixed bytes.

Name	Value
`WT_VARINT`	`0`
`WT_FIXED64`	`1`
`WT_LEN`	`2`
`WT_FIXED32`	`5`

encoding-protobuf-Wire.salam

import encoding.protobuf
import compress

func main:
    mut b := compress.NewByteBuf()
    protobuf.WriteTag(b, 1, protobuf.WT_VARINT)
    protobuf.WriteVarint(b, 150 as u64)
    println compress.BufToBase64(b), protobuf.MakeTag(1, 0), protobuf.MakeTag(2, protobuf.WT_LEN)
    protobuf.WriteTag(b, 2, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(b, "hello")
    protobuf.WriteTag(b, 3, protobuf.WT_FIXED32)
    protobuf.WriteFixed32(b, 7 as u32)
    protobuf.WriteTag(b, 4, protobuf.WT_FIXED64)
    protobuf.WriteFixed64(b, 9 as u64)
    protobuf.WriteFloat(b, 0.5)
    protobuf.WriteDouble(b, 2.25)
    protobuf.WriteLenPrefixedBytes(b, b.data, 2)
    mut r := compress.ReaderFromBuf(b)
    mut good := true
    mut number := 0
    mut wire := 0
    protobuf.ReadTag(r, number, wire, good)
    println number, wire, protobuf.ReadVarint(r, good)
    protobuf.ReadTag(r, number, wire, good)
    println number, protobuf.ReadLenPrefixedStr(r, good)
    protobuf.ReadTag(r, number, wire, good)
    protobuf.SkipField(r, wire, good)
    protobuf.ReadTag(r, number, wire, good)
    println number, protobuf.ReadFixed64(r, good)
    println protobuf.ReadFloat(r, good), protobuf.ReadDouble(r, good)
    b2 := protobuf.ReadLenPrefixedBuf(r, good)
    println compress.BufLen(b2), good
    _ := protobuf.ReadVarint(r, good)
    println good
    println protobuf.ZigZagEncode32(-1), protobuf.ZigZagDecode32(3 as u32), protobuf.ZigZagEncode64(-2), protobuf.ZigZagDecode64(4 as u64)
    mut b3 := compress.NewByteBuf()
    protobuf.WriteFixed32(b3, 42 as u32)
    mut r3 := compress.ReaderFromBuf(b3)
    good = true
    println protobuf.ReadFixed32(r3, good)
    compress.BufFree(b)
    compress.BufFree(b3)
end

OutputCJYB 8 18 1 0 150 2 hello 4 9 0.5 2.25 2 true false 1 -2 3 2 42

Token kinds `PTK_…`

The kind of the tokens from Tokenize.

Name	Value
`PTK_EOF`	`0`
`PTK_IDENT`	`1`
`PTK_INT`	`2`
`PTK_FLOAT`	`3`
`PTK_STRING`	`4`
`PTK_SEMI`	`5`
`PTK_LBRACE`	`6`
`PTK_RBRACE`	`7`
`PTK_LPAREN`	`8`
`PTK_RPAREN`	`9`
`PTK_LBRACKET`	`10`
`PTK_RBRACKET`	`11`
`PTK_LANGLE`	`12`
`PTK_RANGLE`	`13`
`PTK_EQ`	`14`
`PTK_COMMA`	`15`
`PTK_DOT`	`16`
`PTK_MINUS`	`17`
`PTK_ERROR`	`18`

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Functions

`FieldKindName`

func FieldKindName(k: int): str

The name of a field kind, such as int64.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`ScalarKindFromName`

func ScalarKindFromName(name: str): int

The field kind for a plain type name; zero for a message name.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`ParseFile`

func ParseFile(source: str, ok &: bool): PFile

Reads .proto text in the proto3 syntax.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`Tokenize`

func Tokenize(source: str, ok &: bool): PTokenStream

Breaks the text into tokens.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`GenerateSingle`

func GenerateSingle(protoSource: str, ok &: bool): str

Generates Salam code from .proto text: one struct per message, with its serialising functions. Message, enum and map fields are left out, and the generated reader expects repeated numbers unpacked.

encoding-protobuf-Generate.salam

import encoding.protobuf
import str

func main:
    src := "syntax = \"proto3\";\nmessage Ping { string text = 1; }\nservice Echo {\n  rpc Say (Ping) returns (Ping);\n  rpc Stream (stream Ping) returns (stream Ping);\n}\n"
    mut ok := false
    code := protobuf.GenerateSingle(src, ok)
    println ok, str.Contains(code, "struct Ping")
    p := protobuf.ParseFile(src, ok)
    println protobuf.ServiceCount(p), protobuf.ServiceNameAt(p, 0), protobuf.ServiceMethodCount(p, 0)
    println protobuf.ServiceMethodSignature(p, 0, 0)
    println protobuf.ServiceMethodSignature(p, 0, 1)
    println str.Contains(protobuf.GenerateServices(p), "Echo")
    all := protobuf.GenerateWithServices(src, ok)
    println ok, str.Len(all) > str.Len(code)
    println protobuf.ServiceSummary(src, ok)
end

Outputtrue true 1 Echo 2 Say(Ping) returns (Ping) Stream(stream Ping) returns (stream Ping) true true true service Echo (2 methods) rpc Say(Ping) returns (Ping) rpc Stream(stream Ping) returns (stream Ping)

`ServiceCount`

func ServiceCount(f: PFile): int

The number of services.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceNameAt`

func ServiceNameAt(f: PFile, i: int): str

The name of service number i.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceMethodCount`

func ServiceMethodCount(f: PFile, i: int): int

The number of methods of a service.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceMethodSignature`

func ServiceMethodSignature(f: PFile, si: int, mi: int): str

The signature of one method, such as Say(Ping) returns (Ping).

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`GenerateServices`

func GenerateServices(f: PFile): str

Generates the service interface code.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`GenerateWithServices`

func GenerateWithServices(protoSource: str, ok &: bool): str

Generates the message and service code together.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceSummary`

func ServiceSummary(protoSource: str, ok &: bool): str

A readable summary of the services and their methods.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`MakeTag`

func MakeTag(fieldNum: int, wireType: int): u64

A field tag: the field number and the wire type in one number.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteVarint`

func WriteVarint(buf &: compress.ByteBuf, n: u64)

Writes the number with variable length encoding (varint): 7 bits per byte, so small numbers are shorter.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadVarint`

func ReadVarint(r &: compress.ByteReader, ok &: bool): u64

Reads one varint as a u64; ok turns false when the bytes run out.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteTag`

func WriteTag(buf &: compress.ByteBuf, fieldNum: int, wireType: int)

Writes the tag as a varint.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadTag`

func ReadTag(r &: compress.ByteReader, fieldNum &: int, wireType &: int, ok &: bool)

Reads a tag and splits out the field number and the wire type.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagEncode32`

func ZigZagEncode32(n: i32): u32

Zigzag encoding: it moves small negative numbers to small positive ones so they get a short varint.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagDecode32`

func ZigZagDecode32(n: u32): i32

Undoes 32 bit zigzag.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagEncode64`

func ZigZagEncode64(n: i64): u64

64 bit zigzag encoding.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagDecode64`

func ZigZagDecode64(n: u64): i64

Undoes 64 bit zigzag.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFixed32`

func WriteFixed32(buf &: compress.ByteBuf, v: u32)

Writes a 32 bit number as 4 little-endian bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFixed32`

func ReadFixed32(r &: compress.ByteReader, ok &: bool): u32

Reads 4 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFixed64`

func WriteFixed64(buf &: compress.ByteBuf, v: u64)

Writes a 64 bit number as 8 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFixed64`

func ReadFixed64(r &: compress.ByteReader, ok &: bool): u64

Reads 8 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFloat`

func WriteFloat(buf &: compress.ByteBuf, v: f64)

Writes a 32 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFloat`

func ReadFloat(r &: compress.ByteReader, ok &: bool): f64

Reads a 32 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteDouble`

func WriteDouble(buf &: compress.ByteBuf, v: f64)

Writes a 64 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadDouble`

func ReadDouble(r &: compress.ByteReader, ok &: bool): f64

Reads a 64 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteLenPrefixedBytes`

func WriteLenPrefixedBytes(buf &: compress.ByteBuf, p: void*, n: int)

Writes the length and then the bytes; for bytes fields and nested messages.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteLenPrefixedStr`

func WriteLenPrefixedStr(buf &: compress.ByteBuf, s: str)

Writes the length and then the string.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadLenPrefixedBuf`

func ReadLenPrefixedBuf(r &: compress.ByteReader, ok &: bool): compress.ByteBuf

Reads one length prefixed part into a buffer.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadLenPrefixedStr`

func ReadLenPrefixedStr(r &: compress.ByteReader, ok &: bool): str

Reads one length prefixed part into a string.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`SkipField`

func SkipField(r &: compress.ByteReader, wireType: int, ok &: bool)

Steps over the value of a field of the given wire type; for unknown fields.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

encoding/protobuf

What trips people up

Writing a message by hand

Reading a message and skipping unknown fields

Nested and repeated messages

Packed repeated numbers

Negative numbers: int64 or sint64

Generating the code from a .proto file

Which function for which field type

Types

PField

POneof

PEnumValue

PEnum

PMessage

PMethod

PService

PFile

PTokenStream

PToken

Constants and variables

Field kinds FK_…

Wire types WT_…

Token kinds PTK_…

Functions

FieldKindName

ScalarKindFromName

ParseFile

Tokenize

GenerateSingle

ServiceCount

ServiceNameAt

ServiceMethodCount

ServiceMethodSignature

GenerateServices

GenerateWithServices

ServiceSummary

MakeTag

WriteVarint

ReadVarint

WriteTag

ReadTag

ZigZagEncode32

ZigZagDecode32

ZigZagEncode64

ZigZagDecode64

WriteFixed32

ReadFixed32

WriteFixed64

ReadFixed64

WriteFloat

ReadFloat

WriteDouble

ReadDouble

WriteLenPrefixedBytes

WriteLenPrefixedStr

ReadLenPrefixedBuf

ReadLenPrefixedStr

SkipField

`PField`

`POneof`

`PEnumValue`

`PEnum`

`PMessage`

`PMethod`

`PService`

`PFile`

`PTokenStream`

`PToken`

Field kinds `FK_…`

Wire types `WT_…`

Token kinds `PTK_…`

`FieldKindName`

`ScalarKindFromName`

`ParseFile`

`Tokenize`

`GenerateSingle`

`ServiceCount`

`ServiceNameAt`

`ServiceMethodCount`

`ServiceMethodSignature`

`GenerateServices`

`GenerateWithServices`

`ServiceSummary`

`MakeTag`

`WriteVarint`

`ReadVarint`

`WriteTag`

`ReadTag`

`ZigZagEncode32`

`ZigZagDecode32`

`ZigZagEncode64`

`ZigZagDecode64`

`WriteFixed32`

`ReadFixed32`

`WriteFixed64`

`ReadFixed64`

`WriteFloat`

`ReadFloat`

`WriteDouble`

`ReadDouble`

`WriteLenPrefixedBytes`

`WriteLenPrefixedStr`

`ReadLenPrefixedBuf`

`ReadLenPrefixedStr`

`SkipField`