Standard library · Encoding and data

encoding/protobuf

Reading .proto files, the Protocol Buffers wire encoding, and generating Salam code from them.

import encoding.protobuf

On this page

Types
Constants and variables
Functions

Protocol Buffers is Google's binary format for structured messages. The shape of the messages is defined in a .proto file, and this package reads it, generates Salam code for the messages and services, and also carries the low level tools for writing and reading fields.

In the bytes, every field comes as a tag, the field number and the wire type, followed by its value. Numbers are usually written as a varint: each byte holds 7 bits of the number and its top bit says whether another byte follows.

Types

`PTokenStream`

struct PTokenStream

The tokens of a text.

Name	Type	Description
`toks`	`Vector`	The tokens.

encoding-protobuf-Parse.salam

import encoding.protobuf

func main:
    src := "syntax = \"proto3\";\npackage shop;\n\nenum Status { OPEN = 0; DONE = 1; }\n\nmessage Order {\n  int64 id = 1;\n  repeated string items = 2;\n  map<string, int32> counts = 3;\n  Status status = 4;\n  oneof pay { string card = 5; bool cash = 6; }\n}\n\nservice Orders {\n  rpc Get (Order) returns (Order);\n  rpc Watch (Order) returns (stream Order);\n}\n"
    mut ok := false
    p := protobuf.ParseFile(src, ok)
    println ok, p.packageName, p.messages.len(), p.enums.len(), p.services.len(), p.imports.len()
    msg := p.messages[0]
    println msg.name, msg.fields.len(), msg.oneofs[0].name, msg.oneofs[0].fieldIndices
    each f in msg.fields:
        println f.number, f.name, protobuf.FieldKindName(f.kind), f.typeName, f.repeated, f.oneofIndex
    end
    m := msg.fields[2]
    println m.kind == protobuf.FK_MAP, protobuf.FieldKindName(m.keyKind), protobuf.FieldKindName(m.mapValueKind)
    e := p.enums[0]
    println e.name, e.values[1].name, e.values[1].number
    s := p.services[0]
    println s.name, s.methods[1].name, s.methods[1].inputType, s.methods[1].serverStream, s.methods[1].clientStream
    println protobuf.ScalarKindFromName("sint64") == protobuf.FK_SINT64, protobuf.ScalarKindFromName("Order")
    mut toks := protobuf.Tokenize("message A { int32 x = 1; }", ok)
    println ok, toks.toks.len(), toks.toks[0].lexeme, toks.toks[0].kind == protobuf.PTK_IDENT, toks.toks[2].kind == protobuf.PTK_LBRACE
    println toks.toks[6].ival, toks.toks[6].line
    _ := protobuf.ParseFile("message {", ok)
    println ok
end

Outputtrue shop 1 1 1 0 Order 6 pay [4, 5] 1 id int64 false -1 2 items string true -1 3 counts map false -1 4 status message Status false -1 5 card string false 0 6 cash bool false 0 true string int32 Status DONE 1 Orders Watch Order true false true 0 true 10 message true true 1 1 false

`PField`

struct PField

One message field.

Name	Type	Description
`name`	`str = ""`	The name.
`kind`	`int = 0`	The field kind; one of the `FK_` constants.
`typeName`	`str = ""`	The type name for a message and an enum. The parser does not resolve names, so an enum comes out as a message kind too.
`keyKind`	`int = 0`	The key kind of a `map`.
`mapValueKind`	`int = 0`	The value kind of a `map`.
`mapValueTypeName`	`str = ""`	The type name of the `map` value.
`number`	`int = 0`	The field number.
`repeated`	`bool = false`	The field is repeated.
`packedOpt`	`bool = true`	Repeated numbers are written packed.
`oneofIndex`	`int = -1`	The index of the `oneof` group; -1 when there is none.
`is_deprecated`	`bool = false`	The field is marked deprecated.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`POneof`

struct POneof

One oneof group: only one of its fields has a value.

Name	Type	Description
`name`	`str = ""`	The name.
`fieldIndices`	`Vector`	The indices of the member fields in the message.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PEnumValue`

struct PEnumValue

One enum value.

Name	Type	Description
`name`	`str = ""`	The name.
`number`	`int = 0`	The number.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PEnum`

struct PEnum

One enum.

Name	Type	Description
`name`	`str = ""`	The name.
`values`	`Vector`	The values.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PMessage`

struct PMessage

One message.

Name	Type	Description
`name`	`str = ""`	The name.
`fields`	`Vector`	The fields in the order they were defined.
`oneofs`	`Vector`	The `oneof` groups.
`nestedMessages`	`Vector`	The nested messages.
`nestedEnums`	`Vector`	The nested enums.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PMethod`

struct PMethod

One service method.

Name	Type	Description
`name`	`str = ""`	The name.
`inputType`	`str = ""`	The input type.
`outputType`	`str = ""`	The output type.
`clientStream`	`bool = false`	The input is a stream.
`serverStream`	`bool = false`	The output is a stream.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PService`

struct PService

One gRPC service.

Name	Type	Description
`name`	`str = ""`	The name.
`methods`	`Vector`	The methods.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PFile`

struct PFile

One parsed .proto file.

Name	Type	Description
`packageName`	`str = ""`	The package name.
`messages`	`Vector`	The messages.
`enums`	`Vector`	The top level enums.
`imports`	`Vector`	The imported files.
`services`	`Vector`	The services.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`PToken`

struct PToken

One token.

Name	Type	Description
`kind`	`int`	The token kind.
`lexeme`	`str = ""`	The text of the token.
`ival`	`i64 = 0`	The integer value.
`fval`	`f64 = 0`	The float value.
`sval`	`str = ""`	The string value.
`line`	`int = 0`	The line number.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Constants and variables

Wire types `WT_…`

How a field value is written in the bytes: varint, 8 fixed bytes, length prefixed or 4 fixed bytes.

Name	Value
`WT_VARINT`	`0`
`WT_FIXED64`	`1`
`WT_LEN`	`2`
`WT_FIXED32`	`5`

encoding-protobuf-Wire.salam

import encoding.protobuf
import compress

func main:
    mut b := compress.NewByteBuf()
    protobuf.WriteTag(b, 1, protobuf.WT_VARINT)
    protobuf.WriteVarint(b, 150 as u64)
    println compress.BufToBase64(b), protobuf.MakeTag(1, 0), protobuf.MakeTag(2, protobuf.WT_LEN)
    protobuf.WriteTag(b, 2, protobuf.WT_LEN)
    protobuf.WriteLenPrefixedStr(b, "hello")
    protobuf.WriteTag(b, 3, protobuf.WT_FIXED32)
    protobuf.WriteFixed32(b, 7 as u32)
    protobuf.WriteTag(b, 4, protobuf.WT_FIXED64)
    protobuf.WriteFixed64(b, 9 as u64)
    protobuf.WriteFloat(b, 0.5)
    protobuf.WriteDouble(b, 2.25)
    protobuf.WriteLenPrefixedBytes(b, b.data, 2)
    mut r := compress.ReaderFromBuf(b)
    mut good := true
    mut number := 0
    mut wire := 0
    protobuf.ReadTag(r, number, wire, good)
    println number, wire, protobuf.ReadVarint(r, good)
    protobuf.ReadTag(r, number, wire, good)
    println number, protobuf.ReadLenPrefixedStr(r, good)
    protobuf.ReadTag(r, number, wire, good)
    protobuf.SkipField(r, wire, good)
    protobuf.ReadTag(r, number, wire, good)
    println number, protobuf.ReadFixed64(r, good)
    println protobuf.ReadFloat(r, good), protobuf.ReadDouble(r, good)
    b2 := protobuf.ReadLenPrefixedBuf(r, good)
    println compress.BufLen(b2), good
    _ := protobuf.ReadVarint(r, good)
    println good
    println protobuf.ZigZagEncode32(-1), protobuf.ZigZagDecode32(3 as u32), protobuf.ZigZagEncode64(-2), protobuf.ZigZagDecode64(4 as u64)
    mut b3 := compress.NewByteBuf()
    protobuf.WriteFixed32(b3, 42 as u32)
    mut r3 := compress.ReaderFromBuf(b3)
    good = true
    println protobuf.ReadFixed32(r3, good)
    compress.BufFree(b)
    compress.BufFree(b3)
end

OutputCJYB 8 18 1 0 150 2 hello 4 9 0.5 2.25 2 true false 1 -2 3 2 42

Field kinds `FK_…`

The kind of a field: the plain protobuf types, message, enum and map.

Name	Value
`FK_DOUBLE`	`1`
`FK_FLOAT`	`2`
`FK_INT32`	`3`
`FK_INT64`	`4`
`FK_UINT32`	`5`
`FK_UINT64`	`6`
`FK_SINT32`	`7`
`FK_SINT64`	`8`
`FK_FIXED32`	`9`
`FK_FIXED64`	`10`
`FK_SFIXED32`	`11`
`FK_SFIXED64`	`12`
`FK_BOOL`	`13`
`FK_STRING`	`14`
`FK_BYTES`	`15`
`FK_MESSAGE`	`16`
`FK_ENUM`	`17`
`FK_MAP`	`18`

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Token kinds `PTK_…`

The kind of the tokens from Tokenize.

Name	Value
`PTK_EOF`	`0`
`PTK_IDENT`	`1`
`PTK_INT`	`2`
`PTK_FLOAT`	`3`
`PTK_STRING`	`4`
`PTK_SEMI`	`5`
`PTK_LBRACE`	`6`
`PTK_RBRACE`	`7`
`PTK_LPAREN`	`8`
`PTK_RPAREN`	`9`
`PTK_LBRACKET`	`10`
`PTK_RBRACKET`	`11`
`PTK_LANGLE`	`12`
`PTK_RANGLE`	`13`
`PTK_EQ`	`14`
`PTK_COMMA`	`15`
`PTK_DOT`	`16`
`PTK_MINUS`	`17`
`PTK_ERROR`	`18`

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

Functions

`ParseFile`

func ParseFile(source: str, ok &: bool): PFile

Reads .proto text in the proto3 syntax.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`MakeTag`

func MakeTag(fieldNum: int, wireType: int): u64

A field tag: the field number and the wire type in one number.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteVarint`

func WriteVarint(buf &: compress.ByteBuf, n: u64)

Writes the number with variable length encoding (varint): 7 bits per byte, so small numbers are shorter.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadVarint`

func ReadVarint(r &: compress.ByteReader, ok &: bool): u64

Reads one varint.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteTag`

func WriteTag(buf &: compress.ByteBuf, fieldNum: int, wireType: int)

Writes the tag as a varint.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadTag`

func ReadTag(r &: compress.ByteReader, fieldNum &: int, wireType &: int, ok &: bool)

Reads a tag and splits out the field number and the wire type.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagEncode32`

func ZigZagEncode32(n: i32): u32

Zigzag encoding: it moves small negative numbers to small positive ones so they get a short varint.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagDecode32`

func ZigZagDecode32(n: u32): i32

Undoes 32 bit zigzag.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagEncode64`

func ZigZagEncode64(n: i64): u64

64 bit zigzag encoding.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ZigZagDecode64`

func ZigZagDecode64(n: u64): i64

Undoes 64 bit zigzag.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFixed32`

func WriteFixed32(buf &: compress.ByteBuf, v: u32)

Writes a 32 bit number as 4 little-endian bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFixed32`

func ReadFixed32(r &: compress.ByteReader, ok &: bool): u32

Reads 4 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFixed64`

func WriteFixed64(buf &: compress.ByteBuf, v: u64)

Writes a 64 bit number as 8 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFixed64`

func ReadFixed64(r &: compress.ByteReader, ok &: bool): u64

Reads 8 bytes.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteFloat`

func WriteFloat(buf &: compress.ByteBuf, v: f64)

Writes a 32 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadFloat`

func ReadFloat(r &: compress.ByteReader, ok &: bool): f64

Reads a 32 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteDouble`

func WriteDouble(buf &: compress.ByteBuf, v: f64)

Writes a 64 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadDouble`

func ReadDouble(r &: compress.ByteReader, ok &: bool): f64

Reads a 64 bit float.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteLenPrefixedBytes`

func WriteLenPrefixedBytes(buf &: compress.ByteBuf, p: void*, n: int)

Writes the length and then the bytes; for bytes fields and nested messages.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`WriteLenPrefixedStr`

func WriteLenPrefixedStr(buf &: compress.ByteBuf, s: str)

Writes the length and then the string.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadLenPrefixedBuf`

func ReadLenPrefixedBuf(r &: compress.ByteReader, ok &: bool): compress.ByteBuf

Reads one length prefixed part into a buffer.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`ReadLenPrefixedStr`

func ReadLenPrefixedStr(r &: compress.ByteReader, ok &: bool): str

Reads one length prefixed part into a string.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`SkipField`

func SkipField(r &: compress.ByteReader, wireType: int, ok &: bool)

Steps over the value of a field of the given wire type; for unknown fields.

The example for this item is shown with the ones above: encoding-protobuf-Wire.salam

`Tokenize`

func Tokenize(source: str, ok &: bool): PTokenStream

Breaks the text into tokens.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`GenerateSingle`

func GenerateSingle(protoSource: str, ok &: bool): str

Generates Salam code from .proto text: one struct per message, with its serialising functions.

encoding-protobuf-Generate.salam

import encoding.protobuf
import str

func main:
    src := "syntax = \"proto3\";\nmessage Ping { string text = 1; }\nservice Echo {\n  rpc Say (Ping) returns (Ping);\n  rpc Stream (stream Ping) returns (stream Ping);\n}\n"
    mut ok := false
    code := protobuf.GenerateSingle(src, ok)
    println ok, str.Contains(code, "struct Ping")
    p := protobuf.ParseFile(src, ok)
    println protobuf.ServiceCount(p), protobuf.ServiceNameAt(p, 0), protobuf.ServiceMethodCount(p, 0)
    println protobuf.ServiceMethodSignature(p, 0, 0)
    println protobuf.ServiceMethodSignature(p, 0, 1)
    println str.Contains(protobuf.GenerateServices(p), "Echo")
    all := protobuf.GenerateWithServices(src, ok)
    println ok, str.Len(all) > str.Len(code)
    println protobuf.ServiceSummary(src, ok)
end

Outputtrue true 1 Echo 2 Say(Ping) returns (Ping) Stream(stream Ping) returns (stream Ping) true true true service Echo (2 methods) rpc Say(Ping) returns (Ping) rpc Stream(stream Ping) returns (stream Ping)

`FieldKindName`

func FieldKindName(k: int): str

The name of a field kind, such as int64.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`ScalarKindFromName`

func ScalarKindFromName(name: str): int

The field kind for a plain type name; zero for a message name.

The example for this item is shown with the ones above: encoding-protobuf-Parse.salam

`ServiceCount`

func ServiceCount(f: PFile): int

The number of services.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceNameAt`

func ServiceNameAt(f: PFile, i: int): str

The name of service number i.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceMethodCount`

func ServiceMethodCount(f: PFile, i: int): int

The number of methods of a service.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceMethodSignature`

func ServiceMethodSignature(f: PFile, si: int, mi: int): str

The signature of one method, such as Say(Ping) returns (Ping).

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`GenerateServices`

func GenerateServices(f: PFile): str

Generates the service interface code.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`GenerateWithServices`

func GenerateWithServices(protoSource: str, ok &: bool): str

Generates the message and service code together.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

`ServiceSummary`

func ServiceSummary(protoSource: str, ok &: bool): str

A readable summary of the services and their methods.

The example for this item is shown with the ones above: encoding-protobuf-Generate.salam

encoding/protobuf

Types

PTokenStream

PField

POneof

PEnumValue

PEnum

PMessage

PMethod

PService

PFile

PToken

Constants and variables

Wire types WT_…

Field kinds FK_…

Token kinds PTK_…

Functions

ParseFile

MakeTag

WriteVarint

ReadVarint

WriteTag

ReadTag

ZigZagEncode32

ZigZagDecode32

ZigZagEncode64

ZigZagDecode64

WriteFixed32

ReadFixed32

WriteFixed64

ReadFixed64

WriteFloat

ReadFloat

WriteDouble

ReadDouble

WriteLenPrefixedBytes

WriteLenPrefixedStr

ReadLenPrefixedBuf

ReadLenPrefixedStr

SkipField

Tokenize

GenerateSingle

FieldKindName

ScalarKindFromName

ServiceCount

ServiceNameAt

ServiceMethodCount

ServiceMethodSignature

GenerateServices

GenerateWithServices

ServiceSummary

`PTokenStream`

`PField`

`POneof`

`PEnumValue`

`PEnum`

`PMessage`

`PMethod`

`PService`

`PFile`

`PToken`

Wire types `WT_…`

Field kinds `FK_…`

Token kinds `PTK_…`

`ParseFile`

`MakeTag`

`WriteVarint`

`ReadVarint`

`WriteTag`

`ReadTag`

`ZigZagEncode32`

`ZigZagDecode32`

`ZigZagEncode64`

`ZigZagDecode64`

`WriteFixed32`

`ReadFixed32`

`WriteFixed64`

`ReadFixed64`

`WriteFloat`

`ReadFloat`

`WriteDouble`

`ReadDouble`

`WriteLenPrefixedBytes`

`WriteLenPrefixedStr`

`ReadLenPrefixedBuf`

`ReadLenPrefixedStr`

`SkipField`

`Tokenize`

`GenerateSingle`

`FieldKindName`

`ScalarKindFromName`

`ServiceCount`

`ServiceNameAt`

`ServiceMethodCount`

`ServiceMethodSignature`

`GenerateServices`

`GenerateWithServices`

`ServiceSummary`