Standard library · Encoding and data

encoding/binary

Writing and reading numbers as bytes, big-endian or little-endian.

import encoding.binary

Binary file formats and network protocols store each number as a fixed number of bytes in a fixed order. This package writes and reads such fields: 8, 16, 32 and 64 bit integers, signed or unsigned, and 32 or 64 bit IEEE 754 floats. Big-endian (BE) puts the most significant byte first and is the usual order on the network and in PNG; little-endian (LE) is the other way round and is what x86 and ARM processors use, and formats such as ZIP and WAV.

Writers append to a ByteBuf and readers advance a ByteReader, both from compress; that package also has BufPushBytes and RdBytes for raw byte runs such as magic numbers. Use this package when the layout is fixed by a specification or by you. For self-describing data use encoding/msgpack or encoding/protobuf; for hex or Base64 text, encoding.

What trips people up

  • Reading past the end does not stop the program. The missing bytes read as 0 and the reader's bad field turns true and stays true. Read a whole record, then check r.bad once, as the first guide does; or check compress.RdLeft before reading.
  • 32-bit unsigned values come back in an int, which is signed, so values from 2,147,483,648 up look negative. ReadU32BE and ReadI32BE return the same bits. Widen with (v as i64) & 4294967295.
  • 64-bit unsigned is an i64 too, so values above 9,223,372,036,854,775,807 are negative.
  • Writers do not range check. WriteU8(b, 300) writes 44, the low 8 bits.
  • BinaryOptions is not used by any function yet; the float functions take a big_endian bool instead, and the integer functions carry the order in their names.

Guide: your own file format

A binary format usually starts with a few magic bytes that identify it, then a version number and a count, then the records. Write each field with the function for its size and byte order, and read them back in exactly the same order. compress.BufPushBytes writes a fixed text such as the magic, and compress.RdBytes reads it back.

encoding-binary-guide-header.salam
import encoding.binary
import compress
import str

func write_file(scores: Vector<int>): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    compress.BufPushBytes(b, "SLM1" as void*, 4)
    binary.WriteU16LE(b, 2)
    binary.WriteU32LE(b, scores.len())
    each s in scores:
        binary.WriteI32LE(b, s)
    end
    ret b
end

func read_file(data: compress.ByteBuf, size: int): str:
    mut r := compress.NewByteReader(data.data, size)
    mut head := compress.RdBytes(r, 4)
    defer compress.BufFree(head)
    if compress.BufToStrUnsafe(head) != "SLM1": ret "not our file" end
    version := binary.ReadU16LE(r)
    if version != 2: ret "unknown version " + str.FromInt(version) end
    count := binary.ReadU32LE(r)
    if count < 0 or count > 100000: ret "implausible count" end
    mut total := 0 as i64
    repeat count:
        total += binary.ReadI32LE(r) as i64
    end
    if r.bad: ret "cut short after " + str.FromInt(r.pos) + " bytes" end
    ret str.FromInt(count) + " scores, total " + str.FromInt(total)
end

func main:
    mut scores := Vector {} as Vector<int>
    scores.push(120)
    scores.push(-15)
    scores.push(98000)
    mut data := write_file(scores)
    println "file is", compress.BufLen(data), "bytes"

    println "whole file:", read_file(data, compress.BufLen(data))
    println "first 18:  ", read_file(data, 18)
    compress.BufSet(data, 4, 9)
    println "edited:    ", read_file(data, compress.BufLen(data))
    compress.BufFree(data)
end
Outputfile is 22 bytes whole file: 3 scores, total 98105 first 18: cut short after 18 bytes edited: unknown version 9

read_file checks the magic and the version before trusting the rest, and refuses a count that is too large to be real, so a damaged file cannot make it loop for a long time. It then reads every record and checks r.bad once at the end: the 18-byte copy has room for only two of the three scores, and the reader says so. Changing byte 4 turns the version into 9, which is refused. Keep one byte order for the whole format and write it down in its description.

Guide: read a header of a real format

PNG is a good example of a big-endian format. A file starts with an 8 byte signature, then chunks, each with a 4 byte length, a 4 letter type and the data. The first chunk, IHDR, holds the width and height as big-endian 32 bit numbers. The program builds those first bytes itself so it runs without a file; with a real image, read the file into a buffer first and the parsing is the same.

encoding-binary-guide-png.salam
import encoding.binary
import compress

func fake_png(): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    sig := [137, 80, 78, 71, 13, 10, 26, 10]
    each x in sig:
        binary.WriteU8(b, x)
    end
    binary.WriteU32BE(b, 13)
    compress.BufPushBytes(b, "IHDR" as void*, 4)
    binary.WriteU32BE(b, 1280)
    binary.WriteU32BE(b, 720)
    binary.WriteU8(b, 8)
    binary.WriteU8(b, 6)
    ret b
end

func main:
    mut png := fake_png()
    mut r := compress.ReaderFromBuf(png)
    if binary.ReadU8(r) != 137 or binary.ReadU8(r) != 80:
        println "not a PNG"
        ret 1
    end
    repeat 6:
        _ := binary.ReadU8(r)
    end
    length := binary.ReadU32BE(r)
    mut kind := compress.RdBytes(r, 4)
    println "first chunk:", compress.BufToStrUnsafe(kind), "of", length, "bytes"
    width := binary.ReadU32BE(r)
    height := binary.ReadU32BE(r)
    println "size:", width, "x", height
    println "bit depth:", binary.ReadU8(r), "color type:", binary.ReadU8(r)
    compress.BufFree(kind)
    compress.BufFree(png)
end
Outputfirst chunk: IHDR of 13 bytes size: 1280 x 720 bit depth: 8 color type: 6

Guide: length-prefixed messages

A TCP connection delivers a stream of bytes, not separate messages. The usual fix is to send each message with its length in front, here as a 16 bit big-endian number. The reader takes the length, checks that the whole message has arrived with RdLeft, and only then reads it.

encoding-binary-guide-frames.salam
import encoding.binary
import compress

func send(b &: compress.ByteBuf, msg: str):
    binary.WriteU16BE(b, msg.len())
    compress.BufPushBytes(b, msg as void*, msg.len())
end

func main:
    mut wire := compress.NewByteBuf()
    send(wire, "HELLO")
    send(wire, "سلام")
    send(wire, "")
    send(wire, "BYE")
    binary.WriteU16BE(wire, 10)
    compress.BufPushBytes(wire, "par" as void*, 3)
    println "on the wire:", compress.BufLen(wire), "bytes"

    mut r := compress.ReaderFromBuf(wire)
    until compress.RdLeft(r) >= 2:
        n := binary.ReadU16BE(r)
        if compress.RdLeft(r) < n:
            println "incomplete frame, wait for more data"
            break
        end
        mut body := compress.RdBytes(r, n)
        println n, "bytes: [" + compress.BufToStrUnsafe(body) + "]"
        compress.BufFree(body)
    end
    compress.BufFree(wire)
end
Outputon the wire: 29 bytes 5 bytes: [HELLO] 8 bytes: [سلام] 0 bytes: [] 3 bytes: [BYE] incomplete frame, wait for more data

The length counts bytes, so سلام is 8, not 4. The last frame says 10 bytes but only 3 have arrived; a real server keeps those bytes and tries again when more data comes in.

Guide: byte order and sign mistakes

Reading with the wrong byte order does not fail, it just gives a wrong number, so these bugs are easy to miss. Unsigned 32 bit values are another trap: they come back in an int, so anything at or above 2,147,483,648 looks negative. Widen it to i64 and mask it to get the unsigned value.

encoding-binary-guide-endian.salam
import encoding.binary
import compress
import str

func bytes_of(b: compress.ByteBuf): str:
    mut out := ""
    repeat compress.BufLen(b) in i:
        if i > 0:
            out = out + " "
        end
        out = out + str.FromInt(compress.BufGet(b, i))
    end
    ret out
end

func main:
    mut be := compress.NewByteBuf()
    binary.WriteU32BE(be, 1000)
    mut le := compress.NewByteBuf()
    binary.WriteU32LE(le, 1000)
    println "1000 big-endian:   ", bytes_of(be)
    println "1000 little-endian:", bytes_of(le)

    mut r := compress.ReaderFromBuf(le)
    println "read with the wrong order:", binary.ReadU32BE(r)

    mut big := compress.NewByteBuf()
    binary.WriteU32BE(big, -1)
    mut r2 := compress.ReaderFromBuf(big)
    v := binary.ReadU32BE(r2)
    println "u32 0xFFFFFFFF as int:", v, "as unsigned:", (v as i64) & 4294967295

    mut r3 := compress.ReaderFromBuf(be)
    _ := binary.ReadU32BE(r3)
    println "bytes left after reading:", compress.RdLeft(r3)

    compress.BufFree(big)
    compress.BufFree(le)
    compress.BufFree(be)
end
Output1000 big-endian: 0 0 3 232 1000 little-endian: 232 3 0 0 read with the wrong order: -402456576 u32 0xFFFFFFFF as int: -1 as unsigned: 4294967295 bytes left after reading: 0

Reading past the end does not stop the program either: the missing bytes read as zero and r.bad turns true. When the input comes from outside, check r.bad after each record, or compress.RdLeft before it, as the framing example does.

Types

BinaryOptions

struct BinaryOptions
Byte order options. No function in the package takes this struct yet.
NameTypeDescription
big_endianbool = falseBig-endian when true; false by default.
encoding-binary-ReadWrite.salam
import encoding.binary
import compress

func main:
    mut b := compress.NewByteBuf()
    binary.WriteU8(b, 255)
    binary.WriteU16BE(b, 4660)
    binary.WriteU16LE(b, 4660)
    binary.WriteU32BE(b, 1)
    binary.WriteI16LE(b, -2)
    println compress.BufToBase64(b)
    binary.WriteU32LE(b, 7)
    binary.WriteU64BE(b, 8)
    binary.WriteU64LE(b, 9)
    binary.WriteI16BE(b, -3)
    binary.WriteI32BE(b, -4)
    binary.WriteI32LE(b, -5)
    binary.WriteI64BE(b, -6)
    binary.WriteI64LE(b, -7)
    binary.WriteF64(b, 1.5, true)
    binary.WriteF32(b, 0.25, false)
    println compress.BufLen(b)
    mut r := compress.ReaderFromBuf(b)
    println binary.ReadU8(r), binary.ReadU16BE(r), binary.ReadU16LE(r), binary.ReadU32BE(r), binary.ReadI16LE(r)
    println binary.ReadU32LE(r), binary.ReadU64BE(r), binary.ReadU64LE(r)
    println binary.ReadI16BE(r), binary.ReadI32BE(r), binary.ReadI32LE(r), binary.ReadI64BE(r), binary.ReadI64LE(r)
    println binary.ReadF64(r, true), binary.ReadF32(r, false)
    o := binary.BinaryOptions {}
    println o.big_endian
    compress.BufFree(b)
end
Output/xI0NBIAAAAB/v8= 69 255 4660 4660 1 -2 7 8 9 -3 -4 -5 -6 -7 1.5 0.25 false

Functions

WriteU8

func WriteU8(b &: compress.ByteBuf, v: int)
Writes one byte: the low 8 bits of v.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU16BE

func WriteU16BE(b &: compress.ByteBuf, v: int)
Writes an unsigned 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU16LE

func WriteU16LE(b &: compress.ByteBuf, v: int)
Writes an unsigned 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU32BE

func WriteU32BE(b &: compress.ByteBuf, v: int)
Writes an unsigned 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU32LE

func WriteU32LE(b &: compress.ByteBuf, v: int)
Writes an unsigned 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU64BE

func WriteU64BE(b &: compress.ByteBuf, v: i64)
Writes an unsigned 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU64LE

func WriteU64LE(b &: compress.ByteBuf, v: i64)
Writes an unsigned 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI16BE

func WriteI16BE(b &: compress.ByteBuf, v: int)
Writes a signed 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI16LE

func WriteI16LE(b &: compress.ByteBuf, v: int)
Writes a signed 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI32BE

func WriteI32BE(b &: compress.ByteBuf, v: int)
Writes a signed 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI32LE

func WriteI32LE(b &: compress.ByteBuf, v: int)
Writes a signed 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI64BE

func WriteI64BE(b &: compress.ByteBuf, v: i64)
Writes a signed 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI64LE

func WriteI64LE(b &: compress.ByteBuf, v: i64)
Writes a signed 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteF64

func WriteF64(b &: compress.ByteBuf, v: f64, big_endian: bool)
Writes a 64 bit IEEE 754 float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteF32

func WriteF32(b &: compress.ByteBuf, v: f64, big_endian: bool)
Writes a 32 bit float; the value loses precision down to 32 bits as it is written.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU8

func ReadU8(r &: compress.ByteReader): int
Reads one byte, 0 to 255; past the end it gives 0 and sets r.bad.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU16BE

func ReadU16BE(r &: compress.ByteReader): int
Reads an unsigned 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU16LE

func ReadU16LE(r &: compress.ByteReader): int
Reads an unsigned 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU32BE

func ReadU32BE(r &: compress.ByteReader): int
Reads a 32 bit integer big-endian, most significant byte first. The result is an int, so values from 231 up come back negative; mask with (v as i64) & 4294967295.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU32LE

func ReadU32LE(r &: compress.ByteReader): int
Reads a 32 bit integer little-endian, least significant byte first. Values from 231 up come back negative, as with ReadU32BE.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU64BE

func ReadU64BE(r &: compress.ByteReader): i64
Reads an unsigned 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU64LE

func ReadU64LE(r &: compress.ByteReader): i64
Reads an unsigned 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI16BE

func ReadI16BE(r &: compress.ByteReader): int
Reads a signed 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI16LE

func ReadI16LE(r &: compress.ByteReader): int
Reads a signed 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI32BE

func ReadI32BE(r &: compress.ByteReader): int
Reads a signed 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI32LE

func ReadI32LE(r &: compress.ByteReader): int
Reads a signed 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI64BE

func ReadI64BE(r &: compress.ByteReader): i64
Reads a signed 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI64LE

func ReadI64LE(r &: compress.ByteReader): i64
Reads a signed 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadF64

func ReadF64(r &: compress.ByteReader, big_endian: bool): f64
Reads a 64 bit IEEE 754 float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadF32

func ReadF32(r &: compress.ByteReader, big_endian: bool): f64
Reads a 32 bit float and returns it as a 64 bit float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam