encoding/binary
Writing and reading numbers as bytes, big-endian or little-endian.
import encoding.binary
Binary file formats and network protocols store each number as a fixed number of bytes in a fixed order. This package writes and reads such fields: 8, 16, 32 and 64 bit integers, signed or unsigned, and 32 or 64 bit IEEE 754 floats. Big-endian (BE) puts the most significant byte first and is the usual order on the network and in PNG; little-endian (LE) is the other way round and is what x86 and ARM processors use, and formats such as ZIP and WAV.
Writers append to a ByteBuf and readers advance a ByteReader, both from compress; that package also has BufPushBytes and RdBytes for raw byte runs such as magic numbers. Use this package when the layout is fixed by a specification or by you. For self-describing data use encoding/msgpack or encoding/protobuf; for hex or Base64 text, encoding.
What trips people up
- Reading past the end does not stop the program. The missing bytes read as 0 and the reader's
badfield turns true and stays true. Read a whole record, then checkr.badonce, as the first guide does; or checkcompress.RdLeftbefore reading. - 32-bit unsigned values come back in an
int, which is signed, so values from 2,147,483,648 up look negative.ReadU32BEandReadI32BEreturn the same bits. Widen with(v as i64) & 4294967295. - 64-bit unsigned is an
i64too, so values above 9,223,372,036,854,775,807 are negative. - Writers do not range check.
WriteU8(b, 300)writes 44, the low 8 bits. BinaryOptionsis not used by any function yet; the float functions take abig_endianbool instead, and the integer functions carry the order in their names.
Guide: your own file format
A binary format usually starts with a few magic bytes that identify it, then a version number and a count, then the records. Write each field with the function for its size and byte order, and read them back in exactly the same order. compress.BufPushBytes writes a fixed text such as the magic, and compress.RdBytes reads it back.
import encoding.binary
import compress
import str
func write_file(scores: Vector<int>): compress.ByteBuf:
mut b := compress.NewByteBuf()
compress.BufPushBytes(b, "SLM1" as void*, 4)
binary.WriteU16LE(b, 2)
binary.WriteU32LE(b, scores.len())
each s in scores:
binary.WriteI32LE(b, s)
end
ret b
end
func read_file(data: compress.ByteBuf, size: int): str:
mut r := compress.NewByteReader(data.data, size)
mut head := compress.RdBytes(r, 4)
defer compress.BufFree(head)
if compress.BufToStrUnsafe(head) != "SLM1": ret "not our file" end
version := binary.ReadU16LE(r)
if version != 2: ret "unknown version " + str.FromInt(version) end
count := binary.ReadU32LE(r)
if count < 0 or count > 100000: ret "implausible count" end
mut total := 0 as i64
repeat count:
total += binary.ReadI32LE(r) as i64
end
if r.bad: ret "cut short after " + str.FromInt(r.pos) + " bytes" end
ret str.FromInt(count) + " scores, total " + str.FromInt(total)
end
func main:
mut scores := Vector {} as Vector<int>
scores.push(120)
scores.push(-15)
scores.push(98000)
mut data := write_file(scores)
println "file is", compress.BufLen(data), "bytes"
println "whole file:", read_file(data, compress.BufLen(data))
println "first 18: ", read_file(data, 18)
compress.BufSet(data, 4, 9)
println "edited: ", read_file(data, compress.BufLen(data))
compress.BufFree(data)
endread_file checks the magic and the version before trusting the rest, and refuses a count that is too large to be real, so a damaged file cannot make it loop for a long time. It then reads every record and checks r.bad once at the end: the 18-byte copy has room for only two of the three scores, and the reader says so. Changing byte 4 turns the version into 9, which is refused. Keep one byte order for the whole format and write it down in its description.
Guide: read a header of a real format
PNG is a good example of a big-endian format. A file starts with an 8 byte signature, then chunks, each with a 4 byte length, a 4 letter type and the data. The first chunk, IHDR, holds the width and height as big-endian 32 bit numbers. The program builds those first bytes itself so it runs without a file; with a real image, read the file into a buffer first and the parsing is the same.
import encoding.binary
import compress
func fake_png(): compress.ByteBuf:
mut b := compress.NewByteBuf()
sig := [137, 80, 78, 71, 13, 10, 26, 10]
each x in sig:
binary.WriteU8(b, x)
end
binary.WriteU32BE(b, 13)
compress.BufPushBytes(b, "IHDR" as void*, 4)
binary.WriteU32BE(b, 1280)
binary.WriteU32BE(b, 720)
binary.WriteU8(b, 8)
binary.WriteU8(b, 6)
ret b
end
func main:
mut png := fake_png()
mut r := compress.ReaderFromBuf(png)
if binary.ReadU8(r) != 137 or binary.ReadU8(r) != 80:
println "not a PNG"
ret 1
end
repeat 6:
_ := binary.ReadU8(r)
end
length := binary.ReadU32BE(r)
mut kind := compress.RdBytes(r, 4)
println "first chunk:", compress.BufToStrUnsafe(kind), "of", length, "bytes"
width := binary.ReadU32BE(r)
height := binary.ReadU32BE(r)
println "size:", width, "x", height
println "bit depth:", binary.ReadU8(r), "color type:", binary.ReadU8(r)
compress.BufFree(kind)
compress.BufFree(png)
endGuide: length-prefixed messages
A TCP connection delivers a stream of bytes, not separate messages. The usual fix is to send each message with its length in front, here as a 16 bit big-endian number. The reader takes the length, checks that the whole message has arrived with RdLeft, and only then reads it.
import encoding.binary
import compress
func send(b &: compress.ByteBuf, msg: str):
binary.WriteU16BE(b, msg.len())
compress.BufPushBytes(b, msg as void*, msg.len())
end
func main:
mut wire := compress.NewByteBuf()
send(wire, "HELLO")
send(wire, "سلام")
send(wire, "")
send(wire, "BYE")
binary.WriteU16BE(wire, 10)
compress.BufPushBytes(wire, "par" as void*, 3)
println "on the wire:", compress.BufLen(wire), "bytes"
mut r := compress.ReaderFromBuf(wire)
until compress.RdLeft(r) >= 2:
n := binary.ReadU16BE(r)
if compress.RdLeft(r) < n:
println "incomplete frame, wait for more data"
break
end
mut body := compress.RdBytes(r, n)
println n, "bytes: [" + compress.BufToStrUnsafe(body) + "]"
compress.BufFree(body)
end
compress.BufFree(wire)
endThe length counts bytes, so سلام is 8, not 4. The last frame says 10 bytes but only 3 have arrived; a real server keeps those bytes and tries again when more data comes in.
Guide: byte order and sign mistakes
Reading with the wrong byte order does not fail, it just gives a wrong number, so these bugs are easy to miss. Unsigned 32 bit values are another trap: they come back in an int, so anything at or above 2,147,483,648 looks negative. Widen it to i64 and mask it to get the unsigned value.
import encoding.binary
import compress
import str
func bytes_of(b: compress.ByteBuf): str:
mut out := ""
repeat compress.BufLen(b) in i:
if i > 0:
out = out + " "
end
out = out + str.FromInt(compress.BufGet(b, i))
end
ret out
end
func main:
mut be := compress.NewByteBuf()
binary.WriteU32BE(be, 1000)
mut le := compress.NewByteBuf()
binary.WriteU32LE(le, 1000)
println "1000 big-endian: ", bytes_of(be)
println "1000 little-endian:", bytes_of(le)
mut r := compress.ReaderFromBuf(le)
println "read with the wrong order:", binary.ReadU32BE(r)
mut big := compress.NewByteBuf()
binary.WriteU32BE(big, -1)
mut r2 := compress.ReaderFromBuf(big)
v := binary.ReadU32BE(r2)
println "u32 0xFFFFFFFF as int:", v, "as unsigned:", (v as i64) & 4294967295
mut r3 := compress.ReaderFromBuf(be)
_ := binary.ReadU32BE(r3)
println "bytes left after reading:", compress.RdLeft(r3)
compress.BufFree(big)
compress.BufFree(le)
compress.BufFree(be)
endReading past the end does not stop the program either: the missing bytes read as zero and r.bad turns true. When the input comes from outside, check r.bad after each record, or compress.RdLeft before it, as the framing example does.
Types
BinaryOptions
struct BinaryOptions| Name | Type | Description |
|---|---|---|
big_endian | bool = false | Big-endian when true; false by default. |
import encoding.binary
import compress
func main:
mut b := compress.NewByteBuf()
binary.WriteU8(b, 255)
binary.WriteU16BE(b, 4660)
binary.WriteU16LE(b, 4660)
binary.WriteU32BE(b, 1)
binary.WriteI16LE(b, -2)
println compress.BufToBase64(b)
binary.WriteU32LE(b, 7)
binary.WriteU64BE(b, 8)
binary.WriteU64LE(b, 9)
binary.WriteI16BE(b, -3)
binary.WriteI32BE(b, -4)
binary.WriteI32LE(b, -5)
binary.WriteI64BE(b, -6)
binary.WriteI64LE(b, -7)
binary.WriteF64(b, 1.5, true)
binary.WriteF32(b, 0.25, false)
println compress.BufLen(b)
mut r := compress.ReaderFromBuf(b)
println binary.ReadU8(r), binary.ReadU16BE(r), binary.ReadU16LE(r), binary.ReadU32BE(r), binary.ReadI16LE(r)
println binary.ReadU32LE(r), binary.ReadU64BE(r), binary.ReadU64LE(r)
println binary.ReadI16BE(r), binary.ReadI32BE(r), binary.ReadI32LE(r), binary.ReadI64BE(r), binary.ReadI64LE(r)
println binary.ReadF64(r, true), binary.ReadF32(r, false)
o := binary.BinaryOptions {}
println o.big_endian
compress.BufFree(b)
endFunctions
WriteU8
func WriteU8(b &: compress.ByteBuf, v: int)v.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU16BE
func WriteU16BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU16LE
func WriteU16LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU32BE
func WriteU32BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU32LE
func WriteU32LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU64BE
func WriteU64BE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU64LE
func WriteU64LE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI16BE
func WriteI16BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI16LE
func WriteI16LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI32BE
func WriteI32BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI32LE
func WriteI32LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI64BE
func WriteI64BE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI64LE
func WriteI64LE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteF64
func WriteF64(b &: compress.ByteBuf, v: f64, big_endian: bool)big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteF32
func WriteF32(b &: compress.ByteBuf, v: f64, big_endian: bool)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU8
func ReadU8(r &: compress.ByteReader): intr.bad.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU16BE
func ReadU16BE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU16LE
func ReadU16LE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU32BE
func ReadU32BE(r &: compress.ByteReader): intint, so values from 231 up come back negative; mask with (v as i64) & 4294967295.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU32LE
func ReadU32LE(r &: compress.ByteReader): intReadU32BE.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU64BE
func ReadU64BE(r &: compress.ByteReader): i64The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU64LE
func ReadU64LE(r &: compress.ByteReader): i64The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI16BE
func ReadI16BE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI16LE
func ReadI16LE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI32BE
func ReadI32BE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI32LE
func ReadI32LE(r &: compress.ByteReader): intThe example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI64BE
func ReadI64BE(r &: compress.ByteReader): i64The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI64LE
func ReadI64LE(r &: compress.ByteReader): i64The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadF64
func ReadF64(r &: compress.ByteReader, big_endian: bool): f64big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadF32
func ReadF32(r &: compress.ByteReader, big_endian: bool): f64big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam