Standard library · Network

net.censor

Helpers for networks that interfere with traffic: protocol sniffing and TLS server name reading, ClientHello fragmentation, padding, domain rule sets with Iran lists, a probe guard and a replay cache.

import net.censor

On this page

Fragmenting a TLS hello
Domain rules and the Iran lists
Hiding a server from probes
What to know
Types
Constants and variables
Functions

Some networks do not just carry traffic. They look at it: the first bytes of a connection tell a filter which protocol it is, and the server name in the first TLS message tells it which site you are going to. This package is a set of small helpers, each independent of the others, for working with that. Sniff and ParseClientHello read what a filter would read. FragmentClientHello and SNISplitSizes cut that first message into pieces, so that a filter that only looks at one packet never sees the whole name. DomainSet and Classifier decide, from rules, which hosts go direct, through a proxy or nowhere. ProbeGuard, ReplayCache and DecoyResponse are the pieces of a server that does not give itself away to someone who is trying it. To see how the pieces fit together in an app, read Building a v2ray style or VPN app.

It does not open any connection and has no state of its own. It works on bytes and text you give it, so it fits in front of any of the proxy packages: net.tunnel, net.socks, net.httpproxy, net.trojan. None of them calls it for you; you wire it in where you need it.

Core ideas:

Sniff needs only the first few bytes and says KindTLS, KindHTTP, KindSSH, KindSOCKS5 or KindSOCKS4, or KindTooShort when it cannot tell yet.
ParseClientHello reads a TLS ClientHello without a TLS library: server name, offered protocols, whether Encrypted Client Hello is present.
Fragmenting is a plan plus a cut. SplitSizes or SNISplitSizes make the plan (sizes that add up to the total), Slice cuts bytes into plain pieces and FragmentRecords cuts one TLS record into several records. FragmentClientHello does the whole job for a ClientHello.
Padding hides sizes: WrapRecord and WrapStream add random padding inside records that look like TLS application data, and UnwrapRecord takes it off.
A DomainSet holds domain rules (example.com, full:name, keyword:word); a Classifier has three of them and a default and answers DecisionProxy, DecisionDirect or DecisionBlock.

Things that catch people out:

Fragmentation is not encryption and not magic. It helps against filters that look at the first packet or the first record only. Filters change, and many networks reassemble. Test it on the network you actually use, and do not rely on it alone.
Padding is not encryption either. WrapStream hides how large the pieces are, not what is in them. Use it inside TLS or another encrypted channel, never in place of one. The padded records use TLS looking headers of this package's own making: they will not pass for real TLS to a careful observer.
Fragment before TLS goes out, at the start only. The cut goes on the first record of a handshake. A connection already running is not affected.
The Iran lists are starting points. IranDirectList and FilteredList are short, human edited lists that age. Treat them as examples to edit, and keep your own list in a file for AddRules.
ProbeGuard and ReplayCache have no lock. They are plain data. When several threads use one, wrap the calls in your own lock. They do not stop a probe by themselves: you call them from your server's accept path.
ShouldFallback is a single comparison. A server that has a real website to show is what does the hiding; see the Trojan package.
Randomness is secure. RandomInt and RandomBytes draw from the system's secure source. SplitRandom is the one exception: with a seed it is repeatable on purpose, which makes it fine for tests and a poor choice for hiding a pattern.

Fragmenting a TLS hello

A client sends its first TLS message as one record. SNISplitSizes finds the server name in it and plans a cut in the middle of the name; Slice cuts the bytes, and the pieces are sent one after another, each in its own packet. A server reads them as one stream and sees the same message. The example sends a ClientHello in two pieces over loopback and shows what the receiving side reads.

net-censor-guide-fragment.salam

import net.censor
import net.tcp
import sync

const HELLO := "160301008c010000880303000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f206465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f80818283000613011302c02b0100003900000014001200000f7777772e6578616d706c652e636f6d000a00040002001d0010000e000c02683208687474702f312e31002b0003020304"

mut port := 0
mut seen := ""
mut ready := sync.NewWaitGroup()

func server():
    srv := tcp.Bind(0)
    port = tcp.Port(srv)
    sync.Done(ready)
    c := tcp.Accept(srv)
    tcp.SetReadDeadline(c, 1500)
    until true:
        data := tcp.Read(c, 4096)
        if data.len() == 0:
            break
        end
        seen += data.len() + " "
    end
    tcp.Close(c)
end

func main:
    sync.Add(ready, 1 as i64)
    t := spawn(server)
    sync.Wait(ready)

    hello := censor.FromHex(HELLO)
    sizes := censor.SNISplitSizes(hello.data, hello.len, false)
    mut pieces := censor.Slice(hello.data, hello.len, sizes)
    println "server name starts at byte", censor.ParseClientHello(hello.data, hello.len).sni_offset, "and is cut after", sizes.get(0), "bytes"

    conn := tcp.Dial("127.0.0.1", port)
    tcp.SetNoDelay(conn, true)
    repeat pieces.len() in i:
        piece := pieces.get(i)
        tcp.WriteBytes(conn, piece.data, piece.len)
        sync.SleepMs(60)
    end
    tcp.Close(conn)
    join(t)
    println "the server read:", seen
    println "pieces sent:", pieces.len()
    censor.FreePieces(pieces)
end

Outputserver name starts at byte 97 and is cut after 104 bytes the server read: 3 8 pieces sent: 2

That cuts the bytes of the stream, which any TCP server reassembles. FragmentClientHello does something different: it cuts the record into several TLS records, so even a filter that reassembles packets sees the server name split between records. A server that follows TLS accepts both. Send the pieces with a short pause or with the TCP no-delay option on, so that each leaves in its own packet.

Domain rules and the Iran lists

A DomainSet reads rules from text, one per line: example.com matches that domain and everything under it, full:name only that name, keyword:word any host containing the word. A Classifier holds a block set, a proxy set and a direct set and answers in a fixed order: block, then proxy, then direct (the direct set or any private address), then its default. So a rule in the proxy set wins over the direct set, and a private address is direct unless you say otherwise.

mut classifier := censor.NewClassifier(censor.DecisionProxy)
classifier.direct = censor.IranDirectSet()
println censor.Classify(classifier, "www.digikala.com")   // direct
println censor.Classify(classifier, "mail.sharif.ir")     // direct
println censor.Classify(classifier, "youtube.com")        // proxy (the default)

IranDirectList holds .ir and a few large domestic sites; FilteredList holds sites commonly filtered. Both are short and will age: copy them into a file, edit them, and load your own with AddRules. For a router that works on tags, net.tunnel has tunnel.IranRouter and tunnel.ParseRules.

Hiding a server from probes

A server that is being probed sees connections that do not authenticate. The safe answer is to look like an ordinary web server: hand the connection to a real site (ShouldFallback says yes for anything not authenticated), or at least send a plain 404 (DecoyResponse). A ProbeGuard adds a memory: after max_fails failures from one address within window_ms, GuardAllowed says no for ban_ms. A ReplayCache remembers the first bytes of authentications, so that a recorded handshake sent again is noticed (ReplaySeen). A small random wait (DelayMs) before answering a failed client keeps the timing of the failure path from standing out.

if not censor.GuardAllowed(guard, client_ip, now):
    close(connection)
else if not authenticated:
    censor.GuardRecordFail(guard, client_ip, now)
    hand_to_real_website(connection)
end

The net.trojan server already hands failed clients to a real website. These helpers are the extra layer for servers you build yourself.

What to know

Every function works on bytes and text you pass in. Nothing here opens a socket or keeps global state.
Fragmenting and padding are partial measures, not encryption; use them inside an encrypted channel and test them on your own network.
The Iran lists are short and out of date quickly. Keep your own.
ProbeGuard and ReplayCache are plain data without a lock and must be freed (FreeProbeGuard, FreeReplayCache); so must the sets, pieces and buffers you create (FreeDomainSet, FreePieces).
Addresses are IPv4 text or IPv6 text; the helpers do not resolve names.
No logging happens by itself.

Types

`DomainSet`

struct DomainSet

A set of domain rules: whole domains (with all their subdomains), exact names and keywords. Make it with NewDomainSet and fill it with AddRule or AddRules.

Name	Type	Description
`suffixes`	`HashMap`	Domains that match themselves and everything under them.
`fulls`	`HashMap`	Names that must match exactly.
`keywords`	`Vector`	Words that match any host containing them.
`count`	`int = 0`	How many rules were added.

net-censor-Domains.salam

import net.censor

func main:
    mut set := censor.NewDomainSet()
    println censor.AddRule(set, "example.com"), censor.AddRule(set, "full:only.test.org"), censor.AddRule(set, "keyword:torrent")
    println censor.AddRule(set, "domain:*.cdn.net"), censor.AddRule(set, "regexp:a.*"), censor.AddRule(set, "  ")
    println censor.AddRules(set, "# a comment\nblocked.io # trailing\n\nfoo.bar\n"), set.count
    println censor.MatchDomain(set, "example.com"), censor.MatchDomain(set, "WWW.Example.COM."), censor.MatchDomain(set, "badexample.com")
    println censor.MatchDomain(set, "only.test.org"), censor.MatchDomain(set, "x.only.test.org"), censor.MatchDomain(set, "my-torrent-site.xyz")
    println censor.MatchDomain(set, "a.b.cdn.net"), censor.MatchDomain(set, "x.blocked.io"), censor.MatchDomain(set, "net")
    println set.suffixes.size(), set.fulls.size(), set.keywords.len()
    censor.FreeDomainSet(set)

    println censor.IsPrivateHost("192.168.1.5"), censor.IsPrivateHost("172.20.0.1"), censor.IsPrivateHost("172.40.0.1"), censor.IsPrivateHost("localhost"), censor.IsPrivateHost("::1"), censor.IsPrivateHost("example.com")

    mut classifier := censor.NewClassifier(censor.DecisionProxy)
    classifier.direct = censor.IranDirectSet()
    classifier.block = censor.NewDomainSet()
    censor.AddRule(classifier.block, "ads.example.org")
    each host in ["www.digikala.com", "mail.sharif.ir", "youtube.com", "ads.example.org", "192.168.1.5", "172.40.0.1"]:
        println host, censor.Classify(classifier, host)
    end
    println censor.DecisionProxy, censor.DecisionDirect, censor.DecisionBlock

    filtered := censor.FilteredSet()
    println censor.MatchDomain(filtered, "m.youtube.com"), censor.MatchDomain(filtered, "digikala.com")
    println censor.IranDirectList().starts_with("#"), censor.FilteredList().includes("youtube.com")
end

Outputtrue true true true false false 2 6 true true false true false true true true false 4 1 1 true true false true true false www.digikala.com 1 mail.sharif.ir 1 youtube.com 0 ads.example.org 2 192.168.1.5 1 172.40.0.1 0 0 1 2 true false true true

`Classifier`

struct Classifier

Three domain sets and a default: Classify asks them in order and returns a decision. Make it with NewClassifier and fill the sets.

Name	Type	Description
`block`	`DomainSet`	Hosts to refuse. Checked first.
`direct`	`DomainSet`	Hosts to reach directly.
`proxy`	`DomainSet`	Hosts that must use the proxy. Checked second, so it wins over direct and over private addresses.
`default_decision`	`int = 0`	What to do with a host that nothing matched.

The example for this item is shown with the ones above: net-censor-Domains.salam

`ClientHello`

struct ClientHello

What ParseClientHello reads from the first TLS message a client sends: the server name, the offered protocols and a few facts about it.

Name	Type	Description
`ok`	`bool = false`	True when the bytes are a valid ClientHello (possibly cut short, see `truncated`).
`truncated`	`bool = false`	True when the bytes ended before the whole message did. The fields read so far are still filled in.
`record_version`	`int = 0`	The version number in the TLS record header.
`hello_version`	`int = 0`	The version number in the ClientHello itself.
`record_len`	`int = 0`	The length the record header announces.
`sni`	`str = ""`	The server name from the Server Name Indication extension, lower case, or empty when there is none.
`sni_offset`	`int = -1`	Where in the bytes the server name starts, or -1. This is where a filter looks, and where `SNISplitSizes` cuts.
`sni_len`	`int = 0`	The length of the server name.
`alpn`	`Vector`	The application protocols the client offers, such as `h2` and `http/1.1`.
`cipher_count`	`int = 0`	How many cipher suites are offered.
`ext_count`	`int = 0`	How many extensions the message has.
`has_tls13`	`bool = false`	True when TLS 1.3 is offered in the supported versions extension.
`has_ech`	`bool = false`	True when the message carries an Encrypted Client Hello extension, which hides the real server name.
`session_id_len`	`int = 0`	The length of the session id the client sent. TLS 1.3 clients send 32 bytes of it.

net-censor-Sniff.salam

import net.censor

const HELLO := "160301008c010000880303000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f206465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f80818283000613011302c02b0100003900000014001200000f7777772e6578616d706c652e636f6d000a00040002001d0010000e000c02683208687474702f312e31002b0003020304"

func main:
    hello := censor.FromHex(HELLO)
    println "hello bytes:", hello.len, censor.ToHex(hello).len()
    println censor.KindName(censor.Sniff(hello.data, hello.len))
    println censor.KindName(censor.Sniff("GET / HTTP/1.1" as void*, 14))
    println censor.KindName(censor.Sniff("SSH-2.0-OpenSSH_9.6" as void*, 19))
    println censor.KindName(censor.Sniff(censor.FromHex("050100").data, 3))
    println censor.KindName(censor.Sniff(censor.FromHex("0401005000000001").data, 8))
    println censor.KindName(censor.Sniff("GE" as void*, 2))
    println censor.KindName(censor.Sniff("zzzz" as void*, 4))
    println censor.Sniff(hello.data, hello.len) == censor.KindTLS, censor.KindUnknown, censor.KindHTTP, censor.KindSSH, censor.KindSOCKS5, censor.KindSOCKS4, censor.KindTooShort

    ch := censor.ParseClientHello(hello.data, hello.len)
    println ch.ok, ch.truncated, ch.record_len
    println ch.sni, ch.sni_offset, ch.sni_len
    println ch.record_version, ch.hello_version, ch.session_id_len, ch.cipher_count, ch.ext_count
    println ch.has_tls13, ch.has_ech, ch.alpn.len(), ch.alpn.get(0), ch.alpn.get(1)
    println censor.ExtractSNI(hello.data, hello.len)

    cut := censor.ParseClientHello(hello.data, 60)
    println "cut short:", cut.ok, cut.truncated, cut.sni.len()
    println "not tls:", censor.ParseClientHello("GET / HTTP/1.1" as void*, 14).ok
end

Outputhello bytes: 145 290 tls http ssh socks5 socks4 short unknown true 0 2 3 4 5 6 true false 145 www.example.com 97 15 769 771 32 3 4 true false 2 h2 http/1.1 www.example.com cut short: false true 0 not tls: false

`ProbeGuard`

struct ProbeGuard

Counts failed authentications per IP address and bans an address that fails too often. Make it with NewProbeGuard. It is plain data with no lock, so share it between threads only under your own lock.

Name	Type	Description
`max_fails`	`int = 5`	How many failures within the window cause a ban.
`window_ms`	`i64 = 60000`	The length of the counting window, in milliseconds.
`ban_ms`	`i64 = 600000`	How long a ban lasts, in milliseconds.
`fails`	`HashMap`	The failure count of each address.
`first_ms`	`HashMap`	When each address's current window started.
`banned_until`	`HashMap`	Until when each banned address is banned.

net-censor-Probe.salam

import net.censor

func main:
    mut guard := censor.NewProbeGuard(3, 1000 as i64, 5000 as i64)
    println guard.max_fails, guard.window_ms, guard.ban_ms
    println censor.GuardAllowed(guard, "1.2.3.4", 0 as i64)
    println censor.GuardRecordFail(guard, "1.2.3.4", 100 as i64), censor.GuardRecordFail(guard, "1.2.3.4", 200 as i64), censor.GuardRecordFail(guard, "1.2.3.4", 300 as i64)
    println censor.GuardAllowed(guard, "1.2.3.4", 400 as i64), censor.GuardAllowed(guard, "5.6.7.8", 400 as i64), censor.GuardAllowed(guard, "1.2.3.4", 5400 as i64)
    println censor.GuardRecordFail(guard, "9.9.9.9", 0 as i64), censor.GuardRecordFail(guard, "9.9.9.9", 2000 as i64), censor.GuardRecordFail(guard, "9.9.9.9", 2100 as i64)
    censor.GuardRecordFail(guard, "7.7.7.7", 0 as i64)
    censor.GuardRecordFail(guard, "7.7.7.7", 10 as i64)
    censor.GuardRecordOK(guard, "7.7.7.7")
    println censor.GuardRecordFail(guard, "7.7.7.7", 20 as i64), guard.fails.get("7.7.7.7")
    censor.FreeProbeGuard(guard)

    mut cache := censor.NewReplayCache(1000 as i64, 4)
    println cache.ttl_ms, cache.max_entries
    println censor.ReplaySeen(cache, "a", 0 as i64), censor.ReplaySeen(cache, "a", 500 as i64), censor.ReplaySeen(cache, "a", 1600 as i64)
    println cache.seen.size()
    censor.FreeReplayCache(cache)

    println censor.ShouldFallback(false), censor.ShouldFallback(true)
    wait := censor.DelayMs(5, 20)
    println wait >= 5, wait <= 20
    println censor.DecoyResponse().substr(0, 22)
    println censor.DecoyResponse().includes("nginx")
end

Output3 1000 5000 true false false true false true true false false false false 1 1000 4 false true false 1 true false true true HTTP/1.1 404 Not Found true

`ReplayCache`

struct ReplayCache

Remembers recent keys (for example the first bytes of authentication) to tell when one comes a second time. Make it with NewReplayCache. It is plain data with no lock.

Name	Type	Description
`ttl_ms`	`i64 = 120000`	How long a key is remembered, in milliseconds. The default is 120000.
`max_entries`	`int = 4096`	The size at which expired keys are cleared out. The default is 4096.
`seen`	`HashMap`	The keys and the time each was last seen.

The example for this item is shown with the ones above: net-censor-Probe.salam

Constants and variables

Decisions `Decision…`

What Classify returns.

Name	Value
`DecisionProxy`	`0`
`DecisionDirect`	`1`
`DecisionBlock`	`2`

Protocol kinds `Kind…`

What Sniff returns.

Name	Value
`KindUnknown`	`0`
`KindTLS`	`1`
`KindHTTP`	`2`
`KindSSH`	`3`
`KindSOCKS5`	`4`
`KindSOCKS4`	`5`
`KindTooShort`	`6`

Split policies `Split…`

The policies of SplitSizes.

Name	Value
`SplitEven`	`0`
`SplitRandom`	`1`
`SplitFirst`	`2`

`MaxRecordPayload`

const MaxRecordPayload: i32 = 16000

The most data WrapStream puts in one record: 16000 bytes.

net-censor-Fragment.salam

import compress
import net.censor

const HELLO := "160301008c010000880303000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f206465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f80818283000613011302c02b0100003900000014001200000f7777772e6578616d706c652e636f6d000a00040002001d0010000e000c02683208687474702f312e31002b0003020304"

func total(v: Vector<int>): int:
    mut sum := 0
    repeat v.len() in i:
        sum += v.get(i)
    end
    ret sum
end

func main:
    hello := censor.FromHex(HELLO)

    even := censor.SplitSizes(100, censor.SplitEven, 3, 0, 1 as i64)
    println even.get(0), even.get(1), even.get(2), total(even)
    random := censor.SplitSizes(1000, censor.SplitRandom, 5, 0, 42 as i64)
    again := censor.SplitSizes(1000, censor.SplitRandom, 5, 0, 42 as i64)
    println random.len(), total(random), random.get(0) == again.get(0), random.get(4) == again.get(4)
    first := censor.SplitSizes(517, censor.SplitFirst, 2, 5, 0 as i64)
    println first.get(0), first.get(1)
    println censor.SplitSizes(3, censor.SplitEven, 10, 0, 0 as i64).len()

    cuts := censor.SNISplitSizes(hello.data, hello.len, false)
    println cuts.get(0), cuts.get(1), total(cuts)
    lone := censor.SNISplitSizes(hello.data, hello.len, true)
    println lone.len(), lone.get(0), lone.get(1), total(lone)

    mut pieces := censor.Slice(hello.data, hello.len, cuts)
    println pieces.len(), compress.BufEquals(censor.Concat(pieces), hello)
    censor.FreePieces(pieces)

    mut records := censor.FragmentClientHello(hello.data, hello.len, 4, censor.SplitEven, 0 as i64)
    println records.len()
    head := records.get(0)
    println head.len, compress.BufGet(head, 0), compress.BufGet(head, 1), compress.BufGet(head, 2)
    all := censor.Concat(records)
    println all.len, all.len == hello.len + 15
    println censor.ParseClientHello(all.data, all.len).ok, censor.ParseClientHello(head.data, head.len).truncated
    censor.FreePieces(records)

    mut by_hand := censor.FragmentRecords(hello.data, hello.len, cuts)
    println by_hand.len(), by_hand.get(0).len, by_hand.get(1).len
    censor.FreePieces(by_hand)
    println "not a handshake:", censor.FragmentClientHello("GET / HTTP/1.1" as void*, 14, 3, censor.SplitEven, 0 as i64).len()
    println censor.MaxRecordPayload
end

Output34 33 33 100 5 1000 true true 5 512 3 104 41 145 3 1 103 145 2 true 4 40 22 3 1 160 true false false 2 109 41 not a handshake: 0 16000

Functions

`NewDomainSet`

func NewDomainSet(): DomainSet

Returns an empty set.

The example for this item is shown with the ones above: net-censor-Domains.salam

`AddRule`

func AddRule(s &: DomainSet, rule: str): bool

Adds one rule: example.com or domain:example.com (that domain and all below it), full:name (exactly that name) or keyword:word. A leading *. is allowed. False for an empty line or a kind it does not support, such as regexp:.

The example for this item is shown with the ones above: net-censor-Domains.salam

`AddRules`

func AddRules(s &: DomainSet, text: str): int

Adds many rules from text, one per line. Blank lines and # comments, also after a rule, are skipped. Returns how many were added.

The example for this item is shown with the ones above: net-censor-Domains.salam

`MatchDomain`

func MatchDomain(s: DomainSet, host: str): bool

True when a host matches the set. Case and a trailing dot are ignored, and badexample.com does not match the rule example.com.

The example for this item is shown with the ones above: net-censor-Domains.salam

`FreeDomainSet`

func FreeDomainSet(s &: DomainSet)

Frees a set.

The example for this item is shown with the ones above: net-censor-Domains.salam

`IsPrivateHost`

func IsPrivateHost(host: str): bool

True for localhost and for private, loopback and link local addresses: 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, ::1 and the IPv6 private and link local ranges.

The example for this item is shown with the ones above: net-censor-Domains.salam

`NewClassifier`

func NewClassifier(default_decision: int): Classifier

Returns a classifier with empty sets and the given default decision.

The example for this item is shown with the ones above: net-censor-Domains.salam

`Classify`

func Classify(c: Classifier, host: str): int

Decides for a host: block first, then proxy, then direct (the direct set or any private address), then the default.

The example for this item is shown with the ones above: net-censor-Domains.salam

`IranDirectList`

func IranDirectList(): str

A short list, as rule text, of domestic sites that work best without a proxy: .ir and a handful of well known services. A starting point, to be edited.

The example for this item is shown with the ones above: net-censor-Domains.salam

`FilteredList`

func FilteredList(): str

A short list, as rule text, of sites commonly filtered in Iran. A starting point only: it ages and is not complete, so check it before you rely on it.

The example for this item is shown with the ones above: net-censor-Domains.salam

`IranDirectSet`

func IranDirectSet(): DomainSet

IranDirectList loaded into a DomainSet.

The example for this item is shown with the ones above: net-censor-Domains.salam

`FilteredSet`

func FilteredSet(): DomainSet

FilteredList loaded into a DomainSet.

The example for this item is shown with the ones above: net-censor-Domains.salam

`Sniff`

func Sniff(p: void*, n: int): int

Looks at the first bytes of a connection and says which protocol they start: one of the Kind values. It needs only a few bytes, and says KindTooShort when they are not enough.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`KindName`

func KindName(k: int): str

Returns a short name for a Kind value: tls, http, ssh, socks5, socks4, short or unknown.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`ParseClientHello`

func ParseClientHello(p: void*, n: int): ClientHello

Parses a TLS ClientHello from memory without any TLS library. Bytes that are cut short give a partial result with truncated set; anything that is not a ClientHello gives ok false.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`ExtractSNI`

func ExtractSNI(p: void*, n: int): str

Returns just the server name of a ClientHello, or an empty text.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`FromHex`

func FromHex(text: str): compress.ByteBuf

Turns hex text into a byte buffer. Spaces and line breaks are ignored.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`ToHex`

func ToHex(b: compress.ByteBuf): str

Returns the bytes of a buffer as lowercase hex.

The example for this item is shown with the ones above: net-censor-Sniff.salam

`RandomInt`

func RandomInt(lo: int, hi: int): int

Returns a random number from lo to hi, both included, from the system's secure random source.

net-censor-Padding.salam

import compress
import net.censor

func main:
    body := "secret payload"
    wrapped := censor.WrapRecord(body as void*, body.len(), 20)
    println wrapped.len, compress.BufGet(wrapped, 0), compress.BufGet(wrapped, 1), compress.BufGet(wrapped, 2)
    mut got := compress.NewByteBuf()
    println censor.UnwrapRecord(wrapped.data, wrapped.len, got), got.len
    println "needs more:", censor.UnwrapRecord(wrapped.data, wrapped.len - 1, got)
    println "not a record:", censor.UnwrapRecord("GET / HTTP/1.1" as void*, 14, got)
    println "too big:", censor.WrapRecord(body as void*, 20000, 0).len

    mut big := compress.NewByteBuf()
    repeat 40000 in i:
        compress.BufPush(big, i % 251)
    end
    stream := censor.WrapStream(big.data, big.len, 5, 50)
    mut back := compress.NewByteBuf()
    mut offset := 0
    mut records := 0
    until offset < stream.len:
        used := censor.UnwrapRecord(((stream.data as i64) + (offset as i64)) as void*, stream.len - offset, back)
        if used <= 0:
            break
        end
        offset += used
        records++
    end
    println records, compress.BufEquals(back, big), offset == stream.len

    println censor.PadLength(100, 10, 10, 64)
    println censor.PadLength(100, 0, 0, 1)
    println censor.RandomBytes(32).len, censor.RandomInt(5, 5)
    r := censor.RandomInt(10, 20)
    println r >= 10, r <= 20
end

Output43 23 3 3 43 14 needs more: 0 not a record: -1 too big: 0 3 true true 28 0 32 5 true true

`RandomBytes`

func RandomBytes(n: int): compress.ByteBuf

Returns a buffer of random bytes from the secure random source.

The example for this item is shown with the ones above: net-censor-Padding.salam

`PadLength`

func PadLength(payload_len: int, min_pad: int, max_pad: int, block: int): int

Chooses a random padding length between a minimum and a maximum and, when a block size above 1 is given, rounds the total up to a multiple of it.

The example for this item is shown with the ones above: net-censor-Padding.salam

`WrapRecord`

func WrapRecord(p: void*, n: int, pad: int): compress.ByteBuf

Wraps some bytes in a record that looks like TLS application data (type 23, version 3.3) and adds random padding after them. The record has a 4 byte inner header with the real and padding lengths. Empty when it would be over 16384 bytes.

The example for this item is shown with the ones above: net-censor-Padding.salam

`UnwrapRecord`

func UnwrapRecord(p: void*, n: int, payload &: compress.ByteBuf): int

Reads one record made by WrapRecord: puts its real bytes in a buffer and returns how many bytes the record took, 0 when more bytes are needed and -1 when it is not a valid record.

The example for this item is shown with the ones above: net-censor-Padding.salam

`WrapStream`

func WrapStream(p: void*, n: int, min_pad: int, max_pad: int): compress.ByteBuf

Wraps a whole stream of bytes in WrapRecord records of up to MaxRecordPayload bytes, each with random padding between the two limits. It hides sizes, not content: it is not encryption.

The example for this item is shown with the ones above: net-censor-Padding.salam

`SplitSizes`

func SplitSizes(total: int, policy: int, pieces: int, param: int, seed: i64): Vector<int>

Plans how to cut a total length into pieces: returns the sizes. The policy is SplitEven, SplitRandom or SplitFirst; pieces is how many, param the size of the first piece for SplitFirst, seed makes the random policy repeatable. The sizes always add up to the total.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`SNISplitSizes`

func SNISplitSizes(p: void*, n: int, head_first: bool): Vector<int>

Plans a cut in the middle of the server name of a ClientHello, so that no piece holds the whole name. With head_first the first byte goes alone. Without a readable server name it falls back to two even pieces.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`Slice`

func Slice(p: void*, n: int, sizes: Vector<int>): Vector<compress.ByteBuf>

Cuts memory into pieces of the given sizes (the last piece takes the rest) and returns them as buffers.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`FragmentRecords`

func FragmentRecords(p: void*, n: int, sizes: Vector<int>): Vector<compress.ByteBuf>

Cuts one TLS record into several records of the given payload sizes, each with its own header, so the receiver sees the same message in several records. Its type and version are kept.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`FragmentClientHello`

func FragmentClientHello(p: void*, n: int, pieces: int, policy: int, seed: i64): Vector<compress.ByteBuf>

Splits the first TLS message of a connection into the given number of records, by a policy and a seed. The result, sent in order, is still one valid ClientHello for the server, with 5 bytes of extra header per extra record. Empty when the bytes are not a handshake record.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`Concat`

func Concat(pieces: Vector<compress.ByteBuf>): compress.ByteBuf

Joins buffers into one, in order.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`FreePieces`

func FreePieces(pieces &: Vector<compress.ByteBuf>)

Frees a vector of buffers and the buffers in it.

The example for this item is shown with the ones above: net-censor-Fragment.salam

`NewProbeGuard`

func NewProbeGuard(max_fails: int, window_ms: i64, ban_ms: i64): ProbeGuard

Makes a guard: how many failures, in how long, give how long a ban.

The example for this item is shown with the ones above: net-censor-Probe.salam

`GuardAllowed`

func GuardAllowed(g: ProbeGuard, ip: str, now_ms: i64): bool

True when an address is not banned at the given time, in milliseconds.

The example for this item is shown with the ones above: net-censor-Probe.salam

`GuardRecordFail`

func GuardRecordFail(g &: ProbeGuard, ip: str, now_ms: i64): bool

Records a failed attempt from an address at a time. Returns true when this one caused a ban. Failures older than the window do not count.

The example for this item is shown with the ones above: net-censor-Probe.salam

`GuardRecordOK`

func GuardRecordOK(g &: ProbeGuard, ip: str)

Records a success from an address and clears its failure count.

The example for this item is shown with the ones above: net-censor-Probe.salam

`FreeProbeGuard`

func FreeProbeGuard(g &: ProbeGuard)

Frees a guard.

The example for this item is shown with the ones above: net-censor-Probe.salam

`NewReplayCache`

func NewReplayCache(ttl_ms: i64, max_entries: int): ReplayCache

Makes a cache with a lifetime for keys and a size at which it cleans up.

The example for this item is shown with the ones above: net-censor-Probe.salam

`ReplaySeen`

func ReplaySeen(c &: ReplayCache, key: str, now_ms: i64): bool

Returns true when the key was seen within the lifetime. Otherwise it remembers the key now and returns false, so asking twice tells you about the repeat.

The example for this item is shown with the ones above: net-censor-Probe.salam

`FreeReplayCache`

func FreeReplayCache(c &: ReplayCache)

Frees a cache.

The example for this item is shown with the ones above: net-censor-Probe.salam

`ShouldFallback`

func ShouldFallback(authenticated: bool): bool

True for a connection that did not authenticate: it should be handed to the decoy site instead of being dropped. It is the one decision a probe resistant server makes.

The example for this item is shown with the ones above: net-censor-Probe.salam

`DelayMs`

func DelayMs(min_ms: int, max_ms: int): int

Returns a random delay between two limits, in milliseconds. Waiting a little before answering a failed client hides how fast the check was.

The example for this item is shown with the ones above: net-censor-Probe.salam

`DecoyResponse`

func DecoyResponse(): str

Returns a ready HTTP response that looks like a stock 404 Not Found page of a common web server, for a server that has no real site to show.

The example for this item is shown with the ones above: net-censor-Probe.salam

net.censor

Fragmenting a TLS hello

Domain rules and the Iran lists

Hiding a server from probes

What to know

Types

DomainSet

Classifier

ClientHello

ProbeGuard

ReplayCache

Constants and variables

Decisions Decision…

Protocol kinds Kind…

Split policies Split…

MaxRecordPayload

Functions

NewDomainSet

AddRule

AddRules

MatchDomain

FreeDomainSet

IsPrivateHost

NewClassifier

Classify

IranDirectList

FilteredList

IranDirectSet

FilteredSet

Sniff

KindName

ParseClientHello

ExtractSNI

FromHex

ToHex

RandomInt

RandomBytes

PadLength

WrapRecord

UnwrapRecord

WrapStream

SplitSizes

SNISplitSizes

Slice

FragmentRecords

FragmentClientHello

Concat

FreePieces

NewProbeGuard

GuardAllowed

GuardRecordFail

GuardRecordOK

FreeProbeGuard

NewReplayCache

ReplaySeen

FreeReplayCache

ShouldFallback

DelayMs

DecoyResponse

`DomainSet`

`Classifier`

`ClientHello`

`ProbeGuard`

`ReplayCache`

Decisions `Decision…`

Protocol kinds `Kind…`

Split policies `Split…`

`MaxRecordPayload`

`NewDomainSet`

`AddRule`

`AddRules`

`MatchDomain`

`FreeDomainSet`

`IsPrivateHost`

`NewClassifier`

`Classify`

`IranDirectList`

`FilteredList`

`IranDirectSet`

`FilteredSet`

`Sniff`

`KindName`

`ParseClientHello`

`ExtractSNI`

`FromHex`

`ToHex`

`RandomInt`

`RandomBytes`

`PadLength`

`WrapRecord`

`UnwrapRecord`

`WrapStream`

`SplitSizes`

`SNISplitSizes`

`Slice`

`FragmentRecords`

`FragmentClientHello`

`Concat`

`FreePieces`

`NewProbeGuard`

`GuardAllowed`

`GuardRecordFail`

`GuardRecordOK`

`FreeProbeGuard`

`NewReplayCache`

`ReplaySeen`

`FreeReplayCache`

`ShouldFallback`

`DelayMs`

`DecoyResponse`