Standard library · Network

net.httpproxy

An HTTP and HTTPS proxy: a client with Basic login and CONNECT tunnels, a server that forwards plain requests and opens tunnels, with access rules, TLS and a connector for proxy chains.

import net.httpproxy

On this page

Using an HTTP proxy from a program
Running an HTTP proxy
What to know
Types
Constants and variables
Functions

An HTTP proxy handles two kinds of traffic. A plain request (GET http://example.com/ HTTP/1.1) is forwarded by the proxy, which opens the connection to the website itself. A CONNECT host:port request asks the proxy to open a raw tunnel, and then bytes pass untouched in both directions, which is how HTTPS and any other TCP protocol cross a proxy. This package has both sides: Get, Post, Request and Connect are the client, Start runs a server. It also plugs into net.tunnel as an HttpInbound that can share one port with SOCKS, and as an HttpConnector that is one hop of a proxy chain. To see how the pieces fit together in an app, read Building a v2ray style or VPN app.

Core ideas:

Get, Post and Request fetch a plain http:// URL through the proxy and return a Response. For https:// use Connect, which returns a stream, and run TLS over it yourself.
Failure is a value. ok and ok are false, with err or status saying why. A 404 or a 407 is a successful round trip with that status.
A server is a ServerConfig given to Start. SetConnector or SetDialer decide how targets are reached, so it can be joined to routing rules and proxy chains; allow refuses methods, hosts or ports.
The parser is strict on purpose: a request with both Content-Length and Transfer-Encoding, two different lengths, folded or malformed headers, a bad method or a head over MaxHeadBytes is refused instead of guessed at. That is how request smuggling is kept out.

Things that catch people out:

A server listens on bind_host, which is 127.0.0.1 by default. Set 0.0.0.0 only on purpose. Without a login, an open HTTP proxy reachable from outside will be used by strangers. Use AddUser, a firewall, or both.
A CONNECT proxy can reach internal addresses. A client that may open tunnels to any host and port can reach services on your own network (a database on 10.0.0.5, a metadata address) or send mail on port 25. Restrict it with allow, for example to port 443 and to public hosts.
Basic login is not encrypted. The user name and password are Base64, which is only an encoding. Use SetTLS to make the proxy itself speak TLS, or keep it on a trusted link.
Bodies are text. body stops at the first NUL byte, and through a HttpInbound a plain request body is limited to 1 MiB of text. Tunnels have no such limit.
Only absolute http:// URLs. Request refuses https:// and relative URLs. ParseURL reads both schemes, but only http can go the plain way.
HTTP/1.1 only. There is no HTTP/2 and no compressed transfer encoding (a gzip transfer encoding is answered with 501).
Close streams returned in stream.

Using an HTTP proxy from a program

For a plain website, Get sends the request through the proxy. For anything that is not plain HTTP, and for HTTPS, Connect opens a tunnel and gives you the stream; run TLS or any other protocol over it. The example uses a local proxy and two local servers so that it runs anywhere.

net-httpproxy-guide-proxy.salam

import net.httpproxy
import net.tcp
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    mut server_cfg := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(server_cfg, true)
    server := httpproxy.Start(server_cfg)
    client := httpproxy.NewClient("127.0.0.1", server.port)

    plain := httpproxy.Get(client, "http://127.0.0.1:" + origin_port + "/news")
    println "plain http:", plain.status, plain.body

    tunnel := httpproxy.Connect(client, "127.0.0.1", echo_port)
    tunnel.stream.Write("anything, even TLS")
    println "tunnel:", tunnel.status, tunnel.stream.Read(100)
    tunnel.stream.Close()
    httpproxy.Stop(server)
end

Outputplain http: 200 origin saw: GET /news HTTP/1.1 tunnel: 200 echo:anything, even TLS

If the proxy needs a login use NewClientAuth; a 407 status means the login was missing or wrong. To use the proxy as one hop of several, see HttpConnector and tunnel.Chain.

Running an HTTP proxy

NewServerConfig gives a config for a port, AddUser turns the login on and Start serves in the background. The server keeps connections alive, handles pipelined requests, chunked request bodies, Expect: 100-continue and Upgrade, and strips the headers that belong to one hop (IsHopByHop) before it forwards a request. Slow or silent clients are cut by handshake_timeout_ms and idle_timeout_ms.

Decide what the proxy may do before you open it to anyone: allow_connect and allow_plain switch the two kinds of traffic, and allow sees every method, host and port. A proxy for a site should usually allow CONNECT only to port 443 and refuse private addresses.

mut config := httpproxy.NewServerConfig(3128)
httpproxy.AddUser(config, "alice", "a long random password")
config.allow = (method: str, _host: str, port: int): ret port == 443 or method != "CONNECT" end
httpproxy.SetTLS(config, "proxy.crt", "proxy.key")
server := httpproxy.Start(config)

SetTLS turns the proxy into an HTTPS proxy: clients connect to it over TLS (set tls) and only then send their requests, so the login and the targets are no longer visible on the wire. Use a certificate for the name clients connect to.

What to know

The server listens on bind_host (loopback by default). Use a login or a firewall, and restrict CONNECT targets.
Basic login is Base64 only. Use SetTLS or a trusted link.
Through HttpInbound (the tunnel adapter) each plain request gets its own connection and a body of up to 1 MiB of text; the standalone Start server keeps connections alive.
Head limits are MaxHeadBytes bytes and MaxHeaders header lines. Bigger heads get 431.
HTTP/1.1 only, no HTTP/2, no compressed transfer encoding.
No logging happens by itself.

Types

`Head`

struct Head

A parsed request or response head: the first line and the headers. ParseRequestHead and ParseResponseHead make it.

Name	Type	Description
`ok`	`bool = false`	True when the head was valid.
`err`	`str = ""`	Why it was not valid.
`method`	`str = ""`	The request method, such as `GET` or `CONNECT`.
`target`	`str = ""`	The request target: an absolute URL for a proxied request, `host:port` for `CONNECT`.
`version`	`str = ""`	The HTTP version text, such as `HTTP/1.1`.
`status`	`int = 0`	The status code of a response head.
`reason`	`str = ""`	The reason phrase of a response head. Empty is allowed.
`names`	`Vector`	The header names in order, as they were written.
`values`	`Vector`	The header values, one for each name.

net-httpproxy-Parse.salam

import net.httpproxy
import str

func main:
    request := "GET http://example.com:8080/a/b?x=1 HTTP/1.1\r\nHost: example.com:8080\r\nProxy-Authorization: Basic dXNlcjpwYXNz\r\nConnection: keep-alive, X-Drop\r\nX-Drop: 1\r\n\r\n"
    head := httpproxy.ParseRequestHead(request)
    println head.ok, head.method, head.target, head.version, head.names.len()
    println httpproxy.HeaderGet(head, "HOST"), httpproxy.HeaderHas(head, "x-drop"), httpproxy.HeaderCount(head, "host")
    println httpproxy.HeaderHasToken(head, "connection", "KEEP-ALIVE"), httpproxy.HeaderHasToken(head, "connection", "close")
    println head.names.get(0), head.values.get(0)

    each text in ["G@T / HTTP/1.1\r\n\r\n", "GET /\r\n\r\n", "GET / HTTP/1.1\r\nBroken header\r\n\r\n", "GET / HTTP/1.1\r\nA: b\r\n c\r\n\r\n"]:
        bad := httpproxy.ParseRequestHead(text)
        println bad.ok, bad.err
    end
    println httpproxy.MaxHeadBytes, httpproxy.MaxHeaders

    response := httpproxy.ParseResponseHead("HTTP/1.1 407 Proxy Authentication Required\r\nProxy-Authenticate: Basic realm=\"x\"\r\n\r\n")
    println response.ok, response.status, response.reason, httpproxy.HeaderGet(response, "proxy-authenticate")

    each u in ["http://example.com", "HTTP://Example.COM:81/p?q=1#frag", "http://[::1]:8080/x", "https://secure.example/a", "ftp://example.com/", "/relative"]:
        x := httpproxy.ParseURL(u)
        if x.ok:
            println u, "->", x.scheme, x.host, x.port, x.path
        else:
            println u, "-> invalid"
        end
    end
    each t in ["example.com:443", "example.com", "[::1]:8443", "::1", "host:0"]:
        hp := httpproxy.SplitHostPort(t, 80)
        println t, "->", hp.ok, hp.host, hp.port, hp.has_port
    end

    header := httpproxy.BasicHeader("user", "pass")
    println header
    mut user := ""
    mut password := ""
    println httpproxy.BasicCredentials(header, user, password), user, password
    println httpproxy.BasicCredentials("Bearer abc", user, password)

    each name in ["Connection", "Transfer-Encoding", "Upgrade", "Host", "Content-Length"]:
        println name, httpproxy.IsHopByHop(name)
    end
    println str.Replace(httpproxy.ResponseText(407, "Proxy-Authenticate: Basic realm=\"r\"\r\n", "", false), "\r\n", "|")
    println httpproxy.StatusReason(403), "|", httpproxy.StatusReason(502), "|", httpproxy.StatusReason(999)
end

Outputtrue GET http://example.com:8080/a/b?x=1 HTTP/1.1 4 example.com:8080 true 1 true false Host example.com:8080 false invalid method false malformed request line false malformed header line false folded header lines are not allowed 65536 200 true 407 Proxy Authentication Required Basic realm="x" http://example.com -> http example.com 80 / HTTP://Example.COM:81/p?q=1#frag -> http Example.COM 81 /p?q=1 http://[::1]:8080/x -> http ::1 8080 /x https://secure.example/a -> https secure.example 443 /a ftp://example.com/ -> invalid /relative -> invalid example.com:443 -> true example.com 443 true example.com -> true example.com 80 false [::1]:8443 -> true ::1 8443 true ::1 -> false 0 false host:0 -> false 0 false Basic dXNlcjpwYXNz true user pass false Connection true Transfer-Encoding true Upgrade true Host false Content-Length false HTTP/1.1 407 Proxy Authentication Required|Proxy-Authenticate: Basic realm="r"|Content-Length: 0|| Forbidden | Bad Gateway | Status

`HostPort`

struct HostPort

A host and a port taken from text such as example.com:443.

Name	Type	Description
`ok`	`bool = false`	True when the text was a valid host and port.
`host`	`str = ""`	The host, without brackets for IPv6.
`port`	`int = 0`	The port, or the default when the text had none.
`has_port`	`bool = false`	True when the text named the port itself.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`URL`

struct URL

A parsed http:// or https:// URL. ParseURL makes it.

Name	Type	Description
`ok`	`bool = false`	True when the URL was valid.
`scheme`	`str = ""`	`http` or `https`, in lower case.
`host`	`str = ""`	The host, as written.
`port`	`int = 0`	The port, or 80 or 443 when the URL had none.
`path`	`str = ""`	The path with the query, at least `/`. A fragment is dropped.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`Client`

struct Client

An HTTP proxy client: which proxy to use, the login and the TLS options. Make it with NewClient or NewClientAuth.

Name	Type	Description
`host`	`str = ""`	The proxy's host name or IP.
`port`	`int = 0`	The proxy's port.
`username`	`str = ""`	The user name for `Proxy-Authorization: Basic`. Empty means no login.
`password`	`str = ""`	The password that goes with the user name.
`tls`	`bool = false`	True to reach the proxy itself over TLS (an HTTPS proxy). The default is false.
`server_name`	`str = ""`	The name to send and check in the proxy's certificate when `tls` is on. Empty means the host.
`insecure`	`bool = false`	True to skip the certificate check. For tests only.
`timeout_ms`	`int = 15000`	How long connecting and each answer may take, in milliseconds. The default is 15000.
`max_body`	`int = 16777216`	The largest response body `Request` will read, in bytes. The default is 16777216 (16 MiB); a bigger body makes the response fail.

net-httpproxy-Get.salam

import net.httpproxy
import net.tcp
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    mut server_cfg := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(server_cfg, true)
    server := httpproxy.Start(server_cfg)
    println "proxy up:", server.ok
    client := httpproxy.NewClient("127.0.0.1", server.port)
    println client.timeout_ms, client.max_body, client.tls

    url := "http://127.0.0.1:" + origin_port
    r := httpproxy.Get(client, url + "/hello?x=1")
    println r.ok, r.status, r.length, r.body
    println httpproxy.HeaderGet(r.head, "content-length"), r.head.reason

    p := httpproxy.Post(client, url + "/submit", "text/plain", "payload")
    println p.ok, p.status, p.body
    custom := httpproxy.Request(client, "PUT", url + "/thing", "X-Trace: 1\r\n", "data")
    println custom.status, custom.body

    https := httpproxy.Get(client, "https://127.0.0.1/")
    println https.ok, https.err
    httpproxy.Stop(server)
end

Outputproxy up: true 15000 16777216 false true 200 35 origin saw: GET /hello?x=1 HTTP/1.1 35 OK true 200 origin saw: POST /submit HTTP/1.1 200 origin saw: PUT /thing HTTP/1.1 false only absolute http URLs can go through the proxy this way

`HandshakeResult`

struct HandshakeResult

The outcome of a CONNECT on an open stream.

Name	Type	Description
`ok`	`bool = false`	True when the proxy answered with a 2xx status.
`status`	`int = 0`	The HTTP status the proxy answered with, 0 when there was no answer.
`err`	`str = ""`	A readable reason when it failed.

net-httpproxy-Connect.salam

import net.httpproxy
import net.tcp
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    mut server_cfg := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(server_cfg, true)
    server := httpproxy.Start(server_cfg)
    client := httpproxy.NewClient("127.0.0.1", server.port)

    tunnel := httpproxy.Connect(client, "127.0.0.1", echo_port)
    println "connect:", tunnel.ok, tunnel.status
    tunnel.stream.Write("bytes pass through untouched")
    println tunnel.stream.Read(100)
    tunnel.stream.Close()

    conn := httpproxy.DialConn(client, "127.0.0.1", echo_port)
    tcp.Write(conn, "plain conn")
    println tcp.Read(conn, 100)
    tcp.Close(conn)

    refused := httpproxy.Connect(client, "127.0.0.1", tcp.FreePort())
    println "refused:", refused.ok, refused.status, refused.err
    httpproxy.Stop(server)

    mut first_cfg := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(first_cfg, true)
    first := httpproxy.Start(first_cfg)
    mut second_cfg := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(second_cfg, true)
    second := httpproxy.Start(second_cfg)
    hop := httpproxy.Connect(httpproxy.NewClient("127.0.0.1", first.port), "127.0.0.1", second.port)
    onward := httpproxy.ClientHandshake(hop.stream, httpproxy.NewClient("", 0), "127.0.0.1", echo_port)
    println "second hop:", onward.ok, onward.status
    hop.stream.Write("two hops")
    println hop.stream.Read(100)
    hop.stream.Close()
    httpproxy.Stop(first)
    httpproxy.Stop(second)
end

Outputconnect: true 200 echo:bytes pass through untouched echo:plain conn refused: false 502 proxy answered 502 Bad Gateway second hop: true 200 echo:two hops

`ConnectResult`

struct ConnectResult

The outcome of Connect: the tunnel stream when it worked, and the status and reason when it did not.

Name	Type	Description
`ok`	`bool = false`	True when the tunnel is open.
`status`	`int = 0`	The proxy's HTTP status, such as 200, 407 (login needed), 403 or 502.
`err`	`str = ""`	A readable reason when it failed.
`stream`	`dyn tunnel.Stream`	The open stream to the target, valid when `ok` is true. Bytes in both directions pass through untouched, so TLS can run inside it.

The example for this item is shown with the ones above: net-httpproxy-Connect.salam

`Response`

struct Response

A whole HTTP response fetched through the proxy by Get, Post or Request.

Name	Type	Description
`ok`	`bool = false`	True when a complete response arrived. A 404 is still ok; a cut off body is not.
`err`	`str = ""`	A readable reason when `ok` is false.
`status`	`int = 0`	The HTTP status code.
`head`	`Head`	The parsed status line and headers. Read them with `HeaderGet`.
`body`	`str = ""`	The body as text. It stops at the first NUL byte.
`length`	`int = 0`	The body length in bytes.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`ServerConfig`

struct ServerConfig

The settings of an HTTP proxy server: port, login, what is allowed, TLS, time limits and how targets are reached. Start from NewServerConfig.

Name	Type	Description
`port`	`int = 0`	The port to listen on; 0 picks a free one.
`bind_host`	`str = "127.0.0.1"`	The address to listen on. The default is `127.0.0.1`; use `0.0.0.0` or a public address on purpose.
`require_auth`	`bool = false`	True when clients must send `Proxy-Authorization`. `AddUser` and `SetAuth` turn it on.
`realm`	`str = "Salam proxy"`	The realm named in the 407 challenge. The default is `Salam proxy`.
`users`	`HashMap`	The accepted user names and passwords.
`auth`	`func(str, str): bool`	A custom check `(user, password) -> bool` used instead of `users`. Set it with `SetAuth`.
`has_auth`	`bool = false`	True when `auth` is in use.
`allow_connect`	`bool = true`	True to allow `CONNECT` tunnels. The default is true. False answers 403.
`allow_plain`	`bool = true`	True to forward plain `http://` requests. The default is true. False answers 403.
`handshake_timeout_ms`	`int = 15000`	How long a client has to send its request head, in milliseconds. The default is 15000.
`idle_timeout_ms`	`int = 60000`	How long a keep-alive connection may wait for the next request, in milliseconds. The default is 60000.
`relay_idle_ms`	`int = 0`	Cut a `CONNECT` tunnel after this many silent milliseconds. 0, the default, means never.
`use_tls`	`bool = false`	True when the proxy itself speaks TLS, which `SetTLS` turns on.
`tls_config`	`tls.ServerConfig`	The TLS settings used when `use_tls` is true.
`connect`	`func(str, int): socks.Outbound`	The callback that reaches a target: `(host, port) -> socks.Outbound`. The default dials directly. Set it with `SetConnector` or `SetDialer`.
`allow`	`func(str, str, int): bool`	An access rule `(method, host, port) -> bool` asked before each request; false answers 403. The default allows everything.
`policy`	`tunnel.DestPolicy`	Which destinations may be reached. By default loopback, private, link local, metadata and other special addresses are refused. See `DestPolicy`.
`has_connect`	`bool = false`	True when a custom connector is set. Then the policy checks names and literal addresses only; the connector must resolve safely.
`max_connections`	`int = 1024`	The most connections served at once; more are closed (SOCKS) or answered with 503 (HTTP). The default is 1024; 0 turns the cap off.
`max_per_source`	`int = 32`	The most handshakes one source address may have in progress at once. The default is 32; 0 turns the limit off.
`handshake_deadline_ms`	`int = 10000`	The total time a connection has to finish its handshake, in milliseconds, however slowly it sends. The default is 10000; 0 turns it off.

net-httpproxy-Policy.salam

import net.httpproxy
import net.socks
import net.tcp
import net.tunnel
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    url := "http://127.0.0.1:" + origin_port + "/"

    mut acl := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(acl, true)
    acl.allow = (method: str, _host: str, port: int): ret method == "GET" and port != 25 end
    acl_server := httpproxy.Start(acl)
    acl_client := httpproxy.NewClient("127.0.0.1", acl_server.port)
    println "GET:", httpproxy.Get(acl_client, url).status
    println "POST:", httpproxy.Post(acl_client, url, "text/plain", "x").status
    println "port 25:", httpproxy.Get(acl_client, "http://127.0.0.1:25/").status
    httpproxy.Stop(acl_server)

    mut no_connect := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(no_connect, true)
    no_connect.allow_connect = false
    nc := httpproxy.Start(no_connect)
    println "connect off:", httpproxy.Connect(httpproxy.NewClient("127.0.0.1", nc.port), "127.0.0.1", echo_port).status, httpproxy.Get(httpproxy.NewClient("127.0.0.1", nc.port), url).status
    httpproxy.Stop(nc)

    mut no_plain := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(no_plain, true)
    no_plain.allow_plain = false
    np := httpproxy.Start(no_plain)
    opened := httpproxy.Connect(httpproxy.NewClient("127.0.0.1", np.port), "127.0.0.1", echo_port)
    println "plain off:", httpproxy.Get(httpproxy.NewClient("127.0.0.1", np.port), url).status, opened.status
    opened.stream.Close()
    httpproxy.Stop(np)

    mut routed := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(routed, true)
    httpproxy.SetConnector(routed, (host: str, _port: int): if host == "echo.internal": ret socks.DirectConnect("127.0.0.1", echo_port, 2000) end ret socks.Outbound { ok = false, code = socks.RepNetUnreachable, err = "no route" } end)
    rs := httpproxy.Start(routed)
    named := httpproxy.Connect(httpproxy.NewClient("127.0.0.1", rs.port), "echo.internal", 1)
    named.stream.Write("by name")
    println "routed:", named.status, named.stream.Read(100)
    println "unknown:", httpproxy.Connect(httpproxy.NewClient("127.0.0.1", rs.port), "other.internal", 1).status
    httpproxy.Stop(rs)

    mut blocked := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(blocked, true)
    httpproxy.SetDialer(blocked, tunnel.Block())
    bs := httpproxy.Start(blocked)
    println "block dialer:", httpproxy.Connect(httpproxy.NewClient("127.0.0.1", bs.port), "127.0.0.1", echo_port).status
    httpproxy.Stop(bs)

    mut config := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(config, true)
    println config.handshake_timeout_ms, config.idle_timeout_ms, config.relay_idle_ms, config.allow_connect, config.allow_plain
end

OutputGET: 200 POST: 403 port 25: 403 connect off: 403 200 plain off: 403 200 routed: 200 echo:by name unknown: 502 block dialer: 502 15000 60000 0 true true

`Server`

struct Server

A running HTTP proxy. Start returns it and Stop ends it.

Name	Type	Description
`ok`	`bool = false`	True when the port was bound and the server is running.
`port`	`int = 0`	The port it listens on, useful when you asked for 0.
`host`	`str = ""`	The address the server listens on.
`listener`	`tcp.Server`	The underlying TCP listener.
`stop_cell`	`i64*`	The flag that `Stop` clears to end the accept loop.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`HttpInbound`

struct HttpInbound

An HTTP proxy server for tunnel.InboundProtocol: it reads a CONNECT or an absolute URL request from a stream and turns it into a tunnel.Session, so it can share a port in a tunnel.MixedProtocol. Make one with NewInbound.

Name	Type	Description
`config`	`ServerConfig`	The server settings: login, which methods are allowed and the limits.

net-httpproxy-Inbound.salam

import net.httpproxy
import net.tcp
import net.tunnel
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    mut config := httpproxy.NewServerConfig(0)
    httpproxy.AddUser(config, "u", "p")
    inbound_protocol := httpproxy.NewInbound(config)
    println inbound_protocol.Name()

    forward := tunnel.NewForwarder(tunnel.Direct(), inbound_protocol)
    listener := tunnel.ListenTcp(0)
    tunnel.ServeBackground(listener, inbound_protocol, forward)

    client := httpproxy.NewClientAuth("127.0.0.1", listener.port, "u", "p")
    println httpproxy.Get(client, "http://127.0.0.1:" + origin_port + "/page").body
    tunnel_result := httpproxy.Connect(client, "127.0.0.1", echo_port)
    tunnel_result.stream.Write("connect through the inbound")
    println tunnel_result.status, tunnel_result.stream.Read(100)
    tunnel_result.stream.Close()
    println "without login:", httpproxy.Get(httpproxy.NewClient("127.0.0.1", listener.port), "http://127.0.0.1:" + origin_port + "/").status
    listener.Stop()

    hop := httpproxy.NewConnector(httpproxy.NewClient("127.0.0.1", 3128))
    println hop.Name(), hop.Server().String()
    via := tunnel.Via(tunnel.Direct(), hop)
    println via.Name()
end

Outputhttp origin saw: GET /page HTTP/1.1 200 echo:connect through the inbound without login: 407 http 127.0.0.1:3128 http

`Name`

func Name(): str

Returns http.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`Handshake`

func Handshake(s: dyn tunnel.Stream): tunnel.Session

Reads and checks the request head, checks the proxy login and the access rule, and returns the target. A plain request is rewritten to the path form with Connection: close and passed on as tunnel.initial, so through this inbound each connection carries one request, with a body of up to 1 MiB of text; chunked bodies get 501. A refused request is answered here (407, 403, 400, 431, 501).

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`Reply`

func Reply(s: dyn tunnel.Stream, sess: tunnel.Session, ok: bool)

For a CONNECT request, sends 200 Connection established or 502 Bad Gateway. For a plain request only the failure is answered (502): on success the target's own response is what the client gets.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`HttpConnector`

struct HttpConnector

One HTTP proxy hop of a tunnel.Chain: it connects to a proxy and asks it for a CONNECT tunnel to the next place. Make one with NewConnector.

Name	Type	Description
`client`	`Client`	The client settings: proxy address, login and TLS.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`Server`

func Server(): tunnel.Addr

The address of the HTTP proxy.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`Handshake`

func Handshake(s: dyn tunnel.Stream, target: tunnel.Addr): bool

Sends CONNECT for a target over the open stream and checks for a 2xx answer. True when the proxy agreed.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`Name`

func Name(): str

Returns http.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

Constants and variables

`MaxHeadBytes`

const MaxHeadBytes: i32 = 65536

The largest request or response head, in bytes: 65536. A bigger one is refused (431 on the server).

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`MaxHeaders`

const MaxHeaders: i32 = 200

The largest number of header lines: 200.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

Functions

`ParseRequestHead`

func ParseRequestHead(text: str): Head

Parses a request head: the request line and header lines. It is strict: a bad method, a missing version, folded or malformed headers, more than MaxHeaders headers or a head over MaxHeadBytes all fail with a reason in err.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`ParseResponseHead`

func ParseResponseHead(text: str): Head

Parses a response head: the status line and headers. The reason phrase may be empty.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`HeaderGet`

func HeaderGet(h: Head, name: str): str

Returns the first value of a header, or an empty text. The name is compared without regard to case.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`HeaderHas`

func HeaderHas(h: Head, name: str): bool

True when the header exists.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`HeaderCount`

func HeaderCount(h: Head, name: str): int

How many lines carry this header name.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`HeaderHasToken`

func HeaderHasToken(h: Head, name: str, token: str): bool

True when a comma separated header value contains a token, such as keep-alive in Connection. Case is ignored.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`IsHopByHop`

func IsHopByHop(name: str): bool

True for headers that belong to one connection and must not be forwarded: Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, Proxy-Connection, TE, Trailer, Transfer-Encoding, Upgrade.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`SplitHostPort`

func SplitHostPort(s: str, default_port: int): HostPort

Splits host:port or [v6]:port and gives a default port when none is written. Invalid text (a bad port, extra colons, spaces, / or @) gives ok false.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`ParseURL`

func ParseURL(s: str): URL

Parses an http:// or https:// URL into scheme, host, port and path. Other schemes and relative URLs are invalid.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`BasicHeader`

func BasicHeader(user: str, pass: str): str

Returns the Basic ... value for a Proxy-Authorization header: the user name and password joined by a colon, in Base64.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`BasicCredentials`

func BasicCredentials(value: str, user &: str, pass &: str): bool

Reads a Basic header value back into a user name and a password (the password may contain colons). False when it is not a valid Basic value.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`StatusReason`

func StatusReason(code: int): str

Returns the standard reason phrase for a status code, or Status for one it does not know.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`ResponseText`

func ResponseText(code: int, extra_headers: str, body: str, close: bool): str

Builds a whole small response: the status line, any extra header lines, Content-Length, optional Connection: close and the body.

The example for this item is shown with the ones above: net-httpproxy-Parse.salam

`NewClient`

func NewClient(host: str, port: int): Client

Makes a client for a proxy with no login.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`NewClientAuth`

func NewClientAuth(host: str, port: int, username: str, password: str): Client

Makes a client that logs in to the proxy with Basic authentication.

net-httpproxy-Auth.salam

import net.httpproxy
import net.tcp
import sync

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func main:
    start_backends()
    mut config := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(config, true)
    config.realm = "demo"
    httpproxy.AddUser(config, "alice", "s3cret")
    println config.require_auth, config.realm
    server := httpproxy.Start(config)
    url := "http://127.0.0.1:" + origin_port + "/"

    anonymous := httpproxy.Get(httpproxy.NewClient("127.0.0.1", server.port), url)
    println "no login:", anonymous.status, httpproxy.HeaderGet(anonymous.head, "proxy-authenticate")
    wrong := httpproxy.Get(httpproxy.NewClientAuth("127.0.0.1", server.port, "alice", "nope"), url)
    println "wrong password:", wrong.status
    right := httpproxy.Get(httpproxy.NewClientAuth("127.0.0.1", server.port, "alice", "s3cret"), url)
    println "right password:", right.status, right.body
    println "connect without login:", httpproxy.Connect(httpproxy.NewClient("127.0.0.1", server.port), "127.0.0.1", echo_port).status
    httpproxy.Stop(server)

    mut custom := httpproxy.NewServerConfig(0)
    httpproxy.AllowPrivate(custom, true)
    httpproxy.SetAuth(custom, (user: str, password: str): ret user == "bob" and password == "pw" end)
    println custom.has_auth
    second := httpproxy.Start(custom)
    println "callback good:", httpproxy.Get(httpproxy.NewClientAuth("127.0.0.1", second.port, "bob", "pw"), url).status
    println "callback bad:", httpproxy.Get(httpproxy.NewClientAuth("127.0.0.1", second.port, "bob", "x"), url).status
    httpproxy.Stop(second)
end

Outputtrue demo no login: 407 Basic realm="demo" wrong password: 407 right password: 200 origin saw: GET / HTTP/1.1 connect without login: 407 true callback good: 200 callback bad: 407

`ClientHandshake`

func ClientHandshake(s: dyn tunnel.Stream, c: Client, host: str, port: int): HandshakeResult

Sends CONNECT host:port on a stream that is already open to a proxy and reads the answer. This is how a second hop is added on top of the first.

The example for this item is shown with the ones above: net-httpproxy-Connect.salam

`Connect`

func Connect(c: Client, host: str, port: int): ConnectResult

Connects to the proxy and opens a CONNECT tunnel to a host and port. Use it for HTTPS and any other TCP protocol.

The example for this item is shown with the ones above: net-httpproxy-Connect.salam

`DialConn`

func DialConn(c: Client, host: str, port: int): tcp.Conn

Like Connect but returns a plain TCP connection for the net.tcp functions. An empty connection when it failed.

The example for this item is shown with the ones above: net-httpproxy-Connect.salam

`Request`

func Request(c: Client, method: str, url: str, headers: str, body: str): Response

Sends one plain HTTP request through the proxy: a method, an absolute http:// URL, extra header lines (each ending in a line break) and a body. For https:// use Connect and run TLS over the tunnel.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`Get`

func Get(c: Client, url: str): Response

Fetches an http:// URL through the proxy with GET.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`Post`

func Post(c: Client, url: str, content_type: str, body: str): Response

Sends a POST with a content type and a body through the proxy.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`NewServerConfig`

func NewServerConfig(port: int): ServerConfig

Returns a config for a port with the defaults above.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`AddUser`

func AddUser(cfg &: ServerConfig, user: str, pass: str)

Adds a user and password and turns the login on.

The example for this item is shown with the ones above: net-httpproxy-Auth.salam

`SetAuth`

func SetAuth(cfg &: ServerConfig, check: func(str, str): bool)

Sets a custom login check and turns the login on.

The example for this item is shown with the ones above: net-httpproxy-Auth.salam

`SetConnector`

func SetConnector(cfg &: ServerConfig, connect: func(str, int): socks.Outbound)

Sets the callback that reaches targets. Use it to route by name or to refuse hosts.

The example for this item is shown with the ones above: net-httpproxy-Policy.salam

`SetPolicy`

func SetPolicy(cfg &: ServerConfig, p: tunnel.DestPolicy)

Sets the destination policy of the server.

`AllowPrivate`

func AllowPrivate(cfg &: ServerConfig, enabled: bool)

Lets the server reach loopback and private addresses. It only takes effect when the server is bound to loopback; on any other address it is ignored.

`SetTLS`

func SetTLS(cfg &: ServerConfig, cert_file: str, key_file: str): bool

Loads a certificate and key and makes the proxy itself speak TLS (an HTTPS proxy). False when the files cannot be used.

net-httpproxy-Tls.salam

import net.httpproxy

func main:
    mut config := httpproxy.NewServerConfig(8443)
    httpproxy.AddUser(config, "alice", "a long random password")
    if not httpproxy.SetTLS(config, "proxy.crt", "proxy.key"):
        println "certificate or key could not be loaded"
        ret
    end
    println config.use_tls
    server := httpproxy.Start(config)
    println "https proxy up:", server.ok

    mut client := httpproxy.NewClientAuth("proxy.example.com", 8443, "alice", "a long random password")
    client.tls = true
    client.server_name = "proxy.example.com"
    client.insecure = false
    tunnel := httpproxy.Connect(client, "example.com", 443)
    println tunnel.ok, tunnel.status
    httpproxy.Stop(server)
end

`Handle`

func Handle(cfg: ServerConfig, s: dyn tunnel.Stream)

Serves one client on an open stream: reads requests, forwards them or opens tunnels, until the connection ends.

net-httpproxy-Handle.salam

import net.httpproxy
import net.tcp
import net.tunnel
import sync

mut proxy_port := 0

mut origin_port := 0
mut echo_port := 0
mut ready := sync.NewWaitGroup()

func origin():
    srv := tcp.Bind(0)
    origin_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        request := tcp.Read(c, 4096)
        first := request.substr(0, request.find("\r\n"))
        body := "origin saw: " + first
        tcp.Write(c, "HTTP/1.1 200 OK\r\nContent-Length: " + body.len() + "\r\nConnection: close\r\n\r\n" + body)
        tcp.Close(c)
    end
end

func echo():
    srv := tcp.Bind(0)
    echo_port = tcp.Port(srv)
    sync.Done(ready)
    until true:
        c := tcp.Accept(srv)
        tcp.Write(c, "echo:" + tcp.Read(c, 100))
        tcp.Close(c)
    end
end

func start_backends():
    sync.Add(ready, 2 as i64)
    _a := spawn(origin)
    _b := spawn(echo)
    sync.Wait(ready)
end

func serve_one():
    listener := tcp.Bind(0)
    proxy_port = tcp.Port(listener)
    sync.Done(ready)
    c := tcp.Accept(listener)
    httpproxy.Handle(httpproxy.NewServerConfig(0), tunnel.WrapTcp(c))
end

func main:
    start_backends()
    sync.Add(ready, 1 as i64)
    _t := spawn(serve_one)
    sync.Wait(ready)
    client := httpproxy.NewClient("127.0.0.1", proxy_port)
    println httpproxy.Get(client, "http://127.0.0.1:" + origin_port + "/by-hand").body
end

Outputorigin saw: GET /by-hand HTTP/1.1

`Start`

func Start(cfg: ServerConfig): Server

Binds the port and serves clients on background threads. Check ok.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`Stop`

func Stop(sv: Server)

Stops accepting new clients. Connections already running finish on their own.

The example for this item is shown with the ones above: net-httpproxy-Get.salam

`Serve`

func Serve(cfg: ServerConfig): bool

Starts a server and then never returns, for a program whose only job is to be the proxy. False when the port could not be bound.

`NewInbound`

func NewInbound(cfg: ServerConfig): HttpInbound

Makes an HTTP proxy inbound from a server config.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`NewConnector`

func NewConnector(c: Client): HttpConnector

Makes an HttpConnector from a client. Use it with tunnel.Via or tunnel.ChainAdd.

The example for this item is shown with the ones above: net-httpproxy-Inbound.salam

`SetDialer`

func SetDialer(cfg &: ServerConfig, d: dyn tunnel.Dialer)

Makes the server reach targets through a tunnel.Dialer, so it can use rules, chains or a block list. It replaces the connect callback.

The example for this item is shown with the ones above: net-httpproxy-Policy.salam