Search with Typesense
Most applications need a search box, and a good one forgives typos, ranks the best match first and answers in milliseconds. Typesense is an open source search engine built for exactly that, and the db.typesense package lets a Salam program use all of it. This lesson builds a small book catalog from an empty server to a scoped key a web page can use.
Running a server
Typesense is a separate program that keeps your data in its own index. The quickest way to start one on your machine is Docker. The --api-key value is the admin key your program sends with every request; use your own secret instead of xyz outside your own computer.
docker run -p 8108:8108 typesense/typesense:30.2 --data-dir /tmp --api-key=xyzThe examples in this lesson talk to that server, so they are checked by the compiler but not run when this site is built. The output under each one is what it printed against a real Typesense 30.2 server, in the order of the lesson: each example builds on the data the earlier ones left behind.
Connecting
typesense.New takes the server address and the API key and gives back a Client. Nothing is sent yet; every function that talks to the server takes the client first and an Error last. When a call fails the error is filled in: Ok tells you whether it worked, ErrorString gives one readable line, and IsNotFound and IsConflict answer the two questions code asks most often. Release the client with Free when you are done.
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
if not typesense.Health(c, err):
println "Typesense is not ready:", typesense.ErrorString(err)
ret 1
end
println "connected, server", typesense.ServerVersion(c, err)
_ := typesense.GetCollection(c, "no_such_collection", err)
if not typesense.Ok(err):
println "status", err.status, "not found:", typesense.IsNotFound(err)
println typesense.ErrorString(err)
end
endThe error has three parts you can use: status is the HTTP status (0 when the server could not be reached at all), kind says what kind of failure it was, and message is the text the server sent.
Collections and fields
Documents live in a collection, and the collection's schema says which fields exist and what type each one has. NewField makes a plain field, FacetField one you can count values of, and a Field literal lets you set everything else: optional for fields some documents leave out, sort, locale for the language of the text, infix for matches in the middle of a word, and more. The types are constants such as TYPE_STRING, TYPE_INT32, TYPE_FLOAT, TYPE_STRING_ARRAY and TYPE_GEOPOINT.
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
_ := typesense.DeleteCollection(c, "lesson_books", err)
mut s := typesense.Schema { name = "lesson_books", default_sorting_field = "year" }
s.fields.push(typesense.NewField("title", typesense.TYPE_STRING))
s.fields.push(typesense.NewField("author", typesense.TYPE_STRING))
s.fields.push(typesense.FacetField("genre", typesense.TYPE_STRING))
s.fields.push(typesense.NewField("year", typesense.TYPE_INT32))
s.fields.push(typesense.Field { name = "rating", kind = typesense.TYPE_FLOAT, optional = true })
col := typesense.CreateCollection(c, s, err)
if not typesense.Ok(err):
println typesense.ErrorString(err)
ret 1
end
println "created", col.name, "with", col.fields.len(), "fields"
each f in col.fields:
println " ", f.name, f.kind, "facet:", f.facet
end
endA numeric default_sorting_field decides the order when a search does not ask for one. Change a collection later with AddFields and DropFields, look at it with GetCollection and ListCollections, and remove it with DeleteCollection. To rebuild a collection with no downtime, build the new one next to the old one and move an alias with UpsertAlias; programs that search the alias never notice the switch.
Adding documents
A document is a JSON object whose fields match the schema. You can write that JSON yourself, but usually your program already has the data in a struct. ImportOf takes a Vector of any struct, turns each one into JSON with the compiler's own encoder and sends them all in one request. GetDocumentAs does the opposite and reads a stored document back into your struct.
import db.typesense
struct Book:
pub id: str = ""
pub title: str = ""
pub author: str = ""
pub genre: str = ""
pub year: int = 0
end
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
mut books := Vector {} as Vector<Book>
books.push(Book { id = "1", title = "Dune", author = "Frank Herbert", genre = "scifi", year = 1965 })
books.push(Book { id = "2", title = "Dune Messiah", author = "Frank Herbert", genre = "scifi", year = 1969 })
books.push(Book { id = "3", title = "Emma", author = "Jane Austen", genre = "novel", year = 1815 })
books.push(Book { id = "4", title = "Foundation", author = "Isaac Asimov", genre = "scifi", year = 1951 })
books.push(Book { id = "5", title = "Persuasion", author = "Jane Austen", genre = "novel", year = 1817 })
r := typesense.ImportOf(c, "lesson_books", books, typesense.WriteOptions { action = typesense.ACTION_UPSERT }, err)
println "imported", r.success, "failed", r.failed
_ := typesense.CreateDocument(c, "lesson_books", `{"id": "6", "title": "Walden", "author": "Henry Thoreau", "genre": "essay", "year": "1854"}`, err)
println "wrong type:", typesense.ErrorString(err)
println typesense.UpsertDocument(c, "lesson_books", `{"id": "6", "title": "Walden", "author": "Henry Thoreau", "genre": "essay", "year": 1854}`, err)
mut b := Book {}
if typesense.GetDocumentAs(c, "lesson_books", "1", b, err):
println "read back:", b.title, "by", b.author
end
println typesense.UpdateDocument(c, "lesson_books", "6", `{"rating": 4.2}`, err)
endAn import answers even when some documents fail, so check failed; errors lists every failed line with its reason. WriteOptions chooses what happens to existing documents: ACTION_CREATE refuses them, ACTION_UPSERT replaces them, ACTION_UPDATE and ACTION_EMPLACE change only the fields you send. For one document at a time there are CreateDocument, UpsertDocument, UpdateDocument and DeleteDocument, and DeleteByFilter and UpdateByFilter change many at once.
Searching
NewSearch(q, query_by) makes the parameters for a search: the text to look for and the fields to look in. * as the text matches every document, which is useful together with a filter. The result has found, the number of matches, and hits, one per document on this page; hit.document is the document as JSON.
import db.typesense
func show(label: str, r: typesense.SearchResult):
println label, "found", r.found
each h in r.hits:
println " ", h.document
end
end
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
show("dune:", typesense.Search(c, "lesson_books", typesense.NewSearch("dune", "title"), err))
show("typo:", typesense.Search(c, "lesson_books", typesense.NewSearch("persuasoin", "title"), err))
mut p := typesense.NewSearch("*", "")
p.filter_by = "genre:=scifi && year:<1966"
p.sort_by = "year:desc"
p.include_fields = "title,year"
show("filtered:", typesense.Search(c, "lesson_books", p, err))
mut h := typesense.NewSearch("austen", "author,title")
h.per_page = 1
r := typesense.Search(c, "lesson_books", h, err)
println "page", r.page, "of", r.found, "results, highlight:", typesense.Highlight(r.hits.get(0), "author")
endThree things happened that you did not have to ask for. persuasoin still found Persuasion, because Typesense forgives a typo or two in each word. The best match came first. And Highlight shows which part of a field matched. The other parameters are fields on SearchParams: filter_by and sort_by as in the example, page and per_page for paging, include_fields to return less, num_typos to be stricter. A parameter without a field of its own goes through typesense.Set(p, name, value).
| You want | Write |
|---|---|
| Only some documents | p.filter_by = "year:>1950 && genre:=[scifi, novel]" |
| A different order | p.sort_by = "year:desc" |
| Page three, ten per page | p.page = 3 and p.per_page = 10 |
| No typos at all | p.num_typos = "0" |
| Your structs instead of JSON | typesense.HitsAs(result, books, err) |
Facets and groups
A shop search usually shows a sidebar: how many results are shoes and how many are shirts, how many are under 100. Those counts are facets. Name the facet fields in facet_by and every result carries a Facet for each one, with the values and how many hits have them. group_by answers a different question: it keeps the best few hits for each value, here the oldest book of every genre.
import str
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
mut p := typesense.NewSearch("*", "")
p.facet_by = "genre"
r := typesense.Search(c, "lesson_books", p, err)
each f in r.facets:
each fc in f.counts:
println f.field_name, fc.value, fc.count
end
end
mut g := typesense.NewSearch("*", "")
g.group_by = "genre"
g.group_limit = 1
g.sort_by = "year:asc"
rg := typesense.Search(c, "lesson_books", g, err)
each grp in rg.groups:
println "oldest", str.Join(grp.key, ""), "book:", grp.hits.get(0).document
end
endMany searches in one request
A page that searches books and authors at the same time does not need two round trips. MultiSearch sends a list of SearchRequests in one request and returns one result per search; each result has its own code and error, so one bad search does not spoil the others. The parameters you pass as the common ones apply to all of them. UnionSearch goes one step further and merges the hits of all the searches into a single ranked list, and each hit says which search found it.
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
mut searches := Vector {} as Vector<typesense.SearchRequest>
searches.push(typesense.SearchRequest { collection = "lesson_books", params = typesense.NewSearch("dune", "title") })
searches.push(typesense.SearchRequest { collection = "lesson_books", params = typesense.NewSearch("austen", "author") })
common := typesense.SearchParams { per_page = 3 }
each r in typesense.MultiSearch(c, searches, common, err):
println "search found", r.found
end
u := typesense.UnionSearch(c, searches, common, err)
println "union found", u.found
each h in u.hits:
println " from search", h.search_index, h.document
end
endThe union matched four documents but returned three, because the common parameters asked for three per page.
Synonyms and curation
Search engines only know the words in your documents. A synonym set teaches them that some words mean the same thing, so a search for sf also finds books whose genre is scifi. A curation set is for the queries you care about most: when a query matches a rule, the rule can pin documents to positions, hide others, add a filter or change the sort. Both kinds of set are created once and then attached to any collection that should use them.
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
mut words := Vector {} as Vector<str>
words.push("scifi")
words.push("science fiction")
words.push("sf")
mut syns := Vector {} as Vector<typesense.Synonym>
syns.push(typesense.Synonym { id = "scifi", synonyms = words })
_ := typesense.UpsertSynonymSet(c, "lesson_words", syns, err)
mut pins := Vector {} as Vector<typesense.Pin>
pins.push(typesense.Pin { id = "4", position = 1 })
mut rules := Vector {} as Vector<typesense.CurationRule>
rules.push(typesense.CurationRule { id = "classic-first", query = "dune", match_type = typesense.MATCH_EXACT, includes = pins })
_ := typesense.UpsertCurationSet(c, "lesson_rules", rules, err)
_ := typesense.UpdateCollection(c, "lesson_books", `{"synonym_sets": ["lesson_words"], "curation_sets": ["lesson_rules"]}`, err)
println "sf finds", typesense.Search(c, "lesson_books", typesense.NewSearch("sf", "genre"), err).found, "books"
r := typesense.Search(c, "lesson_books", typesense.NewSearch("dune", "title"), err)
println "first hit:", r.hits.get(0).document, "curated:", r.hits.get(0).curated
endThe curated flag tells your page that a hit was placed by a rule, for example to label it as promoted. A search can turn the rules off with typesense.Set(p, "enable_curations", "false"). Typesense 29 and older keep synonyms and rules per collection instead of in sets; for those servers use UpsertSynonym and UpsertOverride.
A key for the browser
The admin key can do anything, so it must never reach a web page. SearchOnlyKey makes a key that may only search the collections you name. A scoped key goes further: ScopedSearchKey signs a search key together with search parameters, such as a filter that keeps every user to their own data. It is made in your program with no request to the server, and the browser cannot remove the filter, because changing it breaks the signature.
import db.typesense
func main:
mut admin := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(admin)
mut err := typesense.Error {}
mut cols := Vector {} as Vector<str>
cols.push("lesson_books")
k := typesense.SearchOnlyKey(admin, "website search", cols, err)
println "search key made, may:", k.actions.get(0)
scoped := typesense.ScopedSearchKey(k.value, typesense.SearchParams { filter_by = "genre:=novel" }, 0)
mut browser := typesense.New("http://localhost:8108", scoped)
defer typesense.Free(browser)
println "the browser sees", typesense.Search(browser, "lesson_books", typesense.NewSearch("*", ""), err).found, "books"
_ := typesense.DeleteDocument(browser, "lesson_books", "1", err)
println "and cannot delete:", err.status
_ := typesense.DeleteKey(admin, k.id, err)
endGive the scoped key an expiry time (the last argument, a Unix time) when it is handed out per session, and delete keys you no longer need with DeleteKey.
Vectors and places
Words are not the only way to find things. A TYPE_GEOPOINT field holds a latitude and longitude, and a filter such as location:(35.70, 51.42, 2 km) keeps only what is near a point, while sorting by the same point puts the nearest first. A vector field (VectorField) holds a list of numbers that describes a document, usually made by a machine learning model, and vector_query finds the documents whose vectors are closest to yours. With EmbeddingField, Typesense makes the vectors itself from the text of other fields.
import db.typesense
func main:
mut c := typesense.New("http://localhost:8108", "xyz")
defer typesense.Free(c)
mut err := typesense.Error {}
_ := typesense.DeleteCollection(c, "lesson_places", err)
mut s := typesense.Schema { name = "lesson_places" }
s.fields.push(typesense.NewField("name", typesense.TYPE_STRING))
s.fields.push(typesense.NewField("location", typesense.TYPE_GEOPOINT))
s.fields.push(typesense.VectorField("style", 2))
_ := typesense.CreateCollection(c, s, err)
mut docs := Vector {} as Vector<str>
docs.push(`{"name": "Quiet library cafe", "location": [35.7019, 51.4223], "style": [0.9, 0.1]}`)
docs.push(`{"name": "Loud sports bar", "location": [35.6990, 51.4180], "style": [0.1, 0.9]}`)
docs.push(`{"name": "Garden tea house", "location": [35.8040, 51.4330], "style": [0.8, 0.2]}`)
_ := typesense.ImportDocuments(c, "lesson_places", docs, typesense.WriteOptions {}, err)
mut near := typesense.NewSearch("*", "")
near.filter_by = "location:(35.7000, 51.4200, 2 km)"
near.sort_by = "location(35.7000, 51.4200):asc"
near.include_fields = "name"
each h in typesense.Search(c, "lesson_places", near, err).hits:
println "near:", h.document, h.geo_distance_meters
end
mut like := typesense.NewSearch("*", "")
like.vector_query = "style:([1.0, 0.0], k: 2)"
like.include_fields = "name"
each h in typesense.Search(c, "lesson_places", like, err).hits:
println "calm:", h.document
end
_ := typesense.DeleteCollection(c, "lesson_places", err)
endRunning it for real
- A Typesense cluster has several nodes. Give
NewClusterall of their addresses: a node that does not answer is skipped forhealthcheck_msand the request moves on to the next.SetNearestNodenames one to try first. - Tune the client with its fields:
timeout_msfor a single request,retriesandretry_interval_msfor how hard it tries. - Use an alias in your searches, not a collection name, so you can rebuild the collection without downtime.
- Back up with
CreateSnapshot, watch the server withHealth,StatsandMetrics, and count clicks or popular queries with analytics rules (CreateAnalyticsRuleandSendEvent). - For an endpoint this package has no function for,
Requestsends any method and path with a JSON body.
Every function, type and field is listed with its own example on the db/typesense reference page. The repository also has a complete, tested program for each feature in tests/en/db, in the files that start with tslive_ex_.