Lexical search over notes, without a vector store.
A backend finds candidates and a raw rank. This module then scores them the same way for every backend, so ranking does not depend on which one answered.
Backends
The backend follows the storage backend, and details of memory_search names the one that
answered:
| Storage | Backend | Rank |
|---|---|---|
| SQLite | SuperPi.Memory.Search.FTS5 (fts5) | bm25() with title 10, topics 5, body 1 |
| PostgreSQL | SuperPi.Memory.Search.Tsvector (tsvector) | ts_rank_cd on the generated column |
| any other, and tests that ask for it | SuperPi.Memory.Search.Like (like) | weighted word hits |
A backend that errors (a missing index table, say) does not fail the search: it falls back to
Like and the result says degraded: true. A search with no words and only a topic filter uses
Like by design and is not degraded. Set config :super_pi, :memory, search: Module to force a backend.
The query is split into words. A note matches when it has any word, and a note with more words
ranks higher. FTS5 and tsvector get quoted words, never raw query syntax, so a model cannot
break the query.
Score
score = text + recency_weight * 0.5^(age_days / half_life_days) + (pinned ? pinned_bonus : 0) - superseded penaltytext is the raw rank divided by the best raw rank of the candidates, in (0, 1]. BM25 is near
zero on a tiny corpus (every word is in most notes), and relative rank keeps the order. A
superseded note takes a penalty of 10, which is above any other term. Then apply_supersession/1
clamps a superseded note below its superseder when both are in the result, so a corrected fact
always outranks the fact it corrected, also along a chain.
Summary
Callbacks
Candidate ids with a raw rank (higher is better). topic is a normalized topic or nil.
Backend name for details.
Functions
Clamps every superseded entry below its superseder when both are present. Repeats until stable, so a chain A < B < C ends ordered C, B, A. The number of passes is bounded by the number of entries, which also stops a corrupt cycle.
The backend module for the configured storage.
Escapes %, _ and the escape character \ for a LIKE with ESCAPE '\'.
The query words that reach a backend: at most 12.
Scores and orders entries %{memory, text} at time now. Pure: it reads only its arguments and
the :memory configuration. Each result gains :recency and :score.
Searches a scope. Options: :limit (default 8, at most 25) and :topic.
A LIKE pattern for a topic: matches the topic as a whole word of the padded topics column.
Words of a text: lowercase letters and digits, no duplicates, in order.
Callbacks
@callback candidates( scope :: SuperPi.Memory.scope(), words :: [String.t()], topic :: String.t() | nil, limit :: pos_integer() ) :: {:ok, [{integer(), number()}]} | {:error, term()}
Candidate ids with a raw rank (higher is better). topic is a normalized topic or nil.
@callback name() :: String.t()
Backend name for details.
Functions
Clamps every superseded entry below its superseder when both are present. Repeats until stable, so a chain A < B < C ends ordered C, B, A. The number of passes is bounded by the number of entries, which also stops a corrupt cycle.
@spec backend() :: module()
The backend module for the configured storage.
Escapes %, _ and the escape character \ for a LIKE with ESCAPE '\'.
The query words that reach a backend: at most 12.
@spec rank([map()], DateTime.t()) :: [map()]
Scores and orders entries %{memory, text} at time now. Pure: it reads only its arguments and
the :memory configuration. Each result gains :recency and :score.
@spec run(SuperPi.Memory.scope(), String.t(), keyword()) :: {:ok, map()} | {:error, :empty_query}
Searches a scope. Options: :limit (default 8, at most 25) and :topic.
Returns {:ok, %{backend: name, degraded: boolean, words: [..], results: [%{memory, text, recency, score}]}}
(best first) or {:error, :empty_query}.
A LIKE pattern for a topic: matches the topic as a whole word of the padded topics column.
Words of a text: lowercase letters and digits, no duplicates, in order.