94 lines
4.5 KiB
Markdown
94 lines
4.5 KiB
Markdown
# QuickSearch design notes
|
|
|
|
A brief record of the design choices and the reasoning behind them. The code
|
|
and its comments specify exact behavior.
|
|
|
|
## Shape of the codebase
|
|
|
|
`quicksearch-core` is the library (indexing, storage, search), and
|
|
`quicksearch-gui` is the app (an immediate-mode egui desktop). On Linux the
|
|
app binary also searches from a terminal; on Windows the separate
|
|
`quicksearch-cli` does. The backend is synchronous Rust: plain threads and
|
|
mpsc channels, no async runtime, because a small, fixed set of long-lived
|
|
workers is simpler as threads.
|
|
|
|
## Optimization
|
|
|
|
Performance is measured, not guessed: a benchmark suite tracks CPU time, peak
|
|
memory, memory churn, and indexing and search speed.
|
|
|
|
## Storage
|
|
|
|
The index is a single SQLite database: file metadata, an FTS5 full-text index
|
|
over document contents, and the extracted text itself, stored zstd-compressed
|
|
for snippets, ranking and verification. The database runs in WAL mode with exactly one
|
|
writer at a time, so read-only searches are never blocked by indexing. Schema
|
|
changes wipe and rebuild the index rather than migrating: the index is a cache
|
|
of the filesystem, so rebuilding is always safe, and only the indexer may do
|
|
it. FTS5 runs in trigram mode for full-text search and also speeds up regex
|
|
and fuzzy searches. By default it records no word positions: every candidate
|
|
is checked against the stored text anyway, so positions only narrow the
|
|
candidates, and dropping them makes the index several times smaller and
|
|
faster to build, for content searches up to two or three times slower. A
|
|
setting trades the other way. Tuning this database is central to QuickSearch's
|
|
performance.
|
|
|
|
## The search path never stalls
|
|
|
|
Searches answer from the index and never wait on the filesystem or the
|
|
indexer. Each keystroke cancels the previous query; results stream out in rank
|
|
order and weaker matches only append, so with the default rank sort nothing
|
|
you have seen reshuffles. A small watcher then checks the visible rows against
|
|
the disk and feeds any corrections to the indexer, which keeps results honest
|
|
even when indexing is stopped or periodic.
|
|
|
|
## Indexing
|
|
|
|
Each indexed root gets its own directory walker and content-extraction pool,
|
|
but all database writes go through one thread and one connection. That writer
|
|
serves the walkers first and slices extraction work, so one root's large
|
|
documents never stall another root's progress. Changing what is indexed does
|
|
not throw the index away: the configuration difference becomes a plan to
|
|
delete what fell out of scope and walk what is newly covered, applied while
|
|
searching and indexing keep running.
|
|
|
|
## Freshness
|
|
|
|
Filesystem watchers turn changes into small incremental updates. When roots
|
|
are too large for watchers to scale, a full reindex on a configurable interval
|
|
is the backstop. There is no background daemon and no "start with your
|
|
session" option: QuickSearch starts and indexes in moments, and a search tool
|
|
has no business running when it is not being used.
|
|
|
|
## Security
|
|
|
|
QuickSearch has no more permissions than any other user program, but by
|
|
gathering what it can read into one searchable place it concentrates the risk
|
|
of data theft. So the index can be encrypted on disk with SQLCipher, keyed
|
|
from a password through Argon2id; the OS keychain can remember the key.
|
|
SQLCipher's per-page HMAC is off: it only detects tampering (not reading) by
|
|
someone who could already read the indexed files, and it costs significant
|
|
search performance. Encryption protects against a stolen disk, a synced backup
|
|
or a copied index, not against software running as the user, which could read
|
|
the original files anyway.
|
|
|
|
## Deliberate omissions
|
|
|
|
- Files whose names are not valid UTF-8 are silently skipped. A lossily
|
|
converted path must never become a database key, and the corner case does
|
|
not justify UI.
|
|
- Duplicate groups are a strong suspicion, not a certainty: grouping hashes
|
|
only each file's size and head, which the walk reads anyway to detect the
|
|
file's type, so it costs little. The Duplicates tab compares files byte for
|
|
byte on request and never deletes or modifies them; it tells you where to
|
|
look.
|
|
- Only one instance may open an index for writing. A second launch raises the
|
|
running window instead, and refuses if it cannot rather than risk
|
|
corruption. The guard is a kernel lock, so a crash never leaves you locked
|
|
out.
|
|
|
|
## Where the detail lives
|
|
|
|
Tests, benchmarks and profiling probes live in each crate (see `cargo test`
|
|
and `cargo bench` in the crate directories). Release and CI mechanics are
|
|
documented in `.forgejo/workflows/ci.yml` and the scripts under `packaging/`.
|