jeremy/quick_search
jeremy/quick_search
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages Wiki Activity Actions

Release v1.1: live results, duplicate verification, and a Settings tab.
Some checks failed
CI / linux (push) Failing after 5m46s
Details
CI / windows-cross (push) Successful in 5m40s
Details
CI / release (push) Has been skipped
Details

Browse source 
Search results now track the disk as it changes (live.rs), duplicate
groups can be confirmed byte-for-byte before deletion (verify.rs), the
Options window becomes a Settings tab with configurable/sortable result
columns, and a first-start tutorial explains the basics.
This commit is contained in:
= 2026-08-17 19:26:18 -04:00
parent 03d0b6a50f
commit 9760503a3f
76 changed files with 9666 additions and 1414 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.forgejo/workflows/ci.yml
View file

@ -63,12 +63,6 @@ jobs:
HOME: /root
# The highest libc6 version the .deb is allowed to require.
MAX_GLIBC: '2.35'
# full_index.rs asserts a heavy indexing root cannot stall a light one, and
# measures that as wall-clock stall. The 100 ms default is calibrated on a
# developer machine; this runner measured 188 ms for the same correct
# behaviour. 600 ms keeps the check meaningful - the regression it exists to
# catch is ~6x the healthy figure, so it would land near 1.2 s here.
QSB_STALL_BUDGET_MS: '600'
steps:
- uses: actions/checkout@v4

4
Cargo.lock generated
View file

@ -3188,7 +3188,7 @@ dependencies = [
[[package]]
name = "quicksearch-core"
version = "1.0.6"
version = "1.1.0"
dependencies = [
"argon2",
"cfb",
@ -3222,7 +3222,7 @@ dependencies = [
[[package]]
name = "quicksearch-gui"
version = "1.0.6"
version = "1.1.0"
dependencies = [
"ashpd",
"chrono",

2
Cargo.toml
View file

@ -25,7 +25,7 @@ members = [
# into a killed process instead of one skipped file.
[workspace.package]
version = "1.0.6"
version = "1.1.0"
edition = "2021"
license = "GPL-3.0-or-later"
authors = ["Jeremy <jeremy@karsttech.com>"]

158
README.md
View file

@ -187,7 +187,9 @@ The install is per-machine and asks for elevation. Into
| `uninstall.exe` | written by the installer; Add/Remove Programs runs it |
The components page offers a Start menu shortcut (on) and a desktop shortcut
(off); both are created for all users. No `config.toml` is installed, for the
(off); both are created for all users. The final page lists what was installed
and where — the install itself takes about a second, which without saying so
reads as a failure — and offers to start QuickSearch, ticked. No `config.toml` is installed, for the
same reason the `.deb` ships none — one next to the binaries is portable mode
(see [Configuration](#configuration)) and would override the personal config
of every account. The app writes `%APPDATA%\quicksearch\config.toml` on first
@ -224,12 +226,32 @@ inside that folder.
`quicksearch` with no query arguments opens the app:
- **Search**: results appear as you type; every keystroke cancels the
previous search. One checkbox enables the two fuzzy passes. Sort by
rank, name, path, size, or modified. Double-click a result to open it;
previous search. One checkbox enables the two fuzzy passes, and once a
search has finished a button inside the right of the search box re-runs
it. Click a column heading to sort by it; **right-click any heading to
choose which columns are shown** — the path is always there, and size
and modified date start hidden, which is what buys the width the path
and the match get instead. The choice is saved (`[search.columns]`,
also in Settings → Search) and applies immediately. Sorting by a column
you then hide falls back to rank. Double-click a result to open it;
right-click it to reveal it in the file manager, open it, copy its
path, or build an ignore filter from it (session-only by default,
optionally persisted to the config). Result text can be selected and
copied in place. Matches in file contents show highlighted snippets.
copied in place. A match in a file's **name** or **path** is
highlighted in that column; a match in its **contents** shows a
highlighted snippet in the Content Match column, with more of the
surrounding text on hover. Rows matched on name or path show a dash
there instead. With `[search] live_results` on (the default) the rows
actually on screen are watched, and what they show is read from the
files themselves: a rename, a deletion or an edit lands within a
second, whether or not indexing is running. The rows coming on screen
are also checked against the disk as they are watched, so one the index
was already out of date about corrects itself; the index is then
brought back in line for those files alone. Over a network share, where
the system reports no events, that check is all you get — the row is
right when it comes on screen and then holds still. Nothing is ever
added, removed or re-ordered underneath you; a file that disappears is
struck through where it sits. Editing the query drops every watch.
- **Manage Index**: full indexing status, Start/Stop/Automatic controls,
indexed folder list, full-text extension filters, ignore patterns, and
the indexing options. Stopping switches to manual mode and saves that
@ -244,25 +266,45 @@ inside that folder.
folder nothing has finished indexing reads "not yet indexed" rather
than zero, and because the figures come from completed runs they do
not move as live updates apply single changes in between.
- **Duplicates**: files sharing a content hash, grouped.
- **Duplicates**: files sharing a content hash, grouped. That hash covers
each file's size and its first `processing.hash_length` bytes and nothing
else, which is the whole reason indexing is affordable — and the reason a
group is a strong suspicion rather than a fact. Right-click a group, or any
file in one, to settle it: every member is read through and compared byte
for byte, with progress and a Cancel button in a modal that then names each
file as identical, differing at a given byte, a different size, or
unreadable. Nothing is deleted or changed either way; the point is to know
before you delete something yourself.
- **Logs**: the lines the app would have printed to a terminal — warnings
from indexing, folder watching and opening files, newest last, with a
filter box and Copy button. Launched from a desktop launcher (or on
Windows, where the app has no console at all) this is the only place
they are visible.
- **Help**: an in-app quickstart — first indexing run, example queries,
what each tab does — pointing here for everything technical.
what each tab does — pointing here for everything technical. A brand-new
installation is shown a short click-through introduction covering the
same ground on its first launch; the Help tab brings it back. Upgrading
into this version does not raise it (see `[ui] tutorial_seen`).
- **Settings**: every configuration control in one place — the database
path, indexing and processing limits, search behaviour, the interface
(scale, shortcut, color scheme) and password protection. Each row
explains itself on hover. Edits are staged and applied together by
**Apply & Save**; leaving the tab with unapplied edits asks first. The
column choices and the password controls are the exceptions, acting the
moment they are used, since the Search tab's own header menu writes the
same settings. The indexed folder list and the indexing mode live on
Manage Index instead, next to the controls that act on them.
**Ctrl+Shift+F from anywhere** brings QuickSearch to the front, restoring
it if it was minimized, and puts the cursor in the search box with the
previous search selected, so the next thing you type is the new one. The
Options window's Interface section rebinds it — click the button and press
Settings tab's Interface section rebinds it — click the button and press
the keys — or switches it off. It is a system-wide shortcut, registered
with Windows or with the X server, so it works while another application
has focus. Wayland does not let an application claim a key, so there the
shortcut is registered with your desktop through the XDG desktop portal
instead; your desktop then has the final say over which key it is, and its
own keyboard settings are where to change it. The Options window says which
own keyboard settings are where to change it. The Settings tab says which
key it settled on. Wayland likewise gives no application a way to put itself
in front of what you are doing, so under it the shortcut selects the Search
tab and the search box but leaves raising the window to the desktop; on X11
@ -301,7 +343,7 @@ processing; Windows Terminal has it, and older consoles get plain text.
The index contains the names and (by default) the full text of everything
it indexes — for most setups, your entire home directory. That is a lot of
concentrated risk in one file. **Options → Security → Enable password
concentrated risk in one file. **Settings → Security → Enable password
protection** encrypts the index on disk with SQLCipher; from then on
QuickSearch asks for the password every time it starts, in the GUI (an
unlock screen before anything opens the index) and in the terminal (a
@ -315,6 +357,10 @@ rebuilds the index — there is no in-place conversion.
in the OS keychain — Secret Service/KWallet on Linux, Credential Manager
on Windows — and skips the prompt. Without a keychain daemon the option
quietly falls back to prompting.
- **Show database key** asks for the password, then shows the raw SQLCipher
key as `0x…` (64 hex digits) with a copy button, for opening the index in
other SQLCipher tools. That key alone reads the index, so treat a copy of
it as carefully as the password.
- Scripts can set `QUICKSEARCH_PASSWORD` for non-interactive terminal
search. Environment variables are readable by other processes of the
same user (`/proc/<pid>/environ`) — prefer the keychain.
@ -397,13 +443,13 @@ containing the binary, its config, and its index can be moved wholesale.
The GUI edits the config live; external edits apply on next start.
`[ui] search_hotkey` is the system-wide search shortcut, written the way
the Options window prints it (`Ctrl+Shift+F`): Ctrl, Alt and Shift in any
the Settings tab prints it (`Ctrl+Shift+F`): Ctrl, Alt and Shift in any
combination, plus one key, joined with `+`. An empty string switches it
off. A value that is not a shortcut is not a config error — the app loads,
says so in the Options window, and runs without one.
says so on the Settings tab, and runs without one.
`[ui] color_scheme` is `dark` (the default) or `light`, changeable in the
Options window and applied without a restart. It does not follow the
`[ui] color_scheme` is `dark` (the default) or `light`, changeable on the
Settings tab and applied without a restart. It does not follow the
desktop's own light/dark setting: on Linux nothing in the window system
reports that, so the only way to know is to connect to the session message
bus and subscribe to the user's settings feed — more of your session than a
@ -499,7 +545,25 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
bigger index — `indexing.content_extensions` remains the throttle. Files no larger than `processing.hash_length` skip that second
pass entirely: the head the walk reads to hash them is already their
whole content, so a plaintext body is extracted in the same `read` and
stored complete. Every run ends — whether
stored complete. Every root runs its own pipeline — its own walker pool
and, once the walk ends, its own extraction pool — but every root's
*writes* go through one thread and one connection
(`indexing/pipeline.rs`), because that is what a single SQLite file
allows. That thread is where FTS5 tokenizes, up to `maximum_text_size`
of text per document inside the insert, and it is the run's dominant
cost. So its loop is scheduled around the walk, the disk-bound phase and
the one whose stall shows: each round serves every walking root first,
then one extracting root, and no turn runs past a 100 ms slice — an
extraction turn commits at the slice and carries the rows it did not
reach to its next turn. A walk therefore waits at most one slice per
round, which its walkers' channel absorbs, so a root walking a large tree
runs at its own rate while another root tokenizes big documents beside
it. What a root has left to extract is counted by its own content pass,
on that pass's read connection, rather than on the writer: on a large
root the count is seconds, and seconds of writer time is every other
root's walk standing still. Total write throughput is what one connection
tokenizing can do; the scheduling shares it fairly and keeps the walk
first, it does not raise it. Every run ends — whether
it completed or was stopped — with an optimize pass on its own connection:
checkpoint, VACUUM if the file has at least 10% slack to reclaim, `PRAGMA
optimize`, checkpoint again. Progress streams through a polled
@ -555,6 +619,26 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
handle and takes a single watch per root, filtering the events instead.
Either way a tree too large to watch degrades to periodic reindexing
rather than going silently stale.
- **Live results** (`live.rs`): a second, much smaller watcher, owned by the
frontend rather than the coordinator, pointed at the parent directories of
the result rows *currently on screen* once they have held still for a
moment. It watches directories, not the result files: editors save by
writing a temporary file and renaming it over the target, so the event
lands on the directory and a watch on the file is left holding an orphaned
inode. What a row shows is read from the **file**, never from the index —
metadata from `stat`, and for a content match the same MIME sniffing and
extractors the indexer uses, re-cut through the same `cascade::text_snippet`
the search itself does. That is what makes it work with indexing stopped.
Arming also sweeps each target once against the size and modified time the
row is displaying, which on a fresh result is what the index said: so
bringing a row on screen *is* a check of the index against the disk, and it
is the only thing that reports anything where the platform sends no events.
It still writes nothing itself; the paths it has just read go to
`IndexCoordinator::update_paths`, which applies them on the coordinator's
own thread — in any mode, so a stopped index does not drift from the screen
— leaving the single-writer rule intact. Caps at 64 directories and 256
rows, rate-limited per path, and dropped wholesale the moment the query is
edited.
- **Search** (`search/`): `SearchService` runs one worker thread; each
query is a *generation*. New queries interrupt the in-flight SQLite
statement (`InterruptHandle`) and stale generations stop cooperatively,
@ -598,6 +682,21 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
case-insensitive literal branch used to allocate a lowercased copy of its
haystack, which the filename pass asked for twice per row of a full-table
scan.
- **Duplicate verification** (`verify.rs`): the second opinion on a group from
`search/duplicates.rs`, which groups by `sha256(size ‖ head)` and so cannot
tell two pre-allocated disk images apart — same size, same zeroes at the
front, everything that distinguishes them in a footer. One lockstep pass:
open every member, drop the ones whose length already disagrees without
reading them, then read a chunk from the first that opened and the same
span from each of the others, reporting the offset of the first byte that
differs and dropping that file from the walk. Deliberately not a hash —
"the same digest" is a probabilistic answer, and a probabilistic answer is
what the head hash already gave. The reference is the first member that
*opens*, so one unreadable file costs its own verdict and nobody else's,
and termination follows what that file actually reads rather than the
length it claimed, so a file truncated mid-run degrades to a short
comparison. The read buffers share a fixed 8 MiB between them however many
members a group has, because a hardlink farm's group runs to thousands.
- **Baloo compatibility** (`cli.rs`, `mime.rs`): the read API this repo's
parent consumes — `status_for_path`, `list_failed`,
`index_size_breakdown`, `pending_content_count`, `clear_path` — plus a
@ -630,14 +729,17 @@ core threads ─────────────▶ ctx.request_repaint() (w
```
Modules map one-to-one onto what you see: `app.rs` (shell and config
routing, with `app/` submodules for the status bar, the security flow and
the confirmation modals), `search_tab.rs` (query strip and virtualized
routing, with `app/` submodules for the status bar, the security flow, the
confirmation modals and the duplicate-verification modal — the one place a
worker's progress is shown in a window rather than the status bar),
`search_tab.rs` (query strip and virtualized
results table; snippet rendering via `LayoutJob` byte ranges, the ignore
dialog and the syntax help live in `search_tab/`), `manage_tab.rs` (status
detail + `tracker.rs` rate estimation, roots and filter editors),
`duplicates_tab.rs`, `logs_tab.rs` (a virtualized view of the core log
ring), `options.rs` (draft-based settings editor shared between the window
and the Manage tab), `platform.rs` (open / reveal-in-file-manager, and the
ring), `settings_tab.rs` (the draft-based config editor, the second of the
two tabs that stage their edits behind an Apply & Save), `platform.rs`
(open / reveal-in-file-manager, and the
Windows stdio setup a window-subsystem process needs before anything
prints), `hotkey/` (the system-wide search shortcut: one key table feeding
both a `RegisterHotKey` / `XGrabKey` registration and, on Wayland, an XDG
@ -650,12 +752,26 @@ microseconds regardless of row count.
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
ranking, cancellation, incremental indexing, coordinator modes, config
resolution, fuzzy matcher vs. brute-force oracle).
resolution, fuzzy matcher vs. brute-force oracle, `verify.rs`'s byte-for-byte
comparison — the shared-head-different-tail case the head hash cannot see, an
unreadable first member, a difference past the first chunk, cancellation —
and `live.rs`'s event
classification, where the platform-specific rename and atomic-save shapes are
synthesized rather than provoked, so they are checked on every platform).
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units plus
headless egui tests that drive the real widgets — building an input frame,
synthesizing clicks and reading back the painted text (`test_ui.rs`) — over
the search and manage tabs, the options editor, the unlock gate, the logs
and duplicates tabs, and query highlighting.
the search, manage and settings tabs, the unlock gate, the logs
and duplicates tabs, the first-start tour, and query highlighting. The search
tab's cover the column picker (including that the path column survives all
32 combinations of the others), which column a match is highlighted in, and
that the repeat-search button appearing inside the query box does not cost it
keyboard focus. The duplicates tab's open the real context menus and click
the entries inside them, so "the verification asks for the whole group, from
either menu, and not at all while one is running" is checked rather than
assumed; the verification modal is rendered in each of its states, and the
tour's footer is probed for where its three buttons actually landed rather
than for the numbers they were expected to land on.
- `cargo bench -p quicksearch-core --bench search` and `--bench index`: divan
microbenchmarks over the two hot paths. Each group runs *what the code does
today* against *the change being considered*, in one process on one corpus,

41
config_example.toml
View file

@ -133,7 +133,7 @@ store_text_for_snippets = true
[security]
# Encrypt the index with a password (SQLCipher). The password is asked
# for every time QuickSearch starts; turning this on or off deletes and
# rebuilds the index. Change it from the GUI (Options → Security), not by
# rebuilds the index. Change it from the GUI (Settings → Security), not by
# hand: enabling protection also generates the KDF salt below.
password_protected = false
# Store the derived key in the OS keychain (Secret Service / KWallet on
@ -163,12 +163,18 @@ watch_cap_warned_roots = []
# registered with your desktop, which may assign a different key and lets
# you change it in its own keyboard settings.
search_hotkey = "Ctrl+Shift+F"
# 'dark' or 'light'. Applied as soon as it is changed in the Options
# window. Your desktop's own light/dark setting is not consulted: reading
# 'dark' or 'light'. Applied as soon as it is changed on the Settings
# tab. Your desktop's own light/dark setting is not consulted: reading
# it would mean connecting to your session's message bus and subscribing to
# your settings, which is more than a search tool should ask for. Anything
# other than 'light' is dark.
color_scheme = "dark"
# Written by QuickSearch, not by you: whether the short introduction shown
# on a brand-new installation has been dismissed. Absent means this config
# predates that introduction - an installation that upgraded into this
# version, which is not offered it. The Help tab can show it again at any
# time.
tutorial_seen = false
[search]
# Start with the fuzzy passes enabled.
@ -185,3 +191,32 @@ display_limit = 1000
results_per_page = 100
# How long the GUI waits after the last keystroke before searching (ms).
debounce_ms = 150
# Watch the search results on screen and show renames, deletions and
# content changes as they happen. Only the rows actually visible are
# watched, and editing the query drops the watches. What a row shows is
# read from the file itself, so this works whether or not indexing is
# running — and the files it reads are then brought up to date in the
# index, so what is stored cannot drift from what you are looking at.
# Rows are also checked against the disk as they come on screen, which is
# all you get over a network share, where the system does not report other
# machines' writes. Nothing is ever added, removed or re-ordered while you
# read.
live_results = true
# Which columns the Search tab shows. The same choices are on the
# right-click menu of any column header, and in Settings → Search; both
# write here immediately, without an Apply.
#
# There is deliberately no 'path' key: the path is always shown, because
# it is the only column that identifies a result on its own.
[search.columns]
name = true
# The excerpt of a file's contents around the match. Rows that matched on
# their name or path show a dash there instead.
content_match = true
# Off by default: the width these take is usually better spent on the path
# and the matched text. Turning one on also makes it available to sort by;
# sorting by a column that is hidden falls back to sorting by rank.
size = false
modified = false
rank = true

1
crates/quicksearch-core/examples/walkprobe.rs
View file

@ -122,7 +122,6 @@ fn parallel(root: &str, config: &Config, db_path: &str) -> (usize, usize) {
config.clone(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
) {
let WalkEvent::File(file) = event else {

62
crates/quicksearch-core/src/config/mod.rs
View file

@ -169,6 +169,47 @@ pub struct SearchConfig {
pub results_per_page: usize,
/// How long the GUI waits after the last keystroke before searching.
pub debounce_ms: u64,
/// Watch the search results currently on screen and show renames,
/// deletions and content changes as they happen. Only the rows actually
/// visible are watched, and any edit to the query drops the watches.
/// What a row shows is read from the file itself, so this holds whether
/// or not indexing is running; the files it reads are then brought up to
/// date in the index, so what is stored cannot drift from what is on
/// screen. See [`crate::live`].
pub live_results: bool,
/// Which columns the Search tab shows.
pub columns: ColumnsConfig,
}
/// Which columns the Search tab shows, as picked from the right-click menu on
/// any column header or from the Settings tab.
///
/// The path column is deliberately not represented: it is always shown, so
/// "no columns at all" is not a state this can hold. Size and modified are off
/// by default — the width they cost is better spent on the path and the match,
/// and both are one click away.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(default)]
pub struct ColumnsConfig {
pub name: bool,
/// The matched excerpt from a file's contents. Rows that matched on their
/// name or path instead show a dash there.
pub content_match: bool,
pub size: bool,
pub modified: bool,
pub rank: bool,
}
impl Default for ColumnsConfig {
fn default() -> Self {
ColumnsConfig {
name: true,
content_match: true,
size: false,
modified: false,
rank: true,
}
}
}
impl SearchConfig {
@ -233,6 +274,8 @@ impl Default for SearchConfig {
display_limit: 1000,
results_per_page: 100,
debounce_ms: 150,
live_results: true,
columns: ColumnsConfig::default(),
}
}
}
@ -302,6 +345,21 @@ pub struct UiConfig {
/// recognises falls back to dark, where a typed-out enum would fail to
/// deserialize and take the whole config file down with it.
pub color_scheme: String,
/// Whether the first-start tour has been dismissed.
///
/// Three-valued on purpose. `None` means the key predates the tour — an
/// installation that upgraded into this version, which has already found
/// its way around — so only a config file this version *created* (which
/// gets `Some(false)` from [`UiConfig::default`]) is ever offered the tour.
/// A plain `bool` could not tell those apart.
///
/// The field-level `default` is load-bearing and not redundant with the
/// `#[serde(default)]` on the struct: that one fills a missing field from
/// `UiConfig::default()`, which says `Some(false)` — and would hand every
/// upgrading installation the tour. This one fills it from
/// `Option::default()`, which is `None`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tutorial_seen: Option<bool>,
}
impl Default for UiConfig {
@ -311,6 +369,10 @@ impl Default for UiConfig {
watch_cap_warned_roots: Vec::new(),
search_hotkey: "Ctrl+Shift+F".to_string(),
color_scheme: "dark".to_string(),
// Not `None`: a config built from these defaults is a config being
// written for the first time, and that is exactly who the tour is
// for. `None` is reserved for a file that predates the key.
tutorial_seen: Some(false),
}
}
}

58
crates/quicksearch-core/src/config/tests.rs
View file

@ -985,3 +985,61 @@ fn fuzzy_edits_warning_only_above_the_threshold() {
assert!(msg.contains(&FUZZY_EDITS_WARN_ABOVE.to_string()));
}
}
/// `config_example.toml` is the documentation for every setting, so a key
/// renamed in the struct and not here would silently ship a config file that
/// does nothing. Parsing it also proves the `[search.columns]` sub-table is
/// spelled the way serde expects.
#[test]
fn the_documented_example_config_parses_to_the_defaults() {
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../config_example.toml")
.canonicalize()
.expect("config_example.toml sits at the repository root");
let text = std::fs::read_to_string(&path).expect("readable");
let parsed: Config = toml::from_str(&text).expect("config_example.toml parses");
// The example documents the shipped defaults for everything that has one
// that does not depend on the machine (paths and the hotkey do).
let d = Config::default();
assert_eq!(
parsed.search, d.search,
"[search] drifted from the defaults"
);
assert_eq!(parsed.processing, d.processing);
assert_eq!(parsed.ui.scale, d.ui.scale);
assert_eq!(parsed.ui.color_scheme, d.ui.color_scheme);
assert_eq!(parsed.ui.tutorial_seen, Some(false));
}
/// The tour is offered to an installation this version created, and to no
/// other. A config written before the key existed reads as `None`, which is
/// how "already found their way around" is distinguished from "brand new".
#[test]
fn only_a_freshly_written_config_asks_for_the_tour() {
assert_eq!(UiConfig::default().tutorial_seen, Some(false));
let older: Config = toml::from_str("[ui]\nscale = 1.1\n").expect("parses");
assert_eq!(
older.ui.tutorial_seen, None,
"a config predating the tour must not be offered it"
);
let dismissed: Config = toml::from_str("[ui]\ntutorial_seen = true\n").expect("parses");
assert_eq!(dismissed.ui.tutorial_seen, Some(true));
}
/// Size and modified cost more width than they earn for most searches.
#[test]
fn the_search_table_ships_without_size_or_modified() {
let cols = ColumnsConfig::default();
assert!(cols.name && cols.content_match && cols.rank);
assert!(!cols.size, "the size column is on by default");
assert!(!cols.modified, "the modified column is on by default");
// A `[search]` block written before the picker existed still gets them.
let older: Config = toml::from_str("[search]\ndisplay_limit = 500\n").expect("parses");
assert_eq!(older.search.columns, cols);
assert_eq!(older.search.display_limit, 500);
assert!(older.search.live_results, "live results default to on");
}

174
crates/quicksearch-core/src/content.rs
View file

@ -1,22 +1,24 @@
//! Parallel content extraction for one indexing root.
//!
//! The second half of a root's pipeline, and the sibling of [`crate::walk`]:
//! a pool of worker threads produces finished work over a bounded channel, and
//! the single writer drains it round-robin against every other root.
//! a pool of worker threads produces finished work over a bounded channel,
//! and the single writer drains it in time-bounded turns. Walking roots are
//! served first and only one extracting root per round, so a pass that is
//! producing faster than the writer can tokenize waits rather than holding up
//! anyone's walk — see `indexing::pipeline`.
//!
//! **One feeder thread owns the only database connection**, paging through
//! the root's pending rows, while N workers do nothing but filesystem work. A
//! connection per worker would multiply SQLite's page cache by the pool size
//! (see [`crate::db::schema::PRAGMAS_WALK_READER`]).
use std::sync::atomic::AtomicBool;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc, Condvar, Mutex};
use std::thread::JoinHandle;
use crate::config::Config;
use crate::extract::Registry;
use crate::file_handling::{decide_content, ContentOutcome, ExtractCursor};
use crate::indexing::should_abort;
use crate::file_handling::{decide_content, ContentOutcome, ExtractCursor, ExtractScope};
use crate::walk::{try_recv_next, TryNext, WorkerStats};
/// Finished rows waiting for the writer.
@ -69,6 +71,11 @@ struct Queue {
struct Shared {
queue: Mutex<Queue>,
idle: Condvar,
/// What the range held when the pass began: rows still to extract and
/// rows already done. Set by the feeder before it pages anything, so
/// `already_done + rows written this pass` stays exact; never set if the
/// feeder could not count.
totals: std::sync::OnceLock<ExtractScope>,
}
impl Shared {
@ -163,6 +170,16 @@ impl ContentPass {
self.stats.clone()
}
/// The range's pending and already-done counts as they stood when the
/// pass began.
///
/// `None` until the feeder has counted — a scan that takes seconds on a
/// large root, which is why it happens here on the pass's own connection
/// and not on the indexer's writer — and forever if it could not.
pub fn totals(&self) -> Option<ExtractScope> {
self.shared.totals.get().copied()
}
/// Join the workers and report whether every one finished cleanly.
/// See [`crate::walk::ParallelWalk::finish`].
pub fn finish(&mut self) -> bool {
@ -197,7 +214,7 @@ impl Drop for ContentPass {
/// A failed query ends the pass rather than retrying: the rows stay
/// `content_state = 0` and the next run picks them up, which is the same
/// outcome as being interrupted.
fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, max_size: i64) {
fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, config: &Config) {
let conn = match crate::db::open::open_walk_reader(db_path) {
Ok(conn) => conn,
Err(e) => {
@ -207,6 +224,18 @@ fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, max_size: i
}
};
// Before the first page, so nothing this pass writes is inside the count.
// The workers cannot run ahead of this: they block in `take` until the
// first page lands. A failure here costs the progress figure, not the
// pass.
match crate::file_handling::count_extract_scope(&conn, &cursor, config) {
Ok(totals) => {
let _ = shared.totals.set(totals);
}
Err(e) => crate::log_warn!("content reader: {}", e),
}
let max_size = crate::file_handling::max_text_file_size(config);
while shared.take_feed_slot().is_some() {
let page =
match crate::db::repo::pending_content_page(&conn, &cursor, max_size, FEED_PAGE as i64)
@ -244,14 +273,13 @@ fn worker(
registry: &Registry,
config: &Config,
stop_flag: &Arc<AtomicBool>,
suspend_flag: &Arc<AtomicBool>,
stats: &WorkerStats,
) {
while let Some(row) = shared.take() {
// Held for the whole of `decide_content`; that is the work the
// progress line reports.
let _busy = stats.enter();
if should_abort(stop_flag, suspend_flag) {
if stop_flag.load(Ordering::Relaxed) {
shared.shutdown();
return;
}
@ -279,14 +307,13 @@ pub fn extract_content(
registry: Arc<Registry>,
config: Config,
stop_flag: Arc<AtomicBool>,
suspend_flag: Arc<AtomicBool>,
workers: usize,
) -> ContentPass {
let shared = Arc::new(Shared {
queue: Mutex::new(Queue::default()),
idle: Condvar::new(),
totals: std::sync::OnceLock::new(),
});
let max_size = i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX);
let (tx, rx) = mpsc::sync_channel(READY_CAP);
let stats = WorkerStats::new(workers.clamp(1, 64));
@ -294,19 +321,11 @@ pub fn extract_content(
.map(|_| {
let (shared, tx) = (shared.clone(), tx.clone());
let (registry, config) = (registry.clone(), config.clone());
let (stop_flag, suspend_flag) = (stop_flag.clone(), suspend_flag.clone());
let stop_flag = stop_flag.clone();
let stats = stats.clone();
crate::platform::spawn_worker("qs-extract", move || {
crate::platform::set_background_priority();
worker(
&shared,
&tx,
&registry,
&config,
&stop_flag,
&suspend_flag,
&stats,
)
worker(&shared, &tx, &registry, &config, &stop_flag, &stats)
})
})
.collect();
@ -318,7 +337,7 @@ pub fn extract_content(
let (shared, db_path, cursor) = (shared.clone(), db_path.to_string(), cursor.clone());
crate::platform::spawn_worker("qs-feeder", move || {
crate::platform::set_background_priority();
feeder(&shared, &db_path, cursor, max_size)
feeder(&shared, &db_path, cursor, &config)
})
};
@ -338,9 +357,23 @@ mod tests {
use crate::db::open_or_recreate;
use crate::db::repo::{self, insert_file, NewFile};
use crate::file_handling::{extract_scope_prepare, store_extracted};
use crate::file_handling::{store_extracted, ExtractScope, Stored};
use crate::mime::FileType;
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
/// The removed `extract_scope_prepare`: the sweep on the writer, then the
/// count the content pass now does on its own connection. Composed here
/// because these tests want both halves in one call.
fn extract_scope_prepare(
conn_mutex: &Arc<Mutex<rusqlite::Connection>>,
cursor: &ExtractCursor,
config: &Config,
) -> Result<ExtractScope, String> {
let conn = crate::lock_ok(conn_mutex);
crate::file_handling::mark_oversize_pending_na(&conn, cursor, config)?;
crate::file_handling::count_extract_scope(&conn, cursor, config)
}
/// A path that does not exist yet — the caller builds the tree under it.
fn tmp(tag: &str) -> PathBuf {
crate::testutil::scratch_dir(tag).join("tree")
@ -391,7 +424,6 @@ mod tests {
Arc::new(Registry::default_set()),
Config::default(),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
workers,
)
}
@ -450,9 +482,13 @@ mod tests {
assert_eq!(rows.len(), 3);
let stop = Arc::new(AtomicBool::new(false));
let far = Instant::now() + Duration::from_secs(60);
assert_eq!(
store_extracted(&conn_mutex, &rows, &stop, &config).unwrap(),
3
store_extracted(&conn_mutex, &rows, &stop, &config, far).unwrap(),
Stored {
consumed: 3,
written: 3
}
);
let state = |p: &Path| -> i64 {
@ -501,6 +537,93 @@ mod tests {
let mut pass = pass_for(&tree, &db, "nonexistent", 4);
assert!(drain(&mut pass).is_empty());
assert!(pass.finish());
// The pass still counted: an empty range is a known zero, not an
// unknown.
assert_eq!(
pass.totals(),
Some(ExtractScope {
pending: 0,
already_done: 0
})
);
std::fs::remove_dir_all(&tree).ok();
std::fs::remove_file(&db).ok();
}
/// The pass counts its range on its own connection, before it pages —
/// which is what lets the writer thread stop doing it. The count is what
/// stood at the start: rows this pass writes are not inside it.
#[test]
fn the_pass_counts_its_range_before_it_starts() {
let (tree, db) = seed("totals", &[("r1", 3), ("r2", 2)]);
let mut pass = pass_for(&tree, &db, "r1", 2);
let rows = drain(&mut pass);
assert!(pass.finish());
assert_eq!(rows.len(), 3);
assert_eq!(
pass.totals(),
Some(ExtractScope {
pending: 3,
already_done: 0
}),
"only r1's rows, all of them pending when the pass began"
);
std::fs::remove_dir_all(&tree).ok();
std::fs::remove_file(&db).ok();
}
/// The writer's turn is bounded by time, not by batch: `store_extracted`
/// stops at its deadline, tells the caller how far it got, and always
/// gets at least one row down so a caller looping on it cannot spin.
#[test]
fn store_extracted_honours_its_deadline_but_always_makes_progress() {
let (tree, db) = seed("deadline", &[("r1", 5)]);
let conn_mutex = Arc::new(Mutex::new(
open_or_recreate(db.to_str().unwrap(), "trigram").unwrap(),
));
let config = Config::default();
let mut pass = pass_for(&tree, &db, "r1", 2);
let rows = drain(&mut pass);
assert!(pass.finish());
assert_eq!(rows.len(), 5);
let stop = Arc::new(AtomicBool::new(false));
// A deadline already gone by: one row, then out.
let past = Instant::now() - Duration::from_secs(1);
assert_eq!(
store_extracted(&conn_mutex, &rows, &stop, &config, past).unwrap(),
Stored {
consumed: 1,
written: 1
}
);
// Plenty of time: the rest, in one call.
let far = Instant::now() + Duration::from_secs(60);
assert_eq!(
store_extracted(&conn_mutex, &rows[1..], &stop, &config, far).unwrap(),
Stored {
consumed: 4,
written: 4
}
);
let done: i64 = conn_mutex
.lock()
.unwrap()
.query_row(
"SELECT COUNT(*) FROM files WHERE content_state = 1",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(done, 5, "every row landed across the two calls");
// Stopped before it starts: nothing consumed, and the caller can tell.
stop.store(true, std::sync::atomic::Ordering::Relaxed);
assert_eq!(
store_extracted(&conn_mutex, &rows, &stop, &config, far).unwrap(),
Stored::default()
);
std::fs::remove_dir_all(&tree).ok();
std::fs::remove_file(&db).ok();
}
@ -514,7 +637,6 @@ mod tests {
Arc::new(Registry::default_set()),
Config::default(),
Arc::new(AtomicBool::new(true)),
Arc::new(AtomicBool::new(false)),
4,
);
assert!(drain(&mut pass).len() < 400);

22
crates/quicksearch-core/src/coordinator.rs
View file

@ -130,6 +130,7 @@ enum CoordCmd {
ConfigChanged(Config),
RebuildIndex,
ClearIndex,
UpdatePaths(Vec<PathBuf>),
Shutdown,
}
@ -228,6 +229,7 @@ impl IndexCoordinator {
watcher_rx: None,
watcher_gen: 0,
pending: HashMap::new(),
targeted: HashMap::new(),
last_event_at: None,
pending_since: None,
needs_full_run: false,
@ -303,6 +305,26 @@ impl IndexCoordinator {
let _ = self.cmd_tx.send(CoordCmd::ClearIndex);
}
/// Bring the index up to date for these paths and nothing else.
///
/// For [`crate::live`]: a frontend that has just read a displayed file
/// from disk hands the path here so the index agrees with what the user
/// is looking at. Deliberately **not** gated on [`IndexMode`] — the whole
/// point is that the rows on screen stay honest with indexing stopped —
/// but still applied on the coordinator's own thread, so the
/// single-writer rule holds and a full run is never raced.
///
/// Each path is re-read and rewritten only if its modified time has moved
/// (see [`crate::incremental::apply_fs_event`]), so submitting a path that
/// is already current costs a `stat` and a row lookup. A path that no
/// longer exists is removed from the index.
pub fn update_paths(&self, paths: Vec<PathBuf>) {
if paths.is_empty() {
return;
}
let _ = self.cmd_tx.send(CoordCmd::UpdatePaths(paths));
}
/// Compare `config` against what the index was built with. Read-only.
pub fn check_config_validation(
&self,

111
crates/quicksearch-core/src/coordinator/inner.rs
View file

@ -22,6 +22,12 @@ pub(super) struct Inner {
pub(super) watcher_rx: Option<mpsc::Receiver<(u64, Result<Watcher, WatchError>)>>,
pub(super) watcher_gen: u64,
pub(super) pending: HashMap<PathBuf, FsEvent>,
/// Paths a frontend asked for by name — see
/// [`IndexCoordinator::update_paths`]. Kept apart from [`Inner::pending`]
/// on purpose: this queue survives [`Inner::clear_pending`] and is applied
/// in manual mode, because it exists to keep the rows a user is *reading*
/// in step with the disk however the indexer is configured.
pub(super) targeted: HashMap<PathBuf, FsEvent>,
/// When the most recent event arrived; the burst is over once this is
/// `pending_settle` old.
pub(super) last_event_at: Option<Instant>,
@ -158,6 +164,36 @@ impl Inner {
drop(shared);
self.files_at = None;
}
CoordCmd::UpdatePaths(paths) => {
// Only paths under an indexed root: the watcher never
// delivers anything else, so nothing downstream checks, and
// a file renamed *out* of every root would otherwise be
// written into the index at its new home. Roots in the same
// spelling `files.path` uses — the caller's paths are.
let prefixes: Vec<String> = self
.config
.normalized_indexing_paths()
.iter()
.map(|root| crate::file_handling::ExtractCursor::for_root(root).lo)
.collect();
for path in paths {
let spelled = path.to_string_lossy();
if !prefixes.iter().any(|lo| spelled.starts_with(lo.as_str())) {
continue;
}
// Existence decides the verb. The caller knows a file
// changed, not what it changed into, and a `Modify` for a
// path that is gone would be silently skipped rather than
// removing the row.
let event = if path.is_file() {
FsEvent::Modify(path)
} else {
FsEvent::Remove(path)
};
enqueue(&mut self.targeted, event);
}
self.was_busy = true;
}
CoordCmd::Shutdown => unreachable!("handled in run()"),
}
}
@ -197,6 +233,14 @@ impl Inner {
self.refresh_file_count();
// Ahead of both the reconcile and the mode gate, and ahead of the
// settle window the watcher queue waits out: these are rows a user is
// looking at right now, there are at most a screenful, and a stopped
// indexer is exactly when the frontend most needs them to be current.
if !self.targeted.is_empty() {
self.apply_targeted();
}
// Ahead of the mode gate: a config edit is reconciled in manual mode
// too.
if self.pending_work.is_some() {
@ -469,6 +513,71 @@ impl Inner {
}
}
/// Apply the by-name queue: the paths a frontend is displaying.
///
/// Shaped like [`Inner::apply_pending`] — removals first, same budget —
/// but it never escalates to [`Inner::needs_full_run`]. A frontend reads
/// what it shows from the file itself, so a failure here leaves the screen
/// correct and only the index behind; reindexing the world over that would
/// be wildly out of proportion.
fn apply_targeted(&mut self) {
self.was_busy = true;
let mut conn = match self.ensure_write_conn() {
Ok(conn) => conn,
Err(e) => {
crate::log_warn!("coordinator: targeted update unavailable: {}", e);
self.targeted.clear();
return;
}
};
let deadline = Instant::now() + APPLY_BUDGET;
let chunk = self.config.processing.batch_size.max(1);
// Removals lead for the same reason they do in `apply_pending`: the
// queue is an unordered map, and a rename enqueues both halves.
let removals: Vec<PathBuf> = self
.targeted
.iter()
.filter(|(_, ev)| is_removal(ev))
.map(|(p, _)| p.clone())
.collect();
for batch in removals.chunks(chunk) {
if let Err(e) = crate::incremental::remove_paths(&mut conn, batch, chunk) {
crate::log_warn!("coordinator: targeted remove: {}", e);
}
for path in batch {
self.targeted.remove(path);
}
if Instant::now() >= deadline {
break;
}
}
if Instant::now() < deadline {
let upserts: Vec<PathBuf> = self
.targeted
.iter()
.filter(|(_, ev)| !is_removal(ev))
.map(|(p, _)| p.clone())
.collect();
for path in upserts {
let Some(ev) = self.targeted.remove(&path) else {
continue;
};
if let Err(e) =
apply_fs_event(&mut conn, &ev, &self.config, &self.ignore, &self.registry)
{
crate::log_warn!("coordinator: targeted apply {:?}: {}", ev, e);
}
if Instant::now() >= deadline {
break;
}
}
}
self.write_conn = Some(conn);
}
fn ensure_write_conn(&mut self) -> Result<Connection, String> {
if let Some(conn) = self.write_conn.take() {
return Ok(conn);
@ -799,7 +908,7 @@ impl Inner {
);
let mut shared = crate::lock_ok(&self.shared);
shared.mode = self.mode;
shared.queued_events = self.pending.len();
shared.queued_events = self.pending.len() + self.targeted.len();
shared.reconcile = reconcile;
drop(shared);

143
crates/quicksearch-core/src/coordinator/tests.rs
View file

@ -505,6 +505,149 @@ fn a_run_it_schedules_itself_wakes_the_frontend() {
coord.shutdown();
}
// --- targeted updates (see `IndexCoordinator::update_paths`) --------------
impl Fixture {
/// The `mtime` the index holds for one path, or `None` if it has no row.
fn stored_mtime(&self, path: &std::path::Path) -> Option<i64> {
let conn = db::open_existing(&self.db.to_string_lossy(), false).ok()?;
conn.query_row(
"SELECT mtime FROM files WHERE path = ?1",
[path.to_string_lossy().as_ref()],
|r| r.get(0),
)
.ok()
}
}
/// The point of the whole thing: the frontend has just read a file the user is
/// looking at, and the index catches up even though indexing is stopped — with
/// no watcher running and no full run scheduled.
#[test]
fn update_paths_indexes_one_file_with_indexing_stopped() {
let f = Fixture::new(false);
std::fs::write(f.dir.join("seed.txt"), "initial content").unwrap();
f.seed_index();
assert_eq!(f.file_count(), 1);
let coord = start_coord(f.config.clone());
let added = f.dir.join("added.txt");
std::fs::write(&added, "written while indexing was stopped").unwrap();
coord.update_paths(vec![added.clone()]);
wait_for("targeted insert", Duration::from_secs(20), || {
f.stored_mtime(&added).is_some()
});
assert_eq!(
coord.state().mode,
IndexMode::ManualStopped,
"a targeted update started a run"
);
assert!(
coord.state().last_full_index.is_some(),
"the seed stamp was disturbed"
);
coord.shutdown();
}
/// The same call is how a row is *validated*: submitting a path the index
/// already agrees with must not rewrite it, which is what makes it cheap
/// enough for the frontend to submit whatever it just looked at.
#[test]
fn update_paths_leaves_a_row_that_already_agrees_alone() {
let f = Fixture::new(false);
let file = f.dir.join("steady.txt");
std::fs::write(&file, "unchanged").unwrap();
f.seed_index();
let before = f.stored_mtime(&file).expect("seeded");
let coord = start_coord(f.config.clone());
coord.update_paths(vec![file.clone()]);
// No state change to wait on, so wait out a few ticks instead.
std::thread::sleep(Duration::from_secs(3));
assert_eq!(f.stored_mtime(&file), Some(before));
assert_eq!(f.file_count(), 1);
coord.shutdown();
}
/// A path outside every indexed root is not the index's to hold, however it
/// was submitted: a result renamed into an un-indexed folder must not follow
/// the row into the index at its new home.
#[test]
fn update_paths_ignores_a_path_outside_every_root() {
let f = Fixture::new(false);
std::fs::write(f.dir.join("seed.txt"), "initial content").unwrap();
f.seed_index();
assert_eq!(f.file_count(), 1);
// A sibling of the indexed tree, under the same scratch parent.
let outside = f.dir.parent().unwrap().join("elsewhere");
std::fs::create_dir_all(&outside).unwrap();
let stray = outside.join("moved-here.txt");
std::fs::write(&stray, "renamed out of the index").unwrap();
let coord = start_coord(f.config.clone());
coord.update_paths(vec![stray.clone()]);
std::thread::sleep(Duration::from_secs(3));
assert_eq!(
f.stored_mtime(&stray),
None,
"an un-indexed folder gained a row"
);
assert_eq!(f.file_count(), 1);
coord.shutdown();
std::fs::remove_dir_all(&outside).ok();
}
/// A row whose file has gone leaves the index too — the frontend hands over
/// the path, not a verb, so the coordinator decides from what is on disk.
#[test]
fn update_paths_removes_a_row_whose_file_is_gone() {
let f = Fixture::new(false);
let file = f.dir.join("doomed.txt");
std::fs::write(&file, "not for long").unwrap();
f.seed_index();
assert!(f.stored_mtime(&file).is_some());
let coord = start_coord(f.config.clone());
std::fs::remove_file(&file).unwrap();
coord.update_paths(vec![file.clone()]);
wait_for("targeted remove", Duration::from_secs(20), || {
f.stored_mtime(&file).is_none()
});
coord.shutdown();
}
/// The single-writer rule still holds: a targeted update submitted while a
/// full run owns the database waits for it rather than opening a second
/// writer beside it.
#[test]
fn update_paths_waits_for_a_full_run_rather_than_racing_it() {
let f = Fixture::new(false);
for i in 0..400 {
std::fs::write(f.dir.join(format!("f{i}.txt")), "body").unwrap();
}
let coord = start_coord(f.config.clone());
coord.reindex_now();
let added = f.dir.join("late.txt");
std::fs::write(&added, "submitted mid-run").unwrap();
coord.update_paths(vec![added.clone()]);
wait_for("run finished", Duration::from_secs(60), || {
coord.state().last_full_index.is_some()
});
wait_for(
"targeted insert after the run",
Duration::from_secs(20),
|| f.stored_mtime(&added).is_some(),
);
coord.shutdown();
}
#[test]
fn auto_mode_runs_initial_index_and_applies_watcher_events() {
let f = Fixture::new(true);

79
crates/quicksearch-core/src/db/repo.rs
View file

@ -217,6 +217,85 @@ pub fn set_content_done(
set_state_clearing_failure(tx, file_id, STATE_DONE, "update DONE")
}
/// Reusable decode buffer and decompression context for the readers of
/// `documents_text` — the read side's mirror of [`DocEncoder`].
///
/// Shared by the cascade's full-text passes and by [`crate::live`], which
/// re-reads one row's body when a file under a visible result changes.
///
/// `zstd::decode_all` builds and tears down a `ZSTD_DCtx` *and* allocates a
/// fresh output `Vec` on every call, and it is called once per candidate row.
/// One context and one buffer, reused across a whole scan, make that a
/// per-scan cost instead of a per-row one.
pub struct DocDecoder {
dctx: zstd::bulk::Decompressor<'static>,
buf: Vec<u8>,
}
/// Where [`DocDecoder::decode`]'s buffer starts before it has seen a document.
/// Most extracted text is well under this, so the doubling below rarely runs.
const INITIAL_DOC_CAPACITY: usize = 64 * 1024;
/// Where the doubling stops. Stored text is capped at
/// `processing.maximum_text_size` (256 KiB by default), so this is far above
/// any legitimate document even if that setting is raised — past it, a failure
/// is a corrupt frame rather than a buffer that is too small.
const MAX_DOC_CAPACITY: usize = 64 * 1024 * 1024;
impl DocDecoder {
pub fn new() -> Result<Self, String> {
Ok(DocDecoder {
dctx: zstd::bulk::Decompressor::new().map_err(|e| e.to_string())?,
buf: Vec::new(),
})
}
/// Decompress `blob` and borrow the result as text.
///
/// Returns `None` for a corrupt frame or non-UTF-8 content. Nothing is
/// copied: the indexer stores UTF-8, so the bytes are borrowed in place
/// rather than run through `String::from_utf8_lossy(..).into_owned()`,
/// which duplicated the whole document even when it was already valid.
pub fn decode(&mut self, blob: &[u8]) -> Option<&str> {
self.buf.clear();
// `decompress_to_buffer` writes into spare capacity and fails rather
// than growing, so the room has to be there first.
//
// The frame header would say how much is needed, but the indexer
// writes with `zstd::encode_all`, which is *stream*-based and so
// records no content size — `get_frame_content_size` says `None` for
// every row this ever sees. Falling back to `zstd::decode_all` there
// looked harmless and was not: it builds a streaming decoder per call,
// which measured as one ~2.4 MiB allocation per document and 27 of the
// 30 GiB a fuzzy search moved through the allocator.
//
// So grow this buffer instead and keep reusing it. It settles at the
// largest document in the scan within the first few rows, after which
// decoding a row allocates nothing at all.
if let Ok(Some(size)) = zstd::zstd_safe::get_frame_content_size(blob) {
self.buf.reserve(usize::try_from(size).ok()?);
}
loop {
if self.buf.capacity() == 0 {
self.buf.reserve(INITIAL_DOC_CAPACITY);
}
match self.dctx.decompress_to_buffer(blob, &mut self.buf) {
Ok(_) => break,
// Too small, or corrupt — the bulk API cannot tell us which.
// Growing is only worth trying while the buffer is still
// smaller than any document could legitimately be.
Err(_) if self.buf.capacity() < MAX_DOC_CAPACITY => {
let bigger = self.buf.capacity().saturating_mul(2);
self.buf.clear();
self.buf.reserve(bigger);
}
Err(_) => return None,
}
}
std::str::from_utf8(&self.buf).ok()
}
}
/// Level 3 hits ~3-5× on English prose at high throughput (hundreds of
/// MB/s); level 9+ would shave a few percent more at 10× the CPU cost, and
/// readers decompress far faster than writers compress.

19
crates/quicksearch-core/src/db/schema.rs
View file

@ -22,7 +22,7 @@
//! | [`PRAGMAS_SEARCH`] | search worker | held across a typing session | 32 MiB |
//! | [`PRAGMAS_READONLY`] | one-shot readers | a single query | 4 MiB |
//! | [`PRAGMAS_MAINTENANCE`] | VACUUM | one bulk copy | 8 MiB |
//! | [`PRAGMAS_WALK_READER`] | per-root row prefetch | the walk | 1 MiB |
//! | [`PRAGMAS_WALK_READER`] | per-root walk prefetch and content feeder | the run | 1 MiB |
//!
//! `PRAGMA mmap_size` is absent from all of them: SQLCipher's codec disables
//! mmap at runtime only when a key is set, mapped pages still count in
@ -124,12 +124,19 @@ pub const PRAGMAS_READONLY: &str = "
PRAGMA foreign_keys = ON;
";
/// Pragmas for a walk's row-prefetch connection.
/// Pragmas for a root's own reader: the walk's row prefetch, and then the
/// content pass's feeder ([`crate::content`]), which reuses this profile for
/// the rest of the run.
///
/// One of these exists per indexing root, so the cache size is multiplied by
/// the root count. 1 MiB holds the upper levels of `idx_files_parent` hot,
/// which is all these queries touch: each is a single index range lookup,
/// and the pages under it are read once and not revisited.
/// Two of these can exist per indexing root, so the cache size is multiplied
/// by the root count. 1 MiB is sized for the walk's queries, which each read
/// one range of `idx_files_parent` once and never revisit it. The feeder's
/// paging is the same shape, but its one-off `count_extract_scope` at pass
/// start is not: that scans the root's whole path range fetching a row per
/// entry, so on a large root it is a cold read all the way through. It is
/// deliberately here rather than on the writer — the writer holding still for
/// it stopped every other root's walk — and this is the connection that pays
/// for that, once per root.
pub const PRAGMAS_WALK_READER: &str = "
PRAGMA busy_timeout = 5000;
PRAGMA cache_size = -1024;

11
crates/quicksearch-core/src/extract/mod.rs
View file

@ -41,11 +41,6 @@ impl ExtractedContent {
}
}
pub fn with_property(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
self.properties.insert(key.into(), value.into());
self
}
/// Convert properties into the `Vec<(String, String)>` shape expected by
/// [`crate::db::repo::set_content_done`]. Keys are sorted for determinism
/// in tests and snapshots.
@ -300,9 +295,9 @@ mod tests {
#[test]
fn properties_sorted_is_deterministic() {
let c = ExtractedContent::with_text("hi")
.with_property("b", "2")
.with_property("a", "1");
let mut c = ExtractedContent::with_text("hi");
c.properties.insert("b".into(), "2".into());
c.properties.insert("a".into(), "1".into());
assert_eq!(
c.properties_sorted(),
vec![

174
crates/quicksearch-core/src/file_handling/batch.rs
View file

@ -9,7 +9,6 @@ use rusqlite::Connection;
use super::*;
use crate::config::Config;
use crate::db::repo::{self};
use crate::indexing::should_abort;
/// The compressed sidecar for one row, or `None` where there is none to write
/// — an empty body, or `store_text_for_snippets` turned off.
@ -29,12 +28,13 @@ type Body = Result<Option<Vec<u8>>, String>;
///
/// What that lock does *not* gate, so the benefit is not overclaimed: search
/// holds its own connection (`db::open::open_search_reader`) and the database
/// is WAL, where a reader never blocks on a writer. `conn_mutex` serializes
/// the indexer against itself — one root's content stores against another's
/// walk inserts, the scope reconciler's slices, and WAL checkpointing. A
/// whole-tree wall-clock run is dominated by FTS5 trigram tokenization and
/// does not move measurably from this change; it is contention that improves,
/// not throughput.
/// is WAL, where a reader never blocks on a writer. Nor does it separate one
/// root from another — every root's writes already run on the single writer
/// thread, so two of them are never inside the lock at once. What it actually
/// serializes the run against is WAL checkpointing, which `run_indexing`
/// forces from the same thread between turns. A whole-tree wall-clock run is
/// dominated by FTS5 trigram tokenization and does not move measurably from
/// this change; it is the length of the hold that improves, not throughput.
fn compress_bodies<'a>(
texts: impl Iterator<Item = Option<&'a str>>,
config: &Config,
@ -198,15 +198,13 @@ pub fn process_batch_inserts(
/// Delete the rows a completed run found no file behind, in chunked
/// transactions. Returns how many went.
///
/// `should_abort` *blocks* while the indexer is suspended, so it must only be
/// observed between chunks with nothing held — checking it mid-transaction
/// pins the shared connection for the whole suspension and freezes the GUI.
/// The stop flag, which never blocks, guards the inner loop.
/// The stop flag is checked between chunks and again per path, never with a
/// transaction open: a chunk either commits whole or is not begun, so a stop
/// cannot leave the index half-reconciled.
pub fn cleanup_stale_index_entries(
conn_mutex: &Arc<Mutex<Connection>>,
stale_paths: &[String],
stop_flag: &Arc<AtomicBool>,
suspend_flag: &Arc<AtomicBool>,
config: &Config,
) -> Result<usize, String> {
if stale_paths.is_empty() {
@ -216,8 +214,8 @@ pub fn cleanup_stale_index_entries(
let mut deleted_count = 0usize;
for batch in stale_paths.chunks(chunk) {
// Outside the lock, so a suspend parks here rather than mid-transaction.
if should_abort(stop_flag, suspend_flag) {
// Outside the lock, so a stop is seen before a transaction is begun.
if stop_flag.load(Ordering::Relaxed) {
return Ok(deleted_count);
}
let conn = crate::lock_ok(conn_mutex);
@ -241,7 +239,7 @@ pub fn cleanup_stale_index_entries(
}
}
if deleted_count > 0 && !should_abort(stop_flag, suspend_flag) {
if deleted_count > 0 && !stop_flag.load(Ordering::Relaxed) {
let conn = crate::lock_ok(conn_mutex);
fts_finalize_after_text_indexing(&conn);
}
@ -296,71 +294,107 @@ pub struct ExtractScope {
pub already_done: usize,
}
/// Prepare a root's extraction scope: flip oversize pending rows to NA
/// (idempotent) and count what is pending vs. already extracted in the range.
/// The `maximum_text_file_size` bound as the SQL below compares it.
pub(crate) fn max_text_file_size(config: &Config) -> i64 {
i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX)
}
/// Flip a root's oversize pending rows to NA. Idempotent.
///
/// The oversize sweep covers what walk-time decisions cannot: a
/// `maximum_text_file_size` *lowered* between runs (which does not force a
/// rebuild), and rows left pending by an older build.
pub fn extract_scope_prepare(
conn_mutex: &Arc<Mutex<Connection>>,
/// Covers what walk-time decisions cannot: a `maximum_text_file_size`
/// *lowered* between runs (which does not force a rebuild), and rows left
/// pending by an older build. Rows this misses would stay pending forever, so
/// it runs on the writer before a root's content pass starts.
pub fn mark_oversize_pending_na(
conn: &Connection,
cursor: &ExtractCursor,
config: &Config,
) -> Result<ExtractScope, String> {
let max_size = i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX);
let conn = crate::lock_ok(conn_mutex);
) -> Result<(), String> {
conn.execute(
"UPDATE files SET content_state = 3 \
WHERE content_state = 0 AND size > ?1 AND path >= ?2 AND path < ?3",
rusqlite::params![max_size, cursor.lo, cursor.hi],
rusqlite::params![max_text_file_size(config), cursor.lo, cursor.hi],
)
.map_err(|e| format!("mark oversize files NA: {}", e))?;
let pending: i64 = conn
Ok(())
}
/// Count what a root's range holds: rows still to extract this run, and rows
/// whose text is already searchable from earlier runs.
///
/// One range scan for both figures. Deliberately callable on any connection
/// — the content pass runs it on its own read connection rather than on the
/// indexer's writer, because on a large root it takes seconds, and seconds of
/// writer time is every other root's walk standing still.
pub fn count_extract_scope(
conn: &Connection,
cursor: &ExtractCursor,
config: &Config,
) -> Result<ExtractScope, String> {
let (pending, already_done): (i64, i64) = conn
.query_row(
"SELECT COUNT(*) FROM files \
WHERE content_state = 0 AND size <= ?1 AND path >= ?2 AND path < ?3",
rusqlite::params![max_size, cursor.lo, cursor.hi],
|row| row.get(0),
"SELECT COALESCE(SUM(content_state = 0 AND size <= ?1), 0), \
COALESCE(SUM(content_state = 1), 0) \
FROM files WHERE path >= ?2 AND path < ?3",
rusqlite::params![max_text_file_size(config), cursor.lo, cursor.hi],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.map_err(|e| format!("Failed to count pending text files: {}", e))?;
let already_done: i64 = conn
.query_row(
"SELECT COUNT(*) FROM files \
WHERE content_state = 1 AND path >= ?1 AND path < ?2",
rusqlite::params![cursor.lo, cursor.hi],
|row| row.get(0),
)
.map_err(|e| format!("Failed to count extracted files: {}", e))?;
.map_err(|e| format!("Failed to count text files: {}", e))?;
Ok(ExtractScope {
pending: pending.max(0) as usize,
already_done: already_done.max(0) as usize,
})
}
/// Write a batch of already-extracted rows — the cheap half of the content
/// pass, and all that runs with the connection held. Chunked so each
/// transaction stays short.
/// Rows per compression chunk and per transaction inside [`store_extracted`].
///
/// Returns how many rows were written. A row whose write fails is logged and
/// skipped rather than failing the run: its `content_state` stays pending, so
/// the next run retries it.
/// Half of what a writer turn may hand in (`pipeline::READY_TOPUP` is 64), so
/// a full turn commits twice rather than once — short holds of the connection
/// being the point. It also bounds the compression thrown away when the
/// deadline cuts a chunk short, to at most `STORE_CHUNK - 1` bodies.
///
/// Note the two buffers are additive: a root can hold `READY_TOPUP` extracted
/// rows waiting for the writer *and* `content::READY_CAP` more in its
/// channel, so in-flight text per root is bounded by their sum, not by either
/// alone.
const STORE_CHUNK: usize = 32;
/// What one [`store_extracted`] call did with the rows it was handed.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct Stored {
/// Rows the caller must now drop from its buffer, written or not.
pub consumed: usize,
/// Rows whose write succeeded — whose `content_state` moved.
pub written: usize,
}
/// Write already-extracted rows — the cheap half of the content pass, and all
/// that runs with the connection held — until `deadline`.
///
/// This is where a document's FTS5 trigram tokenization happens, up to
/// `maximum_text_size` of it per row, and it is the writer thread's dominant
/// cost. The deadline is checked after every row, so a turn on the writer
/// overruns it by at most one document; the rows not reached are left for the
/// caller to hand back next turn. At least one row is always consumed unless
/// the run is already stopped, so a caller looping on this cannot spin.
///
/// A row whose write fails is logged and consumed rather than failing the
/// run: its `content_state` stays pending, so the next run retries it.
pub fn store_extracted(
conn_mutex: &Arc<Mutex<Connection>>,
rows: &[crate::content::ExtractedRow],
stop_flag: &Arc<AtomicBool>,
config: &Config,
) -> Result<usize, String> {
if rows.is_empty() {
return Ok(0);
}
let mut written = 0usize;
for batch in rows.chunks(config.processing.batch_size.max(1)) {
deadline: std::time::Instant,
) -> Result<Stored, String> {
let mut done = Stored::default();
for chunk in rows.chunks(STORE_CHUNK) {
if stop_flag.load(Ordering::Relaxed) {
return Ok(written);
break;
}
// Outside the lock — see `compress_bodies`.
let bodies = compress_bodies(
batch
chunk
.iter()
.map(|r| crate::file_handling::outcome_body(&r.outcome)),
config,
@ -369,16 +403,34 @@ pub fn store_extracted(
let tx = conn
.unchecked_transaction()
.map_err(|e| format!("Failed to begin transaction: {}", e))?;
for (i, row) in batch.iter().enumerate() {
let zstd = body_or_skip!(bodies, i, row.name);
if let Err(e) = store_content_outcome(&tx, row.file_id, &row.name, &row.outcome, zstd) {
crate::log_warn!("content indexing for {}: {}", row.name, e);
continue;
let mut cut = false;
for (i, row) in chunk.iter().enumerate() {
// Counted before anything can skip it: consumed is what the
// caller drains, and a row that failed still has to leave.
done.consumed += 1;
match &bodies[i] {
Err(e) => crate::log_warn!("compress text for {}: {}", row.name, e),
Ok(zstd) => match store_content_outcome(
&tx,
row.file_id,
&row.name,
&row.outcome,
zstd.as_deref(),
) {
Ok(()) => done.written += 1,
Err(e) => crate::log_warn!("content indexing for {}: {}", row.name, e),
},
}
if stop_flag.load(Ordering::Relaxed) || std::time::Instant::now() >= deadline {
cut = true;
break;
}
written += 1;
}
tx.commit()
.map_err(|e| format!("Failed to commit transaction: {}", e))?;
if cut {
break;
}
}
Ok(written)
Ok(done)
}

11
crates/quicksearch-core/src/file_handling/count_and_extract_tests.rs
View file

@ -8,6 +8,17 @@ use super::*;
use std::sync::atomic::{AtomicBool, Ordering};
/// A path that does not exist yet — these tests build the tree themselves.
/// The removed `extract_scope_prepare`: the oversize sweep, then the count.
fn extract_scope_prepare(
conn_mutex: &std::sync::Arc<std::sync::Mutex<rusqlite::Connection>>,
cursor: &ExtractCursor,
config: &Config,
) -> Result<ExtractScope, String> {
let conn = crate::lock_ok(conn_mutex);
super::mark_oversize_pending_na(&conn, cursor, config)?;
super::count_extract_scope(&conn, cursor, config)
}
fn tmp(tag: &str) -> std::path::PathBuf {
crate::testutil::scratch_dir(tag).join("tree")
}

7
crates/quicksearch-core/src/file_handling/mod.rs
View file

@ -11,11 +11,12 @@ mod count_and_extract_tests;
#[cfg(test)]
mod tests;
pub(crate) use batch::store_inline_text;
pub use batch::{
cleanup_stale_index_entries, extract_scope_prepare, process_batch_inserts,
process_batch_updates, store_extracted, ExtractCursor, ExtractScope,
cleanup_stale_index_entries, count_extract_scope, mark_oversize_pending_na,
process_batch_inserts, process_batch_updates, store_extracted, ExtractCursor, ExtractScope,
Stored,
};
pub(crate) use batch::{max_text_file_size, store_inline_text};
pub use counting::count_tree_entries_fast;
pub use paths::{db_key_for_missing_path, filtered_dirs, filtered_walk, UnreadableDirs};
pub(crate) use paths::{normalize_root_string, path_to_db_string, warn_if_unrepresentable};

67
crates/quicksearch-core/src/incremental.rs
View file

@ -159,25 +159,6 @@ fn remove_path(conn: &mut Connection, path: &Path) -> Result<(), String> {
remove_paths(conn, std::slice::from_ref(&path.to_path_buf()), 1)
}
/// Drop removals that a removal of one of their ancestors already covers.
///
/// `rm -rf dir/` reports `dir` *and* every file beneath it; removing `dir`
/// sweeps its whole path range, so each descendant event is duplicate work.
/// For callers holding a raw removal set — the coordinator collapses on
/// arrival instead (`collapse_pending_removals`). Containment is
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::PathBuf> {
if paths.len() < 2 {
return paths;
}
let all: std::collections::HashSet<&Path> = paths.iter().map(|p| p.as_path()).collect();
paths
.iter()
.filter(|p| !p.ancestors().skip(1).any(|a| all.contains(a)))
.cloned()
.collect()
}
/// Delete `paths` and everything indexed beneath them, in transactions of at
/// most `chunk` paths.
///
@ -314,47 +295,6 @@ mod tests {
}
}
fn collapse(paths: &[&str]) -> Vec<String> {
let mut out: Vec<String> =
collapse_removal_roots(paths.iter().map(std::path::PathBuf::from).collect())
.iter()
.map(|p| p.to_string_lossy().into_owned())
.collect();
out.sort();
out
}
#[test]
fn removal_roots_collapse_to_the_shallowest_ancestor() {
assert_eq!(
collapse(&["/dir", "/dir/a.txt", "/dir/b/c.txt", "/dir/b"]),
vec!["/dir"]
);
// Component-wise, so a name-prefix sibling is not swallowed.
assert_eq!(
collapse(&["/a/b", "/a/bc"]),
vec!["/a/b", "/a/bc"],
"/a/bc does not live under /a/b"
);
assert_eq!(
collapse(&["/a/b", "/a/b.txt"]),
vec!["/a/b", "/a/b.txt"],
"a sibling file sorting between a dir and its children survives"
);
// Unrelated removals all survive; order of input does not matter.
assert_eq!(
collapse(&["/x/deep/f", "/y", "/x"]),
vec!["/x", "/y"],
"/x/deep/f is covered by /x, /y is independent"
);
// Degenerate inputs.
assert!(collapse(&[]).is_empty());
assert_eq!(collapse(&["/only"]), vec!["/only"]);
}
/// The collapse must not change what ends up deleted — only how much work
/// it takes to get there.
#[test]
@ -378,9 +318,10 @@ mod tests {
format!("{}/a.txt", canonical_tree).into(),
format!("{}/deep/b.txt", canonical_tree).into(),
];
let roots = collapse_removal_roots(reported);
assert_eq!(roots.len(), 1, "one range covers the whole tree");
// Collapsed to its root the way the coordinator collapses an
// arriving queue (`collapse_pending_removals`): one range covers the
// whole tree, which is what makes `remove_paths` cheap.
let roots = vec![reported[0].clone()];
remove_paths(&mut f.conn, &roots, 200).unwrap();
assert_eq!(f.counts(), (1, 1, 1), "only tree2 survives");
let survivor = f.canonical(&f.dir.join("tree2").join("keep.txt"));

49
crates/quicksearch-core/src/indexing/mod.rs
View file

@ -6,7 +6,7 @@ use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc, Mutex};
use std::thread;
use std::time::{Duration, Instant};
use std::time::Instant;
use crate::config::Config;
use crate::db;
@ -84,7 +84,6 @@ pub struct IndexingService {
status: Arc<Mutex<IndexingStatus>>,
command_tx: mpsc::Sender<IndexingCommand>,
db_connection: Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
suspend_flag: Arc<AtomicBool>,
/// The long single statement a run is inside, if any: the prologue's
/// reconcile scan or the epilogue's VACUUM — never both, so one slot
/// serves and [`IndexingService::cancel_db_work`] reaches either.
@ -92,9 +91,6 @@ pub struct IndexingService {
_handle: thread::JoinHandle<()>,
}
/// Polling interval for `should_abort` while suspended.
const SUSPEND_POLL_MS: u64 = 100;
/// `indexing.root_workers` rekeyed from the spellings the user typed to the
/// canonical roots the indexer walks, so an override survives a `~`, a
/// trailing slash, a relative path or a symlinked root. Entries naming a
@ -112,39 +108,21 @@ fn resolved_root_workers(config: &Config) -> HashMap<String, usize> {
.collect()
}
/// Combined stop/suspend check used by worker loops; `true` iff the caller
/// should abort. While suspended (and not stopped) it parks the thread in
/// short sleeps until `resume()`.
pub(crate) fn should_abort(stop: &Arc<AtomicBool>, suspend: &Arc<AtomicBool>) -> bool {
loop {
if stop.load(Ordering::Relaxed) {
return true;
}
if !suspend.load(Ordering::Relaxed) {
return false;
}
thread::sleep(Duration::from_millis(SUSPEND_POLL_MS));
}
}
impl IndexingService {
pub fn new() -> Self {
let status = Arc::new(Mutex::new(IndexingStatus::Idle));
let (command_tx, command_rx) = mpsc::channel();
let db_connection = Arc::new(Mutex::new(None));
let suspend_flag = Arc::new(AtomicBool::new(false));
let interrupt: Arc<db::InterruptSlot> = Arc::new(db::InterruptSlot::default());
let status_clone = status.clone();
let db_connection_clone = db_connection.clone();
let suspend_clone = suspend_flag.clone();
let interrupt_clone = interrupt.clone();
let handle = thread::spawn(move || {
Self::indexing_thread(
status_clone,
command_rx,
db_connection_clone,
suspend_clone,
interrupt_clone,
);
});
@ -153,7 +131,6 @@ impl IndexingService {
status,
command_tx,
db_connection,
suspend_flag,
interrupt,
_handle: handle,
}
@ -171,22 +148,6 @@ impl IndexingService {
db::interrupt(&self.interrupt)
}
/// Pause the indexer: worker loops calling [`should_abort`] block until
/// [`resume`](Self::resume). Does not stop the worker.
pub fn suspend(&self) {
self.suspend_flag.store(true, Ordering::Relaxed);
}
/// Resume indexing after [`suspend`](Self::suspend). No-op if not
/// suspended.
pub fn resume(&self) {
self.suspend_flag.store(false, Ordering::Relaxed);
}
pub fn is_suspended(&self) -> bool {
self.suspend_flag.load(Ordering::Relaxed)
}
/// Start indexing one or more roots; all walk concurrently, funnelling
/// into one writer thread. Duplicate roots collapse to one walk, and a
/// file reachable from more than one is written once. Returns `Err` if a
@ -301,11 +262,6 @@ impl IndexingService {
}
}
/// Force graceful shutdown - used for signal handling
pub fn graceful_shutdown(&self) -> Result<(), String> {
self.stop_indexing()
}
/// Check if configuration changes require index recreation. A pure
/// *read* check that never wipes; a missing or incompatible DB means
/// there is nothing to validate.
@ -367,7 +323,6 @@ impl IndexingService {
status: Arc<Mutex<IndexingStatus>>,
command_rx: mpsc::Receiver<IndexingCommand>,
db_connection: Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
suspend_flag: Arc<AtomicBool>,
interrupt: Arc<db::InterruptSlot>,
) {
let stop_flag = Arc::new(AtomicBool::new(false));
@ -396,7 +351,6 @@ impl IndexingService {
let config_owned = config.clone();
let db_connection_clone = db_connection.clone();
let suspend_clone = suspend_flag.clone();
let interrupt_clone = interrupt.clone();
indexing_handle = Some(thread::spawn(move || {
// The writer thread: every DB write and every text
@ -407,7 +361,6 @@ impl IndexingService {
&paths_owned,
&db_path_owned,
&stop_flag_clone,
&suspend_clone,
&config_owned,
&db_connection_clone,
&interrupt_clone,

342
crates/quicksearch-core/src/indexing/pipeline.rs
View file

@ -13,9 +13,9 @@ use crate::db;
use crate::db::repo;
use crate::extract::Registry;
use crate::file_handling::{
cleanup_stale_index_entries, count_tree_entries_fast, extract_scope_prepare,
fts_finalize_after_text_indexing, normalize_root_string, process_batch_inserts,
process_batch_updates, store_extracted, ExtractCursor, FileIndexAction, OwnedNewFile,
cleanup_stale_index_entries, count_tree_entries_fast, fts_finalize_after_text_indexing,
mark_oversize_pending_na, normalize_root_string, process_batch_inserts, process_batch_updates,
store_extracted, ExtractCursor, ExtractScope, FileIndexAction, OwnedNewFile,
};
use crate::walk::{thread_count_for, walk_indexable_files, ParallelWalk, TryNext, WalkEvent};
@ -84,6 +84,20 @@ impl Drop for CancelOnDrop {
}
}
/// Writer time one root's turn may take before the round moves on.
///
/// This is the bound on how long any root can hold up the others. Before
/// there was one, an extraction turn ran to the end of whatever was ready —
/// half a second to two seconds of FTS5 trigram tokenization for a batch of
/// large documents — while a walking root's rows sat in its channel and its
/// walkers parked behind them. Reads as "4/4 workers busy, no progress".
const TURN_SLICE: Duration = Duration::from_millis(100);
/// Most extracted rows a root holds back between turns. Not `quantum`: a row
/// carries up to `maximum_text_size` of text, and 500 of those would be
/// 128 MiB per root. At 64 it is 16 MiB.
const READY_TOPUP: usize = 64;
/// One root's in-flight indexing state, owned by the writer loop.
pub(super) struct RootPipeline {
pub(super) root: String,
@ -100,8 +114,14 @@ pub(super) struct RootPipeline {
pub(super) phase: RootPhase,
/// The running content pass, once this root's walk has finished.
pub(super) content: Option<crate::content::ContentPass>,
pub(super) extract_total: usize,
pub(super) extracted: usize,
/// Extracted rows pulled off the pass and not yet written. A turn writes
/// for its slice, not for its batch, so it may leave some behind.
pub(super) ready: Vec<crate::content::ExtractedRow>,
/// Rows this run's content pass has written for this root.
pub(super) written: usize,
/// The pass's range counts, cached once known so a `Done` root still has
/// them after its pass is gone.
pub(super) totals: Option<ExtractScope>,
pub(super) current_file: Option<String>,
/// When this root's current phase began, for the one line each phase logs
/// when it ends.
@ -173,8 +193,16 @@ impl RootPipeline {
stats.map_or((0, 0), |s| (s.active(), s.total()))
}
fn snapshot(&self) -> RootProgress {
/// The pass's counts, from the cache or — until the cache is filled — from
/// the pass itself.
fn extract_totals(&self) -> Option<ExtractScope> {
self.totals
.or_else(|| self.content.as_ref().and_then(|p| p.totals()))
}
pub(super) fn snapshot(&self) -> RootProgress {
let (active_workers, total_workers) = self.worker_counts();
let totals = self.extract_totals();
RootProgress {
root: self.root.clone(),
phase: self.phase,
@ -183,30 +211,58 @@ impl RootPipeline {
0 => None,
n => Some(n),
},
extracted: self.extracted,
extract_total: self.extract_total,
// Earlier runs' rows count once the pass has counted them; until
// then only this run's, so the figure never goes backwards.
extracted: totals.map_or(self.written, |t| t.already_done + self.written),
extract_total: totals.map(|t| t.pending + t.already_done),
current_file: self.current_file.clone(),
active_workers,
total_workers,
}
}
/// Drain up to one quantum of walk events into the pending batches,
/// Drain walk events into the pending batches for up to one slice,
/// finishing the walk if it ends. Returns whether anything happened.
fn service_walking(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
///
/// Batches still land per quantum; the slice only decides how many of
/// them one turn may write. A walk slower than the writer ends its turn at
/// `Empty` well inside the slice; only a walk that has the writer
/// saturated uses all of it.
pub(super) fn service_walking(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
let deadline = Instant::now() + cx.slice;
let mut took = 0usize;
let mut finished = false;
while took < cx.quantum {
while !finished {
let quantum_end = took + cx.quantum;
let more = self.walk_quantum(cx, &mut took, quantum_end, &mut finished)?;
if !more || Instant::now() >= deadline {
break;
}
}
Ok(finished || took > 0)
}
/// One quantum of [`RootPipeline::service_walking`]. Returns whether the
/// channel still had events when the quantum ended — false on `Empty` or
/// on the walk finishing.
fn walk_quantum(
&mut self,
cx: &mut RunCx<'_>,
took: &mut usize,
quantum_end: usize,
finished: &mut bool,
) -> Result<bool, String> {
while *took < quantum_end {
match self.walk.try_next() {
TryNext::Item(WalkEvent::Stale(paths)) => {
took += 1;
*took += 1;
// Applied at the end of the run: deleting mid-walk would
// break "a stopped run deletes nothing", and an aliased
// sighting that exempts a path may still be ahead.
cx.stale_candidates.extend(paths);
}
TryNext::Item(WalkEvent::File(file)) => {
took += 1;
*took += 1;
self.walked += 1;
if self.walked.is_multiple_of(64) {
self.current_file = Some(file.path.clone());
@ -247,15 +303,15 @@ impl RootPipeline {
}
}
}
TryNext::Empty => break,
TryNext::Empty => return Ok(false),
TryNext::Finished => {
self.finish_walk(cx)?;
finished = true;
break;
*finished = true;
return Ok(false);
}
}
}
Ok(finished || took > 0)
Ok(true)
}
/// The walk ended: land the buffered batches, then either hand the root
@ -314,55 +370,89 @@ impl RootPipeline {
}
}
let cursor = ExtractCursor::for_root(&self.root);
let scope = extract_scope_prepare(&cx.conn_mutex, &cursor, cx.config)?;
// Progress counts the root's whole searchable set: files extracted
// in earlier runs start the counter, so an unchanged root shows
// "X of X" rather than "0 of 0".
self.extract_total = scope.pending + scope.already_done;
self.extracted = scope.already_done;
if scope.pending == 0 {
self.phase = RootPhase::Done;
} else {
// Starts only now: the rows have to exist before the feeder
// can page over them.
self.content = Some(crate::content::extract_content(
cx.db_path,
&cursor,
cx.registry.clone(),
cx.config.clone(),
cx.stop_flag.clone(),
cx.suspend_flag.clone(),
self.workers,
));
self.phase = RootPhase::Extracting;
// Only the sweep runs on the writer. Counting the range is the
// pass's own job, on its own connection: on a large root it is
// seconds, and here that was seconds of every other root's walk
// standing still.
{
let conn = crate::lock_ok(&cx.conn_mutex);
mark_oversize_pending_na(&conn, &cursor, cx.config)?;
}
self.totals = None;
self.written = 0;
self.ready.clear();
// Starts only now: the rows have to exist before the feeder can
// page over them. Started even when nothing may be pending — the
// count that would say so is the pass's — and an empty range
// finishes on its own next turn.
self.content = Some(crate::content::extract_content(
cx.db_path,
&cursor,
cx.registry.clone(),
cx.config.clone(),
cx.stop_flag.clone(),
self.workers,
));
self.phase = RootPhase::Extracting;
}
Ok(())
}
/// Drain up to a quantum of finished extraction work, then write it;
/// extraction runs on this root's own pool. Returns whether anything
/// happened.
fn service_extracting(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
let pass = self.content.as_mut().expect("extracting root has a pass");
let mut batch: Vec<crate::content::ExtractedRow> = Vec::new();
/// Write finished extraction work for up to one slice; extraction itself
/// runs on this root's own pool. Returns whether anything happened.
///
/// Rows the slice does not reach stay in `ready` for the next turn, and
/// the pass is not declared done until they have all landed.
pub(super) fn service_extracting(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
let deadline = Instant::now() + cx.slice;
let mut finished = false;
while batch.len() < cx.quantum {
match pass.try_next() {
TryNext::Item(row) => batch.push(row),
TryNext::Empty => break,
TryNext::Finished => {
finished = true;
break;
let mut consumed = 0usize;
// Disjoint borrows: the pass is held across the store.
let Self {
content,
ready,
written,
totals,
current_file,
..
} = self;
let pass = content.as_mut().expect("extracting root has a pass");
if totals.is_none() {
*totals = pass.totals();
}
loop {
while ready.len() < READY_TOPUP {
match pass.try_next() {
TryNext::Item(row) => ready.push(row),
TryNext::Empty => break,
TryNext::Finished => {
finished = true;
break;
}
}
}
if ready.is_empty() {
break;
}
let stored = store_extracted(&cx.conn_mutex, ready, cx.stop_flag, cx.config, deadline)?;
if stored.consumed > 0 {
// The last row *written*, not the last fetched: with leftovers
// the two can be a slice apart.
*current_file = Some(ready[stored.consumed - 1].name.clone());
}
ready.drain(..stored.consumed);
*written += stored.written;
consumed += stored.consumed;
// Stopped, out of time, or still holding rows the deadline cut
// short — the next turn takes it from here.
if stored.consumed == 0 || !ready.is_empty() || Instant::now() >= deadline {
break;
}
}
if let Some(row) = batch.last() {
self.current_file = Some(row.name.clone());
}
let took = batch.len();
self.extracted += store_extracted(&cx.conn_mutex, &batch, cx.stop_flag, cx.config)?;
if finished {
if finished && ready.is_empty() {
if totals.is_none() {
*totals = pass.totals();
}
// Join before deciding; see `ParallelWalk::finish`.
if !pass.finish() {
crate::log_warn!("a content worker for {} terminated abnormally", self.root);
@ -370,39 +460,67 @@ impl RootPipeline {
self.content = None;
self.phase = RootPhase::Done;
let extract_time = self.phase_elapsed();
crate::log_info!(
"{}: content done — {}",
self.root,
phase_summary(self.extracted, "files with text", extract_time)
);
// Quiet for the pass that found nothing to do: every root passes
// through here now, changed or not.
if self.written > 0 {
crate::log_info!(
"{}: content done — {}",
self.root,
phase_summary(self.written, "files with text", extract_time)
);
}
}
Ok(finished || took > 0)
Ok(finished || consumed > 0)
}
}
/// One run's shared environment and cross-root state, threaded through the
/// per-phase [`RootPipeline`] service methods.
struct RunCx<'a> {
conn_mutex: Arc<Mutex<Connection>>,
config: &'a Config,
db_path: &'a str,
stop_flag: &'a Arc<AtomicBool>,
suspend_flag: &'a Arc<AtomicBool>,
pub(super) struct RunCx<'a> {
pub(super) conn_mutex: Arc<Mutex<Connection>>,
pub(super) config: &'a Config,
pub(super) db_path: &'a str,
pub(super) stop_flag: &'a Arc<AtomicBool>,
/// Shared with every root's walk workers, which use it to finish small
/// text files without handing them to the content pass.
registry: Arc<Registry>,
quantum: usize,
pub(super) registry: Arc<Registry>,
pub(super) quantum: usize,
/// See [`TURN_SLICE`]; a field so tests can shrink it.
pub(super) slice: Duration,
/// 128-bit path digests, not paths: at millions of files, owning every
/// path string again was the single largest allocation in a run. See
/// `walk::path_digest`.
seen_paths: HashSet<u128>,
pub(super) seen_paths: HashSet<u128>,
/// Rows the per-directory reconciliation found no file behind, plus
/// whatever the vanished-directory sweep adds once the walks end.
stale_candidates: Vec<String>,
pub(super) stale_candidates: Vec<String>,
/// Paths reached by resolving a symlink, whose row lives under a parent
/// that may be outside every root.
aliased_paths: HashSet<String>,
stale_cleanup_ok: bool,
pub(super) aliased_paths: HashSet<String>,
pub(super) stale_cleanup_ok: bool,
}
impl<'a> RunCx<'a> {
pub(super) fn new(
conn_mutex: Arc<Mutex<Connection>>,
config: &'a Config,
db_path: &'a str,
stop_flag: &'a Arc<AtomicBool>,
) -> RunCx<'a> {
RunCx {
conn_mutex,
config,
db_path,
stop_flag,
registry: Arc::new(Registry::default_set()),
quantum: config.processing.batch_size.max(1),
slice: TURN_SLICE,
seen_paths: HashSet::new(),
stale_candidates: Vec::new(),
aliased_paths: HashSet::new(),
stale_cleanup_ok: true,
}
}
}
/// Publish a status snapshot. Never clobbers Stopping — the command thread
@ -447,7 +565,6 @@ fn build_pipeline(
cx.config.clone(),
cx.registry.clone(),
cx.stop_flag.clone(),
cx.suspend_flag.clone(),
workers,
);
@ -491,8 +608,9 @@ fn build_pipeline(
phase: RootPhase::Walking,
workers,
content: None,
extract_total: 0,
extracted: 0,
ready: Vec::new(),
written: 0,
totals: None,
current_file: None,
phase_started: Instant::now(),
})
@ -557,7 +675,6 @@ fn cleanup_stale(pipelines: &mut [RootPipeline], cx: &mut RunCx<'_>) -> Result<(
&cx.conn_mutex,
stale_paths.as_slice(),
cx.stop_flag,
cx.suspend_flag,
cx.config,
)?;
crate::log_info!(
@ -575,7 +692,6 @@ impl IndexingService {
paths: &[String],
db_path: &str,
stop_flag: &Arc<AtomicBool>,
suspend_flag: &Arc<AtomicBool>,
config: &Config,
db_connection: &Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
interrupt: &db::InterruptSlot,
@ -637,19 +753,7 @@ impl IndexingService {
let count_cancel = Arc::new(AtomicBool::new(false));
let _count_guard = CancelOnDrop(count_cancel.clone());
let mut cx = RunCx {
conn_mutex,
config,
db_path,
stop_flag,
suspend_flag,
registry: Arc::new(Registry::default_set()),
quantum: config.processing.batch_size.max(1),
seen_paths: HashSet::new(),
stale_candidates: Vec::new(),
aliased_paths: HashSet::new(),
stale_cleanup_ok: true,
};
let mut cx = RunCx::new(conn_mutex, config, db_path, stop_flag);
// Read stored counts up front, under one lock, before the walks
// compete for the connection.
@ -689,10 +793,19 @@ impl IndexingService {
};
let mut checkpoint_at = wal_cap;
// Round-robin with skipping: each round takes at most one quantum
// from every root that has work ready.
// Walks first, one slice each, then a single extraction slice.
//
// The walk is the disk-bound phase and the one whose stall shows: its
// workers can only run as far ahead as their channel, so a writer that
// does not come back to it soon enough parks a whole pool behind one
// root's tokenizing. Serving every walking root before any extraction
// caps a walk's wait at one slice per round; taking one extraction
// slice per round, not one per root, keeps that cap independent of
// how many roots are extracting — while still handing extraction a
// slice every round, so it is never starved either. Any root's turn
// ends early the moment it has nothing ready.
loop {
if should_abort(stop_flag, suspend_flag) {
if stop_flag.load(Ordering::Relaxed) {
aborted = true;
break;
}
@ -700,11 +813,16 @@ impl IndexingService {
let n = pipelines.len();
for k in 0..n {
let p = &mut pipelines[(rr + k) % n];
progressed |= match p.phase {
RootPhase::Walking => p.service_walking(&mut cx)?,
RootPhase::Extracting => p.service_extracting(&mut cx)?,
RootPhase::Done => false,
};
if p.phase == RootPhase::Walking {
progressed |= p.service_walking(&mut cx)?;
}
}
for k in 0..n {
let p = &mut pipelines[(rr + k) % n];
if p.phase == RootPhase::Extracting {
progressed |= p.service_extracting(&mut cx)?;
break;
}
}
rr = rr.wrapping_add(1);
@ -766,15 +884,19 @@ impl IndexingService {
}
if aborted {
// Buffered records are valid work — land them before leaving.
for p in &mut pipelines {
process_batch_updates(&cx.conn_mutex, &p.pending_updates, stop_flag, config)?;
p.pending_updates.clear();
process_batch_inserts(&cx.conn_mutex, &p.pending_inserts, stop_flag, config)?;
p.pending_inserts.clear();
}
// No stale cleanup: a partial walk's seen set would delete most
// of the index.
// Nothing is landed on the way out, and there used to be a loop
// here that looked as though it did: `aborted` implies the stop
// flag is set, and both batch writers return on it before their
// first chunk, so it wrote nothing. What a stop drops is each
// root's part-filled insert/update batch (under `batch_size`
// rows) and whatever extraction had ready — all of it still
// `content_state = 0` or absent, so the next run finds it again.
// That is what "a stopped run promises nothing" already means,
// and it is cheaper than tokenizing a slice's worth of documents
// while someone waits for the window to close.
//
// No stale cleanup either: a partial walk's seen set would delete
// most of the index.
report_run_warnings();
crate::log_info!(
"indexing stopped after {:.1}s",

25
crates/quicksearch-core/src/indexing/progress.rs
View file

@ -26,12 +26,15 @@ pub struct RootProgress {
/// counts tree *entries* and so reads high. Read it through
/// [`RootProgress::walk_denominator`].
pub walk_total: Option<usize>,
/// Rows with searchable text: extracted in earlier runs plus this one.
/// Rows with searchable text: this run's, plus earlier runs' once
/// `extract_total` is known.
pub extracted: usize,
/// The root's whole searchable set: pending + already-extracted rows when
/// the walk finished — the count of files that have or will have text,
/// not of files under the root.
pub extract_total: usize,
/// not of files under the root. `None` until the root's content pass has
/// counted its range: a scan that takes seconds on a large root, and one
/// that used to run on the writer thread with every other root waiting.
pub extract_total: Option<usize>,
pub current_file: Option<String>,
/// Threads busy right now / pool size, for the pool this root's current
/// phase is running. Both zero once the root is done: its threads are
@ -80,14 +83,22 @@ impl OverallProgress {
}
/// Aggregate every root's progress into the one pair the status bar shows.
/// `extract_total` is exact the moment a root's walk ends; before that a
/// root contributes only its walk.
///
/// A root contributes its extraction half only once `extract_total` is known
/// — both to `processed` and to `total`, so the two stay in step. Until then
/// (during the walk, and for the moments after it while the pass counts) it
/// contributes its walk alone.
pub fn overall_progress(roots: &[RootProgress]) -> OverallProgress {
let processed = roots.iter().map(|r| r.walked + r.extracted).sum();
let processed = roots
.iter()
.map(|r| r.walked + r.extract_total.map_or(0, |_| r.extracted))
.sum();
let mut total = Some(0usize);
for r in roots {
match (total, r.walk_denominator()) {
(Some(acc), Some(walk)) => total = Some(acc + walk + r.extract_total),
(Some(acc), Some(walk)) => {
total = Some(acc + walk + r.extract_total.unwrap_or(0));
}
_ => {
total = None;
break;

183
crates/quicksearch-core/src/indexing/tests.rs
View file

@ -4,6 +4,7 @@ use crate::extract::Registry;
use crate::file_handling::ExtractCursor;
use crate::walk::walk_indexable_files;
use std::sync::atomic::AtomicUsize;
use std::time::Duration;
fn tmp_dir(tag: &str) -> std::path::PathBuf {
// Canonical: the temp dir itself may sit behind a symlink
@ -86,7 +87,6 @@ fn worker_counts_follow_the_phase() {
let root = dir.to_string_lossy().into_owned();
let stop = Arc::new(AtomicBool::new(false));
let suspend = Arc::new(AtomicBool::new(false));
let walk = walk_indexable_files(
std::slice::from_ref(&root),
false,
@ -96,7 +96,6 @@ fn worker_counts_follow_the_phase() {
Config::default(),
Arc::new(Registry::default_set()),
stop.clone(),
suspend.clone(),
3,
);
// An empty range, so the pass ends immediately — but its pool size is
@ -107,7 +106,6 @@ fn worker_counts_follow_the_phase() {
Arc::new(Registry::default_set()),
Config::default(),
stop,
suspend,
2,
);
@ -123,8 +121,9 @@ fn worker_counts_follow_the_phase() {
phase: RootPhase::Walking,
phase_started: Instant::now(),
content: Some(content),
extract_total: 0,
extracted: 0,
ready: Vec::new(),
written: 0,
totals: None,
current_file: None,
};
@ -138,6 +137,157 @@ fn worker_counts_follow_the_phase() {
std::fs::remove_dir_all(&dir).ok();
}
/// The writer's extraction turn is bounded by its slice, not by what is
/// ready: rows the slice does not reach are carried to the next turn, and the
/// root is not `Done` until they have all landed. Pinned with a zero slice,
/// under which every turn writes exactly one row.
#[test]
fn an_extracting_turn_lands_its_leftovers_one_slice_at_a_time() {
use super::pipeline::RunCx;
use crate::content::ExtractedRow;
use crate::db::repo::{insert_file, NewFile};
use crate::file_handling::ContentOutcome;
use crate::mime::FileType;
let dir = tmp_dir("slice-leftovers");
let db_path = dir.join("index.db").to_string_lossy().into_owned();
let mut conn = db::open_or_recreate(&db_path, "trigram").unwrap();
let tree = dir.join("tree");
std::fs::create_dir_all(&tree).unwrap();
// Five rows the walk would have written, whose extracted text is
// hand-built below rather than read back — the pass is not the subject.
let mut ready: Vec<ExtractedRow> = Vec::new();
{
let tx = conn.transaction().unwrap();
for i in 0..5 {
let path = tree.join(format!("f{}.txt", i));
std::fs::write(&path, "sphinx of black quartz").unwrap();
let file_id = insert_file(
&tx,
&NewFile {
name: &format!("f{}.txt", i),
path: &path.to_string_lossy(),
parent: &tree.to_string_lossy(),
size: 22,
mtime: 1,
inode: None,
device_id: None,
mime: Some("text/plain"),
ftype: FileType::TEXT,
hash: None,
needs_content: true,
},
)
.unwrap()
.expect("unique path");
ready.push(ExtractedRow {
file_id,
name: format!("f{}.txt", i),
outcome: ContentOutcome::Done {
text: format!("sphinx of black quartz {}", i),
properties: Vec::new(),
},
});
}
tx.commit().unwrap();
}
let conn_mutex = Arc::new(Mutex::new(conn));
let root = dir.to_string_lossy().into_owned();
let stop = Arc::new(AtomicBool::new(false));
let config = Config::default();
let walk = walk_indexable_files(
std::slice::from_ref(&root),
false,
false,
crate::config::IgnoreSet::compile(&[]).unwrap(),
&db_path,
config.clone(),
Arc::new(Registry::default_set()),
stop.clone(),
1,
);
// An empty range: the pass reports `Finished` on its own, and the turn
// has to keep going past that until `ready` is empty.
let content = crate::content::extract_content(
&db_path,
&ExtractCursor::for_root(&dir.join("nothing").to_string_lossy()),
Arc::new(Registry::default_set()),
config.clone(),
stop.clone(),
1,
);
let mut p = RootPipeline {
root,
walk,
count_total: Arc::new(AtomicUsize::new(0)),
workers: 1,
pending_updates: Vec::new(),
pending_inserts: Vec::new(),
walked: 0,
walk_clean: true,
phase: RootPhase::Extracting,
phase_started: Instant::now(),
content: Some(content),
ready,
written: 0,
totals: None,
current_file: None,
};
let mut cx = RunCx::new(conn_mutex.clone(), &config, &db_path, &stop);
cx.slice = Duration::ZERO;
let mut turns = 0;
while p.phase == RootPhase::Extracting {
turns += 1;
assert!(
turns < 200,
"the root never finished: {} written",
p.written
);
let before = p.written;
let progressed = p.service_extracting(&mut cx).unwrap();
assert!(
p.written - before <= 1,
"a zero slice wrote {} rows in one turn",
p.written - before
);
assert!(
p.phase != RootPhase::Done || p.ready.is_empty(),
"Done with {} rows still to write",
p.ready.len()
);
if !progressed {
// The empty pass has not reported `Finished` yet.
std::thread::sleep(Duration::from_millis(1));
}
}
assert_eq!(p.written, 5);
assert!(p.ready.is_empty());
assert!(
turns >= 5,
"five rows cannot land in {} zero-slice turns",
turns
);
let done: i64 = conn_mutex
.lock()
.unwrap()
.query_row(
"SELECT COUNT(*) FROM files WHERE content_state = 1",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(done, 5, "every row reached the index");
// The empty pass counted its (empty) range, so the totals are known and
// the snapshot reports this run's rows on top of the range's zero.
assert_eq!(p.snapshot().extracted, 5);
assert_eq!(p.snapshot().extract_total, Some(0));
drop(p);
std::fs::remove_dir_all(&dir).ok();
}
/// One full run over `config`'s roots, driven directly so the caller owns
/// the stop flag. Returns when the run does.
fn run_with(config: &Config, db_path: &str, stop: &Arc<AtomicBool>) -> Result<(), String> {
@ -146,7 +296,6 @@ fn run_with(config: &Config, db_path: &str, stop: &Arc<AtomicBool>) -> Result<()
&config.paths.indexing_paths,
db_path,
stop,
&Arc::new(AtomicBool::new(false)),
config,
&Arc::new(Mutex::new(None)),
&db::InterruptSlot::default(),
@ -374,7 +523,7 @@ fn progress(phase: RootPhase, walked: usize, walk_total: Option<usize>) -> RootP
walked,
walk_total,
extracted: 0,
extract_total: 0,
extract_total: None,
current_file: None,
active_workers: 0,
total_workers: 0,
@ -426,7 +575,7 @@ fn overall_progress_sums_both_halves_of_every_root() {
let mut walking = progress(RootPhase::Walking, 100, Some(1000));
let mut extracting = progress(RootPhase::Extracting, 500, Some(9999));
extracting.extracted = 200;
extracting.extract_total = 400;
extracting.extract_total = Some(400);
walking.extracted = 0;
let o = overall_progress(&[walking, extracting]);
@ -435,6 +584,19 @@ fn overall_progress_sums_both_halves_of_every_root() {
assert_eq!(o.total, Some(1900));
}
/// A root whose content pass has not counted its range yet contributes only
/// its walk — to both halves, so processed and total stay in step and the
/// bar cannot jump when the count lands.
#[test]
fn an_uncounted_extraction_contributes_only_its_walk() {
let mut counting = progress(RootPhase::Extracting, 500, None);
counting.extracted = 7;
counting.extract_total = None;
let o = overall_progress(&[counting]);
assert_eq!(o.processed, 500);
assert_eq!(o.total, Some(500));
}
#[test]
fn one_uncounted_walking_root_leaves_the_whole_total_unknown() {
let known = progress(RootPhase::Done, 10, Some(10));
@ -472,7 +634,7 @@ fn a_finished_run_reaches_exactly_one_hundred_percent() {
.map(|&(walked, extracted)| {
let mut p = progress(RootPhase::Done, walked, Some(walked * 2));
p.extracted = extracted;
p.extract_total = extracted;
p.extract_total = Some(extracted);
p
})
.collect();
@ -496,6 +658,9 @@ fn a_run_with_nothing_to_do_has_no_fraction_to_show() {
fn the_fraction_never_exceeds_one() {
let mut p = progress(RootPhase::Done, 10, None);
p.extracted = 100;
// A counted scope the writes then overran; an uncounted one would be
// left out of both halves and prove nothing here.
p.extract_total = Some(0);
let o = overall_progress(&[p]);
assert_eq!(o.processed, 110);
assert_eq!(o.total, Some(10));

2
crates/quicksearch-core/src/lib.rs
View file

@ -7,6 +7,7 @@ pub mod extract;
pub mod file_handling;
pub mod incremental;
pub mod indexing;
pub mod live;
pub mod log;
pub mod mime;
pub mod platform;
@ -19,6 +20,7 @@ pub mod snippet;
#[doc(hidden)]
pub mod testutil;
pub mod textenc;
pub mod verify;
pub mod walk;
pub mod watcher;

734
crates/quicksearch-core/src/live.rs Normal file
View file

@ -0,0 +1,734 @@
//! Watching the search results a frontend is actually showing.
//!
//! What a row shows is read from the *file*, never from the index. That is
//! what lets this work with indexing stopped, with a file outside every
//! indexed root, or against a row the indexer has not caught up with yet —
//! and it is the whole point of the feature: the list on screen describes the
//! disk, not a snapshot of it.
//!
//! Nothing here writes to the index. Keeping it so is what lets this run
//! alongside the indexer without a second writer; a frontend that wants the
//! index brought back in line with what it just displayed hands the paths to
//! [`crate::coordinator::IndexCoordinator::update_paths`], which does the
//! write on its own thread.
//!
//! # Why directories, not files
//!
//! The obvious design is a watch per result file. It does not work. Editors
//! save by writing a temporary file and renaming it over the target, so the
//! event lands on the *directory* and the old inode — the one a file watch is
//! attached to — is simply orphaned. A file watch also cannot report the new
//! name of a rename. Watching the deduplicated set of parent directories
//! `NonRecursive` sees both, on inotify and on `ReadDirectoryChangesW` alike.
//!
//! # Why this is not [`crate::watcher`]
//!
//! That module exists to cover *subtrees*: it walks each root registering
//! every directory beneath it, adds directories that appear later, and backs a
//! 128k budget with an all-or-nothing guarantee. Pointing it at a result's
//! parent would register that parent's whole tree. This is a flat, fixed, tiny
//! set with no growth and no budget, and its timings are a tenth of that one's
//! — the indexer can afford to coalesce for thirty seconds, a cursor blinking
//! next to a stale filename cannot.
//!
//! # What an event turns into
//!
//! A rename is applied from the event itself. A content change is answered by
//! reading the file: `metadata` for size and modified time, and — for a row
//! whose cell shows body text — the same MIME sniffing and extractors the
//! indexer uses, re-cut through the same [`crate::search::cascade::text_snippet`]
//! (or, for a fuzzy hit, [`crate::search::cascade::fuzzy_snippet`]) the
//! search itself uses.
//!
//! Arming also sweeps every target once, comparing the file on disk against
//! the size and modified time the row is *currently displaying*. Since a fresh
//! result carries what the index said, that sweep is exactly a check of the
//! index against the disk, and it is what makes a row corrected while it was
//! scrolled out of view right itself the moment it comes back. It is also the
//! only thing that works on a filesystem the platform reports no events for.
use std::collections::HashMap;
use std::io::Read;
use std::path::{Path, PathBuf};
use std::sync::mpsc;
use std::sync::Arc;
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
use notify::{
Config as NotifyConfig, Event as NotifyEvent, EventKind, RecommendedWatcher, RecursiveMode,
Watcher as NotifyWatcher,
};
use crate::config::Config;
use crate::extract::Registry;
use crate::query::split::split_for_cascade;
use crate::search::fuzzy::{edit_budget, Bitap};
use crate::search::ContentTier;
use crate::snippet::Snippet;
/// How long events for one path are pooled before being acted on, so the
/// several notify emits behind a single save collapse into one update.
const SETTLE: Duration = Duration::from_millis(150);
/// Floor on how often any one path may produce an update. A file being written
/// in a loop — a log, a build artifact — cannot spin the UI.
const MIN_INTERVAL: Duration = Duration::from_millis(750);
/// How often the loop wakes while it has work pending. It blocks outright when
/// it has none, so an idle QuickSearch does not tick at all.
const TICK: Duration = Duration::from_millis(50);
/// Ceiling on updates emitted per tick, so a directory-wide change (an
/// unpack, a `chmod -R`) drains over several frames instead of one.
const MAX_PER_TICK: usize = 4;
/// Most watches to register. Results cluster hard — a query's hits usually
/// share a handful of directories — so this is generous for the visible rows
/// while staying negligible against the indexer's 128k budget.
const MAX_DIRS: usize = 64;
/// Most rows to track, whatever the frontend asks for.
const MAX_TARGETS: usize = 256;
/// One row the frontend is showing and wants kept current.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Target {
/// The row's path, spelled exactly as the search returned it. Event paths
/// are compared against this byte for byte — see the note in [`watch`].
///
/// [`watch`]: LiveWatcher::watch
pub path: String,
/// `Some` when this row displays text from the file's *body*, and so
/// needs its snippet re-cut when the file changes — and how, since an
/// exact-tier window is cut around the literal term and a fuzzy one
/// around a bitap match the literal is usually absent from. `None` for a
/// filename or path match, which costs one `metadata` call per change and
/// never opens the file.
pub text: Option<ContentTier>,
/// The size the row is displaying. The arm-time sweep compares the file
/// against this, so on a fresh result — where it is whatever the index
/// said — the sweep doubles as a check of the index against the disk.
pub size: u64,
/// The modified time the row is displaying; see [`Target::size`].
pub mtime: i64,
}
/// What a change did to a row's Content Match window.
///
/// Three states, not an `Option`, because "I did not look" and "I looked and
/// it is not there any more" have to reach the frontend as different answers.
/// Blanking a cell because the file was too large to re-read would lose a
/// window the search legitimately found.
#[derive(Debug, Clone, PartialEq)]
pub enum WindowUpdate {
/// Nothing to say: not a body-text row, or its body could not be re-read
/// (too large, no extractor, unreadable). The cell keeps what it has.
Unchanged,
/// Re-cut from the file as it is now.
Cut(Snippet),
/// The body was read and the query is no longer in it. The cell has
/// nothing to show and falls back to its dash.
NoMatch,
}
/// A ready-to-apply change to one row on screen.
#[derive(Debug, Clone, PartialEq)]
pub enum LiveUpdate {
/// The file moved. `path` is the row's old path — the frontend's key.
Renamed {
path: String,
to: String,
name: String,
},
/// The file's contents changed, as read from the file itself.
Changed {
path: String,
size: u64,
mtime: i64,
window: WindowUpdate,
},
/// The file is no longer there. Reversible: the directory watch stays, so
/// a file recreated at the same path reports [`LiveUpdate::Changed`].
Gone { path: String },
}
impl LiveUpdate {
/// The row this update is keyed by — the path the frontend knows it as.
pub fn path(&self) -> &str {
match self {
LiveUpdate::Renamed { path, .. }
| LiveUpdate::Changed { path, .. }
| LiveUpdate::Gone { path } => path,
}
}
}
/// What one settled event window decided about a path, before the filesystem
/// is consulted.
#[derive(Debug, Clone, PartialEq)]
enum Op {
Changed,
Renamed(PathBuf),
/// Provisional: on Linux the `From` half of a rename arrives before the
/// paired event that names the destination, so this may still be upgraded
/// to [`Op::Renamed`] inside the same window.
Gone,
}
/// Commands and events share one channel so the loop can block on `recv()`
/// whenever nothing is pending.
enum Msg {
Event(NotifyEvent),
Watch {
query: String,
targets: Vec<Target>,
/// Boxed: this is by far the largest variant, and a `Watch` is rare
/// next to the events sharing the channel with it.
config: Box<Config>,
},
Clear,
Stop,
}
/// Handle on the watcher thread. Dropping it stops the thread.
pub struct LiveWatcher {
tx: mpsc::Sender<Msg>,
handle: Option<JoinHandle<()>>,
}
impl LiveWatcher {
/// Spawn the watcher. `notify` is called after every update is queued, so
/// an egui frontend can `request_repaint`; pass a no-op for headless use.
pub fn start(notify: Arc<dyn Fn() + Send + Sync>) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>) {
let (tx, rx) = mpsc::channel::<Msg>();
let (update_tx, update_rx) = mpsc::channel::<LiveUpdate>();
let event_tx = tx.clone();
let handle = thread::Builder::new()
.name("qs-live".into())
.spawn(move || {
Loop {
rx,
event_tx,
update_tx,
notify,
watcher: None,
targets: HashMap::new(),
pending: HashMap::new(),
last_emit: HashMap::new(),
orphan_to: Vec::new(),
settle_at: None,
query: None,
fuzzy: None,
config: None,
registry: Registry::default_set(),
}
.run()
})
.expect("spawn live watcher");
(
LiveWatcher {
tx,
handle: Some(handle),
},
update_rx,
)
}
/// Replace the watched set wholesale.
///
/// `query` is the search these rows came from; it is what a re-cut snippet
/// is marked against. `config` supplies the extraction limits and filters,
/// so a snippet cut here is the text the indexer would have stored.
/// Registration happens on the watcher's own thread, so this never blocks
/// the caller on a spun-down disk or a stale mount.
pub fn watch(&self, query: &str, targets: Vec<Target>, config: &Config) {
let _ = self.tx.send(Msg::Watch {
query: query.to_string(),
targets,
config: Box::new(config.clone()),
});
}
/// Drop every watch and forget every pending update.
pub fn clear(&self) {
let _ = self.tx.send(Msg::Clear);
}
/// Stop the thread and join it. Idempotent; [`Drop`] calls it.
pub fn stop(&mut self) {
let _ = self.tx.send(Msg::Stop);
if let Some(handle) = self.handle.take() {
let _ = handle.join();
}
}
}
impl Drop for LiveWatcher {
fn drop(&mut self) {
self.stop();
}
}
struct Loop {
rx: mpsc::Receiver<Msg>,
event_tx: mpsc::Sender<Msg>,
update_tx: mpsc::Sender<LiveUpdate>,
notify: Arc<dyn Fn() + Send + Sync>,
watcher: Option<RecommendedWatcher>,
targets: HashMap<String, Target>,
pending: HashMap<String, Op>,
last_emit: HashMap<String, Instant>,
/// Rename destinations seen in this window whose source is not a target —
/// the other half of a Windows rename, which carries no pairing cookie.
orphan_to: Vec<PathBuf>,
settle_at: Option<Instant>,
query: Option<crate::query::split::CascadeQuery>,
/// The fuzzy matcher for `query`, built at arm time — building it is the
/// cost, running it is cheap — for re-cutting [`ContentTier::Fuzzy`]
/// rows. `None` when the term does not fuzz (too short, wildcarded, or a
/// zero edit budget), which is when the fuzzy pass would not have run.
fuzzy: Option<Bitap>,
config: Option<Box<Config>>,
/// Built once and reused: the extractors are stateless, and the frontend
/// re-arms often enough that rebuilding the table per arm would be waste.
registry: Registry,
}
impl Loop {
fn run(mut self) {
loop {
// Block outright when there is nothing to time out on: an idle
// window costs no wakeups at all.
let msg = match self.settle_at {
None => match self.rx.recv() {
Ok(msg) => Some(msg),
Err(_) => return,
},
Some(deadline) => {
let wait = deadline.saturating_duration_since(Instant::now());
match self.rx.recv_timeout(wait.min(TICK)) {
Ok(msg) => Some(msg),
Err(mpsc::RecvTimeoutError::Timeout) => None,
Err(mpsc::RecvTimeoutError::Disconnected) => return,
}
}
};
match msg {
Some(Msg::Stop) => return,
Some(Msg::Clear) => self.reset(),
Some(Msg::Watch {
query,
targets,
config,
}) => self.rearm(&query, targets, config),
Some(Msg::Event(event)) => {
classify(
&event,
&self.targets,
&mut self.pending,
&mut self.orphan_to,
);
if !self.pending.is_empty() && self.settle_at.is_none() {
self.settle_at = Some(Instant::now() + SETTLE);
}
}
None => {}
}
self.flush_settled();
}
}
fn reset(&mut self) {
self.watcher = None;
self.targets.clear();
self.pending.clear();
self.last_emit.clear();
self.orphan_to.clear();
self.settle_at = None;
self.query = None;
self.fuzzy = None;
self.config = None;
}
/// Point the watcher at a new set of rows, dropping everything about the
/// old one, then check each one against the disk. Registration failures
/// are per-directory and silent beyond the log: this is a cosmetic
/// feature, and a modal about a missing highlight would be worse than the
/// missing highlight.
fn rearm(&mut self, query: &str, targets: Vec<Target>, config: Box<Config>) {
self.reset();
if targets.is_empty() {
return;
}
self.query = split_for_cascade(query).ok();
// The same construction the fuzzy pass makes for a scan, so a fuzzy
// row is re-cut against exactly what matched it.
self.fuzzy = self.query.as_ref().and_then(|q| {
if q.pattern.is_wildcard() {
return None;
}
let folded = q.term.to_ascii_lowercase();
let k = edit_budget(folded.len(), config.search.fuzzy_max_edits)?;
Bitap::new(folded.as_bytes(), k)
});
self.config = Some(config);
let mut watcher = {
let tx = self.event_tx.clone();
let sink = move |res: notify::Result<NotifyEvent>| {
if let Ok(event) = res {
let _ = tx.send(Msg::Event(event));
}
};
match RecommendedWatcher::new(sink, NotifyConfig::default()) {
Ok(w) => w,
Err(e) => {
crate::log_warn!("live results: no watcher available: {}", e);
return;
}
}
};
let mut dirs: Vec<PathBuf> = Vec::new();
for target in targets.into_iter().take(MAX_TARGETS) {
// Derived from the row's own path, never canonicalized: notify
// builds each event path as `watched_dir.join(name)`, so leaving
// this spelled as the index spells it is what lets event paths be
// compared to `Target::path` as plain strings.
let Some(dir) = Path::new(&target.path).parent().map(Path::to_path_buf) else {
continue;
};
if !dirs.contains(&dir) {
if dirs.len() >= MAX_DIRS {
continue;
}
if let Err(e) = watcher.watch(&dir, RecursiveMode::NonRecursive) {
// Deliberately not the indexer's all-or-nothing: partial
// coverage of a display nicety is fine. Rows under this
// directory are dropped rather than swept — a row nothing
// can follow is better left alone than corrected once and
// then silently frozen.
crate::log_warn!("live results: not watching {}: {}", dir.display(), e);
continue;
}
dirs.push(dir);
}
self.targets.insert(target.path.clone(), target);
}
if self.targets.is_empty() {
return;
}
self.watcher = Some(watcher);
self.sweep();
}
/// Compare every target against the disk once and emit what disagrees.
///
/// Nothing here waits for an event, which is the point: a row whose file
/// changed while it was scrolled out of view — or one whose index row was
/// simply out of date when the search returned it — is corrected the
/// moment it is watched.
fn sweep(&mut self) {
let mut updates: Vec<LiveUpdate> = Vec::new();
for target in self.targets.values() {
match std::fs::metadata(&target.path) {
Ok(meta) if meta.is_file() => {
let (size, mtime) = (meta.len(), mtime_of(&meta));
if size == target.size && mtime == target.mtime {
continue;
}
updates.push(self.changed_update(&target.path, target.text, size, mtime));
}
// Unreadable counts as gone: the row cannot be shown as
// current when we cannot see the file at all.
_ => updates.push(LiveUpdate::Gone {
path: target.path.clone(),
}),
}
}
let now = Instant::now();
for update in updates {
// Recorded so an event arriving right behind the sweep — a save
// still in flight when the row came on screen — does not repeat
// the same answer a moment later.
self.last_emit.insert(update.path().to_string(), now);
self.note_emitted(&update);
self.send(update);
}
}
/// Turn the settled window's decisions into updates.
fn flush_settled(&mut self) {
let Some(at) = self.settle_at else { return };
if Instant::now() < at {
return;
}
self.settle_at = None;
// Windows never reports the two halves of a rename as one event and
// gives no cookie to pair them by. When exactly one target went away
// and exactly one unclaimed destination appeared in the same window,
// they are the same file; anything more ambiguous resolves to the
// truthful "gone".
let orphans = std::mem::take(&mut self.orphan_to);
let gone: Vec<String> = self
.pending
.iter()
.filter(|(_, op)| **op == Op::Gone)
.map(|(path, _)| path.clone())
.collect();
if gone.len() == 1 && orphans.len() == 1 {
self.pending
.insert(gone[0].clone(), Op::Renamed(orphans[0].clone()));
}
let now = Instant::now();
let ready: Vec<(String, Op)> = self
.pending
.iter()
.filter(|(path, _)| {
self.last_emit
.get(*path)
.is_none_or(|t| now.duration_since(*t) >= MIN_INTERVAL)
})
.take(MAX_PER_TICK)
.map(|(path, op)| (path.clone(), op.clone()))
.collect();
for (path, op) in ready {
self.pending.remove(&path);
self.last_emit.insert(path.clone(), now);
self.apply(path, op);
}
if !self.pending.is_empty() {
self.settle_at = Some(now + SETTLE);
}
}
fn apply(&mut self, path: String, op: Op) {
let update = match op {
Op::Gone => LiveUpdate::Gone { path },
Op::Renamed(to) => {
let name = to
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_default();
LiveUpdate::Renamed {
path,
to: to.to_string_lossy().into_owned(),
name,
}
}
Op::Changed => {
// Read from the file, not from the index: this has to land
// whatever the indexer is doing, or not doing.
match std::fs::metadata(&path) {
Ok(meta) if meta.is_file() => {
let text = self.targets.get(&path).and_then(|t| t.text);
self.changed_update(&path, text, meta.len(), mtime_of(&meta))
}
_ => LiveUpdate::Gone { path },
}
}
};
self.note_emitted(&update);
self.send(update);
}
/// What the file at `path` implies for the row showing it.
fn changed_update(
&self,
path: &str,
text: Option<ContentTier>,
size: u64,
mtime: i64,
) -> LiveUpdate {
LiveUpdate::Changed {
path: path.to_string(),
size,
mtime,
window: text.map_or(WindowUpdate::Unchanged, |tier| {
self.window_from_disk(path, size, tier)
}),
}
}
/// Re-cut this row's Content Match window from the file on disk.
///
/// Deliberately the indexer's own path — [`crate::mime::guess_mime_from_head`]
/// then [`crate::file_handling::decide_content`], which already applies
/// `content_extensions` and the `maximum_text_size` truncation — so the
/// text a window is cut from is the text the index would have stored, and
/// a refreshed row cannot disagree with a re-run search about anything but
/// timing. Then the tier's own matcher, for the same reason.
fn window_from_disk(&self, path: &str, size: u64, tier: ContentTier) -> WindowUpdate {
let (Some(config), Some(query)) = (self.config.as_deref(), self.query.as_ref()) else {
return WindowUpdate::Unchanged;
};
// The indexer would not have stored text for a file this large, so
// neither does the row — reading it would stall this thread over a
// window nobody can see the whole of anyway.
if size > config.processing.maximum_text_file_size {
return WindowUpdate::Unchanged;
}
let file = Path::new(path);
let Some(head) = read_head(file, config.processing.hash_length) else {
return WindowUpdate::Unchanged;
};
let mime = crate::mime::guess_mime_from_head(file, &head);
// Extractors run third-party parsers over whatever the file now
// holds. One that panics must not take this thread — and with it every
// live row for the rest of the session — down with it.
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
crate::file_handling::decide_content(path, mime.as_deref(), &self.registry, config)
}));
let outcome = match outcome {
Ok(outcome) => outcome,
Err(_) => {
crate::log_warn!("live results: extracting {} panicked", path);
return WindowUpdate::Unchanged;
}
};
let Some(text) = crate::file_handling::outcome_body(&outcome) else {
return WindowUpdate::Unchanged;
};
let folded = text.to_ascii_lowercase();
let cut = match tier {
ContentTier::Exact => {
// A literal term always yields a window, marked or not,
// because the passes only ever call this for a body FTS
// already matched. Here the body may genuinely have stopped
// matching, and an unmarked window is how that reads.
crate::search::cascade::text_snippet(&query.pattern, text, &folded)
.filter(|snip| !snip.ranges.is_empty())
}
ContentTier::Fuzzy => match &self.fuzzy {
Some(bitap) => {
crate::search::cascade::fuzzy_snippet(bitap, text, &folded).map(|(_, s)| s)
}
// The term does not fuzz, so a fuzzy row cannot be re-judged;
// leaving it is the honest reading.
None => return WindowUpdate::Unchanged,
},
};
match cut {
Some(snip) => WindowUpdate::Cut(snip),
None => WindowUpdate::NoMatch,
}
}
/// Keep the target's baseline in step with what the frontend was just
/// told, so a later sweep over the same arm does not repeat itself.
fn note_emitted(&mut self, update: &LiveUpdate) {
let LiveUpdate::Changed {
path, size, mtime, ..
} = update
else {
return;
};
if let Some(target) = self.targets.get_mut(path) {
target.size = *size;
target.mtime = *mtime;
}
}
fn send(&self, update: LiveUpdate) {
if self.update_tx.send(update).is_ok() {
(self.notify)();
}
}
}
/// The first `limit` bytes of a file, for MIME sniffing. A short read is the
/// whole file and is not an error; an unreadable file simply has no MIME.
fn read_head(path: &Path, limit: usize) -> Option<Vec<u8>> {
let file = std::fs::File::open(path).ok()?;
let mut head = Vec::new();
file.take(limit as u64).read_to_end(&mut head).ok()?;
Some(head)
}
fn mtime_of(meta: &std::fs::Metadata) -> i64 {
meta.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
/// Fold one notify event into the pending decisions for this window.
///
/// Pure and filesystem-free, which is what makes the platform differences
/// testable: every shape below is a real emission from `notify` 6.1 on one
/// platform or the other, and the coalescing window is what reconciles them.
fn classify(
event: &NotifyEvent,
targets: &HashMap<String, Target>,
pending: &mut HashMap<String, Op>,
orphan_to: &mut Vec<PathBuf>,
) {
use notify::event::{ModifyKind, RenameMode};
let key = |p: &PathBuf| p.to_string_lossy().into_owned();
let is_target = |p: &PathBuf| targets.contains_key(&key(p));
match event.kind {
EventKind::Modify(ModifyKind::Name(RenameMode::Both)) => {
// Linux pairs the halves and emits this *after* the From/To pair,
// so it lands in the same window and overwrites the provisional
// Gone recorded below.
let (Some(from), Some(to)) = (event.paths.first(), event.paths.get(1)) else {
return;
};
if is_target(from) {
pending.insert(key(from), Op::Renamed(to.clone()));
} else if is_target(to) {
// The atomic-save shape: a temporary file renamed over a row
// we are watching. The row did not move; its contents changed.
pending.insert(key(to), Op::Changed);
}
}
EventKind::Modify(ModifyKind::Name(RenameMode::To)) => {
for path in &event.paths {
if is_target(path) {
pending.insert(key(path), Op::Changed);
} else {
orphan_to.push(path.clone());
}
}
}
EventKind::Modify(ModifyKind::Name(RenameMode::From)) => {
for path in &event.paths {
if is_target(path) {
// Provisional; a Both in this same window upgrades it.
pending.entry(key(path)).or_insert(Op::Gone);
}
}
}
EventKind::Remove(_) => {
for path in &event.paths {
if is_target(path) {
pending.insert(key(path), Op::Gone);
}
}
}
EventKind::Create(_) | EventKind::Modify(_) => {
for path in &event.paths {
if is_target(path) {
// A Create at a watched path un-deletes the row.
pending.insert(key(path), Op::Changed);
}
}
}
// Access events, and anything for a path we are not showing. Live
// results never *add* rows: we have no way to know an unrelated new
// file matches the query, and guessing would be a second, unranked
// search wearing the first one's clothes.
_ => {}
}
}
#[cfg(test)]
#[path = "live_tests.rs"]
mod tests;

655
crates/quicksearch-core/src/live_tests.rs Normal file
View file

@ -0,0 +1,655 @@
//! Tests for live result watching.
//!
//! The [`classify`] tests are pure: they feed the exact event shapes `notify`
//! 6.1 emits on each platform and check what one settled window decides. They
//! are the ones that pin the design — in particular that a watch on a
//! *directory* sees an atomic save, which a watch on the file would not.
//!
//! The end-to-end tests drive a real filesystem through a real watcher.
use super::*;
use notify::event::{CreateKind, ModifyKind, RemoveKind, RenameMode};
fn target(path: &str) -> (String, Target) {
(
path.to_string(),
Target {
path: path.to_string(),
text: Some(ContentTier::Exact),
size: 0,
mtime: 0,
},
)
}
fn targets(paths: &[&str]) -> HashMap<String, Target> {
paths.iter().map(|p| target(p)).collect()
}
fn event(kind: EventKind, paths: &[&str]) -> NotifyEvent {
NotifyEvent {
kind,
paths: paths.iter().map(PathBuf::from).collect(),
attrs: Default::default(),
}
}
/// Feed a window of events and report what it decided, after the same
/// orphan-pairing `flush_settled` applies.
fn window(targets: &HashMap<String, Target>, events: Vec<NotifyEvent>) -> HashMap<String, Op> {
let mut pending = HashMap::new();
let mut orphan_to = Vec::new();
for event in &events {
classify(event, targets, &mut pending, &mut orphan_to);
}
let gone: Vec<String> = pending
.iter()
.filter(|(_, op)| **op == Op::Gone)
.map(|(p, _)| p.clone())
.collect();
if gone.len() == 1 && orphan_to.len() == 1 {
pending.insert(gone[0].clone(), Op::Renamed(orphan_to[0].clone()));
}
pending
}
/// An editor saving a file writes a temporary and renames it over the target.
/// The row did not move — its contents changed — and a watch on the file
/// itself would have seen none of this, because the inode it was attached to
/// is the one that got orphaned. This test is why the watches are on
/// directories.
#[test]
fn an_atomic_save_reads_as_a_content_change() {
let t = targets(&["/docs/report.txt"]);
let decided = window(
&t,
vec![
event(
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
&["/docs/.report.txt.swp"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
&["/docs/report.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::Both)),
&["/docs/.report.txt.swp", "/docs/report.txt"],
),
],
);
assert_eq!(decided.get("/docs/report.txt"), Some(&Op::Changed));
assert_eq!(decided.len(), 1, "nothing else was decided: {decided:?}");
}
/// Linux emits From, To and then Both for one in-directory rename. The
/// provisional Gone recorded for the From half must not escape the window.
#[test]
fn a_linux_rename_pairs_without_leaking_a_gone() {
let t = targets(&["/docs/old.txt"]);
let decided = window(
&t,
vec![
event(
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
&["/docs/old.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
&["/docs/new.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::Both)),
&["/docs/old.txt", "/docs/new.txt"],
),
],
);
assert_eq!(
decided.get("/docs/old.txt"),
Some(&Op::Renamed(PathBuf::from("/docs/new.txt")))
);
assert!(
!decided.values().any(|op| *op == Op::Gone),
"a provisional Gone escaped: {decided:?}"
);
}
/// Windows never emits `Both` and gives no cookie to pair the halves by, so
/// one unclaimed destination beside one departed target is paired by position.
#[test]
fn a_windows_rename_pairs_by_elimination() {
let t = targets(&["/docs/old.txt"]);
let decided = window(
&t,
vec![
event(
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
&["/docs/old.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
&["/docs/new.txt"],
),
],
);
assert_eq!(
decided.get("/docs/old.txt"),
Some(&Op::Renamed(PathBuf::from("/docs/new.txt")))
);
}
/// Two departures and two arrivals in one window cannot be paired without
/// guessing. Guessing wrong renames a row to someone else's file, so the
/// ambiguous case resolves to the truthful answer instead.
#[test]
fn an_ambiguous_windows_window_reports_gone_rather_than_guessing() {
let t = targets(&["/docs/a.txt", "/docs/b.txt"]);
let decided = window(
&t,
vec![
event(
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
&["/docs/a.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
&["/docs/b.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
&["/docs/x.txt"],
),
event(
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
&["/docs/y.txt"],
),
],
);
assert_eq!(decided.get("/docs/a.txt"), Some(&Op::Gone));
assert_eq!(decided.get("/docs/b.txt"), Some(&Op::Gone));
}
/// A watched directory is full of files that are not on screen. None of them
/// may produce an update — live results never add rows.
#[test]
fn events_for_paths_that_are_not_shown_decide_nothing() {
let t = targets(&["/docs/shown.txt"]);
let decided = window(
&t,
vec![
event(EventKind::Create(CreateKind::File), &["/docs/other.txt"]),
event(EventKind::Modify(ModifyKind::Any), &["/docs/another.txt"]),
event(EventKind::Remove(RemoveKind::File), &["/docs/third.txt"]),
],
);
assert!(decided.is_empty(), "{decided:?}");
}
/// A delete marks the row; a file recreated at the same path un-marks it,
/// which is what makes the mark reversible without re-registering anything.
#[test]
fn a_delete_marks_the_row_and_a_recreate_clears_it() {
let t = targets(&["/docs/report.txt"]);
let gone = window(
&t,
vec![event(
EventKind::Remove(RemoveKind::File),
&["/docs/report.txt"],
)],
);
assert_eq!(gone.get("/docs/report.txt"), Some(&Op::Gone));
let back = window(
&t,
vec![event(
EventKind::Create(CreateKind::File),
&["/docs/report.txt"],
)],
);
assert_eq!(back.get("/docs/report.txt"), Some(&Op::Changed));
}
/// Several writes to one file inside a window are one decision, not several.
#[test]
fn repeated_writes_in_one_window_coalesce() {
let t = targets(&["/docs/log.txt"]);
let decided = window(
&t,
vec![
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
],
);
assert_eq!(decided.len(), 1);
assert_eq!(decided.get("/docs/log.txt"), Some(&Op::Changed));
}
// --- end to end ----------------------------------------------------------
use crate::testutil::scratch_dir;
/// Collect updates until `want` of them arrive or the timeout expires.
fn collect(rx: &mpsc::Receiver<LiveUpdate>, want: usize, timeout: Duration) -> Vec<LiveUpdate> {
let deadline = Instant::now() + timeout;
let mut out = Vec::new();
while out.len() < want {
let left = deadline.saturating_duration_since(Instant::now());
if left.is_zero() {
break;
}
match rx.recv_timeout(left) {
Ok(update) => out.push(update),
Err(_) => break,
}
}
out
}
/// A target describing `path` exactly as it is on disk right now, so the
/// arm-time sweep finds nothing to report and the test sees only what it
/// provokes afterwards.
fn current_target(path: &str, text: Option<ContentTier>) -> Target {
let meta = std::fs::metadata(path).expect("target file exists");
Target {
path: path.to_string(),
text,
size: meta.len(),
mtime: mtime_of(&meta),
}
}
fn watch_one(dir: &Path, name: &str) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>, String) {
watch_one_matching(dir, name, "hello world", None)
}
/// Write `body` at `dir/name` and watch it for the query `hello`.
fn watch_one_matching(
dir: &Path,
name: &str,
body: &str,
text: Option<ContentTier>,
) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>, String) {
let path = dir.join(name).to_string_lossy().into_owned();
std::fs::write(&path, body).unwrap();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
watcher.watch(
"hello",
vec![current_target(&path, text)],
&Config::default(),
);
// Registration happens on the watcher thread.
std::thread::sleep(Duration::from_millis(300));
(watcher, rx, path)
}
/// Stop the watcher, then take the tree with it.
///
/// The stop is not optional: the watcher holds an inotify registration on the
/// directory, and pulling the directory out from under a live one is a race
/// worth not having. Unlike most of this crate's tests these do clean up on
/// the way out — the trees are a file or two apiece, generated identically
/// every run, so they hold no evidence the assertion message does not already
/// carry.
fn stop_and_clean(mut watcher: LiveWatcher, dir: &Path) {
watcher.stop();
std::fs::remove_dir_all(dir).ok();
}
#[test]
fn e2e_a_rename_surfaces_with_the_new_name() {
let dir = scratch_dir("live-rename");
let (watcher, rx, path) = watch_one(&dir, "before.txt");
let renamed = dir.join("after.txt");
std::fs::rename(&path, &renamed).unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let Some(LiveUpdate::Renamed {
path: from, name, ..
}) = updates.first()
else {
panic!("expected a rename, got {updates:?}");
};
assert_eq!(from, &path);
assert_eq!(name, "after.txt");
}
#[test]
fn e2e_a_delete_surfaces_as_gone() {
let dir = scratch_dir("live-delete");
let (watcher, rx, path) = watch_one(&dir, "doomed.txt");
std::fs::remove_file(&path).unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
assert!(
updates
.iter()
.any(|u| matches!(u, LiveUpdate::Gone { path: p } if *p == path)),
"expected a Gone for {path}, got {updates:?}"
);
}
/// The design test, on a real filesystem: write a temporary and rename it over
/// the target, the way an editor saves. It must read as a change to the row,
/// not as the row disappearing — which is exactly what a watch on the file
/// itself would have reported.
#[test]
fn e2e_an_atomic_save_does_not_read_as_a_delete() {
let dir = scratch_dir("live-atomic");
let (watcher, rx, path) = watch_one(&dir, "report.txt");
let tmp = dir.join("report.txt.tmp");
std::fs::write(&tmp, "hello, replaced").unwrap();
std::fs::rename(&tmp, &path).unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
assert!(
!updates.iter().any(|u| matches!(u, LiveUpdate::Gone { .. })),
"an atomic save was reported as a deletion: {updates:?}"
);
}
/// A row whose directory does not exist must not panic, and must not stop the
/// watcher from covering the rows whose directories do.
#[test]
fn a_missing_directory_does_not_stop_the_others() {
let dir = scratch_dir("live-missing-dir");
let good = dir.join("present.txt");
std::fs::write(&good, "hello world").unwrap();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
watcher.watch(
"hello",
vec![
Target {
path: dir
.join("nowhere")
.join("ghost.txt")
.to_string_lossy()
.into_owned(),
text: None,
size: 0,
mtime: 0,
},
current_target(&good.to_string_lossy(), None),
],
&Config::default(),
);
std::thread::sleep(Duration::from_millis(300));
std::fs::remove_file(&good).unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
assert!(
updates.iter().any(|u| matches!(u, LiveUpdate::Gone { .. })),
"the reachable row stopped working: {updates:?}"
);
}
// --- content, read from the file rather than from the index ---------------
//
// None of these open a database. That is the assertion they all share: a row
// on screen tracks the disk with no indexer involved, which is what the
// feature is for.
/// Pull the one `Changed` out of a batch, failing loudly on anything else.
fn one_change(updates: &[LiveUpdate], path: &str) -> (u64, i64, WindowUpdate) {
let found = updates.iter().find_map(|u| match u {
LiveUpdate::Changed {
path: p,
size,
mtime,
window,
} if p == path => Some((*size, *mtime, window.clone())),
_ => None,
});
found.unwrap_or_else(|| panic!("expected a Changed for {path}, got {updates:?}"))
}
/// The test the old index-backed design could not have: edit a watched file
/// with no index anywhere, and the row's size, modified time and Content Match
/// window all follow it.
#[test]
fn e2e_a_content_change_re_cuts_the_snippet_with_no_index() {
let dir = scratch_dir("live-content");
let (watcher, rx, path) =
watch_one_matching(&dir, "notes.txt", "hello world", Some(ContentTier::Exact));
std::fs::write(&path, "hello there, a much longer world").unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (size, mtime, window) = one_change(&updates, &path);
assert_eq!(size, "hello there, a much longer world".len() as u64);
assert!(mtime > 0, "no modified time was read");
let WindowUpdate::Cut(snippet) = window else {
panic!("the body still matches, so it should carry a window: {window:?}");
};
assert!(
snippet.window.contains("much longer"),
"the window is stale: {:?}",
snippet.window
);
assert!(!snippet.ranges.is_empty(), "the match was not marked");
}
/// Edited until it no longer matches, the row keeps its place and its metadata
/// but loses its window — the Content Match cell falls back to its dash rather
/// than showing text that is no longer a hit.
#[test]
fn e2e_an_edit_that_removes_the_match_clears_the_window() {
let dir = scratch_dir("live-unmatch");
let (watcher, rx, path) =
watch_one_matching(&dir, "notes.txt", "hello world", Some(ContentTier::Exact));
std::fs::write(&path, "nothing of interest here").unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (size, _, window) = one_change(&updates, &path);
assert_eq!(size, "nothing of interest here".len() as u64);
assert_eq!(
window,
WindowUpdate::NoMatch,
"a window survived the match it no longer has"
);
}
/// A fuzzy content hit is re-cut with the fuzzy matcher, not the literal one.
/// The literal is absent from the body by construction — that is what made
/// it a fuzzy hit — so re-cutting it as an exact row would read as "no longer
/// matches" and blank a cell that still has a hit in it.
#[test]
fn e2e_a_fuzzy_content_hit_is_re_cut_with_the_fuzzy_matcher() {
let dir = scratch_dir("live-fuzzy");
let path = dir.join("notes.txt").to_string_lossy().into_owned();
// "quartz" within one edit of "quarts": a stage-8 hit for that query.
std::fs::write(&path, "sphinx of black quarts judge my vow").unwrap();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
watcher.watch(
"quartz",
vec![current_target(&path, Some(ContentTier::Fuzzy))],
&Config::default(),
);
std::thread::sleep(Duration::from_millis(300));
std::fs::write(&path, "the sphinx of black quarts judged my vow again").unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (_, _, window) = one_change(&updates, &path);
let WindowUpdate::Cut(snippet) = window else {
panic!("a fuzzy hit was re-judged as exact: {window:?}");
};
assert!(
snippet.window.contains("quarts judged"),
"the window was not re-cut from the new body: {:?}",
snippet.window
);
assert!(!snippet.ranges.is_empty(), "the fuzzy match was not marked");
}
/// A file no extractor claims still reports what `metadata` knows. Size and
/// Modified are not the text columns' to withhold.
#[test]
fn e2e_a_file_with_no_extractable_text_still_reports_its_metadata() {
let dir = scratch_dir("live-binary");
let path = dir.join("blob.bin").to_string_lossy().into_owned();
std::fs::write(&path, [0x00u8, 0x01, 0x02, 0xFF]).unwrap();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
watcher.watch(
"hello",
vec![current_target(&path, Some(ContentTier::Exact))],
&Config::default(),
);
std::thread::sleep(Duration::from_millis(300));
std::fs::write(&path, [0x00u8, 0x01, 0x02, 0xFF, 0xFE, 0xFD]).unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (size, _, window) = one_change(&updates, &path);
assert_eq!(size, 6);
// `Unchanged`, not `NoMatch`: nothing readable came back, so there is no
// evidence the row's window is wrong — only that it is unverifiable.
assert_eq!(window, WindowUpdate::Unchanged);
}
/// Past `maximum_text_file_size` the indexer stores no text, so neither does
/// the row — but it still says how big the file got.
#[test]
fn e2e_a_file_over_the_text_size_limit_reports_size_but_no_window() {
let dir = scratch_dir("live-oversize");
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
let path = dir.join("huge.txt").to_string_lossy().into_owned();
std::fs::write(&path, "hello world").unwrap();
let mut config = Config::default();
config.processing.maximum_text_file_size = 16;
watcher.watch(
"hello",
vec![current_target(&path, Some(ContentTier::Exact))],
&config,
);
std::thread::sleep(Duration::from_millis(300));
std::fs::write(&path, "hello world, and rather more of it besides").unwrap();
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (size, _, window) = one_change(&updates, &path);
assert_eq!(
size,
"hello world, and rather more of it besides".len() as u64
);
// Not read, so not disproved: the window the search found stands.
assert_eq!(window, WindowUpdate::Unchanged);
}
// --- the arm-time sweep ---------------------------------------------------
/// A row armed with what the *index* said about a file that has since moved on
/// is corrected the moment it is watched. This is the check of the index
/// against the disk, and it is also the only thing that reports anything at
/// all on a filesystem the platform sends no events for.
#[test]
fn arming_corrects_a_row_that_went_stale_while_it_was_not_watched() {
let dir = scratch_dir("live-sweep-stale");
let path = dir.join("drifted.txt").to_string_lossy().into_owned();
std::fs::write(&path, "hello, a body the index never saw").unwrap();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
// What a stale index row would have claimed.
watcher.watch(
"hello",
vec![Target {
path: path.clone(),
text: Some(ContentTier::Exact),
size: 5,
mtime: 1,
}],
&Config::default(),
);
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
let (size, mtime, window) = one_change(&updates, &path);
assert_eq!(size, "hello, a body the index never saw".len() as u64);
assert!(mtime > 1, "the stale modified time survived");
assert!(
matches!(window, WindowUpdate::Cut(_)),
"the window was not re-cut: {window:?}"
);
}
/// Same sweep, for a row whose file is simply not there any more.
#[test]
fn arming_reports_a_row_whose_file_vanished_while_it_was_not_watched() {
let dir = scratch_dir("live-sweep-gone");
// A sibling keeps the directory watchable, so the ghost is dropped for
// being missing rather than for its directory being missing.
let sibling = dir.join("present.txt");
std::fs::write(&sibling, "hello world").unwrap();
let ghost = dir.join("vanished.txt").to_string_lossy().into_owned();
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
watcher.watch(
"hello",
vec![Target {
path: ghost.clone(),
text: None,
size: 11,
mtime: 1,
}],
&Config::default(),
);
let updates = collect(&rx, 1, Duration::from_secs(5));
stop_and_clean(watcher, &dir);
assert!(
updates
.iter()
.any(|u| matches!(u, LiveUpdate::Gone { path } if *path == ghost)),
"expected a Gone for {ghost}, got {updates:?}"
);
}
/// The other half of the sweep, and the one that keeps it quiet: a row that
/// already agrees with the disk is not touched. Without this the watcher would
/// repaint every visible row on every scroll.
#[test]
fn arming_says_nothing_about_a_row_that_already_agrees_with_the_disk() {
let dir = scratch_dir("live-sweep-quiet");
let (watcher, rx, _path) =
watch_one_matching(&dir, "steady.txt", "hello world", Some(ContentTier::Exact));
// `watch_one_matching` already waited out registration; anything the
// sweep decided has been sent by now.
let updates = collect(&rx, 1, Duration::from_millis(500));
stop_and_clean(watcher, &dir);
assert!(
updates.is_empty(),
"the sweep invented an update: {updates:?}"
);
}
/// Re-arming replaces the set wholesale; an event for a path that is no longer
/// shown decides nothing.
#[test]
fn re_arming_drops_the_previous_targets() {
let t = targets(&["/docs/new.txt"]);
let decided = window(
&t,
vec![event(
EventKind::Modify(ModifyKind::Any),
&["/docs/old.txt"],
)],
);
assert!(decided.is_empty(), "{decided:?}");
}

20
crates/quicksearch-core/src/platform.rs
View file

@ -490,26 +490,6 @@ pub fn release_free_heap() {
// spans to the kernel on free.
}
/// Live and free-but-retained heap bytes, as `(in_use, free)`: `free` is
/// memory already given back to the allocator that glibc still charges the
/// process for. `None` where the platform has no way to answer.
pub fn heap_stats() -> Option<(u64, u64)> {
#[cfg(all(target_os = "linux", target_env = "gnu"))]
{
// `mallinfo2`, not `mallinfo`: the older struct is `int`-typed and
// silently wraps past 2 GiB, which is exactly the size where the
// answer starts to matter.
//
// SAFETY: no arguments, returns a plain struct by value.
let info = unsafe { libc::mallinfo2() };
Some((info.uordblks as u64, info.fordblks as u64))
}
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
{
None
}
}
/// How long to keep retrying a delete that fails because something else holds
/// the file open.
#[cfg(windows)]

9
crates/quicksearch-core/src/scope.rs
View file

@ -89,15 +89,6 @@ impl Scope {
})
}
/// The configured root `path` lives under, if any. Containment is
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
pub fn owning_root(&self, path: &Path) -> Option<&Path> {
self.roots
.iter()
.map(|r| r.path.as_path())
.find(|root| path.starts_with(root) && path != *root)
}
/// Whether the walker would still emit `path` while walking `root`.
///
/// Mirrors `read_directory`'s three `continue`s. Full-path ignore

41
crates/quicksearch-core/src/scope_tests.rs
View file

@ -31,7 +31,6 @@ fn walked(config: &Config, db: &Path) -> HashSet<PathBuf> {
config.clone(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
2,
)
.filter_map(|e| match e {
@ -155,28 +154,6 @@ fn a_root_is_never_filtered_but_its_children_still_are() {
std::fs::remove_dir_all(&base).ok();
}
/// Root ownership compares whole components, so a sibling whose name
/// merely starts with a root's is not inside it — a prune that got this
/// wrong would delete a neighbouring folder's entire index.
#[test]
fn owning_root_does_not_match_name_prefixes() {
let base = tmp_tree("prefix");
let root = base.join("data");
let sibling = base.join("database");
std::fs::create_dir_all(&root).unwrap();
std::fs::create_dir_all(&sibling).unwrap();
let mut config = Config::default();
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
let scope = Scope::from_config(&config).unwrap();
assert_eq!(scope.owning_root(&root.join("f.txt")), Some(root.as_path()));
assert_eq!(scope.owning_root(&sibling.join("f.txt")), None);
// The root itself is a directory, never a row, and owns nothing.
assert_eq!(scope.owning_root(&root), None);
std::fs::remove_dir_all(&base).ok();
}
/// The counters the status display reads: a scan that reports nothing is
/// indistinguishable from a hang.
#[test]
@ -323,21 +300,3 @@ fn cancelling_stops_the_scan_without_finishing_it() {
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// A path under no configured root has no rules to apply — a followed
/// symlink's target is the real case; `owning_root` returning `None` is
/// what keeps it alive.
#[test]
fn a_path_outside_every_root_has_no_owner() {
let base = tmp_tree("outside");
let root = base.join("indexed");
std::fs::create_dir_all(&root).unwrap();
let mut config = Config::default();
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
config.indexing.ignore_patterns = vec!["*".into()];
let scope = Scope::from_config(&config).unwrap();
assert_eq!(scope.owning_root(Path::new("/elsewhere/target.txt")), None);
std::fs::remove_dir_all(&base).ok();
}

56
crates/quicksearch-core/src/search/cascade.rs
View file

@ -80,6 +80,62 @@ const CANCEL_CHECK_ROWS: usize = 256;
/// around the match, and the mouseover shows the rest as extended context.
const SNIPPET_WINDOW_CHARS: usize = 600;
/// The Content Match snippet for one document body, cut exactly as the
/// full-text passes cut it.
///
/// `folded` must be `text` ASCII-lowercased. That fold is byte-length
/// preserving, which is the whole reason offsets found in it can slice `text`;
/// the passes hold one reusable fold buffer per scan and hand it in here
/// rather than paying for a second copy.
///
/// Shared so that [`crate::live`], re-cutting a snippet for a file that
/// changed under a result already on screen, produces the same window the
/// search itself would — otherwise a row would visibly re-frame its own match
/// the moment the file was touched.
pub fn text_snippet(
pattern: &crate::query::pattern::TermPattern,
text: &str,
folded: &str,
) -> Option<snippet::Snippet> {
let opts = snippet::Options {
approx_chars: SNIPPET_WINDOW_CHARS,
};
match pattern.literal() {
// Literal terms keep the richer multi-occurrence extract; a wildcard
// match marks its own first range.
Some(term) => Some(snippet::extract_folded(text, folded, &[term], &opts)),
None => pattern.find_first_folded(folded).map(|r| {
// A greedy pattern can match megabytes; clamp before the window.
let r = clamp_match_range(text, r, SNIPPET_WINDOW_CHARS);
snippet::window_around(text, (r.start, r.end), &opts)
}),
}
}
/// The fuzzy full-text match in one document body: how many times the term
/// occurs within the edit budget, and the Content Match window cut around
/// the first occurrence at the cascade's own width. `None` when it does not
/// occur at all.
///
/// Shared with [`crate::live`] for the same reason as [`text_snippet`]: a
/// fuzzy row whose file changes has to be re-cut the way it was cut, and
/// bitap's range is what it was cut around. `bitap` is built once by the
/// caller — per scan in the pass, per arm in the live watcher — since
/// building it is the cost, and `folded` must be `text` ASCII-lowercased.
pub fn fuzzy_snippet(
bitap: &crate::search::fuzzy::Bitap,
text: &str,
folded: &str,
) -> Option<(usize, snippet::Snippet)> {
let opts = snippet::Options {
approx_chars: SNIPPET_WINDOW_CHARS,
};
// `first` is `Some` exactly when `count` is non-zero: it *is* the first
// of them.
let (count, first) = bitap.count_and_first(folded.as_bytes());
first.map(|range| (count, snippet::window_around(text, range, &opts)))
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Outcome {
pub total: usize,

165
crates/quicksearch-core/src/search/cascade/passes.rs
View file

@ -10,82 +10,6 @@ enum RowHit {
Defer(SearchHit),
}
/// Reusable decode buffer and decompression context for the passes that read
/// document text.
///
/// `zstd::decode_all` builds and tears down a `ZSTD_DCtx` *and* allocates a
/// fresh output `Vec` on every call, and it is called once per candidate row.
/// One context and one buffer, reused across a whole scan, make that a
/// per-scan cost instead of a per-row one.
struct DocDecoder {
dctx: zstd::bulk::Decompressor<'static>,
buf: Vec<u8>,
}
/// Where [`DocDecoder::decode`]'s buffer starts before it has seen a document.
/// Most extracted text is well under this, so the doubling below rarely runs.
const INITIAL_DOC_CAPACITY: usize = 64 * 1024;
/// Where the doubling stops. Stored text is capped at
/// `processing.maximum_text_size` (256 KiB by default), so this is far above
/// any legitimate document even if that setting is raised — past it, a failure
/// is a corrupt frame rather than a buffer that is too small.
const MAX_DOC_CAPACITY: usize = 64 * 1024 * 1024;
impl DocDecoder {
fn new() -> Result<Self, String> {
Ok(DocDecoder {
dctx: zstd::bulk::Decompressor::new().map_err(|e| e.to_string())?,
buf: Vec::new(),
})
}
/// Decompress `blob` and borrow the result as text.
///
/// Returns `None` for a corrupt frame or non-UTF-8 content. Nothing is
/// copied: the indexer stores UTF-8, so the bytes are borrowed in place
/// rather than run through `String::from_utf8_lossy(..).into_owned()`,
/// which duplicated the whole document even when it was already valid.
fn decode(&mut self, blob: &[u8]) -> Option<&str> {
self.buf.clear();
// `decompress_to_buffer` writes into spare capacity and fails rather
// than growing, so the room has to be there first.
//
// The frame header would say how much is needed, but the indexer
// writes with `zstd::encode_all`, which is *stream*-based and so
// records no content size — `get_frame_content_size` says `None` for
// every row this ever sees. Falling back to `zstd::decode_all` there
// looked harmless and was not: it builds a streaming decoder per call,
// which measured as one ~2.4 MiB allocation per document and 27 of the
// 30 GiB a fuzzy search moved through the allocator.
//
// So grow this buffer instead and keep reusing it. It settles at the
// largest document in the scan within the first few rows, after which
// decoding a row allocates nothing at all.
if let Ok(Some(size)) = zstd::zstd_safe::get_frame_content_size(blob) {
self.buf.reserve(usize::try_from(size).ok()?);
}
loop {
if self.buf.capacity() == 0 {
self.buf.reserve(INITIAL_DOC_CAPACITY);
}
match self.dctx.decompress_to_buffer(blob, &mut self.buf) {
Ok(_) => break,
// Too small, or corrupt — the bulk API cannot tell us which.
// Growing is only worth trying while the buffer is still
// smaller than any document could legitimately be.
Err(_) if self.buf.capacity() < MAX_DOC_CAPACITY => {
let bigger = self.buf.capacity().saturating_mul(2);
self.buf.clear();
self.buf.reserve(bigger);
}
Err(_) => return None,
}
}
std::str::from_utf8(&self.buf).ok()
}
}
/// Fold `text` into `dst` in place, reusing its allocation.
///
/// The ASCII fold is byte-length preserving, which is what lets the cascade
@ -242,16 +166,10 @@ impl<'a> Cx<'a> {
let is_path_tier = rank >= 9.0;
// The "snippet" of a name or path hit is that field itself
// with the matched span marked.
let snip = snippet::Snippet {
ranges: vec![match_range],
window: if is_path_tier {
path.to_string()
} else {
name.clone()
},
truncated_start: false,
truncated_end: false,
};
let snip = snippet::whole_field(
if is_path_tier { path } else { name.as_str() },
match_range,
);
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
file_id,
@ -327,12 +245,9 @@ impl<'a> Cx<'a> {
self.params_with_filters(Vec::new()),
),
};
let snippet_opts = snippet::Options {
approx_chars: SNIPPET_WINDOW_CHARS,
};
// One decoder and one fold buffer for the whole scan; both are reused
// per row rather than reallocated.
let mut doc = DocDecoder::new()?;
let mut doc = crate::db::repo::DocDecoder::new()?;
let mut lower = String::new();
// Decompression dominates: check cancellation every row.
self.scan_pass(&sql, params, 1, None, |cx, row, file_id, path| {
@ -369,20 +284,7 @@ impl<'a> Cx<'a> {
if !folded {
fold_into(&mut lower, text);
}
// Literal terms keep the richer multi-occurrence
// extract; a wildcard match marks its own first range.
let snip = match pattern.literal() {
Some(term) => Some(snippet::extract_folded(
text,
&lower,
&[term],
&snippet_opts,
)),
None => pattern.find_first_folded(&lower).map(|r| {
let r = clamp_match_range(text, r, SNIPPET_WINDOW_CHARS);
snippet::window_around(text, (r.start, r.end), &snippet_opts)
}),
};
let snip = super::text_snippet(pattern, text, &lower);
(stage as f64 + count_frac(count), stage as u8, snip)
}
// No stored text: can't case-verify or count. On the
@ -465,15 +367,14 @@ impl<'a> Cx<'a> {
if !cx.regex_accepts(file_id, path, None)? {
return Ok(RowHit::Skip);
}
// Mark the approximate matched span in the matched field;
// window_around clamps and aligns.
let snip = Some(snippet::window_around(
field,
range,
&snippet::Options {
approx_chars: field.len().saturating_mul(2).max(8),
},
));
// The matched field itself with the fuzzy span marked — the
// same shape pass A emits, and what `SearchHit::snippet`
// documents for the name and path tiers. Windowing it here
// used to hand back a *suffix* whenever the match sat past
// two thirds of the way through, which broke that contract
// and left a frontend unable to line the ranges up against
// the field it paints.
let snip = Some(snippet::whole_field(field, range));
let is_path_tier = rank >= 11.0;
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
@ -483,7 +384,14 @@ impl<'a> Cx<'a> {
size,
mtime,
rank,
stage: rank as u8,
// Stamped, not truncated from `rank`: this is the one pass
// whose ranks carry a fraction large enough to reach the
// next integer. `edit_budget` is only warned about above
// 3, so a distance of 10 makes rank 8.0 — and truncating
// that would file a *filename* hit under stage 8, the
// fuzzy full-text tier, telling every frontend to render
// it as a content match.
stage: if is_path_tier { 11 } else { 7 },
snippet: snip,
};
Ok(if is_path_tier {
@ -518,11 +426,8 @@ impl<'a> Cx<'a> {
HIT_COLUMNS, self.query.filter_sql
);
let params = self.params_with_filters(Vec::new());
let snippet_opts = snippet::Options {
approx_chars: SNIPPET_WINDOW_CHARS,
};
// One decoder and one fold buffer for the whole scan, reused per row.
let mut doc = DocDecoder::new()?;
let mut doc = crate::db::repo::DocDecoder::new()?;
let mut folded = String::new();
// Decompression dominates: check cancellation every row.
self.scan_pass(&sql, params, 1, None, |cx, row, file_id, path| {
@ -537,14 +442,12 @@ impl<'a> Cx<'a> {
// ASCII folding is byte-length preserving, so ranges found in
// the folded buffer are valid in the original.
fold_into(&mut folded, text);
let (count, first) = bitap.count_and_first(folded.as_bytes());
if count == 0 {
let Some((count, snip)) = super::fuzzy_snippet(&bitap, text, &folded) else {
return Ok(RowHit::Skip);
}
};
if !cx.regex_accepts(file_id, path, Some(text))? {
return Ok(RowHit::Skip);
}
let snip = first.map(|range| snippet::window_around(text, range, &snippet_opts));
let (size, mtime) = size_and_mtime(row)?;
Ok(RowHit::Emit(SearchHit {
file_id,
@ -554,7 +457,7 @@ impl<'a> Cx<'a> {
mtime,
rank: 8.0 + count_frac(count),
stage: 8,
snippet: snip,
snippet: Some(snip),
}))
})
}
@ -587,16 +490,10 @@ impl<'a> Cx<'a> {
None => return Ok(RowHit::Skip),
},
};
let snip = snippet::Snippet {
ranges: vec![match_range],
window: if is_path_tier {
path.to_string()
} else {
name.clone()
},
truncated_start: false,
truncated_end: false,
};
let snip = snippet::whole_field(
if is_path_tier { path } else { name.as_str() },
match_range,
);
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
file_id,
@ -631,7 +528,7 @@ impl<'a> Cx<'a> {
approx_chars: SNIPPET_WINDOW_CHARS,
};
// One decoder for the whole scan, reused per row.
let mut doc = DocDecoder::new()?;
let mut doc = crate::db::repo::DocDecoder::new()?;
// Decompression dominates: check cancellation every row.
self.scan_pass(&sql, params, 1, None, |_cx, row, file_id, path| {
let blob: Option<&[u8]> = row

314
crates/quicksearch-core/src/search/fuzzy.rs
View file

@ -21,6 +21,10 @@ const MAX_REGISTERS: usize = 22;
pub struct Bitap {
/// `masks[c]` has bit `i` set iff `pattern[i] == c`.
masks: [u64; 256],
/// The same table for the *reversed* pattern, which is what lets
/// [`Bitap::match_start`] find where a match began by scanning backwards
/// from where it ended.
rev_masks: [u64; 256],
/// Pattern length in bytes (1..=64).
len: usize,
/// Maximum edit distance.
@ -36,11 +40,14 @@ impl Bitap {
return None;
}
let mut masks = [0u64; 256];
let mut rev_masks = [0u64; 256];
for (i, &b) in pattern.iter().enumerate() {
masks[b as usize] |= 1u64 << i;
rev_masks[b as usize] |= 1u64 << (pattern.len() - 1 - i);
}
Some(Bitap {
masks,
rev_masks,
len: pattern.len(),
k,
})
@ -58,9 +65,14 @@ impl Bitap {
/// Advance all registers by one haystack byte. Returns the smallest
/// error count d for which the full pattern just matched, if any.
///
/// `masks` selects the direction: [`Bitap::masks`] to scan forwards,
/// [`Bitap::rev_masks`] to scan backwards. Everything else — `len`, `k`,
/// the `done` bit, `reset` — is the same either way, since a reversed
/// pattern is still a pattern of the same length.
#[inline]
fn step(&self, r: &mut [u64], byte: u8) -> Option<usize> {
let mask = self.masks[byte as usize];
fn step(&self, masks: &[u64; 256], r: &mut [u64], byte: u8) -> Option<usize> {
let mask = masks[byte as usize];
let done = 1u64 << (self.len - 1);
let mut hit = None;
let mut prev_old = r[0]; // R_old[d-1] for the d-th iteration
@ -83,52 +95,127 @@ impl Bitap {
hit
}
/// Minimum edit distance (≤ k) of any occurrence of the pattern in
/// `hay`, or `None` if nothing matches within k edits.
pub fn best_distance(&self, hay: &[u8]) -> Option<usize> {
self.best_distance_and_first(hay).map(|(d, _)| d)
}
/// [`best_distance`](Self::best_distance) plus the first match's
/// approximate byte range, from one sweep. The range carries the same
/// caveat as [`count_and_first`](Self::count_and_first): it assumes a
/// pattern-length match, so edits can shift the true start by up to `k`.
pub fn best_distance_and_first(&self, hay: &[u8]) -> Option<(usize, (usize, usize))> {
/// Where the match that ended at `end` with `errors` edits began.
///
/// The forward scan knows an occurrence's *end* exactly — that is the bit
/// it tests — but not its start, and with an insertion or a deletion the
/// match is not `len` bytes long, so `end - len` is simply the wrong
/// offset. Highlighting it put the marks a byte or two off the match and
/// over whatever preceded it: `repot` against `1Reporter` marked `1Repo`.
///
/// So the same automaton runs over the *reversed* pattern, backwards from
/// `end`. The first position it accepts **within `errors` edits** is the
/// start. That bound is what makes this correct rather than merely
/// plausible: the reversed pattern will also accept far shorter spans by
/// spending its whole budget on deletions — against `xabc` with a 2-edit
/// budget it accepts `c` alone on the very first byte — and taking that
/// would mark one letter of an exact three-letter match. The true
/// alignment costs the same read either way, so requiring `≤ errors`
/// rejects the cheap wrong answers and is still guaranteed to fire at or
/// before the real start.
///
/// Which occurrence gets marked is settled elsewhere — see
/// [`Bitap::refine_end`]. The rule both producers land on is *the earliest
/// alignment at the smallest edit distance*, so a mark is never longer
/// than the term and never shorter by more than the budget.
fn match_start(&self, hay: &[u8], end: usize, errors: usize) -> usize {
// A ≤k-edit alignment of a len-byte pattern is at most len+k long,
// so nothing before this can be the start.
let floor = end.saturating_sub(self.len + self.k);
let mut r = [0u64; MAX_REGISTERS];
self.reset(&mut r);
let mut best: Option<(usize, (usize, usize))> = None;
for (i, &b) in hay.iter().enumerate() {
if let Some(d) = self.step(&mut r, b) {
let end = i + 1;
let range = (end.saturating_sub(self.len), end);
if d == 0 {
return Some((0, range));
}
if best.is_none_or(|(cur, _)| d < cur) {
best = Some((d, range));
for (back, &b) in hay[floor..end].iter().rev().enumerate() {
if self
.step(&self.rev_masks, &mut r, b)
.is_some_and(|d| d <= errors)
{
return end - (back + 1);
}
}
// Unreachable: the forward scan proved an alignment ends here, and
// reversed it costs the same. Falling back to the floor keeps a
// hypothetical miss inside the haystack.
floor
}
/// Improve on the *earliest* accepting end by looking a little past it.
///
/// The automaton accepts as soon as a leading part of the pattern has
/// matched, paying for the rest with trailing deletions — so the first
/// end it reports is systematically short. Searching `abcdef` over
/// `zzabcdefzz` accepts after `abcd`, two deletions, with the whole word
/// sitting right there.
///
/// Each further byte can turn one of those deletions into a match, so a
/// better alignment ends at most `errors` bytes later and never more.
/// Stepping a *copy* of the registers that far finds it without
/// disturbing the caller's scan, or its count.
fn refine_end(
&self,
hay: &[u8],
r: &[u64; MAX_REGISTERS],
end: usize,
errors: usize,
) -> (usize, usize) {
let mut best = (errors, end);
let mut probe = *r;
for (ahead, &b) in hay[end..].iter().take(errors).enumerate() {
if let Some(d) = self.step(&self.masks, &mut probe, b) {
if d < best.0 {
best = (d, end + ahead + 1);
}
}
}
best
}
/// Count non-overlapping occurrences (at ≤ k edits) and report the
/// first match's approximate byte range in `hay`. After each hit the
/// automaton resets, so an exact match followed by trailing bytes
/// counts once, and overlapping suffix matches don't inflate counts.
/// The reported range assumes pattern-length matches — edits can shift
/// the true start by up to k bytes, which is fine for snippet windows.
/// The smallest edit distance (≤ k) at which the pattern occurs in `hay`,
/// and that occurrence's byte range — the span a frontend marks.
///
/// This one sweeps the whole haystack, so it finds the best alignment
/// without help; [`Bitap::count_and_first`] resets after every hit and
/// needs [`Bitap::refine_end`] instead.
pub fn best_distance_and_first(&self, hay: &[u8]) -> Option<(usize, (usize, usize))> {
let mut r = [0u64; MAX_REGISTERS];
self.reset(&mut r);
// (errors, end). The start is resolved once, at the end, rather than
// per improvement — `match_start` is a second scan, however short.
let mut best: Option<(usize, usize)> = None;
for (i, &b) in hay.iter().enumerate() {
if let Some(d) = self.step(&self.masks, &mut r, b) {
let end = i + 1;
if d == 0 {
best = Some((0, end));
break;
}
if best.is_none_or(|(cur, _)| d < cur) {
best = Some((d, end));
}
}
}
best.map(|(d, end)| (d, (self.match_start(hay, end, d), end)))
}
/// Count non-overlapping occurrences (at ≤ k edits) and report the first
/// one's byte range in `hay`. After each hit the automaton resets, so an
/// exact match followed by trailing bytes counts once, and overlapping
/// suffix matches don't inflate counts.
///
/// The range is the occurrence itself: [`Bitap::refine_end`] settles which
/// end, then [`Bitap::match_start`] finds where it began. Both run for the
/// first hit only, and both are bounded by the edit budget, so the cost is
/// O(len + k) per row rather than per byte.
pub fn count_and_first(&self, hay: &[u8]) -> (usize, Option<(usize, usize)>) {
let mut r = [0u64; MAX_REGISTERS];
self.reset(&mut r);
let mut count = 0usize;
let mut first: Option<(usize, usize)> = None;
for (i, &b) in hay.iter().enumerate() {
if self.step(&mut r, b).is_some() {
if let Some(d) = self.step(&self.masks, &mut r, b) {
count += 1;
if first.is_none() {
let end = i + 1;
first = Some((end.saturating_sub(self.len), end));
let (errors, end) = self.refine_end(hay, &r, i + 1, d);
first = Some((self.match_start(hay, end, errors), end));
}
self.reset(&mut r);
}
@ -158,7 +245,28 @@ mod tests {
fn best(pattern: &str, hay: &str, k: usize) -> Option<usize> {
Bitap::new(pattern.as_bytes(), k)
.unwrap()
.best_distance(hay.as_bytes())
.best_distance_and_first(hay.as_bytes())
.map(|(d, _)| d)
}
/// The slice of `hay` that a pattern's first occurrence marks — what a
/// frontend highlights.
fn marked<'h>(pattern: &str, hay: &'h str, k: usize) -> &'h str {
let (_, first) = Bitap::new(pattern.as_bytes(), k)
.unwrap()
.count_and_first(hay.as_bytes());
let (s, e) = first.expect("the pattern occurs");
&hay[s..e]
}
/// [`marked`] through the other range producer, which shares
/// `match_start` but reaches it by a different route.
fn marked_best<'h>(pattern: &str, hay: &'h str, k: usize) -> &'h str {
let (_, (s, e)) = Bitap::new(pattern.as_bytes(), k)
.unwrap()
.best_distance_and_first(hay.as_bytes())
.expect("the pattern occurs");
&hay[s..e]
}
#[test]
@ -223,19 +331,136 @@ mod tests {
}
#[test]
fn count_fuzzy_and_range_sane() {
fn count_fuzzy_and_range_is_the_occurrence_itself() {
let b = Bitap::new(b"hello", 1).unwrap();
let hay = b"say helo and hxllo again";
let (count, first) = b.count_and_first(hay);
assert_eq!(count, 2);
let (s, e) = first.unwrap();
assert!(s < e && e <= hay.len());
let window = &hay[s..e];
assert!(
std::str::from_utf8(window).unwrap().contains("hel"),
"first range should cover the first hit, got {:?}",
std::str::from_utf8(window)
);
// The occurrence, not a five-byte window ending where it ends: that
// reached back over the space and marked " helo".
assert_eq!(first, Some((4, 8)));
assert_eq!(&hay[4..8], b"helo");
}
/// The reported bug, exactly: `repot` marked `1Repo` in `1Reporter`.
///
/// The match is `repo` — one deletion, dropping the `t` — so it is four
/// bytes where the term is five, and a range assumed to be term-length
/// reached one byte too far left, over the `1`. Both producers, since
/// they share `match_start`.
#[test]
fn a_match_shorter_than_the_term_is_still_marked_exactly() {
assert_eq!(marked("repot", "1reporter", 1), "repo");
assert_eq!(marked_best("repot", "1reporter", 1), "repo");
// Substitution keeps the length, which is the case that always
// worked — worth holding, since it is the one the old arithmetic got
// right by accident.
assert_eq!(marked("hello", "xx hxllo xx", 1), "hxllo");
assert_eq!(marked_best("hello", "xx hxllo xx", 1), "hxllo");
}
/// Text with a byte inserted into the term marks up to the insertion, not
/// across it: `abxc` is a one-edit alignment of `abc`, but so is the `ab`
/// that ends two bytes earlier, and the rule is the *earliest* alignment
/// at the best distance. Nothing longer than the term can win — spanning
/// an inserted byte costs an edit, and deleting instead costs the same and
/// ends sooner.
#[test]
fn an_insertion_marks_up_to_it_rather_than_over_it() {
assert_eq!(marked("abc", "abxcd", 1), "ab");
assert_eq!(marked_best("abc", "zzabxczz", 1), "ab");
}
/// The trap in resolving the start backwards: the reversed pattern will
/// happily accept a much shorter span by spending its budget on
/// deletions, so taking its *first* acceptance marks one letter of an
/// exact match. `abc` occurs verbatim in `xabc`, and a 2-edit budget lets
/// the reverse pass accept `c` alone one byte in.
///
/// Only the whole-haystack producer can reach the trap — it is the one
/// that reports an exact match while the budget is still generous, so
/// `errors` is 0 where `k` is 2.
#[test]
fn a_generous_budget_does_not_shrink_an_exact_match() {
assert_eq!(marked_best("abc", "xabc", 2), "abc");
assert_eq!(marked_best("abcdef", "zzabcdefzz", 2), "abcdef");
assert_eq!(marked("hello", "say hello world", 2), "hello");
}
/// The automaton accepts as soon as a leading part of the term has
/// matched, spending the rest of the budget on trailing deletions — so
/// the earliest end is short, and marking it highlighted `abcd` for a
/// search for `abcdef` with the whole word right there. `refine_end`
/// looks the budget's worth of bytes past the first acceptance.
///
/// Checked against a brute-force Levenshtein oracle over every span.
#[test]
fn the_mark_is_not_truncated_to_a_leading_part_of_the_term() {
assert_eq!(marked("abcdef", "zzabcdefzz", 2), "abcdef");
assert_eq!(marked("abc", "xabc", 1), "abc");
assert_eq!(marked("reports", "the report went out", 2), "report");
// Not every term can be extended: `repot` against `1reporter` stops
// at `repo` because the next byte (`r`) costs an edit of its own, so
// one is the best it does either way.
assert_eq!(marked("repot", "1reporter", 1), "repo");
// And an alignment already at zero errors has nothing to improve.
assert_eq!(marked("hello", "say hello world", 0), "hello");
}
/// However the span is chosen it is a real alignment at the best distance,
/// so it is never longer than the term and never shorter by more than the
/// budget. That bound is what keeps a mark recognisable: at the production
/// ladder of one edit per three characters, a mark is always at least two
/// thirds of the term.
#[test]
fn the_marked_span_is_within_the_budget_of_the_terms_length() {
for (term, hay) in [
("repot", "1reporter"),
("abcdef", "zzabcdefzz"),
("quarterly", "the quartrly budget"),
("hello", "say helo and hxllo again"),
("reports", "the report went out"),
] {
let k = edit_budget(term.len(), 2).expect("a real budget");
let span = marked(term, hay, k).len();
assert!(
span <= term.len() && term.len() - span <= k,
"{term:?} in {hay:?} (k={k}) marked {span} bytes"
);
}
}
#[test]
fn a_zero_budget_marks_exactly_the_term() {
assert_eq!(marked("hello", "say hello world", 0), "hello");
assert_eq!(marked("ab", "ab ab", 0), "ab");
}
/// A match at the very start, and one whose end is inside the term's own
/// length, are where the offset arithmetic can underflow.
#[test]
fn a_match_at_the_start_of_the_haystack_stays_in_bounds() {
assert_eq!(marked("repot", "reporter", 1), "repo");
// The haystack is shorter than the term: "ab" matches "abc" with one
// deletion, ending at 2.
let (_, first) = Bitap::new(b"abc", 1).unwrap().count_and_first(b"ab");
assert_eq!(first, Some((0, 2)));
}
/// Bitap works on bytes over an ASCII-folded copy, so a range can land
/// inside a multi-byte character. `snippet::aligned_range` is what widens
/// it before anything slices; this only pins that the range stays inside
/// the haystack so that alignment has something valid to work from.
#[test]
fn a_range_over_multibyte_text_stays_within_the_haystack() {
let hay = "café notes — le rapport";
let (_, first) = Bitap::new(b"raport", 1)
.unwrap()
.count_and_first(hay.as_bytes());
let (s, e) = first.expect("one deletion from 'rapport'");
assert!(s < e && e <= hay.len(), "({s}, {e}) outside {}", hay.len());
}
#[test]
@ -335,7 +560,10 @@ mod tests {
let pattern: Vec<u8> = (0..plen).map(|_| alphabet[rng() % 4]).collect();
let hay: Vec<u8> = (0..hlen).map(|_| alphabet[rng() % 4]).collect();
for k in 0..=4 {
let got = Bitap::new(&pattern, k).unwrap().best_distance(&hay);
let got = Bitap::new(&pattern, k)
.unwrap()
.best_distance_and_first(&hay)
.map(|(d, _)| d);
let want = oracle(&pattern, &hay, k);
assert_eq!(
got,

49
crates/quicksearch-core/src/search/mod.rs
View file

@ -69,6 +69,55 @@ pub struct SearchHit {
pub snippet: Option<Snippet>,
}
/// Which field a hit's [`SearchHit::snippet`] excerpts, derived from the
/// cascade stage. See the rank table at the top of [`crate::search::cascade`].
///
/// Frontends branch on this rather than on the raw stage number, so a new tier
/// is classified in one place instead of in every renderer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MatchField {
Name,
Contents,
Path,
}
/// How a [`MatchField::Contents`] hit matched its body — what has to be
/// re-run to cut its snippet again from the file as it now stands.
///
/// The two are not interchangeable: an exact tier's snippet is cut around the
/// literal term, and a fuzzy tier's around a bitap match the literal is
/// usually *absent* from. Re-cutting a fuzzy hit as if it were exact finds
/// nothing and reads as "the file no longer matches".
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ContentTier {
/// Stages 5 and 6: the body contains the term as written.
Exact,
/// Stage 8: the body contains something within the fuzzy edit budget of
/// the term.
Fuzzy,
}
impl SearchHit {
pub fn match_field(&self) -> MatchField {
match self.stage {
1..=4 | 7 => MatchField::Name,
5 | 6 | 8 => MatchField::Contents,
// 9..=11, and whatever a later tier adds: the path is the safe
// reading, since it is the one field every hit carries in full.
_ => MatchField::Path,
}
}
/// `Some` for a hit whose snippet is a window on the file's body.
pub fn content_tier(&self) -> Option<ContentTier> {
match self.stage {
5 | 6 => Some(ContentTier::Exact),
8 => Some(ContentTier::Fuzzy),
_ => None,
}
}
}
#[derive(Debug, Clone)]
pub enum SearchUpdate {
Started {

66
crates/quicksearch-core/src/snippet.rs
View file

@ -52,13 +52,6 @@ impl Snippet {
}
}
/// Extract a snippet from `text` marking every occurrence of any term in
/// `terms` (ASCII-case-insensitive). With no terms or no matches, returns
/// the head of the text as the window with no ranges.
pub fn extract(text: &str, terms: &[&str], opts: &Options) -> Snippet {
extract_folded(text, &text.to_ascii_lowercase(), terms, opts)
}
/// [`extract`] against a haystack the caller has already ASCII-folded.
/// `folded` must be `text.to_ascii_lowercase()` — the fold is byte-length
/// preserving, which is what lets offsets found in it slice the original.
@ -139,6 +132,48 @@ pub fn extract_folded(text: &str, folded: &str, terms: &[&str], opts: &Options)
}
}
/// Clamp `range` into `text` and widen it to the nearest char boundaries.
///
/// Both callers below take ranges from matchers that work on bytes — bitap
/// over an ASCII-folded copy — so an endpoint can land inside a multi-byte
/// character. Slicing there panics, and `Snippet::ranges` promises boundaries.
fn aligned_range(text: &str, range: (usize, usize)) -> (usize, usize) {
let (mut start, mut end) = range;
start = start.min(text.len());
end = end.clamp(start, text.len());
while start > 0 && !text.is_char_boundary(start) {
start -= 1;
}
while end < text.len() && !text.is_char_boundary(end) {
end += 1;
}
(start, end)
}
/// The whole of `text` as the window, with `range` marked.
///
/// This is the shape [`crate::search::SearchHit::snippet`] documents for the
/// name and path tiers, and what lets a frontend highlight the matched span
/// inside its own Name or Path column: `window` is that field verbatim, so the
/// ranges index the field the column is already painting. A filename or a path
/// is short enough to carry whole, so there is nothing to gain by windowing it.
pub fn whole_field(text: &str, range: (usize, usize)) -> Snippet {
if text.is_empty() {
return Snippet::empty();
}
let (start, end) = aligned_range(text, range);
Snippet {
window: text.to_string(),
ranges: if end > start {
vec![(start, end)]
} else {
Vec::new()
},
truncated_start: false,
truncated_end: false,
}
}
/// Build a snippet window around one known match range in `text` (byte
/// offsets into `text`). Used by fuzzy full-text search, where the match
/// was located by the fuzzy matcher rather than exact term search. The
@ -147,15 +182,7 @@ pub fn window_around(text: &str, range: (usize, usize), opts: &Options) -> Snipp
if text.is_empty() {
return Snippet::empty();
}
let (mut ms, mut me) = range;
ms = ms.min(text.len());
me = me.clamp(ms, text.len());
while ms > 0 && !text.is_char_boundary(ms) {
ms -= 1;
}
while me < text.len() && !text.is_char_boundary(me) {
me += 1;
}
let (ms, me) = aligned_range(text, range);
let pre_pad = opts.approx_chars / 3;
let mut win_start = ms.saturating_sub(pre_pad);
@ -244,6 +271,13 @@ fn coalesce_overlapping(v: Vec<(usize, usize)>) -> Vec<(usize, usize)> {
mod tests {
use super::*;
/// The tests were written against a since-removed `extract` wrapper.
/// Production always holds a fold buffer already, so the wrapper earned
/// nothing; folding here keeps its coverage of the window logic.
fn extract(text: &str, terms: &[&str], opts: &Options) -> Snippet {
extract_folded(text, &text.to_ascii_lowercase(), terms, opts)
}
fn opts_small() -> Options {
Options { approx_chars: 40 }
}

141
crates/quicksearch-core/src/testutil.rs
View file

@ -21,12 +21,78 @@ pub fn zstd_of(text: &str) -> Option<Vec<u8>> {
crate::db::repo::encode_one(text, true).expect("zstd encode")
}
/// How old a leftover scratch directory must be before [`sweep_stale`] takes
/// it. Far longer than any test run, so a failure investigated the same day —
/// or the next morning — still has its tree.
const STALE_AFTER: std::time::Duration = std::time::Duration::from_secs(12 * 60 * 60);
/// Whether `name` is one of [`scratch_dir`]'s own directories.
///
/// Matched on the *shape* — `quicksearch-{tag}-{pid}-{seq}`, so the last two
/// dash-separated components must be numbers — rather than on the
/// `quicksearch-` prefix alone. `packaging/capture.sh` keeps its output in
/// `quicksearch-capture` in the same directory, and a prefix match would eat a
/// capture run's screenshots along with the litter.
fn is_scratch_name(name: &str) -> bool {
let Some(rest) = name.strip_prefix("quicksearch-") else {
return false;
};
let numeric = |part: Option<&str>| {
part.is_some_and(|p| !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit()))
};
let mut tail = rest.rsplitn(3, '-');
// seq, then pid, and a tag must remain in front of them.
numeric(tail.next()) && numeric(tail.next()) && tail.next().is_some_and(|tag| !tag.is_empty())
}
/// Remove scratch directories left by runs that are long over.
///
/// Nothing here cleans up on the way *out*: a failed test's tree is most of
/// the evidence, which is why [`scratch_dir`] deliberately leaves it. But
/// passing tests leave theirs too, and most never remove it — so the temp
/// directory grew by roughly three hundred directories per full run and had
/// accumulated some nine thousand of them. Where `/tmp` is a tmpfs that is
/// gigabytes of RAM, which slows the whole suite and pushes the
/// timing-sensitive tests toward their budgets.
///
/// Sweeping on the way *in* keeps both halves: this run's evidence survives,
/// and so does yesterday's, while nothing accumulates without bound. Only
/// [`scratch_dir`]'s own naming is touched.
fn sweep_stale() {
let Ok(entries) = std::fs::read_dir(std::env::temp_dir()) else {
return;
};
let now = std::time::SystemTime::now();
for entry in entries.flatten() {
let name = entry.file_name();
if !name.to_str().is_some_and(is_scratch_name) {
continue;
}
let stale = entry
.metadata()
.and_then(|m| m.modified())
.ok()
.and_then(|t| now.duration_since(t).ok())
.is_some_and(|age| age >= STALE_AFTER);
if stale {
// Best effort throughout: two test binaries starting together race
// on the same directory and one of them loses, which is fine.
std::fs::remove_dir_all(entry.path()).ok();
}
}
}
/// A fresh, empty directory under the system temp dir, named for `tag`.
///
/// Not cleaned up on drop: when a test fails, the tree it built is most of
/// the evidence. Panics — a test that cannot create a directory has nothing
/// left to assert.
/// the evidence. Long-dead runs' trees are swept once per process instead —
/// see [`sweep_stale`]. Panics — a test that cannot create a directory has
/// nothing left to assert.
#[doc(hidden)]
pub fn scratch_dir(tag: &str) -> PathBuf {
static SWEPT: std::sync::Once = std::sync::Once::new();
SWEPT.call_once(sweep_stale);
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-{}-{}-{}",
@ -95,4 +161,75 @@ mod tests {
touch(&deep, b"hi");
assert_eq!(std::fs::read(&deep).unwrap(), b"hi");
}
/// The sweep runs against a shared temp directory, so what it matches is
/// the whole safety argument. `quicksearch-capture` is the one that would
/// hurt: `packaging/capture.sh` puts a run's screenshots and screencasts
/// there, and a prefix match would delete them mid-capture.
#[test]
fn only_scratch_directories_are_swept() {
for ours in [
"quicksearch-coord-1234-0",
"quicksearch-stall-heavy-1001402-7",
"quicksearch-a-0-0",
// Tags contain dashes of their own; only the last two components
// are read as numbers.
"quicksearch-sniff-binary-db-2621744-1",
] {
assert!(is_scratch_name(ours), "{ours} should be swept");
}
for theirs in [
// The capture output directory, the reason this is a shape match.
"quicksearch-capture",
"quicksearch",
"quicksearch-",
// A tag but no pid/seq pair.
"quicksearch-coord",
"quicksearch-coord-1234",
// Numbers, but nothing in front of them to be a tag.
"quicksearch-1234-0",
// Not ours at all.
"cargo-install-abc-1-2",
"tmp-quicksearch-coord-1-2",
] {
assert!(!is_scratch_name(theirs), "{theirs} must not be swept");
}
}
/// Fresh directories survive; only long-dead runs are collected. Uses a
/// hand-built name rather than `scratch_dir` so the assertion is about the
/// age gate and not about whatever else the suite has left lying around.
#[test]
fn the_sweep_keeps_recent_trees_and_takes_old_ones() {
let fresh = scratch_dir("sweep-fresh");
touch(&fresh.join("evidence.txt"), b"kept");
// Same shape, but back-dated past the threshold. `set_times` is the
// only way to age a directory without waiting twelve hours for it.
let old = std::env::temp_dir().join(format!(
"quicksearch-sweep-old-{}-{}",
std::process::id(),
NEXT.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&old).expect("create the aged directory");
let long_ago =
std::time::SystemTime::now() - STALE_AFTER - std::time::Duration::from_secs(60);
std::fs::File::open(&old)
.and_then(|d| {
d.set_times(
std::fs::FileTimes::new()
.set_accessed(long_ago)
.set_modified(long_ago),
)
})
.expect("back-date the aged directory");
sweep_stale();
assert!(fresh.exists(), "a fresh scratch tree was swept away");
assert!(!old.exists(), "a long-dead scratch tree survived the sweep");
std::fs::remove_dir_all(&fresh).ok();
}
}

299
crates/quicksearch-core/src/verify.rs Normal file
View file

@ -0,0 +1,299 @@
//! Byte-for-byte verification that a set of files really is identical.
//!
//! The index groups duplicates by `sha256(size ‖ first hash_length bytes)`
//! (see [`crate::file_handling`]), which reads a file's head and nothing else
//! — a deliberate trade, since hashing every byte on a disk is most of the
//! cost of indexing it. Files of the same size whose heads agree are therefore
//! listed as duplicates whether or not they are: a fixed-size VHD keeps what
//! makes it unique in a footer, and a freshly pre-allocated disk image is
//! zeroes as far as the head can see. This turns that advisory grouping into
//! an answer, for the moment before someone deletes something.
//!
//! No hashing here, by policy. A digest per file would be shorter code and the
//! same answer nearly always — but "nearly always" is what the head hash
//! already offers, and the whole point of asking a second time is that this
//! time the bytes are compared.
use std::fs::File;
use std::io::Read;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
/// Total read-buffer memory, split across the files being compared. A group is
/// usually two files and can be thousands — a hardlink farm, which is exactly
/// what `[indexing] ignore_patterns` warns about — so a per-file buffer of any
/// fixed size would become the largest allocation the process ever makes.
const CHUNK_BUDGET: usize = 8 * 1024 * 1024;
const MIN_CHUNK: usize = 16 * 1024;
const MAX_CHUNK: usize = 256 * 1024;
/// How often progress is emitted. Each one repaints the UI, and a chunk off a
/// warm page cache takes microseconds.
const PROGRESS_INTERVAL: Duration = Duration::from_millis(100);
/// What one member turned out to be, against the reference.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MemberVerdict {
/// Same length, every byte agreed. The reference itself reads this.
Identical,
/// Offset of the first byte that disagreed.
DiffersAt(u64),
/// Lengths disagree, so nothing was read. Within a duplicate group this
/// can only mean a stale index — the hash covers the size.
LengthDiffers { len: u64, reference_len: u64 },
/// Could not be opened, or stopped being readable part way through.
Unreadable(String),
}
impl MemberVerdict {
pub fn is_identical(&self) -> bool {
matches!(self, MemberVerdict::Identical)
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct VerifyReport {
/// Index into the input paths of the file everything else was compared
/// against: the first one that opened. `None` when none of them did.
pub reference: Option<usize>,
/// One verdict per input path, in the input order.
pub verdicts: Vec<MemberVerdict>,
/// Bytes actually read from disk, across every file.
pub bytes_read: u64,
}
impl VerifyReport {
/// Whether every member was read and matched. An empty or single-file set
/// is vacuously identical.
pub fn all_identical(&self) -> bool {
self.verdicts.iter().all(MemberVerdict::is_identical)
}
pub fn differing(&self) -> usize {
self.verdicts.iter().filter(|v| !v.is_identical()).count()
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum VerifyUpdate {
Progress { bytes_read: u64, bytes_total: u64 },
Done(VerifyReport),
Cancelled,
}
/// A file still in the running, with its own read buffer.
struct Live {
index: usize,
file: File,
buf: Vec<u8>,
}
/// Compare every path against the first one that opens, byte for byte, and
/// report what each turned out to be.
///
/// Emits `Progress` while it works and exactly one terminal update — `Done`,
/// or `Cancelled` if `cancel` went up before the comparison finished.
pub fn verify_identical(paths: &[PathBuf], cancel: &AtomicBool, on: &mut dyn FnMut(VerifyUpdate)) {
// Checked before the files are even opened, so a run cancelled before it
// starts reports the cancellation rather than a verdict nobody waited for.
if cancel.load(Ordering::Relaxed) {
on(VerifyUpdate::Cancelled);
return;
}
let mut verdicts = vec![MemberVerdict::Identical; paths.len()];
// The reference is the first path that both opens *and* stats, not simply
// the first path: one unreadable member must not cost the answer about all
// the others.
let mut reference: Option<(usize, File, u64)> = None;
let mut rest: Vec<(usize, File)> = Vec::new();
for (i, path) in paths.iter().enumerate() {
let file = match File::open(path) {
Ok(f) => f,
Err(e) => {
verdicts[i] = MemberVerdict::Unreadable(describe(path, &e));
continue;
}
};
if reference.is_some() {
rest.push((i, file));
continue;
}
match file.metadata() {
Ok(m) => reference = Some((i, file, m.len())),
Err(e) => verdicts[i] = MemberVerdict::Unreadable(describe(path, &e)),
}
}
let Some((reference, mut reference_file, reference_len)) = reference else {
on(VerifyUpdate::Done(VerifyReport {
reference: None,
verdicts,
bytes_read: 0,
}));
return;
};
// A length mismatch is decided from the handles, before a byte is read.
let mut live: Vec<Live> = Vec::with_capacity(rest.len());
for (i, file) in rest {
match file.metadata() {
Ok(m) if m.len() != reference_len => {
verdicts[i] = MemberVerdict::LengthDiffers {
len: m.len(),
reference_len,
};
}
Ok(_) => live.push(Live {
index: i,
file,
buf: Vec::new(),
}),
Err(e) => verdicts[i] = MemberVerdict::Unreadable(describe(&paths[i], &e)),
}
}
let chunk = (CHUNK_BUDGET / (live.len() + 1)).clamp(MIN_CHUNK, MAX_CHUNK);
let mut reference_buf = vec![0u8; chunk];
for l in live.iter_mut() {
l.buf = vec![0u8; chunk];
}
let bytes_total = match live.len() {
0 => 0,
n => reference_len.saturating_mul(n as u64 + 1),
};
let mut bytes_read = 0u64;
let mut offset = 0u64;
// Backdated so the first chunk reports: a progress bar that only appears
// after the first interval reads as a frozen window on a slow disk, which
// is the case this is for.
let mut last_progress = Instant::now()
.checked_sub(PROGRESS_INTERVAL)
.unwrap_or_else(Instant::now);
while !live.is_empty() {
if cancel.load(Ordering::Relaxed) {
on(VerifyUpdate::Cancelled);
return;
}
// Termination is driven by what the reference actually reads rather
// than by the length it claimed, so a file truncated underneath us
// degrades to a short comparison instead of a hang or a false match.
let n = match read_chunk(&mut reference_file, &mut reference_buf) {
Ok(0) => break, // EOF: everything still live matched all the way.
Ok(n) => n,
Err(e) => {
verdicts[reference] = MemberVerdict::Unreadable(describe(&paths[reference], &e));
// Survivors agreed up to here but cannot be finished. Saying
// so is the only honest answer; "identical" would not be.
for l in live.iter() {
verdicts[l.index] = MemberVerdict::Unreadable(format!(
"compared only to byte {offset}: {} could not be read to the end",
paths[reference].display()
));
}
break;
}
};
bytes_read += n as u64;
let mut i = 0;
while i < live.len() {
let (got, verdict) = {
let l = &mut live[i];
match read_chunk(&mut l.file, &mut l.buf[..n]) {
Ok(m) => {
let common = n.min(m);
if let Some(k) =
first_difference(&reference_buf[..common], &l.buf[..common])
{
(m, Some(MemberVerdict::DiffersAt(offset + k as u64)))
} else if m < n {
// Same length a moment ago, shorter now.
(
m,
Some(MemberVerdict::Unreadable(format!(
"{}: ended at byte {} while the file it was compared \
against had more",
paths[l.index].display(),
offset + m as u64
))),
)
} else {
(m, None)
}
}
Err(e) => (
0,
Some(MemberVerdict::Unreadable(describe(&paths[l.index], &e))),
),
}
};
bytes_read += got as u64;
match verdict {
Some(v) => {
verdicts[live[i].index] = v;
live.swap_remove(i);
}
None => i += 1,
}
}
offset += n as u64;
if last_progress.elapsed() >= PROGRESS_INTERVAL {
last_progress = Instant::now();
on(VerifyUpdate::Progress {
bytes_read,
bytes_total,
});
}
}
on(VerifyUpdate::Done(VerifyReport {
reference: Some(reference),
verdicts,
bytes_read,
}));
}
/// Fill `buf` as far as the file allows, returning how much. Short reads are
/// resumed and `Interrupted` retried, the way `extract::plaintext` does, so a
/// short return really does mean end of file.
fn read_chunk(f: &mut File, buf: &mut [u8]) -> std::io::Result<usize> {
let mut filled = 0;
while filled < buf.len() {
match f.read(&mut buf[filled..]) {
Ok(0) => break,
Ok(n) => filled += n,
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(e),
}
}
Ok(filled)
}
/// Offset of the first byte that differs. The equality test comes first
/// because it is a `memcmp`; the byte walk only ever runs on the one chunk
/// that turned out to differ.
fn first_difference(a: &[u8], b: &[u8]) -> Option<usize> {
if a == b {
return None;
}
Some(
a.iter()
.zip(b.iter())
.position(|(x, y)| x != y)
.unwrap_or(a.len().min(b.len())),
)
}
fn describe(path: &Path, e: &std::io::Error) -> String {
format!("{}: {}", path.display(), e)
}
#[cfg(test)]
#[path = "verify_tests.rs"]
mod tests;

281
crates/quicksearch-core/src/verify_tests.rs Normal file
View file

@ -0,0 +1,281 @@
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
use super::*;
use crate::testutil::{scratch_dir, touch};
/// Run to completion with cancellation switched off, returning every update.
fn run(paths: &[PathBuf]) -> Vec<VerifyUpdate> {
let cancel = AtomicBool::new(false);
let mut seen = Vec::new();
verify_identical(paths, &cancel, &mut |u| seen.push(u));
seen
}
/// The report from a run, asserting it produced exactly one terminal update
/// and that the update was `Done`.
fn report(paths: &[PathBuf]) -> VerifyReport {
let seen = run(paths);
let terminal: Vec<&VerifyUpdate> = seen
.iter()
.filter(|u| !matches!(u, VerifyUpdate::Progress { .. }))
.collect();
assert_eq!(terminal.len(), 1, "expected one terminal update: {seen:?}");
match terminal[0] {
VerifyUpdate::Done(r) => r.clone(),
other => panic!("expected Done, got {other:?}"),
}
}
/// `n` files in a fresh directory, each with the body it is given.
fn files(tag: &str, bodies: &[&[u8]]) -> Vec<PathBuf> {
let dir = scratch_dir(tag);
bodies
.iter()
.enumerate()
.map(|(i, body)| {
let p = dir.join(format!("copy{i}.bin"));
touch(&p, body);
p
})
.collect()
}
#[test]
fn identical_files_all_report_identical() {
for count in [2, 3] {
let bodies = vec![&b"the same bytes in every copy"[..]; count];
let paths = files("verify-same", &bodies);
let r = report(&paths);
assert_eq!(r.reference, Some(0));
assert!(r.all_identical(), "{r:?}");
assert_eq!(r.differing(), 0);
// Every file was read through: the head hash alone would not do.
assert_eq!(r.bytes_read, 28 * count as u64);
}
}
#[test]
fn a_difference_is_reported_at_its_offset() {
// First byte, mid-file, and the very last byte: the last is the one a
// head-only hash can never see, and the reason this module exists.
for (label, a, b, at) in [
("first", &b"Xbcdefgh"[..], &b"abcdefgh"[..], 0),
("middle", &b"abcdefgh"[..], &b"abcXefgh"[..], 3),
("last", &b"abcdefgh"[..], &b"abcdefgX"[..], 7),
] {
let paths = files("verify-diff", &[a, b]);
let r = report(&paths);
assert_eq!(r.verdicts[0], MemberVerdict::Identical, "{label}");
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(at), "{label}");
assert!(!r.all_identical(), "{label}");
assert_eq!(r.differing(), 1, "{label}");
}
}
/// The realistic false positive: same size, same head, different tail — a
/// pre-allocated disk image, which is what `[processing] hash_length`
/// documents as the known limitation.
#[test]
fn a_shared_head_with_a_different_tail_is_caught() {
let head = vec![0u8; 64 * 1024];
let mut a = head.clone();
let mut b = head;
a.extend_from_slice(b"footer-a");
b.extend_from_slice(b"footer-b");
let paths = files("verify-tail", &[a.as_slice(), b.as_slice()]);
let r = report(&paths);
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(64 * 1024 + 7));
}
/// Bigger than one chunk, so the multi-chunk path and the running offset are
/// both exercised rather than assumed.
#[test]
fn a_difference_past_the_first_chunk_is_found() {
// Two files, so the chunk is the 256 KiB ceiling and the difference sits
// in the second one.
let size = MAX_CHUNK * 2 + 1234;
let a = vec![7u8; size];
let mut b = a.clone();
let at = MAX_CHUNK + 500;
b[at] = 8;
let paths = files("verify-chunks", &[a.as_slice(), b.as_slice()]);
let r = report(&paths);
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(at as u64));
// Two chunks out of each file and then it stops: with nothing left to
// compare against, reading the remainder of the reference would be work
// that cannot change the answer.
assert_eq!(
r.bytes_read,
4 * MAX_CHUNK as u64,
"did not stop once the last live file dropped out"
);
}
#[test]
fn different_lengths_are_decided_without_reading() {
let paths = files("verify-len", &[b"abcdefgh", b"abcdefghij"]);
let r = report(&paths);
assert_eq!(
r.verdicts[1],
MemberVerdict::LengthDiffers {
len: 10,
reference_len: 8
}
);
assert_eq!(r.bytes_read, 0, "a length mismatch reads nothing");
}
#[test]
fn empty_files_are_identical() {
let paths = files("verify-empty", &[b"", b""]);
let r = report(&paths);
assert!(r.all_identical(), "{r:?}");
assert_eq!(r.bytes_read, 0);
}
#[test]
fn a_missing_member_is_unreadable_and_the_rest_still_compare() {
let mut paths = files("verify-missing", &[b"same", b"same"]);
paths.insert(1, PathBuf::from("/nonexistent/quicksearch-verify-missing"));
let r = report(&paths);
assert_eq!(r.reference, Some(0));
assert!(matches!(r.verdicts[1], MemberVerdict::Unreadable(_)));
assert_eq!(
r.verdicts[2],
MemberVerdict::Identical,
"an unreadable member stopped the others being compared"
);
assert_eq!(r.differing(), 1);
}
/// The first path is the obvious reference, but not a required one: an
/// unreadable first member must not sink the whole run.
#[test]
fn the_reference_falls_through_to_the_first_readable_member() {
let mut paths = files("verify-refmissing", &[b"same", b"same"]);
paths.insert(0, PathBuf::from("/nonexistent/quicksearch-verify-ref"));
let r = report(&paths);
assert_eq!(r.reference, Some(1));
assert!(matches!(r.verdicts[0], MemberVerdict::Unreadable(_)));
assert_eq!(r.verdicts[1], MemberVerdict::Identical);
assert_eq!(r.verdicts[2], MemberVerdict::Identical);
}
#[test]
fn nothing_readable_reports_no_reference() {
let paths = vec![
PathBuf::from("/nonexistent/quicksearch-verify-a"),
PathBuf::from("/nonexistent/quicksearch-verify-b"),
];
let r = report(&paths);
assert_eq!(r.reference, None);
assert_eq!(r.verdicts.len(), 2);
assert!(r.verdicts.iter().all(|v| !v.is_identical()));
}
#[test]
fn a_single_file_and_an_empty_set_are_vacuously_identical() {
let paths = files("verify-one", &[b"alone"]);
let r = report(&paths);
assert_eq!(r.reference, Some(0));
assert!(r.all_identical());
assert_eq!(r.bytes_read, 0, "nothing to compare it against");
let r = report(&[]);
assert_eq!(r.reference, None);
assert!(r.verdicts.is_empty());
assert!(r.all_identical());
}
#[test]
fn a_run_cancelled_before_it_starts_reports_only_that() {
let paths = files("verify-cancel", &[b"same", b"same"]);
let cancel = AtomicBool::new(true);
let mut seen = Vec::new();
verify_identical(&paths, &cancel, &mut |u| seen.push(u));
assert_eq!(seen, vec![VerifyUpdate::Cancelled]);
}
/// Cancelling part way through ends the run there, with no `Done` claiming a
/// verdict it never reached. The first chunk always reports progress, so the
/// flag goes up between two chunks rather than at a time the test has to race
/// for.
#[test]
fn cancelling_mid_run_ends_it_without_a_verdict() {
let body = vec![3u8; MAX_CHUNK * 4];
let paths = files("verify-cancel-mid", &[body.as_slice(), body.as_slice()]);
let cancel = AtomicBool::new(false);
let mut seen = Vec::new();
verify_identical(&paths, &cancel, &mut |u| {
cancel.store(true, Ordering::Relaxed);
seen.push(u);
});
assert!(
matches!(seen.first(), Some(VerifyUpdate::Progress { .. })),
"the first chunk did not report progress: {seen:?}"
);
assert_eq!(seen.last(), Some(&VerifyUpdate::Cancelled), "{seen:?}");
assert!(
!seen.iter().any(|u| matches!(u, VerifyUpdate::Done(_))),
"a cancelled run still reported a verdict: {seen:?}"
);
}
/// Whatever progress reports, it is a fraction that makes sense: monotonic,
/// and never past its own denominator.
#[test]
fn progress_climbs_and_never_overruns_its_denominator() {
let body = vec![9u8; MAX_CHUNK * 6];
let paths = files(
"verify-progress",
&[body.as_slice(), body.as_slice(), body.as_slice()],
);
let mut last = 0;
let mut count = 0;
for update in run(&paths) {
if let VerifyUpdate::Progress {
bytes_read,
bytes_total,
} = update
{
assert!(
bytes_read <= bytes_total,
"{bytes_read} read of {bytes_total}"
);
assert!(bytes_read >= last, "progress went backwards");
last = bytes_read;
count += 1;
}
}
assert!(count > 0, "a six-chunk comparison reported no progress");
}
/// A directory is not a file this can compare, however the platform refuses
/// it — `File::open` fails outright on Windows, while on Linux it opens and
/// then refuses to be read. Either way it is that member's problem, not the
/// run's.
#[test]
fn an_unreadable_member_does_not_stop_the_run() {
let dir = scratch_dir("verify-dir");
let a = dir.join("a.bin");
let b = dir.join("b.bin");
touch(&a, b"identical bytes");
touch(&b, b"identical bytes");
let sub = dir.join("subdir");
std::fs::create_dir_all(&sub).unwrap();
let r = report(&[a, sub, b]);
assert_eq!(r.reference, Some(0));
assert!(
!r.verdicts[1].is_identical(),
"a directory was called an identical file"
);
assert_eq!(r.verdicts[2], MemberVerdict::Identical);
}

8
crates/quicksearch-core/src/walk.rs
View file

@ -29,7 +29,6 @@ use crate::file_handling::{
classify_by_mtime, classify_for_indexing, path_to_db_string, prepare_file_record,
warn_if_unrepresentable, DirRows, FileIndexAction, OwnedNewFile, UnreadableDirs,
};
use crate::indexing::should_abort;
mod pool;
#[cfg(test)]
@ -170,7 +169,6 @@ struct Ctx {
registry: Arc<Registry>,
unreadable: UnreadableDirs,
stop_flag: Arc<AtomicBool>,
suspend_flag: Arc<AtomicBool>,
}
/// Individual unreadable-directory warnings allowed per run before only the
@ -422,7 +420,7 @@ fn prepare(file: PendingFile, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
while let Some((job, slot)) = shared.take() {
let _busy = shared.stats.enter();
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
if ctx.stop_flag.load(Ordering::Relaxed) {
shared.shutdown();
return;
}
@ -459,7 +457,7 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
}
for file in files {
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
if ctx.stop_flag.load(Ordering::Relaxed) {
shared.shutdown();
return;
}
@ -703,7 +701,6 @@ pub fn walk_indexable_files(
config: Config,
registry: Arc<Registry>,
stop_flag: Arc<AtomicBool>,
suspend_flag: Arc<AtomicBool>,
workers: usize,
) -> ParallelWalk {
let mut queue = Queue::default();
@ -746,7 +743,6 @@ pub fn walk_indexable_files(
registry,
unreadable: UnreadableDirs::default(),
stop_flag,
suspend_flag,
});
for root in unresolvable {

11
crates/quicksearch-core/src/walk/tests.rs
View file

@ -54,7 +54,6 @@ fn walk_with(
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
))
}
@ -304,7 +303,6 @@ fn unreadable_directory_is_recorded_not_silently_empty() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = w
@ -462,7 +460,6 @@ fn hidden_and_ignored_entries_are_pruned() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
));
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
@ -508,7 +505,6 @@ fn pruned_entries_are_counted_by_reason() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = (&mut walk)
@ -557,7 +553,6 @@ fn a_tree_with_nothing_pruned_reports_no_summary() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = (&mut walk)
@ -603,7 +598,6 @@ fn a_directory_reports_rows_with_no_file_behind_them() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
));
stale.sort();
@ -643,7 +637,6 @@ fn an_unreadable_directory_reports_nothing_stale() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
));
fs::set_permissions(&locked, fs::Permissions::from_mode(0o755)).ok();
@ -684,7 +677,6 @@ fn stop_flag_ends_the_walk_without_hanging() {
Config::default(),
Arc::new(Registry::default_set()),
stop,
Arc::new(AtomicBool::new(false)),
4,
));
@ -713,7 +705,6 @@ fn dropping_the_walk_early_does_not_hang() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
assert!(w.next().is_some());
@ -739,7 +730,6 @@ fn overlapping_roots_yield_each_file_once() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
));
@ -787,7 +777,6 @@ fn finish_reports_a_clean_walk_and_is_idempotent() {
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = w

172
crates/quicksearch-core/tests/cascade.rs
View file

@ -9,7 +9,9 @@ use quicksearch_core::db::open_or_recreate;
use quicksearch_core::db::repo::{insert_file, set_content_done, NewFile};
use quicksearch_core::mime::FileType;
use quicksearch_core::query::split::split_for_cascade;
use quicksearch_core::search::{cascade, SearchHit, SearchOptions, SearchService, SearchUpdate};
use quicksearch_core::search::{
cascade, MatchField, SearchHit, SearchOptions, SearchService, SearchUpdate,
};
use quicksearch_core::testutil::zstd_of;
mod common;
@ -355,6 +357,39 @@ fn fuzzy_max_edits_widens_and_narrows_the_budget() {
std::fs::remove_file(&p).ok();
}
/// Regression: the fuzzy filename tier stamps stage 7 rather than truncating
/// its own rank. `7.0 + 0.1 * distance` reaches 8.0 at ten edits — the fuzzy
/// *full-text* tier — and every frontend reads `match_field()`, so a filename
/// hit would have been rendered as a match on the file's contents.
#[test]
fn a_distant_fuzzy_filename_hit_stays_a_name_hit() {
let p = tmp_db("fuzzystage");
let mut s = Seeder::new(&p, true);
// Ten substitutions against a 30-character term, whose budget is ten.
let far = s.add("abcdefghijklmnopqrst##########", "/d", 1, None);
let conn = s.done();
let (hits, _) = run_collect(
&conn,
"abcdefghijklmnopqrstuvwxyz0123",
&fuzzy_options_with_edits(10),
);
assert_eq!(
hits.iter().map(|h| h.file_id).collect::<Vec<_>>(),
vec![far]
);
assert!(
(hits[0].rank - 8.0).abs() < 1e-9,
"rank {} is not the 8.0 that used to truncate into the next stage",
hits[0].rank
);
assert_eq!(hits[0].stage, 7, "the name tier is stage 7 at any distance");
assert_eq!(hits[0].match_field(), MatchField::Name);
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test]
fn dedup_keeps_best_rank() {
let p = tmp_db("dedup");
@ -1400,3 +1435,138 @@ fn streaming_does_not_change_the_result_set() {
assert_eq!(outcome.total, 3, "batch size {}", batch);
}
}
/// `SearchHit::snippet` is documented as "the filename for name stages, the
/// full path for path stages", and a frontend relies on it to highlight the
/// match inside the Name or Path column it is already painting: the ranges
/// index that field, so they only line up if the window *is* that field.
///
/// The fuzzy tiers used to window it instead, which silently broke the
/// contract whenever the match landed past two thirds of the way through a
/// long name — the ranges then indexed a suffix, and a column that trusted
/// them would mark the wrong glyphs.
#[test]
fn fuzzy_name_and_path_snippets_carry_the_whole_field() {
let p = tmp_db("fuzzy-whole-field");
let mut s = Seeder::new(&p, true);
// The match sits in the last third of the name, which is what used to
// push the window's start off zero.
s.add(
"a_long_and_deliberately_padded_out_quarterly_repot.txt",
"/home/me/documents/archive",
1,
None,
);
let conn = s.done();
let (hits, _) = run_collect(&conn, "report", &fuzzy_options());
let hit = hits
.iter()
.find(|h| h.stage == 7)
.expect("a fuzzy filename hit");
let snip = hit.snippet.as_ref().expect("a name hit carries a snippet");
assert_eq!(snip.window, hit.name, "the window is not the whole name");
assert!(!snip.truncated_start, "the name was windowed");
assert!(!snip.truncated_end, "the name was windowed");
for &(a, b) in &snip.ranges {
assert!(b <= hit.name.len(), "range {a}..{b} runs past the name");
assert!(
hit.name.is_char_boundary(a) && hit.name.is_char_boundary(b),
"range {a}..{b} is not on char boundaries"
);
}
}
/// A fuzzy tier's marks have to cover the *matched* text and nothing else.
///
/// The bug this pins: bitap reports where a match ends, and the range took
/// its start to be `end - term.len()`, which is only right when the match
/// happens to be as long as the term. Searching `repot` marked `1Repo` inside
/// `1Reporter` — one byte too far left, over a character that matched nothing.
///
/// Asserted on the *text* rather than on offsets, so it reads as the symptom
/// and survives the fixture being reworded.
#[test]
fn a_fuzzy_mark_covers_the_matched_text_and_nothing_else() {
let p = tmp_db("fuzzy-mark-span");
let mut s = Seeder::new(&p, true);
// The leading digit is the point: it is what the old range reached back
// over. In the body too, for the full-text tier.
s.add("1Reporter.txt", "/home/me/docs", 1, None);
s.add(
"body.txt",
"/home/me/docs",
2,
Some("filed under 1Reporter last week"),
);
let conn = s.done();
let (hits, _) = run_collect(&conn, "repot", &fuzzy_options());
// Stage 7 — fuzzy filename, marked inside the whole name.
let name_hit = hits
.iter()
.find(|h| h.stage == 7)
.expect("a fuzzy filename hit");
let snip = name_hit.snippet.as_ref().expect("name tiers carry one");
let (a, b) = snip.ranges[0];
assert_eq!(
&snip.window[a..b],
"Repo",
"the mark is {:?}; it must cover the match and not the leading digit",
&snip.window[a..b]
);
// Stage 8 — fuzzy full text, marked inside the snippet window.
let body_hit = hits
.iter()
.find(|h| h.stage == 8)
.expect("a fuzzy full-text hit");
let snip = body_hit.snippet.as_ref().expect("content tiers carry one");
let (a, b) = snip.ranges[0];
assert_eq!(
&snip.window[a..b],
"Repo",
"the mark is {:?}; it must cover the match and not the leading digit",
&snip.window[a..b]
);
}
/// The other half of the same defect: the automaton accepts the moment a
/// leading part of the term has matched, paying for the term's tail with
/// deletions — so the mark stopped short of the text that actually matched.
///
/// `quarterly` against a body holding `quartrly` accepts after `quartrl`,
/// spending both edits on the missing `y` and the dropped `e`. One byte
/// further is a *better* alignment (one edit) covering the whole word, which
/// is what a reader expects to see lit up.
#[test]
fn a_fuzzy_mark_is_not_truncated_to_a_leading_part_of_the_term() {
let p = tmp_db("fuzzy-mark-full");
let mut s = Seeder::new(&p, true);
// The name must not match at all: a row the filename tier claims never
// reaches the full-text tier. And the body must hold a fuzzy *variant* —
// the term verbatim would be an exact content match, stage 5 or 6.
s.add(
"notes.txt",
"/home/me/docs",
1,
Some("the quartrly budget was revised"),
);
let conn = s.done();
let (hits, _) = run_collect(&conn, "quarterly", &fuzzy_options());
let hit = hits
.iter()
.find(|h| h.stage == 8)
.expect("a fuzzy full-text hit");
let snip = hit.snippet.as_ref().expect("content tiers carry one");
let (a, b) = snip.ranges[0];
assert_eq!(
&snip.window[a..b],
"quartrly",
"the mark is {:?}, a leading part of what matched",
&snip.window[a..b]
);
}

384
crates/quicksearch-core/tests/full_index.rs
View file

@ -11,12 +11,26 @@ use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant, SystemTime};
use quicksearch_core::config::Config;
use quicksearch_core::file_handling::{extract_scope_prepare, ExtractCursor};
use quicksearch_core::file_handling::{
count_extract_scope, mark_oversize_pending_na, ExtractCursor, ExtractScope,
};
use quicksearch_core::indexing::{IndexingService, IndexingStatus, RootPhase};
mod common;
use common::{scratch_dir as tmp_dir, touch};
/// The removed `extract_scope_prepare`: the oversize sweep the writer still
/// does, then the count the content pass now does on its own connection.
fn extract_scope_prepare(
conn_mutex: &Arc<Mutex<rusqlite::Connection>>,
cursor: &ExtractCursor,
config: &Config,
) -> Result<ExtractScope, String> {
let conn = conn_mutex.lock().unwrap();
mark_oversize_pending_na(&conn, cursor, config).unwrap();
count_extract_scope(&conn, cursor, config)
}
/// Run one full index over `root` and wait for it to finish.
fn index_once(root: &Path, db: &Path, config: &Config) {
common::IndexOnce {
@ -1230,6 +1244,80 @@ fn contentless_mode_still_indexes_inlined_files_without_storing_bodies() {
std::fs::remove_dir_all(&db_dir).ok();
}
/// What one watch of a heavy/light overlap saw; see [`observe_overlap`].
struct Overlap {
/// Light files drained and heavy rows stored, across the window in which
/// the heavy root extracted while the light root walked.
light_drained: usize,
heavy_stored: usize,
/// The heavy root's `extract_total` and pool size, for the fixture guards.
heavy_pending: usize,
heavy_pool: usize,
}
/// Watch a two-root run until the heavy root has finished extracting and
/// report how the two counters moved while both were in flight, then stop
/// the run. Removing the fixture is the caller's.
///
/// Deltas across the overlap, never durations. Sparse samples cost only the
/// window's edges, and they trim both counters together. Panics if the window
/// never opened — a fixture that does not exercise the case proves nothing.
fn observe_overlap(service: &IndexingService, heavy_tag: &str, light_tag: &str) -> Overlap {
let mut opened: Option<(usize, usize)> = None; // (light.walked, heavy.extracted)
let mut last = (0usize, 0usize);
let mut heavy_pending = 0usize;
let mut heavy_pool = 0usize;
let deadline = Instant::now() + Duration::from_secs(120);
while Instant::now() < deadline {
let mut in_window = false;
match service.get_status() {
IndexingStatus::Running { roots, .. } => {
let heavy_p = roots.iter().find(|r| r.root.contains(heavy_tag));
let light_p = roots.iter().find(|r| r.root.contains(light_tag));
if let (Some(h), Some(l)) = (heavy_p, light_p) {
// The light root's *walk* is what used to be starved, so the
// window closes with it — past that there is no drain left
// to observe, and `walked` is the only counter in play.
in_window = h.phase == RootPhase::Extracting && l.phase == RootPhase::Walking;
if in_window {
last = (l.walked, h.extracted);
opened.get_or_insert(last);
if let Some(total) = h.extract_total {
heavy_pending = total;
}
heavy_pool = h.total_workers;
}
}
}
// The run is claimed but has not reached its walk yet; there is
// nothing to sample, and breaking here would end the watch before
// the run it is watching had started.
IndexingStatus::Preparing { .. } => {}
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
_ => break,
}
// Both phases are monotone, so a closed window will not reopen.
if opened.is_some() && !in_window {
break;
}
std::thread::sleep(Duration::from_millis(2));
}
service.stop_indexing().unwrap();
let Some((light_open, heavy_open)) = opened else {
panic!(
"never observed the heavy root extracting while the light root walked; \
the fixture is not exercising the case"
);
};
Overlap {
light_drained: last.0 - light_open,
heavy_stored: last.1 - heavy_open,
heavy_pending,
heavy_pool,
}
}
/// A slow root must not stall the others.
///
/// This is the complaint stated directly: one root doing heavy extraction used
@ -1241,120 +1329,252 @@ fn contentless_mode_still_indexes_inlined_files_without_storing_bodies() {
/// wrong measure: writing is serial by construction (one SQLite connection),
/// so on a local disk the writer, not extraction, is the bottleneck and a
/// wall-clock comparison would mostly measure the machine.
///
/// A stall is therefore counted in *work*, not in milliseconds: while the heavy
/// root extracts, how many files the light root's walk was drained of, against
/// how many rows the heavy root's extraction landed. Both counters are advanced
/// by the same writer loop, each root's turn bounded by one slice
/// (`service_walking`, `service_extracting` in `indexing/pipeline.rs`), so their
/// ratio *is* the interleaving.
///
/// - Serialised — the regression — the writer reads the heavy batch itself and
/// drains nobody meanwhile. Whatever shape that takes it obeys
/// `light < heavy + quantum`: one quantum of each per round is the most a
/// single thread taking turns can manage. Its own time budget says the same
/// from the other side, since time spent reading is time not spent inserting.
/// - As built, extraction is off on the root's own pool and the writer's turn
/// for the heavy root is a store and nothing more, so the light root is
/// drained at the writer's full rate throughout — on this fixture several
/// times the bound.
///
/// Counting rather than timing is what makes the verdict the same on a loaded
/// CI runner and an idle workstation. Every way a host can be slow — a
/// preempted writer, a checkpoint, a long round — freezes *both* counters, and
/// cancels. The wall-clock figure this replaced did not cancel: the same
/// correct behaviour measured ~20 ms here and 188 ms on the CI runner, which is
/// *more* than the 130 ms the broken design measured here. At that point CI was
/// overriding the budget six-fold and the check had stopped telling the two
/// designs apart. A bound that has to be calibrated per host is not an
/// assertion.
#[test]
fn a_heavy_root_does_not_stall_a_light_one() {
// HEAVY: few files, each big enough that reading it is real work, with a
// The writer's round-robin quantum. Set here rather than inherited from the
// default 500 because the bound below is arithmetic in it, and because a
// 500-file round is a coarse enough publish interval to look like a stall
// on a slow host all by itself.
const QUANTUM: usize = 16;
// HEAVY: few files, each big enough that reading one is real work, with a
// small `maximum_text_size` so the cost lands in extraction rather than in
// the writer's tokenising.
// the writer's tokenising. Few and large rather than many and small: the
// margin is set by the cost of *one* heavy file against one light one,
// while the fixture's size on disk is their product — so for a given number
// of bytes, bigger files discriminate better. If a host ever comes in
// short, raise this size; raising the count only lengthens the window.
const HEAVY_FILES: usize = 32;
// LIGHT: a wide tree of tiny files, so its walk outlasts the heavy root's
// extraction and its counter moves finely. Each is inlined by its walk
// worker, so this root has no extraction phase of its own to confuse the
// window with. Sized for a walk the writer never holds up: at 6000 it was
// over before half the heavy rows had landed.
const LIGHT_FILES: usize = 16_000;
// Light files drained per (heavy row + quantum). Three times a bound the
// serialised design provably cannot reach, and roughly a ninth of what the
// built one reaches here.
const MIN_INTERLEAVE: usize = 3;
let heavy = tmp_dir("stall-heavy");
let body: Vec<u8> = "sphinx of black quartz judge my vow "
.repeat(40_000)
.repeat(80_000)
.into_bytes();
for i in 0..200 {
for i in 0..HEAVY_FILES {
touch(&heavy.join(format!("d{}/big{:04}.txt", i % 8, i)), &body);
}
// LIGHT: a wide tree of tiny files, so its walk runs long enough to sample
// and its progress counter moves finely.
let light = tmp_dir("stall-light");
for i in 0..6000 {
for i in 0..LIGHT_FILES {
touch(&light.join(format!("d{}/f{:05}.txt", i % 60, i)), b"x");
}
let db_dir = tmp_dir("stall-db");
let db = db_dir.join("index.sqlite");
let roots = vec![
heavy.to_string_lossy().into_owned(),
light.to_string_lossy().into_owned(),
];
let mut config = test_config();
config.processing.maximum_text_size = 1024;
config.processing.maximum_text_file_size = 8 * 1024 * 1024;
config.processing.batch_size = QUANTUM;
// One extraction thread for the heavy root, so its pass costs about what
// the broken design's inline read would and the two differ only in *which*
// thread pays for it. `root_workers` is keyed by the `indexing_paths`
// spelling; both sides canonicalize before matching.
config.paths.indexing_paths = roots.clone();
config.indexing.root_workers.insert(roots[0].clone(), 1);
// The default WAL cap is far above anything this run writes, so no forced
// checkpoint lands inside the window. That stops being true if the fixture
// ever grows by an order of magnitude.
let service = IndexingService::new();
service
.start_indexing(
vec![
heavy.to_string_lossy().into_owned(),
light.to_string_lossy().into_owned(),
],
db.to_string_lossy().into_owned(),
config.clone(),
)
.start_indexing(roots, db.to_string_lossy().into_owned(), config.clone())
.unwrap();
// Sample the light root's progress while the heavy one is extracting, and
// keep the longest interval over which it did not move.
let mut worst = Duration::ZERO;
let mut last_change = Instant::now();
let mut last_seen = 0usize;
let mut sampled_together = false;
let deadline = Instant::now() + Duration::from_secs(120);
while Instant::now() < deadline {
match service.get_status() {
IndexingStatus::Running { roots, .. } => {
let heavy_p = roots.iter().find(|r| r.root.contains("stall-heavy"));
let light_p = roots.iter().find(|r| r.root.contains("stall-light"));
if let (Some(h), Some(l)) = (heavy_p, light_p) {
let light_busy = l.phase != RootPhase::Done;
if h.phase == RootPhase::Extracting && light_busy {
sampled_together = true;
let now = l.walked + l.extracted;
if now != last_seen {
last_seen = now;
last_change = Instant::now();
} else {
worst = worst.max(last_change.elapsed());
}
}
}
}
// The run is claimed but has not reached its walk yet; there is
// nothing to sample, and breaking here would end the watch before
// the run it is watching had started.
IndexingStatus::Preparing { .. } => {}
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
_ => break,
}
std::thread::sleep(Duration::from_millis(2));
}
service.stop_indexing().unwrap();
let seen = observe_overlap(&service, "stall-heavy", "stall-light");
drop(service);
// Before the assertions, unlike the rest of this file: those tests keep
// their trees because a failing test's tree is the evidence, but this
// fixture is generated and identical every run, and its evidence is the two
// counters printed below. Leaving 92 MB of it in a RAM-backed /tmp behind a
// failure is itself a reason for the next run to fail.
std::fs::remove_dir_all(&heavy).ok();
std::fs::remove_dir_all(&light).ok();
std::fs::remove_dir_all(&db_dir).ok();
// The fixture is as configured. Each of these silently costs a factor of
// the margin below if it stops holding, so they are checked before the
// ratio is read as a verdict on the design.
assert_eq!(
seen.heavy_pool, 1,
"the heavy root must extract on the single worker root_workers asked for; \
with the default four its pass is four times shorter and so is the margin"
);
assert_eq!(
seen.heavy_pending, HEAVY_FILES,
"every heavy file must reach the content pass; one inlined by its walk \
worker never produces an extraction phase to overlap with"
);
assert!(
sampled_together,
"never observed the two roots overlapping; the fixture is not exercising the case"
);
// Measured on this fixture: ~20 ms with the extraction pools, ~130 ms when
// the file reading is forced back onto the writer thread (and unbounded in
// the real failure, where the heavy root is on a network share). The bound
// sits between, with several times the observed headroom.
//
// The fixed design's stall does not grow with the heavy root's cost — it is
// one round-robin pass plus one commit — so making that root heavier only
// widens the margin.
// The figures above are wall-clock, so they scale with the host: a small CI
// VM running the other tests in this binary alongside this one measures
// several times the developer-machine number without the design having
// changed at all. QSB_STALL_BUDGET_MS lets that environment say so out loud
// instead of the bound being quietly loosened for everyone. Keep any override
// well under the broken design's figure scaled by the same factor, or the
// test stops discriminating between the two.
let budget = Duration::from_millis(
std::env::var("QSB_STALL_BUDGET_MS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(100),
seen.heavy_stored * 2 >= HEAVY_FILES,
"only {} of {} heavy rows landed inside the observed window; the sample \
did not cover the pass",
seen.heavy_stored,
HEAVY_FILES
);
eprintln!(
"longest light-root stall while heavy extracted: {:?} (budget {:?})",
worst, budget
"light files drained while the heavy root extracted: {} against {} heavy \
rows (quantum {}) — {}x the {}x required; the serialised design cannot \
exceed 1x",
seen.light_drained,
seen.heavy_stored,
QUANTUM,
seen.light_drained / (seen.heavy_stored + QUANTUM),
MIN_INTERLEAVE
);
assert!(
worst < budget,
"the light root stalled for {:?} while the heavy root extracted (budget {:?})",
worst,
budget
seen.light_drained >= MIN_INTERLEAVE * (seen.heavy_stored + QUANTUM),
"the light root was drained of only {} files while the heavy root landed \
{} rows; one quantum of each per round is all a writer that extracts \
inline can manage, so anything near {} means the extraction is back on \
the writer thread",
seen.light_drained,
seen.heavy_stored,
seen.heavy_stored + QUANTUM
);
}
/// The sibling of [`a_heavy_root_does_not_stall_a_light_one`] for the cost that
/// test deliberately keeps small: the writer's own tokenising.
///
/// There the heavy files are expensive to *read* and cheap to *write*
/// (`maximum_text_size = 1024`), so it never exercised the writer. Here each
/// heavy row carries the default 256 KiB of text and its FTS5 trigram insert is
/// the expensive step — and it runs on the writer thread, inside the
/// transaction, where nothing can take it off. Four workers keep the ready
/// channel full, so what one turn finds waiting is a whole channel of them.
///
/// Before turns had a slice, an extraction turn wrote everything it found —
/// half a second to two seconds of tokenising — and the light root's walk got
/// one quantum in between: the ratio below came in under one. With turns
/// bounded by `TURN_SLICE` and walks served first, the light root drains at
/// its own rate while the heavy root lands a row or two per round.
#[test]
fn a_heavy_root_does_not_stall_a_light_one_at_the_writer() {
const QUANTUM: usize = 16;
// Over the walk's inline threshold, and enough that the stored text is the
// full `maximum_text_size` (256 KiB) — the tokenising is what is measured.
const HEAVY_FILES: usize = 32;
// Wider than the sibling's: with the walk no longer waiting on the writer
// it drains so fast that 6000 files were gone before half the heavy rows
// had landed, and the window closed on a sample too short to trust.
const LIGHT_FILES: usize = 16_000;
// As in the sibling: three times a bound the unsliced writer cannot reach.
const MIN_INTERLEAVE: usize = 3;
let heavy = tmp_dir("stall-writer-heavy");
let body: Vec<u8> = "sphinx of black quartz judge my vow "
.repeat(9_000)
.into_bytes();
for i in 0..HEAVY_FILES {
touch(&heavy.join(format!("d{}/big{:04}.txt", i % 8, i)), &body);
}
let light = tmp_dir("stall-writer-light");
for i in 0..LIGHT_FILES {
touch(&light.join(format!("d{}/f{:05}.txt", i % 60, i)), b"x");
}
let db_dir = tmp_dir("stall-writer-db");
let db = db_dir.join("index.sqlite");
let roots = vec![
heavy.to_string_lossy().into_owned(),
light.to_string_lossy().into_owned(),
];
let mut config = test_config();
config.processing.batch_size = QUANTUM;
config.paths.indexing_paths = roots.clone();
// Four readers, so the heavy rows reach the writer faster than it can
// tokenise them and the ready channel is full when its turn comes.
config.indexing.root_workers.insert(roots[0].clone(), 4);
let service = IndexingService::new();
service
.start_indexing(roots, db.to_string_lossy().into_owned(), config.clone())
.unwrap();
let seen = observe_overlap(&service, "stall-writer-heavy", "stall-writer-light");
drop(service);
std::fs::remove_dir_all(&heavy).ok();
std::fs::remove_dir_all(&light).ok();
std::fs::remove_dir_all(&db_dir).ok();
assert_eq!(
seen.heavy_pool, 4,
"the heavy root must extract on four workers"
);
assert_eq!(
seen.heavy_pending, HEAVY_FILES,
"every heavy file must reach the content pass"
);
// A quarter, not the sibling's half: the light walk now outruns the heavy
// pass by design, and a window over eight 256 KiB rows is evidence enough
// that the writer yielded between them.
assert!(
seen.heavy_stored * 4 >= HEAVY_FILES,
"only {} of {} heavy rows landed inside the observed window; the sample \
did not cover the pass",
seen.heavy_stored,
HEAVY_FILES
);
eprintln!(
"light files drained while the heavy root tokenised: {} against {} heavy \
rows (quantum {}) — {}x the {}x required",
seen.light_drained,
seen.heavy_stored,
QUANTUM,
seen.light_drained / (seen.heavy_stored + QUANTUM),
MIN_INTERLEAVE
);
assert!(
seen.light_drained >= MIN_INTERLEAVE * (seen.heavy_stored + QUANTUM),
"the light root was drained of only {} files while the heavy root landed \
{} rows; an extraction turn is writing to the end of its batch again \
instead of yielding at its slice",
seen.light_drained,
seen.heavy_stored
);
}
/// The write-ahead log must not grow for the length of a run.

3
crates/quicksearch-core/tests/snippet_perf.rs
View file

@ -289,7 +289,8 @@ fn snippet_paths_perf_comparison() {
}
None => String::new(),
};
let _snip = snippet::extract(&text, &[q], &opts);
let folded = text.to_ascii_lowercase();
let _snip = snippet::extract_folded(&text, &folded, &[q], &opts);
rows_b_total += 1;
}
}

246
crates/quicksearch-gui/src/app.rs
View file

@ -1,6 +1,7 @@
//! Application shell: tab strip, per-frame event drains, debounce,
//! status bar, and config-change routing.
use std::path::PathBuf;
use std::sync::mpsc;
use std::time::Duration;
@ -22,8 +23,8 @@ use crate::format::{fmt_interval, group_thousands};
use crate::keychain;
use crate::logs_tab::LogsTab;
use crate::manage_tab::ManageTab;
use crate::options::{OptionsWindow, SecurityAction};
use crate::search_tab::SearchTab;
use crate::settings_tab::{SecurityAction, SettingsTab};
use crate::unlock::KeySource;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@ -33,6 +34,17 @@ pub(crate) enum Tab {
Duplicates,
Logs,
Help,
Settings,
}
/// The editor a tab holds, if it stages its edits on a draft rather than
/// saving them the moment they change.
fn tab_editor(tab: Tab) -> Option<UnsavedSource> {
match tab {
Tab::Manage => Some(UnsavedSource::Manage),
Tab::Settings => Some(UnsavedSource::Settings),
Tab::Search | Tab::Duplicates | Tab::Logs | Tab::Help => None,
}
}
/// A navigation the unsaved-changes guard put on hold; once nothing relevant
@ -40,7 +52,6 @@ pub(crate) enum Tab {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum NavIntent {
SwitchTab(Tab),
CloseOptions,
Quit,
}
@ -48,21 +59,30 @@ enum NavIntent {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum UnsavedSource {
Manage,
Options,
Settings,
}
/// Which editor the guard must ask about for `intent`, if any. Quit asks
/// about Options before Manage, one prompt at a time: each draft is a full
/// `Config` snapshot, so applying both at once would revert the first.
/// The editor `from` holds, if it has one and it is holding unapplied edits.
fn dirty_editor(from: Tab, manage_dirty: bool, settings_dirty: bool) -> Option<UnsavedSource> {
match tab_editor(from)? {
UnsavedSource::Manage => manage_dirty.then_some(UnsavedSource::Manage),
UnsavedSource::Settings => settings_dirty.then_some(UnsavedSource::Settings),
}
}
/// Which editor the guard must ask about for `intent` while sitting on
/// `from`, if any. A tab switch asks only about the tab being left; Quit asks
/// about Settings before Manage, one prompt at a time, because each draft is
/// a full `Config` snapshot and applying both at once would revert the first.
fn guard_source(
intent: NavIntent,
from: Tab,
manage_dirty: bool,
options_dirty: bool,
settings_dirty: bool,
) -> Option<UnsavedSource> {
match intent {
NavIntent::SwitchTab(_) => manage_dirty.then_some(UnsavedSource::Manage),
NavIntent::CloseOptions => options_dirty.then_some(UnsavedSource::Options),
NavIntent::Quit if options_dirty => Some(UnsavedSource::Options),
NavIntent::SwitchTab(_) => dirty_editor(from, manage_dirty, settings_dirty),
NavIntent::Quit if settings_dirty => Some(UnsavedSource::Settings),
NavIntent::Quit if manage_dirty => Some(UnsavedSource::Manage),
NavIntent::Quit => None,
}
@ -78,8 +98,13 @@ fn quit_needs_reconcile_warning(intent: NavIntent, reconciling: bool) -> bool {
/// Whether leaving the current tab has to go through the unsaved-changes
/// guard. A navigation already on hold wins: a second intent would replace
/// the answer the guard is waiting for.
fn switch_needs_guard(from: Tab, manage_dirty: bool, nav_pending: bool) -> bool {
from == Tab::Manage && manage_dirty && !nav_pending
fn switch_needs_guard(
from: Tab,
manage_dirty: bool,
settings_dirty: bool,
nav_pending: bool,
) -> bool {
!nav_pending && dirty_editor(from, manage_dirty, settings_dirty).is_some()
}
pub struct QuickSearchApp {
@ -90,9 +115,12 @@ pub struct QuickSearchApp {
manage: ManageTab,
dups: DuplicatesTab,
logs: LogsTab,
options: OptionsWindow,
settings: SettingsTab,
/// Set when applying a config that invalidates the stored index.
rebuild_prompt: Option<Vec<ConfigChange>>,
/// The first-start tour, while it is open. Only ever `Some` for a config
/// file this version created — see [`crate::tutorial`].
tutorial: Option<crate::tutorial::Tutorial>,
/// Set while the "delete the index?" confirmation is open.
clear_prompt: bool,
/// Nested roots found in the loaded config; shown as a modal over the
@ -112,8 +140,12 @@ pub struct QuickSearchApp {
/// Set when the watcher gave up on the directory budget and live
/// updates are off.
watch_cap_prompt: Option<WatchError>,
/// The byte-for-byte check of one duplicate group, while its modal is up.
verify: Option<VerifyModal>,
/// In-flight security flow (enable/disable/change password).
security_prompt: Option<SecurityPrompt>,
/// In-flight show-key flow (confirm password, then reveal).
key_prompt: Option<KeyPrompt>,
/// A navigation held by the unsaved-changes guard; see [`NavIntent`].
pending_nav: Option<NavIntent>,
/// The guard resolved a Quit: let the next close request through.
@ -129,8 +161,10 @@ mod security;
mod status_bar;
#[cfg(test)]
mod tests;
mod verify;
use security::SecurityPrompt;
use security::{KeyPrompt, SecurityPrompt};
use verify::VerifyModal;
impl QuickSearchApp {
/// `initial_query` pre-fills the search box and fires a search on the
@ -178,7 +212,12 @@ impl QuickSearchApp {
} else {
(Tab::Manage, Some(nested))
};
let mut search = SearchTab::new(fuzzy);
// `Some(false)` means a config file *this version wrote*, which is
// the only thing that counts as a first start. A key that is absent
// (`None`) belongs to an installation that upgraded into this version
// and has already found its way around.
let tutorial = (cfg.ui.tutorial_seen == Some(false)).then(crate::tutorial::Tutorial::new);
let mut search = SearchTab::new(fuzzy, cfg.search.columns.clone(), cfg.search.live_results);
if let Some(query) = initial_query {
search.seed(query);
}
@ -190,8 +229,9 @@ impl QuickSearchApp {
manage: ManageTab::new(),
dups: DuplicatesTab::new(),
logs: LogsTab::new(),
options: OptionsWindow::new(),
settings: SettingsTab::new(),
rebuild_prompt: None,
tutorial,
clear_prompt: false,
nested_prompt,
key_source,
@ -199,7 +239,9 @@ impl QuickSearchApp {
reconcile_owed,
reconcile_owed_since,
watch_cap_prompt: None,
verify: None,
security_prompt: None,
key_prompt: None,
pending_nav: None,
quit_confirmed: false,
config_error,
@ -222,6 +264,10 @@ impl QuickSearchApp {
let Some(search) = self.backend.search() else {
return;
};
// The single funnel every search goes through — the debounce, `seed`,
// and every `actions.rerun` producer — so it is the one place the old
// results' watches have to be dropped.
self.backend.clear_live();
let generation = search.search(&self.search.query, self.search_options());
self.search.on_search_started(generation);
}
@ -232,6 +278,32 @@ impl QuickSearchApp {
self.backend.start_duplicates(&cfg, ctx.clone());
}
/// Move to another tab, running what leaving one tab and arriving at the
/// other owe. Every switch goes through here — including the ones the
/// unsaved-changes guard completes a frame later, which is why this is a
/// funnel rather than a comparison against the previous frame's tab.
fn switch_tab(&mut self, ctx: &egui::Context, to: Tab) {
if self.tab == to {
return;
}
match self.tab {
// Watching rows nobody is looking at costs descriptors for
// nothing.
Tab::Search => {
self.backend.clear_live();
self.search.reset_live();
}
// A draft kept while the config is edited elsewhere would go
// stale, and applying it later would revert those edits.
Tab::Settings => self.settings.discard(),
_ => {}
}
self.tab = to;
if to == Tab::Duplicates {
self.start_duplicates_scan(ctx);
}
}
/// Save + route an edited config to the running services. Reports
/// whether the config was accepted — a `false` means nothing was saved
/// and the caller must keep any staged edits alive.
@ -290,6 +362,16 @@ impl QuickSearchApp {
self.rebuild_prompt = Some(changes);
}
}
self.search.live_enabled = new.search.live_results;
if new.search.live_results {
// The watcher holds a copy of the config for its extraction
// limits and filters, so a config edit has to re-arm; dropping
// the tab-side state is what makes the next frame do it.
self.search.reset_live();
} else {
self.backend.clear_live();
self.search.reset_live();
}
self.cfg = new;
true
}
@ -297,18 +379,23 @@ impl QuickSearchApp {
/// What the system-wide search shortcut does once the window is up:
/// show the Search tab with the caret in the query box and any existing
/// text selected.
pub(crate) fn activate_search(&mut self) {
if switch_needs_guard(self.tab, self.manage.is_dirty(), self.pending_nav.is_some()) {
pub(crate) fn activate_search(&mut self, ctx: &egui::Context) {
if switch_needs_guard(
self.tab,
self.manage.is_dirty(),
self.settings.is_dirty(&self.cfg),
self.pending_nav.is_some(),
) {
self.pending_nav = Some(NavIntent::SwitchTab(Tab::Search));
} else {
self.tab = Tab::Search;
self.switch_tab(ctx, Tab::Search);
}
self.search.request_focus();
}
/// Whether the Options window is currently reading a key press to bind.
/// Whether the Settings tab is currently reading a key press to bind.
pub(crate) fn capturing_hotkey(&self) -> bool {
self.options.capturing_hotkey()
self.tab == Tab::Settings && self.settings.capturing_hotkey()
}
/// Switch the indexing mode and write it to the config immediately: a
@ -334,6 +421,21 @@ impl QuickSearchApp {
self.search
.apply_update(update, self.cfg.search.display_limit);
}
// Every live update is something the watcher read off the disk that
// the index has not been told about. Handing the paths back keeps the
// index from drifting away from the rows on screen — and it is the
// only thing that does so while indexing is stopped.
let mut touched: Vec<PathBuf> = Vec::new();
while let Ok(update) = self.backend.live_rx.try_recv() {
touched.push(PathBuf::from(update.path()));
// A rename has two sides: the old path leaves the index and the
// new one enters it.
if let quicksearch_core::live::LiveUpdate::Renamed { to, .. } = &update {
touched.push(PathBuf::from(to));
}
self.search.apply_live(update);
}
self.backend.reindex_live_paths(touched);
// Duplicates worker.
if let Some(rx) = &self.backend.dup_job {
use std::sync::mpsc::TryRecvError;
@ -350,6 +452,7 @@ impl QuickSearchApp {
self.backend.dup_job = None;
}
}
self.drain_verify();
}
fn tick_debounce(&mut self, ctx: &egui::Context) {
@ -389,7 +492,7 @@ impl QuickSearchApp {
}
pub(crate) fn capture_search_settled(&self) -> bool {
self.search.capture_settled()
self.search.settled()
}
pub(crate) fn capture_dups_done(&self) -> bool {
@ -419,6 +522,12 @@ impl QuickSearchApp {
pub(crate) fn pin_live_fields(new: &mut Config, live: &Config) {
new.security = live.security.clone();
new.indexing.auto_index = live.indexing.auto_index;
// The column picker writes straight to the live config the moment a
// checkbox moves — from the table header *or* from the Settings tab,
// which is why the Settings controls for it are not draft-backed. Pinning
// here is what stops a draft taken before a header-menu change from
// undoing it on Apply.
new.search.columns = live.search.columns.clone();
}
/// Keep the configured UI scale within sane, recoverable bounds.
@ -448,7 +557,8 @@ pub(crate) fn apply_theme(ctx: &egui::Context, setting: &str) {
impl eframe::App for QuickSearchApp {
fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
// First, so `previous_tab` below sees pre-navigation state.
// First, so a scripted navigation is held before the tab strip reads
// this frame's state.
#[cfg(feature = "capture")]
self.capture_tick(ctx);
@ -461,7 +571,7 @@ impl eframe::App for QuickSearchApp {
if ctx.input(|i| i.viewport().close_requested())
&& !self.quit_confirmed
&& (self.manage.is_dirty()
|| self.options.is_dirty(&self.cfg)
|| self.settings.is_dirty(&self.cfg)
|| self.backend.coordinator.reconciling())
{
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
@ -471,7 +581,6 @@ impl eframe::App for QuickSearchApp {
self.status_bar(ctx);
let previous_tab = self.tab;
// Tab clicks land on a local first so the unsaved-changes guard can
// hold them.
let mut requested = self.tab;
@ -482,31 +591,21 @@ impl eframe::App for QuickSearchApp {
ui.selectable_value(&mut requested, Tab::Duplicates, "Duplicates");
ui.selectable_value(&mut requested, Tab::Logs, "Logs");
ui.selectable_value(&mut requested, Tab::Help, "Help");
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if ui.button("⚙").on_hover_text("Options").clicked() {
if !self.options.open {
self.options.open_with(&self.cfg);
} else if self.options.is_dirty(&self.cfg) {
if self.pending_nav.is_none() {
self.pending_nav = Some(NavIntent::CloseOptions);
}
} else {
self.options.close_discard();
}
}
});
ui.selectable_value(&mut requested, Tab::Settings, "Settings");
});
});
if requested != self.tab {
if switch_needs_guard(self.tab, self.manage.is_dirty(), self.pending_nav.is_some()) {
if switch_needs_guard(
self.tab,
self.manage.is_dirty(),
self.settings.is_dirty(&self.cfg),
self.pending_nav.is_some(),
) {
self.pending_nav = Some(NavIntent::SwitchTab(requested));
} else {
self.tab = requested;
self.switch_tab(ctx, requested);
}
}
if self.tab == Tab::Duplicates && previous_tab != Tab::Duplicates {
self.start_duplicates_scan(ctx);
}
if let Some(err) = &self.config_error {
let err = err.clone();
@ -533,6 +632,17 @@ impl eframe::App for QuickSearchApp {
self.config_error = Some(e);
}
}
// Live state, saved the moment it changes — like the fuzzy
// default above, and unlike anything edited through the
// Settings draft. The Settings tab's own column controls take
// this same path, so the two editors cannot disagree and a
// stale draft cannot revert either of them.
if let Some(columns) = actions.save_columns {
self.cfg.search.columns = columns;
if let Err(e) = self.cfg.save() {
self.config_error = Some(e);
}
}
if let Some(pattern) = actions.persist_ignore {
let mut new_cfg = self.cfg.clone();
if !new_cfg.indexing.ignore_patterns.contains(&pattern) {
@ -540,6 +650,10 @@ impl eframe::App for QuickSearchApp {
self.apply_new_config(ctx, new_cfg);
}
}
if let Some(targets) = actions.live_targets {
self.backend
.watch_live(&self.search.query, targets, &self.cfg);
}
if actions.rerun {
self.start_search();
}
@ -571,32 +685,54 @@ impl eframe::App for QuickSearchApp {
}
}
Tab::Duplicates => {
let actions = self.dups.ui(ui);
let actions = self.dups.ui(ui, self.verify.is_some());
if actions.refresh {
self.start_duplicates_scan(ctx);
}
if let Some(paths) = actions.verify {
let paths: Vec<std::path::PathBuf> =
paths.into_iter().map(std::path::PathBuf::from).collect();
self.backend.start_verify(paths.clone(), ctx.clone());
self.verify = Some(VerifyModal::new(paths));
}
}
Tab::Logs => self.logs.ui(ui),
Tab::Help => crate::help_tab::ui(ui),
Tab::Help => {
if crate::help_tab::ui(ui) {
self.show_tutorial();
}
}
Tab::Settings => {
let out = self.settings.ui(ui, &self.cfg);
if let Some(new_cfg) = out.applied {
self.apply_new_config(ctx, new_cfg);
}
if let Some(action) = out.security {
self.handle_security_action(action);
}
// Same live path the table header's picker takes, so the two
// controls stay in step and neither needs an Apply.
if let Some(columns) = out.columns {
self.cfg.search.columns = columns.clone();
self.search.columns = columns;
self.search.mark_sort_dirty();
if let Err(e) = self.cfg.save() {
self.config_error = Some(e);
}
}
}
});
let options_out = self.options.ui(ctx, &self.cfg);
if let Some(new_cfg) = options_out.applied {
self.apply_new_config(ctx, new_cfg);
}
if let Some(action) = options_out.security {
self.handle_security_action(action);
}
if options_out.close_requested && self.pending_nav.is_none() {
self.pending_nav = Some(NavIntent::CloseOptions);
}
self.rebuild_prompt_ui(ctx);
self.security_prompt_ui(ctx);
self.key_prompt_ui(ctx);
self.clear_prompt_ui(ctx);
self.nested_prompt_ui(ctx);
// Ahead of the watch-cap warning: on a fresh upgrade both can be true.
self.stale_index_prompt_ui(ctx);
self.watch_cap_prompt_ui(ctx);
self.verify_modal_ui(ctx);
self.tutorial_ui(ctx);
// Last: the guard must sit above everything else on screen.
self.unsaved_prompt_ui(ctx);
}

52
crates/quicksearch-gui/src/app/modals.rs
View file

@ -84,7 +84,7 @@ impl QuickSearchApp {
});
if close == Some(true) {
self.nested_prompt = None;
self.tab = Tab::Manage;
self.switch_tab(ctx, Tab::Manage);
}
}
@ -187,6 +187,29 @@ impl QuickSearchApp {
}
}
/// The first-start tour. Dismissal is written straight to the config, the
/// way the fuzzy default is — not through the Settings draft, which this
/// has nothing to do with.
pub(super) fn tutorial_ui(&mut self, ctx: &egui::Context) {
let Some(tour) = &mut self.tutorial else {
return;
};
let roots = self.cfg.paths.indexing_paths.clone();
if tour.ui(ctx, &roots) {
self.tutorial = None;
self.cfg.ui.tutorial_seen = Some(true);
if let Err(e) = self.cfg.save() {
self.config_error = Some(e);
}
}
}
/// Re-open the tour from the Help tab. Nothing is written until it is
/// dismissed again, so a re-read costs the config nothing.
pub(crate) fn show_tutorial(&mut self) {
self.tutorial = Some(crate::tutorial::Tutorial::new());
}
pub(super) fn clear_prompt_ui(&mut self, ctx: &egui::Context) {
if !self.clear_prompt {
return;
@ -225,8 +248,10 @@ impl QuickSearchApp {
let Some(intent) = self.pending_nav else {
return;
};
let dirty = (self.manage.is_dirty(), self.options.is_dirty(&self.cfg));
let Some(source) = guard_source(intent, dirty.0, dirty.1) else {
let dirty = (self.manage.is_dirty(), self.settings.is_dirty(&self.cfg));
// `self.tab` is the tab being left: the switch itself is what the
// intent is holding back.
let Some(source) = guard_source(intent, self.tab, dirty.0, dirty.1) else {
// Inside the guard: the Discard-then-quit path sets
// `quit_confirmed` and never returns to the close-request check,
// so a warning living only there would be skipped.
@ -249,7 +274,7 @@ impl QuickSearchApp {
Some(UnsavedChoice::Cancel) => self.pending_nav = None,
Some(UnsavedChoice::Discard) => match source {
UnsavedSource::Manage => self.manage.discard(),
UnsavedSource::Options => self.options.close_discard(),
UnsavedSource::Settings => self.settings.discard(),
},
Some(UnsavedChoice::Apply) => {
let ok = match source {
@ -263,11 +288,11 @@ impl QuickSearchApp {
}
None => true,
},
UnsavedSource::Options => match self.options.draft_config() {
UnsavedSource::Settings => match self.settings.draft_config() {
Some(cfg) => {
let ok = self.apply_new_config(ctx, cfg);
if ok {
self.options.close_discard();
self.settings.discard();
}
ok
}
@ -287,14 +312,7 @@ impl QuickSearchApp {
pub(super) fn complete_nav(&mut self, ctx: &egui::Context, intent: NavIntent) {
self.pending_nav = None;
match intent {
NavIntent::SwitchTab(tab) => {
let was = self.tab;
self.tab = tab;
if tab == Tab::Duplicates && was != Tab::Duplicates {
self.start_duplicates_scan(ctx);
}
}
NavIntent::CloseOptions => self.options.close_discard(),
NavIntent::SwitchTab(tab) => self.switch_tab(ctx, tab),
NavIntent::Quit => {
self.quit_confirmed = true;
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
@ -308,7 +326,7 @@ impl QuickSearchApp {
///
/// Unlike the centered `egui::Window` the other prompts use, `egui::Modal`'s
/// backdrop blocks input to everything behind it — a click landing on the
/// tab strip or the Options ✕ would re-trigger or bypass the guard.
/// tab strip would re-trigger or bypass the guard.
fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<UnsavedChoice> {
let mut choice = None;
let modal = egui::Modal::new(egui::Id::new("unsaved-guard")).show(ctx, |ui| {
@ -316,7 +334,7 @@ fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<U
ui.heading("Unsaved changes");
ui.label(match source {
UnsavedSource::Manage => "The Manage Index tab has edits that have not been applied.",
UnsavedSource::Options => "The Options window has edits that have not been applied.",
UnsavedSource::Settings => "The Settings tab has edits that have not been applied.",
});
ui.add_space(6.0);
ui.horizontal(|ui| {
@ -527,7 +545,7 @@ mod tests {
/// buttons fire, Esc cancels, and an untouched frame decides nothing.
#[test]
fn the_unsaved_modal_reports_each_choice() {
for source in [UnsavedSource::Manage, UnsavedSource::Options] {
for source in [UnsavedSource::Manage, UnsavedSource::Settings] {
let ctx = egui::Context::default();
assert_eq!(
modal_frame(&ctx, source, Vec::new()),

273
crates/quicksearch-gui/src/app/security.rs
View file

@ -3,7 +3,9 @@
use super::*;
use crate::ui_util::centered_modal;
use quicksearch_core::security::SALT_LEN;
use crate::ui_util::{centered_modal, hint};
/// The two-step security flow: collect a password (enable/change), derive
/// its key off the UI thread, then confirm the mandatory index rebuild.
@ -16,7 +18,10 @@ pub(super) enum SecurityPrompt {
change: bool,
},
Deriving {
rx: mpsc::Receiver<(SecurityConfig, IndexKey)>,
rx: mpsc::Receiver<IndexKey>,
/// Built with the salt the pending key is being derived from, so the
/// two always describe each other.
new_security: SecurityConfig,
},
ConfirmRebuild {
new_security: SecurityConfig,
@ -33,8 +38,69 @@ impl Drop for SecurityPrompt {
}
}
/// The show-key flow: confirm the password, re-derive from it, then reveal
/// the installed key. Nothing here can change the key or the config.
pub(super) enum KeyPrompt {
Confirm {
pw: String,
wrong: bool,
},
Deriving {
rx: mpsc::Receiver<IndexKey>,
},
/// The key as displayed: `0x` followed by 64 hex digits.
Reveal {
display: String,
},
}
impl Drop for KeyPrompt {
fn drop(&mut self) {
match self {
KeyPrompt::Confirm { pw, .. } => pw.zeroize(),
KeyPrompt::Reveal { display } => display.zeroize(),
KeyPrompt::Deriving { .. } => {}
}
}
}
/// Derive a key off the UI thread. The password is consumed and dropped
/// there, so it never outlives the derivation.
fn spawn_derive(
ctx: &egui::Context,
password: Zeroizing<String>,
salt: [u8; SALT_LEN],
) -> mpsc::Receiver<IndexKey> {
let (tx, rx) = mpsc::channel();
let repaint = ctx.clone();
std::thread::spawn(move || {
let key = derive_key(&password, &salt);
drop(password);
let _ = tx.send(key);
repaint.request_repaint();
});
rx
}
/// Paint the spinner shown while a derivation runs. Not `centered_modal`:
/// this one hides its title bar.
fn deriving_window(ctx: &egui::Context) {
egui::Window::new("Deriving key")
.collapsible(false)
.resizable(false)
.title_bar(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.horizontal(|ui| {
ui.spinner();
ui.label("Deriving key…");
});
});
ctx.request_repaint_after(Duration::from_millis(100));
}
impl QuickSearchApp {
/// Route a click in the Options window's Security block. Keychain
/// Route a click in the Settings tab's Security block. Keychain
/// toggles act immediately; everything else opens the two-step flow.
pub(super) fn handle_security_action(&mut self, action: SecurityAction) {
match action {
@ -52,6 +118,12 @@ impl QuickSearchApp {
new_key: None,
});
}
SecurityAction::ShowKey => {
self.key_prompt = Some(KeyPrompt::Confirm {
pw: String::new(),
wrong: false,
});
}
SecurityAction::SetKeychain(remember) => {
let db_path = self.cfg.resolved_database_path();
if remember {
@ -139,47 +211,25 @@ impl QuickSearchApp {
} else if submit {
let password = Zeroizing::new(std::mem::take(pw1));
pw2.zeroize();
let remember = *remember;
let salt = generate_salt();
let new_security = SecurityConfig {
password_protected: true,
salt: Some(salt_to_hex(&salt)),
use_keychain: *remember,
};
purge_security_field_state(ctx);
let (tx, rx) = mpsc::channel();
let repaint = ctx.clone();
std::thread::spawn(move || {
let salt = generate_salt();
let key = derive_key(&password, &salt);
drop(password);
let new_security = SecurityConfig {
password_protected: true,
salt: Some(salt_to_hex(&salt)),
use_keychain: remember,
};
let _ = tx.send((new_security, key));
repaint.request_repaint();
});
self.security_prompt = Some(SecurityPrompt::Deriving { rx });
let rx = spawn_derive(ctx, password, salt);
self.security_prompt = Some(SecurityPrompt::Deriving { rx, new_security });
}
}
SecurityPrompt::Deriving { rx } => match rx.try_recv() {
Ok((new_security, key)) => {
SecurityPrompt::Deriving { rx, new_security } => match rx.try_recv() {
Ok(key) => {
self.security_prompt = Some(SecurityPrompt::ConfirmRebuild {
new_security,
new_security: new_security.clone(),
new_key: Some(key),
});
}
Err(mpsc::TryRecvError::Empty) => {
// Not `centered_modal`: this one hides its title bar.
egui::Window::new("Deriving key")
.collapsible(false)
.resizable(false)
.title_bar(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.horizontal(|ui| {
ui.spinner();
ui.label("Deriving key…");
});
});
ctx.request_repaint_after(Duration::from_millis(100));
}
Err(mpsc::TryRecvError::Empty) => deriving_window(ctx),
Err(mpsc::TryRecvError::Disconnected) => {
self.config_error = Some("key derivation thread died".to_string());
self.security_prompt = None;
@ -222,6 +272,74 @@ impl QuickSearchApp {
}
}
/// Render the show-key flow (drawn with the other modals). Only ever
/// open while protection is on, so a salt and a process key both exist.
pub(super) fn key_prompt_ui(&mut self, ctx: &egui::Context) {
let Some(prompt) = &mut self.key_prompt else {
return;
};
match prompt {
KeyPrompt::Confirm { pw, wrong } => {
let (submit, cancel) = confirm_key_modal(ctx, pw, *wrong);
if cancel {
self.key_prompt = None; // Drop impl zeroizes
purge_security_field_state(ctx);
} else if submit {
let password = Zeroizing::new(std::mem::take(pw));
purge_security_field_state(ctx);
match self.cfg.security.salt_bytes() {
Ok(salt) => {
let rx = spawn_derive(ctx, password, salt);
self.key_prompt = Some(KeyPrompt::Deriving { rx });
}
Err(e) => {
self.config_error = Some(e);
self.key_prompt = None;
}
}
}
}
KeyPrompt::Deriving { rx } => match rx.try_recv() {
Ok(key) => match db::process_key_hex() {
// What is shown is the installed key, not the derived
// one: it is the key that actually opens the index.
Some(installed) => match reveal_display(&installed, &key.to_hex()) {
Some(display) => {
self.key_prompt = Some(KeyPrompt::Reveal { display });
}
None => {
self.key_prompt = Some(KeyPrompt::Confirm {
pw: String::new(),
wrong: true,
});
}
},
None => {
// Unreachable while protected — the gate always
// installs a key before the app starts.
self.config_error =
Some("no key installed; restart and unlock first".to_string());
self.key_prompt = None;
}
},
Err(mpsc::TryRecvError::Empty) => deriving_window(ctx),
Err(mpsc::TryRecvError::Disconnected) => {
self.config_error = Some("key derivation thread died".to_string());
self.key_prompt = None;
}
},
KeyPrompt::Reveal { display } => {
let (copy, close) = reveal_key_modal(ctx, display);
if copy {
ctx.copy_text(display.clone());
}
if close {
self.key_prompt = None; // Drop impl zeroizes
}
}
}
}
/// Commit a confirmed security change: config, keychain, process key —
/// in that order, before the rebuild so the fresh index is created
/// under the new key (or none).
@ -255,11 +373,92 @@ impl QuickSearchApp {
}
}
/// Id of the show-key confirmation field, shared by the widget and the
/// purge below.
const SHOW_KEY_FIELD: &str = "show-key-pw";
/// Drop egui's retained text-field state (buffer + undo history) for the
/// password dialog fields.
fn purge_security_field_state(ctx: &egui::Context) {
ctx.data_mut(|d| {
d.remove::<egui::text_edit::TextEditState>(egui::Id::new("security-pw1"));
d.remove::<egui::text_edit::TextEditState>(egui::Id::new("security-pw2"));
d.remove::<egui::text_edit::TextEditState>(egui::Id::new(SHOW_KEY_FIELD));
});
}
/// The display form of the installed key, or `None` when the password the
/// user typed does not derive it. Both arguments come from
/// [`IndexKey::to_hex`], which is always lowercase, so a plain comparison is
/// exact; nothing secret is learned from its timing, since the caller
/// already holds the guess.
fn reveal_display(installed_hex: &str, derived_hex: &str) -> Option<String> {
(installed_hex == derived_hex).then(|| format!("0x{}", installed_hex))
}
/// Paint the password confirmation; `(submit, cancel)` from its buttons.
/// Free, like the reveal below, so both halves of the flow can be rendered
/// against a bare context.
fn confirm_key_modal(ctx: &egui::Context, pw: &mut String, wrong: bool) -> (bool, bool) {
centered_modal(ctx, "Show database key", |ui| {
ui.set_max_width(360.0);
ui.label(
"Confirm your password to show the raw key the index is \
encrypted with.",
);
let field = ui.add(
egui::TextEdit::singleline(pw)
.id(egui::Id::new(SHOW_KEY_FIELD))
.password(true)
.hint_text("Password")
.desired_width(240.0),
);
// On open, and again after a wrong attempt. Never steals focus from
// something the user moved to themselves.
if ui.memory(|m| m.focused().is_none()) {
field.request_focus();
}
if wrong {
ui.colored_label(ui.visuals().error_fg_color, "That password is not correct.");
}
ui.horizontal(|ui| {
let ok = !pw.is_empty();
// Enter in the field submits, like the unlock screen.
let entered = field.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter));
let submit =
ui.add_enabled(ok, egui::Button::new("Show key")).clicked() || (ok && entered);
(submit, ui.button("Cancel").clicked())
})
.inner
})
.unwrap_or((false, false))
}
/// Paint the revealed key; `(copy, close)` from its buttons. A free function
/// rather than a method so it can be rendered against a bare context.
fn reveal_key_modal(ctx: &egui::Context, display: &str) -> (bool, bool) {
centered_modal(ctx, "Database key", |ui| {
ui.set_max_width(420.0);
ui.label(
"This is the SQLCipher raw key for the index. Anyone holding it can \
read the index without the password.",
);
ui.add_space(6.0);
ui.horizontal_wrapped(|ui| {
ui.label(egui::RichText::new(display).monospace());
});
ui.add_space(6.0);
ui.label(hint(
"Other SQLCipher tools take the key in this form. A copy stays on the \
clipboard until something else replaces it.",
));
ui.add_space(6.0);
ui.horizontal(|ui| (ui.button("Copy").clicked(), ui.button("Close").clicked()))
.inner
})
.unwrap_or((false, false))
}
#[cfg(test)]
#[path = "security_tests.rs"]
mod tests;

169
crates/quicksearch-gui/src/app/security_tests.rs Normal file
View file

@ -0,0 +1,169 @@
use super::*;
use crate::test_ui::{click_at, painted_text, painted_text_center, raw_input};
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
/// Two 64-digit keys that differ, in the lowercase form [`IndexKey::to_hex`]
/// produces.
const KEY: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
const OTHER: &str = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210";
/// Two passes: an `egui::Window` is measured on its first frame and placed on
/// the next, so a single pass paints nothing to read back. Same shape as the
/// verify modal's test frame.
fn frame(
ctx: &egui::Context,
display: &str,
events: Vec<egui::Event>,
) -> (egui::FullOutput, (bool, bool)) {
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
reveal_key_modal(ctx, display);
});
let mut buttons = (false, false);
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
buttons = reveal_key_modal(ctx, display);
});
(out, buttons)
}
/// The confirmation half, in the same two passes.
fn confirm_frame(
ctx: &egui::Context,
pw: &mut String,
wrong: bool,
events: Vec<egui::Event>,
) -> (egui::FullOutput, (bool, bool)) {
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
confirm_key_modal(ctx, pw, wrong);
});
let mut buttons = (false, false);
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
buttons = confirm_key_modal(ctx, pw, wrong);
});
(out, buttons)
}
/// An empty field cannot submit: there is nothing to derive from, and a
/// dead button says so more clearly than a rejected attempt would.
#[test]
fn the_confirmation_will_not_submit_an_empty_password() {
let ctx = egui::Context::default();
let mut pw = String::new();
let (out, buttons) = confirm_frame(&ctx, &mut pw, false, Vec::new());
assert_eq!(buttons, (false, false));
let pos = painted_text_center(&out, "Show key").expect("no submit button painted");
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(pos));
assert!(!buttons.0, "an empty password was submitted");
}
#[test]
fn the_confirmation_submits_a_typed_password_and_cancels_on_request() {
let ctx = egui::Context::default();
let mut pw = "hunter2".to_string();
let (out, _) = confirm_frame(&ctx, &mut pw, false, Vec::new());
assert!(
!painted_text(&out).contains(&pw),
"the password was painted in the clear: {:?}",
painted_text(&out)
);
let submit = painted_text_center(&out, "Show key").expect("no submit button painted");
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(submit));
assert_eq!(buttons, (true, false));
let cancel = painted_text_center(&out, "Cancel").expect("no cancel button painted");
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(cancel));
assert_eq!(buttons, (false, true));
}
/// A retry has to say why it is asking again, or it reads as the dialog
/// having ignored the first attempt.
#[test]
fn a_retry_says_the_password_was_wrong() {
let ctx = egui::Context::default();
let mut pw = String::new();
let (quiet, _) = confirm_frame(&ctx, &mut pw, false, Vec::new());
assert!(
!painted_text(&quiet)
.iter()
.any(|t| t.contains("not correct")),
"the first attempt was called wrong before it was made"
);
let (out, _) = confirm_frame(&ctx, &mut pw, true, Vec::new());
assert!(
painted_text(&out)
.iter()
.any(|t| t.contains("That password is not correct")),
"{:?}",
painted_text(&out)
);
}
/// The right password derives the installed key, and the key is shown in the
/// `0x` form other SQLCipher tools take.
#[test]
fn the_matching_password_reveals_the_installed_key() {
assert_eq!(reveal_display(KEY, KEY), Some(format!("0x{KEY}")));
}
/// A wrong password derives some other key. Nothing about the real one may
/// leak from the attempt, so the caller gets no display string at all.
#[test]
fn a_password_that_derives_another_key_reveals_nothing() {
assert_eq!(reveal_display(KEY, OTHER), None);
assert_eq!(reveal_display(KEY, ""), None);
// A prefix must not pass: the whole key is compared, not the start of it.
assert_eq!(reveal_display(KEY, &KEY[..62]), None);
// Both sides come from `to_hex`, which is always lowercase, so an
// uppercase spelling is a mismatch rather than a value to normalise.
assert_eq!(reveal_display(KEY, &KEY.to_uppercase()), None);
}
#[test]
fn the_reveal_shows_the_key_and_what_holding_it_means() {
let ctx = egui::Context::default();
let display = format!("0x{KEY}");
let painted = painted_text(&frame(&ctx, &display, Vec::new()).0);
assert!(
painted.contains(&display),
"the key itself is not on screen: {painted:?}"
);
assert!(
painted
.iter()
.any(|t| t.contains("read the index without the password")),
"no warning about what the key is: {painted:?}"
);
assert!(painted.contains(&"Copy".to_string()), "{painted:?}");
assert!(painted.contains(&"Close".to_string()), "{painted:?}");
}
#[test]
fn both_of_the_reveal_buttons_report_their_click() {
let display = format!("0x{KEY}");
for (label, expected) in [("Copy", (true, false)), ("Close", (false, true))] {
let ctx = egui::Context::default();
let (out, _) = frame(&ctx, &display, Vec::new());
let pos =
painted_text_center(&out, label).unwrap_or_else(|| panic!("no {label} button painted"));
let (_, buttons) = frame(&ctx, &display, click_at(pos));
assert_eq!(
buttons, expected,
"clicking {label} reported the wrong pair"
);
}
}
/// The displayed string is the whole key and nothing else: a truncated or
/// annotated form would be pasted into other tools and fail there.
#[test]
fn the_display_form_is_the_prefix_and_the_whole_key() {
let display = reveal_display(KEY, KEY).expect("a match reveals");
assert_eq!(display.len(), 66);
assert!(display.starts_with("0x"));
assert!(display[2..].bytes().all(|b| b.is_ascii_hexdigit()));
}

6
crates/quicksearch-gui/src/app/status_bar.rs
View file

@ -261,7 +261,7 @@ mod tests {
walked,
walk_total,
extracted: 0,
extract_total: 0,
extract_total: None,
current_file: None,
active_workers: 2,
total_workers: 4,
@ -299,11 +299,11 @@ mod tests {
let mut extracting = root(RootPhase::Extracting, 1_000, None);
extracting.extracted = 200;
extracting.extract_total = 800;
extracting.extract_total = Some(800);
extracting.active_workers = 3;
let mut done = root(RootPhase::Done, 500, None);
done.extracted = 500;
done.extract_total = 500;
done.extract_total = Some(500);
done.active_workers = 0;
done.total_workers = 0;
assert_eq!(

124
crates/quicksearch-gui/src/app/tests.rs
View file

@ -30,61 +30,96 @@ fn a_stale_draft_cannot_revert_the_indexing_mode_or_security() {
}
/// The guard decision table for leaving a tab, however it is asked for.
/// Both draft-backed tabs guard their own departure, and neither answers for
/// the other.
#[test]
fn leaving_a_dirty_manage_tab_is_guarded_however_it_is_asked_for() {
assert!(switch_needs_guard(Tab::Manage, true, false));
assert!(
!switch_needs_guard(Tab::Manage, false, false),
"a clean editor has nothing to ask about"
);
assert!(
!switch_needs_guard(Tab::Manage, true, true),
"one held navigation at a time"
);
fn leaving_a_dirty_editor_tab_is_guarded_however_it_is_asked_for() {
// (the tab, whether *its* editor is the dirty one in the pair below)
for (tab, manage_dirty, settings_dirty) in
[(Tab::Manage, true, false), (Tab::Settings, false, true)]
{
assert!(
switch_needs_guard(tab, manage_dirty, settings_dirty, false),
"{tab:?} must guard its own unapplied edits"
);
assert!(
!switch_needs_guard(tab, false, false, false),
"{tab:?}: a clean editor has nothing to ask about"
);
assert!(
!switch_needs_guard(tab, manage_dirty, settings_dirty, true),
"{tab:?}: one held navigation at a time"
);
assert!(
!switch_needs_guard(tab, !manage_dirty, !settings_dirty, false),
"{tab:?} must not answer for the other editor"
);
}
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
assert!(
!switch_needs_guard(tab, true, false),
!switch_needs_guard(tab, true, true, false),
"{tab:?} holds no unapplied edits of its own"
);
}
}
#[test]
fn guard_source_orders_quit_prompts_options_first() {
fn guard_source_orders_quit_prompts_settings_first() {
use super::NavIntent::*;
let tab = SwitchTab(Tab::Search);
let leave = SwitchTab(Tab::Search);
assert_eq!(guard_source(tab, true, true), Some(UnsavedSource::Manage));
assert_eq!(guard_source(tab, true, false), Some(UnsavedSource::Manage));
// A switch asks about the tab being left, and only about that one.
assert_eq!(
guard_source(tab, false, true),
guard_source(leave, Tab::Manage, true, true),
Some(UnsavedSource::Manage)
);
assert_eq!(
guard_source(leave, Tab::Manage, true, false),
Some(UnsavedSource::Manage)
);
assert_eq!(
guard_source(leave, Tab::Manage, false, true),
None,
"options guard its own close"
);
assert_eq!(guard_source(tab, false, false), None);
assert_eq!(
guard_source(CloseOptions, true, true),
Some(UnsavedSource::Options)
"the Settings draft is not what leaving Manage disturbs"
);
assert_eq!(
guard_source(CloseOptions, false, true),
Some(UnsavedSource::Options)
guard_source(leave, Tab::Settings, true, true),
Some(UnsavedSource::Settings)
);
assert_eq!(
guard_source(CloseOptions, true, false),
guard_source(leave, Tab::Settings, false, true),
Some(UnsavedSource::Settings)
);
assert_eq!(
guard_source(leave, Tab::Settings, true, false),
None,
"manage guards tab switches"
"the Manage draft is not what leaving Settings disturbs"
);
assert_eq!(guard_source(CloseOptions, false, false), None);
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
assert_eq!(
guard_source(leave, tab, true, true),
None,
"{tab:?} stages nothing, so leaving it asks nothing"
);
}
assert_eq!(guard_source(Quit, true, true), Some(UnsavedSource::Options));
assert_eq!(
guard_source(Quit, false, true),
Some(UnsavedSource::Options)
);
assert_eq!(guard_source(Quit, true, false), Some(UnsavedSource::Manage));
assert_eq!(guard_source(Quit, false, false), None);
// Quit asks about both, Settings first.
for from in [Tab::Search, Tab::Manage, Tab::Settings] {
assert_eq!(
guard_source(Quit, from, true, true),
Some(UnsavedSource::Settings),
"{from:?}: quitting asks about Settings before Manage"
);
assert_eq!(
guard_source(Quit, from, false, true),
Some(UnsavedSource::Settings)
);
assert_eq!(
guard_source(Quit, from, true, false),
Some(UnsavedSource::Manage)
);
assert_eq!(guard_source(Quit, from, false, false), None);
}
}
/// Only a Quit during a running reconcile warns; a tab switch does not end
@ -95,10 +130,25 @@ fn only_quitting_during_a_reconcile_warns() {
assert!(quit_needs_reconcile_warning(Quit, true));
assert!(!quit_needs_reconcile_warning(Quit, false));
assert!(!quit_needs_reconcile_warning(SwitchTab(Tab::Search), true));
assert!(!quit_needs_reconcile_warning(CloseOptions, true));
assert!(!quit_needs_reconcile_warning(
SwitchTab(Tab::Settings),
true
));
}
/// The two values the Options window writes, plus hand-edited variants.
/// Every tab is placed on exactly one side of the guard, so a tab added
/// later cannot quietly inherit "stages nothing".
#[test]
fn only_the_two_draft_backed_tabs_have_an_editor() {
assert_eq!(tab_editor(Tab::Manage), Some(UnsavedSource::Manage));
assert_eq!(tab_editor(Tab::Settings), Some(UnsavedSource::Settings));
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
assert_eq!(tab_editor(tab), None, "{tab:?} saves as it goes");
}
}
/// The two values the Settings tab's color-scheme box writes, plus
/// hand-edited variants.
#[test]
fn only_light_is_light() {
assert_eq!(theme_for("light"), egui::Theme::Light);

239
crates/quicksearch-gui/src/app/verify.rs Normal file
View file

@ -0,0 +1,239 @@
//! Byte-for-byte verification of one duplicate group, and the modal that
//! reports it.
//!
//! The Duplicates tab groups by a hash of each file's size and head, which is
//! all the indexer ever reads (see [`quicksearch_core::verify`]). This is the
//! second opinion, asked for one group at a time, and it exists because the
//! action it precedes is usually deletion.
use super::*;
use std::path::PathBuf;
use quicksearch_core::verify::{MemberVerdict, VerifyReport, VerifyUpdate};
use crate::format::{group_thousands, human_size};
use crate::ui_util::{centered_modal, hint, progress_bar};
const MODAL_WIDTH: f32 = 560.0;
pub(crate) enum VerifyState {
Running {
bytes_read: u64,
/// Zero until the worker's first progress update lands, which is what
/// puts the bar in its indeterminate state to begin with.
bytes_total: u64,
},
Done(Box<VerifyReport>),
Cancelled,
}
pub(crate) struct VerifyModal {
pub paths: Vec<PathBuf>,
pub state: VerifyState,
}
impl VerifyModal {
pub(crate) fn new(paths: Vec<PathBuf>) -> VerifyModal {
VerifyModal {
paths,
state: VerifyState::Running {
bytes_read: 0,
bytes_total: 0,
},
}
}
}
/// One line of the report: what happened to `path`, in the words the modal
/// paints. Split out from the rendering so the wording is testable without a
/// frame.
pub(crate) fn verdict_line(verdict: &MemberVerdict, reference: bool) -> String {
match verdict {
MemberVerdict::Identical if reference => "compared against".to_string(),
MemberVerdict::Identical => "identical".to_string(),
MemberVerdict::DiffersAt(offset) => {
format!("differs at byte {}", group_thousands(*offset))
}
MemberVerdict::LengthDiffers { len, reference_len } => format!(
"size differs: {} against {}",
human_size(*len),
human_size(*reference_len)
),
MemberVerdict::Unreadable(e) => format!("could not be read — {e}"),
}
}
/// The headline the report earns.
pub(crate) fn summary_line(report: &VerifyReport) -> String {
let total = report.verdicts.len();
if report.reference.is_none() {
return "None of these files could be read.".to_string();
}
let differing = report.differing();
if differing == 0 {
return match total {
0 | 1 => "Nothing to compare: the group holds one file.".to_string(),
n => format!("All {n} files are byte-for-byte identical."),
};
}
format!(
"{} of {} files {} not identical.",
differing,
total,
if differing == 1 { "is" } else { "are" }
)
}
impl QuickSearchApp {
/// Drain the worker and fold its updates into the modal.
pub(super) fn drain_verify(&mut self) {
use std::sync::mpsc::TryRecvError;
let Some(job) = &self.backend.verify_job else {
return;
};
let mut finished = false;
loop {
match job.rx.try_recv() {
Ok(VerifyUpdate::Progress {
bytes_read: read,
bytes_total: total,
}) => {
if let Some(modal) = &mut self.verify {
modal.state = VerifyState::Running {
bytes_read: read,
bytes_total: total,
};
}
}
Ok(VerifyUpdate::Done(report)) => {
if let Some(modal) = &mut self.verify {
modal.state = VerifyState::Done(Box::new(report));
}
finished = true;
break;
}
Ok(VerifyUpdate::Cancelled) => {
if let Some(modal) = &mut self.verify {
modal.state = VerifyState::Cancelled;
}
finished = true;
break;
}
Err(TryRecvError::Empty) => break,
// The worker died without a terminal update. Nothing else can
// arrive, so say so rather than spinning on an empty channel.
Err(TryRecvError::Disconnected) => {
if let Some(modal) = &mut self.verify {
if matches!(modal.state, VerifyState::Running { .. }) {
modal.state = VerifyState::Cancelled;
}
}
finished = true;
break;
}
}
}
if finished {
self.backend.verify_job = None;
}
}
pub(super) fn verify_modal_ui(&mut self, ctx: &egui::Context) {
let Some(modal) = &self.verify else {
return;
};
if verify_modal(ctx, modal) {
self.backend.cancel_verify();
self.verify = None;
}
}
}
/// Paint the modal; `true` when its dismiss button was clicked. A free
/// function rather than a method so it can be rendered against a bare
/// context, without an app and the index behind it.
pub(crate) fn verify_modal(ctx: &egui::Context, modal: &VerifyModal) -> bool {
centered_modal(ctx, "Verify duplicates", |ui| {
ui.set_max_width(MODAL_WIDTH);
match &modal.state {
VerifyState::Running {
bytes_read,
bytes_total,
} => {
ui.label(format!(
"Comparing {} files byte for byte…",
modal.paths.len()
));
// No denominator until the first update lands, which is what
// the indeterminate bar is for.
let fraction =
(*bytes_total > 0).then(|| (*bytes_read as f64 / *bytes_total as f64) as f32);
progress_bar(ui, fraction, MODAL_WIDTH);
ui.label(hint(match bytes_total {
0 => format!("{} read", human_size(*bytes_read)),
total => format!("{} of {}", human_size(*bytes_read), human_size(*total)),
}));
ui.add_space(6.0);
ui.horizontal(|ui| ui.button("Cancel").clicked()).inner
}
VerifyState::Cancelled => {
ui.label("Verification cancelled.");
ui.add_space(6.0);
ui.horizontal(|ui| ui.button("Close").clicked()).inner
}
VerifyState::Done(report) => {
let p = crate::color::palette(ui.visuals().dark_mode);
let identical = report.all_identical() && report.reference.is_some();
let color = if identical {
p.green
} else {
ui.visuals().error_fg_color
};
ui.colored_label(color, summary_line(report));
if !identical {
ui.label(hint(
"Files are grouped by size and how they begin, which is all \
indexing reads. This compared every byte.",
));
}
ui.add_space(6.0);
// Listed even when everything matched: it is the record of
// what was actually read.
egui::ScrollArea::vertical()
.max_height(260.0)
.auto_shrink([false, true])
.show(ui, |ui| {
for (i, path) in modal.paths.iter().enumerate() {
let Some(verdict) = report.verdicts.get(i) else {
continue;
};
let is_reference = report.reference == Some(i);
ui.horizontal_wrapped(|ui| {
ui.label(
egui::RichText::new(path.display().to_string()).monospace(),
);
let line = verdict_line(verdict, is_reference);
if verdict.is_identical() {
ui.label(hint(line));
} else {
ui.colored_label(
ui.visuals().error_fg_color,
egui::RichText::new(line).small(),
);
}
});
}
});
ui.add_space(6.0);
ui.label(hint(format!("{} read", human_size(report.bytes_read))));
ui.horizontal(|ui| ui.button("Close").clicked()).inner
}
}
})
.unwrap_or(false)
}
#[cfg(test)]
#[path = "verify_tests.rs"]
mod tests;

216
crates/quicksearch-gui/src/app/verify_tests.rs Normal file
View file

@ -0,0 +1,216 @@
use super::*;
use quicksearch_core::verify::MemberVerdict::{
DiffersAt, Identical, LengthDiffers, Unreadable as CannotRead,
};
use crate::test_ui::{click_at, painted_text, painted_text_center, raw_input};
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
fn modal(state: VerifyState, n: usize) -> VerifyModal {
VerifyModal {
paths: (0..n)
.map(|i| PathBuf::from(format!("/d/copy{i}.bin")))
.collect(),
state,
}
}
fn report(verdicts: Vec<MemberVerdict>, bytes_read: u64) -> VerifyState {
VerifyState::Done(Box::new(VerifyReport {
reference: Some(0),
verdicts,
bytes_read,
}))
}
/// Two passes: an `egui::Window` is measured on its first frame and placed on
/// the next, so a single pass paints nothing to read back.
fn frame(
ctx: &egui::Context,
m: &VerifyModal,
events: Vec<egui::Event>,
) -> (egui::FullOutput, bool) {
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
verify_modal(ctx, m);
});
let mut closed = false;
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
closed = verify_modal(ctx, m);
});
(out, closed)
}
#[test]
fn a_run_in_progress_says_what_it_is_doing_and_offers_a_way_out() {
let ctx = egui::Context::default();
let m = modal(
VerifyState::Running {
bytes_read: 5 * 1024 * 1024,
bytes_total: 20 * 1024 * 1024,
},
3,
);
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
assert!(
painted.contains(&"Comparing 3 files byte for byte…".to_string()),
"{painted:?}"
);
assert!(
painted.iter().any(|t| t.contains("5.2 MB of 21.0 MB")),
"no byte counter: {painted:?}"
);
assert!(painted.contains(&"Cancel".to_string()), "{painted:?}");
}
/// Before the worker's first update there is no denominator, so the modal
/// reports what it has rather than dividing by zero.
#[test]
fn a_run_with_no_denominator_yet_still_reports() {
let ctx = egui::Context::default();
let m = modal(
VerifyState::Running {
bytes_read: 0,
bytes_total: 0,
},
2,
);
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
assert!(
painted.iter().any(|t| t.contains("0 B read")),
"{painted:?}"
);
}
#[test]
fn a_clean_result_says_so_and_lists_what_was_read() {
let ctx = egui::Context::default();
let m = modal(report(vec![Identical, Identical, Identical], 300), 3);
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
assert!(
painted.contains(&"All 3 files are byte-for-byte identical.".to_string()),
"{painted:?}"
);
assert!(painted.contains(&"/d/copy2.bin".to_string()), "{painted:?}");
assert!(
painted.contains(&"compared against".to_string()),
"{painted:?}"
);
assert!(painted.contains(&"Close".to_string()), "{painted:?}");
}
/// The case the feature exists for: same size, same head, different bytes.
#[test]
fn a_mismatch_names_the_file_and_the_offset() {
let ctx = egui::Context::default();
let m = modal(report(vec![Identical, DiffersAt(1_234_567)], 2), 2);
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
assert!(
painted.contains(&"1 of 2 files is not identical.".to_string()),
"{painted:?}"
);
assert!(
painted.contains(&"differs at byte 1,234,567".to_string()),
"{painted:?}"
);
assert!(painted.contains(&"/d/copy1.bin".to_string()), "{painted:?}");
}
#[test]
fn a_cancelled_run_says_so_rather_than_showing_a_verdict() {
let ctx = egui::Context::default();
let m = modal(VerifyState::Cancelled, 2);
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
assert!(
painted.contains(&"Verification cancelled.".to_string()),
"{painted:?}"
);
assert!(
!painted.iter().any(|t| t.contains("identical")),
"a cancelled run claimed a verdict: {painted:?}"
);
}
/// Every state's dismiss button reports the dismissal, whatever it is called.
#[test]
fn both_dismiss_buttons_report_the_dismissal() {
for (label, state) in [
(
"Cancel",
VerifyState::Running {
bytes_read: 1,
bytes_total: 2,
},
),
("Close", VerifyState::Cancelled),
("Close", report(vec![Identical, Identical], 8)),
] {
let ctx = egui::Context::default();
let m = modal(state, 2);
let (out, _) = frame(&ctx, &m, Vec::new());
let pos =
painted_text_center(&out, label).unwrap_or_else(|| panic!("no {label} button painted"));
let (_, closed) = frame(&ctx, &m, click_at(pos));
assert!(closed, "clicking {label} did not dismiss the modal");
}
}
#[test]
fn every_verdict_reads_as_a_sentence_about_the_file() {
assert_eq!(verdict_line(&Identical, false), "identical");
assert_eq!(verdict_line(&Identical, true), "compared against");
assert_eq!(verdict_line(&DiffersAt(0), false), "differs at byte 0");
assert_eq!(
verdict_line(&DiffersAt(1_048_576), false),
"differs at byte 1,048,576"
);
assert_eq!(
verdict_line(
&LengthDiffers {
len: 2048,
reference_len: 1024
},
false
),
"size differs: 2.0 KB against 1.0 KB"
);
assert!(verdict_line(&CannotRead("/d/x: denied".into()), false)
.contains("could not be read — /d/x: denied"));
}
#[test]
fn the_summary_counts_what_it_found() {
let of = |verdicts: Vec<MemberVerdict>, reference| {
summary_line(&VerifyReport {
reference,
verdicts,
bytes_read: 0,
})
};
assert_eq!(
of(vec![Identical, Identical], Some(0)),
"All 2 files are byte-for-byte identical."
);
assert_eq!(
of(vec![Identical, DiffersAt(4)], Some(0)),
"1 of 2 files is not identical."
);
assert_eq!(
of(vec![Identical, DiffersAt(4), DiffersAt(9)], Some(0)),
"2 of 3 files are not identical."
);
// A group of one cannot disagree with itself, and saying "all 1 files are
// identical" would read as an answer to a question nobody asked.
assert_eq!(
of(vec![Identical], Some(0)),
"Nothing to compare: the group holds one file."
);
assert_eq!(
of(
vec![CannotRead("gone".into()), CannotRead("gone".into())],
None
),
"None of these files could be read."
);
}

101
crates/quicksearch-gui/src/backend.rs
View file

@ -6,22 +6,48 @@
//! Every core thread wakes the UI through `ctx.request_repaint()`, which is
//! what makes polling enough.
//!
//! The duplicates scan is the only throwaway thread, and it fires on a user
//! action, not a timer: a thread per refresh opens its own connection — a
//! page cache and an allocator arena glibc never gives back.
//! The duplicates scan and the byte-for-byte verification of one of its
//! groups are the throwaway threads, and both fire on a user action rather
//! than a timer: a thread per refresh opens its own connection — a page cache
//! and an allocator arena glibc never gives back. The verification opens no
//! connection at all, but it can hold a large group's worth of file handles,
//! so it carries a cancel flag and shutdown raises it.
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc};
use quicksearch_core::config::Config;
use quicksearch_core::coordinator::IndexCoordinator;
use quicksearch_core::live::{LiveUpdate, LiveWatcher};
use quicksearch_core::search::{DuplicateGroup, SearchService, SearchUpdate};
use quicksearch_core::shutdown;
use quicksearch_core::verify::{verify_identical, VerifyUpdate};
/// A duplicate group being read through. The thread is detached and owns
/// nothing the app needs back, so cancelling is just raising the flag: the
/// worker notices between chunks and drops the receiver's other end.
pub struct VerifyJob {
pub rx: mpsc::Receiver<VerifyUpdate>,
cancel: Arc<AtomicBool>,
}
impl VerifyJob {
pub fn cancel(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
}
pub struct Backend {
pub coordinator: Arc<IndexCoordinator>,
pub search: Option<SearchService>,
pub search_rx: mpsc::Receiver<SearchUpdate>,
pub dup_job: Option<mpsc::Receiver<Result<Vec<DuplicateGroup>, String>>>,
pub verify_job: Option<VerifyJob>,
/// Watches the results currently on screen; see [`quicksearch_core::live`].
/// `None` only after [`Backend::shutdown`].
pub live: Option<LiveWatcher>,
pub live_rx: mpsc::Receiver<LiveUpdate>,
}
impl Backend {
@ -44,14 +70,49 @@ impl Backend {
Arc::new(move || repaint_ctx.request_repaint()),
);
let live_ctx = ctx.clone();
let (live, live_rx) = LiveWatcher::start(Arc::new(move || live_ctx.request_repaint()));
Ok(Backend {
coordinator,
search: Some(search),
search_rx,
dup_job: None,
verify_job: None,
live: Some(live),
live_rx,
})
}
/// Point the live watcher at the rows currently on screen, or clear it
/// with an empty `targets`.
pub fn watch_live(
&self,
query: &str,
targets: Vec<quicksearch_core::live::Target>,
config: &Config,
) {
let Some(live) = &self.live else { return };
if targets.is_empty() {
live.clear();
} else {
live.watch(query, targets, config);
}
}
/// Ask the coordinator to bring the index in line with these paths — the
/// files the live watcher has just read from disk on the frontend's
/// behalf, so the index does not drift from what is on screen.
pub fn reindex_live_paths(&self, paths: Vec<PathBuf>) {
self.coordinator.update_paths(paths);
}
pub fn clear_live(&self) {
if let Some(live) = &self.live {
live.clear();
}
}
/// `None` only after [`Backend::shutdown`], i.e. during teardown frames.
pub fn search(&self) -> Option<&SearchService> {
self.search.as_ref()
@ -70,12 +131,46 @@ impl Backend {
self.dup_job = Some(rx);
}
/// Read a duplicate group through on a worker thread, comparing every
/// member against the first byte for byte. Replaces any run already going.
pub fn start_verify(&mut self, paths: Vec<PathBuf>, ctx: egui::Context) {
if let Some(job) = &self.verify_job {
job.cancel();
}
let (tx, rx) = mpsc::channel();
let cancel = Arc::new(AtomicBool::new(false));
let worker_cancel = cancel.clone();
std::thread::spawn(move || {
verify_identical(&paths, &worker_cancel, &mut |update| {
// A closed receiver means the app moved on; the cancel flag
// is what stops the work, so there is nothing to do here.
let _ = tx.send(update);
ctx.request_repaint();
});
});
self.verify_job = Some(VerifyJob { rx, cancel });
}
/// Stop a verification and forget it. The worker sees the flag between
/// chunks and exits on its own.
pub fn cancel_verify(&mut self) {
if let Some(job) = self.verify_job.take() {
job.cancel();
}
}
/// Join the search worker and stop the coordinator. Called once from
/// `on_exit`.
pub fn shutdown(&mut self) {
// Detached and holding open file handles: the flag is what makes a
// verification of a slow, large group let go on the way out.
self.cancel_verify();
if let Some(search) = self.search.take() {
search.shutdown();
}
if let Some(mut live) = self.live.take() {
live.stop();
}
self.coordinator.shutdown();
}
}

2
crates/quicksearch-gui/src/capture.rs
View file

@ -82,7 +82,7 @@ pub(crate) struct CaptureDriver {
/// Screenshot in flight: requested, PNG not yet written.
shot: Option<PathBuf>,
rec: Option<Recorder>,
/// Match-cell row the pointer is pinned to (`hover_match`), and the
/// Content Match cell row the pointer is pinned to (`hover_match`), and the
/// on-screen position it resolved to on the last rendered frame.
hover: Option<usize>,
hover_pos: Option<egui::Pos2>,

14
crates/quicksearch-gui/src/capture/script.rs
View file

@ -11,9 +11,11 @@ use crate::app::Tab;
/// window INT INT # resize to width x height, in the same
/// # logical points as the startup size
/// hover_match INT # pin the pointer over the Nth visible
/// # Match cell (0-based) until hover_off
/// # Content Match cell (0-based) until
/// # hover_off. Counts every visible row,
/// # including those showing a dash.
/// hover_off # release the pinned pointer
/// tab (search|manage|duplicates|logs|help)
/// tab (search|manage|duplicates|logs|help|settings)
/// wait_index_running [max INT] # caps in ms; a capped wait cannot fail
/// wait_index_idle [max INT]
/// wait_search_done [max INT]
@ -187,9 +189,11 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
"duplicates" => Tab::Duplicates,
"logs" => Tab::Logs,
"help" => Tab::Help,
"settings" => Tab::Settings,
other => {
return Err(err(format!(
"unknown tab {other:?}: expected search, manage, duplicates, logs or help"
"unknown tab {other:?}: expected search, manage, duplicates, \
logs, help or settings"
)));
}
}),
@ -302,6 +306,7 @@ mod tests {
tab duplicates
tab logs
tab help
tab settings
wait_index_running
wait_index_running max 15000
wait_index_idle max 13000
@ -335,6 +340,7 @@ mod tests {
Cmd::Tab(Tab::Duplicates),
Cmd::Tab(Tab::Logs),
Cmd::Tab(Tab::Help),
Cmd::Tab(Tab::Settings),
Cmd::WaitIndexRunning { max_ms: None },
Cmd::WaitIndexRunning {
max_ms: Some(15000)
@ -447,7 +453,7 @@ mod tests {
#[test]
fn unknown_tab_and_bad_cps_are_rejected() {
assert!(parse_err("tab settings").msg.contains("unknown tab"));
assert!(parse_err("tab preferences").msg.contains("unknown tab"));
assert!(parse_err(r#"type "x" cps 0"#).msg.contains("positive"));
assert!(parse_err(r#"type "x" cps -3"#).msg.contains("positive"));
}

100
crates/quicksearch-gui/src/duplicates_tab.rs
View file

@ -21,8 +21,18 @@ pub struct DuplicatesTab {
#[derive(Default)]
pub struct DuplicatesActions {
pub refresh: bool,
/// Every member of one group, to be read through and compared byte for
/// byte. Group-scoped whichever row it was asked for from: the question
/// "is this row really a duplicate" is a question about the group.
pub verify: Option<Vec<String>>,
}
/// The entry both context menus carry. Named for what it settles, since the
/// grouping itself never claimed more than a shared size and head.
const VERIFY_LABEL: &str = "Verify copies are identical…";
const VERIFY_TIP: &str = "Reads every file in the group in full and compares them byte for \
byte. Grouping only reads each file's size and how it begins.";
impl DuplicatesTab {
pub fn new() -> DuplicatesTab {
DuplicatesTab {
@ -30,7 +40,10 @@ impl DuplicatesTab {
}
}
pub fn ui(&mut self, ui: &mut egui::Ui) -> DuplicatesActions {
/// `verify_open` is the verification window being up — running or showing
/// a result. There is one of it, so the entry greys out rather than
/// replacing what someone is reading.
pub fn ui(&mut self, ui: &mut egui::Ui, verify_open: bool) -> DuplicatesActions {
let mut actions = DuplicatesActions::default();
ui.horizontal(|ui| {
@ -84,34 +97,48 @@ impl DuplicatesTab {
human_size(group.redundant_size.max(0) as u64),
human_size(group.total_size.max(0) as u64),
);
egui::CollapsingHeader::new(title)
.id_salt(i)
.show(ui, |ui| {
for (_, _, path, size, _) in &group.members {
ui.horizontal(|ui| {
ui.label(human_size(*size));
let response = ui.add(
egui::Label::new(
egui::RichText::new(path).monospace(),
)
.sense(egui::Sense::click()),
);
if response.double_clicked() {
platform::open_file(path);
}
response.context_menu(|ui| {
if ui.button("Open").clicked() {
let header =
egui::CollapsingHeader::new(title)
.id_salt(i)
.show(ui, |ui| {
for (_, _, path, size, _) in &group.members {
ui.horizontal(|ui| {
ui.label(human_size(*size));
let response = ui.add(
egui::Label::new(
egui::RichText::new(path).monospace(),
)
.sense(egui::Sense::click()),
);
if response.double_clicked() {
platform::open_file(path);
ui.close();
}
if ui.button("Open containing folder").clicked() {
platform::reveal_in_folder(path);
ui.close();
}
response.context_menu(|ui| {
if ui.button("Open File").clicked() {
platform::open_file(path);
ui.close();
}
if ui.button("Open containing folder").clicked()
{
platform::reveal_in_folder(path);
ui.close();
}
ui.separator();
if verify_entry(ui, verify_open) {
actions.verify = Some(member_paths(group));
}
});
});
});
}
});
}
});
// Also on the group's own row: the question is
// about the group, and the rows it is about are
// behind a collapsed header until they are not.
header.header_response.context_menu(|ui| {
if verify_entry(ui, verify_open) {
actions.verify = Some(member_paths(group));
}
});
}
});
crate::ui_util::more_below_hint(ui, &scroll);
@ -120,3 +147,24 @@ impl DuplicatesTab {
actions
}
}
/// The shared menu entry. Returns whether it was clicked, and closes the menu
/// when it was.
fn verify_entry(ui: &mut egui::Ui, open: bool) -> bool {
let clicked = ui
.add_enabled(!open, egui::Button::new(VERIFY_LABEL))
.on_hover_text(VERIFY_TIP)
.on_disabled_hover_text("Close the verification window first.")
.clicked();
if clicked {
ui.close();
}
clicked
}
fn member_paths(group: &DuplicateGroup) -> Vec<String> {
group.members.iter().map(|m| m.2.clone()).collect()
}
#[cfg(test)]
mod tests;

186
crates/quicksearch-gui/src/duplicates_tab/tests.rs Normal file
View file

@ -0,0 +1,186 @@
use super::*;
use crate::test_ui::{painted_text, painted_text_center, raw_input};
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
fn group(paths: &[&str]) -> DuplicateGroup {
DuplicateGroup {
hash: vec![0xab; 32],
count: paths.len() as i64,
total_size: 100 * paths.len() as i64,
redundant_size: 100 * (paths.len() as i64 - 1),
members: paths
.iter()
.enumerate()
.map(|(i, p)| {
let name = p.rsplit('/').next().unwrap_or(p).to_string();
(i as i64, name, p.to_string(), 100u64, 1_700_000_000i64)
})
.collect(),
}
}
fn loaded(paths: &[&str]) -> DuplicatesTab {
DuplicatesTab {
state: DupState::Loaded(vec![group(paths)]),
}
}
fn frame(
ctx: &egui::Context,
tab: &mut DuplicatesTab,
busy: bool,
events: Vec<egui::Event>,
) -> (egui::FullOutput, DuplicatesActions) {
let mut actions = DuplicatesActions::default();
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
actions = tab.ui(ui, busy);
});
});
(out, actions)
}
fn click(pos: egui::Pos2, button: egui::PointerButton) -> Vec<egui::Event> {
let mut events = vec![egui::Event::PointerMoved(pos)];
events.extend(
[true, false]
.into_iter()
.map(|pressed| egui::Event::PointerButton {
pos,
button,
pressed,
modifiers: egui::Modifiers::default(),
}),
);
events
}
/// Right-click `needle` and return what the menu it opened painted, plus the
/// actions from that frame.
fn context_menu_on(
ctx: &egui::Context,
tab: &mut DuplicatesTab,
busy: bool,
needle: &str,
) -> (Vec<String>, egui::Pos2) {
let (out, _) = frame(ctx, tab, busy, Vec::new());
let target = painted_text_center(&out, needle)
.unwrap_or_else(|| panic!("nothing painted for {needle:?}"));
frame(
ctx,
tab,
busy,
click(target, egui::PointerButton::Secondary),
);
// The menu is its own area, painted on the frame after the click.
let (out, _) = frame(ctx, tab, busy, Vec::new());
(painted_text(&out), target)
}
/// The title line carries the group; find it without rebuilding its wording.
fn header_of(ctx: &egui::Context, tab: &mut DuplicatesTab) -> String {
let (out, _) = frame(ctx, tab, false, Vec::new());
painted_text(&out)
.into_iter()
.find(|t| t.contains("reclaimable"))
.expect("no group header painted")
}
const PATHS: [&str; 3] = ["/a/img.raw", "/b/img.raw", "/c/img.raw"];
#[test]
fn a_group_header_offers_the_verification() {
let ctx = egui::Context::default();
let mut tab = loaded(&PATHS);
let header = header_of(&ctx, &mut tab);
let (menu, _) = context_menu_on(&ctx, &mut tab, false, &header);
assert!(
menu.contains(&VERIFY_LABEL.to_string()),
"the group's own row does not offer it: {menu:?}"
);
}
/// Clicking it asks for the whole group, not the one row it was asked from.
#[test]
fn verifying_asks_for_every_member_of_the_group() {
let ctx = egui::Context::default();
let mut tab = loaded(&PATHS);
let header = header_of(&ctx, &mut tab);
context_menu_on(&ctx, &mut tab, false, &header);
let (out, _) = frame(&ctx, &mut tab, false, Vec::new());
let entry = painted_text_center(&out, VERIFY_LABEL).expect("no verify entry painted");
let (_, actions) = frame(
&ctx,
&mut tab,
false,
click(entry, egui::PointerButton::Primary),
);
assert_eq!(
actions.verify,
Some(PATHS.iter().map(|p| p.to_string()).collect::<Vec<_>>())
);
}
/// There is one verification window, so a second run is refused where it is
/// asked for rather than replacing what someone is reading.
#[test]
fn a_second_verification_is_refused_while_the_window_is_open() {
let ctx = egui::Context::default();
let mut tab = loaded(&PATHS);
let header = header_of(&ctx, &mut tab);
context_menu_on(&ctx, &mut tab, true, &header);
let (out, _) = frame(&ctx, &mut tab, true, Vec::new());
let entry = painted_text_center(&out, VERIFY_LABEL).expect("the entry should still be listed");
let (_, actions) = frame(
&ctx,
&mut tab,
true,
click(entry, egui::PointerButton::Primary),
);
assert_eq!(actions.verify, None, "a disabled entry still fired");
}
/// Expanding a group and right-clicking one of its files offers the same
/// thing: the rows are what someone is looking at when the question occurs.
#[test]
fn a_member_row_offers_the_verification_too() {
let ctx = egui::Context::default();
let mut tab = loaded(&PATHS);
let header = header_of(&ctx, &mut tab);
let (out, _) = frame(&ctx, &mut tab, false, Vec::new());
let pos = painted_text_center(&out, &header).expect("no header painted");
frame(
&ctx,
&mut tab,
false,
click(pos, egui::PointerButton::Primary),
);
let (menu, _) = context_menu_on(&ctx, &mut tab, false, PATHS[1]);
assert!(
menu.contains(&VERIFY_LABEL.to_string()),
"an expanded member row does not offer it: {menu:?}"
);
assert!(
menu.contains(&"Open File".to_string()),
"the existing entries went missing: {menu:?}"
);
}
#[test]
fn an_empty_result_says_so_rather_than_showing_an_empty_list() {
let ctx = egui::Context::default();
let mut tab = DuplicatesTab {
state: DupState::Loaded(Vec::new()),
};
let painted = painted_text(&frame(&ctx, &mut tab, false, Vec::new()).0);
assert!(
painted.contains(&"No duplicate files found.".to_string()),
"{painted:?}"
);
}

30
crates/quicksearch-gui/src/help_tab.rs
View file

@ -1,7 +1,9 @@
//! The Help tab: a quickstart guide for first-time users. The complete
//! technical reference stays in README.md.
pub fn ui(ui: &mut egui::Ui) {
/// Returns true when the "Show the introduction again" button was clicked.
pub fn ui(ui: &mut egui::Ui) -> bool {
let mut replay = false;
let scroll = egui::ScrollArea::vertical()
.auto_shrink([false; 2])
.show(ui, |ui| {
@ -16,6 +18,11 @@ pub fn ui(ui: &mut egui::Ui) {
files by name and by what is inside them, as you type.",
);
ui.add_space(6.0);
if ui.button("Show the introduction again").clicked() {
replay = true;
}
ui.add_space(12.0);
ui.heading(egui::RichText::new("Getting started").strong());
ui.add_space(4.0);
@ -44,23 +51,26 @@ pub fn ui(ui: &mut egui::Ui) {
words:",
);
ui.monospace("type:Document modified:>=2024-01-01 report");
ui.label("The ? button next to the search box shows the full query syntax.");
ui.label("The ? button left of the search box shows the full query syntax.");
ui.add_space(6.0);
ui.label(
"• Tick Fuzzy to also find matches with typos in them, at some \
cost in speed.",
);
ui.label(
"• Click a column header — Name, Path, Size, Modified, Rank — to \
sort the results; click it again to reverse the order.",
"• Click a column header to sort the results; click it again to \
reverse the order. Right-click any header to choose which \
columns are shown — size and modified date start hidden.",
);
ui.label(
"• Right-click a result to open it, open its containing folder, \
or hide files like it from the results.",
);
ui.label(
"• Matches inside a file's contents show a snippet of the \
surrounding text under the file name.",
"• A match in a file's name or path is highlighted in that \
column; a match in its contents shows a snippet of the \
surrounding text in the Content Match column, with the rest on \
hover.",
);
ui.add_space(12.0);
@ -92,7 +102,12 @@ pub fn ui(ui: &mut egui::Ui) {
"warnings from indexing and folder watching that a \
terminal would have shown",
);
row(ui, "⚙ (top right)", "application options");
row(
ui,
"Settings",
"everything QuickSearch can be told to do, in one \
place; hover any control for what it means",
);
});
ui.add_space(12.0);
@ -131,6 +146,7 @@ pub fn ui(ui: &mut egui::Ui) {
});
});
crate::ui_util::more_below_hint(ui, &scroll);
replay
}
/// Where this build left the README: under the install prefix's `share/doc`

16
crates/quicksearch-gui/src/hotkey/binding.rs
View file

@ -2,10 +2,10 @@
//! produce.
//!
//! A shortcut is written in three different vocabularies before it reaches an
//! operating system: the text in `config.toml` and on the Options button, the
//! token `global-hotkey` parses for `RegisterHotKey`/`XGrabKey`, and the
//! xkbcommon keysym name the XDG *shortcuts* specification wants for the
//! Wayland portal. All three come out of [`KEYS`], so a key cannot be
//! operating system: the text in `config.toml` and on the Settings tab's
//! button, the token `global-hotkey` parses for `RegisterHotKey`/`XGrabKey`,
//! and the xkbcommon keysym name the XDG *shortcuts* specification wants for
//! the Wayland portal. All three come out of [`KEYS`], so a key cannot be
//! spelled correctly for one backend and wrongly for the other.
//!
//! The config text and the `global-hotkey` token are the same string: every
@ -103,7 +103,7 @@ const KEYS: &[(Key, &str, &str)] = &[
(Key::CloseBracket, "BracketRight", "bracketright"),
];
/// Escape is reserved: it cancels the Options window's capture, and a
/// Escape is reserved: it cancels the Settings tab's capture, and a
/// system-wide Escape would be unusable anyway.
const RESERVED: &[Key] = &[Key::Escape];
@ -111,7 +111,7 @@ const RESERVED: &[Key] = &[Key::Escape];
///
/// Super/Meta is absent because `egui::Modifiers` has no field for it — egui
/// reports alt, ctrl, shift and the Mac command key only — so a Super combo
/// could never be captured in the Options window even if a backend could
/// could never be captured in the Settings tab even if a backend could
/// register it.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Binding {
@ -122,7 +122,7 @@ pub struct Binding {
}
/// Why a string or a key press is not a usable shortcut; the wording is
/// shown in the Options window.
/// shown in the Settings tab.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum BindingError {
Empty,
@ -149,7 +149,7 @@ impl fmt::Display for BindingError {
}
impl Binding {
/// Build from a key press egui reported, for the Options window's capture
/// Build from a key press egui reported, for the Settings tab's capture
/// widget. `None` for a press that cannot be a shortcut: a key with no
/// row in [`KEYS`], a reserved key, or a bare key with no modifier held.
///

4
crates/quicksearch-gui/src/hotkey/mod.rs
View file

@ -44,7 +44,7 @@ thread_local! {
static REGISTRY: RefCell<Option<Registry>> = const { RefCell::new(None) };
}
/// What the Options window says about the shortcut.
/// What the Settings tab says about the shortcut.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Status {
/// The setting is empty: no shortcut, by choice.
@ -256,7 +256,7 @@ mod tests {
use super::*;
/// Nothing may touch an OS registration before `init`, so that the
/// headless UI tests can render the Options row.
/// headless UI tests can render the Settings row.
#[test]
fn an_uninitialised_registry_is_inert() {
apply("Ctrl+Shift+F");

6
crates/quicksearch-gui/src/hotkey/portal.rs
View file

@ -7,7 +7,7 @@
//! `preferred_trigger`, and the compositor is free to bind something else, to
//! ask the user first, or to let them change it later in its own settings.
//! What it actually bound comes back as a human-readable
//! `trigger_description`, which is what the Options window shows.
//! `trigger_description`, which is what the Settings tab shows.
//!
//! All of this lives on its own thread. The portal is D-Bus, so every call
//! is a round trip that could block for as long as a dialog stays on screen,
@ -56,7 +56,7 @@ impl Portal {
.spawn(move || pollster::block_on(run(ctx, status, rx)))
{
// Not worth taking the app down for — but the status must say
// so, or Options shows "Waiting for your desktop…" forever.
// so, or the Settings tab shows "Waiting for your desktop…" forever.
quicksearch_core::log_warn!("global shortcut portal thread: {}", e);
*lock_ok(&portal.status) =
Status::Error(format!("the shortcut thread could not be started: {}", e));
@ -194,7 +194,7 @@ fn unavailable(e: &ashpd::Error) -> String {
fn set(ctx: &egui::Context, status: &Mutex<Status>, next: Status) {
*lock_ok(status) = next;
// The Options window may be open and waiting for this.
// The Settings tab may be on screen and waiting for this.
ctx.request_repaint();
}

3
crates/quicksearch-gui/src/main.rs
View file

@ -22,14 +22,15 @@ mod hotkey;
mod keychain;
mod logs_tab;
mod manage_tab;
mod options;
mod platform;
mod query_highlight;
mod search_tab;
mod settings_tab;
#[cfg(test)]
mod test_ui;
mod tips;
mod tracker;
mod tutorial;
mod ui_util;
mod unlock;
mod version;

50
crates/quicksearch-gui/src/manage_tab.rs
View file

@ -434,7 +434,7 @@ impl ManageTab {
ui.label(hint(
"Reindex interval, symlinks, hidden files, tokenizer, and size \
limits are in Options (⚙ in the toolbar).",
limits are on the Settings tab.",
));
ui.add_space(8.0);
@ -542,10 +542,10 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
"Remove indexed folders you do not need, in Indexed folders above.",
"Narrow the full-text extension whitelist, so text is only extracted \
from the file types you actually search.",
"Turn off \"Store text for snippets\" in Options: full-text search keeps \
working, but without previews, occurrence ranking or fuzzy matching \
"Turn off \"Store text for snippets\" on the Settings tab: full-text search \
keeps working, but without previews, occurrence ranking or fuzzy matching \
inside file contents.",
"Lower \"Max text file size\" and \"Max stored text\" in Options.",
"Lower \"Max text file size\" and \"Max stored text\", both on the Settings tab.",
] {
ui.label(format!("• {}", lever));
}
@ -809,20 +809,34 @@ fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
RootPhase::Extracting => {
ui.label(egui::RichText::new("extracting text").color(phase.green));
divider(ui);
let frac = if r.extract_total > 0 {
(r.extracted as f32 / r.extract_total as f32).clamp(0.0, 1.0)
} else {
1.0
};
ui.label(format!(
"{} / {} ({:.0}%) · {}/{} workers",
group_thousands(r.extracted as u64),
group_thousands(r.extract_total as u64),
frac * 100.0,
r.active_workers,
r.total_workers
));
crate::ui_util::progress_bar(ui, Some(frac), 160.0);
let workers = format!("{}/{} workers", r.active_workers, r.total_workers);
match r.extract_total {
Some(total) => {
let frac = if total > 0 {
(r.extracted as f32 / total as f32).clamp(0.0, 1.0)
} else {
1.0
};
ui.label(format!(
"{} / {} ({:.0}%) · {}",
group_thousands(r.extracted as u64),
group_thousands(total as u64),
frac * 100.0,
workers
));
crate::ui_util::progress_bar(ui, Some(frac), 160.0);
}
// The pass is still counting its range — the same shape
// as a walk without a denominator yet.
None => {
ui.label(format!(
"{} files · {}",
group_thousands(r.extracted as u64),
workers
));
crate::ui_util::progress_bar(ui, None, 160.0);
}
}
}
RootPhase::Done => {
// Whole-root totals: `walked` counts every file the walk saw

8
crates/quicksearch-gui/src/manage_tab/tests.rs
View file

@ -64,7 +64,7 @@ fn running_state(roots: &[&str], current_file: Option<&str>) -> IndexerState {
walked: 100,
walk_total: Some(1000),
extracted: 0,
extract_total: 0,
extract_total: None,
current_file: current_file.map(str::to_string),
active_workers: 4,
total_workers: 4,
@ -304,7 +304,7 @@ fn root_progress(phase: RootPhase, walked: usize, walk_total: Option<usize>) ->
walked,
walk_total,
extracted: 0,
extract_total: 0,
extract_total: None,
current_file: None,
active_workers: 4,
total_workers: 4,
@ -683,7 +683,7 @@ fn the_probe_caches_until_the_refresh_interval_is_up() {
let _ = std::fs::remove_dir_all(&dir);
}
/// A database path edited in Options must not keep showing the old
/// A database path edited on the Settings tab must not keep showing the old
/// database's size for the rest of the interval.
#[test]
fn the_probe_follows_a_changed_database_path() {
@ -762,7 +762,7 @@ fn hovering_the_size_explains_how_to_shrink_the_index() {
"Indexed folders",
"whitelist",
"Store text for snippets",
"Options",
"Settings tab",
] {
assert!(text.contains(lever), "tooltip never mentions {}", lever);
}

1182
crates/quicksearch-gui/src/search_tab.rs
View file

File diff suppressed because it is too large Load diff

88
crates/quicksearch-gui/src/search_tab/snippet_render.rs
View file

@ -37,6 +37,11 @@ const SNIPPET_LEAD: &str = "… ";
/// Append `window[range]` to `job`, highlighting whatever parts of `ranges`
/// (byte offsets into `window`) fall inside it.
///
/// Ranges are clipped to the slice, so a caller rendering a string in pieces
/// can hand each piece the *whole* set: a range inside this one survives, one
/// straddling an edge survives as the part that is here, and one wholly
/// outside disappears.
fn append_marked(
job: &mut LayoutJob,
fmt: &SnippetFormats,
@ -61,6 +66,18 @@ fn append_marked(
}
}
/// A whole field — a filename — with its matched spans marked.
///
/// Wrapping is left at the job's defaults on purpose: `egui::Label` overwrites
/// only `wrap.max_width`, so this is laid out exactly like the plain string it
/// replaces, and the cell keeps the height and clipping it had before.
pub(super) fn marked_field_job(ui: &egui::Ui, text: &str, ranges: &[(usize, usize)]) -> LayoutJob {
let fmt = snippet_formats(ui);
let mut job = LayoutJob::default();
append_marked(&mut job, &fmt, text, ranges, 0..text.len());
job
}
/// The byte offset in `snip.window` that rendering has to start at for the
/// first match to land on a row that survives `max_rows`; `0` when it
/// already does. epaint stops at `wrap.max_rows` and *every* `\n` costs a
@ -159,16 +176,14 @@ pub(super) fn snippet_job(ui: &egui::Ui, snip: &Snippet, max_rows: usize) -> Lay
job
}
/// The Match column cell: one line with the (first) matched span centered
/// and an equal amount of context on both sides, trimmed to what fits the
/// column width. Matches on a whole field — a filename or a path — are
/// wrapped in brackets: `[name]`.
pub(super) fn centered_match_job(
ui: &egui::Ui,
snip: &Snippet,
width_px: f32,
whole_field: bool,
) -> LayoutJob {
/// The Content Match column cell: one line with the (first) matched span
/// centered and an equal amount of context on both sides, trimmed to what fits
/// the column width.
///
/// Only ever called with a content snippet. Name and path matches are
/// highlighted in their own columns and leave a dash here, so the bracketed
/// `[whole field]` rendering this used to carry is gone.
pub(super) fn centered_match_job(ui: &egui::Ui, snip: &Snippet, width_px: f32) -> LayoutJob {
let fmt = snippet_formats(ui);
// Newlines force line breaks even in a one-row LayoutJob; flatten them
@ -189,11 +204,6 @@ pub(super) fn centered_match_job(
let font_id = &fmt.normal.font_id;
let width_of = |c: char| f.glyph_width(font_id, c);
let ellipsis = width_of('…');
let brackets = if whole_field {
width_of('[') + width_of(']')
} else {
0.0
};
let mut marks = 0.0;
if snip.truncated_start {
marks += ellipsis;
@ -201,13 +211,13 @@ pub(super) fn centered_match_job(
if snip.truncated_end {
marks += ellipsis;
}
if fits_within(window, width_px - brackets - marks, width_of) {
if fits_within(window, width_px - marks, width_of) {
return (0, window.len(), true);
}
// Something has to go, so either end may gain a mark; reserve for
// both so a cut never overflows the column.
let budget = width_px - brackets - 2.0 * ellipsis;
let budget = width_px - 2.0 * ellipsis;
let Some(&(a, b)) = snip.ranges.first() else {
// No ranges (shouldn't happen for match cells) — head trim.
return (0, take_forward(window, 0, budget.max(0.0), width_of), true);
@ -261,22 +271,52 @@ pub(super) fn centered_match_job(
let mut job = LayoutJob::default();
job.wrap.max_rows = 1;
job.wrap.break_anywhere = true;
if whole_field && decorate {
job.append("[", 0.0, fmt.weak.clone());
}
if decorate && (start > 0 || snip.truncated_start) {
job.append("…", 0.0, fmt.weak.clone());
}
append_marked(&mut job, &fmt, window, &snip.ranges, start..end);
if decorate && (end < window.len() || snip.truncated_end) {
job.append("…", 0.0, fmt.weak.clone());
}
if whole_field && decorate {
job.append("]", 0.0, fmt.weak);
job.append("…", 0.0, fmt.weak);
}
job
}
/// The Path column cell: middle-elided to `width_px`, with whatever of a
/// path-tier match survives the cut highlighted.
///
/// The path reads at full strength — it is the one column that identifies a
/// result on its own. Only the elision mark is weak, since it is punctuation
/// this renderer added rather than anything the file is named.
///
/// Returns the job and whether anything was actually elided — the caller's
/// trigger for a full-path tooltip, since egui offers one only when *it* did
/// the eliding and it is handed an already-shortened string.
pub(super) fn path_cell_job(
ui: &egui::Ui,
path: &str,
ranges: &[(usize, usize)],
width_px: f32,
font_id: &egui::FontId,
) -> (LayoutJob, bool) {
let fmt = snippet_formats(ui);
let mut job = LayoutJob::default();
match crate::ui_util::middle_elide_cut(ui, path, width_px, font_id) {
// It fits: the whole path, marked — which is exactly
// [`marked_field_job`].
None => (marked_field_job(ui, path, ranges), false),
// The two surviving ends are appended straight from `path` at their
// original offsets: `append_marked` clips the ranges to each end, so a
// match that fell in the dropped middle drops with it rather than
// landing on whatever glyphs moved into those offsets.
Some((head, tail)) => {
append_marked(&mut job, &fmt, path, ranges, 0..head);
job.append("…", 0.0, fmt.weak.clone());
append_marked(&mut job, &fmt, path, ranges, tail..path.len());
(job, true)
}
}
}
/// Whether the whole of `text` fits in `budget` pixels; stops at the first
/// character that does not.
fn fits_within(text: &str, budget: f32, width_of: impl Fn(char) -> f32) -> bool {

1611
crates/quicksearch-gui/src/search_tab/tests.rs
View file

File diff suppressed because it is too large Load diff

268
crates/quicksearch-gui/src/options.rs → crates/quicksearch-gui/src/settings_tab.rs
View file

@ -1,11 +1,11 @@
//! The Options window and the shared config editor used by both the
//! window and the Manage Index tab. Edits happen on a draft; Apply
//! validates, saves, and hands the new config to the app.
//! The Settings tab: every configuration control the GUI offers, grouped
//! into sections. Edits happen on a draft; Apply validates, saves, and hands
//! the new config to the app.
use crate::keychain;
use crate::tips::{self, tip_row, Tipped};
use crate::ui_util::hint;
use quicksearch_core::config::Config;
use quicksearch_core::config::{ColumnsConfig, Config};
/// A [`tip_row`] holding one numeric [`egui::DragValue`] — the shape of most
/// rows in the config editor.
@ -22,7 +22,7 @@ fn drag_row<N: egui::emath::Numeric>(
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Section {
enum Section {
Indexing,
Processing,
Search,
@ -36,24 +36,26 @@ pub enum SecurityAction {
Disable,
ChangePassword,
SetKeychain(bool),
ShowKey,
}
/// What one frame of the Options window produced.
/// What one frame of the Settings tab produced.
#[derive(Default)]
pub struct OptionsOutput {
pub struct SettingsOutput {
/// "Apply & Save" was clicked with this draft.
pub applied: Option<Config>,
/// A Security block action was clicked.
pub security: Option<SecurityAction>,
/// The title-bar close was clicked while the draft holds unapplied
/// edits; the window is held open and the app raises the guard.
pub close_requested: bool,
/// The Columns block changed. Like Security, it edits the live config
/// rather than the draft, so it takes effect without Apply.
pub columns: Option<ColumnsConfig>,
}
pub struct OptionsWindow {
pub open: bool,
pub struct SettingsTab {
/// The staged config, built from the live one the first frame the tab is
/// shown and dropped again when it is left.
draft: Option<Config>,
/// Cached answer from [`OptionsWindow::keychain_active`], with the
/// Cached answer from [`SettingsTab::keychain_active`], with the
/// `use_keychain` preference it was probed under.
keychain_probed_for: Option<bool>,
keychain_active: bool,
@ -61,10 +63,9 @@ pub struct OptionsWindow {
capturing_hotkey: bool,
}
impl OptionsWindow {
pub fn new() -> OptionsWindow {
OptionsWindow {
open: false,
impl SettingsTab {
pub fn new() -> SettingsTab {
SettingsTab {
draft: None,
keychain_probed_for: None,
keychain_active: false,
@ -73,21 +74,16 @@ impl OptionsWindow {
}
/// Whether the shortcut button is reading a key press right now, so the
/// app can hold the shortcut it is about to replace. See
/// app can hold the shortcut it is about to replace. The app gates this
/// on the tab being the one on screen. See
/// [`crate::unlock::Gate::handle_hotkey`].
pub fn capturing_hotkey(&self) -> bool {
self.open && self.capturing_hotkey
}
pub fn open_with(&mut self, current: &Config) {
self.open = true;
self.draft = Some(current.clone());
self.keychain_probed_for = None;
self.capturing_hotkey
}
/// Whether the draft differs from the live config. The fields the app
/// pins on apply are neutralized first, so the Security block never
/// makes the window read as dirty.
/// makes the tab read as dirty.
pub fn is_dirty(&self, current: &Config) -> bool {
let Some(draft) = &self.draft else {
return false;
@ -102,34 +98,27 @@ impl OptionsWindow {
self.draft.clone()
}
/// Close and drop the draft (Discard, or a clean close).
pub fn close_discard(&mut self) {
self.open = false;
/// Drop the draft: Discard, or leaving the tab. The next frame that
/// shows the tab stages a fresh copy of the live config, which is what
/// keeps a draft from going stale against edits made on Manage Index.
pub fn discard(&mut self) {
self.draft = None;
self.capturing_hotkey = false;
self.keychain_probed_for = None;
}
/// Adopt the window's open flag for this frame. A dirty close is
/// intercepted: the window is held open and the caller is told to raise
/// the unsaved-changes guard instead.
fn intercept_close(&mut self, still_open: bool, current: &Config) -> bool {
self.open = still_open;
if self.open {
return false;
}
if self.is_dirty(current) {
self.open = true;
true
} else {
self.draft = None;
false
/// Take a draft if there is none: the first frame the tab is shown, and
/// the first frame after it was left.
fn stage(&mut self, current: &Config) {
if self.draft.is_none() {
self.draft = Some(current.clone());
}
}
/// True when this index's key really is in the OS keychain: the
/// preference is on *and* the keychain answers with an entry (a dead
/// daemon, a locked keyring or a denied prompt all read as "no").
/// Probed when the window opens and when the preference changes — a
/// Probed when the tab is entered and when the preference changes — a
/// keychain read is an IPC round trip.
fn keychain_active(&mut self, current: &Config) -> bool {
if self.keychain_probed_for != Some(current.security.use_keychain) {
@ -142,85 +131,79 @@ impl OptionsWindow {
}
/// Render; reports an applied draft config and/or a security action.
pub fn ui(&mut self, ctx: &egui::Context, current: &Config) -> OptionsOutput {
if !self.open {
self.draft = None;
return OptionsOutput::default();
}
if self.draft.is_none() {
self.draft = Some(current.clone());
}
let mut out = OptionsOutput::default();
let mut open = self.open;
pub fn ui(&mut self, ui: &mut egui::Ui, current: &Config) -> SettingsOutput {
self.stage(current);
let mut out = SettingsOutput::default();
let keychain_active = self.keychain_active(current);
let dirty = self.is_dirty(current);
let capturing = &mut self.capturing_hotkey;
let draft = self.draft.as_mut().unwrap();
egui::Window::new("Options")
.open(&mut open)
.resizable(false)
.default_width(420.0)
.show(ctx, |ui| {
let scroll = egui::ScrollArea::vertical()
.max_height(480.0)
.show(ui, |ui| {
ui.heading(egui::RichText::new("Paths").strong());
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
tip_row(ui, "Database file", &tips::DATABASE_PATH, |ui| {
ui.add(
egui::TextEdit::singleline(&mut draft.paths.database_path)
.desired_width(260.0),
)
});
});
ui.label(hint("Indexed folders are managed on the Manage Index tab."));
ui.separator();
let scroll = egui::ScrollArea::vertical()
.auto_shrink([false; 2])
.show(ui, |ui| {
// Cap the column like a document page: a maximized window
// would otherwise stretch every hint into one long line.
ui.set_max_width(620.0);
ui.heading(egui::RichText::new("Indexing").strong());
config_editor_ui(ui, draft, Section::Indexing);
ui.label(hint(
"Automatic and manual indexing are switched on the \
Manage Index tab.",
));
ui.separator();
ui.heading(egui::RichText::new("Processing").strong());
config_editor_ui(ui, draft, Section::Processing);
ui.separator();
ui.heading(egui::RichText::new("Search").strong());
config_editor_ui(ui, draft, Section::Search);
ui.separator();
ui.heading(egui::RichText::new("Interface").strong());
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
tip_row(ui, "UI scale", &tips::UI_SCALE, |ui| {
ui.add(
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
.step_by(0.05)
.fixed_decimals(2),
)
});
tip_row(ui, "Search shortcut", &tips::SEARCH_HOTKEY, |ui| {
hotkey_edit(ui, &mut draft.ui.search_hotkey, capturing)
});
tip_row(ui, "Color scheme", &tips::COLOR_SCHEME, |ui| {
color_scheme_edit(ui, &mut draft.ui.color_scheme)
});
});
hotkey_note(ui, &draft.ui.search_hotkey, &current.ui.search_hotkey);
ui.separator();
// Security acts on the live config, not the draft.
// The KDF salt is never shown here or anywhere else
// in the GUI.
ui.heading(egui::RichText::new("Security").strong());
out.security = security_ui(ui, current, keychain_active);
ui.heading(egui::RichText::new("Paths").strong());
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
tip_row(ui, "Database file", &tips::DATABASE_PATH, |ui| {
ui.add(
egui::TextEdit::singleline(&mut draft.paths.database_path)
.desired_width(260.0),
)
});
crate::ui_util::more_below_hint(ui, &scroll);
});
ui.label(hint("Indexed folders are managed on the Manage Index tab."));
ui.separator();
ui.heading(egui::RichText::new("Indexing").strong());
config_editor_ui(ui, draft, Section::Indexing);
ui.label(hint(
"Automatic and manual indexing are switched on the \
Manage Index tab.",
));
ui.separator();
ui.heading(egui::RichText::new("Processing").strong());
config_editor_ui(ui, draft, Section::Processing);
ui.separator();
ui.heading(egui::RichText::new("Search").strong());
config_editor_ui(ui, draft, Section::Search);
ui.add_space(6.0);
// Live, not drafted — see `columns_ui`.
out.columns = columns_ui(ui, &current.search.columns);
ui.separator();
ui.heading(egui::RichText::new("Interface").strong());
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
tip_row(ui, "UI scale", &tips::UI_SCALE, |ui| {
ui.add(
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
.step_by(0.05)
.fixed_decimals(2),
)
});
tip_row(ui, "Search shortcut", &tips::SEARCH_HOTKEY, |ui| {
hotkey_edit(ui, &mut draft.ui.search_hotkey, capturing)
});
tip_row(ui, "Color scheme", &tips::COLOR_SCHEME, |ui| {
color_scheme_edit(ui, &mut draft.ui.color_scheme)
});
});
hotkey_note(ui, &draft.ui.search_hotkey, &current.ui.search_hotkey);
ui.separator();
// Security acts on the live config, not the draft. The KDF
// salt is never shown here or anywhere else in the GUI.
ui.heading(egui::RichText::new("Security").strong());
out.security = security_ui(ui, current, keychain_active);
ui.separator();
// Last in the scroll, where the Manage Index tab also puts
// it, so the two draft-backed editors read the same way.
let p = crate::color::palette(ui.visuals().dark_mode);
ui.horizontal(|ui| {
let apply = ui
@ -246,12 +229,12 @@ impl OptionsWindow {
});
ui.label(hint(
"Narrowing a filter removes the entries it excludes; widening \
one reindexes to find what it now allows. Only the tokenizer \
and hash length require a full rebuild.",
one reindexes to find what it now allows. Only the tokenizer \
and hash length require a full rebuild.",
));
});
crate::ui_util::more_below_hint(ui, &scroll);
out.close_requested = self.intercept_close(open, current);
out
}
}
@ -398,6 +381,39 @@ fn hotkey_note(ui: &mut egui::Ui, draft: &str, live: &str) {
});
}
/// The Search-tab column picker, mirroring the right-click menu on the table
/// headers. Returns the new set when a checkbox moved.
///
/// Acts on the **live** config, not the draft, for the same reason the
/// Security block does: the header menu writes columns the instant they
/// change, and a draft-backed copy here would silently revert that on the next
/// Apply. `app::pin_live_fields` keeps the draft out of this field entirely.
fn columns_ui(ui: &mut egui::Ui, current: &ColumnsConfig) -> Option<ColumnsConfig> {
let mut next = current.clone();
ui.label("Search columns").on_hover_text(tips::COLUMNS.body);
ui.horizontal_wrapped(|ui| {
ui.checkbox(&mut next.name, "Name").tip(&tips::COLUMNS);
// Checked and greyed rather than absent: an omitted entry reads as an
// oversight, a disabled one answers the question.
ui.add_enabled(false, egui::Checkbox::new(&mut true, "Path"))
.on_disabled_hover_text(
"The path is always shown — it is the only column that \
identifies a result on its own.",
);
ui.checkbox(&mut next.content_match, "Content Match")
.tip(&tips::COLUMNS);
ui.checkbox(&mut next.size, "Size").tip(&tips::COLUMNS);
ui.checkbox(&mut next.modified, "Modified")
.tip(&tips::COLUMNS);
ui.checkbox(&mut next.rank, "Rank").tip(&tips::COLUMNS);
});
ui.label(hint(
"Also on the Search tab: right-click any column header. Applied and \
saved immediately.",
));
(next != *current).then_some(next)
}
/// The Security block: status plus action buttons. Never renders the salt.
fn security_ui(
ui: &mut egui::Ui,
@ -430,6 +446,14 @@ fn security_ui(
action = Some(SecurityAction::Disable);
}
});
// Its own row: three buttons do not fit the window's width.
if ui
.button("Show database key…")
.tip(&tips::SHOW_KEY)
.clicked()
{
action = Some(SecurityAction::ShowKey);
}
let mut remember = current.security.use_keychain;
if ui
.checkbox(&mut remember, "Remember on this device")
@ -455,10 +479,10 @@ fn security_ui(
action
}
/// The per-section config controls of the Options window. Every row goes
/// The per-section config controls of the Settings tab. Every row goes
/// through [`crate::tips::tip_row`], so a setting cannot arrive here
/// without a tooltip.
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
match section {
Section::Indexing => {
egui::Grid::new("cfg-indexing")
@ -598,6 +622,10 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
&mut config.search.debounce_ms,
0..=2000,
);
tip_row(ui, "Live results", &tips::LIVE_RESULTS, |ui| {
ui.checkbox(&mut config.search.live_results, "")
});
});
// The warning comes and goes as the value is edited; keep it off
// the ids of what follows (`ui_util::stable_section`).

279
crates/quicksearch-gui/src/options/tests.rs → crates/quicksearch-gui/src/settings_tab/tests.rs
View file

@ -5,62 +5,74 @@ use super::*;
#[test]
fn a_fresh_draft_is_not_dirty() {
let mut w = OptionsWindow::new();
let mut w = SettingsTab::new();
let cfg = Config::default();
assert!(!w.is_dirty(&cfg), "no draft at all");
w.open_with(&cfg);
w.stage(&cfg);
assert!(!w.is_dirty(&cfg));
}
#[test]
fn an_edited_draft_is_dirty_until_discarded() {
let mut w = OptionsWindow::new();
let mut w = SettingsTab::new();
let cfg = Config::default();
w.open_with(&cfg);
w.stage(&cfg);
w.draft.as_mut().unwrap().search.debounce_ms += 100;
assert!(w.is_dirty(&cfg));
w.close_discard();
assert!(!w.open);
w.discard();
assert!(w.draft.is_none());
assert!(!w.is_dirty(&cfg), "the draft is gone");
}
/// The Security block and the mode buttons act on the live config while
/// the window sits open; the stale copies in the draft are not edits.
/// the tab is on screen; the stale copies in the draft are not edits.
#[test]
fn live_security_and_mode_changes_are_not_dirty() {
let mut w = OptionsWindow::new();
let mut w = SettingsTab::new();
let mut cfg = Config::default();
w.open_with(&cfg);
w.stage(&cfg);
cfg.security.use_keychain = !cfg.security.use_keychain;
cfg.indexing.auto_index = !cfg.indexing.auto_index;
assert!(!w.is_dirty(&cfg));
}
/// Leaving the tab drops the draft, so the next visit stages the config as
/// it stands *then*. Without this an edit made on the Manage Index tab in
/// between would be reverted by a later Apply: `pin_live_fields` protects
/// the fields saved live, but not the indexed folders or the filters.
#[test]
fn a_dirty_close_is_held_and_a_clean_one_drops_the_draft() {
let mut w = OptionsWindow::new();
let cfg = Config::default();
w.open_with(&cfg);
w.draft.as_mut().unwrap().search.debounce_ms += 100;
fn a_draft_is_restaged_from_the_live_config_after_leaving() {
let mut w = SettingsTab::new();
let mut cfg = Config::default();
w.stage(&cfg);
w.discard();
assert!(
w.intercept_close(false, &cfg),
"dirty close raises the guard"
cfg.indexing.ignore_patterns.push("*.tmp".to_string());
w.stage(&cfg);
assert!(!w.is_dirty(&cfg), "the fresh draft matches the live config");
assert_eq!(
w.draft_config().unwrap().indexing.ignore_patterns,
cfg.indexing.ignore_patterns,
"the filter added while the tab was away survives"
);
assert!(w.open, "the window is held open until the user decides");
assert!(w.draft.is_some(), "the draft survives");
}
assert!(!w.intercept_close(true, &cfg), "still open: nothing to do");
w.draft = Some(cfg.clone());
assert!(!w.intercept_close(false, &cfg), "a clean close just closes");
assert!(!w.open);
assert!(w.draft.is_none());
/// A key capture in progress cannot outlive the tab: the app stops asking
/// [`SettingsTab::capturing_hotkey`] once another tab is up, and the button
/// must not be waiting when the tab comes back either.
#[test]
fn leaving_the_tab_ends_a_shortcut_capture() {
let mut w = SettingsTab::new();
let cfg = Config::default();
w.stage(&cfg);
w.capturing_hotkey = true;
w.discard();
assert!(!w.capturing_hotkey());
}
use crate::test_ui::{click_at, painted_text, painted_text_center};
/// One frame of the shortcut control on its own, outside the window's
/// One frame of the shortcut control on its own, outside the tab's
/// scroll area so it is never below the fold.
fn run_hotkey_edit(
ctx: &egui::Context,
@ -295,10 +307,11 @@ const ROWS: &[(Section, &str, &tips::Tip)] = &[
&tips::RESULTS_PER_PAGE,
),
(Section::Search, "Debounce (ms)", &tips::DEBOUNCE),
(Section::Search, "Live results", &tips::LIVE_RESULTS),
];
/// Hovering a row's name paints that row's own explanation. Rendered
/// without the window's scroll area so nothing sits below the fold.
/// without the tab's scroll area so nothing sits below the fold.
#[test]
fn every_row_shows_its_own_tip() {
for (section, label, tip) in ROWS {
@ -348,27 +361,20 @@ fn hovering_a_setting_label_explains_it() {
s.interaction.show_tooltips_only_when_still = false;
});
let cfg = Config::default();
let mut w = OptionsWindow::new();
w.open_with(&cfg);
let mut w = SettingsTab::new();
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
let run = |w: &mut SettingsTab, events: Vec<egui::Event>| {
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
ctx.run(input, |ctx| {
w.ui(ctx, &cfg);
egui::CentralPanel::default().show(ctx, |ui| {
w.ui(ui, &cfg);
});
})
};
// The window spends its first frames sizing itself and painting
// nothing; run until the label is on screen.
let mut target = None;
for _ in 0..5 {
let full = run(&mut w, vec![]);
target = painted_text_center(&full, "Tokenizer");
if target.is_some() {
break;
}
}
let target = target.expect("the Tokenizer label was not painted");
let full = run(&mut w, vec![]);
let target =
painted_text_center(&full, "Tokenizer").expect("the Tokenizer label was not painted");
// The tooltip is an area of its own, so it can land a frame late.
let mut out = run(&mut w, vec![egui::Event::PointerMoved(target)]);
@ -384,37 +390,29 @@ fn hovering_a_setting_label_explains_it() {
panic!("no tooltip appeared over the Tokenizer label");
}
/// One real frame of the window in a headless context: it renders, and
/// the Apply & Save click comes back out as `applied`.
/// One real frame of the tab in a headless context: it renders, and the
/// Apply & Save click comes back out as `applied`.
#[test]
fn the_window_renders_and_apply_reports_the_draft() {
fn the_tab_renders_and_apply_reports_the_draft() {
let ctx = egui::Context::default();
let cfg = Config::default();
let mut w = OptionsWindow::new();
w.open_with(&cfg);
let mut w = SettingsTab::new();
w.stage(&cfg);
w.draft.as_mut().unwrap().search.debounce_ms += 100;
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
let run = |w: &mut SettingsTab, events: Vec<egui::Event>| {
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
let mut out = OptionsOutput::default();
let full = ctx.run(input, |ctx| out = w.ui(ctx, &cfg));
let mut out = SettingsOutput::default();
let full = ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| out = w.ui(ui, &cfg));
});
(out, full)
};
// A new egui window spends its first frames in sizing passes that
// suppress painting; run untouched frames until the settled button
// is actually on screen.
let mut target = None;
for _ in 0..5 {
let (untouched, full) = run(&mut w, vec![]);
assert!(untouched.applied.is_none());
assert!(!untouched.close_requested);
target = painted_text_center(&full, "Apply & Save");
if target.is_some() {
break;
}
}
let target = target.expect("the Apply & Save button was not painted");
let (untouched, full) = run(&mut w, vec![]);
assert!(untouched.applied.is_none());
let target = painted_text_center(&full, "Apply & Save")
.expect("the Apply & Save button was not painted");
let clicks = [true, false]
.into_iter()
.map(|pressed| egui::Event::PointerButton {
@ -432,3 +430,156 @@ fn the_window_renders_and_apply_reports_the_draft() {
"the click reported the edited draft"
);
}
/// One frame of the column picker on its own, outside the tab's scroll
/// area so it is never below the fold — the same shape as [`run_hotkey_edit`].
fn run_columns(
ctx: &egui::Context,
current: &ColumnsConfig,
events: Vec<egui::Event>,
) -> (Option<ColumnsConfig>, egui::FullOutput) {
let input = crate::test_ui::raw_input(egui::vec2(700.0, 200.0), events);
let mut picked = None;
let full = ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
picked = columns_ui(ui, current);
});
});
(picked, full)
}
/// The Settings copy of the column picker acts on the *live* config, not the
/// draft — the same arrangement the Security block uses, and the reason it and
/// the table header's menu cannot end up disagreeing. So it reports a change
/// the moment a box moves, with no Apply.
#[test]
fn the_columns_block_reports_a_change_immediately() {
let ctx = egui::Context::default();
let current = ColumnsConfig::default();
assert!(!current.size, "the fixture assumes Size ships off");
let (quiet, full) = run_columns(&ctx, &current, vec![]);
assert!(quiet.is_none(), "reported a change nobody made");
let target = painted_text_center(&full, "Size").expect("no Size checkbox");
let (picked, _) = run_columns(&ctx, &current, click_at(target));
let picked = picked.expect("the click reported nothing");
assert!(picked.size, "clicking Size did not switch it on");
// Only that one moved.
assert_eq!(
picked,
ColumnsConfig {
size: true,
..current
}
);
}
/// The path is not offered: it is the one column that identifies a result on
/// its own. It is shown checked and greyed rather than left out, so the
/// question "why can I not remove it?" has an answer on screen.
#[test]
fn the_columns_block_offers_every_column_but_the_path() {
let ctx = egui::Context::default();
let (_, full) = run_columns(&ctx, &ColumnsConfig::default(), vec![]);
let painted = painted_text(&full);
for label in ["Name", "Path", "Content Match", "Size", "Modified", "Rank"] {
assert!(
painted.iter().any(|t| t == label),
"{label} missing: {painted:?}"
);
}
// Clicking it does nothing, because it is disabled.
let target = painted_text_center(&full, "Path").expect("no Path entry");
let (picked, _) = run_columns(&ctx, &ColumnsConfig::default(), click_at(target));
assert!(picked.is_none(), "the path column was switched off");
}
/// One frame of the Security block on its own, in the shape of
/// [`run_columns`]. `keychain_active` is passed straight through, so nothing
/// here touches the OS keychain.
fn run_security(
ctx: &egui::Context,
current: &Config,
events: Vec<egui::Event>,
) -> (Option<SecurityAction>, egui::FullOutput) {
let input = crate::test_ui::raw_input(egui::vec2(700.0, 300.0), events);
let mut action = None;
let full = ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
action = security_ui(ui, current, false);
});
});
(action, full)
}
/// An unprotected index has no key at all, so there is nothing the button
/// could show and it is left out rather than shown dead.
#[test]
fn the_key_button_appears_only_while_the_index_is_encrypted() {
let ctx = egui::Context::default();
let mut cfg = Config::default();
assert!(
!cfg.security.password_protected,
"the fixture assumes protection ships off"
);
let (_, full) = run_security(&ctx, &cfg, vec![]);
assert!(
painted_text_center(&full, "Show database key…").is_none(),
"offered the key of an unencrypted index: {:?}",
painted_text(&full)
);
cfg.security.password_protected = true;
let (_, full) = run_security(&ctx, &cfg, vec![]);
assert!(
painted_text_center(&full, "Show database key…").is_some(),
"no key button while encrypted: {:?}",
painted_text(&full)
);
}
/// The click only asks for the flow; the password confirmation and the reveal
/// both live in the app, so nothing about the key is decided here.
#[test]
fn clicking_the_key_button_reports_show_key() {
let ctx = egui::Context::default();
let cfg = Config {
security: quicksearch_core::config::SecurityConfig {
password_protected: true,
..Default::default()
},
..Default::default()
};
let (quiet, full) = run_security(&ctx, &cfg, vec![]);
assert!(quiet.is_none(), "reported an action nobody clicked");
let target = painted_text_center(&full, "Show database key…").expect("no key button");
let (action, _) = run_security(&ctx, &cfg, click_at(target));
assert_eq!(action, Some(SecurityAction::ShowKey));
}
/// Columns are live state, so a draft taken before one changed must not carry
/// the old set back on Apply — `app::pin_live_fields` is what prevents that,
/// and this is the assertion that it covers this field.
#[test]
fn a_stale_draft_cannot_revert_the_columns() {
let mut w = SettingsTab::new();
let mut cfg = Config::default();
w.stage(&cfg);
// The header menu switches a column on while the tab is on screen.
cfg.search.columns.size = true;
assert!(!w.is_dirty(&cfg), "a live column change read as an edit");
let draft = w.draft_config().expect("a draft");
let mut applied = draft;
crate::app::pin_live_fields(&mut applied, &cfg);
assert!(
applied.search.columns.size,
"applying the stale draft reverted the column"
);
}

25
crates/quicksearch-gui/src/test_ui.rs
View file

@ -101,6 +101,31 @@ pub fn painted_spans(out: &egui::FullOutput) -> Vec<(String, egui::Color32)> {
.collect()
}
/// Every styled run painted this frame that has a background behind it, with
/// that background, in paint order.
///
/// The distinguishing mark of a highlighted match: the column headers and the
/// strong parts of a snippet are painted in the same *text* color, so
/// [`painted_spans`] alone cannot tell a match from a header.
pub fn painted_backgrounds(out: &egui::FullOutput) -> Vec<(String, egui::Color32)> {
painted_galleys(out)
.into_iter()
.flat_map(|(g, _)| {
g.job
.sections
.iter()
.filter(|s| s.format.background != egui::Color32::TRANSPARENT)
.map(|s| {
(
g.job.text[s.byte_range.clone()].to_string(),
s.format.background,
)
})
.collect::<Vec<_>>()
})
.collect()
}
/// Every *visible* row of every galley painted this frame, in paint order.
/// Not the same as [`painted_text`]: a galley's `text()` is the job it was
/// laid out from, including the rows epaint dropped at `wrap.max_rows` —

66
crates/quicksearch-gui/src/tips.rs
View file

@ -1,5 +1,5 @@
//! Plain-language tooltips for the configuration controls: every setting in
//! the Options window, and every configuration control on the Manage Index
//! the Settings tab, and every configuration control on the Manage Index
//! tab, explains itself on hover.
/// How wide a tooltip may get; matches `manage_tab::db_size_tooltip`.
@ -73,7 +73,7 @@ pub fn tip_row(
ui.end_row();
}
// --- Options: Paths ------------------------------------------------------
// --- Settings: Paths ------------------------------------------------------
pub static DATABASE_PATH: Tip = Tip {
title: "Database file",
@ -90,7 +90,7 @@ pub static DATABASE_PATH: Tip = Tip {
caution: None,
};
// --- Options: Indexing ---------------------------------------------------
// --- Settings: Indexing ---------------------------------------------------
pub static REINDEX_INTERVAL: Tip = Tip {
title: "Full reindex every",
@ -139,7 +139,7 @@ pub static INCLUDE_HIDDEN: Tip = Tip {
caution: None,
};
// --- Options: Processing -------------------------------------------------
// --- Settings: Processing -------------------------------------------------
pub static TOKENIZER: Tip = Tip {
title: "Tokenizer",
@ -257,7 +257,7 @@ pub static STORE_TEXT: Tip = Tip {
caution: None,
};
// --- Options: Search -----------------------------------------------------
// --- Settings: Search -----------------------------------------------------
pub static FUZZY_DEFAULT: Tip = Tip {
title: "Fuzzy search ON by default",
@ -329,7 +329,43 @@ pub static DEBOUNCE: Tip = Tip {
caution: None,
};
// --- Options: Interface --------------------------------------------------
pub static LIVE_RESULTS: Tip = Tip {
title: "Live results",
body: "Watches the results currently on screen and updates them as the \
files change, so a file you rename or edit in another window does \
not sit there showing its old name or its old text.\n\n\
Only the rows you can actually see are watched, and every one of \
them is dropped the moment you change the search. Nothing is ever \
added, removed or re-sorted while you read — a file that stops \
matching stays where it is until you search again.",
examples: &[
"Renames, deletions and edited contents all show up within a second, \
whatever the indexer is doing — what a row says is read from the \
file, not from the index.",
"A row is also checked against the disk as it comes on screen, so one \
the index was out of date about puts itself right. That check is all \
you get over a network share, where the system does not report other \
machines' writes.",
],
caution: Some(
"The files behind the rows you are looking at are kept up to date in \
the index too, even while indexing is stopped. Turn this off if a \
stopped index must mean nothing is written at all.",
),
};
pub static COLUMNS: Tip = Tip {
title: "Search columns",
body: "Which columns the results table shows. The path is always there — \
it is the only column that identifies a result on its own.\n\n\
Size and modified date start switched off: the width they take is \
usually better spent on the path and the matched text. Turning a \
column on also makes it available to sort by.",
examples: &["Right-clicking any column header on the Search tab does the same thing."],
caution: None,
};
// --- Settings: Interface --------------------------------------------------
pub static UI_SCALE: Tip = Tip {
title: "UI scale",
@ -380,7 +416,7 @@ pub static COLOR_SCHEME: Tip = Tip {
caution: None,
};
// --- Options: Security ---------------------------------------------------
// --- Settings: Security ---------------------------------------------------
pub static ENABLE_PASSWORD: Tip = Tip {
title: "Enable password protection",
@ -419,6 +455,21 @@ pub static DISABLE_PASSWORD: Tip = Tip {
),
};
pub static SHOW_KEY: Tip = Tip {
title: "Show database key",
body: "Reveals the raw SQLCipher key the index is encrypted with, once you \
have confirmed your password. Tools such as DB Browser for SQLCipher \
accept it in the 0x form shown and can then open the index file \
directly.\n\n\
The key is worked out from your password and the salt in the config \
file, so it stays the same until the password changes.",
examples: &[],
caution: Some(
"Anyone holding this key can read the index without the password. Treat a copy \
of it as carefully as the password itself.",
),
};
pub static REMEMBER_KEYCHAIN: Tip = Tip {
title: "Remember on this device",
body: "Hands the key to the password store your system already has, such \
@ -616,6 +667,7 @@ mod tests {
&ENABLE_PASSWORD,
&CHANGE_PASSWORD,
&DISABLE_PASSWORD,
&SHOW_KEY,
&REMEMBER_KEYCHAIN,
&START_NOW,
&STOP_INDEXING,

441
crates/quicksearch-gui/src/tutorial.rs Normal file
View file

@ -0,0 +1,441 @@
//! The first-start tour: a few pages explaining what QuickSearch indexes,
//! how results are ranked, and what the parts of the Search tab do.
//!
//! Shown once, to an installation that has never run before — `[ui]
//! tutorial_seen` is `Some(false)` only in a config file this version created,
//! so upgrading into this version does not summon it. The Help tab can bring
//! it back afterwards, which is also what keeps this from being write-only.
use crate::ui_util::{centered_modal, hint};
/// One page of the tour. Static text, so the pages are a table rather than a
/// match arm each.
struct Page {
title: &'static str,
/// Paragraphs. Rendered in order with a little space between them.
body: &'static [&'static str],
/// Rendered small and de-emphasised under the body — where to go, rather
/// than what the thing is.
pointer: Option<&'static str>,
}
const PAGES: &[Page] = &[
Page {
title: "Welcome to QuickSearch",
body: &[
"QuickSearch keeps an index of the folders you choose, and searches \
as you type.",
"By default only your user folder is indexed and searchable.",
"Because the answers come from the index rather than from reading \
your disk, results appear as fast as you can type, even across \
hundreds of thousands of files.",
],
pointer: Some("A quick tutorial for new users! Hit Skip to Exit."),
},
Page {
title: "What is indexed",
body: &[
"Indexing is QuickSearch reading through your folders once and \
remembering what it found, so that searching later is instant. It \
runs on its own in the background and keeps up with changes as you \
make them.",
"QuickSearch never connects to the internet, and always respects your privacy. \
QuickSearch can encrypt your index to make this remembered data more secure.",
"These are the folders being indexed right now:",
],
pointer: Some(
"To index more locations, open the Manage Index tab and add a folder. \
To set an index password, look near the bottom of the Settings tab.",
),
},
Page {
title: "How results are ranked",
body: &[
"The best search matches come first (have the lowest rank). \
Exact file-name matchs are best, then names that contain what you typed; then \
files whose contents contain the search terms, the ones mentioning it most often \
first; and last, files matched only by their full folder path.",
"The coloured number in the Rank column is which of those tiers a \
result came from: blue is a great match, red is a distant one. \
Clicking a column heading sorts by something else instead.",
],
pointer: Some("Right-click any column heading to choose which columns are shown."),
},
Page {
title: "The status bar",
body: &[
"The line along the bottom of the window is what QuickSearch is \
doing. While it is indexing it shows the phase, how far through it \
is, and how fast; when it has nothing to do it shows how many files \
are indexed.",
"Searching works the whole time, including during that first indexing run, \
but some files might not be shown in the results until the scan completes.",
],
pointer: None,
},
Page {
title: "Typos, and what a result can do",
body: &[
"Tick \"Fuzzy\"beside the search box to also match words with typos \
in them — \"repot\" will find \"report\". It searches more \
thoroughly, so it is a little slower; leave it off until you need \
it.",
"Right-click any result for more: open it, open the folder holding \
it, copy its path, or build a filter that hides files like it from \
future searches.",
],
pointer: Some(
"The ? button left of the search box lists the filters you can type \
into a query, like type:Document or modified:>=2024-01-01.",
),
},
Page {
title: "Duplicates",
body: &[
"The Duplicates tab looks for files across all indexed folders for identical copies. \
They are shown grouped together, with the largest wasted space first.",
"It is a quick way to find the same download sitting in three \
places. QuickSearch only shows you the groups; deleting anything is \
left to you.",
],
pointer: Some(
"For speed, files are compared by size and by how they begin (first 8KB). \
This is not a guarantee of an exact match. You can right click a result to verify before you delete anything.",
),
},
Page {
title: "Settings",
body: &[
"The Settings tab, at the right-hand end of the tab strip, is where \
you can tweak and tune the software. Mouse over any of \
the settings for a brief description of what they do.",
"Most changes wait for the Apply & Save button at the bottom.",
"QuickSearch is completely free for anyone to use. If you love it, please let your friends know about us!",
],
pointer: None,
},
];
/// The open tour.
pub struct Tutorial {
page: usize,
}
impl Tutorial {
pub fn new() -> Tutorial {
Tutorial { page: 0 }
}
/// Render. `roots` is the live indexed-folder list, named on the page
/// about indexing so the tour describes this installation rather than a
/// generic one.
/// Returns true once the tour is finished with — skipped or read to the
/// end — which is the caller's cue to remember that and drop it.
pub fn ui(&mut self, ctx: &egui::Context, roots: &[String]) -> bool {
let page = &PAGES[self.page.min(PAGES.len() - 1)];
let (first, last) = (self.page == 0, self.page + 1 == PAGES.len());
let mut dismissed = false;
let mut step: i64 = 0;
centered_modal(ctx, page.title, |ui| {
ui.set_max_width(520.0);
for paragraph in page.body {
ui.label(*paragraph);
ui.add_space(6.0);
}
// The one page that shows live state rather than static text.
if self.page == 1 {
if roots.is_empty() {
ui.label(hint("No folders are indexed yet."));
} else {
for root in roots {
ui.monospace(root);
}
}
ui.add_space(6.0);
}
if let Some(pointer) = page.pointer {
ui.label(hint(pointer));
}
ui.add_space(10.0);
ui.separator();
// Three equal thirds rather than one row: it is the only layout
// that puts Skip in the middle without measuring the buttons
// either side of it, whose widths change with the page ("Next"
// becoming "Finish") and with the counter's digits.
ui.columns(3, |cols| {
// A column lays its contents out *justified*, so a button put
// straight into one is stretched to the full third. The other
// two escape that by nesting their own layout; this one has to
// say so.
cols[0].with_layout(egui::Layout::left_to_right(egui::Align::Min), |ui| {
if ui.add_enabled(!first, egui::Button::new("Back")).clicked() {
step = -1;
}
});
cols[1].vertical_centered(|ui| {
if ui.button("Skip").clicked() {
dismissed = true;
}
});
// `Align::Min`, not `Center`: a column is as tall as the rest
// of the window, so centring in it drops the button a hundred
// points below the two beside it.
cols[2].with_layout(egui::Layout::right_to_left(egui::Align::Min), |ui| {
let p = crate::color::palette(ui.visuals().dark_mode);
let next = if last { "Finish" } else { "Next" };
if ui
.add(crate::ui_util::bordered_button(next, p.blue))
.clicked()
{
if last {
dismissed = true;
} else {
step = 1;
}
}
ui.label(hint(format!("{} of {}", self.page + 1, PAGES.len())));
});
});
});
// Applied after the closure so the page a frame rendered stays the page
// its buttons were laid out for.
if step != 0 {
let next = self.page as i64 + step;
self.page = next.clamp(0, PAGES.len() as i64 - 1) as usize;
}
dismissed
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::test_ui::{click_at, painted_text, raw_input};
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
/// Two passes: an `egui::Window` is measured on its first frame and only
/// placed on the next, so a single pass paints nothing to read back and
/// has nothing at a known position to click.
fn frame(
ctx: &egui::Context,
tour: &mut Tutorial,
events: Vec<egui::Event>,
) -> (egui::FullOutput, bool) {
let roots = ["/home/me".to_string()];
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
tour.ui(ctx, &roots);
});
let mut dismissed = false;
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
dismissed = tour.ui(ctx, &roots);
});
(out, dismissed)
}
/// Every page has something to say, and says it.
#[test]
fn every_page_paints_its_own_title_and_body() {
for (page, spec) in PAGES.iter().enumerate() {
let ctx = egui::Context::default();
let mut tour = Tutorial { page };
let (out, _) = frame(&ctx, &mut tour, Vec::new());
let painted = painted_text(&out);
assert!(
painted.iter().any(|t| t == spec.title),
"page {page} painted no title: {painted:?}"
);
assert!(!spec.body.is_empty(), "page {page} has an empty body");
assert!(
painted
.iter()
.any(|t| t == &format!("{} of {}", page + 1, PAGES.len())),
"page {page} did not say where it is: {painted:?}"
);
}
}
/// The page about indexing names *this* installation's folders, not a
/// generic example.
#[test]
fn the_indexing_page_lists_the_configured_folders() {
let ctx = egui::Context::default();
let mut tour = Tutorial { page: 1 };
let roots = ["/srv/projects".to_string()];
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
tour.ui(ctx, &roots);
});
let out = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
tour.ui(ctx, &roots);
});
let painted = painted_text(&out);
assert!(painted.iter().any(|t| t == "/srv/projects"), "{painted:?}");
}
/// The y of the footer row on `page`, found by walking down the middle
/// column, which only Skip occupies. Wrapped text height moves the row
/// from page to page, so it is probed rather than guessed.
fn footer_y(ctx: &egui::Context, page: usize) -> f32 {
for y in (150..600).step_by(2) {
let mut t = Tutorial { page };
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(500.0, y as f32)));
if dismissed {
return y as f32;
}
}
panic!("no Skip button down the middle of page {page}");
}
/// The stretch of x along the footer row that one button answers a click
/// on — where it is, and how wide.
#[derive(Debug, Clone, Copy)]
struct Span {
lo: f32,
hi: f32,
}
impl Span {
fn width(&self) -> f32 {
self.hi - self.lo
}
}
/// The three footer buttons, found by what clicking each one does: Back
/// steps a page back, Next steps one forward, Skip dismisses.
///
/// Must be run on a middle page — on the last page Finish and Skip both
/// dismiss without moving, and on the first Back is disabled.
fn footer_spans(ctx: &egui::Context, page: usize) -> [Option<Span>; 3] {
assert!(page > 0 && page + 1 < PAGES.len(), "probe a middle page");
let y = footer_y(ctx, page);
let mut spans: [Option<Span>; 3] = [None; 3];
for x in 150..850 {
let mut t = Tutorial { page };
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(x as f32, y)));
let x = x as f32;
let which = if dismissed {
1 // Skip
} else if t.page + 1 == page {
0 // Back
} else if t.page == page + 1 {
2 // Next
} else {
continue;
};
match &mut spans[which] {
Some(span) => span.hi = x,
slot => *slot = Some(Span { lo: x, hi: x }),
}
}
spans
}
/// The footer reads Back, then Skip, then Next — and each does what its
/// label says. Positions are probed rather than asserted against numbers:
/// the window auto-sizes to the page's text, so the thirds move.
///
/// The widths are the other half of it. `Ui::columns` lays a column out
/// justified, so a button dropped straight into one comes out as wide as
/// the whole third — which is what Back was until it was given a layout of
/// its own. Two buttons with four-letter labels either side of the row
/// have to come out the same size.
#[test]
fn the_footer_runs_back_then_skip_then_next_at_the_same_size() {
let ctx = egui::Context::default();
let [back, skip, next] = footer_spans(&ctx, 1);
let back = back.expect("no Back button in the footer");
let skip = skip.expect("no Skip button in the footer");
let next = next.expect("no Next button in the footer");
assert!(
back.lo < skip.lo,
"Back ({back:?}) is not left of Skip ({skip:?})"
);
assert!(
skip.lo < next.lo,
"Skip ({skip:?}) is not left of Next ({next:?})"
);
assert!(
(back.width() - next.width()).abs() <= 2.0,
"Back is {} wide against Next's {}",
back.width(),
next.width()
);
// Belt and braces on the shape of the bug: a stretched button fills
// its third of a 520-point modal, which no four-letter label does.
assert!(
back.width() < 80.0,
"Back is stretched to {} points",
back.width()
);
}
/// Back is disabled on the first page, so nothing in the footer can walk
/// the tour off the front.
#[test]
fn the_first_page_cannot_go_back() {
let ctx = egui::Context::default();
let y = footer_y(&ctx, 0);
for x in (150..850).step_by(4) {
let mut t = Tutorial { page: 0 };
let _ = frame(&ctx, &mut t, click_at(egui::pos2(x as f32, y)));
assert!(
t.page == 0 || t.page == 1,
"clicking x={x} left page {}",
t.page
);
}
}
/// Clicking anywhere in the button row, on any page; collects what fired.
fn sweep(ctx: &egui::Context, tour: &mut Tutorial) -> Vec<usize> {
let mut seen = Vec::new();
for y in (200..500).step_by(4) {
for x in (240..760).step_by(8) {
let mut t = Tutorial { page: tour.page };
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(x as f32, y as f32)));
if dismissed {
seen.push(t.page);
}
}
}
seen
}
/// Both ways out of the tour report the dismissal, so the flag gets set
/// whichever the user takes. Positions depend on wrapped text height, so
/// the button row is swept rather than guessed at — the same approach the
/// confirmation modals' tests take.
#[test]
fn skip_and_finish_both_dismiss() {
let ctx = egui::Context::default();
// Skip is on every page.
let mut tour = Tutorial { page: 0 };
assert!(
!sweep(&ctx, &mut tour).is_empty(),
"Skip never fired on the first page"
);
// Finish only on the last, where it replaces Next.
let mut tour = Tutorial {
page: PAGES.len() - 1,
};
assert!(
!sweep(&ctx, &mut tour).is_empty(),
"Finish never fired on the last page"
);
let ctx = egui::Context::default();
let mut tour = Tutorial {
page: PAGES.len() - 1,
};
let (out, _) = frame(&ctx, &mut tour, Vec::new());
assert!(
painted_text(&out).contains(&"Finish".to_string()),
"the last page still offers Next"
);
}
}

52
crates/quicksearch-gui/src/ui_util.rs
View file

@ -122,8 +122,10 @@ pub fn pattern_edit(
(response, valid)
}
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
/// `font_id`, returning it borrowed and untouched when it already fits.
/// Where a middle-elide cuts `text` to fit `max_width` pixels in `font_id`:
/// the byte offset the head keeps up to, and the one the tail resumes from,
/// with a single `…` standing for everything between. `None` when the whole
/// string fits and nothing is dropped.
///
/// The budget is in pixels, summed from the font's own glyph advances (the
/// same numbers egui's layout adds up), not a character count scaled by one
@ -131,18 +133,20 @@ pub fn pattern_edit(
/// directions: overshoot and egui elides the result a *second* time,
/// painting two ellipses; undershoot and the column sits visibly short.
///
/// The borrowed/owned distinction is the caller's signal that something was
/// dropped, which is what a "full text on hover" tooltip keys off.
pub fn middle_elide<'a>(
/// Split out from [`middle_elide`] because a caller that also has *ranges* to
/// highlight needs the cut itself, not just the shortened string: it renders
/// the two surviving ends separately so its marks keep the offsets they had
/// (see `snippet_render::path_cell_job`).
pub fn middle_elide_cut(
ui: &egui::Ui,
text: &'a str,
text: &str,
max_width: f32,
font_id: &egui::FontId,
) -> Cow<'a, str> {
) -> Option<(usize, usize)> {
ui.fonts(|f| {
let width_of = |c: char| f.glyph_width(font_id, c);
if text.chars().map(width_of).sum::<f32>() <= max_width {
return Cow::Borrowed(text);
return None;
}
let budget = max_width - width_of('…');
@ -178,17 +182,35 @@ pub fn middle_elide<'a>(
}
if head >= tail {
// The two halves met without dropping anything.
return Cow::Borrowed(text);
return None;
}
let mut out = String::with_capacity(head + '…'.len_utf8() + (text.len() - tail));
out.push_str(&text[..head]);
out.push('…');
out.push_str(&text[tail..]);
Cow::Owned(out)
Some((head, tail))
})
}
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
/// `font_id`, returning it borrowed and untouched when it already fits.
///
/// The borrowed/owned distinction is the caller's signal that something was
/// dropped, which is what a "full text on hover" tooltip keys off.
pub fn middle_elide<'a>(
ui: &egui::Ui,
text: &'a str,
max_width: f32,
font_id: &egui::FontId,
) -> Cow<'a, str> {
match middle_elide_cut(ui, text, max_width, font_id) {
None => Cow::Borrowed(text),
Some((head, tail)) => {
let mut out = String::with_capacity(head + '…'.len_utf8() + (text.len() - tail));
out.push_str(&text[..head]);
out.push('…');
out.push_str(&text[tail..]);
Cow::Owned(out)
}
}
}
/// Paint a semitransparent down-arrow near the bottom edge of a scroll
/// area while more content lies below the fold. Painter-only, so it can
/// never swallow clicks; the bundled fonts have no ▼ glyph, so it is a

6
crates/quicksearch-gui/src/unlock.rs
View file

@ -72,12 +72,12 @@ impl Gate {
return;
}
if let Gate::Running(app) = self {
// The Options window is waiting for a key press to bind; the
// shortcut must not reshuffle the window underneath that dialog.
// The Settings tab is waiting for a key press to bind; the
// shortcut must not reshuffle the window underneath it.
if app.capturing_hotkey() {
return;
}
app.activate_search();
app.activate_search(ctx);
}
crate::hotkey::raise(ctx, frame);
}

14
packaging/capture-scenario.txt
View file

@ -3,7 +3,7 @@
# "capture"). Edit this file to re-choreograph the captures; timings are in
# milliseconds and `max` caps bound clip length without failing the run.
# The demo config sets [ui] scale = 1.5, so every window size here is 1.5x
# The demo config sets [ui] scale = 1.25, so every window size here is 1.25x
# the layout it shows: high-resolution captures of an unchanged layout.
# --- 1. manage-indexing.webm: the fresh auto-index in progress -------------
@ -26,8 +26,12 @@ focus_search # tab switches drop egui focus; re-arm it
clear_query
window 1200 600 # compact clip: the smallest layout at
# which every results column still fits
# (any narrower clips the Match column
# away, defeating the demo), at 1.5x
# (any narrower clips the Content Match
# column away, defeating the demo), at
# 1.25x. Sized for the four columns the
# demo config pins — Name, Path, Content
# Match, Rank — so turning another one on
# there means revisiting this width.
wait_ms 800 # the resize lands asynchronously
record_start search
# The query is typed in quick bursts with human hesitations; each pause
@ -42,8 +46,8 @@ wait_ms 300
type "ition" cps 10
wait_search_done max 8000
wait_ms 1000
hover_match 2 # pin the pointer on the 3rd result's Match
# cell: the tooltip expands the snippet
hover_match 2 # pin the pointer on the 3rd result's Content
# Match cell: the tooltip expands the snippet
# with surrounding file content
wait_ms 2500 # tooltip delay, then linger on it
record_stop # end the clip with the tooltip on screen

30
packaging/capture.sh
View file

@ -68,8 +68,34 @@ auto_index = true
include_hidden = true
ignore_patterns = []
# 1.5x zoom + proportionally larger windows (set in the scenario) render the
# same layout at ~1.5x the pixel density, for crisper website assets.
[search]
# Off for the run: on, the watcher arms over the visible rows mid-recording,
# re-reads them from disk and hands the paths back for reindexing — index
# writes inside a clip that is supposed to be reproducible.
live_results = false
[search.columns]
# Exactly the four the captures are choreographed around: Name, Path, Content
# Match, Rank. Every one is pinned rather than defaulted — the scenario's
# window widths are chosen for this column set, and `hover_match` addresses
# Content Match cells by index, so a shipped default that moved would silently
# re-frame or break the clips.
#
# Path is absent because it is not optional: it is always drawn (see
# `ColumnsConfig`, which deliberately does not represent it).
name = true
content_match = true
rank = true
size = false
modified = false
# 1.25x zoom + proportionally larger windows (set in the scenario) render the
# same layout at ~1.25x the pixel density, for crisper website assets.
#
# No `tutorial_seen` key here, deliberately: absent it deserializes to `None`,
# which reads as "an installation that upgraded into this version". Writing
# `false` would open the first-run tour over the first capture and wedge the
# run.
[ui]
scale = 1.25
EOF

2
packaging/quicksearch.1
View file

@ -136,7 +136,7 @@ text stays part of the search phrase.
.BR AND ", " OR
and parentheses are treated as plain words.
.SH PASSWORD PROTECTION
The index can be encrypted with a password (application Options, Security).
The index can be encrypted with a password (the Settings tab, Security).
A protected index must be unlocked every time either binary starts. The
application shows an unlock screen; terminal mode resolves the key from, in
order: the OS keychain (when \(lqRemember on this device\(rq is enabled),

12
packaging/quicksearch.nsi
View file

@ -61,6 +61,18 @@ VIAddVersionKey "FileDescription" "${APP} ${VERSION} installer"
!insertmacro MUI_PAGE_COMPONENTS
!insertmacro MUI_PAGE_DIRECTORY
!insertmacro MUI_PAGE_INSTFILES
; The payload is two binaries and three documents, so this install finishes in
; about a second - fast enough that people reported it as a failure. The finish
; page is the only place that can say otherwise, so it lists what was installed
; and where instead of the stock "Setup was completed successfully."
; Kept to six lines: the finish page's text field is a fixed height, and MUI
; clips rather than scrolls what does not fit.
!define MUI_FINISHPAGE_TITLE "${APP} ${VERSION} is installed"
!define MUI_FINISHPAGE_TEXT "Installed into $INSTDIR:$\r$\n\
quicksearch.exe (the app), quicksearch-cli.exe (terminal search),$\r$\n\
README.md, LICENSE.txt and config_example.toml.$\r$\n$\r$\n\
Your settings and search index are created on first run, under your \
own account. Upgrading and uninstalling leave both alone."
!define MUI_FINISHPAGE_RUN
!define MUI_FINISHPAGE_RUN_TEXT "Run ${APP}"
!define MUI_FINISHPAGE_RUN_FUNCTION LaunchAsUser