jeremy/quick_search
jeremy/quick_search
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages Wiki Activity Actions

Improved Windows handling of Cloud folders. Improved indexing status reporting. Cleaned up some comments. Added support for old M$ word documents. Improved idle memory consumption.
All checks were successful
CI / linux (push) Successful in 3m2s
Details
CI / windows-cross (push) Successful in 2m7s
Details
CI / release (push) Successful in 13s
Details

Browse source
This commit is contained in:
= 2026-08-05 18:05:04 -04:00
parent ea224eed40
commit e8057e596c
59 changed files with 7841 additions and 1658 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
Cargo.lock generated
View file

@ -3190,9 +3190,10 @@ dependencies = [
[[package]]
name = "quicksearch-core"
version = "0.9.2"
version = "1.0.2"
dependencies = [
"argon2",
"cfb",
"chardetng",
"ctrlc",
"encoding_rs",
@ -3222,7 +3223,7 @@ dependencies = [
[[package]]
name = "quicksearch-gui"
version = "0.9.2"
version = "1.0.2"
dependencies = [
"chrono",
"eframe",

2
Cargo.toml
View file

@ -6,7 +6,7 @@ members = [
]
[workspace.package]
version = "1.0.1"
version = "1.0.2"
edition = "2021"
license = "GPL-3.0-or-later"
authors = ["Jeremy <jeremy@karsttech.com>"]

72
README.md
View file

@ -66,7 +66,7 @@ and exit codes behave normally. On Unix `quicksearch` also does both, and
```sh
./packaging/build-deb.sh
sudo apt install ./dist/quicksearch_0.1.0_amd64.deb
sudo apt install ./dist/quicksearch_1.0.2_amd64.deb
```
The script builds the release binary, strips it, and assembles a `.deb` with
@ -172,7 +172,7 @@ installer, so it takes `/S` for a silent install and `/D=` for the directory
(last argument, unquoted):
```bat
quicksearch-0.9.1-windows-x86_64-setup.exe /S /D=C:\Tools\QuickSearch
quicksearch-1.0.2-windows-x86_64-setup.exe /S /D=C:\Tools\QuickSearch
```
The `.zip` on the release page is the alternative to all of this: the same two
@ -211,7 +211,17 @@ inside that folder.
what each tab does — pointing here for everything technical.
The bottom status bar always shows what the indexer is doing (phase,
percent, files/sec) or the total indexed file count when idle.
percent, files/sec) or the total indexed file count when idle. Applying a
settings change to the index counts as something the indexer is doing: it
reports its progress there and in the Manage Index tab, and says what it
removed for a few seconds after it finishes, so a change that takes a
millisecond is as visible as one that takes minutes.
Quitting while a settings change is still being applied asks first. Leaving
is never refused — the work stops promptly and the index stays consistent —
but it stops part-way, so entries you excluded can still turn up in search
results until indexing runs again. The next launch says so, with a button to
start that run; in automatic mode the periodic reindex does it for you.
### Terminal
@ -317,10 +327,12 @@ default index goes to `~/.local/share/quicksearch/index.sqlite`
Defaults follow the platform. The first indexing root is your home
directory or `%USERPROFILE%`; `include_hidden = false` skips dot-files
everywhere and additionally anything marked Hidden or System on Windows,
which is what keeps `AppData`, `$RECYCLE.BIN` and `System Volume
Information` out of the index; and ignore patterns are matched
case-insensitively on Windows and macOS, matching the filesystem.
everywhere and additionally anything marked Hidden on Windows, which is
what keeps `AppData`, `$RECYCLE.BIN` and `System Volume Information` out
of the index — the System attribute alone is not enough, because cloud
sync roots carry it purely to get a branded folder icon; and ignore
patterns are matched case-insensitively on Windows and macOS, matching
the filesystem.
**Portable mode**: a `config.toml` sitting next to the `quicksearch`
binary overrides the user config entirely, and relative paths inside any
@ -381,8 +393,10 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
- **Indexing** (`indexing.rs`, `file_handling.rs`): full runs walk each
root (`filtered_walk` prunes hidden/ignored subtrees before descending),
classify files by mtime into insert/update/skip, batch-write metadata,
sweep stale rows, then extract content (plaintext, RTF, Office, PDF,
audio tags, EXIF; see `extract/`) for FTS. Files whose extension no MIME
sweep stale rows, then extract content (plaintext, RTF, Office — both the
OOXML/ODF zip formats and the pre-2007 binary `.doc`/`.xls`/`.ppt`, whose
OLE2 streams are read in `extract/ole.rs` — PDF, audio tags, EXIF; see
`extract/`) for FTS. Files whose extension no MIME
table knows — including extensionless ones like `README` or `Makefile` —
are sniffed from their head bytes and indexed as text only when that head
is provably text: valid UTF-8, or BOM-marked (`mime.rs`, `textenc.rs`).
@ -397,7 +411,12 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
it completed or was stopped — with an optimize pass on its own connection:
checkpoint, VACUUM if the file has at least 10% slack to reclaim, `PRAGMA
optimize`, checkpoint again. Progress streams through a polled
`IndexingStatus`, which reads `Optimizing` for the duration of that pass.
`IndexingStatus`, which reads `Optimizing` for the duration of that pass —
and `Preparing` for everything a run does before its first file is walked:
waiting on the previous run's thread, opening the index (a WAL recovery
lands here), and reconciling a changed configuration. Each carries the run's
start time, so a prologue that outlasts the walk on a large index reads as
slow work rather than a hang.
- **Scope reconciliation** (`scope.rs`): the index is a cache of what a walk
under the configured roots would produce, so a configuration change is a
difference between the two rather than a reason to start over.
@ -412,6 +431,20 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
closed behave like one edited live. The scan is per-root, by `[lo, hi)`
range: a symlink target stored outside every root has no owning root and
therefore no rules that could be applied to it, so it is never visited.
Whichever of the two applies it, a pass that *finishes* records what it
reconciled against — everything but the three rebuild-only keys, which no
scan can satisfy. That record is the whole convergence condition: an
abandoned pass leaves it alone and the next run picks the work back up,
while a completed one stops every later run from re-deriving the same plan
and rescanning every row to redo work already done. Both report a live
`ReconcileProgress` while they scan, since on a large index this is minutes
of work with no files moving to show for it. Both can also be abandoned:
`advance` reads a cancel flag before every statement, and the statement
already running — one `DELETE` can cover a whole root — is ended by
`sqlite3_interrupt`, since a flag alone cannot reach inside SQLite. That is
what makes closing the window during a prune immediate instead of a wait
the desktop offers to kill. `scope::outstanding_work` asks the record what
is still owed, which is how the GUI knows to remind you at the next launch.
- **Coordinator** (`coordinator.rs`): the object binaries construct.
Owns the `IndexingService`, the debouncing filesystem watcher
(`watcher.rs`), and the mode state machine (Auto / Manual, persisted as
@ -444,10 +477,12 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
flush last, so weaker matches only ever append. All SQL is
parameterized; structured filters from the query language (`query/`)
are ANDed onto every pass.
- **Baloo compatibility** (`cli.rs`, `mime.rs`): read-only endpoints
(`status_for_path`, `list_failed`, `index_size_breakdown`, …) and a
Baloo-shaped type model, groundwork for a future `balooctl`-compatible
layer.
- **Baloo compatibility** (`cli.rs`, `mime.rs`): the read API this repo's
parent consumes — `status_for_path`, `list_failed`,
`index_size_breakdown`, `pending_content_count`, `clear_path` — plus a
Baloo-shaped type model. Only `index_counts` has a caller inside this
repository; the rest are a compatibility surface for the parent's
`balooctl` layer and are not dead code.
- **Logging** (`log.rs`): background reporting goes through `log_info!` /
`log_warn!` rather than `println!`/`eprintln!`. Each writes its line to
stderr *and* appends it to a bounded in-memory ring (newest 5000 lines,
@ -456,7 +491,8 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
search hits, usage, the error a command exits with — stays on stdio.
- **Platform differences** (`platform.rs`): the single home for `#[cfg]`.
Home directory lookup, what counts as a hidden entry (dot-prefix, plus
the Hidden/System attributes on Windows), network-filesystem detection
the Hidden attribute on Windows — System deliberately excluded, since
cloud sync roots set it to get a folder icon), network-filesystem detection
(`/proc/mounts` against `GetDriveTypeW`), path collation, and the
watch-registration strategy all live here, so the rest of the crate can
ask a question rather than test a target. Anything decidable from a
@ -490,7 +526,11 @@ pagination: the table is virtualized, so a single scroll list capped at
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
ranking, cancellation, incremental indexing, coordinator modes, config
resolution, fuzzy matcher vs. brute-force oracle).
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units.
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units plus
headless egui tests that drive the real widgets — building an input frame,
synthesizing clicks and reading back the painted text (`test_ui.rs`) — over
the search and manage tabs, the options editor, the unlock gate, the logs
and duplicates tabs, and query highlighting.
- `QSB_SNIPPET_PERF=1 cargo test --release -p quicksearch-core --test
snippet_perf -- --nocapture`: snippet pipeline benchmark.
- `.forgejo/workflows/ci.yml`: builds both platforms on every push to `master`

17
config_example.toml
View file

@ -34,7 +34,7 @@ database_path = "~/.local/share/quicksearch/index.sqlite"
# buttons on the Manage Index tab write this value, so the mode you left the
# app in is the mode it starts in.
auto_index = true
reindex_interval_minutes = 1440
reindex_interval_minutes = 60
# Follow symbolic links during directory walks. Applies to links pointing at
# files as well as at directories: with this off a symlink is not resolved at
# all, so its target is never indexed — which matters because a target can
@ -43,8 +43,12 @@ reindex_interval_minutes = 1440
# that are no longer in scope; turning it on reindexes to find them.
follow_symlinks = false
# Index hidden files and directories. That means dot-files everywhere, and
# additionally anything carrying the Hidden or System attribute on Windows
# (AppData, $RECYCLE.BIN, System Volume Information, pagefile.sys ...).
# additionally anything carrying the Hidden attribute on Windows (AppData,
# $RECYCLE.BIN, System Volume Information, pagefile.sys ...). The System
# attribute on its own does not count: Windows honours the desktop.ini inside
# a folder only if the folder carries System or Read-only, so cloud sync roots
# (ownCloud, Nextcloud, OneDrive, Google Drive) and any folder given a custom
# icon carry it purely to get that icon, and are indexed normally.
# Turning this off removes the hidden entries already indexed.
include_hidden = false
# Empty = extract text from every supported format. Non-empty = content
@ -145,6 +149,13 @@ use_keychain = false
# together (0.5 – 2.5). Ctrl +/- and Ctrl 0 adjust it temporarily at
# runtime; this value is the persistent baseline.
scale = 1.1
# Written by QuickSearch, not by you: the folders that have already shown
# the "more subfolders than the watcher can follow" warning, so restarting
# does not repeat it. Keyed by folder rather than a single flag so that
# adding a folder warns again — the trade-off changed — and pruned to the
# current folder list whenever it is applied. Deleting it just means the
# warnings come back once each.
watch_cap_warned_roots = []
[search]
# Start with the fuzzy passes enabled.

14
crates/quicksearch-core/Cargo.toml
View file

@ -29,6 +29,11 @@ walkdir = "2.5.0"
# stored, so zstd/bzip2/aes-crypto are all dead weight here.
zip = { version = "0.6", default-features = false, features = ["deflate"] }
quick-xml = "0.31"
# OLE2 compound-file reader, for the pre-2007 binary Office formats
# (.doc/.xls/.ppt) whose text lives in named streams rather than a zip. Already
# in the lockfile transitively via infer, so naming it directly compiles
# nothing new.
cfb = "0.7"
serde = { version = "1.0", features = ["derive"] }
toml = "0.8"
mime_guess = "2.0"
@ -62,8 +67,10 @@ regex = "1"
libc = "0.2"
# `GetDriveTypeW` (a mapped drive letter is the only way to spot an SMB share
# that isn't written as UNC) plus the FILE_ATTRIBUTE_* constants for hidden
# detection. Pinned to 0.52 deliberately: walkdir → winapi-util already
# that isn't written as UNC) plus the FILE_ATTRIBUTE_* constants, which
# `platform.rs` spells out for itself so its tests run on Linux and then
# const-asserts against this crate on Windows builds.
# Pinned to 0.52 deliberately: walkdir → winapi-util already
# resolves exactly that version, so this adds no new crate compilations.
[target.'cfg(windows)'.dependencies]
# GetDriveTypeW and the FILE_ATTRIBUTE_* constants live in
@ -73,6 +80,9 @@ libc = "0.2"
# GetCurrentThread need Win32_Foundation for HANDLE/BOOL.
windows-sys = { version = "0.52", features = [
"Win32_Foundation",
# SECURITY_ATTRIBUTES, which `CreateFileW`'s signature names even though
# the directory-count path passes null for it.
"Win32_Security",
"Win32_Storage_FileSystem",
"Win32_System_Threading",
"Win32_System_WindowsProgramming",

2
crates/quicksearch-core/examples/indexprobe.rs
View file

@ -198,7 +198,7 @@ fn prose(rng: &mut Rng, target: usize) -> String {
let mut s = String::with_capacity(target + 16);
while s.len() < target {
s.push_str(WORDS[rng.next() as usize % WORDS.len()]);
s.push(if rng.next() % 12 == 0 { '\n' } else { ' ' });
s.push(if rng.next().is_multiple_of(12) { '\n' } else { ' ' });
}
s.truncate(target);
s

9
crates/quicksearch-core/examples/memprobe.rs
View file

@ -207,7 +207,7 @@ fn run(mode: &str, root: &str, db: &Path, interval: Duration, config_path: Optio
file,
});
if ticks % marker_every == 0 && db.exists() {
if ticks.is_multiple_of(marker_every) && db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
@ -254,6 +254,9 @@ fn progress(status: &IndexingStatus) -> (usize, usize, &'static str, String) {
(walked, extracted, phase, file)
}
/// Eight columns of one probe run, printed as a line. Grouping them into a
/// struct would only move the same eight names one level out.
#[allow(clippy::too_many_arguments)]
fn report(
mode: &str,
elapsed: Duration,
@ -267,8 +270,8 @@ fn report(
// One line per 5% of the run, so the shape is visible at any duration.
let step = (samples.len() / 20).max(1);
eprintln!(
"\n {:>8} {:>10} {:>9} {:>10} {}",
"t", "RSS", "walked", "extracted", "phase"
"\n {:>8} {:>10} {:>9} {:>10} phase",
"t", "RSS", "walked", "extracted"
);
for s in samples.iter().step_by(step) {
eprintln!(

40
crates/quicksearch-core/examples/oleprobe.rs Normal file
View file

@ -0,0 +1,40 @@
//! Print what the legacy-Office extractor gets out of a real file.
//!
//! The unit tests build their own fixtures, which proves the parsers agree
//! with the format specs as read. This runs them against files a real
//! producer wrote, which is the other half of the question.
//!
//! cargo run --example oleprobe -- some.doc some.xls some.ppt
use std::path::Path;
use quicksearch_core::extract::{Extractor, Registry};
fn main() {
let mut failures = 0;
for arg in std::env::args().skip(1) {
let path = Path::new(&arg);
println!("=== {} ===", path.display());
match quicksearch_core::extract::office::OfficeExtractor.extract(path) {
Ok(content) => {
let text = content.text;
println!("{} chars", text.chars().count());
let preview: String = text.chars().take(400).collect();
println!("{}", preview);
}
Err(e) => {
failures += 1;
println!("FAILED: {}", e);
}
}
// The dispatch a real index would take: MIME, not extension.
let registry = Registry::default_set();
let mime = mime_guess::from_path(path)
.first()
.map(|m| m.essence_str().to_string())
.unwrap_or_default();
println!("(mime {} claimed: {})", mime, registry.supports(&mime));
println!();
}
std::process::exit(if failures > 0 { 1 } else { 0 });
}

111
crates/quicksearch-core/src/cli.rs
View file

@ -1,10 +1,27 @@
//! Programmatic read-only query helpers.
//! Programmatic query helpers.
//!
//! Pure functions that open a DB, run a query, and return structured data.
//! No stdout, no CLI framing — callers (GUI, future CLI binaries, Set B
//! `balooctl`) format the result as they see fit. Mutating operations live
//! on [`crate::indexing::IndexingService`] since they require a running
//! worker thread.
//! No stdout, no CLI framing — the GUI and `quicksearch-cli` format the
//! result as they see fit. Indexing operations live on
//! [`crate::indexing::IndexingService`] since they require a running worker
//! thread.
//!
//! # These are consumed from outside this repository
//!
//! QuickSearch is a sub-repo. Of everything here only [`index_counts`] has a
//! caller in this tree (the GUI status bar); [`status_for_path`],
//! [`list_failed`], [`index_size_breakdown`], [`pending_content_count`] and
//! [`clear_path`] are called by the parent repository's Baloo compat daemon,
//! which is what reports them to `balooctl` and mirrors them into LMDB.
//!
//! So they are **not dead code**, and their signatures are a compatibility
//! surface rather than an internal detail: a search of this repository alone
//! will not turn up the callers that break when one changes.
//!
//! One exception to the "query helpers" framing: [`clear_path`] mutates. It
//! opens its own writer, which sidesteps the single-writer discipline the
//! coordinator maintains, so it is safe only against an index no local
//! coordinator is running against.
use rusqlite::{params, OptionalExtension};
@ -233,16 +250,7 @@ mod tests {
use crate::mime::FileType;
fn tmp_path() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-cli-test-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
crate::testutil::scratch_dir("cli").join("index.sqlite")
}
fn seed_fixture(db_path: &str) -> (i64, i64) {
@ -328,6 +336,79 @@ mod tests {
std::fs::remove_file(&p).ok();
}
/// The count is "outstanding extraction work", which is a narrower thing
/// than "rows without text": a file nothing extracts is settled, not
/// waiting, and would otherwise be reported as a backlog that never
/// drains.
#[test]
fn pending_content_count_counts_only_outstanding_work() {
let p = tmp_path();
let dbp = p.to_str().unwrap();
// a is Done, b is Failed — both resolved, neither pending.
let _ = seed_fixture(dbp);
assert_eq!(pending_content_count(dbp).unwrap(), 0);
let mut conn = open_or_recreate(dbp, "trigram").unwrap();
{
let tx = conn.transaction().unwrap();
// Claimed by an extractor, text not read yet: this is the backlog.
insert_file(
&tx,
&NewFile {
name: "c.txt",
path: "/tmp/c.txt",
parent: "/tmp",
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: Some("text/plain"),
ftype: FileType::TEXT,
hash: None,
needs_content: true,
},
)
.unwrap()
.expect("unique path");
// Nothing extracts this one, so it is NA on arrival and must not
// inflate the figure.
insert_file(
&tx,
&NewFile {
name: "d.bin",
path: "/tmp/d.bin",
parent: "/tmp",
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: None,
ftype: FileType::EMPTY,
hash: None,
needs_content: false,
},
)
.unwrap()
.expect("unique path");
tx.commit().unwrap();
}
drop(conn);
assert_eq!(
pending_content_count(dbp).unwrap(),
1,
"only the file awaiting extraction counts"
);
std::fs::remove_file(&p).ok();
}
#[test]
fn pending_content_count_on_a_missing_db_is_an_error() {
let missing = crate::testutil::scratch_dir("cli-missing").join("nope.sqlite");
assert!(pending_content_count(missing.to_str().unwrap()).is_err());
}
#[test]
fn index_size_breakdown_counts_rows() {
let p = tmp_path();

273
crates/quicksearch-core/src/config.rs
View file

@ -19,6 +19,7 @@ use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(default)]
#[derive(Default)]
pub struct Config {
pub paths: PathConfig,
pub indexing: IndexingConfig,
@ -192,7 +193,7 @@ impl Default for IndexingConfig {
fn default() -> Self {
IndexingConfig {
auto_index: true,
reindex_interval_minutes: 24 * 60,
reindex_interval_minutes: 60,
follow_symlinks: false,
include_hidden: false,
content_extensions: Vec::new(),
@ -293,19 +294,6 @@ impl Default for UiConfig {
}
}
impl Default for Config {
fn default() -> Self {
Config {
paths: PathConfig::default(),
indexing: IndexingConfig::default(),
processing: ProcessingConfig::default(),
search: SearchConfig::default(),
ui: UiConfig::default(),
security: SecurityConfig::default(),
source: None,
}
}
}
/// Directories and files excluded from a fresh index.
///
@ -451,8 +439,10 @@ impl Config {
cfg.source = Some(path.to_path_buf());
Ok(cfg)
} else {
let mut cfg = Config::default();
cfg.source = Some(path.to_path_buf());
let cfg = Config {
source: Some(path.to_path_buf()),
..Config::default()
};
cfg.save()?;
Ok(cfg)
}
@ -511,7 +501,7 @@ impl Config {
}
/// `indexing_paths` with the same resolution rules as
/// [`resolved_database_path`].
/// [`Config::resolved_database_path`].
pub fn resolved_indexing_paths(&self) -> Vec<PathBuf> {
self.paths
.indexing_paths
@ -567,11 +557,44 @@ pub fn content_allowed(path: &Path, cfg: &Config) -> bool {
}
}
/// Longest name folded without allocating. Comfortably above `NAME_MAX` on the
/// filesystems that matter (255 bytes), so the heap path is effectively dead
/// code kept for correctness rather than for use.
const FOLD_BUF: usize = 256;
/// Whether `pat` is a plain name with no glob syntax in it.
///
/// `\` is included even though it is not glob syntax everywhere: a pattern
/// containing one is routed to the path set rather than the component set, so
/// treating it as literal here would put it in the wrong place.
fn is_literal_name(pat: &str) -> bool {
!pat.contains(['*', '?', '[', ']', '{', '}', '/', '\\'])
}
/// Compiled ignore patterns, split by matching scope: patterns without a
/// path separator match any single path component; the rest match the full
/// path. Both use glob syntax.
#[derive(Debug)]
pub struct IgnoreSet {
/// Component patterns that are plain ASCII names — `.git`, `node_modules`,
/// `System Volume Information`. Held apart from `component` purely for
/// speed, and it is a large difference on Windows.
///
/// globset compiles a case-insensitive glob by giving up every fast path it
/// has: `Glob::literal`, `ext`, `prefix`, `suffix` and `basename_tokens` all
/// return `None` the moment `case_insensitive` is set, so every pattern
/// falls through to a `RegexSet` scan. Case-sensitively, `.git` and
/// `node_modules` compile to a hash lookup and `*.tmp` to an extension
/// lookup. That left Windows — which is case-insensitive *and* carries seven
/// extra default patterns — running a 12-pattern regex DFA over every single
/// directory entry, where Linux did five hash lookups.
///
/// Folded and compared as **ASCII**, matching [`PATH_COLLATION`]'s reasoning
/// exactly: SQLite's `NOCASE` and `LIKE` fold ASCII only, so the glob layer
/// agreeing with them is what keeps a path filter from disagreeing with the
/// ignore rules. Patterns that are not ASCII stay in `component` and keep
/// globset's Unicode folding, so nothing that matched before stops matching.
literal_components: std::collections::HashSet<String>,
component: globset::GlobSet,
path: globset::GlobSet,
empty: bool,
@ -579,6 +602,7 @@ pub struct IgnoreSet {
impl IgnoreSet {
pub fn compile(patterns: &[String]) -> Result<IgnoreSet, String> {
let mut literal_components = std::collections::HashSet::new();
let mut component = globset::GlobSetBuilder::new();
let mut path = globset::GlobSetBuilder::new();
for pat in patterns {
@ -605,6 +629,17 @@ impl IgnoreSet {
if pat.is_empty() {
continue;
}
// A plain ASCII name needs no glob machinery at all — see
// `literal_components`. Everything else, including every non-ASCII
// pattern, goes on to globset unchanged.
if pat.is_ascii() && is_literal_name(pat) {
literal_components.insert(if crate::platform::PATHS_ARE_CASE_INSENSITIVE {
pat.to_ascii_lowercase()
} else {
pat.to_string()
});
continue;
}
let glob = globset::GlobBuilder::new(pat)
.literal_separator(false)
// Match the filesystem's own rules, or `node_modules` fails to
@ -612,7 +647,7 @@ impl IgnoreSet {
// half of Windows compatibility on its own: `Candidate` folds
// `\` to `/` when matching, and backslash-as-escape is off
// wherever `\` is a separator.
.case_insensitive(cfg!(any(windows, target_os = "macos")))
.case_insensitive(crate::platform::PATHS_ARE_CASE_INSENSITIVE)
.build()
.map_err(|e| format!("invalid ignore pattern {:?}: {}", pat, e))?;
if pat.contains('/') || pat.contains('\\') {
@ -627,18 +662,54 @@ impl IgnoreSet {
let path = path
.build()
.map_err(|e| format!("compile ignore patterns: {}", e))?;
let empty = component.is_empty() && path.is_empty();
let empty = literal_components.is_empty() && component.is_empty() && path.is_empty();
Ok(IgnoreSet {
literal_components,
component,
path,
empty,
})
}
/// Whether `name` is one of the plain-name patterns, folded per platform.
///
/// Allocation-free for any name that fits [`FOLD_BUF`], which is every name
/// a real filesystem can produce. This runs on every directory entry the
/// walker sees, so it is the one place in the ignore path worth keeping off
/// the heap.
fn matches_literal(&self, name: &str) -> bool {
if self.literal_components.is_empty() {
return false;
}
if !crate::platform::PATHS_ARE_CASE_INSENSITIVE {
return self.literal_components.contains(name);
}
// Every stored literal is ASCII, and ASCII case folding maps ASCII to
// ASCII, so a name containing any non-ASCII byte cannot equal one.
if !name.is_ascii() {
return false;
}
if name.len() <= FOLD_BUF {
let mut buf = [0u8; FOLD_BUF];
let buf = &mut buf[..name.len()];
buf.copy_from_slice(name.as_bytes());
buf.make_ascii_lowercase();
// Lowercasing ASCII yields ASCII, which is always valid UTF-8.
let folded = std::str::from_utf8(buf).expect("ascii stays utf-8");
return self.literal_components.contains(folded);
}
self.literal_components.contains(&name.to_ascii_lowercase())
}
/// Match a single file/directory name. Used by the walker to prune
/// subtrees before descending.
pub fn matches_component(&self, name: &str) -> bool {
!self.empty && self.component.is_match(name)
if self.empty {
return false;
}
// The literal set answers almost every call — the defaults are all
// plain names — and answers it without touching the regex engine.
self.matches_literal(name) || self.component.is_match(name)
}
/// Match a path against the full-path patterns only. The path *and its
@ -669,9 +740,12 @@ impl IgnoreSet {
if self.matches_path_pattern(path) {
return true;
}
path.components().any(|c| {
matches!(c, std::path::Component::Normal(name)
if self.component.is_match(Path::new(name)))
path.components().any(|c| match c {
// Routed through `matches_component` rather than `self.component`
// directly, or the literal patterns would be invisible here and the
// watcher would index what the walker prunes.
std::path::Component::Normal(name) => self.matches_component(&name.to_string_lossy()),
_ => false,
})
}
@ -764,6 +838,42 @@ impl IndexWork {
pub fn scans_rows(&self) -> bool {
self.prune_scope || self.reconcile_content || self.restore_text || self.drop_text
}
/// The plan in one line, for the log entry that announces the scan.
///
/// Names what changed rather than what will happen to the rows: the
/// reader is someone asking why a run has not started walking yet, and
/// the answer they need is which edit of theirs caused it.
pub fn summary(&self) -> String {
let mut parts: Vec<String> = Vec::new();
if !self.drop_roots.is_empty() {
parts.push(format!(
"{} root(s) no longer indexed",
self.drop_roots.len()
));
}
if self.prune_scope {
parts.push("narrowed ignore or hidden-file rules".into());
}
if self.drop_aliases {
parts.push("symlinks no longer followed".into());
}
if self.reconcile_content {
parts.push("changed content extensions".into());
}
if self.restore_text {
parts.push("snippet text turned on".into());
}
if self.drop_text {
parts.push("snippet text turned off".into());
}
if parts.is_empty() {
// `touches_index` is false here, so no caller logs this; a
// placeholder beats an empty pair of parentheses if one ever does.
return "no stored rows affected".into();
}
parts.join("; ")
}
}
/// What running services must do after a config edit. Computed by the GUI
@ -788,8 +898,8 @@ pub struct ConfigActions {
/// duplicates are reported once). Nested roots are disallowed: with one
/// walker per root they would race for the same files and split progress
/// attribution. Comparison is on best-effort canonicalized paths (an
/// unresolvable root is compared as spelled) and is component-boundary
/// aware — `/a/bc` is not under `/a/b`.
/// unresolvable root is compared as spelled), component-wise per
/// [`crate::file_handling::UnreadableDirs::covers`].
pub fn nested_roots(roots: &[String]) -> Vec<(String, String)> {
let resolved: Vec<PathBuf> = roots
.iter()
@ -932,17 +1042,7 @@ mod tests {
use super::*;
fn tmp_dir() -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-config-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
fs::create_dir_all(&p).unwrap();
p
crate::testutil::scratch_dir("config")
}
#[test]
@ -1298,6 +1398,89 @@ mod tests {
);
}
/// Which patterns take the fast path, and that taking it changes nothing
/// observable. A plain name is matched whole — never as a prefix, a
/// substring or a wildcard — and only its case is allowed to vary.
#[test]
fn the_literal_fast_path_matches_whole_names_only() {
let literal = IgnoreSet::compile(&["node_modules".to_string()]).unwrap();
assert!(
!literal.literal_components.is_empty(),
"a plain name belongs on the fast path"
);
for globby in ["node_module?", "node_*", "*.tmp", "a[bc]d", "x{1,2}"] {
assert!(
IgnoreSet::compile(&[globby.to_string()])
.unwrap()
.literal_components
.is_empty(),
"{} has glob syntax and must stay with globset",
globby
);
}
assert!(literal.matches_component("node_modules"));
for cased in ["Node_Modules", "NODE_MODULES", "node_moduleS"] {
assert_eq!(
literal.matches_component(cased),
cfg!(any(windows, target_os = "macos")),
"only case may vary, and only where the filesystem says so: {}",
cased
);
}
for name in [
"node_modules_",
"_node_modules",
"nodemodules",
"node_module",
"src",
"",
] {
assert!(
!literal.matches_component(name),
"{} is not the ignored name",
name
);
}
}
/// A non-ASCII pattern keeps globset's Unicode folding rather than being
/// silently downgraded to the ASCII fast path.
#[test]
fn non_ascii_patterns_stay_on_the_glob_path() {
let set = IgnoreSet::compile(&["café".to_string()]).unwrap();
assert!(
set.literal_components.is_empty(),
"a non-ASCII name must not join the ASCII-folded set"
);
assert!(set.matches_component("café"));
assert!(!set.matches_component("cafe"));
}
/// Names longer than the stack fold buffer take the heap path, and must
/// come back with the same answer.
#[test]
fn overlong_names_still_fold_correctly() {
let long = "a".repeat(FOLD_BUF + 10);
let set = IgnoreSet::compile(std::slice::from_ref(&long)).unwrap();
assert!(set.matches_component(&long));
assert_eq!(
set.matches_component(&long.to_uppercase()),
cfg!(any(windows, target_os = "macos"))
);
assert!(!set.matches_component(&"a".repeat(FOLD_BUF + 9)));
}
/// Watcher events are matched by whole path, and the literal patterns have
/// to be visible on that route too — otherwise the watcher indexes exactly
/// what the walker prunes and the index churns every cycle.
#[test]
fn full_path_matching_sees_literal_component_patterns() {
let set = IgnoreSet::compile(&["node_modules".to_string()]).unwrap();
assert!(set.matches_path(Path::new("/home/me/proj/node_modules/pkg/index.js")));
assert!(!set.matches_path(Path::new("/home/me/proj/src/index.js")));
}
#[test]
fn default_ignore_patterns_cover_the_platform() {
let d = IndexingConfig::default().ignore_patterns;
@ -1374,8 +1557,10 @@ mod tests {
fn root_workers_round_trip() {
let dir = tmp_dir();
let path = dir.join("config.toml");
let mut cfg = Config::default();
cfg.source = Some(path.clone());
let mut cfg = Config {
source: Some(path.clone()),
..Config::default()
};
cfg.paths.indexing_paths = vec!["/data".into(), "/share".into()];
cfg.indexing.root_workers.insert("/share".into(), 24);
cfg.save().unwrap();
@ -1705,8 +1890,10 @@ mod tests {
fn watch_cap_warned_roots_round_trips() {
let dir = tmp_dir();
let path = dir.join("config.toml");
let mut cfg = Config::default();
cfg.source = Some(path.clone());
let mut cfg = Config {
source: Some(path.clone()),
..Config::default()
};
cfg.ui.watch_cap_warned_roots =
vec!["/media/ApolloStore".to_string(), "/media/GSSD".to_string()];
cfg.save().unwrap();
@ -1740,8 +1927,10 @@ mod tests {
fn fuzzy_max_edits_round_trips() {
let dir = tmp_dir();
let path = dir.join("config.toml");
let mut cfg = Config::default();
cfg.source = Some(path.clone());
let mut cfg = Config {
source: Some(path.clone()),
..Config::default()
};
cfg.search.fuzzy_max_edits = 4;
cfg.save().unwrap();

23
crates/quicksearch-core/src/content.rs
View file

@ -175,11 +175,9 @@ impl ContentPass {
self.stats.clone()
}
/// Join the workers and report whether every one finished cleanly.
///
/// The caller needs this for the same reason the walk does: a dead worker
/// and a finished worker both close the channel, so from the receiving end
/// they are indistinguishable.
/// Join the workers and report whether every one finished cleanly. Needed
/// for the same reason as [`crate::walk::ParallelWalk::finish`], which
/// states it.
pub fn finish(&mut self) -> bool {
// Dropping the receiver first releases any worker parked in `send`.
self.rx = None;
@ -357,20 +355,9 @@ mod tests {
use crate::file_handling::{extract_scope_prepare, store_extracted};
use crate::mime::FileType;
use std::path::{Path, PathBuf};
use std::time::UNIX_EPOCH;
/// A path that does not exist yet — the caller builds the tree under it.
fn tmp(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-content-{}-{}-{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
crate::testutil::scratch_dir(tag).join("tree")
}
/// A tree of `n` text files under `root/sub`, plus an index holding a

469
crates/quicksearch-core/src/coordinator.rs
View file

@ -35,7 +35,7 @@ use crate::config::{diff_actions, Config, IgnoreSet, IndexWork};
use crate::db;
use crate::extract::Registry;
use crate::incremental::apply_fs_event;
use crate::indexing::{ConfigChange, IndexingService, IndexingStatus};
use crate::indexing::{ConfigChange, IndexingService, IndexingStatus, PrepStep, ReconcileProgress};
use crate::scope::WorkCursor;
use crate::watcher::{FsEvent, WatchError, WatchFilters, Watcher, WatcherConfig};
@ -67,6 +67,28 @@ pub enum WatcherStatus {
Disabled { reason: WatchError },
}
/// A config reconciliation the coordinator applies between runs.
///
/// Separate from [`IndexingStatus`] on purpose. That enum is what every caller
/// reads to decide whether a full run owns the database, and this work is the
/// coordinator's own — putting it there would make the tick that performs it
/// believe it must keep off the file. On a large index it is minutes of
/// scanning that used to report nothing but `Idle`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReconcileState {
/// Scanning now; the counters move every slice.
Running(ReconcileProgress),
/// Finished within the last [`RECONCILE_SUMMARY_LINGER`].
///
/// The tail is the whole point: narrowing a filter on a small index is
/// over in a millisecond, and a display that only existed while the work
/// did would leave the user with no sign it ever happened.
Finished(ReconcileProgress),
}
/// How long a finished reconciliation keeps reporting itself.
pub const RECONCILE_SUMMARY_LINGER: Duration = Duration::from_secs(10);
/// One-stop poll surface for the GUI.
#[derive(Debug, Clone)]
pub struct IndexerState {
@ -78,8 +100,21 @@ pub struct IndexerState {
pub queued_events: usize,
/// Live-update health; see [`WatcherStatus`].
pub watcher: WatcherStatus,
/// The between-runs reconciliation, while it runs and briefly after.
///
/// A run's *own* reconciliation is not here — it is a step of the run, and
/// reads as [`IndexingStatus::Preparing`] with a
/// [`PrepStep::Reconciling`]. Both report the same
/// [`ReconcileProgress`], so the display does not have to care which one
/// it is looking at.
pub reconcile: Option<ReconcileState>,
}
// `ConfigChanged(Config)` dwarfs the unit variants, but these are sent one
// at a time down an mpsc channel at human cadence — a mode flip, a settings
// save. Boxing would trade a rare oversized move for an allocation on a path
// whose whole job is to be simple.
#[allow(clippy::large_enum_variant)]
enum CoordCmd {
SetMode(IndexMode),
ReindexNow,
@ -95,6 +130,34 @@ pub struct IndexCoordinator {
shared: Arc<Mutex<Shared>>,
handle: Mutex<Option<JoinHandle<()>>>,
stopped: AtomicBool,
/// How [`Self::shutdown`] reaches a reconciliation in progress; see
/// [`Inner::apply_work`].
reconcile_stop: Arc<ReconcileStop>,
}
/// The two halves of cutting the coordinator's reconciliation short — the pair
/// [`db::InterruptSlot`] describes.
///
/// A command cannot do it: the thread that would read the command is the
/// thread inside the scan. Closing the window used to wait for both.
#[derive(Default)]
struct ReconcileStop {
cancel: AtomicBool,
interrupt: db::InterruptSlot,
}
impl ReconcileStop {
/// Flag first, then interrupt: the flag is what stops the *next*
/// statement, and setting it after the interrupt would leave a window in
/// which the pass starts one more.
fn stop(&self) {
self.cancel.store(true, Ordering::SeqCst);
db::interrupt(&self.interrupt);
}
fn cancelled(&self) -> bool {
self.cancel.load(Ordering::SeqCst)
}
}
/// State mirrored out of the coordinator thread for `state()`.
@ -103,6 +166,7 @@ struct Shared {
last_full_index: Option<u64>,
queued_events: usize,
watcher: WatcherStatus,
reconcile: Option<ReconcileState>,
}
impl IndexCoordinator {
@ -130,12 +194,15 @@ impl IndexCoordinator {
last_full_index: None,
queued_events: 0,
watcher: WatcherStatus::Off,
reconcile: None,
}));
let reconcile_stop = Arc::new(ReconcileStop::default());
let mut inner = Inner {
config,
indexing: indexing.clone(),
shared: shared.clone(),
reconcile_stop: reconcile_stop.clone(),
event_tx,
event_rx,
watcher: None,
@ -147,6 +214,8 @@ impl IndexCoordinator {
pending_since: None,
needs_full_run: false,
pending_work: None,
reconcile_done: None,
reconcile_cut_short: false,
saw_running: false,
write_conn: None,
ignore: Arc::new(IgnoreSet::compile(&[]).expect("empty ignore set")),
@ -167,6 +236,7 @@ impl IndexCoordinator {
shared,
handle: Mutex::new(Some(handle)),
stopped: AtomicBool::new(false),
reconcile_stop,
})
}
@ -178,6 +248,7 @@ impl IndexCoordinator {
last_full_index: shared.last_full_index,
queued_events: shared.queued_events,
watcher: shared.watcher.clone(),
reconcile: shared.reconcile,
}
}
@ -223,15 +294,41 @@ impl IndexCoordinator {
/// Stop the watcher, any running index pass, and the coordinator
/// thread. Idempotent; usable from a signal handler through an Arc.
///
/// Cancelling the reconciliation comes *before* the command, because the
/// command is read by the thread the scan is running on: a pass part-way
/// through a large index would otherwise hold this join — and with it the
/// window close that called it — for as long as the scan had left.
pub fn shutdown(&self) {
if self.stopped.swap(true, Ordering::SeqCst) {
return;
}
self.reconcile_stop.stop();
let _ = self.cmd_tx.send(CoordCmd::Shutdown);
if let Some(handle) = self.handle.lock().unwrap().take() {
let _ = handle.join();
}
}
/// Whether a configuration change is being applied to the index right
/// now, by either of the two places that can be doing it.
///
/// For a caller deciding whether to warn before quitting: an abandoned
/// pass leaves entries the user excluded still in the index until the next
/// indexing run redoes it.
pub fn reconciling(&self) -> bool {
// The lock is released before the service is asked, so this never
// holds two of them at once.
let between_runs = self.shared.lock().unwrap().reconcile;
matches!(between_runs, Some(ReconcileState::Running(_)))
|| matches!(
self.indexing.get_status(),
IndexingStatus::Preparing {
step: PrepStep::Reconciling(_),
..
}
)
}
}
impl Drop for IndexCoordinator {
@ -281,9 +378,7 @@ fn collapse_pending_removals(pending: &mut HashMap<PathBuf, FsEvent>) {
.filter(|(_, ev)| is_removal(ev))
.map(|(p, _)| p.clone())
.collect();
// `Path::ancestors` walks whole components, so `/a/bc` is never treated as
// living under `/a/b` — the rule `remove_tree` and `UnreadableDirs::covers`
// also use.
// Component-wise containment, per `UnreadableDirs::covers`.
pending.retain(|path, ev| {
!is_removal(ev) || !path.ancestors().skip(1).any(|a| removed.contains(a))
});
@ -303,6 +398,9 @@ struct Inner {
config: Config,
indexing: Arc<IndexingService>,
shared: Arc<Mutex<Shared>>,
/// Read inside the reconciliation, set from the thread that shuts this
/// one down; see [`ReconcileStop`].
reconcile_stop: Arc<ReconcileStop>,
event_tx: mpsc::Sender<FsEvent>,
event_rx: mpsc::Receiver<FsEvent>,
watcher: Option<Watcher>,
@ -325,6 +423,11 @@ struct Inner {
/// manual mode, whereas a config change the user just made is acted on in
/// either mode.
pending_work: Option<WorkCursor>,
/// The last reconciliation to finish, and when. Published until it is
/// [`RECONCILE_SUMMARY_LINGER`] old; see [`ReconcileState::Finished`].
reconcile_done: Option<(ReconcileProgress, Instant)>,
/// A reconciliation was abandoned part-way; read by [`Inner::teardown`].
reconcile_cut_short: bool,
/// A start was requested; set false once the service reports running,
/// so idle-after-running transitions are detectable.
saw_running: bool,
@ -407,8 +510,11 @@ impl Inner {
CoordCmd::RebuildIndex => {
let db = self.db_path();
self.write_conn = None;
// Nothing to reconcile against once the file is gone.
// Nothing to reconcile against once the file is gone — and
// nothing to report about what was reconciled in the index
// that is about to stop existing.
self.pending_work = None;
self.reconcile_done = None;
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
crate::log_warn!("coordinator: rebuild: {}", e);
}
@ -423,6 +529,7 @@ impl Inner {
self.enter_manual_stopped();
self.write_conn = None;
self.pending_work = None;
self.reconcile_done = None;
let db = self.db_path();
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
crate::log_warn!("coordinator: clear index: {}", e);
@ -439,7 +546,8 @@ impl Inner {
let status = self.indexing.get_status();
match status {
IndexingStatus::Running { .. }
IndexingStatus::Preparing { .. }
| IndexingStatus::Running { .. }
| IndexingStatus::Stopping
| IndexingStatus::Optimizing => {
// Single-writer rule: never touch the DB while a full run
@ -545,15 +653,34 @@ impl Inner {
}
};
let mut cursor = self.pending_work.take().expect("caller checked");
let outcome = crate::scope::advance(
&mut conn,
&self.config,
&self.registry,
&mut cursor,
Instant::now() + crate::scope::SLICE,
);
let outcome = {
// Held only for the slice: the handle names whatever statement
// this connection is running, and outside `advance` that is
// nothing this cancellation has any business ending.
let _armed = db::InterruptGuard::arm(&self.reconcile_stop.interrupt, &conn);
crate::scope::advance(
&mut conn,
&self.config,
&self.registry,
&mut cursor,
Instant::now() + crate::scope::SLICE,
&self.reconcile_stop.cancel,
)
};
self.write_conn = Some(conn);
if let Err(e) = outcome {
// A cancelled statement fails like any other, and telling the two
// apart from a stringified error is guesswork — so ask the flag we
// set ourselves. Shutting down mid-scan is not a fault to report.
if self.reconcile_stop.cancelled() {
self.reconcile_cut_short = true;
crate::log_info!(
"configuration change interrupted after {} index entries; \
the next indexing run starts it again",
cursor.progress().examined
);
return;
}
// Abandoned rather than retried: the cursor is already dropped,
// and a database error that persists would otherwise spin this
// loop for the life of the process. The next full run reconciles
@ -566,9 +693,34 @@ impl Inner {
return;
}
if !cursor.done() {
// Nothing is recorded for a pass that stopped early, cancelled or
// not: the stale record is what makes the next run redo it.
self.pending_work = Some(cursor);
return;
}
// Held for the linger so the display outlives the work: on a small
// index this whole pass is over between two frames.
self.reconcile_done = Some((cursor.progress(), Instant::now()));
// Record what the pass just brought the index into line with. Only
// here, on the path where the work finished and nothing errored: the
// stale record is what makes the next full run redo an abandoned
// reconcile, and it is the documented backstop for the error path
// above. Without this stamp a completed prune left the index still
// describing itself with the old configuration, so every later run
// rescanned every row to re-apply work already done — the silent wait
// a large index spends before its walk starts.
if let Some(conn) = self.write_conn.as_ref() {
// The same spelling a run records: canonicalized, and sorted into
// one string by `config_validation_entries`.
let roots: Vec<String> = self
.config
.normalized_indexing_paths()
.into_iter()
.collect();
if let Err(e) = IndexingService::stamp_reconciled(conn, &self.config, &roots) {
crate::log_warn!("coordinator: record reconciled configuration: {}", e);
}
}
if cursor.deleted > 0 || cursor.recontented > 0 {
crate::log_info!(
"configuration change: {} index entries removed, {} re-examined \
@ -617,7 +769,7 @@ impl Inner {
/// create-then-delete ends with it absent, because the upsert half consults
/// the filesystem and finds nothing there.
fn apply_pending(&mut self) {
let conn = match self.ensure_write_conn() {
let mut conn = match self.ensure_write_conn() {
Ok(conn) => conn,
Err(e) => {
// Missing or stale DB: incremental can't help, rebuild.
@ -629,8 +781,6 @@ impl Inner {
return;
}
};
// Borrow dance: pull the connection out while applying.
let mut conn = conn;
let deadline = Instant::now() + APPLY_BUDGET;
let chunk = self.config.processing.batch_size.max(1);
@ -642,7 +792,11 @@ impl Inner {
.collect();
for batch in removals.chunks(chunk) {
if let Err(e) = crate::incremental::remove_paths(&mut conn, batch, chunk) {
crate::log_warn!("coordinator: remove: {}", e);
// The batch leaves `pending` either way — replaying a write
// that just failed, once a tick forever, is the worse
// failure. A full run is what recovers the rows instead.
crate::log_warn!("coordinator: remove: {}; scheduling full run", e);
self.needs_full_run = true;
}
for path in batch {
self.pending.remove(path);
@ -666,7 +820,11 @@ impl Inner {
if let Err(e) =
apply_fs_event(&mut conn, &ev, &self.config, &self.ignore, &self.registry)
{
crate::log_warn!("coordinator: apply {:?}: {}", ev, e);
// Same reasoning as the removal half above: the event is
// already out of `pending`, so a full run is the only
// thing that still picks the file up.
crate::log_warn!("coordinator: apply {:?}: {}; scheduling full run", ev, e);
self.needs_full_run = true;
}
if Instant::now() >= deadline {
break;
@ -918,23 +1076,51 @@ impl Inner {
Ok(())
}
/// Re-read the stamp the last completed full run left behind.
///
/// A failure to open is deliberately *not* published as `None`. Only a
/// successful read means "never indexed", and `periodic_due` answers that
/// by starting a full run immediately — so treating a locked or
/// contended database as "never" would schedule a fresh run every tick
/// for as long as the condition lasts. Keeping the previous value leaves
/// the schedule where it was until a read succeeds.
fn refresh_last_full_index(&self) {
let last = db::open_existing(&self.db_path(), false)
.ok()
.and_then(|conn| db::repo::get_last_full_index(&conn));
self.shared.lock().unwrap().last_full_index = last;
match db::open_existing(&self.db_path(), false) {
Ok(conn) => {
let last = db::repo::get_last_full_index(&conn);
self.shared.lock().unwrap().last_full_index = last;
}
Err(e) => crate::log_warn!("coordinator: last-full-index unreadable: {}", e),
}
}
fn publish(&self) {
fn publish(&mut self) {
let reconcile = match &self.pending_work {
Some(cursor) => Some(ReconcileState::Running(cursor.progress())),
None => {
// The tail ages out here rather than in `tick`, which returns
// early for the whole length of a run; this runs every turn of
// the loop, so it expires within a second of its deadline
// whatever else is going on.
let now = Instant::now();
self.reconcile_done = self
.reconcile_done
.filter(|(_, at)| summary_is_fresh(*at, now));
self.reconcile_done
.map(|(progress, _)| ReconcileState::Finished(progress))
}
};
let mut shared = self.shared.lock().unwrap();
shared.mode = self.mode;
shared.queued_events = self.pending.len();
shared.reconcile = reconcile;
}
/// Must stay fast: it runs (transitively) on the GUI thread during
/// window close, and desktops show a "terminate this application?"
/// dialog after a few unresponsive seconds. Signal, don't wait — an
/// abandoned run is safe under WAL.
/// abandoned run is safe under WAL, and so is an abandoned reconcile:
/// nothing recorded it, so the next run does it again.
fn teardown(mut self) {
self.stop_watcher();
let status = self.indexing.get_status();
@ -946,10 +1132,18 @@ impl Inner {
// window during an optimize pass would wait out a rewrite of the
// whole index. The interrupted VACUUM rolls back, and the next
// run's checkpoints land the log.
self.indexing.cancel_optimizing();
self.indexing.cancel_db_work();
}
// Unfinished either way it can end: still holding its cursor, or
// abandoned mid-statement by the cancellation.
let cut_short = self.reconcile_cut_short || self.pending_work.is_some();
if let Some(conn) = self.write_conn.take() {
if idle {
// A reconciliation cut short is the one idle case that must not
// checkpoint. It can have written a great deal of WAL — deleting
// a root's rows is all log — and a TRUNCATE checkpoint of it is
// more of exactly the wait the cancellation just spared the user.
// Dropping is safe: WAL keeps the log and the next run lands it.
if idle && !cut_short {
db::repo::checkpoint_and_close(conn);
}
// Otherwise just drop: a TRUNCATE checkpoint would block
@ -958,13 +1152,15 @@ impl Inner {
}
}
fn now_unix() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
/// Whether a reconciliation that finished at `at` is still worth reporting.
///
/// Its own function so the rule can be tested without waiting the linger out.
fn summary_is_fresh(at: Instant, now: Instant) -> bool {
now.duration_since(at) < RECONCILE_SUMMARY_LINGER
}
use crate::log::now_unix;
#[cfg(test)]
mod tests {
use super::*;
@ -988,17 +1184,10 @@ mod tests {
impl Fixture {
fn new(auto: bool) -> Fixture {
let stamp = format!(
"{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let dir = std::env::temp_dir().join(format!("qs-coord-{}", stamp));
let scratch = crate::testutil::scratch_dir("coord");
let dir = scratch.join("tree");
std::fs::create_dir_all(&dir).unwrap();
let db = std::env::temp_dir().join(format!("qs-coord-{}.sqlite", stamp));
let db = scratch.join("index.sqlite");
let mut config = Config::default();
config.paths.indexing_paths = vec![dir.to_string_lossy().into_owned()];
config.paths.database_path = db.to_string_lossy().into_owned();
@ -1014,6 +1203,21 @@ mod tests {
Err(_) => -1,
}
}
/// What a run starting now would still find to reconcile.
fn outstanding_work(&self, config: &Config) -> IndexWork {
crate::scope::outstanding_work(&self.db.to_string_lossy(), config).unwrap()
}
fn stored_value(&self, key: &str) -> Option<String> {
let conn = db::open_existing(&self.db.to_string_lossy(), false).unwrap();
conn.query_row(
"SELECT value FROM config_validation WHERE key = ?1",
[key],
|r| r.get(0),
)
.ok()
}
}
impl Drop for Fixture {
@ -1042,6 +1246,51 @@ mod tests {
coord.shutdown();
}
/// Closing the window during a prune must not wait the prune out. The
/// thread that reads the shutdown command is the thread inside the scan,
/// so before this the join sat behind however many rows were left — on a
/// large index, minutes of it, with the window still on screen and the
/// desktop offering to kill the app.
#[test]
fn shutdown_during_a_prune_does_not_wait_for_it() {
let f = Fixture::new(false);
// Enough rows that the scan cannot plausibly finish between the
// config landing and the shutdown two lines later.
for i in 0..400 {
std::fs::write(f.dir.join(format!("f{}.log", i)), "dropped content").unwrap();
}
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
coord.reindex_now();
wait_for("initial run", Duration::from_secs(60), || {
coord.state().last_full_index.is_some() && f.file_count() == 401
});
let mut narrowed = f.config.clone();
narrowed.indexing.ignore_patterns.push("*.log".into());
// One row per page, so the scan is as many statements as there are
// rows: the shape a huge index has, at a size a test can afford.
narrowed.processing.batch_size = 1;
coord.apply_config(narrowed.clone());
let asked = Instant::now();
coord.shutdown();
let took = asked.elapsed();
assert!(
took < Duration::from_secs(5),
"shutdown waited {:?} for the prune",
took
);
// And it really did leave the work unfinished rather than racing
// through it: the stored record still describes the old settings, so
// the next run derives the same plan and applies it.
assert!(
f.outstanding_work(&narrowed).touches_index(),
"the prune ran to completion, so this proves nothing about waiting"
);
}
/// A narrowed filter is applied to the stored index without a prompt and
/// without a run — including in manual mode, where the user has said not
/// to index anything. Deleting entries they just excluded is not indexing
@ -1085,6 +1334,144 @@ mod tests {
coord.shutdown();
}
/// A prune that finishes must record what it reconciled against, or every
/// later run re-derives the same plan and rescans every row under every
/// root to redo work already done. On a multi-million-file index that
/// rescan is minutes of silence before the walk starts — the whole reason
/// indexing looked hung after a prune.
#[test]
fn a_completed_prune_records_what_it_reconciled() {
let f = Fixture::new(false);
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
coord.reindex_now();
wait_for("initial run", Duration::from_secs(20), || {
let s = coord.state();
s.last_full_index.is_some() && f.file_count() == 2
});
let mut narrowed = f.config.clone();
narrowed.indexing.ignore_patterns.push("*.log".into());
assert!(
f.outstanding_work(&narrowed).touches_index(),
"the edit must be one the index does not yet reflect"
);
coord.apply_config(narrowed.clone());
wait_for(
"the log entry to be pruned",
Duration::from_secs(20),
|| f.file_count() == 1,
);
// The prune and the stamp are two steps of one tick; the count above
// can be observed between them.
wait_for("the prune to be recorded", Duration::from_secs(20), || {
!f.outstanding_work(&narrowed).touches_index()
});
assert!(
f.outstanding_work(&narrowed).is_empty(),
"a run starting now has nothing left to reconcile"
);
coord.shutdown();
}
/// A prune of a two-file index is one transaction, over well inside a
/// frame. Reporting it only while it runs would mean the user changes a
/// setting and sees nothing at all — so the result outlives the work.
#[test]
fn a_finished_prune_keeps_reporting_itself_for_a_while() {
let f = Fixture::new(false);
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
coord.reindex_now();
wait_for("initial run", Duration::from_secs(20), || {
let s = coord.state();
s.last_full_index.is_some() && f.file_count() == 2
});
let mut narrowed = f.config.clone();
narrowed.indexing.ignore_patterns.push("*.log".into());
coord.apply_config(narrowed);
wait_for("the summary to appear", Duration::from_secs(20), || {
matches!(coord.state().reconcile, Some(ReconcileState::Finished(_)))
});
let Some(ReconcileState::Finished(progress)) = coord.state().reconcile else {
panic!("the summary went away as soon as it arrived");
};
assert_eq!(progress.deleted, 1, "the log entry, and only it");
coord.shutdown();
}
/// And it does go away: a summary is a report of what just happened, not
/// a state the app sits in. The rule is tested directly rather than by
/// sleeping out the linger.
#[test]
fn a_summary_stops_being_fresh_once_the_linger_is_up() {
let now = Instant::now();
assert!(summary_is_fresh(now, now));
assert!(summary_is_fresh(
now,
now + RECONCILE_SUMMARY_LINGER - Duration::from_millis(1)
));
assert!(!summary_is_fresh(now, now + RECONCILE_SUMMARY_LINGER));
assert!(!summary_is_fresh(now, now + RECONCILE_SUMMARY_LINGER * 60));
}
/// The three settings no scan can satisfy stay at the values the index was
/// *built* with. A prune that stamped them would clear a rebuild the user
/// was prompted for and declined — and would clear it from an unrelated
/// later edit at that.
#[test]
fn a_prune_never_records_settings_only_a_rebuild_can_satisfy() {
let f = Fixture::new(false);
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
coord.reindex_now();
wait_for("initial run", Duration::from_secs(20), || {
let s = coord.state();
s.last_full_index.is_some() && f.file_count() == 2
});
let built_with = f.stored_value("hash_length").unwrap();
// The user changes the hash length and declines the rebuild it needs.
// The coordinator takes the config either way — a wipe is the caller's
// decision — so from here its copy disagrees with the stored hashes,
// and nothing short of a rebuild can make them agree.
let mut rebuilt = f.config.clone();
rebuilt.processing.hash_length = f.config.processing.hash_length * 2;
coord.apply_config(rebuilt.clone());
std::thread::sleep(Duration::from_millis(500));
// Then they edit a filter. This one *is* reconcilable, and the pass
// that applies it stamps — with the coordinator's config, whose hash
// length is the one the index does not have.
let mut narrowed = rebuilt.clone();
narrowed.indexing.ignore_patterns.push("*.log".into());
coord.apply_config(narrowed);
wait_for(
"the log entry to be pruned",
Duration::from_secs(20),
|| f.file_count() == 1,
);
std::thread::sleep(Duration::from_millis(500));
assert_eq!(
f.stored_value("hash_length"),
Some(built_with),
"the recorded hash length still describes the stored hashes, so the \
rebuild prompt survives an unrelated prune"
);
coord.shutdown();
}
/// Widening it does the opposite: nothing is deleted, and the walk that
/// finds the newly-eligible files starts on its own, returning manual mode
/// to stopped afterwards the way `reindex_now` does.

66
crates/quicksearch-core/src/db/mod.rs
View file

@ -8,6 +8,10 @@
//! size, `clear`) instead use [`open::open_existing`], which never creates or
//! wipes — a tokenizer difference or stale version is an error, not data loss.
use std::sync::Mutex;
use rusqlite::{Connection, InterruptHandle};
pub mod key;
pub mod open;
pub mod repo;
@ -18,3 +22,65 @@ pub use open::{
index_needs_rebuild, open_existing, open_or_recreate, verify_process_key,
CURRENT_SCHEMA_VERSION, KEY_MISMATCH_PREFIX,
};
/// A shared slot holding the interrupt handle of whatever long statement is
/// running, so another thread can cut it short.
///
/// # Why a flag is not enough
///
/// SQLite's own interrupt is the only way out of a statement already in
/// flight: a `DELETE` over a whole root's range, an FTS merge or a VACUUM
/// answer to nothing else, and on a large index each can run for minutes. A
/// flag can only stop the *next* statement from starting.
///
/// So cancelling anything long is two halves — the flag, which prevents the
/// next statement, and this, which ends the current one — and a caller that
/// wants a bounded wait needs both. Every user of this type is one of those
/// pairs: [`crate::scope::advance`] and its `cancel` argument,
/// `coordinator::ReconcileStop`, and the stop flag alongside
/// `IndexingService::cancel_db_work`. This is the reference statement of the
/// rule; those sites record only what it means locally.
///
/// One further consequence, which is why [`InterruptGuard`] exists rather than
/// a set/clear pair: an interrupted statement fails like any other and cannot
/// be told apart from a real failure by its error message. The flag the
/// canceller set is the only reliable answer, so callers re-read it on the
/// error path before deciding whether they were cancelled or broke.
pub type InterruptSlot = Mutex<Option<InterruptHandle>>;
/// Publish `conn`'s interrupt handle in `slot` for as long as this lives.
///
/// The guard, rather than a set/clear pair, because every path out matters:
/// a handle left behind after an early `?` would let a later interrupt —
/// aimed at the next long statement, or at a VACUUM — land on whatever that
/// connection happens to be running by then.
pub struct InterruptGuard<'a> {
slot: &'a InterruptSlot,
}
impl<'a> InterruptGuard<'a> {
pub fn arm(slot: &'a InterruptSlot, conn: &Connection) -> InterruptGuard<'a> {
if let Ok(mut held) = slot.lock() {
*held = Some(conn.get_interrupt_handle());
}
InterruptGuard { slot }
}
}
impl Drop for InterruptGuard<'_> {
fn drop(&mut self) {
if let Ok(mut held) = self.slot.lock() {
*held = None;
}
}
}
/// Interrupt the statement `slot` holds a handle for, if there is one.
/// A no-op otherwise, and on a connection that has since closed.
pub fn interrupt(slot: &InterruptSlot) {
if let Ok(held) = slot.lock() {
if let Some(handle) = held.as_ref() {
handle.interrupt();
}
}
}

38
crates/quicksearch-core/src/db/open.rs
View file

@ -7,11 +7,11 @@
//! migrations.
//!
//! The tradeoff: users pay a re-index cost every time the shipped schema
//! changes. Our indexing is fast (see `bench/`) and schema changes are
//! rare in practice, so the code-complexity cost of maintaining real
//! migration paths wasn't worth it. A single `open_or_recreate` replaces
//! what used to be version detection + tokenizer-drift FTS rebuild +
//! legacy-layout recovery, all of which ultimately wiped anyway.
//! changes. Indexing is fast and schema changes are rare in practice, so the
//! code-complexity cost of maintaining real migration paths wasn't worth it.
//! A single `open_or_recreate` replaces what used to be version detection +
//! tokenizer-drift FTS rebuild + legacy-layout recovery, all of which
//! ultimately wiped anyway.
use std::path::Path;
@ -414,10 +414,7 @@ fn apply_current_schema(conn: &Connection, tokenizer: &str) -> Result<(), String
conn.execute_batch(&fts)
.map_err(|e| format!("Failed to create searchabletext: {}", e))?;
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let now = crate::log::now_unix();
let effective = effective_tokenizer(tokenizer);
conn.execute(
"INSERT INTO schema_info(key, value) VALUES ('version', ?1), ('created_at', ?2), ('tokenize', ?3)",
@ -437,16 +434,7 @@ mod tests {
use super::*;
fn tmp_db_path() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-test-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
crate::testutil::scratch_dir("open").join("index.sqlite")
}
#[test]
@ -548,7 +536,7 @@ mod tests {
assert_eq!(count, 0);
// New columns should exist (just prepare the SELECT — an
// unknown column name would parse-error here).
let _ = conn
conn
.query_row(
"SELECT basic_state, content_state, type, mime FROM files LIMIT 0",
[],
@ -667,15 +655,7 @@ mod tests {
fn open_or_recreate_creates_missing_parent_dirs() {
// Fresh installs point at ~/.local/share/quicksearch/… which
// doesn't exist yet; the owner open must create it.
let mut dir = std::env::temp_dir();
dir.push(format!(
"qs-mkdir-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let dir = crate::testutil::scratch_dir("mkdir");
let db = dir.join("nested/deeper/index.sqlite");
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
drop(conn);

274
crates/quicksearch-core/src/db/repo.rs
View file

@ -23,6 +23,57 @@ pub const STATE_DONE: i64 = 1;
pub const STATE_FAILED: i64 = 2;
pub const STATE_NA: i64 = 3;
/// `prepare_cached` + `execute`, returning the affected row count.
///
/// `what` is a closure rather than a string so the message is built only on
/// the error path — several callers run this per property or per file, where
/// formatting a description that is then discarded is the dominant cost.
///
/// Takes `&Connection`; a `Transaction` derefs to one, so the same call works
/// inside and outside a transaction.
fn exec(
conn: &Connection,
sql: &str,
params: impl rusqlite::Params,
what: impl FnOnce() -> String,
) -> Result<usize, String> {
conn.prepare_cached(sql)
.and_then(|mut stmt| stmt.execute(params))
.map_err(|e| format!("{}: {}", what(), e))
}
/// Move a file to a content state that means "nothing is currently wrong with
/// this row", clearing any failure record along with it.
///
/// The pair is one operation, not two that happen to be adjacent: a row that
/// succeeded, that nothing can extract, or that has been queued for another
/// attempt must not keep a `failed_files` entry explaining a failure that no
/// longer applies. `list-failed` reads that table directly, so a stale entry
/// is a file the user is told is broken after it stopped being broken.
///
/// [`set_content_failed`] is deliberately not routed through here — it is the
/// one transition that *writes* a failure record.
fn set_state_clearing_failure(
tx: &Transaction<'_>,
file_id: i64,
state: i64,
transition: &'static str,
) -> Result<(), String> {
exec(
tx,
"UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2",
params![state, file_id],
|| format!("{} content_state {}", transition, file_id),
)?;
exec(
tx,
"DELETE FROM failed_files WHERE file_id = ?1",
params![file_id],
|| format!("clear failed_files {}", file_id),
)?;
Ok(())
}
/// Everything needed to insert a fresh file row.
#[derive(Debug, Clone)]
pub struct NewFile<'a> {
@ -162,18 +213,22 @@ pub fn set_content_done(
remove_content_for_id(tx, file_id)?;
for (k, v) in properties {
tx.prepare_cached("INSERT INTO properties(file_id, key, value) VALUES (?1, ?2, ?3)")
.and_then(|mut stmt| stmt.execute(params![file_id, k, v]))
.map_err(|e| format!("insert property {}={}: {}", k, v, e))?;
exec(
tx,
"INSERT INTO properties(file_id, key, value) VALUES (?1, ?2, ?3)",
params![file_id, k, v],
|| format!("insert property {}={}", k, v),
)?;
}
let props_blob = encode_properties_for_fts(properties);
// Contentless FTS5 still accepts values on INSERT — the tokenizer needs
// them — it simply doesn't persist the raw column values.
tx.prepare_cached(
exec(
tx,
"INSERT INTO searchabletext(rowid, name, text, properties) VALUES (?1, ?2, ?3, ?4)",
)
.and_then(|mut stmt| stmt.execute(params![file_id, name, text, props_blob]))
.map_err(|e| format!("insert FTS row {}: {}", file_id, e))?;
params![file_id, name, text, props_blob],
|| format!("insert FTS row {}", file_id),
)?;
// Skip the compressed sidecar when: the config disables snippet storage
// outright, or there's no body text (e.g. an image whose extractor
@ -182,21 +237,15 @@ pub fn set_content_done(
if store_text && !text.is_empty() {
let compressed = zstd::encode_all(text.as_bytes(), ZSTD_LEVEL)
.map_err(|e| format!("zstd encode for file {}: {}", file_id, e))?;
tx.prepare_cached(
exec(
tx,
"INSERT INTO documents_text(file_id, text_zstd, text_len) VALUES (?1, ?2, ?3)",
)
.and_then(|mut stmt| stmt.execute(params![file_id, compressed, text.len() as i64]))
.map_err(|e| format!("insert documents_text {}: {}", file_id, e))?;
params![file_id, compressed, text.len() as i64],
|| format!("insert documents_text {}", file_id),
)?;
}
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
.and_then(|mut stmt| stmt.execute(params![STATE_DONE, file_id]))
.map_err(|e| format!("update content_state DONE {}: {}", file_id, e))?;
// Clear any prior failed-file record.
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
.and_then(|mut stmt| stmt.execute(params![file_id]))
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
Ok(())
set_state_clearing_failure(tx, file_id, STATE_DONE, "update DONE")
}
/// zstd level tuned for extracted-text prose. Level 3 hits ~3-5× on English
@ -208,31 +257,26 @@ const ZSTD_LEVEL: i32 = 3;
/// Mark a file's content extraction as failed. Keeps the basic row in place.
pub fn set_content_failed(tx: &Transaction<'_>, file_id: i64, reason: &str) -> Result<(), String> {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0) as i64;
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = ?2 WHERE id = ?3")
.and_then(|mut stmt| stmt.execute(params![STATE_FAILED, reason, file_id]))
.map_err(|e| format!("update content_state FAILED {}: {}", file_id, e))?;
tx.prepare_cached(
let now = crate::log::now_unix() as i64;
exec(
tx,
"UPDATE files SET content_state = ?1, failure_msg = ?2 WHERE id = ?3",
params![STATE_FAILED, reason, file_id],
|| format!("update content_state FAILED {}", file_id),
)?;
exec(
tx,
"INSERT OR REPLACE INTO failed_files(file_id, reason, ts) VALUES (?1, ?2, ?3)",
)
.and_then(|mut stmt| stmt.execute(params![file_id, reason, now]))
.map_err(|e| format!("insert failed_files {}: {}", file_id, e))?;
params![file_id, reason, now],
|| format!("insert failed_files {}", file_id),
)?;
Ok(())
}
/// Mark content extraction as not applicable (e.g. binary format we don't
/// support). The file row still contributes to filename search.
pub fn set_content_na(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
.and_then(|mut stmt| stmt.execute(params![STATE_NA, file_id]))
.map_err(|e| format!("update content_state NA {}: {}", file_id, e))?;
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
.and_then(|mut stmt| stmt.execute(params![file_id]))
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
Ok(())
set_state_clearing_failure(tx, file_id, STATE_NA, "update NA")
}
/// Delete a file row by path, keeping FTS in sync. Returns whether a row was
@ -276,15 +320,16 @@ pub fn delete_subtree(tx: &Transaction<'_>, lo: &str, hi: &str) -> Result<usize,
(SELECT id FROM files WHERE path >= ?1 AND path < ?2)",
table, key
);
tx.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params![lo, hi]))
.map_err(|e| format!("delete {} under {}: {}", table, lo, e))?;
exec(tx, &sql, params![lo, hi], || {
format!("delete {} under {}", table, lo)
})?;
}
let removed = tx
.prepare_cached("DELETE FROM files WHERE path >= ?1 AND path < ?2")
.and_then(|mut stmt| stmt.execute(params![lo, hi]))
.map_err(|e| format!("delete files under {}: {}", lo, e))?;
Ok(removed)
exec(
tx,
"DELETE FROM files WHERE path >= ?1 AND path < ?2",
params![lo, hi],
|| format!("delete files under {}", lo),
)
}
/// Delete every row whose path falls in *none* of `ranges`, keeping the four
@ -326,14 +371,14 @@ pub fn delete_outside_ranges(
"DELETE FROM {} WHERE {} IN (SELECT id FROM files WHERE {})",
table, key, predicate
);
tx.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params_from_iter(bounds.iter())))
.map_err(|e| format!("delete {} outside the roots: {}", table, e))?;
exec(tx, &sql, params_from_iter(bounds.iter()), || {
format!("delete {} outside the roots", table)
})?;
}
let sql = format!("DELETE FROM files WHERE {}", predicate);
tx.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params_from_iter(bounds.iter())))
.map_err(|e| format!("delete files outside the roots: {}", e))
exec(tx, &sql, params_from_iter(bounds.iter()), || {
"delete files outside the roots".to_string()
})
}
/// The tables a file id owns, in the order they must be cleared: everything
@ -381,15 +426,14 @@ pub fn delete_ids(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, String> {
}
for (table, key) in DEPENDENT_TABLES {
let sql = format!("DELETE FROM {} WHERE {} IN ({})", table, key, placeholders);
tx.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
.map_err(|e| format!("delete {} for {} ids: {}", table, chunk.len(), e))?;
exec(tx, &sql, params_from_iter(chunk.iter()), || {
format!("delete {} for {} ids", table, chunk.len())
})?;
}
let sql = format!("DELETE FROM files WHERE id IN ({})", placeholders);
removed += tx
.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
.map_err(|e| format!("delete {} file rows: {}", chunk.len(), e))?;
removed += exec(tx, &sql, params_from_iter(chunk.iter()), || {
format!("delete {} file rows", chunk.len())
})?;
}
Ok(removed)
}
@ -420,8 +464,11 @@ pub fn dir_rows(
Ok(out)
}
/// A row the content pass has yet to extract: `(id, name, path, mime)`.
pub type PendingContentRow = (i64, String, String, Option<String>);
/// One page of rows still awaiting content extraction under `cursor`'s range,
/// as `(id, name, path, mime)` ordered by id.
/// ordered by id.
///
/// Keyset, not `OFFSET`: `id > cursor.last_id` means each page is an index
/// seek rather than a re-scan of everything already handed out, and — because
@ -434,7 +481,7 @@ pub fn pending_content_page(
cursor: &crate::file_handling::ExtractCursor,
max_size: i64,
limit: i64,
) -> Result<Vec<(i64, String, String, Option<String>)>, String> {
) -> Result<Vec<PendingContentRow>, String> {
let mut stmt = conn
.prepare_cached(
"SELECT id, name, path, mime FROM files
@ -472,6 +519,17 @@ pub struct ScopeRow {
pub content_state: i64,
}
/// How many files the index holds.
///
/// A denominator for a scan that pages over all of them, so the cost is paid
/// once against work measured in pages. SQLite answers it from the smallest
/// index rather than the table, so it is a key scan and not a row fetch.
pub fn row_count(conn: &Connection) -> Result<usize, String> {
conn.query_row("SELECT COUNT(*) FROM files", [], |r| r.get::<_, i64>(0))
.map(|n| n.max(0) as usize)
.map_err(|e| format!("count indexed files: {}", e))
}
/// One page of rows whose path is `> after` and `< hi`, in path order.
///
/// Keyset on `path` rather than on `id`: the range is already a seek on
@ -529,10 +587,9 @@ pub fn drop_stored_text(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, Stri
"DELETE FROM documents_text WHERE file_id IN ({})",
placeholders
);
removed += tx
.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
.map_err(|e| format!("drop stored text for {} ids: {}", chunk.len(), e))?;
removed += exec(tx, &sql, params_from_iter(chunk.iter()), || {
format!("drop stored text for {} ids", chunk.len())
})?;
}
Ok(removed)
}
@ -545,13 +602,7 @@ pub fn drop_stored_text(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, Stri
/// and hash from a fresh stat), but its content must be produced again.
pub fn reset_content_pending(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
remove_content_for_id(tx, file_id)?;
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
.and_then(|mut stmt| stmt.execute(params![STATE_PENDING, file_id]))
.map_err(|e| format!("reset content_state pending {}: {}", file_id, e))?;
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
.and_then(|mut stmt| stmt.execute(params![file_id]))
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
Ok(())
set_state_clearing_failure(tx, file_id, STATE_PENDING, "reset pending")
}
/// The stored mtime for one exact path, or `None` if it isn't indexed.
@ -621,9 +672,9 @@ pub fn remove_content_for_id(tx: &Transaction<'_>, file_id: i64) -> Result<(), S
("properties", "file_id"),
] {
let sql = format!("DELETE FROM {} WHERE {} = ?1", table, key);
tx.prepare_cached(&sql)
.and_then(|mut stmt| stmt.execute(params![file_id]))
.map_err(|e| format!("delete {} for {}: {}", table, file_id, e))?;
exec(tx, &sql, params![file_id], || {
format!("delete {} for {}", table, file_id)
})?;
}
Ok(())
}
@ -779,6 +830,70 @@ pub fn set_last_full_index(conn: &Connection, ts: u64) -> Result<(), String> {
Ok(())
}
/// `schema_info` key holding one root's last known file count.
fn walk_count_key(root: &str) -> String {
format!("walk_count:{}", root)
}
/// How many files the last clean walk of `root` reported.
///
/// The progress bar's denominator. Absent means this root has never been walked
/// to completion — a new root, a fresh index, or a run that was stopped — and
/// the caller falls back to counting the tree as it goes.
///
/// Deliberately last run's *file* count rather than a tree-entry count: it is
/// the same quantity the numerator counts, where an entry count includes
/// directories and ignore-pruned subtrees and so reads high (over 1.6x on a home
/// directory). See [`crate::indexing::RootProgress::walk_denominator`].
pub fn get_root_walk_count(conn: &Connection, root: &str) -> Option<usize> {
conn.query_row(
"SELECT value FROM schema_info WHERE key = ?1",
params![walk_count_key(root)],
|r| r.get::<_, String>(0),
)
.optional()
.ok()
.flatten()
.and_then(|v| v.parse().ok())
}
/// Record `n` as `root`'s file count, for the next run's progress bar.
///
/// Written only after a walk that finished cleanly: a stopped or partially
/// unreadable walk saw only part of the tree, and storing its count would leave
/// every later run dividing by a number that is too small.
pub fn set_root_walk_count(conn: &Connection, root: &str, n: usize) -> Result<(), String> {
conn.execute(
"INSERT OR REPLACE INTO schema_info(key, value) VALUES (?1, ?2)",
params![walk_count_key(root), n.to_string()],
)
.map_err(|e| format!("write walk count for {}: {}", root, e))?;
Ok(())
}
/// Forget the stored counts of roots that are no longer configured.
///
/// Without this, `schema_info` accumulates a row per root the user ever had —
/// harmless in size, but it also means a root removed and later re-added would
/// start from a count that predates everything that happened in between.
pub fn prune_root_walk_counts(conn: &Connection, keep: &[String]) -> Result<(), String> {
let keep: std::collections::HashSet<String> = keep.iter().map(|r| walk_count_key(r)).collect();
let mut stmt = conn
.prepare("SELECT key FROM schema_info WHERE key LIKE 'walk_count:%'")
.map_err(|e| format!("read walk counts: {}", e))?;
let stored: Vec<String> = stmt
.query_map([], |r| r.get::<_, String>(0))
.map_err(|e| format!("read walk counts: {}", e))?
.filter_map(|r| r.ok())
.collect();
drop(stmt);
for key in stored.iter().filter(|k| !keep.contains(*k)) {
conn.execute("DELETE FROM schema_info WHERE key = ?1", params![key])
.map_err(|e| format!("drop walk count {}: {}", key, e))?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
@ -788,16 +903,7 @@ mod tests {
use crate::db::open_or_recreate;
fn tmp_path() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-repo-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
crate::testutil::scratch_dir("repo").join("index.sqlite")
}
#[test]

7
crates/quicksearch-core/src/db/schema.rs
View file

@ -1,5 +1,8 @@
//! SQL strings for the current schema. Versioned; [`migrate`](super::migrate)
//! drives the upgrade path.
//! SQL strings for the current schema.
//!
//! Versioned by [`super::open::CURRENT_SCHEMA_VERSION`], but there is no
//! upgrade path: a database written under any other version is wiped and
//! recreated from [`SCHEMA_CURRENT`]. See [`super::open`] for why.
/// Pragmas applied on every writable connection open.
///

373
crates/quicksearch-core/src/document_extraction.rs
View file

@ -1,373 +0,0 @@
use std::ffi::OsString;
use std::fs::File;
use std::io::{BufReader, Read};
use quick_xml::events::Event;
use quick_xml::Reader;
use zip::ZipArchive;
/// Extract text from DOCX files by parsing the word/document.xml
pub fn extract_text_from_docx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut document_xml = archive.by_name("word/document.xml")?;
let mut content = String::new();
document_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut text_content = String::new();
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
if e.name().as_ref() == b"w:t" {
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
text_content.push_str(&e.unescape()?.into_owned());
}
Ok(Event::End(ref e)) => {
if e.name().as_ref() == b"w:t" {
in_text = false;
} else if e.name().as_ref() == b"w:p" {
text_content.push('\n');
}
}
Ok(Event::Eof) => break,
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
_ => {}
}
buf.clear();
}
Ok(text_content)
}
/// Extract text from XLSX files by parsing worksheet XML files
pub fn extract_text_from_xlsx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut text_content = String::new();
// First, read shared strings if they exist
let mut shared_strings = Vec::new();
if let Ok(mut shared_strings_xml) = archive.by_name("xl/sharedStrings.xml") {
let mut content = String::new();
shared_strings_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
if e.name().as_ref() == b"t" {
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
shared_strings.push(e.unescape()?.into_owned());
}
Ok(Event::End(ref e)) => {
if e.name().as_ref() == b"t" {
in_text = false;
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
}
// Read worksheets
for i in 0..archive.len() {
let file_name = archive.by_index(i)?.name().to_string();
if file_name.starts_with("xl/worksheets/sheet") && file_name.ends_with(".xml") {
let mut sheet_xml = archive.by_index(i)?;
let mut content = String::new();
sheet_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut buf = Vec::new();
let mut in_cell = false;
let mut cell_type = String::new();
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
if e.name().as_ref() == b"c" {
in_cell = true;
cell_type.clear();
for attr in e.attributes() {
let attr = attr?;
if attr.key.as_ref() == b"t" {
cell_type = String::from_utf8_lossy(&attr.value).to_string();
}
}
} else if e.name().as_ref() == b"v" && in_cell {
// Value element
}
}
Ok(Event::Text(e)) if in_cell => {
let text = e.unescape()?.into_owned();
if cell_type == "s" {
// Shared string reference
if let Ok(index) = text.parse::<usize>() {
if index < shared_strings.len() {
text_content.push_str(&shared_strings[index]);
text_content.push(' ');
}
}
} else {
text_content.push_str(&text);
text_content.push(' ');
}
}
Ok(Event::End(ref e)) => {
if e.name().as_ref() == b"c" {
in_cell = false;
} else if e.name().as_ref() == b"row" {
text_content.push('\n');
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
}
}
Ok(text_content)
}
/// Extract text from PPTX files by parsing slide XML files
pub fn extract_text_from_pptx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut text_content = String::new();
// Read all slide files
for i in 0..archive.len() {
let file_name = archive.by_index(i)?.name().to_string();
if file_name.starts_with("ppt/slides/slide") && file_name.ends_with(".xml") {
let mut slide_xml = archive.by_index(i)?;
let mut content = String::new();
slide_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
if e.name().as_ref() == b"a:t" {
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
text_content.push_str(&e.unescape()?.into_owned());
}
Ok(Event::End(ref e)) => {
if e.name().as_ref() == b"a:t" {
in_text = false;
} else if e.name().as_ref() == b"a:p" {
text_content.push('\n');
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
text_content.push_str("\n--- New Slide ---\n");
}
}
Ok(text_content)
}
/// Extract text from ODT files (OpenDocument Text)
pub fn extract_text_from_odt(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut content_xml = archive.by_name("content.xml")?;
let mut content = String::new();
content_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut text_content = String::new();
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p"
|| name.as_ref() == b"text:h"
|| name.as_ref() == b"text:span"
{
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
text_content.push_str(&e.unescape()?.into_owned());
}
Ok(Event::End(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p" || name.as_ref() == b"text:h" {
text_content.push('\n');
in_text = false;
} else if name.as_ref() == b"text:span" {
in_text = false;
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
Ok(text_content)
}
/// Extract text from ODP files (OpenDocument Presentation)
pub fn extract_text_from_odp(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut content_xml = archive.by_name("content.xml")?;
let mut content = String::new();
content_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut text_content = String::new();
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p"
|| name.as_ref() == b"text:h"
|| name.as_ref() == b"text:span"
{
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
text_content.push_str(&e.unescape()?.into_owned());
}
Ok(Event::End(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p" || name.as_ref() == b"text:h" {
text_content.push('\n');
in_text = false;
} else if name.as_ref() == b"text:span" {
in_text = false;
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
Ok(text_content)
}
/// Extract text from ODS files (OpenDocument Spreadsheet)
pub fn extract_text_from_ods(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
let file = File::open(file_path)?;
let mut archive = ZipArchive::new(BufReader::new(file))?;
let mut content_xml = archive.by_name("content.xml")?;
let mut content = String::new();
content_xml.read_to_string(&mut content)?;
let mut reader = Reader::from_str(&content);
reader.trim_text(true);
let mut text_content = String::new();
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p" || name.as_ref() == b"text:span" {
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
text_content.push_str(&e.unescape()?.into_owned());
text_content.push(' ');
}
Ok(Event::End(ref e)) => {
let name = e.name();
if name.as_ref() == b"text:p" {
text_content.push('\n');
in_text = false;
} else if name.as_ref() == b"text:span" {
in_text = false;
}
}
Ok(Event::Eof) => break,
_ => {}
}
buf.clear();
}
Ok(text_content)
}
/// Extract text from various document formats
pub fn extract_document_text(
file_path: &OsString,
extension: &str,
) -> Result<String, Box<dyn std::error::Error>> {
match extension {
"docx" => extract_text_from_docx(file_path),
"doc" => {
// DOC format is binary and complex to parse without external tools
// For now, return an empty result
Ok(String::new())
}
"xlsx" => extract_text_from_xlsx(file_path),
"xls" => {
// XLS format is binary and complex to parse without external tools
Ok(String::new())
}
"pptx" => extract_text_from_pptx(file_path),
"ppt" => {
// PPT format is binary and complex to parse without external tools
Ok(String::new())
}
"odt" => extract_text_from_odt(file_path),
"odp" => extract_text_from_odp(file_path),
"ods" => extract_text_from_ods(file_path),
_ => Ok(String::new()),
}
}

2
crates/quicksearch-core/src/extract/image.rs
View file

@ -1,4 +1,4 @@
//! Image metadata extraction via [`kamadak_exif`]. Reads EXIF tags (camera
//! Image metadata extraction via `kamadak-exif`. Reads EXIF tags (camera
//! make/model, date, GPS, dimensions) into properties. `text` is left empty
//! — this extractor does not OCR.

1
crates/quicksearch-core/src/extract/mod.rs
View file

@ -18,6 +18,7 @@ use std::path::Path;
pub mod audio;
pub mod image;
pub mod office;
pub mod ole;
pub mod pdf;
pub mod plaintext;
pub mod rtf;

505
crates/quicksearch-core/src/extract/office.rs
View file

@ -1,13 +1,27 @@
//! Office document extraction: DOCX, XLSX, PPTX, ODT, ODP, ODS.
//!
//! Delegates to [`crate::document_extraction`] — a single entry point based
//! on file extension rather than MIME. We translate the MIME to the
//! extension expected by that module.
//! All six are zip containers holding XML, and five of the six want the same
//! thing from it: the character data of a few named elements, with a newline
//! where a paragraph closes. That shape lives in [`collect_xml_text`], driven
//! by a per-format [`TextSpec`], so there is one event loop rather than one
//! per format.
//!
//! XLSX is the exception and keeps its own two loops: its text is not in the
//! sheet at all but in a shared-string table the cells index into, which is a
//! different machine, not a different table of element names.
//!
//! Dispatch is by file extension rather than MIME. `.docm` carries the same
//! MIME as `.docx` but needs the same reader, and the extension is what
//! distinguishes them.
use std::ffi::OsString;
use std::error::Error;
use std::fs::File;
use std::io::{BufReader, Read, Seek};
use std::path::Path;
use crate::document_extraction::extract_document_text;
use quick_xml::events::Event;
use quick_xml::Reader;
use zip::ZipArchive;
use super::{ExtractError, ExtractedContent, Extractor};
@ -28,22 +42,293 @@ fn mime_to_ext(mime: &str) -> Option<&'static str> {
}
}
// ---------------------------------------------------------------------------
// The shared XML text walk
// ---------------------------------------------------------------------------
/// Which elements of a format's XML carry text, and where paragraphs end.
///
/// `text` and `breaks` are matched independently on a closing tag: an element
/// can be in both (ODF's `text:p` both holds text and ends a paragraph), in
/// only one (`w:p` breaks but holds nothing directly), or in `text` alone
/// (`text:span`, which ends a run without ending the line).
struct TextSpec {
/// Elements whose character data is body text.
text: &'static [&'static [u8]],
/// Elements that close a paragraph, emitting `'\n'`.
breaks: &'static [&'static [u8]],
/// Emitted after each text run. Spreadsheets separate cells with it;
/// prose formats leave it `None` so runs within a paragraph stay joined.
separator: Option<char>,
}
const DOCX: TextSpec = TextSpec {
text: &[b"w:t"],
breaks: &[b"w:p"],
separator: None,
};
const PPTX: TextSpec = TextSpec {
text: &[b"a:t"],
breaks: &[b"a:p"],
separator: None,
};
/// ODT and ODP are the same format as far as text extraction is concerned —
/// both are ODF prose with headings, paragraphs and spans.
const ODF_TEXT: TextSpec = TextSpec {
text: &[b"text:p", b"text:h", b"text:span"],
breaks: &[b"text:p", b"text:h"],
separator: None,
};
const ODF_SHEET: TextSpec = TextSpec {
text: &[b"text:p", b"text:span"],
breaks: &[b"text:p"],
separator: Some(' '),
};
/// Append the text `spec` selects out of `xml` to `out`.
///
/// `in_text` is a flag rather than a depth count, which means a closing
/// `</text:span>` ends the run even though its enclosing `<text:p>` is still
/// open. That is how every one of the six extractors this replaces behaved.
fn collect_xml_text(xml: &str, spec: &TextSpec, out: &mut String) -> Result<(), Box<dyn Error>> {
let mut reader = Reader::from_str(xml);
reader.trim_text(true);
let mut buf = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) => {
if spec.text.contains(&e.name().as_ref()) {
in_text = true;
}
}
Ok(Event::Text(e)) if in_text => {
out.push_str(&e.unescape()?);
if let Some(sep) = spec.separator {
out.push(sep);
}
}
Ok(Event::End(ref e)) => {
let name = e.name();
if spec.text.contains(&name.as_ref()) {
in_text = false;
}
if spec.breaks.contains(&name.as_ref()) {
out.push('\n');
}
}
Ok(Event::Eof) => break,
// Propagated rather than ignored. Five of the six loops this
// replaces had no error arm at all, so a malformed member sent
// them round the loop on an error the reader kept re-reporting
// without advancing — a hang on a file the user merely happened
// to have on disk.
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
_ => {}
}
buf.clear();
}
Ok(())
}
// ---------------------------------------------------------------------------
// Container access
// ---------------------------------------------------------------------------
type Archive = ZipArchive<BufReader<File>>;
fn open_container(path: &Path) -> Result<Archive, Box<dyn Error>> {
Ok(ZipArchive::new(BufReader::new(File::open(path)?))?)
}
/// One member's bytes as a string.
fn member_text<R: Read + Seek>(
archive: &mut ZipArchive<R>,
name: &str,
) -> Result<String, Box<dyn Error>> {
let mut member = archive.by_name(name)?;
let mut body = String::new();
member.read_to_string(&mut body)?;
Ok(body)
}
/// Names of the `.xml` members under `prefix`, in archive order.
///
/// Indexed rather than taken from `file_names()`, which iterates a hash map:
/// slide order is the archive's order, and hashing it would shuffle the
/// slides of every presentation.
fn xml_members_under<R: Read + Seek>(
archive: &mut ZipArchive<R>,
prefix: &str,
) -> Result<Vec<String>, Box<dyn Error>> {
let mut names = Vec::new();
for i in 0..archive.len() {
let name = archive.by_index(i)?.name().to_string();
if name.starts_with(prefix) && name.ends_with(".xml") {
names.push(name);
}
}
Ok(names)
}
/// A format whose whole text lives in one member under one spec.
fn single_member(
path: &Path,
member: &str,
spec: &TextSpec,
) -> Result<String, Box<dyn Error>> {
let mut archive = open_container(path)?;
let xml = member_text(&mut archive, member)?;
let mut out = String::new();
collect_xml_text(&xml, spec, &mut out)?;
Ok(out)
}
fn extract_pptx(path: &Path) -> Result<String, Box<dyn Error>> {
let mut archive = open_container(path)?;
let mut out = String::new();
for name in xml_members_under(&mut archive, "ppt/slides/slide")? {
let xml = member_text(&mut archive, &name)?;
collect_xml_text(&xml, &PPTX, &mut out)?;
out.push_str("\n--- New Slide ---\n");
}
Ok(out)
}
// ---------------------------------------------------------------------------
// XLSX: shared strings plus cells
// ---------------------------------------------------------------------------
/// The workbook's shared-string table, in index order. Absent or unreadable
/// is not an error: a sheet of nothing but numbers has no table at all.
fn shared_strings<R: Read + Seek>(archive: &mut ZipArchive<R>) -> Vec<String> {
let Ok(xml) = member_text(archive, "xl/sharedStrings.xml") else {
return Vec::new();
};
let mut reader = Reader::from_str(&xml);
reader.trim_text(true);
let mut buf = Vec::new();
let mut strings = Vec::new();
let mut in_text = false;
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) if e.name().as_ref() == b"t" => in_text = true,
Ok(Event::Text(e)) if in_text => match e.unescape() {
Ok(s) => strings.push(s.into_owned()),
Err(_) => return strings,
},
Ok(Event::End(ref e)) if e.name().as_ref() == b"t" => in_text = false,
Ok(Event::Eof) | Err(_) => break,
_ => {}
}
buf.clear();
}
strings
}
/// One worksheet's cells. A `t="s"` cell holds an index into `strings`
/// rather than text of its own; every other type holds its value inline.
fn collect_sheet(xml: &str, strings: &[String], out: &mut String) -> Result<(), Box<dyn Error>> {
let mut reader = Reader::from_str(xml);
reader.trim_text(true);
let mut buf = Vec::new();
let mut in_cell = false;
let mut cell_type = String::new();
loop {
match reader.read_event_into(&mut buf) {
Ok(Event::Start(ref e)) if e.name().as_ref() == b"c" => {
in_cell = true;
cell_type.clear();
for attr in e.attributes() {
let attr = attr?;
if attr.key.as_ref() == b"t" {
cell_type = String::from_utf8_lossy(&attr.value).to_string();
}
}
}
Ok(Event::Text(e)) if in_cell => {
let text = e.unescape()?;
if cell_type == "s" {
// A shared-string reference. An index past the end of the
// table is a corrupt workbook, not something to guess at.
if let Some(s) = text.parse::<usize>().ok().and_then(|i| strings.get(i)) {
out.push_str(s);
out.push(' ');
}
} else {
out.push_str(&text);
out.push(' ');
}
}
Ok(Event::End(ref e)) => {
let name = e.name();
if name.as_ref() == b"c" {
in_cell = false;
} else if name.as_ref() == b"row" {
out.push('\n');
}
}
Ok(Event::Eof) => break,
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
_ => {}
}
buf.clear();
}
Ok(())
}
fn extract_xlsx(path: &Path) -> Result<String, Box<dyn Error>> {
let mut archive = open_container(path)?;
let strings = shared_strings(&mut archive);
let mut out = String::new();
for name in xml_members_under(&mut archive, "xl/worksheets/sheet")? {
let xml = member_text(&mut archive, &name)?;
collect_sheet(&xml, &strings, &mut out)?;
}
Ok(out)
}
// ---------------------------------------------------------------------------
// Dispatch
// ---------------------------------------------------------------------------
/// Extract text from an office document, chosen by lowercase extension.
///
/// An extension nothing here handles yields empty text rather than an error:
/// the caller reaches this only for a MIME [`mime_to_ext`] claimed, so an
/// unrecognized extension means the file was named unlike its type.
fn extract_document_text(path: &Path, extension: &str) -> Result<String, Box<dyn Error>> {
match extension {
"docx" => single_member(path, "word/document.xml", &DOCX),
"xlsx" => extract_xlsx(path),
"pptx" => extract_pptx(path),
"odt" | "odp" => single_member(path, "content.xml", &ODF_TEXT),
"ods" => single_member(path, "content.xml", &ODF_SHEET),
// Pre-2007 binary formats: a different container entirely.
"doc" | "xls" | "ppt" => super::ole::extract_ole_text(path, extension),
_ => Ok(String::new()),
}
}
impl Extractor for OfficeExtractor {
fn supports(&self, mime: &str) -> bool {
mime_to_ext(mime).is_some()
}
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> {
// Recompute the extension from MIME at call time by asking the path.
// Using the filesystem extension directly is more robust than round-
// tripping through MIME: `.docm` has the same MIME as `.docx` but
// `extract_document_text` looks up by extension.
// The extension from the path, not from the MIME: `.docm` and `.docx`
// share a MIME but the dispatch above is by extension.
let ext = path
.extension()
.and_then(|s| s.to_str())
.map(|s| s.to_ascii_lowercase())
.unwrap_or_default();
let text = extract_document_text(&OsString::from(path.as_os_str()), &ext)
let text = extract_document_text(path, &ext)
.map_err(|e| format!("office extractor {}: {}", path.display(), e))?;
Ok(ExtractedContent::with_text(text))
}
@ -52,6 +337,7 @@ impl Extractor for OfficeExtractor {
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
#[test]
fn supports_docx_and_friends() {
@ -66,4 +352,203 @@ mod tests {
}
assert!(!e.supports("image/png"));
}
/// A zip container holding `members`, written to a scratch file.
fn container(tag: &str, ext: &str, members: &[(&str, &str)]) -> std::path::PathBuf {
let path = crate::testutil::scratch_dir(tag).join(format!("doc.{ext}"));
let file = File::create(&path).unwrap();
let mut zip = zip::ZipWriter::new(file);
for (name, body) in members {
zip.start_file(*name, zip::write::FileOptions::default())
.unwrap();
zip.write_all(body.as_bytes()).unwrap();
}
zip.finish().unwrap();
path
}
const DOCX_BODY: &str = "<w:document><w:body>\
<w:p><w:r><w:t>Hello</w:t></w:r><w:r><w:t>world</w:t></w:r></w:p>\
<w:p><w:r><w:t>Second</w:t></w:r></w:p>\
</w:body></w:document>";
const PPTX_SLIDE: &str = "<p:sld><p:cSld><p:spTree><p:sp><p:txBody>\
<a:p><a:r><a:t>Title</a:t></a:r></a:p>\
<a:p><a:r><a:t>Body</a:t></a:r></a:p>\
</p:txBody></p:sp></p:spTree></p:cSld></p:sld>";
const ODT_BODY: &str = "<office:document-content><office:body><office:text>\
<text:h>Heading</text:h>\
<text:p>Para<text:span>span</text:span></text:p>\
</office:text></office:body></office:document-content>";
const ODS_BODY: &str = "<office:document-content><office:body><office:spreadsheet>\
<table:table><table:table-row>\
<table:table-cell><text:p>A1</text:p></table:table-cell>\
<table:table-cell><text:p>B1</text:p></table:table-cell>\
</table:table-row></table:table>\
</office:spreadsheet></office:body></office:document-content>";
const XLSX_SHARED: &str = "<sst><si><t>Shared</t></si><si><t>Second</t></si></sst>";
const XLSX_SHEET: &str = "<worksheet><sheetData>\
<row><c t=\"s\"><v>0</v></c><c t=\"n\"><v>42</v></c></row>\
<row><c t=\"s\"><v>1</v></c></row>\
</sheetData></worksheet>";
// The golden set. These strings were recorded from the six hand-written
// extractors this module replaced, so they pin the rewrite to exactly
// what shipped rather than to what it ought to have produced.
#[test]
fn docx() {
let p = container("docx", "docx", &[("word/document.xml", DOCX_BODY)]);
assert_eq!(
extract_document_text(&p, "docx").unwrap(),
"Helloworld\nSecond\n"
);
}
#[test]
fn pptx_marks_each_slide_and_keeps_archive_order() {
let p = container(
"pptx",
"pptx",
&[
("ppt/slides/slide1.xml", PPTX_SLIDE),
("ppt/slides/slide2.xml", PPTX_SLIDE),
],
);
assert_eq!(
extract_document_text(&p, "pptx").unwrap(),
"Title\nBody\n\n--- New Slide ---\nTitle\nBody\n\n--- New Slide ---\n"
);
}
#[test]
fn odt_and_odp_are_the_same_extraction() {
let odt = container("odt", "odt", &[("content.xml", ODT_BODY)]);
let odp = container("odp", "odp", &[("content.xml", ODT_BODY)]);
assert_eq!(
extract_document_text(&odt, "odt").unwrap(),
"Heading\nParaspan\n"
);
assert_eq!(
extract_document_text(&odt, "odt").unwrap(),
extract_document_text(&odp, "odp").unwrap(),
);
}
#[test]
fn ods_separates_cells_with_a_space() {
let p = container("ods", "ods", &[("content.xml", ODS_BODY)]);
assert_eq!(extract_document_text(&p, "ods").unwrap(), "A1 \nB1 \n");
}
#[test]
fn xlsx_resolves_shared_strings() {
let p = container(
"xlsx",
"xlsx",
&[
("xl/sharedStrings.xml", XLSX_SHARED),
("xl/worksheets/sheet1.xml", XLSX_SHEET),
],
);
assert_eq!(
extract_document_text(&p, "xlsx").unwrap(),
"Shared 42 \nSecond \n"
);
}
#[test]
fn an_extension_nothing_handles_is_empty() {
let p = container("none", "bin", &[("whatever", "x")]);
assert_eq!(extract_document_text(&p, "zzz").unwrap(), "");
}
/// A workbook of pure numbers has no shared-string table. Its absence is
/// normal, not a failure.
#[test]
fn xlsx_without_a_shared_string_table_still_reads_its_cells() {
let p = container(
"xlsx-nosst",
"xlsx",
&[("xl/worksheets/sheet1.xml", "<worksheet><sheetData>\
<row><c t=\"n\"><v>7</v></c></row></sheetData></worksheet>")],
);
assert_eq!(extract_document_text(&p, "xlsx").unwrap(), "7 \n");
}
/// A shared-string index past the end of the table is dropped rather than
/// panicking on the slice.
#[test]
fn an_out_of_range_shared_string_index_is_dropped() {
let p = container(
"xlsx-oob",
"xlsx",
&[
("xl/sharedStrings.xml", "<sst><si><t>only</t></si></sst>"),
(
"xl/worksheets/sheet1.xml",
"<worksheet><sheetData><row>\
<c t=\"s\"><v>0</v></c><c t=\"s\"><v>99</v></c>\
</row></sheetData></worksheet>",
),
],
);
assert_eq!(extract_document_text(&p, "xlsx").unwrap(), "only \n");
}
/// Malformed XML is an error, not a hang. Five of the six extractors this
/// replaced had no error arm, so the reader re-reported the same failure
/// forever without advancing.
///
/// An undefined entity inside a text run is the cheapest way to reach that
/// arm, and it is a real shape: a document written by a tool that emitted
/// HTML entities into OOXML.
#[test]
fn malformed_xml_returns_an_error_rather_than_looping() {
for (ext, member, body) in [
("docx", "word/document.xml", "<w:t>bad &nonsuch; entity</w:t>"),
("odt", "content.xml", "<text:p>bad &nonsuch; entity</text:p>"),
("ods", "content.xml", "<text:p>bad &nonsuch; entity</text:p>"),
("pptx", "ppt/slides/slide1.xml", "<a:t>bad &nonsuch; entity</a:t>"),
] {
let p = container(&format!("bad-{ext}"), ext, &[(member, body)]);
assert!(
extract_document_text(&p, ext).is_err(),
"{ext} should report malformed XML"
);
}
}
/// Mismatched tags are caught too — quick_xml checks closing names, and
/// that error now reaches the caller instead of being swallowed.
#[test]
fn mismatched_tags_are_an_error() {
let p = container(
"mismatch",
"docx",
&[("word/document.xml", "<w:body><w:t>x</w:body>")],
);
assert!(extract_document_text(&p, "docx").is_err());
}
/// A container missing the member the format is defined by.
#[test]
fn a_missing_member_is_an_error() {
let p = container("empty", "docx", &[("unrelated.xml", "<x/>")]);
assert!(extract_document_text(&p, "docx").is_err());
}
/// Not a zip file at all — the shape a truncated download or a
/// misidentified file arrives in.
#[test]
fn a_non_container_is_an_error() {
let dir = crate::testutil::scratch_dir("notzip");
let p = dir.join("doc.docx");
crate::testutil::touch(&p, b"this is not a zip archive");
assert!(extract_document_text(&p, "docx").is_err());
}
}

1097
crates/quicksearch-core/src/extract/ole.rs Normal file
View file

File diff suppressed because it is too large Load diff

13
crates/quicksearch-core/src/extract/plaintext.rs
View file

@ -124,17 +124,8 @@ mod tests {
use super::*;
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-plaintext-{}-{}-{}.txt",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::write(&p, body).unwrap();
let p = crate::testutil::scratch_dir(tag).join("sample.txt");
crate::testutil::touch(&p, body);
p
}

21
crates/quicksearch-core/src/extract/rtf.rs
View file

@ -58,17 +58,8 @@ mod tests {
use super::*;
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-rtf-{}-{}-{}.rtf",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::write(&p, body).unwrap();
let p = crate::testutil::scratch_dir(tag).join("sample.rtf");
crate::testutil::touch(&p, body);
p
}
@ -98,7 +89,13 @@ mod tests {
fn malformed_input_errors_and_names_the_file() {
let p = tmp("broken", br"{\rtf1 truncated");
let err = RtfExtractor.extract(&p).unwrap_err();
assert!(err.contains("qs-rtf-broken"), "must name the file: {}", err);
// The path itself, not a fixed prefix: this is the message a user sees
// in `list-failed`, and it is useless without naming the file.
assert!(
err.contains(&p.display().to_string()),
"must name the file: {}",
err
);
std::fs::remove_file(&p).ok();
}

364
crates/quicksearch-core/src/file_handling.rs
View file

@ -1,10 +1,10 @@
use std::collections::HashMap;
use std::fs::File;
use std::io::Read;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
// Only the Unix entry-count path shells out; Windows walks the tree directly.
use std::collections::HashMap;
#[cfg(unix)]
use std::process::{Command, Stdio};
use std::time::UNIX_EPOCH;
@ -67,15 +67,12 @@ pub(crate) fn path_to_db_string(path: &Path) -> String {
}
}
/// Canonicalize a root string for storage/comparison, stripping the Windows
/// UNC prefix. Multi-root strings (newline-joined) fail canonicalize and
/// pass through verbatim, which still compares consistently.
///
/// The UNC strip is [`path_to_db_string`]'s, not a hand-rolled one: chopping
/// four characters would turn `\\?\UNC\server\share` into
/// `UNC\server\share`, which is not a path — and no longer looks like a
/// share, so the root would walk with the local thread count instead of the
/// network one.
/// Canonicalize a root string for storage/comparison, via
/// [`path_to_db_string`] rather than a hand-rolled prefix strip — the extra
/// consequence here being that a mis-stripped share stops looking like one, so
/// the root would walk with the local thread count instead of the network one.
/// Multi-root strings (newline-joined) fail canonicalize and pass through
/// verbatim, which still compares consistently.
///
/// This is the spelling `files.path` rows are prefixed with, so it is also the
/// form roots must be compared in: `~/docs` and `/home/me/docs` name one root
@ -196,8 +193,19 @@ impl UnreadableDirs {
/// Whether `path` lies under a directory the walk failed to read, and so
/// must not be treated as deleted.
///
/// Compares by path component, not by string prefix: `/a/bc` does not
/// live under `/a/b`.
/// # Containment is by path component, never by string prefix
///
/// `Path::starts_with` and `Path::ancestors` both compare whole
/// components, so `/a/bc` does not live under `/a/b` and `/a/b.txt` is not
/// swallowed by `/a/b`. A `str::starts_with` would say otherwise, and
/// every use of this rule in the codebase decides whether rows get
/// deleted, whether a watch is dropped, or whether a subtree is in scope —
/// so a sibling matched by accident is a neighbouring folder's index
/// disappearing. This is the reference statement of the rule;
/// [`crate::incremental::collapse_removal_roots`],
/// `coordinator::collapse_pending_removals`, `watcher::WatchRegistry::remove_tree`,
/// [`crate::scope::Scope::owning_root`] and
/// [`crate::config::nested_roots`] apply the same one.
pub fn covers(&self, path: &str) -> bool {
let dirs = self.dirs.lock().unwrap();
if dirs.is_empty() {
@ -217,7 +225,12 @@ impl UnreadableDirs {
///
/// Used as a `walkdir` `filter_entry` predicate, so returning `false` for a
/// directory prunes the whole subtree instead of merely skipping the entry.
fn walk_filter(e: &DirEntry, include_hidden: bool, ignore: &IgnoreSet) -> bool {
fn walk_filter(
e: &DirEntry,
follow_symlinks: bool,
include_hidden: bool,
ignore: &IgnoreSet,
) -> bool {
// Depth 0 is the root itself — a `false` here would silence the
// entire walk, and users explicitly chose their roots.
if e.depth() == 0 {
@ -227,7 +240,21 @@ fn walk_filter(e: &DirEntry, include_hidden: bool, ignore: &IgnoreSet) -> bool {
// Free on Windows (walkdir hands back the attributes `FindNextFileW`
// already returned) and never called on Unix, so the "no extra lstat"
// property below still holds.
if !include_hidden && crate::platform::entry_is_hidden(&name, || e.metadata().ok()) {
//
// The exception is a followed symlink: `DirEntry::metadata` switches to
// `fs::metadata` there and would report the *target's* attributes, where
// the parallel walker reports the link's own. `entry_hidden_reason`
// requires the entry itself, so ask for it explicitly — walkdir was
// already making a real call in that case, so this costs nothing extra.
if !include_hidden
&& crate::platform::entry_is_hidden(&name, || {
if follow_symlinks && e.path_is_symlink() {
std::fs::symlink_metadata(e.path()).ok()
} else {
e.metadata().ok()
}
})
{
return false;
}
!ignore.matches_component(&name) && !ignore.matches_path_pattern(e.path())
@ -256,7 +283,7 @@ fn walk_entries<'a>(
WalkDir::new(root)
.follow_links(follow_symlinks)
.into_iter()
.filter_entry(move |e| walk_filter(e, include_hidden, ignore))
.filter_entry(move |e| walk_filter(e, follow_symlinks, include_hidden, ignore))
.filter_map(move |res| match res {
Ok(e) => Some(e),
Err(err) => {
@ -310,7 +337,6 @@ pub fn filtered_dirs<'a>(
fn parse_wc_l_stdout(bytes: &[u8]) -> Result<usize, String> {
let s = String::from_utf8_lossy(bytes);
let token = s
.trim()
.split_whitespace()
.next()
.ok_or_else(|| "wc: empty output".to_string())?;
@ -405,15 +431,11 @@ fn count_find_pipe_wc(
/// Count tree entries with a plain directory walk.
///
/// Windows has no `find`, and the obvious substitute — `powershell.exe -Command
/// "(Get-ChildItem -Recurse | Measure-Object).Count"` — is a poor trade: 300+
/// ms of interpreter startup before any work, `Get-ChildItem -Recurse` is far
/// slower than a `FindNextFileW` loop, it pops a console window on a windowed
/// process, and the path has to be escaped into a script string. Walking
/// directly is faster, quieter, and cancels immediately instead of at the
/// 50 ms subprocess-poll granularity.
/// Kept as the oracle [`count_tree_entries_win32`] is tested against: the fast
/// path replaces it precisely because it must produce the same number, and an
/// independent implementation is the only way to assert that.
#[cfg(windows)]
fn count_tree_entries_native(
fn count_tree_entries_walkdir(
path: &str,
cancel: &std::sync::atomic::AtomicBool,
) -> Result<usize, String> {
@ -436,10 +458,151 @@ fn count_tree_entries_native(
Ok(n)
}
/// Count tree entries by reading directories in bulk through
/// `GetFileInformationByHandleEx`.
///
/// Windows has no `find`, so the Unix pipeline has nothing to call. The
/// obvious substitute — `powershell.exe -Command "(Get-ChildItem -Recurse |
/// Measure-Object).Count"` — is a poor trade: 300+ ms of interpreter startup
/// before any work, `Get-ChildItem -Recurse` is far slower than a directory
/// read, it pops a console window on a windowed process, and the path has to
/// be escaped into a script string.
///
/// The reason this exists rather than [`count_tree_entries_walkdir`] is the
/// per-entry cost. `std::fs::read_dir` — which walkdir sits on — issues one
/// `FindNextFileW` per entry and allocates a `PathBuf` for each, so counting a
/// million-entry tree means a million syscalls and a million allocations to
/// produce a single integer. `FileIdBothDirectoryInfo` fills a caller-supplied
/// buffer with as many chained records as fit, which turns the syscall count
/// into roughly one per 64 KiB of directory data and the allocations into one
/// per directory. This is the documented Win32 form of what fast indexers use;
/// it needs no new crate, no elevated rights, and no ntdll.
///
/// Recursion is explicit and iterative — a stack of directories still to read
/// — because a deep tree must not be able to exhaust the thread's stack.
#[cfg(windows)]
fn count_tree_entries_win32(
path: &str,
cancel: &std::sync::atomic::AtomicBool,
) -> Result<usize, String> {
use std::ffi::OsString;
use std::os::windows::ffi::{OsStrExt, OsStringExt};
use std::sync::atomic::Ordering;
use windows_sys::Win32::Foundation::{CloseHandle, INVALID_HANDLE_VALUE};
use windows_sys::Win32::Storage::FileSystem::{
CreateFileW, GetFileInformationByHandleEx, FileIdBothDirectoryInfo, FILE_ID_BOTH_DIR_INFO,
FILE_FLAG_BACKUP_SEMANTICS, FILE_LIST_DIRECTORY, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE, OPEN_EXISTING,
};
/// One call returns as many entries as fit here. 64 KiB holds several
/// hundred typical names, so a directory of any ordinary size is one or
/// two syscalls rather than one per file.
const BUFFER_BYTES: usize = 64 * 1024;
/// `FILE_ATTRIBUTE_DIRECTORY`, spelled out for the reason
/// [`crate::platform`] spells out its own attribute constants, and
/// const-asserted against the real header below.
const ATTR_DIRECTORY: u32 = 0x10;
const _: () = assert!(
ATTR_DIRECTORY == windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_DIRECTORY
);
fn wide(path: &std::path::Path) -> Vec<u16> {
path.as_os_str().encode_wide().chain(Some(0)).collect()
}
let mut pending = vec![std::path::PathBuf::from(path)];
let mut count = 0usize;
// One allocation for the whole walk, reused for every directory.
let mut buffer = vec![0u8; BUFFER_BYTES];
while let Some(dir) = pending.pop() {
// Checked per directory *and* per entry below: cancellation has to be
// immediate on a tree large enough to be worth counting.
if cancel.load(Ordering::Relaxed) {
return Err("count cancelled".to_string());
}
// FILE_FLAG_BACKUP_SEMANTICS is what makes CreateFileW open a
// directory rather than fail; the share flags let the tree keep being
// used while we count it.
let handle = unsafe {
CreateFileW(
wide(&dir).as_ptr(),
FILE_LIST_DIRECTORY,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
std::ptr::null(),
OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS,
// hTemplateFile: a HANDLE, which windows-sys spells `isize`.
0,
)
};
if handle == INVALID_HANDLE_VALUE {
// An unreadable directory costs its own entries, not the tree's.
continue;
}
loop {
// Returns zero both on error and at the end of the listing; either
// way this directory is done.
let ok = unsafe {
GetFileInformationByHandleEx(
handle,
FileIdBothDirectoryInfo,
buffer.as_mut_ptr().cast(),
buffer.len() as u32,
)
};
if ok == 0 {
break;
}
let mut offset = 0usize;
loop {
if cancel.load(Ordering::Relaxed) {
unsafe { CloseHandle(handle) };
return Err("count cancelled".to_string());
}
// SAFETY: the API filled `buffer` with a chain of these
// records, each at the offset the previous one gave.
let info =
unsafe { &*(buffer.as_ptr().add(offset) as *const FILE_ID_BOTH_DIR_INFO) };
// FileNameLength is in bytes; FileName is UTF-16 and is *not*
// NUL-terminated, so the length is the only thing that says
// where it ends.
let name_units = (info.FileNameLength as usize) / 2;
let name = unsafe {
std::slice::from_raw_parts(info.FileName.as_ptr(), name_units)
};
let name = OsString::from_wide(name);
// "." and ".." are entries of the listing, not of the tree.
let is_dot = name == "." || name == "..";
if !is_dot {
count += 1;
if info.FileAttributes & ATTR_DIRECTORY != 0 {
pending.push(dir.join(&name));
}
}
match info.NextEntryOffset {
0 => break,
next => offset += next as usize,
}
}
}
unsafe { CloseHandle(handle) };
}
Ok(count)
}
/// Rough tree entry count for progress totals. Setting `cancel` stops it
/// rather than letting it scan an entire root after the run stopped: on Unix
/// that kills the `find`/`wc` subprocesses at ~50 ms granularity, on Windows
/// the native walk notices almost immediately. Runs concurrently with
/// the bulk directory read notices almost immediately. Runs concurrently with
/// indexing — its scope is not identical to the walker's classified file
/// count.
pub fn count_tree_entries_fast(
@ -448,18 +611,18 @@ pub fn count_tree_entries_fast(
) -> Result<usize, String> {
#[cfg(windows)]
{
return count_tree_entries_native(path, cancel);
return count_tree_entries_win32(path, cancel);
}
#[cfg(all(unix, target_os = "linux"))]
{
return count_find_pipe_wc(path, cancel, true).or_else(|e| {
count_find_pipe_wc(path, cancel, true).or_else(|e| {
if e.contains("cancelled") {
Err(e)
} else {
// Non-GNU find without -printf: plain listing.
count_find_pipe_wc(path, cancel, false)
}
});
})
}
#[cfg(all(unix, not(target_os = "linux")))]
{
@ -509,13 +672,13 @@ pub fn classify_by_mtime(stored: Option<u64>, mtime: u64) -> FileIndexAction {
}
}
/// Safely truncate a string to at most max_bytes bytes while respecting UTF-8 character boundaries
/// Truncate to at most `max_bytes` bytes, backing up to a UTF-8 character
/// boundary. Cuts short of the budget rather than over it.
fn safe_truncate_string(s: &str, max_bytes: usize) -> String {
if s.len() <= max_bytes {
return s.to_string();
}
// Find the last valid UTF-8 character boundary at or before max_bytes
let mut end = max_bytes;
while end > 0 && !s.is_char_boundary(end) {
end -= 1;
@ -553,7 +716,7 @@ fn get_file_hash(
f.read_exact(&mut head)?;
let mut hasher = Sha256::new();
hasher.update(&size.to_le_bytes());
hasher.update(size.to_le_bytes());
hasher.update(&head);
Ok((hasher.finalize().to_vec(), head))
}
@ -583,7 +746,11 @@ pub struct OwnedNewFile {
pub device_id: Option<u64>,
pub mime: Option<String>,
pub ftype: FileType,
pub hash: Vec<u8>,
/// `None` only for a file whose contents could not be read without paying
/// for them — a dehydrated cloud placeholder. Stored as SQL NULL, which is
/// what keeps such files out of duplicate detection: an empty or zero hash
/// would make every one of them look identical to every other.
pub hash: Option<Vec<u8>>,
/// Text extracted from the head bytes during the walk, for files small
/// enough that the head *was* the whole file. `Some` means the content
/// pass never has to open this file; `None` leaves it pending as before.
@ -611,12 +778,28 @@ impl OwnedNewFile {
device_id: self.device_id,
mime: self.mime.as_deref(),
ftype: self.ftype,
hash: Some(&self.hash),
hash: self.hash.as_deref(),
needs_content: self.needs_content,
}
}
}
/// Individual "cannot hash" warnings allowed per run before only the count is
/// kept. See [`crate::log::Throttle`]; [`reset_run_warnings`] arms it.
static HASH_FAILURES: crate::log::Throttle = crate::log::Throttle::new(20);
/// Arm the per-run warning throttles. Called once at the start of an indexing
/// run, so the counts a run reports describe that run and not the process.
pub fn reset_run_warnings() {
HASH_FAILURES.reset();
}
/// How many files could not be hashed this run, and how many of those went
/// unlogged. `(0, 0)` when nothing failed.
pub fn hash_failure_counts() -> (u64, u64) {
(HASH_FAILURES.seen(), HASH_FAILURES.suppressed())
}
/// Build the `files` row for one on-disk file from a `stat` the caller
/// already holds. The single implementation behind both full-run batches
/// and incremental watcher updates.
@ -651,11 +834,29 @@ pub fn prepare_file_record(
.and_then(|t| t.duration_since(UNIX_EPOCH).ok())
.map(|d| d.as_secs())?;
let (hash, head) = match get_file_hash(size, Path::new(path), config.processing.hash_length) {
Ok(v) => v,
Err(e) => {
crate::log_warn!("Skipping file (cannot hash) {}: {}", path, e);
return None;
// A dehydrated cloud file is indexed from its metadata alone. Reading even
// the first byte would block on downloading the whole thing, so the head is
// never fetched: no hash, no MIME sniff, no inline extraction, and the row
// is born with nothing pending. The file is still fully searchable by name,
// size and date — only its *contents* wait, and they wait until the user
// hydrates the file themselves. Hydration does not change the mtime, so what
// picks it up is this same test on a later run, once the attribute clears.
let dehydrated = crate::platform::is_cloud_placeholder(meta);
let (hash, head) = if dehydrated {
(None, Vec::new())
} else {
match get_file_hash(size, Path::new(path), config.processing.hash_length) {
Ok((hash, head)) => (Some(hash), head),
Err(e) => {
// Throttled: on Windows a file another process holds open fails
// here as a matter of course, and a large tree can produce
// thousands. `run_indexing` reports the total.
if HASH_FAILURES.allow() {
crate::log_warn!("Skipping file (cannot hash) {}: {}", path, e);
}
return None;
}
}
};
@ -664,7 +865,9 @@ pub fn prepare_file_record(
.map(|n| n.to_string_lossy().into_owned())?;
let parent = parent_str(path);
let (inode, device_id) = inode_and_device(meta);
// Sniff from the bytes hashing already read rather than reopening.
// Sniff from the bytes hashing already read rather than reopening. With no
// head to sniff, the extension is all there is — which is what
// `guess_mime_from_head` already falls back to for an empty slice.
let mime = guess_mime_from_head(Path::new(path), &head);
let ftype = mime.as_deref().map(mime_to_type).unwrap_or(FileType::EMPTY);
@ -672,7 +875,8 @@ pub fn prepare_file_record(
// registry are all in hand — and stored as the row's `content_state`, so
// "pending" downstream means a file that really needs reading rather than
// one the content pass would only mark not-applicable.
let needs_content = size <= config.processing.maximum_text_file_size
let needs_content = !dehydrated
&& size <= config.processing.maximum_text_file_size
&& content_extractable(Path::new(path), mime.as_deref(), config, registry);
// When the head is the whole file, an extractor that works from bytes can
@ -1209,17 +1413,7 @@ mod tests {
use std::path::MAIN_SEPARATOR;
fn tmp_tree() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-walk-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
p
crate::testutil::scratch_dir("walk")
}
fn touch(p: &Path) {
@ -1619,18 +1813,9 @@ mod count_and_extract_tests {
use super::*;
use std::sync::atomic::{AtomicBool, Ordering};
/// A path that does not exist yet — these tests build the tree themselves.
fn tmp(tag: &str) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-ce-{}-{}-{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
crate::testutil::scratch_dir(tag).join("tree")
}
#[test]
@ -1642,8 +1827,59 @@ mod count_and_extract_tests {
}
let cancel = AtomicBool::new(false);
let n = count_tree_entries_fast(root.to_str().unwrap(), &cancel).unwrap();
// find lists the root, the subdir, and the three files.
assert_eq!(n, 5);
// The subdir plus the three files. Unix counts through `find`, which
// also lists the root it was given; the Windows directory read only
// ever sees entries *inside* a directory and so never counts the root.
// Both are correct for a progress denominator — this is an estimate
// running concurrently with the walk, not a total to reconcile against.
assert_eq!(n, if cfg!(windows) { 4 } else { 5 });
std::fs::remove_dir_all(&root).ok();
}
/// The load-bearing property of the Windows fast path: it exists only to
/// be quicker than a plain walk, so it has to agree with one exactly.
///
/// The tree is deliberately wider than one buffer's worth of directory
/// data, so `GetFileInformationByHandleEx` is called repeatedly for the
/// same directory and the resumption between calls is exercised — the part
/// a single-buffer test would never reach.
#[cfg(windows)]
#[test]
fn the_bulk_directory_read_agrees_with_a_plain_walk() {
let root = tmp("count-oracle");
std::fs::create_dir_all(root.join("empty")).unwrap();
std::fs::create_dir_all(root.join("a/b/c")).unwrap();
// Long names so the chained records fill more than one 64 KiB buffer.
for i in 0..600 {
let name = format!("{}-{:04}.txt", "padding".repeat(12), i);
std::fs::write(root.join(&name), b"x").unwrap();
std::fs::write(root.join("a/b/c").join(&name), b"x").unwrap();
}
let cancel = AtomicBool::new(false);
let path = root.to_str().unwrap();
let fast = count_tree_entries_win32(path, &cancel).unwrap();
let plain = count_tree_entries_walkdir(path, &cancel).unwrap();
assert_eq!(fast, plain, "the fast count must match a plain walk");
// 1200 files + `empty` + `a` + `a/b` + `a/b/c`.
assert_eq!(fast, 1204);
std::fs::remove_dir_all(&root).ok();
}
/// Cancellation is checked per entry, not per directory, so a huge single
/// directory stops as promptly as a deep tree.
#[cfg(windows)]
#[test]
fn the_bulk_directory_read_stops_when_cancelled() {
let root = tmp("count-cancel");
std::fs::create_dir_all(&root).unwrap();
for i in 0..200 {
std::fs::write(root.join(format!("f{:03}.txt", i)), b"x").unwrap();
}
let cancel = AtomicBool::new(true);
let err = count_tree_entries_win32(root.to_str().unwrap(), &cancel).unwrap_err();
assert!(err.contains("cancelled"), "{err}");
std::fs::remove_dir_all(&root).ok();
}

47
crates/quicksearch-core/src/incremental.rs
View file

@ -73,8 +73,14 @@ fn upsert_path(
if meta.is_dir() {
// A moved-in tree surfaces as one directory event; walk it with
// the same filters as a full run.
//
// `filtered_walk` wants a `&str`, so a path that is not valid UTF-8
// cannot be walked here. That is a whole subtree missing from the
// index, not one file, so it is an error rather than a quiet `Ok`:
// the caller answers it by scheduling a full run, which walks from
// the configured root and never needs the path as a `str`.
let Some(root) = path.to_str() else {
return Ok(());
return Err(format!("directory path is not valid UTF-8: {:?}", path));
};
let entries: Vec<_> = filtered_walk(
root,
@ -168,11 +174,17 @@ fn remove_path(conn: &mut Connection, path: &Path) -> Result<(), String> {
/// `dir` sweeps its whole path range, each descendant event is duplicate work —
/// 10 000 of them for a 10 000-file tree.
///
/// This is for a caller holding a raw removal set. The coordinator is not one:
/// it collapses as events *arrive*, in `collapse_pending_removals`, because it
/// has to measure its queue after collapsing rather than before (see
/// `Inner::drain_events`). Same rule, two entry points, deliberately — a
/// mass deletion must be one path by the time either of them is done with it.
///
/// Ancestor membership is tested against a set rather than by sorting, which
/// keeps this obviously correct: no ordering argument to get wrong, and
/// `Path::ancestors` walks whole components, so `/a/bc` is never treated as
/// living under `/a/b` — the same rule `remove_tree` and
/// `UnreadableDirs::covers` use. Paths are shallow, so the cost is linear.
/// keeps this obviously correct: there is no ordering argument to get wrong.
/// Containment is component-wise, per
/// [`crate::file_handling::UnreadableDirs::covers`]. Paths are shallow, so the
/// cost is linear.
pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::PathBuf> {
if paths.len() < 2 {
return paths;
@ -188,11 +200,13 @@ pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::
/// Delete `paths` and everything indexed beneath them, in transactions of at
/// most `chunk` paths.
///
/// A mass deletion is the case this exists for. Callers hand over the *roots*
/// of the removal set (see [`collapse_removal_roots`]), so `rm -rf dir/` is one
/// path here rather than one per file, and each one costs a fixed handful of
/// range-driven statements ([`repo::delete_subtree`]) instead of a full table
/// scan plus five statements per file.
/// Correct for any path set, but a mass deletion is the case it exists for,
/// and it pays off only if the caller has already reduced that set to its
/// *roots* — then `rm -rf dir/` is one path here rather than one per file, and
/// costs a fixed handful of range-driven statements
/// ([`repo::delete_subtree`]) instead of a full table scan plus five
/// statements per file. Both callers do: the coordinator collapses on arrival,
/// and anyone else has [`collapse_removal_roots`].
///
/// Chunking bounds how long any single transaction holds the connection, the
/// same way `process_batch_inserts` bounds the indexer's writes.
@ -239,17 +253,10 @@ mod tests {
impl Fixture {
fn new() -> Fixture {
let stamp = format!(
"{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let dir = std::env::temp_dir().join(format!("qs-incr-{}", stamp));
let scratch = crate::testutil::scratch_dir("incr");
let dir = scratch.join("tree");
std::fs::create_dir_all(&dir).unwrap();
let db = std::env::temp_dir().join(format!("qs-incr-{}.sqlite", stamp));
let db = scratch.join("index.sqlite");
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
let config = Config::default();
let ignore = IgnoreSet::compile(&config.indexing.ignore_patterns).unwrap();

841
crates/quicksearch-core/src/indexing.rs
View file

File diff suppressed because it is too large Load diff

3
crates/quicksearch-core/src/lib.rs
View file

@ -3,7 +3,6 @@ pub mod config;
pub mod content;
pub mod coordinator;
pub mod db;
pub mod document_extraction;
pub mod extract;
pub mod file_handling;
pub mod incremental;
@ -17,6 +16,8 @@ pub mod search;
pub mod security;
pub mod shutdown;
pub mod snippet;
#[doc(hidden)]
pub mod testutil;
pub mod textenc;
pub mod walk;
pub mod watcher;

95
crates/quicksearch-core/src/log.rs
View file

@ -7,7 +7,8 @@
//! exactly the ones a user needs when something looks wrong, and they were
//! going nowhere.
//!
//! So background reporting goes through [`log_info!`] and [`log_warn!`]
//! So background reporting goes through [`crate::log_info!`] and
//! [`crate::log_warn!`]
//! instead of `println!`/`eprintln!`: each writes the same line to stderr
//! *and* appends it to a bounded ring the GUI's Logs tab reads. A terminal
//! run looks exactly as it did; a windowed run gains the tab.
@ -66,7 +67,8 @@ macro_rules! log_warn {
/// Write `message` to stderr and to the ring.
///
/// Prefer the [`log_info!`] / [`log_warn!`] macros; this is what they call.
/// Prefer the [`crate::log_info!`] / [`crate::log_warn!`] macros; this is what
/// they call.
///
/// A failed stderr write is ignored rather than propagated: `eprintln!`
/// *panics* when the handle is unwritable, which on a process launched
@ -82,6 +84,56 @@ pub fn record(level: Level, message: String) {
lock().push(level, text);
}
/// A cap on how many times one *kind* of warning is allowed to speak.
///
/// Some failures are per-file and arrive in the thousands: a directory tree the
/// user cannot read, or — the case that motivated this — a Windows machine where
/// `ERROR_SHARING_VIOLATION` from a file some other process holds open is
/// routine and has no Unix equivalent. Logging each one costs a global mutex and
/// an unbuffered stderr write on the walk's hottest path, and it evicts the
/// [`CAPACITY`]-line ring so thoroughly that the warnings worth reading are gone
/// before the run ends.
///
/// So the first few speak and the rest are counted. The count is the part that
/// actually informs: "3 files could not be hashed" and "31,402 files could not
/// be hashed" call for very different responses, and neither is legible as a
/// wall of identical lines.
///
/// Declared as a `static` next to the call site it guards, and reset by whoever
/// owns the run, so the numbers describe one run rather than the process.
pub struct Throttle {
limit: u64,
seen: std::sync::atomic::AtomicU64,
}
impl Throttle {
pub const fn new(limit: u64) -> Throttle {
Throttle {
limit,
seen: std::sync::atomic::AtomicU64::new(0),
}
}
/// Count one occurrence, and answer whether it may be logged individually.
pub fn allow(&self) -> bool {
self.seen.fetch_add(1, std::sync::atomic::Ordering::Relaxed) < self.limit
}
/// Occurrences counted since the last [`Throttle::reset`].
pub fn seen(&self) -> u64 {
self.seen.load(std::sync::atomic::Ordering::Relaxed)
}
/// How many went unlogged — what a summary line should report.
pub fn suppressed(&self) -> u64 {
self.seen().saturating_sub(self.limit)
}
pub fn reset(&self) {
self.seen.store(0, std::sync::atomic::Ordering::Relaxed);
}
}
/// Every retained line, oldest first.
pub fn snapshot() -> Vec<LogLine> {
lock().lines.iter().cloned().collect()
@ -152,7 +204,17 @@ impl Ring {
}
}
fn now_unix() -> u64 {
/// Seconds since the Unix epoch.
///
/// Lives here because this is the lowest-level module in the crate and every
/// layer above it wanted the same four lines — log lines, the `last_full_index`
/// stamp, `schema_info.created_at`, failure timestamps, and the GUI's
/// "5 min ago".
///
/// A clock set before 1970 yields `0` rather than an error. Every caller is
/// stamping a record or measuring an age, and none has anything better to do
/// with a failure than what `0` already does: read as "very long ago".
pub fn now_unix() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
@ -215,6 +277,33 @@ mod tests {
assert_eq!(ring.recorded, 4, "the running total survives a clear");
}
#[test]
fn a_throttle_lets_the_first_few_through_and_counts_the_rest() {
let t = Throttle::new(3);
assert_eq!((t.seen(), t.suppressed()), (0, 0));
for _ in 0..3 {
assert!(t.allow(), "the first `limit` occurrences speak");
}
for _ in 0..7 {
assert!(!t.allow(), "the rest are counted only");
}
assert_eq!(t.seen(), 10);
assert_eq!(t.suppressed(), 7);
t.reset();
assert_eq!((t.seen(), t.suppressed()), (0, 0));
assert!(t.allow(), "a reset throttle speaks again");
}
/// A zero limit must silence rather than divide by anything.
#[test]
fn a_zero_limit_throttle_logs_nothing() {
let t = Throttle::new(0);
assert!(!t.allow());
assert_eq!(t.seen(), 1);
assert_eq!(t.suppressed(), 1);
}
/// Through the global: the macros must land in the snapshot, and a
/// warning must carry the prefix its terminal line has. Written to
/// tolerate lines from tests running in parallel in this process.

454
crates/quicksearch-core/src/platform.rs
View file

@ -29,12 +29,72 @@ pub fn home_dir() -> Option<OsString> {
std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE"))
}
/// Whether a directory entry counts as hidden.
/// `FILE_ATTRIBUTE_HIDDEN`.
///
/// Unix: a leading dot. Windows: a leading dot **or** `FILE_ATTRIBUTE_HIDDEN`
/// / `FILE_ATTRIBUTE_SYSTEM` — without which `include_hidden = false` hides
/// nothing on Windows, and `$RECYCLE.BIN`, `System Volume Information`,
/// `pagefile.sys` and `AppData` all get indexed.
/// Spelled out rather than imported, for the reason in the module header:
/// `windows-sys` is a `cfg(windows)`-only dependency, and
/// [`attributes_are_hidden`] has to compile — and be tested — on Linux, where
/// the suite runs.
const FILE_ATTRIBUTE_HIDDEN: u32 = 0x2;
/// The cross-compiled Windows build is where the spelling above is checked
/// against the real header value, so a typo fails that job rather than quietly
/// indexing `$RECYCLE.BIN`.
#[cfg(windows)]
const _: () = assert!(
FILE_ATTRIBUTE_HIDDEN == windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_HIDDEN
);
/// Whether a Windows attribute word marks an entry hidden.
///
/// `FILE_ATTRIBUTE_HIDDEN` and nothing else. `FILE_ATTRIBUTE_SYSTEM` was part
/// of this test and was removed, because it pruned users' cloud folders:
/// Windows honours the `desktop.ini` inside a folder only if the folder itself
/// carries Read-only or System, so the ownCloud, Nextcloud, OneDrive and Google
/// Drive clients set System on their sync root purely to get a branded icon.
/// Such a folder has no Hidden bit and is plainly visible in Explorer, yet the
/// whole subtree vanished from the index with nothing to explain it — only
/// "Index hidden files" brought it back, which is the opposite of what that
/// setting is for. Any folder given a custom icon is the same case.
///
/// Nothing the System term existed for is lost. `$RECYCLE.BIN`, `System Volume
/// Information`, `pagefile.sys` and the legacy per-user junctions (`My
/// Documents`, `Local Settings`, `Application Data`) are Hidden **and** System —
/// that pairing is Windows' own definition of a protected operating system file
/// — and `AppData` is Hidden alone. Hidden catches every one. The drive-root
/// names are excluded by [`crate::config`]'s default ignore patterns as well, so
/// they have two independent reasons to stay out.
#[cfg_attr(not(windows), allow(dead_code))]
pub(crate) fn attributes_are_hidden(attributes: u32) -> bool {
attributes & FILE_ATTRIBUTE_HIDDEN != 0
}
/// Why an entry counted as hidden.
///
/// The distinction exists for the walk's log line: a dot prefix explains itself
/// and an ignore pattern is something the user typed, but "this visible folder
/// was skipped over an attribute you cannot see" has no discoverability at all,
/// so only that case is worth reporting.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HiddenReason {
DotPrefix,
Attribute,
}
/// Whether a directory entry counts as hidden, and why.
///
/// Unix: a leading dot. Windows: a leading dot **or** `FILE_ATTRIBUTE_HIDDEN` —
/// without which `include_hidden = false` hides nothing on Windows, and
/// `$RECYCLE.BIN`, `System Volume Information`, `pagefile.sys` and `AppData` all
/// get indexed. `FILE_ATTRIBUTE_SYSTEM` is deliberately not part of it; see
/// [`attributes_are_hidden`].
///
/// `meta` must report the attributes of the entry **itself**, never of a link
/// target — callers pass `symlink_metadata` or an already-cached directory
/// entry. A hidden symlink is a hidden alias; a visible symlink to a hidden
/// target is a visible alias. All four call sites have to agree on that, or a
/// full run indexes a file the watcher then refuses to update and the index
/// churns on every cycle.
///
/// `meta` is a closure because on Unix it is never called: the walkers
/// deliberately avoid `metadata()`, which would cost an extra `lstat` per
@ -42,28 +102,207 @@ pub fn home_dir() -> Option<OsString> {
/// zero anyway — both `std::fs::DirEntry::metadata` and
/// `walkdir::DirEntry::metadata` hand back data already cached from
/// `FindNextFileW`.
pub fn entry_is_hidden<F>(name: &str, meta: F) -> bool
pub fn entry_hidden_reason<F>(name: &str, meta: F) -> Option<HiddenReason>
where
F: FnOnce() -> Option<std::fs::Metadata>,
{
if name.starts_with('.') {
return true;
return Some(HiddenReason::DotPrefix);
}
#[cfg(windows)]
{
use std::os::windows::fs::MetadataExt;
use windows_sys::Win32::Storage::FileSystem::{
FILE_ATTRIBUTE_HIDDEN, FILE_ATTRIBUTE_SYSTEM,
};
if let Some(m) = meta() {
return m.file_attributes() & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) != 0;
if attributes_are_hidden(m.file_attributes()) {
return Some(HiddenReason::Attribute);
}
}
}
#[cfg(not(windows))]
{
let _ = meta;
}
false
None
}
/// [`entry_hidden_reason`] for the callers that only need the verdict.
pub fn entry_is_hidden<F>(name: &str, meta: F) -> bool
where
F: FnOnce() -> Option<std::fs::Metadata>,
{
entry_hidden_reason(name, meta).is_some()
}
/// `FILE_ATTRIBUTE_REPARSE_POINT`. Spelled out for [`FILE_ATTRIBUTE_HIDDEN`]'s
/// reason, and checked against the real header value the same way.
const FILE_ATTRIBUTE_REPARSE_POINT: u32 = 0x400;
#[cfg(windows)]
const _: () = assert!(
FILE_ATTRIBUTE_REPARSE_POINT
== windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_REPARSE_POINT
);
/// The attributes a dehydrated cloud file carries.
///
/// `FILE_ATTRIBUTE_OFFLINE` (0x1000) is the old tape-archive bit that OneDrive
/// reused; `RECALL_ON_OPEN` (0x40000) marks a file whose *metadata* is local but
/// whose data is not; `RECALL_ON_DATA_ACCESS` (0x400000) is the modern
/// Files-On-Demand placeholder. Any one of them means opening the file for read
/// pulls it over the network.
const FILE_ATTRIBUTE_OFFLINE: u32 = 0x1000;
const FILE_ATTRIBUTE_RECALL_ON_OPEN: u32 = 0x4_0000;
const FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS: u32 = 0x40_0000;
#[cfg(windows)]
const _: () = {
use windows_sys::Win32::Storage::FileSystem as fs_attrs;
assert!(FILE_ATTRIBUTE_OFFLINE == fs_attrs::FILE_ATTRIBUTE_OFFLINE);
assert!(FILE_ATTRIBUTE_RECALL_ON_OPEN == fs_attrs::FILE_ATTRIBUTE_RECALL_ON_OPEN);
assert!(FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS == fs_attrs::FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS);
};
/// Whether reading this file's contents would pull it down from the cloud.
///
/// OneDrive, and every other Files-On-Demand provider, leaves a placeholder on
/// disk with real metadata and no data. Opening one for read is not a local
/// operation: it blocks on a network download of the entire file. The default
/// indexing root is `%USERPROFILE%`, the OneDrive folder beneath it is not
/// hidden, and the walk hashes the first 8 KiB of every new or changed file — so
/// without this test a first index quietly downloads the user's whole cloud
/// drive, filling their disk with the files they had deliberately offloaded.
///
/// Named for the question the caller is actually asking ("will reading this
/// cost a download?") rather than for the bits, because the answer is what the
/// indexing path branches on. Always `false` off Windows: no other platform this
/// runs on has an equivalent, and a `stat` there tells the truth about the data.
pub fn is_cloud_placeholder(meta: &std::fs::Metadata) -> bool {
#[cfg(windows)]
{
use std::os::windows::fs::MetadataExt;
return attributes_are_dehydrated(meta.file_attributes());
}
#[cfg(not(windows))]
{
let _ = meta;
false
}
}
/// The bit test behind [`is_cloud_placeholder`], split out so the Linux suite
/// exercises it. See this module's second rule.
#[cfg_attr(not(windows), allow(dead_code))]
pub(crate) fn attributes_are_dehydrated(attributes: u32) -> bool {
attributes
& (FILE_ATTRIBUTE_OFFLINE
| FILE_ATTRIBUTE_RECALL_ON_OPEN
| FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS)
!= 0
}
/// Whether a Windows attribute word marks an entry as a reparse point.
///
/// Split out from [`entry_cached_metadata`] so the bit test is exercised by the
/// Linux suite, per this module's second rule. Deliberately *not* the same
/// question as `FileType::is_symlink`, which additionally requires the
/// name-surrogate bit: see [`entry_cached_metadata`] for why that distinction is
/// the whole point.
#[cfg_attr(not(windows), allow(dead_code))]
pub(crate) fn attributes_are_reparse_point(attributes: u32) -> bool {
attributes & FILE_ATTRIBUTE_REPARSE_POINT != 0
}
/// Metadata a directory read already handed back, on the platforms that hand
/// any back at all.
///
/// `std::fs::Metadata` on Windows: `FindFirstFileW`/`FindNextFileW` return size,
/// timestamps and attributes with every name, and `std::fs::DirEntry::metadata`
/// is a copy of that buffer rather than a syscall. A `fs::metadata(path)` on the
/// same entry is therefore a whole extra `CreateFileW` +
/// `GetFileInformationByHandle` + `CloseHandle` — an `IRP_MJ_CREATE` through
/// every antivirus and EDR minifilter on the machine — for data already in hand.
///
/// Uninhabited everywhere else, because `getdents64` returns only `d_type`: size
/// and mtime genuinely need the `statx`, and `DirEntry::metadata` there is a
/// *second* syscall rather than a saved one. Uninhabited rather than a unit
/// struct so `Option<CachedMetadata>` is zero-sized off Windows — the walker
/// queues one per pending file and nothing throttles file chunks (see
/// `walk::Job::Files`), so carrying `Option<std::fs::Metadata>` would cost Linux
/// 176 bytes per queued file to hold nothing at all.
#[cfg(windows)]
pub(crate) type CachedMetadata = std::fs::Metadata;
#[cfg(not(windows))]
pub(crate) type CachedMetadata = std::convert::Infallible;
/// The zero-cost half of the claim above, enforced rather than asserted in
/// prose: a layout change in a future toolchain becomes a compile error instead
/// of a silent regression in the walker's memory profile.
#[cfg(not(windows))]
const _: () = assert!(std::mem::size_of::<Option<CachedMetadata>>() == 0);
/// The metadata a directory read already produced for one entry — but only
/// where trusting it is both free *and* indistinguishable from a fresh `stat`.
///
/// `meta` is a closure for exactly [`entry_hidden_reason`]'s reason: on Unix it
/// is never called, because `std::fs::DirEntry::metadata` there is a real
/// `lstat` and the walk spends exactly one `statx` per file by design.
///
/// `None` for a reparse point even on Windows, and that is the whole subtlety.
/// `fs::metadata` *follows* a reparse point; the cached buffer describes the
/// link itself. The tags std does not classify as symlinks reach the walk's
/// ordinary file arm — `IO_REPARSE_TAG_APPEXECLINK`, the zero-byte Store app
/// stubs under `WindowsApps`, and the OneDrive `IO_REPARSE_TAG_CLOUD_*` family —
/// and for those the two answers differ: today an AppExecLink fails
/// `CreateFileW` with `ERROR_CANT_ACCESS_FILE` and is skipped, while its
/// directory entry looks like an ordinary empty file and would get indexed.
/// Sending every reparse point back to the path-based `fs::metadata` keeps the
/// semantics identical and leaves only the common case on the fast path.
///
/// Tested against the raw `FILE_ATTRIBUTE_REPARSE_POINT` bit, deliberately not
/// against `file_type().is_symlink()`: std's `is_symlink` additionally requires
/// the name-surrogate bit `0x20000000`, which is precisely the test that lets
/// AppExecLink and the cloud tags through.
pub(crate) fn entry_cached_metadata<F>(meta: F) -> Option<CachedMetadata>
where
F: FnOnce() -> Option<std::fs::Metadata>,
{
#[cfg(windows)]
{
use std::os::windows::fs::MetadataExt;
if let Some(m) = meta().filter(|m| !attributes_are_reparse_point(m.file_attributes())) {
return Some(m);
}
}
#[cfg(not(windows))]
{
let _ = meta;
}
None
}
/// A file's `std::fs::Metadata`, from the directory read where that read
/// supplied it and from a `stat` where it did not.
///
/// The `#[cfg]` lives here rather than at the call site, per this module's first
/// rule. Off Windows `cached` is uninhabited and therefore provably `None`,
/// which is why the walk still costs exactly one `statx` per file there.
#[cfg(windows)]
pub(crate) fn metadata_or_stat(
path: &Path,
cached: Option<CachedMetadata>,
) -> std::io::Result<std::fs::Metadata> {
match cached {
Some(m) => Ok(m),
None => std::fs::metadata(path),
}
}
#[cfg(not(windows))]
pub(crate) fn metadata_or_stat(
path: &Path,
_cached: Option<CachedMetadata>,
) -> std::io::Result<std::fs::Metadata> {
std::fs::metadata(path)
}
/// Whether `path` has a hidden component *below* the root that contains it.
@ -101,7 +340,11 @@ pub fn path_has_hidden_component_under(path: &Path, roots: &[PathBuf]) -> bool {
current.push(component);
if let Component::Normal(name) = component {
let name = name.to_string_lossy();
if entry_is_hidden(&name, || std::fs::metadata(&current).ok()) {
// `symlink_metadata`, not `metadata`: each component is judged as
// itself, which is what the walkers do. Only the final component
// stops being followed, so intermediate ones still resolve
// normally. See `entry_hidden_reason`.
if entry_is_hidden(&name, || std::fs::symlink_metadata(&current).ok()) {
return true;
}
}
@ -234,6 +477,15 @@ pub const WATCH_ROOTS_RECURSIVELY: bool = cfg!(windows);
/// on both sides, consistently.
pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" };
/// Whether this platform's filesystem matches names without regard to case.
///
/// What ignore patterns compile against: on Windows and macOS `node_modules`
/// has to exclude `Node_Modules`, and on Linux it must not. Named here rather
/// than spelled `cfg!(any(windows, target_os = "macos"))` at each use, so the
/// two places that must agree — [`crate::config::IgnoreSet`]'s literal set and
/// its glob set — cannot drift apart.
pub const PATHS_ARE_CASE_INSENSITIVE: bool = cfg!(any(windows, target_os = "macos"));
/// Drop the **calling thread** to background scheduling priority.
///
/// Per-thread, not per-process. The GUI shares this process, so lowering the
@ -245,6 +497,19 @@ pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" }
///
/// Best-effort and idempotent: a refusal is not worth reporting, since the
/// only consequence is that indexing competes on equal terms.
///
/// **CPU only, on every platform.** This used to be
/// `THREAD_MODE_BACKGROUND_BEGIN` on Windows, which is background *mode*: it
/// lowers the thread to base priority 4 and drops it to `IoPriorityVeryLow`, a
/// tier the kernel does not merely deprioritise but actively rate-limits — the
/// same one SuperFetch and defrag run in. Linux's `nice` has no I/O half at all
/// under the usual schedulers, so the two platforms were not doing remotely the
/// same thing: every walker, prefetcher, extractor *and* the single SQLite
/// writer thread were throttled on Windows and full-speed on Linux, which is
/// most of why Windows indexing was so much slower. Matching `nice`'s CPU-only
/// semantics is the deliberate choice; a machine that genuinely needs I/O
/// throttling needs it as a user-visible setting, not as a silent per-platform
/// difference.
pub fn set_background_priority() {
#[cfg(target_os = "linux")]
{
@ -256,12 +521,12 @@ pub fn set_background_priority() {
#[cfg(windows)]
{
use windows_sys::Win32::System::Threading::{
GetCurrentThread, SetThreadPriority, THREAD_MODE_BACKGROUND_BEGIN,
GetCurrentThread, SetThreadPriority, THREAD_PRIORITY_BELOW_NORMAL,
};
// Background *mode*, not merely a lower priority number: it drops I/O
// priority as well, which is what actually keeps a walk from starving
// the foreground on a spinning disk.
unsafe { SetThreadPriority(GetCurrentThread(), THREAD_MODE_BACKGROUND_BEGIN) };
// Yields the CPU to the foreground without touching I/O priority — the
// closest Windows equivalent of `nice(10)`. See the note above for why
// this is not `THREAD_MODE_BACKGROUND_BEGIN`.
unsafe { SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_BELOW_NORMAL) };
}
// Elsewhere (macOS, BSD): deliberately nothing. `nice` there applies to the
// whole process, so it would hit the GUI. The right call is
@ -426,12 +691,165 @@ mod tests {
assert!(!called, "a dot prefix must short-circuit before any stat");
}
/// Each of the three attributes on its own must be enough: providers do not
/// agree on which they set, and getting this wrong means silently
/// downloading someone's entire cloud drive.
#[test]
fn any_recall_attribute_marks_a_file_dehydrated() {
for bit in [OFFLINE, RECALL_ON_OPEN, RECALL_ON_DATA_ACCESS] {
assert!(attributes_are_dehydrated(bit));
assert!(attributes_are_dehydrated(ARCHIVE | REPARSE_POINT | bit));
}
// A synced-down file keeps the reparse point but drops the recall bits.
assert!(!attributes_are_dehydrated(ARCHIVE | REPARSE_POINT));
assert!(!attributes_are_dehydrated(ARCHIVE));
assert!(!attributes_are_dehydrated(NORMAL));
assert!(!attributes_are_dehydrated(0));
}
/// Off Windows there is no such thing, and a local `stat` tells the truth.
#[test]
#[cfg(not(windows))]
fn nothing_is_a_cloud_placeholder_off_windows() {
let meta = std::fs::metadata(env!("CARGO_MANIFEST_DIR")).unwrap();
assert!(!is_cloud_placeholder(&meta));
}
/// The bit test behind the walk's fast path, exercised where the suite
/// actually runs. A junction, an AppExecLink stub and a OneDrive
/// placeholder all carry this bit; an ordinary file does not.
#[test]
fn reparse_points_are_recognised_by_attribute() {
assert!(attributes_are_reparse_point(REPARSE_POINT));
assert!(attributes_are_reparse_point(DIRECTORY | REPARSE_POINT));
// A dehydrated cloud file: reparse point plus the recall attributes.
assert!(attributes_are_reparse_point(
ARCHIVE | REPARSE_POINT | 0x40_0000
));
assert!(!attributes_are_reparse_point(ARCHIVE));
assert!(!attributes_are_reparse_point(NORMAL));
assert!(!attributes_are_reparse_point(DIRECTORY));
assert!(!attributes_are_reparse_point(0));
}
/// Off Windows the directory read supplies nothing, and asking it for
/// anything would be the `lstat` per entry the walker exists to avoid.
#[test]
#[cfg(not(windows))]
fn nothing_is_served_from_a_directory_read_off_windows() {
let mut called = false;
let got = entry_cached_metadata(|| {
called = true;
None
});
assert!(got.is_none());
assert!(
!called,
"DirEntry::metadata here is an lstat, which is the syscall the walk exists to avoid"
);
}
/// `fs::metadata` follows a reparse point and the cached buffer does not,
/// so the fast path must decline every one of them — including the tags std
/// does not call symlinks, which are precisely the ones that reach the
/// walk's ordinary file arm.
#[test]
#[cfg(windows)]
fn a_reparse_point_is_never_served_from_the_directory_read() {
let dir = std::env::temp_dir().join(format!("qs-reparse-{}", std::process::id()));
let target = dir.join("target");
let link = dir.join("link");
let plain = dir.join("plain.txt");
std::fs::create_dir_all(&target).unwrap();
std::fs::write(&plain, b"x").unwrap();
let made = std::process::Command::new("cmd")
.args(["/C", "mklink", "/J"])
.arg(&link)
.arg(&target)
.output()
.map(|o| o.status.success())
.unwrap_or(false);
if made {
let m = std::fs::symlink_metadata(&link).unwrap();
assert!(
entry_cached_metadata(|| Some(m)).is_none(),
"a junction must fall back to the path-based stat"
);
}
let m = std::fs::metadata(&plain).unwrap();
assert!(
entry_cached_metadata(|| Some(m)).is_some(),
"an ordinary file must take the fast path"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn ordinary_names_are_not_hidden() {
assert!(!entry_is_hidden("Documents", || None));
assert!(!entry_is_hidden("report.txt", || None));
}
/// The real `FILE_ATTRIBUTE_*` bits, so the cases below read as the files
/// they stand for.
const READONLY: u32 = 0x1;
const HIDDEN: u32 = 0x2;
const SYSTEM: u32 = 0x4;
const DIRECTORY: u32 = 0x10;
const ARCHIVE: u32 = 0x20;
const NORMAL: u32 = 0x80;
const REPARSE_POINT: u32 = 0x400;
const OFFLINE: u32 = 0x1000;
const RECALL_ON_OPEN: u32 = 0x4_0000;
const RECALL_ON_DATA_ACCESS: u32 = 0x40_0000;
/// The attribute half of `entry_is_hidden`, which the Windows arm cannot
/// be asked about from Linux. Split out precisely so this test runs
/// everywhere.
#[test]
fn only_the_hidden_bit_hides_an_entry() {
// AppData: Hidden alone, and `std::env::temp_dir()` lives under it.
assert!(attributes_are_hidden(HIDDEN | DIRECTORY));
// $RECYCLE.BIN, System Volume Information, and the legacy per-user
// junctions: Hidden+System, Windows' own definition of a protected
// operating system file.
assert!(attributes_are_hidden(HIDDEN | SYSTEM | DIRECTORY));
// pagefile.sys.
assert!(attributes_are_hidden(HIDDEN | SYSTEM | ARCHIVE));
assert!(!attributes_are_hidden(0));
assert!(!attributes_are_hidden(NORMAL));
assert!(!attributes_are_hidden(DIRECTORY));
assert!(!attributes_are_hidden(READONLY | DIRECTORY));
}
/// Regression: a cloud sync root carries System and *not* Hidden — Windows
/// will not honour the `desktop.ini` supplying its branded icon otherwise —
/// and is fully visible in Explorer. Keying on System pruned the folder and
/// every file beneath it, silently, and only "include hidden files" brought
/// it back.
#[test]
fn a_sync_root_marked_system_but_not_hidden_is_not_hidden() {
assert!(!attributes_are_hidden(SYSTEM | DIRECTORY));
// Read-only is the other attribute that enables desktop.ini, and
// Explorer sets it when a user picks a custom folder icon.
assert!(!attributes_are_hidden(READONLY | SYSTEM | DIRECTORY));
assert!(!attributes_are_hidden(SYSTEM));
}
/// The walk announces an attribute prune and stays quiet about a dot
/// prefix, so the two must stay distinguishable.
#[test]
fn a_dot_prefix_reports_itself_as_the_reason() {
assert_eq!(
entry_hidden_reason(".git", || None),
Some(HiddenReason::DotPrefix)
);
assert_eq!(entry_hidden_reason("Documents", || None), None);
}
#[test]
fn hidden_components_are_measured_from_the_innermost_root() {
let root = PathBuf::from(format!("{}.config", sep_prefix()));

7
crates/quicksearch-core/src/query/pattern.rs
View file

@ -236,7 +236,8 @@ impl TermPattern {
}
}
/// Case-insensitive [`find_first`] against an already-folded haystack.
/// Case-insensitive [`TermPattern::find_first`] against an already-folded
/// haystack.
///
/// The literal path would otherwise fold the haystack itself, and the
/// cascade's full-text passes need the same fold for counting, searching
@ -256,8 +257,8 @@ impl TermPattern {
}
}
/// Case-insensitive [`count`] against an already-folded haystack. See
/// [`TermPattern::find_first_folded`].
/// Case-insensitive [`TermPattern::count`] against an already-folded
/// haystack. See [`TermPattern::find_first_folded`].
pub fn count_folded(&self, folded: &str) -> usize {
match self {
TermPattern::Empty => 0,

431
crates/quicksearch-core/src/scope.rs
View file

@ -28,8 +28,25 @@
//! indexed, has no owning root, and so has no filtering rules that can be
//! applied to it — scanning by range means it is simply never visited, the
//! same exemption `aliased_paths` gives it during a full run's stale sweep.
//!
//! ## Reporting and giving up
//!
//! On a multi-million-row index this pass is minutes of work with no files
//! moving to show for it, so the cursor counts what it has examined against
//! the rows in the index and its driver publishes that
//! ([`crate::indexing::ReconcileProgress`]).
//!
//! It can also be abandoned, through the flag-plus-interrupt pair
//! [`crate::db::InterruptSlot`] describes — before this, closing the window
//! during a prune waited the prune out. What makes giving up safe is that
//! nothing here records anything: the stored configuration is stamped by the
//! *caller*, only once the cursor reports itself finished, so an abandoned
//! pass leaves the index describing the settings it was last reconciled to and
//! the next run derives the same plan again. See [`outstanding_work`], which
//! is that question asked directly.
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use rusqlite::Connection;
@ -38,12 +55,17 @@ use crate::config::{Config, IgnoreSet, IndexWork};
use crate::db::repo;
use crate::extract::Registry;
use crate::file_handling::{content_extractable, fts_finalize_after_text_indexing, ExtractCursor};
use crate::indexing::ReconcileProgress;
/// How long [`advance`] may work before handing control back.
///
/// Not a throughput knob — the caller decides when to come back — but a bound
/// on how long a Stop, a search or a further config edit waits behind a scan
/// in progress. The same budget the coordinator gives its watcher queue.
///
/// It bounds the wait *between* statements only; one statement can outlast the
/// whole budget, which is why the pass also publishes its connection through
/// [`crate::db::InterruptGuard`]. See [`crate::db::InterruptSlot`].
pub const SLICE: Duration = Duration::from_millis(250);
/// One configured root, with the `files.path` range it owns precomputed.
@ -89,9 +111,8 @@ impl Scope {
})
}
/// The configured root `path` lives under, if any. `Path::starts_with`
/// compares whole components, so `/a/bc` is never read as living under
/// `/a/b`.
/// The configured root `path` lives under, if any. Containment is
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
pub fn owning_root(&self, path: &Path) -> Option<&Path> {
self.roots
.iter()
@ -107,8 +128,23 @@ impl Scope {
/// which is exactly the union of the per-level tests the walker performs
/// on its way down. The hidden and component-pattern rules are tested per
/// component *below* the root: a root is never filtered, because the user
/// chose it (see `walk_parallel`).
/// chose it (see [`crate::walk::walk_indexable_files`]).
pub fn covers(&self, root: &Path, path: &Path) -> bool {
self.covers_cached(root, path, &mut CoverCache::default())
}
/// [`Scope::covers`], reusing the verdicts already reached for the
/// directories on the way down.
///
/// A scan calls this once per stored row, and rows under one root share
/// their ancestors densely — every file in `~/src/project/src` asks the same
/// four questions before it asks its own. On Unix that repetition is free,
/// because `entry_is_hidden` never calls the closure and the whole loop is
/// string comparisons. On Windows each component is a `symlink_metadata`,
/// which is a `CreateFileW` through the full filter-driver stack: a 3M-row
/// index at depth 8 was ~24M file opens where Linux did none. Remembering
/// each directory's verdict collapses that to roughly one per directory.
pub fn covers_cached(&self, root: &Path, path: &Path, cache: &mut CoverCache) -> bool {
if self.ignore.matches_path_pattern(path) {
return false;
}
@ -116,28 +152,85 @@ impl Scope {
return false;
};
let mut current = root.to_path_buf();
for component in relative.components() {
let depth = relative.components().count();
for (i, component) in relative.components().enumerate() {
let std::path::Component::Normal(name) = component else {
// Stored paths are canonical, so stripping a canonical root
// leaves plain names. Anything else did not come from a walk.
return false;
};
current.push(name);
let name = name.to_string_lossy();
// The metadata closure is only consulted on Windows, where hidden
// is an attribute rather than a leading dot; on Unix this stays at
// zero syscalls, exactly as it does in the walker.
if !self.include_hidden
&& crate::platform::entry_is_hidden(&name, || std::fs::metadata(&current).ok())
{
return false;
// Only the ancestors are worth remembering. The last component is
// this row's own file name, asked once and never again, so caching
// it would grow the map by one entry per row for no hits at all.
let is_leaf = i + 1 == depth;
if !is_leaf {
if let Some(allowed) = cache.get(&current) {
if !allowed {
return false;
}
continue;
}
}
if self.ignore.matches_component(&name) {
let allowed = self.component_allowed(&current, &name.to_string_lossy());
if !is_leaf {
cache.insert(current.clone(), allowed);
}
if !allowed {
return false;
}
}
true
}
/// Whether one path component passes the hidden and component-pattern
/// rules. `current` is its full path, which the attribute test needs.
fn component_allowed(&self, current: &Path, name: &str) -> bool {
// The metadata closure is only consulted on Windows, where hidden
// is an attribute rather than a leading dot; on Unix this stays at
// zero syscalls, exactly as it does in the walker.
//
// `symlink_metadata` for the same reason the walker uses
// `DirEntry::metadata`: the component is judged as itself, never as
// what it points at. Following here and not there is exactly the
// disagreement this whole type exists to avoid.
if !self.include_hidden
&& crate::platform::entry_is_hidden(name, || std::fs::symlink_metadata(current).ok())
{
return false;
}
!self.ignore.matches_component(name)
}
}
/// Directory verdicts already reached by [`Scope::covers_cached`].
///
/// Bounded rather than unbounded: a scan of a multi-million-row index would
/// otherwise hold every directory under every root at once, and the pass is
/// already paged precisely so it does not have to. Past the cap the map is
/// cleared outright instead of evicting one entry — rows arrive in roughly
/// insertion order, so the entries that matter are the ones just added, and a
/// clear costs one rebuild of a working set that is small by construction.
#[derive(Default)]
pub struct CoverCache {
dirs: std::collections::HashMap<PathBuf, bool>,
}
impl CoverCache {
/// Directories remembered before the map is cleared. Roughly 100 bytes per
/// entry, so this is a few megabytes at most.
const CAP: usize = 20_000;
fn get(&self, dir: &Path) -> Option<bool> {
self.dirs.get(dir).copied()
}
fn insert(&mut self, dir: PathBuf, allowed: bool) {
if self.dirs.len() >= Self::CAP {
self.dirs.clear();
}
self.dirs.insert(dir, allowed);
}
}
/// How far an in-progress [`advance`] has got.
@ -146,6 +239,13 @@ impl Scope {
/// coordinator's command loop for the length of a full scan; the caller hands
/// back the same cursor each tick with a fresh deadline, the way
/// `apply_pending` drains the watcher queue.
///
/// Resumable within one pass only. A cursor abandoned — cancelled, or dropped
/// on an error — takes its position with it, and the next attempt starts from
/// the beginning of a freshly derived plan; every part of the pass is
/// idempotent precisely so that this costs time and nothing else. A config edit
/// arriving mid-pass has the same effect (see `Inner::start_work`), which is
/// why the counters can go backwards between two published snapshots.
pub struct WorkCursor {
work: IndexWork,
scope: Scope,
@ -165,6 +265,15 @@ pub struct WorkCursor {
pub deleted: usize,
/// Rows whose content state or stored text was re-decided.
pub recontented: usize,
/// Rows the scan has re-tested against the current configuration.
examined: usize,
/// Rows in the index, counted once when the scan first needs a page and
/// then left alone — a denominator that moved would make the display walk
/// backwards. `None` until then, and for a plan that reads no rows at all:
/// removing a root is one whole-range delete with no intermediate state to
/// report, so it gets the indeterminate bar rather than an invented
/// percentage.
total: Option<usize>,
}
impl WorkCursor {
@ -179,6 +288,8 @@ impl WorkCursor {
finalized: false,
deleted: 0,
recontented: 0,
examined: 0,
total: None,
})
}
@ -186,6 +297,20 @@ impl WorkCursor {
self.finalized
}
/// A snapshot for the status the caller publishes.
///
/// Both places this work runs — a run's prologue and the coordinator's
/// between-runs pass — report the same figures from the same counters,
/// so the user sees one thing however the reconcile was reached.
pub fn progress(&self) -> ReconcileProgress {
ReconcileProgress {
examined: self.examined,
total: self.total,
deleted: self.deleted,
recontented: self.recontented,
}
}
/// Whether a full walk must follow this reconciliation.
pub fn reindex(&self) -> bool {
self.work.reindex
@ -207,16 +332,31 @@ impl WorkCursor {
/// Apply as much of `cursor` as fits before `deadline`, one page of rows per
/// transaction. Returns with the cursor advanced; call again until
/// [`WorkCursor::done`].
///
/// `cancel` means "do not start another statement": it is read before every
/// one, and a set flag returns immediately with the cursor un-finished. It is
/// only half of cutting the pass short — the statement already running answers
/// to [`crate::db::interrupt`] and nothing else — and it is deliberately not
/// the same thing as `deadline`, which hands control back to a caller that
/// intends to come straight back.
///
/// A cancelled pass leaves the work owed. Nothing here records what it did:
/// the stored configuration is stamped only by a caller that saw the cursor
/// finish, so the next run re-derives the same plan and picks it up.
pub fn advance(
conn: &mut Connection,
config: &Config,
registry: &Registry,
cursor: &mut WorkCursor,
deadline: Instant,
cancel: &AtomicBool,
) -> Result<(), String> {
// Whole ranges first: a removed root's rows can never satisfy the scan's
// filters anyway, and deleting them by range spares the scan the work.
while cursor.drop_idx < cursor.work.drop_roots.len() {
if cancelled(cancel) {
return Ok(());
}
let range = ExtractCursor::for_root(&cursor.work.drop_roots[cursor.drop_idx]);
let tx = conn
.transaction()
@ -231,6 +371,10 @@ pub fn advance(
}
}
if cancelled(cancel) {
return Ok(());
}
// Before the per-root scan and after the root deletions: the ranges it
// spares must already be the final set of roots.
if !cursor.dropped_aliases && cursor.work.drop_aliases {
@ -254,8 +398,24 @@ pub fn advance(
}
if cursor.work.scans_rows() {
// One count, the first time a page is actually needed. The scan it
// measures is a keyset walk of every row under every root, so on the
// indexes where this matters it is minutes of work against a count of
// seconds — and without it the display has no denominator at all.
if cursor.total.is_none() {
if cancelled(cancel) {
return Ok(());
}
cursor.total = Some(repo::row_count(conn)?);
}
let page = config.processing.batch_size.max(1) as i64;
// Lives across pages, not per page: consecutive pages walk the same
// directories, which is exactly the repetition worth remembering.
let mut covered = CoverCache::default();
while cursor.root_idx < cursor.scope.roots.len() {
if cancelled(cancel) {
return Ok(());
}
let root = &cursor.scope.roots[cursor.root_idx];
if cursor.after.is_empty() {
cursor.after = root.lo.clone();
@ -267,6 +427,7 @@ pub fn advance(
continue;
};
cursor.after = last.path.clone();
cursor.examined += rows.len();
let root = cursor.scope.roots[cursor.root_idx].path.clone();
let (deleted, recontented) = apply_page(
conn,
@ -276,6 +437,7 @@ pub fn advance(
&cursor.work,
&root,
&rows,
&mut covered,
)?;
cursor.deleted += deleted;
cursor.recontented += recontented;
@ -287,6 +449,12 @@ pub fn advance(
// Deletions leave the FTS index with tombstones and a long segment list;
// the same automerge that follows a run's stale cleanup collapses them.
// It is one statement that can run for a while, so it is the last thing
// the flag can spare the caller — and skipping it costs only tidiness,
// since the next run's own automerge collapses the same segments.
if cancelled(cancel) {
return Ok(());
}
if cursor.deleted > 0 || cursor.recontented > 0 {
fts_finalize_after_text_indexing(conn);
}
@ -294,7 +462,12 @@ pub fn advance(
Ok(())
}
fn cancelled(cancel: &AtomicBool) -> bool {
cancel.load(Ordering::Relaxed)
}
/// Decide and write one page of rows. Returns `(deleted, recontented)`.
#[allow(clippy::too_many_arguments)]
fn apply_page(
conn: &mut Connection,
config: &Config,
@ -303,6 +476,7 @@ fn apply_page(
work: &IndexWork,
root: &Path,
rows: &[repo::ScopeRow],
covered: &mut CoverCache,
) -> Result<(usize, usize), String> {
let mut doomed: Vec<i64> = Vec::new();
let mut stale_text: Vec<i64> = Vec::new();
@ -311,7 +485,7 @@ fn apply_page(
for row in rows {
let path = Path::new(&row.path);
if work.prune_scope && !scope.covers(root, path) {
if work.prune_scope && !scope.covers_cached(root, path, covered) {
doomed.push(row.id);
continue;
}
@ -388,18 +562,44 @@ pub fn stored_config(conn: &Connection, config: &Config) -> Result<Config, Strin
"store_text_for_snippets" => {
stored.processing.store_text_for_snippets = value == "true"
}
"hash_length" => {
if let Ok(n) = value.parse() {
stored.processing.hash_length = n;
}
}
"hash_length" => match value.parse() {
Ok(n) => stored.processing.hash_length = n,
// Keeping the caller's value is the safest of bad options,
// but it makes the reconciliation diff describe a config the
// index was not built under, so it must not pass in silence.
Err(e) => crate::log_warn!("stored hash_length {:?} unreadable: {}", value, e),
},
"tokenize" => stored.processing.tokenize = value,
// An unrecognized key is a record written by a newer build.
// Ignoring it leaves that setting at the caller's value, which is
// what a build that does not know the key would have used anyway.
_ => {}
}
}
Ok(stored)
}
/// The reconciliation the index still owes `config`, derived from its own
/// record of what it was last brought into line with.
///
/// The one question three callers ask in the same words: a run, to decide what
/// its prologue must do; a test, to check a pass recorded itself; and the GUI
/// at startup, to tell the user their settings have not reached the index yet.
/// Empty is the normal answer — every pass that finishes stamps the record —
/// so a non-empty one means a pass was abandoned, or the config was edited
/// while the app was closed. Either way the remedy is the same: run indexing.
///
/// The roots are canonicalized first, because that is the spelling the record
/// holds; comparing raw ones would report a `~` or a trailing slash as a
/// changed root.
pub fn outstanding_work(db_path: &str, config: &Config) -> Result<IndexWork, String> {
let conn = crate::db::open_existing(db_path, false)?;
let mut current = config.clone();
current.paths.indexing_paths = config.normalized_indexing_paths().into_iter().collect();
let stored = stored_config(&conn, &current)?;
Ok(crate::config::diff_actions(&stored, &current).work)
}
#[cfg(test)]
mod tests {
use super::*;
@ -409,18 +609,7 @@ mod tests {
use std::sync::Arc;
fn tmp_tree(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-scope-{}-{}-{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
std::fs::canonicalize(&p).unwrap()
crate::testutil::scratch_dir_canonical(tag)
}
fn touch(p: &Path) {
@ -456,6 +645,34 @@ mod tests {
.collect()
}
/// One `files` row per path, which is all the scan reads.
fn seed(conn: &mut Connection, paths: &[PathBuf]) {
let tx = conn.transaction().unwrap();
for path in paths {
let path = path.to_string_lossy();
let (parent, name) = path.rsplit_once('/').unwrap();
repo::insert_file(
&tx,
&repo::NewFile {
name,
path: &path,
parent,
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: Some("text/plain"),
ftype: crate::mime::FileType::TEXT,
hash: None,
needs_content: false,
},
)
.unwrap()
.expect("unique path");
}
tx.commit().unwrap();
}
/// Every file that physically exists under `root`, walker or no walker.
fn on_disk(root: &Path) -> Vec<PathBuf> {
walkdir::WalkDir::new(root)
@ -566,6 +783,156 @@ mod tests {
std::fs::remove_dir_all(&base).ok();
}
/// The counters the status display reads. A scan that reports nothing is
/// indistinguishable from a hang, and on the indexes where this pass takes
/// minutes that is exactly what it looked like.
#[test]
fn the_scan_reports_its_way_through_every_row() {
let root = tmp_tree("progress");
for i in 0..7 {
touch(&root.join(format!("f{}.log", i)));
}
touch(&root.join("keep.txt"));
let db_dir = tmp_tree("progress-db");
let db = empty_db(&db_dir);
let mut conn = crate::db::open_existing(db.to_str().unwrap(), true).unwrap();
let mut config = Config::default();
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
// One row per page, so the scan takes as many steps as there are rows
// and a counter that only moved at the end would be visible.
config.processing.batch_size = 1;
seed(&mut conn, &on_disk(&root));
let mut narrowed = config.clone();
narrowed.indexing.ignore_patterns = vec!["*.log".into()];
let work = crate::config::diff_actions(&config, &narrowed).work;
let mut cursor = WorkCursor::new(work, &narrowed).unwrap();
assert_eq!(
cursor.progress(),
ReconcileProgress::default(),
"nothing counted before the first slice"
);
let registry = Registry::default_set();
let run = AtomicBool::new(false);
let mut seen: Vec<usize> = Vec::new();
while !cursor.done() {
// A deadline already past, so each call does the least it can and
// the counters are sampled at their finest granularity.
advance(
&mut conn,
&narrowed,
&registry,
&mut cursor,
Instant::now(),
&run,
)
.unwrap();
seen.push(cursor.progress().examined);
}
let end = cursor.progress();
assert_eq!(end.total, Some(8), "counted once, before the first page");
assert_eq!(end.examined, 8, "every row was re-tested");
assert_eq!(end.deleted, 7, "the logs, and only the logs");
assert!(
seen.windows(2).all(|w| w[0] <= w[1]),
"the count never goes backwards: {:?}",
seen
);
assert!(
seen.len() > 2 && seen[0] < end.examined,
"progress was reported during the scan, not only at its end: {:?}",
seen
);
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// Cancelling stops the pass at the next statement boundary and leaves the
/// cursor un-finished, so nothing downstream can mistake it for done and
/// record the configuration as reconciled. The rows it had already reached
/// stay gone — every part of the pass is idempotent, and the next run
/// re-derives the same plan and finishes it.
#[test]
fn cancelling_stops_the_scan_without_finishing_it() {
let root = tmp_tree("cancel");
for i in 0..6 {
touch(&root.join(format!("f{}.log", i)));
}
touch(&root.join("keep.txt"));
let db_dir = tmp_tree("cancel-db");
let db = empty_db(&db_dir);
let mut conn = crate::db::open_existing(db.to_str().unwrap(), true).unwrap();
let mut config = Config::default();
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
config.processing.batch_size = 1;
seed(&mut conn, &on_disk(&root));
let mut narrowed = config.clone();
narrowed.indexing.ignore_patterns = vec!["*.log".into()];
let work = crate::config::diff_actions(&config, &narrowed).work;
let registry = Registry::default_set();
// Cancelled from the outset: not one statement runs.
let stop = AtomicBool::new(true);
let mut cursor = WorkCursor::new(work.clone(), &narrowed).unwrap();
advance(
&mut conn,
&narrowed,
&registry,
&mut cursor,
Instant::now() + SLICE,
&stop,
)
.unwrap();
assert!(!cursor.done(), "a cancelled pass is never finished");
assert_eq!(
cursor.progress(),
ReconcileProgress::default(),
"a cancelled pass touched the index"
);
// And part-way through: one slice with the flag clear, the rest with
// it set. The counters keep what the first slice earned.
let stop = AtomicBool::new(false);
let mut cursor = WorkCursor::new(work, &narrowed).unwrap();
advance(
&mut conn,
&narrowed,
&registry,
&mut cursor,
Instant::now(),
&stop,
)
.unwrap();
let part_way = cursor.progress();
assert!(part_way.examined > 0 && !cursor.done(), "nothing to cancel");
stop.store(true, Ordering::Relaxed);
advance(
&mut conn,
&narrowed,
&registry,
&mut cursor,
Instant::now() + SLICE,
&stop,
)
.unwrap();
assert!(!cursor.done(), "the pass finished despite the cancellation");
assert_eq!(
cursor.progress(),
part_way,
"the cancelled slice did more work"
);
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// A path under no configured root has no rules that could be applied to
/// it — a followed symlink's target is the real case. The scan reaches it
/// by never visiting it, so `owning_root` returning `None` is what keeps

80
crates/quicksearch-core/src/search/cascade.rs
View file

@ -213,6 +213,29 @@ fn count_frac(count: usize) -> f64 {
(1000usize.saturating_sub(count.min(1000))) as f64 / 1000.0
}
/// The `files` columns every pass selects, in the order the passes index
/// them: `0` id, `1` name, `2` path, `3` size, `4` mtime. Passes that also
/// want the stored document text append `dt.text_zstd` as column `5`.
///
/// One string rather than seven copies, because the column *order* is what
/// every `row.get(n)` in this file is written against — a pass that spelled
/// its own list in a different order would compile and then quietly serve
/// paths as names.
const HIT_COLUMNS: &str = "f.id, f.name, f.path, f.size, f.mtime";
/// Columns 3 and 4: the two every pass reads identically and stores without
/// inspecting.
///
/// The clamp is the point of having this in one place. `size` is `INTEGER` in
/// SQLite and so signed; a corrupt or hand-edited row holding `-1` would
/// otherwise become 18 exabytes on the way to `u64` and sort to the top of
/// every size-ordered result.
fn size_and_mtime(row: &rusqlite::Row<'_>) -> Result<(u64, i64), String> {
let size = row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64;
let mtime = row.get(4).map_err(|e| e.to_string())?;
Ok((size, mtime))
}
/// The path tiers only make sense with enough term to be specific — same
/// floor the trigram full-text pass uses. Wildcards count only their
/// literal content (`a*b` is two characters of specificity, not three).
@ -439,8 +462,9 @@ impl<'a> Cx<'a> {
// A path always ends in its own name, so `path LIKE` is the
// superset that feeds both the name and the path tiers.
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f \
"SELECT {} FROM files f \
WHERE {} LIKE ? ESCAPE '\\'{}",
HIT_COLUMNS,
if with_paths { "f.path" } else { "f.name" },
query.filter_sql
);
@ -469,7 +493,7 @@ impl<'a> Cx<'a> {
let mut clock = FlushClock::new();
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % CANCEL_CHECK_ROWS == 0 && self.cancelled() {
if scanned.is_multiple_of(CANCEL_CHECK_ROWS) && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
@ -519,12 +543,13 @@ impl<'a> Cx<'a> {
truncated_start: false,
truncated_end: false,
};
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank,
stage: rank as u8,
snippet: Some(snip),
@ -586,20 +611,22 @@ impl<'a> Cx<'a> {
let (sql, params) = match match_expr {
Some(expr) => (
format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
"SELECT {}, dt.text_zstd \
FROM searchabletext \
JOIN files f ON f.id = searchabletext.rowid \
LEFT JOIN documents_text dt ON dt.file_id = f.id \
WHERE searchabletext MATCH ?{}",
HIT_COLUMNS,
query.filter_sql
),
self.params_with_filters(vec![rusqlite::types::Value::Text(expr)]),
),
None => (
format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
"SELECT {}, dt.text_zstd \
FROM documents_text dt \
JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
HIT_COLUMNS,
query.filter_sql
),
self.params_with_filters(Vec::new()),
@ -692,12 +719,13 @@ impl<'a> Cx<'a> {
continue;
}
let (size, mtime) = size_and_mtime(row)?;
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank,
stage,
snippet: snip,
@ -733,7 +761,8 @@ impl<'a> Cx<'a> {
let with_paths = path_tiers_enabled(&self.query.pattern);
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
"SELECT {} FROM files f WHERE 1=1{}",
HIT_COLUMNS,
self.query.filter_sql
);
let params = self.params_with_filters(Vec::new());
@ -750,7 +779,7 @@ impl<'a> Cx<'a> {
let mut clock = FlushClock::new();
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % 1024 == 0 && self.cancelled() {
if scanned.is_multiple_of(1024) && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
@ -789,12 +818,13 @@ impl<'a> Cx<'a> {
},
));
let is_path_tier = rank >= 11.0;
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank,
stage: rank as u8,
snippet: snip,
@ -838,8 +868,9 @@ impl<'a> Cx<'a> {
};
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
"SELECT {}, dt.text_zstd \
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
HIT_COLUMNS,
self.query.filter_sql
);
let params = self.params_with_filters(Vec::new());
@ -882,12 +913,13 @@ impl<'a> Cx<'a> {
continue;
}
let snip = first.map(|range| snippet::window_around(&text, range, &snippet_opts));
let (size, mtime) = size_and_mtime(row)?;
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank: 8.0 + count_frac(count),
stage: 8,
snippet: snip,
@ -911,7 +943,8 @@ impl<'a> Cx<'a> {
let query = self.query;
let re = query.regex.as_ref().expect("regex-only pass list");
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
"SELECT {} FROM files f WHERE 1=1{}",
HIT_COLUMNS,
query.filter_sql
);
let params = self.params_with_filters(Vec::new());
@ -928,7 +961,7 @@ impl<'a> Cx<'a> {
let mut clock = FlushClock::new();
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % 1024 == 0 && self.cancelled() {
if scanned.is_multiple_of(1024) && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
@ -956,12 +989,13 @@ impl<'a> Cx<'a> {
truncated_start: false,
truncated_end: false,
};
let (size, mtime) = size_and_mtime(row)?;
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank,
stage: rank as u8,
snippet: Some(snip),
@ -992,8 +1026,9 @@ impl<'a> Cx<'a> {
let query = self.query;
let re = query.regex.as_ref().expect("regex-only pass list");
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
"SELECT {}, dt.text_zstd \
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
HIT_COLUMNS,
query.filter_sql
);
let params = self.params_with_filters(Vec::new());
@ -1033,12 +1068,13 @@ impl<'a> Cx<'a> {
let r = clamp_match_range(&text, r, SNIPPET_WINDOW_CHARS);
snippet::window_around(&text, (r.start, r.end), &snippet_opts)
});
let (size, mtime) = size_and_mtime(row)?;
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
size,
mtime,
rank: 6.0 + count_frac(count),
stage: 6,
snippet: snip,

10
crates/quicksearch-core/src/search/duplicates.rs
View file

@ -88,15 +88,7 @@ mod tests {
use crate::mime::FileType;
fn seed_db() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-dups-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let p = crate::testutil::scratch_dir("dups").join("index.sqlite");
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
let tx = conn.transaction().unwrap();
let add = |name: &str, path: &str, size: u64, hash: Option<&[u8]>| {

6
crates/quicksearch-core/src/search/fuzzy.rs
View file

@ -320,9 +320,9 @@ mod tests {
}
std::mem::swap(&mut prev, &mut cur);
}
for j in 0..=hay.len() {
best = best.min(prev[j]);
}
// The best occurrence may end at any text position, so the answer is
// the smallest value in the final row.
best = prev.iter().copied().min().unwrap_or(best);
if best <= k {
Some(best)
} else {

4
crates/quicksearch-core/src/security.rs
View file

@ -47,7 +47,7 @@ impl IndexKey {
hex_encode(&self.0)
}
/// Strict inverse of [`to_hex`]: exactly 64 hex digits, any case.
/// Strict inverse of [`IndexKey::to_hex`]: exactly 64 hex digits, any case.
pub fn from_hex(hex: &str) -> Result<IndexKey, String> {
let bytes = hex_decode(hex)?;
let arr: [u8; KEY_LEN] = bytes
@ -102,7 +102,7 @@ fn hex_encode(bytes: &[u8]) -> String {
}
fn hex_decode(hex: &str) -> Result<Vec<u8>, String> {
if hex.len() % 2 != 0 {
if !hex.len().is_multiple_of(2) {
return Err("hex string has odd length".to_string());
}
if !hex.bytes().all(|b| b.is_ascii_hexdigit()) {

76
crates/quicksearch-core/src/testutil.rs Normal file
View file

@ -0,0 +1,76 @@
//! Scratch directories for tests.
//!
//! Public and `#[doc(hidden)]` rather than `#[cfg(test)]`: the `tests/`
//! integration binaries and the GUI crate are separate compilation units, so a
//! test-gated item here would be invisible to them. This is the only reason it
//! is not private.
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
/// Distinguishes directories requested within one process. Two tests running
/// on different threads in the same millisecond would otherwise collide — the
/// hand-rolled helpers this replaces all keyed off a timestamp, which made
/// that rare rather than impossible.
static NEXT: AtomicUsize = AtomicUsize::new(0);
/// A fresh, empty directory under the system temp dir, named for `tag`.
///
/// Not cleaned up on drop, deliberately: when a test fails, the tree it built
/// is most of the evidence. The OS clears the temp dir eventually.
///
/// Panics rather than returning a `Result` — a test that cannot create a
/// directory has nothing left to assert.
#[doc(hidden)]
pub fn scratch_dir(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-{}-{}-{}",
tag,
std::process::id(),
NEXT.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&p).expect("create scratch dir");
p
}
/// [`scratch_dir`] canonicalized, for the tests that compare walked paths
/// against the root they were given. On macOS `/tmp` is a symlink to
/// `/private/tmp`, so an uncanonicalized root and a walked path disagree.
#[doc(hidden)]
pub fn scratch_dir_canonical(tag: &str) -> PathBuf {
std::fs::canonicalize(scratch_dir(tag)).expect("canonicalize scratch dir")
}
/// Write `body` to `path`, creating parent directories as needed.
#[doc(hidden)]
pub fn touch(path: &std::path::Path, body: &[u8]) {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent).expect("create parent dir");
}
std::fs::write(path, body).expect("write file");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn every_call_gets_its_own_empty_directory() {
let a = scratch_dir("selftest");
let b = scratch_dir("selftest");
assert_ne!(a, b, "two calls must not collide");
for d in [&a, &b] {
assert!(d.is_dir());
assert_eq!(std::fs::read_dir(d).unwrap().count(), 0, "starts empty");
}
}
#[test]
fn touch_creates_missing_parents() {
let dir = scratch_dir("selftest-touch");
let deep = dir.join("a/b/c.txt");
touch(&deep, b"hi");
assert_eq!(std::fs::read(&deep).unwrap(), b"hi");
}
}

461
crates/quicksearch-core/src/walk.rs
View file

@ -27,7 +27,7 @@
use std::collections::HashSet;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::{mpsc, Arc, Condvar, Mutex};
use std::thread::{self, JoinHandle};
use std::time::UNIX_EPOCH;
@ -100,6 +100,11 @@ impl WalkedFile {
/// What the walk emits. Files as they are classified, plus the per-directory
/// verdict on which index rows no longer have a file behind them.
// `File` is far larger than the deletion variant, and deliberately so: this
// is the walk's hot path, one event per file on a tree of millions. Boxing to
// even out the variants would add exactly the per-file allocation the
// owned-record design exists to avoid.
#[allow(clippy::large_enum_variant)]
#[derive(Debug)]
pub enum WalkEvent {
File(WalkedFile),
@ -109,6 +114,27 @@ pub enum WalkEvent {
Stale(Vec<String>),
}
/// One file a directory read produced: its path, plus whatever that read
/// already told us about it.
///
/// The metadata half is `Some` only on Windows, where `FindNextFileW` returned
/// size, mtime and attributes alongside the name. On Unix `getdents64` returns
/// only `d_type`, so it is always `None`, [`prepare`] does the single `statx`
/// the walk has always done, and `Option<CachedMetadata>` is zero-sized — a
/// queued file costs exactly the `PathBuf` it cost before.
struct PendingFile {
path: PathBuf,
cached: Option<crate::platform::CachedMetadata>,
}
impl PendingFile {
/// A path that did not come from a directory entry — a resolved symlink
/// target — so there is nothing cached to carry.
fn uncached(path: PathBuf) -> Self {
PendingFile { path, cached: None }
}
}
/// Work waiting for a thread.
enum Job {
/// Read this directory and process its files. Carries the directory's
@ -116,8 +142,10 @@ enum Job {
Dir(PathBuf, Arc<DirRows>),
/// Process this slice of one directory's files, split off because the
/// directory was too wide for one worker to be worth serialising on.
/// Shares its directory's rows.
Files(Vec<PathBuf>, Arc<DirRows>),
/// Shares its directory's rows. On Windows each entry also carries the
/// metadata its directory read returned, so classifying these files costs
/// no syscall at all.
Files(Vec<PendingFile>, Arc<DirRows>),
/// A resolved symlink target, with the stored mtime for its own path.
/// Classified against that rather than against any directory's rows.
Alias(PathBuf, Option<u64>),
@ -151,6 +179,12 @@ struct Queue {
/// directory pushes many of them. Bounding on the total would let file
/// chunks starve directory prefetching and leave the pool waiting on
/// rows that were never fetched.
///
/// The consequence is that file chunks, not this counter, are what bounds
/// the walker's memory: nothing throttles them, so one very wide directory
/// can hold its whole listing in `jobs` at once — and on Windows each of
/// those entries also carries its directory read's metadata
/// ([`PendingFile`]), so that is the larger cost of the two.
dirs_ready: usize,
/// Workers currently holding a job — that is, workers that may still push
/// more. The walk is over when this is zero and `jobs` is empty.
@ -342,7 +376,7 @@ enum Found {
/// A resolved symlink target. Needs an exact-path mtime lookup.
Alias(PathBuf),
/// Overflow files from the directory just read, which already has rows.
Files(Vec<PathBuf>, Arc<DirRows>),
Files(Vec<PendingFile>, Arc<DirRows>),
}
impl Shared {
@ -398,10 +432,57 @@ impl Drop for ActiveJob<'_> {
}
}
/// How many entries each filter rejected, for the one-line summary a run logs
/// when it finishes.
///
/// Counted rather than logged per entry, deliberately. The process log is a
/// 5,000-line ring ([`crate::log::CAPACITY`]) that evicts oldest-first with no
/// level protection, so a line per ignored entry would push every warning —
/// including the unreadable-directory ones that distinguish a network blip from
/// a deletion — out of the buffer before the run ended. `log::record` also takes
/// a global mutex and writes to stderr, which would serialize the worker pool on
/// the one path the walker is built to keep syscall-free.
///
/// A pruned *directory* is one increment, not one per file beneath it: the
/// subtree is never enumerated.
#[derive(Debug, Default)]
pub struct PruneCounts {
/// Names beginning with a dot.
pub dot_named: AtomicU64,
/// Windows entries carrying `FILE_ATTRIBUTE_HIDDEN`.
pub attribute: AtomicU64,
/// Rejected by a configured ignore pattern, of either kind.
pub ignored: AtomicU64,
}
impl PruneCounts {
pub fn total(&self) -> u64 {
self.dot_named.load(Ordering::Relaxed)
+ self.attribute.load(Ordering::Relaxed)
+ self.ignored.load(Ordering::Relaxed)
}
/// The summary line, or `None` when nothing was pruned — a clean tree
/// should not add noise to the log.
pub fn summary(&self) -> Option<String> {
if self.total() == 0 {
return None;
}
Some(format!(
"pruned {} entries: {} hidden by attribute, {} dot-named, {} by ignore pattern",
self.total(),
self.attribute.load(Ordering::Relaxed),
self.dot_named.load(Ordering::Relaxed),
self.ignored.load(Ordering::Relaxed),
))
}
}
struct Ctx {
follow_symlinks: bool,
include_hidden: bool,
ignore: IgnoreSet,
pruned: PruneCounts,
config: Config,
/// Lets a worker finish small text files outright: the head it reads to
/// hash them is already their entire contents, so an extractor that works
@ -412,6 +493,16 @@ struct Ctx {
suspend_flag: Arc<AtomicBool>,
}
/// Individual unreadable-directory warnings allowed per run before only the
/// count is kept. Reset by [`reset_run_warnings`].
static UNREADABLE_WARNINGS: crate::log::Throttle = crate::log::Throttle::new(20);
/// Arm this module's per-run warning throttle. See
/// [`crate::file_handling::reset_run_warnings`], which the same caller invokes.
pub fn reset_run_warnings() {
UNREADABLE_WARNINGS.reset();
}
/// Read one directory, apply the hidden/ignore rules, and split the result:
/// subdirectories and overflow file chunks go to `found` for the pool, the
/// remaining files come back for this worker to handle immediately.
@ -425,18 +516,27 @@ struct Ctx {
/// A directory that cannot be read returns before reconciling, so nothing
/// under it is ever deleted — an unreadable directory must not read as an
/// empty one.
///
/// Each returned file carries whatever this read already told us about it, so
/// that [`prepare`] does not have to ask the filesystem twice. See
/// [`PendingFile`].
fn read_directory(
dir: &Path,
rows: &Arc<DirRows>,
ctx: &Ctx,
found: &mut Vec<Found>,
stale: &mut Vec<String>,
) -> Vec<PathBuf> {
) -> Vec<PendingFile> {
let entries = match fs::read_dir(dir) {
Ok(entries) => entries,
Err(e) => {
// Not the same as "this directory is empty": see UnreadableDirs.
crate::log_warn!("cannot read {}: {}", dir.display(), e);
// Throttled because a permission-denied subtree produces one of
// these per directory; `UnreadableDirs` records every one of them
// regardless, and the run reports the total.
if UNREADABLE_WARNINGS.allow() {
crate::log_warn!("cannot read {}: {}", dir.display(), e);
}
ctx.unreadable.record(dir.to_path_buf());
return Vec::new();
}
@ -452,7 +552,9 @@ fn read_directory(
let entry = match entry {
Ok(entry) => entry,
Err(e) => {
crate::log_warn!("cannot read an entry of {}: {}", dir.display(), e);
if UNREADABLE_WARNINGS.allow() {
crate::log_warn!("cannot read an entry of {}: {}", dir.display(), e);
}
ctx.unreadable.record(dir.to_path_buf());
// The listing is now incomplete, so it cannot be used to
// decide what is missing: an entry we failed to read would
@ -465,17 +567,45 @@ fn read_directory(
let name = entry.file_name();
let name = name.to_string_lossy();
// `entry.metadata()` is only consulted on Windows, where it is free —
// the attributes came back with the directory read. On Unix the
// closure is never called, so this stays at zero extra syscalls.
if !ctx.include_hidden && crate::platform::entry_is_hidden(&name, || entry.metadata().ok())
{
continue;
// the attributes came back with the directory read, and it reports the
// entry itself rather than a link target, which is what
// `entry_hidden_reason` requires. On Unix the closure is never called,
// so this stays at zero extra syscalls.
if !ctx.include_hidden {
if let Some(reason) =
crate::platform::entry_hidden_reason(&name, || entry.metadata().ok())
{
match reason {
crate::platform::HiddenReason::DotPrefix => {
ctx.pruned.dot_named.fetch_add(1, Ordering::Relaxed);
}
crate::platform::HiddenReason::Attribute => {
ctx.pruned.attribute.fetch_add(1, Ordering::Relaxed);
// Only the attribute case is announced, and only for a
// directory. A dot prefix explains itself and an ignore
// pattern is something the user typed, but a plainly
// visible folder skipped over an attribute Explorer does
// not show has no other way of being discovered — which
// is how a whole cloud-sync tree went missing in silence.
if entry.file_type().is_ok_and(|ft| ft.is_dir()) {
crate::log_info!(
"skipping {}: hidden attribute set (enable \"include hidden \
files\" to index it)",
entry.path().display()
);
}
}
}
continue;
}
}
if ctx.ignore.matches_component(&name) {
ctx.pruned.ignored.fetch_add(1, Ordering::Relaxed);
continue;
}
let path = entry.path();
if ctx.ignore.matches_path_pattern(&path) {
ctx.pruned.ignored.fetch_add(1, Ordering::Relaxed);
continue;
}
@ -507,12 +637,12 @@ fn read_directory(
// path is what the index stores, and pushing only canonical
// directories is what keeps `seen_dirs` able to break cycles.
//
// Normalized like the roots (walk_parallel), or on Windows the
// target keeps `canonicalize`'s `\\?\` prefix: every path below
// it would be spelled differently from the plainly-spelled
// roots, so full-path ignore patterns would never match under a
// followed junction and `seen_dirs` could not dedup against an
// overlapping root.
// Normalized like the roots (`walk_indexable_files`), or on
// Windows the target keeps `canonicalize`'s `\\?\` prefix:
// every path below it would be spelled differently from the
// plainly-spelled roots, so full-path ignore patterns would
// never match under a followed junction and `seen_dirs` could
// not dedup against an overlapping root.
if let Ok(target) = path.canonicalize() {
let target = PathBuf::from(path_to_db_string(&target));
match fs::metadata(&target) {
@ -528,7 +658,13 @@ fn read_directory(
}
Ok(_) => {
present.insert(name.into_owned());
files.push(path);
// Windows already sent this file's size and mtime back with its
// name, and `DirEntry::metadata` there is a copy of that buffer
// rather than a syscall — so carrying it means `prepare` never
// opens the file just to ask the same question again. `None` on
// Unix, and on any reparse point: see `entry_cached_metadata`.
let cached = crate::platform::entry_cached_metadata(|| entry.metadata().ok());
files.push(PendingFile { path, cached });
}
// Type unknown: the entry exists but we could not classify it.
// Mark it present so an existing row survives — seen, not deleted.
@ -583,9 +719,14 @@ pub fn path_digest(path: &str) -> u128 {
u128::from_be_bytes(bytes)
}
/// One `stat`, then classify; only files that are actually going to be
/// At most one `stat`, then classify; only files that are actually going to be
/// written get opened, and small text files are finished outright.
fn prepare(path: PathBuf, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
///
/// "At most", because on Windows the directory read already answered the
/// question and [`PendingFile::cached`] carries the answer — see
/// [`crate::platform::metadata_or_stat`].
fn prepare(file: PendingFile, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
let PendingFile { path, cached } = file;
let db_path = path_to_db_string(&path);
let digest = path_digest(&db_path);
let aliased = matches!(known, Known::Exact(_));
@ -598,7 +739,11 @@ fn prepare(path: PathBuf, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
return WalkedFile::skipped(db_path, digest, aliased);
}
let Ok(meta) = fs::metadata(&path) else {
// On Windows this is the directory read's own copy of the entry: an
// unchanged file now costs no syscall at all, and a changed one costs only
// the open the hasher was going to do anyway. On Unix, and for every
// reparse point, it is the same single `stat` as before.
let Ok(meta) = crate::platform::metadata_or_stat(&path, cached) else {
// Seen but unreadable. Emitting it anyway keeps its index row alive:
// a transient stat failure must not read as "deleted".
return WalkedFile::skipped(db_path, digest, aliased);
@ -663,8 +808,11 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
Job::Files(files, rows) => (files, rows),
Job::Alias(path, stored) => {
slot.finish(found);
// Reached through `canonicalize`, not through a directory
// entry, so there is nothing cached to carry.
let file = PendingFile::uncached(path);
if tx
.send(WalkEvent::File(prepare(path, Known::Exact(stored), ctx)))
.send(WalkEvent::File(prepare(file, Known::Exact(stored), ctx)))
.is_err()
{
shared.shutdown();
@ -684,13 +832,13 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
return;
}
for path in files {
for file in files {
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
shared.shutdown();
return;
}
if tx
.send(WalkEvent::File(prepare(path, Known::InDir(&rows), ctx)))
.send(WalkEvent::File(prepare(file, Known::InDir(&rows), ctx)))
.is_err()
{
// Receiver gone: the run was stopped or failed. Not an error.
@ -705,7 +853,8 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
/// read-only connection.
///
/// One per walk. The alternative — a connection per worker — would multiply
/// SQLite's page cache by the pool size; see [`PRAGMAS_WALK_READER`]. Every
/// SQLite's page cache by the pool size; see
/// [`crate::db::schema::PRAGMAS_WALK_READER`]. Every
/// query here is a single index lookup, so one thread stays far ahead of a
/// pool bound by `stat` latency.
///
@ -753,6 +902,9 @@ fn prefetcher(shared: &Shared, db_path: &str) {
/// stops the workers and joins them.
pub struct ParallelWalk {
rx: Option<mpsc::Receiver<WalkEvent>>,
/// One event pulled off the channel by [`ParallelWalk::wait_ready`] and not
/// yet handed to [`ParallelWalk::try_next`]. See `wait_ready`.
pending: Option<WalkEvent>,
handles: Vec<JoinHandle<()>>,
/// Joined by [`ParallelWalk::finish`] alongside the workers. Held
/// separately only so a failure to open its connection is attributable.
@ -768,6 +920,12 @@ impl ParallelWalk {
&self.ctx.unreadable
}
/// How many entries each filter rejected. Final on the same terms as
/// [`ParallelWalk::unreadable`].
pub fn pruned(&self) -> &PruneCounts {
&self.ctx.pruned
}
/// Every canonical directory the walk queued, in `files.parent` spelling.
///
/// The caller's vanished-directory sweep needs this: a directory deleted
@ -802,7 +960,10 @@ impl ParallelWalk {
/// identical from the receiving end: both close the channel, so iteration
/// simply ends. Treating a panicked walk as a completed one would hand
/// stale cleanup a partial file set and delete everything the dead workers
/// never reached.
/// never reached. So: join before deciding anything about what the walk
/// saw. This is the reference statement of that rule; the content pass
/// ([`crate::content::ContentPass::finish`]) and the writer loop's
/// `TryNext::Finished` arms follow it.
pub fn finish(&mut self) -> bool {
// Dropping the receiver first releases any worker parked in `send`.
self.rx = None;
@ -852,18 +1013,73 @@ pub(crate) fn try_recv_next<T>(rx: Option<&mpsc::Receiver<T>>) -> TryNext<T> {
}
}
/// [`try_recv_next`], but willing to wait up to `timeout` for something to
/// arrive.
///
/// What the writer loop backs off with instead of `thread::sleep`. A sleep is
/// the wrong instrument twice over: it ignores work that lands a microsecond
/// later, and on Windows the default timer resolution is 15.6 ms, so a 2 ms
/// backoff actually stalls for 15.6 — nearly eight times the intended pause,
/// every time the channels run momentarily dry. `recv_timeout` parks on the
/// channel's own condition variable, so a sender wakes it immediately and the
/// timeout is only the ceiling.
pub(crate) fn recv_next_timeout<T>(
rx: Option<&mpsc::Receiver<T>>,
timeout: std::time::Duration,
) -> TryNext<T> {
match rx {
None => TryNext::Finished,
Some(rx) => match rx.recv_timeout(timeout) {
Ok(item) => TryNext::Item(item),
Err(mpsc::RecvTimeoutError::Timeout) => TryNext::Empty,
Err(mpsc::RecvTimeoutError::Disconnected) => TryNext::Finished,
},
}
}
impl ParallelWalk {
/// Non-blocking variant of `next`, for callers multiplexing several
/// walks (the per-root writer loop).
pub fn try_next(&mut self) -> TryNext<WalkEvent> {
if let Some(event) = self.pending.take() {
return TryNext::Item(event);
}
try_recv_next(self.rx.as_ref())
}
/// Wait up to `timeout` for this walk to produce something, holding
/// whatever arrives for the next [`ParallelWalk::try_next`].
///
/// The writer loop's idle backoff. It multiplexes several walks, so it
/// cannot simply block on one of them and consume the result — hence the
/// one-slot pushback: the event is taken off the channel, but the loop still
/// sees it in its normal round-robin order.
///
/// Returns whether anything is now ready.
pub fn wait_ready(&mut self, timeout: std::time::Duration) -> bool {
if self.pending.is_some() {
return true;
}
match recv_next_timeout(self.rx.as_ref(), timeout) {
TryNext::Item(event) => {
self.pending = Some(event);
true
}
// A finished walk is "ready" in the sense the caller cares about:
// there is something to do (notice it ended), so do not keep waiting.
TryNext::Finished => true,
TryNext::Empty => false,
}
}
}
impl Iterator for ParallelWalk {
type Item = WalkEvent;
fn next(&mut self) -> Option<WalkEvent> {
if let Some(event) = self.pending.take() {
return Some(event);
}
self.rx.as_ref()?.recv().ok()
}
}
@ -938,6 +1154,7 @@ pub fn walk_indexable_files(
follow_symlinks,
include_hidden,
ignore,
pruned: PruneCounts::default(),
config,
registry,
unreadable: UnreadableDirs::default(),
@ -976,6 +1193,7 @@ pub fn walk_indexable_files(
ParallelWalk {
rx: Some(rx),
pending: None,
handles,
prefetch: Some(prefetch),
shared,
@ -1007,18 +1225,7 @@ mod tests {
use super::*;
fn tmp_tree(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-pwalk-{}-{}-{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
fs::create_dir_all(&p).unwrap();
p
crate::testutil::scratch_dir(tag)
}
fn touch(p: &Path) {
@ -1032,16 +1239,7 @@ mod tests {
/// caller passes in, so these tests build the state they are testing
/// against the same way the indexer does.
fn db_with(tag: &str, rows: &[(String, u64)]) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-pwalk-db-{}-{}-{}.sqlite",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let p = crate::testutil::scratch_dir(tag).join("index.sqlite");
let conn = crate::db::open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
for (path, mtime) in rows {
let as_path = Path::new(path);
@ -1234,6 +1432,74 @@ mod tests {
fs::remove_dir_all(&root).ok();
}
/// The walk's mtime and a `stat`'s mtime must be the same number.
///
/// On Windows the walk now reads mtime out of the directory entry while the
/// *watcher* writes its rows from `fs::metadata`; if the two ever disagreed,
/// every run would reclassify files nothing had touched and the index would
/// churn forever. Seeding the index the watcher's way and demanding the walk
/// call every file `Skip` is what pins them together.
///
/// `unchanged_files_are_never_opened` cannot catch this on its own: both of
/// its walks read from the same source, so they agree with each other even
/// when both disagree with a stat. Vacuous on Unix, where there is only ever
/// one source; on Windows it is the whole guarantee.
#[test]
fn a_walk_agrees_with_a_stat_seeded_index() {
let root = tmp_tree("stat-seeded");
touch(&root.join("a.txt"));
touch(&root.join("sub/b.txt"));
let seeded: Vec<(String, u64)> = [root.join("a.txt"), root.join("sub/b.txt")]
.iter()
.map(|p| {
let mtime = fs::metadata(p)
.unwrap()
.modified()
.unwrap()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs();
(path_to_db_string(p), mtime)
})
.collect();
let files = walk(&root, &db_with("stat-seeded", &seeded));
assert_eq!(files.len(), 2);
for f in &files {
assert_eq!(
f.action,
FileIndexAction::Skip,
"the directory read disagreed with a stat about {}",
f.path
);
assert!(f.record.is_none());
}
fs::remove_dir_all(&root).ok();
}
/// Windows: the three fields `prepare` and `prepare_file_record` read out of
/// the cached buffer, pinned against what a `stat` would have said — for the
/// case that now skips the `stat` entirely.
#[test]
#[cfg(windows)]
fn cached_directory_metadata_matches_a_stat_field_for_field() {
let root = tmp_tree("cached-meta");
touch(&root.join("a.txt"));
fs::write(root.join("b.bin"), vec![0u8; 5000]).unwrap();
for entry in fs::read_dir(&root).unwrap() {
let entry = entry.unwrap();
let cached = crate::platform::entry_cached_metadata(|| entry.metadata().ok())
.expect("a plain file is served from the directory read");
let fresh = fs::metadata(entry.path()).unwrap();
assert_eq!(cached.is_file(), fresh.is_file());
assert_eq!(cached.len(), fresh.len());
assert_eq!(cached.modified().unwrap(), fresh.modified().unwrap());
}
fs::remove_dir_all(&root).ok();
}
#[test]
fn every_seen_file_is_reported_even_when_it_cannot_be_read() {
// A path missing from the stream gets its index row deleted, so
@ -1472,6 +1738,103 @@ mod tests {
fs::remove_dir_all(&root).ok();
}
/// The counters behind the one-line summary a run logs.
///
/// The property worth pinning is that a pruned *directory* costs one
/// increment rather than one per file beneath it — the subtree is never
/// enumerated, which is exactly why logging per entry was rejected and
/// counting was not.
#[test]
fn pruned_entries_are_counted_by_reason() {
let root = tmp_tree("prune-counts");
touch(&root.join("keep.txt"));
touch(&root.join("sub/keep2.txt"));
touch(&root.join("sub/skip.tmp"));
// Two files below, one prune.
touch(&root.join(".hidden/inside.txt"));
touch(&root.join(".hidden/also-inside.txt"));
touch(&root.join(".dotfile"));
// Three levels below, still one prune.
touch(&root.join("node_modules/dep/lib/index.js"));
let ignore =
IgnoreSet::compile(&["*.tmp".to_string(), "node_modules".to_string()]).unwrap();
let mut walk = walk_indexable_files(
&[root.to_string_lossy().into_owned()],
false,
false,
ignore,
empty_db("prune-counts").to_str().unwrap(),
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = (&mut walk)
.filter_map(|e| match e {
WalkEvent::File(f) => Some(f),
WalkEvent::Stale(_) => None,
})
.collect();
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
let pruned = walk.pruned();
assert_eq!(
pruned.dot_named.load(Ordering::Relaxed),
2,
"`.hidden` and `.dotfile` — not the two files inside `.hidden`"
);
assert_eq!(
pruned.ignored.load(Ordering::Relaxed),
2,
"`skip.tmp` and `node_modules` — not `index.js` three levels down"
);
// Attributes are a Windows concept; on Linux nothing can reach this
// counter, and on Windows a temp tree carries no Hidden bit.
assert_eq!(pruned.attribute.load(Ordering::Relaxed), 0);
assert_eq!(pruned.total(), 4);
let summary = pruned.summary().expect("something was pruned");
assert!(summary.contains("4 entries"), "{}", summary);
fs::remove_dir_all(&root).ok();
}
/// A clean tree must add no line to a log whose whole budget is 5,000
/// entries.
#[test]
fn a_tree_with_nothing_pruned_reports_no_summary() {
let root = tmp_tree("prune-none");
touch(&root.join("keep.txt"));
touch(&root.join("sub/keep2.txt"));
let mut walk = walk_indexable_files(
&[root.to_string_lossy().into_owned()],
false,
false,
IgnoreSet::compile(&[]).unwrap(),
empty_db("prune-none").to_str().unwrap(),
Config::default(),
Arc::new(Registry::default_set()),
Arc::new(AtomicBool::new(false)),
Arc::new(AtomicBool::new(false)),
4,
);
let files: Vec<WalkedFile> = (&mut walk)
.filter_map(|e| match e {
WalkEvent::File(f) => Some(f),
WalkEvent::Stale(_) => None,
})
.collect();
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
assert_eq!(walk.pruned().total(), 0);
assert!(walk.pruned().summary().is_none());
fs::remove_dir_all(&root).ok();
}
#[test]
fn a_directory_reports_rows_with_no_file_behind_them() {
// The per-directory diff, at the level it is computed: one listing

33
crates/quicksearch-core/src/watcher.rs
View file

@ -160,7 +160,7 @@ impl std::error::Error for WatchError {}
/// Render the directory cap compactly: the default 128_000 reads "128k".
fn fmt_cap(cap: usize) -> String {
if cap >= 1000 && cap % 1000 == 0 {
if cap >= 1000 && cap.is_multiple_of(1000) {
format!("{}k", cap / 1000)
} else {
cap.to_string()
@ -266,12 +266,12 @@ impl WatchRegistry {
}
/// Forget `dir` and every watched directory beneath it, returning how
/// many were dropped.
/// many were dropped. Containment is component-wise, per
/// [`crate::file_handling::UnreadableDirs::covers`].
///
/// `Path::starts_with` compares whole components, so `/a/bc` is not
/// treated as living under `/a/b`. The kernel drops watches for deleted
/// directories on its own; unwatching anyway keeps notify's internal
/// descriptor map from growing across a long session of directory churn.
/// The kernel drops watches for deleted directories on its own; unwatching
/// anyway keeps notify's internal descriptor map from growing across a long
/// session of directory churn.
fn remove_tree(&mut self, dir: &Path) -> usize {
// The scan below is O(watched dirs), and the event loop calls this for
// every Remove — files included. Deleting a directory of 10k files
@ -564,7 +564,7 @@ fn run_loop(rx: mpsc::Receiver<NotifyEvent>, ctx: LoopCtx) {
// Periodic GC of abandoned throttle entries.
tick_counter = tick_counter.wrapping_add(1);
if tick_counter % prune_interval_ticks == 0 {
if tick_counter.is_multiple_of(prune_interval_ticks) {
let max_age = ctx
.config
.throttle_window
@ -809,19 +809,8 @@ mod tests {
}
}
/// Unique temp directory; the repo has no `tempfile` dev-dependency.
fn tmp_dir(tag: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!(
"qs-watch-{}-{}-{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&dir).unwrap();
dir
crate::testutil::scratch_dir(tag)
}
/// A registry with `dirs` seeded directly, so the pure set logic can be
@ -903,8 +892,10 @@ mod tests {
);
}
let (sink, got) = sink_to_vec();
let mut config = WatcherConfig::default();
config.max_dirs_per_tick = 3;
let config = WatcherConfig {
max_dirs_per_tick: 3,
..WatcherConfig::default()
};
flush_ready(&mut map, &sink, &config);
// Each dir contributes one event because each entry has one path.
assert_eq!(got.lock().unwrap().len(), 3);

19
crates/quicksearch-core/tests/cascade.rs
View file

@ -1,7 +1,7 @@
//! Integration tests for the ranked search cascade and the streaming
//! search service, against real temp databases.
use std::path::PathBuf;
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
@ -11,19 +11,8 @@ use quicksearch_core::mime::FileType;
use quicksearch_core::query::split::split_for_cascade;
use quicksearch_core::search::{cascade, SearchHit, SearchOptions, SearchService, SearchUpdate};
fn tmp_db(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-cascade-{}-{}-{}.sqlite",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
}
mod common;
use common::scratch_db as tmp_db;
struct Seeder {
conn: rusqlite::Connection,
@ -31,7 +20,7 @@ struct Seeder {
}
impl Seeder {
fn new(path: &PathBuf, store_text: bool) -> Seeder {
fn new(path: &Path, store_text: bool) -> Seeder {
Seeder {
conn: open_or_recreate(path.to_str().unwrap(), "trigram").unwrap(),
store_text,

92
crates/quicksearch-core/tests/common/mod.rs Normal file
View file

@ -0,0 +1,92 @@
//! Shared scaffolding for the integration-test binaries.
//!
//! Cargo compiles this into each `tests/*.rs` that declares `mod common;`, so
//! every binary gets its own copy and each one uses a different subset.
#![allow(dead_code)]
use std::path::Path;
use std::time::{Duration, Instant};
use quicksearch_core::config::Config;
use quicksearch_core::db;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
#[allow(unused_imports)]
pub use quicksearch_core::testutil::{scratch_dir, scratch_dir_canonical, touch};
/// A scratch database path under a fresh directory. The sidecars SQLite
/// creates alongside it (`-wal`, `-shm`) land in the same directory.
pub fn scratch_db(tag: &str) -> std::path::PathBuf {
scratch_dir(tag).join("index.sqlite")
}
/// How long a single indexing run may take before the test gives up. Generous:
/// CI runs these in a container against a cold page cache.
const INDEX_TIMEOUT: Duration = Duration::from_secs(120);
/// One full indexing run, awaited to completion.
///
/// Completion is read from the `last_full_index` marker rather than the status
/// enum, because `run_indexing` writes that marker only on a successful finish.
/// Polling for `IndexingStatus::Idle` instead would race: a small tree finishes
/// between two polls, leaving `Idle` ambiguous between "not started yet" and
/// "already done".
pub struct IndexOnce<'a> {
pub db: &'a Path,
pub roots: Vec<String>,
pub config: &'a Config,
/// Delete any existing completion marker first, so a second run over the
/// same index is distinguishable from the first. Off for suites that index
/// into a database whose lifecycle they are themselves testing.
pub fresh_marker: bool,
/// Poll the marker through the keyed open. An encrypted index cannot be
/// read by a plain `rusqlite::Connection::open`, so a run against one would
/// otherwise never observe its own completion and time out.
pub encrypted: bool,
}
impl IndexOnce<'_> {
pub fn run(mut self) {
if self.fresh_marker && self.db.exists() {
let conn = rusqlite::Connection::open(self.db).unwrap();
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.unwrap();
}
let service = IndexingService::new();
service
.start_indexing(
std::mem::take(&mut self.roots),
self.db.to_string_lossy().into_owned(),
self.config.clone(),
)
.unwrap();
let deadline = Instant::now() + INDEX_TIMEOUT;
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if self.db.exists() && self.completed() {
done = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(done, "indexing did not finish within {:?}", INDEX_TIMEOUT);
service.stop_indexing().unwrap();
}
/// Whether the completion marker is present. A database mid-creation is
/// simply "not yet", not a failure — the poll comes round again.
fn completed(&self) -> bool {
let conn = if self.encrypted {
db::open_existing(&self.db.to_string_lossy(), false).ok()
} else {
rusqlite::Connection::open(self.db).ok()
};
conn.is_some_and(|c| db::repo::get_last_full_index(&c).is_some())
}
}

60
crates/quicksearch-core/tests/encrypted.rs
View file

@ -7,59 +7,31 @@
//! do. Everything runs inside a single #[test] so the key transitions are
//! strictly ordered.
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use std::path::Path;
use quicksearch_core::config::Config;
use quicksearch_core::db;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
use quicksearch_core::indexing::IndexingService;
use quicksearch_core::security::{derive_key, salt_from_hex};
fn tmp_dir(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-enc-{}-{}-{}",
tag,
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
p
}
mod common;
use common::scratch_dir as tmp_dir;
/// Run one full index over `root` and wait for the completion marker,
/// reading it through the keyed open so the poll works on encrypted DBs.
/// reading it through the keyed open so the poll works on encrypted indexes.
///
/// The marker is deliberately *not* cleared first: this suite indexes into a
/// database whose enable/disable rebuild cycle it is itself testing, and each
/// rebuild already starts from a fresh file.
fn index_once(root: &Path, db_path: &Path, config: &Config) {
let service = IndexingService::new();
service
.start_indexing(
vec![root.to_string_lossy().into_owned()],
db_path.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
let deadline = Instant::now() + Duration::from_secs(120);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db_path.exists() {
if let Ok(conn) = db::open_existing(&db_path.to_string_lossy(), false) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(10));
common::IndexOnce {
db: db_path,
roots: vec![root.to_string_lossy().into_owned()],
config,
fresh_marker: false,
encrypted: true,
}
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().unwrap();
.run()
}
fn header(db_path: &Path) -> [u8; 16] {

130
crates/quicksearch-core/tests/full_index.rs
View file

@ -6,74 +6,27 @@
//! invisible on a first index — `existing_files` is empty, so nothing is
//! stale — and only appears on the second run.
use std::path::{Path, PathBuf};
use std::path::Path;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use std::time::{Duration, Instant, SystemTime};
use quicksearch_core::config::Config;
use quicksearch_core::file_handling::{extract_scope_prepare, ExtractCursor};
use quicksearch_core::indexing::{IndexingService, IndexingStatus, RootPhase};
fn tmp_dir(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-e2e-{}-{}-{}",
tag,
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
p
}
mod common;
use common::{scratch_dir as tmp_dir, touch};
fn touch(p: &Path, body: &[u8]) {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
}
/// Run one full index and wait for it to finish.
///
/// Completion is detected via the `last_full_index` marker, which
/// `run_indexing` writes only on a successful finish. Polling the status
/// enum instead would race: a small tree finishes between two polls, so
/// `Idle` is ambiguous between "not started yet" and "already done".
/// Run one full index over `root` and wait for it to finish.
fn index_once(root: &Path, db: &Path, config: &Config) {
if db.exists() {
let conn = rusqlite::Connection::open(db).unwrap();
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.unwrap();
common::IndexOnce {
db,
roots: vec![root.to_string_lossy().into_owned()],
config,
fresh_marker: true,
encrypted: false,
}
let service = IndexingService::new();
service
.start_indexing(
vec![root.to_string_lossy().into_owned()],
db.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
let deadline = Instant::now() + Duration::from_secs(120);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().unwrap();
.run()
}
/// (path, mtime, content_state) for every indexed row, ordered by path.
@ -91,10 +44,10 @@ fn rows(db: &Path) -> Vec<(String, i64, i64)> {
}
fn test_config() -> Config {
let config = Config::default();
// Keep the run to phase 1 semantics we're asserting on; extraction is
// covered elsewhere.
config
Config::default()
}
#[test]
@ -393,9 +346,12 @@ fn starting_a_run_claims_the_status_before_it_returns() {
)
.unwrap();
// No sleep, no poll: the very next observation must already be Running.
// No sleep, no poll: the very next observation must already show the run.
// `Preparing` is what a claim looks like before the command thread has
// even picked the start up — it is still joining the previous run — and
// it holds the index exactly as `Running` does.
assert!(
matches!(service.get_status(), IndexingStatus::Running { .. }),
matches!(service.get_status(), IndexingStatus::Preparing { .. }),
"status must be claimed synchronously, got {:?}",
service.get_status()
);
@ -781,7 +737,9 @@ fn a_directory_that_becomes_unreadable_deletes_nothing() {
/// Everything about a file's indexed content that a user can observe: its
/// state, the stored snippet body, and its property rows.
fn content_rows(db: &Path) -> Vec<(String, i64, Option<String>, Option<i64>, String)> {
type ContentRow = (String, i64, Option<String>, Option<i64>, String);
fn content_rows(db: &Path) -> Vec<ContentRow> {
let conn = rusqlite::Connection::open(db).unwrap();
let mut stmt = conn
.prepare(
@ -948,6 +906,41 @@ fn undecodable_small_files_are_reported_as_failures_not_silently_skipped() {
std::fs::remove_dir_all(&db_dir).ok();
}
/// A `.doc` that is not a readable OLE2 compound file — a truncated download,
/// or something misnamed — records a failure with a reason.
///
/// This is the end-to-end shape of the legacy-Office support: the walk types
/// the file from its extension, the office extractor claims `application/
/// msword`, and the OLE2 reader either produces text or says why it could not.
/// Until that reader existed, every `.doc` took the third path instead —
/// `DONE` with empty text — which reads as "indexed, contains nothing" and is
/// indistinguishable from a genuinely empty document.
#[test]
fn an_unreadable_legacy_office_file_fails_with_a_reason() {
let root = tmp_dir("legacy-doc");
let db_dir = tmp_dir("legacy-doc-db");
let db = db_dir.join("index.sqlite");
touch(&root.join("broken.doc"), b"D0CF11E0 this is not really a compound file");
index_once(&root, &db, &Config::default());
let conn = rusqlite::Connection::open(&db).unwrap();
let (state, msg): (i64, Option<String>) = conn
.query_row(
"SELECT content_state, failure_msg FROM files WHERE path LIKE '%broken.doc'",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
assert_eq!(state, 2, "an unreadable .doc is FAILED, not DONE-with-no-text");
let msg = msg.unwrap_or_default();
assert!(msg.contains("broken.doc"), "names the file: {msg}");
assert!(msg.contains("compound file"), "says what went wrong: {msg}");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// The text sniff end-to-end: extensionless text files (README, Makefile,
/// go.sum) are content-indexed off their head bytes, while an extensionless
/// binary blob stays NA.
@ -1306,6 +1299,10 @@ fn a_heavy_root_does_not_stall_a_light_one() {
}
}
}
// The run is claimed but has not reached its walk yet; there is
// nothing to sample, and breaking here would end the watch before
// the run it is watching had started.
IndexingStatus::Preparing { .. } => {}
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
_ => break,
}
@ -1410,7 +1407,10 @@ fn the_wal_stays_bounded_during_a_run() {
}
last = len;
match service.get_status() {
IndexingStatus::Running { .. } => {}
// Preparing included: the run is claimed but has not opened the
// database yet, so there is no log to watch and nothing to stop
// watching for either.
IndexingStatus::Running { .. } | IndexingStatus::Preparing { .. } => {}
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
_ => break,
}

67
crates/quicksearch-core/tests/reconcile.rs
View file

@ -8,69 +8,28 @@
//! assertion a regression that quietly reintroduces the rebuild would pass
//! every other check in this file.
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use std::path::Path;
use std::sync::atomic::AtomicBool;
use std::time::Instant;
use quicksearch_core::config::{diff_actions, Config};
use quicksearch_core::db;
use quicksearch_core::extract::Registry;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
use quicksearch_core::scope::{advance, WorkCursor, SLICE};
fn tmp_dir(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-reconcile-{}-{}-{}",
tag,
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
std::fs::canonicalize(&p).unwrap()
}
fn touch(p: &Path, body: &[u8]) {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
}
mod common;
use common::{scratch_dir_canonical as tmp_dir, touch};
/// Run one full index over `config`'s roots and wait for it to finish.
fn index_once(db: &Path, config: &Config) {
if db.exists() {
let conn = rusqlite::Connection::open(db).unwrap();
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.unwrap();
common::IndexOnce {
db,
roots: config.paths.indexing_paths.clone(),
config,
fresh_marker: true,
encrypted: false,
}
let service = IndexingService::new();
service
.start_indexing(
config.paths.indexing_paths.clone(),
db.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
let deadline = Instant::now() + Duration::from_secs(120);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().unwrap();
.run()
}
/// Apply the reconciliation `old -> new` implies, exactly as the coordinator
@ -84,6 +43,7 @@ fn reconcile(db: &Path, old: &Config, new: &Config) -> (usize, usize) {
let mut conn = db::open_existing(&db.to_string_lossy(), true).unwrap();
let registry = Registry::default_set();
let mut cursor = WorkCursor::new(actions.work, new).unwrap();
let run = AtomicBool::new(false);
while !cursor.done() {
advance(
&mut conn,
@ -91,6 +51,7 @@ fn reconcile(db: &Path, old: &Config, new: &Config) -> (usize, usize) {
&registry,
&mut cursor,
Instant::now() + SLICE,
&run,
)
.unwrap();
}

16
crates/quicksearch-core/tests/snippet_perf.rs
View file

@ -16,7 +16,6 @@
//! --test snippet_perf -- --nocapture
//! ```
use std::path::PathBuf;
use std::time::Instant;
use quicksearch_core::snippet;
@ -123,19 +122,8 @@ fn seed_text(rng: &mut u64, target_words: usize) -> String {
out
}
fn tmp_path(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-snippet-perf-{}-{}-{}.sqlite",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
}
mod common;
use common::scratch_db as tmp_path;
#[test]
fn snippet_paths_perf_comparison() {

441
crates/quicksearch-gui/src/app.rs
View file

@ -6,9 +6,11 @@ use std::time::{Duration, Instant};
use quicksearch_core::cli::IndexCounts;
use quicksearch_core::config::{diff_actions, nested_roots, Config, SecurityConfig};
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
use quicksearch_core::coordinator::{IndexMode, IndexerState, ReconcileState, WatcherStatus};
use quicksearch_core::db;
use quicksearch_core::indexing::{overall_progress, ConfigChange, IndexingStatus, RootPhase};
use quicksearch_core::indexing::{
overall_progress, ConfigChange, IndexingStatus, PrepStep, RootPhase,
};
use quicksearch_core::search::SearchOptions;
use quicksearch_core::security::{derive_key, generate_salt, salt_to_hex, IndexKey};
use quicksearch_core::watcher::WatchError;
@ -79,6 +81,18 @@ fn guard_source(
}
}
/// Whether quitting now needs the "settings are still being applied" warning.
///
/// Only a Quit, and only while a reconciliation is actually running: leaving
/// mid-pass leaves entries the user excluded still in the index until an
/// indexing run redoes the work, which in manual mode means until they ask for
/// one. Its own function because it is a rule, not a rendering decision, and
/// because the guard it belongs to has two entrances — the close request, and
/// the unsaved-changes prompt resolving to Quit.
fn quit_needs_reconcile_warning(intent: NavIntent, reconciling: bool) -> bool {
intent == NavIntent::Quit && reconciling
}
pub struct QuickSearchApp {
cfg: Config,
backend: Backend,
@ -103,6 +117,13 @@ pub struct QuickSearchApp {
/// and the next run will replace it; see
/// [`QuickSearchApp::stale_index_prompt_ui`].
stale_index_prompt: bool,
/// Set at startup when the index has not caught up with the settings —
/// a reconciliation cut short by a previous quit, or a config edited
/// while the app was closed. See [`QuickSearchApp::reconcile_owed_ui`].
reconcile_owed: bool,
/// `last_full_index` as it read at startup; the run that moves it past
/// this is the run that clears `reconcile_owed`.
reconcile_owed_since: Option<u64>,
/// Set when the watcher gave up on the directory budget and live
/// updates are off; see [`QuickSearchApp::check_watch_cap_warning`].
watch_cap_prompt: Option<WatchError>,
@ -177,7 +198,20 @@ impl QuickSearchApp {
let stale_index_prompt =
db::index_needs_rebuild(&cfg.resolved_database_path().to_string_lossy());
// Also before the backend, and for the same reason: in automatic mode
// the coordinator's first run can reconcile — and clear the answer —
// before the first frame is drawn. A missing or unreadable index owes
// nothing; it has never been reconciled against anything.
let db_path = cfg.resolved_database_path().to_string_lossy().into_owned();
let reconcile_owed = quicksearch_core::scope::outstanding_work(&db_path, &cfg)
.map(|work| work.touches_index())
.unwrap_or(false);
let backend = Backend::start(&cfg, ctx.clone())?;
// Read from the coordinator rather than the file: it stamps this at
// startup, before its thread can run anything, so it is the same
// number the frames below compare against.
let reconcile_owed_since = backend.coordinator.state().last_full_index;
let fuzzy = cfg.search.fuzzy_default;
// Startup validation: a hand-edited config can nest roots, which
// per-root pipelines can't accept. Redirect straight to the folder
@ -208,6 +242,8 @@ impl QuickSearchApp {
nested_prompt,
key_source,
stale_index_prompt,
reconcile_owed,
reconcile_owed_since,
watch_cap_prompt: None,
security_prompt: None,
pending_nav: None,
@ -261,8 +297,13 @@ impl QuickSearchApp {
.watch_cap_warned_roots
.retain(|root| new.paths.indexing_paths.contains(root));
let actions = diff_actions(&self.cfg, &new);
// A config that could not be written must not take effect either: a
// read-only config directory would otherwise apply the settings to
// this process, revert them on restart, and — because `is_dirty`
// compares against `self.cfg` — show nothing unsaved in between.
if let Err(e) = new.save() {
self.config_error = Some(e);
return false;
}
if (new.ui.scale - self.cfg.ui.scale).abs() > f32::EPSILON {
ctx.set_zoom_factor(clamp_scale(new.ui.scale));
@ -321,13 +362,9 @@ impl QuickSearchApp {
fn drain_events(&mut self) {
// Streamed search results.
loop {
match self.backend.search_rx.try_recv() {
Ok(update) => self
.search
.apply_update(update, self.cfg.search.display_limit),
Err(_) => break,
}
while let Ok(update) = self.backend.search_rx.try_recv() {
self.search
.apply_update(update, self.cfg.search.display_limit);
}
// Status-bar counts worker.
if let Some(rx) = &self.backend.counts_job {
@ -430,6 +467,65 @@ impl QuickSearchApp {
egui::TopBottomPanel::bottom("status-bar").show(ctx, |ui| {
ui.horizontal(|ui| {
match &state.activity {
// A reconcile the coordinator applies between runs: no run
// holds the index, but it is scanning every row of it —
// and for a moment after, so a pass shorter than a frame
// still leaves a trace.
IndexingStatus::Idle if state.reconcile.is_some() => {
match state.reconcile.expect("matched Some") {
ReconcileState::Running(r) => {
ui.label(
egui::RichText::new(match (r.total, r.fraction()) {
(Some(total), Some(frac)) => format!(
"Applying configuration change · {} / {} ({:.0}%)",
group_thousands(r.examined as u64),
group_thousands(total as u64),
frac * 100.0
),
_ => format!(
"Applying configuration change · {} entries",
group_thousands(r.examined as u64)
),
})
.small(),
);
progress_widget(ui, r.fraction());
}
ReconcileState::Finished(r) => {
ui.label(
egui::RichText::new(crate::format::fmt_reconcile_summary(
r.deleted,
r.recontented,
))
.small(),
);
}
}
}
IndexingStatus::Preparing { start_time, step } => {
let (label, frac) = match step {
PrepStep::PreviousRun => {
("Finishing the previous run…".to_string(), None)
}
PrepStep::OpeningIndex => ("Opening the index…".to_string(), None),
PrepStep::Reconciling(r) => (
format!(
"Applying configuration change · {} entries",
group_thousands(r.examined as u64)
),
r.fraction(),
),
};
ui.label(
egui::RichText::new(format!(
"{} · {}",
label,
crate::format::fmt_duration_clock(start_time.elapsed())
))
.small(),
);
progress_widget(ui, frac);
}
IndexingStatus::Idle => {
let mode = match state.mode {
IndexMode::Auto => "Auto",
@ -497,14 +593,7 @@ impl QuickSearchApp {
text.push_str(&format!(" · {}/{} workers", active, total_workers));
}
ui.label(egui::RichText::new(text).small());
match frac {
Some(frac) => {
ui.add(egui::ProgressBar::new(frac as f32).desired_width(120.0));
}
None => {
ui.add(egui::Spinner::new().size(12.0));
}
}
progress_widget(ui, frac);
}
}
@ -525,11 +614,15 @@ impl QuickSearchApp {
});
});
// Keep painting while anything is moving.
// Keep painting while anything is moving. The reconcile clause is not
// redundant: the coordinator's own pass runs with the activity `Idle`,
// so without it the counters would freeze mid-scan until the pointer
// moved — and the summary that follows would never age off screen.
if !matches!(
state.activity,
IndexingStatus::Idle | IndexingStatus::Error(_)
) {
) || state.reconcile.is_some()
{
ctx.request_repaint_after(Duration::from_millis(250));
}
// Watcher registration walks every root, so its verdict can land
@ -631,8 +724,12 @@ impl QuickSearchApp {
return;
}
}
} else {
keychain::delete_key(&db_path.to_string_lossy());
} else if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
// Mirrors the store half: the preference describes what
// is on the keychain, so it must not claim the key is
// gone while it is still there.
self.config_error = Some(e);
return;
}
self.cfg.security.use_keychain = remember;
if let Err(e) = self.cfg.save() {
@ -823,8 +920,13 @@ impl QuickSearchApp {
}
}
// Disabling protection, or "remember" off: no stored key may
// survive pointing at the previous encryption state.
_ => keychain::delete_key(&db_path),
// survive pointing at the previous encryption state. A failure
// here leaves one that does, which is worth saying out loud.
_ => {
if let Err(e) = keychain::delete_key(&db_path) {
self.config_error = Some(e);
}
}
}
db::set_process_key(new_key);
self.backend.coordinator.rebuild_index();
@ -833,6 +935,20 @@ impl QuickSearchApp {
}
}
/// The status bar's trailing progress indicator: a bar when the work has a
/// denominator, a spinner when it does not. One helper so every kind of
/// activity the bar reports ends the same way.
fn progress_widget(ui: &mut egui::Ui, fraction: Option<f64>) {
match fraction {
Some(frac) => {
ui.add(egui::ProgressBar::new(frac as f32).desired_width(120.0));
}
None => {
ui.add(egui::Spinner::new().size(12.0));
}
}
}
/// Drop egui's retained text-field state (buffer + undo history) for the
/// password dialog fields.
fn purge_security_field_state(ctx: &egui::Context) {
@ -899,6 +1015,49 @@ impl QuickSearchApp {
}
}
/// Tell the user their settings have not reached the index, and offer the
/// one thing that fixes it.
///
/// The condition is the index's own record: a reconciliation that finishes
/// stamps it, so work still owed means a pass was abandoned — quitting
/// during one is the ordinary way — or the config was edited while the app
/// was closed. In automatic mode the periodic run clears it without the
/// user doing anything, which is why the banner is a line and a button
/// rather than a modal; in manual mode nothing happens until they ask.
///
/// Held out of the way while a run or a reconcile is in progress: that is
/// the work itself, and it can only be answered by waiting.
fn reconcile_owed_ui(&mut self, ctx: &egui::Context) {
if !self.reconcile_owed {
return;
}
let state = self.backend.coordinator.state();
// A completed run is the proof: it reconciles from the same record
// and stamps it. A run the user stops does not move this, and the
// banner correctly comes back.
if state.last_full_index > self.reconcile_owed_since {
self.reconcile_owed = false;
return;
}
if !matches!(
state.activity,
IndexingStatus::Idle | IndexingStatus::Error(_)
) || state.reconcile.is_some()
{
return;
}
match reconcile_owed_banner(ctx) {
None => {}
Some(ReconcileOwedChoice::StartIndexing) => {
self.backend.coordinator.reindex_now();
// Not cleared here: the run that finishes clears it, and one
// that is stopped half-way leaves the reminder standing.
ctx.request_repaint_after(Duration::from_millis(100));
}
Some(ReconcileOwedChoice::Dismiss) => self.reconcile_owed = false,
}
}
fn watch_cap_prompt_ui(&mut self, ctx: &egui::Context) {
let Some(reason) = &self.watch_cap_prompt else {
return;
@ -1020,6 +1179,23 @@ impl QuickSearchApp {
};
let dirty = (self.manage.is_dirty(), self.options.is_dirty(&self.cfg));
let Some(source) = guard_source(intent, dirty.0, dirty.1) else {
// Second in line, and inside the guard rather than beside it: the
// Discard-then-quit path sets `quit_confirmed` and never returns
// to the close-request check, so a warning that lived only there
// would be skipped by exactly the user who dirtied an editor.
if quit_needs_reconcile_warning(intent, self.backend.coordinator.reconciling()) {
// Repaint on its own: a reconcile that ends while the modal is
// up should take the modal with it.
ctx.request_repaint_after(Duration::from_millis(250));
match reconcile_quit_modal(ctx) {
None => return,
Some(false) => {
self.pending_nav = None;
return;
}
Some(true) => {}
}
}
return self.complete_nav(ctx, intent);
};
match unsaved_changes_modal(ctx, source) {
@ -1101,6 +1277,11 @@ impl QuickSearchApp {
self.backend.coordinator.state().activity
}
/// Let the close request a scripted quit sends through both guards.
pub(crate) fn capture_confirm_quit(&mut self) {
self.quit_confirmed = true;
}
pub(crate) fn capture_search_settled(&self) -> bool {
self.search.capture_settled()
}
@ -1186,6 +1367,85 @@ fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<U
choice
}
/// What the user chose in the "settings not applied yet" banner.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum ReconcileOwedChoice {
StartIndexing,
Dismiss,
}
/// The banner's body, as a top panel under the config-error one.
///
/// A free function for the same reason as [`stale_index_window`]: a test can
/// render it and click its buttons without a coordinator behind it.
fn reconcile_owed_banner(ctx: &egui::Context) -> Option<ReconcileOwedChoice> {
let mut choice = None;
egui::TopBottomPanel::top("reconcile-owed").show(ctx, |ui| {
ui.horizontal(|ui| {
ui.colored_label(
ui.visuals().warn_fg_color,
"⚠ Your indexing settings have not been applied to the index yet.",
);
if ui.small_button("Start indexing now").clicked() {
choice = Some(ReconcileOwedChoice::StartIndexing);
}
if ui.small_button("Dismiss").clicked() {
choice = Some(ReconcileOwedChoice::Dismiss);
}
});
});
choice
}
/// Body of the quit-during-a-reconcile guard; `Some(true)` to quit anyway,
/// `Some(false)` to stay. Esc and a backdrop click count as staying.
///
/// The same blocking `egui::Modal` the unsaved guard uses, and for the same
/// reason: this is a decision, not a notice. Quitting is not refused — the
/// pass is cancellable and the index stays consistent either way — but the
/// consequence is invisible otherwise, since the entries the user excluded go
/// on appearing in search results until an indexing run finishes the job.
fn reconcile_quit_modal(ctx: &egui::Context) -> Option<bool> {
let mut choice = None;
let modal = egui::Modal::new(egui::Id::new("reconcile-quit-guard")).show(ctx, |ui| {
ui.set_max_width(460.0);
ui.heading("Settings are still being applied");
ui.label(
"QuickSearch is still applying your indexing settings to the index. If you \
quit now it stops part-way, and entries you excluded can still turn up in \
search results.",
);
ui.add_space(4.0);
ui.label(
"Nothing is lost: the next indexing run picks the work up again. In manual \
mode, choose \"Start indexing now\" on the Manage Index tab after the next \
launch.",
);
ui.add_space(6.0);
ui.horizontal(|ui| {
if ui
.button(egui::RichText::new("Quit anyway").color(ui.visuals().error_fg_color))
.clicked()
{
choice = Some(true);
}
if ui
.add(crate::ui_util::bordered_button(
"Cancel",
crate::ui_util::BLUE,
))
.clicked()
{
choice = Some(false);
}
});
});
if choice.is_none() && modal.should_close() {
choice = Some(false);
}
choice
}
/// The stale-index window's body. Returns whether the user asked for the
/// rebuild.
///
@ -1274,7 +1534,12 @@ impl eframe::App for QuickSearchApp {
// window is gone there is nothing left to ask — and re-sent from
// `complete_nav` if the user chooses to leave.
if ctx.input(|i| i.viewport().close_requested()) && !self.quit_confirmed {
if self.manage.is_dirty() || self.options.is_dirty(&self.cfg) {
// A reconciliation in flight gets the same treatment as an unsaved
// editor: hold the close and say what leaving now costs.
if self.manage.is_dirty()
|| self.options.is_dirty(&self.cfg)
|| self.backend.coordinator.reconciling()
{
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
// Quitting subsumes any narrower pending navigation.
self.pending_nav = Some(NavIntent::Quit);
@ -1335,6 +1600,7 @@ impl eframe::App for QuickSearchApp {
});
});
}
self.reconcile_owed_ui(ctx);
egui::CentralPanel::default().show(ctx, |ui| match self.tab {
Tab::Search => {
@ -1426,30 +1692,16 @@ mod tests {
use super::*;
fn frame(ctx: &egui::Context, source: KeySource, events: Vec<egui::Event>) -> bool {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 700.0),
)),
events,
..Default::default()
};
let input = crate::test_ui::raw_input(SCREEN, events);
let mut clicked = false;
let _ = ctx.run(input, |ctx| clicked = stale_index_window(ctx, source));
clicked
}
fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
[true, false]
.into_iter()
.map(|pressed| egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed,
modifiers: egui::Modifiers::default(),
})
.collect()
}
use crate::test_ui::click_at;
/// The viewport every modal in this module is centred in.
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
/// The modal the user sees after unlocking onto an index from an older
/// version. It must render under every key source — each produces a
@ -1566,14 +1818,7 @@ mod tests {
source: UnsavedSource,
events: Vec<egui::Event>,
) -> Option<UnsavedChoice> {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 700.0),
)),
events,
..Default::default()
};
let input = crate::test_ui::raw_input(SCREEN, events);
let mut choice = None;
let _ = ctx.run(input, |ctx| choice = unsaved_changes_modal(ctx, source));
choice
@ -1627,4 +1872,98 @@ mod tests {
assert_eq!(esc, Some(UnsavedChoice::Cancel), "Esc must cancel");
}
}
/// Quitting mid-reconcile is the one case that needs saying out loud: the
/// index is left describing settings the user has already changed, and in
/// manual mode nothing fixes that until they ask for a run. Nothing else
/// warrants the prompt — a tab switch does not end the pass, and a quit
/// with no pass running has nothing to warn about.
#[test]
fn only_quitting_during_a_reconcile_warns() {
use NavIntent::*;
assert!(quit_needs_reconcile_warning(Quit, true));
assert!(!quit_needs_reconcile_warning(Quit, false));
assert!(!quit_needs_reconcile_warning(SwitchTab(Tab::Search), true));
assert!(!quit_needs_reconcile_warning(CloseOptions, true));
}
fn reconcile_modal_frame(ctx: &egui::Context, events: Vec<egui::Event>) -> Option<bool> {
let input = crate::test_ui::raw_input(SCREEN, events);
let mut choice = None;
let _ = ctx.run(input, |ctx| choice = reconcile_quit_modal(ctx));
choice
}
/// Both ways out of the quit warning work, and neither is the default: a
/// modal whose "Quit anyway" did nothing would trap the user in an app
/// they asked to close, and one that quit on Esc would make the warning
/// pointless.
#[test]
fn the_quit_warning_reports_both_answers() {
let ctx = egui::Context::default();
assert_eq!(
reconcile_modal_frame(&ctx, Vec::new()),
None,
"an untouched frame must not decide"
);
let mut seen = std::collections::HashSet::new();
for y in (250..450).step_by(3) {
for x in (250..760).step_by(6) {
if let Some(choice) =
reconcile_modal_frame(&ctx, click_at(egui::pos2(x as f32, y as f32)))
{
seen.insert(choice);
}
}
}
assert!(seen.contains(&true), "\"Quit anyway\" never fired");
assert!(seen.contains(&false), "Cancel never fired");
let esc = reconcile_modal_frame(
&ctx,
vec![egui::Event::Key {
key: egui::Key::Escape,
physical_key: None,
pressed: true,
repeat: false,
modifiers: egui::Modifiers::NONE,
}],
);
assert_eq!(esc, Some(false), "Esc must keep the app open");
}
fn banner_frame(ctx: &egui::Context, events: Vec<egui::Event>) -> Option<ReconcileOwedChoice> {
let input = crate::test_ui::raw_input(SCREEN, events);
let mut choice = None;
let _ = ctx.run(input, |ctx| choice = reconcile_owed_banner(ctx));
choice
}
/// The reminder a quit mid-prune leaves behind. Its Start button is the
/// whole point — the banner exists because in manual mode nothing else
/// will finish the work — and Dismiss must not be the only live control.
#[test]
fn the_reconcile_banner_reports_both_buttons() {
let ctx = egui::Context::default();
assert_eq!(banner_frame(&ctx, Vec::new()), None);
let mut seen = std::collections::HashSet::new();
// A top panel, so it sits in the first rows of the window.
for y in (0..60).step_by(2) {
for x in (0..1000).step_by(4) {
if let Some(choice) = banner_frame(&ctx, click_at(egui::pos2(x as f32, y as f32))) {
seen.insert(choice);
}
}
}
assert!(
seen.contains(&ReconcileOwedChoice::StartIndexing),
"\"Start indexing now\" never fired"
);
assert!(
seen.contains(&ReconcileOwedChoice::Dismiss),
"Dismiss never fired"
);
}
}

34
crates/quicksearch-gui/src/capture.rs
View file

@ -193,8 +193,16 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
"clear_query" => Cmd::ClearQuery,
"focus_search" => Cmd::FocusSearch,
"window" => {
let w = parse_int("width", next_word(rest, line_no, "width in points")?, line_no)?;
let h = parse_int("height", next_word(rest, line_no, "height in points")?, line_no)?;
let w = parse_int(
"width",
next_word(rest, line_no, "width in points")?,
line_no,
)?;
let h = parse_int(
"height",
next_word(rest, line_no, "height in points")?,
line_no,
)?;
if w == 0 || h == 0 {
return Err(err("window dimensions must be positive".to_string()));
}
@ -203,9 +211,11 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
h: h as f32,
}
}
"hover_match" => Cmd::HoverMatch(
parse_int("row", next_word(rest, line_no, "row index")?, line_no)? as usize,
),
"hover_match" => {
Cmd::HoverMatch(
parse_int("row", next_word(rest, line_no, "row index")?, line_no)? as usize,
)
}
"hover_off" => Cmd::HoverOff,
"tab" => Cmd::Tab(match next_word(rest, line_no, "tab name")? {
"search" => Tab::Search,
@ -452,7 +462,7 @@ impl CaptureDriver {
}
let Some(cmd) = self.cmds.get(self.pc).cloned() else {
self.quit(ctx);
self.quit(app, ctx);
return;
};
let started = match self.cmd_started {
@ -542,7 +552,7 @@ impl CaptureDriver {
ShotTag,
)));
}
Cmd::Quit => self.quit(ctx),
Cmd::Quit => self.quit(app, ctx),
}
if matches!(cmd, Cmd::RecordStop) {
self.stop_recorder();
@ -572,7 +582,8 @@ impl CaptureDriver {
capped(max_ms)
|| matches!(
app.capture_indexing_status(),
IndexingStatus::Running { .. }
IndexingStatus::Preparing { .. }
| IndexingStatus::Running { .. }
| IndexingStatus::Stopping
| IndexingStatus::Optimizing
)
@ -590,8 +601,13 @@ impl CaptureDriver {
}
}
fn quit(&mut self, ctx: &egui::Context) {
fn quit(&mut self, app: &mut QuickSearchApp, ctx: &egui::Context) {
self.stop_recorder();
// A scripted quit answers the guards up front. Nothing a scenario does
// dirties an editor, but a scenario that changed the indexed folders
// can leave a reconcile running, and its modal would hold the window
// open until the run timed out.
app.capture_confirm_quit();
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
self.finished = true;
}

2
crates/quicksearch-gui/src/duplicates_tab.rs
View file

@ -54,6 +54,8 @@ impl DuplicatesTab {
.weak(),
);
}
// The spinner and its label are already on the header row above,
// so a second progress line here would just repeat them.
DupState::Loading => {}
DupState::Error(e) => {
ui.colored_label(ui.visuals().error_fg_color, e);

90
crates/quicksearch-gui/src/format.rs
View file

@ -29,11 +29,7 @@ pub fn fmt_mtime(unix_secs: i64) -> String {
/// "5 min ago", "3 h ago", else `YYYY-MM-DD HH:MM`. Gives instant
/// feedback that an action (like a fast index run) actually happened.
pub fn fmt_ago(unix_secs: u64) -> String {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let age = now.saturating_sub(unix_secs);
let age = quicksearch_core::log::now_unix().saturating_sub(unix_secs);
if age < 60 {
"just now".to_string()
} else if age < 3600 {
@ -59,7 +55,8 @@ pub fn fmt_interval(minutes: u64) -> String {
if minutes.is_multiple_of(1440) {
let days = minutes / 1440;
return if days == 1 {
// "24 h" reads better than "1 day" for the shipped default.
// A staleness window reads better in hours than in days: "1 day"
// invites rounding to "about a day", "24 h" does not.
"24 h".to_string()
} else {
format!("{} days", days)
@ -76,7 +73,7 @@ pub fn group_thousands(n: u64) -> String {
let digits = n.to_string();
let mut out = String::with_capacity(digits.len() + digits.len() / 3);
for (i, c) in digits.chars().enumerate() {
if i > 0 && (digits.len() - i) % 3 == 0 {
if i > 0 && (digits.len() - i).is_multiple_of(3) {
out.push(',');
}
out.push(c);
@ -108,6 +105,50 @@ pub fn fmt_elapsed(d: std::time::Duration) -> String {
}
}
/// A running clock: `0:07`, `4:32`, `1:04:12`.
///
/// For work that is still going, where the question is "how long has this
/// been like this?" — [`fmt_elapsed`] answers a different one and would
/// render a twenty-minute wait as `1234.5 s`. Seconds are always two digits
/// so the text does not change width every tick.
pub fn fmt_duration_clock(d: std::time::Duration) -> String {
let secs = d.as_secs();
let (h, m, s) = (secs / 3600, (secs % 3600) / 60, secs % 60);
if h > 0 {
format!("{}:{:02}:{:02}", h, m, s)
} else {
format!("{}:{:02}", m, s)
}
}
/// What a finished configuration reconciliation did, in one line.
///
/// The counts are the point: a prune of a small index is over in a
/// millisecond, so this line is the only evidence the user gets that the
/// setting they changed reached the index at all. A clause whose count is
/// zero is left out rather than printed as "0", and a pass that found nothing
/// to change still reports that it ran — that it ran is the answer.
pub fn fmt_reconcile_summary(deleted: usize, recontented: usize) -> String {
let entries = |n: usize| {
format!(
"{} {}",
group_thousands(n as u64),
if n == 1 { "entry" } else { "entries" }
)
};
let mut parts: Vec<String> = Vec::new();
if deleted > 0 {
parts.push(format!("{} removed", entries(deleted)));
}
if recontented > 0 {
parts.push(format!("{} re-examined", entries(recontented)));
}
if parts.is_empty() {
return "Configuration change applied".to_string();
}
format!("Configuration change applied · {}", parts.join(" · "))
}
/// Middle-truncate a path to at most `max_chars` characters.
pub fn middle_truncate(s: &str, max_chars: usize) -> String {
let chars: Vec<char> = s.chars().collect();
@ -141,10 +182,11 @@ mod tests {
assert_eq!(fmt_interval(0), "run");
assert_eq!(fmt_interval(1), "1 min");
assert_eq!(fmt_interval(59), "59 min");
assert_eq!(fmt_interval(60), "1 h");
assert_eq!(fmt_interval(60), "1 h", "the shipped default");
assert_eq!(fmt_interval(90), "1 h 30 min");
assert_eq!(fmt_interval(120), "2 h");
assert_eq!(fmt_interval(1440), "24 h", "the shipped default");
// A whole day is the one multiple-of-1440 case that stays in hours.
assert_eq!(fmt_interval(1440), "24 h");
assert_eq!(fmt_interval(2880), "2 days");
assert_eq!(fmt_interval(10_080), "7 days");
}
@ -179,10 +221,7 @@ mod tests {
#[test]
fn ago_buckets() {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_secs();
let now = quicksearch_core::log::now_unix();
assert_eq!(fmt_ago(now), "just now");
assert_eq!(fmt_ago(now - 59), "just now");
assert_eq!(fmt_ago(now - 120), "2 min ago");
@ -190,6 +229,31 @@ mod tests {
assert!(fmt_ago(now - 200_000).contains('-'), "old = absolute date");
}
#[test]
fn reconcile_summaries_omit_what_did_not_happen() {
assert_eq!(
fmt_reconcile_summary(0, 0),
"Configuration change applied",
"a pass with nothing to do still reports that it ran"
);
assert_eq!(
fmt_reconcile_summary(1, 0),
"Configuration change applied · 1 entry removed"
);
assert_eq!(
fmt_reconcile_summary(1204, 0),
"Configuration change applied · 1,204 entries removed"
);
assert_eq!(
fmt_reconcile_summary(0, 7),
"Configuration change applied · 7 entries re-examined"
);
assert_eq!(
fmt_reconcile_summary(2, 3),
"Configuration change applied · 2 entries removed · 3 entries re-examined"
);
}
#[test]
fn truncation() {
assert_eq!(middle_truncate("short", 20), "short");

16
crates/quicksearch-gui/src/keychain.rs
View file

@ -33,10 +33,16 @@ pub fn load_key(db_path: &str) -> Result<Option<String>, String> {
}
}
/// Forget the remembered key. Best-effort: an entry that never existed or
/// a dead keychain daemon are both fine outcomes for "forget".
pub fn delete_key(db_path: &str) {
if let Ok(entry) = entry(db_path) {
let _ = entry.delete_credential();
/// Forget the remembered key. An entry that never existed is a fine outcome
/// for "forget" and reports success.
///
/// A real failure is not: the derived SQLCipher key is still sitting in the
/// OS keychain, so a caller that goes on to record "not remembered" would be
/// describing a machine state that isn't true. Callers surface this rather
/// than assuming the key is gone.
pub fn delete_key(db_path: &str) -> Result<(), String> {
match entry(db_path)?.delete_credential() {
Ok(()) | Err(keyring::Error::NoEntry) => Ok(()),
Err(e) => Err(format!("keychain delete failed: {}", e)),
}
}

3
crates/quicksearch-gui/src/main.rs
View file

@ -24,6 +24,9 @@ mod options;
mod platform;
mod query_highlight;
mod search_tab;
#[cfg(test)]
mod test_ui;
mod tips;
mod tracker;
mod ui_util;
mod unlock;

460
crates/quicksearch-gui/src/manage_tab.rs
View file

@ -5,11 +5,18 @@ use std::path::Path;
use std::time::{Duration, Instant};
use quicksearch_core::config::Config;
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
use quicksearch_core::indexing::{IndexingStatus, RootPhase, RootProgress};
use quicksearch_core::coordinator::{IndexMode, IndexerState, ReconcileState, WatcherStatus};
use quicksearch_core::indexing::{
IndexingStatus, PrepStep, ReconcileProgress, RootPhase, RootProgress,
};
use crate::format::{fmt_interval, fmt_rate, group_thousands, human_size, middle_truncate};
use crate::format::{
fmt_duration_clock, fmt_interval, fmt_rate, fmt_reconcile_summary, group_thousands, human_size,
middle_truncate,
};
use crate::tips::{self, Tipped};
use crate::tracker::SpeedTracker;
use crate::ui_util::middle_elide;
/// What the tab asks the app to do after this frame.
#[derive(Default)]
@ -65,9 +72,13 @@ impl ManageTab {
let total: usize = roots.iter().map(|r| r.walked + r.extracted).sum();
self.speed.record(total);
}
IndexingStatus::Idle | IndexingStatus::Error(_) | IndexingStatus::Optimizing => {
self.speed.reset()
}
// Preparing included: the prologue has its own counters and no
// files to rate, and a stale files/sec left over from the last
// run would read as progress that is not happening.
IndexingStatus::Idle
| IndexingStatus::Error(_)
| IndexingStatus::Optimizing
| IndexingStatus::Preparing { .. } => self.speed.reset(),
_ => {}
}
}
@ -202,6 +213,7 @@ impl ManageTab {
);
if ui
.add_enabled(!running, egui::Button::new("Start indexing now"))
.tip(&tips::START_NOW)
.clicked()
{
actions.start_now = true;
@ -211,10 +223,7 @@ impl ManageTab {
running || state.mode == IndexMode::Auto,
egui::Button::new("Stop"),
)
.on_hover_text(
"Stop indexing and switch to manual. Saved right away: it \
stays manual on the next launch too.",
)
.tip(&tips::STOP_INDEXING)
.clicked()
{
actions.stop = true;
@ -224,10 +233,7 @@ impl ManageTab {
state.mode != IndexMode::Auto,
egui::Button::new("Return to Automatic"),
)
.on_hover_text(
"Watch for changes and reindex periodically again. Also \
saved, so this is how the app starts from now on.",
)
.tip(&tips::RETURN_TO_AUTO)
.clicked()
{
actions.auto = true;
@ -243,7 +249,7 @@ impl ManageTab {
.button(
egui::RichText::new("Clear index…").color(ui.visuals().error_fg_color),
)
.on_hover_text("Delete the index database (asks for confirmation)")
.tip(&tips::CLEAR_INDEX)
.clicked()
{
actions.clear_index = true;
@ -268,7 +274,7 @@ impl ManageTab {
// never push them out of view; the path truncates into
// whatever width remains (full path on hover).
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if ui.small_button("Remove").clicked() {
if ui.small_button("Remove").tip(&tips::REMOVE_ROOT).clicked() {
remove = Some(i);
}
// Per-root walker override; 0 = auto (4 local / 16
@ -295,11 +301,7 @@ impl ManageTab {
}
}),
)
.on_hover_text(
"Walker threads for this folder. auto = 4 on local \
storage, 16 on network mounts. Takes effect on \
the next indexing run.",
);
.tip(&tips::ROOT_WORKERS);
#[cfg(test)]
tests::record_widget("workers", &response);
if response.changed() {
@ -316,12 +318,9 @@ impl ManageTab {
egui::Layout::left_to_right(egui::Align::Center),
|ui| {
let font_id = egui::TextStyle::Monospace.resolve(ui.style());
let char_width =
ui.fonts(|f| f.glyph_width(&font_id, '0')).max(1.0);
let budget =
((ui.available_width() / char_width) as usize).max(16);
ui.monospace(middle_truncate(root, budget))
.on_hover_text(root);
let shown =
middle_elide(ui, root, ui.available_width(), &font_id);
ui.monospace(shown.as_ref()).on_hover_text(root);
},
);
});
@ -332,7 +331,7 @@ impl ManageTab {
draft.indexing.root_workers.remove(&removed);
}
ui.horizontal(|ui| {
if ui.button("Add folder…").clicked() {
if ui.button("Add folder…").tip(&tips::ADD_ROOT).clicked() {
if let Some(dir) = rfd::FileDialog::new().pick_folder() {
let path = dir.to_string_lossy().into_owned();
try_add_root(draft, path, &mut self.root_error);
@ -342,8 +341,11 @@ impl ManageTab {
egui::TextEdit::singleline(&mut self.new_root)
.desired_width(240.0)
.hint_text("or type a path"),
);
if ui.button("Add").clicked() && !self.new_root.trim().is_empty() {
)
.tip(&tips::ADD_ROOT);
if ui.button("Add").tip(&tips::ADD_ROOT).clicked()
&& !self.new_root.trim().is_empty()
{
let path = self.new_root.trim().to_string();
if try_add_root(draft, path, &mut self.root_error) {
self.new_root.clear();
@ -369,7 +371,9 @@ impl ManageTab {
// --- Filters ---------------------------------------------------
ui.heading(egui::RichText::new("Content filters").strong());
ui.columns(2, |cols| {
cols[0].label("Full-text extensions whitelist (empty = all supported):");
cols[0]
.label("Full-text extensions whitelist (empty = all supported):")
.tip(&tips::EXT_WHITELIST);
cols[0]
.add(
egui::TextEdit::multiline(&mut self.ext_filter_text)
@ -377,15 +381,10 @@ impl ManageTab {
.desired_width(f32::INFINITY)
.hint_text("txt\nmd\npdf # comments allowed\n(none)"),
)
.on_hover_text(
"One extension per line, leading dot optional. A non-empty \
list also excludes files that have no extension at all \
(Makefile, README, .bashrc) — add the line \"(none)\" to \
keep extracting text from those.\n\n\
\"#\" starts a comment, either on its own line or after an \
entry, so a type can be commented out without losing it.",
);
cols[1].label("Ignore patterns (excluded entirely):");
.tip(&tips::EXT_WHITELIST);
cols[1]
.label("Ignore patterns (excluded entirely):")
.tip(&tips::IGNORE_PATTERNS);
let mut remove_pat: Option<usize> = None;
// The list grows and shrinks — including from outside
// this tab — so it is kept off the id of the editor
@ -425,7 +424,11 @@ impl ManageTab {
);
let submitted =
response.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter));
if ui.add_enabled(valid, egui::Button::new("Add")).clicked()
response.tip(&tips::IGNORE_PATTERNS);
if ui
.add_enabled(valid, egui::Button::new("Add"))
.tip(&tips::IGNORE_PATTERNS)
.clicked()
|| (submitted && valid)
{
let pat = self.new_ignore.trim().to_string();
@ -463,14 +466,16 @@ impl ManageTab {
let dirty = self.is_dirty();
ui.horizontal(|ui| {
let apply = ui.add(crate::ui_util::bordered_button(
"Apply & Save",
if dirty {
crate::ui_util::ORANGE
} else {
crate::ui_util::BLUE
},
));
let apply = ui
.add(crate::ui_util::bordered_button(
"Apply & Save",
if dirty {
crate::ui_util::ORANGE
} else {
crate::ui_util::BLUE
},
))
.tip(&tips::APPLY_SAVE);
#[cfg(test)]
tests::record_widget("apply", &apply);
// The label comes and goes with the dirty state; keep it
@ -572,7 +577,7 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
ui.set_max_width(440.0);
ui.strong("To reduce the index size");
for lever in [
"Add ignore filters for files and folders you never search — the ignore \
"Add ignore filters for files and folders you never search, in the ignore \
pattern list further down this tab.",
"Remove indexed folders you do not need, in Indexed folders above.",
"Narrow the full-text extension whitelist, so text is only extracted \
@ -588,7 +593,7 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
ui.label(
egui::RichText::new(
"Narrowing any of these removes the entries it excludes straight away, \
but the file does not shrink on its own: the freed space is reused by \
but the file does not shrink on its own. The freed space is reused by \
the index rather than returned to the disk, until an indexing run's \
optimize pass compacts it.",
)
@ -682,6 +687,21 @@ fn status_panel(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
match &state.activity {
IndexingStatus::Idle => {
// A reconcile the coordinator is applying between runs. No run
// owns the index, so the activity really is Idle — but the thread
// is scanning every row, which on a large index is minutes of
// work that used to be reported as nothing at all.
match &state.reconcile {
Some(ReconcileState::Running(r)) => return reconcile_row(ui, r, None),
// Kept on screen for a few seconds after the work ends: a
// narrowed filter on a small index is applied faster than the
// display could show it happening.
Some(ReconcileState::Finished(r)) => {
ui.label(fmt_reconcile_summary(r.deleted, r.recontented));
return;
}
None => {}
}
// Relative wording makes even a milliseconds-fast run visibly
// register ("just now") instead of looking like a dead button.
let last = state
@ -690,6 +710,9 @@ fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker
.unwrap_or_else(|| "never".to_string());
ui.label(format!("Idle; last full index: {}", last));
}
IndexingStatus::Preparing { start_time, step } => {
prep_row(ui, step, start_time.elapsed());
}
IndexingStatus::Error(e) => {
ui.colored_label(ui.visuals().error_fg_color, format!("Error: {}", e));
}
@ -716,6 +739,99 @@ fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker
}
}
/// What a run is doing before it walks its first file.
///
/// Every one of these can outlast the walk itself on a large index, and they
/// all used to render as one motionless "Starting…". The elapsed clock is the
/// point of the row: it is the only thing that distinguishes slow work from a
/// hang, and it is why each label carries one even when there is nothing to
/// count.
fn prep_row(ui: &mut egui::Ui, step: &PrepStep, elapsed: Duration) {
match step {
PrepStep::PreviousRun => waiting_row(ui, "Finishing the previous run…", elapsed),
PrepStep::OpeningIndex => waiting_row(ui, "Opening the index…", elapsed),
PrepStep::Reconciling(r) => reconcile_row(ui, r, Some(elapsed)),
}
}
/// A prologue step with no counters: label, clock, indeterminate bar.
fn waiting_row(ui: &mut egui::Ui, label: &str, elapsed: Duration) {
ui.horizontal(|ui| {
ui.label(label);
ui.label(
egui::RichText::new(fmt_duration_clock(elapsed))
.small()
.weak(),
);
ui.add(
egui::ProgressBar::new(0.0)
.animate(true)
.desired_width(160.0),
);
});
}
/// A configuration reconciliation, from either place one runs.
///
/// `elapsed` is `Some` for a run's prologue, which has a start time to
/// measure from; the coordinator's between-runs pass has none, and its
/// examined count moves often enough to serve the same purpose.
fn reconcile_row(ui: &mut egui::Ui, r: &ReconcileProgress, elapsed: Option<Duration>) {
ui.horizontal(|ui| {
ui.label("Applying configuration change");
ui.label(egui::RichText::new("|").weak());
match (r.total, r.fraction()) {
(Some(total), Some(frac)) => {
ui.label(format!(
"{} / {} ({:.0}%) entries checked",
group_thousands(r.examined as u64),
group_thousands(total as u64),
frac * 100.0
));
}
_ => {
ui.label(format!(
"{} entries checked",
group_thousands(r.examined as u64)
));
}
}
if let Some(elapsed) = elapsed {
ui.label(
egui::RichText::new(fmt_duration_clock(elapsed))
.small()
.weak(),
);
}
match r.fraction() {
Some(frac) => {
ui.add(egui::ProgressBar::new(frac as f32).desired_width(160.0));
}
// Whole-range deletions read no rows, so they reach the bar with
// no denominator — the same indeterminate form a walk uses before
// its count lands.
None => {
ui.add(
egui::ProgressBar::new(0.0)
.animate(true)
.desired_width(160.0),
);
}
}
});
if r.deleted > 0 || r.recontented > 0 {
ui.label(
egui::RichText::new(format!(
"{} entries removed, {} re-examined for text extraction",
group_thousands(r.deleted as u64),
group_thousands(r.recontented as u64)
))
.small()
.weak(),
);
}
}
/// One root's progress: path, phase, bar, counters, current file.
fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
// Weak "|" separators split the row into folder | status | numbers.
@ -830,6 +946,7 @@ mod tests {
last_full_index: Some(0),
queued_events: 0,
watcher: WatcherStatus::Active { dirs: 10 },
reconcile: None,
}
}
@ -858,28 +975,31 @@ mod tests {
/// contents — not just their widget ids — are under test.
fn state_with(roots: Vec<RootProgress>) -> IndexerState {
IndexerState {
mode: IndexMode::Auto,
activity: IndexingStatus::Running {
start_time: std::time::Instant::now(),
roots,
},
last_full_index: Some(0),
queued_events: 0,
watcher: WatcherStatus::Active { dirs: 10 },
..idle_state()
}
}
fn raw_input(events: Vec<egui::Event>) -> egui::RawInput {
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 900.0),
)),
events,
..Default::default()
/// A run still in its prologue, before the walk has produced anything.
fn preparing_state(step: PrepStep) -> IndexerState {
IndexerState {
activity: IndexingStatus::Preparing {
start_time: std::time::Instant::now(),
step,
},
..idle_state()
}
}
use crate::test_ui::{click_at, painted_text};
fn raw_input(events: Vec<egui::Event>) -> egui::RawInput {
crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events)
}
/// One frame of the real tab, with `events` delivered to it.
fn frame(
ctx: &egui::Context,
@ -917,23 +1037,7 @@ mod tests {
tab.ui(ui, state, cfg);
});
});
let mut text = Vec::new();
for clipped in &out.shapes {
collect_text(&clipped.shape, &mut text);
}
text
}
fn collect_text(shape: &egui::epaint::Shape, out: &mut Vec<String>) {
match shape {
egui::epaint::Shape::Text(t) => out.push(t.galley.text().to_string()),
egui::epaint::Shape::Vec(shapes) => {
for s in shapes {
collect_text(s, out);
}
}
_ => {}
}
painted_text(&out)
}
fn pointer(pos: egui::Pos2, pressed: bool) -> egui::Event {
@ -945,14 +1049,6 @@ mod tests {
}
}
fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
vec![
egui::Event::PointerMoved(pos),
pointer(pos, true),
pointer(pos, false),
]
}
fn cfg_with_root() -> Config {
let mut cfg = Config::default();
cfg.paths.indexing_paths = vec!["/data".into()];
@ -1181,14 +1277,154 @@ mod tests {
assert!(!text.contains(" / "), "invented a denominator: {}", text);
}
/// Every step of the prologue names itself. These used to be one static
/// "Starting…" with no clock, which on a large index is indistinguishable
/// from a hang for as long as the step takes.
#[test]
fn each_prologue_step_says_what_it_is_waiting_on() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
for (step, expected) in [
(PrepStep::PreviousRun, "Finishing the previous run…"),
(PrepStep::OpeningIndex, "Opening the index…"),
] {
let text = frame_text(&ctx, &mut tab, &preparing_state(step)).join(" | ");
assert!(text.contains(expected), "{}", text);
// The clock is the point of the row.
assert!(text.contains("0:00"), "no elapsed time shown: {}", text);
}
}
/// The reconcile is the long one, and the only prologue step with
/// something to count. It reports its position in the scan.
#[test]
fn a_reconcile_reports_how_far_through_the_index_it_is() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
let text = frame_text(
&ctx,
&mut tab,
&preparing_state(PrepStep::Reconciling(ReconcileProgress {
examined: 2_500_000,
total: Some(8_000_000),
deleted: 1_204,
recontented: 0,
})),
)
.join(" | ");
assert!(text.contains("Applying configuration change"), "{}", text);
assert!(
text.contains("2,500,000 / 8,000,000 (31%) entries checked"),
"{}",
text
);
assert!(text.contains("1,204 entries removed"), "{}", text);
}
/// Whole-range deletions read no rows, so the scan can reach the display
/// with nothing to divide by. It must not invent a denominator — the same
/// rule the walk follows before its count lands.
#[test]
fn a_reconcile_without_a_row_count_shows_no_denominator() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
let text = frame_text(
&ctx,
&mut tab,
&preparing_state(PrepStep::Reconciling(ReconcileProgress::default())),
)
.join(" | ");
assert!(text.contains("0 entries checked"), "{}", text);
assert!(!text.contains(" / "), "invented a denominator: {}", text);
}
/// The coordinator applies a prune between runs, so no run owns the index
/// and the activity really is `Idle` — but the thread is scanning every
/// row of it. Reporting "Idle" for the minutes that takes is what made a
/// working app look like a stopped one.
#[test]
fn a_prune_between_runs_is_reported_instead_of_idle() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
let state = IndexerState {
reconcile: Some(ReconcileState::Running(ReconcileProgress {
examined: 40_000,
total: Some(80_000),
deleted: 0,
recontented: 0,
})),
..idle_state()
};
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
assert!(
text.contains("Applying configuration change"),
"the scan is invisible: {}",
text
);
assert!(text.contains("40,000 / 80,000 (50%)"), "{}", text);
assert!(
!text.contains("Idle; last full index"),
"reported idle while scanning: {}",
text
);
}
/// The pass itself can be over between two frames — narrowing a filter on
/// a small index is one transaction. Without a tail the user changes a
/// setting, sees nothing move, and has no way to tell whether it took.
#[test]
fn a_finished_prune_reports_what_it_did() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
let state = IndexerState {
reconcile: Some(ReconcileState::Finished(ReconcileProgress {
examined: 80_000,
total: Some(80_000),
deleted: 1_204,
recontented: 7,
})),
..idle_state()
};
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
assert!(
text.contains("Configuration change applied"),
"the finished pass left no trace: {}",
text
);
assert!(text.contains("1,204 entries removed"), "{}", text);
assert!(text.contains("7 entries re-examined"), "{}", text);
assert!(
!text.contains("Idle; last full index"),
"the summary was replaced by the idle line: {}",
text
);
}
/// The placeholder this whole prologue display replaced. It carried no
/// phase, no counters and no clock, and it was on screen for every one of
/// the steps above.
#[test]
fn the_starting_placeholder_is_gone() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
for state in [
preparing_state(PrepStep::PreviousRun),
preparing_state(PrepStep::OpeningIndex),
preparing_state(PrepStep::Reconciling(ReconcileProgress::default())),
idle_state(),
] {
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
assert!(!text.contains("Starting"), "{}", text);
}
}
/// A scratch directory of its own for each size test, so two of them
/// running in parallel cannot see each other's files.
fn scratch_dir(tag: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("qs-dbsize-{}-{}", std::process::id(), tag));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).expect("scratch dir");
dir
}
use quicksearch_core::testutil::scratch_dir;
fn write_bytes(path: &Path, len: usize) {
std::fs::write(path, vec![b'x'; len]).expect("write");
@ -1600,4 +1836,40 @@ mod tests {
);
assert_eq!(widget("apply").0, clean_id, "the label renamed the button");
}
/// The status area gains a row while a prune runs, another while its
/// summary lingers, and loses both afterwards. Every one of those frames
/// must leave the widgets below it with the ids they had: an editor whose
/// id changes mid-edit loses its buffer, and a prune is exactly when the
/// user is looking at the filter list they just changed.
#[test]
fn the_prune_rows_come_and_go_without_renaming_anything_below() {
let ctx = egui::Context::default();
let mut tab = ManageTab::new();
let cfg = cfg_with_root();
let progress = ReconcileProgress {
examined: 40_000,
total: Some(80_000),
deleted: 12,
recontented: 0,
};
frame_text_with(&ctx, &mut tab, &cfg, &idle_state());
let (apply, _) = widget("apply");
let (workers, _) = widget("workers");
for reconcile in [
Some(ReconcileState::Running(progress)),
Some(ReconcileState::Finished(progress)),
None,
] {
let state = IndexerState {
reconcile,
..idle_state()
};
frame_text_with(&ctx, &mut tab, &cfg, &state);
assert_eq!(widget("apply").0, apply, "renamed by {reconcile:?}");
assert_eq!(widget("workers").0, workers, "renamed by {reconcile:?}");
}
}
}

451
crates/quicksearch-gui/src/options.rs
View file

@ -3,6 +3,7 @@
//! validates, saves, and hands the new config to the app.
use crate::keychain;
use crate::tips::{self, tip_row, Tipped};
use quicksearch_core::config::Config;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@ -144,12 +145,12 @@ impl OptionsWindow {
.show(ui, |ui| {
ui.heading(egui::RichText::new("Paths").strong());
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
ui.label("Database file");
ui.add(
egui::TextEdit::singleline(&mut draft.paths.database_path)
.desired_width(260.0),
);
ui.end_row();
tip_row(ui, "Database file", &tips::DATABASE_PATH, |ui| {
ui.add(
egui::TextEdit::singleline(&mut draft.paths.database_path)
.desired_width(260.0),
)
});
});
ui.label(
egui::RichText::new(
@ -182,18 +183,13 @@ impl OptionsWindow {
ui.heading(egui::RichText::new("Interface").strong());
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
ui.label("UI scale");
ui.add(
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
.step_by(0.05)
.fixed_decimals(2),
)
.on_hover_text(
"Zooms the whole interface: fonts, spacing, and \
widgets. Ctrl +/- and Ctrl 0 adjust it temporarily \
at runtime.",
);
ui.end_row();
tip_row(ui, "UI scale", &tips::UI_SCALE, |ui| {
ui.add(
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
.step_by(0.05)
.fixed_decimals(2),
)
});
});
ui.separator();
@ -208,14 +204,16 @@ impl OptionsWindow {
ui.separator();
ui.horizontal(|ui| {
let apply = ui.add(crate::ui_util::bordered_button(
"Apply & Save",
if dirty {
crate::ui_util::ORANGE
} else {
crate::ui_util::BLUE
},
));
let apply = ui
.add(crate::ui_util::bordered_button(
"Apply & Save",
if dirty {
crate::ui_util::ORANGE
} else {
crate::ui_util::BLUE
},
))
.tip(&tips::APPLY_SAVE);
if apply.clicked() {
out.applied = Some(draft.clone());
}
@ -264,27 +262,36 @@ fn security_ui(
ui.label("The index is encrypted; a password is required at startup.");
}
ui.horizontal(|ui| {
if ui.button("Change password…").clicked() {
if ui
.button("Change password…")
.tip(&tips::CHANGE_PASSWORD)
.clicked()
{
action = Some(SecurityAction::ChangePassword);
}
if ui.button("Disable protection…").clicked() {
if ui
.button("Disable protection…")
.tip(&tips::DISABLE_PASSWORD)
.clicked()
{
action = Some(SecurityAction::Disable);
}
});
let mut remember = current.security.use_keychain;
if ui
.checkbox(&mut remember, "Remember on this device")
.on_hover_text(
"Stores the derived key (not the password) in the OS keychain \
and skips the startup prompt on this machine.",
)
.tip(&tips::REMEMBER_KEYCHAIN)
.changed()
{
action = Some(SecurityAction::SetKeychain(remember));
}
} else {
ui.label("The index is not encrypted.");
if ui.button("Enable password protection…").clicked() {
if ui
.button("Enable password protection…")
.tip(&tips::ENABLE_PASSWORD)
.clicked()
{
action = Some(SecurityAction::Enable);
}
ui.label(
@ -299,8 +306,11 @@ fn security_ui(
action
}
/// One implementation of the per-section config controls, shared by the
/// Options window and the Manage tab.
/// The per-section config controls of the Options window.
///
/// Every row goes through [`crate::tips::tip_row`], which takes the tooltip
/// that explains it: a setting cannot arrive here without one, and hovering
/// the name works as well as hovering the control.
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
match section {
Section::Indexing => {
@ -311,42 +321,44 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
// state, switched (and saved) by the Stop / Return to
// Automatic buttons on the Manage Index tab. A staged copy
// of it here would fight those buttons.
ui.label("Full reindex every");
ui.horizontal(|ui| {
ui.add(
egui::DragValue::new(&mut config.indexing.reindex_interval_minutes)
.range(5..=60 * 24 * 30),
);
ui.label("minutes");
tip_row(ui, "Full reindex every", &tips::REINDEX_INTERVAL, |ui| {
ui.horizontal(|ui| {
ui.add(
egui::DragValue::new(&mut config.indexing.reindex_interval_minutes)
.range(5..=60 * 24 * 30),
);
ui.label("minutes");
})
.response
});
ui.end_row();
ui.label("Follow symlinks");
ui.checkbox(&mut config.indexing.follow_symlinks, "");
ui.end_row();
tip_row(ui, "Follow symlinks", &tips::FOLLOW_SYMLINKS, |ui| {
ui.checkbox(&mut config.indexing.follow_symlinks, "")
});
ui.label("Include hidden files");
ui.checkbox(&mut config.indexing.include_hidden, "");
ui.end_row();
tip_row(ui, "Include hidden files", &tips::INCLUDE_HIDDEN, |ui| {
ui.checkbox(&mut config.indexing.include_hidden, "")
});
});
}
Section::Processing => {
egui::Grid::new("cfg-processing")
.num_columns(2)
.show(ui, |ui| {
ui.label("Tokenizer");
egui::ComboBox::from_id_salt("cfg-tokenize")
.selected_text(&config.processing.tokenize)
.show_ui(ui, |ui| {
for opt in ["trigram", "unicode61", "porter"] {
ui.selectable_value(
&mut config.processing.tokenize,
opt.to_string(),
opt,
);
}
});
ui.end_row();
tip_row(ui, "Tokenizer", &tips::TOKENIZER, |ui| {
egui::ComboBox::from_id_salt("cfg-tokenize")
.selected_text(&config.processing.tokenize)
.show_ui(ui, |ui| {
for opt in ["trigram", "unicode61", "porter"] {
ui.selectable_value(
&mut config.processing.tokenize,
opt.to_string(),
opt,
);
}
})
.response
});
ui.label("");
ui.hyperlink_to(
@ -355,101 +367,85 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
);
ui.end_row();
ui.label("Hash sample size (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.hash_length)
.range(512..=1_048_576),
);
ui.end_row();
tip_row(ui, "Hash sample size (bytes)", &tips::HASH_LENGTH, |ui| {
ui.add(
egui::DragValue::new(&mut config.processing.hash_length)
.range(512..=1_048_576),
)
});
ui.label("Max stored text (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_size)
.range(1024..=16_777_216),
tip_row(
ui,
"Max stored text (bytes)",
&tips::MAX_STORED_TEXT,
|ui| {
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_size)
.range(1024..=16_777_216),
)
},
);
ui.end_row();
ui.label("Max text file size (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_file_size)
.range(1024..=1_073_741_824),
tip_row(
ui,
"Max text file size (bytes)",
&tips::MAX_TEXT_FILE_SIZE,
|ui| {
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_file_size)
.range(1024..=1_073_741_824),
)
},
);
ui.end_row();
ui.label("Batch size");
ui.add(
egui::DragValue::new(&mut config.processing.batch_size).range(10..=100_000),
);
ui.end_row();
tip_row(ui, "Batch size", &tips::BATCH_SIZE, |ui| {
ui.add(
egui::DragValue::new(&mut config.processing.batch_size)
.range(10..=100_000),
)
});
ui.label("Max WAL size (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.maximum_wal_size)
.range(0u64..=8_589_934_592u64),
)
.on_hover_text(
"How large index.sqlite-wal may grow during a run before the \
indexer forces a checkpoint. 0 disables forced checkpoints; \
anything below 16 MiB is raised to it.",
);
ui.end_row();
tip_row(ui, "Max WAL size (bytes)", &tips::MAX_WAL_SIZE, |ui| {
ui.add(
egui::DragValue::new(&mut config.processing.maximum_wal_size)
.range(0u64..=8_589_934_592u64),
)
});
ui.label("Store text for snippets");
ui.checkbox(&mut config.processing.store_text_for_snippets, "")
.on_hover_text(
"Off: smaller index, but no previews, occurrence ranking, \
case verification, or fuzzy full-text search",
);
ui.end_row();
tip_row(ui, "Store text for snippets", &tips::STORE_TEXT, |ui| {
ui.checkbox(&mut config.processing.store_text_for_snippets, "")
});
});
}
Section::Search => {
egui::Grid::new("cfg-search").num_columns(2).show(ui, |ui| {
ui.label("Fuzzy search ON by default");
ui.checkbox(&mut config.search.fuzzy_default, "");
ui.end_row();
ui.label("Fuzzy edit distance");
ui.vertical(|ui| {
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
.on_hover_text(
"Ceiling on the typo budget. The allowance grows with the \
search term, one edit per three characters, up to this \
value, so 2 means \"1 edit for short terms, 2 for longer \
ones\". 0 turns the fuzzy stages off.",
);
if let Some(warning) = config.search.fuzzy_edits_warning() {
ui.label(
egui::RichText::new(warning)
.small()
.color(crate::ui_util::ORANGE),
);
}
});
ui.end_row();
ui.label("Display limit");
ui.add(egui::DragValue::new(&mut config.search.display_limit).range(50..=100_000));
ui.end_row();
ui.label("Stream batch size");
ui.add(
egui::DragValue::new(&mut config.search.results_per_page).range(10..=10_000),
tip_row(
ui,
"Fuzzy search ON by default",
&tips::FUZZY_DEFAULT,
|ui| ui.checkbox(&mut config.search.fuzzy_default, ""),
);
ui.end_row();
ui.label("Debounce (ms)");
ui.add(egui::DragValue::new(&mut config.search.debounce_ms).range(0..=2000));
ui.end_row();
tip_row(ui, "Fuzzy edit distance", &tips::FUZZY_EDITS, |ui| {
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
});
ui.label("Fuzzy max edits");
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
.on_hover_text(
"Ceiling on fuzzy edit distance (the budget grows one \
edit per three characters of the term). 0 disables \
the fuzzy passes.",
);
ui.end_row();
tip_row(ui, "Display limit", &tips::DISPLAY_LIMIT, |ui| {
ui.add(
egui::DragValue::new(&mut config.search.display_limit).range(50..=100_000),
)
});
tip_row(ui, "Stream batch size", &tips::RESULTS_PER_PAGE, |ui| {
ui.add(
egui::DragValue::new(&mut config.search.results_per_page)
.range(10..=10_000),
)
});
tip_row(ui, "Debounce (ms)", &tips::DEBOUNCE, |ui| {
ui.add(egui::DragValue::new(&mut config.search.debounce_ms).range(0..=2000))
});
});
// A warning that comes and goes as the value is edited would
// otherwise move every widget below it in the window; see
@ -525,29 +521,151 @@ mod tests {
assert!(w.draft.is_none());
}
/// Where `needle` was painted this frame, as the center of its galley —
/// a click target that follows the layout instead of pinning it.
fn painted_text_center(out: &egui::FullOutput, needle: &str) -> Option<egui::Pos2> {
fn walk(shape: &egui::epaint::Shape, needle: &str, found: &mut Option<egui::Pos2>) {
match shape {
egui::epaint::Shape::Text(t) => {
if t.galley.text() == needle {
*found = Some(t.pos + t.galley.size() / 2.0);
}
use crate::test_ui::{painted_text, painted_text_center};
/// Every row of every section, with the tip it must show. `tip_row`
/// makes a row without *a* tooltip impossible; this table is what makes
/// a row with the *wrong* one impossible.
const ROWS: &[(Section, &str, &tips::Tip)] = &[
(
Section::Indexing,
"Full reindex every",
&tips::REINDEX_INTERVAL,
),
(Section::Indexing, "Follow symlinks", &tips::FOLLOW_SYMLINKS),
(
Section::Indexing,
"Include hidden files",
&tips::INCLUDE_HIDDEN,
),
(Section::Processing, "Tokenizer", &tips::TOKENIZER),
(
Section::Processing,
"Hash sample size (bytes)",
&tips::HASH_LENGTH,
),
(
Section::Processing,
"Max stored text (bytes)",
&tips::MAX_STORED_TEXT,
),
(
Section::Processing,
"Max text file size (bytes)",
&tips::MAX_TEXT_FILE_SIZE,
),
(Section::Processing, "Batch size", &tips::BATCH_SIZE),
(
Section::Processing,
"Max WAL size (bytes)",
&tips::MAX_WAL_SIZE,
),
(
Section::Processing,
"Store text for snippets",
&tips::STORE_TEXT,
),
(
Section::Search,
"Fuzzy search ON by default",
&tips::FUZZY_DEFAULT,
),
(Section::Search, "Fuzzy edit distance", &tips::FUZZY_EDITS),
(Section::Search, "Display limit", &tips::DISPLAY_LIMIT),
(
Section::Search,
"Stream batch size",
&tips::RESULTS_PER_PAGE,
),
(Section::Search, "Debounce (ms)", &tips::DEBOUNCE),
];
/// Hovering a row's name paints that row's own explanation. Rendered
/// without the window's scroll area so nothing sits below the fold.
#[test]
fn every_row_shows_its_own_tip() {
for (section, label, tip) in ROWS {
let ctx = egui::Context::default();
ctx.style_mut(|s| {
s.interaction.tooltip_delay = 0.0;
s.interaction.show_tooltips_only_when_still = false;
});
let mut cfg = Config::default();
let mut run = |events: Vec<egui::Event>| {
let input = crate::test_ui::raw_input(egui::vec2(600.0, 800.0), events);
ctx.run(input, |ctx| {
egui::CentralPanel::default()
.show(ctx, |ui| config_editor_ui(ui, &mut cfg, *section));
})
};
let first = run(vec![]);
let pos = painted_text_center(&first, label)
.unwrap_or_else(|| panic!("{label} was not painted"));
// Enough of the body to be unique, and short enough to survive
// an edit to the sentence it starts.
let opening: String = tip.body.chars().take(40).collect();
let mut out = run(vec![egui::Event::PointerMoved(pos)]);
let mut found = false;
for _ in 0..3 {
// The tooltip is an area of its own, so it can land a frame
// late.
if painted_text(&out).join("\n").contains(&opening) {
found = true;
break;
}
egui::epaint::Shape::Vec(v) => {
for s in v {
walk(s, needle, found);
}
}
_ => {}
out = run(vec![]);
}
assert!(found, "hovering {label:?} did not show {:?}", tip.title);
}
}
/// Hovering a setting's *name* explains it, not just its control: the
/// label is the larger target and the one a reader's eye is already on.
/// Checks the wiring, so the tooltip timing is turned off.
#[test]
fn hovering_a_setting_label_explains_it() {
let ctx = egui::Context::default();
ctx.style_mut(|s| {
s.interaction.tooltip_delay = 0.0;
s.interaction.show_tooltips_only_when_still = false;
});
let cfg = Config::default();
let mut w = OptionsWindow::new();
w.open_with(&cfg);
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
ctx.run(input, |ctx| {
w.ui(ctx, &cfg);
})
};
// The window spends its first frames sizing itself and painting
// nothing; run until the label is on screen.
let mut target = None;
for _ in 0..5 {
let full = run(&mut w, vec![]);
target = painted_text_center(&full, "Tokenizer");
if target.is_some() {
break;
}
}
let mut found = None;
for clipped in &out.shapes {
walk(&clipped.shape, needle, &mut found);
let target = target.expect("the Tokenizer label was not painted");
// The tooltip is an area of its own, so it can land a frame late.
let mut out = run(&mut w, vec![egui::Event::PointerMoved(target)]);
for _ in 0..3 {
let painted = painted_text(&out).join("\n");
if painted.contains(crate::tips::TOKENIZER.title)
&& painted.contains("cut up so that it can be")
{
return;
}
out = run(&mut w, vec![]);
}
found
panic!("no tooltip appeared over the Tokenizer label");
}
/// One real frame of the window in a headless context: it renders, and
@ -561,14 +679,7 @@ mod tests {
w.draft.as_mut().unwrap().search.debounce_ms += 100;
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 900.0),
)),
events,
..Default::default()
};
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
let mut out = OptionsOutput::default();
let full = ctx.run(input, |ctx| out = w.ui(ctx, &cfg));
(out, full)

126
crates/quicksearch-gui/src/search_tab.rs
View file

@ -10,6 +10,7 @@ use quicksearch_core::snippet::Snippet;
use crate::format::{fmt_elapsed, fmt_mtime, human_size};
use crate::platform;
use crate::ui_util::middle_elide;
/// Width of the query strip's status slot, in points. Wide enough for the
/// longest query time `fmt_elapsed` produces, and held whether the slot is
@ -496,7 +497,8 @@ impl SearchTab {
let preview_height = if preview_snippet.is_some() { 44.0 } else { 0.0 };
let table_height = (ui.available_height() - preview_height).max(60.0);
let text_height = egui::TextStyle::Body.resolve(ui.style()).size + 4.0;
let body_font = egui::TextStyle::Body.resolve(ui.style());
let text_height = body_font.size + 4.0;
let mut open_ignore_dialog: Option<usize> = None;
let mut hovered_now: Option<usize> = None;
// Moved out of `self` rather than cloned. The row closure needs `&mut
@ -564,8 +566,37 @@ impl SearchTab {
cell_responses.push(ui.label(&hit.name));
});
row.col(|ui| {
cell_responses
.push(ui.label(egui::RichText::new(&hit.path).weak()));
// Center-elided: egui's own truncation keeps
// the head and drops the deepest directories
// — the half that actually says where the
// file lives. A sizing pass hands out cell
// rects that are not final yet, so nothing is
// measured against one.
let shown = if ui.is_sizing_pass() {
std::borrow::Cow::Borrowed(hit.path.as_str())
} else {
middle_elide(
ui,
&hit.path,
// A point of slack, so a rounding
// disagreement with egui's layout
// cannot cost a second ellipsis.
ui.available_width() - 1.0,
&body_font,
)
};
let elided = matches!(shown, std::borrow::Cow::Owned(_));
// egui offers a full-text tooltip only when
// *it* elided the galley, and it would be
// handed the string already shortened here.
let mut response = ui.add(
egui::Label::new(egui::RichText::new(shown.as_ref()).weak())
.show_tooltip_when_elided(false),
);
if elided {
response = response.on_hover_text(&hit.path);
}
cell_responses.push(response);
});
if self.has_snippets {
// Borrowed, not cloned: a snippet window runs
@ -643,7 +674,7 @@ impl SearchTab {
let mut response = row.response();
for r in cell_responses {
response = response | r;
response |= r;
}
if response.contains_pointer() {
hovered_now = Some(display_ix);
@ -1098,10 +1129,7 @@ fn rank_tier_color(stage: u8) -> egui::Color32 {
/// Timestamp color: fresh files get a green tint that fades into the weak
/// text color over ~2 years on a log scale.
fn recency_color(ui: &egui::Ui, mtime: i64) -> egui::Color32 {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0);
let now = quicksearch_core::log::now_unix() as i64;
let age_hours = ((now - mtime).max(0) as f32 / 3600.0).max(1.0);
const HORIZON_HOURS: f32 = 24.0 * 365.0 * 2.0;
let t = (age_hours.ln() / HORIZON_HOURS.ln()).clamp(0.0, 1.0);
@ -1144,14 +1172,7 @@ mod tests {
tab: &mut SearchTab,
events: Vec<egui::Event>,
) -> egui::FullOutput {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 700.0),
)),
events,
..Default::default()
};
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 700.0), events);
ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
tab.ui(ui);
@ -1175,6 +1196,79 @@ mod tests {
panic!("never landed on row {row}'s label text");
}
use crate::test_ui::painted_text;
/// Far too long for the Path column at the test's 1000pt screen width.
fn deep_path() -> String {
concat!(
"/media/shared/QuickSearch/crates/quicksearch-gui/src/",
"deeply/nested/under/several/more/directories/alpha_widget_0.txt"
)
.to_string()
}
/// The Path column drops out of the *middle*, so the volume the file
/// sits on and the directories right above it both stay on screen.
/// egui's own truncation would keep the head and throw the tail away —
/// and the tail is the half that says where the file lives.
#[test]
fn long_paths_elide_from_the_middle_of_the_path_column() {
let ctx = egui::Context::default();
let mut tab = tab_with_results(1);
let path = deep_path();
tab.results[0].path = path.clone();
let painted = painted_text(&run_frame(&ctx, &mut tab, vec![]));
let cell = painted
.iter()
.find(|t| t.starts_with("/media") && t.contains('…'))
.unwrap_or_else(|| panic!("no elided path cell among {painted:?}"));
assert!(!painted.contains(&path), "painted in full");
assert_eq!(cell.matches('…').count(), 1, "elided twice: {cell}");
let (head, tail) = cell.split_once('…').expect("one ellipsis");
assert!(path.starts_with(head), "{cell}");
assert!(path.ends_with(tail), "{cell}");
assert!(
tail.ends_with("alpha_widget_0.txt"),
"the deep end survives: {cell}"
);
}
/// The whole path stays reachable on hover. egui hands out that tooltip
/// for free only while *it* did the eliding, so text shortened ahead of
/// time has to bring its own — and it is easy to lose silently.
#[test]
fn an_elided_path_still_shows_the_whole_thing_on_hover() {
let ctx = egui::Context::default();
// Testing that the tooltip is wired up, not egui's hover timing.
ctx.style_mut(|s| {
s.interaction.tooltip_delay = 0.0;
s.interaction.show_tooltips_only_when_still = false;
});
let mut tab = tab_with_results(1);
let path = deep_path();
tab.results[0].path = path.clone();
run_frame(&ctx, &mut tab, vec![]); // settle the table's layout
for y in 40..250 {
// x lands in the Path column, past the 220pt Name column.
let pos = egui::pos2(300.0, y as f32);
let mut out = run_frame(&ctx, &mut tab, vec![egui::Event::PointerMoved(pos)]);
if tab.hovered_row != Some(0) {
continue;
}
// The tooltip is its own area, so it may land a frame behind.
for _ in 0..3 {
if painted_text(&out).contains(&path) {
return;
}
out = run_frame(&ctx, &mut tab, vec![]);
}
}
panic!("the full path never appeared on hover");
}
fn click(pos: egui::Pos2, button: egui::PointerButton) -> Vec<egui::Event> {
[true, false]
.into_iter()

85
crates/quicksearch-gui/src/test_ui.rs Normal file
View file

@ -0,0 +1,85 @@
//! Driving egui headlessly from tests: build an input frame, synthesize a
//! click, read back what was painted.
//!
//! Every tab's test module wants the same three things, and each had grown its
//! own copy. The per-tab `frame(…)` wrappers stay where they are — each drives
//! a different widget with a different return type — but they are built on
//! these.
/// A frame of input at `size`, carrying `events`.
///
/// The viewport size is explicit rather than defaulted because it is load
/// bearing: a modal is centred in it, so tests that locate a button by where
/// it was painted get different coordinates from a different size, and a panel
/// that does not fit is simply not painted at all.
pub fn raw_input(size: egui::Vec2, events: Vec<egui::Event>) -> egui::RawInput {
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(egui::Pos2::ZERO, size)),
events,
..Default::default()
}
}
/// A primary-button press and release at `pos`, preceded by the pointer moving
/// there.
///
/// The move is not decoration: egui hit-tests against the pointer's *current*
/// position, so a press delivered without one lands wherever the pointer was
/// last frame — which for the first frame of a fresh context is nowhere.
pub fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
let button = |pressed| egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed,
modifiers: egui::Modifiers::NONE,
};
vec![
egui::Event::PointerMoved(pos),
button(true),
button(false),
]
}
/// Every text galley painted this frame, each with the rectangle it occupies.
///
/// Labels carry no widget id worth recording, so reading the shapes back is
/// the only way to check the text a user actually sees — and the only way to
/// find a click target that follows the layout instead of pinning it.
pub fn painted(out: &egui::FullOutput) -> Vec<(String, egui::Rect)> {
fn walk(shape: &egui::epaint::Shape, into: &mut Vec<(String, egui::Rect)>) {
match shape {
egui::epaint::Shape::Text(t) => into.push((
t.galley.text().to_string(),
egui::Rect::from_min_size(t.pos, t.galley.size()),
)),
egui::epaint::Shape::Vec(shapes) => {
for s in shapes {
walk(s, into);
}
}
_ => {}
}
}
let mut out_text = Vec::new();
for clipped in &out.shapes {
walk(&clipped.shape, &mut out_text);
}
out_text
}
/// Every string painted this frame, in paint order.
pub fn painted_text(out: &egui::FullOutput) -> Vec<String> {
painted(out).into_iter().map(|(text, _)| text).collect()
}
/// The centre of `needle`'s galley, as a click target.
///
/// The *last* match wins, so a string painted both behind a modal and on it
/// resolves to the one on top — which is the one a click would reach.
pub fn painted_text_center(out: &egui::FullOutput, needle: &str) -> Option<egui::Pos2> {
painted(out)
.iter()
.rev()
.find(|(text, _)| text == needle)
.map(|(_, rect)| rect.center())
}

758
crates/quicksearch-gui/src/tips.rs Normal file
View file

@ -0,0 +1,758 @@
//! Plain-language tooltips for the configuration controls.
//!
//! Every setting in the Options window, and every configuration control on
//! the Manage Index tab, explains itself on hover: what it does in ordinary
//! words first, then what changing it costs, then concrete values and when
//! someone would pick them. The text lives here rather than inline so the
//! layout code stays readable and the wording can be checked on its own.
//!
//! Written for someone who does not know what a tokenizer or a write-ahead
//! log is, and should not have to.
use crate::ui_util::ORANGE;
/// How wide a tooltip may get. Matches `manage_tab::db_size_tooltip`: wide
/// enough that a sentence is not shredded into three lines, narrow enough
/// that the eye finds the next line.
const TIP_WIDTH: f32 = 420.0;
/// One control's tooltip.
pub struct Tip {
/// The setting's name, in bold at the top.
pub title: &'static str,
/// The explanation. One or more paragraphs separated by `"\n\n"`.
pub body: &'static str,
/// Concrete values and when to choose them, rendered small underneath.
/// Empty where a setting has nothing to weigh up.
pub examples: &'static [&'static str],
/// A consequence worth seeing before the click, rendered small and
/// orange. Reserved for rebuilds and deletions, so that it keeps
/// meaning something.
pub caution: Option<&'static str>,
}
impl Tip {
/// Render into a hover popup.
pub fn show(&self, ui: &mut egui::Ui) {
ui.set_max_width(TIP_WIDTH);
ui.strong(self.title);
ui.label(self.body);
if !self.examples.is_empty() {
ui.add_space(4.0);
// One example reads as a sentence; several read as a list.
let single = self.examples.len() == 1;
for example in self.examples {
let line = if single {
format!("Example: {}", example)
} else {
format!("• {}", example)
};
ui.label(egui::RichText::new(line).small());
}
}
if let Some(caution) = self.caution {
ui.add_space(4.0);
ui.label(egui::RichText::new(caution).small().color(ORANGE));
}
}
}
/// Attach a [`Tip`] to any widget.
pub trait Tipped {
/// Show `tip` on hover, whether or not the widget is enabled: a greyed
/// out button is exactly when someone wants to know what it would do.
fn tip(self, tip: &'static Tip) -> Self;
}
impl Tipped for egui::Response {
fn tip(self, tip: &'static Tip) -> egui::Response {
self.on_hover_ui(|ui| tip.show(ui))
.on_disabled_hover_ui(|ui| tip.show(ui))
}
}
/// One row of a two-column settings grid: the label and the control share a
/// tooltip, so hovering the name works as well as hovering the widget.
///
/// Taking the tip by value rather than as an option is deliberate: a row
/// added later cannot forget to explain itself.
pub fn tip_row(
ui: &mut egui::Ui,
label: &str,
tip: &'static Tip,
widget: impl FnOnce(&mut egui::Ui) -> egui::Response,
) {
ui.label(label).tip(tip);
widget(ui).tip(tip);
ui.end_row();
}
// --- Options: Paths ------------------------------------------------------
pub static DATABASE_PATH: Tip = Tip {
title: "Database file",
body: "Where QuickSearch keeps its index: a single file holding the \
names, locations and text of everything it has indexed. It grows \
with the number of files indexed, so it wants a drive with room \
to spare.\n\n\
Pointing this somewhere else switches to the index at the new \
location, building a fresh one there if nothing exists yet. The \
old file stays on disk until you delete it. Keep it off a folder \
that syncs to the cloud, such as OneDrive or Dropbox: it is far \
too large and too busy to synchronise.",
examples: &["a path on a second, larger drive when the system drive is tight on space."],
caution: None,
};
// --- Options: Indexing ---------------------------------------------------
pub static REINDEX_INTERVAL: Tip = Tip {
title: "Full reindex every",
body: "Indexed folders are watched, so ordinary changes appear within \
seconds. This is the safety net behind that: a full sweep that \
picks up anything the watching missed, such as changes made while \
QuickSearch was closed, or on a drive that does not report them.\n\n\
A sweep costs some disk activity for a few minutes, and searching \
keeps working throughout.",
examples: &[
"1440, once a day, suits an ordinary laptop.",
"60 when you work on a network share, or a drive other machines write to.",
"10080, once a week, for an archive that barely changes.",
],
caution: None,
};
pub static FOLLOW_SYMLINKS: Tip = Tip {
title: "Follow symlinks",
body: "Symlinks are entries that stand in for a file or folder living \
somewhere else. Off, QuickSearch steps over them, so nothing is \
indexed twice under two names. On, it looks through them and \
indexes what they point at, stored under the real location rather \
than the link's.\n\n\
Turning this off removes the entries that are no longer in scope; \
turning it on indexes whatever it now reaches. Neither rebuilds \
the index.",
examples: &[
"on when a folder you search lives outside your indexed folders and is reached \
through a link.",
],
caution: None,
};
pub static INCLUDE_HIDDEN: Tip = Tip {
title: "Include hidden files",
body: "Files and folders the system keeps out of sight: names beginning \
with a dot on Linux and macOS, plus anything carrying the Hidden \
attribute on Windows, such as AppData and $RECYCLE.BIN. They are \
mostly program settings and caches, so leaving them out keeps the \
index smaller and the results cleaner.\n\n\
Folders marked System but not Hidden are indexed either way. Cloud \
sync folders and folders given a custom icon carry that mark only \
to get the icon, and are ordinary folders otherwise.",
examples: &["on when you want to find configuration files such as .bashrc or .gitconfig."],
caution: None,
};
// --- Options: Processing -------------------------------------------------
pub static TOKENIZER: Tip = Tip {
title: "Tokenizer",
body: "How the text inside your files is cut up so that it can be \
searched.\n\n\
trigram indexes every run of three characters, so a search for \
\"port\" also finds \"airport\", and it works for languages that \
do not put spaces between words. The other two index whole words \
instead: the index is smaller and faster, but a search matches \
only from the start of a word.",
examples: &[
"trigram, the default, for finding a fragment anywhere inside a word.",
"unicode61 when you only ever search whole words and want the smallest, \
fastest index.",
"porter to also match English word endings, so \"running\" finds \"run\".",
],
caution: Some("Changing this builds the whole index again from scratch."),
};
pub static HASH_LENGTH: Tip = Tip {
title: "Hash sample size",
body: "How much of the start of each file QuickSearch reads in order to \
recognise it. Those first bytes give the file its fingerprint, \
which is how the Duplicates tab knows two files are identical; \
they also say what kind of file it is, and for a small text file \
they are the whole text.\n\n\
Reading more is more reliable and slower, particularly over a \
network.",
examples: &[
"8192, the default, is right for almost everyone.",
"higher when the Duplicates tab groups files that are not really identical, \
which happens with disk images and other formats that begin with a lot of \
empty space.",
],
caution: Some("Changing this builds the whole index again from scratch."),
};
pub static MAX_STORED_TEXT: Tip = Tip {
title: "Max stored text",
body: "How much text QuickSearch keeps out of any one file. Text past \
this point is not stored, so a search will not find a word that \
appears only deep inside a very long document. File names, sizes \
and dates are unaffected.\n\n\
Along with the two settings below it, this is one of the largest \
influences on how big the index becomes.",
examples: &[
"262144, 256 KB, covers the whole of most documents.",
"65536, 64 KB, to shrink the index when you mostly search the opening pages.",
"higher when you search long books, transcripts or logs and expect to find \
words near the end.",
],
caution: None,
};
pub static MAX_TEXT_FILE_SIZE: Tip = Tip {
title: "Max text file size",
body: "Files larger than this are indexed by name only: QuickSearch does \
not open them to read the text inside. It keeps one stray huge \
file from holding up an indexing run.\n\n\
Those files still appear in results, found by their name, size or \
date.",
examples: &[
"2097152, 2 MB, skips very few ordinary documents.",
"52428800, 50 MB, when you search inside large log files or scanned PDFs.",
],
caution: None,
};
pub static BATCH_SIZE: Tip = Tip {
title: "Batch size",
body: "How many files QuickSearch handles per write to the index while \
indexing. Larger batches mean fewer, bigger writes, which is a \
little faster and uses more memory.\n\n\
A speed setting only: it changes nothing about what you can find, \
and most people never need to touch it.",
examples: &[
"500, the default, balances speed against memory.",
"lower, around 50, on a machine with very little memory to spare.",
],
caution: None,
};
pub static MAX_WAL_SIZE: Tip = Tip {
title: "Max WAL size",
body: "While indexing, changes are written to a companion file beside \
the index and folded in afterwards. That normally happens by \
itself, but during a long run with searches going on at the same \
time the companion file keeps growing, sometimes past the size of \
the index. This is the point at which QuickSearch pauses and folds \
it in regardless.\n\n\
Another speed setting; the default suits most machines.",
examples: &[
"536870912, 512 MB, is the default.",
"67108864, 64 MB, when disk space is tight.",
"0 to never force it and let the database decide. Any other value below 16 MB \
is treated as 16 MB.",
],
caution: None,
};
pub static STORE_TEXT: Tip = Tip {
title: "Store text for snippets",
body: "Keeps the text QuickSearch reads out of your files, rather than \
only the index needed to search it. It is what makes the preview \
line underneath a result possible.\n\n\
Off, searching inside files still works, but there are no \
previews, no ranking by how often a word appears, no telling \
Report from report, and no allowance for typos inside file \
contents. The index shrinks considerably.\n\n\
Turning it off discards the stored text at once; turning it back \
on reads your files again.",
examples: &[
"off when the index has grown larger than you want and you can do without previews.",
],
caution: None,
};
// --- Options: Search -----------------------------------------------------
pub static FUZZY_DEFAULT: Tip = Tip {
title: "Fuzzy search ON by default",
body: "Whether the Fuzzy box on the Search tab starts ticked each time \
QuickSearch opens. Fuzzy search also finds matches with typos in \
them, at some cost in speed. Either way you can tick and untick \
it whenever you like.",
examples: &["on when you often look for names you are not sure how to spell."],
caution: None,
};
pub static FUZZY_EDITS: Tip = Tip {
title: "Fuzzy edit distance",
body: "How far a word may sit from what you typed and still count as a \
match while Fuzzy is on. One edit is one letter added, removed or \
changed, so \"reciept\" is one edit away from \"receipt\".\n\n\
The allowance grows with the length of what you type, one edit per \
three characters, up to the value set here. Short searches stay \
strict, so that three letters do not match half the index.",
examples: &[
"2, the default, allows one edit for short words and two for longer ones.",
"0 turns typo matching off altogether, even with the Fuzzy box ticked.",
"3 or more is allowed, but searches get slower and pull in a lot of \
unrelated files.",
],
caution: None,
};
pub static DISPLAY_LIMIT: Tip = Tip {
title: "Display limit",
body: "The most results one search will gather and show. A search \
matching thousands of files stops here, which keeps the list quick \
to scroll and cheap to hold in memory.\n\n\
The best matches come first, so a lower limit rarely hides what \
you were looking for.",
examples: &[
"1000, the default, is more than anyone scrolls through.",
"higher when you use QuickSearch to list every file of a kind, such as \
type:Image, and want them all at once.",
],
caution: None,
};
pub static RESULTS_PER_PAGE: Tip = Tip {
title: "Stream batch size",
body: "Results arrive in batches while a search runs, and this is how \
many are in each one. Smaller batches put the first results on \
screen sooner and update the list more often; larger ones do less \
work in total.\n\n\
This is not a page size: scrolling the results does not go back \
for more.",
examples: &[
"100, the default, feels immediate on most machines.",
"25 when the first results are slow to appear on a large index.",
],
caution: None,
};
pub static DEBOUNCE: Tip = Tip {
title: "Debounce",
body: "How long QuickSearch waits after your last keystroke before it \
searches, so that typing a word does not fire off a search for \
every letter in it. 1000 milliseconds is one second.",
examples: &[
"150, the default, keeps up with ordinary typing.",
"0 to chase every keystroke on a fast machine with a modest index.",
"300 or more when the results flicker or stutter as you type.",
],
caution: None,
};
// --- Options: Interface --------------------------------------------------
pub static UI_SCALE: Tip = Tip {
title: "UI scale",
body: "Zooms the whole window: text, spacing and controls together. 1.00 \
is the ordinary size for your screen.\n\n\
Ctrl with + or - changes it for the moment without saving, and \
Ctrl 0 puts it back. This slider is the size QuickSearch starts \
at.",
examples: &[
"1.40 or more on a high resolution screen where the text looks small.",
"0.80 to fit more results on screen at once.",
],
caution: None,
};
// --- Options: Security ---------------------------------------------------
pub static ENABLE_PASSWORD: Tip = Tip {
title: "Enable password protection",
body: "Encrypts the index with a password of your choosing. The index \
holds the names and the text of your files, so anyone who can \
read that file can read those; encrypting it means they cannot.\n\n\
QuickSearch then asks for the password each time it starts, unless \
you let it remember.",
examples: &[],
caution: Some(
"Turning protection on deletes the index and builds it again. Your files are \
not touched.",
),
};
pub static CHANGE_PASSWORD: Tip = Tip {
title: "Change password",
body: "Replaces the password the index is encrypted with. You are asked \
for a new one, and the index is encrypted again under it.",
examples: &[],
caution: Some(
"Changing the password deletes the index and builds it again. Your files are \
not touched.",
),
};
pub static DISABLE_PASSWORD: Tip = Tip {
title: "Disable protection",
body: "Removes the password and leaves the index unencrypted on disk. \
Anyone able to read that file can then see the names of your files \
and the text inside them.",
examples: &[],
caution: Some(
"Turning protection off deletes the index and builds it again. Your files are \
not touched.",
),
};
pub static REMEMBER_KEYCHAIN: Tip = Tip {
title: "Remember on this device",
body: "Hands the key to the password store your system already has, such \
as GNOME Keyring, KWallet, or Windows Credential Manager, so that \
QuickSearch can unlock the index without asking at startup.\n\n\
The password itself is never stored, only the key worked out from \
it, and only on this machine. Off, you type the password each time \
QuickSearch starts.",
examples: &[],
caution: None,
};
// --- Manage Index tab: indexing controls ---------------------------------
pub static START_NOW: Tip = Tip {
title: "Start indexing now",
body: "Runs a full pass over your indexed folders straight away instead \
of waiting for the next scheduled one. Worth doing after adding a \
folder, after changing a filter, or when the computer has been off \
for a while.\n\n\
Searching carries on working while it runs. Unavailable while a \
run is already under way.",
examples: &[],
caution: None,
};
pub static STOP_INDEXING: Tip = Tip {
title: "Stop",
body: "Stops the run in progress and switches to manual, so QuickSearch \
no longer watches for changes or reindexes on a schedule. The \
index stays as it is and searching still works, but it drifts out \
of date as your files change.\n\n\
Saved immediately: QuickSearch is still in manual the next time it \
starts.",
examples: &[],
caution: None,
};
pub static RETURN_TO_AUTO: Tip = Tip {
title: "Return to Automatic",
body: "Goes back to watching your folders and reindexing on a schedule, \
catching up on everything that changed while indexing was manual.\n\n\
Also saved, so this is how QuickSearch starts from now on.",
examples: &[],
caution: None,
};
pub static CLEAR_INDEX: Tip = Tip {
title: "Clear index",
body: "Deletes the index database. Searching finds nothing until it is \
built again, which for a large folder takes a while. Your own \
files are never touched.\n\n\
QuickSearch asks for confirmation first, then drops to manual so \
that it does not immediately rebuild what you just deleted.",
examples: &[],
caution: Some("This cannot be undone: the index has to be built from scratch again."),
};
// --- Manage Index tab: indexed folders -----------------------------------
pub static ADD_ROOT: Tip = Tip {
title: "Add an indexed folder",
body: "Adds a folder for QuickSearch to index, along with everything \
inside it. Choose it with the browser, or type the path and press \
Add.\n\n\
Indexed folders may not overlap, so a folder already inside \
another one is refused. Adding a folder starts an indexing pass to \
pick it up and leaves the rest of the index alone.",
examples: &["a second drive, or a network share you search often."],
caution: None,
};
pub static REMOVE_ROOT: Tip = Tip {
title: "Remove this folder",
body: "Stops indexing this folder and removes its entries from the \
index. The rest of the index is left alone, and the files \
themselves are not touched.\n\n\
Takes effect when you click Apply & Save.",
examples: &[],
caution: None,
};
pub static ROOT_WORKERS: Tip = Tip {
title: "Workers",
body: "How many folders QuickSearch explores at once inside this indexed \
folder. More of them finish sooner on storage that answers many \
requests at a time, which network drives do especially well, but \
they compete for the same disk.\n\n\
auto reads 4 on local storage and 16 on a network mount. Takes \
effect on the next indexing run.",
examples: &[
"auto unless indexing is slower than you would expect.",
"16 or more for a network share that is slow to answer each request.",
"2 to keep indexing out of the way on an older machine.",
],
caution: None,
};
// --- Manage Index tab: content filters -----------------------------------
pub static EXT_WHITELIST: Tip = Tip {
title: "Full-text extensions whitelist",
body: "Which kinds of file QuickSearch reads the text out of, one \
extension per line, the leading dot optional. Empty means every \
kind it understands.\n\n\
Every file is still indexed by name whatever you put here. A list \
also leaves out files with no extension at all, such as Makefile \
or README, unless you add the line (none). Anything after a # is a \
comment, so a file type can be switched off without losing the \
line.\n\n\
Narrowing the list discards the text it now excludes; widening it \
reads those files again.",
examples: &[
"txt, md and pdf to keep the index small and focused on documents.",
"empty to search inside everything QuickSearch can read.",
],
caution: None,
};
pub static IGNORE_PATTERNS: Tip = Tip {
title: "Ignore patterns",
body: "Files and folders left out of the index entirely, by name and by \
content alike. Type one pattern and click Add.\n\n\
A pattern without a slash matches a file or folder name anywhere, \
and must match the whole name: .jpg matches only something called \
exactly that, while *.jpg matches every JPEG. A pattern with a \
slash in it is matched against the whole path, and skips \
everything underneath. * stands for any run of characters and ? \
for a single one.\n\n\
Adding a pattern removes the entries it matches; removing one \
indexes them again.",
examples: &[
"node_modules to skip that folder wherever it turns up.",
"*.tmp to skip temporary files by extension.",
"a full path such as the Videos folder to skip it and everything inside it.",
],
caution: None,
};
// --- Shared --------------------------------------------------------------
pub static APPLY_SAVE: Tip = Tip {
title: "Apply & Save",
body: "Writes these settings to the configuration file and puts them to \
work straight away. Until you click here, your edits are only \
staged.\n\n\
Narrowing a setting removes the entries it now excludes; widening \
one indexes whatever it now allows. Only the tokenizer, the hash \
sample size and password protection need the index built again \
from scratch, and those ask first.",
examples: &[],
caution: None,
};
#[cfg(test)]
mod tests {
use super::*;
/// Every tip in the file. A tip missing from here is only missing from
/// the checks below, so keep it in step when adding one.
const ALL: &[&Tip] = &[
&DATABASE_PATH,
&REINDEX_INTERVAL,
&FOLLOW_SYMLINKS,
&INCLUDE_HIDDEN,
&TOKENIZER,
&HASH_LENGTH,
&MAX_STORED_TEXT,
&MAX_TEXT_FILE_SIZE,
&BATCH_SIZE,
&MAX_WAL_SIZE,
&STORE_TEXT,
&FUZZY_DEFAULT,
&FUZZY_EDITS,
&DISPLAY_LIMIT,
&RESULTS_PER_PAGE,
&DEBOUNCE,
&UI_SCALE,
&ENABLE_PASSWORD,
&CHANGE_PASSWORD,
&DISABLE_PASSWORD,
&REMEMBER_KEYCHAIN,
&START_NOW,
&STOP_INDEXING,
&RETURN_TO_AUTO,
&CLEAR_INDEX,
&ADD_ROOT,
&REMOVE_ROOT,
&ROOT_WORKERS,
&EXT_WHITELIST,
&IGNORE_PATTERNS,
&APPLY_SAVE,
];
/// Everything a tip can put on screen, as one string.
fn all_text(tip: &Tip) -> String {
let mut text = format!("{}\n{}", tip.title, tip.body);
for example in tip.examples {
text.push('\n');
text.push_str(example);
}
if let Some(caution) = tip.caution {
text.push('\n');
text.push_str(caution);
}
text
}
/// House style, and the one rule that is easy to break by pasting from
/// the source comments: these tooltips use no em-dashes.
#[test]
fn no_tip_uses_an_em_dash() {
for tip in ALL {
assert!(
!all_text(tip).contains('—'),
"{} uses an em-dash",
tip.title
);
}
}
#[test]
fn every_tip_is_filled_in() {
for tip in ALL {
assert!(!tip.title.trim().is_empty(), "a tip has no title");
assert!(
!tip.title.ends_with('.'),
"{}: title is not a sentence",
tip.title
);
assert!(
tip.body.trim().len() > 40,
"{}: body says too little",
tip.title
);
assert!(
tip.body.trim_end().ends_with('.'),
"{}: body is not a finished sentence",
tip.title
);
// "Stops the run in progress" under the title "Stop" is fine;
// "Stop. Stops the run" is the restatement worth catching, so
// the title only counts as repeated when a word ends there.
let restates = tip
.body
.strip_prefix(tip.title)
.is_some_and(|rest| !rest.starts_with(|c: char| c.is_alphanumeric()));
assert!(!restates, "{}: body repeats the title", tip.title);
for example in tip.examples {
assert!(
!example.trim().is_empty() && example.trim_end().ends_with('.'),
"{}: bad example {:?}",
tip.title,
example
);
}
if let Some(caution) = tip.caution {
assert!(
caution.trim_end().ends_with('.'),
"{}: caution is not a finished sentence",
tip.title
);
}
}
}
/// Descriptive, but a tooltip nobody reads to the end helps nobody.
#[test]
fn no_tip_is_a_wall_of_text() {
for tip in ALL {
let len = all_text(tip).chars().count();
assert!(len <= 900, "{} is {} characters long", tip.title, len);
}
}
/// Two controls sharing a title means one of them was pasted from the
/// other and never renamed.
#[test]
fn titles_are_distinct() {
let mut seen: Vec<&str> = ALL.iter().map(|t| t.title).collect();
seen.sort_unstable();
let count = seen.len();
seen.dedup();
assert_eq!(count, seen.len(), "two tips share a title: {:?}", seen);
}
/// The renderer puts every part on screen: title, body, examples and
/// caution. Written against the tip with all four.
#[test]
fn show_paints_every_part() {
let ctx = egui::Context::default();
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
let full = ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| TOKENIZER.show(ui));
});
let painted = crate::test_ui::painted_text(&full).join("\n");
assert!(painted.contains(TOKENIZER.title), "no title: {painted}");
assert!(painted.contains("trigram indexes every run"), "no body");
assert!(painted.contains("• trigram, the default"), "no examples");
assert!(
painted.contains(TOKENIZER.caution.unwrap()),
"no caution line"
);
}
/// A lone example reads as a sentence rather than a one-item list.
#[test]
fn a_single_example_is_prefixed_with_example() {
let ctx = egui::Context::default();
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
let full = ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| INCLUDE_HIDDEN.show(ui));
});
let painted = crate::test_ui::painted_text(&full).join("\n");
assert!(
painted.contains(&format!("Example: {}", INCLUDE_HIDDEN.examples[0])),
"{painted}"
);
}
/// A greyed-out control still explains itself. egui shows nothing on a
/// disabled widget unless the *disabled* tooltip is set too, and "why
/// can I not click this" is exactly when the answer is wanted: Stop,
/// Start indexing now and Return to Automatic are greyed out by turns.
#[test]
fn a_disabled_control_still_explains_itself() {
let ctx = egui::Context::default();
ctx.style_mut(|s| {
s.interaction.tooltip_delay = 0.0;
s.interaction.show_tooltips_only_when_still = false;
});
let run = |events: Vec<egui::Event>| {
let input = crate::test_ui::raw_input(egui::vec2(600.0, 400.0), events);
ctx.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
ui.add_enabled(false, egui::Button::new("Stop"))
.tip(&STOP_INDEXING);
});
})
};
run(vec![]);
let settled = run(vec![]);
let pos = crate::test_ui::painted_text_center(&settled, "Stop").expect("button painted");
let opening: String = STOP_INDEXING.body.chars().take(40).collect();
let mut out = run(vec![egui::Event::PointerMoved(pos)]);
for _ in 0..3 {
if crate::test_ui::painted_text(&out)
.join("\n")
.contains(&opening)
{
return;
}
out = run(vec![]);
}
panic!("a disabled control said nothing on hover");
}
}

188
crates/quicksearch-gui/src/ui_util.rs
View file

@ -1,7 +1,8 @@
//! Shared UI helpers: emphasis colors, bordered widgets, ignore-pattern
//! validation, and the "more content below" scroll hint.
//! validation, text eliding, and the "more content below" scroll hint.
use quicksearch_core::config::IgnoreSet;
use std::borrow::Cow;
/// Warning/emphasis orange, also used for the fuzzy-edit-distance warning.
pub const ORANGE: egui::Color32 = egui::Color32::from_rgb(220, 150, 40);
@ -136,6 +137,81 @@ pub fn pattern_edit(
(response, valid)
}
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
/// `font_id`, returning it borrowed and untouched when it already fits.
///
/// A path's two ends are the informative ones — the head says which volume
/// or home it lives under, the tail names the deepest directories — so a
/// column too narrow for the whole thing should drop out of the middle
/// rather than tail-truncate the way egui does by default.
///
/// The budget is in pixels, summed from the font's own glyph advances (the
/// same numbers egui's layout adds up), not a character count scaled by the
/// width of one sample glyph. The proportional body font makes that estimate
/// wrong in both directions: overshoot and egui elides the result a *second*
/// time, painting two ellipses; undershoot and the column sits visibly short
/// of full.
///
/// The borrowed/owned distinction is also the caller's signal that something
/// was dropped, which is what a "full text on hover" tooltip keys off.
pub fn middle_elide<'a>(
ui: &egui::Ui,
text: &'a str,
max_width: f32,
font_id: &egui::FontId,
) -> Cow<'a, str> {
ui.fonts(|f| {
let width_of = |c: char| f.glyph_width(font_id, c);
if text.chars().map(width_of).sum::<f32>() <= max_width {
return Cow::Borrowed(text);
}
let budget = max_width - width_of('…');
// Grow a head and a tail toward each other through the middle,
// each step feeding whichever side is currently narrower so the cut
// lands near the middle. Indices advance by whole characters, so
// they always land on UTF-8 boundaries.
let (mut head, mut tail) = (0usize, text.len());
let (mut head_w, mut tail_w) = (0.0f32, 0.0f32);
while head < tail {
let rest = &text[head..tail];
let front = rest.chars().next().expect("head < tail");
let back = rest.chars().next_back().expect("head < tail");
let (front_w, back_w) = (width_of(front), width_of(back));
let used = head_w + tail_w;
let (front_fits, back_fits) = (used + front_w <= budget, used + back_w <= budget);
if !front_fits && !back_fits {
break;
}
// The preferred side wins when it fits; otherwise the other one
// does, since at least one of them just did.
let take_front = if head_w <= tail_w {
front_fits
} else {
!back_fits
};
if take_front {
head += front.len_utf8();
head_w += front_w;
} else {
tail -= back.len_utf8();
tail_w += back_w;
}
}
if head >= tail {
// The two halves met without dropping anything — splicing an
// ellipsis in now would only lengthen a string that fits.
return Cow::Borrowed(text);
}
let mut out = String::with_capacity(head + '…'.len_utf8() + (text.len() - tail));
out.push_str(&text[..head]);
out.push('…');
out.push_str(&text[tail..]);
Cow::Owned(out)
})
}
/// Paint a semitransparent down-arrow near the bottom edge of a scroll
/// area while more content lies below the fold. Painter-only, so it can
/// never swallow clicks. (The bundled fonts have no ▼ glyph — this is a
@ -175,7 +251,115 @@ pub fn more_below_hint<R>(ui: &egui::Ui, out: &egui::scroll_area::ScrollAreaOutp
#[cfg(test)]
mod tests {
use super::{ignore_pattern_valid, pattern_border, pattern_hint, INVALID_RED, VALID_GREEN};
use super::{
ignore_pattern_valid, middle_elide, pattern_border, pattern_hint, Cow, INVALID_RED,
VALID_GREEN,
};
/// A `Ui` from a real (headless) egui pass, so `middle_elide` measures
/// with the same fonts the app paints with — the whole point of the
/// helper is that its arithmetic agrees with egui's layout.
fn with_ui<R>(f: impl FnOnce(&mut egui::Ui) -> R) -> R {
let ctx = egui::Context::default();
let mut f = Some(f);
let mut out = None;
let _ = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
if let Some(f) = f.take() {
out = Some(f(ui));
}
});
});
out.expect("the central panel ran")
}
fn body_font(ui: &egui::Ui) -> egui::FontId {
egui::TextStyle::Body.resolve(ui.style())
}
const DEEP: &str = "/media/shared/QuickSearch/crates/quicksearch-gui/src/search_tab.rs";
#[test]
fn text_that_fits_comes_back_untouched() {
with_ui(|ui| {
let font = body_font(ui);
let out = middle_elide(ui, DEEP, 10_000.0, &font);
assert!(matches!(out, Cow::Borrowed(_)), "borrowed when it fits");
assert_eq!(out, DEEP);
// Nothing to elide, however little room there is.
assert!(matches!(middle_elide(ui, "", 0.0, &font), Cow::Borrowed(_)));
});
}
#[test]
fn eliding_keeps_both_ends_and_cuts_once() {
with_ui(|ui| {
let out = middle_elide(ui, DEEP, 200.0, &body_font(ui));
assert!(matches!(out, Cow::Owned(_)), "{out}");
assert_eq!(out.matches('…').count(), 1, "{out}");
let (head, tail) = out.split_once('…').expect("one ellipsis");
assert!(
!head.is_empty() && !tail.is_empty(),
"both ends survive: {out}"
);
assert!(DEEP.starts_with(head), "{out}");
assert!(DEEP.ends_with(tail), "{out}");
assert!(
tail.ends_with("search_tab.rs"),
"the filename survives: {out}"
);
});
}
/// The load-bearing property. If the result laid out in the same font
/// were wider than the budget, egui would elide it a *second* time and
/// paint two ellipses.
#[test]
fn the_result_fits_the_budget_it_was_given() {
with_ui(|ui| {
let font = body_font(ui);
for width in [40.0f32, 60.0, 121.5, 200.0, 337.5, 480.0] {
let out = middle_elide(ui, DEEP, width, &font);
let painted = ui.fonts(|f| {
f.layout_no_wrap(out.to_string(), font.clone(), egui::Color32::WHITE)
.size()
.x
});
assert!(
painted <= width,
"at {width}: {out:?} lays out at {painted}"
);
}
});
}
/// An ellipsis is the least that can stand for the text; a column too
/// narrow even for that gets it anyway rather than a panic.
#[test]
fn degenerate_widths_never_panic() {
with_ui(|ui| {
let font = body_font(ui);
for width in [f32::NEG_INFINITY, -50.0, 0.0] {
assert_eq!(middle_elide(ui, DEEP, width, &font), "…", "at {width}");
}
});
}
/// Indices walk by whole characters, so a path of multi-byte glyphs
/// slices cleanly instead of panicking mid-codepoint.
#[test]
fn multi_byte_paths_split_on_character_boundaries() {
with_ui(|ui| {
let font = body_font(ui);
let path = "/srv/données/日本語/архив/файл-très-long.txt";
for width in [30.0f32, 55.0, 90.0, 140.0, 210.0, 400.0] {
let out = middle_elide(ui, path, width, &font);
let (head, tail) = out.split_once('…').unwrap_or((out.as_ref(), ""));
assert!(path.starts_with(head), "at {width}: {out}");
assert!(path.ends_with(tail), "at {width}: {out}");
}
});
}
/// The trap behind "my ignore filters don't work" reports: ".jpg" is an
/// exact-name pattern, and the hint must say so and offer "*.jpg".

40
crates/quicksearch-gui/src/unlock.rs
View file

@ -38,6 +38,9 @@ pub enum KeySource {
/// The application shell handed to eframe: locked (unlock screen) or
/// running (the real app).
// Exactly one of these exists for the lifetime of the process, and it is
// already boxed on the side that would matter.
#[allow(clippy::large_enum_variant)]
pub enum Gate {
Locked(UnlockScreen),
Running(Box<QuickSearchApp>),
@ -339,12 +342,11 @@ impl UnlockScreen {
/// UI-side buffers.
fn submit(&mut self, ctx: &egui::Context) {
self.error = None;
if matches!(self.mode, Mode::Create) {
if self.password.is_empty() {
if matches!(self.mode, Mode::Create)
&& self.password.is_empty() {
self.error = Some("The password may not be empty.".to_string());
return;
}
}
let Ok(salt) = self.cfg.security.salt_bytes() else {
return; // BrokenSalt mode never reaches submit
};
@ -381,8 +383,10 @@ impl UnlockScreen {
// stick. Surface it in the running app's banner.
self.config_error = Some(e);
}
} else {
keychain::delete_key(&db_path.to_string_lossy());
} else if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
// Same treatment as the store half above: unlock proceeds, but
// the banner says the old key is still on the keychain.
self.config_error = Some(e);
}
if self.cfg.security.use_keychain != self.remember {
self.cfg.security.use_keychain = self.remember;
@ -447,7 +451,12 @@ impl UnlockScreen {
if let Err(e) = delete_index_files(&db_path) {
self.error = Some(e);
} else {
keychain::delete_key(&db_path.to_string_lossy());
// The index files are already gone; a surviving
// entry would point at a database that no longer
// exists. Non-fatal, but not silent.
if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
self.config_error = Some(e);
}
db::set_process_key(None);
self.cfg.security = SecurityConfig::default();
if let Err(e) = self.cfg.save() {
@ -508,6 +517,9 @@ fn delete_index_files(db_path: &std::path::Path) -> Result<(), String> {
mod tests {
use super::*;
/// The unlock screen is the whole window, so its viewport is the window's.
const SCREEN: egui::Vec2 = egui::vec2(900.0, 600.0);
/// A protected config whose salt parses, so the screen lands in a real
/// password mode rather than `BrokenSalt`.
fn locked_config() -> Config {
@ -522,13 +534,7 @@ mod tests {
}
fn frame(ctx: &egui::Context, screen: &mut UnlockScreen) {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(900.0, 600.0),
)),
..Default::default()
};
let input = crate::test_ui::raw_input(SCREEN, Vec::new());
let _ = ctx.run(input, |ctx| {
// `update` only builds the app on a successful unlock, which needs
// a derived key — so with nothing submitted this stays on screen.
@ -591,13 +597,7 @@ mod tests {
fn the_lock_screen_shows_the_build_id() {
let ctx = egui::Context::default();
let mut screen = UnlockScreen::new(locked_config(), None, None);
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(900.0, 600.0),
)),
..Default::default()
};
let input = crate::test_ui::raw_input(SCREEN, Vec::new());
let out = ctx.run(input, |ctx| {
assert!(screen.update(ctx).is_none());

9
packaging/build-deb.sh
View file

@ -122,9 +122,16 @@ say "Installed ${#icons[@]} icon sizes plus the scalable SVG"
# Debian wants man pages and the changelog compressed, with no gzip timestamp so
# repeat builds are byte-identical. quicksearch-cli.1 is a one-line .so stub
# pointing at quicksearch.1, which documents both binaries.
#
# The .TH version field is rewritten from $version on the way through, so the
# footer of an installed page tracks [workspace.package] version instead of
# needing a hand-edit every release. The stub has no .TH, so the sed is a
# no-op there.
install -dm755 "$stage/usr/share/man/man1"
for page in "${BINARIES[@]}"; do
gzip -9nc "$REPO_ROOT/packaging/$page.1" > "$stage/usr/share/man/man1/$page.1.gz"
sed "/^\.TH /s/\"quicksearch [^\"]*\"/\"quicksearch $version\"/" \
"$REPO_ROOT/packaging/$page.1" \
| gzip -9nc > "$stage/usr/share/man/man1/$page.1.gz"
chmod 644 "$stage/usr/share/man/man1/$page.1.gz"
done

66
packaging/quicksearch.1
View file

@ -1,4 +1,7 @@
.TH QUICKSEARCH 1 "2026-08-02" "quicksearch 0.1.0" "User Commands"
.\" The version in the .TH line above is rewritten from [workspace.package]
.\" version by packaging/build-deb.sh at package time; the literal here is
.\" only what an uninstalled read of this file shows.
.TH QUICKSEARCH 1 "2026-08-05" "quicksearch 1.0.2" "User Commands"
.SH NAME
quicksearch, quicksearch\-cli \- fast full\-text search across your files
.SH SYNOPSIS
@ -60,29 +63,78 @@ of bare paths. Highlights the match in bold when stdout is a terminal.
.TP
.BR \-h ", " \-\-help
Print usage and exit.
.TP
.BR \-V ", " \-\-version
Print the version and the commit it was built from, then exit. Quote this in
bug reports.
.PP
An unrecognised option given without any query terms is passed through and the
application is opened, since it may be an option for the windowing backend.
.SH QUERY SYNTAX
Plain words form a single phrase. Filters may be combined with it:
Plain words form a single phrase, matched against names, contents and paths.
Filters may be combined with it:
.TP
.B \(dq\fIexact phrase\fB\(dq
Quoting keeps spaces, stars and filter\-like words literal.
.B \(dq\(dq
escapes a quote.
.TP
.BI bud * port
.B *
matches any run of characters, staying within one line of content.
.B %
and
.B _
are always literal.
.TP
.B regex:\fIpattern\fR
Match a regular expression against names, contents and paths, for example
.BR regex:\(dq(foo|bar)\ed+\(dq .
Case\-insensitive by default;
.B (?\-i:\(dq\(dq)
overrides. Quote patterns containing spaces or
.BR "( ) : = < > \(dq" .
.TP
.B type:\fIName\fR
Match a file class, for example
.IR type:Document ", " type:Image ", " type:Audio .
Match a file class: one of
.IR Audio ", " Image ", " Video ", " Document ", " Text ", " Archive ", "
.IR Spreadsheet ", " Presentation ", " Folder .
.TP
.B modified:\fIexpr\fR
Compare against the modification date, for example
.IR modified:>=2024-01-01 .
Also
.BR < ", " <= ", " > " and " = ;
dates are
.IR yyyy-mm-dd .
.B mtime:
is an alias.
.TP
.B path:\fI/dir\fR
Restrict results to a directory.
Restrict results to a directory and its subdirectories.
.B folder:
and
.B includefolder:
are aliases.
.B *
is literal here.
.TP
.B mime:\fItype\fR
Match a MIME type, for example
Match a MIME type exactly, for example
.IR mime:application/pdf .
.TP
.B name:\fIfragment\fR
Match a fragment of the filename.
Match a fragment of the filename. A filter, so it does not affect ranking.
.B filename:
is an alias; an unquoted
.B *
globs.
.PP
Unrecognised
.I key:value
text stays part of the search phrase.
.BR AND ", " OR
and parentheses are treated as plain words.
.SH PASSWORD PROTECTION
The index can be encrypted with a password (application Options, Security).
A protected index must be unlocked every time either binary starts. The