Improved Windows handling of Cloud folders. Improved indexing status reporting. Cleaned up some comments. Added support for old M$ word documents. Improved idle memory consumption.
This commit is contained in:
parent
ea224eed40
commit
e8057e596c
59 changed files with 7841 additions and 1658 deletions
5
Cargo.lock
generated
5
Cargo.lock
generated
|
|
@ -3190,9 +3190,10 @@ dependencies = [
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "quicksearch-core"
|
name = "quicksearch-core"
|
||||||
version = "0.9.2"
|
version = "1.0.2"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"argon2",
|
"argon2",
|
||||||
|
"cfb",
|
||||||
"chardetng",
|
"chardetng",
|
||||||
"ctrlc",
|
"ctrlc",
|
||||||
"encoding_rs",
|
"encoding_rs",
|
||||||
|
|
@ -3222,7 +3223,7 @@ dependencies = [
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "quicksearch-gui"
|
name = "quicksearch-gui"
|
||||||
version = "0.9.2"
|
version = "1.0.2"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"chrono",
|
"chrono",
|
||||||
"eframe",
|
"eframe",
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ members = [
|
||||||
]
|
]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "1.0.1"
|
version = "1.0.2"
|
||||||
edition = "2021"
|
edition = "2021"
|
||||||
license = "GPL-3.0-or-later"
|
license = "GPL-3.0-or-later"
|
||||||
authors = ["Jeremy <jeremy@karsttech.com>"]
|
authors = ["Jeremy <jeremy@karsttech.com>"]
|
||||||
|
|
|
||||||
72
README.md
72
README.md
|
|
@ -66,7 +66,7 @@ and exit codes behave normally. On Unix `quicksearch` also does both, and
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
./packaging/build-deb.sh
|
./packaging/build-deb.sh
|
||||||
sudo apt install ./dist/quicksearch_0.1.0_amd64.deb
|
sudo apt install ./dist/quicksearch_1.0.2_amd64.deb
|
||||||
```
|
```
|
||||||
|
|
||||||
The script builds the release binary, strips it, and assembles a `.deb` with
|
The script builds the release binary, strips it, and assembles a `.deb` with
|
||||||
|
|
@ -172,7 +172,7 @@ installer, so it takes `/S` for a silent install and `/D=` for the directory
|
||||||
(last argument, unquoted):
|
(last argument, unquoted):
|
||||||
|
|
||||||
```bat
|
```bat
|
||||||
quicksearch-0.9.1-windows-x86_64-setup.exe /S /D=C:\Tools\QuickSearch
|
quicksearch-1.0.2-windows-x86_64-setup.exe /S /D=C:\Tools\QuickSearch
|
||||||
```
|
```
|
||||||
|
|
||||||
The `.zip` on the release page is the alternative to all of this: the same two
|
The `.zip` on the release page is the alternative to all of this: the same two
|
||||||
|
|
@ -211,7 +211,17 @@ inside that folder.
|
||||||
what each tab does — pointing here for everything technical.
|
what each tab does — pointing here for everything technical.
|
||||||
|
|
||||||
The bottom status bar always shows what the indexer is doing (phase,
|
The bottom status bar always shows what the indexer is doing (phase,
|
||||||
percent, files/sec) or the total indexed file count when idle.
|
percent, files/sec) or the total indexed file count when idle. Applying a
|
||||||
|
settings change to the index counts as something the indexer is doing: it
|
||||||
|
reports its progress there and in the Manage Index tab, and says what it
|
||||||
|
removed for a few seconds after it finishes, so a change that takes a
|
||||||
|
millisecond is as visible as one that takes minutes.
|
||||||
|
|
||||||
|
Quitting while a settings change is still being applied asks first. Leaving
|
||||||
|
is never refused — the work stops promptly and the index stays consistent —
|
||||||
|
but it stops part-way, so entries you excluded can still turn up in search
|
||||||
|
results until indexing runs again. The next launch says so, with a button to
|
||||||
|
start that run; in automatic mode the periodic reindex does it for you.
|
||||||
|
|
||||||
### Terminal
|
### Terminal
|
||||||
|
|
||||||
|
|
@ -317,10 +327,12 @@ default index goes to `~/.local/share/quicksearch/index.sqlite`
|
||||||
|
|
||||||
Defaults follow the platform. The first indexing root is your home
|
Defaults follow the platform. The first indexing root is your home
|
||||||
directory or `%USERPROFILE%`; `include_hidden = false` skips dot-files
|
directory or `%USERPROFILE%`; `include_hidden = false` skips dot-files
|
||||||
everywhere and additionally anything marked Hidden or System on Windows,
|
everywhere and additionally anything marked Hidden on Windows, which is
|
||||||
which is what keeps `AppData`, `$RECYCLE.BIN` and `System Volume
|
what keeps `AppData`, `$RECYCLE.BIN` and `System Volume Information` out
|
||||||
Information` out of the index; and ignore patterns are matched
|
of the index — the System attribute alone is not enough, because cloud
|
||||||
case-insensitively on Windows and macOS, matching the filesystem.
|
sync roots carry it purely to get a branded folder icon; and ignore
|
||||||
|
patterns are matched case-insensitively on Windows and macOS, matching
|
||||||
|
the filesystem.
|
||||||
|
|
||||||
**Portable mode**: a `config.toml` sitting next to the `quicksearch`
|
**Portable mode**: a `config.toml` sitting next to the `quicksearch`
|
||||||
binary overrides the user config entirely, and relative paths inside any
|
binary overrides the user config entirely, and relative paths inside any
|
||||||
|
|
@ -381,8 +393,10 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
||||||
- **Indexing** (`indexing.rs`, `file_handling.rs`): full runs walk each
|
- **Indexing** (`indexing.rs`, `file_handling.rs`): full runs walk each
|
||||||
root (`filtered_walk` prunes hidden/ignored subtrees before descending),
|
root (`filtered_walk` prunes hidden/ignored subtrees before descending),
|
||||||
classify files by mtime into insert/update/skip, batch-write metadata,
|
classify files by mtime into insert/update/skip, batch-write metadata,
|
||||||
sweep stale rows, then extract content (plaintext, RTF, Office, PDF,
|
sweep stale rows, then extract content (plaintext, RTF, Office — both the
|
||||||
audio tags, EXIF; see `extract/`) for FTS. Files whose extension no MIME
|
OOXML/ODF zip formats and the pre-2007 binary `.doc`/`.xls`/`.ppt`, whose
|
||||||
|
OLE2 streams are read in `extract/ole.rs` — PDF, audio tags, EXIF; see
|
||||||
|
`extract/`) for FTS. Files whose extension no MIME
|
||||||
table knows — including extensionless ones like `README` or `Makefile` —
|
table knows — including extensionless ones like `README` or `Makefile` —
|
||||||
are sniffed from their head bytes and indexed as text only when that head
|
are sniffed from their head bytes and indexed as text only when that head
|
||||||
is provably text: valid UTF-8, or BOM-marked (`mime.rs`, `textenc.rs`).
|
is provably text: valid UTF-8, or BOM-marked (`mime.rs`, `textenc.rs`).
|
||||||
|
|
@ -397,7 +411,12 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
||||||
it completed or was stopped — with an optimize pass on its own connection:
|
it completed or was stopped — with an optimize pass on its own connection:
|
||||||
checkpoint, VACUUM if the file has at least 10% slack to reclaim, `PRAGMA
|
checkpoint, VACUUM if the file has at least 10% slack to reclaim, `PRAGMA
|
||||||
optimize`, checkpoint again. Progress streams through a polled
|
optimize`, checkpoint again. Progress streams through a polled
|
||||||
`IndexingStatus`, which reads `Optimizing` for the duration of that pass.
|
`IndexingStatus`, which reads `Optimizing` for the duration of that pass —
|
||||||
|
and `Preparing` for everything a run does before its first file is walked:
|
||||||
|
waiting on the previous run's thread, opening the index (a WAL recovery
|
||||||
|
lands here), and reconciling a changed configuration. Each carries the run's
|
||||||
|
start time, so a prologue that outlasts the walk on a large index reads as
|
||||||
|
slow work rather than a hang.
|
||||||
- **Scope reconciliation** (`scope.rs`): the index is a cache of what a walk
|
- **Scope reconciliation** (`scope.rs`): the index is a cache of what a walk
|
||||||
under the configured roots would produce, so a configuration change is a
|
under the configured roots would produce, so a configuration change is a
|
||||||
difference between the two rather than a reason to start over.
|
difference between the two rather than a reason to start over.
|
||||||
|
|
@ -412,6 +431,20 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
||||||
closed behave like one edited live. The scan is per-root, by `[lo, hi)`
|
closed behave like one edited live. The scan is per-root, by `[lo, hi)`
|
||||||
range: a symlink target stored outside every root has no owning root and
|
range: a symlink target stored outside every root has no owning root and
|
||||||
therefore no rules that could be applied to it, so it is never visited.
|
therefore no rules that could be applied to it, so it is never visited.
|
||||||
|
Whichever of the two applies it, a pass that *finishes* records what it
|
||||||
|
reconciled against — everything but the three rebuild-only keys, which no
|
||||||
|
scan can satisfy. That record is the whole convergence condition: an
|
||||||
|
abandoned pass leaves it alone and the next run picks the work back up,
|
||||||
|
while a completed one stops every later run from re-deriving the same plan
|
||||||
|
and rescanning every row to redo work already done. Both report a live
|
||||||
|
`ReconcileProgress` while they scan, since on a large index this is minutes
|
||||||
|
of work with no files moving to show for it. Both can also be abandoned:
|
||||||
|
`advance` reads a cancel flag before every statement, and the statement
|
||||||
|
already running — one `DELETE` can cover a whole root — is ended by
|
||||||
|
`sqlite3_interrupt`, since a flag alone cannot reach inside SQLite. That is
|
||||||
|
what makes closing the window during a prune immediate instead of a wait
|
||||||
|
the desktop offers to kill. `scope::outstanding_work` asks the record what
|
||||||
|
is still owed, which is how the GUI knows to remind you at the next launch.
|
||||||
- **Coordinator** (`coordinator.rs`): the object binaries construct.
|
- **Coordinator** (`coordinator.rs`): the object binaries construct.
|
||||||
Owns the `IndexingService`, the debouncing filesystem watcher
|
Owns the `IndexingService`, the debouncing filesystem watcher
|
||||||
(`watcher.rs`), and the mode state machine (Auto / Manual, persisted as
|
(`watcher.rs`), and the mode state machine (Auto / Manual, persisted as
|
||||||
|
|
@ -444,10 +477,12 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
||||||
flush last, so weaker matches only ever append. All SQL is
|
flush last, so weaker matches only ever append. All SQL is
|
||||||
parameterized; structured filters from the query language (`query/`)
|
parameterized; structured filters from the query language (`query/`)
|
||||||
are ANDed onto every pass.
|
are ANDed onto every pass.
|
||||||
- **Baloo compatibility** (`cli.rs`, `mime.rs`): read-only endpoints
|
- **Baloo compatibility** (`cli.rs`, `mime.rs`): the read API this repo's
|
||||||
(`status_for_path`, `list_failed`, `index_size_breakdown`, …) and a
|
parent consumes — `status_for_path`, `list_failed`,
|
||||||
Baloo-shaped type model, groundwork for a future `balooctl`-compatible
|
`index_size_breakdown`, `pending_content_count`, `clear_path` — plus a
|
||||||
layer.
|
Baloo-shaped type model. Only `index_counts` has a caller inside this
|
||||||
|
repository; the rest are a compatibility surface for the parent's
|
||||||
|
`balooctl` layer and are not dead code.
|
||||||
- **Logging** (`log.rs`): background reporting goes through `log_info!` /
|
- **Logging** (`log.rs`): background reporting goes through `log_info!` /
|
||||||
`log_warn!` rather than `println!`/`eprintln!`. Each writes its line to
|
`log_warn!` rather than `println!`/`eprintln!`. Each writes its line to
|
||||||
stderr *and* appends it to a bounded in-memory ring (newest 5000 lines,
|
stderr *and* appends it to a bounded in-memory ring (newest 5000 lines,
|
||||||
|
|
@ -456,7 +491,8 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
||||||
search hits, usage, the error a command exits with — stays on stdio.
|
search hits, usage, the error a command exits with — stays on stdio.
|
||||||
- **Platform differences** (`platform.rs`): the single home for `#[cfg]`.
|
- **Platform differences** (`platform.rs`): the single home for `#[cfg]`.
|
||||||
Home directory lookup, what counts as a hidden entry (dot-prefix, plus
|
Home directory lookup, what counts as a hidden entry (dot-prefix, plus
|
||||||
the Hidden/System attributes on Windows), network-filesystem detection
|
the Hidden attribute on Windows — System deliberately excluded, since
|
||||||
|
cloud sync roots set it to get a folder icon), network-filesystem detection
|
||||||
(`/proc/mounts` against `GetDriveTypeW`), path collation, and the
|
(`/proc/mounts` against `GetDriveTypeW`), path collation, and the
|
||||||
watch-registration strategy all live here, so the rest of the crate can
|
watch-registration strategy all live here, so the rest of the crate can
|
||||||
ask a question rather than test a target. Anything decidable from a
|
ask a question rather than test a target. Anything decidable from a
|
||||||
|
|
@ -490,7 +526,11 @@ pagination: the table is virtualized, so a single scroll list capped at
|
||||||
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
|
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
|
||||||
ranking, cancellation, incremental indexing, coordinator modes, config
|
ranking, cancellation, incremental indexing, coordinator modes, config
|
||||||
resolution, fuzzy matcher vs. brute-force oracle).
|
resolution, fuzzy matcher vs. brute-force oracle).
|
||||||
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units.
|
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units plus
|
||||||
|
headless egui tests that drive the real widgets — building an input frame,
|
||||||
|
synthesizing clicks and reading back the painted text (`test_ui.rs`) — over
|
||||||
|
the search and manage tabs, the options editor, the unlock gate, the logs
|
||||||
|
and duplicates tabs, and query highlighting.
|
||||||
- `QSB_SNIPPET_PERF=1 cargo test --release -p quicksearch-core --test
|
- `QSB_SNIPPET_PERF=1 cargo test --release -p quicksearch-core --test
|
||||||
snippet_perf -- --nocapture`: snippet pipeline benchmark.
|
snippet_perf -- --nocapture`: snippet pipeline benchmark.
|
||||||
- `.forgejo/workflows/ci.yml`: builds both platforms on every push to `master`
|
- `.forgejo/workflows/ci.yml`: builds both platforms on every push to `master`
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,7 @@ database_path = "~/.local/share/quicksearch/index.sqlite"
|
||||||
# buttons on the Manage Index tab write this value, so the mode you left the
|
# buttons on the Manage Index tab write this value, so the mode you left the
|
||||||
# app in is the mode it starts in.
|
# app in is the mode it starts in.
|
||||||
auto_index = true
|
auto_index = true
|
||||||
reindex_interval_minutes = 1440
|
reindex_interval_minutes = 60
|
||||||
# Follow symbolic links during directory walks. Applies to links pointing at
|
# Follow symbolic links during directory walks. Applies to links pointing at
|
||||||
# files as well as at directories: with this off a symlink is not resolved at
|
# files as well as at directories: with this off a symlink is not resolved at
|
||||||
# all, so its target is never indexed — which matters because a target can
|
# all, so its target is never indexed — which matters because a target can
|
||||||
|
|
@ -43,8 +43,12 @@ reindex_interval_minutes = 1440
|
||||||
# that are no longer in scope; turning it on reindexes to find them.
|
# that are no longer in scope; turning it on reindexes to find them.
|
||||||
follow_symlinks = false
|
follow_symlinks = false
|
||||||
# Index hidden files and directories. That means dot-files everywhere, and
|
# Index hidden files and directories. That means dot-files everywhere, and
|
||||||
# additionally anything carrying the Hidden or System attribute on Windows
|
# additionally anything carrying the Hidden attribute on Windows (AppData,
|
||||||
# (AppData, $RECYCLE.BIN, System Volume Information, pagefile.sys ...).
|
# $RECYCLE.BIN, System Volume Information, pagefile.sys ...). The System
|
||||||
|
# attribute on its own does not count: Windows honours the desktop.ini inside
|
||||||
|
# a folder only if the folder carries System or Read-only, so cloud sync roots
|
||||||
|
# (ownCloud, Nextcloud, OneDrive, Google Drive) and any folder given a custom
|
||||||
|
# icon carry it purely to get that icon, and are indexed normally.
|
||||||
# Turning this off removes the hidden entries already indexed.
|
# Turning this off removes the hidden entries already indexed.
|
||||||
include_hidden = false
|
include_hidden = false
|
||||||
# Empty = extract text from every supported format. Non-empty = content
|
# Empty = extract text from every supported format. Non-empty = content
|
||||||
|
|
@ -145,6 +149,13 @@ use_keychain = false
|
||||||
# together (0.5 – 2.5). Ctrl +/- and Ctrl 0 adjust it temporarily at
|
# together (0.5 – 2.5). Ctrl +/- and Ctrl 0 adjust it temporarily at
|
||||||
# runtime; this value is the persistent baseline.
|
# runtime; this value is the persistent baseline.
|
||||||
scale = 1.1
|
scale = 1.1
|
||||||
|
# Written by QuickSearch, not by you: the folders that have already shown
|
||||||
|
# the "more subfolders than the watcher can follow" warning, so restarting
|
||||||
|
# does not repeat it. Keyed by folder rather than a single flag so that
|
||||||
|
# adding a folder warns again — the trade-off changed — and pruned to the
|
||||||
|
# current folder list whenever it is applied. Deleting it just means the
|
||||||
|
# warnings come back once each.
|
||||||
|
watch_cap_warned_roots = []
|
||||||
|
|
||||||
[search]
|
[search]
|
||||||
# Start with the fuzzy passes enabled.
|
# Start with the fuzzy passes enabled.
|
||||||
|
|
|
||||||
|
|
@ -29,6 +29,11 @@ walkdir = "2.5.0"
|
||||||
# stored, so zstd/bzip2/aes-crypto are all dead weight here.
|
# stored, so zstd/bzip2/aes-crypto are all dead weight here.
|
||||||
zip = { version = "0.6", default-features = false, features = ["deflate"] }
|
zip = { version = "0.6", default-features = false, features = ["deflate"] }
|
||||||
quick-xml = "0.31"
|
quick-xml = "0.31"
|
||||||
|
# OLE2 compound-file reader, for the pre-2007 binary Office formats
|
||||||
|
# (.doc/.xls/.ppt) whose text lives in named streams rather than a zip. Already
|
||||||
|
# in the lockfile transitively via infer, so naming it directly compiles
|
||||||
|
# nothing new.
|
||||||
|
cfb = "0.7"
|
||||||
serde = { version = "1.0", features = ["derive"] }
|
serde = { version = "1.0", features = ["derive"] }
|
||||||
toml = "0.8"
|
toml = "0.8"
|
||||||
mime_guess = "2.0"
|
mime_guess = "2.0"
|
||||||
|
|
@ -62,8 +67,10 @@ regex = "1"
|
||||||
libc = "0.2"
|
libc = "0.2"
|
||||||
|
|
||||||
# `GetDriveTypeW` (a mapped drive letter is the only way to spot an SMB share
|
# `GetDriveTypeW` (a mapped drive letter is the only way to spot an SMB share
|
||||||
# that isn't written as UNC) plus the FILE_ATTRIBUTE_* constants for hidden
|
# that isn't written as UNC) plus the FILE_ATTRIBUTE_* constants, which
|
||||||
# detection. Pinned to 0.52 deliberately: walkdir → winapi-util already
|
# `platform.rs` spells out for itself so its tests run on Linux and then
|
||||||
|
# const-asserts against this crate on Windows builds.
|
||||||
|
# Pinned to 0.52 deliberately: walkdir → winapi-util already
|
||||||
# resolves exactly that version, so this adds no new crate compilations.
|
# resolves exactly that version, so this adds no new crate compilations.
|
||||||
[target.'cfg(windows)'.dependencies]
|
[target.'cfg(windows)'.dependencies]
|
||||||
# GetDriveTypeW and the FILE_ATTRIBUTE_* constants live in
|
# GetDriveTypeW and the FILE_ATTRIBUTE_* constants live in
|
||||||
|
|
@ -73,6 +80,9 @@ libc = "0.2"
|
||||||
# GetCurrentThread need Win32_Foundation for HANDLE/BOOL.
|
# GetCurrentThread need Win32_Foundation for HANDLE/BOOL.
|
||||||
windows-sys = { version = "0.52", features = [
|
windows-sys = { version = "0.52", features = [
|
||||||
"Win32_Foundation",
|
"Win32_Foundation",
|
||||||
|
# SECURITY_ATTRIBUTES, which `CreateFileW`'s signature names even though
|
||||||
|
# the directory-count path passes null for it.
|
||||||
|
"Win32_Security",
|
||||||
"Win32_Storage_FileSystem",
|
"Win32_Storage_FileSystem",
|
||||||
"Win32_System_Threading",
|
"Win32_System_Threading",
|
||||||
"Win32_System_WindowsProgramming",
|
"Win32_System_WindowsProgramming",
|
||||||
|
|
|
||||||
|
|
@ -198,7 +198,7 @@ fn prose(rng: &mut Rng, target: usize) -> String {
|
||||||
let mut s = String::with_capacity(target + 16);
|
let mut s = String::with_capacity(target + 16);
|
||||||
while s.len() < target {
|
while s.len() < target {
|
||||||
s.push_str(WORDS[rng.next() as usize % WORDS.len()]);
|
s.push_str(WORDS[rng.next() as usize % WORDS.len()]);
|
||||||
s.push(if rng.next() % 12 == 0 { '\n' } else { ' ' });
|
s.push(if rng.next().is_multiple_of(12) { '\n' } else { ' ' });
|
||||||
}
|
}
|
||||||
s.truncate(target);
|
s.truncate(target);
|
||||||
s
|
s
|
||||||
|
|
|
||||||
|
|
@ -207,7 +207,7 @@ fn run(mode: &str, root: &str, db: &Path, interval: Duration, config_path: Optio
|
||||||
file,
|
file,
|
||||||
});
|
});
|
||||||
|
|
||||||
if ticks % marker_every == 0 && db.exists() {
|
if ticks.is_multiple_of(marker_every) && db.exists() {
|
||||||
if let Ok(conn) = rusqlite::Connection::open(db) {
|
if let Ok(conn) = rusqlite::Connection::open(db) {
|
||||||
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
|
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
|
||||||
done = true;
|
done = true;
|
||||||
|
|
@ -254,6 +254,9 @@ fn progress(status: &IndexingStatus) -> (usize, usize, &'static str, String) {
|
||||||
(walked, extracted, phase, file)
|
(walked, extracted, phase, file)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Eight columns of one probe run, printed as a line. Grouping them into a
|
||||||
|
/// struct would only move the same eight names one level out.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
fn report(
|
fn report(
|
||||||
mode: &str,
|
mode: &str,
|
||||||
elapsed: Duration,
|
elapsed: Duration,
|
||||||
|
|
@ -267,8 +270,8 @@ fn report(
|
||||||
// One line per 5% of the run, so the shape is visible at any duration.
|
// One line per 5% of the run, so the shape is visible at any duration.
|
||||||
let step = (samples.len() / 20).max(1);
|
let step = (samples.len() / 20).max(1);
|
||||||
eprintln!(
|
eprintln!(
|
||||||
"\n {:>8} {:>10} {:>9} {:>10} {}",
|
"\n {:>8} {:>10} {:>9} {:>10} phase",
|
||||||
"t", "RSS", "walked", "extracted", "phase"
|
"t", "RSS", "walked", "extracted"
|
||||||
);
|
);
|
||||||
for s in samples.iter().step_by(step) {
|
for s in samples.iter().step_by(step) {
|
||||||
eprintln!(
|
eprintln!(
|
||||||
|
|
|
||||||
40
crates/quicksearch-core/examples/oleprobe.rs
Normal file
40
crates/quicksearch-core/examples/oleprobe.rs
Normal file
|
|
@ -0,0 +1,40 @@
|
||||||
|
//! Print what the legacy-Office extractor gets out of a real file.
|
||||||
|
//!
|
||||||
|
//! The unit tests build their own fixtures, which proves the parsers agree
|
||||||
|
//! with the format specs as read. This runs them against files a real
|
||||||
|
//! producer wrote, which is the other half of the question.
|
||||||
|
//!
|
||||||
|
//! cargo run --example oleprobe -- some.doc some.xls some.ppt
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use quicksearch_core::extract::{Extractor, Registry};
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let mut failures = 0;
|
||||||
|
for arg in std::env::args().skip(1) {
|
||||||
|
let path = Path::new(&arg);
|
||||||
|
println!("=== {} ===", path.display());
|
||||||
|
match quicksearch_core::extract::office::OfficeExtractor.extract(path) {
|
||||||
|
Ok(content) => {
|
||||||
|
let text = content.text;
|
||||||
|
println!("{} chars", text.chars().count());
|
||||||
|
let preview: String = text.chars().take(400).collect();
|
||||||
|
println!("{}", preview);
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
failures += 1;
|
||||||
|
println!("FAILED: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The dispatch a real index would take: MIME, not extension.
|
||||||
|
let registry = Registry::default_set();
|
||||||
|
let mime = mime_guess::from_path(path)
|
||||||
|
.first()
|
||||||
|
.map(|m| m.essence_str().to_string())
|
||||||
|
.unwrap_or_default();
|
||||||
|
println!("(mime {} claimed: {})", mime, registry.supports(&mime));
|
||||||
|
println!();
|
||||||
|
}
|
||||||
|
std::process::exit(if failures > 0 { 1 } else { 0 });
|
||||||
|
}
|
||||||
|
|
@ -1,10 +1,27 @@
|
||||||
//! Programmatic read-only query helpers.
|
//! Programmatic query helpers.
|
||||||
//!
|
//!
|
||||||
//! Pure functions that open a DB, run a query, and return structured data.
|
//! Pure functions that open a DB, run a query, and return structured data.
|
||||||
//! No stdout, no CLI framing — callers (GUI, future CLI binaries, Set B
|
//! No stdout, no CLI framing — the GUI and `quicksearch-cli` format the
|
||||||
//! `balooctl`) format the result as they see fit. Mutating operations live
|
//! result as they see fit. Indexing operations live on
|
||||||
//! on [`crate::indexing::IndexingService`] since they require a running
|
//! [`crate::indexing::IndexingService`] since they require a running worker
|
||||||
//! worker thread.
|
//! thread.
|
||||||
|
//!
|
||||||
|
//! # These are consumed from outside this repository
|
||||||
|
//!
|
||||||
|
//! QuickSearch is a sub-repo. Of everything here only [`index_counts`] has a
|
||||||
|
//! caller in this tree (the GUI status bar); [`status_for_path`],
|
||||||
|
//! [`list_failed`], [`index_size_breakdown`], [`pending_content_count`] and
|
||||||
|
//! [`clear_path`] are called by the parent repository's Baloo compat daemon,
|
||||||
|
//! which is what reports them to `balooctl` and mirrors them into LMDB.
|
||||||
|
//!
|
||||||
|
//! So they are **not dead code**, and their signatures are a compatibility
|
||||||
|
//! surface rather than an internal detail: a search of this repository alone
|
||||||
|
//! will not turn up the callers that break when one changes.
|
||||||
|
//!
|
||||||
|
//! One exception to the "query helpers" framing: [`clear_path`] mutates. It
|
||||||
|
//! opens its own writer, which sidesteps the single-writer discipline the
|
||||||
|
//! coordinator maintains, so it is safe only against an index no local
|
||||||
|
//! coordinator is running against.
|
||||||
|
|
||||||
use rusqlite::{params, OptionalExtension};
|
use rusqlite::{params, OptionalExtension};
|
||||||
|
|
||||||
|
|
@ -233,16 +250,7 @@ mod tests {
|
||||||
use crate::mime::FileType;
|
use crate::mime::FileType;
|
||||||
|
|
||||||
fn tmp_path() -> std::path::PathBuf {
|
fn tmp_path() -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir("cli").join("index.sqlite")
|
||||||
p.push(format!(
|
|
||||||
"qs-cli-test-{}-{}.sqlite",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn seed_fixture(db_path: &str) -> (i64, i64) {
|
fn seed_fixture(db_path: &str) -> (i64, i64) {
|
||||||
|
|
@ -328,6 +336,79 @@ mod tests {
|
||||||
std::fs::remove_file(&p).ok();
|
std::fs::remove_file(&p).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The count is "outstanding extraction work", which is a narrower thing
|
||||||
|
/// than "rows without text": a file nothing extracts is settled, not
|
||||||
|
/// waiting, and would otherwise be reported as a backlog that never
|
||||||
|
/// drains.
|
||||||
|
#[test]
|
||||||
|
fn pending_content_count_counts_only_outstanding_work() {
|
||||||
|
let p = tmp_path();
|
||||||
|
let dbp = p.to_str().unwrap();
|
||||||
|
// a is Done, b is Failed — both resolved, neither pending.
|
||||||
|
let _ = seed_fixture(dbp);
|
||||||
|
assert_eq!(pending_content_count(dbp).unwrap(), 0);
|
||||||
|
|
||||||
|
let mut conn = open_or_recreate(dbp, "trigram").unwrap();
|
||||||
|
{
|
||||||
|
let tx = conn.transaction().unwrap();
|
||||||
|
// Claimed by an extractor, text not read yet: this is the backlog.
|
||||||
|
insert_file(
|
||||||
|
&tx,
|
||||||
|
&NewFile {
|
||||||
|
name: "c.txt",
|
||||||
|
path: "/tmp/c.txt",
|
||||||
|
parent: "/tmp",
|
||||||
|
size: 1,
|
||||||
|
mtime: 1,
|
||||||
|
inode: None,
|
||||||
|
device_id: None,
|
||||||
|
mime: Some("text/plain"),
|
||||||
|
ftype: FileType::TEXT,
|
||||||
|
hash: None,
|
||||||
|
needs_content: true,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
.expect("unique path");
|
||||||
|
// Nothing extracts this one, so it is NA on arrival and must not
|
||||||
|
// inflate the figure.
|
||||||
|
insert_file(
|
||||||
|
&tx,
|
||||||
|
&NewFile {
|
||||||
|
name: "d.bin",
|
||||||
|
path: "/tmp/d.bin",
|
||||||
|
parent: "/tmp",
|
||||||
|
size: 1,
|
||||||
|
mtime: 1,
|
||||||
|
inode: None,
|
||||||
|
device_id: None,
|
||||||
|
mime: None,
|
||||||
|
ftype: FileType::EMPTY,
|
||||||
|
hash: None,
|
||||||
|
needs_content: false,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
.expect("unique path");
|
||||||
|
tx.commit().unwrap();
|
||||||
|
}
|
||||||
|
drop(conn);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
pending_content_count(dbp).unwrap(),
|
||||||
|
1,
|
||||||
|
"only the file awaiting extraction counts"
|
||||||
|
);
|
||||||
|
|
||||||
|
std::fs::remove_file(&p).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pending_content_count_on_a_missing_db_is_an_error() {
|
||||||
|
let missing = crate::testutil::scratch_dir("cli-missing").join("nope.sqlite");
|
||||||
|
assert!(pending_content_count(missing.to_str().unwrap()).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn index_size_breakdown_counts_rows() {
|
fn index_size_breakdown_counts_rows() {
|
||||||
let p = tmp_path();
|
let p = tmp_path();
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,7 @@ use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
|
#[derive(Default)]
|
||||||
pub struct Config {
|
pub struct Config {
|
||||||
pub paths: PathConfig,
|
pub paths: PathConfig,
|
||||||
pub indexing: IndexingConfig,
|
pub indexing: IndexingConfig,
|
||||||
|
|
@ -192,7 +193,7 @@ impl Default for IndexingConfig {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
IndexingConfig {
|
IndexingConfig {
|
||||||
auto_index: true,
|
auto_index: true,
|
||||||
reindex_interval_minutes: 24 * 60,
|
reindex_interval_minutes: 60,
|
||||||
follow_symlinks: false,
|
follow_symlinks: false,
|
||||||
include_hidden: false,
|
include_hidden: false,
|
||||||
content_extensions: Vec::new(),
|
content_extensions: Vec::new(),
|
||||||
|
|
@ -293,19 +294,6 @@ impl Default for UiConfig {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for Config {
|
|
||||||
fn default() -> Self {
|
|
||||||
Config {
|
|
||||||
paths: PathConfig::default(),
|
|
||||||
indexing: IndexingConfig::default(),
|
|
||||||
processing: ProcessingConfig::default(),
|
|
||||||
search: SearchConfig::default(),
|
|
||||||
ui: UiConfig::default(),
|
|
||||||
security: SecurityConfig::default(),
|
|
||||||
source: None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Directories and files excluded from a fresh index.
|
/// Directories and files excluded from a fresh index.
|
||||||
///
|
///
|
||||||
|
|
@ -451,8 +439,10 @@ impl Config {
|
||||||
cfg.source = Some(path.to_path_buf());
|
cfg.source = Some(path.to_path_buf());
|
||||||
Ok(cfg)
|
Ok(cfg)
|
||||||
} else {
|
} else {
|
||||||
let mut cfg = Config::default();
|
let cfg = Config {
|
||||||
cfg.source = Some(path.to_path_buf());
|
source: Some(path.to_path_buf()),
|
||||||
|
..Config::default()
|
||||||
|
};
|
||||||
cfg.save()?;
|
cfg.save()?;
|
||||||
Ok(cfg)
|
Ok(cfg)
|
||||||
}
|
}
|
||||||
|
|
@ -511,7 +501,7 @@ impl Config {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `indexing_paths` with the same resolution rules as
|
/// `indexing_paths` with the same resolution rules as
|
||||||
/// [`resolved_database_path`].
|
/// [`Config::resolved_database_path`].
|
||||||
pub fn resolved_indexing_paths(&self) -> Vec<PathBuf> {
|
pub fn resolved_indexing_paths(&self) -> Vec<PathBuf> {
|
||||||
self.paths
|
self.paths
|
||||||
.indexing_paths
|
.indexing_paths
|
||||||
|
|
@ -567,11 +557,44 @@ pub fn content_allowed(path: &Path, cfg: &Config) -> bool {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Longest name folded without allocating. Comfortably above `NAME_MAX` on the
|
||||||
|
/// filesystems that matter (255 bytes), so the heap path is effectively dead
|
||||||
|
/// code kept for correctness rather than for use.
|
||||||
|
const FOLD_BUF: usize = 256;
|
||||||
|
|
||||||
|
/// Whether `pat` is a plain name with no glob syntax in it.
|
||||||
|
///
|
||||||
|
/// `\` is included even though it is not glob syntax everywhere: a pattern
|
||||||
|
/// containing one is routed to the path set rather than the component set, so
|
||||||
|
/// treating it as literal here would put it in the wrong place.
|
||||||
|
fn is_literal_name(pat: &str) -> bool {
|
||||||
|
!pat.contains(['*', '?', '[', ']', '{', '}', '/', '\\'])
|
||||||
|
}
|
||||||
|
|
||||||
/// Compiled ignore patterns, split by matching scope: patterns without a
|
/// Compiled ignore patterns, split by matching scope: patterns without a
|
||||||
/// path separator match any single path component; the rest match the full
|
/// path separator match any single path component; the rest match the full
|
||||||
/// path. Both use glob syntax.
|
/// path. Both use glob syntax.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub struct IgnoreSet {
|
pub struct IgnoreSet {
|
||||||
|
/// Component patterns that are plain ASCII names — `.git`, `node_modules`,
|
||||||
|
/// `System Volume Information`. Held apart from `component` purely for
|
||||||
|
/// speed, and it is a large difference on Windows.
|
||||||
|
///
|
||||||
|
/// globset compiles a case-insensitive glob by giving up every fast path it
|
||||||
|
/// has: `Glob::literal`, `ext`, `prefix`, `suffix` and `basename_tokens` all
|
||||||
|
/// return `None` the moment `case_insensitive` is set, so every pattern
|
||||||
|
/// falls through to a `RegexSet` scan. Case-sensitively, `.git` and
|
||||||
|
/// `node_modules` compile to a hash lookup and `*.tmp` to an extension
|
||||||
|
/// lookup. That left Windows — which is case-insensitive *and* carries seven
|
||||||
|
/// extra default patterns — running a 12-pattern regex DFA over every single
|
||||||
|
/// directory entry, where Linux did five hash lookups.
|
||||||
|
///
|
||||||
|
/// Folded and compared as **ASCII**, matching [`PATH_COLLATION`]'s reasoning
|
||||||
|
/// exactly: SQLite's `NOCASE` and `LIKE` fold ASCII only, so the glob layer
|
||||||
|
/// agreeing with them is what keeps a path filter from disagreeing with the
|
||||||
|
/// ignore rules. Patterns that are not ASCII stay in `component` and keep
|
||||||
|
/// globset's Unicode folding, so nothing that matched before stops matching.
|
||||||
|
literal_components: std::collections::HashSet<String>,
|
||||||
component: globset::GlobSet,
|
component: globset::GlobSet,
|
||||||
path: globset::GlobSet,
|
path: globset::GlobSet,
|
||||||
empty: bool,
|
empty: bool,
|
||||||
|
|
@ -579,6 +602,7 @@ pub struct IgnoreSet {
|
||||||
|
|
||||||
impl IgnoreSet {
|
impl IgnoreSet {
|
||||||
pub fn compile(patterns: &[String]) -> Result<IgnoreSet, String> {
|
pub fn compile(patterns: &[String]) -> Result<IgnoreSet, String> {
|
||||||
|
let mut literal_components = std::collections::HashSet::new();
|
||||||
let mut component = globset::GlobSetBuilder::new();
|
let mut component = globset::GlobSetBuilder::new();
|
||||||
let mut path = globset::GlobSetBuilder::new();
|
let mut path = globset::GlobSetBuilder::new();
|
||||||
for pat in patterns {
|
for pat in patterns {
|
||||||
|
|
@ -605,6 +629,17 @@ impl IgnoreSet {
|
||||||
if pat.is_empty() {
|
if pat.is_empty() {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// A plain ASCII name needs no glob machinery at all — see
|
||||||
|
// `literal_components`. Everything else, including every non-ASCII
|
||||||
|
// pattern, goes on to globset unchanged.
|
||||||
|
if pat.is_ascii() && is_literal_name(pat) {
|
||||||
|
literal_components.insert(if crate::platform::PATHS_ARE_CASE_INSENSITIVE {
|
||||||
|
pat.to_ascii_lowercase()
|
||||||
|
} else {
|
||||||
|
pat.to_string()
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
let glob = globset::GlobBuilder::new(pat)
|
let glob = globset::GlobBuilder::new(pat)
|
||||||
.literal_separator(false)
|
.literal_separator(false)
|
||||||
// Match the filesystem's own rules, or `node_modules` fails to
|
// Match the filesystem's own rules, or `node_modules` fails to
|
||||||
|
|
@ -612,7 +647,7 @@ impl IgnoreSet {
|
||||||
// half of Windows compatibility on its own: `Candidate` folds
|
// half of Windows compatibility on its own: `Candidate` folds
|
||||||
// `\` to `/` when matching, and backslash-as-escape is off
|
// `\` to `/` when matching, and backslash-as-escape is off
|
||||||
// wherever `\` is a separator.
|
// wherever `\` is a separator.
|
||||||
.case_insensitive(cfg!(any(windows, target_os = "macos")))
|
.case_insensitive(crate::platform::PATHS_ARE_CASE_INSENSITIVE)
|
||||||
.build()
|
.build()
|
||||||
.map_err(|e| format!("invalid ignore pattern {:?}: {}", pat, e))?;
|
.map_err(|e| format!("invalid ignore pattern {:?}: {}", pat, e))?;
|
||||||
if pat.contains('/') || pat.contains('\\') {
|
if pat.contains('/') || pat.contains('\\') {
|
||||||
|
|
@ -627,18 +662,54 @@ impl IgnoreSet {
|
||||||
let path = path
|
let path = path
|
||||||
.build()
|
.build()
|
||||||
.map_err(|e| format!("compile ignore patterns: {}", e))?;
|
.map_err(|e| format!("compile ignore patterns: {}", e))?;
|
||||||
let empty = component.is_empty() && path.is_empty();
|
let empty = literal_components.is_empty() && component.is_empty() && path.is_empty();
|
||||||
Ok(IgnoreSet {
|
Ok(IgnoreSet {
|
||||||
|
literal_components,
|
||||||
component,
|
component,
|
||||||
path,
|
path,
|
||||||
empty,
|
empty,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether `name` is one of the plain-name patterns, folded per platform.
|
||||||
|
///
|
||||||
|
/// Allocation-free for any name that fits [`FOLD_BUF`], which is every name
|
||||||
|
/// a real filesystem can produce. This runs on every directory entry the
|
||||||
|
/// walker sees, so it is the one place in the ignore path worth keeping off
|
||||||
|
/// the heap.
|
||||||
|
fn matches_literal(&self, name: &str) -> bool {
|
||||||
|
if self.literal_components.is_empty() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if !crate::platform::PATHS_ARE_CASE_INSENSITIVE {
|
||||||
|
return self.literal_components.contains(name);
|
||||||
|
}
|
||||||
|
// Every stored literal is ASCII, and ASCII case folding maps ASCII to
|
||||||
|
// ASCII, so a name containing any non-ASCII byte cannot equal one.
|
||||||
|
if !name.is_ascii() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if name.len() <= FOLD_BUF {
|
||||||
|
let mut buf = [0u8; FOLD_BUF];
|
||||||
|
let buf = &mut buf[..name.len()];
|
||||||
|
buf.copy_from_slice(name.as_bytes());
|
||||||
|
buf.make_ascii_lowercase();
|
||||||
|
// Lowercasing ASCII yields ASCII, which is always valid UTF-8.
|
||||||
|
let folded = std::str::from_utf8(buf).expect("ascii stays utf-8");
|
||||||
|
return self.literal_components.contains(folded);
|
||||||
|
}
|
||||||
|
self.literal_components.contains(&name.to_ascii_lowercase())
|
||||||
|
}
|
||||||
|
|
||||||
/// Match a single file/directory name. Used by the walker to prune
|
/// Match a single file/directory name. Used by the walker to prune
|
||||||
/// subtrees before descending.
|
/// subtrees before descending.
|
||||||
pub fn matches_component(&self, name: &str) -> bool {
|
pub fn matches_component(&self, name: &str) -> bool {
|
||||||
!self.empty && self.component.is_match(name)
|
if self.empty {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
// The literal set answers almost every call — the defaults are all
|
||||||
|
// plain names — and answers it without touching the regex engine.
|
||||||
|
self.matches_literal(name) || self.component.is_match(name)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Match a path against the full-path patterns only. The path *and its
|
/// Match a path against the full-path patterns only. The path *and its
|
||||||
|
|
@ -669,9 +740,12 @@ impl IgnoreSet {
|
||||||
if self.matches_path_pattern(path) {
|
if self.matches_path_pattern(path) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
path.components().any(|c| {
|
path.components().any(|c| match c {
|
||||||
matches!(c, std::path::Component::Normal(name)
|
// Routed through `matches_component` rather than `self.component`
|
||||||
if self.component.is_match(Path::new(name)))
|
// directly, or the literal patterns would be invisible here and the
|
||||||
|
// watcher would index what the walker prunes.
|
||||||
|
std::path::Component::Normal(name) => self.matches_component(&name.to_string_lossy()),
|
||||||
|
_ => false,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -764,6 +838,42 @@ impl IndexWork {
|
||||||
pub fn scans_rows(&self) -> bool {
|
pub fn scans_rows(&self) -> bool {
|
||||||
self.prune_scope || self.reconcile_content || self.restore_text || self.drop_text
|
self.prune_scope || self.reconcile_content || self.restore_text || self.drop_text
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The plan in one line, for the log entry that announces the scan.
|
||||||
|
///
|
||||||
|
/// Names what changed rather than what will happen to the rows: the
|
||||||
|
/// reader is someone asking why a run has not started walking yet, and
|
||||||
|
/// the answer they need is which edit of theirs caused it.
|
||||||
|
pub fn summary(&self) -> String {
|
||||||
|
let mut parts: Vec<String> = Vec::new();
|
||||||
|
if !self.drop_roots.is_empty() {
|
||||||
|
parts.push(format!(
|
||||||
|
"{} root(s) no longer indexed",
|
||||||
|
self.drop_roots.len()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if self.prune_scope {
|
||||||
|
parts.push("narrowed ignore or hidden-file rules".into());
|
||||||
|
}
|
||||||
|
if self.drop_aliases {
|
||||||
|
parts.push("symlinks no longer followed".into());
|
||||||
|
}
|
||||||
|
if self.reconcile_content {
|
||||||
|
parts.push("changed content extensions".into());
|
||||||
|
}
|
||||||
|
if self.restore_text {
|
||||||
|
parts.push("snippet text turned on".into());
|
||||||
|
}
|
||||||
|
if self.drop_text {
|
||||||
|
parts.push("snippet text turned off".into());
|
||||||
|
}
|
||||||
|
if parts.is_empty() {
|
||||||
|
// `touches_index` is false here, so no caller logs this; a
|
||||||
|
// placeholder beats an empty pair of parentheses if one ever does.
|
||||||
|
return "no stored rows affected".into();
|
||||||
|
}
|
||||||
|
parts.join("; ")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What running services must do after a config edit. Computed by the GUI
|
/// What running services must do after a config edit. Computed by the GUI
|
||||||
|
|
@ -788,8 +898,8 @@ pub struct ConfigActions {
|
||||||
/// duplicates are reported once). Nested roots are disallowed: with one
|
/// duplicates are reported once). Nested roots are disallowed: with one
|
||||||
/// walker per root they would race for the same files and split progress
|
/// walker per root they would race for the same files and split progress
|
||||||
/// attribution. Comparison is on best-effort canonicalized paths (an
|
/// attribution. Comparison is on best-effort canonicalized paths (an
|
||||||
/// unresolvable root is compared as spelled) and is component-boundary
|
/// unresolvable root is compared as spelled), component-wise per
|
||||||
/// aware — `/a/bc` is not under `/a/b`.
|
/// [`crate::file_handling::UnreadableDirs::covers`].
|
||||||
pub fn nested_roots(roots: &[String]) -> Vec<(String, String)> {
|
pub fn nested_roots(roots: &[String]) -> Vec<(String, String)> {
|
||||||
let resolved: Vec<PathBuf> = roots
|
let resolved: Vec<PathBuf> = roots
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -932,17 +1042,7 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn tmp_dir() -> PathBuf {
|
fn tmp_dir() -> PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir("config")
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-config-{}-{}",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
fs::create_dir_all(&p).unwrap();
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -1298,6 +1398,89 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Which patterns take the fast path, and that taking it changes nothing
|
||||||
|
/// observable. A plain name is matched whole — never as a prefix, a
|
||||||
|
/// substring or a wildcard — and only its case is allowed to vary.
|
||||||
|
#[test]
|
||||||
|
fn the_literal_fast_path_matches_whole_names_only() {
|
||||||
|
let literal = IgnoreSet::compile(&["node_modules".to_string()]).unwrap();
|
||||||
|
assert!(
|
||||||
|
!literal.literal_components.is_empty(),
|
||||||
|
"a plain name belongs on the fast path"
|
||||||
|
);
|
||||||
|
for globby in ["node_module?", "node_*", "*.tmp", "a[bc]d", "x{1,2}"] {
|
||||||
|
assert!(
|
||||||
|
IgnoreSet::compile(&[globby.to_string()])
|
||||||
|
.unwrap()
|
||||||
|
.literal_components
|
||||||
|
.is_empty(),
|
||||||
|
"{} has glob syntax and must stay with globset",
|
||||||
|
globby
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
assert!(literal.matches_component("node_modules"));
|
||||||
|
for cased in ["Node_Modules", "NODE_MODULES", "node_moduleS"] {
|
||||||
|
assert_eq!(
|
||||||
|
literal.matches_component(cased),
|
||||||
|
cfg!(any(windows, target_os = "macos")),
|
||||||
|
"only case may vary, and only where the filesystem says so: {}",
|
||||||
|
cased
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for name in [
|
||||||
|
"node_modules_",
|
||||||
|
"_node_modules",
|
||||||
|
"nodemodules",
|
||||||
|
"node_module",
|
||||||
|
"src",
|
||||||
|
"",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!literal.matches_component(name),
|
||||||
|
"{} is not the ignored name",
|
||||||
|
name
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A non-ASCII pattern keeps globset's Unicode folding rather than being
|
||||||
|
/// silently downgraded to the ASCII fast path.
|
||||||
|
#[test]
|
||||||
|
fn non_ascii_patterns_stay_on_the_glob_path() {
|
||||||
|
let set = IgnoreSet::compile(&["café".to_string()]).unwrap();
|
||||||
|
assert!(
|
||||||
|
set.literal_components.is_empty(),
|
||||||
|
"a non-ASCII name must not join the ASCII-folded set"
|
||||||
|
);
|
||||||
|
assert!(set.matches_component("café"));
|
||||||
|
assert!(!set.matches_component("cafe"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Names longer than the stack fold buffer take the heap path, and must
|
||||||
|
/// come back with the same answer.
|
||||||
|
#[test]
|
||||||
|
fn overlong_names_still_fold_correctly() {
|
||||||
|
let long = "a".repeat(FOLD_BUF + 10);
|
||||||
|
let set = IgnoreSet::compile(std::slice::from_ref(&long)).unwrap();
|
||||||
|
assert!(set.matches_component(&long));
|
||||||
|
assert_eq!(
|
||||||
|
set.matches_component(&long.to_uppercase()),
|
||||||
|
cfg!(any(windows, target_os = "macos"))
|
||||||
|
);
|
||||||
|
assert!(!set.matches_component(&"a".repeat(FOLD_BUF + 9)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Watcher events are matched by whole path, and the literal patterns have
|
||||||
|
/// to be visible on that route too — otherwise the watcher indexes exactly
|
||||||
|
/// what the walker prunes and the index churns every cycle.
|
||||||
|
#[test]
|
||||||
|
fn full_path_matching_sees_literal_component_patterns() {
|
||||||
|
let set = IgnoreSet::compile(&["node_modules".to_string()]).unwrap();
|
||||||
|
assert!(set.matches_path(Path::new("/home/me/proj/node_modules/pkg/index.js")));
|
||||||
|
assert!(!set.matches_path(Path::new("/home/me/proj/src/index.js")));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn default_ignore_patterns_cover_the_platform() {
|
fn default_ignore_patterns_cover_the_platform() {
|
||||||
let d = IndexingConfig::default().ignore_patterns;
|
let d = IndexingConfig::default().ignore_patterns;
|
||||||
|
|
@ -1374,8 +1557,10 @@ mod tests {
|
||||||
fn root_workers_round_trip() {
|
fn root_workers_round_trip() {
|
||||||
let dir = tmp_dir();
|
let dir = tmp_dir();
|
||||||
let path = dir.join("config.toml");
|
let path = dir.join("config.toml");
|
||||||
let mut cfg = Config::default();
|
let mut cfg = Config {
|
||||||
cfg.source = Some(path.clone());
|
source: Some(path.clone()),
|
||||||
|
..Config::default()
|
||||||
|
};
|
||||||
cfg.paths.indexing_paths = vec!["/data".into(), "/share".into()];
|
cfg.paths.indexing_paths = vec!["/data".into(), "/share".into()];
|
||||||
cfg.indexing.root_workers.insert("/share".into(), 24);
|
cfg.indexing.root_workers.insert("/share".into(), 24);
|
||||||
cfg.save().unwrap();
|
cfg.save().unwrap();
|
||||||
|
|
@ -1705,8 +1890,10 @@ mod tests {
|
||||||
fn watch_cap_warned_roots_round_trips() {
|
fn watch_cap_warned_roots_round_trips() {
|
||||||
let dir = tmp_dir();
|
let dir = tmp_dir();
|
||||||
let path = dir.join("config.toml");
|
let path = dir.join("config.toml");
|
||||||
let mut cfg = Config::default();
|
let mut cfg = Config {
|
||||||
cfg.source = Some(path.clone());
|
source: Some(path.clone()),
|
||||||
|
..Config::default()
|
||||||
|
};
|
||||||
cfg.ui.watch_cap_warned_roots =
|
cfg.ui.watch_cap_warned_roots =
|
||||||
vec!["/media/ApolloStore".to_string(), "/media/GSSD".to_string()];
|
vec!["/media/ApolloStore".to_string(), "/media/GSSD".to_string()];
|
||||||
cfg.save().unwrap();
|
cfg.save().unwrap();
|
||||||
|
|
@ -1740,8 +1927,10 @@ mod tests {
|
||||||
fn fuzzy_max_edits_round_trips() {
|
fn fuzzy_max_edits_round_trips() {
|
||||||
let dir = tmp_dir();
|
let dir = tmp_dir();
|
||||||
let path = dir.join("config.toml");
|
let path = dir.join("config.toml");
|
||||||
let mut cfg = Config::default();
|
let mut cfg = Config {
|
||||||
cfg.source = Some(path.clone());
|
source: Some(path.clone()),
|
||||||
|
..Config::default()
|
||||||
|
};
|
||||||
cfg.search.fuzzy_max_edits = 4;
|
cfg.search.fuzzy_max_edits = 4;
|
||||||
cfg.save().unwrap();
|
cfg.save().unwrap();
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -175,11 +175,9 @@ impl ContentPass {
|
||||||
self.stats.clone()
|
self.stats.clone()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Join the workers and report whether every one finished cleanly.
|
/// Join the workers and report whether every one finished cleanly. Needed
|
||||||
///
|
/// for the same reason as [`crate::walk::ParallelWalk::finish`], which
|
||||||
/// The caller needs this for the same reason the walk does: a dead worker
|
/// states it.
|
||||||
/// and a finished worker both close the channel, so from the receiving end
|
|
||||||
/// they are indistinguishable.
|
|
||||||
pub fn finish(&mut self) -> bool {
|
pub fn finish(&mut self) -> bool {
|
||||||
// Dropping the receiver first releases any worker parked in `send`.
|
// Dropping the receiver first releases any worker parked in `send`.
|
||||||
self.rx = None;
|
self.rx = None;
|
||||||
|
|
@ -357,20 +355,9 @@ mod tests {
|
||||||
use crate::file_handling::{extract_scope_prepare, store_extracted};
|
use crate::file_handling::{extract_scope_prepare, store_extracted};
|
||||||
use crate::mime::FileType;
|
use crate::mime::FileType;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::time::UNIX_EPOCH;
|
/// A path that does not exist yet — the caller builds the tree under it.
|
||||||
|
|
||||||
fn tmp(tag: &str) -> PathBuf {
|
fn tmp(tag: &str) -> PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir(tag).join("tree")
|
||||||
p.push(format!(
|
|
||||||
"qs-content-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A tree of `n` text files under `root/sub`, plus an index holding a
|
/// A tree of `n` text files under `root/sub`, plus an index holding a
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ use crate::config::{diff_actions, Config, IgnoreSet, IndexWork};
|
||||||
use crate::db;
|
use crate::db;
|
||||||
use crate::extract::Registry;
|
use crate::extract::Registry;
|
||||||
use crate::incremental::apply_fs_event;
|
use crate::incremental::apply_fs_event;
|
||||||
use crate::indexing::{ConfigChange, IndexingService, IndexingStatus};
|
use crate::indexing::{ConfigChange, IndexingService, IndexingStatus, PrepStep, ReconcileProgress};
|
||||||
use crate::scope::WorkCursor;
|
use crate::scope::WorkCursor;
|
||||||
use crate::watcher::{FsEvent, WatchError, WatchFilters, Watcher, WatcherConfig};
|
use crate::watcher::{FsEvent, WatchError, WatchFilters, Watcher, WatcherConfig};
|
||||||
|
|
||||||
|
|
@ -67,6 +67,28 @@ pub enum WatcherStatus {
|
||||||
Disabled { reason: WatchError },
|
Disabled { reason: WatchError },
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A config reconciliation the coordinator applies between runs.
|
||||||
|
///
|
||||||
|
/// Separate from [`IndexingStatus`] on purpose. That enum is what every caller
|
||||||
|
/// reads to decide whether a full run owns the database, and this work is the
|
||||||
|
/// coordinator's own — putting it there would make the tick that performs it
|
||||||
|
/// believe it must keep off the file. On a large index it is minutes of
|
||||||
|
/// scanning that used to report nothing but `Idle`.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum ReconcileState {
|
||||||
|
/// Scanning now; the counters move every slice.
|
||||||
|
Running(ReconcileProgress),
|
||||||
|
/// Finished within the last [`RECONCILE_SUMMARY_LINGER`].
|
||||||
|
///
|
||||||
|
/// The tail is the whole point: narrowing a filter on a small index is
|
||||||
|
/// over in a millisecond, and a display that only existed while the work
|
||||||
|
/// did would leave the user with no sign it ever happened.
|
||||||
|
Finished(ReconcileProgress),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How long a finished reconciliation keeps reporting itself.
|
||||||
|
pub const RECONCILE_SUMMARY_LINGER: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
/// One-stop poll surface for the GUI.
|
/// One-stop poll surface for the GUI.
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct IndexerState {
|
pub struct IndexerState {
|
||||||
|
|
@ -78,8 +100,21 @@ pub struct IndexerState {
|
||||||
pub queued_events: usize,
|
pub queued_events: usize,
|
||||||
/// Live-update health; see [`WatcherStatus`].
|
/// Live-update health; see [`WatcherStatus`].
|
||||||
pub watcher: WatcherStatus,
|
pub watcher: WatcherStatus,
|
||||||
|
/// The between-runs reconciliation, while it runs and briefly after.
|
||||||
|
///
|
||||||
|
/// A run's *own* reconciliation is not here — it is a step of the run, and
|
||||||
|
/// reads as [`IndexingStatus::Preparing`] with a
|
||||||
|
/// [`PrepStep::Reconciling`]. Both report the same
|
||||||
|
/// [`ReconcileProgress`], so the display does not have to care which one
|
||||||
|
/// it is looking at.
|
||||||
|
pub reconcile: Option<ReconcileState>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// `ConfigChanged(Config)` dwarfs the unit variants, but these are sent one
|
||||||
|
// at a time down an mpsc channel at human cadence — a mode flip, a settings
|
||||||
|
// save. Boxing would trade a rare oversized move for an allocation on a path
|
||||||
|
// whose whole job is to be simple.
|
||||||
|
#[allow(clippy::large_enum_variant)]
|
||||||
enum CoordCmd {
|
enum CoordCmd {
|
||||||
SetMode(IndexMode),
|
SetMode(IndexMode),
|
||||||
ReindexNow,
|
ReindexNow,
|
||||||
|
|
@ -95,6 +130,34 @@ pub struct IndexCoordinator {
|
||||||
shared: Arc<Mutex<Shared>>,
|
shared: Arc<Mutex<Shared>>,
|
||||||
handle: Mutex<Option<JoinHandle<()>>>,
|
handle: Mutex<Option<JoinHandle<()>>>,
|
||||||
stopped: AtomicBool,
|
stopped: AtomicBool,
|
||||||
|
/// How [`Self::shutdown`] reaches a reconciliation in progress; see
|
||||||
|
/// [`Inner::apply_work`].
|
||||||
|
reconcile_stop: Arc<ReconcileStop>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The two halves of cutting the coordinator's reconciliation short — the pair
|
||||||
|
/// [`db::InterruptSlot`] describes.
|
||||||
|
///
|
||||||
|
/// A command cannot do it: the thread that would read the command is the
|
||||||
|
/// thread inside the scan. Closing the window used to wait for both.
|
||||||
|
#[derive(Default)]
|
||||||
|
struct ReconcileStop {
|
||||||
|
cancel: AtomicBool,
|
||||||
|
interrupt: db::InterruptSlot,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ReconcileStop {
|
||||||
|
/// Flag first, then interrupt: the flag is what stops the *next*
|
||||||
|
/// statement, and setting it after the interrupt would leave a window in
|
||||||
|
/// which the pass starts one more.
|
||||||
|
fn stop(&self) {
|
||||||
|
self.cancel.store(true, Ordering::SeqCst);
|
||||||
|
db::interrupt(&self.interrupt);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cancelled(&self) -> bool {
|
||||||
|
self.cancel.load(Ordering::SeqCst)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// State mirrored out of the coordinator thread for `state()`.
|
/// State mirrored out of the coordinator thread for `state()`.
|
||||||
|
|
@ -103,6 +166,7 @@ struct Shared {
|
||||||
last_full_index: Option<u64>,
|
last_full_index: Option<u64>,
|
||||||
queued_events: usize,
|
queued_events: usize,
|
||||||
watcher: WatcherStatus,
|
watcher: WatcherStatus,
|
||||||
|
reconcile: Option<ReconcileState>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl IndexCoordinator {
|
impl IndexCoordinator {
|
||||||
|
|
@ -130,12 +194,15 @@ impl IndexCoordinator {
|
||||||
last_full_index: None,
|
last_full_index: None,
|
||||||
queued_events: 0,
|
queued_events: 0,
|
||||||
watcher: WatcherStatus::Off,
|
watcher: WatcherStatus::Off,
|
||||||
|
reconcile: None,
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
let reconcile_stop = Arc::new(ReconcileStop::default());
|
||||||
let mut inner = Inner {
|
let mut inner = Inner {
|
||||||
config,
|
config,
|
||||||
indexing: indexing.clone(),
|
indexing: indexing.clone(),
|
||||||
shared: shared.clone(),
|
shared: shared.clone(),
|
||||||
|
reconcile_stop: reconcile_stop.clone(),
|
||||||
event_tx,
|
event_tx,
|
||||||
event_rx,
|
event_rx,
|
||||||
watcher: None,
|
watcher: None,
|
||||||
|
|
@ -147,6 +214,8 @@ impl IndexCoordinator {
|
||||||
pending_since: None,
|
pending_since: None,
|
||||||
needs_full_run: false,
|
needs_full_run: false,
|
||||||
pending_work: None,
|
pending_work: None,
|
||||||
|
reconcile_done: None,
|
||||||
|
reconcile_cut_short: false,
|
||||||
saw_running: false,
|
saw_running: false,
|
||||||
write_conn: None,
|
write_conn: None,
|
||||||
ignore: Arc::new(IgnoreSet::compile(&[]).expect("empty ignore set")),
|
ignore: Arc::new(IgnoreSet::compile(&[]).expect("empty ignore set")),
|
||||||
|
|
@ -167,6 +236,7 @@ impl IndexCoordinator {
|
||||||
shared,
|
shared,
|
||||||
handle: Mutex::new(Some(handle)),
|
handle: Mutex::new(Some(handle)),
|
||||||
stopped: AtomicBool::new(false),
|
stopped: AtomicBool::new(false),
|
||||||
|
reconcile_stop,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -178,6 +248,7 @@ impl IndexCoordinator {
|
||||||
last_full_index: shared.last_full_index,
|
last_full_index: shared.last_full_index,
|
||||||
queued_events: shared.queued_events,
|
queued_events: shared.queued_events,
|
||||||
watcher: shared.watcher.clone(),
|
watcher: shared.watcher.clone(),
|
||||||
|
reconcile: shared.reconcile,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -223,15 +294,41 @@ impl IndexCoordinator {
|
||||||
|
|
||||||
/// Stop the watcher, any running index pass, and the coordinator
|
/// Stop the watcher, any running index pass, and the coordinator
|
||||||
/// thread. Idempotent; usable from a signal handler through an Arc.
|
/// thread. Idempotent; usable from a signal handler through an Arc.
|
||||||
|
///
|
||||||
|
/// Cancelling the reconciliation comes *before* the command, because the
|
||||||
|
/// command is read by the thread the scan is running on: a pass part-way
|
||||||
|
/// through a large index would otherwise hold this join — and with it the
|
||||||
|
/// window close that called it — for as long as the scan had left.
|
||||||
pub fn shutdown(&self) {
|
pub fn shutdown(&self) {
|
||||||
if self.stopped.swap(true, Ordering::SeqCst) {
|
if self.stopped.swap(true, Ordering::SeqCst) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
self.reconcile_stop.stop();
|
||||||
let _ = self.cmd_tx.send(CoordCmd::Shutdown);
|
let _ = self.cmd_tx.send(CoordCmd::Shutdown);
|
||||||
if let Some(handle) = self.handle.lock().unwrap().take() {
|
if let Some(handle) = self.handle.lock().unwrap().take() {
|
||||||
let _ = handle.join();
|
let _ = handle.join();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether a configuration change is being applied to the index right
|
||||||
|
/// now, by either of the two places that can be doing it.
|
||||||
|
///
|
||||||
|
/// For a caller deciding whether to warn before quitting: an abandoned
|
||||||
|
/// pass leaves entries the user excluded still in the index until the next
|
||||||
|
/// indexing run redoes it.
|
||||||
|
pub fn reconciling(&self) -> bool {
|
||||||
|
// The lock is released before the service is asked, so this never
|
||||||
|
// holds two of them at once.
|
||||||
|
let between_runs = self.shared.lock().unwrap().reconcile;
|
||||||
|
matches!(between_runs, Some(ReconcileState::Running(_)))
|
||||||
|
|| matches!(
|
||||||
|
self.indexing.get_status(),
|
||||||
|
IndexingStatus::Preparing {
|
||||||
|
step: PrepStep::Reconciling(_),
|
||||||
|
..
|
||||||
|
}
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Drop for IndexCoordinator {
|
impl Drop for IndexCoordinator {
|
||||||
|
|
@ -281,9 +378,7 @@ fn collapse_pending_removals(pending: &mut HashMap<PathBuf, FsEvent>) {
|
||||||
.filter(|(_, ev)| is_removal(ev))
|
.filter(|(_, ev)| is_removal(ev))
|
||||||
.map(|(p, _)| p.clone())
|
.map(|(p, _)| p.clone())
|
||||||
.collect();
|
.collect();
|
||||||
// `Path::ancestors` walks whole components, so `/a/bc` is never treated as
|
// Component-wise containment, per `UnreadableDirs::covers`.
|
||||||
// living under `/a/b` — the rule `remove_tree` and `UnreadableDirs::covers`
|
|
||||||
// also use.
|
|
||||||
pending.retain(|path, ev| {
|
pending.retain(|path, ev| {
|
||||||
!is_removal(ev) || !path.ancestors().skip(1).any(|a| removed.contains(a))
|
!is_removal(ev) || !path.ancestors().skip(1).any(|a| removed.contains(a))
|
||||||
});
|
});
|
||||||
|
|
@ -303,6 +398,9 @@ struct Inner {
|
||||||
config: Config,
|
config: Config,
|
||||||
indexing: Arc<IndexingService>,
|
indexing: Arc<IndexingService>,
|
||||||
shared: Arc<Mutex<Shared>>,
|
shared: Arc<Mutex<Shared>>,
|
||||||
|
/// Read inside the reconciliation, set from the thread that shuts this
|
||||||
|
/// one down; see [`ReconcileStop`].
|
||||||
|
reconcile_stop: Arc<ReconcileStop>,
|
||||||
event_tx: mpsc::Sender<FsEvent>,
|
event_tx: mpsc::Sender<FsEvent>,
|
||||||
event_rx: mpsc::Receiver<FsEvent>,
|
event_rx: mpsc::Receiver<FsEvent>,
|
||||||
watcher: Option<Watcher>,
|
watcher: Option<Watcher>,
|
||||||
|
|
@ -325,6 +423,11 @@ struct Inner {
|
||||||
/// manual mode, whereas a config change the user just made is acted on in
|
/// manual mode, whereas a config change the user just made is acted on in
|
||||||
/// either mode.
|
/// either mode.
|
||||||
pending_work: Option<WorkCursor>,
|
pending_work: Option<WorkCursor>,
|
||||||
|
/// The last reconciliation to finish, and when. Published until it is
|
||||||
|
/// [`RECONCILE_SUMMARY_LINGER`] old; see [`ReconcileState::Finished`].
|
||||||
|
reconcile_done: Option<(ReconcileProgress, Instant)>,
|
||||||
|
/// A reconciliation was abandoned part-way; read by [`Inner::teardown`].
|
||||||
|
reconcile_cut_short: bool,
|
||||||
/// A start was requested; set false once the service reports running,
|
/// A start was requested; set false once the service reports running,
|
||||||
/// so idle-after-running transitions are detectable.
|
/// so idle-after-running transitions are detectable.
|
||||||
saw_running: bool,
|
saw_running: bool,
|
||||||
|
|
@ -407,8 +510,11 @@ impl Inner {
|
||||||
CoordCmd::RebuildIndex => {
|
CoordCmd::RebuildIndex => {
|
||||||
let db = self.db_path();
|
let db = self.db_path();
|
||||||
self.write_conn = None;
|
self.write_conn = None;
|
||||||
// Nothing to reconcile against once the file is gone.
|
// Nothing to reconcile against once the file is gone — and
|
||||||
|
// nothing to report about what was reconciled in the index
|
||||||
|
// that is about to stop existing.
|
||||||
self.pending_work = None;
|
self.pending_work = None;
|
||||||
|
self.reconcile_done = None;
|
||||||
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
|
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
|
||||||
crate::log_warn!("coordinator: rebuild: {}", e);
|
crate::log_warn!("coordinator: rebuild: {}", e);
|
||||||
}
|
}
|
||||||
|
|
@ -423,6 +529,7 @@ impl Inner {
|
||||||
self.enter_manual_stopped();
|
self.enter_manual_stopped();
|
||||||
self.write_conn = None;
|
self.write_conn = None;
|
||||||
self.pending_work = None;
|
self.pending_work = None;
|
||||||
|
self.reconcile_done = None;
|
||||||
let db = self.db_path();
|
let db = self.db_path();
|
||||||
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
|
if let Err(e) = self.indexing.delete_index_for_rebuild(&db) {
|
||||||
crate::log_warn!("coordinator: clear index: {}", e);
|
crate::log_warn!("coordinator: clear index: {}", e);
|
||||||
|
|
@ -439,7 +546,8 @@ impl Inner {
|
||||||
|
|
||||||
let status = self.indexing.get_status();
|
let status = self.indexing.get_status();
|
||||||
match status {
|
match status {
|
||||||
IndexingStatus::Running { .. }
|
IndexingStatus::Preparing { .. }
|
||||||
|
| IndexingStatus::Running { .. }
|
||||||
| IndexingStatus::Stopping
|
| IndexingStatus::Stopping
|
||||||
| IndexingStatus::Optimizing => {
|
| IndexingStatus::Optimizing => {
|
||||||
// Single-writer rule: never touch the DB while a full run
|
// Single-writer rule: never touch the DB while a full run
|
||||||
|
|
@ -545,15 +653,34 @@ impl Inner {
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
let mut cursor = self.pending_work.take().expect("caller checked");
|
let mut cursor = self.pending_work.take().expect("caller checked");
|
||||||
let outcome = crate::scope::advance(
|
let outcome = {
|
||||||
|
// Held only for the slice: the handle names whatever statement
|
||||||
|
// this connection is running, and outside `advance` that is
|
||||||
|
// nothing this cancellation has any business ending.
|
||||||
|
let _armed = db::InterruptGuard::arm(&self.reconcile_stop.interrupt, &conn);
|
||||||
|
crate::scope::advance(
|
||||||
&mut conn,
|
&mut conn,
|
||||||
&self.config,
|
&self.config,
|
||||||
&self.registry,
|
&self.registry,
|
||||||
&mut cursor,
|
&mut cursor,
|
||||||
Instant::now() + crate::scope::SLICE,
|
Instant::now() + crate::scope::SLICE,
|
||||||
);
|
&self.reconcile_stop.cancel,
|
||||||
|
)
|
||||||
|
};
|
||||||
self.write_conn = Some(conn);
|
self.write_conn = Some(conn);
|
||||||
if let Err(e) = outcome {
|
if let Err(e) = outcome {
|
||||||
|
// A cancelled statement fails like any other, and telling the two
|
||||||
|
// apart from a stringified error is guesswork — so ask the flag we
|
||||||
|
// set ourselves. Shutting down mid-scan is not a fault to report.
|
||||||
|
if self.reconcile_stop.cancelled() {
|
||||||
|
self.reconcile_cut_short = true;
|
||||||
|
crate::log_info!(
|
||||||
|
"configuration change interrupted after {} index entries; \
|
||||||
|
the next indexing run starts it again",
|
||||||
|
cursor.progress().examined
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
// Abandoned rather than retried: the cursor is already dropped,
|
// Abandoned rather than retried: the cursor is already dropped,
|
||||||
// and a database error that persists would otherwise spin this
|
// and a database error that persists would otherwise spin this
|
||||||
// loop for the life of the process. The next full run reconciles
|
// loop for the life of the process. The next full run reconciles
|
||||||
|
|
@ -566,9 +693,34 @@ impl Inner {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if !cursor.done() {
|
if !cursor.done() {
|
||||||
|
// Nothing is recorded for a pass that stopped early, cancelled or
|
||||||
|
// not: the stale record is what makes the next run redo it.
|
||||||
self.pending_work = Some(cursor);
|
self.pending_work = Some(cursor);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// Held for the linger so the display outlives the work: on a small
|
||||||
|
// index this whole pass is over between two frames.
|
||||||
|
self.reconcile_done = Some((cursor.progress(), Instant::now()));
|
||||||
|
// Record what the pass just brought the index into line with. Only
|
||||||
|
// here, on the path where the work finished and nothing errored: the
|
||||||
|
// stale record is what makes the next full run redo an abandoned
|
||||||
|
// reconcile, and it is the documented backstop for the error path
|
||||||
|
// above. Without this stamp a completed prune left the index still
|
||||||
|
// describing itself with the old configuration, so every later run
|
||||||
|
// rescanned every row to re-apply work already done — the silent wait
|
||||||
|
// a large index spends before its walk starts.
|
||||||
|
if let Some(conn) = self.write_conn.as_ref() {
|
||||||
|
// The same spelling a run records: canonicalized, and sorted into
|
||||||
|
// one string by `config_validation_entries`.
|
||||||
|
let roots: Vec<String> = self
|
||||||
|
.config
|
||||||
|
.normalized_indexing_paths()
|
||||||
|
.into_iter()
|
||||||
|
.collect();
|
||||||
|
if let Err(e) = IndexingService::stamp_reconciled(conn, &self.config, &roots) {
|
||||||
|
crate::log_warn!("coordinator: record reconciled configuration: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
if cursor.deleted > 0 || cursor.recontented > 0 {
|
if cursor.deleted > 0 || cursor.recontented > 0 {
|
||||||
crate::log_info!(
|
crate::log_info!(
|
||||||
"configuration change: {} index entries removed, {} re-examined \
|
"configuration change: {} index entries removed, {} re-examined \
|
||||||
|
|
@ -617,7 +769,7 @@ impl Inner {
|
||||||
/// create-then-delete ends with it absent, because the upsert half consults
|
/// create-then-delete ends with it absent, because the upsert half consults
|
||||||
/// the filesystem and finds nothing there.
|
/// the filesystem and finds nothing there.
|
||||||
fn apply_pending(&mut self) {
|
fn apply_pending(&mut self) {
|
||||||
let conn = match self.ensure_write_conn() {
|
let mut conn = match self.ensure_write_conn() {
|
||||||
Ok(conn) => conn,
|
Ok(conn) => conn,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
// Missing or stale DB: incremental can't help, rebuild.
|
// Missing or stale DB: incremental can't help, rebuild.
|
||||||
|
|
@ -629,8 +781,6 @@ impl Inner {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
// Borrow dance: pull the connection out while applying.
|
|
||||||
let mut conn = conn;
|
|
||||||
let deadline = Instant::now() + APPLY_BUDGET;
|
let deadline = Instant::now() + APPLY_BUDGET;
|
||||||
let chunk = self.config.processing.batch_size.max(1);
|
let chunk = self.config.processing.batch_size.max(1);
|
||||||
|
|
||||||
|
|
@ -642,7 +792,11 @@ impl Inner {
|
||||||
.collect();
|
.collect();
|
||||||
for batch in removals.chunks(chunk) {
|
for batch in removals.chunks(chunk) {
|
||||||
if let Err(e) = crate::incremental::remove_paths(&mut conn, batch, chunk) {
|
if let Err(e) = crate::incremental::remove_paths(&mut conn, batch, chunk) {
|
||||||
crate::log_warn!("coordinator: remove: {}", e);
|
// The batch leaves `pending` either way — replaying a write
|
||||||
|
// that just failed, once a tick forever, is the worse
|
||||||
|
// failure. A full run is what recovers the rows instead.
|
||||||
|
crate::log_warn!("coordinator: remove: {}; scheduling full run", e);
|
||||||
|
self.needs_full_run = true;
|
||||||
}
|
}
|
||||||
for path in batch {
|
for path in batch {
|
||||||
self.pending.remove(path);
|
self.pending.remove(path);
|
||||||
|
|
@ -666,7 +820,11 @@ impl Inner {
|
||||||
if let Err(e) =
|
if let Err(e) =
|
||||||
apply_fs_event(&mut conn, &ev, &self.config, &self.ignore, &self.registry)
|
apply_fs_event(&mut conn, &ev, &self.config, &self.ignore, &self.registry)
|
||||||
{
|
{
|
||||||
crate::log_warn!("coordinator: apply {:?}: {}", ev, e);
|
// Same reasoning as the removal half above: the event is
|
||||||
|
// already out of `pending`, so a full run is the only
|
||||||
|
// thing that still picks the file up.
|
||||||
|
crate::log_warn!("coordinator: apply {:?}: {}; scheduling full run", ev, e);
|
||||||
|
self.needs_full_run = true;
|
||||||
}
|
}
|
||||||
if Instant::now() >= deadline {
|
if Instant::now() >= deadline {
|
||||||
break;
|
break;
|
||||||
|
|
@ -918,23 +1076,51 @@ impl Inner {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Re-read the stamp the last completed full run left behind.
|
||||||
|
///
|
||||||
|
/// A failure to open is deliberately *not* published as `None`. Only a
|
||||||
|
/// successful read means "never indexed", and `periodic_due` answers that
|
||||||
|
/// by starting a full run immediately — so treating a locked or
|
||||||
|
/// contended database as "never" would schedule a fresh run every tick
|
||||||
|
/// for as long as the condition lasts. Keeping the previous value leaves
|
||||||
|
/// the schedule where it was until a read succeeds.
|
||||||
fn refresh_last_full_index(&self) {
|
fn refresh_last_full_index(&self) {
|
||||||
let last = db::open_existing(&self.db_path(), false)
|
match db::open_existing(&self.db_path(), false) {
|
||||||
.ok()
|
Ok(conn) => {
|
||||||
.and_then(|conn| db::repo::get_last_full_index(&conn));
|
let last = db::repo::get_last_full_index(&conn);
|
||||||
self.shared.lock().unwrap().last_full_index = last;
|
self.shared.lock().unwrap().last_full_index = last;
|
||||||
}
|
}
|
||||||
|
Err(e) => crate::log_warn!("coordinator: last-full-index unreadable: {}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn publish(&self) {
|
fn publish(&mut self) {
|
||||||
|
let reconcile = match &self.pending_work {
|
||||||
|
Some(cursor) => Some(ReconcileState::Running(cursor.progress())),
|
||||||
|
None => {
|
||||||
|
// The tail ages out here rather than in `tick`, which returns
|
||||||
|
// early for the whole length of a run; this runs every turn of
|
||||||
|
// the loop, so it expires within a second of its deadline
|
||||||
|
// whatever else is going on.
|
||||||
|
let now = Instant::now();
|
||||||
|
self.reconcile_done = self
|
||||||
|
.reconcile_done
|
||||||
|
.filter(|(_, at)| summary_is_fresh(*at, now));
|
||||||
|
self.reconcile_done
|
||||||
|
.map(|(progress, _)| ReconcileState::Finished(progress))
|
||||||
|
}
|
||||||
|
};
|
||||||
let mut shared = self.shared.lock().unwrap();
|
let mut shared = self.shared.lock().unwrap();
|
||||||
shared.mode = self.mode;
|
shared.mode = self.mode;
|
||||||
shared.queued_events = self.pending.len();
|
shared.queued_events = self.pending.len();
|
||||||
|
shared.reconcile = reconcile;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Must stay fast: it runs (transitively) on the GUI thread during
|
/// Must stay fast: it runs (transitively) on the GUI thread during
|
||||||
/// window close, and desktops show a "terminate this application?"
|
/// window close, and desktops show a "terminate this application?"
|
||||||
/// dialog after a few unresponsive seconds. Signal, don't wait — an
|
/// dialog after a few unresponsive seconds. Signal, don't wait — an
|
||||||
/// abandoned run is safe under WAL.
|
/// abandoned run is safe under WAL, and so is an abandoned reconcile:
|
||||||
|
/// nothing recorded it, so the next run does it again.
|
||||||
fn teardown(mut self) {
|
fn teardown(mut self) {
|
||||||
self.stop_watcher();
|
self.stop_watcher();
|
||||||
let status = self.indexing.get_status();
|
let status = self.indexing.get_status();
|
||||||
|
|
@ -946,10 +1132,18 @@ impl Inner {
|
||||||
// window during an optimize pass would wait out a rewrite of the
|
// window during an optimize pass would wait out a rewrite of the
|
||||||
// whole index. The interrupted VACUUM rolls back, and the next
|
// whole index. The interrupted VACUUM rolls back, and the next
|
||||||
// run's checkpoints land the log.
|
// run's checkpoints land the log.
|
||||||
self.indexing.cancel_optimizing();
|
self.indexing.cancel_db_work();
|
||||||
}
|
}
|
||||||
|
// Unfinished either way it can end: still holding its cursor, or
|
||||||
|
// abandoned mid-statement by the cancellation.
|
||||||
|
let cut_short = self.reconcile_cut_short || self.pending_work.is_some();
|
||||||
if let Some(conn) = self.write_conn.take() {
|
if let Some(conn) = self.write_conn.take() {
|
||||||
if idle {
|
// A reconciliation cut short is the one idle case that must not
|
||||||
|
// checkpoint. It can have written a great deal of WAL — deleting
|
||||||
|
// a root's rows is all log — and a TRUNCATE checkpoint of it is
|
||||||
|
// more of exactly the wait the cancellation just spared the user.
|
||||||
|
// Dropping is safe: WAL keeps the log and the next run lands it.
|
||||||
|
if idle && !cut_short {
|
||||||
db::repo::checkpoint_and_close(conn);
|
db::repo::checkpoint_and_close(conn);
|
||||||
}
|
}
|
||||||
// Otherwise just drop: a TRUNCATE checkpoint would block
|
// Otherwise just drop: a TRUNCATE checkpoint would block
|
||||||
|
|
@ -958,13 +1152,15 @@ impl Inner {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn now_unix() -> u64 {
|
/// Whether a reconciliation that finished at `at` is still worth reporting.
|
||||||
std::time::SystemTime::now()
|
///
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
/// Its own function so the rule can be tested without waiting the linger out.
|
||||||
.map(|d| d.as_secs())
|
fn summary_is_fresh(at: Instant, now: Instant) -> bool {
|
||||||
.unwrap_or(0)
|
now.duration_since(at) < RECONCILE_SUMMARY_LINGER
|
||||||
}
|
}
|
||||||
|
|
||||||
|
use crate::log::now_unix;
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
@ -988,17 +1184,10 @@ mod tests {
|
||||||
|
|
||||||
impl Fixture {
|
impl Fixture {
|
||||||
fn new(auto: bool) -> Fixture {
|
fn new(auto: bool) -> Fixture {
|
||||||
let stamp = format!(
|
let scratch = crate::testutil::scratch_dir("coord");
|
||||||
"{}-{}",
|
let dir = scratch.join("tree");
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
);
|
|
||||||
let dir = std::env::temp_dir().join(format!("qs-coord-{}", stamp));
|
|
||||||
std::fs::create_dir_all(&dir).unwrap();
|
std::fs::create_dir_all(&dir).unwrap();
|
||||||
let db = std::env::temp_dir().join(format!("qs-coord-{}.sqlite", stamp));
|
let db = scratch.join("index.sqlite");
|
||||||
let mut config = Config::default();
|
let mut config = Config::default();
|
||||||
config.paths.indexing_paths = vec![dir.to_string_lossy().into_owned()];
|
config.paths.indexing_paths = vec![dir.to_string_lossy().into_owned()];
|
||||||
config.paths.database_path = db.to_string_lossy().into_owned();
|
config.paths.database_path = db.to_string_lossy().into_owned();
|
||||||
|
|
@ -1014,6 +1203,21 @@ mod tests {
|
||||||
Err(_) => -1,
|
Err(_) => -1,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// What a run starting now would still find to reconcile.
|
||||||
|
fn outstanding_work(&self, config: &Config) -> IndexWork {
|
||||||
|
crate::scope::outstanding_work(&self.db.to_string_lossy(), config).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn stored_value(&self, key: &str) -> Option<String> {
|
||||||
|
let conn = db::open_existing(&self.db.to_string_lossy(), false).unwrap();
|
||||||
|
conn.query_row(
|
||||||
|
"SELECT value FROM config_validation WHERE key = ?1",
|
||||||
|
[key],
|
||||||
|
|r| r.get(0),
|
||||||
|
)
|
||||||
|
.ok()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Drop for Fixture {
|
impl Drop for Fixture {
|
||||||
|
|
@ -1042,6 +1246,51 @@ mod tests {
|
||||||
coord.shutdown();
|
coord.shutdown();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Closing the window during a prune must not wait the prune out. The
|
||||||
|
/// thread that reads the shutdown command is the thread inside the scan,
|
||||||
|
/// so before this the join sat behind however many rows were left — on a
|
||||||
|
/// large index, minutes of it, with the window still on screen and the
|
||||||
|
/// desktop offering to kill the app.
|
||||||
|
#[test]
|
||||||
|
fn shutdown_during_a_prune_does_not_wait_for_it() {
|
||||||
|
let f = Fixture::new(false);
|
||||||
|
// Enough rows that the scan cannot plausibly finish between the
|
||||||
|
// config landing and the shutdown two lines later.
|
||||||
|
for i in 0..400 {
|
||||||
|
std::fs::write(f.dir.join(format!("f{}.log", i)), "dropped content").unwrap();
|
||||||
|
}
|
||||||
|
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
|
||||||
|
|
||||||
|
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
|
||||||
|
coord.reindex_now();
|
||||||
|
wait_for("initial run", Duration::from_secs(60), || {
|
||||||
|
coord.state().last_full_index.is_some() && f.file_count() == 401
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut narrowed = f.config.clone();
|
||||||
|
narrowed.indexing.ignore_patterns.push("*.log".into());
|
||||||
|
// One row per page, so the scan is as many statements as there are
|
||||||
|
// rows: the shape a huge index has, at a size a test can afford.
|
||||||
|
narrowed.processing.batch_size = 1;
|
||||||
|
coord.apply_config(narrowed.clone());
|
||||||
|
|
||||||
|
let asked = Instant::now();
|
||||||
|
coord.shutdown();
|
||||||
|
let took = asked.elapsed();
|
||||||
|
assert!(
|
||||||
|
took < Duration::from_secs(5),
|
||||||
|
"shutdown waited {:?} for the prune",
|
||||||
|
took
|
||||||
|
);
|
||||||
|
// And it really did leave the work unfinished rather than racing
|
||||||
|
// through it: the stored record still describes the old settings, so
|
||||||
|
// the next run derives the same plan and applies it.
|
||||||
|
assert!(
|
||||||
|
f.outstanding_work(&narrowed).touches_index(),
|
||||||
|
"the prune ran to completion, so this proves nothing about waiting"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// A narrowed filter is applied to the stored index without a prompt and
|
/// A narrowed filter is applied to the stored index without a prompt and
|
||||||
/// without a run — including in manual mode, where the user has said not
|
/// without a run — including in manual mode, where the user has said not
|
||||||
/// to index anything. Deleting entries they just excluded is not indexing
|
/// to index anything. Deleting entries they just excluded is not indexing
|
||||||
|
|
@ -1085,6 +1334,144 @@ mod tests {
|
||||||
coord.shutdown();
|
coord.shutdown();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A prune that finishes must record what it reconciled against, or every
|
||||||
|
/// later run re-derives the same plan and rescans every row under every
|
||||||
|
/// root to redo work already done. On a multi-million-file index that
|
||||||
|
/// rescan is minutes of silence before the walk starts — the whole reason
|
||||||
|
/// indexing looked hung after a prune.
|
||||||
|
#[test]
|
||||||
|
fn a_completed_prune_records_what_it_reconciled() {
|
||||||
|
let f = Fixture::new(false);
|
||||||
|
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
|
||||||
|
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
|
||||||
|
|
||||||
|
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
|
||||||
|
coord.reindex_now();
|
||||||
|
wait_for("initial run", Duration::from_secs(20), || {
|
||||||
|
let s = coord.state();
|
||||||
|
s.last_full_index.is_some() && f.file_count() == 2
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut narrowed = f.config.clone();
|
||||||
|
narrowed.indexing.ignore_patterns.push("*.log".into());
|
||||||
|
assert!(
|
||||||
|
f.outstanding_work(&narrowed).touches_index(),
|
||||||
|
"the edit must be one the index does not yet reflect"
|
||||||
|
);
|
||||||
|
|
||||||
|
coord.apply_config(narrowed.clone());
|
||||||
|
wait_for(
|
||||||
|
"the log entry to be pruned",
|
||||||
|
Duration::from_secs(20),
|
||||||
|
|| f.file_count() == 1,
|
||||||
|
);
|
||||||
|
// The prune and the stamp are two steps of one tick; the count above
|
||||||
|
// can be observed between them.
|
||||||
|
wait_for("the prune to be recorded", Duration::from_secs(20), || {
|
||||||
|
!f.outstanding_work(&narrowed).touches_index()
|
||||||
|
});
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
f.outstanding_work(&narrowed).is_empty(),
|
||||||
|
"a run starting now has nothing left to reconcile"
|
||||||
|
);
|
||||||
|
coord.shutdown();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A prune of a two-file index is one transaction, over well inside a
|
||||||
|
/// frame. Reporting it only while it runs would mean the user changes a
|
||||||
|
/// setting and sees nothing at all — so the result outlives the work.
|
||||||
|
#[test]
|
||||||
|
fn a_finished_prune_keeps_reporting_itself_for_a_while() {
|
||||||
|
let f = Fixture::new(false);
|
||||||
|
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
|
||||||
|
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
|
||||||
|
|
||||||
|
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
|
||||||
|
coord.reindex_now();
|
||||||
|
wait_for("initial run", Duration::from_secs(20), || {
|
||||||
|
let s = coord.state();
|
||||||
|
s.last_full_index.is_some() && f.file_count() == 2
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut narrowed = f.config.clone();
|
||||||
|
narrowed.indexing.ignore_patterns.push("*.log".into());
|
||||||
|
coord.apply_config(narrowed);
|
||||||
|
wait_for("the summary to appear", Duration::from_secs(20), || {
|
||||||
|
matches!(coord.state().reconcile, Some(ReconcileState::Finished(_)))
|
||||||
|
});
|
||||||
|
|
||||||
|
let Some(ReconcileState::Finished(progress)) = coord.state().reconcile else {
|
||||||
|
panic!("the summary went away as soon as it arrived");
|
||||||
|
};
|
||||||
|
assert_eq!(progress.deleted, 1, "the log entry, and only it");
|
||||||
|
coord.shutdown();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// And it does go away: a summary is a report of what just happened, not
|
||||||
|
/// a state the app sits in. The rule is tested directly rather than by
|
||||||
|
/// sleeping out the linger.
|
||||||
|
#[test]
|
||||||
|
fn a_summary_stops_being_fresh_once_the_linger_is_up() {
|
||||||
|
let now = Instant::now();
|
||||||
|
assert!(summary_is_fresh(now, now));
|
||||||
|
assert!(summary_is_fresh(
|
||||||
|
now,
|
||||||
|
now + RECONCILE_SUMMARY_LINGER - Duration::from_millis(1)
|
||||||
|
));
|
||||||
|
assert!(!summary_is_fresh(now, now + RECONCILE_SUMMARY_LINGER));
|
||||||
|
assert!(!summary_is_fresh(now, now + RECONCILE_SUMMARY_LINGER * 60));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The three settings no scan can satisfy stay at the values the index was
|
||||||
|
/// *built* with. A prune that stamped them would clear a rebuild the user
|
||||||
|
/// was prompted for and declined — and would clear it from an unrelated
|
||||||
|
/// later edit at that.
|
||||||
|
#[test]
|
||||||
|
fn a_prune_never_records_settings_only_a_rebuild_can_satisfy() {
|
||||||
|
let f = Fixture::new(false);
|
||||||
|
std::fs::write(f.dir.join("keep.txt"), "kept content").unwrap();
|
||||||
|
std::fs::write(f.dir.join("drop.log"), "dropped content").unwrap();
|
||||||
|
|
||||||
|
let coord = IndexCoordinator::start(f.config.clone()).unwrap();
|
||||||
|
coord.reindex_now();
|
||||||
|
wait_for("initial run", Duration::from_secs(20), || {
|
||||||
|
let s = coord.state();
|
||||||
|
s.last_full_index.is_some() && f.file_count() == 2
|
||||||
|
});
|
||||||
|
let built_with = f.stored_value("hash_length").unwrap();
|
||||||
|
|
||||||
|
// The user changes the hash length and declines the rebuild it needs.
|
||||||
|
// The coordinator takes the config either way — a wipe is the caller's
|
||||||
|
// decision — so from here its copy disagrees with the stored hashes,
|
||||||
|
// and nothing short of a rebuild can make them agree.
|
||||||
|
let mut rebuilt = f.config.clone();
|
||||||
|
rebuilt.processing.hash_length = f.config.processing.hash_length * 2;
|
||||||
|
coord.apply_config(rebuilt.clone());
|
||||||
|
std::thread::sleep(Duration::from_millis(500));
|
||||||
|
|
||||||
|
// Then they edit a filter. This one *is* reconcilable, and the pass
|
||||||
|
// that applies it stamps — with the coordinator's config, whose hash
|
||||||
|
// length is the one the index does not have.
|
||||||
|
let mut narrowed = rebuilt.clone();
|
||||||
|
narrowed.indexing.ignore_patterns.push("*.log".into());
|
||||||
|
coord.apply_config(narrowed);
|
||||||
|
wait_for(
|
||||||
|
"the log entry to be pruned",
|
||||||
|
Duration::from_secs(20),
|
||||||
|
|| f.file_count() == 1,
|
||||||
|
);
|
||||||
|
std::thread::sleep(Duration::from_millis(500));
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
f.stored_value("hash_length"),
|
||||||
|
Some(built_with),
|
||||||
|
"the recorded hash length still describes the stored hashes, so the \
|
||||||
|
rebuild prompt survives an unrelated prune"
|
||||||
|
);
|
||||||
|
coord.shutdown();
|
||||||
|
}
|
||||||
|
|
||||||
/// Widening it does the opposite: nothing is deleted, and the walk that
|
/// Widening it does the opposite: nothing is deleted, and the walk that
|
||||||
/// finds the newly-eligible files starts on its own, returning manual mode
|
/// finds the newly-eligible files starts on its own, returning manual mode
|
||||||
/// to stopped afterwards the way `reindex_now` does.
|
/// to stopped afterwards the way `reindex_now` does.
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,10 @@
|
||||||
//! size, `clear`) instead use [`open::open_existing`], which never creates or
|
//! size, `clear`) instead use [`open::open_existing`], which never creates or
|
||||||
//! wipes — a tokenizer difference or stale version is an error, not data loss.
|
//! wipes — a tokenizer difference or stale version is an error, not data loss.
|
||||||
|
|
||||||
|
use std::sync::Mutex;
|
||||||
|
|
||||||
|
use rusqlite::{Connection, InterruptHandle};
|
||||||
|
|
||||||
pub mod key;
|
pub mod key;
|
||||||
pub mod open;
|
pub mod open;
|
||||||
pub mod repo;
|
pub mod repo;
|
||||||
|
|
@ -18,3 +22,65 @@ pub use open::{
|
||||||
index_needs_rebuild, open_existing, open_or_recreate, verify_process_key,
|
index_needs_rebuild, open_existing, open_or_recreate, verify_process_key,
|
||||||
CURRENT_SCHEMA_VERSION, KEY_MISMATCH_PREFIX,
|
CURRENT_SCHEMA_VERSION, KEY_MISMATCH_PREFIX,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/// A shared slot holding the interrupt handle of whatever long statement is
|
||||||
|
/// running, so another thread can cut it short.
|
||||||
|
///
|
||||||
|
/// # Why a flag is not enough
|
||||||
|
///
|
||||||
|
/// SQLite's own interrupt is the only way out of a statement already in
|
||||||
|
/// flight: a `DELETE` over a whole root's range, an FTS merge or a VACUUM
|
||||||
|
/// answer to nothing else, and on a large index each can run for minutes. A
|
||||||
|
/// flag can only stop the *next* statement from starting.
|
||||||
|
///
|
||||||
|
/// So cancelling anything long is two halves — the flag, which prevents the
|
||||||
|
/// next statement, and this, which ends the current one — and a caller that
|
||||||
|
/// wants a bounded wait needs both. Every user of this type is one of those
|
||||||
|
/// pairs: [`crate::scope::advance`] and its `cancel` argument,
|
||||||
|
/// `coordinator::ReconcileStop`, and the stop flag alongside
|
||||||
|
/// `IndexingService::cancel_db_work`. This is the reference statement of the
|
||||||
|
/// rule; those sites record only what it means locally.
|
||||||
|
///
|
||||||
|
/// One further consequence, which is why [`InterruptGuard`] exists rather than
|
||||||
|
/// a set/clear pair: an interrupted statement fails like any other and cannot
|
||||||
|
/// be told apart from a real failure by its error message. The flag the
|
||||||
|
/// canceller set is the only reliable answer, so callers re-read it on the
|
||||||
|
/// error path before deciding whether they were cancelled or broke.
|
||||||
|
pub type InterruptSlot = Mutex<Option<InterruptHandle>>;
|
||||||
|
|
||||||
|
/// Publish `conn`'s interrupt handle in `slot` for as long as this lives.
|
||||||
|
///
|
||||||
|
/// The guard, rather than a set/clear pair, because every path out matters:
|
||||||
|
/// a handle left behind after an early `?` would let a later interrupt —
|
||||||
|
/// aimed at the next long statement, or at a VACUUM — land on whatever that
|
||||||
|
/// connection happens to be running by then.
|
||||||
|
pub struct InterruptGuard<'a> {
|
||||||
|
slot: &'a InterruptSlot,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<'a> InterruptGuard<'a> {
|
||||||
|
pub fn arm(slot: &'a InterruptSlot, conn: &Connection) -> InterruptGuard<'a> {
|
||||||
|
if let Ok(mut held) = slot.lock() {
|
||||||
|
*held = Some(conn.get_interrupt_handle());
|
||||||
|
}
|
||||||
|
InterruptGuard { slot }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for InterruptGuard<'_> {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
if let Ok(mut held) = self.slot.lock() {
|
||||||
|
*held = None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Interrupt the statement `slot` holds a handle for, if there is one.
|
||||||
|
/// A no-op otherwise, and on a connection that has since closed.
|
||||||
|
pub fn interrupt(slot: &InterruptSlot) {
|
||||||
|
if let Ok(held) = slot.lock() {
|
||||||
|
if let Some(handle) = held.as_ref() {
|
||||||
|
handle.interrupt();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,11 +7,11 @@
|
||||||
//! migrations.
|
//! migrations.
|
||||||
//!
|
//!
|
||||||
//! The tradeoff: users pay a re-index cost every time the shipped schema
|
//! The tradeoff: users pay a re-index cost every time the shipped schema
|
||||||
//! changes. Our indexing is fast (see `bench/`) and schema changes are
|
//! changes. Indexing is fast and schema changes are rare in practice, so the
|
||||||
//! rare in practice, so the code-complexity cost of maintaining real
|
//! code-complexity cost of maintaining real migration paths wasn't worth it.
|
||||||
//! migration paths wasn't worth it. A single `open_or_recreate` replaces
|
//! A single `open_or_recreate` replaces what used to be version detection +
|
||||||
//! what used to be version detection + tokenizer-drift FTS rebuild +
|
//! tokenizer-drift FTS rebuild + legacy-layout recovery, all of which
|
||||||
//! legacy-layout recovery, all of which ultimately wiped anyway.
|
//! ultimately wiped anyway.
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
|
|
@ -414,10 +414,7 @@ fn apply_current_schema(conn: &Connection, tokenizer: &str) -> Result<(), String
|
||||||
conn.execute_batch(&fts)
|
conn.execute_batch(&fts)
|
||||||
.map_err(|e| format!("Failed to create searchabletext: {}", e))?;
|
.map_err(|e| format!("Failed to create searchabletext: {}", e))?;
|
||||||
|
|
||||||
let now = std::time::SystemTime::now()
|
let now = crate::log::now_unix();
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_secs())
|
|
||||||
.unwrap_or(0);
|
|
||||||
let effective = effective_tokenizer(tokenizer);
|
let effective = effective_tokenizer(tokenizer);
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO schema_info(key, value) VALUES ('version', ?1), ('created_at', ?2), ('tokenize', ?3)",
|
"INSERT INTO schema_info(key, value) VALUES ('version', ?1), ('created_at', ?2), ('tokenize', ?3)",
|
||||||
|
|
@ -437,16 +434,7 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn tmp_db_path() -> std::path::PathBuf {
|
fn tmp_db_path() -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir("open").join("index.sqlite")
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-test-{}-{}.sqlite",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -548,7 +536,7 @@ mod tests {
|
||||||
assert_eq!(count, 0);
|
assert_eq!(count, 0);
|
||||||
// New columns should exist (just prepare the SELECT — an
|
// New columns should exist (just prepare the SELECT — an
|
||||||
// unknown column name would parse-error here).
|
// unknown column name would parse-error here).
|
||||||
let _ = conn
|
conn
|
||||||
.query_row(
|
.query_row(
|
||||||
"SELECT basic_state, content_state, type, mime FROM files LIMIT 0",
|
"SELECT basic_state, content_state, type, mime FROM files LIMIT 0",
|
||||||
[],
|
[],
|
||||||
|
|
@ -667,15 +655,7 @@ mod tests {
|
||||||
fn open_or_recreate_creates_missing_parent_dirs() {
|
fn open_or_recreate_creates_missing_parent_dirs() {
|
||||||
// Fresh installs point at ~/.local/share/quicksearch/… which
|
// Fresh installs point at ~/.local/share/quicksearch/… which
|
||||||
// doesn't exist yet; the owner open must create it.
|
// doesn't exist yet; the owner open must create it.
|
||||||
let mut dir = std::env::temp_dir();
|
let dir = crate::testutil::scratch_dir("mkdir");
|
||||||
dir.push(format!(
|
|
||||||
"qs-mkdir-{}-{}",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
let db = dir.join("nested/deeper/index.sqlite");
|
let db = dir.join("nested/deeper/index.sqlite");
|
||||||
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
|
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
|
||||||
drop(conn);
|
drop(conn);
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,57 @@ pub const STATE_DONE: i64 = 1;
|
||||||
pub const STATE_FAILED: i64 = 2;
|
pub const STATE_FAILED: i64 = 2;
|
||||||
pub const STATE_NA: i64 = 3;
|
pub const STATE_NA: i64 = 3;
|
||||||
|
|
||||||
|
/// `prepare_cached` + `execute`, returning the affected row count.
|
||||||
|
///
|
||||||
|
/// `what` is a closure rather than a string so the message is built only on
|
||||||
|
/// the error path — several callers run this per property or per file, where
|
||||||
|
/// formatting a description that is then discarded is the dominant cost.
|
||||||
|
///
|
||||||
|
/// Takes `&Connection`; a `Transaction` derefs to one, so the same call works
|
||||||
|
/// inside and outside a transaction.
|
||||||
|
fn exec(
|
||||||
|
conn: &Connection,
|
||||||
|
sql: &str,
|
||||||
|
params: impl rusqlite::Params,
|
||||||
|
what: impl FnOnce() -> String,
|
||||||
|
) -> Result<usize, String> {
|
||||||
|
conn.prepare_cached(sql)
|
||||||
|
.and_then(|mut stmt| stmt.execute(params))
|
||||||
|
.map_err(|e| format!("{}: {}", what(), e))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Move a file to a content state that means "nothing is currently wrong with
|
||||||
|
/// this row", clearing any failure record along with it.
|
||||||
|
///
|
||||||
|
/// The pair is one operation, not two that happen to be adjacent: a row that
|
||||||
|
/// succeeded, that nothing can extract, or that has been queued for another
|
||||||
|
/// attempt must not keep a `failed_files` entry explaining a failure that no
|
||||||
|
/// longer applies. `list-failed` reads that table directly, so a stale entry
|
||||||
|
/// is a file the user is told is broken after it stopped being broken.
|
||||||
|
///
|
||||||
|
/// [`set_content_failed`] is deliberately not routed through here — it is the
|
||||||
|
/// one transition that *writes* a failure record.
|
||||||
|
fn set_state_clearing_failure(
|
||||||
|
tx: &Transaction<'_>,
|
||||||
|
file_id: i64,
|
||||||
|
state: i64,
|
||||||
|
transition: &'static str,
|
||||||
|
) -> Result<(), String> {
|
||||||
|
exec(
|
||||||
|
tx,
|
||||||
|
"UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2",
|
||||||
|
params![state, file_id],
|
||||||
|
|| format!("{} content_state {}", transition, file_id),
|
||||||
|
)?;
|
||||||
|
exec(
|
||||||
|
tx,
|
||||||
|
"DELETE FROM failed_files WHERE file_id = ?1",
|
||||||
|
params![file_id],
|
||||||
|
|| format!("clear failed_files {}", file_id),
|
||||||
|
)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// Everything needed to insert a fresh file row.
|
/// Everything needed to insert a fresh file row.
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct NewFile<'a> {
|
pub struct NewFile<'a> {
|
||||||
|
|
@ -162,18 +213,22 @@ pub fn set_content_done(
|
||||||
remove_content_for_id(tx, file_id)?;
|
remove_content_for_id(tx, file_id)?;
|
||||||
|
|
||||||
for (k, v) in properties {
|
for (k, v) in properties {
|
||||||
tx.prepare_cached("INSERT INTO properties(file_id, key, value) VALUES (?1, ?2, ?3)")
|
exec(
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id, k, v]))
|
tx,
|
||||||
.map_err(|e| format!("insert property {}={}: {}", k, v, e))?;
|
"INSERT INTO properties(file_id, key, value) VALUES (?1, ?2, ?3)",
|
||||||
|
params![file_id, k, v],
|
||||||
|
|| format!("insert property {}={}", k, v),
|
||||||
|
)?;
|
||||||
}
|
}
|
||||||
let props_blob = encode_properties_for_fts(properties);
|
let props_blob = encode_properties_for_fts(properties);
|
||||||
// Contentless FTS5 still accepts values on INSERT — the tokenizer needs
|
// Contentless FTS5 still accepts values on INSERT — the tokenizer needs
|
||||||
// them — it simply doesn't persist the raw column values.
|
// them — it simply doesn't persist the raw column values.
|
||||||
tx.prepare_cached(
|
exec(
|
||||||
|
tx,
|
||||||
"INSERT INTO searchabletext(rowid, name, text, properties) VALUES (?1, ?2, ?3, ?4)",
|
"INSERT INTO searchabletext(rowid, name, text, properties) VALUES (?1, ?2, ?3, ?4)",
|
||||||
)
|
params![file_id, name, text, props_blob],
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id, name, text, props_blob]))
|
|| format!("insert FTS row {}", file_id),
|
||||||
.map_err(|e| format!("insert FTS row {}: {}", file_id, e))?;
|
)?;
|
||||||
|
|
||||||
// Skip the compressed sidecar when: the config disables snippet storage
|
// Skip the compressed sidecar when: the config disables snippet storage
|
||||||
// outright, or there's no body text (e.g. an image whose extractor
|
// outright, or there's no body text (e.g. an image whose extractor
|
||||||
|
|
@ -182,21 +237,15 @@ pub fn set_content_done(
|
||||||
if store_text && !text.is_empty() {
|
if store_text && !text.is_empty() {
|
||||||
let compressed = zstd::encode_all(text.as_bytes(), ZSTD_LEVEL)
|
let compressed = zstd::encode_all(text.as_bytes(), ZSTD_LEVEL)
|
||||||
.map_err(|e| format!("zstd encode for file {}: {}", file_id, e))?;
|
.map_err(|e| format!("zstd encode for file {}: {}", file_id, e))?;
|
||||||
tx.prepare_cached(
|
exec(
|
||||||
|
tx,
|
||||||
"INSERT INTO documents_text(file_id, text_zstd, text_len) VALUES (?1, ?2, ?3)",
|
"INSERT INTO documents_text(file_id, text_zstd, text_len) VALUES (?1, ?2, ?3)",
|
||||||
)
|
params![file_id, compressed, text.len() as i64],
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id, compressed, text.len() as i64]))
|
|| format!("insert documents_text {}", file_id),
|
||||||
.map_err(|e| format!("insert documents_text {}: {}", file_id, e))?;
|
)?;
|
||||||
}
|
}
|
||||||
|
|
||||||
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
|
set_state_clearing_failure(tx, file_id, STATE_DONE, "update DONE")
|
||||||
.and_then(|mut stmt| stmt.execute(params![STATE_DONE, file_id]))
|
|
||||||
.map_err(|e| format!("update content_state DONE {}: {}", file_id, e))?;
|
|
||||||
// Clear any prior failed-file record.
|
|
||||||
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
|
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id]))
|
|
||||||
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
|
|
||||||
Ok(())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// zstd level tuned for extracted-text prose. Level 3 hits ~3-5× on English
|
/// zstd level tuned for extracted-text prose. Level 3 hits ~3-5× on English
|
||||||
|
|
@ -208,31 +257,26 @@ const ZSTD_LEVEL: i32 = 3;
|
||||||
|
|
||||||
/// Mark a file's content extraction as failed. Keeps the basic row in place.
|
/// Mark a file's content extraction as failed. Keeps the basic row in place.
|
||||||
pub fn set_content_failed(tx: &Transaction<'_>, file_id: i64, reason: &str) -> Result<(), String> {
|
pub fn set_content_failed(tx: &Transaction<'_>, file_id: i64, reason: &str) -> Result<(), String> {
|
||||||
let now = std::time::SystemTime::now()
|
let now = crate::log::now_unix() as i64;
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
exec(
|
||||||
.map(|d| d.as_secs())
|
tx,
|
||||||
.unwrap_or(0) as i64;
|
"UPDATE files SET content_state = ?1, failure_msg = ?2 WHERE id = ?3",
|
||||||
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = ?2 WHERE id = ?3")
|
params![STATE_FAILED, reason, file_id],
|
||||||
.and_then(|mut stmt| stmt.execute(params![STATE_FAILED, reason, file_id]))
|
|| format!("update content_state FAILED {}", file_id),
|
||||||
.map_err(|e| format!("update content_state FAILED {}: {}", file_id, e))?;
|
)?;
|
||||||
tx.prepare_cached(
|
exec(
|
||||||
|
tx,
|
||||||
"INSERT OR REPLACE INTO failed_files(file_id, reason, ts) VALUES (?1, ?2, ?3)",
|
"INSERT OR REPLACE INTO failed_files(file_id, reason, ts) VALUES (?1, ?2, ?3)",
|
||||||
)
|
params![file_id, reason, now],
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id, reason, now]))
|
|| format!("insert failed_files {}", file_id),
|
||||||
.map_err(|e| format!("insert failed_files {}: {}", file_id, e))?;
|
)?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Mark content extraction as not applicable (e.g. binary format we don't
|
/// Mark content extraction as not applicable (e.g. binary format we don't
|
||||||
/// support). The file row still contributes to filename search.
|
/// support). The file row still contributes to filename search.
|
||||||
pub fn set_content_na(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
|
pub fn set_content_na(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
|
||||||
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
|
set_state_clearing_failure(tx, file_id, STATE_NA, "update NA")
|
||||||
.and_then(|mut stmt| stmt.execute(params![STATE_NA, file_id]))
|
|
||||||
.map_err(|e| format!("update content_state NA {}: {}", file_id, e))?;
|
|
||||||
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
|
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id]))
|
|
||||||
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
|
|
||||||
Ok(())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Delete a file row by path, keeping FTS in sync. Returns whether a row was
|
/// Delete a file row by path, keeping FTS in sync. Returns whether a row was
|
||||||
|
|
@ -276,15 +320,16 @@ pub fn delete_subtree(tx: &Transaction<'_>, lo: &str, hi: &str) -> Result<usize,
|
||||||
(SELECT id FROM files WHERE path >= ?1 AND path < ?2)",
|
(SELECT id FROM files WHERE path >= ?1 AND path < ?2)",
|
||||||
table, key
|
table, key
|
||||||
);
|
);
|
||||||
tx.prepare_cached(&sql)
|
exec(tx, &sql, params![lo, hi], || {
|
||||||
.and_then(|mut stmt| stmt.execute(params![lo, hi]))
|
format!("delete {} under {}", table, lo)
|
||||||
.map_err(|e| format!("delete {} under {}: {}", table, lo, e))?;
|
})?;
|
||||||
}
|
}
|
||||||
let removed = tx
|
exec(
|
||||||
.prepare_cached("DELETE FROM files WHERE path >= ?1 AND path < ?2")
|
tx,
|
||||||
.and_then(|mut stmt| stmt.execute(params![lo, hi]))
|
"DELETE FROM files WHERE path >= ?1 AND path < ?2",
|
||||||
.map_err(|e| format!("delete files under {}: {}", lo, e))?;
|
params![lo, hi],
|
||||||
Ok(removed)
|
|| format!("delete files under {}", lo),
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Delete every row whose path falls in *none* of `ranges`, keeping the four
|
/// Delete every row whose path falls in *none* of `ranges`, keeping the four
|
||||||
|
|
@ -326,14 +371,14 @@ pub fn delete_outside_ranges(
|
||||||
"DELETE FROM {} WHERE {} IN (SELECT id FROM files WHERE {})",
|
"DELETE FROM {} WHERE {} IN (SELECT id FROM files WHERE {})",
|
||||||
table, key, predicate
|
table, key, predicate
|
||||||
);
|
);
|
||||||
tx.prepare_cached(&sql)
|
exec(tx, &sql, params_from_iter(bounds.iter()), || {
|
||||||
.and_then(|mut stmt| stmt.execute(params_from_iter(bounds.iter())))
|
format!("delete {} outside the roots", table)
|
||||||
.map_err(|e| format!("delete {} outside the roots: {}", table, e))?;
|
})?;
|
||||||
}
|
}
|
||||||
let sql = format!("DELETE FROM files WHERE {}", predicate);
|
let sql = format!("DELETE FROM files WHERE {}", predicate);
|
||||||
tx.prepare_cached(&sql)
|
exec(tx, &sql, params_from_iter(bounds.iter()), || {
|
||||||
.and_then(|mut stmt| stmt.execute(params_from_iter(bounds.iter())))
|
"delete files outside the roots".to_string()
|
||||||
.map_err(|e| format!("delete files outside the roots: {}", e))
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The tables a file id owns, in the order they must be cleared: everything
|
/// The tables a file id owns, in the order they must be cleared: everything
|
||||||
|
|
@ -381,15 +426,14 @@ pub fn delete_ids(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, String> {
|
||||||
}
|
}
|
||||||
for (table, key) in DEPENDENT_TABLES {
|
for (table, key) in DEPENDENT_TABLES {
|
||||||
let sql = format!("DELETE FROM {} WHERE {} IN ({})", table, key, placeholders);
|
let sql = format!("DELETE FROM {} WHERE {} IN ({})", table, key, placeholders);
|
||||||
tx.prepare_cached(&sql)
|
exec(tx, &sql, params_from_iter(chunk.iter()), || {
|
||||||
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
|
format!("delete {} for {} ids", table, chunk.len())
|
||||||
.map_err(|e| format!("delete {} for {} ids: {}", table, chunk.len(), e))?;
|
})?;
|
||||||
}
|
}
|
||||||
let sql = format!("DELETE FROM files WHERE id IN ({})", placeholders);
|
let sql = format!("DELETE FROM files WHERE id IN ({})", placeholders);
|
||||||
removed += tx
|
removed += exec(tx, &sql, params_from_iter(chunk.iter()), || {
|
||||||
.prepare_cached(&sql)
|
format!("delete {} file rows", chunk.len())
|
||||||
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
|
})?;
|
||||||
.map_err(|e| format!("delete {} file rows: {}", chunk.len(), e))?;
|
|
||||||
}
|
}
|
||||||
Ok(removed)
|
Ok(removed)
|
||||||
}
|
}
|
||||||
|
|
@ -420,8 +464,11 @@ pub fn dir_rows(
|
||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A row the content pass has yet to extract: `(id, name, path, mime)`.
|
||||||
|
pub type PendingContentRow = (i64, String, String, Option<String>);
|
||||||
|
|
||||||
/// One page of rows still awaiting content extraction under `cursor`'s range,
|
/// One page of rows still awaiting content extraction under `cursor`'s range,
|
||||||
/// as `(id, name, path, mime)` ordered by id.
|
/// ordered by id.
|
||||||
///
|
///
|
||||||
/// Keyset, not `OFFSET`: `id > cursor.last_id` means each page is an index
|
/// Keyset, not `OFFSET`: `id > cursor.last_id` means each page is an index
|
||||||
/// seek rather than a re-scan of everything already handed out, and — because
|
/// seek rather than a re-scan of everything already handed out, and — because
|
||||||
|
|
@ -434,7 +481,7 @@ pub fn pending_content_page(
|
||||||
cursor: &crate::file_handling::ExtractCursor,
|
cursor: &crate::file_handling::ExtractCursor,
|
||||||
max_size: i64,
|
max_size: i64,
|
||||||
limit: i64,
|
limit: i64,
|
||||||
) -> Result<Vec<(i64, String, String, Option<String>)>, String> {
|
) -> Result<Vec<PendingContentRow>, String> {
|
||||||
let mut stmt = conn
|
let mut stmt = conn
|
||||||
.prepare_cached(
|
.prepare_cached(
|
||||||
"SELECT id, name, path, mime FROM files
|
"SELECT id, name, path, mime FROM files
|
||||||
|
|
@ -472,6 +519,17 @@ pub struct ScopeRow {
|
||||||
pub content_state: i64,
|
pub content_state: i64,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// How many files the index holds.
|
||||||
|
///
|
||||||
|
/// A denominator for a scan that pages over all of them, so the cost is paid
|
||||||
|
/// once against work measured in pages. SQLite answers it from the smallest
|
||||||
|
/// index rather than the table, so it is a key scan and not a row fetch.
|
||||||
|
pub fn row_count(conn: &Connection) -> Result<usize, String> {
|
||||||
|
conn.query_row("SELECT COUNT(*) FROM files", [], |r| r.get::<_, i64>(0))
|
||||||
|
.map(|n| n.max(0) as usize)
|
||||||
|
.map_err(|e| format!("count indexed files: {}", e))
|
||||||
|
}
|
||||||
|
|
||||||
/// One page of rows whose path is `> after` and `< hi`, in path order.
|
/// One page of rows whose path is `> after` and `< hi`, in path order.
|
||||||
///
|
///
|
||||||
/// Keyset on `path` rather than on `id`: the range is already a seek on
|
/// Keyset on `path` rather than on `id`: the range is already a seek on
|
||||||
|
|
@ -529,10 +587,9 @@ pub fn drop_stored_text(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, Stri
|
||||||
"DELETE FROM documents_text WHERE file_id IN ({})",
|
"DELETE FROM documents_text WHERE file_id IN ({})",
|
||||||
placeholders
|
placeholders
|
||||||
);
|
);
|
||||||
removed += tx
|
removed += exec(tx, &sql, params_from_iter(chunk.iter()), || {
|
||||||
.prepare_cached(&sql)
|
format!("drop stored text for {} ids", chunk.len())
|
||||||
.and_then(|mut stmt| stmt.execute(params_from_iter(chunk.iter())))
|
})?;
|
||||||
.map_err(|e| format!("drop stored text for {} ids: {}", chunk.len(), e))?;
|
|
||||||
}
|
}
|
||||||
Ok(removed)
|
Ok(removed)
|
||||||
}
|
}
|
||||||
|
|
@ -545,13 +602,7 @@ pub fn drop_stored_text(tx: &Transaction<'_>, ids: &[i64]) -> Result<usize, Stri
|
||||||
/// and hash from a fresh stat), but its content must be produced again.
|
/// and hash from a fresh stat), but its content must be produced again.
|
||||||
pub fn reset_content_pending(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
|
pub fn reset_content_pending(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
|
||||||
remove_content_for_id(tx, file_id)?;
|
remove_content_for_id(tx, file_id)?;
|
||||||
tx.prepare_cached("UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2")
|
set_state_clearing_failure(tx, file_id, STATE_PENDING, "reset pending")
|
||||||
.and_then(|mut stmt| stmt.execute(params![STATE_PENDING, file_id]))
|
|
||||||
.map_err(|e| format!("reset content_state pending {}: {}", file_id, e))?;
|
|
||||||
tx.prepare_cached("DELETE FROM failed_files WHERE file_id = ?1")
|
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id]))
|
|
||||||
.map_err(|e| format!("clear failed_files {}: {}", file_id, e))?;
|
|
||||||
Ok(())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The stored mtime for one exact path, or `None` if it isn't indexed.
|
/// The stored mtime for one exact path, or `None` if it isn't indexed.
|
||||||
|
|
@ -621,9 +672,9 @@ pub fn remove_content_for_id(tx: &Transaction<'_>, file_id: i64) -> Result<(), S
|
||||||
("properties", "file_id"),
|
("properties", "file_id"),
|
||||||
] {
|
] {
|
||||||
let sql = format!("DELETE FROM {} WHERE {} = ?1", table, key);
|
let sql = format!("DELETE FROM {} WHERE {} = ?1", table, key);
|
||||||
tx.prepare_cached(&sql)
|
exec(tx, &sql, params![file_id], || {
|
||||||
.and_then(|mut stmt| stmt.execute(params![file_id]))
|
format!("delete {} for {}", table, file_id)
|
||||||
.map_err(|e| format!("delete {} for {}: {}", table, file_id, e))?;
|
})?;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
@ -779,6 +830,70 @@ pub fn set_last_full_index(conn: &Connection, ts: u64) -> Result<(), String> {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `schema_info` key holding one root's last known file count.
|
||||||
|
fn walk_count_key(root: &str) -> String {
|
||||||
|
format!("walk_count:{}", root)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How many files the last clean walk of `root` reported.
|
||||||
|
///
|
||||||
|
/// The progress bar's denominator. Absent means this root has never been walked
|
||||||
|
/// to completion — a new root, a fresh index, or a run that was stopped — and
|
||||||
|
/// the caller falls back to counting the tree as it goes.
|
||||||
|
///
|
||||||
|
/// Deliberately last run's *file* count rather than a tree-entry count: it is
|
||||||
|
/// the same quantity the numerator counts, where an entry count includes
|
||||||
|
/// directories and ignore-pruned subtrees and so reads high (over 1.6x on a home
|
||||||
|
/// directory). See [`crate::indexing::RootProgress::walk_denominator`].
|
||||||
|
pub fn get_root_walk_count(conn: &Connection, root: &str) -> Option<usize> {
|
||||||
|
conn.query_row(
|
||||||
|
"SELECT value FROM schema_info WHERE key = ?1",
|
||||||
|
params![walk_count_key(root)],
|
||||||
|
|r| r.get::<_, String>(0),
|
||||||
|
)
|
||||||
|
.optional()
|
||||||
|
.ok()
|
||||||
|
.flatten()
|
||||||
|
.and_then(|v| v.parse().ok())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Record `n` as `root`'s file count, for the next run's progress bar.
|
||||||
|
///
|
||||||
|
/// Written only after a walk that finished cleanly: a stopped or partially
|
||||||
|
/// unreadable walk saw only part of the tree, and storing its count would leave
|
||||||
|
/// every later run dividing by a number that is too small.
|
||||||
|
pub fn set_root_walk_count(conn: &Connection, root: &str, n: usize) -> Result<(), String> {
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR REPLACE INTO schema_info(key, value) VALUES (?1, ?2)",
|
||||||
|
params![walk_count_key(root), n.to_string()],
|
||||||
|
)
|
||||||
|
.map_err(|e| format!("write walk count for {}: {}", root, e))?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Forget the stored counts of roots that are no longer configured.
|
||||||
|
///
|
||||||
|
/// Without this, `schema_info` accumulates a row per root the user ever had —
|
||||||
|
/// harmless in size, but it also means a root removed and later re-added would
|
||||||
|
/// start from a count that predates everything that happened in between.
|
||||||
|
pub fn prune_root_walk_counts(conn: &Connection, keep: &[String]) -> Result<(), String> {
|
||||||
|
let keep: std::collections::HashSet<String> = keep.iter().map(|r| walk_count_key(r)).collect();
|
||||||
|
let mut stmt = conn
|
||||||
|
.prepare("SELECT key FROM schema_info WHERE key LIKE 'walk_count:%'")
|
||||||
|
.map_err(|e| format!("read walk counts: {}", e))?;
|
||||||
|
let stored: Vec<String> = stmt
|
||||||
|
.query_map([], |r| r.get::<_, String>(0))
|
||||||
|
.map_err(|e| format!("read walk counts: {}", e))?
|
||||||
|
.filter_map(|r| r.ok())
|
||||||
|
.collect();
|
||||||
|
drop(stmt);
|
||||||
|
for key in stored.iter().filter(|k| !keep.contains(*k)) {
|
||||||
|
conn.execute("DELETE FROM schema_info WHERE key = ?1", params![key])
|
||||||
|
.map_err(|e| format!("drop walk count {}: {}", key, e))?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
|
|
@ -788,16 +903,7 @@ mod tests {
|
||||||
use crate::db::open_or_recreate;
|
use crate::db::open_or_recreate;
|
||||||
|
|
||||||
fn tmp_path() -> std::path::PathBuf {
|
fn tmp_path() -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir("repo").join("index.sqlite")
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-repo-{}-{}.sqlite",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,8 @@
|
||||||
//! SQL strings for the current schema. Versioned; [`migrate`](super::migrate)
|
//! SQL strings for the current schema.
|
||||||
//! drives the upgrade path.
|
//!
|
||||||
|
//! Versioned by [`super::open::CURRENT_SCHEMA_VERSION`], but there is no
|
||||||
|
//! upgrade path: a database written under any other version is wiped and
|
||||||
|
//! recreated from [`SCHEMA_CURRENT`]. See [`super::open`] for why.
|
||||||
|
|
||||||
/// Pragmas applied on every writable connection open.
|
/// Pragmas applied on every writable connection open.
|
||||||
///
|
///
|
||||||
|
|
|
||||||
|
|
@ -1,373 +0,0 @@
|
||||||
use std::ffi::OsString;
|
|
||||||
use std::fs::File;
|
|
||||||
use std::io::{BufReader, Read};
|
|
||||||
|
|
||||||
use quick_xml::events::Event;
|
|
||||||
use quick_xml::Reader;
|
|
||||||
use zip::ZipArchive;
|
|
||||||
|
|
||||||
/// Extract text from DOCX files by parsing the word/document.xml
|
|
||||||
pub fn extract_text_from_docx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut document_xml = archive.by_name("word/document.xml")?;
|
|
||||||
let mut content = String::new();
|
|
||||||
document_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"w:t" {
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
text_content.push_str(&e.unescape()?.into_owned());
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"w:t" {
|
|
||||||
in_text = false;
|
|
||||||
} else if e.name().as_ref() == b"w:p" {
|
|
||||||
text_content.push('\n');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from XLSX files by parsing worksheet XML files
|
|
||||||
pub fn extract_text_from_xlsx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
|
|
||||||
// First, read shared strings if they exist
|
|
||||||
let mut shared_strings = Vec::new();
|
|
||||||
if let Ok(mut shared_strings_xml) = archive.by_name("xl/sharedStrings.xml") {
|
|
||||||
let mut content = String::new();
|
|
||||||
shared_strings_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"t" {
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
shared_strings.push(e.unescape()?.into_owned());
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"t" {
|
|
||||||
in_text = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read worksheets
|
|
||||||
for i in 0..archive.len() {
|
|
||||||
let file_name = archive.by_index(i)?.name().to_string();
|
|
||||||
if file_name.starts_with("xl/worksheets/sheet") && file_name.ends_with(".xml") {
|
|
||||||
let mut sheet_xml = archive.by_index(i)?;
|
|
||||||
let mut content = String::new();
|
|
||||||
sheet_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_cell = false;
|
|
||||||
let mut cell_type = String::new();
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"c" {
|
|
||||||
in_cell = true;
|
|
||||||
cell_type.clear();
|
|
||||||
for attr in e.attributes() {
|
|
||||||
let attr = attr?;
|
|
||||||
if attr.key.as_ref() == b"t" {
|
|
||||||
cell_type = String::from_utf8_lossy(&attr.value).to_string();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if e.name().as_ref() == b"v" && in_cell {
|
|
||||||
// Value element
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_cell => {
|
|
||||||
let text = e.unescape()?.into_owned();
|
|
||||||
if cell_type == "s" {
|
|
||||||
// Shared string reference
|
|
||||||
if let Ok(index) = text.parse::<usize>() {
|
|
||||||
if index < shared_strings.len() {
|
|
||||||
text_content.push_str(&shared_strings[index]);
|
|
||||||
text_content.push(' ');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
text_content.push_str(&text);
|
|
||||||
text_content.push(' ');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"c" {
|
|
||||||
in_cell = false;
|
|
||||||
} else if e.name().as_ref() == b"row" {
|
|
||||||
text_content.push('\n');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from PPTX files by parsing slide XML files
|
|
||||||
pub fn extract_text_from_pptx(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
|
|
||||||
// Read all slide files
|
|
||||||
for i in 0..archive.len() {
|
|
||||||
let file_name = archive.by_index(i)?.name().to_string();
|
|
||||||
if file_name.starts_with("ppt/slides/slide") && file_name.ends_with(".xml") {
|
|
||||||
let mut slide_xml = archive.by_index(i)?;
|
|
||||||
let mut content = String::new();
|
|
||||||
slide_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"a:t" {
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
text_content.push_str(&e.unescape()?.into_owned());
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
if e.name().as_ref() == b"a:t" {
|
|
||||||
in_text = false;
|
|
||||||
} else if e.name().as_ref() == b"a:p" {
|
|
||||||
text_content.push('\n');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
text_content.push_str("\n--- New Slide ---\n");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from ODT files (OpenDocument Text)
|
|
||||||
pub fn extract_text_from_odt(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut content_xml = archive.by_name("content.xml")?;
|
|
||||||
let mut content = String::new();
|
|
||||||
content_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p"
|
|
||||||
|| name.as_ref() == b"text:h"
|
|
||||||
|| name.as_ref() == b"text:span"
|
|
||||||
{
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
text_content.push_str(&e.unescape()?.into_owned());
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p" || name.as_ref() == b"text:h" {
|
|
||||||
text_content.push('\n');
|
|
||||||
in_text = false;
|
|
||||||
} else if name.as_ref() == b"text:span" {
|
|
||||||
in_text = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from ODP files (OpenDocument Presentation)
|
|
||||||
pub fn extract_text_from_odp(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut content_xml = archive.by_name("content.xml")?;
|
|
||||||
let mut content = String::new();
|
|
||||||
content_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p"
|
|
||||||
|| name.as_ref() == b"text:h"
|
|
||||||
|| name.as_ref() == b"text:span"
|
|
||||||
{
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
text_content.push_str(&e.unescape()?.into_owned());
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p" || name.as_ref() == b"text:h" {
|
|
||||||
text_content.push('\n');
|
|
||||||
in_text = false;
|
|
||||||
} else if name.as_ref() == b"text:span" {
|
|
||||||
in_text = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from ODS files (OpenDocument Spreadsheet)
|
|
||||||
pub fn extract_text_from_ods(file_path: &OsString) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
let file = File::open(file_path)?;
|
|
||||||
let mut archive = ZipArchive::new(BufReader::new(file))?;
|
|
||||||
|
|
||||||
let mut content_xml = archive.by_name("content.xml")?;
|
|
||||||
let mut content = String::new();
|
|
||||||
content_xml.read_to_string(&mut content)?;
|
|
||||||
|
|
||||||
let mut reader = Reader::from_str(&content);
|
|
||||||
reader.trim_text(true);
|
|
||||||
|
|
||||||
let mut text_content = String::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut in_text = false;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
match reader.read_event_into(&mut buf) {
|
|
||||||
Ok(Event::Start(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p" || name.as_ref() == b"text:span" {
|
|
||||||
in_text = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Text(e)) if in_text => {
|
|
||||||
text_content.push_str(&e.unescape()?.into_owned());
|
|
||||||
text_content.push(' ');
|
|
||||||
}
|
|
||||||
Ok(Event::End(ref e)) => {
|
|
||||||
let name = e.name();
|
|
||||||
if name.as_ref() == b"text:p" {
|
|
||||||
text_content.push('\n');
|
|
||||||
in_text = false;
|
|
||||||
} else if name.as_ref() == b"text:span" {
|
|
||||||
in_text = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Event::Eof) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
buf.clear();
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(text_content)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract text from various document formats
|
|
||||||
pub fn extract_document_text(
|
|
||||||
file_path: &OsString,
|
|
||||||
extension: &str,
|
|
||||||
) -> Result<String, Box<dyn std::error::Error>> {
|
|
||||||
match extension {
|
|
||||||
"docx" => extract_text_from_docx(file_path),
|
|
||||||
"doc" => {
|
|
||||||
// DOC format is binary and complex to parse without external tools
|
|
||||||
// For now, return an empty result
|
|
||||||
Ok(String::new())
|
|
||||||
}
|
|
||||||
"xlsx" => extract_text_from_xlsx(file_path),
|
|
||||||
"xls" => {
|
|
||||||
// XLS format is binary and complex to parse without external tools
|
|
||||||
Ok(String::new())
|
|
||||||
}
|
|
||||||
"pptx" => extract_text_from_pptx(file_path),
|
|
||||||
"ppt" => {
|
|
||||||
// PPT format is binary and complex to parse without external tools
|
|
||||||
Ok(String::new())
|
|
||||||
}
|
|
||||||
"odt" => extract_text_from_odt(file_path),
|
|
||||||
"odp" => extract_text_from_odp(file_path),
|
|
||||||
"ods" => extract_text_from_ods(file_path),
|
|
||||||
_ => Ok(String::new()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
//! Image metadata extraction via [`kamadak_exif`]. Reads EXIF tags (camera
|
//! Image metadata extraction via `kamadak-exif`. Reads EXIF tags (camera
|
||||||
//! make/model, date, GPS, dimensions) into properties. `text` is left empty
|
//! make/model, date, GPS, dimensions) into properties. `text` is left empty
|
||||||
//! — this extractor does not OCR.
|
//! — this extractor does not OCR.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -18,6 +18,7 @@ use std::path::Path;
|
||||||
pub mod audio;
|
pub mod audio;
|
||||||
pub mod image;
|
pub mod image;
|
||||||
pub mod office;
|
pub mod office;
|
||||||
|
pub mod ole;
|
||||||
pub mod pdf;
|
pub mod pdf;
|
||||||
pub mod plaintext;
|
pub mod plaintext;
|
||||||
pub mod rtf;
|
pub mod rtf;
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,27 @@
|
||||||
//! Office document extraction: DOCX, XLSX, PPTX, ODT, ODP, ODS.
|
//! Office document extraction: DOCX, XLSX, PPTX, ODT, ODP, ODS.
|
||||||
//!
|
//!
|
||||||
//! Delegates to [`crate::document_extraction`] — a single entry point based
|
//! All six are zip containers holding XML, and five of the six want the same
|
||||||
//! on file extension rather than MIME. We translate the MIME to the
|
//! thing from it: the character data of a few named elements, with a newline
|
||||||
//! extension expected by that module.
|
//! where a paragraph closes. That shape lives in [`collect_xml_text`], driven
|
||||||
|
//! by a per-format [`TextSpec`], so there is one event loop rather than one
|
||||||
|
//! per format.
|
||||||
|
//!
|
||||||
|
//! XLSX is the exception and keeps its own two loops: its text is not in the
|
||||||
|
//! sheet at all but in a shared-string table the cells index into, which is a
|
||||||
|
//! different machine, not a different table of element names.
|
||||||
|
//!
|
||||||
|
//! Dispatch is by file extension rather than MIME. `.docm` carries the same
|
||||||
|
//! MIME as `.docx` but needs the same reader, and the extension is what
|
||||||
|
//! distinguishes them.
|
||||||
|
|
||||||
use std::ffi::OsString;
|
use std::error::Error;
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io::{BufReader, Read, Seek};
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use crate::document_extraction::extract_document_text;
|
use quick_xml::events::Event;
|
||||||
|
use quick_xml::Reader;
|
||||||
|
use zip::ZipArchive;
|
||||||
|
|
||||||
use super::{ExtractError, ExtractedContent, Extractor};
|
use super::{ExtractError, ExtractedContent, Extractor};
|
||||||
|
|
||||||
|
|
@ -28,22 +42,293 @@ fn mime_to_ext(mime: &str) -> Option<&'static str> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// The shared XML text walk
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// Which elements of a format's XML carry text, and where paragraphs end.
|
||||||
|
///
|
||||||
|
/// `text` and `breaks` are matched independently on a closing tag: an element
|
||||||
|
/// can be in both (ODF's `text:p` both holds text and ends a paragraph), in
|
||||||
|
/// only one (`w:p` breaks but holds nothing directly), or in `text` alone
|
||||||
|
/// (`text:span`, which ends a run without ending the line).
|
||||||
|
struct TextSpec {
|
||||||
|
/// Elements whose character data is body text.
|
||||||
|
text: &'static [&'static [u8]],
|
||||||
|
/// Elements that close a paragraph, emitting `'\n'`.
|
||||||
|
breaks: &'static [&'static [u8]],
|
||||||
|
/// Emitted after each text run. Spreadsheets separate cells with it;
|
||||||
|
/// prose formats leave it `None` so runs within a paragraph stay joined.
|
||||||
|
separator: Option<char>,
|
||||||
|
}
|
||||||
|
|
||||||
|
const DOCX: TextSpec = TextSpec {
|
||||||
|
text: &[b"w:t"],
|
||||||
|
breaks: &[b"w:p"],
|
||||||
|
separator: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
const PPTX: TextSpec = TextSpec {
|
||||||
|
text: &[b"a:t"],
|
||||||
|
breaks: &[b"a:p"],
|
||||||
|
separator: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// ODT and ODP are the same format as far as text extraction is concerned —
|
||||||
|
/// both are ODF prose with headings, paragraphs and spans.
|
||||||
|
const ODF_TEXT: TextSpec = TextSpec {
|
||||||
|
text: &[b"text:p", b"text:h", b"text:span"],
|
||||||
|
breaks: &[b"text:p", b"text:h"],
|
||||||
|
separator: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
const ODF_SHEET: TextSpec = TextSpec {
|
||||||
|
text: &[b"text:p", b"text:span"],
|
||||||
|
breaks: &[b"text:p"],
|
||||||
|
separator: Some(' '),
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Append the text `spec` selects out of `xml` to `out`.
|
||||||
|
///
|
||||||
|
/// `in_text` is a flag rather than a depth count, which means a closing
|
||||||
|
/// `</text:span>` ends the run even though its enclosing `<text:p>` is still
|
||||||
|
/// open. That is how every one of the six extractors this replaces behaved.
|
||||||
|
fn collect_xml_text(xml: &str, spec: &TextSpec, out: &mut String) -> Result<(), Box<dyn Error>> {
|
||||||
|
let mut reader = Reader::from_str(xml);
|
||||||
|
reader.trim_text(true);
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
let mut in_text = false;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
match reader.read_event_into(&mut buf) {
|
||||||
|
Ok(Event::Start(ref e)) => {
|
||||||
|
if spec.text.contains(&e.name().as_ref()) {
|
||||||
|
in_text = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::Text(e)) if in_text => {
|
||||||
|
out.push_str(&e.unescape()?);
|
||||||
|
if let Some(sep) = spec.separator {
|
||||||
|
out.push(sep);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::End(ref e)) => {
|
||||||
|
let name = e.name();
|
||||||
|
if spec.text.contains(&name.as_ref()) {
|
||||||
|
in_text = false;
|
||||||
|
}
|
||||||
|
if spec.breaks.contains(&name.as_ref()) {
|
||||||
|
out.push('\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::Eof) => break,
|
||||||
|
// Propagated rather than ignored. Five of the six loops this
|
||||||
|
// replaces had no error arm at all, so a malformed member sent
|
||||||
|
// them round the loop on an error the reader kept re-reporting
|
||||||
|
// without advancing — a hang on a file the user merely happened
|
||||||
|
// to have on disk.
|
||||||
|
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Container access
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
type Archive = ZipArchive<BufReader<File>>;
|
||||||
|
|
||||||
|
fn open_container(path: &Path) -> Result<Archive, Box<dyn Error>> {
|
||||||
|
Ok(ZipArchive::new(BufReader::new(File::open(path)?))?)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One member's bytes as a string.
|
||||||
|
fn member_text<R: Read + Seek>(
|
||||||
|
archive: &mut ZipArchive<R>,
|
||||||
|
name: &str,
|
||||||
|
) -> Result<String, Box<dyn Error>> {
|
||||||
|
let mut member = archive.by_name(name)?;
|
||||||
|
let mut body = String::new();
|
||||||
|
member.read_to_string(&mut body)?;
|
||||||
|
Ok(body)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Names of the `.xml` members under `prefix`, in archive order.
|
||||||
|
///
|
||||||
|
/// Indexed rather than taken from `file_names()`, which iterates a hash map:
|
||||||
|
/// slide order is the archive's order, and hashing it would shuffle the
|
||||||
|
/// slides of every presentation.
|
||||||
|
fn xml_members_under<R: Read + Seek>(
|
||||||
|
archive: &mut ZipArchive<R>,
|
||||||
|
prefix: &str,
|
||||||
|
) -> Result<Vec<String>, Box<dyn Error>> {
|
||||||
|
let mut names = Vec::new();
|
||||||
|
for i in 0..archive.len() {
|
||||||
|
let name = archive.by_index(i)?.name().to_string();
|
||||||
|
if name.starts_with(prefix) && name.ends_with(".xml") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(names)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A format whose whole text lives in one member under one spec.
|
||||||
|
fn single_member(
|
||||||
|
path: &Path,
|
||||||
|
member: &str,
|
||||||
|
spec: &TextSpec,
|
||||||
|
) -> Result<String, Box<dyn Error>> {
|
||||||
|
let mut archive = open_container(path)?;
|
||||||
|
let xml = member_text(&mut archive, member)?;
|
||||||
|
let mut out = String::new();
|
||||||
|
collect_xml_text(&xml, spec, &mut out)?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn extract_pptx(path: &Path) -> Result<String, Box<dyn Error>> {
|
||||||
|
let mut archive = open_container(path)?;
|
||||||
|
let mut out = String::new();
|
||||||
|
for name in xml_members_under(&mut archive, "ppt/slides/slide")? {
|
||||||
|
let xml = member_text(&mut archive, &name)?;
|
||||||
|
collect_xml_text(&xml, &PPTX, &mut out)?;
|
||||||
|
out.push_str("\n--- New Slide ---\n");
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// XLSX: shared strings plus cells
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// The workbook's shared-string table, in index order. Absent or unreadable
|
||||||
|
/// is not an error: a sheet of nothing but numbers has no table at all.
|
||||||
|
fn shared_strings<R: Read + Seek>(archive: &mut ZipArchive<R>) -> Vec<String> {
|
||||||
|
let Ok(xml) = member_text(archive, "xl/sharedStrings.xml") else {
|
||||||
|
return Vec::new();
|
||||||
|
};
|
||||||
|
let mut reader = Reader::from_str(&xml);
|
||||||
|
reader.trim_text(true);
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
let mut strings = Vec::new();
|
||||||
|
let mut in_text = false;
|
||||||
|
loop {
|
||||||
|
match reader.read_event_into(&mut buf) {
|
||||||
|
Ok(Event::Start(ref e)) if e.name().as_ref() == b"t" => in_text = true,
|
||||||
|
Ok(Event::Text(e)) if in_text => match e.unescape() {
|
||||||
|
Ok(s) => strings.push(s.into_owned()),
|
||||||
|
Err(_) => return strings,
|
||||||
|
},
|
||||||
|
Ok(Event::End(ref e)) if e.name().as_ref() == b"t" => in_text = false,
|
||||||
|
Ok(Event::Eof) | Err(_) => break,
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
strings
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One worksheet's cells. A `t="s"` cell holds an index into `strings`
|
||||||
|
/// rather than text of its own; every other type holds its value inline.
|
||||||
|
fn collect_sheet(xml: &str, strings: &[String], out: &mut String) -> Result<(), Box<dyn Error>> {
|
||||||
|
let mut reader = Reader::from_str(xml);
|
||||||
|
reader.trim_text(true);
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
let mut in_cell = false;
|
||||||
|
let mut cell_type = String::new();
|
||||||
|
|
||||||
|
loop {
|
||||||
|
match reader.read_event_into(&mut buf) {
|
||||||
|
Ok(Event::Start(ref e)) if e.name().as_ref() == b"c" => {
|
||||||
|
in_cell = true;
|
||||||
|
cell_type.clear();
|
||||||
|
for attr in e.attributes() {
|
||||||
|
let attr = attr?;
|
||||||
|
if attr.key.as_ref() == b"t" {
|
||||||
|
cell_type = String::from_utf8_lossy(&attr.value).to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::Text(e)) if in_cell => {
|
||||||
|
let text = e.unescape()?;
|
||||||
|
if cell_type == "s" {
|
||||||
|
// A shared-string reference. An index past the end of the
|
||||||
|
// table is a corrupt workbook, not something to guess at.
|
||||||
|
if let Some(s) = text.parse::<usize>().ok().and_then(|i| strings.get(i)) {
|
||||||
|
out.push_str(s);
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
out.push_str(&text);
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::End(ref e)) => {
|
||||||
|
let name = e.name();
|
||||||
|
if name.as_ref() == b"c" {
|
||||||
|
in_cell = false;
|
||||||
|
} else if name.as_ref() == b"row" {
|
||||||
|
out.push('\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Event::Eof) => break,
|
||||||
|
Err(e) => return Err(format!("Error parsing XML: {}", e).into()),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn extract_xlsx(path: &Path) -> Result<String, Box<dyn Error>> {
|
||||||
|
let mut archive = open_container(path)?;
|
||||||
|
let strings = shared_strings(&mut archive);
|
||||||
|
let mut out = String::new();
|
||||||
|
for name in xml_members_under(&mut archive, "xl/worksheets/sheet")? {
|
||||||
|
let xml = member_text(&mut archive, &name)?;
|
||||||
|
collect_sheet(&xml, &strings, &mut out)?;
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Dispatch
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// Extract text from an office document, chosen by lowercase extension.
|
||||||
|
///
|
||||||
|
/// An extension nothing here handles yields empty text rather than an error:
|
||||||
|
/// the caller reaches this only for a MIME [`mime_to_ext`] claimed, so an
|
||||||
|
/// unrecognized extension means the file was named unlike its type.
|
||||||
|
fn extract_document_text(path: &Path, extension: &str) -> Result<String, Box<dyn Error>> {
|
||||||
|
match extension {
|
||||||
|
"docx" => single_member(path, "word/document.xml", &DOCX),
|
||||||
|
"xlsx" => extract_xlsx(path),
|
||||||
|
"pptx" => extract_pptx(path),
|
||||||
|
"odt" | "odp" => single_member(path, "content.xml", &ODF_TEXT),
|
||||||
|
"ods" => single_member(path, "content.xml", &ODF_SHEET),
|
||||||
|
// Pre-2007 binary formats: a different container entirely.
|
||||||
|
"doc" | "xls" | "ppt" => super::ole::extract_ole_text(path, extension),
|
||||||
|
_ => Ok(String::new()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl Extractor for OfficeExtractor {
|
impl Extractor for OfficeExtractor {
|
||||||
fn supports(&self, mime: &str) -> bool {
|
fn supports(&self, mime: &str) -> bool {
|
||||||
mime_to_ext(mime).is_some()
|
mime_to_ext(mime).is_some()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> {
|
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> {
|
||||||
// Recompute the extension from MIME at call time by asking the path.
|
// The extension from the path, not from the MIME: `.docm` and `.docx`
|
||||||
// Using the filesystem extension directly is more robust than round-
|
// share a MIME but the dispatch above is by extension.
|
||||||
// tripping through MIME: `.docm` has the same MIME as `.docx` but
|
|
||||||
// `extract_document_text` looks up by extension.
|
|
||||||
let ext = path
|
let ext = path
|
||||||
.extension()
|
.extension()
|
||||||
.and_then(|s| s.to_str())
|
.and_then(|s| s.to_str())
|
||||||
.map(|s| s.to_ascii_lowercase())
|
.map(|s| s.to_ascii_lowercase())
|
||||||
.unwrap_or_default();
|
.unwrap_or_default();
|
||||||
let text = extract_document_text(&OsString::from(path.as_os_str()), &ext)
|
let text = extract_document_text(path, &ext)
|
||||||
.map_err(|e| format!("office extractor {}: {}", path.display(), e))?;
|
.map_err(|e| format!("office extractor {}: {}", path.display(), e))?;
|
||||||
Ok(ExtractedContent::with_text(text))
|
Ok(ExtractedContent::with_text(text))
|
||||||
}
|
}
|
||||||
|
|
@ -52,6 +337,7 @@ impl Extractor for OfficeExtractor {
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use std::io::Write;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn supports_docx_and_friends() {
|
fn supports_docx_and_friends() {
|
||||||
|
|
@ -66,4 +352,203 @@ mod tests {
|
||||||
}
|
}
|
||||||
assert!(!e.supports("image/png"));
|
assert!(!e.supports("image/png"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A zip container holding `members`, written to a scratch file.
|
||||||
|
fn container(tag: &str, ext: &str, members: &[(&str, &str)]) -> std::path::PathBuf {
|
||||||
|
let path = crate::testutil::scratch_dir(tag).join(format!("doc.{ext}"));
|
||||||
|
let file = File::create(&path).unwrap();
|
||||||
|
let mut zip = zip::ZipWriter::new(file);
|
||||||
|
for (name, body) in members {
|
||||||
|
zip.start_file(*name, zip::write::FileOptions::default())
|
||||||
|
.unwrap();
|
||||||
|
zip.write_all(body.as_bytes()).unwrap();
|
||||||
|
}
|
||||||
|
zip.finish().unwrap();
|
||||||
|
path
|
||||||
|
}
|
||||||
|
|
||||||
|
const DOCX_BODY: &str = "<w:document><w:body>\
|
||||||
|
<w:p><w:r><w:t>Hello</w:t></w:r><w:r><w:t>world</w:t></w:r></w:p>\
|
||||||
|
<w:p><w:r><w:t>Second</w:t></w:r></w:p>\
|
||||||
|
</w:body></w:document>";
|
||||||
|
|
||||||
|
const PPTX_SLIDE: &str = "<p:sld><p:cSld><p:spTree><p:sp><p:txBody>\
|
||||||
|
<a:p><a:r><a:t>Title</a:t></a:r></a:p>\
|
||||||
|
<a:p><a:r><a:t>Body</a:t></a:r></a:p>\
|
||||||
|
</p:txBody></p:sp></p:spTree></p:cSld></p:sld>";
|
||||||
|
|
||||||
|
const ODT_BODY: &str = "<office:document-content><office:body><office:text>\
|
||||||
|
<text:h>Heading</text:h>\
|
||||||
|
<text:p>Para<text:span>span</text:span></text:p>\
|
||||||
|
</office:text></office:body></office:document-content>";
|
||||||
|
|
||||||
|
const ODS_BODY: &str = "<office:document-content><office:body><office:spreadsheet>\
|
||||||
|
<table:table><table:table-row>\
|
||||||
|
<table:table-cell><text:p>A1</text:p></table:table-cell>\
|
||||||
|
<table:table-cell><text:p>B1</text:p></table:table-cell>\
|
||||||
|
</table:table-row></table:table>\
|
||||||
|
</office:spreadsheet></office:body></office:document-content>";
|
||||||
|
|
||||||
|
const XLSX_SHARED: &str = "<sst><si><t>Shared</t></si><si><t>Second</t></si></sst>";
|
||||||
|
|
||||||
|
const XLSX_SHEET: &str = "<worksheet><sheetData>\
|
||||||
|
<row><c t=\"s\"><v>0</v></c><c t=\"n\"><v>42</v></c></row>\
|
||||||
|
<row><c t=\"s\"><v>1</v></c></row>\
|
||||||
|
</sheetData></worksheet>";
|
||||||
|
|
||||||
|
// The golden set. These strings were recorded from the six hand-written
|
||||||
|
// extractors this module replaced, so they pin the rewrite to exactly
|
||||||
|
// what shipped rather than to what it ought to have produced.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn docx() {
|
||||||
|
let p = container("docx", "docx", &[("word/document.xml", DOCX_BODY)]);
|
||||||
|
assert_eq!(
|
||||||
|
extract_document_text(&p, "docx").unwrap(),
|
||||||
|
"Helloworld\nSecond\n"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pptx_marks_each_slide_and_keeps_archive_order() {
|
||||||
|
let p = container(
|
||||||
|
"pptx",
|
||||||
|
"pptx",
|
||||||
|
&[
|
||||||
|
("ppt/slides/slide1.xml", PPTX_SLIDE),
|
||||||
|
("ppt/slides/slide2.xml", PPTX_SLIDE),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
extract_document_text(&p, "pptx").unwrap(),
|
||||||
|
"Title\nBody\n\n--- New Slide ---\nTitle\nBody\n\n--- New Slide ---\n"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn odt_and_odp_are_the_same_extraction() {
|
||||||
|
let odt = container("odt", "odt", &[("content.xml", ODT_BODY)]);
|
||||||
|
let odp = container("odp", "odp", &[("content.xml", ODT_BODY)]);
|
||||||
|
assert_eq!(
|
||||||
|
extract_document_text(&odt, "odt").unwrap(),
|
||||||
|
"Heading\nParaspan\n"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
extract_document_text(&odt, "odt").unwrap(),
|
||||||
|
extract_document_text(&odp, "odp").unwrap(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ods_separates_cells_with_a_space() {
|
||||||
|
let p = container("ods", "ods", &[("content.xml", ODS_BODY)]);
|
||||||
|
assert_eq!(extract_document_text(&p, "ods").unwrap(), "A1 \nB1 \n");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn xlsx_resolves_shared_strings() {
|
||||||
|
let p = container(
|
||||||
|
"xlsx",
|
||||||
|
"xlsx",
|
||||||
|
&[
|
||||||
|
("xl/sharedStrings.xml", XLSX_SHARED),
|
||||||
|
("xl/worksheets/sheet1.xml", XLSX_SHEET),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
extract_document_text(&p, "xlsx").unwrap(),
|
||||||
|
"Shared 42 \nSecond \n"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_extension_nothing_handles_is_empty() {
|
||||||
|
let p = container("none", "bin", &[("whatever", "x")]);
|
||||||
|
assert_eq!(extract_document_text(&p, "zzz").unwrap(), "");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A workbook of pure numbers has no shared-string table. Its absence is
|
||||||
|
/// normal, not a failure.
|
||||||
|
#[test]
|
||||||
|
fn xlsx_without_a_shared_string_table_still_reads_its_cells() {
|
||||||
|
let p = container(
|
||||||
|
"xlsx-nosst",
|
||||||
|
"xlsx",
|
||||||
|
&[("xl/worksheets/sheet1.xml", "<worksheet><sheetData>\
|
||||||
|
<row><c t=\"n\"><v>7</v></c></row></sheetData></worksheet>")],
|
||||||
|
);
|
||||||
|
assert_eq!(extract_document_text(&p, "xlsx").unwrap(), "7 \n");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A shared-string index past the end of the table is dropped rather than
|
||||||
|
/// panicking on the slice.
|
||||||
|
#[test]
|
||||||
|
fn an_out_of_range_shared_string_index_is_dropped() {
|
||||||
|
let p = container(
|
||||||
|
"xlsx-oob",
|
||||||
|
"xlsx",
|
||||||
|
&[
|
||||||
|
("xl/sharedStrings.xml", "<sst><si><t>only</t></si></sst>"),
|
||||||
|
(
|
||||||
|
"xl/worksheets/sheet1.xml",
|
||||||
|
"<worksheet><sheetData><row>\
|
||||||
|
<c t=\"s\"><v>0</v></c><c t=\"s\"><v>99</v></c>\
|
||||||
|
</row></sheetData></worksheet>",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert_eq!(extract_document_text(&p, "xlsx").unwrap(), "only \n");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Malformed XML is an error, not a hang. Five of the six extractors this
|
||||||
|
/// replaced had no error arm, so the reader re-reported the same failure
|
||||||
|
/// forever without advancing.
|
||||||
|
///
|
||||||
|
/// An undefined entity inside a text run is the cheapest way to reach that
|
||||||
|
/// arm, and it is a real shape: a document written by a tool that emitted
|
||||||
|
/// HTML entities into OOXML.
|
||||||
|
#[test]
|
||||||
|
fn malformed_xml_returns_an_error_rather_than_looping() {
|
||||||
|
for (ext, member, body) in [
|
||||||
|
("docx", "word/document.xml", "<w:t>bad &nonsuch; entity</w:t>"),
|
||||||
|
("odt", "content.xml", "<text:p>bad &nonsuch; entity</text:p>"),
|
||||||
|
("ods", "content.xml", "<text:p>bad &nonsuch; entity</text:p>"),
|
||||||
|
("pptx", "ppt/slides/slide1.xml", "<a:t>bad &nonsuch; entity</a:t>"),
|
||||||
|
] {
|
||||||
|
let p = container(&format!("bad-{ext}"), ext, &[(member, body)]);
|
||||||
|
assert!(
|
||||||
|
extract_document_text(&p, ext).is_err(),
|
||||||
|
"{ext} should report malformed XML"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Mismatched tags are caught too — quick_xml checks closing names, and
|
||||||
|
/// that error now reaches the caller instead of being swallowed.
|
||||||
|
#[test]
|
||||||
|
fn mismatched_tags_are_an_error() {
|
||||||
|
let p = container(
|
||||||
|
"mismatch",
|
||||||
|
"docx",
|
||||||
|
&[("word/document.xml", "<w:body><w:t>x</w:body>")],
|
||||||
|
);
|
||||||
|
assert!(extract_document_text(&p, "docx").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A container missing the member the format is defined by.
|
||||||
|
#[test]
|
||||||
|
fn a_missing_member_is_an_error() {
|
||||||
|
let p = container("empty", "docx", &[("unrelated.xml", "<x/>")]);
|
||||||
|
assert!(extract_document_text(&p, "docx").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Not a zip file at all — the shape a truncated download or a
|
||||||
|
/// misidentified file arrives in.
|
||||||
|
#[test]
|
||||||
|
fn a_non_container_is_an_error() {
|
||||||
|
let dir = crate::testutil::scratch_dir("notzip");
|
||||||
|
let p = dir.join("doc.docx");
|
||||||
|
crate::testutil::touch(&p, b"this is not a zip archive");
|
||||||
|
assert!(extract_document_text(&p, "docx").is_err());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
1097
crates/quicksearch-core/src/extract/ole.rs
Normal file
1097
crates/quicksearch-core/src/extract/ole.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -124,17 +124,8 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
|
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
let p = crate::testutil::scratch_dir(tag).join("sample.txt");
|
||||||
p.push(format!(
|
crate::testutil::touch(&p, body);
|
||||||
"qs-plaintext-{}-{}-{}.txt",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::write(&p, body).unwrap();
|
|
||||||
p
|
p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -58,17 +58,8 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
|
fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
let p = crate::testutil::scratch_dir(tag).join("sample.rtf");
|
||||||
p.push(format!(
|
crate::testutil::touch(&p, body);
|
||||||
"qs-rtf-{}-{}-{}.rtf",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::write(&p, body).unwrap();
|
|
||||||
p
|
p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -98,7 +89,13 @@ mod tests {
|
||||||
fn malformed_input_errors_and_names_the_file() {
|
fn malformed_input_errors_and_names_the_file() {
|
||||||
let p = tmp("broken", br"{\rtf1 truncated");
|
let p = tmp("broken", br"{\rtf1 truncated");
|
||||||
let err = RtfExtractor.extract(&p).unwrap_err();
|
let err = RtfExtractor.extract(&p).unwrap_err();
|
||||||
assert!(err.contains("qs-rtf-broken"), "must name the file: {}", err);
|
// The path itself, not a fixed prefix: this is the message a user sees
|
||||||
|
// in `list-failed`, and it is useless without naming the file.
|
||||||
|
assert!(
|
||||||
|
err.contains(&p.display().to_string()),
|
||||||
|
"must name the file: {}",
|
||||||
|
err
|
||||||
|
);
|
||||||
std::fs::remove_file(&p).ok();
|
std::fs::remove_file(&p).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
|
use std::collections::HashMap;
|
||||||
use std::fs::File;
|
use std::fs::File;
|
||||||
use std::io::Read;
|
use std::io::Read;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::{Arc, Mutex};
|
||||||
// Only the Unix entry-count path shells out; Windows walks the tree directly.
|
// Only the Unix entry-count path shells out; Windows walks the tree directly.
|
||||||
use std::collections::HashMap;
|
|
||||||
#[cfg(unix)]
|
#[cfg(unix)]
|
||||||
use std::process::{Command, Stdio};
|
use std::process::{Command, Stdio};
|
||||||
use std::time::UNIX_EPOCH;
|
use std::time::UNIX_EPOCH;
|
||||||
|
|
@ -67,15 +67,12 @@ pub(crate) fn path_to_db_string(path: &Path) -> String {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Canonicalize a root string for storage/comparison, stripping the Windows
|
/// Canonicalize a root string for storage/comparison, via
|
||||||
/// UNC prefix. Multi-root strings (newline-joined) fail canonicalize and
|
/// [`path_to_db_string`] rather than a hand-rolled prefix strip — the extra
|
||||||
/// pass through verbatim, which still compares consistently.
|
/// consequence here being that a mis-stripped share stops looking like one, so
|
||||||
///
|
/// the root would walk with the local thread count instead of the network one.
|
||||||
/// The UNC strip is [`path_to_db_string`]'s, not a hand-rolled one: chopping
|
/// Multi-root strings (newline-joined) fail canonicalize and pass through
|
||||||
/// four characters would turn `\\?\UNC\server\share` into
|
/// verbatim, which still compares consistently.
|
||||||
/// `UNC\server\share`, which is not a path — and no longer looks like a
|
|
||||||
/// share, so the root would walk with the local thread count instead of the
|
|
||||||
/// network one.
|
|
||||||
///
|
///
|
||||||
/// This is the spelling `files.path` rows are prefixed with, so it is also the
|
/// This is the spelling `files.path` rows are prefixed with, so it is also the
|
||||||
/// form roots must be compared in: `~/docs` and `/home/me/docs` name one root
|
/// form roots must be compared in: `~/docs` and `/home/me/docs` name one root
|
||||||
|
|
@ -196,8 +193,19 @@ impl UnreadableDirs {
|
||||||
/// Whether `path` lies under a directory the walk failed to read, and so
|
/// Whether `path` lies under a directory the walk failed to read, and so
|
||||||
/// must not be treated as deleted.
|
/// must not be treated as deleted.
|
||||||
///
|
///
|
||||||
/// Compares by path component, not by string prefix: `/a/bc` does not
|
/// # Containment is by path component, never by string prefix
|
||||||
/// live under `/a/b`.
|
///
|
||||||
|
/// `Path::starts_with` and `Path::ancestors` both compare whole
|
||||||
|
/// components, so `/a/bc` does not live under `/a/b` and `/a/b.txt` is not
|
||||||
|
/// swallowed by `/a/b`. A `str::starts_with` would say otherwise, and
|
||||||
|
/// every use of this rule in the codebase decides whether rows get
|
||||||
|
/// deleted, whether a watch is dropped, or whether a subtree is in scope —
|
||||||
|
/// so a sibling matched by accident is a neighbouring folder's index
|
||||||
|
/// disappearing. This is the reference statement of the rule;
|
||||||
|
/// [`crate::incremental::collapse_removal_roots`],
|
||||||
|
/// `coordinator::collapse_pending_removals`, `watcher::WatchRegistry::remove_tree`,
|
||||||
|
/// [`crate::scope::Scope::owning_root`] and
|
||||||
|
/// [`crate::config::nested_roots`] apply the same one.
|
||||||
pub fn covers(&self, path: &str) -> bool {
|
pub fn covers(&self, path: &str) -> bool {
|
||||||
let dirs = self.dirs.lock().unwrap();
|
let dirs = self.dirs.lock().unwrap();
|
||||||
if dirs.is_empty() {
|
if dirs.is_empty() {
|
||||||
|
|
@ -217,7 +225,12 @@ impl UnreadableDirs {
|
||||||
///
|
///
|
||||||
/// Used as a `walkdir` `filter_entry` predicate, so returning `false` for a
|
/// Used as a `walkdir` `filter_entry` predicate, so returning `false` for a
|
||||||
/// directory prunes the whole subtree instead of merely skipping the entry.
|
/// directory prunes the whole subtree instead of merely skipping the entry.
|
||||||
fn walk_filter(e: &DirEntry, include_hidden: bool, ignore: &IgnoreSet) -> bool {
|
fn walk_filter(
|
||||||
|
e: &DirEntry,
|
||||||
|
follow_symlinks: bool,
|
||||||
|
include_hidden: bool,
|
||||||
|
ignore: &IgnoreSet,
|
||||||
|
) -> bool {
|
||||||
// Depth 0 is the root itself — a `false` here would silence the
|
// Depth 0 is the root itself — a `false` here would silence the
|
||||||
// entire walk, and users explicitly chose their roots.
|
// entire walk, and users explicitly chose their roots.
|
||||||
if e.depth() == 0 {
|
if e.depth() == 0 {
|
||||||
|
|
@ -227,7 +240,21 @@ fn walk_filter(e: &DirEntry, include_hidden: bool, ignore: &IgnoreSet) -> bool {
|
||||||
// Free on Windows (walkdir hands back the attributes `FindNextFileW`
|
// Free on Windows (walkdir hands back the attributes `FindNextFileW`
|
||||||
// already returned) and never called on Unix, so the "no extra lstat"
|
// already returned) and never called on Unix, so the "no extra lstat"
|
||||||
// property below still holds.
|
// property below still holds.
|
||||||
if !include_hidden && crate::platform::entry_is_hidden(&name, || e.metadata().ok()) {
|
//
|
||||||
|
// The exception is a followed symlink: `DirEntry::metadata` switches to
|
||||||
|
// `fs::metadata` there and would report the *target's* attributes, where
|
||||||
|
// the parallel walker reports the link's own. `entry_hidden_reason`
|
||||||
|
// requires the entry itself, so ask for it explicitly — walkdir was
|
||||||
|
// already making a real call in that case, so this costs nothing extra.
|
||||||
|
if !include_hidden
|
||||||
|
&& crate::platform::entry_is_hidden(&name, || {
|
||||||
|
if follow_symlinks && e.path_is_symlink() {
|
||||||
|
std::fs::symlink_metadata(e.path()).ok()
|
||||||
|
} else {
|
||||||
|
e.metadata().ok()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
!ignore.matches_component(&name) && !ignore.matches_path_pattern(e.path())
|
!ignore.matches_component(&name) && !ignore.matches_path_pattern(e.path())
|
||||||
|
|
@ -256,7 +283,7 @@ fn walk_entries<'a>(
|
||||||
WalkDir::new(root)
|
WalkDir::new(root)
|
||||||
.follow_links(follow_symlinks)
|
.follow_links(follow_symlinks)
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter_entry(move |e| walk_filter(e, include_hidden, ignore))
|
.filter_entry(move |e| walk_filter(e, follow_symlinks, include_hidden, ignore))
|
||||||
.filter_map(move |res| match res {
|
.filter_map(move |res| match res {
|
||||||
Ok(e) => Some(e),
|
Ok(e) => Some(e),
|
||||||
Err(err) => {
|
Err(err) => {
|
||||||
|
|
@ -310,7 +337,6 @@ pub fn filtered_dirs<'a>(
|
||||||
fn parse_wc_l_stdout(bytes: &[u8]) -> Result<usize, String> {
|
fn parse_wc_l_stdout(bytes: &[u8]) -> Result<usize, String> {
|
||||||
let s = String::from_utf8_lossy(bytes);
|
let s = String::from_utf8_lossy(bytes);
|
||||||
let token = s
|
let token = s
|
||||||
.trim()
|
|
||||||
.split_whitespace()
|
.split_whitespace()
|
||||||
.next()
|
.next()
|
||||||
.ok_or_else(|| "wc: empty output".to_string())?;
|
.ok_or_else(|| "wc: empty output".to_string())?;
|
||||||
|
|
@ -405,15 +431,11 @@ fn count_find_pipe_wc(
|
||||||
|
|
||||||
/// Count tree entries with a plain directory walk.
|
/// Count tree entries with a plain directory walk.
|
||||||
///
|
///
|
||||||
/// Windows has no `find`, and the obvious substitute — `powershell.exe -Command
|
/// Kept as the oracle [`count_tree_entries_win32`] is tested against: the fast
|
||||||
/// "(Get-ChildItem -Recurse | Measure-Object).Count"` — is a poor trade: 300+
|
/// path replaces it precisely because it must produce the same number, and an
|
||||||
/// ms of interpreter startup before any work, `Get-ChildItem -Recurse` is far
|
/// independent implementation is the only way to assert that.
|
||||||
/// slower than a `FindNextFileW` loop, it pops a console window on a windowed
|
|
||||||
/// process, and the path has to be escaped into a script string. Walking
|
|
||||||
/// directly is faster, quieter, and cancels immediately instead of at the
|
|
||||||
/// 50 ms subprocess-poll granularity.
|
|
||||||
#[cfg(windows)]
|
#[cfg(windows)]
|
||||||
fn count_tree_entries_native(
|
fn count_tree_entries_walkdir(
|
||||||
path: &str,
|
path: &str,
|
||||||
cancel: &std::sync::atomic::AtomicBool,
|
cancel: &std::sync::atomic::AtomicBool,
|
||||||
) -> Result<usize, String> {
|
) -> Result<usize, String> {
|
||||||
|
|
@ -436,10 +458,151 @@ fn count_tree_entries_native(
|
||||||
Ok(n)
|
Ok(n)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Count tree entries by reading directories in bulk through
|
||||||
|
/// `GetFileInformationByHandleEx`.
|
||||||
|
///
|
||||||
|
/// Windows has no `find`, so the Unix pipeline has nothing to call. The
|
||||||
|
/// obvious substitute — `powershell.exe -Command "(Get-ChildItem -Recurse |
|
||||||
|
/// Measure-Object).Count"` — is a poor trade: 300+ ms of interpreter startup
|
||||||
|
/// before any work, `Get-ChildItem -Recurse` is far slower than a directory
|
||||||
|
/// read, it pops a console window on a windowed process, and the path has to
|
||||||
|
/// be escaped into a script string.
|
||||||
|
///
|
||||||
|
/// The reason this exists rather than [`count_tree_entries_walkdir`] is the
|
||||||
|
/// per-entry cost. `std::fs::read_dir` — which walkdir sits on — issues one
|
||||||
|
/// `FindNextFileW` per entry and allocates a `PathBuf` for each, so counting a
|
||||||
|
/// million-entry tree means a million syscalls and a million allocations to
|
||||||
|
/// produce a single integer. `FileIdBothDirectoryInfo` fills a caller-supplied
|
||||||
|
/// buffer with as many chained records as fit, which turns the syscall count
|
||||||
|
/// into roughly one per 64 KiB of directory data and the allocations into one
|
||||||
|
/// per directory. This is the documented Win32 form of what fast indexers use;
|
||||||
|
/// it needs no new crate, no elevated rights, and no ntdll.
|
||||||
|
///
|
||||||
|
/// Recursion is explicit and iterative — a stack of directories still to read
|
||||||
|
/// — because a deep tree must not be able to exhaust the thread's stack.
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn count_tree_entries_win32(
|
||||||
|
path: &str,
|
||||||
|
cancel: &std::sync::atomic::AtomicBool,
|
||||||
|
) -> Result<usize, String> {
|
||||||
|
use std::ffi::OsString;
|
||||||
|
use std::os::windows::ffi::{OsStrExt, OsStringExt};
|
||||||
|
use std::sync::atomic::Ordering;
|
||||||
|
use windows_sys::Win32::Foundation::{CloseHandle, INVALID_HANDLE_VALUE};
|
||||||
|
use windows_sys::Win32::Storage::FileSystem::{
|
||||||
|
CreateFileW, GetFileInformationByHandleEx, FileIdBothDirectoryInfo, FILE_ID_BOTH_DIR_INFO,
|
||||||
|
FILE_FLAG_BACKUP_SEMANTICS, FILE_LIST_DIRECTORY, FILE_SHARE_DELETE, FILE_SHARE_READ,
|
||||||
|
FILE_SHARE_WRITE, OPEN_EXISTING,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// One call returns as many entries as fit here. 64 KiB holds several
|
||||||
|
/// hundred typical names, so a directory of any ordinary size is one or
|
||||||
|
/// two syscalls rather than one per file.
|
||||||
|
const BUFFER_BYTES: usize = 64 * 1024;
|
||||||
|
|
||||||
|
/// `FILE_ATTRIBUTE_DIRECTORY`, spelled out for the reason
|
||||||
|
/// [`crate::platform`] spells out its own attribute constants, and
|
||||||
|
/// const-asserted against the real header below.
|
||||||
|
const ATTR_DIRECTORY: u32 = 0x10;
|
||||||
|
const _: () = assert!(
|
||||||
|
ATTR_DIRECTORY == windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_DIRECTORY
|
||||||
|
);
|
||||||
|
|
||||||
|
fn wide(path: &std::path::Path) -> Vec<u16> {
|
||||||
|
path.as_os_str().encode_wide().chain(Some(0)).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut pending = vec![std::path::PathBuf::from(path)];
|
||||||
|
let mut count = 0usize;
|
||||||
|
// One allocation for the whole walk, reused for every directory.
|
||||||
|
let mut buffer = vec![0u8; BUFFER_BYTES];
|
||||||
|
|
||||||
|
while let Some(dir) = pending.pop() {
|
||||||
|
// Checked per directory *and* per entry below: cancellation has to be
|
||||||
|
// immediate on a tree large enough to be worth counting.
|
||||||
|
if cancel.load(Ordering::Relaxed) {
|
||||||
|
return Err("count cancelled".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
// FILE_FLAG_BACKUP_SEMANTICS is what makes CreateFileW open a
|
||||||
|
// directory rather than fail; the share flags let the tree keep being
|
||||||
|
// used while we count it.
|
||||||
|
let handle = unsafe {
|
||||||
|
CreateFileW(
|
||||||
|
wide(&dir).as_ptr(),
|
||||||
|
FILE_LIST_DIRECTORY,
|
||||||
|
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
|
||||||
|
std::ptr::null(),
|
||||||
|
OPEN_EXISTING,
|
||||||
|
FILE_FLAG_BACKUP_SEMANTICS,
|
||||||
|
// hTemplateFile: a HANDLE, which windows-sys spells `isize`.
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
};
|
||||||
|
if handle == INVALID_HANDLE_VALUE {
|
||||||
|
// An unreadable directory costs its own entries, not the tree's.
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
loop {
|
||||||
|
// Returns zero both on error and at the end of the listing; either
|
||||||
|
// way this directory is done.
|
||||||
|
let ok = unsafe {
|
||||||
|
GetFileInformationByHandleEx(
|
||||||
|
handle,
|
||||||
|
FileIdBothDirectoryInfo,
|
||||||
|
buffer.as_mut_ptr().cast(),
|
||||||
|
buffer.len() as u32,
|
||||||
|
)
|
||||||
|
};
|
||||||
|
if ok == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut offset = 0usize;
|
||||||
|
loop {
|
||||||
|
if cancel.load(Ordering::Relaxed) {
|
||||||
|
unsafe { CloseHandle(handle) };
|
||||||
|
return Err("count cancelled".to_string());
|
||||||
|
}
|
||||||
|
// SAFETY: the API filled `buffer` with a chain of these
|
||||||
|
// records, each at the offset the previous one gave.
|
||||||
|
let info =
|
||||||
|
unsafe { &*(buffer.as_ptr().add(offset) as *const FILE_ID_BOTH_DIR_INFO) };
|
||||||
|
|
||||||
|
// FileNameLength is in bytes; FileName is UTF-16 and is *not*
|
||||||
|
// NUL-terminated, so the length is the only thing that says
|
||||||
|
// where it ends.
|
||||||
|
let name_units = (info.FileNameLength as usize) / 2;
|
||||||
|
let name = unsafe {
|
||||||
|
std::slice::from_raw_parts(info.FileName.as_ptr(), name_units)
|
||||||
|
};
|
||||||
|
let name = OsString::from_wide(name);
|
||||||
|
|
||||||
|
// "." and ".." are entries of the listing, not of the tree.
|
||||||
|
let is_dot = name == "." || name == "..";
|
||||||
|
if !is_dot {
|
||||||
|
count += 1;
|
||||||
|
if info.FileAttributes & ATTR_DIRECTORY != 0 {
|
||||||
|
pending.push(dir.join(&name));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
match info.NextEntryOffset {
|
||||||
|
0 => break,
|
||||||
|
next => offset += next as usize,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
unsafe { CloseHandle(handle) };
|
||||||
|
}
|
||||||
|
Ok(count)
|
||||||
|
}
|
||||||
|
|
||||||
/// Rough tree entry count for progress totals. Setting `cancel` stops it
|
/// Rough tree entry count for progress totals. Setting `cancel` stops it
|
||||||
/// rather than letting it scan an entire root after the run stopped: on Unix
|
/// rather than letting it scan an entire root after the run stopped: on Unix
|
||||||
/// that kills the `find`/`wc` subprocesses at ~50 ms granularity, on Windows
|
/// that kills the `find`/`wc` subprocesses at ~50 ms granularity, on Windows
|
||||||
/// the native walk notices almost immediately. Runs concurrently with
|
/// the bulk directory read notices almost immediately. Runs concurrently with
|
||||||
/// indexing — its scope is not identical to the walker's classified file
|
/// indexing — its scope is not identical to the walker's classified file
|
||||||
/// count.
|
/// count.
|
||||||
pub fn count_tree_entries_fast(
|
pub fn count_tree_entries_fast(
|
||||||
|
|
@ -448,18 +611,18 @@ pub fn count_tree_entries_fast(
|
||||||
) -> Result<usize, String> {
|
) -> Result<usize, String> {
|
||||||
#[cfg(windows)]
|
#[cfg(windows)]
|
||||||
{
|
{
|
||||||
return count_tree_entries_native(path, cancel);
|
return count_tree_entries_win32(path, cancel);
|
||||||
}
|
}
|
||||||
#[cfg(all(unix, target_os = "linux"))]
|
#[cfg(all(unix, target_os = "linux"))]
|
||||||
{
|
{
|
||||||
return count_find_pipe_wc(path, cancel, true).or_else(|e| {
|
count_find_pipe_wc(path, cancel, true).or_else(|e| {
|
||||||
if e.contains("cancelled") {
|
if e.contains("cancelled") {
|
||||||
Err(e)
|
Err(e)
|
||||||
} else {
|
} else {
|
||||||
// Non-GNU find without -printf: plain listing.
|
// Non-GNU find without -printf: plain listing.
|
||||||
count_find_pipe_wc(path, cancel, false)
|
count_find_pipe_wc(path, cancel, false)
|
||||||
}
|
}
|
||||||
});
|
})
|
||||||
}
|
}
|
||||||
#[cfg(all(unix, not(target_os = "linux")))]
|
#[cfg(all(unix, not(target_os = "linux")))]
|
||||||
{
|
{
|
||||||
|
|
@ -509,13 +672,13 @@ pub fn classify_by_mtime(stored: Option<u64>, mtime: u64) -> FileIndexAction {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Safely truncate a string to at most max_bytes bytes while respecting UTF-8 character boundaries
|
/// Truncate to at most `max_bytes` bytes, backing up to a UTF-8 character
|
||||||
|
/// boundary. Cuts short of the budget rather than over it.
|
||||||
fn safe_truncate_string(s: &str, max_bytes: usize) -> String {
|
fn safe_truncate_string(s: &str, max_bytes: usize) -> String {
|
||||||
if s.len() <= max_bytes {
|
if s.len() <= max_bytes {
|
||||||
return s.to_string();
|
return s.to_string();
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find the last valid UTF-8 character boundary at or before max_bytes
|
|
||||||
let mut end = max_bytes;
|
let mut end = max_bytes;
|
||||||
while end > 0 && !s.is_char_boundary(end) {
|
while end > 0 && !s.is_char_boundary(end) {
|
||||||
end -= 1;
|
end -= 1;
|
||||||
|
|
@ -553,7 +716,7 @@ fn get_file_hash(
|
||||||
f.read_exact(&mut head)?;
|
f.read_exact(&mut head)?;
|
||||||
|
|
||||||
let mut hasher = Sha256::new();
|
let mut hasher = Sha256::new();
|
||||||
hasher.update(&size.to_le_bytes());
|
hasher.update(size.to_le_bytes());
|
||||||
hasher.update(&head);
|
hasher.update(&head);
|
||||||
Ok((hasher.finalize().to_vec(), head))
|
Ok((hasher.finalize().to_vec(), head))
|
||||||
}
|
}
|
||||||
|
|
@ -583,7 +746,11 @@ pub struct OwnedNewFile {
|
||||||
pub device_id: Option<u64>,
|
pub device_id: Option<u64>,
|
||||||
pub mime: Option<String>,
|
pub mime: Option<String>,
|
||||||
pub ftype: FileType,
|
pub ftype: FileType,
|
||||||
pub hash: Vec<u8>,
|
/// `None` only for a file whose contents could not be read without paying
|
||||||
|
/// for them — a dehydrated cloud placeholder. Stored as SQL NULL, which is
|
||||||
|
/// what keeps such files out of duplicate detection: an empty or zero hash
|
||||||
|
/// would make every one of them look identical to every other.
|
||||||
|
pub hash: Option<Vec<u8>>,
|
||||||
/// Text extracted from the head bytes during the walk, for files small
|
/// Text extracted from the head bytes during the walk, for files small
|
||||||
/// enough that the head *was* the whole file. `Some` means the content
|
/// enough that the head *was* the whole file. `Some` means the content
|
||||||
/// pass never has to open this file; `None` leaves it pending as before.
|
/// pass never has to open this file; `None` leaves it pending as before.
|
||||||
|
|
@ -611,12 +778,28 @@ impl OwnedNewFile {
|
||||||
device_id: self.device_id,
|
device_id: self.device_id,
|
||||||
mime: self.mime.as_deref(),
|
mime: self.mime.as_deref(),
|
||||||
ftype: self.ftype,
|
ftype: self.ftype,
|
||||||
hash: Some(&self.hash),
|
hash: self.hash.as_deref(),
|
||||||
needs_content: self.needs_content,
|
needs_content: self.needs_content,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Individual "cannot hash" warnings allowed per run before only the count is
|
||||||
|
/// kept. See [`crate::log::Throttle`]; [`reset_run_warnings`] arms it.
|
||||||
|
static HASH_FAILURES: crate::log::Throttle = crate::log::Throttle::new(20);
|
||||||
|
|
||||||
|
/// Arm the per-run warning throttles. Called once at the start of an indexing
|
||||||
|
/// run, so the counts a run reports describe that run and not the process.
|
||||||
|
pub fn reset_run_warnings() {
|
||||||
|
HASH_FAILURES.reset();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How many files could not be hashed this run, and how many of those went
|
||||||
|
/// unlogged. `(0, 0)` when nothing failed.
|
||||||
|
pub fn hash_failure_counts() -> (u64, u64) {
|
||||||
|
(HASH_FAILURES.seen(), HASH_FAILURES.suppressed())
|
||||||
|
}
|
||||||
|
|
||||||
/// Build the `files` row for one on-disk file from a `stat` the caller
|
/// Build the `files` row for one on-disk file from a `stat` the caller
|
||||||
/// already holds. The single implementation behind both full-run batches
|
/// already holds. The single implementation behind both full-run batches
|
||||||
/// and incremental watcher updates.
|
/// and incremental watcher updates.
|
||||||
|
|
@ -651,12 +834,30 @@ pub fn prepare_file_record(
|
||||||
.and_then(|t| t.duration_since(UNIX_EPOCH).ok())
|
.and_then(|t| t.duration_since(UNIX_EPOCH).ok())
|
||||||
.map(|d| d.as_secs())?;
|
.map(|d| d.as_secs())?;
|
||||||
|
|
||||||
let (hash, head) = match get_file_hash(size, Path::new(path), config.processing.hash_length) {
|
// A dehydrated cloud file is indexed from its metadata alone. Reading even
|
||||||
Ok(v) => v,
|
// the first byte would block on downloading the whole thing, so the head is
|
||||||
|
// never fetched: no hash, no MIME sniff, no inline extraction, and the row
|
||||||
|
// is born with nothing pending. The file is still fully searchable by name,
|
||||||
|
// size and date — only its *contents* wait, and they wait until the user
|
||||||
|
// hydrates the file themselves. Hydration does not change the mtime, so what
|
||||||
|
// picks it up is this same test on a later run, once the attribute clears.
|
||||||
|
let dehydrated = crate::platform::is_cloud_placeholder(meta);
|
||||||
|
|
||||||
|
let (hash, head) = if dehydrated {
|
||||||
|
(None, Vec::new())
|
||||||
|
} else {
|
||||||
|
match get_file_hash(size, Path::new(path), config.processing.hash_length) {
|
||||||
|
Ok((hash, head)) => (Some(hash), head),
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
|
// Throttled: on Windows a file another process holds open fails
|
||||||
|
// here as a matter of course, and a large tree can produce
|
||||||
|
// thousands. `run_indexing` reports the total.
|
||||||
|
if HASH_FAILURES.allow() {
|
||||||
crate::log_warn!("Skipping file (cannot hash) {}: {}", path, e);
|
crate::log_warn!("Skipping file (cannot hash) {}: {}", path, e);
|
||||||
|
}
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let name = Path::new(path)
|
let name = Path::new(path)
|
||||||
|
|
@ -664,7 +865,9 @@ pub fn prepare_file_record(
|
||||||
.map(|n| n.to_string_lossy().into_owned())?;
|
.map(|n| n.to_string_lossy().into_owned())?;
|
||||||
let parent = parent_str(path);
|
let parent = parent_str(path);
|
||||||
let (inode, device_id) = inode_and_device(meta);
|
let (inode, device_id) = inode_and_device(meta);
|
||||||
// Sniff from the bytes hashing already read rather than reopening.
|
// Sniff from the bytes hashing already read rather than reopening. With no
|
||||||
|
// head to sniff, the extension is all there is — which is what
|
||||||
|
// `guess_mime_from_head` already falls back to for an empty slice.
|
||||||
let mime = guess_mime_from_head(Path::new(path), &head);
|
let mime = guess_mime_from_head(Path::new(path), &head);
|
||||||
let ftype = mime.as_deref().map(mime_to_type).unwrap_or(FileType::EMPTY);
|
let ftype = mime.as_deref().map(mime_to_type).unwrap_or(FileType::EMPTY);
|
||||||
|
|
||||||
|
|
@ -672,7 +875,8 @@ pub fn prepare_file_record(
|
||||||
// registry are all in hand — and stored as the row's `content_state`, so
|
// registry are all in hand — and stored as the row's `content_state`, so
|
||||||
// "pending" downstream means a file that really needs reading rather than
|
// "pending" downstream means a file that really needs reading rather than
|
||||||
// one the content pass would only mark not-applicable.
|
// one the content pass would only mark not-applicable.
|
||||||
let needs_content = size <= config.processing.maximum_text_file_size
|
let needs_content = !dehydrated
|
||||||
|
&& size <= config.processing.maximum_text_file_size
|
||||||
&& content_extractable(Path::new(path), mime.as_deref(), config, registry);
|
&& content_extractable(Path::new(path), mime.as_deref(), config, registry);
|
||||||
|
|
||||||
// When the head is the whole file, an extractor that works from bytes can
|
// When the head is the whole file, an extractor that works from bytes can
|
||||||
|
|
@ -1209,17 +1413,7 @@ mod tests {
|
||||||
use std::path::MAIN_SEPARATOR;
|
use std::path::MAIN_SEPARATOR;
|
||||||
|
|
||||||
fn tmp_tree() -> std::path::PathBuf {
|
fn tmp_tree() -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir("walk")
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-walk-{}-{}",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&p).unwrap();
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn touch(p: &Path) {
|
fn touch(p: &Path) {
|
||||||
|
|
@ -1619,18 +1813,9 @@ mod count_and_extract_tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
|
|
||||||
|
/// A path that does not exist yet — these tests build the tree themselves.
|
||||||
fn tmp(tag: &str) -> std::path::PathBuf {
|
fn tmp(tag: &str) -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir(tag).join("tree")
|
||||||
p.push(format!(
|
|
||||||
"qs-ce-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -1642,8 +1827,59 @@ mod count_and_extract_tests {
|
||||||
}
|
}
|
||||||
let cancel = AtomicBool::new(false);
|
let cancel = AtomicBool::new(false);
|
||||||
let n = count_tree_entries_fast(root.to_str().unwrap(), &cancel).unwrap();
|
let n = count_tree_entries_fast(root.to_str().unwrap(), &cancel).unwrap();
|
||||||
// find lists the root, the subdir, and the three files.
|
// The subdir plus the three files. Unix counts through `find`, which
|
||||||
assert_eq!(n, 5);
|
// also lists the root it was given; the Windows directory read only
|
||||||
|
// ever sees entries *inside* a directory and so never counts the root.
|
||||||
|
// Both are correct for a progress denominator — this is an estimate
|
||||||
|
// running concurrently with the walk, not a total to reconcile against.
|
||||||
|
assert_eq!(n, if cfg!(windows) { 4 } else { 5 });
|
||||||
|
std::fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The load-bearing property of the Windows fast path: it exists only to
|
||||||
|
/// be quicker than a plain walk, so it has to agree with one exactly.
|
||||||
|
///
|
||||||
|
/// The tree is deliberately wider than one buffer's worth of directory
|
||||||
|
/// data, so `GetFileInformationByHandleEx` is called repeatedly for the
|
||||||
|
/// same directory and the resumption between calls is exercised — the part
|
||||||
|
/// a single-buffer test would never reach.
|
||||||
|
#[cfg(windows)]
|
||||||
|
#[test]
|
||||||
|
fn the_bulk_directory_read_agrees_with_a_plain_walk() {
|
||||||
|
let root = tmp("count-oracle");
|
||||||
|
std::fs::create_dir_all(root.join("empty")).unwrap();
|
||||||
|
std::fs::create_dir_all(root.join("a/b/c")).unwrap();
|
||||||
|
// Long names so the chained records fill more than one 64 KiB buffer.
|
||||||
|
for i in 0..600 {
|
||||||
|
let name = format!("{}-{:04}.txt", "padding".repeat(12), i);
|
||||||
|
std::fs::write(root.join(&name), b"x").unwrap();
|
||||||
|
std::fs::write(root.join("a/b/c").join(&name), b"x").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let cancel = AtomicBool::new(false);
|
||||||
|
let path = root.to_str().unwrap();
|
||||||
|
let fast = count_tree_entries_win32(path, &cancel).unwrap();
|
||||||
|
let plain = count_tree_entries_walkdir(path, &cancel).unwrap();
|
||||||
|
assert_eq!(fast, plain, "the fast count must match a plain walk");
|
||||||
|
// 1200 files + `empty` + `a` + `a/b` + `a/b/c`.
|
||||||
|
assert_eq!(fast, 1204);
|
||||||
|
|
||||||
|
std::fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cancellation is checked per entry, not per directory, so a huge single
|
||||||
|
/// directory stops as promptly as a deep tree.
|
||||||
|
#[cfg(windows)]
|
||||||
|
#[test]
|
||||||
|
fn the_bulk_directory_read_stops_when_cancelled() {
|
||||||
|
let root = tmp("count-cancel");
|
||||||
|
std::fs::create_dir_all(&root).unwrap();
|
||||||
|
for i in 0..200 {
|
||||||
|
std::fs::write(root.join(format!("f{:03}.txt", i)), b"x").unwrap();
|
||||||
|
}
|
||||||
|
let cancel = AtomicBool::new(true);
|
||||||
|
let err = count_tree_entries_win32(root.to_str().unwrap(), &cancel).unwrap_err();
|
||||||
|
assert!(err.contains("cancelled"), "{err}");
|
||||||
std::fs::remove_dir_all(&root).ok();
|
std::fs::remove_dir_all(&root).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -73,8 +73,14 @@ fn upsert_path(
|
||||||
if meta.is_dir() {
|
if meta.is_dir() {
|
||||||
// A moved-in tree surfaces as one directory event; walk it with
|
// A moved-in tree surfaces as one directory event; walk it with
|
||||||
// the same filters as a full run.
|
// the same filters as a full run.
|
||||||
|
//
|
||||||
|
// `filtered_walk` wants a `&str`, so a path that is not valid UTF-8
|
||||||
|
// cannot be walked here. That is a whole subtree missing from the
|
||||||
|
// index, not one file, so it is an error rather than a quiet `Ok`:
|
||||||
|
// the caller answers it by scheduling a full run, which walks from
|
||||||
|
// the configured root and never needs the path as a `str`.
|
||||||
let Some(root) = path.to_str() else {
|
let Some(root) = path.to_str() else {
|
||||||
return Ok(());
|
return Err(format!("directory path is not valid UTF-8: {:?}", path));
|
||||||
};
|
};
|
||||||
let entries: Vec<_> = filtered_walk(
|
let entries: Vec<_> = filtered_walk(
|
||||||
root,
|
root,
|
||||||
|
|
@ -168,11 +174,17 @@ fn remove_path(conn: &mut Connection, path: &Path) -> Result<(), String> {
|
||||||
/// `dir` sweeps its whole path range, each descendant event is duplicate work —
|
/// `dir` sweeps its whole path range, each descendant event is duplicate work —
|
||||||
/// 10 000 of them for a 10 000-file tree.
|
/// 10 000 of them for a 10 000-file tree.
|
||||||
///
|
///
|
||||||
|
/// This is for a caller holding a raw removal set. The coordinator is not one:
|
||||||
|
/// it collapses as events *arrive*, in `collapse_pending_removals`, because it
|
||||||
|
/// has to measure its queue after collapsing rather than before (see
|
||||||
|
/// `Inner::drain_events`). Same rule, two entry points, deliberately — a
|
||||||
|
/// mass deletion must be one path by the time either of them is done with it.
|
||||||
|
///
|
||||||
/// Ancestor membership is tested against a set rather than by sorting, which
|
/// Ancestor membership is tested against a set rather than by sorting, which
|
||||||
/// keeps this obviously correct: no ordering argument to get wrong, and
|
/// keeps this obviously correct: there is no ordering argument to get wrong.
|
||||||
/// `Path::ancestors` walks whole components, so `/a/bc` is never treated as
|
/// Containment is component-wise, per
|
||||||
/// living under `/a/b` — the same rule `remove_tree` and
|
/// [`crate::file_handling::UnreadableDirs::covers`]. Paths are shallow, so the
|
||||||
/// `UnreadableDirs::covers` use. Paths are shallow, so the cost is linear.
|
/// cost is linear.
|
||||||
pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::PathBuf> {
|
pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::PathBuf> {
|
||||||
if paths.len() < 2 {
|
if paths.len() < 2 {
|
||||||
return paths;
|
return paths;
|
||||||
|
|
@ -188,11 +200,13 @@ pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::
|
||||||
/// Delete `paths` and everything indexed beneath them, in transactions of at
|
/// Delete `paths` and everything indexed beneath them, in transactions of at
|
||||||
/// most `chunk` paths.
|
/// most `chunk` paths.
|
||||||
///
|
///
|
||||||
/// A mass deletion is the case this exists for. Callers hand over the *roots*
|
/// Correct for any path set, but a mass deletion is the case it exists for,
|
||||||
/// of the removal set (see [`collapse_removal_roots`]), so `rm -rf dir/` is one
|
/// and it pays off only if the caller has already reduced that set to its
|
||||||
/// path here rather than one per file, and each one costs a fixed handful of
|
/// *roots* — then `rm -rf dir/` is one path here rather than one per file, and
|
||||||
/// range-driven statements ([`repo::delete_subtree`]) instead of a full table
|
/// costs a fixed handful of range-driven statements
|
||||||
/// scan plus five statements per file.
|
/// ([`repo::delete_subtree`]) instead of a full table scan plus five
|
||||||
|
/// statements per file. Both callers do: the coordinator collapses on arrival,
|
||||||
|
/// and anyone else has [`collapse_removal_roots`].
|
||||||
///
|
///
|
||||||
/// Chunking bounds how long any single transaction holds the connection, the
|
/// Chunking bounds how long any single transaction holds the connection, the
|
||||||
/// same way `process_batch_inserts` bounds the indexer's writes.
|
/// same way `process_batch_inserts` bounds the indexer's writes.
|
||||||
|
|
@ -239,17 +253,10 @@ mod tests {
|
||||||
|
|
||||||
impl Fixture {
|
impl Fixture {
|
||||||
fn new() -> Fixture {
|
fn new() -> Fixture {
|
||||||
let stamp = format!(
|
let scratch = crate::testutil::scratch_dir("incr");
|
||||||
"{}-{}",
|
let dir = scratch.join("tree");
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
);
|
|
||||||
let dir = std::env::temp_dir().join(format!("qs-incr-{}", stamp));
|
|
||||||
std::fs::create_dir_all(&dir).unwrap();
|
std::fs::create_dir_all(&dir).unwrap();
|
||||||
let db = std::env::temp_dir().join(format!("qs-incr-{}.sqlite", stamp));
|
let db = scratch.join("index.sqlite");
|
||||||
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
|
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
|
||||||
let config = Config::default();
|
let config = Config::default();
|
||||||
let ignore = IgnoreSet::compile(&config.indexing.ignore_patterns).unwrap();
|
let ignore = IgnoreSet::compile(&config.indexing.ignore_patterns).unwrap();
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -3,7 +3,6 @@ pub mod config;
|
||||||
pub mod content;
|
pub mod content;
|
||||||
pub mod coordinator;
|
pub mod coordinator;
|
||||||
pub mod db;
|
pub mod db;
|
||||||
pub mod document_extraction;
|
|
||||||
pub mod extract;
|
pub mod extract;
|
||||||
pub mod file_handling;
|
pub mod file_handling;
|
||||||
pub mod incremental;
|
pub mod incremental;
|
||||||
|
|
@ -17,6 +16,8 @@ pub mod search;
|
||||||
pub mod security;
|
pub mod security;
|
||||||
pub mod shutdown;
|
pub mod shutdown;
|
||||||
pub mod snippet;
|
pub mod snippet;
|
||||||
|
#[doc(hidden)]
|
||||||
|
pub mod testutil;
|
||||||
pub mod textenc;
|
pub mod textenc;
|
||||||
pub mod walk;
|
pub mod walk;
|
||||||
pub mod watcher;
|
pub mod watcher;
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,8 @@
|
||||||
//! exactly the ones a user needs when something looks wrong, and they were
|
//! exactly the ones a user needs when something looks wrong, and they were
|
||||||
//! going nowhere.
|
//! going nowhere.
|
||||||
//!
|
//!
|
||||||
//! So background reporting goes through [`log_info!`] and [`log_warn!`]
|
//! So background reporting goes through [`crate::log_info!`] and
|
||||||
|
//! [`crate::log_warn!`]
|
||||||
//! instead of `println!`/`eprintln!`: each writes the same line to stderr
|
//! instead of `println!`/`eprintln!`: each writes the same line to stderr
|
||||||
//! *and* appends it to a bounded ring the GUI's Logs tab reads. A terminal
|
//! *and* appends it to a bounded ring the GUI's Logs tab reads. A terminal
|
||||||
//! run looks exactly as it did; a windowed run gains the tab.
|
//! run looks exactly as it did; a windowed run gains the tab.
|
||||||
|
|
@ -66,7 +67,8 @@ macro_rules! log_warn {
|
||||||
|
|
||||||
/// Write `message` to stderr and to the ring.
|
/// Write `message` to stderr and to the ring.
|
||||||
///
|
///
|
||||||
/// Prefer the [`log_info!`] / [`log_warn!`] macros; this is what they call.
|
/// Prefer the [`crate::log_info!`] / [`crate::log_warn!`] macros; this is what
|
||||||
|
/// they call.
|
||||||
///
|
///
|
||||||
/// A failed stderr write is ignored rather than propagated: `eprintln!`
|
/// A failed stderr write is ignored rather than propagated: `eprintln!`
|
||||||
/// *panics* when the handle is unwritable, which on a process launched
|
/// *panics* when the handle is unwritable, which on a process launched
|
||||||
|
|
@ -82,6 +84,56 @@ pub fn record(level: Level, message: String) {
|
||||||
lock().push(level, text);
|
lock().push(level, text);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A cap on how many times one *kind* of warning is allowed to speak.
|
||||||
|
///
|
||||||
|
/// Some failures are per-file and arrive in the thousands: a directory tree the
|
||||||
|
/// user cannot read, or — the case that motivated this — a Windows machine where
|
||||||
|
/// `ERROR_SHARING_VIOLATION` from a file some other process holds open is
|
||||||
|
/// routine and has no Unix equivalent. Logging each one costs a global mutex and
|
||||||
|
/// an unbuffered stderr write on the walk's hottest path, and it evicts the
|
||||||
|
/// [`CAPACITY`]-line ring so thoroughly that the warnings worth reading are gone
|
||||||
|
/// before the run ends.
|
||||||
|
///
|
||||||
|
/// So the first few speak and the rest are counted. The count is the part that
|
||||||
|
/// actually informs: "3 files could not be hashed" and "31,402 files could not
|
||||||
|
/// be hashed" call for very different responses, and neither is legible as a
|
||||||
|
/// wall of identical lines.
|
||||||
|
///
|
||||||
|
/// Declared as a `static` next to the call site it guards, and reset by whoever
|
||||||
|
/// owns the run, so the numbers describe one run rather than the process.
|
||||||
|
pub struct Throttle {
|
||||||
|
limit: u64,
|
||||||
|
seen: std::sync::atomic::AtomicU64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Throttle {
|
||||||
|
pub const fn new(limit: u64) -> Throttle {
|
||||||
|
Throttle {
|
||||||
|
limit,
|
||||||
|
seen: std::sync::atomic::AtomicU64::new(0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Count one occurrence, and answer whether it may be logged individually.
|
||||||
|
pub fn allow(&self) -> bool {
|
||||||
|
self.seen.fetch_add(1, std::sync::atomic::Ordering::Relaxed) < self.limit
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Occurrences counted since the last [`Throttle::reset`].
|
||||||
|
pub fn seen(&self) -> u64 {
|
||||||
|
self.seen.load(std::sync::atomic::Ordering::Relaxed)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How many went unlogged — what a summary line should report.
|
||||||
|
pub fn suppressed(&self) -> u64 {
|
||||||
|
self.seen().saturating_sub(self.limit)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn reset(&self) {
|
||||||
|
self.seen.store(0, std::sync::atomic::Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Every retained line, oldest first.
|
/// Every retained line, oldest first.
|
||||||
pub fn snapshot() -> Vec<LogLine> {
|
pub fn snapshot() -> Vec<LogLine> {
|
||||||
lock().lines.iter().cloned().collect()
|
lock().lines.iter().cloned().collect()
|
||||||
|
|
@ -152,7 +204,17 @@ impl Ring {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn now_unix() -> u64 {
|
/// Seconds since the Unix epoch.
|
||||||
|
///
|
||||||
|
/// Lives here because this is the lowest-level module in the crate and every
|
||||||
|
/// layer above it wanted the same four lines — log lines, the `last_full_index`
|
||||||
|
/// stamp, `schema_info.created_at`, failure timestamps, and the GUI's
|
||||||
|
/// "5 min ago".
|
||||||
|
///
|
||||||
|
/// A clock set before 1970 yields `0` rather than an error. Every caller is
|
||||||
|
/// stamping a record or measuring an age, and none has anything better to do
|
||||||
|
/// with a failure than what `0` already does: read as "very long ago".
|
||||||
|
pub fn now_unix() -> u64 {
|
||||||
SystemTime::now()
|
SystemTime::now()
|
||||||
.duration_since(UNIX_EPOCH)
|
.duration_since(UNIX_EPOCH)
|
||||||
.map(|d| d.as_secs())
|
.map(|d| d.as_secs())
|
||||||
|
|
@ -215,6 +277,33 @@ mod tests {
|
||||||
assert_eq!(ring.recorded, 4, "the running total survives a clear");
|
assert_eq!(ring.recorded, 4, "the running total survives a clear");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_throttle_lets_the_first_few_through_and_counts_the_rest() {
|
||||||
|
let t = Throttle::new(3);
|
||||||
|
assert_eq!((t.seen(), t.suppressed()), (0, 0));
|
||||||
|
for _ in 0..3 {
|
||||||
|
assert!(t.allow(), "the first `limit` occurrences speak");
|
||||||
|
}
|
||||||
|
for _ in 0..7 {
|
||||||
|
assert!(!t.allow(), "the rest are counted only");
|
||||||
|
}
|
||||||
|
assert_eq!(t.seen(), 10);
|
||||||
|
assert_eq!(t.suppressed(), 7);
|
||||||
|
|
||||||
|
t.reset();
|
||||||
|
assert_eq!((t.seen(), t.suppressed()), (0, 0));
|
||||||
|
assert!(t.allow(), "a reset throttle speaks again");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A zero limit must silence rather than divide by anything.
|
||||||
|
#[test]
|
||||||
|
fn a_zero_limit_throttle_logs_nothing() {
|
||||||
|
let t = Throttle::new(0);
|
||||||
|
assert!(!t.allow());
|
||||||
|
assert_eq!(t.seen(), 1);
|
||||||
|
assert_eq!(t.suppressed(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
/// Through the global: the macros must land in the snapshot, and a
|
/// Through the global: the macros must land in the snapshot, and a
|
||||||
/// warning must carry the prefix its terminal line has. Written to
|
/// warning must carry the prefix its terminal line has. Written to
|
||||||
/// tolerate lines from tests running in parallel in this process.
|
/// tolerate lines from tests running in parallel in this process.
|
||||||
|
|
|
||||||
|
|
@ -29,12 +29,72 @@ pub fn home_dir() -> Option<OsString> {
|
||||||
std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE"))
|
std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE"))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether a directory entry counts as hidden.
|
/// `FILE_ATTRIBUTE_HIDDEN`.
|
||||||
///
|
///
|
||||||
/// Unix: a leading dot. Windows: a leading dot **or** `FILE_ATTRIBUTE_HIDDEN`
|
/// Spelled out rather than imported, for the reason in the module header:
|
||||||
/// / `FILE_ATTRIBUTE_SYSTEM` — without which `include_hidden = false` hides
|
/// `windows-sys` is a `cfg(windows)`-only dependency, and
|
||||||
/// nothing on Windows, and `$RECYCLE.BIN`, `System Volume Information`,
|
/// [`attributes_are_hidden`] has to compile — and be tested — on Linux, where
|
||||||
/// `pagefile.sys` and `AppData` all get indexed.
|
/// the suite runs.
|
||||||
|
const FILE_ATTRIBUTE_HIDDEN: u32 = 0x2;
|
||||||
|
|
||||||
|
/// The cross-compiled Windows build is where the spelling above is checked
|
||||||
|
/// against the real header value, so a typo fails that job rather than quietly
|
||||||
|
/// indexing `$RECYCLE.BIN`.
|
||||||
|
#[cfg(windows)]
|
||||||
|
const _: () = assert!(
|
||||||
|
FILE_ATTRIBUTE_HIDDEN == windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_HIDDEN
|
||||||
|
);
|
||||||
|
|
||||||
|
/// Whether a Windows attribute word marks an entry hidden.
|
||||||
|
///
|
||||||
|
/// `FILE_ATTRIBUTE_HIDDEN` and nothing else. `FILE_ATTRIBUTE_SYSTEM` was part
|
||||||
|
/// of this test and was removed, because it pruned users' cloud folders:
|
||||||
|
/// Windows honours the `desktop.ini` inside a folder only if the folder itself
|
||||||
|
/// carries Read-only or System, so the ownCloud, Nextcloud, OneDrive and Google
|
||||||
|
/// Drive clients set System on their sync root purely to get a branded icon.
|
||||||
|
/// Such a folder has no Hidden bit and is plainly visible in Explorer, yet the
|
||||||
|
/// whole subtree vanished from the index with nothing to explain it — only
|
||||||
|
/// "Index hidden files" brought it back, which is the opposite of what that
|
||||||
|
/// setting is for. Any folder given a custom icon is the same case.
|
||||||
|
///
|
||||||
|
/// Nothing the System term existed for is lost. `$RECYCLE.BIN`, `System Volume
|
||||||
|
/// Information`, `pagefile.sys` and the legacy per-user junctions (`My
|
||||||
|
/// Documents`, `Local Settings`, `Application Data`) are Hidden **and** System —
|
||||||
|
/// that pairing is Windows' own definition of a protected operating system file
|
||||||
|
/// — and `AppData` is Hidden alone. Hidden catches every one. The drive-root
|
||||||
|
/// names are excluded by [`crate::config`]'s default ignore patterns as well, so
|
||||||
|
/// they have two independent reasons to stay out.
|
||||||
|
#[cfg_attr(not(windows), allow(dead_code))]
|
||||||
|
pub(crate) fn attributes_are_hidden(attributes: u32) -> bool {
|
||||||
|
attributes & FILE_ATTRIBUTE_HIDDEN != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Why an entry counted as hidden.
|
||||||
|
///
|
||||||
|
/// The distinction exists for the walk's log line: a dot prefix explains itself
|
||||||
|
/// and an ignore pattern is something the user typed, but "this visible folder
|
||||||
|
/// was skipped over an attribute you cannot see" has no discoverability at all,
|
||||||
|
/// so only that case is worth reporting.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum HiddenReason {
|
||||||
|
DotPrefix,
|
||||||
|
Attribute,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether a directory entry counts as hidden, and why.
|
||||||
|
///
|
||||||
|
/// Unix: a leading dot. Windows: a leading dot **or** `FILE_ATTRIBUTE_HIDDEN` —
|
||||||
|
/// without which `include_hidden = false` hides nothing on Windows, and
|
||||||
|
/// `$RECYCLE.BIN`, `System Volume Information`, `pagefile.sys` and `AppData` all
|
||||||
|
/// get indexed. `FILE_ATTRIBUTE_SYSTEM` is deliberately not part of it; see
|
||||||
|
/// [`attributes_are_hidden`].
|
||||||
|
///
|
||||||
|
/// `meta` must report the attributes of the entry **itself**, never of a link
|
||||||
|
/// target — callers pass `symlink_metadata` or an already-cached directory
|
||||||
|
/// entry. A hidden symlink is a hidden alias; a visible symlink to a hidden
|
||||||
|
/// target is a visible alias. All four call sites have to agree on that, or a
|
||||||
|
/// full run indexes a file the watcher then refuses to update and the index
|
||||||
|
/// churns on every cycle.
|
||||||
///
|
///
|
||||||
/// `meta` is a closure because on Unix it is never called: the walkers
|
/// `meta` is a closure because on Unix it is never called: the walkers
|
||||||
/// deliberately avoid `metadata()`, which would cost an extra `lstat` per
|
/// deliberately avoid `metadata()`, which would cost an extra `lstat` per
|
||||||
|
|
@ -42,28 +102,207 @@ pub fn home_dir() -> Option<OsString> {
|
||||||
/// zero anyway — both `std::fs::DirEntry::metadata` and
|
/// zero anyway — both `std::fs::DirEntry::metadata` and
|
||||||
/// `walkdir::DirEntry::metadata` hand back data already cached from
|
/// `walkdir::DirEntry::metadata` hand back data already cached from
|
||||||
/// `FindNextFileW`.
|
/// `FindNextFileW`.
|
||||||
pub fn entry_is_hidden<F>(name: &str, meta: F) -> bool
|
pub fn entry_hidden_reason<F>(name: &str, meta: F) -> Option<HiddenReason>
|
||||||
where
|
where
|
||||||
F: FnOnce() -> Option<std::fs::Metadata>,
|
F: FnOnce() -> Option<std::fs::Metadata>,
|
||||||
{
|
{
|
||||||
if name.starts_with('.') {
|
if name.starts_with('.') {
|
||||||
return true;
|
return Some(HiddenReason::DotPrefix);
|
||||||
}
|
}
|
||||||
#[cfg(windows)]
|
#[cfg(windows)]
|
||||||
{
|
{
|
||||||
use std::os::windows::fs::MetadataExt;
|
use std::os::windows::fs::MetadataExt;
|
||||||
use windows_sys::Win32::Storage::FileSystem::{
|
|
||||||
FILE_ATTRIBUTE_HIDDEN, FILE_ATTRIBUTE_SYSTEM,
|
|
||||||
};
|
|
||||||
if let Some(m) = meta() {
|
if let Some(m) = meta() {
|
||||||
return m.file_attributes() & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) != 0;
|
if attributes_are_hidden(m.file_attributes()) {
|
||||||
|
return Some(HiddenReason::Attribute);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
#[cfg(not(windows))]
|
#[cfg(not(windows))]
|
||||||
{
|
{
|
||||||
let _ = meta;
|
let _ = meta;
|
||||||
}
|
}
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`entry_hidden_reason`] for the callers that only need the verdict.
|
||||||
|
pub fn entry_is_hidden<F>(name: &str, meta: F) -> bool
|
||||||
|
where
|
||||||
|
F: FnOnce() -> Option<std::fs::Metadata>,
|
||||||
|
{
|
||||||
|
entry_hidden_reason(name, meta).is_some()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `FILE_ATTRIBUTE_REPARSE_POINT`. Spelled out for [`FILE_ATTRIBUTE_HIDDEN`]'s
|
||||||
|
/// reason, and checked against the real header value the same way.
|
||||||
|
const FILE_ATTRIBUTE_REPARSE_POINT: u32 = 0x400;
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
const _: () = assert!(
|
||||||
|
FILE_ATTRIBUTE_REPARSE_POINT
|
||||||
|
== windows_sys::Win32::Storage::FileSystem::FILE_ATTRIBUTE_REPARSE_POINT
|
||||||
|
);
|
||||||
|
|
||||||
|
/// The attributes a dehydrated cloud file carries.
|
||||||
|
///
|
||||||
|
/// `FILE_ATTRIBUTE_OFFLINE` (0x1000) is the old tape-archive bit that OneDrive
|
||||||
|
/// reused; `RECALL_ON_OPEN` (0x40000) marks a file whose *metadata* is local but
|
||||||
|
/// whose data is not; `RECALL_ON_DATA_ACCESS` (0x400000) is the modern
|
||||||
|
/// Files-On-Demand placeholder. Any one of them means opening the file for read
|
||||||
|
/// pulls it over the network.
|
||||||
|
const FILE_ATTRIBUTE_OFFLINE: u32 = 0x1000;
|
||||||
|
const FILE_ATTRIBUTE_RECALL_ON_OPEN: u32 = 0x4_0000;
|
||||||
|
const FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS: u32 = 0x40_0000;
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
const _: () = {
|
||||||
|
use windows_sys::Win32::Storage::FileSystem as fs_attrs;
|
||||||
|
assert!(FILE_ATTRIBUTE_OFFLINE == fs_attrs::FILE_ATTRIBUTE_OFFLINE);
|
||||||
|
assert!(FILE_ATTRIBUTE_RECALL_ON_OPEN == fs_attrs::FILE_ATTRIBUTE_RECALL_ON_OPEN);
|
||||||
|
assert!(FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS == fs_attrs::FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS);
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Whether reading this file's contents would pull it down from the cloud.
|
||||||
|
///
|
||||||
|
/// OneDrive, and every other Files-On-Demand provider, leaves a placeholder on
|
||||||
|
/// disk with real metadata and no data. Opening one for read is not a local
|
||||||
|
/// operation: it blocks on a network download of the entire file. The default
|
||||||
|
/// indexing root is `%USERPROFILE%`, the OneDrive folder beneath it is not
|
||||||
|
/// hidden, and the walk hashes the first 8 KiB of every new or changed file — so
|
||||||
|
/// without this test a first index quietly downloads the user's whole cloud
|
||||||
|
/// drive, filling their disk with the files they had deliberately offloaded.
|
||||||
|
///
|
||||||
|
/// Named for the question the caller is actually asking ("will reading this
|
||||||
|
/// cost a download?") rather than for the bits, because the answer is what the
|
||||||
|
/// indexing path branches on. Always `false` off Windows: no other platform this
|
||||||
|
/// runs on has an equivalent, and a `stat` there tells the truth about the data.
|
||||||
|
pub fn is_cloud_placeholder(meta: &std::fs::Metadata) -> bool {
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
use std::os::windows::fs::MetadataExt;
|
||||||
|
return attributes_are_dehydrated(meta.file_attributes());
|
||||||
|
}
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
{
|
||||||
|
let _ = meta;
|
||||||
false
|
false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The bit test behind [`is_cloud_placeholder`], split out so the Linux suite
|
||||||
|
/// exercises it. See this module's second rule.
|
||||||
|
#[cfg_attr(not(windows), allow(dead_code))]
|
||||||
|
pub(crate) fn attributes_are_dehydrated(attributes: u32) -> bool {
|
||||||
|
attributes
|
||||||
|
& (FILE_ATTRIBUTE_OFFLINE
|
||||||
|
| FILE_ATTRIBUTE_RECALL_ON_OPEN
|
||||||
|
| FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS)
|
||||||
|
!= 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether a Windows attribute word marks an entry as a reparse point.
|
||||||
|
///
|
||||||
|
/// Split out from [`entry_cached_metadata`] so the bit test is exercised by the
|
||||||
|
/// Linux suite, per this module's second rule. Deliberately *not* the same
|
||||||
|
/// question as `FileType::is_symlink`, which additionally requires the
|
||||||
|
/// name-surrogate bit: see [`entry_cached_metadata`] for why that distinction is
|
||||||
|
/// the whole point.
|
||||||
|
#[cfg_attr(not(windows), allow(dead_code))]
|
||||||
|
pub(crate) fn attributes_are_reparse_point(attributes: u32) -> bool {
|
||||||
|
attributes & FILE_ATTRIBUTE_REPARSE_POINT != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Metadata a directory read already handed back, on the platforms that hand
|
||||||
|
/// any back at all.
|
||||||
|
///
|
||||||
|
/// `std::fs::Metadata` on Windows: `FindFirstFileW`/`FindNextFileW` return size,
|
||||||
|
/// timestamps and attributes with every name, and `std::fs::DirEntry::metadata`
|
||||||
|
/// is a copy of that buffer rather than a syscall. A `fs::metadata(path)` on the
|
||||||
|
/// same entry is therefore a whole extra `CreateFileW` +
|
||||||
|
/// `GetFileInformationByHandle` + `CloseHandle` — an `IRP_MJ_CREATE` through
|
||||||
|
/// every antivirus and EDR minifilter on the machine — for data already in hand.
|
||||||
|
///
|
||||||
|
/// Uninhabited everywhere else, because `getdents64` returns only `d_type`: size
|
||||||
|
/// and mtime genuinely need the `statx`, and `DirEntry::metadata` there is a
|
||||||
|
/// *second* syscall rather than a saved one. Uninhabited rather than a unit
|
||||||
|
/// struct so `Option<CachedMetadata>` is zero-sized off Windows — the walker
|
||||||
|
/// queues one per pending file and nothing throttles file chunks (see
|
||||||
|
/// `walk::Job::Files`), so carrying `Option<std::fs::Metadata>` would cost Linux
|
||||||
|
/// 176 bytes per queued file to hold nothing at all.
|
||||||
|
#[cfg(windows)]
|
||||||
|
pub(crate) type CachedMetadata = std::fs::Metadata;
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
pub(crate) type CachedMetadata = std::convert::Infallible;
|
||||||
|
|
||||||
|
/// The zero-cost half of the claim above, enforced rather than asserted in
|
||||||
|
/// prose: a layout change in a future toolchain becomes a compile error instead
|
||||||
|
/// of a silent regression in the walker's memory profile.
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
const _: () = assert!(std::mem::size_of::<Option<CachedMetadata>>() == 0);
|
||||||
|
|
||||||
|
/// The metadata a directory read already produced for one entry — but only
|
||||||
|
/// where trusting it is both free *and* indistinguishable from a fresh `stat`.
|
||||||
|
///
|
||||||
|
/// `meta` is a closure for exactly [`entry_hidden_reason`]'s reason: on Unix it
|
||||||
|
/// is never called, because `std::fs::DirEntry::metadata` there is a real
|
||||||
|
/// `lstat` and the walk spends exactly one `statx` per file by design.
|
||||||
|
///
|
||||||
|
/// `None` for a reparse point even on Windows, and that is the whole subtlety.
|
||||||
|
/// `fs::metadata` *follows* a reparse point; the cached buffer describes the
|
||||||
|
/// link itself. The tags std does not classify as symlinks reach the walk's
|
||||||
|
/// ordinary file arm — `IO_REPARSE_TAG_APPEXECLINK`, the zero-byte Store app
|
||||||
|
/// stubs under `WindowsApps`, and the OneDrive `IO_REPARSE_TAG_CLOUD_*` family —
|
||||||
|
/// and for those the two answers differ: today an AppExecLink fails
|
||||||
|
/// `CreateFileW` with `ERROR_CANT_ACCESS_FILE` and is skipped, while its
|
||||||
|
/// directory entry looks like an ordinary empty file and would get indexed.
|
||||||
|
/// Sending every reparse point back to the path-based `fs::metadata` keeps the
|
||||||
|
/// semantics identical and leaves only the common case on the fast path.
|
||||||
|
///
|
||||||
|
/// Tested against the raw `FILE_ATTRIBUTE_REPARSE_POINT` bit, deliberately not
|
||||||
|
/// against `file_type().is_symlink()`: std's `is_symlink` additionally requires
|
||||||
|
/// the name-surrogate bit `0x20000000`, which is precisely the test that lets
|
||||||
|
/// AppExecLink and the cloud tags through.
|
||||||
|
pub(crate) fn entry_cached_metadata<F>(meta: F) -> Option<CachedMetadata>
|
||||||
|
where
|
||||||
|
F: FnOnce() -> Option<std::fs::Metadata>,
|
||||||
|
{
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
use std::os::windows::fs::MetadataExt;
|
||||||
|
if let Some(m) = meta().filter(|m| !attributes_are_reparse_point(m.file_attributes())) {
|
||||||
|
return Some(m);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
{
|
||||||
|
let _ = meta;
|
||||||
|
}
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A file's `std::fs::Metadata`, from the directory read where that read
|
||||||
|
/// supplied it and from a `stat` where it did not.
|
||||||
|
///
|
||||||
|
/// The `#[cfg]` lives here rather than at the call site, per this module's first
|
||||||
|
/// rule. Off Windows `cached` is uninhabited and therefore provably `None`,
|
||||||
|
/// which is why the walk still costs exactly one `statx` per file there.
|
||||||
|
#[cfg(windows)]
|
||||||
|
pub(crate) fn metadata_or_stat(
|
||||||
|
path: &Path,
|
||||||
|
cached: Option<CachedMetadata>,
|
||||||
|
) -> std::io::Result<std::fs::Metadata> {
|
||||||
|
match cached {
|
||||||
|
Some(m) => Ok(m),
|
||||||
|
None => std::fs::metadata(path),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
pub(crate) fn metadata_or_stat(
|
||||||
|
path: &Path,
|
||||||
|
_cached: Option<CachedMetadata>,
|
||||||
|
) -> std::io::Result<std::fs::Metadata> {
|
||||||
|
std::fs::metadata(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether `path` has a hidden component *below* the root that contains it.
|
/// Whether `path` has a hidden component *below* the root that contains it.
|
||||||
|
|
@ -101,7 +340,11 @@ pub fn path_has_hidden_component_under(path: &Path, roots: &[PathBuf]) -> bool {
|
||||||
current.push(component);
|
current.push(component);
|
||||||
if let Component::Normal(name) = component {
|
if let Component::Normal(name) = component {
|
||||||
let name = name.to_string_lossy();
|
let name = name.to_string_lossy();
|
||||||
if entry_is_hidden(&name, || std::fs::metadata(¤t).ok()) {
|
// `symlink_metadata`, not `metadata`: each component is judged as
|
||||||
|
// itself, which is what the walkers do. Only the final component
|
||||||
|
// stops being followed, so intermediate ones still resolve
|
||||||
|
// normally. See `entry_hidden_reason`.
|
||||||
|
if entry_is_hidden(&name, || std::fs::symlink_metadata(¤t).ok()) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -234,6 +477,15 @@ pub const WATCH_ROOTS_RECURSIVELY: bool = cfg!(windows);
|
||||||
/// on both sides, consistently.
|
/// on both sides, consistently.
|
||||||
pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" };
|
pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" };
|
||||||
|
|
||||||
|
/// Whether this platform's filesystem matches names without regard to case.
|
||||||
|
///
|
||||||
|
/// What ignore patterns compile against: on Windows and macOS `node_modules`
|
||||||
|
/// has to exclude `Node_Modules`, and on Linux it must not. Named here rather
|
||||||
|
/// than spelled `cfg!(any(windows, target_os = "macos"))` at each use, so the
|
||||||
|
/// two places that must agree — [`crate::config::IgnoreSet`]'s literal set and
|
||||||
|
/// its glob set — cannot drift apart.
|
||||||
|
pub const PATHS_ARE_CASE_INSENSITIVE: bool = cfg!(any(windows, target_os = "macos"));
|
||||||
|
|
||||||
/// Drop the **calling thread** to background scheduling priority.
|
/// Drop the **calling thread** to background scheduling priority.
|
||||||
///
|
///
|
||||||
/// Per-thread, not per-process. The GUI shares this process, so lowering the
|
/// Per-thread, not per-process. The GUI shares this process, so lowering the
|
||||||
|
|
@ -245,6 +497,19 @@ pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" }
|
||||||
///
|
///
|
||||||
/// Best-effort and idempotent: a refusal is not worth reporting, since the
|
/// Best-effort and idempotent: a refusal is not worth reporting, since the
|
||||||
/// only consequence is that indexing competes on equal terms.
|
/// only consequence is that indexing competes on equal terms.
|
||||||
|
///
|
||||||
|
/// **CPU only, on every platform.** This used to be
|
||||||
|
/// `THREAD_MODE_BACKGROUND_BEGIN` on Windows, which is background *mode*: it
|
||||||
|
/// lowers the thread to base priority 4 and drops it to `IoPriorityVeryLow`, a
|
||||||
|
/// tier the kernel does not merely deprioritise but actively rate-limits — the
|
||||||
|
/// same one SuperFetch and defrag run in. Linux's `nice` has no I/O half at all
|
||||||
|
/// under the usual schedulers, so the two platforms were not doing remotely the
|
||||||
|
/// same thing: every walker, prefetcher, extractor *and* the single SQLite
|
||||||
|
/// writer thread were throttled on Windows and full-speed on Linux, which is
|
||||||
|
/// most of why Windows indexing was so much slower. Matching `nice`'s CPU-only
|
||||||
|
/// semantics is the deliberate choice; a machine that genuinely needs I/O
|
||||||
|
/// throttling needs it as a user-visible setting, not as a silent per-platform
|
||||||
|
/// difference.
|
||||||
pub fn set_background_priority() {
|
pub fn set_background_priority() {
|
||||||
#[cfg(target_os = "linux")]
|
#[cfg(target_os = "linux")]
|
||||||
{
|
{
|
||||||
|
|
@ -256,12 +521,12 @@ pub fn set_background_priority() {
|
||||||
#[cfg(windows)]
|
#[cfg(windows)]
|
||||||
{
|
{
|
||||||
use windows_sys::Win32::System::Threading::{
|
use windows_sys::Win32::System::Threading::{
|
||||||
GetCurrentThread, SetThreadPriority, THREAD_MODE_BACKGROUND_BEGIN,
|
GetCurrentThread, SetThreadPriority, THREAD_PRIORITY_BELOW_NORMAL,
|
||||||
};
|
};
|
||||||
// Background *mode*, not merely a lower priority number: it drops I/O
|
// Yields the CPU to the foreground without touching I/O priority — the
|
||||||
// priority as well, which is what actually keeps a walk from starving
|
// closest Windows equivalent of `nice(10)`. See the note above for why
|
||||||
// the foreground on a spinning disk.
|
// this is not `THREAD_MODE_BACKGROUND_BEGIN`.
|
||||||
unsafe { SetThreadPriority(GetCurrentThread(), THREAD_MODE_BACKGROUND_BEGIN) };
|
unsafe { SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_BELOW_NORMAL) };
|
||||||
}
|
}
|
||||||
// Elsewhere (macOS, BSD): deliberately nothing. `nice` there applies to the
|
// Elsewhere (macOS, BSD): deliberately nothing. `nice` there applies to the
|
||||||
// whole process, so it would hit the GUI. The right call is
|
// whole process, so it would hit the GUI. The right call is
|
||||||
|
|
@ -426,12 +691,165 @@ mod tests {
|
||||||
assert!(!called, "a dot prefix must short-circuit before any stat");
|
assert!(!called, "a dot prefix must short-circuit before any stat");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Each of the three attributes on its own must be enough: providers do not
|
||||||
|
/// agree on which they set, and getting this wrong means silently
|
||||||
|
/// downloading someone's entire cloud drive.
|
||||||
|
#[test]
|
||||||
|
fn any_recall_attribute_marks_a_file_dehydrated() {
|
||||||
|
for bit in [OFFLINE, RECALL_ON_OPEN, RECALL_ON_DATA_ACCESS] {
|
||||||
|
assert!(attributes_are_dehydrated(bit));
|
||||||
|
assert!(attributes_are_dehydrated(ARCHIVE | REPARSE_POINT | bit));
|
||||||
|
}
|
||||||
|
// A synced-down file keeps the reparse point but drops the recall bits.
|
||||||
|
assert!(!attributes_are_dehydrated(ARCHIVE | REPARSE_POINT));
|
||||||
|
assert!(!attributes_are_dehydrated(ARCHIVE));
|
||||||
|
assert!(!attributes_are_dehydrated(NORMAL));
|
||||||
|
assert!(!attributes_are_dehydrated(0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Off Windows there is no such thing, and a local `stat` tells the truth.
|
||||||
|
#[test]
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
fn nothing_is_a_cloud_placeholder_off_windows() {
|
||||||
|
let meta = std::fs::metadata(env!("CARGO_MANIFEST_DIR")).unwrap();
|
||||||
|
assert!(!is_cloud_placeholder(&meta));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The bit test behind the walk's fast path, exercised where the suite
|
||||||
|
/// actually runs. A junction, an AppExecLink stub and a OneDrive
|
||||||
|
/// placeholder all carry this bit; an ordinary file does not.
|
||||||
|
#[test]
|
||||||
|
fn reparse_points_are_recognised_by_attribute() {
|
||||||
|
assert!(attributes_are_reparse_point(REPARSE_POINT));
|
||||||
|
assert!(attributes_are_reparse_point(DIRECTORY | REPARSE_POINT));
|
||||||
|
// A dehydrated cloud file: reparse point plus the recall attributes.
|
||||||
|
assert!(attributes_are_reparse_point(
|
||||||
|
ARCHIVE | REPARSE_POINT | 0x40_0000
|
||||||
|
));
|
||||||
|
assert!(!attributes_are_reparse_point(ARCHIVE));
|
||||||
|
assert!(!attributes_are_reparse_point(NORMAL));
|
||||||
|
assert!(!attributes_are_reparse_point(DIRECTORY));
|
||||||
|
assert!(!attributes_are_reparse_point(0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Off Windows the directory read supplies nothing, and asking it for
|
||||||
|
/// anything would be the `lstat` per entry the walker exists to avoid.
|
||||||
|
#[test]
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
fn nothing_is_served_from_a_directory_read_off_windows() {
|
||||||
|
let mut called = false;
|
||||||
|
let got = entry_cached_metadata(|| {
|
||||||
|
called = true;
|
||||||
|
None
|
||||||
|
});
|
||||||
|
assert!(got.is_none());
|
||||||
|
assert!(
|
||||||
|
!called,
|
||||||
|
"DirEntry::metadata here is an lstat, which is the syscall the walk exists to avoid"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `fs::metadata` follows a reparse point and the cached buffer does not,
|
||||||
|
/// so the fast path must decline every one of them — including the tags std
|
||||||
|
/// does not call symlinks, which are precisely the ones that reach the
|
||||||
|
/// walk's ordinary file arm.
|
||||||
|
#[test]
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn a_reparse_point_is_never_served_from_the_directory_read() {
|
||||||
|
let dir = std::env::temp_dir().join(format!("qs-reparse-{}", std::process::id()));
|
||||||
|
let target = dir.join("target");
|
||||||
|
let link = dir.join("link");
|
||||||
|
let plain = dir.join("plain.txt");
|
||||||
|
std::fs::create_dir_all(&target).unwrap();
|
||||||
|
std::fs::write(&plain, b"x").unwrap();
|
||||||
|
|
||||||
|
let made = std::process::Command::new("cmd")
|
||||||
|
.args(["/C", "mklink", "/J"])
|
||||||
|
.arg(&link)
|
||||||
|
.arg(&target)
|
||||||
|
.output()
|
||||||
|
.map(|o| o.status.success())
|
||||||
|
.unwrap_or(false);
|
||||||
|
if made {
|
||||||
|
let m = std::fs::symlink_metadata(&link).unwrap();
|
||||||
|
assert!(
|
||||||
|
entry_cached_metadata(|| Some(m)).is_none(),
|
||||||
|
"a junction must fall back to the path-based stat"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let m = std::fs::metadata(&plain).unwrap();
|
||||||
|
assert!(
|
||||||
|
entry_cached_metadata(|| Some(m)).is_some(),
|
||||||
|
"an ordinary file must take the fast path"
|
||||||
|
);
|
||||||
|
std::fs::remove_dir_all(&dir).ok();
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn ordinary_names_are_not_hidden() {
|
fn ordinary_names_are_not_hidden() {
|
||||||
assert!(!entry_is_hidden("Documents", || None));
|
assert!(!entry_is_hidden("Documents", || None));
|
||||||
assert!(!entry_is_hidden("report.txt", || None));
|
assert!(!entry_is_hidden("report.txt", || None));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The real `FILE_ATTRIBUTE_*` bits, so the cases below read as the files
|
||||||
|
/// they stand for.
|
||||||
|
const READONLY: u32 = 0x1;
|
||||||
|
const HIDDEN: u32 = 0x2;
|
||||||
|
const SYSTEM: u32 = 0x4;
|
||||||
|
const DIRECTORY: u32 = 0x10;
|
||||||
|
const ARCHIVE: u32 = 0x20;
|
||||||
|
const NORMAL: u32 = 0x80;
|
||||||
|
const REPARSE_POINT: u32 = 0x400;
|
||||||
|
const OFFLINE: u32 = 0x1000;
|
||||||
|
const RECALL_ON_OPEN: u32 = 0x4_0000;
|
||||||
|
const RECALL_ON_DATA_ACCESS: u32 = 0x40_0000;
|
||||||
|
|
||||||
|
/// The attribute half of `entry_is_hidden`, which the Windows arm cannot
|
||||||
|
/// be asked about from Linux. Split out precisely so this test runs
|
||||||
|
/// everywhere.
|
||||||
|
#[test]
|
||||||
|
fn only_the_hidden_bit_hides_an_entry() {
|
||||||
|
// AppData: Hidden alone, and `std::env::temp_dir()` lives under it.
|
||||||
|
assert!(attributes_are_hidden(HIDDEN | DIRECTORY));
|
||||||
|
// $RECYCLE.BIN, System Volume Information, and the legacy per-user
|
||||||
|
// junctions: Hidden+System, Windows' own definition of a protected
|
||||||
|
// operating system file.
|
||||||
|
assert!(attributes_are_hidden(HIDDEN | SYSTEM | DIRECTORY));
|
||||||
|
// pagefile.sys.
|
||||||
|
assert!(attributes_are_hidden(HIDDEN | SYSTEM | ARCHIVE));
|
||||||
|
|
||||||
|
assert!(!attributes_are_hidden(0));
|
||||||
|
assert!(!attributes_are_hidden(NORMAL));
|
||||||
|
assert!(!attributes_are_hidden(DIRECTORY));
|
||||||
|
assert!(!attributes_are_hidden(READONLY | DIRECTORY));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Regression: a cloud sync root carries System and *not* Hidden — Windows
|
||||||
|
/// will not honour the `desktop.ini` supplying its branded icon otherwise —
|
||||||
|
/// and is fully visible in Explorer. Keying on System pruned the folder and
|
||||||
|
/// every file beneath it, silently, and only "include hidden files" brought
|
||||||
|
/// it back.
|
||||||
|
#[test]
|
||||||
|
fn a_sync_root_marked_system_but_not_hidden_is_not_hidden() {
|
||||||
|
assert!(!attributes_are_hidden(SYSTEM | DIRECTORY));
|
||||||
|
// Read-only is the other attribute that enables desktop.ini, and
|
||||||
|
// Explorer sets it when a user picks a custom folder icon.
|
||||||
|
assert!(!attributes_are_hidden(READONLY | SYSTEM | DIRECTORY));
|
||||||
|
assert!(!attributes_are_hidden(SYSTEM));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The walk announces an attribute prune and stays quiet about a dot
|
||||||
|
/// prefix, so the two must stay distinguishable.
|
||||||
|
#[test]
|
||||||
|
fn a_dot_prefix_reports_itself_as_the_reason() {
|
||||||
|
assert_eq!(
|
||||||
|
entry_hidden_reason(".git", || None),
|
||||||
|
Some(HiddenReason::DotPrefix)
|
||||||
|
);
|
||||||
|
assert_eq!(entry_hidden_reason("Documents", || None), None);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn hidden_components_are_measured_from_the_innermost_root() {
|
fn hidden_components_are_measured_from_the_innermost_root() {
|
||||||
let root = PathBuf::from(format!("{}.config", sep_prefix()));
|
let root = PathBuf::from(format!("{}.config", sep_prefix()));
|
||||||
|
|
|
||||||
|
|
@ -236,7 +236,8 @@ impl TermPattern {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Case-insensitive [`find_first`] against an already-folded haystack.
|
/// Case-insensitive [`TermPattern::find_first`] against an already-folded
|
||||||
|
/// haystack.
|
||||||
///
|
///
|
||||||
/// The literal path would otherwise fold the haystack itself, and the
|
/// The literal path would otherwise fold the haystack itself, and the
|
||||||
/// cascade's full-text passes need the same fold for counting, searching
|
/// cascade's full-text passes need the same fold for counting, searching
|
||||||
|
|
@ -256,8 +257,8 @@ impl TermPattern {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Case-insensitive [`count`] against an already-folded haystack. See
|
/// Case-insensitive [`TermPattern::count`] against an already-folded
|
||||||
/// [`TermPattern::find_first_folded`].
|
/// haystack. See [`TermPattern::find_first_folded`].
|
||||||
pub fn count_folded(&self, folded: &str) -> usize {
|
pub fn count_folded(&self, folded: &str) -> usize {
|
||||||
match self {
|
match self {
|
||||||
TermPattern::Empty => 0,
|
TermPattern::Empty => 0,
|
||||||
|
|
|
||||||
|
|
@ -28,8 +28,25 @@
|
||||||
//! indexed, has no owning root, and so has no filtering rules that can be
|
//! indexed, has no owning root, and so has no filtering rules that can be
|
||||||
//! applied to it — scanning by range means it is simply never visited, the
|
//! applied to it — scanning by range means it is simply never visited, the
|
||||||
//! same exemption `aliased_paths` gives it during a full run's stale sweep.
|
//! same exemption `aliased_paths` gives it during a full run's stale sweep.
|
||||||
|
//!
|
||||||
|
//! ## Reporting and giving up
|
||||||
|
//!
|
||||||
|
//! On a multi-million-row index this pass is minutes of work with no files
|
||||||
|
//! moving to show for it, so the cursor counts what it has examined against
|
||||||
|
//! the rows in the index and its driver publishes that
|
||||||
|
//! ([`crate::indexing::ReconcileProgress`]).
|
||||||
|
//!
|
||||||
|
//! It can also be abandoned, through the flag-plus-interrupt pair
|
||||||
|
//! [`crate::db::InterruptSlot`] describes — before this, closing the window
|
||||||
|
//! during a prune waited the prune out. What makes giving up safe is that
|
||||||
|
//! nothing here records anything: the stored configuration is stamped by the
|
||||||
|
//! *caller*, only once the cursor reports itself finished, so an abandoned
|
||||||
|
//! pass leaves the index describing the settings it was last reconciled to and
|
||||||
|
//! the next run derives the same plan again. See [`outstanding_work`], which
|
||||||
|
//! is that question asked directly.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use rusqlite::Connection;
|
use rusqlite::Connection;
|
||||||
|
|
@ -38,12 +55,17 @@ use crate::config::{Config, IgnoreSet, IndexWork};
|
||||||
use crate::db::repo;
|
use crate::db::repo;
|
||||||
use crate::extract::Registry;
|
use crate::extract::Registry;
|
||||||
use crate::file_handling::{content_extractable, fts_finalize_after_text_indexing, ExtractCursor};
|
use crate::file_handling::{content_extractable, fts_finalize_after_text_indexing, ExtractCursor};
|
||||||
|
use crate::indexing::ReconcileProgress;
|
||||||
|
|
||||||
/// How long [`advance`] may work before handing control back.
|
/// How long [`advance`] may work before handing control back.
|
||||||
///
|
///
|
||||||
/// Not a throughput knob — the caller decides when to come back — but a bound
|
/// Not a throughput knob — the caller decides when to come back — but a bound
|
||||||
/// on how long a Stop, a search or a further config edit waits behind a scan
|
/// on how long a Stop, a search or a further config edit waits behind a scan
|
||||||
/// in progress. The same budget the coordinator gives its watcher queue.
|
/// in progress. The same budget the coordinator gives its watcher queue.
|
||||||
|
///
|
||||||
|
/// It bounds the wait *between* statements only; one statement can outlast the
|
||||||
|
/// whole budget, which is why the pass also publishes its connection through
|
||||||
|
/// [`crate::db::InterruptGuard`]. See [`crate::db::InterruptSlot`].
|
||||||
pub const SLICE: Duration = Duration::from_millis(250);
|
pub const SLICE: Duration = Duration::from_millis(250);
|
||||||
|
|
||||||
/// One configured root, with the `files.path` range it owns precomputed.
|
/// One configured root, with the `files.path` range it owns precomputed.
|
||||||
|
|
@ -89,9 +111,8 @@ impl Scope {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The configured root `path` lives under, if any. `Path::starts_with`
|
/// The configured root `path` lives under, if any. Containment is
|
||||||
/// compares whole components, so `/a/bc` is never read as living under
|
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
|
||||||
/// `/a/b`.
|
|
||||||
pub fn owning_root(&self, path: &Path) -> Option<&Path> {
|
pub fn owning_root(&self, path: &Path) -> Option<&Path> {
|
||||||
self.roots
|
self.roots
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -107,8 +128,23 @@ impl Scope {
|
||||||
/// which is exactly the union of the per-level tests the walker performs
|
/// which is exactly the union of the per-level tests the walker performs
|
||||||
/// on its way down. The hidden and component-pattern rules are tested per
|
/// on its way down. The hidden and component-pattern rules are tested per
|
||||||
/// component *below* the root: a root is never filtered, because the user
|
/// component *below* the root: a root is never filtered, because the user
|
||||||
/// chose it (see `walk_parallel`).
|
/// chose it (see [`crate::walk::walk_indexable_files`]).
|
||||||
pub fn covers(&self, root: &Path, path: &Path) -> bool {
|
pub fn covers(&self, root: &Path, path: &Path) -> bool {
|
||||||
|
self.covers_cached(root, path, &mut CoverCache::default())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`Scope::covers`], reusing the verdicts already reached for the
|
||||||
|
/// directories on the way down.
|
||||||
|
///
|
||||||
|
/// A scan calls this once per stored row, and rows under one root share
|
||||||
|
/// their ancestors densely — every file in `~/src/project/src` asks the same
|
||||||
|
/// four questions before it asks its own. On Unix that repetition is free,
|
||||||
|
/// because `entry_is_hidden` never calls the closure and the whole loop is
|
||||||
|
/// string comparisons. On Windows each component is a `symlink_metadata`,
|
||||||
|
/// which is a `CreateFileW` through the full filter-driver stack: a 3M-row
|
||||||
|
/// index at depth 8 was ~24M file opens where Linux did none. Remembering
|
||||||
|
/// each directory's verdict collapses that to roughly one per directory.
|
||||||
|
pub fn covers_cached(&self, root: &Path, path: &Path, cache: &mut CoverCache) -> bool {
|
||||||
if self.ignore.matches_path_pattern(path) {
|
if self.ignore.matches_path_pattern(path) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
@ -116,28 +152,85 @@ impl Scope {
|
||||||
return false;
|
return false;
|
||||||
};
|
};
|
||||||
let mut current = root.to_path_buf();
|
let mut current = root.to_path_buf();
|
||||||
for component in relative.components() {
|
let depth = relative.components().count();
|
||||||
|
for (i, component) in relative.components().enumerate() {
|
||||||
let std::path::Component::Normal(name) = component else {
|
let std::path::Component::Normal(name) = component else {
|
||||||
// Stored paths are canonical, so stripping a canonical root
|
// Stored paths are canonical, so stripping a canonical root
|
||||||
// leaves plain names. Anything else did not come from a walk.
|
// leaves plain names. Anything else did not come from a walk.
|
||||||
return false;
|
return false;
|
||||||
};
|
};
|
||||||
current.push(name);
|
current.push(name);
|
||||||
let name = name.to_string_lossy();
|
// Only the ancestors are worth remembering. The last component is
|
||||||
// The metadata closure is only consulted on Windows, where hidden
|
// this row's own file name, asked once and never again, so caching
|
||||||
// is an attribute rather than a leading dot; on Unix this stays at
|
// it would grow the map by one entry per row for no hits at all.
|
||||||
// zero syscalls, exactly as it does in the walker.
|
let is_leaf = i + 1 == depth;
|
||||||
if !self.include_hidden
|
if !is_leaf {
|
||||||
&& crate::platform::entry_is_hidden(&name, || std::fs::metadata(¤t).ok())
|
if let Some(allowed) = cache.get(¤t) {
|
||||||
{
|
if !allowed {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if self.ignore.matches_component(&name) {
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let allowed = self.component_allowed(¤t, &name.to_string_lossy());
|
||||||
|
if !is_leaf {
|
||||||
|
cache.insert(current.clone(), allowed);
|
||||||
|
}
|
||||||
|
if !allowed {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
true
|
true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether one path component passes the hidden and component-pattern
|
||||||
|
/// rules. `current` is its full path, which the attribute test needs.
|
||||||
|
fn component_allowed(&self, current: &Path, name: &str) -> bool {
|
||||||
|
// The metadata closure is only consulted on Windows, where hidden
|
||||||
|
// is an attribute rather than a leading dot; on Unix this stays at
|
||||||
|
// zero syscalls, exactly as it does in the walker.
|
||||||
|
//
|
||||||
|
// `symlink_metadata` for the same reason the walker uses
|
||||||
|
// `DirEntry::metadata`: the component is judged as itself, never as
|
||||||
|
// what it points at. Following here and not there is exactly the
|
||||||
|
// disagreement this whole type exists to avoid.
|
||||||
|
if !self.include_hidden
|
||||||
|
&& crate::platform::entry_is_hidden(name, || std::fs::symlink_metadata(current).ok())
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
!self.ignore.matches_component(name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Directory verdicts already reached by [`Scope::covers_cached`].
|
||||||
|
///
|
||||||
|
/// Bounded rather than unbounded: a scan of a multi-million-row index would
|
||||||
|
/// otherwise hold every directory under every root at once, and the pass is
|
||||||
|
/// already paged precisely so it does not have to. Past the cap the map is
|
||||||
|
/// cleared outright instead of evicting one entry — rows arrive in roughly
|
||||||
|
/// insertion order, so the entries that matter are the ones just added, and a
|
||||||
|
/// clear costs one rebuild of a working set that is small by construction.
|
||||||
|
#[derive(Default)]
|
||||||
|
pub struct CoverCache {
|
||||||
|
dirs: std::collections::HashMap<PathBuf, bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CoverCache {
|
||||||
|
/// Directories remembered before the map is cleared. Roughly 100 bytes per
|
||||||
|
/// entry, so this is a few megabytes at most.
|
||||||
|
const CAP: usize = 20_000;
|
||||||
|
|
||||||
|
fn get(&self, dir: &Path) -> Option<bool> {
|
||||||
|
self.dirs.get(dir).copied()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn insert(&mut self, dir: PathBuf, allowed: bool) {
|
||||||
|
if self.dirs.len() >= Self::CAP {
|
||||||
|
self.dirs.clear();
|
||||||
|
}
|
||||||
|
self.dirs.insert(dir, allowed);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How far an in-progress [`advance`] has got.
|
/// How far an in-progress [`advance`] has got.
|
||||||
|
|
@ -146,6 +239,13 @@ impl Scope {
|
||||||
/// coordinator's command loop for the length of a full scan; the caller hands
|
/// coordinator's command loop for the length of a full scan; the caller hands
|
||||||
/// back the same cursor each tick with a fresh deadline, the way
|
/// back the same cursor each tick with a fresh deadline, the way
|
||||||
/// `apply_pending` drains the watcher queue.
|
/// `apply_pending` drains the watcher queue.
|
||||||
|
///
|
||||||
|
/// Resumable within one pass only. A cursor abandoned — cancelled, or dropped
|
||||||
|
/// on an error — takes its position with it, and the next attempt starts from
|
||||||
|
/// the beginning of a freshly derived plan; every part of the pass is
|
||||||
|
/// idempotent precisely so that this costs time and nothing else. A config edit
|
||||||
|
/// arriving mid-pass has the same effect (see `Inner::start_work`), which is
|
||||||
|
/// why the counters can go backwards between two published snapshots.
|
||||||
pub struct WorkCursor {
|
pub struct WorkCursor {
|
||||||
work: IndexWork,
|
work: IndexWork,
|
||||||
scope: Scope,
|
scope: Scope,
|
||||||
|
|
@ -165,6 +265,15 @@ pub struct WorkCursor {
|
||||||
pub deleted: usize,
|
pub deleted: usize,
|
||||||
/// Rows whose content state or stored text was re-decided.
|
/// Rows whose content state or stored text was re-decided.
|
||||||
pub recontented: usize,
|
pub recontented: usize,
|
||||||
|
/// Rows the scan has re-tested against the current configuration.
|
||||||
|
examined: usize,
|
||||||
|
/// Rows in the index, counted once when the scan first needs a page and
|
||||||
|
/// then left alone — a denominator that moved would make the display walk
|
||||||
|
/// backwards. `None` until then, and for a plan that reads no rows at all:
|
||||||
|
/// removing a root is one whole-range delete with no intermediate state to
|
||||||
|
/// report, so it gets the indeterminate bar rather than an invented
|
||||||
|
/// percentage.
|
||||||
|
total: Option<usize>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl WorkCursor {
|
impl WorkCursor {
|
||||||
|
|
@ -179,6 +288,8 @@ impl WorkCursor {
|
||||||
finalized: false,
|
finalized: false,
|
||||||
deleted: 0,
|
deleted: 0,
|
||||||
recontented: 0,
|
recontented: 0,
|
||||||
|
examined: 0,
|
||||||
|
total: None,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -186,6 +297,20 @@ impl WorkCursor {
|
||||||
self.finalized
|
self.finalized
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A snapshot for the status the caller publishes.
|
||||||
|
///
|
||||||
|
/// Both places this work runs — a run's prologue and the coordinator's
|
||||||
|
/// between-runs pass — report the same figures from the same counters,
|
||||||
|
/// so the user sees one thing however the reconcile was reached.
|
||||||
|
pub fn progress(&self) -> ReconcileProgress {
|
||||||
|
ReconcileProgress {
|
||||||
|
examined: self.examined,
|
||||||
|
total: self.total,
|
||||||
|
deleted: self.deleted,
|
||||||
|
recontented: self.recontented,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether a full walk must follow this reconciliation.
|
/// Whether a full walk must follow this reconciliation.
|
||||||
pub fn reindex(&self) -> bool {
|
pub fn reindex(&self) -> bool {
|
||||||
self.work.reindex
|
self.work.reindex
|
||||||
|
|
@ -207,16 +332,31 @@ impl WorkCursor {
|
||||||
/// Apply as much of `cursor` as fits before `deadline`, one page of rows per
|
/// Apply as much of `cursor` as fits before `deadline`, one page of rows per
|
||||||
/// transaction. Returns with the cursor advanced; call again until
|
/// transaction. Returns with the cursor advanced; call again until
|
||||||
/// [`WorkCursor::done`].
|
/// [`WorkCursor::done`].
|
||||||
|
///
|
||||||
|
/// `cancel` means "do not start another statement": it is read before every
|
||||||
|
/// one, and a set flag returns immediately with the cursor un-finished. It is
|
||||||
|
/// only half of cutting the pass short — the statement already running answers
|
||||||
|
/// to [`crate::db::interrupt`] and nothing else — and it is deliberately not
|
||||||
|
/// the same thing as `deadline`, which hands control back to a caller that
|
||||||
|
/// intends to come straight back.
|
||||||
|
///
|
||||||
|
/// A cancelled pass leaves the work owed. Nothing here records what it did:
|
||||||
|
/// the stored configuration is stamped only by a caller that saw the cursor
|
||||||
|
/// finish, so the next run re-derives the same plan and picks it up.
|
||||||
pub fn advance(
|
pub fn advance(
|
||||||
conn: &mut Connection,
|
conn: &mut Connection,
|
||||||
config: &Config,
|
config: &Config,
|
||||||
registry: &Registry,
|
registry: &Registry,
|
||||||
cursor: &mut WorkCursor,
|
cursor: &mut WorkCursor,
|
||||||
deadline: Instant,
|
deadline: Instant,
|
||||||
|
cancel: &AtomicBool,
|
||||||
) -> Result<(), String> {
|
) -> Result<(), String> {
|
||||||
// Whole ranges first: a removed root's rows can never satisfy the scan's
|
// Whole ranges first: a removed root's rows can never satisfy the scan's
|
||||||
// filters anyway, and deleting them by range spares the scan the work.
|
// filters anyway, and deleting them by range spares the scan the work.
|
||||||
while cursor.drop_idx < cursor.work.drop_roots.len() {
|
while cursor.drop_idx < cursor.work.drop_roots.len() {
|
||||||
|
if cancelled(cancel) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
let range = ExtractCursor::for_root(&cursor.work.drop_roots[cursor.drop_idx]);
|
let range = ExtractCursor::for_root(&cursor.work.drop_roots[cursor.drop_idx]);
|
||||||
let tx = conn
|
let tx = conn
|
||||||
.transaction()
|
.transaction()
|
||||||
|
|
@ -231,6 +371,10 @@ pub fn advance(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if cancelled(cancel) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
// Before the per-root scan and after the root deletions: the ranges it
|
// Before the per-root scan and after the root deletions: the ranges it
|
||||||
// spares must already be the final set of roots.
|
// spares must already be the final set of roots.
|
||||||
if !cursor.dropped_aliases && cursor.work.drop_aliases {
|
if !cursor.dropped_aliases && cursor.work.drop_aliases {
|
||||||
|
|
@ -254,8 +398,24 @@ pub fn advance(
|
||||||
}
|
}
|
||||||
|
|
||||||
if cursor.work.scans_rows() {
|
if cursor.work.scans_rows() {
|
||||||
|
// One count, the first time a page is actually needed. The scan it
|
||||||
|
// measures is a keyset walk of every row under every root, so on the
|
||||||
|
// indexes where this matters it is minutes of work against a count of
|
||||||
|
// seconds — and without it the display has no denominator at all.
|
||||||
|
if cursor.total.is_none() {
|
||||||
|
if cancelled(cancel) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
cursor.total = Some(repo::row_count(conn)?);
|
||||||
|
}
|
||||||
let page = config.processing.batch_size.max(1) as i64;
|
let page = config.processing.batch_size.max(1) as i64;
|
||||||
|
// Lives across pages, not per page: consecutive pages walk the same
|
||||||
|
// directories, which is exactly the repetition worth remembering.
|
||||||
|
let mut covered = CoverCache::default();
|
||||||
while cursor.root_idx < cursor.scope.roots.len() {
|
while cursor.root_idx < cursor.scope.roots.len() {
|
||||||
|
if cancelled(cancel) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
let root = &cursor.scope.roots[cursor.root_idx];
|
let root = &cursor.scope.roots[cursor.root_idx];
|
||||||
if cursor.after.is_empty() {
|
if cursor.after.is_empty() {
|
||||||
cursor.after = root.lo.clone();
|
cursor.after = root.lo.clone();
|
||||||
|
|
@ -267,6 +427,7 @@ pub fn advance(
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
cursor.after = last.path.clone();
|
cursor.after = last.path.clone();
|
||||||
|
cursor.examined += rows.len();
|
||||||
let root = cursor.scope.roots[cursor.root_idx].path.clone();
|
let root = cursor.scope.roots[cursor.root_idx].path.clone();
|
||||||
let (deleted, recontented) = apply_page(
|
let (deleted, recontented) = apply_page(
|
||||||
conn,
|
conn,
|
||||||
|
|
@ -276,6 +437,7 @@ pub fn advance(
|
||||||
&cursor.work,
|
&cursor.work,
|
||||||
&root,
|
&root,
|
||||||
&rows,
|
&rows,
|
||||||
|
&mut covered,
|
||||||
)?;
|
)?;
|
||||||
cursor.deleted += deleted;
|
cursor.deleted += deleted;
|
||||||
cursor.recontented += recontented;
|
cursor.recontented += recontented;
|
||||||
|
|
@ -287,6 +449,12 @@ pub fn advance(
|
||||||
|
|
||||||
// Deletions leave the FTS index with tombstones and a long segment list;
|
// Deletions leave the FTS index with tombstones and a long segment list;
|
||||||
// the same automerge that follows a run's stale cleanup collapses them.
|
// the same automerge that follows a run's stale cleanup collapses them.
|
||||||
|
// It is one statement that can run for a while, so it is the last thing
|
||||||
|
// the flag can spare the caller — and skipping it costs only tidiness,
|
||||||
|
// since the next run's own automerge collapses the same segments.
|
||||||
|
if cancelled(cancel) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
if cursor.deleted > 0 || cursor.recontented > 0 {
|
if cursor.deleted > 0 || cursor.recontented > 0 {
|
||||||
fts_finalize_after_text_indexing(conn);
|
fts_finalize_after_text_indexing(conn);
|
||||||
}
|
}
|
||||||
|
|
@ -294,7 +462,12 @@ pub fn advance(
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn cancelled(cancel: &AtomicBool) -> bool {
|
||||||
|
cancel.load(Ordering::Relaxed)
|
||||||
|
}
|
||||||
|
|
||||||
/// Decide and write one page of rows. Returns `(deleted, recontented)`.
|
/// Decide and write one page of rows. Returns `(deleted, recontented)`.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
fn apply_page(
|
fn apply_page(
|
||||||
conn: &mut Connection,
|
conn: &mut Connection,
|
||||||
config: &Config,
|
config: &Config,
|
||||||
|
|
@ -303,6 +476,7 @@ fn apply_page(
|
||||||
work: &IndexWork,
|
work: &IndexWork,
|
||||||
root: &Path,
|
root: &Path,
|
||||||
rows: &[repo::ScopeRow],
|
rows: &[repo::ScopeRow],
|
||||||
|
covered: &mut CoverCache,
|
||||||
) -> Result<(usize, usize), String> {
|
) -> Result<(usize, usize), String> {
|
||||||
let mut doomed: Vec<i64> = Vec::new();
|
let mut doomed: Vec<i64> = Vec::new();
|
||||||
let mut stale_text: Vec<i64> = Vec::new();
|
let mut stale_text: Vec<i64> = Vec::new();
|
||||||
|
|
@ -311,7 +485,7 @@ fn apply_page(
|
||||||
|
|
||||||
for row in rows {
|
for row in rows {
|
||||||
let path = Path::new(&row.path);
|
let path = Path::new(&row.path);
|
||||||
if work.prune_scope && !scope.covers(root, path) {
|
if work.prune_scope && !scope.covers_cached(root, path, covered) {
|
||||||
doomed.push(row.id);
|
doomed.push(row.id);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
@ -388,18 +562,44 @@ pub fn stored_config(conn: &Connection, config: &Config) -> Result<Config, Strin
|
||||||
"store_text_for_snippets" => {
|
"store_text_for_snippets" => {
|
||||||
stored.processing.store_text_for_snippets = value == "true"
|
stored.processing.store_text_for_snippets = value == "true"
|
||||||
}
|
}
|
||||||
"hash_length" => {
|
"hash_length" => match value.parse() {
|
||||||
if let Ok(n) = value.parse() {
|
Ok(n) => stored.processing.hash_length = n,
|
||||||
stored.processing.hash_length = n;
|
// Keeping the caller's value is the safest of bad options,
|
||||||
}
|
// but it makes the reconciliation diff describe a config the
|
||||||
}
|
// index was not built under, so it must not pass in silence.
|
||||||
|
Err(e) => crate::log_warn!("stored hash_length {:?} unreadable: {}", value, e),
|
||||||
|
},
|
||||||
"tokenize" => stored.processing.tokenize = value,
|
"tokenize" => stored.processing.tokenize = value,
|
||||||
|
// An unrecognized key is a record written by a newer build.
|
||||||
|
// Ignoring it leaves that setting at the caller's value, which is
|
||||||
|
// what a build that does not know the key would have used anyway.
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(stored)
|
Ok(stored)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The reconciliation the index still owes `config`, derived from its own
|
||||||
|
/// record of what it was last brought into line with.
|
||||||
|
///
|
||||||
|
/// The one question three callers ask in the same words: a run, to decide what
|
||||||
|
/// its prologue must do; a test, to check a pass recorded itself; and the GUI
|
||||||
|
/// at startup, to tell the user their settings have not reached the index yet.
|
||||||
|
/// Empty is the normal answer — every pass that finishes stamps the record —
|
||||||
|
/// so a non-empty one means a pass was abandoned, or the config was edited
|
||||||
|
/// while the app was closed. Either way the remedy is the same: run indexing.
|
||||||
|
///
|
||||||
|
/// The roots are canonicalized first, because that is the spelling the record
|
||||||
|
/// holds; comparing raw ones would report a `~` or a trailing slash as a
|
||||||
|
/// changed root.
|
||||||
|
pub fn outstanding_work(db_path: &str, config: &Config) -> Result<IndexWork, String> {
|
||||||
|
let conn = crate::db::open_existing(db_path, false)?;
|
||||||
|
let mut current = config.clone();
|
||||||
|
current.paths.indexing_paths = config.normalized_indexing_paths().into_iter().collect();
|
||||||
|
let stored = stored_config(&conn, ¤t)?;
|
||||||
|
Ok(crate::config::diff_actions(&stored, ¤t).work)
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
@ -409,18 +609,7 @@ mod tests {
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
fn tmp_tree(tag: &str) -> PathBuf {
|
fn tmp_tree(tag: &str) -> PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir_canonical(tag)
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-scope-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&p).unwrap();
|
|
||||||
std::fs::canonicalize(&p).unwrap()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn touch(p: &Path) {
|
fn touch(p: &Path) {
|
||||||
|
|
@ -456,6 +645,34 @@ mod tests {
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One `files` row per path, which is all the scan reads.
|
||||||
|
fn seed(conn: &mut Connection, paths: &[PathBuf]) {
|
||||||
|
let tx = conn.transaction().unwrap();
|
||||||
|
for path in paths {
|
||||||
|
let path = path.to_string_lossy();
|
||||||
|
let (parent, name) = path.rsplit_once('/').unwrap();
|
||||||
|
repo::insert_file(
|
||||||
|
&tx,
|
||||||
|
&repo::NewFile {
|
||||||
|
name,
|
||||||
|
path: &path,
|
||||||
|
parent,
|
||||||
|
size: 1,
|
||||||
|
mtime: 1,
|
||||||
|
inode: None,
|
||||||
|
device_id: None,
|
||||||
|
mime: Some("text/plain"),
|
||||||
|
ftype: crate::mime::FileType::TEXT,
|
||||||
|
hash: None,
|
||||||
|
needs_content: false,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
.expect("unique path");
|
||||||
|
}
|
||||||
|
tx.commit().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
/// Every file that physically exists under `root`, walker or no walker.
|
/// Every file that physically exists under `root`, walker or no walker.
|
||||||
fn on_disk(root: &Path) -> Vec<PathBuf> {
|
fn on_disk(root: &Path) -> Vec<PathBuf> {
|
||||||
walkdir::WalkDir::new(root)
|
walkdir::WalkDir::new(root)
|
||||||
|
|
@ -566,6 +783,156 @@ mod tests {
|
||||||
std::fs::remove_dir_all(&base).ok();
|
std::fs::remove_dir_all(&base).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The counters the status display reads. A scan that reports nothing is
|
||||||
|
/// indistinguishable from a hang, and on the indexes where this pass takes
|
||||||
|
/// minutes that is exactly what it looked like.
|
||||||
|
#[test]
|
||||||
|
fn the_scan_reports_its_way_through_every_row() {
|
||||||
|
let root = tmp_tree("progress");
|
||||||
|
for i in 0..7 {
|
||||||
|
touch(&root.join(format!("f{}.log", i)));
|
||||||
|
}
|
||||||
|
touch(&root.join("keep.txt"));
|
||||||
|
|
||||||
|
let db_dir = tmp_tree("progress-db");
|
||||||
|
let db = empty_db(&db_dir);
|
||||||
|
let mut conn = crate::db::open_existing(db.to_str().unwrap(), true).unwrap();
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
|
||||||
|
// One row per page, so the scan takes as many steps as there are rows
|
||||||
|
// and a counter that only moved at the end would be visible.
|
||||||
|
config.processing.batch_size = 1;
|
||||||
|
seed(&mut conn, &on_disk(&root));
|
||||||
|
|
||||||
|
let mut narrowed = config.clone();
|
||||||
|
narrowed.indexing.ignore_patterns = vec!["*.log".into()];
|
||||||
|
let work = crate::config::diff_actions(&config, &narrowed).work;
|
||||||
|
let mut cursor = WorkCursor::new(work, &narrowed).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
cursor.progress(),
|
||||||
|
ReconcileProgress::default(),
|
||||||
|
"nothing counted before the first slice"
|
||||||
|
);
|
||||||
|
|
||||||
|
let registry = Registry::default_set();
|
||||||
|
let run = AtomicBool::new(false);
|
||||||
|
let mut seen: Vec<usize> = Vec::new();
|
||||||
|
while !cursor.done() {
|
||||||
|
// A deadline already past, so each call does the least it can and
|
||||||
|
// the counters are sampled at their finest granularity.
|
||||||
|
advance(
|
||||||
|
&mut conn,
|
||||||
|
&narrowed,
|
||||||
|
®istry,
|
||||||
|
&mut cursor,
|
||||||
|
Instant::now(),
|
||||||
|
&run,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
seen.push(cursor.progress().examined);
|
||||||
|
}
|
||||||
|
|
||||||
|
let end = cursor.progress();
|
||||||
|
assert_eq!(end.total, Some(8), "counted once, before the first page");
|
||||||
|
assert_eq!(end.examined, 8, "every row was re-tested");
|
||||||
|
assert_eq!(end.deleted, 7, "the logs, and only the logs");
|
||||||
|
assert!(
|
||||||
|
seen.windows(2).all(|w| w[0] <= w[1]),
|
||||||
|
"the count never goes backwards: {:?}",
|
||||||
|
seen
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
seen.len() > 2 && seen[0] < end.examined,
|
||||||
|
"progress was reported during the scan, not only at its end: {:?}",
|
||||||
|
seen
|
||||||
|
);
|
||||||
|
|
||||||
|
std::fs::remove_dir_all(&root).ok();
|
||||||
|
std::fs::remove_dir_all(&db_dir).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cancelling stops the pass at the next statement boundary and leaves the
|
||||||
|
/// cursor un-finished, so nothing downstream can mistake it for done and
|
||||||
|
/// record the configuration as reconciled. The rows it had already reached
|
||||||
|
/// stay gone — every part of the pass is idempotent, and the next run
|
||||||
|
/// re-derives the same plan and finishes it.
|
||||||
|
#[test]
|
||||||
|
fn cancelling_stops_the_scan_without_finishing_it() {
|
||||||
|
let root = tmp_tree("cancel");
|
||||||
|
for i in 0..6 {
|
||||||
|
touch(&root.join(format!("f{}.log", i)));
|
||||||
|
}
|
||||||
|
touch(&root.join("keep.txt"));
|
||||||
|
|
||||||
|
let db_dir = tmp_tree("cancel-db");
|
||||||
|
let db = empty_db(&db_dir);
|
||||||
|
let mut conn = crate::db::open_existing(db.to_str().unwrap(), true).unwrap();
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
|
||||||
|
config.processing.batch_size = 1;
|
||||||
|
seed(&mut conn, &on_disk(&root));
|
||||||
|
|
||||||
|
let mut narrowed = config.clone();
|
||||||
|
narrowed.indexing.ignore_patterns = vec!["*.log".into()];
|
||||||
|
let work = crate::config::diff_actions(&config, &narrowed).work;
|
||||||
|
let registry = Registry::default_set();
|
||||||
|
|
||||||
|
// Cancelled from the outset: not one statement runs.
|
||||||
|
let stop = AtomicBool::new(true);
|
||||||
|
let mut cursor = WorkCursor::new(work.clone(), &narrowed).unwrap();
|
||||||
|
advance(
|
||||||
|
&mut conn,
|
||||||
|
&narrowed,
|
||||||
|
®istry,
|
||||||
|
&mut cursor,
|
||||||
|
Instant::now() + SLICE,
|
||||||
|
&stop,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert!(!cursor.done(), "a cancelled pass is never finished");
|
||||||
|
assert_eq!(
|
||||||
|
cursor.progress(),
|
||||||
|
ReconcileProgress::default(),
|
||||||
|
"a cancelled pass touched the index"
|
||||||
|
);
|
||||||
|
|
||||||
|
// And part-way through: one slice with the flag clear, the rest with
|
||||||
|
// it set. The counters keep what the first slice earned.
|
||||||
|
let stop = AtomicBool::new(false);
|
||||||
|
let mut cursor = WorkCursor::new(work, &narrowed).unwrap();
|
||||||
|
advance(
|
||||||
|
&mut conn,
|
||||||
|
&narrowed,
|
||||||
|
®istry,
|
||||||
|
&mut cursor,
|
||||||
|
Instant::now(),
|
||||||
|
&stop,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let part_way = cursor.progress();
|
||||||
|
assert!(part_way.examined > 0 && !cursor.done(), "nothing to cancel");
|
||||||
|
|
||||||
|
stop.store(true, Ordering::Relaxed);
|
||||||
|
advance(
|
||||||
|
&mut conn,
|
||||||
|
&narrowed,
|
||||||
|
®istry,
|
||||||
|
&mut cursor,
|
||||||
|
Instant::now() + SLICE,
|
||||||
|
&stop,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert!(!cursor.done(), "the pass finished despite the cancellation");
|
||||||
|
assert_eq!(
|
||||||
|
cursor.progress(),
|
||||||
|
part_way,
|
||||||
|
"the cancelled slice did more work"
|
||||||
|
);
|
||||||
|
|
||||||
|
std::fs::remove_dir_all(&root).ok();
|
||||||
|
std::fs::remove_dir_all(&db_dir).ok();
|
||||||
|
}
|
||||||
|
|
||||||
/// A path under no configured root has no rules that could be applied to
|
/// A path under no configured root has no rules that could be applied to
|
||||||
/// it — a followed symlink's target is the real case. The scan reaches it
|
/// it — a followed symlink's target is the real case. The scan reaches it
|
||||||
/// by never visiting it, so `owning_root` returning `None` is what keeps
|
/// by never visiting it, so `owning_root` returning `None` is what keeps
|
||||||
|
|
|
||||||
|
|
@ -213,6 +213,29 @@ fn count_frac(count: usize) -> f64 {
|
||||||
(1000usize.saturating_sub(count.min(1000))) as f64 / 1000.0
|
(1000usize.saturating_sub(count.min(1000))) as f64 / 1000.0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The `files` columns every pass selects, in the order the passes index
|
||||||
|
/// them: `0` id, `1` name, `2` path, `3` size, `4` mtime. Passes that also
|
||||||
|
/// want the stored document text append `dt.text_zstd` as column `5`.
|
||||||
|
///
|
||||||
|
/// One string rather than seven copies, because the column *order* is what
|
||||||
|
/// every `row.get(n)` in this file is written against — a pass that spelled
|
||||||
|
/// its own list in a different order would compile and then quietly serve
|
||||||
|
/// paths as names.
|
||||||
|
const HIT_COLUMNS: &str = "f.id, f.name, f.path, f.size, f.mtime";
|
||||||
|
|
||||||
|
/// Columns 3 and 4: the two every pass reads identically and stores without
|
||||||
|
/// inspecting.
|
||||||
|
///
|
||||||
|
/// The clamp is the point of having this in one place. `size` is `INTEGER` in
|
||||||
|
/// SQLite and so signed; a corrupt or hand-edited row holding `-1` would
|
||||||
|
/// otherwise become 18 exabytes on the way to `u64` and sort to the top of
|
||||||
|
/// every size-ordered result.
|
||||||
|
fn size_and_mtime(row: &rusqlite::Row<'_>) -> Result<(u64, i64), String> {
|
||||||
|
let size = row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64;
|
||||||
|
let mtime = row.get(4).map_err(|e| e.to_string())?;
|
||||||
|
Ok((size, mtime))
|
||||||
|
}
|
||||||
|
|
||||||
/// The path tiers only make sense with enough term to be specific — same
|
/// The path tiers only make sense with enough term to be specific — same
|
||||||
/// floor the trigram full-text pass uses. Wildcards count only their
|
/// floor the trigram full-text pass uses. Wildcards count only their
|
||||||
/// literal content (`a*b` is two characters of specificity, not three).
|
/// literal content (`a*b` is two characters of specificity, not three).
|
||||||
|
|
@ -439,8 +462,9 @@ impl<'a> Cx<'a> {
|
||||||
// A path always ends in its own name, so `path LIKE` is the
|
// A path always ends in its own name, so `path LIKE` is the
|
||||||
// superset that feeds both the name and the path tiers.
|
// superset that feeds both the name and the path tiers.
|
||||||
let sql = format!(
|
let sql = format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f \
|
"SELECT {} FROM files f \
|
||||||
WHERE {} LIKE ? ESCAPE '\\'{}",
|
WHERE {} LIKE ? ESCAPE '\\'{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
if with_paths { "f.path" } else { "f.name" },
|
if with_paths { "f.path" } else { "f.name" },
|
||||||
query.filter_sql
|
query.filter_sql
|
||||||
);
|
);
|
||||||
|
|
@ -469,7 +493,7 @@ impl<'a> Cx<'a> {
|
||||||
let mut clock = FlushClock::new();
|
let mut clock = FlushClock::new();
|
||||||
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
||||||
scanned += 1;
|
scanned += 1;
|
||||||
if scanned % CANCEL_CHECK_ROWS == 0 && self.cancelled() {
|
if scanned.is_multiple_of(CANCEL_CHECK_ROWS) && self.cancelled() {
|
||||||
return Ok(false);
|
return Ok(false);
|
||||||
}
|
}
|
||||||
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
||||||
|
|
@ -519,12 +543,13 @@ impl<'a> Cx<'a> {
|
||||||
truncated_start: false,
|
truncated_start: false,
|
||||||
truncated_end: false,
|
truncated_end: false,
|
||||||
};
|
};
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
let hit = SearchHit {
|
let hit = SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name,
|
name,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank,
|
rank,
|
||||||
stage: rank as u8,
|
stage: rank as u8,
|
||||||
snippet: Some(snip),
|
snippet: Some(snip),
|
||||||
|
|
@ -586,20 +611,22 @@ impl<'a> Cx<'a> {
|
||||||
let (sql, params) = match match_expr {
|
let (sql, params) = match match_expr {
|
||||||
Some(expr) => (
|
Some(expr) => (
|
||||||
format!(
|
format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
|
"SELECT {}, dt.text_zstd \
|
||||||
FROM searchabletext \
|
FROM searchabletext \
|
||||||
JOIN files f ON f.id = searchabletext.rowid \
|
JOIN files f ON f.id = searchabletext.rowid \
|
||||||
LEFT JOIN documents_text dt ON dt.file_id = f.id \
|
LEFT JOIN documents_text dt ON dt.file_id = f.id \
|
||||||
WHERE searchabletext MATCH ?{}",
|
WHERE searchabletext MATCH ?{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
query.filter_sql
|
query.filter_sql
|
||||||
),
|
),
|
||||||
self.params_with_filters(vec![rusqlite::types::Value::Text(expr)]),
|
self.params_with_filters(vec![rusqlite::types::Value::Text(expr)]),
|
||||||
),
|
),
|
||||||
None => (
|
None => (
|
||||||
format!(
|
format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
|
"SELECT {}, dt.text_zstd \
|
||||||
FROM documents_text dt \
|
FROM documents_text dt \
|
||||||
JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
query.filter_sql
|
query.filter_sql
|
||||||
),
|
),
|
||||||
self.params_with_filters(Vec::new()),
|
self.params_with_filters(Vec::new()),
|
||||||
|
|
@ -692,12 +719,13 @@ impl<'a> Cx<'a> {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
buf.push(SearchHit {
|
buf.push(SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name: row.get(1).map_err(|e| e.to_string())?,
|
name: row.get(1).map_err(|e| e.to_string())?,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank,
|
rank,
|
||||||
stage,
|
stage,
|
||||||
snippet: snip,
|
snippet: snip,
|
||||||
|
|
@ -733,7 +761,8 @@ impl<'a> Cx<'a> {
|
||||||
let with_paths = path_tiers_enabled(&self.query.pattern);
|
let with_paths = path_tiers_enabled(&self.query.pattern);
|
||||||
|
|
||||||
let sql = format!(
|
let sql = format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
|
"SELECT {} FROM files f WHERE 1=1{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
self.query.filter_sql
|
self.query.filter_sql
|
||||||
);
|
);
|
||||||
let params = self.params_with_filters(Vec::new());
|
let params = self.params_with_filters(Vec::new());
|
||||||
|
|
@ -750,7 +779,7 @@ impl<'a> Cx<'a> {
|
||||||
let mut clock = FlushClock::new();
|
let mut clock = FlushClock::new();
|
||||||
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
||||||
scanned += 1;
|
scanned += 1;
|
||||||
if scanned % 1024 == 0 && self.cancelled() {
|
if scanned.is_multiple_of(1024) && self.cancelled() {
|
||||||
return Ok(false);
|
return Ok(false);
|
||||||
}
|
}
|
||||||
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
||||||
|
|
@ -789,12 +818,13 @@ impl<'a> Cx<'a> {
|
||||||
},
|
},
|
||||||
));
|
));
|
||||||
let is_path_tier = rank >= 11.0;
|
let is_path_tier = rank >= 11.0;
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
let hit = SearchHit {
|
let hit = SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name,
|
name,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank,
|
rank,
|
||||||
stage: rank as u8,
|
stage: rank as u8,
|
||||||
snippet: snip,
|
snippet: snip,
|
||||||
|
|
@ -838,8 +868,9 @@ impl<'a> Cx<'a> {
|
||||||
};
|
};
|
||||||
|
|
||||||
let sql = format!(
|
let sql = format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
|
"SELECT {}, dt.text_zstd \
|
||||||
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
self.query.filter_sql
|
self.query.filter_sql
|
||||||
);
|
);
|
||||||
let params = self.params_with_filters(Vec::new());
|
let params = self.params_with_filters(Vec::new());
|
||||||
|
|
@ -882,12 +913,13 @@ impl<'a> Cx<'a> {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let snip = first.map(|range| snippet::window_around(&text, range, &snippet_opts));
|
let snip = first.map(|range| snippet::window_around(&text, range, &snippet_opts));
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
buf.push(SearchHit {
|
buf.push(SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name: row.get(1).map_err(|e| e.to_string())?,
|
name: row.get(1).map_err(|e| e.to_string())?,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank: 8.0 + count_frac(count),
|
rank: 8.0 + count_frac(count),
|
||||||
stage: 8,
|
stage: 8,
|
||||||
snippet: snip,
|
snippet: snip,
|
||||||
|
|
@ -911,7 +943,8 @@ impl<'a> Cx<'a> {
|
||||||
let query = self.query;
|
let query = self.query;
|
||||||
let re = query.regex.as_ref().expect("regex-only pass list");
|
let re = query.regex.as_ref().expect("regex-only pass list");
|
||||||
let sql = format!(
|
let sql = format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
|
"SELECT {} FROM files f WHERE 1=1{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
query.filter_sql
|
query.filter_sql
|
||||||
);
|
);
|
||||||
let params = self.params_with_filters(Vec::new());
|
let params = self.params_with_filters(Vec::new());
|
||||||
|
|
@ -928,7 +961,7 @@ impl<'a> Cx<'a> {
|
||||||
let mut clock = FlushClock::new();
|
let mut clock = FlushClock::new();
|
||||||
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
|
||||||
scanned += 1;
|
scanned += 1;
|
||||||
if scanned % 1024 == 0 && self.cancelled() {
|
if scanned.is_multiple_of(1024) && self.cancelled() {
|
||||||
return Ok(false);
|
return Ok(false);
|
||||||
}
|
}
|
||||||
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
|
||||||
|
|
@ -956,12 +989,13 @@ impl<'a> Cx<'a> {
|
||||||
truncated_start: false,
|
truncated_start: false,
|
||||||
truncated_end: false,
|
truncated_end: false,
|
||||||
};
|
};
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
let hit = SearchHit {
|
let hit = SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name,
|
name,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank,
|
rank,
|
||||||
stage: rank as u8,
|
stage: rank as u8,
|
||||||
snippet: Some(snip),
|
snippet: Some(snip),
|
||||||
|
|
@ -992,8 +1026,9 @@ impl<'a> Cx<'a> {
|
||||||
let query = self.query;
|
let query = self.query;
|
||||||
let re = query.regex.as_ref().expect("regex-only pass list");
|
let re = query.regex.as_ref().expect("regex-only pass list");
|
||||||
let sql = format!(
|
let sql = format!(
|
||||||
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
|
"SELECT {}, dt.text_zstd \
|
||||||
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
|
||||||
|
HIT_COLUMNS,
|
||||||
query.filter_sql
|
query.filter_sql
|
||||||
);
|
);
|
||||||
let params = self.params_with_filters(Vec::new());
|
let params = self.params_with_filters(Vec::new());
|
||||||
|
|
@ -1033,12 +1068,13 @@ impl<'a> Cx<'a> {
|
||||||
let r = clamp_match_range(&text, r, SNIPPET_WINDOW_CHARS);
|
let r = clamp_match_range(&text, r, SNIPPET_WINDOW_CHARS);
|
||||||
snippet::window_around(&text, (r.start, r.end), &snippet_opts)
|
snippet::window_around(&text, (r.start, r.end), &snippet_opts)
|
||||||
});
|
});
|
||||||
|
let (size, mtime) = size_and_mtime(row)?;
|
||||||
buf.push(SearchHit {
|
buf.push(SearchHit {
|
||||||
file_id,
|
file_id,
|
||||||
name: row.get(1).map_err(|e| e.to_string())?,
|
name: row.get(1).map_err(|e| e.to_string())?,
|
||||||
path,
|
path,
|
||||||
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
|
size,
|
||||||
mtime: row.get(4).map_err(|e| e.to_string())?,
|
mtime,
|
||||||
rank: 6.0 + count_frac(count),
|
rank: 6.0 + count_frac(count),
|
||||||
stage: 6,
|
stage: 6,
|
||||||
snippet: snip,
|
snippet: snip,
|
||||||
|
|
|
||||||
|
|
@ -88,15 +88,7 @@ mod tests {
|
||||||
use crate::mime::FileType;
|
use crate::mime::FileType;
|
||||||
|
|
||||||
fn seed_db() -> std::path::PathBuf {
|
fn seed_db() -> std::path::PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
let p = crate::testutil::scratch_dir("dups").join("index.sqlite");
|
||||||
p.push(format!(
|
|
||||||
"qs-dups-{}-{}.sqlite",
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
|
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
|
||||||
let tx = conn.transaction().unwrap();
|
let tx = conn.transaction().unwrap();
|
||||||
let add = |name: &str, path: &str, size: u64, hash: Option<&[u8]>| {
|
let add = |name: &str, path: &str, size: u64, hash: Option<&[u8]>| {
|
||||||
|
|
|
||||||
|
|
@ -320,9 +320,9 @@ mod tests {
|
||||||
}
|
}
|
||||||
std::mem::swap(&mut prev, &mut cur);
|
std::mem::swap(&mut prev, &mut cur);
|
||||||
}
|
}
|
||||||
for j in 0..=hay.len() {
|
// The best occurrence may end at any text position, so the answer is
|
||||||
best = best.min(prev[j]);
|
// the smallest value in the final row.
|
||||||
}
|
best = prev.iter().copied().min().unwrap_or(best);
|
||||||
if best <= k {
|
if best <= k {
|
||||||
Some(best)
|
Some(best)
|
||||||
} else {
|
} else {
|
||||||
|
|
|
||||||
|
|
@ -47,7 +47,7 @@ impl IndexKey {
|
||||||
hex_encode(&self.0)
|
hex_encode(&self.0)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Strict inverse of [`to_hex`]: exactly 64 hex digits, any case.
|
/// Strict inverse of [`IndexKey::to_hex`]: exactly 64 hex digits, any case.
|
||||||
pub fn from_hex(hex: &str) -> Result<IndexKey, String> {
|
pub fn from_hex(hex: &str) -> Result<IndexKey, String> {
|
||||||
let bytes = hex_decode(hex)?;
|
let bytes = hex_decode(hex)?;
|
||||||
let arr: [u8; KEY_LEN] = bytes
|
let arr: [u8; KEY_LEN] = bytes
|
||||||
|
|
@ -102,7 +102,7 @@ fn hex_encode(bytes: &[u8]) -> String {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hex_decode(hex: &str) -> Result<Vec<u8>, String> {
|
fn hex_decode(hex: &str) -> Result<Vec<u8>, String> {
|
||||||
if hex.len() % 2 != 0 {
|
if !hex.len().is_multiple_of(2) {
|
||||||
return Err("hex string has odd length".to_string());
|
return Err("hex string has odd length".to_string());
|
||||||
}
|
}
|
||||||
if !hex.bytes().all(|b| b.is_ascii_hexdigit()) {
|
if !hex.bytes().all(|b| b.is_ascii_hexdigit()) {
|
||||||
|
|
|
||||||
76
crates/quicksearch-core/src/testutil.rs
Normal file
76
crates/quicksearch-core/src/testutil.rs
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
//! Scratch directories for tests.
|
||||||
|
//!
|
||||||
|
//! Public and `#[doc(hidden)]` rather than `#[cfg(test)]`: the `tests/`
|
||||||
|
//! integration binaries and the GUI crate are separate compilation units, so a
|
||||||
|
//! test-gated item here would be invisible to them. This is the only reason it
|
||||||
|
//! is not private.
|
||||||
|
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||||
|
|
||||||
|
/// Distinguishes directories requested within one process. Two tests running
|
||||||
|
/// on different threads in the same millisecond would otherwise collide — the
|
||||||
|
/// hand-rolled helpers this replaces all keyed off a timestamp, which made
|
||||||
|
/// that rare rather than impossible.
|
||||||
|
static NEXT: AtomicUsize = AtomicUsize::new(0);
|
||||||
|
|
||||||
|
/// A fresh, empty directory under the system temp dir, named for `tag`.
|
||||||
|
///
|
||||||
|
/// Not cleaned up on drop, deliberately: when a test fails, the tree it built
|
||||||
|
/// is most of the evidence. The OS clears the temp dir eventually.
|
||||||
|
///
|
||||||
|
/// Panics rather than returning a `Result` — a test that cannot create a
|
||||||
|
/// directory has nothing left to assert.
|
||||||
|
#[doc(hidden)]
|
||||||
|
pub fn scratch_dir(tag: &str) -> PathBuf {
|
||||||
|
let mut p = std::env::temp_dir();
|
||||||
|
p.push(format!(
|
||||||
|
"quicksearch-{}-{}-{}",
|
||||||
|
tag,
|
||||||
|
std::process::id(),
|
||||||
|
NEXT.fetch_add(1, Ordering::Relaxed)
|
||||||
|
));
|
||||||
|
std::fs::create_dir_all(&p).expect("create scratch dir");
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`scratch_dir`] canonicalized, for the tests that compare walked paths
|
||||||
|
/// against the root they were given. On macOS `/tmp` is a symlink to
|
||||||
|
/// `/private/tmp`, so an uncanonicalized root and a walked path disagree.
|
||||||
|
#[doc(hidden)]
|
||||||
|
pub fn scratch_dir_canonical(tag: &str) -> PathBuf {
|
||||||
|
std::fs::canonicalize(scratch_dir(tag)).expect("canonicalize scratch dir")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Write `body` to `path`, creating parent directories as needed.
|
||||||
|
#[doc(hidden)]
|
||||||
|
pub fn touch(path: &std::path::Path, body: &[u8]) {
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
std::fs::create_dir_all(parent).expect("create parent dir");
|
||||||
|
}
|
||||||
|
std::fs::write(path, body).expect("write file");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_call_gets_its_own_empty_directory() {
|
||||||
|
let a = scratch_dir("selftest");
|
||||||
|
let b = scratch_dir("selftest");
|
||||||
|
assert_ne!(a, b, "two calls must not collide");
|
||||||
|
for d in [&a, &b] {
|
||||||
|
assert!(d.is_dir());
|
||||||
|
assert_eq!(std::fs::read_dir(d).unwrap().count(), 0, "starts empty");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn touch_creates_missing_parents() {
|
||||||
|
let dir = scratch_dir("selftest-touch");
|
||||||
|
let deep = dir.join("a/b/c.txt");
|
||||||
|
touch(&deep, b"hi");
|
||||||
|
assert_eq!(std::fs::read(&deep).unwrap(), b"hi");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -27,7 +27,7 @@
|
||||||
use std::collections::HashSet;
|
use std::collections::HashSet;
|
||||||
use std::fs;
|
use std::fs;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
|
||||||
use std::sync::{mpsc, Arc, Condvar, Mutex};
|
use std::sync::{mpsc, Arc, Condvar, Mutex};
|
||||||
use std::thread::{self, JoinHandle};
|
use std::thread::{self, JoinHandle};
|
||||||
use std::time::UNIX_EPOCH;
|
use std::time::UNIX_EPOCH;
|
||||||
|
|
@ -100,6 +100,11 @@ impl WalkedFile {
|
||||||
|
|
||||||
/// What the walk emits. Files as they are classified, plus the per-directory
|
/// What the walk emits. Files as they are classified, plus the per-directory
|
||||||
/// verdict on which index rows no longer have a file behind them.
|
/// verdict on which index rows no longer have a file behind them.
|
||||||
|
// `File` is far larger than the deletion variant, and deliberately so: this
|
||||||
|
// is the walk's hot path, one event per file on a tree of millions. Boxing to
|
||||||
|
// even out the variants would add exactly the per-file allocation the
|
||||||
|
// owned-record design exists to avoid.
|
||||||
|
#[allow(clippy::large_enum_variant)]
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum WalkEvent {
|
pub enum WalkEvent {
|
||||||
File(WalkedFile),
|
File(WalkedFile),
|
||||||
|
|
@ -109,6 +114,27 @@ pub enum WalkEvent {
|
||||||
Stale(Vec<String>),
|
Stale(Vec<String>),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One file a directory read produced: its path, plus whatever that read
|
||||||
|
/// already told us about it.
|
||||||
|
///
|
||||||
|
/// The metadata half is `Some` only on Windows, where `FindNextFileW` returned
|
||||||
|
/// size, mtime and attributes alongside the name. On Unix `getdents64` returns
|
||||||
|
/// only `d_type`, so it is always `None`, [`prepare`] does the single `statx`
|
||||||
|
/// the walk has always done, and `Option<CachedMetadata>` is zero-sized — a
|
||||||
|
/// queued file costs exactly the `PathBuf` it cost before.
|
||||||
|
struct PendingFile {
|
||||||
|
path: PathBuf,
|
||||||
|
cached: Option<crate::platform::CachedMetadata>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PendingFile {
|
||||||
|
/// A path that did not come from a directory entry — a resolved symlink
|
||||||
|
/// target — so there is nothing cached to carry.
|
||||||
|
fn uncached(path: PathBuf) -> Self {
|
||||||
|
PendingFile { path, cached: None }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Work waiting for a thread.
|
/// Work waiting for a thread.
|
||||||
enum Job {
|
enum Job {
|
||||||
/// Read this directory and process its files. Carries the directory's
|
/// Read this directory and process its files. Carries the directory's
|
||||||
|
|
@ -116,8 +142,10 @@ enum Job {
|
||||||
Dir(PathBuf, Arc<DirRows>),
|
Dir(PathBuf, Arc<DirRows>),
|
||||||
/// Process this slice of one directory's files, split off because the
|
/// Process this slice of one directory's files, split off because the
|
||||||
/// directory was too wide for one worker to be worth serialising on.
|
/// directory was too wide for one worker to be worth serialising on.
|
||||||
/// Shares its directory's rows.
|
/// Shares its directory's rows. On Windows each entry also carries the
|
||||||
Files(Vec<PathBuf>, Arc<DirRows>),
|
/// metadata its directory read returned, so classifying these files costs
|
||||||
|
/// no syscall at all.
|
||||||
|
Files(Vec<PendingFile>, Arc<DirRows>),
|
||||||
/// A resolved symlink target, with the stored mtime for its own path.
|
/// A resolved symlink target, with the stored mtime for its own path.
|
||||||
/// Classified against that rather than against any directory's rows.
|
/// Classified against that rather than against any directory's rows.
|
||||||
Alias(PathBuf, Option<u64>),
|
Alias(PathBuf, Option<u64>),
|
||||||
|
|
@ -151,6 +179,12 @@ struct Queue {
|
||||||
/// directory pushes many of them. Bounding on the total would let file
|
/// directory pushes many of them. Bounding on the total would let file
|
||||||
/// chunks starve directory prefetching and leave the pool waiting on
|
/// chunks starve directory prefetching and leave the pool waiting on
|
||||||
/// rows that were never fetched.
|
/// rows that were never fetched.
|
||||||
|
///
|
||||||
|
/// The consequence is that file chunks, not this counter, are what bounds
|
||||||
|
/// the walker's memory: nothing throttles them, so one very wide directory
|
||||||
|
/// can hold its whole listing in `jobs` at once — and on Windows each of
|
||||||
|
/// those entries also carries its directory read's metadata
|
||||||
|
/// ([`PendingFile`]), so that is the larger cost of the two.
|
||||||
dirs_ready: usize,
|
dirs_ready: usize,
|
||||||
/// Workers currently holding a job — that is, workers that may still push
|
/// Workers currently holding a job — that is, workers that may still push
|
||||||
/// more. The walk is over when this is zero and `jobs` is empty.
|
/// more. The walk is over when this is zero and `jobs` is empty.
|
||||||
|
|
@ -342,7 +376,7 @@ enum Found {
|
||||||
/// A resolved symlink target. Needs an exact-path mtime lookup.
|
/// A resolved symlink target. Needs an exact-path mtime lookup.
|
||||||
Alias(PathBuf),
|
Alias(PathBuf),
|
||||||
/// Overflow files from the directory just read, which already has rows.
|
/// Overflow files from the directory just read, which already has rows.
|
||||||
Files(Vec<PathBuf>, Arc<DirRows>),
|
Files(Vec<PendingFile>, Arc<DirRows>),
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Shared {
|
impl Shared {
|
||||||
|
|
@ -398,10 +432,57 @@ impl Drop for ActiveJob<'_> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// How many entries each filter rejected, for the one-line summary a run logs
|
||||||
|
/// when it finishes.
|
||||||
|
///
|
||||||
|
/// Counted rather than logged per entry, deliberately. The process log is a
|
||||||
|
/// 5,000-line ring ([`crate::log::CAPACITY`]) that evicts oldest-first with no
|
||||||
|
/// level protection, so a line per ignored entry would push every warning —
|
||||||
|
/// including the unreadable-directory ones that distinguish a network blip from
|
||||||
|
/// a deletion — out of the buffer before the run ended. `log::record` also takes
|
||||||
|
/// a global mutex and writes to stderr, which would serialize the worker pool on
|
||||||
|
/// the one path the walker is built to keep syscall-free.
|
||||||
|
///
|
||||||
|
/// A pruned *directory* is one increment, not one per file beneath it: the
|
||||||
|
/// subtree is never enumerated.
|
||||||
|
#[derive(Debug, Default)]
|
||||||
|
pub struct PruneCounts {
|
||||||
|
/// Names beginning with a dot.
|
||||||
|
pub dot_named: AtomicU64,
|
||||||
|
/// Windows entries carrying `FILE_ATTRIBUTE_HIDDEN`.
|
||||||
|
pub attribute: AtomicU64,
|
||||||
|
/// Rejected by a configured ignore pattern, of either kind.
|
||||||
|
pub ignored: AtomicU64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PruneCounts {
|
||||||
|
pub fn total(&self) -> u64 {
|
||||||
|
self.dot_named.load(Ordering::Relaxed)
|
||||||
|
+ self.attribute.load(Ordering::Relaxed)
|
||||||
|
+ self.ignored.load(Ordering::Relaxed)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The summary line, or `None` when nothing was pruned — a clean tree
|
||||||
|
/// should not add noise to the log.
|
||||||
|
pub fn summary(&self) -> Option<String> {
|
||||||
|
if self.total() == 0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(format!(
|
||||||
|
"pruned {} entries: {} hidden by attribute, {} dot-named, {} by ignore pattern",
|
||||||
|
self.total(),
|
||||||
|
self.attribute.load(Ordering::Relaxed),
|
||||||
|
self.dot_named.load(Ordering::Relaxed),
|
||||||
|
self.ignored.load(Ordering::Relaxed),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
struct Ctx {
|
struct Ctx {
|
||||||
follow_symlinks: bool,
|
follow_symlinks: bool,
|
||||||
include_hidden: bool,
|
include_hidden: bool,
|
||||||
ignore: IgnoreSet,
|
ignore: IgnoreSet,
|
||||||
|
pruned: PruneCounts,
|
||||||
config: Config,
|
config: Config,
|
||||||
/// Lets a worker finish small text files outright: the head it reads to
|
/// Lets a worker finish small text files outright: the head it reads to
|
||||||
/// hash them is already their entire contents, so an extractor that works
|
/// hash them is already their entire contents, so an extractor that works
|
||||||
|
|
@ -412,6 +493,16 @@ struct Ctx {
|
||||||
suspend_flag: Arc<AtomicBool>,
|
suspend_flag: Arc<AtomicBool>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Individual unreadable-directory warnings allowed per run before only the
|
||||||
|
/// count is kept. Reset by [`reset_run_warnings`].
|
||||||
|
static UNREADABLE_WARNINGS: crate::log::Throttle = crate::log::Throttle::new(20);
|
||||||
|
|
||||||
|
/// Arm this module's per-run warning throttle. See
|
||||||
|
/// [`crate::file_handling::reset_run_warnings`], which the same caller invokes.
|
||||||
|
pub fn reset_run_warnings() {
|
||||||
|
UNREADABLE_WARNINGS.reset();
|
||||||
|
}
|
||||||
|
|
||||||
/// Read one directory, apply the hidden/ignore rules, and split the result:
|
/// Read one directory, apply the hidden/ignore rules, and split the result:
|
||||||
/// subdirectories and overflow file chunks go to `found` for the pool, the
|
/// subdirectories and overflow file chunks go to `found` for the pool, the
|
||||||
/// remaining files come back for this worker to handle immediately.
|
/// remaining files come back for this worker to handle immediately.
|
||||||
|
|
@ -425,18 +516,27 @@ struct Ctx {
|
||||||
/// A directory that cannot be read returns before reconciling, so nothing
|
/// A directory that cannot be read returns before reconciling, so nothing
|
||||||
/// under it is ever deleted — an unreadable directory must not read as an
|
/// under it is ever deleted — an unreadable directory must not read as an
|
||||||
/// empty one.
|
/// empty one.
|
||||||
|
///
|
||||||
|
/// Each returned file carries whatever this read already told us about it, so
|
||||||
|
/// that [`prepare`] does not have to ask the filesystem twice. See
|
||||||
|
/// [`PendingFile`].
|
||||||
fn read_directory(
|
fn read_directory(
|
||||||
dir: &Path,
|
dir: &Path,
|
||||||
rows: &Arc<DirRows>,
|
rows: &Arc<DirRows>,
|
||||||
ctx: &Ctx,
|
ctx: &Ctx,
|
||||||
found: &mut Vec<Found>,
|
found: &mut Vec<Found>,
|
||||||
stale: &mut Vec<String>,
|
stale: &mut Vec<String>,
|
||||||
) -> Vec<PathBuf> {
|
) -> Vec<PendingFile> {
|
||||||
let entries = match fs::read_dir(dir) {
|
let entries = match fs::read_dir(dir) {
|
||||||
Ok(entries) => entries,
|
Ok(entries) => entries,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
// Not the same as "this directory is empty": see UnreadableDirs.
|
// Not the same as "this directory is empty": see UnreadableDirs.
|
||||||
|
// Throttled because a permission-denied subtree produces one of
|
||||||
|
// these per directory; `UnreadableDirs` records every one of them
|
||||||
|
// regardless, and the run reports the total.
|
||||||
|
if UNREADABLE_WARNINGS.allow() {
|
||||||
crate::log_warn!("cannot read {}: {}", dir.display(), e);
|
crate::log_warn!("cannot read {}: {}", dir.display(), e);
|
||||||
|
}
|
||||||
ctx.unreadable.record(dir.to_path_buf());
|
ctx.unreadable.record(dir.to_path_buf());
|
||||||
return Vec::new();
|
return Vec::new();
|
||||||
}
|
}
|
||||||
|
|
@ -452,7 +552,9 @@ fn read_directory(
|
||||||
let entry = match entry {
|
let entry = match entry {
|
||||||
Ok(entry) => entry,
|
Ok(entry) => entry,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
|
if UNREADABLE_WARNINGS.allow() {
|
||||||
crate::log_warn!("cannot read an entry of {}: {}", dir.display(), e);
|
crate::log_warn!("cannot read an entry of {}: {}", dir.display(), e);
|
||||||
|
}
|
||||||
ctx.unreadable.record(dir.to_path_buf());
|
ctx.unreadable.record(dir.to_path_buf());
|
||||||
// The listing is now incomplete, so it cannot be used to
|
// The listing is now incomplete, so it cannot be used to
|
||||||
// decide what is missing: an entry we failed to read would
|
// decide what is missing: an entry we failed to read would
|
||||||
|
|
@ -465,17 +567,45 @@ fn read_directory(
|
||||||
let name = entry.file_name();
|
let name = entry.file_name();
|
||||||
let name = name.to_string_lossy();
|
let name = name.to_string_lossy();
|
||||||
// `entry.metadata()` is only consulted on Windows, where it is free —
|
// `entry.metadata()` is only consulted on Windows, where it is free —
|
||||||
// the attributes came back with the directory read. On Unix the
|
// the attributes came back with the directory read, and it reports the
|
||||||
// closure is never called, so this stays at zero extra syscalls.
|
// entry itself rather than a link target, which is what
|
||||||
if !ctx.include_hidden && crate::platform::entry_is_hidden(&name, || entry.metadata().ok())
|
// `entry_hidden_reason` requires. On Unix the closure is never called,
|
||||||
|
// so this stays at zero extra syscalls.
|
||||||
|
if !ctx.include_hidden {
|
||||||
|
if let Some(reason) =
|
||||||
|
crate::platform::entry_hidden_reason(&name, || entry.metadata().ok())
|
||||||
{
|
{
|
||||||
|
match reason {
|
||||||
|
crate::platform::HiddenReason::DotPrefix => {
|
||||||
|
ctx.pruned.dot_named.fetch_add(1, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
crate::platform::HiddenReason::Attribute => {
|
||||||
|
ctx.pruned.attribute.fetch_add(1, Ordering::Relaxed);
|
||||||
|
// Only the attribute case is announced, and only for a
|
||||||
|
// directory. A dot prefix explains itself and an ignore
|
||||||
|
// pattern is something the user typed, but a plainly
|
||||||
|
// visible folder skipped over an attribute Explorer does
|
||||||
|
// not show has no other way of being discovered — which
|
||||||
|
// is how a whole cloud-sync tree went missing in silence.
|
||||||
|
if entry.file_type().is_ok_and(|ft| ft.is_dir()) {
|
||||||
|
crate::log_info!(
|
||||||
|
"skipping {}: hidden attribute set (enable \"include hidden \
|
||||||
|
files\" to index it)",
|
||||||
|
entry.path().display()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if ctx.ignore.matches_component(&name) {
|
if ctx.ignore.matches_component(&name) {
|
||||||
|
ctx.pruned.ignored.fetch_add(1, Ordering::Relaxed);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let path = entry.path();
|
let path = entry.path();
|
||||||
if ctx.ignore.matches_path_pattern(&path) {
|
if ctx.ignore.matches_path_pattern(&path) {
|
||||||
|
ctx.pruned.ignored.fetch_add(1, Ordering::Relaxed);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -507,12 +637,12 @@ fn read_directory(
|
||||||
// path is what the index stores, and pushing only canonical
|
// path is what the index stores, and pushing only canonical
|
||||||
// directories is what keeps `seen_dirs` able to break cycles.
|
// directories is what keeps `seen_dirs` able to break cycles.
|
||||||
//
|
//
|
||||||
// Normalized like the roots (walk_parallel), or on Windows the
|
// Normalized like the roots (`walk_indexable_files`), or on
|
||||||
// target keeps `canonicalize`'s `\\?\` prefix: every path below
|
// Windows the target keeps `canonicalize`'s `\\?\` prefix:
|
||||||
// it would be spelled differently from the plainly-spelled
|
// every path below it would be spelled differently from the
|
||||||
// roots, so full-path ignore patterns would never match under a
|
// plainly-spelled roots, so full-path ignore patterns would
|
||||||
// followed junction and `seen_dirs` could not dedup against an
|
// never match under a followed junction and `seen_dirs` could
|
||||||
// overlapping root.
|
// not dedup against an overlapping root.
|
||||||
if let Ok(target) = path.canonicalize() {
|
if let Ok(target) = path.canonicalize() {
|
||||||
let target = PathBuf::from(path_to_db_string(&target));
|
let target = PathBuf::from(path_to_db_string(&target));
|
||||||
match fs::metadata(&target) {
|
match fs::metadata(&target) {
|
||||||
|
|
@ -528,7 +658,13 @@ fn read_directory(
|
||||||
}
|
}
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
present.insert(name.into_owned());
|
present.insert(name.into_owned());
|
||||||
files.push(path);
|
// Windows already sent this file's size and mtime back with its
|
||||||
|
// name, and `DirEntry::metadata` there is a copy of that buffer
|
||||||
|
// rather than a syscall — so carrying it means `prepare` never
|
||||||
|
// opens the file just to ask the same question again. `None` on
|
||||||
|
// Unix, and on any reparse point: see `entry_cached_metadata`.
|
||||||
|
let cached = crate::platform::entry_cached_metadata(|| entry.metadata().ok());
|
||||||
|
files.push(PendingFile { path, cached });
|
||||||
}
|
}
|
||||||
// Type unknown: the entry exists but we could not classify it.
|
// Type unknown: the entry exists but we could not classify it.
|
||||||
// Mark it present so an existing row survives — seen, not deleted.
|
// Mark it present so an existing row survives — seen, not deleted.
|
||||||
|
|
@ -583,9 +719,14 @@ pub fn path_digest(path: &str) -> u128 {
|
||||||
u128::from_be_bytes(bytes)
|
u128::from_be_bytes(bytes)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One `stat`, then classify; only files that are actually going to be
|
/// At most one `stat`, then classify; only files that are actually going to be
|
||||||
/// written get opened, and small text files are finished outright.
|
/// written get opened, and small text files are finished outright.
|
||||||
fn prepare(path: PathBuf, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
|
///
|
||||||
|
/// "At most", because on Windows the directory read already answered the
|
||||||
|
/// question and [`PendingFile::cached`] carries the answer — see
|
||||||
|
/// [`crate::platform::metadata_or_stat`].
|
||||||
|
fn prepare(file: PendingFile, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
|
||||||
|
let PendingFile { path, cached } = file;
|
||||||
let db_path = path_to_db_string(&path);
|
let db_path = path_to_db_string(&path);
|
||||||
let digest = path_digest(&db_path);
|
let digest = path_digest(&db_path);
|
||||||
let aliased = matches!(known, Known::Exact(_));
|
let aliased = matches!(known, Known::Exact(_));
|
||||||
|
|
@ -598,7 +739,11 @@ fn prepare(path: PathBuf, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
|
||||||
return WalkedFile::skipped(db_path, digest, aliased);
|
return WalkedFile::skipped(db_path, digest, aliased);
|
||||||
}
|
}
|
||||||
|
|
||||||
let Ok(meta) = fs::metadata(&path) else {
|
// On Windows this is the directory read's own copy of the entry: an
|
||||||
|
// unchanged file now costs no syscall at all, and a changed one costs only
|
||||||
|
// the open the hasher was going to do anyway. On Unix, and for every
|
||||||
|
// reparse point, it is the same single `stat` as before.
|
||||||
|
let Ok(meta) = crate::platform::metadata_or_stat(&path, cached) else {
|
||||||
// Seen but unreadable. Emitting it anyway keeps its index row alive:
|
// Seen but unreadable. Emitting it anyway keeps its index row alive:
|
||||||
// a transient stat failure must not read as "deleted".
|
// a transient stat failure must not read as "deleted".
|
||||||
return WalkedFile::skipped(db_path, digest, aliased);
|
return WalkedFile::skipped(db_path, digest, aliased);
|
||||||
|
|
@ -663,8 +808,11 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
|
||||||
Job::Files(files, rows) => (files, rows),
|
Job::Files(files, rows) => (files, rows),
|
||||||
Job::Alias(path, stored) => {
|
Job::Alias(path, stored) => {
|
||||||
slot.finish(found);
|
slot.finish(found);
|
||||||
|
// Reached through `canonicalize`, not through a directory
|
||||||
|
// entry, so there is nothing cached to carry.
|
||||||
|
let file = PendingFile::uncached(path);
|
||||||
if tx
|
if tx
|
||||||
.send(WalkEvent::File(prepare(path, Known::Exact(stored), ctx)))
|
.send(WalkEvent::File(prepare(file, Known::Exact(stored), ctx)))
|
||||||
.is_err()
|
.is_err()
|
||||||
{
|
{
|
||||||
shared.shutdown();
|
shared.shutdown();
|
||||||
|
|
@ -684,13 +832,13 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
for path in files {
|
for file in files {
|
||||||
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
|
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
|
||||||
shared.shutdown();
|
shared.shutdown();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if tx
|
if tx
|
||||||
.send(WalkEvent::File(prepare(path, Known::InDir(&rows), ctx)))
|
.send(WalkEvent::File(prepare(file, Known::InDir(&rows), ctx)))
|
||||||
.is_err()
|
.is_err()
|
||||||
{
|
{
|
||||||
// Receiver gone: the run was stopped or failed. Not an error.
|
// Receiver gone: the run was stopped or failed. Not an error.
|
||||||
|
|
@ -705,7 +853,8 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
|
||||||
/// read-only connection.
|
/// read-only connection.
|
||||||
///
|
///
|
||||||
/// One per walk. The alternative — a connection per worker — would multiply
|
/// One per walk. The alternative — a connection per worker — would multiply
|
||||||
/// SQLite's page cache by the pool size; see [`PRAGMAS_WALK_READER`]. Every
|
/// SQLite's page cache by the pool size; see
|
||||||
|
/// [`crate::db::schema::PRAGMAS_WALK_READER`]. Every
|
||||||
/// query here is a single index lookup, so one thread stays far ahead of a
|
/// query here is a single index lookup, so one thread stays far ahead of a
|
||||||
/// pool bound by `stat` latency.
|
/// pool bound by `stat` latency.
|
||||||
///
|
///
|
||||||
|
|
@ -753,6 +902,9 @@ fn prefetcher(shared: &Shared, db_path: &str) {
|
||||||
/// stops the workers and joins them.
|
/// stops the workers and joins them.
|
||||||
pub struct ParallelWalk {
|
pub struct ParallelWalk {
|
||||||
rx: Option<mpsc::Receiver<WalkEvent>>,
|
rx: Option<mpsc::Receiver<WalkEvent>>,
|
||||||
|
/// One event pulled off the channel by [`ParallelWalk::wait_ready`] and not
|
||||||
|
/// yet handed to [`ParallelWalk::try_next`]. See `wait_ready`.
|
||||||
|
pending: Option<WalkEvent>,
|
||||||
handles: Vec<JoinHandle<()>>,
|
handles: Vec<JoinHandle<()>>,
|
||||||
/// Joined by [`ParallelWalk::finish`] alongside the workers. Held
|
/// Joined by [`ParallelWalk::finish`] alongside the workers. Held
|
||||||
/// separately only so a failure to open its connection is attributable.
|
/// separately only so a failure to open its connection is attributable.
|
||||||
|
|
@ -768,6 +920,12 @@ impl ParallelWalk {
|
||||||
&self.ctx.unreadable
|
&self.ctx.unreadable
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// How many entries each filter rejected. Final on the same terms as
|
||||||
|
/// [`ParallelWalk::unreadable`].
|
||||||
|
pub fn pruned(&self) -> &PruneCounts {
|
||||||
|
&self.ctx.pruned
|
||||||
|
}
|
||||||
|
|
||||||
/// Every canonical directory the walk queued, in `files.parent` spelling.
|
/// Every canonical directory the walk queued, in `files.parent` spelling.
|
||||||
///
|
///
|
||||||
/// The caller's vanished-directory sweep needs this: a directory deleted
|
/// The caller's vanished-directory sweep needs this: a directory deleted
|
||||||
|
|
@ -802,7 +960,10 @@ impl ParallelWalk {
|
||||||
/// identical from the receiving end: both close the channel, so iteration
|
/// identical from the receiving end: both close the channel, so iteration
|
||||||
/// simply ends. Treating a panicked walk as a completed one would hand
|
/// simply ends. Treating a panicked walk as a completed one would hand
|
||||||
/// stale cleanup a partial file set and delete everything the dead workers
|
/// stale cleanup a partial file set and delete everything the dead workers
|
||||||
/// never reached.
|
/// never reached. So: join before deciding anything about what the walk
|
||||||
|
/// saw. This is the reference statement of that rule; the content pass
|
||||||
|
/// ([`crate::content::ContentPass::finish`]) and the writer loop's
|
||||||
|
/// `TryNext::Finished` arms follow it.
|
||||||
pub fn finish(&mut self) -> bool {
|
pub fn finish(&mut self) -> bool {
|
||||||
// Dropping the receiver first releases any worker parked in `send`.
|
// Dropping the receiver first releases any worker parked in `send`.
|
||||||
self.rx = None;
|
self.rx = None;
|
||||||
|
|
@ -852,18 +1013,73 @@ pub(crate) fn try_recv_next<T>(rx: Option<&mpsc::Receiver<T>>) -> TryNext<T> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// [`try_recv_next`], but willing to wait up to `timeout` for something to
|
||||||
|
/// arrive.
|
||||||
|
///
|
||||||
|
/// What the writer loop backs off with instead of `thread::sleep`. A sleep is
|
||||||
|
/// the wrong instrument twice over: it ignores work that lands a microsecond
|
||||||
|
/// later, and on Windows the default timer resolution is 15.6 ms, so a 2 ms
|
||||||
|
/// backoff actually stalls for 15.6 — nearly eight times the intended pause,
|
||||||
|
/// every time the channels run momentarily dry. `recv_timeout` parks on the
|
||||||
|
/// channel's own condition variable, so a sender wakes it immediately and the
|
||||||
|
/// timeout is only the ceiling.
|
||||||
|
pub(crate) fn recv_next_timeout<T>(
|
||||||
|
rx: Option<&mpsc::Receiver<T>>,
|
||||||
|
timeout: std::time::Duration,
|
||||||
|
) -> TryNext<T> {
|
||||||
|
match rx {
|
||||||
|
None => TryNext::Finished,
|
||||||
|
Some(rx) => match rx.recv_timeout(timeout) {
|
||||||
|
Ok(item) => TryNext::Item(item),
|
||||||
|
Err(mpsc::RecvTimeoutError::Timeout) => TryNext::Empty,
|
||||||
|
Err(mpsc::RecvTimeoutError::Disconnected) => TryNext::Finished,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl ParallelWalk {
|
impl ParallelWalk {
|
||||||
/// Non-blocking variant of `next`, for callers multiplexing several
|
/// Non-blocking variant of `next`, for callers multiplexing several
|
||||||
/// walks (the per-root writer loop).
|
/// walks (the per-root writer loop).
|
||||||
pub fn try_next(&mut self) -> TryNext<WalkEvent> {
|
pub fn try_next(&mut self) -> TryNext<WalkEvent> {
|
||||||
|
if let Some(event) = self.pending.take() {
|
||||||
|
return TryNext::Item(event);
|
||||||
|
}
|
||||||
try_recv_next(self.rx.as_ref())
|
try_recv_next(self.rx.as_ref())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Wait up to `timeout` for this walk to produce something, holding
|
||||||
|
/// whatever arrives for the next [`ParallelWalk::try_next`].
|
||||||
|
///
|
||||||
|
/// The writer loop's idle backoff. It multiplexes several walks, so it
|
||||||
|
/// cannot simply block on one of them and consume the result — hence the
|
||||||
|
/// one-slot pushback: the event is taken off the channel, but the loop still
|
||||||
|
/// sees it in its normal round-robin order.
|
||||||
|
///
|
||||||
|
/// Returns whether anything is now ready.
|
||||||
|
pub fn wait_ready(&mut self, timeout: std::time::Duration) -> bool {
|
||||||
|
if self.pending.is_some() {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
match recv_next_timeout(self.rx.as_ref(), timeout) {
|
||||||
|
TryNext::Item(event) => {
|
||||||
|
self.pending = Some(event);
|
||||||
|
true
|
||||||
|
}
|
||||||
|
// A finished walk is "ready" in the sense the caller cares about:
|
||||||
|
// there is something to do (notice it ended), so do not keep waiting.
|
||||||
|
TryNext::Finished => true,
|
||||||
|
TryNext::Empty => false,
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Iterator for ParallelWalk {
|
impl Iterator for ParallelWalk {
|
||||||
type Item = WalkEvent;
|
type Item = WalkEvent;
|
||||||
|
|
||||||
fn next(&mut self) -> Option<WalkEvent> {
|
fn next(&mut self) -> Option<WalkEvent> {
|
||||||
|
if let Some(event) = self.pending.take() {
|
||||||
|
return Some(event);
|
||||||
|
}
|
||||||
self.rx.as_ref()?.recv().ok()
|
self.rx.as_ref()?.recv().ok()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -938,6 +1154,7 @@ pub fn walk_indexable_files(
|
||||||
follow_symlinks,
|
follow_symlinks,
|
||||||
include_hidden,
|
include_hidden,
|
||||||
ignore,
|
ignore,
|
||||||
|
pruned: PruneCounts::default(),
|
||||||
config,
|
config,
|
||||||
registry,
|
registry,
|
||||||
unreadable: UnreadableDirs::default(),
|
unreadable: UnreadableDirs::default(),
|
||||||
|
|
@ -976,6 +1193,7 @@ pub fn walk_indexable_files(
|
||||||
|
|
||||||
ParallelWalk {
|
ParallelWalk {
|
||||||
rx: Some(rx),
|
rx: Some(rx),
|
||||||
|
pending: None,
|
||||||
handles,
|
handles,
|
||||||
prefetch: Some(prefetch),
|
prefetch: Some(prefetch),
|
||||||
shared,
|
shared,
|
||||||
|
|
@ -1007,18 +1225,7 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn tmp_tree(tag: &str) -> PathBuf {
|
fn tmp_tree(tag: &str) -> PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
crate::testutil::scratch_dir(tag)
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-pwalk-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
fs::create_dir_all(&p).unwrap();
|
|
||||||
p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn touch(p: &Path) {
|
fn touch(p: &Path) {
|
||||||
|
|
@ -1032,16 +1239,7 @@ mod tests {
|
||||||
/// caller passes in, so these tests build the state they are testing
|
/// caller passes in, so these tests build the state they are testing
|
||||||
/// against the same way the indexer does.
|
/// against the same way the indexer does.
|
||||||
fn db_with(tag: &str, rows: &[(String, u64)]) -> PathBuf {
|
fn db_with(tag: &str, rows: &[(String, u64)]) -> PathBuf {
|
||||||
let mut p = std::env::temp_dir();
|
let p = crate::testutil::scratch_dir(tag).join("index.sqlite");
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-pwalk-db-{}-{}-{}.sqlite",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
let conn = crate::db::open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
|
let conn = crate::db::open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
|
||||||
for (path, mtime) in rows {
|
for (path, mtime) in rows {
|
||||||
let as_path = Path::new(path);
|
let as_path = Path::new(path);
|
||||||
|
|
@ -1234,6 +1432,74 @@ mod tests {
|
||||||
fs::remove_dir_all(&root).ok();
|
fs::remove_dir_all(&root).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The walk's mtime and a `stat`'s mtime must be the same number.
|
||||||
|
///
|
||||||
|
/// On Windows the walk now reads mtime out of the directory entry while the
|
||||||
|
/// *watcher* writes its rows from `fs::metadata`; if the two ever disagreed,
|
||||||
|
/// every run would reclassify files nothing had touched and the index would
|
||||||
|
/// churn forever. Seeding the index the watcher's way and demanding the walk
|
||||||
|
/// call every file `Skip` is what pins them together.
|
||||||
|
///
|
||||||
|
/// `unchanged_files_are_never_opened` cannot catch this on its own: both of
|
||||||
|
/// its walks read from the same source, so they agree with each other even
|
||||||
|
/// when both disagree with a stat. Vacuous on Unix, where there is only ever
|
||||||
|
/// one source; on Windows it is the whole guarantee.
|
||||||
|
#[test]
|
||||||
|
fn a_walk_agrees_with_a_stat_seeded_index() {
|
||||||
|
let root = tmp_tree("stat-seeded");
|
||||||
|
touch(&root.join("a.txt"));
|
||||||
|
touch(&root.join("sub/b.txt"));
|
||||||
|
|
||||||
|
let seeded: Vec<(String, u64)> = [root.join("a.txt"), root.join("sub/b.txt")]
|
||||||
|
.iter()
|
||||||
|
.map(|p| {
|
||||||
|
let mtime = fs::metadata(p)
|
||||||
|
.unwrap()
|
||||||
|
.modified()
|
||||||
|
.unwrap()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.unwrap()
|
||||||
|
.as_secs();
|
||||||
|
(path_to_db_string(p), mtime)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let files = walk(&root, &db_with("stat-seeded", &seeded));
|
||||||
|
assert_eq!(files.len(), 2);
|
||||||
|
for f in &files {
|
||||||
|
assert_eq!(
|
||||||
|
f.action,
|
||||||
|
FileIndexAction::Skip,
|
||||||
|
"the directory read disagreed with a stat about {}",
|
||||||
|
f.path
|
||||||
|
);
|
||||||
|
assert!(f.record.is_none());
|
||||||
|
}
|
||||||
|
fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Windows: the three fields `prepare` and `prepare_file_record` read out of
|
||||||
|
/// the cached buffer, pinned against what a `stat` would have said — for the
|
||||||
|
/// case that now skips the `stat` entirely.
|
||||||
|
#[test]
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn cached_directory_metadata_matches_a_stat_field_for_field() {
|
||||||
|
let root = tmp_tree("cached-meta");
|
||||||
|
touch(&root.join("a.txt"));
|
||||||
|
fs::write(root.join("b.bin"), vec![0u8; 5000]).unwrap();
|
||||||
|
|
||||||
|
for entry in fs::read_dir(&root).unwrap() {
|
||||||
|
let entry = entry.unwrap();
|
||||||
|
let cached = crate::platform::entry_cached_metadata(|| entry.metadata().ok())
|
||||||
|
.expect("a plain file is served from the directory read");
|
||||||
|
let fresh = fs::metadata(entry.path()).unwrap();
|
||||||
|
assert_eq!(cached.is_file(), fresh.is_file());
|
||||||
|
assert_eq!(cached.len(), fresh.len());
|
||||||
|
assert_eq!(cached.modified().unwrap(), fresh.modified().unwrap());
|
||||||
|
}
|
||||||
|
fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn every_seen_file_is_reported_even_when_it_cannot_be_read() {
|
fn every_seen_file_is_reported_even_when_it_cannot_be_read() {
|
||||||
// A path missing from the stream gets its index row deleted, so
|
// A path missing from the stream gets its index row deleted, so
|
||||||
|
|
@ -1472,6 +1738,103 @@ mod tests {
|
||||||
fs::remove_dir_all(&root).ok();
|
fs::remove_dir_all(&root).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The counters behind the one-line summary a run logs.
|
||||||
|
///
|
||||||
|
/// The property worth pinning is that a pruned *directory* costs one
|
||||||
|
/// increment rather than one per file beneath it — the subtree is never
|
||||||
|
/// enumerated, which is exactly why logging per entry was rejected and
|
||||||
|
/// counting was not.
|
||||||
|
#[test]
|
||||||
|
fn pruned_entries_are_counted_by_reason() {
|
||||||
|
let root = tmp_tree("prune-counts");
|
||||||
|
touch(&root.join("keep.txt"));
|
||||||
|
touch(&root.join("sub/keep2.txt"));
|
||||||
|
touch(&root.join("sub/skip.tmp"));
|
||||||
|
// Two files below, one prune.
|
||||||
|
touch(&root.join(".hidden/inside.txt"));
|
||||||
|
touch(&root.join(".hidden/also-inside.txt"));
|
||||||
|
touch(&root.join(".dotfile"));
|
||||||
|
// Three levels below, still one prune.
|
||||||
|
touch(&root.join("node_modules/dep/lib/index.js"));
|
||||||
|
|
||||||
|
let ignore =
|
||||||
|
IgnoreSet::compile(&["*.tmp".to_string(), "node_modules".to_string()]).unwrap();
|
||||||
|
let mut walk = walk_indexable_files(
|
||||||
|
&[root.to_string_lossy().into_owned()],
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
ignore,
|
||||||
|
empty_db("prune-counts").to_str().unwrap(),
|
||||||
|
Config::default(),
|
||||||
|
Arc::new(Registry::default_set()),
|
||||||
|
Arc::new(AtomicBool::new(false)),
|
||||||
|
Arc::new(AtomicBool::new(false)),
|
||||||
|
4,
|
||||||
|
);
|
||||||
|
let files: Vec<WalkedFile> = (&mut walk)
|
||||||
|
.filter_map(|e| match e {
|
||||||
|
WalkEvent::File(f) => Some(f),
|
||||||
|
WalkEvent::Stale(_) => None,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
|
||||||
|
|
||||||
|
let pruned = walk.pruned();
|
||||||
|
assert_eq!(
|
||||||
|
pruned.dot_named.load(Ordering::Relaxed),
|
||||||
|
2,
|
||||||
|
"`.hidden` and `.dotfile` — not the two files inside `.hidden`"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
pruned.ignored.load(Ordering::Relaxed),
|
||||||
|
2,
|
||||||
|
"`skip.tmp` and `node_modules` — not `index.js` three levels down"
|
||||||
|
);
|
||||||
|
// Attributes are a Windows concept; on Linux nothing can reach this
|
||||||
|
// counter, and on Windows a temp tree carries no Hidden bit.
|
||||||
|
assert_eq!(pruned.attribute.load(Ordering::Relaxed), 0);
|
||||||
|
assert_eq!(pruned.total(), 4);
|
||||||
|
|
||||||
|
let summary = pruned.summary().expect("something was pruned");
|
||||||
|
assert!(summary.contains("4 entries"), "{}", summary);
|
||||||
|
|
||||||
|
fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A clean tree must add no line to a log whose whole budget is 5,000
|
||||||
|
/// entries.
|
||||||
|
#[test]
|
||||||
|
fn a_tree_with_nothing_pruned_reports_no_summary() {
|
||||||
|
let root = tmp_tree("prune-none");
|
||||||
|
touch(&root.join("keep.txt"));
|
||||||
|
touch(&root.join("sub/keep2.txt"));
|
||||||
|
|
||||||
|
let mut walk = walk_indexable_files(
|
||||||
|
&[root.to_string_lossy().into_owned()],
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
IgnoreSet::compile(&[]).unwrap(),
|
||||||
|
empty_db("prune-none").to_str().unwrap(),
|
||||||
|
Config::default(),
|
||||||
|
Arc::new(Registry::default_set()),
|
||||||
|
Arc::new(AtomicBool::new(false)),
|
||||||
|
Arc::new(AtomicBool::new(false)),
|
||||||
|
4,
|
||||||
|
);
|
||||||
|
let files: Vec<WalkedFile> = (&mut walk)
|
||||||
|
.filter_map(|e| match e {
|
||||||
|
WalkEvent::File(f) => Some(f),
|
||||||
|
WalkEvent::Stale(_) => None,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
|
||||||
|
|
||||||
|
assert_eq!(walk.pruned().total(), 0);
|
||||||
|
assert!(walk.pruned().summary().is_none());
|
||||||
|
|
||||||
|
fs::remove_dir_all(&root).ok();
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_directory_reports_rows_with_no_file_behind_them() {
|
fn a_directory_reports_rows_with_no_file_behind_them() {
|
||||||
// The per-directory diff, at the level it is computed: one listing
|
// The per-directory diff, at the level it is computed: one listing
|
||||||
|
|
|
||||||
|
|
@ -160,7 +160,7 @@ impl std::error::Error for WatchError {}
|
||||||
|
|
||||||
/// Render the directory cap compactly: the default 128_000 reads "128k".
|
/// Render the directory cap compactly: the default 128_000 reads "128k".
|
||||||
fn fmt_cap(cap: usize) -> String {
|
fn fmt_cap(cap: usize) -> String {
|
||||||
if cap >= 1000 && cap % 1000 == 0 {
|
if cap >= 1000 && cap.is_multiple_of(1000) {
|
||||||
format!("{}k", cap / 1000)
|
format!("{}k", cap / 1000)
|
||||||
} else {
|
} else {
|
||||||
cap.to_string()
|
cap.to_string()
|
||||||
|
|
@ -266,12 +266,12 @@ impl WatchRegistry {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Forget `dir` and every watched directory beneath it, returning how
|
/// Forget `dir` and every watched directory beneath it, returning how
|
||||||
/// many were dropped.
|
/// many were dropped. Containment is component-wise, per
|
||||||
|
/// [`crate::file_handling::UnreadableDirs::covers`].
|
||||||
///
|
///
|
||||||
/// `Path::starts_with` compares whole components, so `/a/bc` is not
|
/// The kernel drops watches for deleted directories on its own; unwatching
|
||||||
/// treated as living under `/a/b`. The kernel drops watches for deleted
|
/// anyway keeps notify's internal descriptor map from growing across a long
|
||||||
/// directories on its own; unwatching anyway keeps notify's internal
|
/// session of directory churn.
|
||||||
/// descriptor map from growing across a long session of directory churn.
|
|
||||||
fn remove_tree(&mut self, dir: &Path) -> usize {
|
fn remove_tree(&mut self, dir: &Path) -> usize {
|
||||||
// The scan below is O(watched dirs), and the event loop calls this for
|
// The scan below is O(watched dirs), and the event loop calls this for
|
||||||
// every Remove — files included. Deleting a directory of 10k files
|
// every Remove — files included. Deleting a directory of 10k files
|
||||||
|
|
@ -564,7 +564,7 @@ fn run_loop(rx: mpsc::Receiver<NotifyEvent>, ctx: LoopCtx) {
|
||||||
|
|
||||||
// Periodic GC of abandoned throttle entries.
|
// Periodic GC of abandoned throttle entries.
|
||||||
tick_counter = tick_counter.wrapping_add(1);
|
tick_counter = tick_counter.wrapping_add(1);
|
||||||
if tick_counter % prune_interval_ticks == 0 {
|
if tick_counter.is_multiple_of(prune_interval_ticks) {
|
||||||
let max_age = ctx
|
let max_age = ctx
|
||||||
.config
|
.config
|
||||||
.throttle_window
|
.throttle_window
|
||||||
|
|
@ -809,19 +809,8 @@ mod tests {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Unique temp directory; the repo has no `tempfile` dev-dependency.
|
|
||||||
fn tmp_dir(tag: &str) -> PathBuf {
|
fn tmp_dir(tag: &str) -> PathBuf {
|
||||||
let dir = std::env::temp_dir().join(format!(
|
crate::testutil::scratch_dir(tag)
|
||||||
"qs-watch-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&dir).unwrap();
|
|
||||||
dir
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A registry with `dirs` seeded directly, so the pure set logic can be
|
/// A registry with `dirs` seeded directly, so the pure set logic can be
|
||||||
|
|
@ -903,8 +892,10 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
let (sink, got) = sink_to_vec();
|
let (sink, got) = sink_to_vec();
|
||||||
let mut config = WatcherConfig::default();
|
let config = WatcherConfig {
|
||||||
config.max_dirs_per_tick = 3;
|
max_dirs_per_tick: 3,
|
||||||
|
..WatcherConfig::default()
|
||||||
|
};
|
||||||
flush_ready(&mut map, &sink, &config);
|
flush_ready(&mut map, &sink, &config);
|
||||||
// Each dir contributes one event because each entry has one path.
|
// Each dir contributes one event because each entry has one path.
|
||||||
assert_eq!(got.lock().unwrap().len(), 3);
|
assert_eq!(got.lock().unwrap().len(), 3);
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
//! Integration tests for the ranked search cascade and the streaming
|
//! Integration tests for the ranked search cascade and the streaming
|
||||||
//! search service, against real temp databases.
|
//! search service, against real temp databases.
|
||||||
|
|
||||||
use std::path::PathBuf;
|
use std::path::Path;
|
||||||
use std::sync::atomic::{AtomicU64, Ordering};
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
|
@ -11,19 +11,8 @@ use quicksearch_core::mime::FileType;
|
||||||
use quicksearch_core::query::split::split_for_cascade;
|
use quicksearch_core::query::split::split_for_cascade;
|
||||||
use quicksearch_core::search::{cascade, SearchHit, SearchOptions, SearchService, SearchUpdate};
|
use quicksearch_core::search::{cascade, SearchHit, SearchOptions, SearchService, SearchUpdate};
|
||||||
|
|
||||||
fn tmp_db(tag: &str) -> PathBuf {
|
mod common;
|
||||||
let mut p = std::env::temp_dir();
|
use common::scratch_db as tmp_db;
|
||||||
p.push(format!(
|
|
||||||
"qs-cascade-{}-{}-{}.sqlite",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
struct Seeder {
|
struct Seeder {
|
||||||
conn: rusqlite::Connection,
|
conn: rusqlite::Connection,
|
||||||
|
|
@ -31,7 +20,7 @@ struct Seeder {
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Seeder {
|
impl Seeder {
|
||||||
fn new(path: &PathBuf, store_text: bool) -> Seeder {
|
fn new(path: &Path, store_text: bool) -> Seeder {
|
||||||
Seeder {
|
Seeder {
|
||||||
conn: open_or_recreate(path.to_str().unwrap(), "trigram").unwrap(),
|
conn: open_or_recreate(path.to_str().unwrap(), "trigram").unwrap(),
|
||||||
store_text,
|
store_text,
|
||||||
|
|
|
||||||
92
crates/quicksearch-core/tests/common/mod.rs
Normal file
92
crates/quicksearch-core/tests/common/mod.rs
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
//! Shared scaffolding for the integration-test binaries.
|
||||||
|
//!
|
||||||
|
//! Cargo compiles this into each `tests/*.rs` that declares `mod common;`, so
|
||||||
|
//! every binary gets its own copy and each one uses a different subset.
|
||||||
|
|
||||||
|
#![allow(dead_code)]
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use quicksearch_core::config::Config;
|
||||||
|
use quicksearch_core::db;
|
||||||
|
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
|
||||||
|
|
||||||
|
#[allow(unused_imports)]
|
||||||
|
pub use quicksearch_core::testutil::{scratch_dir, scratch_dir_canonical, touch};
|
||||||
|
|
||||||
|
/// A scratch database path under a fresh directory. The sidecars SQLite
|
||||||
|
/// creates alongside it (`-wal`, `-shm`) land in the same directory.
|
||||||
|
pub fn scratch_db(tag: &str) -> std::path::PathBuf {
|
||||||
|
scratch_dir(tag).join("index.sqlite")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How long a single indexing run may take before the test gives up. Generous:
|
||||||
|
/// CI runs these in a container against a cold page cache.
|
||||||
|
const INDEX_TIMEOUT: Duration = Duration::from_secs(120);
|
||||||
|
|
||||||
|
/// One full indexing run, awaited to completion.
|
||||||
|
///
|
||||||
|
/// Completion is read from the `last_full_index` marker rather than the status
|
||||||
|
/// enum, because `run_indexing` writes that marker only on a successful finish.
|
||||||
|
/// Polling for `IndexingStatus::Idle` instead would race: a small tree finishes
|
||||||
|
/// between two polls, leaving `Idle` ambiguous between "not started yet" and
|
||||||
|
/// "already done".
|
||||||
|
pub struct IndexOnce<'a> {
|
||||||
|
pub db: &'a Path,
|
||||||
|
pub roots: Vec<String>,
|
||||||
|
pub config: &'a Config,
|
||||||
|
/// Delete any existing completion marker first, so a second run over the
|
||||||
|
/// same index is distinguishable from the first. Off for suites that index
|
||||||
|
/// into a database whose lifecycle they are themselves testing.
|
||||||
|
pub fresh_marker: bool,
|
||||||
|
/// Poll the marker through the keyed open. An encrypted index cannot be
|
||||||
|
/// read by a plain `rusqlite::Connection::open`, so a run against one would
|
||||||
|
/// otherwise never observe its own completion and time out.
|
||||||
|
pub encrypted: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl IndexOnce<'_> {
|
||||||
|
pub fn run(mut self) {
|
||||||
|
if self.fresh_marker && self.db.exists() {
|
||||||
|
let conn = rusqlite::Connection::open(self.db).unwrap();
|
||||||
|
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let service = IndexingService::new();
|
||||||
|
service
|
||||||
|
.start_indexing(
|
||||||
|
std::mem::take(&mut self.roots),
|
||||||
|
self.db.to_string_lossy().into_owned(),
|
||||||
|
self.config.clone(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let deadline = Instant::now() + INDEX_TIMEOUT;
|
||||||
|
let mut done = false;
|
||||||
|
while Instant::now() < deadline {
|
||||||
|
if let IndexingStatus::Error(e) = service.get_status() {
|
||||||
|
panic!("indexing failed: {}", e);
|
||||||
|
}
|
||||||
|
if self.db.exists() && self.completed() {
|
||||||
|
done = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
std::thread::sleep(Duration::from_millis(10));
|
||||||
|
}
|
||||||
|
assert!(done, "indexing did not finish within {:?}", INDEX_TIMEOUT);
|
||||||
|
service.stop_indexing().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether the completion marker is present. A database mid-creation is
|
||||||
|
/// simply "not yet", not a failure — the poll comes round again.
|
||||||
|
fn completed(&self) -> bool {
|
||||||
|
let conn = if self.encrypted {
|
||||||
|
db::open_existing(&self.db.to_string_lossy(), false).ok()
|
||||||
|
} else {
|
||||||
|
rusqlite::Connection::open(self.db).ok()
|
||||||
|
};
|
||||||
|
conn.is_some_and(|c| db::repo::get_last_full_index(&c).is_some())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -7,59 +7,31 @@
|
||||||
//! do. Everything runs inside a single #[test] so the key transitions are
|
//! do. Everything runs inside a single #[test] so the key transitions are
|
||||||
//! strictly ordered.
|
//! strictly ordered.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::Path;
|
||||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
|
||||||
|
|
||||||
use quicksearch_core::config::Config;
|
use quicksearch_core::config::Config;
|
||||||
use quicksearch_core::db;
|
use quicksearch_core::db;
|
||||||
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
|
use quicksearch_core::indexing::IndexingService;
|
||||||
use quicksearch_core::security::{derive_key, salt_from_hex};
|
use quicksearch_core::security::{derive_key, salt_from_hex};
|
||||||
|
|
||||||
fn tmp_dir(tag: &str) -> PathBuf {
|
mod common;
|
||||||
let mut p = std::env::temp_dir();
|
use common::scratch_dir as tmp_dir;
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-enc-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&p).unwrap();
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Run one full index over `root` and wait for the completion marker,
|
/// Run one full index over `root` and wait for the completion marker,
|
||||||
/// reading it through the keyed open so the poll works on encrypted DBs.
|
/// reading it through the keyed open so the poll works on encrypted indexes.
|
||||||
|
///
|
||||||
|
/// The marker is deliberately *not* cleared first: this suite indexes into a
|
||||||
|
/// database whose enable/disable rebuild cycle it is itself testing, and each
|
||||||
|
/// rebuild already starts from a fresh file.
|
||||||
fn index_once(root: &Path, db_path: &Path, config: &Config) {
|
fn index_once(root: &Path, db_path: &Path, config: &Config) {
|
||||||
let service = IndexingService::new();
|
common::IndexOnce {
|
||||||
service
|
db: db_path,
|
||||||
.start_indexing(
|
roots: vec![root.to_string_lossy().into_owned()],
|
||||||
vec![root.to_string_lossy().into_owned()],
|
config,
|
||||||
db_path.to_string_lossy().into_owned(),
|
fresh_marker: false,
|
||||||
config.clone(),
|
encrypted: true,
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let deadline = Instant::now() + Duration::from_secs(120);
|
|
||||||
let mut done = false;
|
|
||||||
while Instant::now() < deadline {
|
|
||||||
if let IndexingStatus::Error(e) = service.get_status() {
|
|
||||||
panic!("indexing failed: {}", e);
|
|
||||||
}
|
}
|
||||||
if db_path.exists() {
|
.run()
|
||||||
if let Ok(conn) = db::open_existing(&db_path.to_string_lossy(), false) {
|
|
||||||
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
|
|
||||||
done = true;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
std::thread::sleep(Duration::from_millis(10));
|
|
||||||
}
|
|
||||||
assert!(done, "indexing did not finish within the timeout");
|
|
||||||
service.stop_indexing().unwrap();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn header(db_path: &Path) -> [u8; 16] {
|
fn header(db_path: &Path) -> [u8; 16] {
|
||||||
|
|
|
||||||
|
|
@ -6,74 +6,27 @@
|
||||||
//! invisible on a first index — `existing_files` is empty, so nothing is
|
//! invisible on a first index — `existing_files` is empty, so nothing is
|
||||||
//! stale — and only appears on the second run.
|
//! stale — and only appears on the second run.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::Path;
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::{Arc, Mutex};
|
||||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
use std::time::{Duration, Instant, SystemTime};
|
||||||
|
|
||||||
use quicksearch_core::config::Config;
|
use quicksearch_core::config::Config;
|
||||||
use quicksearch_core::file_handling::{extract_scope_prepare, ExtractCursor};
|
use quicksearch_core::file_handling::{extract_scope_prepare, ExtractCursor};
|
||||||
use quicksearch_core::indexing::{IndexingService, IndexingStatus, RootPhase};
|
use quicksearch_core::indexing::{IndexingService, IndexingStatus, RootPhase};
|
||||||
|
|
||||||
fn tmp_dir(tag: &str) -> PathBuf {
|
mod common;
|
||||||
let mut p = std::env::temp_dir();
|
use common::{scratch_dir as tmp_dir, touch};
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-e2e-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&p).unwrap();
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
fn touch(p: &Path, body: &[u8]) {
|
/// Run one full index over `root` and wait for it to finish.
|
||||||
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
|
|
||||||
std::fs::write(p, body).unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Run one full index and wait for it to finish.
|
|
||||||
///
|
|
||||||
/// Completion is detected via the `last_full_index` marker, which
|
|
||||||
/// `run_indexing` writes only on a successful finish. Polling the status
|
|
||||||
/// enum instead would race: a small tree finishes between two polls, so
|
|
||||||
/// `Idle` is ambiguous between "not started yet" and "already done".
|
|
||||||
fn index_once(root: &Path, db: &Path, config: &Config) {
|
fn index_once(root: &Path, db: &Path, config: &Config) {
|
||||||
if db.exists() {
|
common::IndexOnce {
|
||||||
let conn = rusqlite::Connection::open(db).unwrap();
|
db,
|
||||||
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
|
roots: vec![root.to_string_lossy().into_owned()],
|
||||||
.unwrap();
|
config,
|
||||||
|
fresh_marker: true,
|
||||||
|
encrypted: false,
|
||||||
}
|
}
|
||||||
|
.run()
|
||||||
let service = IndexingService::new();
|
|
||||||
service
|
|
||||||
.start_indexing(
|
|
||||||
vec![root.to_string_lossy().into_owned()],
|
|
||||||
db.to_string_lossy().into_owned(),
|
|
||||||
config.clone(),
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let deadline = Instant::now() + Duration::from_secs(120);
|
|
||||||
let mut done = false;
|
|
||||||
while Instant::now() < deadline {
|
|
||||||
if let IndexingStatus::Error(e) = service.get_status() {
|
|
||||||
panic!("indexing failed: {}", e);
|
|
||||||
}
|
|
||||||
if db.exists() {
|
|
||||||
if let Ok(conn) = rusqlite::Connection::open(db) {
|
|
||||||
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
|
|
||||||
done = true;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
std::thread::sleep(Duration::from_millis(10));
|
|
||||||
}
|
|
||||||
assert!(done, "indexing did not finish within the timeout");
|
|
||||||
service.stop_indexing().unwrap();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// (path, mtime, content_state) for every indexed row, ordered by path.
|
/// (path, mtime, content_state) for every indexed row, ordered by path.
|
||||||
|
|
@ -91,10 +44,10 @@ fn rows(db: &Path) -> Vec<(String, i64, i64)> {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn test_config() -> Config {
|
fn test_config() -> Config {
|
||||||
let config = Config::default();
|
|
||||||
// Keep the run to phase 1 semantics we're asserting on; extraction is
|
// Keep the run to phase 1 semantics we're asserting on; extraction is
|
||||||
// covered elsewhere.
|
// covered elsewhere.
|
||||||
config
|
Config::default()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -393,9 +346,12 @@ fn starting_a_run_claims_the_status_before_it_returns() {
|
||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
// No sleep, no poll: the very next observation must already be Running.
|
// No sleep, no poll: the very next observation must already show the run.
|
||||||
|
// `Preparing` is what a claim looks like before the command thread has
|
||||||
|
// even picked the start up — it is still joining the previous run — and
|
||||||
|
// it holds the index exactly as `Running` does.
|
||||||
assert!(
|
assert!(
|
||||||
matches!(service.get_status(), IndexingStatus::Running { .. }),
|
matches!(service.get_status(), IndexingStatus::Preparing { .. }),
|
||||||
"status must be claimed synchronously, got {:?}",
|
"status must be claimed synchronously, got {:?}",
|
||||||
service.get_status()
|
service.get_status()
|
||||||
);
|
);
|
||||||
|
|
@ -781,7 +737,9 @@ fn a_directory_that_becomes_unreadable_deletes_nothing() {
|
||||||
|
|
||||||
/// Everything about a file's indexed content that a user can observe: its
|
/// Everything about a file's indexed content that a user can observe: its
|
||||||
/// state, the stored snippet body, and its property rows.
|
/// state, the stored snippet body, and its property rows.
|
||||||
fn content_rows(db: &Path) -> Vec<(String, i64, Option<String>, Option<i64>, String)> {
|
type ContentRow = (String, i64, Option<String>, Option<i64>, String);
|
||||||
|
|
||||||
|
fn content_rows(db: &Path) -> Vec<ContentRow> {
|
||||||
let conn = rusqlite::Connection::open(db).unwrap();
|
let conn = rusqlite::Connection::open(db).unwrap();
|
||||||
let mut stmt = conn
|
let mut stmt = conn
|
||||||
.prepare(
|
.prepare(
|
||||||
|
|
@ -948,6 +906,41 @@ fn undecodable_small_files_are_reported_as_failures_not_silently_skipped() {
|
||||||
std::fs::remove_dir_all(&db_dir).ok();
|
std::fs::remove_dir_all(&db_dir).ok();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A `.doc` that is not a readable OLE2 compound file — a truncated download,
|
||||||
|
/// or something misnamed — records a failure with a reason.
|
||||||
|
///
|
||||||
|
/// This is the end-to-end shape of the legacy-Office support: the walk types
|
||||||
|
/// the file from its extension, the office extractor claims `application/
|
||||||
|
/// msword`, and the OLE2 reader either produces text or says why it could not.
|
||||||
|
/// Until that reader existed, every `.doc` took the third path instead —
|
||||||
|
/// `DONE` with empty text — which reads as "indexed, contains nothing" and is
|
||||||
|
/// indistinguishable from a genuinely empty document.
|
||||||
|
#[test]
|
||||||
|
fn an_unreadable_legacy_office_file_fails_with_a_reason() {
|
||||||
|
let root = tmp_dir("legacy-doc");
|
||||||
|
let db_dir = tmp_dir("legacy-doc-db");
|
||||||
|
let db = db_dir.join("index.sqlite");
|
||||||
|
|
||||||
|
touch(&root.join("broken.doc"), b"D0CF11E0 this is not really a compound file");
|
||||||
|
index_once(&root, &db, &Config::default());
|
||||||
|
|
||||||
|
let conn = rusqlite::Connection::open(&db).unwrap();
|
||||||
|
let (state, msg): (i64, Option<String>) = conn
|
||||||
|
.query_row(
|
||||||
|
"SELECT content_state, failure_msg FROM files WHERE path LIKE '%broken.doc'",
|
||||||
|
[],
|
||||||
|
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(state, 2, "an unreadable .doc is FAILED, not DONE-with-no-text");
|
||||||
|
let msg = msg.unwrap_or_default();
|
||||||
|
assert!(msg.contains("broken.doc"), "names the file: {msg}");
|
||||||
|
assert!(msg.contains("compound file"), "says what went wrong: {msg}");
|
||||||
|
|
||||||
|
std::fs::remove_dir_all(&root).ok();
|
||||||
|
std::fs::remove_dir_all(&db_dir).ok();
|
||||||
|
}
|
||||||
|
|
||||||
/// The text sniff end-to-end: extensionless text files (README, Makefile,
|
/// The text sniff end-to-end: extensionless text files (README, Makefile,
|
||||||
/// go.sum) are content-indexed off their head bytes, while an extensionless
|
/// go.sum) are content-indexed off their head bytes, while an extensionless
|
||||||
/// binary blob stays NA.
|
/// binary blob stays NA.
|
||||||
|
|
@ -1306,6 +1299,10 @@ fn a_heavy_root_does_not_stall_a_light_one() {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// The run is claimed but has not reached its walk yet; there is
|
||||||
|
// nothing to sample, and breaking here would end the watch before
|
||||||
|
// the run it is watching had started.
|
||||||
|
IndexingStatus::Preparing { .. } => {}
|
||||||
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
||||||
_ => break,
|
_ => break,
|
||||||
}
|
}
|
||||||
|
|
@ -1410,7 +1407,10 @@ fn the_wal_stays_bounded_during_a_run() {
|
||||||
}
|
}
|
||||||
last = len;
|
last = len;
|
||||||
match service.get_status() {
|
match service.get_status() {
|
||||||
IndexingStatus::Running { .. } => {}
|
// Preparing included: the run is claimed but has not opened the
|
||||||
|
// database yet, so there is no log to watch and nothing to stop
|
||||||
|
// watching for either.
|
||||||
|
IndexingStatus::Running { .. } | IndexingStatus::Preparing { .. } => {}
|
||||||
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
||||||
_ => break,
|
_ => break,
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -8,69 +8,28 @@
|
||||||
//! assertion a regression that quietly reintroduces the rebuild would pass
|
//! assertion a regression that quietly reintroduces the rebuild would pass
|
||||||
//! every other check in this file.
|
//! every other check in this file.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::Path;
|
||||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
use std::sync::atomic::AtomicBool;
|
||||||
|
use std::time::Instant;
|
||||||
|
|
||||||
use quicksearch_core::config::{diff_actions, Config};
|
use quicksearch_core::config::{diff_actions, Config};
|
||||||
use quicksearch_core::db;
|
use quicksearch_core::db;
|
||||||
use quicksearch_core::extract::Registry;
|
use quicksearch_core::extract::Registry;
|
||||||
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
|
|
||||||
use quicksearch_core::scope::{advance, WorkCursor, SLICE};
|
use quicksearch_core::scope::{advance, WorkCursor, SLICE};
|
||||||
|
|
||||||
fn tmp_dir(tag: &str) -> PathBuf {
|
mod common;
|
||||||
let mut p = std::env::temp_dir();
|
use common::{scratch_dir_canonical as tmp_dir, touch};
|
||||||
p.push(format!(
|
|
||||||
"quicksearch-reconcile-{}-{}-{}",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
std::fs::create_dir_all(&p).unwrap();
|
|
||||||
std::fs::canonicalize(&p).unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn touch(p: &Path, body: &[u8]) {
|
|
||||||
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
|
|
||||||
std::fs::write(p, body).unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Run one full index over `config`'s roots and wait for it to finish.
|
/// Run one full index over `config`'s roots and wait for it to finish.
|
||||||
fn index_once(db: &Path, config: &Config) {
|
fn index_once(db: &Path, config: &Config) {
|
||||||
if db.exists() {
|
common::IndexOnce {
|
||||||
let conn = rusqlite::Connection::open(db).unwrap();
|
db,
|
||||||
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
|
roots: config.paths.indexing_paths.clone(),
|
||||||
.unwrap();
|
config,
|
||||||
|
fresh_marker: true,
|
||||||
|
encrypted: false,
|
||||||
}
|
}
|
||||||
let service = IndexingService::new();
|
.run()
|
||||||
service
|
|
||||||
.start_indexing(
|
|
||||||
config.paths.indexing_paths.clone(),
|
|
||||||
db.to_string_lossy().into_owned(),
|
|
||||||
config.clone(),
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let deadline = Instant::now() + Duration::from_secs(120);
|
|
||||||
let mut done = false;
|
|
||||||
while Instant::now() < deadline {
|
|
||||||
if let IndexingStatus::Error(e) = service.get_status() {
|
|
||||||
panic!("indexing failed: {}", e);
|
|
||||||
}
|
|
||||||
if db.exists() {
|
|
||||||
if let Ok(conn) = rusqlite::Connection::open(db) {
|
|
||||||
if db::repo::get_last_full_index(&conn).is_some() {
|
|
||||||
done = true;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
std::thread::sleep(Duration::from_millis(10));
|
|
||||||
}
|
|
||||||
assert!(done, "indexing did not finish within the timeout");
|
|
||||||
service.stop_indexing().unwrap();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Apply the reconciliation `old -> new` implies, exactly as the coordinator
|
/// Apply the reconciliation `old -> new` implies, exactly as the coordinator
|
||||||
|
|
@ -84,6 +43,7 @@ fn reconcile(db: &Path, old: &Config, new: &Config) -> (usize, usize) {
|
||||||
let mut conn = db::open_existing(&db.to_string_lossy(), true).unwrap();
|
let mut conn = db::open_existing(&db.to_string_lossy(), true).unwrap();
|
||||||
let registry = Registry::default_set();
|
let registry = Registry::default_set();
|
||||||
let mut cursor = WorkCursor::new(actions.work, new).unwrap();
|
let mut cursor = WorkCursor::new(actions.work, new).unwrap();
|
||||||
|
let run = AtomicBool::new(false);
|
||||||
while !cursor.done() {
|
while !cursor.done() {
|
||||||
advance(
|
advance(
|
||||||
&mut conn,
|
&mut conn,
|
||||||
|
|
@ -91,6 +51,7 @@ fn reconcile(db: &Path, old: &Config, new: &Config) -> (usize, usize) {
|
||||||
®istry,
|
®istry,
|
||||||
&mut cursor,
|
&mut cursor,
|
||||||
Instant::now() + SLICE,
|
Instant::now() + SLICE,
|
||||||
|
&run,
|
||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,6 @@
|
||||||
//! --test snippet_perf -- --nocapture
|
//! --test snippet_perf -- --nocapture
|
||||||
//! ```
|
//! ```
|
||||||
|
|
||||||
use std::path::PathBuf;
|
|
||||||
use std::time::Instant;
|
use std::time::Instant;
|
||||||
|
|
||||||
use quicksearch_core::snippet;
|
use quicksearch_core::snippet;
|
||||||
|
|
@ -123,19 +122,8 @@ fn seed_text(rng: &mut u64, target_words: usize) -> String {
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
fn tmp_path(tag: &str) -> PathBuf {
|
mod common;
|
||||||
let mut p = std::env::temp_dir();
|
use common::scratch_db as tmp_path;
|
||||||
p.push(format!(
|
|
||||||
"qs-snippet-perf-{}-{}-{}.sqlite",
|
|
||||||
tag,
|
|
||||||
std::process::id(),
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_nanos()
|
|
||||||
));
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn snippet_paths_perf_comparison() {
|
fn snippet_paths_perf_comparison() {
|
||||||
|
|
|
||||||
|
|
@ -6,9 +6,11 @@ use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use quicksearch_core::cli::IndexCounts;
|
use quicksearch_core::cli::IndexCounts;
|
||||||
use quicksearch_core::config::{diff_actions, nested_roots, Config, SecurityConfig};
|
use quicksearch_core::config::{diff_actions, nested_roots, Config, SecurityConfig};
|
||||||
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
|
use quicksearch_core::coordinator::{IndexMode, IndexerState, ReconcileState, WatcherStatus};
|
||||||
use quicksearch_core::db;
|
use quicksearch_core::db;
|
||||||
use quicksearch_core::indexing::{overall_progress, ConfigChange, IndexingStatus, RootPhase};
|
use quicksearch_core::indexing::{
|
||||||
|
overall_progress, ConfigChange, IndexingStatus, PrepStep, RootPhase,
|
||||||
|
};
|
||||||
use quicksearch_core::search::SearchOptions;
|
use quicksearch_core::search::SearchOptions;
|
||||||
use quicksearch_core::security::{derive_key, generate_salt, salt_to_hex, IndexKey};
|
use quicksearch_core::security::{derive_key, generate_salt, salt_to_hex, IndexKey};
|
||||||
use quicksearch_core::watcher::WatchError;
|
use quicksearch_core::watcher::WatchError;
|
||||||
|
|
@ -79,6 +81,18 @@ fn guard_source(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether quitting now needs the "settings are still being applied" warning.
|
||||||
|
///
|
||||||
|
/// Only a Quit, and only while a reconciliation is actually running: leaving
|
||||||
|
/// mid-pass leaves entries the user excluded still in the index until an
|
||||||
|
/// indexing run redoes the work, which in manual mode means until they ask for
|
||||||
|
/// one. Its own function because it is a rule, not a rendering decision, and
|
||||||
|
/// because the guard it belongs to has two entrances — the close request, and
|
||||||
|
/// the unsaved-changes prompt resolving to Quit.
|
||||||
|
fn quit_needs_reconcile_warning(intent: NavIntent, reconciling: bool) -> bool {
|
||||||
|
intent == NavIntent::Quit && reconciling
|
||||||
|
}
|
||||||
|
|
||||||
pub struct QuickSearchApp {
|
pub struct QuickSearchApp {
|
||||||
cfg: Config,
|
cfg: Config,
|
||||||
backend: Backend,
|
backend: Backend,
|
||||||
|
|
@ -103,6 +117,13 @@ pub struct QuickSearchApp {
|
||||||
/// and the next run will replace it; see
|
/// and the next run will replace it; see
|
||||||
/// [`QuickSearchApp::stale_index_prompt_ui`].
|
/// [`QuickSearchApp::stale_index_prompt_ui`].
|
||||||
stale_index_prompt: bool,
|
stale_index_prompt: bool,
|
||||||
|
/// Set at startup when the index has not caught up with the settings —
|
||||||
|
/// a reconciliation cut short by a previous quit, or a config edited
|
||||||
|
/// while the app was closed. See [`QuickSearchApp::reconcile_owed_ui`].
|
||||||
|
reconcile_owed: bool,
|
||||||
|
/// `last_full_index` as it read at startup; the run that moves it past
|
||||||
|
/// this is the run that clears `reconcile_owed`.
|
||||||
|
reconcile_owed_since: Option<u64>,
|
||||||
/// Set when the watcher gave up on the directory budget and live
|
/// Set when the watcher gave up on the directory budget and live
|
||||||
/// updates are off; see [`QuickSearchApp::check_watch_cap_warning`].
|
/// updates are off; see [`QuickSearchApp::check_watch_cap_warning`].
|
||||||
watch_cap_prompt: Option<WatchError>,
|
watch_cap_prompt: Option<WatchError>,
|
||||||
|
|
@ -177,7 +198,20 @@ impl QuickSearchApp {
|
||||||
let stale_index_prompt =
|
let stale_index_prompt =
|
||||||
db::index_needs_rebuild(&cfg.resolved_database_path().to_string_lossy());
|
db::index_needs_rebuild(&cfg.resolved_database_path().to_string_lossy());
|
||||||
|
|
||||||
|
// Also before the backend, and for the same reason: in automatic mode
|
||||||
|
// the coordinator's first run can reconcile — and clear the answer —
|
||||||
|
// before the first frame is drawn. A missing or unreadable index owes
|
||||||
|
// nothing; it has never been reconciled against anything.
|
||||||
|
let db_path = cfg.resolved_database_path().to_string_lossy().into_owned();
|
||||||
|
let reconcile_owed = quicksearch_core::scope::outstanding_work(&db_path, &cfg)
|
||||||
|
.map(|work| work.touches_index())
|
||||||
|
.unwrap_or(false);
|
||||||
|
|
||||||
let backend = Backend::start(&cfg, ctx.clone())?;
|
let backend = Backend::start(&cfg, ctx.clone())?;
|
||||||
|
// Read from the coordinator rather than the file: it stamps this at
|
||||||
|
// startup, before its thread can run anything, so it is the same
|
||||||
|
// number the frames below compare against.
|
||||||
|
let reconcile_owed_since = backend.coordinator.state().last_full_index;
|
||||||
let fuzzy = cfg.search.fuzzy_default;
|
let fuzzy = cfg.search.fuzzy_default;
|
||||||
// Startup validation: a hand-edited config can nest roots, which
|
// Startup validation: a hand-edited config can nest roots, which
|
||||||
// per-root pipelines can't accept. Redirect straight to the folder
|
// per-root pipelines can't accept. Redirect straight to the folder
|
||||||
|
|
@ -208,6 +242,8 @@ impl QuickSearchApp {
|
||||||
nested_prompt,
|
nested_prompt,
|
||||||
key_source,
|
key_source,
|
||||||
stale_index_prompt,
|
stale_index_prompt,
|
||||||
|
reconcile_owed,
|
||||||
|
reconcile_owed_since,
|
||||||
watch_cap_prompt: None,
|
watch_cap_prompt: None,
|
||||||
security_prompt: None,
|
security_prompt: None,
|
||||||
pending_nav: None,
|
pending_nav: None,
|
||||||
|
|
@ -261,8 +297,13 @@ impl QuickSearchApp {
|
||||||
.watch_cap_warned_roots
|
.watch_cap_warned_roots
|
||||||
.retain(|root| new.paths.indexing_paths.contains(root));
|
.retain(|root| new.paths.indexing_paths.contains(root));
|
||||||
let actions = diff_actions(&self.cfg, &new);
|
let actions = diff_actions(&self.cfg, &new);
|
||||||
|
// A config that could not be written must not take effect either: a
|
||||||
|
// read-only config directory would otherwise apply the settings to
|
||||||
|
// this process, revert them on restart, and — because `is_dirty`
|
||||||
|
// compares against `self.cfg` — show nothing unsaved in between.
|
||||||
if let Err(e) = new.save() {
|
if let Err(e) = new.save() {
|
||||||
self.config_error = Some(e);
|
self.config_error = Some(e);
|
||||||
|
return false;
|
||||||
}
|
}
|
||||||
if (new.ui.scale - self.cfg.ui.scale).abs() > f32::EPSILON {
|
if (new.ui.scale - self.cfg.ui.scale).abs() > f32::EPSILON {
|
||||||
ctx.set_zoom_factor(clamp_scale(new.ui.scale));
|
ctx.set_zoom_factor(clamp_scale(new.ui.scale));
|
||||||
|
|
@ -321,13 +362,9 @@ impl QuickSearchApp {
|
||||||
|
|
||||||
fn drain_events(&mut self) {
|
fn drain_events(&mut self) {
|
||||||
// Streamed search results.
|
// Streamed search results.
|
||||||
loop {
|
while let Ok(update) = self.backend.search_rx.try_recv() {
|
||||||
match self.backend.search_rx.try_recv() {
|
self.search
|
||||||
Ok(update) => self
|
.apply_update(update, self.cfg.search.display_limit);
|
||||||
.search
|
|
||||||
.apply_update(update, self.cfg.search.display_limit),
|
|
||||||
Err(_) => break,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
// Status-bar counts worker.
|
// Status-bar counts worker.
|
||||||
if let Some(rx) = &self.backend.counts_job {
|
if let Some(rx) = &self.backend.counts_job {
|
||||||
|
|
@ -430,6 +467,65 @@ impl QuickSearchApp {
|
||||||
egui::TopBottomPanel::bottom("status-bar").show(ctx, |ui| {
|
egui::TopBottomPanel::bottom("status-bar").show(ctx, |ui| {
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
match &state.activity {
|
match &state.activity {
|
||||||
|
// A reconcile the coordinator applies between runs: no run
|
||||||
|
// holds the index, but it is scanning every row of it —
|
||||||
|
// and for a moment after, so a pass shorter than a frame
|
||||||
|
// still leaves a trace.
|
||||||
|
IndexingStatus::Idle if state.reconcile.is_some() => {
|
||||||
|
match state.reconcile.expect("matched Some") {
|
||||||
|
ReconcileState::Running(r) => {
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(match (r.total, r.fraction()) {
|
||||||
|
(Some(total), Some(frac)) => format!(
|
||||||
|
"Applying configuration change · {} / {} ({:.0}%)",
|
||||||
|
group_thousands(r.examined as u64),
|
||||||
|
group_thousands(total as u64),
|
||||||
|
frac * 100.0
|
||||||
|
),
|
||||||
|
_ => format!(
|
||||||
|
"Applying configuration change · {} entries",
|
||||||
|
group_thousands(r.examined as u64)
|
||||||
|
),
|
||||||
|
})
|
||||||
|
.small(),
|
||||||
|
);
|
||||||
|
progress_widget(ui, r.fraction());
|
||||||
|
}
|
||||||
|
ReconcileState::Finished(r) => {
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(crate::format::fmt_reconcile_summary(
|
||||||
|
r.deleted,
|
||||||
|
r.recontented,
|
||||||
|
))
|
||||||
|
.small(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
IndexingStatus::Preparing { start_time, step } => {
|
||||||
|
let (label, frac) = match step {
|
||||||
|
PrepStep::PreviousRun => {
|
||||||
|
("Finishing the previous run…".to_string(), None)
|
||||||
|
}
|
||||||
|
PrepStep::OpeningIndex => ("Opening the index…".to_string(), None),
|
||||||
|
PrepStep::Reconciling(r) => (
|
||||||
|
format!(
|
||||||
|
"Applying configuration change · {} entries",
|
||||||
|
group_thousands(r.examined as u64)
|
||||||
|
),
|
||||||
|
r.fraction(),
|
||||||
|
),
|
||||||
|
};
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(format!(
|
||||||
|
"{} · {}",
|
||||||
|
label,
|
||||||
|
crate::format::fmt_duration_clock(start_time.elapsed())
|
||||||
|
))
|
||||||
|
.small(),
|
||||||
|
);
|
||||||
|
progress_widget(ui, frac);
|
||||||
|
}
|
||||||
IndexingStatus::Idle => {
|
IndexingStatus::Idle => {
|
||||||
let mode = match state.mode {
|
let mode = match state.mode {
|
||||||
IndexMode::Auto => "Auto",
|
IndexMode::Auto => "Auto",
|
||||||
|
|
@ -497,14 +593,7 @@ impl QuickSearchApp {
|
||||||
text.push_str(&format!(" · {}/{} workers", active, total_workers));
|
text.push_str(&format!(" · {}/{} workers", active, total_workers));
|
||||||
}
|
}
|
||||||
ui.label(egui::RichText::new(text).small());
|
ui.label(egui::RichText::new(text).small());
|
||||||
match frac {
|
progress_widget(ui, frac);
|
||||||
Some(frac) => {
|
|
||||||
ui.add(egui::ProgressBar::new(frac as f32).desired_width(120.0));
|
|
||||||
}
|
|
||||||
None => {
|
|
||||||
ui.add(egui::Spinner::new().size(12.0));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -525,11 +614,15 @@ impl QuickSearchApp {
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// Keep painting while anything is moving.
|
// Keep painting while anything is moving. The reconcile clause is not
|
||||||
|
// redundant: the coordinator's own pass runs with the activity `Idle`,
|
||||||
|
// so without it the counters would freeze mid-scan until the pointer
|
||||||
|
// moved — and the summary that follows would never age off screen.
|
||||||
if !matches!(
|
if !matches!(
|
||||||
state.activity,
|
state.activity,
|
||||||
IndexingStatus::Idle | IndexingStatus::Error(_)
|
IndexingStatus::Idle | IndexingStatus::Error(_)
|
||||||
) {
|
) || state.reconcile.is_some()
|
||||||
|
{
|
||||||
ctx.request_repaint_after(Duration::from_millis(250));
|
ctx.request_repaint_after(Duration::from_millis(250));
|
||||||
}
|
}
|
||||||
// Watcher registration walks every root, so its verdict can land
|
// Watcher registration walks every root, so its verdict can land
|
||||||
|
|
@ -631,8 +724,12 @@ impl QuickSearchApp {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
|
||||||
keychain::delete_key(&db_path.to_string_lossy());
|
// Mirrors the store half: the preference describes what
|
||||||
|
// is on the keychain, so it must not claim the key is
|
||||||
|
// gone while it is still there.
|
||||||
|
self.config_error = Some(e);
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
self.cfg.security.use_keychain = remember;
|
self.cfg.security.use_keychain = remember;
|
||||||
if let Err(e) = self.cfg.save() {
|
if let Err(e) = self.cfg.save() {
|
||||||
|
|
@ -823,8 +920,13 @@ impl QuickSearchApp {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Disabling protection, or "remember" off: no stored key may
|
// Disabling protection, or "remember" off: no stored key may
|
||||||
// survive pointing at the previous encryption state.
|
// survive pointing at the previous encryption state. A failure
|
||||||
_ => keychain::delete_key(&db_path),
|
// here leaves one that does, which is worth saying out loud.
|
||||||
|
_ => {
|
||||||
|
if let Err(e) = keychain::delete_key(&db_path) {
|
||||||
|
self.config_error = Some(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
db::set_process_key(new_key);
|
db::set_process_key(new_key);
|
||||||
self.backend.coordinator.rebuild_index();
|
self.backend.coordinator.rebuild_index();
|
||||||
|
|
@ -833,6 +935,20 @@ impl QuickSearchApp {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The status bar's trailing progress indicator: a bar when the work has a
|
||||||
|
/// denominator, a spinner when it does not. One helper so every kind of
|
||||||
|
/// activity the bar reports ends the same way.
|
||||||
|
fn progress_widget(ui: &mut egui::Ui, fraction: Option<f64>) {
|
||||||
|
match fraction {
|
||||||
|
Some(frac) => {
|
||||||
|
ui.add(egui::ProgressBar::new(frac as f32).desired_width(120.0));
|
||||||
|
}
|
||||||
|
None => {
|
||||||
|
ui.add(egui::Spinner::new().size(12.0));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Drop egui's retained text-field state (buffer + undo history) for the
|
/// Drop egui's retained text-field state (buffer + undo history) for the
|
||||||
/// password dialog fields.
|
/// password dialog fields.
|
||||||
fn purge_security_field_state(ctx: &egui::Context) {
|
fn purge_security_field_state(ctx: &egui::Context) {
|
||||||
|
|
@ -899,6 +1015,49 @@ impl QuickSearchApp {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Tell the user their settings have not reached the index, and offer the
|
||||||
|
/// one thing that fixes it.
|
||||||
|
///
|
||||||
|
/// The condition is the index's own record: a reconciliation that finishes
|
||||||
|
/// stamps it, so work still owed means a pass was abandoned — quitting
|
||||||
|
/// during one is the ordinary way — or the config was edited while the app
|
||||||
|
/// was closed. In automatic mode the periodic run clears it without the
|
||||||
|
/// user doing anything, which is why the banner is a line and a button
|
||||||
|
/// rather than a modal; in manual mode nothing happens until they ask.
|
||||||
|
///
|
||||||
|
/// Held out of the way while a run or a reconcile is in progress: that is
|
||||||
|
/// the work itself, and it can only be answered by waiting.
|
||||||
|
fn reconcile_owed_ui(&mut self, ctx: &egui::Context) {
|
||||||
|
if !self.reconcile_owed {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let state = self.backend.coordinator.state();
|
||||||
|
// A completed run is the proof: it reconciles from the same record
|
||||||
|
// and stamps it. A run the user stops does not move this, and the
|
||||||
|
// banner correctly comes back.
|
||||||
|
if state.last_full_index > self.reconcile_owed_since {
|
||||||
|
self.reconcile_owed = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if !matches!(
|
||||||
|
state.activity,
|
||||||
|
IndexingStatus::Idle | IndexingStatus::Error(_)
|
||||||
|
) || state.reconcile.is_some()
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
match reconcile_owed_banner(ctx) {
|
||||||
|
None => {}
|
||||||
|
Some(ReconcileOwedChoice::StartIndexing) => {
|
||||||
|
self.backend.coordinator.reindex_now();
|
||||||
|
// Not cleared here: the run that finishes clears it, and one
|
||||||
|
// that is stopped half-way leaves the reminder standing.
|
||||||
|
ctx.request_repaint_after(Duration::from_millis(100));
|
||||||
|
}
|
||||||
|
Some(ReconcileOwedChoice::Dismiss) => self.reconcile_owed = false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn watch_cap_prompt_ui(&mut self, ctx: &egui::Context) {
|
fn watch_cap_prompt_ui(&mut self, ctx: &egui::Context) {
|
||||||
let Some(reason) = &self.watch_cap_prompt else {
|
let Some(reason) = &self.watch_cap_prompt else {
|
||||||
return;
|
return;
|
||||||
|
|
@ -1020,6 +1179,23 @@ impl QuickSearchApp {
|
||||||
};
|
};
|
||||||
let dirty = (self.manage.is_dirty(), self.options.is_dirty(&self.cfg));
|
let dirty = (self.manage.is_dirty(), self.options.is_dirty(&self.cfg));
|
||||||
let Some(source) = guard_source(intent, dirty.0, dirty.1) else {
|
let Some(source) = guard_source(intent, dirty.0, dirty.1) else {
|
||||||
|
// Second in line, and inside the guard rather than beside it: the
|
||||||
|
// Discard-then-quit path sets `quit_confirmed` and never returns
|
||||||
|
// to the close-request check, so a warning that lived only there
|
||||||
|
// would be skipped by exactly the user who dirtied an editor.
|
||||||
|
if quit_needs_reconcile_warning(intent, self.backend.coordinator.reconciling()) {
|
||||||
|
// Repaint on its own: a reconcile that ends while the modal is
|
||||||
|
// up should take the modal with it.
|
||||||
|
ctx.request_repaint_after(Duration::from_millis(250));
|
||||||
|
match reconcile_quit_modal(ctx) {
|
||||||
|
None => return,
|
||||||
|
Some(false) => {
|
||||||
|
self.pending_nav = None;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Some(true) => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
return self.complete_nav(ctx, intent);
|
return self.complete_nav(ctx, intent);
|
||||||
};
|
};
|
||||||
match unsaved_changes_modal(ctx, source) {
|
match unsaved_changes_modal(ctx, source) {
|
||||||
|
|
@ -1101,6 +1277,11 @@ impl QuickSearchApp {
|
||||||
self.backend.coordinator.state().activity
|
self.backend.coordinator.state().activity
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Let the close request a scripted quit sends through both guards.
|
||||||
|
pub(crate) fn capture_confirm_quit(&mut self) {
|
||||||
|
self.quit_confirmed = true;
|
||||||
|
}
|
||||||
|
|
||||||
pub(crate) fn capture_search_settled(&self) -> bool {
|
pub(crate) fn capture_search_settled(&self) -> bool {
|
||||||
self.search.capture_settled()
|
self.search.capture_settled()
|
||||||
}
|
}
|
||||||
|
|
@ -1186,6 +1367,85 @@ fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<U
|
||||||
choice
|
choice
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// What the user chose in the "settings not applied yet" banner.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||||
|
enum ReconcileOwedChoice {
|
||||||
|
StartIndexing,
|
||||||
|
Dismiss,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The banner's body, as a top panel under the config-error one.
|
||||||
|
///
|
||||||
|
/// A free function for the same reason as [`stale_index_window`]: a test can
|
||||||
|
/// render it and click its buttons without a coordinator behind it.
|
||||||
|
fn reconcile_owed_banner(ctx: &egui::Context) -> Option<ReconcileOwedChoice> {
|
||||||
|
let mut choice = None;
|
||||||
|
egui::TopBottomPanel::top("reconcile-owed").show(ctx, |ui| {
|
||||||
|
ui.horizontal(|ui| {
|
||||||
|
ui.colored_label(
|
||||||
|
ui.visuals().warn_fg_color,
|
||||||
|
"⚠ Your indexing settings have not been applied to the index yet.",
|
||||||
|
);
|
||||||
|
if ui.small_button("Start indexing now").clicked() {
|
||||||
|
choice = Some(ReconcileOwedChoice::StartIndexing);
|
||||||
|
}
|
||||||
|
if ui.small_button("Dismiss").clicked() {
|
||||||
|
choice = Some(ReconcileOwedChoice::Dismiss);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
choice
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Body of the quit-during-a-reconcile guard; `Some(true)` to quit anyway,
|
||||||
|
/// `Some(false)` to stay. Esc and a backdrop click count as staying.
|
||||||
|
///
|
||||||
|
/// The same blocking `egui::Modal` the unsaved guard uses, and for the same
|
||||||
|
/// reason: this is a decision, not a notice. Quitting is not refused — the
|
||||||
|
/// pass is cancellable and the index stays consistent either way — but the
|
||||||
|
/// consequence is invisible otherwise, since the entries the user excluded go
|
||||||
|
/// on appearing in search results until an indexing run finishes the job.
|
||||||
|
fn reconcile_quit_modal(ctx: &egui::Context) -> Option<bool> {
|
||||||
|
let mut choice = None;
|
||||||
|
let modal = egui::Modal::new(egui::Id::new("reconcile-quit-guard")).show(ctx, |ui| {
|
||||||
|
ui.set_max_width(460.0);
|
||||||
|
ui.heading("Settings are still being applied");
|
||||||
|
ui.label(
|
||||||
|
"QuickSearch is still applying your indexing settings to the index. If you \
|
||||||
|
quit now it stops part-way, and entries you excluded can still turn up in \
|
||||||
|
search results.",
|
||||||
|
);
|
||||||
|
ui.add_space(4.0);
|
||||||
|
ui.label(
|
||||||
|
"Nothing is lost: the next indexing run picks the work up again. In manual \
|
||||||
|
mode, choose \"Start indexing now\" on the Manage Index tab after the next \
|
||||||
|
launch.",
|
||||||
|
);
|
||||||
|
ui.add_space(6.0);
|
||||||
|
ui.horizontal(|ui| {
|
||||||
|
if ui
|
||||||
|
.button(egui::RichText::new("Quit anyway").color(ui.visuals().error_fg_color))
|
||||||
|
.clicked()
|
||||||
|
{
|
||||||
|
choice = Some(true);
|
||||||
|
}
|
||||||
|
if ui
|
||||||
|
.add(crate::ui_util::bordered_button(
|
||||||
|
"Cancel",
|
||||||
|
crate::ui_util::BLUE,
|
||||||
|
))
|
||||||
|
.clicked()
|
||||||
|
{
|
||||||
|
choice = Some(false);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
if choice.is_none() && modal.should_close() {
|
||||||
|
choice = Some(false);
|
||||||
|
}
|
||||||
|
choice
|
||||||
|
}
|
||||||
|
|
||||||
/// The stale-index window's body. Returns whether the user asked for the
|
/// The stale-index window's body. Returns whether the user asked for the
|
||||||
/// rebuild.
|
/// rebuild.
|
||||||
///
|
///
|
||||||
|
|
@ -1274,7 +1534,12 @@ impl eframe::App for QuickSearchApp {
|
||||||
// window is gone there is nothing left to ask — and re-sent from
|
// window is gone there is nothing left to ask — and re-sent from
|
||||||
// `complete_nav` if the user chooses to leave.
|
// `complete_nav` if the user chooses to leave.
|
||||||
if ctx.input(|i| i.viewport().close_requested()) && !self.quit_confirmed {
|
if ctx.input(|i| i.viewport().close_requested()) && !self.quit_confirmed {
|
||||||
if self.manage.is_dirty() || self.options.is_dirty(&self.cfg) {
|
// A reconciliation in flight gets the same treatment as an unsaved
|
||||||
|
// editor: hold the close and say what leaving now costs.
|
||||||
|
if self.manage.is_dirty()
|
||||||
|
|| self.options.is_dirty(&self.cfg)
|
||||||
|
|| self.backend.coordinator.reconciling()
|
||||||
|
{
|
||||||
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
|
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
|
||||||
// Quitting subsumes any narrower pending navigation.
|
// Quitting subsumes any narrower pending navigation.
|
||||||
self.pending_nav = Some(NavIntent::Quit);
|
self.pending_nav = Some(NavIntent::Quit);
|
||||||
|
|
@ -1335,6 +1600,7 @@ impl eframe::App for QuickSearchApp {
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
self.reconcile_owed_ui(ctx);
|
||||||
|
|
||||||
egui::CentralPanel::default().show(ctx, |ui| match self.tab {
|
egui::CentralPanel::default().show(ctx, |ui| match self.tab {
|
||||||
Tab::Search => {
|
Tab::Search => {
|
||||||
|
|
@ -1426,30 +1692,16 @@ mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn frame(ctx: &egui::Context, source: KeySource, events: Vec<egui::Event>) -> bool {
|
fn frame(ctx: &egui::Context, source: KeySource, events: Vec<egui::Event>) -> bool {
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(SCREEN, events);
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(1000.0, 700.0),
|
|
||||||
)),
|
|
||||||
events,
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
let mut clicked = false;
|
let mut clicked = false;
|
||||||
let _ = ctx.run(input, |ctx| clicked = stale_index_window(ctx, source));
|
let _ = ctx.run(input, |ctx| clicked = stale_index_window(ctx, source));
|
||||||
clicked
|
clicked
|
||||||
}
|
}
|
||||||
|
|
||||||
fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
|
use crate::test_ui::click_at;
|
||||||
[true, false]
|
|
||||||
.into_iter()
|
/// The viewport every modal in this module is centred in.
|
||||||
.map(|pressed| egui::Event::PointerButton {
|
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
|
||||||
pos,
|
|
||||||
button: egui::PointerButton::Primary,
|
|
||||||
pressed,
|
|
||||||
modifiers: egui::Modifiers::default(),
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The modal the user sees after unlocking onto an index from an older
|
/// The modal the user sees after unlocking onto an index from an older
|
||||||
/// version. It must render under every key source — each produces a
|
/// version. It must render under every key source — each produces a
|
||||||
|
|
@ -1566,14 +1818,7 @@ mod tests {
|
||||||
source: UnsavedSource,
|
source: UnsavedSource,
|
||||||
events: Vec<egui::Event>,
|
events: Vec<egui::Event>,
|
||||||
) -> Option<UnsavedChoice> {
|
) -> Option<UnsavedChoice> {
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(SCREEN, events);
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(1000.0, 700.0),
|
|
||||||
)),
|
|
||||||
events,
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
let mut choice = None;
|
let mut choice = None;
|
||||||
let _ = ctx.run(input, |ctx| choice = unsaved_changes_modal(ctx, source));
|
let _ = ctx.run(input, |ctx| choice = unsaved_changes_modal(ctx, source));
|
||||||
choice
|
choice
|
||||||
|
|
@ -1627,4 +1872,98 @@ mod tests {
|
||||||
assert_eq!(esc, Some(UnsavedChoice::Cancel), "Esc must cancel");
|
assert_eq!(esc, Some(UnsavedChoice::Cancel), "Esc must cancel");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Quitting mid-reconcile is the one case that needs saying out loud: the
|
||||||
|
/// index is left describing settings the user has already changed, and in
|
||||||
|
/// manual mode nothing fixes that until they ask for a run. Nothing else
|
||||||
|
/// warrants the prompt — a tab switch does not end the pass, and a quit
|
||||||
|
/// with no pass running has nothing to warn about.
|
||||||
|
#[test]
|
||||||
|
fn only_quitting_during_a_reconcile_warns() {
|
||||||
|
use NavIntent::*;
|
||||||
|
assert!(quit_needs_reconcile_warning(Quit, true));
|
||||||
|
assert!(!quit_needs_reconcile_warning(Quit, false));
|
||||||
|
assert!(!quit_needs_reconcile_warning(SwitchTab(Tab::Search), true));
|
||||||
|
assert!(!quit_needs_reconcile_warning(CloseOptions, true));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn reconcile_modal_frame(ctx: &egui::Context, events: Vec<egui::Event>) -> Option<bool> {
|
||||||
|
let input = crate::test_ui::raw_input(SCREEN, events);
|
||||||
|
let mut choice = None;
|
||||||
|
let _ = ctx.run(input, |ctx| choice = reconcile_quit_modal(ctx));
|
||||||
|
choice
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Both ways out of the quit warning work, and neither is the default: a
|
||||||
|
/// modal whose "Quit anyway" did nothing would trap the user in an app
|
||||||
|
/// they asked to close, and one that quit on Esc would make the warning
|
||||||
|
/// pointless.
|
||||||
|
#[test]
|
||||||
|
fn the_quit_warning_reports_both_answers() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
assert_eq!(
|
||||||
|
reconcile_modal_frame(&ctx, Vec::new()),
|
||||||
|
None,
|
||||||
|
"an untouched frame must not decide"
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut seen = std::collections::HashSet::new();
|
||||||
|
for y in (250..450).step_by(3) {
|
||||||
|
for x in (250..760).step_by(6) {
|
||||||
|
if let Some(choice) =
|
||||||
|
reconcile_modal_frame(&ctx, click_at(egui::pos2(x as f32, y as f32)))
|
||||||
|
{
|
||||||
|
seen.insert(choice);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(seen.contains(&true), "\"Quit anyway\" never fired");
|
||||||
|
assert!(seen.contains(&false), "Cancel never fired");
|
||||||
|
|
||||||
|
let esc = reconcile_modal_frame(
|
||||||
|
&ctx,
|
||||||
|
vec![egui::Event::Key {
|
||||||
|
key: egui::Key::Escape,
|
||||||
|
physical_key: None,
|
||||||
|
pressed: true,
|
||||||
|
repeat: false,
|
||||||
|
modifiers: egui::Modifiers::NONE,
|
||||||
|
}],
|
||||||
|
);
|
||||||
|
assert_eq!(esc, Some(false), "Esc must keep the app open");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn banner_frame(ctx: &egui::Context, events: Vec<egui::Event>) -> Option<ReconcileOwedChoice> {
|
||||||
|
let input = crate::test_ui::raw_input(SCREEN, events);
|
||||||
|
let mut choice = None;
|
||||||
|
let _ = ctx.run(input, |ctx| choice = reconcile_owed_banner(ctx));
|
||||||
|
choice
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reminder a quit mid-prune leaves behind. Its Start button is the
|
||||||
|
/// whole point — the banner exists because in manual mode nothing else
|
||||||
|
/// will finish the work — and Dismiss must not be the only live control.
|
||||||
|
#[test]
|
||||||
|
fn the_reconcile_banner_reports_both_buttons() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
assert_eq!(banner_frame(&ctx, Vec::new()), None);
|
||||||
|
|
||||||
|
let mut seen = std::collections::HashSet::new();
|
||||||
|
// A top panel, so it sits in the first rows of the window.
|
||||||
|
for y in (0..60).step_by(2) {
|
||||||
|
for x in (0..1000).step_by(4) {
|
||||||
|
if let Some(choice) = banner_frame(&ctx, click_at(egui::pos2(x as f32, y as f32))) {
|
||||||
|
seen.insert(choice);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
seen.contains(&ReconcileOwedChoice::StartIndexing),
|
||||||
|
"\"Start indexing now\" never fired"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
seen.contains(&ReconcileOwedChoice::Dismiss),
|
||||||
|
"Dismiss never fired"
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -193,8 +193,16 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
|
||||||
"clear_query" => Cmd::ClearQuery,
|
"clear_query" => Cmd::ClearQuery,
|
||||||
"focus_search" => Cmd::FocusSearch,
|
"focus_search" => Cmd::FocusSearch,
|
||||||
"window" => {
|
"window" => {
|
||||||
let w = parse_int("width", next_word(rest, line_no, "width in points")?, line_no)?;
|
let w = parse_int(
|
||||||
let h = parse_int("height", next_word(rest, line_no, "height in points")?, line_no)?;
|
"width",
|
||||||
|
next_word(rest, line_no, "width in points")?,
|
||||||
|
line_no,
|
||||||
|
)?;
|
||||||
|
let h = parse_int(
|
||||||
|
"height",
|
||||||
|
next_word(rest, line_no, "height in points")?,
|
||||||
|
line_no,
|
||||||
|
)?;
|
||||||
if w == 0 || h == 0 {
|
if w == 0 || h == 0 {
|
||||||
return Err(err("window dimensions must be positive".to_string()));
|
return Err(err("window dimensions must be positive".to_string()));
|
||||||
}
|
}
|
||||||
|
|
@ -203,9 +211,11 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
|
||||||
h: h as f32,
|
h: h as f32,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
"hover_match" => Cmd::HoverMatch(
|
"hover_match" => {
|
||||||
|
Cmd::HoverMatch(
|
||||||
parse_int("row", next_word(rest, line_no, "row index")?, line_no)? as usize,
|
parse_int("row", next_word(rest, line_no, "row index")?, line_no)? as usize,
|
||||||
),
|
)
|
||||||
|
}
|
||||||
"hover_off" => Cmd::HoverOff,
|
"hover_off" => Cmd::HoverOff,
|
||||||
"tab" => Cmd::Tab(match next_word(rest, line_no, "tab name")? {
|
"tab" => Cmd::Tab(match next_word(rest, line_no, "tab name")? {
|
||||||
"search" => Tab::Search,
|
"search" => Tab::Search,
|
||||||
|
|
@ -452,7 +462,7 @@ impl CaptureDriver {
|
||||||
}
|
}
|
||||||
|
|
||||||
let Some(cmd) = self.cmds.get(self.pc).cloned() else {
|
let Some(cmd) = self.cmds.get(self.pc).cloned() else {
|
||||||
self.quit(ctx);
|
self.quit(app, ctx);
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
let started = match self.cmd_started {
|
let started = match self.cmd_started {
|
||||||
|
|
@ -542,7 +552,7 @@ impl CaptureDriver {
|
||||||
ShotTag,
|
ShotTag,
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
Cmd::Quit => self.quit(ctx),
|
Cmd::Quit => self.quit(app, ctx),
|
||||||
}
|
}
|
||||||
if matches!(cmd, Cmd::RecordStop) {
|
if matches!(cmd, Cmd::RecordStop) {
|
||||||
self.stop_recorder();
|
self.stop_recorder();
|
||||||
|
|
@ -572,7 +582,8 @@ impl CaptureDriver {
|
||||||
capped(max_ms)
|
capped(max_ms)
|
||||||
|| matches!(
|
|| matches!(
|
||||||
app.capture_indexing_status(),
|
app.capture_indexing_status(),
|
||||||
IndexingStatus::Running { .. }
|
IndexingStatus::Preparing { .. }
|
||||||
|
| IndexingStatus::Running { .. }
|
||||||
| IndexingStatus::Stopping
|
| IndexingStatus::Stopping
|
||||||
| IndexingStatus::Optimizing
|
| IndexingStatus::Optimizing
|
||||||
)
|
)
|
||||||
|
|
@ -590,8 +601,13 @@ impl CaptureDriver {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn quit(&mut self, ctx: &egui::Context) {
|
fn quit(&mut self, app: &mut QuickSearchApp, ctx: &egui::Context) {
|
||||||
self.stop_recorder();
|
self.stop_recorder();
|
||||||
|
// A scripted quit answers the guards up front. Nothing a scenario does
|
||||||
|
// dirties an editor, but a scenario that changed the indexed folders
|
||||||
|
// can leave a reconcile running, and its modal would hold the window
|
||||||
|
// open until the run timed out.
|
||||||
|
app.capture_confirm_quit();
|
||||||
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
|
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
|
||||||
self.finished = true;
|
self.finished = true;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -54,6 +54,8 @@ impl DuplicatesTab {
|
||||||
.weak(),
|
.weak(),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
// The spinner and its label are already on the header row above,
|
||||||
|
// so a second progress line here would just repeat them.
|
||||||
DupState::Loading => {}
|
DupState::Loading => {}
|
||||||
DupState::Error(e) => {
|
DupState::Error(e) => {
|
||||||
ui.colored_label(ui.visuals().error_fg_color, e);
|
ui.colored_label(ui.visuals().error_fg_color, e);
|
||||||
|
|
|
||||||
|
|
@ -29,11 +29,7 @@ pub fn fmt_mtime(unix_secs: i64) -> String {
|
||||||
/// "5 min ago", "3 h ago", else `YYYY-MM-DD HH:MM`. Gives instant
|
/// "5 min ago", "3 h ago", else `YYYY-MM-DD HH:MM`. Gives instant
|
||||||
/// feedback that an action (like a fast index run) actually happened.
|
/// feedback that an action (like a fast index run) actually happened.
|
||||||
pub fn fmt_ago(unix_secs: u64) -> String {
|
pub fn fmt_ago(unix_secs: u64) -> String {
|
||||||
let now = std::time::SystemTime::now()
|
let age = quicksearch_core::log::now_unix().saturating_sub(unix_secs);
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_secs())
|
|
||||||
.unwrap_or(0);
|
|
||||||
let age = now.saturating_sub(unix_secs);
|
|
||||||
if age < 60 {
|
if age < 60 {
|
||||||
"just now".to_string()
|
"just now".to_string()
|
||||||
} else if age < 3600 {
|
} else if age < 3600 {
|
||||||
|
|
@ -59,7 +55,8 @@ pub fn fmt_interval(minutes: u64) -> String {
|
||||||
if minutes.is_multiple_of(1440) {
|
if minutes.is_multiple_of(1440) {
|
||||||
let days = minutes / 1440;
|
let days = minutes / 1440;
|
||||||
return if days == 1 {
|
return if days == 1 {
|
||||||
// "24 h" reads better than "1 day" for the shipped default.
|
// A staleness window reads better in hours than in days: "1 day"
|
||||||
|
// invites rounding to "about a day", "24 h" does not.
|
||||||
"24 h".to_string()
|
"24 h".to_string()
|
||||||
} else {
|
} else {
|
||||||
format!("{} days", days)
|
format!("{} days", days)
|
||||||
|
|
@ -76,7 +73,7 @@ pub fn group_thousands(n: u64) -> String {
|
||||||
let digits = n.to_string();
|
let digits = n.to_string();
|
||||||
let mut out = String::with_capacity(digits.len() + digits.len() / 3);
|
let mut out = String::with_capacity(digits.len() + digits.len() / 3);
|
||||||
for (i, c) in digits.chars().enumerate() {
|
for (i, c) in digits.chars().enumerate() {
|
||||||
if i > 0 && (digits.len() - i) % 3 == 0 {
|
if i > 0 && (digits.len() - i).is_multiple_of(3) {
|
||||||
out.push(',');
|
out.push(',');
|
||||||
}
|
}
|
||||||
out.push(c);
|
out.push(c);
|
||||||
|
|
@ -108,6 +105,50 @@ pub fn fmt_elapsed(d: std::time::Duration) -> String {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A running clock: `0:07`, `4:32`, `1:04:12`.
|
||||||
|
///
|
||||||
|
/// For work that is still going, where the question is "how long has this
|
||||||
|
/// been like this?" — [`fmt_elapsed`] answers a different one and would
|
||||||
|
/// render a twenty-minute wait as `1234.5 s`. Seconds are always two digits
|
||||||
|
/// so the text does not change width every tick.
|
||||||
|
pub fn fmt_duration_clock(d: std::time::Duration) -> String {
|
||||||
|
let secs = d.as_secs();
|
||||||
|
let (h, m, s) = (secs / 3600, (secs % 3600) / 60, secs % 60);
|
||||||
|
if h > 0 {
|
||||||
|
format!("{}:{:02}:{:02}", h, m, s)
|
||||||
|
} else {
|
||||||
|
format!("{}:{:02}", m, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What a finished configuration reconciliation did, in one line.
|
||||||
|
///
|
||||||
|
/// The counts are the point: a prune of a small index is over in a
|
||||||
|
/// millisecond, so this line is the only evidence the user gets that the
|
||||||
|
/// setting they changed reached the index at all. A clause whose count is
|
||||||
|
/// zero is left out rather than printed as "0", and a pass that found nothing
|
||||||
|
/// to change still reports that it ran — that it ran is the answer.
|
||||||
|
pub fn fmt_reconcile_summary(deleted: usize, recontented: usize) -> String {
|
||||||
|
let entries = |n: usize| {
|
||||||
|
format!(
|
||||||
|
"{} {}",
|
||||||
|
group_thousands(n as u64),
|
||||||
|
if n == 1 { "entry" } else { "entries" }
|
||||||
|
)
|
||||||
|
};
|
||||||
|
let mut parts: Vec<String> = Vec::new();
|
||||||
|
if deleted > 0 {
|
||||||
|
parts.push(format!("{} removed", entries(deleted)));
|
||||||
|
}
|
||||||
|
if recontented > 0 {
|
||||||
|
parts.push(format!("{} re-examined", entries(recontented)));
|
||||||
|
}
|
||||||
|
if parts.is_empty() {
|
||||||
|
return "Configuration change applied".to_string();
|
||||||
|
}
|
||||||
|
format!("Configuration change applied · {}", parts.join(" · "))
|
||||||
|
}
|
||||||
|
|
||||||
/// Middle-truncate a path to at most `max_chars` characters.
|
/// Middle-truncate a path to at most `max_chars` characters.
|
||||||
pub fn middle_truncate(s: &str, max_chars: usize) -> String {
|
pub fn middle_truncate(s: &str, max_chars: usize) -> String {
|
||||||
let chars: Vec<char> = s.chars().collect();
|
let chars: Vec<char> = s.chars().collect();
|
||||||
|
|
@ -141,10 +182,11 @@ mod tests {
|
||||||
assert_eq!(fmt_interval(0), "run");
|
assert_eq!(fmt_interval(0), "run");
|
||||||
assert_eq!(fmt_interval(1), "1 min");
|
assert_eq!(fmt_interval(1), "1 min");
|
||||||
assert_eq!(fmt_interval(59), "59 min");
|
assert_eq!(fmt_interval(59), "59 min");
|
||||||
assert_eq!(fmt_interval(60), "1 h");
|
assert_eq!(fmt_interval(60), "1 h", "the shipped default");
|
||||||
assert_eq!(fmt_interval(90), "1 h 30 min");
|
assert_eq!(fmt_interval(90), "1 h 30 min");
|
||||||
assert_eq!(fmt_interval(120), "2 h");
|
assert_eq!(fmt_interval(120), "2 h");
|
||||||
assert_eq!(fmt_interval(1440), "24 h", "the shipped default");
|
// A whole day is the one multiple-of-1440 case that stays in hours.
|
||||||
|
assert_eq!(fmt_interval(1440), "24 h");
|
||||||
assert_eq!(fmt_interval(2880), "2 days");
|
assert_eq!(fmt_interval(2880), "2 days");
|
||||||
assert_eq!(fmt_interval(10_080), "7 days");
|
assert_eq!(fmt_interval(10_080), "7 days");
|
||||||
}
|
}
|
||||||
|
|
@ -179,10 +221,7 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn ago_buckets() {
|
fn ago_buckets() {
|
||||||
let now = std::time::SystemTime::now()
|
let now = quicksearch_core::log::now_unix();
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_secs();
|
|
||||||
assert_eq!(fmt_ago(now), "just now");
|
assert_eq!(fmt_ago(now), "just now");
|
||||||
assert_eq!(fmt_ago(now - 59), "just now");
|
assert_eq!(fmt_ago(now - 59), "just now");
|
||||||
assert_eq!(fmt_ago(now - 120), "2 min ago");
|
assert_eq!(fmt_ago(now - 120), "2 min ago");
|
||||||
|
|
@ -190,6 +229,31 @@ mod tests {
|
||||||
assert!(fmt_ago(now - 200_000).contains('-'), "old = absolute date");
|
assert!(fmt_ago(now - 200_000).contains('-'), "old = absolute date");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reconcile_summaries_omit_what_did_not_happen() {
|
||||||
|
assert_eq!(
|
||||||
|
fmt_reconcile_summary(0, 0),
|
||||||
|
"Configuration change applied",
|
||||||
|
"a pass with nothing to do still reports that it ran"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
fmt_reconcile_summary(1, 0),
|
||||||
|
"Configuration change applied · 1 entry removed"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
fmt_reconcile_summary(1204, 0),
|
||||||
|
"Configuration change applied · 1,204 entries removed"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
fmt_reconcile_summary(0, 7),
|
||||||
|
"Configuration change applied · 7 entries re-examined"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
fmt_reconcile_summary(2, 3),
|
||||||
|
"Configuration change applied · 2 entries removed · 3 entries re-examined"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn truncation() {
|
fn truncation() {
|
||||||
assert_eq!(middle_truncate("short", 20), "short");
|
assert_eq!(middle_truncate("short", 20), "short");
|
||||||
|
|
|
||||||
|
|
@ -33,10 +33,16 @@ pub fn load_key(db_path: &str) -> Result<Option<String>, String> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Forget the remembered key. Best-effort: an entry that never existed or
|
/// Forget the remembered key. An entry that never existed is a fine outcome
|
||||||
/// a dead keychain daemon are both fine outcomes for "forget".
|
/// for "forget" and reports success.
|
||||||
pub fn delete_key(db_path: &str) {
|
///
|
||||||
if let Ok(entry) = entry(db_path) {
|
/// A real failure is not: the derived SQLCipher key is still sitting in the
|
||||||
let _ = entry.delete_credential();
|
/// OS keychain, so a caller that goes on to record "not remembered" would be
|
||||||
|
/// describing a machine state that isn't true. Callers surface this rather
|
||||||
|
/// than assuming the key is gone.
|
||||||
|
pub fn delete_key(db_path: &str) -> Result<(), String> {
|
||||||
|
match entry(db_path)?.delete_credential() {
|
||||||
|
Ok(()) | Err(keyring::Error::NoEntry) => Ok(()),
|
||||||
|
Err(e) => Err(format!("keychain delete failed: {}", e)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,9 @@ mod options;
|
||||||
mod platform;
|
mod platform;
|
||||||
mod query_highlight;
|
mod query_highlight;
|
||||||
mod search_tab;
|
mod search_tab;
|
||||||
|
#[cfg(test)]
|
||||||
|
mod test_ui;
|
||||||
|
mod tips;
|
||||||
mod tracker;
|
mod tracker;
|
||||||
mod ui_util;
|
mod ui_util;
|
||||||
mod unlock;
|
mod unlock;
|
||||||
|
|
|
||||||
|
|
@ -5,11 +5,18 @@ use std::path::Path;
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use quicksearch_core::config::Config;
|
use quicksearch_core::config::Config;
|
||||||
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
|
use quicksearch_core::coordinator::{IndexMode, IndexerState, ReconcileState, WatcherStatus};
|
||||||
use quicksearch_core::indexing::{IndexingStatus, RootPhase, RootProgress};
|
use quicksearch_core::indexing::{
|
||||||
|
IndexingStatus, PrepStep, ReconcileProgress, RootPhase, RootProgress,
|
||||||
|
};
|
||||||
|
|
||||||
use crate::format::{fmt_interval, fmt_rate, group_thousands, human_size, middle_truncate};
|
use crate::format::{
|
||||||
|
fmt_duration_clock, fmt_interval, fmt_rate, fmt_reconcile_summary, group_thousands, human_size,
|
||||||
|
middle_truncate,
|
||||||
|
};
|
||||||
|
use crate::tips::{self, Tipped};
|
||||||
use crate::tracker::SpeedTracker;
|
use crate::tracker::SpeedTracker;
|
||||||
|
use crate::ui_util::middle_elide;
|
||||||
|
|
||||||
/// What the tab asks the app to do after this frame.
|
/// What the tab asks the app to do after this frame.
|
||||||
#[derive(Default)]
|
#[derive(Default)]
|
||||||
|
|
@ -65,9 +72,13 @@ impl ManageTab {
|
||||||
let total: usize = roots.iter().map(|r| r.walked + r.extracted).sum();
|
let total: usize = roots.iter().map(|r| r.walked + r.extracted).sum();
|
||||||
self.speed.record(total);
|
self.speed.record(total);
|
||||||
}
|
}
|
||||||
IndexingStatus::Idle | IndexingStatus::Error(_) | IndexingStatus::Optimizing => {
|
// Preparing included: the prologue has its own counters and no
|
||||||
self.speed.reset()
|
// files to rate, and a stale files/sec left over from the last
|
||||||
}
|
// run would read as progress that is not happening.
|
||||||
|
IndexingStatus::Idle
|
||||||
|
| IndexingStatus::Error(_)
|
||||||
|
| IndexingStatus::Optimizing
|
||||||
|
| IndexingStatus::Preparing { .. } => self.speed.reset(),
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -202,6 +213,7 @@ impl ManageTab {
|
||||||
);
|
);
|
||||||
if ui
|
if ui
|
||||||
.add_enabled(!running, egui::Button::new("Start indexing now"))
|
.add_enabled(!running, egui::Button::new("Start indexing now"))
|
||||||
|
.tip(&tips::START_NOW)
|
||||||
.clicked()
|
.clicked()
|
||||||
{
|
{
|
||||||
actions.start_now = true;
|
actions.start_now = true;
|
||||||
|
|
@ -211,10 +223,7 @@ impl ManageTab {
|
||||||
running || state.mode == IndexMode::Auto,
|
running || state.mode == IndexMode::Auto,
|
||||||
egui::Button::new("Stop"),
|
egui::Button::new("Stop"),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
.tip(&tips::STOP_INDEXING)
|
||||||
"Stop indexing and switch to manual. Saved right away: it \
|
|
||||||
stays manual on the next launch too.",
|
|
||||||
)
|
|
||||||
.clicked()
|
.clicked()
|
||||||
{
|
{
|
||||||
actions.stop = true;
|
actions.stop = true;
|
||||||
|
|
@ -224,10 +233,7 @@ impl ManageTab {
|
||||||
state.mode != IndexMode::Auto,
|
state.mode != IndexMode::Auto,
|
||||||
egui::Button::new("Return to Automatic"),
|
egui::Button::new("Return to Automatic"),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
.tip(&tips::RETURN_TO_AUTO)
|
||||||
"Watch for changes and reindex periodically again. Also \
|
|
||||||
saved, so this is how the app starts from now on.",
|
|
||||||
)
|
|
||||||
.clicked()
|
.clicked()
|
||||||
{
|
{
|
||||||
actions.auto = true;
|
actions.auto = true;
|
||||||
|
|
@ -243,7 +249,7 @@ impl ManageTab {
|
||||||
.button(
|
.button(
|
||||||
egui::RichText::new("Clear index…").color(ui.visuals().error_fg_color),
|
egui::RichText::new("Clear index…").color(ui.visuals().error_fg_color),
|
||||||
)
|
)
|
||||||
.on_hover_text("Delete the index database (asks for confirmation)")
|
.tip(&tips::CLEAR_INDEX)
|
||||||
.clicked()
|
.clicked()
|
||||||
{
|
{
|
||||||
actions.clear_index = true;
|
actions.clear_index = true;
|
||||||
|
|
@ -268,7 +274,7 @@ impl ManageTab {
|
||||||
// never push them out of view; the path truncates into
|
// never push them out of view; the path truncates into
|
||||||
// whatever width remains (full path on hover).
|
// whatever width remains (full path on hover).
|
||||||
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
||||||
if ui.small_button("Remove").clicked() {
|
if ui.small_button("Remove").tip(&tips::REMOVE_ROOT).clicked() {
|
||||||
remove = Some(i);
|
remove = Some(i);
|
||||||
}
|
}
|
||||||
// Per-root walker override; 0 = auto (4 local / 16
|
// Per-root walker override; 0 = auto (4 local / 16
|
||||||
|
|
@ -295,11 +301,7 @@ impl ManageTab {
|
||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
.tip(&tips::ROOT_WORKERS);
|
||||||
"Walker threads for this folder. auto = 4 on local \
|
|
||||||
storage, 16 on network mounts. Takes effect on \
|
|
||||||
the next indexing run.",
|
|
||||||
);
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
tests::record_widget("workers", &response);
|
tests::record_widget("workers", &response);
|
||||||
if response.changed() {
|
if response.changed() {
|
||||||
|
|
@ -316,12 +318,9 @@ impl ManageTab {
|
||||||
egui::Layout::left_to_right(egui::Align::Center),
|
egui::Layout::left_to_right(egui::Align::Center),
|
||||||
|ui| {
|
|ui| {
|
||||||
let font_id = egui::TextStyle::Monospace.resolve(ui.style());
|
let font_id = egui::TextStyle::Monospace.resolve(ui.style());
|
||||||
let char_width =
|
let shown =
|
||||||
ui.fonts(|f| f.glyph_width(&font_id, '0')).max(1.0);
|
middle_elide(ui, root, ui.available_width(), &font_id);
|
||||||
let budget =
|
ui.monospace(shown.as_ref()).on_hover_text(root);
|
||||||
((ui.available_width() / char_width) as usize).max(16);
|
|
||||||
ui.monospace(middle_truncate(root, budget))
|
|
||||||
.on_hover_text(root);
|
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
@ -332,7 +331,7 @@ impl ManageTab {
|
||||||
draft.indexing.root_workers.remove(&removed);
|
draft.indexing.root_workers.remove(&removed);
|
||||||
}
|
}
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
if ui.button("Add folder…").clicked() {
|
if ui.button("Add folder…").tip(&tips::ADD_ROOT).clicked() {
|
||||||
if let Some(dir) = rfd::FileDialog::new().pick_folder() {
|
if let Some(dir) = rfd::FileDialog::new().pick_folder() {
|
||||||
let path = dir.to_string_lossy().into_owned();
|
let path = dir.to_string_lossy().into_owned();
|
||||||
try_add_root(draft, path, &mut self.root_error);
|
try_add_root(draft, path, &mut self.root_error);
|
||||||
|
|
@ -342,8 +341,11 @@ impl ManageTab {
|
||||||
egui::TextEdit::singleline(&mut self.new_root)
|
egui::TextEdit::singleline(&mut self.new_root)
|
||||||
.desired_width(240.0)
|
.desired_width(240.0)
|
||||||
.hint_text("or type a path"),
|
.hint_text("or type a path"),
|
||||||
);
|
)
|
||||||
if ui.button("Add").clicked() && !self.new_root.trim().is_empty() {
|
.tip(&tips::ADD_ROOT);
|
||||||
|
if ui.button("Add").tip(&tips::ADD_ROOT).clicked()
|
||||||
|
&& !self.new_root.trim().is_empty()
|
||||||
|
{
|
||||||
let path = self.new_root.trim().to_string();
|
let path = self.new_root.trim().to_string();
|
||||||
if try_add_root(draft, path, &mut self.root_error) {
|
if try_add_root(draft, path, &mut self.root_error) {
|
||||||
self.new_root.clear();
|
self.new_root.clear();
|
||||||
|
|
@ -369,7 +371,9 @@ impl ManageTab {
|
||||||
// --- Filters ---------------------------------------------------
|
// --- Filters ---------------------------------------------------
|
||||||
ui.heading(egui::RichText::new("Content filters").strong());
|
ui.heading(egui::RichText::new("Content filters").strong());
|
||||||
ui.columns(2, |cols| {
|
ui.columns(2, |cols| {
|
||||||
cols[0].label("Full-text extensions whitelist (empty = all supported):");
|
cols[0]
|
||||||
|
.label("Full-text extensions whitelist (empty = all supported):")
|
||||||
|
.tip(&tips::EXT_WHITELIST);
|
||||||
cols[0]
|
cols[0]
|
||||||
.add(
|
.add(
|
||||||
egui::TextEdit::multiline(&mut self.ext_filter_text)
|
egui::TextEdit::multiline(&mut self.ext_filter_text)
|
||||||
|
|
@ -377,15 +381,10 @@ impl ManageTab {
|
||||||
.desired_width(f32::INFINITY)
|
.desired_width(f32::INFINITY)
|
||||||
.hint_text("txt\nmd\npdf # comments allowed\n(none)"),
|
.hint_text("txt\nmd\npdf # comments allowed\n(none)"),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
.tip(&tips::EXT_WHITELIST);
|
||||||
"One extension per line, leading dot optional. A non-empty \
|
cols[1]
|
||||||
list also excludes files that have no extension at all \
|
.label("Ignore patterns (excluded entirely):")
|
||||||
(Makefile, README, .bashrc) — add the line \"(none)\" to \
|
.tip(&tips::IGNORE_PATTERNS);
|
||||||
keep extracting text from those.\n\n\
|
|
||||||
\"#\" starts a comment, either on its own line or after an \
|
|
||||||
entry, so a type can be commented out without losing it.",
|
|
||||||
);
|
|
||||||
cols[1].label("Ignore patterns (excluded entirely):");
|
|
||||||
let mut remove_pat: Option<usize> = None;
|
let mut remove_pat: Option<usize> = None;
|
||||||
// The list grows and shrinks — including from outside
|
// The list grows and shrinks — including from outside
|
||||||
// this tab — so it is kept off the id of the editor
|
// this tab — so it is kept off the id of the editor
|
||||||
|
|
@ -425,7 +424,11 @@ impl ManageTab {
|
||||||
);
|
);
|
||||||
let submitted =
|
let submitted =
|
||||||
response.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter));
|
response.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter));
|
||||||
if ui.add_enabled(valid, egui::Button::new("Add")).clicked()
|
response.tip(&tips::IGNORE_PATTERNS);
|
||||||
|
if ui
|
||||||
|
.add_enabled(valid, egui::Button::new("Add"))
|
||||||
|
.tip(&tips::IGNORE_PATTERNS)
|
||||||
|
.clicked()
|
||||||
|| (submitted && valid)
|
|| (submitted && valid)
|
||||||
{
|
{
|
||||||
let pat = self.new_ignore.trim().to_string();
|
let pat = self.new_ignore.trim().to_string();
|
||||||
|
|
@ -463,14 +466,16 @@ impl ManageTab {
|
||||||
|
|
||||||
let dirty = self.is_dirty();
|
let dirty = self.is_dirty();
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
let apply = ui.add(crate::ui_util::bordered_button(
|
let apply = ui
|
||||||
|
.add(crate::ui_util::bordered_button(
|
||||||
"Apply & Save",
|
"Apply & Save",
|
||||||
if dirty {
|
if dirty {
|
||||||
crate::ui_util::ORANGE
|
crate::ui_util::ORANGE
|
||||||
} else {
|
} else {
|
||||||
crate::ui_util::BLUE
|
crate::ui_util::BLUE
|
||||||
},
|
},
|
||||||
));
|
))
|
||||||
|
.tip(&tips::APPLY_SAVE);
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
tests::record_widget("apply", &apply);
|
tests::record_widget("apply", &apply);
|
||||||
// The label comes and goes with the dirty state; keep it
|
// The label comes and goes with the dirty state; keep it
|
||||||
|
|
@ -572,7 +577,7 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
|
||||||
ui.set_max_width(440.0);
|
ui.set_max_width(440.0);
|
||||||
ui.strong("To reduce the index size");
|
ui.strong("To reduce the index size");
|
||||||
for lever in [
|
for lever in [
|
||||||
"Add ignore filters for files and folders you never search — the ignore \
|
"Add ignore filters for files and folders you never search, in the ignore \
|
||||||
pattern list further down this tab.",
|
pattern list further down this tab.",
|
||||||
"Remove indexed folders you do not need, in Indexed folders above.",
|
"Remove indexed folders you do not need, in Indexed folders above.",
|
||||||
"Narrow the full-text extension whitelist, so text is only extracted \
|
"Narrow the full-text extension whitelist, so text is only extracted \
|
||||||
|
|
@ -588,7 +593,7 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
|
||||||
ui.label(
|
ui.label(
|
||||||
egui::RichText::new(
|
egui::RichText::new(
|
||||||
"Narrowing any of these removes the entries it excludes straight away, \
|
"Narrowing any of these removes the entries it excludes straight away, \
|
||||||
but the file does not shrink on its own: the freed space is reused by \
|
but the file does not shrink on its own. The freed space is reused by \
|
||||||
the index rather than returned to the disk, until an indexing run's \
|
the index rather than returned to the disk, until an indexing run's \
|
||||||
optimize pass compacts it.",
|
optimize pass compacts it.",
|
||||||
)
|
)
|
||||||
|
|
@ -682,6 +687,21 @@ fn status_panel(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
|
||||||
fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
|
fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
|
||||||
match &state.activity {
|
match &state.activity {
|
||||||
IndexingStatus::Idle => {
|
IndexingStatus::Idle => {
|
||||||
|
// A reconcile the coordinator is applying between runs. No run
|
||||||
|
// owns the index, so the activity really is Idle — but the thread
|
||||||
|
// is scanning every row, which on a large index is minutes of
|
||||||
|
// work that used to be reported as nothing at all.
|
||||||
|
match &state.reconcile {
|
||||||
|
Some(ReconcileState::Running(r)) => return reconcile_row(ui, r, None),
|
||||||
|
// Kept on screen for a few seconds after the work ends: a
|
||||||
|
// narrowed filter on a small index is applied faster than the
|
||||||
|
// display could show it happening.
|
||||||
|
Some(ReconcileState::Finished(r)) => {
|
||||||
|
ui.label(fmt_reconcile_summary(r.deleted, r.recontented));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
None => {}
|
||||||
|
}
|
||||||
// Relative wording makes even a milliseconds-fast run visibly
|
// Relative wording makes even a milliseconds-fast run visibly
|
||||||
// register ("just now") instead of looking like a dead button.
|
// register ("just now") instead of looking like a dead button.
|
||||||
let last = state
|
let last = state
|
||||||
|
|
@ -690,6 +710,9 @@ fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker
|
||||||
.unwrap_or_else(|| "never".to_string());
|
.unwrap_or_else(|| "never".to_string());
|
||||||
ui.label(format!("Idle; last full index: {}", last));
|
ui.label(format!("Idle; last full index: {}", last));
|
||||||
}
|
}
|
||||||
|
IndexingStatus::Preparing { start_time, step } => {
|
||||||
|
prep_row(ui, step, start_time.elapsed());
|
||||||
|
}
|
||||||
IndexingStatus::Error(e) => {
|
IndexingStatus::Error(e) => {
|
||||||
ui.colored_label(ui.visuals().error_fg_color, format!("Error: {}", e));
|
ui.colored_label(ui.visuals().error_fg_color, format!("Error: {}", e));
|
||||||
}
|
}
|
||||||
|
|
@ -716,6 +739,99 @@ fn status_contents(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// What a run is doing before it walks its first file.
|
||||||
|
///
|
||||||
|
/// Every one of these can outlast the walk itself on a large index, and they
|
||||||
|
/// all used to render as one motionless "Starting…". The elapsed clock is the
|
||||||
|
/// point of the row: it is the only thing that distinguishes slow work from a
|
||||||
|
/// hang, and it is why each label carries one even when there is nothing to
|
||||||
|
/// count.
|
||||||
|
fn prep_row(ui: &mut egui::Ui, step: &PrepStep, elapsed: Duration) {
|
||||||
|
match step {
|
||||||
|
PrepStep::PreviousRun => waiting_row(ui, "Finishing the previous run…", elapsed),
|
||||||
|
PrepStep::OpeningIndex => waiting_row(ui, "Opening the index…", elapsed),
|
||||||
|
PrepStep::Reconciling(r) => reconcile_row(ui, r, Some(elapsed)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A prologue step with no counters: label, clock, indeterminate bar.
|
||||||
|
fn waiting_row(ui: &mut egui::Ui, label: &str, elapsed: Duration) {
|
||||||
|
ui.horizontal(|ui| {
|
||||||
|
ui.label(label);
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(fmt_duration_clock(elapsed))
|
||||||
|
.small()
|
||||||
|
.weak(),
|
||||||
|
);
|
||||||
|
ui.add(
|
||||||
|
egui::ProgressBar::new(0.0)
|
||||||
|
.animate(true)
|
||||||
|
.desired_width(160.0),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A configuration reconciliation, from either place one runs.
|
||||||
|
///
|
||||||
|
/// `elapsed` is `Some` for a run's prologue, which has a start time to
|
||||||
|
/// measure from; the coordinator's between-runs pass has none, and its
|
||||||
|
/// examined count moves often enough to serve the same purpose.
|
||||||
|
fn reconcile_row(ui: &mut egui::Ui, r: &ReconcileProgress, elapsed: Option<Duration>) {
|
||||||
|
ui.horizontal(|ui| {
|
||||||
|
ui.label("Applying configuration change");
|
||||||
|
ui.label(egui::RichText::new("|").weak());
|
||||||
|
match (r.total, r.fraction()) {
|
||||||
|
(Some(total), Some(frac)) => {
|
||||||
|
ui.label(format!(
|
||||||
|
"{} / {} ({:.0}%) entries checked",
|
||||||
|
group_thousands(r.examined as u64),
|
||||||
|
group_thousands(total as u64),
|
||||||
|
frac * 100.0
|
||||||
|
));
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
ui.label(format!(
|
||||||
|
"{} entries checked",
|
||||||
|
group_thousands(r.examined as u64)
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(elapsed) = elapsed {
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(fmt_duration_clock(elapsed))
|
||||||
|
.small()
|
||||||
|
.weak(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
match r.fraction() {
|
||||||
|
Some(frac) => {
|
||||||
|
ui.add(egui::ProgressBar::new(frac as f32).desired_width(160.0));
|
||||||
|
}
|
||||||
|
// Whole-range deletions read no rows, so they reach the bar with
|
||||||
|
// no denominator — the same indeterminate form a walk uses before
|
||||||
|
// its count lands.
|
||||||
|
None => {
|
||||||
|
ui.add(
|
||||||
|
egui::ProgressBar::new(0.0)
|
||||||
|
.animate(true)
|
||||||
|
.desired_width(160.0),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
if r.deleted > 0 || r.recontented > 0 {
|
||||||
|
ui.label(
|
||||||
|
egui::RichText::new(format!(
|
||||||
|
"{} entries removed, {} re-examined for text extraction",
|
||||||
|
group_thousands(r.deleted as u64),
|
||||||
|
group_thousands(r.recontented as u64)
|
||||||
|
))
|
||||||
|
.small()
|
||||||
|
.weak(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// One root's progress: path, phase, bar, counters, current file.
|
/// One root's progress: path, phase, bar, counters, current file.
|
||||||
fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
|
fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
|
||||||
// Weak "|" separators split the row into folder | status | numbers.
|
// Weak "|" separators split the row into folder | status | numbers.
|
||||||
|
|
@ -830,6 +946,7 @@ mod tests {
|
||||||
last_full_index: Some(0),
|
last_full_index: Some(0),
|
||||||
queued_events: 0,
|
queued_events: 0,
|
||||||
watcher: WatcherStatus::Active { dirs: 10 },
|
watcher: WatcherStatus::Active { dirs: 10 },
|
||||||
|
reconcile: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -858,28 +975,31 @@ mod tests {
|
||||||
/// contents — not just their widget ids — are under test.
|
/// contents — not just their widget ids — are under test.
|
||||||
fn state_with(roots: Vec<RootProgress>) -> IndexerState {
|
fn state_with(roots: Vec<RootProgress>) -> IndexerState {
|
||||||
IndexerState {
|
IndexerState {
|
||||||
mode: IndexMode::Auto,
|
|
||||||
activity: IndexingStatus::Running {
|
activity: IndexingStatus::Running {
|
||||||
start_time: std::time::Instant::now(),
|
start_time: std::time::Instant::now(),
|
||||||
roots,
|
roots,
|
||||||
},
|
},
|
||||||
last_full_index: Some(0),
|
..idle_state()
|
||||||
queued_events: 0,
|
|
||||||
watcher: WatcherStatus::Active { dirs: 10 },
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn raw_input(events: Vec<egui::Event>) -> egui::RawInput {
|
/// A run still in its prologue, before the walk has produced anything.
|
||||||
egui::RawInput {
|
fn preparing_state(step: PrepStep) -> IndexerState {
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
IndexerState {
|
||||||
egui::Pos2::ZERO,
|
activity: IndexingStatus::Preparing {
|
||||||
egui::vec2(1000.0, 900.0),
|
start_time: std::time::Instant::now(),
|
||||||
)),
|
step,
|
||||||
events,
|
},
|
||||||
..Default::default()
|
..idle_state()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
use crate::test_ui::{click_at, painted_text};
|
||||||
|
|
||||||
|
fn raw_input(events: Vec<egui::Event>) -> egui::RawInput {
|
||||||
|
crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events)
|
||||||
|
}
|
||||||
|
|
||||||
/// One frame of the real tab, with `events` delivered to it.
|
/// One frame of the real tab, with `events` delivered to it.
|
||||||
fn frame(
|
fn frame(
|
||||||
ctx: &egui::Context,
|
ctx: &egui::Context,
|
||||||
|
|
@ -917,23 +1037,7 @@ mod tests {
|
||||||
tab.ui(ui, state, cfg);
|
tab.ui(ui, state, cfg);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
let mut text = Vec::new();
|
painted_text(&out)
|
||||||
for clipped in &out.shapes {
|
|
||||||
collect_text(&clipped.shape, &mut text);
|
|
||||||
}
|
|
||||||
text
|
|
||||||
}
|
|
||||||
|
|
||||||
fn collect_text(shape: &egui::epaint::Shape, out: &mut Vec<String>) {
|
|
||||||
match shape {
|
|
||||||
egui::epaint::Shape::Text(t) => out.push(t.galley.text().to_string()),
|
|
||||||
egui::epaint::Shape::Vec(shapes) => {
|
|
||||||
for s in shapes {
|
|
||||||
collect_text(s, out);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn pointer(pos: egui::Pos2, pressed: bool) -> egui::Event {
|
fn pointer(pos: egui::Pos2, pressed: bool) -> egui::Event {
|
||||||
|
|
@ -945,14 +1049,6 @@ mod tests {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
|
|
||||||
vec![
|
|
||||||
egui::Event::PointerMoved(pos),
|
|
||||||
pointer(pos, true),
|
|
||||||
pointer(pos, false),
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
fn cfg_with_root() -> Config {
|
fn cfg_with_root() -> Config {
|
||||||
let mut cfg = Config::default();
|
let mut cfg = Config::default();
|
||||||
cfg.paths.indexing_paths = vec!["/data".into()];
|
cfg.paths.indexing_paths = vec!["/data".into()];
|
||||||
|
|
@ -1181,14 +1277,154 @@ mod tests {
|
||||||
assert!(!text.contains(" / "), "invented a denominator: {}", text);
|
assert!(!text.contains(" / "), "invented a denominator: {}", text);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Every step of the prologue names itself. These used to be one static
|
||||||
|
/// "Starting…" with no clock, which on a large index is indistinguishable
|
||||||
|
/// from a hang for as long as the step takes.
|
||||||
|
#[test]
|
||||||
|
fn each_prologue_step_says_what_it_is_waiting_on() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
|
||||||
|
for (step, expected) in [
|
||||||
|
(PrepStep::PreviousRun, "Finishing the previous run…"),
|
||||||
|
(PrepStep::OpeningIndex, "Opening the index…"),
|
||||||
|
] {
|
||||||
|
let text = frame_text(&ctx, &mut tab, &preparing_state(step)).join(" | ");
|
||||||
|
assert!(text.contains(expected), "{}", text);
|
||||||
|
// The clock is the point of the row.
|
||||||
|
assert!(text.contains("0:00"), "no elapsed time shown: {}", text);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reconcile is the long one, and the only prologue step with
|
||||||
|
/// something to count. It reports its position in the scan.
|
||||||
|
#[test]
|
||||||
|
fn a_reconcile_reports_how_far_through_the_index_it_is() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
let text = frame_text(
|
||||||
|
&ctx,
|
||||||
|
&mut tab,
|
||||||
|
&preparing_state(PrepStep::Reconciling(ReconcileProgress {
|
||||||
|
examined: 2_500_000,
|
||||||
|
total: Some(8_000_000),
|
||||||
|
deleted: 1_204,
|
||||||
|
recontented: 0,
|
||||||
|
})),
|
||||||
|
)
|
||||||
|
.join(" | ");
|
||||||
|
|
||||||
|
assert!(text.contains("Applying configuration change"), "{}", text);
|
||||||
|
assert!(
|
||||||
|
text.contains("2,500,000 / 8,000,000 (31%) entries checked"),
|
||||||
|
"{}",
|
||||||
|
text
|
||||||
|
);
|
||||||
|
assert!(text.contains("1,204 entries removed"), "{}", text);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whole-range deletions read no rows, so the scan can reach the display
|
||||||
|
/// with nothing to divide by. It must not invent a denominator — the same
|
||||||
|
/// rule the walk follows before its count lands.
|
||||||
|
#[test]
|
||||||
|
fn a_reconcile_without_a_row_count_shows_no_denominator() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
let text = frame_text(
|
||||||
|
&ctx,
|
||||||
|
&mut tab,
|
||||||
|
&preparing_state(PrepStep::Reconciling(ReconcileProgress::default())),
|
||||||
|
)
|
||||||
|
.join(" | ");
|
||||||
|
assert!(text.contains("0 entries checked"), "{}", text);
|
||||||
|
assert!(!text.contains(" / "), "invented a denominator: {}", text);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The coordinator applies a prune between runs, so no run owns the index
|
||||||
|
/// and the activity really is `Idle` — but the thread is scanning every
|
||||||
|
/// row of it. Reporting "Idle" for the minutes that takes is what made a
|
||||||
|
/// working app look like a stopped one.
|
||||||
|
#[test]
|
||||||
|
fn a_prune_between_runs_is_reported_instead_of_idle() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
let state = IndexerState {
|
||||||
|
reconcile: Some(ReconcileState::Running(ReconcileProgress {
|
||||||
|
examined: 40_000,
|
||||||
|
total: Some(80_000),
|
||||||
|
deleted: 0,
|
||||||
|
recontented: 0,
|
||||||
|
})),
|
||||||
|
..idle_state()
|
||||||
|
};
|
||||||
|
|
||||||
|
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
|
||||||
|
assert!(
|
||||||
|
text.contains("Applying configuration change"),
|
||||||
|
"the scan is invisible: {}",
|
||||||
|
text
|
||||||
|
);
|
||||||
|
assert!(text.contains("40,000 / 80,000 (50%)"), "{}", text);
|
||||||
|
assert!(
|
||||||
|
!text.contains("Idle; last full index"),
|
||||||
|
"reported idle while scanning: {}",
|
||||||
|
text
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The pass itself can be over between two frames — narrowing a filter on
|
||||||
|
/// a small index is one transaction. Without a tail the user changes a
|
||||||
|
/// setting, sees nothing move, and has no way to tell whether it took.
|
||||||
|
#[test]
|
||||||
|
fn a_finished_prune_reports_what_it_did() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
let state = IndexerState {
|
||||||
|
reconcile: Some(ReconcileState::Finished(ReconcileProgress {
|
||||||
|
examined: 80_000,
|
||||||
|
total: Some(80_000),
|
||||||
|
deleted: 1_204,
|
||||||
|
recontented: 7,
|
||||||
|
})),
|
||||||
|
..idle_state()
|
||||||
|
};
|
||||||
|
|
||||||
|
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
|
||||||
|
assert!(
|
||||||
|
text.contains("Configuration change applied"),
|
||||||
|
"the finished pass left no trace: {}",
|
||||||
|
text
|
||||||
|
);
|
||||||
|
assert!(text.contains("1,204 entries removed"), "{}", text);
|
||||||
|
assert!(text.contains("7 entries re-examined"), "{}", text);
|
||||||
|
assert!(
|
||||||
|
!text.contains("Idle; last full index"),
|
||||||
|
"the summary was replaced by the idle line: {}",
|
||||||
|
text
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The placeholder this whole prologue display replaced. It carried no
|
||||||
|
/// phase, no counters and no clock, and it was on screen for every one of
|
||||||
|
/// the steps above.
|
||||||
|
#[test]
|
||||||
|
fn the_starting_placeholder_is_gone() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
for state in [
|
||||||
|
preparing_state(PrepStep::PreviousRun),
|
||||||
|
preparing_state(PrepStep::OpeningIndex),
|
||||||
|
preparing_state(PrepStep::Reconciling(ReconcileProgress::default())),
|
||||||
|
idle_state(),
|
||||||
|
] {
|
||||||
|
let text = frame_text(&ctx, &mut tab, &state).join(" | ");
|
||||||
|
assert!(!text.contains("Starting"), "{}", text);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// A scratch directory of its own for each size test, so two of them
|
/// A scratch directory of its own for each size test, so two of them
|
||||||
/// running in parallel cannot see each other's files.
|
/// running in parallel cannot see each other's files.
|
||||||
fn scratch_dir(tag: &str) -> std::path::PathBuf {
|
use quicksearch_core::testutil::scratch_dir;
|
||||||
let dir = std::env::temp_dir().join(format!("qs-dbsize-{}-{}", std::process::id(), tag));
|
|
||||||
let _ = std::fs::remove_dir_all(&dir);
|
|
||||||
std::fs::create_dir_all(&dir).expect("scratch dir");
|
|
||||||
dir
|
|
||||||
}
|
|
||||||
|
|
||||||
fn write_bytes(path: &Path, len: usize) {
|
fn write_bytes(path: &Path, len: usize) {
|
||||||
std::fs::write(path, vec![b'x'; len]).expect("write");
|
std::fs::write(path, vec![b'x'; len]).expect("write");
|
||||||
|
|
@ -1600,4 +1836,40 @@ mod tests {
|
||||||
);
|
);
|
||||||
assert_eq!(widget("apply").0, clean_id, "the label renamed the button");
|
assert_eq!(widget("apply").0, clean_id, "the label renamed the button");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The status area gains a row while a prune runs, another while its
|
||||||
|
/// summary lingers, and loses both afterwards. Every one of those frames
|
||||||
|
/// must leave the widgets below it with the ids they had: an editor whose
|
||||||
|
/// id changes mid-edit loses its buffer, and a prune is exactly when the
|
||||||
|
/// user is looking at the filter list they just changed.
|
||||||
|
#[test]
|
||||||
|
fn the_prune_rows_come_and_go_without_renaming_anything_below() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = ManageTab::new();
|
||||||
|
let cfg = cfg_with_root();
|
||||||
|
let progress = ReconcileProgress {
|
||||||
|
examined: 40_000,
|
||||||
|
total: Some(80_000),
|
||||||
|
deleted: 12,
|
||||||
|
recontented: 0,
|
||||||
|
};
|
||||||
|
|
||||||
|
frame_text_with(&ctx, &mut tab, &cfg, &idle_state());
|
||||||
|
let (apply, _) = widget("apply");
|
||||||
|
let (workers, _) = widget("workers");
|
||||||
|
|
||||||
|
for reconcile in [
|
||||||
|
Some(ReconcileState::Running(progress)),
|
||||||
|
Some(ReconcileState::Finished(progress)),
|
||||||
|
None,
|
||||||
|
] {
|
||||||
|
let state = IndexerState {
|
||||||
|
reconcile,
|
||||||
|
..idle_state()
|
||||||
|
};
|
||||||
|
frame_text_with(&ctx, &mut tab, &cfg, &state);
|
||||||
|
assert_eq!(widget("apply").0, apply, "renamed by {reconcile:?}");
|
||||||
|
assert_eq!(widget("workers").0, workers, "renamed by {reconcile:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@
|
||||||
//! validates, saves, and hands the new config to the app.
|
//! validates, saves, and hands the new config to the app.
|
||||||
|
|
||||||
use crate::keychain;
|
use crate::keychain;
|
||||||
|
use crate::tips::{self, tip_row, Tipped};
|
||||||
use quicksearch_core::config::Config;
|
use quicksearch_core::config::Config;
|
||||||
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
|
@ -144,12 +145,12 @@ impl OptionsWindow {
|
||||||
.show(ui, |ui| {
|
.show(ui, |ui| {
|
||||||
ui.heading(egui::RichText::new("Paths").strong());
|
ui.heading(egui::RichText::new("Paths").strong());
|
||||||
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
|
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
|
||||||
ui.label("Database file");
|
tip_row(ui, "Database file", &tips::DATABASE_PATH, |ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::TextEdit::singleline(&mut draft.paths.database_path)
|
egui::TextEdit::singleline(&mut draft.paths.database_path)
|
||||||
.desired_width(260.0),
|
.desired_width(260.0),
|
||||||
);
|
)
|
||||||
ui.end_row();
|
});
|
||||||
});
|
});
|
||||||
ui.label(
|
ui.label(
|
||||||
egui::RichText::new(
|
egui::RichText::new(
|
||||||
|
|
@ -182,18 +183,13 @@ impl OptionsWindow {
|
||||||
|
|
||||||
ui.heading(egui::RichText::new("Interface").strong());
|
ui.heading(egui::RichText::new("Interface").strong());
|
||||||
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
|
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
|
||||||
ui.label("UI scale");
|
tip_row(ui, "UI scale", &tips::UI_SCALE, |ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
|
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
|
||||||
.step_by(0.05)
|
.step_by(0.05)
|
||||||
.fixed_decimals(2),
|
.fixed_decimals(2),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
});
|
||||||
"Zooms the whole interface: fonts, spacing, and \
|
|
||||||
widgets. Ctrl +/- and Ctrl 0 adjust it temporarily \
|
|
||||||
at runtime.",
|
|
||||||
);
|
|
||||||
ui.end_row();
|
|
||||||
});
|
});
|
||||||
ui.separator();
|
ui.separator();
|
||||||
|
|
||||||
|
|
@ -208,14 +204,16 @@ impl OptionsWindow {
|
||||||
|
|
||||||
ui.separator();
|
ui.separator();
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
let apply = ui.add(crate::ui_util::bordered_button(
|
let apply = ui
|
||||||
|
.add(crate::ui_util::bordered_button(
|
||||||
"Apply & Save",
|
"Apply & Save",
|
||||||
if dirty {
|
if dirty {
|
||||||
crate::ui_util::ORANGE
|
crate::ui_util::ORANGE
|
||||||
} else {
|
} else {
|
||||||
crate::ui_util::BLUE
|
crate::ui_util::BLUE
|
||||||
},
|
},
|
||||||
));
|
))
|
||||||
|
.tip(&tips::APPLY_SAVE);
|
||||||
if apply.clicked() {
|
if apply.clicked() {
|
||||||
out.applied = Some(draft.clone());
|
out.applied = Some(draft.clone());
|
||||||
}
|
}
|
||||||
|
|
@ -264,27 +262,36 @@ fn security_ui(
|
||||||
ui.label("The index is encrypted; a password is required at startup.");
|
ui.label("The index is encrypted; a password is required at startup.");
|
||||||
}
|
}
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
if ui.button("Change password…").clicked() {
|
if ui
|
||||||
|
.button("Change password…")
|
||||||
|
.tip(&tips::CHANGE_PASSWORD)
|
||||||
|
.clicked()
|
||||||
|
{
|
||||||
action = Some(SecurityAction::ChangePassword);
|
action = Some(SecurityAction::ChangePassword);
|
||||||
}
|
}
|
||||||
if ui.button("Disable protection…").clicked() {
|
if ui
|
||||||
|
.button("Disable protection…")
|
||||||
|
.tip(&tips::DISABLE_PASSWORD)
|
||||||
|
.clicked()
|
||||||
|
{
|
||||||
action = Some(SecurityAction::Disable);
|
action = Some(SecurityAction::Disable);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
let mut remember = current.security.use_keychain;
|
let mut remember = current.security.use_keychain;
|
||||||
if ui
|
if ui
|
||||||
.checkbox(&mut remember, "Remember on this device")
|
.checkbox(&mut remember, "Remember on this device")
|
||||||
.on_hover_text(
|
.tip(&tips::REMEMBER_KEYCHAIN)
|
||||||
"Stores the derived key (not the password) in the OS keychain \
|
|
||||||
and skips the startup prompt on this machine.",
|
|
||||||
)
|
|
||||||
.changed()
|
.changed()
|
||||||
{
|
{
|
||||||
action = Some(SecurityAction::SetKeychain(remember));
|
action = Some(SecurityAction::SetKeychain(remember));
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
ui.label("The index is not encrypted.");
|
ui.label("The index is not encrypted.");
|
||||||
if ui.button("Enable password protection…").clicked() {
|
if ui
|
||||||
|
.button("Enable password protection…")
|
||||||
|
.tip(&tips::ENABLE_PASSWORD)
|
||||||
|
.clicked()
|
||||||
|
{
|
||||||
action = Some(SecurityAction::Enable);
|
action = Some(SecurityAction::Enable);
|
||||||
}
|
}
|
||||||
ui.label(
|
ui.label(
|
||||||
|
|
@ -299,8 +306,11 @@ fn security_ui(
|
||||||
action
|
action
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One implementation of the per-section config controls, shared by the
|
/// The per-section config controls of the Options window.
|
||||||
/// Options window and the Manage tab.
|
///
|
||||||
|
/// Every row goes through [`crate::tips::tip_row`], which takes the tooltip
|
||||||
|
/// that explains it: a setting cannot arrive here without one, and hovering
|
||||||
|
/// the name works as well as hovering the control.
|
||||||
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
|
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
|
||||||
match section {
|
match section {
|
||||||
Section::Indexing => {
|
Section::Indexing => {
|
||||||
|
|
@ -311,30 +321,31 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
|
||||||
// state, switched (and saved) by the Stop / Return to
|
// state, switched (and saved) by the Stop / Return to
|
||||||
// Automatic buttons on the Manage Index tab. A staged copy
|
// Automatic buttons on the Manage Index tab. A staged copy
|
||||||
// of it here would fight those buttons.
|
// of it here would fight those buttons.
|
||||||
ui.label("Full reindex every");
|
tip_row(ui, "Full reindex every", &tips::REINDEX_INTERVAL, |ui| {
|
||||||
ui.horizontal(|ui| {
|
ui.horizontal(|ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.indexing.reindex_interval_minutes)
|
egui::DragValue::new(&mut config.indexing.reindex_interval_minutes)
|
||||||
.range(5..=60 * 24 * 30),
|
.range(5..=60 * 24 * 30),
|
||||||
);
|
);
|
||||||
ui.label("minutes");
|
ui.label("minutes");
|
||||||
|
})
|
||||||
|
.response
|
||||||
});
|
});
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Follow symlinks");
|
tip_row(ui, "Follow symlinks", &tips::FOLLOW_SYMLINKS, |ui| {
|
||||||
ui.checkbox(&mut config.indexing.follow_symlinks, "");
|
ui.checkbox(&mut config.indexing.follow_symlinks, "")
|
||||||
ui.end_row();
|
});
|
||||||
|
|
||||||
ui.label("Include hidden files");
|
tip_row(ui, "Include hidden files", &tips::INCLUDE_HIDDEN, |ui| {
|
||||||
ui.checkbox(&mut config.indexing.include_hidden, "");
|
ui.checkbox(&mut config.indexing.include_hidden, "")
|
||||||
ui.end_row();
|
});
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Section::Processing => {
|
Section::Processing => {
|
||||||
egui::Grid::new("cfg-processing")
|
egui::Grid::new("cfg-processing")
|
||||||
.num_columns(2)
|
.num_columns(2)
|
||||||
.show(ui, |ui| {
|
.show(ui, |ui| {
|
||||||
ui.label("Tokenizer");
|
tip_row(ui, "Tokenizer", &tips::TOKENIZER, |ui| {
|
||||||
egui::ComboBox::from_id_salt("cfg-tokenize")
|
egui::ComboBox::from_id_salt("cfg-tokenize")
|
||||||
.selected_text(&config.processing.tokenize)
|
.selected_text(&config.processing.tokenize)
|
||||||
.show_ui(ui, |ui| {
|
.show_ui(ui, |ui| {
|
||||||
|
|
@ -345,8 +356,9 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
|
||||||
opt,
|
opt,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
})
|
||||||
|
.response
|
||||||
});
|
});
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("");
|
ui.label("");
|
||||||
ui.hyperlink_to(
|
ui.hyperlink_to(
|
||||||
|
|
@ -355,101 +367,85 @@ pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section
|
||||||
);
|
);
|
||||||
ui.end_row();
|
ui.end_row();
|
||||||
|
|
||||||
ui.label("Hash sample size (bytes)");
|
tip_row(ui, "Hash sample size (bytes)", &tips::HASH_LENGTH, |ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.processing.hash_length)
|
egui::DragValue::new(&mut config.processing.hash_length)
|
||||||
.range(512..=1_048_576),
|
.range(512..=1_048_576),
|
||||||
);
|
)
|
||||||
ui.end_row();
|
});
|
||||||
|
|
||||||
ui.label("Max stored text (bytes)");
|
tip_row(
|
||||||
|
ui,
|
||||||
|
"Max stored text (bytes)",
|
||||||
|
&tips::MAX_STORED_TEXT,
|
||||||
|
|ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.processing.maximum_text_size)
|
egui::DragValue::new(&mut config.processing.maximum_text_size)
|
||||||
.range(1024..=16_777_216),
|
.range(1024..=16_777_216),
|
||||||
|
)
|
||||||
|
},
|
||||||
);
|
);
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Max text file size (bytes)");
|
tip_row(
|
||||||
|
ui,
|
||||||
|
"Max text file size (bytes)",
|
||||||
|
&tips::MAX_TEXT_FILE_SIZE,
|
||||||
|
|ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.processing.maximum_text_file_size)
|
egui::DragValue::new(&mut config.processing.maximum_text_file_size)
|
||||||
.range(1024..=1_073_741_824),
|
.range(1024..=1_073_741_824),
|
||||||
|
)
|
||||||
|
},
|
||||||
);
|
);
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Batch size");
|
tip_row(ui, "Batch size", &tips::BATCH_SIZE, |ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.processing.batch_size).range(10..=100_000),
|
egui::DragValue::new(&mut config.processing.batch_size)
|
||||||
);
|
.range(10..=100_000),
|
||||||
ui.end_row();
|
)
|
||||||
|
});
|
||||||
|
|
||||||
ui.label("Max WAL size (bytes)");
|
tip_row(ui, "Max WAL size (bytes)", &tips::MAX_WAL_SIZE, |ui| {
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.processing.maximum_wal_size)
|
egui::DragValue::new(&mut config.processing.maximum_wal_size)
|
||||||
.range(0u64..=8_589_934_592u64),
|
.range(0u64..=8_589_934_592u64),
|
||||||
)
|
)
|
||||||
.on_hover_text(
|
});
|
||||||
"How large index.sqlite-wal may grow during a run before the \
|
|
||||||
indexer forces a checkpoint. 0 disables forced checkpoints; \
|
|
||||||
anything below 16 MiB is raised to it.",
|
|
||||||
);
|
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Store text for snippets");
|
tip_row(ui, "Store text for snippets", &tips::STORE_TEXT, |ui| {
|
||||||
ui.checkbox(&mut config.processing.store_text_for_snippets, "")
|
ui.checkbox(&mut config.processing.store_text_for_snippets, "")
|
||||||
.on_hover_text(
|
});
|
||||||
"Off: smaller index, but no previews, occurrence ranking, \
|
|
||||||
case verification, or fuzzy full-text search",
|
|
||||||
);
|
|
||||||
ui.end_row();
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Section::Search => {
|
Section::Search => {
|
||||||
egui::Grid::new("cfg-search").num_columns(2).show(ui, |ui| {
|
egui::Grid::new("cfg-search").num_columns(2).show(ui, |ui| {
|
||||||
ui.label("Fuzzy search ON by default");
|
tip_row(
|
||||||
ui.checkbox(&mut config.search.fuzzy_default, "");
|
ui,
|
||||||
ui.end_row();
|
"Fuzzy search ON by default",
|
||||||
|
&tips::FUZZY_DEFAULT,
|
||||||
|
|ui| ui.checkbox(&mut config.search.fuzzy_default, ""),
|
||||||
|
);
|
||||||
|
|
||||||
ui.label("Fuzzy edit distance");
|
tip_row(ui, "Fuzzy edit distance", &tips::FUZZY_EDITS, |ui| {
|
||||||
ui.vertical(|ui| {
|
|
||||||
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
|
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
|
||||||
.on_hover_text(
|
|
||||||
"Ceiling on the typo budget. The allowance grows with the \
|
|
||||||
search term, one edit per three characters, up to this \
|
|
||||||
value, so 2 means \"1 edit for short terms, 2 for longer \
|
|
||||||
ones\". 0 turns the fuzzy stages off.",
|
|
||||||
);
|
|
||||||
if let Some(warning) = config.search.fuzzy_edits_warning() {
|
|
||||||
ui.label(
|
|
||||||
egui::RichText::new(warning)
|
|
||||||
.small()
|
|
||||||
.color(crate::ui_util::ORANGE),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Display limit");
|
tip_row(ui, "Display limit", &tips::DISPLAY_LIMIT, |ui| {
|
||||||
ui.add(egui::DragValue::new(&mut config.search.display_limit).range(50..=100_000));
|
|
||||||
ui.end_row();
|
|
||||||
|
|
||||||
ui.label("Stream batch size");
|
|
||||||
ui.add(
|
ui.add(
|
||||||
egui::DragValue::new(&mut config.search.results_per_page).range(10..=10_000),
|
egui::DragValue::new(&mut config.search.display_limit).range(50..=100_000),
|
||||||
);
|
)
|
||||||
ui.end_row();
|
});
|
||||||
|
|
||||||
ui.label("Debounce (ms)");
|
tip_row(ui, "Stream batch size", &tips::RESULTS_PER_PAGE, |ui| {
|
||||||
ui.add(egui::DragValue::new(&mut config.search.debounce_ms).range(0..=2000));
|
ui.add(
|
||||||
ui.end_row();
|
egui::DragValue::new(&mut config.search.results_per_page)
|
||||||
|
.range(10..=10_000),
|
||||||
|
)
|
||||||
|
});
|
||||||
|
|
||||||
ui.label("Fuzzy max edits");
|
tip_row(ui, "Debounce (ms)", &tips::DEBOUNCE, |ui| {
|
||||||
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
|
ui.add(egui::DragValue::new(&mut config.search.debounce_ms).range(0..=2000))
|
||||||
.on_hover_text(
|
});
|
||||||
"Ceiling on fuzzy edit distance (the budget grows one \
|
|
||||||
edit per three characters of the term). 0 disables \
|
|
||||||
the fuzzy passes.",
|
|
||||||
);
|
|
||||||
ui.end_row();
|
|
||||||
});
|
});
|
||||||
// A warning that comes and goes as the value is edited would
|
// A warning that comes and goes as the value is edited would
|
||||||
// otherwise move every widget below it in the window; see
|
// otherwise move every widget below it in the window; see
|
||||||
|
|
@ -525,29 +521,151 @@ mod tests {
|
||||||
assert!(w.draft.is_none());
|
assert!(w.draft.is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Where `needle` was painted this frame, as the center of its galley —
|
use crate::test_ui::{painted_text, painted_text_center};
|
||||||
/// a click target that follows the layout instead of pinning it.
|
|
||||||
fn painted_text_center(out: &egui::FullOutput, needle: &str) -> Option<egui::Pos2> {
|
/// Every row of every section, with the tip it must show. `tip_row`
|
||||||
fn walk(shape: &egui::epaint::Shape, needle: &str, found: &mut Option<egui::Pos2>) {
|
/// makes a row without *a* tooltip impossible; this table is what makes
|
||||||
match shape {
|
/// a row with the *wrong* one impossible.
|
||||||
egui::epaint::Shape::Text(t) => {
|
const ROWS: &[(Section, &str, &tips::Tip)] = &[
|
||||||
if t.galley.text() == needle {
|
(
|
||||||
*found = Some(t.pos + t.galley.size() / 2.0);
|
Section::Indexing,
|
||||||
|
"Full reindex every",
|
||||||
|
&tips::REINDEX_INTERVAL,
|
||||||
|
),
|
||||||
|
(Section::Indexing, "Follow symlinks", &tips::FOLLOW_SYMLINKS),
|
||||||
|
(
|
||||||
|
Section::Indexing,
|
||||||
|
"Include hidden files",
|
||||||
|
&tips::INCLUDE_HIDDEN,
|
||||||
|
),
|
||||||
|
(Section::Processing, "Tokenizer", &tips::TOKENIZER),
|
||||||
|
(
|
||||||
|
Section::Processing,
|
||||||
|
"Hash sample size (bytes)",
|
||||||
|
&tips::HASH_LENGTH,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Section::Processing,
|
||||||
|
"Max stored text (bytes)",
|
||||||
|
&tips::MAX_STORED_TEXT,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Section::Processing,
|
||||||
|
"Max text file size (bytes)",
|
||||||
|
&tips::MAX_TEXT_FILE_SIZE,
|
||||||
|
),
|
||||||
|
(Section::Processing, "Batch size", &tips::BATCH_SIZE),
|
||||||
|
(
|
||||||
|
Section::Processing,
|
||||||
|
"Max WAL size (bytes)",
|
||||||
|
&tips::MAX_WAL_SIZE,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Section::Processing,
|
||||||
|
"Store text for snippets",
|
||||||
|
&tips::STORE_TEXT,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Section::Search,
|
||||||
|
"Fuzzy search ON by default",
|
||||||
|
&tips::FUZZY_DEFAULT,
|
||||||
|
),
|
||||||
|
(Section::Search, "Fuzzy edit distance", &tips::FUZZY_EDITS),
|
||||||
|
(Section::Search, "Display limit", &tips::DISPLAY_LIMIT),
|
||||||
|
(
|
||||||
|
Section::Search,
|
||||||
|
"Stream batch size",
|
||||||
|
&tips::RESULTS_PER_PAGE,
|
||||||
|
),
|
||||||
|
(Section::Search, "Debounce (ms)", &tips::DEBOUNCE),
|
||||||
|
];
|
||||||
|
|
||||||
|
/// Hovering a row's name paints that row's own explanation. Rendered
|
||||||
|
/// without the window's scroll area so nothing sits below the fold.
|
||||||
|
#[test]
|
||||||
|
fn every_row_shows_its_own_tip() {
|
||||||
|
for (section, label, tip) in ROWS {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
ctx.style_mut(|s| {
|
||||||
|
s.interaction.tooltip_delay = 0.0;
|
||||||
|
s.interaction.show_tooltips_only_when_still = false;
|
||||||
|
});
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
let mut run = |events: Vec<egui::Event>| {
|
||||||
|
let input = crate::test_ui::raw_input(egui::vec2(600.0, 800.0), events);
|
||||||
|
ctx.run(input, |ctx| {
|
||||||
|
egui::CentralPanel::default()
|
||||||
|
.show(ctx, |ui| config_editor_ui(ui, &mut cfg, *section));
|
||||||
|
})
|
||||||
|
};
|
||||||
|
|
||||||
|
let first = run(vec![]);
|
||||||
|
let pos = painted_text_center(&first, label)
|
||||||
|
.unwrap_or_else(|| panic!("{label} was not painted"));
|
||||||
|
|
||||||
|
// Enough of the body to be unique, and short enough to survive
|
||||||
|
// an edit to the sentence it starts.
|
||||||
|
let opening: String = tip.body.chars().take(40).collect();
|
||||||
|
let mut out = run(vec![egui::Event::PointerMoved(pos)]);
|
||||||
|
let mut found = false;
|
||||||
|
for _ in 0..3 {
|
||||||
|
// The tooltip is an area of its own, so it can land a frame
|
||||||
|
// late.
|
||||||
|
if painted_text(&out).join("\n").contains(&opening) {
|
||||||
|
found = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
out = run(vec![]);
|
||||||
|
}
|
||||||
|
assert!(found, "hovering {label:?} did not show {:?}", tip.title);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
egui::epaint::Shape::Vec(v) => {
|
|
||||||
for s in v {
|
/// Hovering a setting's *name* explains it, not just its control: the
|
||||||
walk(s, needle, found);
|
/// label is the larger target and the one a reader's eye is already on.
|
||||||
|
/// Checks the wiring, so the tooltip timing is turned off.
|
||||||
|
#[test]
|
||||||
|
fn hovering_a_setting_label_explains_it() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
ctx.style_mut(|s| {
|
||||||
|
s.interaction.tooltip_delay = 0.0;
|
||||||
|
s.interaction.show_tooltips_only_when_still = false;
|
||||||
|
});
|
||||||
|
let cfg = Config::default();
|
||||||
|
let mut w = OptionsWindow::new();
|
||||||
|
w.open_with(&cfg);
|
||||||
|
|
||||||
|
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
|
||||||
|
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
|
||||||
|
ctx.run(input, |ctx| {
|
||||||
|
w.ui(ctx, &cfg);
|
||||||
|
})
|
||||||
|
};
|
||||||
|
|
||||||
|
// The window spends its first frames sizing itself and painting
|
||||||
|
// nothing; run until the label is on screen.
|
||||||
|
let mut target = None;
|
||||||
|
for _ in 0..5 {
|
||||||
|
let full = run(&mut w, vec![]);
|
||||||
|
target = painted_text_center(&full, "Tokenizer");
|
||||||
|
if target.is_some() {
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
_ => {}
|
let target = target.expect("the Tokenizer label was not painted");
|
||||||
|
|
||||||
|
// The tooltip is an area of its own, so it can land a frame late.
|
||||||
|
let mut out = run(&mut w, vec![egui::Event::PointerMoved(target)]);
|
||||||
|
for _ in 0..3 {
|
||||||
|
let painted = painted_text(&out).join("\n");
|
||||||
|
if painted.contains(crate::tips::TOKENIZER.title)
|
||||||
|
&& painted.contains("cut up so that it can be")
|
||||||
|
{
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
|
out = run(&mut w, vec![]);
|
||||||
}
|
}
|
||||||
let mut found = None;
|
panic!("no tooltip appeared over the Tokenizer label");
|
||||||
for clipped in &out.shapes {
|
|
||||||
walk(&clipped.shape, needle, &mut found);
|
|
||||||
}
|
|
||||||
found
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One real frame of the window in a headless context: it renders, and
|
/// One real frame of the window in a headless context: it renders, and
|
||||||
|
|
@ -561,14 +679,7 @@ mod tests {
|
||||||
w.draft.as_mut().unwrap().search.debounce_ms += 100;
|
w.draft.as_mut().unwrap().search.debounce_ms += 100;
|
||||||
|
|
||||||
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
|
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(1000.0, 900.0),
|
|
||||||
)),
|
|
||||||
events,
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
let mut out = OptionsOutput::default();
|
let mut out = OptionsOutput::default();
|
||||||
let full = ctx.run(input, |ctx| out = w.ui(ctx, &cfg));
|
let full = ctx.run(input, |ctx| out = w.ui(ctx, &cfg));
|
||||||
(out, full)
|
(out, full)
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,7 @@ use quicksearch_core::snippet::Snippet;
|
||||||
|
|
||||||
use crate::format::{fmt_elapsed, fmt_mtime, human_size};
|
use crate::format::{fmt_elapsed, fmt_mtime, human_size};
|
||||||
use crate::platform;
|
use crate::platform;
|
||||||
|
use crate::ui_util::middle_elide;
|
||||||
|
|
||||||
/// Width of the query strip's status slot, in points. Wide enough for the
|
/// Width of the query strip's status slot, in points. Wide enough for the
|
||||||
/// longest query time `fmt_elapsed` produces, and held whether the slot is
|
/// longest query time `fmt_elapsed` produces, and held whether the slot is
|
||||||
|
|
@ -496,7 +497,8 @@ impl SearchTab {
|
||||||
let preview_height = if preview_snippet.is_some() { 44.0 } else { 0.0 };
|
let preview_height = if preview_snippet.is_some() { 44.0 } else { 0.0 };
|
||||||
let table_height = (ui.available_height() - preview_height).max(60.0);
|
let table_height = (ui.available_height() - preview_height).max(60.0);
|
||||||
|
|
||||||
let text_height = egui::TextStyle::Body.resolve(ui.style()).size + 4.0;
|
let body_font = egui::TextStyle::Body.resolve(ui.style());
|
||||||
|
let text_height = body_font.size + 4.0;
|
||||||
let mut open_ignore_dialog: Option<usize> = None;
|
let mut open_ignore_dialog: Option<usize> = None;
|
||||||
let mut hovered_now: Option<usize> = None;
|
let mut hovered_now: Option<usize> = None;
|
||||||
// Moved out of `self` rather than cloned. The row closure needs `&mut
|
// Moved out of `self` rather than cloned. The row closure needs `&mut
|
||||||
|
|
@ -564,8 +566,37 @@ impl SearchTab {
|
||||||
cell_responses.push(ui.label(&hit.name));
|
cell_responses.push(ui.label(&hit.name));
|
||||||
});
|
});
|
||||||
row.col(|ui| {
|
row.col(|ui| {
|
||||||
cell_responses
|
// Center-elided: egui's own truncation keeps
|
||||||
.push(ui.label(egui::RichText::new(&hit.path).weak()));
|
// the head and drops the deepest directories
|
||||||
|
// — the half that actually says where the
|
||||||
|
// file lives. A sizing pass hands out cell
|
||||||
|
// rects that are not final yet, so nothing is
|
||||||
|
// measured against one.
|
||||||
|
let shown = if ui.is_sizing_pass() {
|
||||||
|
std::borrow::Cow::Borrowed(hit.path.as_str())
|
||||||
|
} else {
|
||||||
|
middle_elide(
|
||||||
|
ui,
|
||||||
|
&hit.path,
|
||||||
|
// A point of slack, so a rounding
|
||||||
|
// disagreement with egui's layout
|
||||||
|
// cannot cost a second ellipsis.
|
||||||
|
ui.available_width() - 1.0,
|
||||||
|
&body_font,
|
||||||
|
)
|
||||||
|
};
|
||||||
|
let elided = matches!(shown, std::borrow::Cow::Owned(_));
|
||||||
|
// egui offers a full-text tooltip only when
|
||||||
|
// *it* elided the galley, and it would be
|
||||||
|
// handed the string already shortened here.
|
||||||
|
let mut response = ui.add(
|
||||||
|
egui::Label::new(egui::RichText::new(shown.as_ref()).weak())
|
||||||
|
.show_tooltip_when_elided(false),
|
||||||
|
);
|
||||||
|
if elided {
|
||||||
|
response = response.on_hover_text(&hit.path);
|
||||||
|
}
|
||||||
|
cell_responses.push(response);
|
||||||
});
|
});
|
||||||
if self.has_snippets {
|
if self.has_snippets {
|
||||||
// Borrowed, not cloned: a snippet window runs
|
// Borrowed, not cloned: a snippet window runs
|
||||||
|
|
@ -643,7 +674,7 @@ impl SearchTab {
|
||||||
|
|
||||||
let mut response = row.response();
|
let mut response = row.response();
|
||||||
for r in cell_responses {
|
for r in cell_responses {
|
||||||
response = response | r;
|
response |= r;
|
||||||
}
|
}
|
||||||
if response.contains_pointer() {
|
if response.contains_pointer() {
|
||||||
hovered_now = Some(display_ix);
|
hovered_now = Some(display_ix);
|
||||||
|
|
@ -1098,10 +1129,7 @@ fn rank_tier_color(stage: u8) -> egui::Color32 {
|
||||||
/// Timestamp color: fresh files get a green tint that fades into the weak
|
/// Timestamp color: fresh files get a green tint that fades into the weak
|
||||||
/// text color over ~2 years on a log scale.
|
/// text color over ~2 years on a log scale.
|
||||||
fn recency_color(ui: &egui::Ui, mtime: i64) -> egui::Color32 {
|
fn recency_color(ui: &egui::Ui, mtime: i64) -> egui::Color32 {
|
||||||
let now = std::time::SystemTime::now()
|
let now = quicksearch_core::log::now_unix() as i64;
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_secs() as i64)
|
|
||||||
.unwrap_or(0);
|
|
||||||
let age_hours = ((now - mtime).max(0) as f32 / 3600.0).max(1.0);
|
let age_hours = ((now - mtime).max(0) as f32 / 3600.0).max(1.0);
|
||||||
const HORIZON_HOURS: f32 = 24.0 * 365.0 * 2.0;
|
const HORIZON_HOURS: f32 = 24.0 * 365.0 * 2.0;
|
||||||
let t = (age_hours.ln() / HORIZON_HOURS.ln()).clamp(0.0, 1.0);
|
let t = (age_hours.ln() / HORIZON_HOURS.ln()).clamp(0.0, 1.0);
|
||||||
|
|
@ -1144,14 +1172,7 @@ mod tests {
|
||||||
tab: &mut SearchTab,
|
tab: &mut SearchTab,
|
||||||
events: Vec<egui::Event>,
|
events: Vec<egui::Event>,
|
||||||
) -> egui::FullOutput {
|
) -> egui::FullOutput {
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 700.0), events);
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(1000.0, 700.0),
|
|
||||||
)),
|
|
||||||
events,
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
ctx.run(input, |ctx| {
|
ctx.run(input, |ctx| {
|
||||||
egui::CentralPanel::default().show(ctx, |ui| {
|
egui::CentralPanel::default().show(ctx, |ui| {
|
||||||
tab.ui(ui);
|
tab.ui(ui);
|
||||||
|
|
@ -1175,6 +1196,79 @@ mod tests {
|
||||||
panic!("never landed on row {row}'s label text");
|
panic!("never landed on row {row}'s label text");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
use crate::test_ui::painted_text;
|
||||||
|
|
||||||
|
/// Far too long for the Path column at the test's 1000pt screen width.
|
||||||
|
fn deep_path() -> String {
|
||||||
|
concat!(
|
||||||
|
"/media/shared/QuickSearch/crates/quicksearch-gui/src/",
|
||||||
|
"deeply/nested/under/several/more/directories/alpha_widget_0.txt"
|
||||||
|
)
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The Path column drops out of the *middle*, so the volume the file
|
||||||
|
/// sits on and the directories right above it both stay on screen.
|
||||||
|
/// egui's own truncation would keep the head and throw the tail away —
|
||||||
|
/// and the tail is the half that says where the file lives.
|
||||||
|
#[test]
|
||||||
|
fn long_paths_elide_from_the_middle_of_the_path_column() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut tab = tab_with_results(1);
|
||||||
|
let path = deep_path();
|
||||||
|
tab.results[0].path = path.clone();
|
||||||
|
|
||||||
|
let painted = painted_text(&run_frame(&ctx, &mut tab, vec![]));
|
||||||
|
let cell = painted
|
||||||
|
.iter()
|
||||||
|
.find(|t| t.starts_with("/media") && t.contains('…'))
|
||||||
|
.unwrap_or_else(|| panic!("no elided path cell among {painted:?}"));
|
||||||
|
|
||||||
|
assert!(!painted.contains(&path), "painted in full");
|
||||||
|
assert_eq!(cell.matches('…').count(), 1, "elided twice: {cell}");
|
||||||
|
let (head, tail) = cell.split_once('…').expect("one ellipsis");
|
||||||
|
assert!(path.starts_with(head), "{cell}");
|
||||||
|
assert!(path.ends_with(tail), "{cell}");
|
||||||
|
assert!(
|
||||||
|
tail.ends_with("alpha_widget_0.txt"),
|
||||||
|
"the deep end survives: {cell}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The whole path stays reachable on hover. egui hands out that tooltip
|
||||||
|
/// for free only while *it* did the eliding, so text shortened ahead of
|
||||||
|
/// time has to bring its own — and it is easy to lose silently.
|
||||||
|
#[test]
|
||||||
|
fn an_elided_path_still_shows_the_whole_thing_on_hover() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
// Testing that the tooltip is wired up, not egui's hover timing.
|
||||||
|
ctx.style_mut(|s| {
|
||||||
|
s.interaction.tooltip_delay = 0.0;
|
||||||
|
s.interaction.show_tooltips_only_when_still = false;
|
||||||
|
});
|
||||||
|
let mut tab = tab_with_results(1);
|
||||||
|
let path = deep_path();
|
||||||
|
tab.results[0].path = path.clone();
|
||||||
|
|
||||||
|
run_frame(&ctx, &mut tab, vec![]); // settle the table's layout
|
||||||
|
for y in 40..250 {
|
||||||
|
// x lands in the Path column, past the 220pt Name column.
|
||||||
|
let pos = egui::pos2(300.0, y as f32);
|
||||||
|
let mut out = run_frame(&ctx, &mut tab, vec![egui::Event::PointerMoved(pos)]);
|
||||||
|
if tab.hovered_row != Some(0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// The tooltip is its own area, so it may land a frame behind.
|
||||||
|
for _ in 0..3 {
|
||||||
|
if painted_text(&out).contains(&path) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
out = run_frame(&ctx, &mut tab, vec![]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
panic!("the full path never appeared on hover");
|
||||||
|
}
|
||||||
|
|
||||||
fn click(pos: egui::Pos2, button: egui::PointerButton) -> Vec<egui::Event> {
|
fn click(pos: egui::Pos2, button: egui::PointerButton) -> Vec<egui::Event> {
|
||||||
[true, false]
|
[true, false]
|
||||||
.into_iter()
|
.into_iter()
|
||||||
|
|
|
||||||
85
crates/quicksearch-gui/src/test_ui.rs
Normal file
85
crates/quicksearch-gui/src/test_ui.rs
Normal file
|
|
@ -0,0 +1,85 @@
|
||||||
|
//! Driving egui headlessly from tests: build an input frame, synthesize a
|
||||||
|
//! click, read back what was painted.
|
||||||
|
//!
|
||||||
|
//! Every tab's test module wants the same three things, and each had grown its
|
||||||
|
//! own copy. The per-tab `frame(…)` wrappers stay where they are — each drives
|
||||||
|
//! a different widget with a different return type — but they are built on
|
||||||
|
//! these.
|
||||||
|
|
||||||
|
/// A frame of input at `size`, carrying `events`.
|
||||||
|
///
|
||||||
|
/// The viewport size is explicit rather than defaulted because it is load
|
||||||
|
/// bearing: a modal is centred in it, so tests that locate a button by where
|
||||||
|
/// it was painted get different coordinates from a different size, and a panel
|
||||||
|
/// that does not fit is simply not painted at all.
|
||||||
|
pub fn raw_input(size: egui::Vec2, events: Vec<egui::Event>) -> egui::RawInput {
|
||||||
|
egui::RawInput {
|
||||||
|
screen_rect: Some(egui::Rect::from_min_size(egui::Pos2::ZERO, size)),
|
||||||
|
events,
|
||||||
|
..Default::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A primary-button press and release at `pos`, preceded by the pointer moving
|
||||||
|
/// there.
|
||||||
|
///
|
||||||
|
/// The move is not decoration: egui hit-tests against the pointer's *current*
|
||||||
|
/// position, so a press delivered without one lands wherever the pointer was
|
||||||
|
/// last frame — which for the first frame of a fresh context is nowhere.
|
||||||
|
pub fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
|
||||||
|
let button = |pressed| egui::Event::PointerButton {
|
||||||
|
pos,
|
||||||
|
button: egui::PointerButton::Primary,
|
||||||
|
pressed,
|
||||||
|
modifiers: egui::Modifiers::NONE,
|
||||||
|
};
|
||||||
|
vec![
|
||||||
|
egui::Event::PointerMoved(pos),
|
||||||
|
button(true),
|
||||||
|
button(false),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every text galley painted this frame, each with the rectangle it occupies.
|
||||||
|
///
|
||||||
|
/// Labels carry no widget id worth recording, so reading the shapes back is
|
||||||
|
/// the only way to check the text a user actually sees — and the only way to
|
||||||
|
/// find a click target that follows the layout instead of pinning it.
|
||||||
|
pub fn painted(out: &egui::FullOutput) -> Vec<(String, egui::Rect)> {
|
||||||
|
fn walk(shape: &egui::epaint::Shape, into: &mut Vec<(String, egui::Rect)>) {
|
||||||
|
match shape {
|
||||||
|
egui::epaint::Shape::Text(t) => into.push((
|
||||||
|
t.galley.text().to_string(),
|
||||||
|
egui::Rect::from_min_size(t.pos, t.galley.size()),
|
||||||
|
)),
|
||||||
|
egui::epaint::Shape::Vec(shapes) => {
|
||||||
|
for s in shapes {
|
||||||
|
walk(s, into);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut out_text = Vec::new();
|
||||||
|
for clipped in &out.shapes {
|
||||||
|
walk(&clipped.shape, &mut out_text);
|
||||||
|
}
|
||||||
|
out_text
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every string painted this frame, in paint order.
|
||||||
|
pub fn painted_text(out: &egui::FullOutput) -> Vec<String> {
|
||||||
|
painted(out).into_iter().map(|(text, _)| text).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The centre of `needle`'s galley, as a click target.
|
||||||
|
///
|
||||||
|
/// The *last* match wins, so a string painted both behind a modal and on it
|
||||||
|
/// resolves to the one on top — which is the one a click would reach.
|
||||||
|
pub fn painted_text_center(out: &egui::FullOutput, needle: &str) -> Option<egui::Pos2> {
|
||||||
|
painted(out)
|
||||||
|
.iter()
|
||||||
|
.rev()
|
||||||
|
.find(|(text, _)| text == needle)
|
||||||
|
.map(|(_, rect)| rect.center())
|
||||||
|
}
|
||||||
758
crates/quicksearch-gui/src/tips.rs
Normal file
758
crates/quicksearch-gui/src/tips.rs
Normal file
|
|
@ -0,0 +1,758 @@
|
||||||
|
//! Plain-language tooltips for the configuration controls.
|
||||||
|
//!
|
||||||
|
//! Every setting in the Options window, and every configuration control on
|
||||||
|
//! the Manage Index tab, explains itself on hover: what it does in ordinary
|
||||||
|
//! words first, then what changing it costs, then concrete values and when
|
||||||
|
//! someone would pick them. The text lives here rather than inline so the
|
||||||
|
//! layout code stays readable and the wording can be checked on its own.
|
||||||
|
//!
|
||||||
|
//! Written for someone who does not know what a tokenizer or a write-ahead
|
||||||
|
//! log is, and should not have to.
|
||||||
|
|
||||||
|
use crate::ui_util::ORANGE;
|
||||||
|
|
||||||
|
/// How wide a tooltip may get. Matches `manage_tab::db_size_tooltip`: wide
|
||||||
|
/// enough that a sentence is not shredded into three lines, narrow enough
|
||||||
|
/// that the eye finds the next line.
|
||||||
|
const TIP_WIDTH: f32 = 420.0;
|
||||||
|
|
||||||
|
/// One control's tooltip.
|
||||||
|
pub struct Tip {
|
||||||
|
/// The setting's name, in bold at the top.
|
||||||
|
pub title: &'static str,
|
||||||
|
/// The explanation. One or more paragraphs separated by `"\n\n"`.
|
||||||
|
pub body: &'static str,
|
||||||
|
/// Concrete values and when to choose them, rendered small underneath.
|
||||||
|
/// Empty where a setting has nothing to weigh up.
|
||||||
|
pub examples: &'static [&'static str],
|
||||||
|
/// A consequence worth seeing before the click, rendered small and
|
||||||
|
/// orange. Reserved for rebuilds and deletions, so that it keeps
|
||||||
|
/// meaning something.
|
||||||
|
pub caution: Option<&'static str>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Tip {
|
||||||
|
/// Render into a hover popup.
|
||||||
|
pub fn show(&self, ui: &mut egui::Ui) {
|
||||||
|
ui.set_max_width(TIP_WIDTH);
|
||||||
|
ui.strong(self.title);
|
||||||
|
ui.label(self.body);
|
||||||
|
if !self.examples.is_empty() {
|
||||||
|
ui.add_space(4.0);
|
||||||
|
// One example reads as a sentence; several read as a list.
|
||||||
|
let single = self.examples.len() == 1;
|
||||||
|
for example in self.examples {
|
||||||
|
let line = if single {
|
||||||
|
format!("Example: {}", example)
|
||||||
|
} else {
|
||||||
|
format!("• {}", example)
|
||||||
|
};
|
||||||
|
ui.label(egui::RichText::new(line).small());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(caution) = self.caution {
|
||||||
|
ui.add_space(4.0);
|
||||||
|
ui.label(egui::RichText::new(caution).small().color(ORANGE));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Attach a [`Tip`] to any widget.
|
||||||
|
pub trait Tipped {
|
||||||
|
/// Show `tip` on hover, whether or not the widget is enabled: a greyed
|
||||||
|
/// out button is exactly when someone wants to know what it would do.
|
||||||
|
fn tip(self, tip: &'static Tip) -> Self;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Tipped for egui::Response {
|
||||||
|
fn tip(self, tip: &'static Tip) -> egui::Response {
|
||||||
|
self.on_hover_ui(|ui| tip.show(ui))
|
||||||
|
.on_disabled_hover_ui(|ui| tip.show(ui))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One row of a two-column settings grid: the label and the control share a
|
||||||
|
/// tooltip, so hovering the name works as well as hovering the widget.
|
||||||
|
///
|
||||||
|
/// Taking the tip by value rather than as an option is deliberate: a row
|
||||||
|
/// added later cannot forget to explain itself.
|
||||||
|
pub fn tip_row(
|
||||||
|
ui: &mut egui::Ui,
|
||||||
|
label: &str,
|
||||||
|
tip: &'static Tip,
|
||||||
|
widget: impl FnOnce(&mut egui::Ui) -> egui::Response,
|
||||||
|
) {
|
||||||
|
ui.label(label).tip(tip);
|
||||||
|
widget(ui).tip(tip);
|
||||||
|
ui.end_row();
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Options: Paths ------------------------------------------------------
|
||||||
|
|
||||||
|
pub static DATABASE_PATH: Tip = Tip {
|
||||||
|
title: "Database file",
|
||||||
|
body: "Where QuickSearch keeps its index: a single file holding the \
|
||||||
|
names, locations and text of everything it has indexed. It grows \
|
||||||
|
with the number of files indexed, so it wants a drive with room \
|
||||||
|
to spare.\n\n\
|
||||||
|
Pointing this somewhere else switches to the index at the new \
|
||||||
|
location, building a fresh one there if nothing exists yet. The \
|
||||||
|
old file stays on disk until you delete it. Keep it off a folder \
|
||||||
|
that syncs to the cloud, such as OneDrive or Dropbox: it is far \
|
||||||
|
too large and too busy to synchronise.",
|
||||||
|
examples: &["a path on a second, larger drive when the system drive is tight on space."],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Options: Indexing ---------------------------------------------------
|
||||||
|
|
||||||
|
pub static REINDEX_INTERVAL: Tip = Tip {
|
||||||
|
title: "Full reindex every",
|
||||||
|
body: "Indexed folders are watched, so ordinary changes appear within \
|
||||||
|
seconds. This is the safety net behind that: a full sweep that \
|
||||||
|
picks up anything the watching missed, such as changes made while \
|
||||||
|
QuickSearch was closed, or on a drive that does not report them.\n\n\
|
||||||
|
A sweep costs some disk activity for a few minutes, and searching \
|
||||||
|
keeps working throughout.",
|
||||||
|
examples: &[
|
||||||
|
"1440, once a day, suits an ordinary laptop.",
|
||||||
|
"60 when you work on a network share, or a drive other machines write to.",
|
||||||
|
"10080, once a week, for an archive that barely changes.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static FOLLOW_SYMLINKS: Tip = Tip {
|
||||||
|
title: "Follow symlinks",
|
||||||
|
body: "Symlinks are entries that stand in for a file or folder living \
|
||||||
|
somewhere else. Off, QuickSearch steps over them, so nothing is \
|
||||||
|
indexed twice under two names. On, it looks through them and \
|
||||||
|
indexes what they point at, stored under the real location rather \
|
||||||
|
than the link's.\n\n\
|
||||||
|
Turning this off removes the entries that are no longer in scope; \
|
||||||
|
turning it on indexes whatever it now reaches. Neither rebuilds \
|
||||||
|
the index.",
|
||||||
|
examples: &[
|
||||||
|
"on when a folder you search lives outside your indexed folders and is reached \
|
||||||
|
through a link.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static INCLUDE_HIDDEN: Tip = Tip {
|
||||||
|
title: "Include hidden files",
|
||||||
|
body: "Files and folders the system keeps out of sight: names beginning \
|
||||||
|
with a dot on Linux and macOS, plus anything carrying the Hidden \
|
||||||
|
attribute on Windows, such as AppData and $RECYCLE.BIN. They are \
|
||||||
|
mostly program settings and caches, so leaving them out keeps the \
|
||||||
|
index smaller and the results cleaner.\n\n\
|
||||||
|
Folders marked System but not Hidden are indexed either way. Cloud \
|
||||||
|
sync folders and folders given a custom icon carry that mark only \
|
||||||
|
to get the icon, and are ordinary folders otherwise.",
|
||||||
|
examples: &["on when you want to find configuration files such as .bashrc or .gitconfig."],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Options: Processing -------------------------------------------------
|
||||||
|
|
||||||
|
pub static TOKENIZER: Tip = Tip {
|
||||||
|
title: "Tokenizer",
|
||||||
|
body: "How the text inside your files is cut up so that it can be \
|
||||||
|
searched.\n\n\
|
||||||
|
trigram indexes every run of three characters, so a search for \
|
||||||
|
\"port\" also finds \"airport\", and it works for languages that \
|
||||||
|
do not put spaces between words. The other two index whole words \
|
||||||
|
instead: the index is smaller and faster, but a search matches \
|
||||||
|
only from the start of a word.",
|
||||||
|
examples: &[
|
||||||
|
"trigram, the default, for finding a fragment anywhere inside a word.",
|
||||||
|
"unicode61 when you only ever search whole words and want the smallest, \
|
||||||
|
fastest index.",
|
||||||
|
"porter to also match English word endings, so \"running\" finds \"run\".",
|
||||||
|
],
|
||||||
|
caution: Some("Changing this builds the whole index again from scratch."),
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static HASH_LENGTH: Tip = Tip {
|
||||||
|
title: "Hash sample size",
|
||||||
|
body: "How much of the start of each file QuickSearch reads in order to \
|
||||||
|
recognise it. Those first bytes give the file its fingerprint, \
|
||||||
|
which is how the Duplicates tab knows two files are identical; \
|
||||||
|
they also say what kind of file it is, and for a small text file \
|
||||||
|
they are the whole text.\n\n\
|
||||||
|
Reading more is more reliable and slower, particularly over a \
|
||||||
|
network.",
|
||||||
|
examples: &[
|
||||||
|
"8192, the default, is right for almost everyone.",
|
||||||
|
"higher when the Duplicates tab groups files that are not really identical, \
|
||||||
|
which happens with disk images and other formats that begin with a lot of \
|
||||||
|
empty space.",
|
||||||
|
],
|
||||||
|
caution: Some("Changing this builds the whole index again from scratch."),
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static MAX_STORED_TEXT: Tip = Tip {
|
||||||
|
title: "Max stored text",
|
||||||
|
body: "How much text QuickSearch keeps out of any one file. Text past \
|
||||||
|
this point is not stored, so a search will not find a word that \
|
||||||
|
appears only deep inside a very long document. File names, sizes \
|
||||||
|
and dates are unaffected.\n\n\
|
||||||
|
Along with the two settings below it, this is one of the largest \
|
||||||
|
influences on how big the index becomes.",
|
||||||
|
examples: &[
|
||||||
|
"262144, 256 KB, covers the whole of most documents.",
|
||||||
|
"65536, 64 KB, to shrink the index when you mostly search the opening pages.",
|
||||||
|
"higher when you search long books, transcripts or logs and expect to find \
|
||||||
|
words near the end.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static MAX_TEXT_FILE_SIZE: Tip = Tip {
|
||||||
|
title: "Max text file size",
|
||||||
|
body: "Files larger than this are indexed by name only: QuickSearch does \
|
||||||
|
not open them to read the text inside. It keeps one stray huge \
|
||||||
|
file from holding up an indexing run.\n\n\
|
||||||
|
Those files still appear in results, found by their name, size or \
|
||||||
|
date.",
|
||||||
|
examples: &[
|
||||||
|
"2097152, 2 MB, skips very few ordinary documents.",
|
||||||
|
"52428800, 50 MB, when you search inside large log files or scanned PDFs.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static BATCH_SIZE: Tip = Tip {
|
||||||
|
title: "Batch size",
|
||||||
|
body: "How many files QuickSearch handles per write to the index while \
|
||||||
|
indexing. Larger batches mean fewer, bigger writes, which is a \
|
||||||
|
little faster and uses more memory.\n\n\
|
||||||
|
A speed setting only: it changes nothing about what you can find, \
|
||||||
|
and most people never need to touch it.",
|
||||||
|
examples: &[
|
||||||
|
"500, the default, balances speed against memory.",
|
||||||
|
"lower, around 50, on a machine with very little memory to spare.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static MAX_WAL_SIZE: Tip = Tip {
|
||||||
|
title: "Max WAL size",
|
||||||
|
body: "While indexing, changes are written to a companion file beside \
|
||||||
|
the index and folded in afterwards. That normally happens by \
|
||||||
|
itself, but during a long run with searches going on at the same \
|
||||||
|
time the companion file keeps growing, sometimes past the size of \
|
||||||
|
the index. This is the point at which QuickSearch pauses and folds \
|
||||||
|
it in regardless.\n\n\
|
||||||
|
Another speed setting; the default suits most machines.",
|
||||||
|
examples: &[
|
||||||
|
"536870912, 512 MB, is the default.",
|
||||||
|
"67108864, 64 MB, when disk space is tight.",
|
||||||
|
"0 to never force it and let the database decide. Any other value below 16 MB \
|
||||||
|
is treated as 16 MB.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static STORE_TEXT: Tip = Tip {
|
||||||
|
title: "Store text for snippets",
|
||||||
|
body: "Keeps the text QuickSearch reads out of your files, rather than \
|
||||||
|
only the index needed to search it. It is what makes the preview \
|
||||||
|
line underneath a result possible.\n\n\
|
||||||
|
Off, searching inside files still works, but there are no \
|
||||||
|
previews, no ranking by how often a word appears, no telling \
|
||||||
|
Report from report, and no allowance for typos inside file \
|
||||||
|
contents. The index shrinks considerably.\n\n\
|
||||||
|
Turning it off discards the stored text at once; turning it back \
|
||||||
|
on reads your files again.",
|
||||||
|
examples: &[
|
||||||
|
"off when the index has grown larger than you want and you can do without previews.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Options: Search -----------------------------------------------------
|
||||||
|
|
||||||
|
pub static FUZZY_DEFAULT: Tip = Tip {
|
||||||
|
title: "Fuzzy search ON by default",
|
||||||
|
body: "Whether the Fuzzy box on the Search tab starts ticked each time \
|
||||||
|
QuickSearch opens. Fuzzy search also finds matches with typos in \
|
||||||
|
them, at some cost in speed. Either way you can tick and untick \
|
||||||
|
it whenever you like.",
|
||||||
|
examples: &["on when you often look for names you are not sure how to spell."],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static FUZZY_EDITS: Tip = Tip {
|
||||||
|
title: "Fuzzy edit distance",
|
||||||
|
body: "How far a word may sit from what you typed and still count as a \
|
||||||
|
match while Fuzzy is on. One edit is one letter added, removed or \
|
||||||
|
changed, so \"reciept\" is one edit away from \"receipt\".\n\n\
|
||||||
|
The allowance grows with the length of what you type, one edit per \
|
||||||
|
three characters, up to the value set here. Short searches stay \
|
||||||
|
strict, so that three letters do not match half the index.",
|
||||||
|
examples: &[
|
||||||
|
"2, the default, allows one edit for short words and two for longer ones.",
|
||||||
|
"0 turns typo matching off altogether, even with the Fuzzy box ticked.",
|
||||||
|
"3 or more is allowed, but searches get slower and pull in a lot of \
|
||||||
|
unrelated files.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static DISPLAY_LIMIT: Tip = Tip {
|
||||||
|
title: "Display limit",
|
||||||
|
body: "The most results one search will gather and show. A search \
|
||||||
|
matching thousands of files stops here, which keeps the list quick \
|
||||||
|
to scroll and cheap to hold in memory.\n\n\
|
||||||
|
The best matches come first, so a lower limit rarely hides what \
|
||||||
|
you were looking for.",
|
||||||
|
examples: &[
|
||||||
|
"1000, the default, is more than anyone scrolls through.",
|
||||||
|
"higher when you use QuickSearch to list every file of a kind, such as \
|
||||||
|
type:Image, and want them all at once.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static RESULTS_PER_PAGE: Tip = Tip {
|
||||||
|
title: "Stream batch size",
|
||||||
|
body: "Results arrive in batches while a search runs, and this is how \
|
||||||
|
many are in each one. Smaller batches put the first results on \
|
||||||
|
screen sooner and update the list more often; larger ones do less \
|
||||||
|
work in total.\n\n\
|
||||||
|
This is not a page size: scrolling the results does not go back \
|
||||||
|
for more.",
|
||||||
|
examples: &[
|
||||||
|
"100, the default, feels immediate on most machines.",
|
||||||
|
"25 when the first results are slow to appear on a large index.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static DEBOUNCE: Tip = Tip {
|
||||||
|
title: "Debounce",
|
||||||
|
body: "How long QuickSearch waits after your last keystroke before it \
|
||||||
|
searches, so that typing a word does not fire off a search for \
|
||||||
|
every letter in it. 1000 milliseconds is one second.",
|
||||||
|
examples: &[
|
||||||
|
"150, the default, keeps up with ordinary typing.",
|
||||||
|
"0 to chase every keystroke on a fast machine with a modest index.",
|
||||||
|
"300 or more when the results flicker or stutter as you type.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Options: Interface --------------------------------------------------
|
||||||
|
|
||||||
|
pub static UI_SCALE: Tip = Tip {
|
||||||
|
title: "UI scale",
|
||||||
|
body: "Zooms the whole window: text, spacing and controls together. 1.00 \
|
||||||
|
is the ordinary size for your screen.\n\n\
|
||||||
|
Ctrl with + or - changes it for the moment without saving, and \
|
||||||
|
Ctrl 0 puts it back. This slider is the size QuickSearch starts \
|
||||||
|
at.",
|
||||||
|
examples: &[
|
||||||
|
"1.40 or more on a high resolution screen where the text looks small.",
|
||||||
|
"0.80 to fit more results on screen at once.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Options: Security ---------------------------------------------------
|
||||||
|
|
||||||
|
pub static ENABLE_PASSWORD: Tip = Tip {
|
||||||
|
title: "Enable password protection",
|
||||||
|
body: "Encrypts the index with a password of your choosing. The index \
|
||||||
|
holds the names and the text of your files, so anyone who can \
|
||||||
|
read that file can read those; encrypting it means they cannot.\n\n\
|
||||||
|
QuickSearch then asks for the password each time it starts, unless \
|
||||||
|
you let it remember.",
|
||||||
|
examples: &[],
|
||||||
|
caution: Some(
|
||||||
|
"Turning protection on deletes the index and builds it again. Your files are \
|
||||||
|
not touched.",
|
||||||
|
),
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static CHANGE_PASSWORD: Tip = Tip {
|
||||||
|
title: "Change password",
|
||||||
|
body: "Replaces the password the index is encrypted with. You are asked \
|
||||||
|
for a new one, and the index is encrypted again under it.",
|
||||||
|
examples: &[],
|
||||||
|
caution: Some(
|
||||||
|
"Changing the password deletes the index and builds it again. Your files are \
|
||||||
|
not touched.",
|
||||||
|
),
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static DISABLE_PASSWORD: Tip = Tip {
|
||||||
|
title: "Disable protection",
|
||||||
|
body: "Removes the password and leaves the index unencrypted on disk. \
|
||||||
|
Anyone able to read that file can then see the names of your files \
|
||||||
|
and the text inside them.",
|
||||||
|
examples: &[],
|
||||||
|
caution: Some(
|
||||||
|
"Turning protection off deletes the index and builds it again. Your files are \
|
||||||
|
not touched.",
|
||||||
|
),
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static REMEMBER_KEYCHAIN: Tip = Tip {
|
||||||
|
title: "Remember on this device",
|
||||||
|
body: "Hands the key to the password store your system already has, such \
|
||||||
|
as GNOME Keyring, KWallet, or Windows Credential Manager, so that \
|
||||||
|
QuickSearch can unlock the index without asking at startup.\n\n\
|
||||||
|
The password itself is never stored, only the key worked out from \
|
||||||
|
it, and only on this machine. Off, you type the password each time \
|
||||||
|
QuickSearch starts.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Manage Index tab: indexing controls ---------------------------------
|
||||||
|
|
||||||
|
pub static START_NOW: Tip = Tip {
|
||||||
|
title: "Start indexing now",
|
||||||
|
body: "Runs a full pass over your indexed folders straight away instead \
|
||||||
|
of waiting for the next scheduled one. Worth doing after adding a \
|
||||||
|
folder, after changing a filter, or when the computer has been off \
|
||||||
|
for a while.\n\n\
|
||||||
|
Searching carries on working while it runs. Unavailable while a \
|
||||||
|
run is already under way.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static STOP_INDEXING: Tip = Tip {
|
||||||
|
title: "Stop",
|
||||||
|
body: "Stops the run in progress and switches to manual, so QuickSearch \
|
||||||
|
no longer watches for changes or reindexes on a schedule. The \
|
||||||
|
index stays as it is and searching still works, but it drifts out \
|
||||||
|
of date as your files change.\n\n\
|
||||||
|
Saved immediately: QuickSearch is still in manual the next time it \
|
||||||
|
starts.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static RETURN_TO_AUTO: Tip = Tip {
|
||||||
|
title: "Return to Automatic",
|
||||||
|
body: "Goes back to watching your folders and reindexing on a schedule, \
|
||||||
|
catching up on everything that changed while indexing was manual.\n\n\
|
||||||
|
Also saved, so this is how QuickSearch starts from now on.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static CLEAR_INDEX: Tip = Tip {
|
||||||
|
title: "Clear index",
|
||||||
|
body: "Deletes the index database. Searching finds nothing until it is \
|
||||||
|
built again, which for a large folder takes a while. Your own \
|
||||||
|
files are never touched.\n\n\
|
||||||
|
QuickSearch asks for confirmation first, then drops to manual so \
|
||||||
|
that it does not immediately rebuild what you just deleted.",
|
||||||
|
examples: &[],
|
||||||
|
caution: Some("This cannot be undone: the index has to be built from scratch again."),
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Manage Index tab: indexed folders -----------------------------------
|
||||||
|
|
||||||
|
pub static ADD_ROOT: Tip = Tip {
|
||||||
|
title: "Add an indexed folder",
|
||||||
|
body: "Adds a folder for QuickSearch to index, along with everything \
|
||||||
|
inside it. Choose it with the browser, or type the path and press \
|
||||||
|
Add.\n\n\
|
||||||
|
Indexed folders may not overlap, so a folder already inside \
|
||||||
|
another one is refused. Adding a folder starts an indexing pass to \
|
||||||
|
pick it up and leaves the rest of the index alone.",
|
||||||
|
examples: &["a second drive, or a network share you search often."],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static REMOVE_ROOT: Tip = Tip {
|
||||||
|
title: "Remove this folder",
|
||||||
|
body: "Stops indexing this folder and removes its entries from the \
|
||||||
|
index. The rest of the index is left alone, and the files \
|
||||||
|
themselves are not touched.\n\n\
|
||||||
|
Takes effect when you click Apply & Save.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static ROOT_WORKERS: Tip = Tip {
|
||||||
|
title: "Workers",
|
||||||
|
body: "How many folders QuickSearch explores at once inside this indexed \
|
||||||
|
folder. More of them finish sooner on storage that answers many \
|
||||||
|
requests at a time, which network drives do especially well, but \
|
||||||
|
they compete for the same disk.\n\n\
|
||||||
|
auto reads 4 on local storage and 16 on a network mount. Takes \
|
||||||
|
effect on the next indexing run.",
|
||||||
|
examples: &[
|
||||||
|
"auto unless indexing is slower than you would expect.",
|
||||||
|
"16 or more for a network share that is slow to answer each request.",
|
||||||
|
"2 to keep indexing out of the way on an older machine.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Manage Index tab: content filters -----------------------------------
|
||||||
|
|
||||||
|
pub static EXT_WHITELIST: Tip = Tip {
|
||||||
|
title: "Full-text extensions whitelist",
|
||||||
|
body: "Which kinds of file QuickSearch reads the text out of, one \
|
||||||
|
extension per line, the leading dot optional. Empty means every \
|
||||||
|
kind it understands.\n\n\
|
||||||
|
Every file is still indexed by name whatever you put here. A list \
|
||||||
|
also leaves out files with no extension at all, such as Makefile \
|
||||||
|
or README, unless you add the line (none). Anything after a # is a \
|
||||||
|
comment, so a file type can be switched off without losing the \
|
||||||
|
line.\n\n\
|
||||||
|
Narrowing the list discards the text it now excludes; widening it \
|
||||||
|
reads those files again.",
|
||||||
|
examples: &[
|
||||||
|
"txt, md and pdf to keep the index small and focused on documents.",
|
||||||
|
"empty to search inside everything QuickSearch can read.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub static IGNORE_PATTERNS: Tip = Tip {
|
||||||
|
title: "Ignore patterns",
|
||||||
|
body: "Files and folders left out of the index entirely, by name and by \
|
||||||
|
content alike. Type one pattern and click Add.\n\n\
|
||||||
|
A pattern without a slash matches a file or folder name anywhere, \
|
||||||
|
and must match the whole name: .jpg matches only something called \
|
||||||
|
exactly that, while *.jpg matches every JPEG. A pattern with a \
|
||||||
|
slash in it is matched against the whole path, and skips \
|
||||||
|
everything underneath. * stands for any run of characters and ? \
|
||||||
|
for a single one.\n\n\
|
||||||
|
Adding a pattern removes the entries it matches; removing one \
|
||||||
|
indexes them again.",
|
||||||
|
examples: &[
|
||||||
|
"node_modules to skip that folder wherever it turns up.",
|
||||||
|
"*.tmp to skip temporary files by extension.",
|
||||||
|
"a full path such as the Videos folder to skip it and everything inside it.",
|
||||||
|
],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- Shared --------------------------------------------------------------
|
||||||
|
|
||||||
|
pub static APPLY_SAVE: Tip = Tip {
|
||||||
|
title: "Apply & Save",
|
||||||
|
body: "Writes these settings to the configuration file and puts them to \
|
||||||
|
work straight away. Until you click here, your edits are only \
|
||||||
|
staged.\n\n\
|
||||||
|
Narrowing a setting removes the entries it now excludes; widening \
|
||||||
|
one indexes whatever it now allows. Only the tokenizer, the hash \
|
||||||
|
sample size and password protection need the index built again \
|
||||||
|
from scratch, and those ask first.",
|
||||||
|
examples: &[],
|
||||||
|
caution: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// Every tip in the file. A tip missing from here is only missing from
|
||||||
|
/// the checks below, so keep it in step when adding one.
|
||||||
|
const ALL: &[&Tip] = &[
|
||||||
|
&DATABASE_PATH,
|
||||||
|
&REINDEX_INTERVAL,
|
||||||
|
&FOLLOW_SYMLINKS,
|
||||||
|
&INCLUDE_HIDDEN,
|
||||||
|
&TOKENIZER,
|
||||||
|
&HASH_LENGTH,
|
||||||
|
&MAX_STORED_TEXT,
|
||||||
|
&MAX_TEXT_FILE_SIZE,
|
||||||
|
&BATCH_SIZE,
|
||||||
|
&MAX_WAL_SIZE,
|
||||||
|
&STORE_TEXT,
|
||||||
|
&FUZZY_DEFAULT,
|
||||||
|
&FUZZY_EDITS,
|
||||||
|
&DISPLAY_LIMIT,
|
||||||
|
&RESULTS_PER_PAGE,
|
||||||
|
&DEBOUNCE,
|
||||||
|
&UI_SCALE,
|
||||||
|
&ENABLE_PASSWORD,
|
||||||
|
&CHANGE_PASSWORD,
|
||||||
|
&DISABLE_PASSWORD,
|
||||||
|
&REMEMBER_KEYCHAIN,
|
||||||
|
&START_NOW,
|
||||||
|
&STOP_INDEXING,
|
||||||
|
&RETURN_TO_AUTO,
|
||||||
|
&CLEAR_INDEX,
|
||||||
|
&ADD_ROOT,
|
||||||
|
&REMOVE_ROOT,
|
||||||
|
&ROOT_WORKERS,
|
||||||
|
&EXT_WHITELIST,
|
||||||
|
&IGNORE_PATTERNS,
|
||||||
|
&APPLY_SAVE,
|
||||||
|
];
|
||||||
|
|
||||||
|
/// Everything a tip can put on screen, as one string.
|
||||||
|
fn all_text(tip: &Tip) -> String {
|
||||||
|
let mut text = format!("{}\n{}", tip.title, tip.body);
|
||||||
|
for example in tip.examples {
|
||||||
|
text.push('\n');
|
||||||
|
text.push_str(example);
|
||||||
|
}
|
||||||
|
if let Some(caution) = tip.caution {
|
||||||
|
text.push('\n');
|
||||||
|
text.push_str(caution);
|
||||||
|
}
|
||||||
|
text
|
||||||
|
}
|
||||||
|
|
||||||
|
/// House style, and the one rule that is easy to break by pasting from
|
||||||
|
/// the source comments: these tooltips use no em-dashes.
|
||||||
|
#[test]
|
||||||
|
fn no_tip_uses_an_em_dash() {
|
||||||
|
for tip in ALL {
|
||||||
|
assert!(
|
||||||
|
!all_text(tip).contains('—'),
|
||||||
|
"{} uses an em-dash",
|
||||||
|
tip.title
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_tip_is_filled_in() {
|
||||||
|
for tip in ALL {
|
||||||
|
assert!(!tip.title.trim().is_empty(), "a tip has no title");
|
||||||
|
assert!(
|
||||||
|
!tip.title.ends_with('.'),
|
||||||
|
"{}: title is not a sentence",
|
||||||
|
tip.title
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
tip.body.trim().len() > 40,
|
||||||
|
"{}: body says too little",
|
||||||
|
tip.title
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
tip.body.trim_end().ends_with('.'),
|
||||||
|
"{}: body is not a finished sentence",
|
||||||
|
tip.title
|
||||||
|
);
|
||||||
|
// "Stops the run in progress" under the title "Stop" is fine;
|
||||||
|
// "Stop. Stops the run" is the restatement worth catching, so
|
||||||
|
// the title only counts as repeated when a word ends there.
|
||||||
|
let restates = tip
|
||||||
|
.body
|
||||||
|
.strip_prefix(tip.title)
|
||||||
|
.is_some_and(|rest| !rest.starts_with(|c: char| c.is_alphanumeric()));
|
||||||
|
assert!(!restates, "{}: body repeats the title", tip.title);
|
||||||
|
for example in tip.examples {
|
||||||
|
assert!(
|
||||||
|
!example.trim().is_empty() && example.trim_end().ends_with('.'),
|
||||||
|
"{}: bad example {:?}",
|
||||||
|
tip.title,
|
||||||
|
example
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if let Some(caution) = tip.caution {
|
||||||
|
assert!(
|
||||||
|
caution.trim_end().ends_with('.'),
|
||||||
|
"{}: caution is not a finished sentence",
|
||||||
|
tip.title
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Descriptive, but a tooltip nobody reads to the end helps nobody.
|
||||||
|
#[test]
|
||||||
|
fn no_tip_is_a_wall_of_text() {
|
||||||
|
for tip in ALL {
|
||||||
|
let len = all_text(tip).chars().count();
|
||||||
|
assert!(len <= 900, "{} is {} characters long", tip.title, len);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Two controls sharing a title means one of them was pasted from the
|
||||||
|
/// other and never renamed.
|
||||||
|
#[test]
|
||||||
|
fn titles_are_distinct() {
|
||||||
|
let mut seen: Vec<&str> = ALL.iter().map(|t| t.title).collect();
|
||||||
|
seen.sort_unstable();
|
||||||
|
let count = seen.len();
|
||||||
|
seen.dedup();
|
||||||
|
assert_eq!(count, seen.len(), "two tips share a title: {:?}", seen);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The renderer puts every part on screen: title, body, examples and
|
||||||
|
/// caution. Written against the tip with all four.
|
||||||
|
#[test]
|
||||||
|
fn show_paints_every_part() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
|
||||||
|
let full = ctx.run(input, |ctx| {
|
||||||
|
egui::CentralPanel::default().show(ctx, |ui| TOKENIZER.show(ui));
|
||||||
|
});
|
||||||
|
let painted = crate::test_ui::painted_text(&full).join("\n");
|
||||||
|
assert!(painted.contains(TOKENIZER.title), "no title: {painted}");
|
||||||
|
assert!(painted.contains("trigram indexes every run"), "no body");
|
||||||
|
assert!(painted.contains("• trigram, the default"), "no examples");
|
||||||
|
assert!(
|
||||||
|
painted.contains(TOKENIZER.caution.unwrap()),
|
||||||
|
"no caution line"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A lone example reads as a sentence rather than a one-item list.
|
||||||
|
#[test]
|
||||||
|
fn a_single_example_is_prefixed_with_example() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
|
||||||
|
let full = ctx.run(input, |ctx| {
|
||||||
|
egui::CentralPanel::default().show(ctx, |ui| INCLUDE_HIDDEN.show(ui));
|
||||||
|
});
|
||||||
|
let painted = crate::test_ui::painted_text(&full).join("\n");
|
||||||
|
assert!(
|
||||||
|
painted.contains(&format!("Example: {}", INCLUDE_HIDDEN.examples[0])),
|
||||||
|
"{painted}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A greyed-out control still explains itself. egui shows nothing on a
|
||||||
|
/// disabled widget unless the *disabled* tooltip is set too, and "why
|
||||||
|
/// can I not click this" is exactly when the answer is wanted: Stop,
|
||||||
|
/// Start indexing now and Return to Automatic are greyed out by turns.
|
||||||
|
#[test]
|
||||||
|
fn a_disabled_control_still_explains_itself() {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
ctx.style_mut(|s| {
|
||||||
|
s.interaction.tooltip_delay = 0.0;
|
||||||
|
s.interaction.show_tooltips_only_when_still = false;
|
||||||
|
});
|
||||||
|
let run = |events: Vec<egui::Event>| {
|
||||||
|
let input = crate::test_ui::raw_input(egui::vec2(600.0, 400.0), events);
|
||||||
|
ctx.run(input, |ctx| {
|
||||||
|
egui::CentralPanel::default().show(ctx, |ui| {
|
||||||
|
ui.add_enabled(false, egui::Button::new("Stop"))
|
||||||
|
.tip(&STOP_INDEXING);
|
||||||
|
});
|
||||||
|
})
|
||||||
|
};
|
||||||
|
|
||||||
|
run(vec![]);
|
||||||
|
let settled = run(vec![]);
|
||||||
|
let pos = crate::test_ui::painted_text_center(&settled, "Stop").expect("button painted");
|
||||||
|
let opening: String = STOP_INDEXING.body.chars().take(40).collect();
|
||||||
|
let mut out = run(vec![egui::Event::PointerMoved(pos)]);
|
||||||
|
for _ in 0..3 {
|
||||||
|
if crate::test_ui::painted_text(&out)
|
||||||
|
.join("\n")
|
||||||
|
.contains(&opening)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
out = run(vec![]);
|
||||||
|
}
|
||||||
|
panic!("a disabled control said nothing on hover");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,7 +1,8 @@
|
||||||
//! Shared UI helpers: emphasis colors, bordered widgets, ignore-pattern
|
//! Shared UI helpers: emphasis colors, bordered widgets, ignore-pattern
|
||||||
//! validation, and the "more content below" scroll hint.
|
//! validation, text eliding, and the "more content below" scroll hint.
|
||||||
|
|
||||||
use quicksearch_core::config::IgnoreSet;
|
use quicksearch_core::config::IgnoreSet;
|
||||||
|
use std::borrow::Cow;
|
||||||
|
|
||||||
/// Warning/emphasis orange, also used for the fuzzy-edit-distance warning.
|
/// Warning/emphasis orange, also used for the fuzzy-edit-distance warning.
|
||||||
pub const ORANGE: egui::Color32 = egui::Color32::from_rgb(220, 150, 40);
|
pub const ORANGE: egui::Color32 = egui::Color32::from_rgb(220, 150, 40);
|
||||||
|
|
@ -136,6 +137,81 @@ pub fn pattern_edit(
|
||||||
(response, valid)
|
(response, valid)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
|
||||||
|
/// `font_id`, returning it borrowed and untouched when it already fits.
|
||||||
|
///
|
||||||
|
/// A path's two ends are the informative ones — the head says which volume
|
||||||
|
/// or home it lives under, the tail names the deepest directories — so a
|
||||||
|
/// column too narrow for the whole thing should drop out of the middle
|
||||||
|
/// rather than tail-truncate the way egui does by default.
|
||||||
|
///
|
||||||
|
/// The budget is in pixels, summed from the font's own glyph advances (the
|
||||||
|
/// same numbers egui's layout adds up), not a character count scaled by the
|
||||||
|
/// width of one sample glyph. The proportional body font makes that estimate
|
||||||
|
/// wrong in both directions: overshoot and egui elides the result a *second*
|
||||||
|
/// time, painting two ellipses; undershoot and the column sits visibly short
|
||||||
|
/// of full.
|
||||||
|
///
|
||||||
|
/// The borrowed/owned distinction is also the caller's signal that something
|
||||||
|
/// was dropped, which is what a "full text on hover" tooltip keys off.
|
||||||
|
pub fn middle_elide<'a>(
|
||||||
|
ui: &egui::Ui,
|
||||||
|
text: &'a str,
|
||||||
|
max_width: f32,
|
||||||
|
font_id: &egui::FontId,
|
||||||
|
) -> Cow<'a, str> {
|
||||||
|
ui.fonts(|f| {
|
||||||
|
let width_of = |c: char| f.glyph_width(font_id, c);
|
||||||
|
if text.chars().map(width_of).sum::<f32>() <= max_width {
|
||||||
|
return Cow::Borrowed(text);
|
||||||
|
}
|
||||||
|
let budget = max_width - width_of('…');
|
||||||
|
|
||||||
|
// Grow a head and a tail toward each other through the middle,
|
||||||
|
// each step feeding whichever side is currently narrower so the cut
|
||||||
|
// lands near the middle. Indices advance by whole characters, so
|
||||||
|
// they always land on UTF-8 boundaries.
|
||||||
|
let (mut head, mut tail) = (0usize, text.len());
|
||||||
|
let (mut head_w, mut tail_w) = (0.0f32, 0.0f32);
|
||||||
|
while head < tail {
|
||||||
|
let rest = &text[head..tail];
|
||||||
|
let front = rest.chars().next().expect("head < tail");
|
||||||
|
let back = rest.chars().next_back().expect("head < tail");
|
||||||
|
let (front_w, back_w) = (width_of(front), width_of(back));
|
||||||
|
let used = head_w + tail_w;
|
||||||
|
let (front_fits, back_fits) = (used + front_w <= budget, used + back_w <= budget);
|
||||||
|
if !front_fits && !back_fits {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
// The preferred side wins when it fits; otherwise the other one
|
||||||
|
// does, since at least one of them just did.
|
||||||
|
let take_front = if head_w <= tail_w {
|
||||||
|
front_fits
|
||||||
|
} else {
|
||||||
|
!back_fits
|
||||||
|
};
|
||||||
|
if take_front {
|
||||||
|
head += front.len_utf8();
|
||||||
|
head_w += front_w;
|
||||||
|
} else {
|
||||||
|
tail -= back.len_utf8();
|
||||||
|
tail_w += back_w;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if head >= tail {
|
||||||
|
// The two halves met without dropping anything — splicing an
|
||||||
|
// ellipsis in now would only lengthen a string that fits.
|
||||||
|
return Cow::Borrowed(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut out = String::with_capacity(head + '…'.len_utf8() + (text.len() - tail));
|
||||||
|
out.push_str(&text[..head]);
|
||||||
|
out.push('…');
|
||||||
|
out.push_str(&text[tail..]);
|
||||||
|
Cow::Owned(out)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Paint a semitransparent down-arrow near the bottom edge of a scroll
|
/// Paint a semitransparent down-arrow near the bottom edge of a scroll
|
||||||
/// area while more content lies below the fold. Painter-only, so it can
|
/// area while more content lies below the fold. Painter-only, so it can
|
||||||
/// never swallow clicks. (The bundled fonts have no ▼ glyph — this is a
|
/// never swallow clicks. (The bundled fonts have no ▼ glyph — this is a
|
||||||
|
|
@ -175,7 +251,115 @@ pub fn more_below_hint<R>(ui: &egui::Ui, out: &egui::scroll_area::ScrollAreaOutp
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{ignore_pattern_valid, pattern_border, pattern_hint, INVALID_RED, VALID_GREEN};
|
use super::{
|
||||||
|
ignore_pattern_valid, middle_elide, pattern_border, pattern_hint, Cow, INVALID_RED,
|
||||||
|
VALID_GREEN,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// A `Ui` from a real (headless) egui pass, so `middle_elide` measures
|
||||||
|
/// with the same fonts the app paints with — the whole point of the
|
||||||
|
/// helper is that its arithmetic agrees with egui's layout.
|
||||||
|
fn with_ui<R>(f: impl FnOnce(&mut egui::Ui) -> R) -> R {
|
||||||
|
let ctx = egui::Context::default();
|
||||||
|
let mut f = Some(f);
|
||||||
|
let mut out = None;
|
||||||
|
let _ = ctx.run(egui::RawInput::default(), |ctx| {
|
||||||
|
egui::CentralPanel::default().show(ctx, |ui| {
|
||||||
|
if let Some(f) = f.take() {
|
||||||
|
out = Some(f(ui));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
out.expect("the central panel ran")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn body_font(ui: &egui::Ui) -> egui::FontId {
|
||||||
|
egui::TextStyle::Body.resolve(ui.style())
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEEP: &str = "/media/shared/QuickSearch/crates/quicksearch-gui/src/search_tab.rs";
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn text_that_fits_comes_back_untouched() {
|
||||||
|
with_ui(|ui| {
|
||||||
|
let font = body_font(ui);
|
||||||
|
let out = middle_elide(ui, DEEP, 10_000.0, &font);
|
||||||
|
assert!(matches!(out, Cow::Borrowed(_)), "borrowed when it fits");
|
||||||
|
assert_eq!(out, DEEP);
|
||||||
|
// Nothing to elide, however little room there is.
|
||||||
|
assert!(matches!(middle_elide(ui, "", 0.0, &font), Cow::Borrowed(_)));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn eliding_keeps_both_ends_and_cuts_once() {
|
||||||
|
with_ui(|ui| {
|
||||||
|
let out = middle_elide(ui, DEEP, 200.0, &body_font(ui));
|
||||||
|
assert!(matches!(out, Cow::Owned(_)), "{out}");
|
||||||
|
assert_eq!(out.matches('…').count(), 1, "{out}");
|
||||||
|
let (head, tail) = out.split_once('…').expect("one ellipsis");
|
||||||
|
assert!(
|
||||||
|
!head.is_empty() && !tail.is_empty(),
|
||||||
|
"both ends survive: {out}"
|
||||||
|
);
|
||||||
|
assert!(DEEP.starts_with(head), "{out}");
|
||||||
|
assert!(DEEP.ends_with(tail), "{out}");
|
||||||
|
assert!(
|
||||||
|
tail.ends_with("search_tab.rs"),
|
||||||
|
"the filename survives: {out}"
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The load-bearing property. If the result laid out in the same font
|
||||||
|
/// were wider than the budget, egui would elide it a *second* time and
|
||||||
|
/// paint two ellipses.
|
||||||
|
#[test]
|
||||||
|
fn the_result_fits_the_budget_it_was_given() {
|
||||||
|
with_ui(|ui| {
|
||||||
|
let font = body_font(ui);
|
||||||
|
for width in [40.0f32, 60.0, 121.5, 200.0, 337.5, 480.0] {
|
||||||
|
let out = middle_elide(ui, DEEP, width, &font);
|
||||||
|
let painted = ui.fonts(|f| {
|
||||||
|
f.layout_no_wrap(out.to_string(), font.clone(), egui::Color32::WHITE)
|
||||||
|
.size()
|
||||||
|
.x
|
||||||
|
});
|
||||||
|
assert!(
|
||||||
|
painted <= width,
|
||||||
|
"at {width}: {out:?} lays out at {painted}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An ellipsis is the least that can stand for the text; a column too
|
||||||
|
/// narrow even for that gets it anyway rather than a panic.
|
||||||
|
#[test]
|
||||||
|
fn degenerate_widths_never_panic() {
|
||||||
|
with_ui(|ui| {
|
||||||
|
let font = body_font(ui);
|
||||||
|
for width in [f32::NEG_INFINITY, -50.0, 0.0] {
|
||||||
|
assert_eq!(middle_elide(ui, DEEP, width, &font), "…", "at {width}");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Indices walk by whole characters, so a path of multi-byte glyphs
|
||||||
|
/// slices cleanly instead of panicking mid-codepoint.
|
||||||
|
#[test]
|
||||||
|
fn multi_byte_paths_split_on_character_boundaries() {
|
||||||
|
with_ui(|ui| {
|
||||||
|
let font = body_font(ui);
|
||||||
|
let path = "/srv/données/日本語/архив/файл-très-long.txt";
|
||||||
|
for width in [30.0f32, 55.0, 90.0, 140.0, 210.0, 400.0] {
|
||||||
|
let out = middle_elide(ui, path, width, &font);
|
||||||
|
let (head, tail) = out.split_once('…').unwrap_or((out.as_ref(), ""));
|
||||||
|
assert!(path.starts_with(head), "at {width}: {out}");
|
||||||
|
assert!(path.ends_with(tail), "at {width}: {out}");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/// The trap behind "my ignore filters don't work" reports: ".jpg" is an
|
/// The trap behind "my ignore filters don't work" reports: ".jpg" is an
|
||||||
/// exact-name pattern, and the hint must say so and offer "*.jpg".
|
/// exact-name pattern, and the hint must say so and offer "*.jpg".
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,9 @@ pub enum KeySource {
|
||||||
|
|
||||||
/// The application shell handed to eframe: locked (unlock screen) or
|
/// The application shell handed to eframe: locked (unlock screen) or
|
||||||
/// running (the real app).
|
/// running (the real app).
|
||||||
|
// Exactly one of these exists for the lifetime of the process, and it is
|
||||||
|
// already boxed on the side that would matter.
|
||||||
|
#[allow(clippy::large_enum_variant)]
|
||||||
pub enum Gate {
|
pub enum Gate {
|
||||||
Locked(UnlockScreen),
|
Locked(UnlockScreen),
|
||||||
Running(Box<QuickSearchApp>),
|
Running(Box<QuickSearchApp>),
|
||||||
|
|
@ -339,12 +342,11 @@ impl UnlockScreen {
|
||||||
/// UI-side buffers.
|
/// UI-side buffers.
|
||||||
fn submit(&mut self, ctx: &egui::Context) {
|
fn submit(&mut self, ctx: &egui::Context) {
|
||||||
self.error = None;
|
self.error = None;
|
||||||
if matches!(self.mode, Mode::Create) {
|
if matches!(self.mode, Mode::Create)
|
||||||
if self.password.is_empty() {
|
&& self.password.is_empty() {
|
||||||
self.error = Some("The password may not be empty.".to_string());
|
self.error = Some("The password may not be empty.".to_string());
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
}
|
|
||||||
let Ok(salt) = self.cfg.security.salt_bytes() else {
|
let Ok(salt) = self.cfg.security.salt_bytes() else {
|
||||||
return; // BrokenSalt mode never reaches submit
|
return; // BrokenSalt mode never reaches submit
|
||||||
};
|
};
|
||||||
|
|
@ -381,8 +383,10 @@ impl UnlockScreen {
|
||||||
// stick. Surface it in the running app's banner.
|
// stick. Surface it in the running app's banner.
|
||||||
self.config_error = Some(e);
|
self.config_error = Some(e);
|
||||||
}
|
}
|
||||||
} else {
|
} else if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
|
||||||
keychain::delete_key(&db_path.to_string_lossy());
|
// Same treatment as the store half above: unlock proceeds, but
|
||||||
|
// the banner says the old key is still on the keychain.
|
||||||
|
self.config_error = Some(e);
|
||||||
}
|
}
|
||||||
if self.cfg.security.use_keychain != self.remember {
|
if self.cfg.security.use_keychain != self.remember {
|
||||||
self.cfg.security.use_keychain = self.remember;
|
self.cfg.security.use_keychain = self.remember;
|
||||||
|
|
@ -447,7 +451,12 @@ impl UnlockScreen {
|
||||||
if let Err(e) = delete_index_files(&db_path) {
|
if let Err(e) = delete_index_files(&db_path) {
|
||||||
self.error = Some(e);
|
self.error = Some(e);
|
||||||
} else {
|
} else {
|
||||||
keychain::delete_key(&db_path.to_string_lossy());
|
// The index files are already gone; a surviving
|
||||||
|
// entry would point at a database that no longer
|
||||||
|
// exists. Non-fatal, but not silent.
|
||||||
|
if let Err(e) = keychain::delete_key(&db_path.to_string_lossy()) {
|
||||||
|
self.config_error = Some(e);
|
||||||
|
}
|
||||||
db::set_process_key(None);
|
db::set_process_key(None);
|
||||||
self.cfg.security = SecurityConfig::default();
|
self.cfg.security = SecurityConfig::default();
|
||||||
if let Err(e) = self.cfg.save() {
|
if let Err(e) = self.cfg.save() {
|
||||||
|
|
@ -508,6 +517,9 @@ fn delete_index_files(db_path: &std::path::Path) -> Result<(), String> {
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
|
/// The unlock screen is the whole window, so its viewport is the window's.
|
||||||
|
const SCREEN: egui::Vec2 = egui::vec2(900.0, 600.0);
|
||||||
|
|
||||||
/// A protected config whose salt parses, so the screen lands in a real
|
/// A protected config whose salt parses, so the screen lands in a real
|
||||||
/// password mode rather than `BrokenSalt`.
|
/// password mode rather than `BrokenSalt`.
|
||||||
fn locked_config() -> Config {
|
fn locked_config() -> Config {
|
||||||
|
|
@ -522,13 +534,7 @@ mod tests {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn frame(ctx: &egui::Context, screen: &mut UnlockScreen) {
|
fn frame(ctx: &egui::Context, screen: &mut UnlockScreen) {
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(SCREEN, Vec::new());
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(900.0, 600.0),
|
|
||||||
)),
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
let _ = ctx.run(input, |ctx| {
|
let _ = ctx.run(input, |ctx| {
|
||||||
// `update` only builds the app on a successful unlock, which needs
|
// `update` only builds the app on a successful unlock, which needs
|
||||||
// a derived key — so with nothing submitted this stays on screen.
|
// a derived key — so with nothing submitted this stays on screen.
|
||||||
|
|
@ -591,13 +597,7 @@ mod tests {
|
||||||
fn the_lock_screen_shows_the_build_id() {
|
fn the_lock_screen_shows_the_build_id() {
|
||||||
let ctx = egui::Context::default();
|
let ctx = egui::Context::default();
|
||||||
let mut screen = UnlockScreen::new(locked_config(), None, None);
|
let mut screen = UnlockScreen::new(locked_config(), None, None);
|
||||||
let input = egui::RawInput {
|
let input = crate::test_ui::raw_input(SCREEN, Vec::new());
|
||||||
screen_rect: Some(egui::Rect::from_min_size(
|
|
||||||
egui::Pos2::ZERO,
|
|
||||||
egui::vec2(900.0, 600.0),
|
|
||||||
)),
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
|
|
||||||
let out = ctx.run(input, |ctx| {
|
let out = ctx.run(input, |ctx| {
|
||||||
assert!(screen.update(ctx).is_none());
|
assert!(screen.update(ctx).is_none());
|
||||||
|
|
|
||||||
|
|
@ -122,9 +122,16 @@ say "Installed ${#icons[@]} icon sizes plus the scalable SVG"
|
||||||
# Debian wants man pages and the changelog compressed, with no gzip timestamp so
|
# Debian wants man pages and the changelog compressed, with no gzip timestamp so
|
||||||
# repeat builds are byte-identical. quicksearch-cli.1 is a one-line .so stub
|
# repeat builds are byte-identical. quicksearch-cli.1 is a one-line .so stub
|
||||||
# pointing at quicksearch.1, which documents both binaries.
|
# pointing at quicksearch.1, which documents both binaries.
|
||||||
|
#
|
||||||
|
# The .TH version field is rewritten from $version on the way through, so the
|
||||||
|
# footer of an installed page tracks [workspace.package] version instead of
|
||||||
|
# needing a hand-edit every release. The stub has no .TH, so the sed is a
|
||||||
|
# no-op there.
|
||||||
install -dm755 "$stage/usr/share/man/man1"
|
install -dm755 "$stage/usr/share/man/man1"
|
||||||
for page in "${BINARIES[@]}"; do
|
for page in "${BINARIES[@]}"; do
|
||||||
gzip -9nc "$REPO_ROOT/packaging/$page.1" > "$stage/usr/share/man/man1/$page.1.gz"
|
sed "/^\.TH /s/\"quicksearch [^\"]*\"/\"quicksearch $version\"/" \
|
||||||
|
"$REPO_ROOT/packaging/$page.1" \
|
||||||
|
| gzip -9nc > "$stage/usr/share/man/man1/$page.1.gz"
|
||||||
chmod 644 "$stage/usr/share/man/man1/$page.1.gz"
|
chmod 644 "$stage/usr/share/man/man1/$page.1.gz"
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,7 @@
|
||||||
.TH QUICKSEARCH 1 "2026-08-02" "quicksearch 0.1.0" "User Commands"
|
.\" The version in the .TH line above is rewritten from [workspace.package]
|
||||||
|
.\" version by packaging/build-deb.sh at package time; the literal here is
|
||||||
|
.\" only what an uninstalled read of this file shows.
|
||||||
|
.TH QUICKSEARCH 1 "2026-08-05" "quicksearch 1.0.2" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
quicksearch, quicksearch\-cli \- fast full\-text search across your files
|
quicksearch, quicksearch\-cli \- fast full\-text search across your files
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
|
|
@ -60,29 +63,78 @@ of bare paths. Highlights the match in bold when stdout is a terminal.
|
||||||
.TP
|
.TP
|
||||||
.BR \-h ", " \-\-help
|
.BR \-h ", " \-\-help
|
||||||
Print usage and exit.
|
Print usage and exit.
|
||||||
|
.TP
|
||||||
|
.BR \-V ", " \-\-version
|
||||||
|
Print the version and the commit it was built from, then exit. Quote this in
|
||||||
|
bug reports.
|
||||||
.PP
|
.PP
|
||||||
An unrecognised option given without any query terms is passed through and the
|
An unrecognised option given without any query terms is passed through and the
|
||||||
application is opened, since it may be an option for the windowing backend.
|
application is opened, since it may be an option for the windowing backend.
|
||||||
.SH QUERY SYNTAX
|
.SH QUERY SYNTAX
|
||||||
Plain words form a single phrase. Filters may be combined with it:
|
Plain words form a single phrase, matched against names, contents and paths.
|
||||||
|
Filters may be combined with it:
|
||||||
|
.TP
|
||||||
|
.B \(dq\fIexact phrase\fB\(dq
|
||||||
|
Quoting keeps spaces, stars and filter\-like words literal.
|
||||||
|
.B \(dq\(dq
|
||||||
|
escapes a quote.
|
||||||
|
.TP
|
||||||
|
.BI bud * port
|
||||||
|
.B *
|
||||||
|
matches any run of characters, staying within one line of content.
|
||||||
|
.B %
|
||||||
|
and
|
||||||
|
.B _
|
||||||
|
are always literal.
|
||||||
|
.TP
|
||||||
|
.B regex:\fIpattern\fR
|
||||||
|
Match a regular expression against names, contents and paths, for example
|
||||||
|
.BR regex:\(dq(foo|bar)\ed+\(dq .
|
||||||
|
Case\-insensitive by default;
|
||||||
|
.B (?\-i:\(dq\(dq)
|
||||||
|
overrides. Quote patterns containing spaces or
|
||||||
|
.BR "( ) : = < > \(dq" .
|
||||||
.TP
|
.TP
|
||||||
.B type:\fIName\fR
|
.B type:\fIName\fR
|
||||||
Match a file class, for example
|
Match a file class: one of
|
||||||
.IR type:Document ", " type:Image ", " type:Audio .
|
.IR Audio ", " Image ", " Video ", " Document ", " Text ", " Archive ", "
|
||||||
|
.IR Spreadsheet ", " Presentation ", " Folder .
|
||||||
.TP
|
.TP
|
||||||
.B modified:\fIexpr\fR
|
.B modified:\fIexpr\fR
|
||||||
Compare against the modification date, for example
|
Compare against the modification date, for example
|
||||||
.IR modified:>=2024-01-01 .
|
.IR modified:>=2024-01-01 .
|
||||||
|
Also
|
||||||
|
.BR < ", " <= ", " > " and " = ;
|
||||||
|
dates are
|
||||||
|
.IR yyyy-mm-dd .
|
||||||
|
.B mtime:
|
||||||
|
is an alias.
|
||||||
.TP
|
.TP
|
||||||
.B path:\fI/dir\fR
|
.B path:\fI/dir\fR
|
||||||
Restrict results to a directory.
|
Restrict results to a directory and its subdirectories.
|
||||||
|
.B folder:
|
||||||
|
and
|
||||||
|
.B includefolder:
|
||||||
|
are aliases.
|
||||||
|
.B *
|
||||||
|
is literal here.
|
||||||
.TP
|
.TP
|
||||||
.B mime:\fItype\fR
|
.B mime:\fItype\fR
|
||||||
Match a MIME type, for example
|
Match a MIME type exactly, for example
|
||||||
.IR mime:application/pdf .
|
.IR mime:application/pdf .
|
||||||
.TP
|
.TP
|
||||||
.B name:\fIfragment\fR
|
.B name:\fIfragment\fR
|
||||||
Match a fragment of the filename.
|
Match a fragment of the filename. A filter, so it does not affect ranking.
|
||||||
|
.B filename:
|
||||||
|
is an alias; an unquoted
|
||||||
|
.B *
|
||||||
|
globs.
|
||||||
|
.PP
|
||||||
|
Unrecognised
|
||||||
|
.I key:value
|
||||||
|
text stays part of the search phrase.
|
||||||
|
.BR AND ", " OR
|
||||||
|
and parentheses are treated as plain words.
|
||||||
.SH PASSWORD PROTECTION
|
.SH PASSWORD PROTECTION
|
||||||
The index can be encrypted with a password (application Options, Security).
|
The index can be encrypted with a password (application Options, Security).
|
||||||
A protected index must be unlocked every time either binary starts. The
|
A protected index must be unlocked every time either binary starts. The
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue