Compare commits
No commits in common. "Release-v1.1" and "master" have entirely different histories.
Release-v1
...
master
99 changed files with 2174 additions and 10913 deletions
|
|
@ -63,6 +63,12 @@ jobs:
|
|||
HOME: /root
|
||||
# The highest libc6 version the .deb is allowed to require.
|
||||
MAX_GLIBC: '2.35'
|
||||
# full_index.rs asserts a heavy indexing root cannot stall a light one, and
|
||||
# measures that as wall-clock stall. The 100 ms default is calibrated on a
|
||||
# developer machine; this runner measured 188 ms for the same correct
|
||||
# behaviour. 600 ms keeps the check meaningful - the regression it exists to
|
||||
# catch is ~6x the healthy figure, so it would land near 1.2 s here.
|
||||
QSB_STALL_BUDGET_MS: '600'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
|
@ -93,10 +99,6 @@ jobs:
|
|||
# for before it will run.
|
||||
# libcap2-bin provides capsh, which the Test step uses to drop the two
|
||||
# DAC capabilities so root obeys permission bits.
|
||||
# jq parses the tags API for the releasable check below. Named rather
|
||||
# than assumed: the release job gets it from its image, and a base
|
||||
# image change that dropped it would turn that check into a silent
|
||||
# pass rather than a failure.
|
||||
# zsync and appstream are for build-appimage.sh: appimagetool shells
|
||||
# out to zsyncmake rather than bundling it, and reports success while
|
||||
# writing nothing when it is absent, so the script checks for it up
|
||||
|
|
@ -105,44 +107,7 @@ jobs:
|
|||
apt-get install -y --no-install-recommends \
|
||||
build-essential perl pkg-config \
|
||||
binutils dpkg-dev desktop-file-utils gzip libcap2-bin \
|
||||
zsync appstream jq
|
||||
|
||||
- name: Check the release tag is free
|
||||
# The other half of the guard above, for the path that actually cuts
|
||||
# most releases: pushing a Release* branch. There the tag comes from
|
||||
# [workspace.package] rather than the ref, so the mistake is not a
|
||||
# mismatched tag but a *forgotten bump* — the version still points at a
|
||||
# release that already shipped.
|
||||
#
|
||||
# The release job checks this too and remains the authority; it just
|
||||
# cannot check it until both build jobs are green, so forgetting the
|
||||
# bump used to cost two full release builds, packaging and an artifact
|
||||
# upload before anything said so. This says so in seconds.
|
||||
#
|
||||
# Deliberately as lenient as the release job: a tag at *this* commit is
|
||||
# a re-run and fine, and an unreachable API reads as "not found" and
|
||||
# lets the build proceed rather than failing on a network hiccup. An
|
||||
# early check that blocks a good release is worse than one that misses
|
||||
# a bad one, because the late check still catches it.
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
env:
|
||||
RELEASE_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
version=$(sed -n '/^\[workspace\.package\]/,/^\[/{ s/^version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p }' Cargo.toml)
|
||||
[ -n "$version" ] || { echo "could not read the version from Cargo.toml" >&2; exit 1; }
|
||||
tag="v$version"
|
||||
api="${GITHUB_API_URL:-$GITHUB_SERVER_URL/api/v1}"
|
||||
at=$(curl -sS -H "Authorization: token $RELEASE_TOKEN" \
|
||||
"$api/repos/$GITHUB_REPOSITORY/tags/$tag" | jq -r '.commit.sha // empty')
|
||||
if [ -n "$at" ] && [ "$at" != "$GITHUB_SHA" ]; then
|
||||
echo "ERROR: $tag already exists at $at, not $GITHUB_SHA." >&2
|
||||
echo "Bump [workspace.package] version in Cargo.toml, refresh Cargo.lock" >&2
|
||||
echo "with 'cargo update -w', commit and push again." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $tag is free (or already at this commit)"
|
||||
zsync appstream
|
||||
|
||||
- name: Trust the workspace
|
||||
# checkout writes as root into a directory git then considers dubiously
|
||||
|
|
@ -203,23 +168,7 @@ jobs:
|
|||
echo "capabilities in test shell: $(grep CapEff /proc/self/status | tr -d "\t")"
|
||||
cargo test --release --locked --workspace'
|
||||
|
||||
# Everything from here down produces release assets, so it runs only
|
||||
# where a release can actually come out: a v* tag or a Release* branch.
|
||||
# On master and pull requests the job stops after Build and Test, which
|
||||
# is what those runs are for — the packaging that used to follow built a
|
||||
# .deb, an AppImage, a tarball and a 14-day artifact upload that nothing
|
||||
# would ever download, because the release job is skipped there anyway.
|
||||
#
|
||||
# The condition is repeated rather than hoisted into an env var: Actions
|
||||
# has no workflow-level expression alias, YAML anchors are not supported,
|
||||
# and `env.X` inside `if:` would fail *closed* on a runner that did not
|
||||
# populate it — silently skipping packaging on a real release. Spelled
|
||||
# out, it is the same form the release job's own gate uses.
|
||||
- name: Build the .deb
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
# --no-build reuses the binaries from the Build step rather than
|
||||
# recompiling. SOURCE_DATE_EPOCH pins the generated changelog date so
|
||||
# repeat builds of the same commit are byte-identical.
|
||||
|
|
@ -228,14 +177,8 @@ jobs:
|
|||
./packaging/build-deb.sh --no-build
|
||||
|
||||
- name: Check the glibc floor
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
# The whole point of pinning the container. If someone bumps the image,
|
||||
# this fails loudly instead of quietly shipping an uninstallable package.
|
||||
# Gated with the .deb it inspects — there is no package to read without
|
||||
# the step above.
|
||||
run: |
|
||||
deb=$(ls dist/*.deb)
|
||||
depends=$(dpkg-deb -f "$deb" Depends)
|
||||
|
|
@ -251,10 +194,6 @@ jobs:
|
|||
echo "OK: glibc floor $floor <= $MAX_GLIBC"
|
||||
|
||||
- name: Build the AppImage
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
# After the glibc gate, so the cheaper check still fails first. --no-build
|
||||
# reuses the binaries from the Build step, as the .deb step does, and
|
||||
# SOURCE_DATE_EPOCH pins the date substituted into the AppStream release
|
||||
|
|
@ -268,10 +207,6 @@ jobs:
|
|||
./packaging/build-appimage.sh --no-build
|
||||
|
||||
- name: Package the binaries
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
# A tarball for anyone not installing the .deb, stripped to match what
|
||||
# build-deb.sh ships.
|
||||
run: |
|
||||
|
|
@ -296,12 +231,6 @@ jobs:
|
|||
# node16, which current Forgejo runner images no longer ship; the
|
||||
# -node20 tags are Forgejo's builds for precisely this combination.
|
||||
- uses: actions/upload-artifact@v3-node20
|
||||
# `if-no-files-found: error` below would fail every master push once the
|
||||
# packaging steps above are gated, so this carries the same gate.
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
with:
|
||||
name: linux-x86_64
|
||||
# The .zsync is not optional: the update URL baked into every AppImage
|
||||
|
|
@ -393,16 +322,7 @@ jobs:
|
|||
fi
|
||||
done
|
||||
|
||||
# As in the linux job, the asset-producing steps run only where a release
|
||||
# can come out. The DLL check above deliberately stays ungated: it
|
||||
# validates the .exe itself rather than packaging it, costs an objdump,
|
||||
# and is exactly the kind of regression worth catching on master rather
|
||||
# than at release time.
|
||||
- name: Build the installer
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
# --no-build reuses the binaries from the Build step rather than
|
||||
# cross-compiling them a second time. The installer and the .zip below
|
||||
# are alternatives, not a two-step download: the installer puts the app
|
||||
|
|
@ -412,10 +332,6 @@ jobs:
|
|||
run: ./packaging/build-installer.sh --no-build
|
||||
|
||||
- name: Package the binaries
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
run: |
|
||||
version=$(sed -n '/^\[workspace\.package\]/,/^\[/{ s/^version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p }' Cargo.toml)
|
||||
[ -n "$version" ] || { echo "could not read the version from Cargo.toml" >&2; exit 1; }
|
||||
|
|
@ -438,12 +354,6 @@ jobs:
|
|||
# node16, which current Forgejo runner images no longer ship; the
|
||||
# -node20 tags are Forgejo's builds for precisely this combination.
|
||||
- uses: actions/upload-artifact@v3-node20
|
||||
# Same gate as the packaging steps, for the same `if-no-files-found`
|
||||
# reason as the linux job.
|
||||
if: >-
|
||||
startsWith(github.ref, 'refs/tags/v')
|
||||
|| startsWith(github.ref, 'refs/heads/Release')
|
||||
|| startsWith(github.ref, 'refs/heads/release')
|
||||
with:
|
||||
name: windows-x86_64
|
||||
path: |
|
||||
|
|
|
|||
159
Cargo.lock
generated
159
Cargo.lock
generated
|
|
@ -1112,6 +1112,7 @@ dependencies = [
|
|||
"egui",
|
||||
"enum-map",
|
||||
"log",
|
||||
"mime_guess2",
|
||||
"profiling",
|
||||
]
|
||||
|
||||
|
|
@ -1221,6 +1222,7 @@ dependencies = [
|
|||
"bytemuck",
|
||||
"ecolor",
|
||||
"emath",
|
||||
"epaint_default_fonts",
|
||||
"log",
|
||||
"nohash-hasher",
|
||||
"parking_lot",
|
||||
|
|
@ -1228,6 +1230,12 @@ dependencies = [
|
|||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "epaint_default_fonts"
|
||||
version = "0.32.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1537accc50c9cab5a272c39300bdd0dd5dca210f6e5e8d70be048df9596e7ca2"
|
||||
|
||||
[[package]]
|
||||
name = "equivalent"
|
||||
version = "1.0.1"
|
||||
|
|
@ -2070,6 +2078,15 @@ dependencies = [
|
|||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "kamadak-exif"
|
||||
version = "0.5.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef4fc70d0ab7e5b6bafa30216a6b48705ea964cdfc29c050f2412295eba58077"
|
||||
dependencies = [
|
||||
"mutate_once",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "keyboard-types"
|
||||
version = "0.7.0"
|
||||
|
|
@ -2320,6 +2337,18 @@ dependencies = [
|
|||
"unicase",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "mime_guess2"
|
||||
version = "2.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1706dc14a2e140dec0a7a07109d9a3d5890b81e85bd6c60b906b249a77adf0ca"
|
||||
dependencies = [
|
||||
"mime",
|
||||
"phf",
|
||||
"phf_shared",
|
||||
"unicase",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "miniz_oxide"
|
||||
version = "0.8.9"
|
||||
|
|
@ -2352,6 +2381,12 @@ dependencies = [
|
|||
"pxfm",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "mutate_once"
|
||||
version = "0.1.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13d2233c9842d08cfe13f9eac96e207ca6a2ea10b80259ebe8ad0268be27d2af"
|
||||
|
||||
[[package]]
|
||||
name = "naga"
|
||||
version = "25.0.1"
|
||||
|
|
@ -2940,6 +2975,50 @@ version = "2.3.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e"
|
||||
|
||||
[[package]]
|
||||
name = "phf"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078"
|
||||
dependencies = [
|
||||
"phf_macros",
|
||||
"phf_shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_generator"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d"
|
||||
dependencies = [
|
||||
"phf_shared",
|
||||
"rand 0.8.5",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_macros"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f84ac04429c13a7ff43785d75ad27569f2951ce0ffd30a3321230db2fc727216"
|
||||
dependencies = [
|
||||
"phf_generator",
|
||||
"phf_shared",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.66",
|
||||
"unicase",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_shared"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5"
|
||||
dependencies = [
|
||||
"siphasher",
|
||||
"unicase",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pin-project"
|
||||
version = "1.1.10"
|
||||
|
|
@ -3059,7 +3138,7 @@ version = "3.5.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f"
|
||||
dependencies = [
|
||||
"toml_edit",
|
||||
"toml_edit 0.25.13+spec-1.1.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3109,7 +3188,7 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "quicksearch-core"
|
||||
version = "1.1.0"
|
||||
version = "1.0.6"
|
||||
dependencies = [
|
||||
"argon2",
|
||||
"cfb",
|
||||
|
|
@ -3120,6 +3199,7 @@ dependencies = [
|
|||
"getrandom 0.2.15",
|
||||
"globset",
|
||||
"infer",
|
||||
"kamadak-exif",
|
||||
"libc",
|
||||
"lofty",
|
||||
"memchr",
|
||||
|
|
@ -3142,7 +3222,7 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "quicksearch-gui"
|
||||
version = "1.1.0"
|
||||
version = "1.0.6"
|
||||
dependencies = [
|
||||
"ashpd",
|
||||
"chrono",
|
||||
|
|
@ -3185,6 +3265,15 @@ version = "6.0.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.8.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
|
||||
dependencies = [
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.9.5"
|
||||
|
|
@ -3532,11 +3621,11 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "serde_spanned"
|
||||
version = "1.1.1"
|
||||
version = "0.6.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6662b5879511e06e8999a8a235d848113e942c9124f211511b16466ee2995f26"
|
||||
checksum = "79e674e01f999af37c49f70a6ede167a8a60b2503e56c5599532a65baa5969a0"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3587,6 +3676,12 @@ version = "0.1.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
|
||||
|
||||
[[package]]
|
||||
name = "siphasher"
|
||||
version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
|
||||
|
||||
[[package]]
|
||||
name = "slab"
|
||||
version = "0.4.9"
|
||||
|
|
@ -3891,17 +3986,23 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
|
|||
|
||||
[[package]]
|
||||
name = "toml"
|
||||
version = "1.1.3+spec-1.1.0"
|
||||
version = "0.8.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "53c96ecdfa941c8fc4fcaed14f99ada8ebed502eef533015095a07e3301d4c3c"
|
||||
checksum = "185d8ab0dfbb35cf1399a6344d8484209c088f75f8f68230da55d48d95d43e3d"
|
||||
dependencies = [
|
||||
"indexmap",
|
||||
"serde_core",
|
||||
"serde",
|
||||
"serde_spanned",
|
||||
"toml_datetime",
|
||||
"toml_parser",
|
||||
"toml_writer",
|
||||
"winnow 1.0.4",
|
||||
"toml_datetime 0.6.3",
|
||||
"toml_edit 0.20.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_datetime"
|
||||
version = "0.6.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7cda73e2f1397b1262d6dfdcef8aafae14d1de7748d66822d3bfeeb6d03e5e4b"
|
||||
dependencies = [
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3913,6 +4014,19 @@ dependencies = [
|
|||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_edit"
|
||||
version = "0.20.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "396e4d48bbb2b7554c944bde63101b5ae446cff6ec4a24227428f15eb72ef338"
|
||||
dependencies = [
|
||||
"indexmap",
|
||||
"serde",
|
||||
"serde_spanned",
|
||||
"toml_datetime 0.6.3",
|
||||
"winnow 0.5.40",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_edit"
|
||||
version = "0.25.13+spec-1.1.0"
|
||||
|
|
@ -3920,7 +4034,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "6975367e4d2ef766d86af01ffad14b622fecc8d4357a998fbc4deb6e9bacaf9b"
|
||||
dependencies = [
|
||||
"indexmap",
|
||||
"toml_datetime",
|
||||
"toml_datetime 1.1.1+spec-1.1.0",
|
||||
"toml_parser",
|
||||
"winnow 1.0.4",
|
||||
]
|
||||
|
|
@ -3934,12 +4048,6 @@ dependencies = [
|
|||
"winnow 1.0.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_writer"
|
||||
version = "1.1.2+spec-1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7d56353a2a665ad0f41a421187180aab746c8c325620617ad883a99a1cbe66d2"
|
||||
|
||||
[[package]]
|
||||
name = "tracing"
|
||||
version = "0.1.40"
|
||||
|
|
@ -4900,6 +5008,15 @@ dependencies = [
|
|||
"xkbcommon-dl",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "winnow"
|
||||
version = "0.5.40"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f593a95398737aeed53e489c785df13f3618e41dbcd6718c6addbf1395aa6876"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "winnow"
|
||||
version = "0.7.15"
|
||||
|
|
|
|||
99
Cargo.toml
99
Cargo.toml
|
|
@ -5,91 +5,28 @@ members = [
|
|||
"crates/quicksearch-gui",
|
||||
]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.1.0"
|
||||
edition = "2021"
|
||||
license = "GPL-3.0-or-later"
|
||||
authors = ["Jeremy <jeremy@karsttech.com>"]
|
||||
repository = "https://code.karsttech.com/jeremy/quick_search.git"
|
||||
|
||||
# Fat LTO, one codegen unit. This reverses an earlier decision, so both
|
||||
# measurements are kept: the old one was right about what it measured, and it
|
||||
# measured the wrong axis on the wrong kind of build.
|
||||
#
|
||||
# WHAT WAS MEASURED BEFORE, against an *incremental* rebuild of the GUI after
|
||||
# touching core (10.2 s at the defaults):
|
||||
# No `[profile.release]` on purpose. Cross-crate inlining looks like it should
|
||||
# pay here — the cascade calls into `snippet`/`query`, both into `memchr` and
|
||||
# `zstd`, everything into `rusqlite`'s FFI wrappers — so it was measured, and
|
||||
# it does not. Against an incremental rebuild of the GUI after touching core
|
||||
# (10.2 s at the defaults):
|
||||
#
|
||||
# lto=thin 67 s search cold 29.9 ms warm best 14.77 ms index cold 459 ms
|
||||
# lto=thin,cgu=1 152 s (build cost alone ruled it out)
|
||||
# lto=fat,cgu=1 200 s search cold 29.2 ms warm best 14.57 ms index cold 479 ms
|
||||
# (defaults) 10 s search cold 30.0 ms warm best 14.92 ms index cold 478 ms
|
||||
#
|
||||
# It concluded that the runtime gain was at the noise floor against a 6.6-19.5x
|
||||
# longer build. Two things were wrong with that as a decision:
|
||||
# Every runtime column moves 0-4% and index-cold is not even monotonic, so the
|
||||
# gain is at the noise floor while the build is 6.6-19.5x longer — on a CI that
|
||||
# builds three targets. Re-measure before concluding otherwise.
|
||||
#
|
||||
# * It never weighed SIZE, which turns out to be where the effect is.
|
||||
# * A 10 s incremental rebuild is the worst possible denominator for a
|
||||
# link-time optimization, because the LTO link is nearly the whole cost and
|
||||
# there is no compile phase to amortize it against. CI never builds that
|
||||
# way; it builds clean.
|
||||
#
|
||||
# RE-MEASURED CLEAN, on 6 cores, stripped `target/*/quicksearch`:
|
||||
#
|
||||
# stripped vs def search cold warm best index cold
|
||||
# (defaults) 28,248,272 - 28.2 ms 14.7 ms 445 ms
|
||||
# cgu=1 25,909,616 -8.3% 28.6 ms 14.8 ms 452 ms
|
||||
# lto=thin 28,233,296 -0.1% (not run: no size effect)
|
||||
# lto=thin,cgu=1 25,991,536 -8.0% (not run: worse than cgu=1 alone)
|
||||
# lto=fat,cgu=1 24,284,912 -14.0% 27.3 ms 14.1 ms 443 ms
|
||||
#
|
||||
# Search is best of three runs; the spreads overlap, so read fat as "3-4%
|
||||
# faster or even", never as a regression. It is both the fastest and by far the
|
||||
# smallest, which is why it wins outright.
|
||||
#
|
||||
# Two surprises worth keeping. `codegen-units = 1` ALONE is worth 2.3 MB - the
|
||||
# old table never isolated it, because every cgu=1 row there also carried LTO.
|
||||
# And `lto = "thin"` alone is worth nothing at all (0.1%), while thin+cgu=1 is
|
||||
# slightly *worse* than cgu=1 by itself. Thin is not a cheaper fat here; it is
|
||||
# a different, useless thing.
|
||||
#
|
||||
# BUILD COST, clean, 6 cores. The second column is what CI actually pays on top
|
||||
# of the binaries, since `cargo test --release --workspace` links every test,
|
||||
# example and bin target - `--release` is `--profile=release`, so this profile
|
||||
# is theirs too:
|
||||
#
|
||||
# bins + all test targets total
|
||||
# (defaults) 143 s 28 s 171 s
|
||||
# cgu=1 158 s 41 s 199 s
|
||||
# lto=fat,cgu=1 198 s 100 s 298 s
|
||||
#
|
||||
# So +127 s on a clean CI run, not the tens of minutes the incremental figure
|
||||
# above implies. Test targets link cheaply because they are small; it is the
|
||||
# 320-crate dependency compile that dominates, and cgu=1 spreads across cores.
|
||||
#
|
||||
# WHAT IS DELIBERATELY NOT SET:
|
||||
#
|
||||
# `strip` would save nothing shipped. The packaging scripts and ci.yml strip the
|
||||
# staged *copy*, which keeps `target/release/` symbolised for `perf` and for the
|
||||
# RUST_BACKTRACE=1 release-test backtraces CI prints, and keeps `--no-strip` on
|
||||
# build-deb.sh and build-appimage.sh meaning something. Setting it here would
|
||||
# move the same 6.5 MB saving to a place where it costs debuggability.
|
||||
#
|
||||
# `panic = "abort"` is the largest win left - it would delete `.gcc_except_table`
|
||||
# (428 KB after LTO) and most of `.eh_frame` (1.33 MB) - and is unavailable for
|
||||
# two independent reasons. `extract/pdf.rs` runs `pdf-extract` inside
|
||||
# `catch_unwind`, so aborting would turn a malformed PDF into a killed process
|
||||
# instead of one skipped file. And cargo forces every dependency to rebuild with
|
||||
# unwind when building tests under an abort profile, so `cargo test --release`
|
||||
# would compile the whole graph a second time.
|
||||
#
|
||||
# Non-PIE would remove most of `.rela.dyn` (1.37 MB) and is rejected on
|
||||
# hardening grounds: this program parses arbitrary user PDFs with a crate known
|
||||
# to panic on malformed input, which is the last place to give up ASLR.
|
||||
#
|
||||
# The vendored OpenSSL (~1.3 MB of `.text`, via SQLCipher) is unreachable by any
|
||||
# profile knob - `openssl-src` pins its own `-O2` - and dropping it would add a
|
||||
# runtime `libcrypto` dependency, which is exactly what the AppImage's
|
||||
# bundles-no-libraries invariant forbids.
|
||||
[profile.release]
|
||||
lto = "fat"
|
||||
codegen-units = 1
|
||||
# `panic = "abort"` is separately unavailable: `extract/pdf.rs` runs
|
||||
# `pdf-extract` inside `catch_unwind`, so aborting would turn a malformed PDF
|
||||
# into a killed process instead of one skipped file.
|
||||
|
||||
[workspace.package]
|
||||
version = "1.0.6"
|
||||
edition = "2021"
|
||||
license = "GPL-3.0-or-later"
|
||||
authors = ["Jeremy <jeremy@karsttech.com>"]
|
||||
repository = "https://code.karsttech.com/jeremy/quick_search.git"
|
||||
|
|
|
|||
202
README.md
202
README.md
|
|
@ -187,9 +187,7 @@ The install is per-machine and asks for elevation. Into
|
|||
| `uninstall.exe` | written by the installer; Add/Remove Programs runs it |
|
||||
|
||||
The components page offers a Start menu shortcut (on) and a desktop shortcut
|
||||
(off); both are created for all users. The final page lists what was installed
|
||||
and where — the install itself takes about a second, which without saying so
|
||||
reads as a failure — and offers to start QuickSearch, ticked. No `config.toml` is installed, for the
|
||||
(off); both are created for all users. No `config.toml` is installed, for the
|
||||
same reason the `.deb` ships none — one next to the binaries is portable mode
|
||||
(see [Configuration](#configuration)) and would override the personal config
|
||||
of every account. The app writes `%APPDATA%\quicksearch\config.toml` on first
|
||||
|
|
@ -226,32 +224,12 @@ inside that folder.
|
|||
`quicksearch` with no query arguments opens the app:
|
||||
|
||||
- **Search**: results appear as you type; every keystroke cancels the
|
||||
previous search. One checkbox enables the two fuzzy passes, and once a
|
||||
search has finished a button inside the right of the search box re-runs
|
||||
it. Click a column heading to sort by it; **right-click any heading to
|
||||
choose which columns are shown** — the path is always there, and size
|
||||
and modified date start hidden, which is what buys the width the path
|
||||
and the match get instead. The choice is saved (`[search.columns]`,
|
||||
also in Settings → Search) and applies immediately. Sorting by a column
|
||||
you then hide falls back to rank. Double-click a result to open it;
|
||||
previous search. One checkbox enables the two fuzzy passes. Sort by
|
||||
rank, name, path, size, or modified. Double-click a result to open it;
|
||||
right-click it to reveal it in the file manager, open it, copy its
|
||||
path, or build an ignore filter from it (session-only by default,
|
||||
optionally persisted to the config). Result text can be selected and
|
||||
copied in place. A match in a file's **name** or **path** is
|
||||
highlighted in that column; a match in its **contents** shows a
|
||||
highlighted snippet in the Content Match column, with more of the
|
||||
surrounding text on hover. Rows matched on name or path show a dash
|
||||
there instead. With `[search] live_results` on (the default) the rows
|
||||
actually on screen are watched, and what they show is read from the
|
||||
files themselves: a rename, a deletion or an edit lands within a
|
||||
second, whether or not indexing is running. The rows coming on screen
|
||||
are also checked against the disk as they are watched, so one the index
|
||||
was already out of date about corrects itself; the index is then
|
||||
brought back in line for those files alone. Over a network share, where
|
||||
the system reports no events, that check is all you get — the row is
|
||||
right when it comes on screen and then holds still. Nothing is ever
|
||||
added, removed or re-ordered underneath you; a file that disappears is
|
||||
struck through where it sits. Editing the query drops every watch.
|
||||
copied in place. Matches in file contents show highlighted snippets.
|
||||
- **Manage Index**: full indexing status, Start/Stop/Automatic controls,
|
||||
indexed folder list, full-text extension filters, ignore patterns, and
|
||||
the indexing options. Stopping switches to manual mode and saves that
|
||||
|
|
@ -266,45 +244,25 @@ inside that folder.
|
|||
folder nothing has finished indexing reads "not yet indexed" rather
|
||||
than zero, and because the figures come from completed runs they do
|
||||
not move as live updates apply single changes in between.
|
||||
- **Duplicates**: files sharing a content hash, grouped. That hash covers
|
||||
each file's size and its first `processing.hash_length` bytes and nothing
|
||||
else, which is the whole reason indexing is affordable — and the reason a
|
||||
group is a strong suspicion rather than a fact. Right-click a group, or any
|
||||
file in one, to settle it: every member is read through and compared byte
|
||||
for byte, with progress and a Cancel button in a modal that then names each
|
||||
file as identical, differing at a given byte, a different size, or
|
||||
unreadable. Nothing is deleted or changed either way; the point is to know
|
||||
before you delete something yourself.
|
||||
- **Duplicates**: files sharing a content hash, grouped.
|
||||
- **Logs**: the lines the app would have printed to a terminal — warnings
|
||||
from indexing, folder watching and opening files, newest last, with a
|
||||
filter box and Copy button. Launched from a desktop launcher (or on
|
||||
Windows, where the app has no console at all) this is the only place
|
||||
they are visible.
|
||||
- **Help**: an in-app quickstart — first indexing run, example queries,
|
||||
what each tab does — pointing here for everything technical. A brand-new
|
||||
installation is shown a short click-through introduction covering the
|
||||
same ground on its first launch; the Help tab brings it back. Upgrading
|
||||
into this version does not raise it (see `[ui] tutorial_seen`).
|
||||
- **Settings**: every configuration control in one place — the database
|
||||
path, indexing and processing limits, search behaviour, the interface
|
||||
(scale, shortcut, color scheme) and password protection. Each row
|
||||
explains itself on hover. Edits are staged and applied together by
|
||||
**Apply & Save**; leaving the tab with unapplied edits asks first. The
|
||||
column choices and the password controls are the exceptions, acting the
|
||||
moment they are used, since the Search tab's own header menu writes the
|
||||
same settings. The indexed folder list and the indexing mode live on
|
||||
Manage Index instead, next to the controls that act on them.
|
||||
what each tab does — pointing here for everything technical.
|
||||
|
||||
**Ctrl+Shift+F from anywhere** brings QuickSearch to the front, restoring
|
||||
it if it was minimized, and puts the cursor in the search box with the
|
||||
previous search selected, so the next thing you type is the new one. The
|
||||
Settings tab's Interface section rebinds it — click the button and press
|
||||
Options window's Interface section rebinds it — click the button and press
|
||||
the keys — or switches it off. It is a system-wide shortcut, registered
|
||||
with Windows or with the X server, so it works while another application
|
||||
has focus. Wayland does not let an application claim a key, so there the
|
||||
shortcut is registered with your desktop through the XDG desktop portal
|
||||
instead; your desktop then has the final say over which key it is, and its
|
||||
own keyboard settings are where to change it. The Settings tab says which
|
||||
own keyboard settings are where to change it. The Options window says which
|
||||
key it settled on. Wayland likewise gives no application a way to put itself
|
||||
in front of what you are doing, so under it the shortcut selects the Search
|
||||
tab and the search box but leaves raising the window to the desktop; on X11
|
||||
|
|
@ -343,7 +301,7 @@ processing; Windows Terminal has it, and older consoles get plain text.
|
|||
|
||||
The index contains the names and (by default) the full text of everything
|
||||
it indexes — for most setups, your entire home directory. That is a lot of
|
||||
concentrated risk in one file. **Settings → Security → Enable password
|
||||
concentrated risk in one file. **Options → Security → Enable password
|
||||
protection** encrypts the index on disk with SQLCipher; from then on
|
||||
QuickSearch asks for the password every time it starts, in the GUI (an
|
||||
unlock screen before anything opens the index) and in the terminal (a
|
||||
|
|
@ -357,10 +315,6 @@ rebuilds the index — there is no in-place conversion.
|
|||
in the OS keychain — Secret Service/KWallet on Linux, Credential Manager
|
||||
on Windows — and skips the prompt. Without a keychain daemon the option
|
||||
quietly falls back to prompting.
|
||||
- **Show database key** asks for the password, then shows the raw SQLCipher
|
||||
key as `0x…` (64 hex digits) with a copy button, for opening the index in
|
||||
other SQLCipher tools. That key alone reads the index, so treat a copy of
|
||||
it as carefully as the password.
|
||||
- Scripts can set `QUICKSEARCH_PASSWORD` for non-interactive terminal
|
||||
search. Environment variables are readable by other processes of the
|
||||
same user (`/proc/<pid>/environ`) — prefer the keychain.
|
||||
|
|
@ -443,13 +397,13 @@ containing the binary, its config, and its index can be moved wholesale.
|
|||
The GUI edits the config live; external edits apply on next start.
|
||||
|
||||
`[ui] search_hotkey` is the system-wide search shortcut, written the way
|
||||
the Settings tab prints it (`Ctrl+Shift+F`): Ctrl, Alt and Shift in any
|
||||
the Options window prints it (`Ctrl+Shift+F`): Ctrl, Alt and Shift in any
|
||||
combination, plus one key, joined with `+`. An empty string switches it
|
||||
off. A value that is not a shortcut is not a config error — the app loads,
|
||||
says so on the Settings tab, and runs without one.
|
||||
says so in the Options window, and runs without one.
|
||||
|
||||
`[ui] color_scheme` is `dark` (the default) or `light`, changeable on the
|
||||
Settings tab and applied without a restart. It does not follow the
|
||||
`[ui] color_scheme` is `dark` (the default) or `light`, changeable in the
|
||||
Options window and applied without a restart. It does not follow the
|
||||
desktop's own light/dark setting: on Linux nothing in the window system
|
||||
reports that, so the only way to know is to connect to the session message
|
||||
bus and subscribe to the user's settings feed — more of your session than a
|
||||
|
|
@ -492,15 +446,11 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
|||
log, because SQLite's own autocheckpoint can only reset the log at an
|
||||
instant no reader holds it — and a run keeps a reader per root querying
|
||||
throughout, so left alone the log grows for the whole run. `files` holds
|
||||
metadata (name, path, size, mtime, hash, MIME/type bitmask, content
|
||||
state); `searchabletext` is a *contentless* FTS5 table over one column,
|
||||
the document body (postings only, configurable tokenizer, trigram by
|
||||
default) — filename ranks come from scanning `files.name`, so a `name`
|
||||
column there would only index the same strings twice; canonical extracted
|
||||
metadata (name, path, size, mtime, hash, MIME/type bitmask, per-row
|
||||
index state); `searchabletext` is a *contentless* FTS5 table (postings
|
||||
only, configurable tokenizer, trigram by default); canonical extracted
|
||||
text lives zstd-compressed in `documents_text`, which powers snippets,
|
||||
occurrence ranking, and fuzzy full-text search, and whose uncompressed
|
||||
length is read back from the zstd frame header rather than stored beside
|
||||
it. Schema changes wipe and
|
||||
occurrence ranking, and fuzzy full-text search. Schema changes wipe and
|
||||
rebuild by policy; the indexer (`open_or_recreate`) is the only code
|
||||
allowed to do that; every consumer uses `open_existing`, which treats
|
||||
drift as an error, never data loss. With password protection on, every
|
||||
|
|
@ -527,14 +477,11 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
|||
classify files by mtime into insert/update/skip, batch-write metadata,
|
||||
sweep stale rows, then extract content (plaintext, RTF, Office — both the
|
||||
OOXML/ODF zip formats and the pre-2007 binary `.doc`/`.xls`/`.ppt`, whose
|
||||
OLE2 streams are read in `extract/ole.rs` — PDF, audio tags; see
|
||||
`extract/`) for FTS. Images are claimed by no extractor: the EXIF reader
|
||||
produced structured properties and never text, and with properties parked
|
||||
(see `extract::ExtractedContent`) leaving `image/*` unclaimed is what keeps
|
||||
the content pass from opening every image on disk. PDFs are parsed once:
|
||||
the two-parse version that preceded it was the largest single memory
|
||||
consumer of a run over a PDF-heavy tree, and it was what pulled a second
|
||||
copy of `lopdf` — and with it rayon's
|
||||
OLE2 streams are read in `extract/ole.rs` — PDF, audio tags, EXIF; see
|
||||
`extract/`) for FTS. PDFs are parsed once, with the text and the `Info`
|
||||
dictionary taken off the same document: the two-parse version that preceded
|
||||
it was the largest single memory consumer of a run over a PDF-heavy tree, and
|
||||
it was what pulled a second copy of `lopdf` — and with it rayon's
|
||||
never-torn-down thread pool — into the build. That is a claim about PDFs
|
||||
rather than about runs in general, and it is worth knowing which tree a
|
||||
number came from: on one with almost no PDFs, a cold run peaks at 130 MiB
|
||||
|
|
@ -552,25 +499,7 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
|||
bigger index — `indexing.content_extensions` remains the throttle. Files no larger than `processing.hash_length` skip that second
|
||||
pass entirely: the head the walk reads to hash them is already their
|
||||
whole content, so a plaintext body is extracted in the same `read` and
|
||||
stored complete. Every root runs its own pipeline — its own walker pool
|
||||
and, once the walk ends, its own extraction pool — but every root's
|
||||
*writes* go through one thread and one connection
|
||||
(`indexing/pipeline.rs`), because that is what a single SQLite file
|
||||
allows. That thread is where FTS5 tokenizes, up to `maximum_text_size`
|
||||
of text per document inside the insert, and it is the run's dominant
|
||||
cost. So its loop is scheduled around the walk, the disk-bound phase and
|
||||
the one whose stall shows: each round serves every walking root first,
|
||||
then one extracting root, and no turn runs past a 100 ms slice — an
|
||||
extraction turn commits at the slice and carries the rows it did not
|
||||
reach to its next turn. A walk therefore waits at most one slice per
|
||||
round, which its walkers' channel absorbs, so a root walking a large tree
|
||||
runs at its own rate while another root tokenizes big documents beside
|
||||
it. What a root has left to extract is counted by its own content pass,
|
||||
on that pass's read connection, rather than on the writer: on a large
|
||||
root the count is seconds, and seconds of writer time is every other
|
||||
root's walk standing still. Total write throughput is what one connection
|
||||
tokenizing can do; the scheduling shares it fairly and keeps the walk
|
||||
first, it does not raise it. Every run ends — whether
|
||||
stored complete. Every run ends — whether
|
||||
it completed or was stopped — with an optimize pass on its own connection:
|
||||
checkpoint, VACUUM if the file has at least 10% slack to reclaim, `PRAGMA
|
||||
optimize`, checkpoint again. Progress streams through a polled
|
||||
|
|
@ -626,26 +555,6 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
|||
handle and takes a single watch per root, filtering the events instead.
|
||||
Either way a tree too large to watch degrades to periodic reindexing
|
||||
rather than going silently stale.
|
||||
- **Live results** (`live.rs`): a second, much smaller watcher, owned by the
|
||||
frontend rather than the coordinator, pointed at the parent directories of
|
||||
the result rows *currently on screen* once they have held still for a
|
||||
moment. It watches directories, not the result files: editors save by
|
||||
writing a temporary file and renaming it over the target, so the event
|
||||
lands on the directory and a watch on the file is left holding an orphaned
|
||||
inode. What a row shows is read from the **file**, never from the index —
|
||||
metadata from `stat`, and for a content match the same MIME sniffing and
|
||||
extractors the indexer uses, re-cut through the same `cascade::text_snippet`
|
||||
the search itself does. That is what makes it work with indexing stopped.
|
||||
Arming also sweeps each target once against the size and modified time the
|
||||
row is displaying, which on a fresh result is what the index said: so
|
||||
bringing a row on screen *is* a check of the index against the disk, and it
|
||||
is the only thing that reports anything where the platform sends no events.
|
||||
It still writes nothing itself; the paths it has just read go to
|
||||
`IndexCoordinator::update_paths`, which applies them on the coordinator's
|
||||
own thread — in any mode, so a stopped index does not drift from the screen
|
||||
— leaving the single-writer rule intact. Caps at 64 directories and 256
|
||||
rows, rate-limited per path, and dropped wholesale the moment the query is
|
||||
edited.
|
||||
- **Search** (`search/`): `SearchService` runs one worker thread; each
|
||||
query is a *generation*. New queries interrupt the in-flight SQLite
|
||||
statement (`InterruptHandle`) and stale generations stop cooperatively,
|
||||
|
|
@ -689,21 +598,6 @@ Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
|
|||
case-insensitive literal branch used to allocate a lowercased copy of its
|
||||
haystack, which the filename pass asked for twice per row of a full-table
|
||||
scan.
|
||||
- **Duplicate verification** (`verify.rs`): the second opinion on a group from
|
||||
`search/duplicates.rs`, which groups by `sha256(size ‖ head)` and so cannot
|
||||
tell two pre-allocated disk images apart — same size, same zeroes at the
|
||||
front, everything that distinguishes them in a footer. One lockstep pass:
|
||||
open every member, drop the ones whose length already disagrees without
|
||||
reading them, then read a chunk from the first that opened and the same
|
||||
span from each of the others, reporting the offset of the first byte that
|
||||
differs and dropping that file from the walk. Deliberately not a hash —
|
||||
"the same digest" is a probabilistic answer, and a probabilistic answer is
|
||||
what the head hash already gave. The reference is the first member that
|
||||
*opens*, so one unreadable file costs its own verdict and nobody else's,
|
||||
and termination follows what that file actually reads rather than the
|
||||
length it claimed, so a file truncated mid-run degrades to a short
|
||||
comparison. The read buffers share a fixed 8 MiB between them however many
|
||||
members a group has, because a hardlink farm's group runs to thousands.
|
||||
- **Baloo compatibility** (`cli.rs`, `mime.rs`): the read API this repo's
|
||||
parent consumes — `status_for_path`, `list_failed`,
|
||||
`index_size_breakdown`, `pending_content_count`, `clear_path` — plus a
|
||||
|
|
@ -736,17 +630,14 @@ core threads ─────────────▶ ctx.request_repaint() (w
|
|||
```
|
||||
|
||||
Modules map one-to-one onto what you see: `app.rs` (shell and config
|
||||
routing, with `app/` submodules for the status bar, the security flow, the
|
||||
confirmation modals and the duplicate-verification modal — the one place a
|
||||
worker's progress is shown in a window rather than the status bar),
|
||||
`search_tab.rs` (query strip and virtualized
|
||||
routing, with `app/` submodules for the status bar, the security flow and
|
||||
the confirmation modals), `search_tab.rs` (query strip and virtualized
|
||||
results table; snippet rendering via `LayoutJob` byte ranges, the ignore
|
||||
dialog and the syntax help live in `search_tab/`), `manage_tab.rs` (status
|
||||
detail + `tracker.rs` rate estimation, roots and filter editors),
|
||||
`duplicates_tab.rs`, `logs_tab.rs` (a virtualized view of the core log
|
||||
ring), `settings_tab.rs` (the draft-based config editor, the second of the
|
||||
two tabs that stage their edits behind an Apply & Save), `platform.rs`
|
||||
(open / reveal-in-file-manager, and the
|
||||
ring), `options.rs` (draft-based settings editor shared between the window
|
||||
and the Manage tab), `platform.rs` (open / reveal-in-file-manager, and the
|
||||
Windows stdio setup a window-subsystem process needs before anything
|
||||
prints), `hotkey/` (the system-wide search shortcut: one key table feeding
|
||||
both a `RegisterHotKey` / `XGrabKey` registration and, on Wayland, an XDG
|
||||
|
|
@ -759,26 +650,12 @@ microseconds regardless of row count.
|
|||
|
||||
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
|
||||
ranking, cancellation, incremental indexing, coordinator modes, config
|
||||
resolution, fuzzy matcher vs. brute-force oracle, `verify.rs`'s byte-for-byte
|
||||
comparison — the shared-head-different-tail case the head hash cannot see, an
|
||||
unreadable first member, a difference past the first chunk, cancellation —
|
||||
and `live.rs`'s event
|
||||
classification, where the platform-specific rename and atomic-save shapes are
|
||||
synthesized rather than provoked, so they are checked on every platform).
|
||||
resolution, fuzzy matcher vs. brute-force oracle).
|
||||
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units plus
|
||||
headless egui tests that drive the real widgets — building an input frame,
|
||||
synthesizing clicks and reading back the painted text (`test_ui.rs`) — over
|
||||
the search, manage and settings tabs, the unlock gate, the logs
|
||||
and duplicates tabs, the first-start tour, and query highlighting. The search
|
||||
tab's cover the column picker (including that the path column survives all
|
||||
32 combinations of the others), which column a match is highlighted in, and
|
||||
that the repeat-search button appearing inside the query box does not cost it
|
||||
keyboard focus. The duplicates tab's open the real context menus and click
|
||||
the entries inside them, so "the verification asks for the whole group, from
|
||||
either menu, and not at all while one is running" is checked rather than
|
||||
assumed; the verification modal is rendered in each of its states, and the
|
||||
tour's footer is probed for where its three buttons actually landed rather
|
||||
than for the numbers they were expected to land on.
|
||||
the search and manage tabs, the options editor, the unlock gate, the logs
|
||||
and duplicates tabs, and query highlighting.
|
||||
- `cargo bench -p quicksearch-core --bench search` and `--bench index`: divan
|
||||
microbenchmarks over the two hot paths. Each group runs *what the code does
|
||||
today* against *the change being considered*, in one process on one corpus,
|
||||
|
|
@ -818,26 +695,13 @@ microseconds regardless of row count.
|
|||
allocation counts, measure that separately before concluding a path is cheap,
|
||||
and measure the GUI rather than a one-shot `quicksearch-cli` run — a
|
||||
short-lived process cannot show what a typing session retains.
|
||||
- `.forgejo/workflows/ci.yml`: builds and tests both platforms on every push to
|
||||
`master` and every pull request. Those runs stop there — packaging (the
|
||||
`.deb`, the AppImage, the tarball, the Windows installer and `.zip`, and the
|
||||
artifact upload) runs only where a release can actually come out, which is a
|
||||
`v*` tag or a `Release...` branch. The Windows non-system-DLL check is
|
||||
deliberately not gated that way: it validates the `.exe` rather than
|
||||
packaging it, so it runs everywhere and catches a regression on `master`
|
||||
rather than at release time.
|
||||
|
||||
To cut a release, bump `[workspace.package] version`
|
||||
- `.forgejo/workflows/ci.yml`: builds both platforms on every push to `master`
|
||||
and every pull request. To cut a release, bump `[workspace.package] version`
|
||||
in `Cargo.toml` and push the commit on a branch named `Release...`; CI runs
|
||||
`cargo update -w` first, so a lockfile still pinning the old member versions
|
||||
is not something you have to remember. That only re-resolves the workspace
|
||||
crates, so the `--locked` build after it still fails on a dependency added or
|
||||
bumped without committing `Cargo.lock`. Forgetting the version bump is caught
|
||||
in seconds rather than after two full release builds: each release path gets
|
||||
a guard before anything is compiled — a `v*` tag is checked against the
|
||||
workspace version, and a `Release...` branch is checked against the tags that
|
||||
already exist. Both are accelerators, not the authority; the release job
|
||||
re-checks and remains the thing that actually refuses. Once both
|
||||
bumped without committing `Cargo.lock`. Once both
|
||||
build jobs are green, CI tags that commit `v<version>` and publishes a release
|
||||
with the `.deb`, an AppImage and its `.zsync` sidecar, a Linux tarball, the
|
||||
Windows installer and a Windows zip attached; pushing a `v*` tag by hand does
|
||||
|
|
|
|||
|
|
@ -109,12 +109,6 @@ maximum_text_size = 262144
|
|||
maximum_text_file_size = 2097152
|
||||
# Files per batch during walks / inserts / extraction.
|
||||
batch_size = 500
|
||||
# Writer time one indexing root's turn may take before the round-robin moves
|
||||
# on (milliseconds). The time half of the knob whose row half is batch_size:
|
||||
# it bounds how long one root can hold up the others, so a root extracting
|
||||
# large documents cannot leave another root's walkers parked behind it. 0
|
||||
# gives each turn one batch_size quantum and no more.
|
||||
writer_turn_slice_ms = 100
|
||||
# Files per transaction for incremental FTS updates.
|
||||
fts_update_batch_size = 1000
|
||||
# How large the write-ahead log (index.sqlite-wal) may grow during an
|
||||
|
|
@ -139,7 +133,7 @@ store_text_for_snippets = true
|
|||
[security]
|
||||
# Encrypt the index with a password (SQLCipher). The password is asked
|
||||
# for every time QuickSearch starts; turning this on or off deletes and
|
||||
# rebuilds the index. Change it from the GUI (Settings → Security), not by
|
||||
# rebuilds the index. Change it from the GUI (Options → Security), not by
|
||||
# hand: enabling protection also generates the KDF salt below.
|
||||
password_protected = false
|
||||
# Store the derived key in the OS keychain (Secret Service / KWallet on
|
||||
|
|
@ -169,18 +163,12 @@ watch_cap_warned_roots = []
|
|||
# registered with your desktop, which may assign a different key and lets
|
||||
# you change it in its own keyboard settings.
|
||||
search_hotkey = "Ctrl+Shift+F"
|
||||
# 'dark' or 'light'. Applied as soon as it is changed on the Settings
|
||||
# tab. Your desktop's own light/dark setting is not consulted: reading
|
||||
# 'dark' or 'light'. Applied as soon as it is changed in the Options
|
||||
# window. Your desktop's own light/dark setting is not consulted: reading
|
||||
# it would mean connecting to your session's message bus and subscribing to
|
||||
# your settings, which is more than a search tool should ask for. Anything
|
||||
# other than 'light' is dark.
|
||||
color_scheme = "dark"
|
||||
# Written by QuickSearch, not by you: whether the short introduction shown
|
||||
# on a brand-new installation has been dismissed. Absent means this config
|
||||
# predates that introduction - an installation that upgraded into this
|
||||
# version, which is not offered it. The Help tab can show it again at any
|
||||
# time.
|
||||
tutorial_seen = false
|
||||
|
||||
[search]
|
||||
# Start with the fuzzy passes enabled.
|
||||
|
|
@ -197,32 +185,3 @@ display_limit = 1000
|
|||
results_per_page = 100
|
||||
# How long the GUI waits after the last keystroke before searching (ms).
|
||||
debounce_ms = 150
|
||||
# Watch the search results on screen and show renames, deletions and
|
||||
# content changes as they happen. Only the rows actually visible are
|
||||
# watched, and editing the query drops the watches. What a row shows is
|
||||
# read from the file itself, so this works whether or not indexing is
|
||||
# running — and the files it reads are then brought up to date in the
|
||||
# index, so what is stored cannot drift from what you are looking at.
|
||||
# Rows are also checked against the disk as they come on screen, which is
|
||||
# all you get over a network share, where the system does not report other
|
||||
# machines' writes. Nothing is ever added, removed or re-ordered while you
|
||||
# read.
|
||||
live_results = true
|
||||
|
||||
# Which columns the Search tab shows. The same choices are on the
|
||||
# right-click menu of any column header, and in Settings → Search; both
|
||||
# write here immediately, without an Apply.
|
||||
#
|
||||
# There is deliberately no 'path' key: the path is always shown, because
|
||||
# it is the only column that identifies a result on its own.
|
||||
[search.columns]
|
||||
name = true
|
||||
# The excerpt of a file's contents around the match. Rows that matched on
|
||||
# their name or path show a dash there instead.
|
||||
content_match = true
|
||||
# Off by default: the width these take is usually better spent on the path
|
||||
# and the matched text. Turning one on also makes it available to sort by;
|
||||
# sorting by a column that is hidden falls back to sorting by rank.
|
||||
size = false
|
||||
modified = false
|
||||
rank = true
|
||||
|
|
|
|||
|
|
@ -35,12 +35,7 @@ quick-xml = "0.31"
|
|||
# nothing new.
|
||||
cfb = "0.7"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
# 1.x, not 0.8: 0.8 pulls `toml_edit`, the whole format-preserving document
|
||||
# model, for what is only `from_str` and `to_string_pretty` here. 1.x parses and
|
||||
# writes as a stream instead - no `toml_edit`, no `winnow` 0.5 alongside the 1.0
|
||||
# already in the tree - which is 252 KB of `.text` this crate was paying for a
|
||||
# document API it never touches.
|
||||
toml = "1"
|
||||
toml = "0.8"
|
||||
mime_guess = "2.0"
|
||||
infer = "0.15"
|
||||
# Charset decoding for non-UTF-8 text (UTF-16 .reg exports, legacy
|
||||
|
|
@ -61,8 +56,7 @@ rtf-parser = { version = "0.4", default-features = false }
|
|||
# global thread pool is never torn down), chrono, time, md5 and a second nom.
|
||||
pdf-extract = "0.12"
|
||||
lofty = "0.19"
|
||||
# Parked with `extract::image` — see `extract::ExtractedContent`.
|
||||
# kamadak-exif = "0.5"
|
||||
kamadak-exif = "0.5"
|
||||
notify = "6.1"
|
||||
ctrlc = "3.4"
|
||||
zstd = "0.13"
|
||||
|
|
|
|||
|
|
@ -122,6 +122,7 @@ fn parallel(root: &str, config: &Config, db_path: &str) -> (usize, usize) {
|
|||
config.clone(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
) {
|
||||
let WalkEvent::File(file) = event else {
|
||||
|
|
|
|||
|
|
@ -77,6 +77,7 @@ pub struct FailedEntry {
|
|||
pub struct SizeReport {
|
||||
pub file_size_bytes: u64,
|
||||
pub files_row_count: i64,
|
||||
pub properties_row_count: i64,
|
||||
pub failed_files_row_count: i64,
|
||||
pub searchabletext_row_count: i64,
|
||||
pub documents_text_row_count: i64,
|
||||
|
|
@ -97,20 +98,13 @@ impl SizeReport {
|
|||
|
||||
/// Query the per-file indexing status. Returns `FileStatus` with
|
||||
/// `basic == NotIndexed` if the path isn't in the database.
|
||||
///
|
||||
/// There is no stored basic state: a `files` row exists only once its
|
||||
/// metadata has been read, so the row *is* the basic-indexed state. The
|
||||
/// failure reason comes from `failed_files`, the one place it is written.
|
||||
pub fn status_for_path(db_path: &str, path: &str) -> Result<FileStatus, String> {
|
||||
let conn = open_existing(db_path, false)?;
|
||||
let row: Option<(i64, Option<String>)> = conn
|
||||
let row: Option<(i64, i64, Option<String>)> = conn
|
||||
.query_row(
|
||||
"SELECT f.content_state, ff.reason \
|
||||
FROM files f \
|
||||
LEFT JOIN failed_files ff ON ff.file_id = f.id \
|
||||
WHERE f.path = ?1",
|
||||
"SELECT basic_state, content_state, failure_msg FROM files WHERE path = ?1",
|
||||
params![path],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)),
|
||||
)
|
||||
.optional()
|
||||
.map_err(|e| format!("status_for_path({}): {}", path, e))?;
|
||||
|
|
@ -121,9 +115,9 @@ pub fn status_for_path(db_path: &str, path: &str) -> Result<FileStatus, String>
|
|||
content: IndexState::NotIndexed,
|
||||
failure_reason: None,
|
||||
},
|
||||
Some((content, reason)) => FileStatus {
|
||||
Some((basic, content, reason)) => FileStatus {
|
||||
path: path.to_string(),
|
||||
basic: IndexState::Done,
|
||||
basic: IndexState::from(basic),
|
||||
content: IndexState::from(content),
|
||||
failure_reason: reason,
|
||||
},
|
||||
|
|
@ -170,29 +164,18 @@ pub fn index_size_breakdown(db_path: &str) -> Result<SizeReport, String> {
|
|||
.map_err(|e| format!("count {}: {}", table, e))
|
||||
};
|
||||
let dt_row_count: i64 = count("documents_text")?;
|
||||
// The uncompressed length is not a column: zstd records it in each frame's
|
||||
// header, so this reads it back (see `repo::raw_text_len`). Only the
|
||||
// header is wanted, and 18 bytes is the most one can occupy — projecting
|
||||
// the prefix keeps this off the document bodies themselves.
|
||||
let mut stmt = conn
|
||||
.prepare("SELECT substr(text_zstd, 1, 18), LENGTH(text_zstd) FROM documents_text")
|
||||
.map_err(|e| format!("documents_text size sum prepare: {}", e))?;
|
||||
let rows = stmt
|
||||
.query_map([], |r| Ok((r.get::<_, Vec<u8>>(0)?, r.get::<_, i64>(1)?)))
|
||||
let (dt_raw, dt_compressed): (i64, i64) = conn
|
||||
.query_row(
|
||||
"SELECT COALESCE(SUM(text_len), 0), COALESCE(SUM(LENGTH(text_zstd)), 0) FROM documents_text",
|
||||
[],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.map_err(|e| format!("documents_text size sum: {}", e))?;
|
||||
let (mut dt_raw, mut dt_compressed) = (0i64, 0i64);
|
||||
for row in rows {
|
||||
let (header, compressed) = row.map_err(|e| format!("documents_text size row: {}", e))?;
|
||||
// A frame with no recorded content size contributes nothing rather
|
||||
// than skewing the ratio with a guess.
|
||||
dt_raw += crate::db::repo::raw_text_len(&header).unwrap_or(0) as i64;
|
||||
dt_compressed += compressed;
|
||||
}
|
||||
drop(stmt);
|
||||
|
||||
Ok(SizeReport {
|
||||
file_size_bytes,
|
||||
files_row_count: count("files")?,
|
||||
properties_row_count: count("properties")?,
|
||||
failed_files_row_count: count("failed_files")?,
|
||||
searchabletext_row_count: count("searchabletext")?,
|
||||
documents_text_row_count: dt_row_count,
|
||||
|
|
@ -220,7 +203,7 @@ pub fn pending_content_count(db_path: &str) -> Result<i64, String> {
|
|||
}
|
||||
|
||||
/// Remove a single file from the index. Returns whether a row was deleted.
|
||||
/// Keeps FTS and `documents_text` in sync via the repo helpers.
|
||||
/// Keeps FTS/documents/properties in sync via the repo helpers.
|
||||
pub fn clear_path(db_path: &str, path: &str) -> Result<bool, String> {
|
||||
let mut conn = open_existing(db_path, true)?;
|
||||
let tx = conn
|
||||
|
|
@ -256,6 +239,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -264,7 +249,7 @@ mod tests {
|
|||
)
|
||||
.unwrap()
|
||||
.expect("unique path");
|
||||
set_content_done(&tx, a, "hello", zstd_of("hello").as_deref()).unwrap();
|
||||
set_content_done(&tx, a, "a.txt", "hello", &[], zstd_of("hello").as_deref()).unwrap();
|
||||
let b = insert_file(
|
||||
&tx,
|
||||
&NewFile {
|
||||
|
|
@ -273,6 +258,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::EMPTY,
|
||||
hash: None,
|
||||
|
|
@ -347,6 +334,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -365,6 +354,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::EMPTY,
|
||||
hash: None,
|
||||
|
|
@ -426,6 +417,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -435,7 +428,7 @@ mod tests {
|
|||
.unwrap()
|
||||
.expect("unique path");
|
||||
let prose = "the quick brown fox jumps over the lazy dog. ".repeat(500);
|
||||
set_content_done(&tx, id, &prose, zstd_of(&prose).as_deref()).unwrap();
|
||||
set_content_done(&tx, id, "big.txt", &prose, &[], zstd_of(&prose).as_deref()).unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
drop(conn);
|
||||
|
|
@ -486,6 +479,8 @@ mod tests {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
|
|||
|
|
@ -31,8 +31,8 @@ pub struct IndexWork {
|
|||
pub drop_aliases: bool,
|
||||
/// The `content_extensions` filter changed. Kept rows are re-tested
|
||||
/// against it in both directions: newly-included files go back to
|
||||
/// pending, newly-excluded ones give up their text and FTS row but keep
|
||||
/// the name/path row that filename search needs.
|
||||
/// pending, newly-excluded ones give up their text, properties and FTS
|
||||
/// row but keep the name/path row that filename search needs.
|
||||
pub reconcile_content: bool,
|
||||
/// `store_text_for_snippets` turned on. Rows that finished extraction
|
||||
/// under the old setting kept no text, so they must run again.
|
||||
|
|
|
|||
|
|
@ -116,22 +116,6 @@ pub struct ProcessingConfig {
|
|||
pub maximum_text_size: usize,
|
||||
pub maximum_text_file_size: u64,
|
||||
pub batch_size: usize,
|
||||
/// Writer time one root's turn may take before the round moves on, in
|
||||
/// milliseconds. The time half of the round-robin whose row half is
|
||||
/// `batch_size`, and so the bound on how long any one root can hold up
|
||||
/// the others.
|
||||
///
|
||||
/// Before there was one, an extraction turn ran to the end of whatever
|
||||
/// was ready — half a second to two seconds of FTS5 trigram tokenization
|
||||
/// for a batch of large documents — while a walking root's rows sat in
|
||||
/// its channel and its walkers parked behind them. Reads as "4/4 workers
|
||||
/// busy, no progress".
|
||||
///
|
||||
/// `0` gives each turn one `batch_size` quantum and no more, which is
|
||||
/// the finest the round-robin goes; the tests that count work per round
|
||||
/// use small values here so a phase cannot begin and end between two
|
||||
/// status snapshots.
|
||||
pub writer_turn_slice_ms: u64,
|
||||
pub fts_update_batch_size: usize,
|
||||
/// How large the write-ahead log may grow during a run before the indexer
|
||||
/// forces a checkpoint, in bytes. `0` disables forced checkpoints;
|
||||
|
|
@ -185,47 +169,6 @@ pub struct SearchConfig {
|
|||
pub results_per_page: usize,
|
||||
/// How long the GUI waits after the last keystroke before searching.
|
||||
pub debounce_ms: u64,
|
||||
/// Watch the search results currently on screen and show renames,
|
||||
/// deletions and content changes as they happen. Only the rows actually
|
||||
/// visible are watched, and any edit to the query drops the watches.
|
||||
/// What a row shows is read from the file itself, so this holds whether
|
||||
/// or not indexing is running; the files it reads are then brought up to
|
||||
/// date in the index, so what is stored cannot drift from what is on
|
||||
/// screen. See [`crate::live`].
|
||||
pub live_results: bool,
|
||||
/// Which columns the Search tab shows.
|
||||
pub columns: ColumnsConfig,
|
||||
}
|
||||
|
||||
/// Which columns the Search tab shows, as picked from the right-click menu on
|
||||
/// any column header or from the Settings tab.
|
||||
///
|
||||
/// The path column is deliberately not represented: it is always shown, so
|
||||
/// "no columns at all" is not a state this can hold. Size and modified are off
|
||||
/// by default — the width they cost is better spent on the path and the match,
|
||||
/// and both are one click away.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(default)]
|
||||
pub struct ColumnsConfig {
|
||||
pub name: bool,
|
||||
/// The matched excerpt from a file's contents. Rows that matched on their
|
||||
/// name or path instead show a dash there.
|
||||
pub content_match: bool,
|
||||
pub size: bool,
|
||||
pub modified: bool,
|
||||
pub rank: bool,
|
||||
}
|
||||
|
||||
impl Default for ColumnsConfig {
|
||||
fn default() -> Self {
|
||||
ColumnsConfig {
|
||||
name: true,
|
||||
content_match: true,
|
||||
size: false,
|
||||
modified: false,
|
||||
rank: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl SearchConfig {
|
||||
|
|
@ -274,7 +217,6 @@ impl Default for ProcessingConfig {
|
|||
maximum_text_size: 1024 * 256,
|
||||
maximum_text_file_size: 1024 * 1024 * 2,
|
||||
batch_size: 500,
|
||||
writer_turn_slice_ms: 100,
|
||||
fts_update_batch_size: 1000,
|
||||
maximum_wal_size: 1024 * 1024 * 512,
|
||||
tokenize: "trigram".to_string(),
|
||||
|
|
@ -291,8 +233,6 @@ impl Default for SearchConfig {
|
|||
display_limit: 1000,
|
||||
results_per_page: 100,
|
||||
debounce_ms: 150,
|
||||
live_results: true,
|
||||
columns: ColumnsConfig::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -362,21 +302,6 @@ pub struct UiConfig {
|
|||
/// recognises falls back to dark, where a typed-out enum would fail to
|
||||
/// deserialize and take the whole config file down with it.
|
||||
pub color_scheme: String,
|
||||
/// Whether the first-start tour has been dismissed.
|
||||
///
|
||||
/// Three-valued on purpose. `None` means the key predates the tour — an
|
||||
/// installation that upgraded into this version, which has already found
|
||||
/// its way around — so only a config file this version *created* (which
|
||||
/// gets `Some(false)` from [`UiConfig::default`]) is ever offered the tour.
|
||||
/// A plain `bool` could not tell those apart.
|
||||
///
|
||||
/// The field-level `default` is load-bearing and not redundant with the
|
||||
/// `#[serde(default)]` on the struct: that one fills a missing field from
|
||||
/// `UiConfig::default()`, which says `Some(false)` — and would hand every
|
||||
/// upgrading installation the tour. This one fills it from
|
||||
/// `Option::default()`, which is `None`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub tutorial_seen: Option<bool>,
|
||||
}
|
||||
|
||||
impl Default for UiConfig {
|
||||
|
|
@ -386,10 +311,6 @@ impl Default for UiConfig {
|
|||
watch_cap_warned_roots: Vec::new(),
|
||||
search_hotkey: "Ctrl+Shift+F".to_string(),
|
||||
color_scheme: "dark".to_string(),
|
||||
// Not `None`: a config built from these defaults is a config being
|
||||
// written for the first time, and that is exactly who the tour is
|
||||
// for. `None` is reserved for a file that predates the key.
|
||||
tutorial_seen: Some(false),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -985,61 +985,3 @@ fn fuzzy_edits_warning_only_above_the_threshold() {
|
|||
assert!(msg.contains(&FUZZY_EDITS_WARN_ABOVE.to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
/// `config_example.toml` is the documentation for every setting, so a key
|
||||
/// renamed in the struct and not here would silently ship a config file that
|
||||
/// does nothing. Parsing it also proves the `[search.columns]` sub-table is
|
||||
/// spelled the way serde expects.
|
||||
#[test]
|
||||
fn the_documented_example_config_parses_to_the_defaults() {
|
||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("../../config_example.toml")
|
||||
.canonicalize()
|
||||
.expect("config_example.toml sits at the repository root");
|
||||
let text = std::fs::read_to_string(&path).expect("readable");
|
||||
let parsed: Config = toml::from_str(&text).expect("config_example.toml parses");
|
||||
|
||||
// The example documents the shipped defaults for everything that has one
|
||||
// that does not depend on the machine (paths and the hotkey do).
|
||||
let d = Config::default();
|
||||
assert_eq!(
|
||||
parsed.search, d.search,
|
||||
"[search] drifted from the defaults"
|
||||
);
|
||||
assert_eq!(parsed.processing, d.processing);
|
||||
assert_eq!(parsed.ui.scale, d.ui.scale);
|
||||
assert_eq!(parsed.ui.color_scheme, d.ui.color_scheme);
|
||||
assert_eq!(parsed.ui.tutorial_seen, Some(false));
|
||||
}
|
||||
|
||||
/// The tour is offered to an installation this version created, and to no
|
||||
/// other. A config written before the key existed reads as `None`, which is
|
||||
/// how "already found their way around" is distinguished from "brand new".
|
||||
#[test]
|
||||
fn only_a_freshly_written_config_asks_for_the_tour() {
|
||||
assert_eq!(UiConfig::default().tutorial_seen, Some(false));
|
||||
|
||||
let older: Config = toml::from_str("[ui]\nscale = 1.1\n").expect("parses");
|
||||
assert_eq!(
|
||||
older.ui.tutorial_seen, None,
|
||||
"a config predating the tour must not be offered it"
|
||||
);
|
||||
|
||||
let dismissed: Config = toml::from_str("[ui]\ntutorial_seen = true\n").expect("parses");
|
||||
assert_eq!(dismissed.ui.tutorial_seen, Some(true));
|
||||
}
|
||||
|
||||
/// Size and modified cost more width than they earn for most searches.
|
||||
#[test]
|
||||
fn the_search_table_ships_without_size_or_modified() {
|
||||
let cols = ColumnsConfig::default();
|
||||
assert!(cols.name && cols.content_match && cols.rank);
|
||||
assert!(!cols.size, "the size column is on by default");
|
||||
assert!(!cols.modified, "the modified column is on by default");
|
||||
|
||||
// A `[search]` block written before the picker existed still gets them.
|
||||
let older: Config = toml::from_str("[search]\ndisplay_limit = 500\n").expect("parses");
|
||||
assert_eq!(older.search.columns, cols);
|
||||
assert_eq!(older.search.display_limit, 500);
|
||||
assert!(older.search.live_results, "live results default to on");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,24 +1,22 @@
|
|||
//! Parallel content extraction for one indexing root.
|
||||
//!
|
||||
//! The second half of a root's pipeline, and the sibling of [`crate::walk`]:
|
||||
//! a pool of worker threads produces finished work over a bounded channel,
|
||||
//! and the single writer drains it in time-bounded turns. Walking roots are
|
||||
//! served first and only one extracting root per round, so a pass that is
|
||||
//! producing faster than the writer can tokenize waits rather than holding up
|
||||
//! anyone's walk — see `indexing::pipeline`.
|
||||
//! a pool of worker threads produces finished work over a bounded channel, and
|
||||
//! the single writer drains it round-robin against every other root.
|
||||
//!
|
||||
//! **One feeder thread owns the only database connection**, paging through
|
||||
//! the root's pending rows, while N workers do nothing but filesystem work. A
|
||||
//! connection per worker would multiply SQLite's page cache by the pool size
|
||||
//! (see [`crate::db::schema::PRAGMAS_WALK_READER`]).
|
||||
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::atomic::AtomicBool;
|
||||
use std::sync::{mpsc, Arc, Condvar, Mutex};
|
||||
use std::thread::JoinHandle;
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::extract::Registry;
|
||||
use crate::file_handling::{decide_content, ContentOutcome, ExtractCursor, ExtractScope};
|
||||
use crate::file_handling::{decide_content, ContentOutcome, ExtractCursor};
|
||||
use crate::indexing::should_abort;
|
||||
use crate::walk::{try_recv_next, TryNext, WorkerStats};
|
||||
|
||||
/// Finished rows waiting for the writer.
|
||||
|
|
@ -71,11 +69,6 @@ struct Queue {
|
|||
struct Shared {
|
||||
queue: Mutex<Queue>,
|
||||
idle: Condvar,
|
||||
/// What the range held when the pass began: rows still to extract and
|
||||
/// rows already done. Set by the feeder before it pages anything, so
|
||||
/// `already_done + rows written this pass` stays exact; never set if the
|
||||
/// feeder could not count.
|
||||
totals: std::sync::OnceLock<ExtractScope>,
|
||||
}
|
||||
|
||||
impl Shared {
|
||||
|
|
@ -170,16 +163,6 @@ impl ContentPass {
|
|||
self.stats.clone()
|
||||
}
|
||||
|
||||
/// The range's pending and already-done counts as they stood when the
|
||||
/// pass began.
|
||||
///
|
||||
/// `None` until the feeder has counted — a scan that takes seconds on a
|
||||
/// large root, which is why it happens here on the pass's own connection
|
||||
/// and not on the indexer's writer — and forever if it could not.
|
||||
pub fn totals(&self) -> Option<ExtractScope> {
|
||||
self.shared.totals.get().copied()
|
||||
}
|
||||
|
||||
/// Join the workers and report whether every one finished cleanly.
|
||||
/// See [`crate::walk::ParallelWalk::finish`].
|
||||
pub fn finish(&mut self) -> bool {
|
||||
|
|
@ -214,7 +197,7 @@ impl Drop for ContentPass {
|
|||
/// A failed query ends the pass rather than retrying: the rows stay
|
||||
/// `content_state = 0` and the next run picks them up, which is the same
|
||||
/// outcome as being interrupted.
|
||||
fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, config: &Config) {
|
||||
fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, max_size: i64) {
|
||||
let conn = match crate::db::open::open_walk_reader(db_path) {
|
||||
Ok(conn) => conn,
|
||||
Err(e) => {
|
||||
|
|
@ -224,18 +207,6 @@ fn feeder(shared: &Shared, db_path: &str, mut cursor: ExtractCursor, config: &Co
|
|||
}
|
||||
};
|
||||
|
||||
// Before the first page, so nothing this pass writes is inside the count.
|
||||
// The workers cannot run ahead of this: they block in `take` until the
|
||||
// first page lands. A failure here costs the progress figure, not the
|
||||
// pass.
|
||||
match crate::file_handling::count_extract_scope(&conn, &cursor, config) {
|
||||
Ok(totals) => {
|
||||
let _ = shared.totals.set(totals);
|
||||
}
|
||||
Err(e) => crate::log_warn!("content reader: {}", e),
|
||||
}
|
||||
|
||||
let max_size = crate::file_handling::max_text_file_size(config);
|
||||
while shared.take_feed_slot().is_some() {
|
||||
let page =
|
||||
match crate::db::repo::pending_content_page(&conn, &cursor, max_size, FEED_PAGE as i64)
|
||||
|
|
@ -273,13 +244,14 @@ fn worker(
|
|||
registry: &Registry,
|
||||
config: &Config,
|
||||
stop_flag: &Arc<AtomicBool>,
|
||||
suspend_flag: &Arc<AtomicBool>,
|
||||
stats: &WorkerStats,
|
||||
) {
|
||||
while let Some(row) = shared.take() {
|
||||
// Held for the whole of `decide_content`; that is the work the
|
||||
// progress line reports.
|
||||
let _busy = stats.enter();
|
||||
if stop_flag.load(Ordering::Relaxed) {
|
||||
if should_abort(stop_flag, suspend_flag) {
|
||||
shared.shutdown();
|
||||
return;
|
||||
}
|
||||
|
|
@ -307,13 +279,14 @@ pub fn extract_content(
|
|||
registry: Arc<Registry>,
|
||||
config: Config,
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
suspend_flag: Arc<AtomicBool>,
|
||||
workers: usize,
|
||||
) -> ContentPass {
|
||||
let shared = Arc::new(Shared {
|
||||
queue: Mutex::new(Queue::default()),
|
||||
idle: Condvar::new(),
|
||||
totals: std::sync::OnceLock::new(),
|
||||
});
|
||||
let max_size = i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX);
|
||||
|
||||
let (tx, rx) = mpsc::sync_channel(READY_CAP);
|
||||
let stats = WorkerStats::new(workers.clamp(1, 64));
|
||||
|
|
@ -321,11 +294,19 @@ pub fn extract_content(
|
|||
.map(|_| {
|
||||
let (shared, tx) = (shared.clone(), tx.clone());
|
||||
let (registry, config) = (registry.clone(), config.clone());
|
||||
let stop_flag = stop_flag.clone();
|
||||
let (stop_flag, suspend_flag) = (stop_flag.clone(), suspend_flag.clone());
|
||||
let stats = stats.clone();
|
||||
crate::platform::spawn_worker("qs-extract", move || {
|
||||
crate::platform::set_background_priority();
|
||||
worker(&shared, &tx, ®istry, &config, &stop_flag, &stats)
|
||||
worker(
|
||||
&shared,
|
||||
&tx,
|
||||
®istry,
|
||||
&config,
|
||||
&stop_flag,
|
||||
&suspend_flag,
|
||||
&stats,
|
||||
)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
|
@ -337,7 +318,7 @@ pub fn extract_content(
|
|||
let (shared, db_path, cursor) = (shared.clone(), db_path.to_string(), cursor.clone());
|
||||
crate::platform::spawn_worker("qs-feeder", move || {
|
||||
crate::platform::set_background_priority();
|
||||
feeder(&shared, &db_path, cursor, &config)
|
||||
feeder(&shared, &db_path, cursor, max_size)
|
||||
})
|
||||
};
|
||||
|
||||
|
|
@ -357,23 +338,9 @@ mod tests {
|
|||
|
||||
use crate::db::open_or_recreate;
|
||||
use crate::db::repo::{self, insert_file, NewFile};
|
||||
use crate::file_handling::{store_extracted, ExtractScope, Stored};
|
||||
use crate::file_handling::{extract_scope_prepare, store_extracted};
|
||||
use crate::mime::FileType;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// The removed `extract_scope_prepare`: the sweep on the writer, then the
|
||||
/// count the content pass now does on its own connection. Composed here
|
||||
/// because these tests want both halves in one call.
|
||||
fn extract_scope_prepare(
|
||||
conn_mutex: &Arc<Mutex<rusqlite::Connection>>,
|
||||
cursor: &ExtractCursor,
|
||||
config: &Config,
|
||||
) -> Result<ExtractScope, String> {
|
||||
let conn = crate::lock_ok(conn_mutex);
|
||||
crate::file_handling::mark_oversize_pending_na(&conn, cursor, config)?;
|
||||
crate::file_handling::count_extract_scope(&conn, cursor, config)
|
||||
}
|
||||
/// A path that does not exist yet — the caller builds the tree under it.
|
||||
fn tmp(tag: &str) -> PathBuf {
|
||||
crate::testutil::scratch_dir(tag).join("tree")
|
||||
|
|
@ -400,6 +367,8 @@ mod tests {
|
|||
parent: d.to_str().unwrap(),
|
||||
size: std::fs::metadata(&f).unwrap().len(),
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -422,6 +391,7 @@ mod tests {
|
|||
Arc::new(Registry::default_set()),
|
||||
Config::default(),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
workers,
|
||||
)
|
||||
}
|
||||
|
|
@ -480,13 +450,9 @@ mod tests {
|
|||
assert_eq!(rows.len(), 3);
|
||||
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let far = Instant::now() + Duration::from_secs(60);
|
||||
assert_eq!(
|
||||
store_extracted(&conn_mutex, &rows, &stop, &config, far).unwrap(),
|
||||
Stored {
|
||||
consumed: 3,
|
||||
written: 3
|
||||
}
|
||||
store_extracted(&conn_mutex, &rows, &stop, &config).unwrap(),
|
||||
3
|
||||
);
|
||||
|
||||
let state = |p: &Path| -> i64 {
|
||||
|
|
@ -535,93 +501,6 @@ mod tests {
|
|||
let mut pass = pass_for(&tree, &db, "nonexistent", 4);
|
||||
assert!(drain(&mut pass).is_empty());
|
||||
assert!(pass.finish());
|
||||
// The pass still counted: an empty range is a known zero, not an
|
||||
// unknown.
|
||||
assert_eq!(
|
||||
pass.totals(),
|
||||
Some(ExtractScope {
|
||||
pending: 0,
|
||||
already_done: 0
|
||||
})
|
||||
);
|
||||
std::fs::remove_dir_all(&tree).ok();
|
||||
std::fs::remove_file(&db).ok();
|
||||
}
|
||||
|
||||
/// The pass counts its range on its own connection, before it pages —
|
||||
/// which is what lets the writer thread stop doing it. The count is what
|
||||
/// stood at the start: rows this pass writes are not inside it.
|
||||
#[test]
|
||||
fn the_pass_counts_its_range_before_it_starts() {
|
||||
let (tree, db) = seed("totals", &[("r1", 3), ("r2", 2)]);
|
||||
let mut pass = pass_for(&tree, &db, "r1", 2);
|
||||
let rows = drain(&mut pass);
|
||||
assert!(pass.finish());
|
||||
assert_eq!(rows.len(), 3);
|
||||
assert_eq!(
|
||||
pass.totals(),
|
||||
Some(ExtractScope {
|
||||
pending: 3,
|
||||
already_done: 0
|
||||
}),
|
||||
"only r1's rows, all of them pending when the pass began"
|
||||
);
|
||||
std::fs::remove_dir_all(&tree).ok();
|
||||
std::fs::remove_file(&db).ok();
|
||||
}
|
||||
|
||||
/// The writer's turn is bounded by time, not by batch: `store_extracted`
|
||||
/// stops at its deadline, tells the caller how far it got, and always
|
||||
/// gets at least one row down so a caller looping on it cannot spin.
|
||||
#[test]
|
||||
fn store_extracted_honours_its_deadline_but_always_makes_progress() {
|
||||
let (tree, db) = seed("deadline", &[("r1", 5)]);
|
||||
let conn_mutex = Arc::new(Mutex::new(
|
||||
open_or_recreate(db.to_str().unwrap(), "trigram").unwrap(),
|
||||
));
|
||||
let config = Config::default();
|
||||
let mut pass = pass_for(&tree, &db, "r1", 2);
|
||||
let rows = drain(&mut pass);
|
||||
assert!(pass.finish());
|
||||
assert_eq!(rows.len(), 5);
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
|
||||
// A deadline already gone by: one row, then out.
|
||||
let past = Instant::now() - Duration::from_secs(1);
|
||||
assert_eq!(
|
||||
store_extracted(&conn_mutex, &rows, &stop, &config, past).unwrap(),
|
||||
Stored {
|
||||
consumed: 1,
|
||||
written: 1
|
||||
}
|
||||
);
|
||||
// Plenty of time: the rest, in one call.
|
||||
let far = Instant::now() + Duration::from_secs(60);
|
||||
assert_eq!(
|
||||
store_extracted(&conn_mutex, &rows[1..], &stop, &config, far).unwrap(),
|
||||
Stored {
|
||||
consumed: 4,
|
||||
written: 4
|
||||
}
|
||||
);
|
||||
let done: i64 = conn_mutex
|
||||
.lock()
|
||||
.unwrap()
|
||||
.query_row(
|
||||
"SELECT COUNT(*) FROM files WHERE content_state = 1",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(done, 5, "every row landed across the two calls");
|
||||
|
||||
// Stopped before it starts: nothing consumed, and the caller can tell.
|
||||
stop.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
assert_eq!(
|
||||
store_extracted(&conn_mutex, &rows, &stop, &config, far).unwrap(),
|
||||
Stored::default()
|
||||
);
|
||||
|
||||
std::fs::remove_dir_all(&tree).ok();
|
||||
std::fs::remove_file(&db).ok();
|
||||
}
|
||||
|
|
@ -635,6 +514,7 @@ mod tests {
|
|||
Arc::new(Registry::default_set()),
|
||||
Config::default(),
|
||||
Arc::new(AtomicBool::new(true)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
assert!(drain(&mut pass).len() < 400);
|
||||
|
|
|
|||
|
|
@ -130,7 +130,6 @@ enum CoordCmd {
|
|||
ConfigChanged(Config),
|
||||
RebuildIndex,
|
||||
ClearIndex,
|
||||
UpdatePaths(Vec<PathBuf>),
|
||||
Shutdown,
|
||||
}
|
||||
|
||||
|
|
@ -229,7 +228,6 @@ impl IndexCoordinator {
|
|||
watcher_rx: None,
|
||||
watcher_gen: 0,
|
||||
pending: HashMap::new(),
|
||||
targeted: HashMap::new(),
|
||||
last_event_at: None,
|
||||
pending_since: None,
|
||||
needs_full_run: false,
|
||||
|
|
@ -305,26 +303,6 @@ impl IndexCoordinator {
|
|||
let _ = self.cmd_tx.send(CoordCmd::ClearIndex);
|
||||
}
|
||||
|
||||
/// Bring the index up to date for these paths and nothing else.
|
||||
///
|
||||
/// For [`crate::live`]: a frontend that has just read a displayed file
|
||||
/// from disk hands the path here so the index agrees with what the user
|
||||
/// is looking at. Deliberately **not** gated on [`IndexMode`] — the whole
|
||||
/// point is that the rows on screen stay honest with indexing stopped —
|
||||
/// but still applied on the coordinator's own thread, so the
|
||||
/// single-writer rule holds and a full run is never raced.
|
||||
///
|
||||
/// Each path is re-read and rewritten only if its modified time has moved
|
||||
/// (see [`crate::incremental::apply_fs_event`]), so submitting a path that
|
||||
/// is already current costs a `stat` and a row lookup. A path that no
|
||||
/// longer exists is removed from the index.
|
||||
pub fn update_paths(&self, paths: Vec<PathBuf>) {
|
||||
if paths.is_empty() {
|
||||
return;
|
||||
}
|
||||
let _ = self.cmd_tx.send(CoordCmd::UpdatePaths(paths));
|
||||
}
|
||||
|
||||
/// Compare `config` against what the index was built with. Read-only.
|
||||
pub fn check_config_validation(
|
||||
&self,
|
||||
|
|
|
|||
|
|
@ -22,12 +22,6 @@ pub(super) struct Inner {
|
|||
pub(super) watcher_rx: Option<mpsc::Receiver<(u64, Result<Watcher, WatchError>)>>,
|
||||
pub(super) watcher_gen: u64,
|
||||
pub(super) pending: HashMap<PathBuf, FsEvent>,
|
||||
/// Paths a frontend asked for by name — see
|
||||
/// [`IndexCoordinator::update_paths`]. Kept apart from [`Inner::pending`]
|
||||
/// on purpose: this queue survives [`Inner::clear_pending`] and is applied
|
||||
/// in manual mode, because it exists to keep the rows a user is *reading*
|
||||
/// in step with the disk however the indexer is configured.
|
||||
pub(super) targeted: HashMap<PathBuf, FsEvent>,
|
||||
/// When the most recent event arrived; the burst is over once this is
|
||||
/// `pending_settle` old.
|
||||
pub(super) last_event_at: Option<Instant>,
|
||||
|
|
@ -164,36 +158,6 @@ impl Inner {
|
|||
drop(shared);
|
||||
self.files_at = None;
|
||||
}
|
||||
CoordCmd::UpdatePaths(paths) => {
|
||||
// Only paths under an indexed root: the watcher never
|
||||
// delivers anything else, so nothing downstream checks, and
|
||||
// a file renamed *out* of every root would otherwise be
|
||||
// written into the index at its new home. Roots in the same
|
||||
// spelling `files.path` uses — the caller's paths are.
|
||||
let prefixes: Vec<String> = self
|
||||
.config
|
||||
.normalized_indexing_paths()
|
||||
.iter()
|
||||
.map(|root| crate::file_handling::ExtractCursor::for_root(root).lo)
|
||||
.collect();
|
||||
for path in paths {
|
||||
let spelled = path.to_string_lossy();
|
||||
if !prefixes.iter().any(|lo| spelled.starts_with(lo.as_str())) {
|
||||
continue;
|
||||
}
|
||||
// Existence decides the verb. The caller knows a file
|
||||
// changed, not what it changed into, and a `Modify` for a
|
||||
// path that is gone would be silently skipped rather than
|
||||
// removing the row.
|
||||
let event = if path.is_file() {
|
||||
FsEvent::Modify(path)
|
||||
} else {
|
||||
FsEvent::Remove(path)
|
||||
};
|
||||
enqueue(&mut self.targeted, event);
|
||||
}
|
||||
self.was_busy = true;
|
||||
}
|
||||
CoordCmd::Shutdown => unreachable!("handled in run()"),
|
||||
}
|
||||
}
|
||||
|
|
@ -233,14 +197,6 @@ impl Inner {
|
|||
|
||||
self.refresh_file_count();
|
||||
|
||||
// Ahead of both the reconcile and the mode gate, and ahead of the
|
||||
// settle window the watcher queue waits out: these are rows a user is
|
||||
// looking at right now, there are at most a screenful, and a stopped
|
||||
// indexer is exactly when the frontend most needs them to be current.
|
||||
if !self.targeted.is_empty() {
|
||||
self.apply_targeted();
|
||||
}
|
||||
|
||||
// Ahead of the mode gate: a config edit is reconciled in manual mode
|
||||
// too.
|
||||
if self.pending_work.is_some() {
|
||||
|
|
@ -513,71 +469,6 @@ impl Inner {
|
|||
}
|
||||
}
|
||||
|
||||
/// Apply the by-name queue: the paths a frontend is displaying.
|
||||
///
|
||||
/// Shaped like [`Inner::apply_pending`] — removals first, same budget —
|
||||
/// but it never escalates to [`Inner::needs_full_run`]. A frontend reads
|
||||
/// what it shows from the file itself, so a failure here leaves the screen
|
||||
/// correct and only the index behind; reindexing the world over that would
|
||||
/// be wildly out of proportion.
|
||||
fn apply_targeted(&mut self) {
|
||||
self.was_busy = true;
|
||||
let mut conn = match self.ensure_write_conn() {
|
||||
Ok(conn) => conn,
|
||||
Err(e) => {
|
||||
crate::log_warn!("coordinator: targeted update unavailable: {}", e);
|
||||
self.targeted.clear();
|
||||
return;
|
||||
}
|
||||
};
|
||||
let deadline = Instant::now() + APPLY_BUDGET;
|
||||
let chunk = self.config.processing.batch_size.max(1);
|
||||
|
||||
// Removals lead for the same reason they do in `apply_pending`: the
|
||||
// queue is an unordered map, and a rename enqueues both halves.
|
||||
let removals: Vec<PathBuf> = self
|
||||
.targeted
|
||||
.iter()
|
||||
.filter(|(_, ev)| is_removal(ev))
|
||||
.map(|(p, _)| p.clone())
|
||||
.collect();
|
||||
for batch in removals.chunks(chunk) {
|
||||
if let Err(e) = crate::incremental::remove_paths(&mut conn, batch, chunk) {
|
||||
crate::log_warn!("coordinator: targeted remove: {}", e);
|
||||
}
|
||||
for path in batch {
|
||||
self.targeted.remove(path);
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if Instant::now() < deadline {
|
||||
let upserts: Vec<PathBuf> = self
|
||||
.targeted
|
||||
.iter()
|
||||
.filter(|(_, ev)| !is_removal(ev))
|
||||
.map(|(p, _)| p.clone())
|
||||
.collect();
|
||||
for path in upserts {
|
||||
let Some(ev) = self.targeted.remove(&path) else {
|
||||
continue;
|
||||
};
|
||||
if let Err(e) =
|
||||
apply_fs_event(&mut conn, &ev, &self.config, &self.ignore, &self.registry)
|
||||
{
|
||||
crate::log_warn!("coordinator: targeted apply {:?}: {}", ev, e);
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.write_conn = Some(conn);
|
||||
}
|
||||
|
||||
fn ensure_write_conn(&mut self) -> Result<Connection, String> {
|
||||
if let Some(conn) = self.write_conn.take() {
|
||||
return Ok(conn);
|
||||
|
|
@ -908,7 +799,7 @@ impl Inner {
|
|||
);
|
||||
let mut shared = crate::lock_ok(&self.shared);
|
||||
shared.mode = self.mode;
|
||||
shared.queued_events = self.pending.len() + self.targeted.len();
|
||||
shared.queued_events = self.pending.len();
|
||||
shared.reconcile = reconcile;
|
||||
drop(shared);
|
||||
|
||||
|
|
|
|||
|
|
@ -505,149 +505,6 @@ fn a_run_it_schedules_itself_wakes_the_frontend() {
|
|||
coord.shutdown();
|
||||
}
|
||||
|
||||
// --- targeted updates (see `IndexCoordinator::update_paths`) --------------
|
||||
|
||||
impl Fixture {
|
||||
/// The `mtime` the index holds for one path, or `None` if it has no row.
|
||||
fn stored_mtime(&self, path: &std::path::Path) -> Option<i64> {
|
||||
let conn = db::open_existing(&self.db.to_string_lossy(), false).ok()?;
|
||||
conn.query_row(
|
||||
"SELECT mtime FROM files WHERE path = ?1",
|
||||
[path.to_string_lossy().as_ref()],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.ok()
|
||||
}
|
||||
}
|
||||
|
||||
/// The point of the whole thing: the frontend has just read a file the user is
|
||||
/// looking at, and the index catches up even though indexing is stopped — with
|
||||
/// no watcher running and no full run scheduled.
|
||||
#[test]
|
||||
fn update_paths_indexes_one_file_with_indexing_stopped() {
|
||||
let f = Fixture::new(false);
|
||||
std::fs::write(f.dir.join("seed.txt"), "initial content").unwrap();
|
||||
f.seed_index();
|
||||
assert_eq!(f.file_count(), 1);
|
||||
|
||||
let coord = start_coord(f.config.clone());
|
||||
let added = f.dir.join("added.txt");
|
||||
std::fs::write(&added, "written while indexing was stopped").unwrap();
|
||||
coord.update_paths(vec![added.clone()]);
|
||||
|
||||
wait_for("targeted insert", Duration::from_secs(20), || {
|
||||
f.stored_mtime(&added).is_some()
|
||||
});
|
||||
assert_eq!(
|
||||
coord.state().mode,
|
||||
IndexMode::ManualStopped,
|
||||
"a targeted update started a run"
|
||||
);
|
||||
assert!(
|
||||
coord.state().last_full_index.is_some(),
|
||||
"the seed stamp was disturbed"
|
||||
);
|
||||
coord.shutdown();
|
||||
}
|
||||
|
||||
/// The same call is how a row is *validated*: submitting a path the index
|
||||
/// already agrees with must not rewrite it, which is what makes it cheap
|
||||
/// enough for the frontend to submit whatever it just looked at.
|
||||
#[test]
|
||||
fn update_paths_leaves_a_row_that_already_agrees_alone() {
|
||||
let f = Fixture::new(false);
|
||||
let file = f.dir.join("steady.txt");
|
||||
std::fs::write(&file, "unchanged").unwrap();
|
||||
f.seed_index();
|
||||
let before = f.stored_mtime(&file).expect("seeded");
|
||||
|
||||
let coord = start_coord(f.config.clone());
|
||||
coord.update_paths(vec![file.clone()]);
|
||||
// No state change to wait on, so wait out a few ticks instead.
|
||||
std::thread::sleep(Duration::from_secs(3));
|
||||
|
||||
assert_eq!(f.stored_mtime(&file), Some(before));
|
||||
assert_eq!(f.file_count(), 1);
|
||||
coord.shutdown();
|
||||
}
|
||||
|
||||
/// A path outside every indexed root is not the index's to hold, however it
|
||||
/// was submitted: a result renamed into an un-indexed folder must not follow
|
||||
/// the row into the index at its new home.
|
||||
#[test]
|
||||
fn update_paths_ignores_a_path_outside_every_root() {
|
||||
let f = Fixture::new(false);
|
||||
std::fs::write(f.dir.join("seed.txt"), "initial content").unwrap();
|
||||
f.seed_index();
|
||||
assert_eq!(f.file_count(), 1);
|
||||
|
||||
// A sibling of the indexed tree, under the same scratch parent.
|
||||
let outside = f.dir.parent().unwrap().join("elsewhere");
|
||||
std::fs::create_dir_all(&outside).unwrap();
|
||||
let stray = outside.join("moved-here.txt");
|
||||
std::fs::write(&stray, "renamed out of the index").unwrap();
|
||||
|
||||
let coord = start_coord(f.config.clone());
|
||||
coord.update_paths(vec![stray.clone()]);
|
||||
std::thread::sleep(Duration::from_secs(3));
|
||||
|
||||
assert_eq!(
|
||||
f.stored_mtime(&stray),
|
||||
None,
|
||||
"an un-indexed folder gained a row"
|
||||
);
|
||||
assert_eq!(f.file_count(), 1);
|
||||
coord.shutdown();
|
||||
std::fs::remove_dir_all(&outside).ok();
|
||||
}
|
||||
|
||||
/// A row whose file has gone leaves the index too — the frontend hands over
|
||||
/// the path, not a verb, so the coordinator decides from what is on disk.
|
||||
#[test]
|
||||
fn update_paths_removes_a_row_whose_file_is_gone() {
|
||||
let f = Fixture::new(false);
|
||||
let file = f.dir.join("doomed.txt");
|
||||
std::fs::write(&file, "not for long").unwrap();
|
||||
f.seed_index();
|
||||
assert!(f.stored_mtime(&file).is_some());
|
||||
|
||||
let coord = start_coord(f.config.clone());
|
||||
std::fs::remove_file(&file).unwrap();
|
||||
coord.update_paths(vec![file.clone()]);
|
||||
|
||||
wait_for("targeted remove", Duration::from_secs(20), || {
|
||||
f.stored_mtime(&file).is_none()
|
||||
});
|
||||
coord.shutdown();
|
||||
}
|
||||
|
||||
/// The single-writer rule still holds: a targeted update submitted while a
|
||||
/// full run owns the database waits for it rather than opening a second
|
||||
/// writer beside it.
|
||||
#[test]
|
||||
fn update_paths_waits_for_a_full_run_rather_than_racing_it() {
|
||||
let f = Fixture::new(false);
|
||||
for i in 0..400 {
|
||||
std::fs::write(f.dir.join(format!("f{i}.txt")), "body").unwrap();
|
||||
}
|
||||
let coord = start_coord(f.config.clone());
|
||||
coord.reindex_now();
|
||||
|
||||
let added = f.dir.join("late.txt");
|
||||
std::fs::write(&added, "submitted mid-run").unwrap();
|
||||
coord.update_paths(vec![added.clone()]);
|
||||
|
||||
wait_for("run finished", Duration::from_secs(60), || {
|
||||
coord.state().last_full_index.is_some()
|
||||
});
|
||||
wait_for(
|
||||
"targeted insert after the run",
|
||||
Duration::from_secs(20),
|
||||
|| f.stored_mtime(&added).is_some(),
|
||||
);
|
||||
coord.shutdown();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auto_mode_runs_initial_index_and_applies_watcher_events() {
|
||||
let f = Fixture::new(true);
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ pub const KEY_MISMATCH_PREFIX: &str = "KEY_MISMATCH: ";
|
|||
/// values go stale: `files.mime`, `files.type` and `content_state` are
|
||||
/// computed at walk time and never re-derived for unchanged files, so a
|
||||
/// classification change needs the wipe to apply everywhere.
|
||||
pub const CURRENT_SCHEMA_VERSION: u32 = 7;
|
||||
pub const CURRENT_SCHEMA_VERSION: u32 = 6;
|
||||
|
||||
/// Open `db_path` and ensure the on-disk schema matches this build; if it
|
||||
/// doesn't (including a changed `tokenizer`), delete the file and recreate it
|
||||
|
|
|
|||
|
|
@ -102,7 +102,7 @@ fn legacy_layout_db_is_wiped_and_recreated() {
|
|||
// New columns should exist (just prepare the SELECT — an
|
||||
// unknown column name would parse-error here).
|
||||
conn.query_row(
|
||||
"SELECT content_state, type, mime FROM files LIMIT 0",
|
||||
"SELECT basic_state, content_state, type, mime FROM files LIMIT 0",
|
||||
[],
|
||||
|_| Ok(()),
|
||||
)
|
||||
|
|
@ -172,8 +172,8 @@ fn open_existing_reads_nondefault_tokenizer_without_wiping() {
|
|||
// Seed the FTS index (rowid = the files row we just inserted) so a
|
||||
// MATCH query can be exercised against the on-disk tokenizer.
|
||||
conn.execute(
|
||||
"INSERT INTO searchabletext (rowid, text) \
|
||||
VALUES (last_insert_rowid(), 'hello world')",
|
||||
"INSERT INTO searchabletext (rowid, name, text, properties) \
|
||||
VALUES (last_insert_rowid(), 'note', 'hello world', '')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
//! Row-level write helpers that keep the FTS5 contentless table in sync with
|
||||
//! `files`/`documents_text`.
|
||||
//! `files`/`documents`/`properties`.
|
||||
//!
|
||||
//! States (mirrors the `content_state` column):
|
||||
//! States (mirrors `basic_state` / `content_state` columns):
|
||||
//!
|
||||
//! | value | meaning |
|
||||
//! |------:|---------|
|
||||
|
|
@ -45,7 +45,7 @@ fn set_state_clearing_failure(
|
|||
) -> Result<(), String> {
|
||||
exec(
|
||||
tx,
|
||||
"UPDATE files SET content_state = ?1 WHERE id = ?2",
|
||||
"UPDATE files SET content_state = ?1, failure_msg = NULL WHERE id = ?2",
|
||||
params![state, file_id],
|
||||
|| format!("{} content_state {}", transition, file_id),
|
||||
)?;
|
||||
|
|
@ -66,6 +66,8 @@ pub struct NewFile<'a> {
|
|||
pub parent: &'a str,
|
||||
pub size: u64,
|
||||
pub mtime: u64,
|
||||
pub inode: Option<u64>,
|
||||
pub device_id: Option<u64>,
|
||||
pub mime: Option<&'a str>,
|
||||
pub ftype: FileType,
|
||||
pub hash: Option<&'a [u8]>,
|
||||
|
|
@ -74,16 +76,17 @@ pub struct NewFile<'a> {
|
|||
pub needs_content: bool,
|
||||
}
|
||||
|
||||
/// Insert a new file row, returning its id. `content_state` comes from
|
||||
/// `needs_content`; there is no separate basic state, because the row
|
||||
/// existing *is* the basic-index state. `INSERT OR IGNORE`: a UNIQUE(path)
|
||||
/// collision returns `None` rather than aborting the batch.
|
||||
/// Insert a new file row, returning its id. `basic_state` is set to DONE
|
||||
/// (the row existing *is* the basic-index state); `content_state` comes from
|
||||
/// `needs_content`. `INSERT OR IGNORE`: a UNIQUE(path) collision returns
|
||||
/// `None` rather than aborting the batch.
|
||||
pub fn insert_file(tx: &Transaction<'_>, f: &NewFile<'_>) -> Result<Option<i64>, String> {
|
||||
let rows = tx
|
||||
.prepare_cached(
|
||||
"INSERT OR IGNORE INTO files (
|
||||
name, path, parent, size, mtime, mime, type, content_state, hash
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
|
||||
name, path, parent, size, mtime, inode, device_id,
|
||||
mime, type, basic_state, content_state, hash
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12)",
|
||||
)
|
||||
.and_then(|mut stmt| {
|
||||
stmt.execute(params![
|
||||
|
|
@ -92,8 +95,11 @@ pub fn insert_file(tx: &Transaction<'_>, f: &NewFile<'_>) -> Result<Option<i64>,
|
|||
f.parent,
|
||||
f.size as i64,
|
||||
f.mtime as i64,
|
||||
f.inode.map(|x| x as i64),
|
||||
f.device_id.map(|x| x as i64),
|
||||
f.mime,
|
||||
f.ftype.bits() as i64,
|
||||
STATE_DONE,
|
||||
initial_content_state(f),
|
||||
f.hash,
|
||||
])
|
||||
|
|
@ -119,14 +125,14 @@ fn initial_content_state(f: &NewFile<'_>) -> i64 {
|
|||
/// Update a file's metadata in place (same path, changed size/mtime/hash) and
|
||||
/// reset its content state from `f.needs_content`, clearing any extracted
|
||||
/// content so the text-indexing pass re-processes it. Writes `size`, `mtime`,
|
||||
/// `hash`, `mime`, `type` and `content_state` — and only those; `name` and
|
||||
/// `parent` are not refreshed here.
|
||||
/// `hash`, `mime`, `type`, `content_state` and `failure_msg` — and only
|
||||
/// those; `name`, `parent`, `inode` and `device_id` are not refreshed here.
|
||||
pub fn update_file_basic(tx: &Transaction<'_>, f: &NewFile<'_>) -> Result<Option<i64>, String> {
|
||||
let id: Option<i64> = tx
|
||||
.prepare_cached(
|
||||
"UPDATE files
|
||||
SET size = ?1, mtime = ?2, hash = ?3, mime = ?4, type = ?5,
|
||||
content_state = ?6
|
||||
content_state = ?6, failure_msg = NULL
|
||||
WHERE path = ?7
|
||||
RETURNING id",
|
||||
)
|
||||
|
|
@ -153,11 +159,13 @@ pub fn update_file_basic(tx: &Transaction<'_>, f: &NewFile<'_>) -> Result<Option
|
|||
Ok(Some(id))
|
||||
}
|
||||
|
||||
/// Mark a file's content indexing as complete and write the extracted text
|
||||
/// atomically. The plaintext feeds the contentless FTS5 tokenizer; when
|
||||
/// `store_text` is `true` it is also stored zstd-compressed in
|
||||
/// Mark a file's content indexing as complete and write the extracted text +
|
||||
/// properties atomically. The plaintext feeds the contentless FTS5 tokenizer;
|
||||
/// when `store_text` is `true` it is also stored zstd-compressed in
|
||||
/// `documents_text` for snippet rendering (`false`: matches still work, but
|
||||
/// result rows can't render snippets).
|
||||
/// result rows can't render snippets). `properties` are stored both as a
|
||||
/// structured side-table (exact retrieval) and concatenated into the FTS
|
||||
/// `properties` column (MATCH).
|
||||
/// `text_zstd` is the already-compressed body for the `documents_text`
|
||||
/// sidecar, or `None` to write no sidecar at all (an empty body, or
|
||||
/// `store_text_for_snippets` off).
|
||||
|
|
@ -170,27 +178,38 @@ pub fn update_file_basic(tx: &Transaction<'_>, f: &NewFile<'_>) -> Result<Option
|
|||
pub fn set_content_done(
|
||||
tx: &Transaction<'_>,
|
||||
file_id: i64,
|
||||
name: &str,
|
||||
text: &str,
|
||||
properties: &[(String, String)],
|
||||
text_zstd: Option<&[u8]>,
|
||||
) -> Result<(), String> {
|
||||
remove_content_for_id(tx, file_id)?;
|
||||
|
||||
for (k, v) in properties {
|
||||
exec(
|
||||
tx,
|
||||
"INSERT INTO properties(file_id, key, value) VALUES (?1, ?2, ?3)",
|
||||
params![file_id, k, v],
|
||||
|| format!("insert property {}={}", k, v),
|
||||
)?;
|
||||
}
|
||||
let props_blob = encode_properties_for_fts(properties);
|
||||
// Contentless FTS5 still accepts values on INSERT — the tokenizer needs
|
||||
// them — it simply doesn't persist the raw column values.
|
||||
exec(
|
||||
tx,
|
||||
"INSERT INTO searchabletext(rowid, text) VALUES (?1, ?2)",
|
||||
params![file_id, text],
|
||||
"INSERT INTO searchabletext(rowid, name, text, properties) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![file_id, name, text, props_blob],
|
||||
|| format!("insert FTS row {}", file_id),
|
||||
)?;
|
||||
|
||||
// No sidecar row for empty body text (an audio file whose tags are all
|
||||
// empty, say) — the caller passes `None` for that.
|
||||
// No sidecar row for empty body text (e.g. an image whose extractor
|
||||
// returned only EXIF properties) — the caller passes `None` for that.
|
||||
if let Some(compressed) = text_zstd {
|
||||
exec(
|
||||
tx,
|
||||
"INSERT INTO documents_text(file_id, text_zstd) VALUES (?1, ?2)",
|
||||
params![file_id, compressed],
|
||||
"INSERT INTO documents_text(file_id, text_zstd, text_len) VALUES (?1, ?2, ?3)",
|
||||
params![file_id, compressed, text.len() as i64],
|
||||
|| format!("insert documents_text {}", file_id),
|
||||
)?;
|
||||
}
|
||||
|
|
@ -198,86 +217,6 @@ pub fn set_content_done(
|
|||
set_state_clearing_failure(tx, file_id, STATE_DONE, "update DONE")
|
||||
}
|
||||
|
||||
/// Reusable decode buffer and decompression context for the readers of
|
||||
/// `documents_text` — the read side's mirror of [`DocEncoder`].
|
||||
///
|
||||
/// Shared by the cascade's full-text passes and by [`crate::live`], which
|
||||
/// re-reads one row's body when a file under a visible result changes.
|
||||
///
|
||||
/// `zstd::decode_all` builds and tears down a `ZSTD_DCtx` *and* allocates a
|
||||
/// fresh output `Vec` on every call, and it is called once per candidate row.
|
||||
/// One context and one buffer, reused across a whole scan, make that a
|
||||
/// per-scan cost instead of a per-row one.
|
||||
pub struct DocDecoder {
|
||||
dctx: zstd::bulk::Decompressor<'static>,
|
||||
buf: Vec<u8>,
|
||||
}
|
||||
|
||||
/// Where [`DocDecoder::decode`]'s buffer starts before it has seen a document.
|
||||
/// Most extracted text is well under this, so the doubling below rarely runs.
|
||||
const INITIAL_DOC_CAPACITY: usize = 64 * 1024;
|
||||
|
||||
/// Where the doubling stops. Stored text is capped at
|
||||
/// `processing.maximum_text_size` (256 KiB by default), so this is far above
|
||||
/// any legitimate document even if that setting is raised — past it, a failure
|
||||
/// is a corrupt frame rather than a buffer that is too small.
|
||||
const MAX_DOC_CAPACITY: usize = 64 * 1024 * 1024;
|
||||
|
||||
impl DocDecoder {
|
||||
pub fn new() -> Result<Self, String> {
|
||||
Ok(DocDecoder {
|
||||
dctx: zstd::bulk::Decompressor::new().map_err(|e| e.to_string())?,
|
||||
buf: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Decompress `blob` and borrow the result as text.
|
||||
///
|
||||
/// Returns `None` for a corrupt frame or non-UTF-8 content. Nothing is
|
||||
/// copied: the indexer stores UTF-8, so the bytes are borrowed in place
|
||||
/// rather than run through `String::from_utf8_lossy(..).into_owned()`,
|
||||
/// which duplicated the whole document even when it was already valid.
|
||||
pub fn decode(&mut self, blob: &[u8]) -> Option<&str> {
|
||||
self.buf.clear();
|
||||
// `decompress_to_buffer` writes into spare capacity and fails rather
|
||||
// than growing, so the room has to be there first.
|
||||
//
|
||||
// [`DocEncoder`] compresses through `ZSTD_compress2`, which knows the
|
||||
// whole input up front and records its length in the frame header, so
|
||||
// this reservation is normally exact and the loop below runs once.
|
||||
//
|
||||
// The loop is still the fallback, and it is not optional: a frame
|
||||
// written by a *stream*-based encoder carries no content size, and
|
||||
// falling back to `zstd::decode_all` for those looked harmless and was
|
||||
// not — it builds a streaming decoder per call, which measured as one
|
||||
// ~2.4 MiB allocation per document and 27 of the 30 GiB a fuzzy search
|
||||
// moved through the allocator. Growing and reusing this buffer instead
|
||||
// settles at the largest document in the scan within the first few
|
||||
// rows, after which decoding a row allocates nothing at all.
|
||||
if let Ok(Some(size)) = zstd::zstd_safe::get_frame_content_size(blob) {
|
||||
self.buf.reserve(usize::try_from(size).ok()?);
|
||||
}
|
||||
loop {
|
||||
if self.buf.capacity() == 0 {
|
||||
self.buf.reserve(INITIAL_DOC_CAPACITY);
|
||||
}
|
||||
match self.dctx.decompress_to_buffer(blob, &mut self.buf) {
|
||||
Ok(_) => break,
|
||||
// Too small, or corrupt — the bulk API cannot tell us which.
|
||||
// Growing is only worth trying while the buffer is still
|
||||
// smaller than any document could legitimately be.
|
||||
Err(_) if self.buf.capacity() < MAX_DOC_CAPACITY => {
|
||||
let bigger = self.buf.capacity().saturating_mul(2);
|
||||
self.buf.clear();
|
||||
self.buf.reserve(bigger);
|
||||
}
|
||||
Err(_) => return None,
|
||||
}
|
||||
}
|
||||
std::str::from_utf8(&self.buf).ok()
|
||||
}
|
||||
}
|
||||
|
||||
/// Level 3 hits ~3-5× on English prose at high throughput (hundreds of
|
||||
/// MB/s); level 9+ would shave a few percent more at 10× the CPU cost, and
|
||||
/// readers decompress far faster than writers compress.
|
||||
|
|
@ -322,31 +261,13 @@ pub fn encode_one(text: &str, store_text: bool) -> Result<Option<Vec<u8>>, Strin
|
|||
DocEncoder::new()?.encode(text).map(Some)
|
||||
}
|
||||
|
||||
/// The uncompressed size of a stored `documents_text` blob, read out of the
|
||||
/// zstd frame header instead of from a column beside it.
|
||||
///
|
||||
/// [`DocEncoder`] compresses through `ZSTD_compress2`, which is handed the
|
||||
/// whole document at once and writes its length into the frame header. A
|
||||
/// `text_len` column would have stored that same number a second time for
|
||||
/// every row, to serve one figure in the size report.
|
||||
///
|
||||
/// `None` for a frame that carries no content size — nothing this writer
|
||||
/// produces — or a corrupt one. The caller only needs the frame *header*, so
|
||||
/// `blob` may be a prefix of the stored value.
|
||||
pub fn raw_text_len(blob: &[u8]) -> Option<u64> {
|
||||
zstd::zstd_safe::get_frame_content_size(blob).ok().flatten()
|
||||
}
|
||||
|
||||
/// Mark a file's content extraction as failed. Keeps the basic row in place.
|
||||
///
|
||||
/// The reason is written once, to `failed_files` — which also carries the
|
||||
/// timestamp, and which `list-failed` and `status` both read.
|
||||
pub fn set_content_failed(tx: &Transaction<'_>, file_id: i64, reason: &str) -> Result<(), String> {
|
||||
let now = crate::log::now_unix() as i64;
|
||||
exec(
|
||||
tx,
|
||||
"UPDATE files SET content_state = ?1 WHERE id = ?2",
|
||||
params![STATE_FAILED, file_id],
|
||||
"UPDATE files SET content_state = ?1, failure_msg = ?2 WHERE id = ?3",
|
||||
params![STATE_FAILED, reason, file_id],
|
||||
|| format!("update content_state FAILED {}", file_id),
|
||||
)?;
|
||||
exec(
|
||||
|
|
@ -448,9 +369,10 @@ pub fn delete_outside_ranges(
|
|||
/// keyed to `files.id` first, then `files` itself. Not left to `ON DELETE
|
||||
/// CASCADE`: `searchabletext` is an FTS5 virtual table with no foreign key at
|
||||
/// all, and cascade only fires on connections with `PRAGMA foreign_keys` on.
|
||||
const DEPENDENT_TABLES: [(&str, &str); 3] = [
|
||||
const DEPENDENT_TABLES: [(&str, &str); 4] = [
|
||||
("searchabletext", "rowid"),
|
||||
("documents_text", "file_id"),
|
||||
("properties", "file_id"),
|
||||
("failed_files", "file_id"),
|
||||
];
|
||||
|
||||
|
|
@ -710,15 +632,16 @@ pub fn paths_in_dir(conn: &Connection, parent: &str) -> Result<Vec<String>, Stri
|
|||
.map_err(|e| format!("read path under {}: {}", parent, e))
|
||||
}
|
||||
|
||||
/// Remove the FTS row and the compressed text blob for a given file id. Does
|
||||
/// not touch the `files` row itself. Idempotent — a missing row is fine.
|
||||
/// Remove the FTS row, compressed text blob, and any `properties` rows for
|
||||
/// a given file id. Does not touch the `files` row itself. Idempotent — a
|
||||
/// missing row is fine.
|
||||
pub fn remove_content_for_id(tx: &Transaction<'_>, file_id: i64) -> Result<(), String> {
|
||||
// `contentless_delete=1` on the FTS5 table makes this work without
|
||||
// re-supplying the old column values (it tombstones the rowid).
|
||||
//
|
||||
// Spelled out rather than built from a (table, key) table: this runs for
|
||||
// every extracted document and every changed file, and `format!`ing two
|
||||
// constant strings per call also handed `prepare_cached` two freshly
|
||||
// every extracted document and every changed file, and `format!`ing three
|
||||
// constant strings per call also handed `prepare_cached` three freshly
|
||||
// allocated keys to hash.
|
||||
for (what, sql) in [
|
||||
(
|
||||
|
|
@ -729,6 +652,7 @@ pub fn remove_content_for_id(tx: &Transaction<'_>, file_id: i64) -> Result<(), S
|
|||
"documents_text",
|
||||
"DELETE FROM documents_text WHERE file_id = ?1",
|
||||
),
|
||||
("properties", "DELETE FROM properties WHERE file_id = ?1"),
|
||||
] {
|
||||
exec(tx, sql, params![file_id], || {
|
||||
format!("delete {} for {}", what, file_id)
|
||||
|
|
@ -737,6 +661,16 @@ pub fn remove_content_for_id(tx: &Transaction<'_>, file_id: i64) -> Result<(), S
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Serialize properties for the FTS `properties` column. `key:value` pairs
|
||||
/// separated by spaces so `MATCH 'properties:artist:beatles'` works.
|
||||
fn encode_properties_for_fts(props: &[(String, String)]) -> String {
|
||||
props
|
||||
.iter()
|
||||
.map(|(k, v)| format!("{}:{}", k, v))
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ")
|
||||
}
|
||||
|
||||
/// Free pages, as a percentage of the file, that make a [`maintain`] VACUUM
|
||||
/// worth its cost: rewriting a multi-gigabyte index to reclaim a few
|
||||
/// megabytes is minutes of I/O for no gain.
|
||||
|
|
|
|||
|
|
@ -9,34 +9,6 @@ fn tmp_path() -> std::path::PathBuf {
|
|||
crate::testutil::scratch_dir("repo").join("index.sqlite")
|
||||
}
|
||||
|
||||
/// The size report reads each body's uncompressed length out of its zstd
|
||||
/// frame header instead of from a stored column, which works only because
|
||||
/// [`DocEncoder`] compresses through `ZSTD_compress2` — the API that is
|
||||
/// handed the whole input up front and records its length. A switch back to
|
||||
/// a streaming encoder would silently zero that figure, so pin it here.
|
||||
#[test]
|
||||
fn a_compressed_body_carries_its_uncompressed_length() {
|
||||
let mut enc = DocEncoder::new().unwrap();
|
||||
let long = "lorem ipsum dolor sit amet ".repeat(4096);
|
||||
for text in ["", "hello world", &long] {
|
||||
let blob = enc.encode(text).unwrap();
|
||||
assert_eq!(
|
||||
raw_text_len(&blob),
|
||||
Some(text.len() as u64),
|
||||
"frame header lost the content size for a {}-byte body",
|
||||
text.len()
|
||||
);
|
||||
// The size report projects only a prefix, never the whole body — the
|
||||
// header fits in 18 bytes and that has to be enough.
|
||||
let prefix = &blob[..blob.len().min(18)];
|
||||
assert_eq!(
|
||||
raw_text_len(prefix),
|
||||
Some(text.len() as u64),
|
||||
"the first 18 bytes must be enough to read the length"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn insert_update_delete_round_trip() {
|
||||
let p = tmp_path();
|
||||
|
|
@ -51,6 +23,8 @@ fn insert_update_delete_round_trip() {
|
|||
parent: "/tmp",
|
||||
size: 42,
|
||||
mtime: 1_700_000_000,
|
||||
inode: Some(7),
|
||||
device_id: Some(64768),
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: Some(&[1, 2, 3]),
|
||||
|
|
@ -59,7 +33,15 @@ fn insert_update_delete_round_trip() {
|
|||
)
|
||||
.unwrap()
|
||||
.expect("unique path");
|
||||
set_content_done(&tx, id, "hello world", zstd_of("hello world").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
"a.txt",
|
||||
"hello world",
|
||||
&[("title".to_string(), "hi".to_string())],
|
||||
zstd_of("hello world").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
|
|
@ -105,6 +87,8 @@ fn insert_writes_content_state_from_needs_content() {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -143,6 +127,8 @@ fn update_writes_content_state_from_needs_content() {
|
|||
parent: "/tmp",
|
||||
size: 10,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::EMPTY,
|
||||
hash: None,
|
||||
|
|
@ -151,7 +137,15 @@ fn update_writes_content_state_from_needs_content() {
|
|||
let id = {
|
||||
let tx = conn.transaction().unwrap();
|
||||
let id = insert_file(&tx, &row).unwrap().expect("unique path");
|
||||
set_content_done(&tx, id, "old text", zstd_of("old text").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
"a.txt",
|
||||
"old text",
|
||||
&[],
|
||||
zstd_of("old text").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
tx.commit().unwrap();
|
||||
id
|
||||
};
|
||||
|
|
@ -178,6 +172,14 @@ fn update_writes_content_state_from_needs_content() {
|
|||
assert_eq!(got, Some(id));
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
let basic: i64 = conn
|
||||
.query_row(
|
||||
"SELECT basic_state FROM files WHERE id = ?1",
|
||||
params![id],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(basic, STATE_DONE);
|
||||
assert_eq!(content_state(&conn), STATE_PENDING);
|
||||
|
||||
let fts_hits: i64 = conn
|
||||
|
|
@ -220,6 +222,8 @@ fn insert_file_twice_on_same_path_is_idempotent() {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -260,6 +264,8 @@ fn delete_subtree_clears_every_dependent_table() {
|
|||
parent,
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -268,7 +274,15 @@ fn delete_subtree_clears_every_dependent_table() {
|
|||
)
|
||||
.unwrap()
|
||||
.expect("unique path");
|
||||
set_content_done(tx, id, "body text", zstd_of("body text").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
tx,
|
||||
id,
|
||||
name,
|
||||
"body text",
|
||||
&[("k".into(), "v".into())],
|
||||
zstd_of("body text").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
id
|
||||
};
|
||||
|
||||
|
|
@ -298,6 +312,7 @@ fn delete_subtree_clears_every_dependent_table() {
|
|||
assert_eq!(count("SELECT COUNT(*) FROM files"), 2, "siblings survive");
|
||||
assert_eq!(count("SELECT COUNT(*) FROM searchabletext"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM documents_text"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM properties"), 2);
|
||||
assert_eq!(
|
||||
count("SELECT COUNT(*) FROM failed_files"),
|
||||
0,
|
||||
|
|
@ -337,6 +352,8 @@ fn seeded(conn: &mut Connection, paths: &[&str]) -> std::collections::HashMap<St
|
|||
parent,
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -345,7 +362,15 @@ fn seeded(conn: &mut Connection, paths: &[&str]) -> std::collections::HashMap<St
|
|||
)
|
||||
.unwrap()
|
||||
.expect("unique path");
|
||||
set_content_done(&tx, id, "body text", zstd_of("body text").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
name,
|
||||
"body text",
|
||||
&[("k".into(), "v".into())],
|
||||
zstd_of("body text").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
ids.insert((*path).to_string(), id);
|
||||
}
|
||||
tx.commit().unwrap();
|
||||
|
|
@ -462,6 +487,7 @@ fn delete_ids_clears_every_dependent_table() {
|
|||
assert_eq!(count("SELECT COUNT(*) FROM files"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM searchabletext"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM documents_text"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM properties"), 2);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM failed_files"), 0);
|
||||
|
||||
// The FTS index really lost them, not just the `files` row: a
|
||||
|
|
@ -537,6 +563,7 @@ fn drop_stored_text_keeps_the_file_searchable() {
|
|||
2,
|
||||
"both files still match on content"
|
||||
);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM properties"), 2);
|
||||
|
||||
drop(conn);
|
||||
std::fs::remove_file(&p).ok();
|
||||
|
|
@ -563,20 +590,22 @@ fn reset_content_pending_clears_the_last_extraction() {
|
|||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
let row: (i64, i64) = conn
|
||||
let row: (i64, i64, Option<String>) = conn
|
||||
.query_row(
|
||||
"SELECT content_state, mtime FROM files WHERE id = ?1",
|
||||
"SELECT content_state, mtime, failure_msg FROM files WHERE id = ?1",
|
||||
params![id],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(row.0, STATE_PENDING);
|
||||
assert_eq!(row.1, 1, "metadata untouched — the file did not change");
|
||||
assert_eq!(row.2, None);
|
||||
|
||||
let count = |sql: &str| -> i64 { conn.query_row(sql, [], |r| r.get(0)).unwrap() };
|
||||
assert_eq!(count("SELECT COUNT(*) FROM files"), 2, "rows stay");
|
||||
assert_eq!(count("SELECT COUNT(*) FROM searchabletext"), 0);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM documents_text"), 0);
|
||||
assert_eq!(count("SELECT COUNT(*) FROM properties"), 0);
|
||||
assert_eq!(
|
||||
count("SELECT COUNT(*) FROM failed_files"),
|
||||
0,
|
||||
|
|
@ -697,6 +726,8 @@ fn checkpoint_and_close_truncates_wal() {
|
|||
parent: "/tmp",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::EMPTY,
|
||||
hash: None,
|
||||
|
|
@ -735,6 +766,8 @@ fn seed_rows(conn: &mut Connection, range: std::ops::Range<usize>) {
|
|||
parent: "/tmp/bulk",
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -746,7 +779,9 @@ fn seed_rows(conn: &mut Connection, range: std::ops::Range<usize>) {
|
|||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
&name,
|
||||
&"lorem ipsum dolor sit amet ".repeat(64),
|
||||
&[],
|
||||
zstd_of(&"lorem ipsum dolor sit amet ".repeat(64)).as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
|
|
@ -825,6 +860,8 @@ fn a_busy_reader_defeats_the_autocheckpoint_but_not_a_forced_one() {
|
|||
parent: "/tmp/bare",
|
||||
size: i as u64,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::TEXT,
|
||||
hash: Some(&[0u8; 32]),
|
||||
|
|
@ -983,6 +1020,8 @@ fn set_content_failed_writes_failed_table() {
|
|||
parent: "/tmp",
|
||||
size: 0,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::EMPTY,
|
||||
hash: None,
|
||||
|
|
@ -1029,6 +1068,8 @@ fn insert_at(tx: &Transaction<'_>, path: &str, needs_content: bool) -> i64 {
|
|||
parent,
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -1060,7 +1101,15 @@ fn count_root_counts_the_fts_rows_it_says_it_does() {
|
|||
// Two searchable, and one of each way a row can fail to be.
|
||||
for path in ["/tree/a.txt", "/tree/b.txt"] {
|
||||
let id = insert_at(&tx, path, true);
|
||||
set_content_done(&tx, id, "body text", zstd_of("body text").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
"n",
|
||||
"body text",
|
||||
&[],
|
||||
zstd_of("body text").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
let failed = insert_at(&tx, "/tree/c.bin", true);
|
||||
set_content_failed(&tx, failed, "bad parse").unwrap();
|
||||
|
|
@ -1085,7 +1134,15 @@ fn count_root_counts_the_fts_rows_it_says_it_does() {
|
|||
{
|
||||
let tx = conn.transaction().unwrap();
|
||||
let id = insert_at(&tx, "/elsewhere/f.txt", true);
|
||||
set_content_done(&tx, id, "body text", zstd_of("body text").as_deref()).unwrap();
|
||||
set_content_done(
|
||||
&tx,
|
||||
id,
|
||||
"n",
|
||||
"body text",
|
||||
&[],
|
||||
zstd_of("body text").as_deref(),
|
||||
)
|
||||
.unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
assert_eq!(fts_rows(&conn), 3);
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@
|
|||
//! | [`PRAGMAS_SEARCH`] | search worker | held across a typing session | 32 MiB |
|
||||
//! | [`PRAGMAS_READONLY`] | one-shot readers | a single query | 4 MiB |
|
||||
//! | [`PRAGMAS_MAINTENANCE`] | VACUUM | one bulk copy | 8 MiB |
|
||||
//! | [`PRAGMAS_WALK_READER`] | per-root walk prefetch and content feeder | the run | 1 MiB |
|
||||
//! | [`PRAGMAS_WALK_READER`] | per-root row prefetch | the walk | 1 MiB |
|
||||
//!
|
||||
//! `PRAGMA mmap_size` is absent from all of them: SQLCipher's codec disables
|
||||
//! mmap at runtime only when a key is set, mapped pages still count in
|
||||
|
|
@ -124,19 +124,12 @@ pub const PRAGMAS_READONLY: &str = "
|
|||
PRAGMA foreign_keys = ON;
|
||||
";
|
||||
|
||||
/// Pragmas for a root's own reader: the walk's row prefetch, and then the
|
||||
/// content pass's feeder ([`crate::content`]), which reuses this profile for
|
||||
/// the rest of the run.
|
||||
/// Pragmas for a walk's row-prefetch connection.
|
||||
///
|
||||
/// Two of these can exist per indexing root, so the cache size is multiplied
|
||||
/// by the root count. 1 MiB is sized for the walk's queries, which each read
|
||||
/// one range of `idx_files_parent` once and never revisit it. The feeder's
|
||||
/// paging is the same shape, but its one-off `count_extract_scope` at pass
|
||||
/// start is not: that scans the root's whole path range fetching a row per
|
||||
/// entry, so on a large root it is a cold read all the way through. It is
|
||||
/// deliberately here rather than on the writer — the writer holding still for
|
||||
/// it stopped every other root's walk — and this is the connection that pays
|
||||
/// for that, once per root.
|
||||
/// One of these exists per indexing root, so the cache size is multiplied by
|
||||
/// the root count. 1 MiB holds the upper levels of `idx_files_parent` hot,
|
||||
/// which is all these queries touch: each is a single index range lookup,
|
||||
/// and the pages under it are read once and not revisited.
|
||||
pub const PRAGMAS_WALK_READER: &str = "
|
||||
PRAGMA busy_timeout = 5000;
|
||||
PRAGMA cache_size = -1024;
|
||||
|
|
@ -163,9 +156,13 @@ CREATE TABLE files (
|
|||
parent TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
mtime INTEGER NOT NULL,
|
||||
inode INTEGER,
|
||||
device_id INTEGER,
|
||||
mime TEXT,
|
||||
type INTEGER NOT NULL DEFAULT 0,
|
||||
basic_state INTEGER NOT NULL DEFAULT 0, -- 0=pending 1=done 2=failed
|
||||
content_state INTEGER NOT NULL DEFAULT 0, -- 0=pending 1=done 2=failed 3=n/a
|
||||
failure_msg TEXT,
|
||||
hash BLOB
|
||||
);
|
||||
|
||||
|
|
@ -184,6 +181,13 @@ CREATE INDEX idx_files_mime ON files(mime);
|
|||
CREATE INDEX idx_files_hash ON files(hash);
|
||||
CREATE INDEX idx_files_content_pending ON files(id) WHERE content_state = 0;
|
||||
|
||||
CREATE TABLE properties (
|
||||
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
||||
key TEXT NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
PRIMARY KEY (file_id, key)
|
||||
);
|
||||
|
||||
CREATE TABLE failed_files (
|
||||
file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE,
|
||||
reason TEXT,
|
||||
|
|
@ -193,16 +197,11 @@ CREATE TABLE failed_files (
|
|||
-- Canonical extracted text for every successfully content-indexed file.
|
||||
-- Compressed with zstd (see `crate::db::repo::set_content_done`). Only
|
||||
-- written when the extractor produced text; absent rows mean "no body
|
||||
-- text" (e.g. an audio file whose tags are all empty). "Has a row here" is
|
||||
-- therefore *not* the same as "content indexed" — `files.content_state` is
|
||||
-- the authority on that.
|
||||
--
|
||||
-- The uncompressed length is not stored: zstd records it in the frame
|
||||
-- header, so `crate::db::repo::raw_text_len` reads it back for the one
|
||||
-- caller (the size report) that wants it.
|
||||
-- text" (e.g. an image with only EXIF properties).
|
||||
CREATE TABLE documents_text (
|
||||
file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE,
|
||||
text_zstd BLOB NOT NULL
|
||||
text_zstd BLOB NOT NULL,
|
||||
text_len INTEGER NOT NULL -- original byte length pre-compression
|
||||
);
|
||||
|
||||
CREATE TABLE config_validation (
|
||||
|
|
@ -218,17 +217,11 @@ CREATE TABLE config_validation (
|
|||
/// rowid=?` without replaying the original row text, at the cost of a modest
|
||||
/// tombstone bitmap. Built-in `snippet()` is unavailable in contentless
|
||||
/// mode — snippets are rendered in Rust from `documents_text` instead.
|
||||
///
|
||||
/// **One column, deliberately.** Document bodies are the only thing anything
|
||||
/// ever MATCHes: the cascade pins its query to the body
|
||||
/// (`crate::search::cascade::passes`) and filename ranks come from scanning
|
||||
/// `files.name`, which the trigram index of a `name` column here would only
|
||||
/// duplicate — at (len − 2) postings per file indexed.
|
||||
pub fn fts_create_sql(tokenizer: &str) -> String {
|
||||
let effective = effective_tokenizer(tokenizer);
|
||||
format!(
|
||||
"CREATE VIRTUAL TABLE searchabletext USING fts5(\
|
||||
text, \
|
||||
name, text, properties, \
|
||||
tokenize='{}', \
|
||||
content='', \
|
||||
contentless_delete=1\
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
//! Audio tag extraction via [`lofty`]. Concatenates the searchable tag
|
||||
//! values — title, artist, album, genre, comment — into `text` so full-text
|
||||
//! search works across them.
|
||||
//! Audio tag extraction via [`lofty`]. Pulls title/artist/album/genre/year/
|
||||
//! track/duration into properties; concatenates tag values into `text` so
|
||||
//! full-text search works across them.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use lofty::{
|
||||
file::TaggedFileExt,
|
||||
file::{AudioFile, TaggedFileExt},
|
||||
probe::Probe,
|
||||
tag::{Accessor, ItemKey},
|
||||
};
|
||||
|
|
@ -25,42 +25,58 @@ impl Extractor for AudioExtractor {
|
|||
.read()
|
||||
.map_err(|e| format!("lofty read {}: {}", path.display(), e))?;
|
||||
|
||||
// properties (parked): year, track and duration went to the property
|
||||
// map alone and never reached `text`, so nothing collects them now.
|
||||
// See `super::ExtractedContent`.
|
||||
let mut pieces: Vec<String> = Vec::new();
|
||||
if let Some(tag) = tagged.primary_tag().or_else(|| tagged.first_tag()) {
|
||||
let mut push = |value: Option<String>| {
|
||||
if let Some(v) = value.filter(|v: &String| !v.is_empty()) {
|
||||
pieces.push(v);
|
||||
}
|
||||
};
|
||||
// `ItemKey` first, falling back to the `Accessor` shortcut for the
|
||||
// three fields that have one — a tag can carry the value under
|
||||
// either.
|
||||
push(
|
||||
tag.get_string(&ItemKey::TrackTitle)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(str::to_string)
|
||||
.or_else(|| tag.title().map(|t| t.to_string())),
|
||||
);
|
||||
push(
|
||||
tag.get_string(&ItemKey::TrackArtist)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(str::to_string)
|
||||
.or_else(|| tag.artist().map(|a| a.to_string())),
|
||||
);
|
||||
push(
|
||||
tag.get_string(&ItemKey::AlbumTitle)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(str::to_string)
|
||||
.or_else(|| tag.album().map(|a| a.to_string())),
|
||||
);
|
||||
push(tag.get_string(&ItemKey::Genre).map(str::to_string));
|
||||
push(tag.get_string(&ItemKey::Comment).map(str::to_string));
|
||||
let mut out = ExtractedContent::default();
|
||||
|
||||
let duration_secs = tagged.properties().duration().as_secs();
|
||||
if duration_secs > 0 {
|
||||
out.properties
|
||||
.insert("duration".to_string(), duration_secs.to_string());
|
||||
}
|
||||
|
||||
Ok(ExtractedContent::with_text(pieces.join(" ")))
|
||||
if let Some(tag) = tagged.primary_tag().or_else(|| tagged.first_tag()) {
|
||||
for (key, item_key) in [
|
||||
("title", ItemKey::TrackTitle),
|
||||
("artist", ItemKey::TrackArtist),
|
||||
("album", ItemKey::AlbumTitle),
|
||||
("genre", ItemKey::Genre),
|
||||
("year", ItemKey::Year),
|
||||
("track", ItemKey::TrackNumber),
|
||||
("comment", ItemKey::Comment),
|
||||
] {
|
||||
if let Some(v) = tag.get_string(&item_key) {
|
||||
if !v.is_empty() {
|
||||
out.properties.insert(key.to_string(), v.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
// Accessor shortcuts for common fields if the ItemKey lookup missed.
|
||||
if !out.properties.contains_key("title") {
|
||||
if let Some(t) = tag.title() {
|
||||
out.properties.insert("title".to_string(), t.to_string());
|
||||
}
|
||||
}
|
||||
if !out.properties.contains_key("artist") {
|
||||
if let Some(a) = tag.artist() {
|
||||
out.properties.insert("artist".to_string(), a.to_string());
|
||||
}
|
||||
}
|
||||
if !out.properties.contains_key("album") {
|
||||
if let Some(a) = tag.album() {
|
||||
out.properties.insert("album".to_string(), a.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Join the searchable tag values into one blob so FTS hits on any of them.
|
||||
let mut pieces: Vec<&str> = Vec::new();
|
||||
for k in ["title", "artist", "album", "genre", "comment"] {
|
||||
if let Some(v) = out.properties.get(k) {
|
||||
pieces.push(v.as_str());
|
||||
}
|
||||
}
|
||||
out.text = pieces.join(" ");
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -68,84 +84,6 @@ impl Extractor for AudioExtractor {
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A minimal but real MPEG file: an ID3v2.3 tag carrying `frames`
|
||||
/// (`("TPE1", "…")` and friends), followed by one silent MPEG-1 Layer III
|
||||
/// frame so the probe recognizes the format from its content.
|
||||
fn write_mp3(tag: &str, frames: &[(&str, &str)]) -> std::path::PathBuf {
|
||||
let mut body = Vec::new();
|
||||
for (id, value) in frames {
|
||||
let mut payload = vec![0x00]; // ISO-8859-1
|
||||
payload.extend_from_slice(value.as_bytes());
|
||||
body.extend_from_slice(id.as_bytes());
|
||||
// ID3v2.3 frame sizes are plain big-endian, unlike the tag size.
|
||||
body.extend_from_slice(&(payload.len() as u32).to_be_bytes());
|
||||
body.extend_from_slice(&[0, 0]); // flags
|
||||
body.extend_from_slice(&payload);
|
||||
}
|
||||
|
||||
let mut out = Vec::new();
|
||||
out.extend_from_slice(b"ID3");
|
||||
out.extend_from_slice(&[0x03, 0x00, 0x00]); // v2.3, no flags
|
||||
// Tag size is syncsafe: seven bits per byte.
|
||||
let n = body.len() as u32;
|
||||
out.extend_from_slice(&[
|
||||
((n >> 21) & 0x7F) as u8,
|
||||
((n >> 14) & 0x7F) as u8,
|
||||
((n >> 7) & 0x7F) as u8,
|
||||
(n & 0x7F) as u8,
|
||||
]);
|
||||
out.extend_from_slice(&body);
|
||||
|
||||
// MPEG-1 Layer III, 128 kbps, 44.1 kHz, no padding: 417-byte frames.
|
||||
// Four of them, because the probe confirms a sync word by finding the
|
||||
// next frame where the first one says it will be.
|
||||
for _ in 0..4 {
|
||||
out.extend_from_slice(&[0xFF, 0xFB, 0x90, 0x00]);
|
||||
out.resize(out.len() + 413, 0);
|
||||
}
|
||||
|
||||
let path = crate::testutil::scratch_dir(tag).join("track.mp3");
|
||||
std::fs::write(&path, &out).expect("write fixture mp3");
|
||||
path
|
||||
}
|
||||
|
||||
/// The searchable text is assembled from the tag values, which is the
|
||||
/// only reason audio files are full-text indexed at all. Pins the
|
||||
/// rewrite that dropped the property map this used to be built from.
|
||||
#[test]
|
||||
fn tag_values_become_searchable_text() {
|
||||
let path = write_mp3(
|
||||
"audio-tags",
|
||||
&[
|
||||
("TIT2", "Blue Monday"),
|
||||
("TPE1", "New Order"),
|
||||
("TALB", "Power Corruption"),
|
||||
("TCON", "Synthpop"),
|
||||
],
|
||||
);
|
||||
let out = AudioExtractor.extract(&path).expect("extract");
|
||||
for expected in ["Blue Monday", "New Order", "Power Corruption", "Synthpop"] {
|
||||
assert!(
|
||||
out.text.contains(expected),
|
||||
"{:?} missing from {:?}",
|
||||
expected,
|
||||
out.text
|
||||
);
|
||||
}
|
||||
// Title first, then artist, album, genre — a stable order so the
|
||||
// stored text does not churn between runs.
|
||||
assert_eq!(out.text, "Blue Monday New Order Power Corruption Synthpop");
|
||||
}
|
||||
|
||||
/// No tags at all is a successful extraction with nothing to store, not
|
||||
/// a failure — `set_content_done` writes no sidecar row for it.
|
||||
#[test]
|
||||
fn an_untagged_file_yields_empty_text() {
|
||||
let path = write_mp3("audio-untagged", &[]);
|
||||
let out = AudioExtractor.extract(&path).expect("extract");
|
||||
assert!(out.text.is_empty(), "unexpected text {:?}", out.text);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn supports_audio_mimes() {
|
||||
let e = AudioExtractor;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
//! Content extractors: the searchable text of a file.
|
||||
//! Content extractors: text plus structured properties (title, artist, EXIF, …).
|
||||
//!
|
||||
//! An [`Extractor`] decides whether it can handle a given MIME type and, if
|
||||
//! so, produces [`ExtractedContent`] for the file. The [`Registry`] picks the
|
||||
|
|
@ -9,59 +9,55 @@
|
|||
//! is decided once, upstream in [`crate::mime::guess_mime_from_head`];
|
||||
//! nothing downstream reopens the file to ask again.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
|
||||
pub mod audio;
|
||||
// pub mod image; // parked — see `ExtractedContent` below
|
||||
pub mod image;
|
||||
pub mod office;
|
||||
pub mod ole;
|
||||
pub mod pdf;
|
||||
pub mod plaintext;
|
||||
pub mod rtf;
|
||||
|
||||
/// Result of a successful extraction: `text` feeds the FTS5 `text` column.
|
||||
/// Result of a successful extraction. `text` feeds the FTS5 `text` column;
|
||||
/// `properties` feeds both the `properties` FTS5 column (as `key:value`
|
||||
/// tokens) and the structured `properties` table for later retrieval.
|
||||
///
|
||||
/// Extractors may return an empty `text` when the file has no narrative
|
||||
/// content (an audio file whose tags are all empty, say). Filename search
|
||||
/// still works in that case.
|
||||
///
|
||||
/// # Structured properties are parked
|
||||
///
|
||||
/// Extractors used to return a `properties: HashMap<String, String>` beside
|
||||
/// the text — EXIF, audio tags, the PDF `Info` dictionary — stored in a
|
||||
/// `properties` table *and* concatenated into a `properties` FTS column.
|
||||
/// Nothing ever read either back: no query, no result row, no UI. So the
|
||||
/// storage is gone and the extraction is commented out rather than deleted.
|
||||
///
|
||||
/// Reviving it means restoring, together: this field and
|
||||
/// `properties_sorted`, the blocks marked "properties (parked)" in
|
||||
/// `image.rs` / `audio.rs` / `pdf.rs`, the `image` module registration in
|
||||
/// [`Registry::default_set`], the `properties` table and FTS column in
|
||||
/// [`crate::db::schema`], the `properties` argument to
|
||||
/// [`crate::db::repo::set_content_done`] — and a consumer that shows them.
|
||||
/// content (e.g. an image where only EXIF matters). Filename search still
|
||||
/// works in that case.
|
||||
#[derive(Debug, Default, Clone)]
|
||||
pub struct ExtractedContent {
|
||||
pub text: String,
|
||||
// pub properties: HashMap<String, String>,
|
||||
pub properties: HashMap<String, String>,
|
||||
}
|
||||
|
||||
impl ExtractedContent {
|
||||
pub fn with_text(text: impl Into<String>) -> Self {
|
||||
Self { text: text.into() }
|
||||
Self {
|
||||
text: text.into(),
|
||||
properties: HashMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
// /// Convert properties into the `Vec<(String, String)>` shape expected by
|
||||
// /// [`crate::db::repo::set_content_done`]. Keys are sorted for determinism
|
||||
// /// in tests and snapshots.
|
||||
// pub fn properties_sorted(&self) -> Vec<(String, String)> {
|
||||
// let mut v: Vec<(String, String)> = self
|
||||
// .properties
|
||||
// .iter()
|
||||
// .map(|(k, v)| (k.clone(), v.clone()))
|
||||
// .collect();
|
||||
// v.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
// v
|
||||
// }
|
||||
pub fn with_property(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
|
||||
self.properties.insert(key.into(), value.into());
|
||||
self
|
||||
}
|
||||
|
||||
/// Convert properties into the `Vec<(String, String)>` shape expected by
|
||||
/// [`crate::db::repo::set_content_done`]. Keys are sorted for determinism
|
||||
/// in tests and snapshots.
|
||||
pub fn properties_sorted(&self) -> Vec<(String, String)> {
|
||||
let mut v: Vec<(String, String)> = self
|
||||
.properties
|
||||
.iter()
|
||||
.map(|(k, v)| (k.clone(), v.clone()))
|
||||
.collect();
|
||||
v.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
/// Boxed error type for extractor failures. A string reason is stored on the
|
||||
|
|
@ -161,21 +157,15 @@ impl Registry {
|
|||
.and_then(|e| e.extract_from_head(path, head))
|
||||
}
|
||||
|
||||
/// The default set: RTF, plaintext, office docs, PDF, audio tags.
|
||||
/// The default set: RTF, plaintext, office docs, PDF, audio tags,
|
||||
/// image EXIF.
|
||||
///
|
||||
/// Order matters — the first extractor whose `supports` accepts a MIME
|
||||
/// wins. RTF precedes plaintext because plaintext claims every `text/*`
|
||||
/// and would swallow `text/rtf` as raw control words. Plaintext
|
||||
/// precedes audio because it deliberately claims playlist
|
||||
/// precedes audio and image because it deliberately claims playlist
|
||||
/// (`audio/x-mpegurl`, `audio/scpls`) and SVG MIMEs whose text is worth
|
||||
/// more than their tags.
|
||||
///
|
||||
/// No image extractor: it produced EXIF properties and never any text,
|
||||
/// so with properties parked it would open and parse every image on
|
||||
/// disk to return nothing. Leaving `image/*` unclaimed is what makes
|
||||
/// [`crate::file_handling::content_extractable`] record images as
|
||||
/// `STATE_NA` at walk time, so the content pass never opens them.
|
||||
/// Filenames are indexed exactly as before.
|
||||
pub fn default_set() -> Self {
|
||||
Self::new()
|
||||
.with(rtf::RtfExtractor)
|
||||
|
|
@ -183,7 +173,7 @@ impl Registry {
|
|||
.with(office::OfficeExtractor)
|
||||
.with(pdf::PdfExtractor)
|
||||
.with(audio::AudioExtractor)
|
||||
// .with(image::ImageExtractor) // parked with `ExtractedContent`
|
||||
.with(image::ImageExtractor)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -220,7 +210,6 @@ mod tests {
|
|||
assert!(r
|
||||
.extract_complete_head(p, "application/pdf", b"%PDF-1.4")
|
||||
.is_none());
|
||||
// No extractor claims images at all now — the head path must agree.
|
||||
assert!(r
|
||||
.extract_complete_head(p, "image/png", b"\x89PNG")
|
||||
.is_none());
|
||||
|
|
@ -309,13 +298,17 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
/// Images are claimed by nothing, so the walk records them `NA` and the
|
||||
/// content pass never opens them. Pins the parked image extractor.
|
||||
#[test]
|
||||
fn images_are_not_claimed_by_any_extractor() {
|
||||
let r = Registry::default_set();
|
||||
for mime in ["image/jpeg", "image/png", "Image/JPEG", "image/tiff"] {
|
||||
assert!(!r.supports(mime), "{} should be unclaimed", mime);
|
||||
}
|
||||
fn properties_sorted_is_deterministic() {
|
||||
let c = ExtractedContent::with_text("hi")
|
||||
.with_property("b", "2")
|
||||
.with_property("a", "1");
|
||||
assert_eq!(
|
||||
c.properties_sorted(),
|
||||
vec![
|
||||
("a".to_string(), "1".to_string()),
|
||||
("b".to_string(), "2".to_string())
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
//! PDF text extraction.
|
||||
//!
|
||||
//! One `Document::load` per file, then the text is taken off it.
|
||||
//! `pdf_extract` can panic or hard-error on malformed files; any failure is
|
||||
//! surfaced to the caller and marks the file's content state as failed.
|
||||
//! One `Document::load` per file, then both the text and the `Info` dictionary
|
||||
//! are taken off it. `pdf_extract` can panic or hard-error on malformed files;
|
||||
//! any failure is surfaced to the caller and marks the file's content state as
|
||||
//! failed.
|
||||
//!
|
||||
//! `lopdf` is reached through `pdf_extract`'s own `pub use lopdf::*` and must
|
||||
//! **not** be declared in `Cargo.toml` again: a direct declaration resolved a
|
||||
|
|
@ -15,7 +16,7 @@ use std::cell::Cell;
|
|||
use std::path::Path;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
use pdf_extract::{Document, PlainTextOutput};
|
||||
use pdf_extract::{Document, Object, PlainTextOutput};
|
||||
|
||||
use super::{ExtractError, ExtractedContent, Extractor};
|
||||
|
||||
|
|
@ -74,18 +75,15 @@ impl Extractor for PdfExtractor {
|
|||
}
|
||||
}
|
||||
|
||||
// properties (parked) — the `Info` dictionary read. See
|
||||
// `super::ExtractedContent`; reviving this also needs the `Object` import
|
||||
// above and `object_to_string` below.
|
||||
//
|
||||
// /// The six `Info` keys worth keeping, in the order they are written.
|
||||
// const INFO_KEYS: [&str; 6] = [
|
||||
// "Title", "Author", "Subject", "Keywords", "Creator", "Producer",
|
||||
// ];
|
||||
/// The six `Info` keys worth keeping, in the order they are written.
|
||||
const INFO_KEYS: [&str; 6] = [
|
||||
"Title", "Author", "Subject", "Keywords", "Creator", "Producer",
|
||||
];
|
||||
|
||||
/// Load the document once and take the text off it. This is
|
||||
/// `pdf_extract::extract_text`'s body (load, decrypt, `output_doc`) spelled
|
||||
/// out, which is also what kept the document in scope for the `Info` read.
|
||||
/// Load the document once; take the text and the `Info` dictionary off it.
|
||||
/// This is `pdf_extract::extract_text`'s body (load, decrypt, `output_doc`)
|
||||
/// spelled out so the `Info` read happens while the document is still in
|
||||
/// scope.
|
||||
fn extract_one_pass(path: &Path) -> Result<ExtractedContent, ExtractError> {
|
||||
let mut doc = Document::load(path).map_err(|e| format!("pdf_extract: {}", e))?;
|
||||
// Decryption must happen before either the content streams or the `Info`
|
||||
|
|
@ -100,35 +98,35 @@ fn extract_one_pass(path: &Path) -> Result<ExtractedContent, ExtractError> {
|
|||
let mut sink = PlainTextOutput::new(&mut text);
|
||||
pdf_extract::output_doc(&doc, &mut sink).map_err(|e| format!("pdf_extract: {}", e))?;
|
||||
}
|
||||
// properties (parked): the `Info` dictionary was read here, soft-failing
|
||||
// when absent because the text is the half that matters. It is all that
|
||||
// held `doc` open past `output_doc`.
|
||||
//
|
||||
// let info = doc
|
||||
// .trailer
|
||||
// .get(b"Info")
|
||||
// .ok()
|
||||
// .and_then(|o| o.as_reference().ok())
|
||||
// .and_then(|id| doc.get_object(id).ok())
|
||||
// .and_then(|o| o.as_dict().ok());
|
||||
// if let Some(dict) = info {
|
||||
// for key in INFO_KEYS {
|
||||
// if let Some(s) = dict.get(key.as_bytes()).ok().and_then(object_to_string) {
|
||||
// if !s.is_empty() {
|
||||
// out.properties.insert(key.to_ascii_lowercase(), s);
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// fn object_to_string(obj: &Object) -> Option<String> {
|
||||
// match obj {
|
||||
// Object::String(bytes, _) => Some(String::from_utf8_lossy(bytes).into_owned()),
|
||||
// Object::Name(bytes) => Some(String::from_utf8_lossy(bytes).into_owned()),
|
||||
// _ => None,
|
||||
// }
|
||||
// }
|
||||
Ok(ExtractedContent::with_text(text))
|
||||
let mut out = ExtractedContent::with_text(text);
|
||||
|
||||
// Soft-fail, unchanged: a document with no readable `Info` dictionary
|
||||
// still has its text, and the text is the half that matters.
|
||||
let info = doc
|
||||
.trailer
|
||||
.get(b"Info")
|
||||
.ok()
|
||||
.and_then(|o| o.as_reference().ok())
|
||||
.and_then(|id| doc.get_object(id).ok())
|
||||
.and_then(|o| o.as_dict().ok());
|
||||
if let Some(dict) = info {
|
||||
for key in INFO_KEYS {
|
||||
if let Some(s) = dict.get(key.as_bytes()).ok().and_then(object_to_string) {
|
||||
if !s.is_empty() {
|
||||
out.properties.insert(key.to_ascii_lowercase(), s);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn object_to_string(obj: &Object) -> Option<String> {
|
||||
match obj {
|
||||
Object::String(bytes, _) => Some(String::from_utf8_lossy(bytes).into_owned()),
|
||||
Object::Name(bytes) => Some(String::from_utf8_lossy(bytes).into_owned()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
@ -155,7 +153,7 @@ mod tests {
|
|||
assert!(!PdfExtractor.supports("application/zip"));
|
||||
}
|
||||
|
||||
use pdf_extract::{dictionary, Dictionary, Object, Stream, StringFormat};
|
||||
use pdf_extract::{dictionary, Dictionary, Stream, StringFormat};
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// Write a one-page PDF drawing `body`, with `info` as its `Info`
|
||||
|
|
@ -215,10 +213,8 @@ mod tests {
|
|||
Object::String(s.as_bytes().to_vec(), StringFormat::Literal)
|
||||
}
|
||||
|
||||
/// A document *with* an `Info` dictionary still extracts its text — the
|
||||
/// dictionary is no longer read, and must not get in the way.
|
||||
#[test]
|
||||
fn extracts_text_from_a_document_with_an_info_dictionary() {
|
||||
fn extracts_text_and_info_properties() {
|
||||
let path = write_pdf(
|
||||
"pdf-full",
|
||||
"Hello QuickSearch",
|
||||
|
|
@ -238,6 +234,32 @@ mod tests {
|
|||
"drawn text missing from {:?}",
|
||||
out.text
|
||||
);
|
||||
// Lowercased keys, which is the contract the rest of the pipeline
|
||||
// stores under.
|
||||
assert_eq!(
|
||||
out.properties.get("title").map(String::as_str),
|
||||
Some("The Title")
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("author").map(String::as_str),
|
||||
Some("An Author")
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("subject").map(String::as_str),
|
||||
Some("A Subject")
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("keywords").map(String::as_str),
|
||||
Some("alpha beta")
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("creator").map(String::as_str),
|
||||
Some("A Creator")
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("producer").map(String::as_str),
|
||||
Some("A Producer")
|
||||
);
|
||||
}
|
||||
|
||||
/// The soft-fail path: no `Info` dictionary is not an extraction failure,
|
||||
|
|
@ -247,6 +269,54 @@ mod tests {
|
|||
let path = write_pdf("pdf-noinfo", "Body Only", None);
|
||||
let out = PdfExtractor.extract(&path).expect("extract");
|
||||
assert!(out.text.contains("Body Only"));
|
||||
assert!(
|
||||
out.properties.is_empty(),
|
||||
"unexpected properties: {:?}",
|
||||
out.properties
|
||||
);
|
||||
}
|
||||
|
||||
/// An empty `Info` value is absence, not an empty property.
|
||||
#[test]
|
||||
fn empty_info_values_are_not_stored() {
|
||||
let path = write_pdf(
|
||||
"pdf-emptyinfo",
|
||||
"Body",
|
||||
Some(dictionary! {
|
||||
"Title" => text_string(""),
|
||||
"Author" => text_string("Real Author"),
|
||||
}),
|
||||
);
|
||||
let out = PdfExtractor.extract(&path).expect("extract");
|
||||
assert!(!out.properties.contains_key("title"), "empty title stored");
|
||||
assert_eq!(
|
||||
out.properties.get("author").map(String::as_str),
|
||||
Some("Real Author")
|
||||
);
|
||||
}
|
||||
|
||||
/// `Info` values that are not strings or names are skipped rather than
|
||||
/// rendered — pins `object_to_string`'s catch-all arm.
|
||||
#[test]
|
||||
fn non_string_info_values_are_skipped() {
|
||||
let path = write_pdf(
|
||||
"pdf-badinfo",
|
||||
"Body",
|
||||
Some(dictionary! {
|
||||
"Producer" => 42,
|
||||
"Title" => text_string("Kept"),
|
||||
}),
|
||||
);
|
||||
let out = PdfExtractor.extract(&path).expect("extract");
|
||||
assert!(
|
||||
!out.properties.contains_key("producer"),
|
||||
"integer Info value was rendered: {:?}",
|
||||
out.properties
|
||||
);
|
||||
assert_eq!(
|
||||
out.properties.get("title").map(String::as_str),
|
||||
Some("Kept")
|
||||
);
|
||||
}
|
||||
|
||||
/// Malformed input must come back as an error, not take the process
|
||||
|
|
|
|||
|
|
@ -145,6 +145,7 @@ mod tests {
|
|||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(from_disk.text, from_head.text);
|
||||
assert_eq!(from_disk.properties, from_head.properties);
|
||||
std::fs::remove_file(&p).ok();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ use rusqlite::Connection;
|
|||
use super::*;
|
||||
use crate::config::Config;
|
||||
use crate::db::repo::{self};
|
||||
use crate::indexing::should_abort;
|
||||
|
||||
/// The compressed sidecar for one row, or `None` where there is none to write
|
||||
/// — an empty body, or `store_text_for_snippets` turned off.
|
||||
|
|
@ -28,13 +29,12 @@ type Body = Result<Option<Vec<u8>>, String>;
|
|||
///
|
||||
/// What that lock does *not* gate, so the benefit is not overclaimed: search
|
||||
/// holds its own connection (`db::open::open_search_reader`) and the database
|
||||
/// is WAL, where a reader never blocks on a writer. Nor does it separate one
|
||||
/// root from another — every root's writes already run on the single writer
|
||||
/// thread, so two of them are never inside the lock at once. What it actually
|
||||
/// serializes the run against is WAL checkpointing, which `run_indexing`
|
||||
/// forces from the same thread between turns. A whole-tree wall-clock run is
|
||||
/// dominated by FTS5 trigram tokenization and does not move measurably from
|
||||
/// this change; it is the length of the hold that improves, not throughput.
|
||||
/// is WAL, where a reader never blocks on a writer. `conn_mutex` serializes
|
||||
/// the indexer against itself — one root's content stores against another's
|
||||
/// walk inserts, the scope reconciler's slices, and WAL checkpointing. A
|
||||
/// whole-tree wall-clock run is dominated by FTS5 trigram tokenization and
|
||||
/// does not move measurably from this change; it is contention that improves,
|
||||
/// not throughput.
|
||||
fn compress_bodies<'a>(
|
||||
texts: impl Iterator<Item = Option<&'a str>>,
|
||||
config: &Config,
|
||||
|
|
@ -125,7 +125,7 @@ pub fn process_batch_updates(
|
|||
|
||||
if let (Some(id), Some(text)) = (id, rec.inline_text.as_deref()) {
|
||||
let zstd = body_or_skip!(bodies, i, rec.path);
|
||||
repo::set_content_done(&tx, id, text, zstd)?;
|
||||
store_inline_text(&tx, id, rec, text, zstd)?;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -136,6 +136,19 @@ pub fn process_batch_updates(
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Store text the walk already extracted, so the content pass skips this row.
|
||||
/// Same [`repo::set_content_done`] the content pass calls, so a row finished
|
||||
/// here is indistinguishable from one finished there.
|
||||
pub(crate) fn store_inline_text(
|
||||
tx: &rusqlite::Transaction<'_>,
|
||||
file_id: i64,
|
||||
rec: &OwnedNewFile,
|
||||
text: &str,
|
||||
text_zstd: Option<&[u8]>,
|
||||
) -> Result<(), String> {
|
||||
repo::set_content_done(tx, file_id, &rec.name, text, &[], text_zstd)
|
||||
}
|
||||
|
||||
/// Write already-prepared records for newly discovered files. Silent, like
|
||||
/// [`process_batch_updates`], and likewise stores any text the walk already
|
||||
/// extracted.
|
||||
|
|
@ -171,7 +184,7 @@ pub fn process_batch_inserts(
|
|||
.map_err(|e| format!("Failed to insert file record: {}", e))?;
|
||||
if let (Some(id), Some(text)) = (id, rec.inline_text.as_deref()) {
|
||||
let zstd = body_or_skip!(bodies, i, rec.path);
|
||||
repo::set_content_done(&tx, id, text, zstd)?;
|
||||
store_inline_text(&tx, id, rec, text, zstd)?;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -185,13 +198,15 @@ pub fn process_batch_inserts(
|
|||
/// Delete the rows a completed run found no file behind, in chunked
|
||||
/// transactions. Returns how many went.
|
||||
///
|
||||
/// The stop flag is checked between chunks and again per path, never with a
|
||||
/// transaction open: a chunk either commits whole or is not begun, so a stop
|
||||
/// cannot leave the index half-reconciled.
|
||||
/// `should_abort` *blocks* while the indexer is suspended, so it must only be
|
||||
/// observed between chunks with nothing held — checking it mid-transaction
|
||||
/// pins the shared connection for the whole suspension and freezes the GUI.
|
||||
/// The stop flag, which never blocks, guards the inner loop.
|
||||
pub fn cleanup_stale_index_entries(
|
||||
conn_mutex: &Arc<Mutex<Connection>>,
|
||||
stale_paths: &[String],
|
||||
stop_flag: &Arc<AtomicBool>,
|
||||
suspend_flag: &Arc<AtomicBool>,
|
||||
config: &Config,
|
||||
) -> Result<usize, String> {
|
||||
if stale_paths.is_empty() {
|
||||
|
|
@ -201,8 +216,8 @@ pub fn cleanup_stale_index_entries(
|
|||
let mut deleted_count = 0usize;
|
||||
|
||||
for batch in stale_paths.chunks(chunk) {
|
||||
// Outside the lock, so a stop is seen before a transaction is begun.
|
||||
if stop_flag.load(Ordering::Relaxed) {
|
||||
// Outside the lock, so a suspend parks here rather than mid-transaction.
|
||||
if should_abort(stop_flag, suspend_flag) {
|
||||
return Ok(deleted_count);
|
||||
}
|
||||
let conn = crate::lock_ok(conn_mutex);
|
||||
|
|
@ -226,7 +241,7 @@ pub fn cleanup_stale_index_entries(
|
|||
}
|
||||
}
|
||||
|
||||
if deleted_count > 0 && !stop_flag.load(Ordering::Relaxed) {
|
||||
if deleted_count > 0 && !should_abort(stop_flag, suspend_flag) {
|
||||
let conn = crate::lock_ok(conn_mutex);
|
||||
fts_finalize_after_text_indexing(&conn);
|
||||
}
|
||||
|
|
@ -281,107 +296,71 @@ pub struct ExtractScope {
|
|||
pub already_done: usize,
|
||||
}
|
||||
|
||||
/// The `maximum_text_file_size` bound as the SQL below compares it.
|
||||
pub(crate) fn max_text_file_size(config: &Config) -> i64 {
|
||||
i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX)
|
||||
}
|
||||
|
||||
/// Flip a root's oversize pending rows to NA. Idempotent.
|
||||
/// Prepare a root's extraction scope: flip oversize pending rows to NA
|
||||
/// (idempotent) and count what is pending vs. already extracted in the range.
|
||||
///
|
||||
/// Covers what walk-time decisions cannot: a `maximum_text_file_size`
|
||||
/// *lowered* between runs (which does not force a rebuild), and rows left
|
||||
/// pending by an older build. Rows this misses would stay pending forever, so
|
||||
/// it runs on the writer before a root's content pass starts.
|
||||
pub fn mark_oversize_pending_na(
|
||||
conn: &Connection,
|
||||
cursor: &ExtractCursor,
|
||||
config: &Config,
|
||||
) -> Result<(), String> {
|
||||
conn.execute(
|
||||
"UPDATE files SET content_state = 3 \
|
||||
WHERE content_state = 0 AND size > ?1 AND path >= ?2 AND path < ?3",
|
||||
rusqlite::params![max_text_file_size(config), cursor.lo, cursor.hi],
|
||||
)
|
||||
.map_err(|e| format!("mark oversize files NA: {}", e))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Count what a root's range holds: rows still to extract this run, and rows
|
||||
/// whose text is already searchable from earlier runs.
|
||||
///
|
||||
/// One range scan for both figures. Deliberately callable on any connection
|
||||
/// — the content pass runs it on its own read connection rather than on the
|
||||
/// indexer's writer, because on a large root it takes seconds, and seconds of
|
||||
/// writer time is every other root's walk standing still.
|
||||
pub fn count_extract_scope(
|
||||
conn: &Connection,
|
||||
/// The oversize sweep covers what walk-time decisions cannot: a
|
||||
/// `maximum_text_file_size` *lowered* between runs (which does not force a
|
||||
/// rebuild), and rows left pending by an older build.
|
||||
pub fn extract_scope_prepare(
|
||||
conn_mutex: &Arc<Mutex<Connection>>,
|
||||
cursor: &ExtractCursor,
|
||||
config: &Config,
|
||||
) -> Result<ExtractScope, String> {
|
||||
let (pending, already_done): (i64, i64) = conn
|
||||
.query_row(
|
||||
"SELECT COALESCE(SUM(content_state = 0 AND size <= ?1), 0), \
|
||||
COALESCE(SUM(content_state = 1), 0) \
|
||||
FROM files WHERE path >= ?2 AND path < ?3",
|
||||
rusqlite::params![max_text_file_size(config), cursor.lo, cursor.hi],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
let max_size = i64::try_from(config.processing.maximum_text_file_size).unwrap_or(i64::MAX);
|
||||
let conn = crate::lock_ok(conn_mutex);
|
||||
conn.execute(
|
||||
"UPDATE files SET content_state = 3 \
|
||||
WHERE content_state = 0 AND size > ?1 AND path >= ?2 AND path < ?3",
|
||||
rusqlite::params![max_size, cursor.lo, cursor.hi],
|
||||
)
|
||||
.map_err(|e| format!("Failed to count text files: {}", e))?;
|
||||
.map_err(|e| format!("mark oversize files NA: {}", e))?;
|
||||
let pending: i64 = conn
|
||||
.query_row(
|
||||
"SELECT COUNT(*) FROM files \
|
||||
WHERE content_state = 0 AND size <= ?1 AND path >= ?2 AND path < ?3",
|
||||
rusqlite::params![max_size, cursor.lo, cursor.hi],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.map_err(|e| format!("Failed to count pending text files: {}", e))?;
|
||||
let already_done: i64 = conn
|
||||
.query_row(
|
||||
"SELECT COUNT(*) FROM files \
|
||||
WHERE content_state = 1 AND path >= ?1 AND path < ?2",
|
||||
rusqlite::params![cursor.lo, cursor.hi],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.map_err(|e| format!("Failed to count extracted files: {}", e))?;
|
||||
Ok(ExtractScope {
|
||||
pending: pending.max(0) as usize,
|
||||
already_done: already_done.max(0) as usize,
|
||||
})
|
||||
}
|
||||
|
||||
/// Rows per compression chunk and per transaction inside [`store_extracted`].
|
||||
/// Write a batch of already-extracted rows — the cheap half of the content
|
||||
/// pass, and all that runs with the connection held. Chunked so each
|
||||
/// transaction stays short.
|
||||
///
|
||||
/// Half of what a writer turn may hand in (`pipeline::READY_TOPUP` is 64), so
|
||||
/// a full turn commits twice rather than once — short holds of the connection
|
||||
/// being the point. It also bounds the compression thrown away when the
|
||||
/// deadline cuts a chunk short, to at most `STORE_CHUNK - 1` bodies.
|
||||
///
|
||||
/// Note the two buffers are additive: a root can hold `READY_TOPUP` extracted
|
||||
/// rows waiting for the writer *and* `content::READY_CAP` more in its
|
||||
/// channel, so in-flight text per root is bounded by their sum, not by either
|
||||
/// alone.
|
||||
const STORE_CHUNK: usize = 32;
|
||||
|
||||
/// What one [`store_extracted`] call did with the rows it was handed.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub struct Stored {
|
||||
/// Rows the caller must now drop from its buffer, written or not.
|
||||
pub consumed: usize,
|
||||
/// Rows whose write succeeded — whose `content_state` moved.
|
||||
pub written: usize,
|
||||
}
|
||||
|
||||
/// Write already-extracted rows — the cheap half of the content pass, and all
|
||||
/// that runs with the connection held — until `deadline`.
|
||||
///
|
||||
/// This is where a document's FTS5 trigram tokenization happens, up to
|
||||
/// `maximum_text_size` of it per row, and it is the writer thread's dominant
|
||||
/// cost. The deadline is checked after every row, so a turn on the writer
|
||||
/// overruns it by at most one document; the rows not reached are left for the
|
||||
/// caller to hand back next turn. At least one row is always consumed unless
|
||||
/// the run is already stopped, so a caller looping on this cannot spin.
|
||||
///
|
||||
/// A row whose write fails is logged and consumed rather than failing the
|
||||
/// run: its `content_state` stays pending, so the next run retries it.
|
||||
/// Returns how many rows were written. A row whose write fails is logged and
|
||||
/// skipped rather than failing the run: its `content_state` stays pending, so
|
||||
/// the next run retries it.
|
||||
pub fn store_extracted(
|
||||
conn_mutex: &Arc<Mutex<Connection>>,
|
||||
rows: &[crate::content::ExtractedRow],
|
||||
stop_flag: &Arc<AtomicBool>,
|
||||
config: &Config,
|
||||
deadline: std::time::Instant,
|
||||
) -> Result<Stored, String> {
|
||||
let mut done = Stored::default();
|
||||
for chunk in rows.chunks(STORE_CHUNK) {
|
||||
) -> Result<usize, String> {
|
||||
if rows.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let mut written = 0usize;
|
||||
for batch in rows.chunks(config.processing.batch_size.max(1)) {
|
||||
if stop_flag.load(Ordering::Relaxed) {
|
||||
break;
|
||||
return Ok(written);
|
||||
}
|
||||
// Outside the lock — see `compress_bodies`.
|
||||
let bodies = compress_bodies(
|
||||
chunk
|
||||
batch
|
||||
.iter()
|
||||
.map(|r| crate::file_handling::outcome_body(&r.outcome)),
|
||||
config,
|
||||
|
|
@ -390,30 +369,16 @@ pub fn store_extracted(
|
|||
let tx = conn
|
||||
.unchecked_transaction()
|
||||
.map_err(|e| format!("Failed to begin transaction: {}", e))?;
|
||||
let mut cut = false;
|
||||
for (i, row) in chunk.iter().enumerate() {
|
||||
// Counted before anything can skip it: consumed is what the
|
||||
// caller drains, and a row that failed still has to leave.
|
||||
done.consumed += 1;
|
||||
match &bodies[i] {
|
||||
Err(e) => crate::log_warn!("compress text for {}: {}", row.name, e),
|
||||
Ok(zstd) => {
|
||||
match store_content_outcome(&tx, row.file_id, &row.outcome, zstd.as_deref()) {
|
||||
Ok(()) => done.written += 1,
|
||||
Err(e) => crate::log_warn!("content indexing for {}: {}", row.name, e),
|
||||
}
|
||||
}
|
||||
}
|
||||
if stop_flag.load(Ordering::Relaxed) || std::time::Instant::now() >= deadline {
|
||||
cut = true;
|
||||
break;
|
||||
for (i, row) in batch.iter().enumerate() {
|
||||
let zstd = body_or_skip!(bodies, i, row.name);
|
||||
if let Err(e) = store_content_outcome(&tx, row.file_id, &row.name, &row.outcome, zstd) {
|
||||
crate::log_warn!("content indexing for {}: {}", row.name, e);
|
||||
continue;
|
||||
}
|
||||
written += 1;
|
||||
}
|
||||
tx.commit()
|
||||
.map_err(|e| format!("Failed to commit transaction: {}", e))?;
|
||||
if cut {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(done)
|
||||
Ok(written)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,17 +8,6 @@ use super::*;
|
|||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
/// A path that does not exist yet — these tests build the tree themselves.
|
||||
/// The removed `extract_scope_prepare`: the oversize sweep, then the count.
|
||||
fn extract_scope_prepare(
|
||||
conn_mutex: &std::sync::Arc<std::sync::Mutex<rusqlite::Connection>>,
|
||||
cursor: &ExtractCursor,
|
||||
config: &Config,
|
||||
) -> Result<ExtractScope, String> {
|
||||
let conn = crate::lock_ok(conn_mutex);
|
||||
super::mark_oversize_pending_na(&conn, cursor, config)?;
|
||||
super::count_extract_scope(&conn, cursor, config)
|
||||
}
|
||||
|
||||
fn tmp(tag: &str) -> std::path::PathBuf {
|
||||
crate::testutil::scratch_dir(tag).join("tree")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -104,20 +104,12 @@ fn count_find_pipe_wc(
|
|||
|
||||
/// Count tree entries with a plain directory walk — the oracle
|
||||
/// [`count_tree_entries_win32`] is tested against.
|
||||
///
|
||||
/// Gated on `test` as well as `windows` because its only caller is a
|
||||
/// Windows-only test: under plain `cargo build --target …-windows-gnu` it
|
||||
/// would otherwise compile as dead code. The `walkdir` import is local for the
|
||||
/// same reason the rest of this module's platform imports are — a file-level
|
||||
/// one is invisible to the host that cannot compile the branch using it, which
|
||||
/// is exactly how it went missing when this module was split out.
|
||||
#[cfg(all(windows, test))]
|
||||
#[cfg(windows)]
|
||||
fn count_tree_entries_walkdir(
|
||||
path: &str,
|
||||
cancel: &std::sync::atomic::AtomicBool,
|
||||
) -> Result<usize, String> {
|
||||
use std::sync::atomic::Ordering;
|
||||
use walkdir::WalkDir;
|
||||
|
||||
let mut n = 0usize;
|
||||
// Unreadable subtrees are skipped rather than fatal: this is a progress
|
||||
|
|
|
|||
|
|
@ -11,11 +11,10 @@ mod count_and_extract_tests;
|
|||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
pub(crate) use batch::max_text_file_size;
|
||||
pub(crate) use batch::store_inline_text;
|
||||
pub use batch::{
|
||||
cleanup_stale_index_entries, count_extract_scope, mark_oversize_pending_na,
|
||||
process_batch_inserts, process_batch_updates, store_extracted, ExtractCursor, ExtractScope,
|
||||
Stored,
|
||||
cleanup_stale_index_entries, extract_scope_prepare, process_batch_inserts,
|
||||
process_batch_updates, store_extracted, ExtractCursor, ExtractScope,
|
||||
};
|
||||
pub use counting::count_tree_entries_fast;
|
||||
pub use paths::{db_key_for_missing_path, filtered_dirs, filtered_walk, UnreadableDirs};
|
||||
|
|
|
|||
|
|
@ -8,6 +8,20 @@ use walkdir::{DirEntry, WalkDir};
|
|||
|
||||
use crate::config::IgnoreSet;
|
||||
|
||||
/// Derive (inode, device_id) from a `std::fs::Metadata` on platforms that
|
||||
/// expose them. Returns `(None, None)` on Windows and other non-Unix targets.
|
||||
pub(super) fn inode_and_device(_meta: &std::fs::Metadata) -> (Option<u64>, Option<u64>) {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
(Some(_meta.ino()), Some(_meta.dev()))
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
(None, None)
|
||||
}
|
||||
}
|
||||
|
||||
/// Render a path as the string stored in `files.path`.
|
||||
///
|
||||
/// `Path::canonicalize` on Windows hands back extended-length paths; the
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ use std::time::UNIX_EPOCH;
|
|||
use rusqlite::Connection;
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use super::paths::parent_str;
|
||||
use super::paths::{inode_and_device, parent_str};
|
||||
use super::*;
|
||||
use crate::config::Config;
|
||||
use crate::db::repo::{self, NewFile};
|
||||
|
|
@ -110,6 +110,8 @@ pub struct OwnedNewFile {
|
|||
pub parent: String,
|
||||
pub size: u64,
|
||||
pub mtime: u64,
|
||||
pub inode: Option<u64>,
|
||||
pub device_id: Option<u64>,
|
||||
pub mime: Option<String>,
|
||||
pub ftype: FileType,
|
||||
/// `None` only for a dehydrated cloud placeholder. Stored as SQL NULL,
|
||||
|
|
@ -134,6 +136,8 @@ impl OwnedNewFile {
|
|||
parent: &self.parent,
|
||||
size: self.size,
|
||||
mtime: self.mtime,
|
||||
inode: self.inode,
|
||||
device_id: self.device_id,
|
||||
mime: self.mime.as_deref(),
|
||||
ftype: self.ftype,
|
||||
hash: self.hash.as_deref(),
|
||||
|
|
@ -214,6 +218,7 @@ pub fn prepare_file_record(
|
|||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())?;
|
||||
let parent = parent_str(path);
|
||||
let (inode, device_id) = inode_and_device(meta);
|
||||
// Sniff from the bytes hashing already read; an empty head falls back to
|
||||
// the extension.
|
||||
let mime = guess_mime_from_head(Path::new(path), &head);
|
||||
|
|
@ -253,6 +258,8 @@ pub fn prepare_file_record(
|
|||
parent,
|
||||
size,
|
||||
mtime,
|
||||
inode,
|
||||
device_id,
|
||||
mime,
|
||||
ftype,
|
||||
hash,
|
||||
|
|
@ -277,9 +284,10 @@ pub fn prepare_file_record_from_path(
|
|||
prepare_file_record(&db_path, &meta, config, registry)
|
||||
}
|
||||
|
||||
/// Extract content for one file and record the outcome on its row: text on
|
||||
/// success, `NA` when no extractor applies or the `content_extensions`
|
||||
/// filter excludes it, `FAILED` with a reason on extractor errors. The single implementation behind the full text-index
|
||||
/// Extract content for one file and record the outcome on its row: text +
|
||||
/// properties on success, `NA` when no extractor applies or the
|
||||
/// `content_extensions` filter excludes it, `FAILED` with a reason on
|
||||
/// extractor errors. The single implementation behind the full text-index
|
||||
/// pass and incremental updates.
|
||||
///
|
||||
/// `mime` is authoritative, including when it is `None`: the head was already
|
||||
|
|
@ -287,6 +295,7 @@ pub fn prepare_file_record_from_path(
|
|||
pub fn extract_and_store(
|
||||
tx: &rusqlite::Transaction<'_>,
|
||||
file_id: i64,
|
||||
name: &str,
|
||||
path: &str,
|
||||
mime: Option<&str>,
|
||||
registry: &Registry,
|
||||
|
|
@ -297,15 +306,18 @@ pub fn extract_and_store(
|
|||
Some(text) => repo::encode_one(text, config.processing.store_text_for_snippets)?,
|
||||
None => None,
|
||||
};
|
||||
store_content_outcome(tx, file_id, &outcome, zstd.as_deref())
|
||||
store_content_outcome(tx, file_id, name, &outcome, zstd.as_deref())
|
||||
}
|
||||
|
||||
/// What should be written for one file's content, decided without touching
|
||||
/// the database.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ContentOutcome {
|
||||
/// Text, already truncated to `maximum_text_size`.
|
||||
Done { text: String },
|
||||
/// Text (already truncated to `maximum_text_size`) and sorted properties.
|
||||
Done {
|
||||
text: String,
|
||||
properties: Vec<(String, String)>,
|
||||
},
|
||||
/// No extractor claims the MIME, or `content_extensions` excludes it.
|
||||
NotApplicable,
|
||||
/// The extractor ran and failed; the reason goes on the row.
|
||||
|
|
@ -355,7 +367,10 @@ pub fn decide_content(
|
|||
content.text =
|
||||
safe_truncate_string(&content.text, config.processing.maximum_text_size);
|
||||
}
|
||||
ContentOutcome::Done { text: content.text }
|
||||
ContentOutcome::Done {
|
||||
properties: content.properties_sorted(),
|
||||
text: content.text,
|
||||
}
|
||||
}
|
||||
Ok(None) => ContentOutcome::NotApplicable,
|
||||
Err(reason) => ContentOutcome::Failed(reason),
|
||||
|
|
@ -370,11 +385,14 @@ pub fn decide_content(
|
|||
pub fn store_content_outcome(
|
||||
tx: &rusqlite::Transaction<'_>,
|
||||
file_id: i64,
|
||||
name: &str,
|
||||
outcome: &ContentOutcome,
|
||||
text_zstd: Option<&[u8]>,
|
||||
) -> Result<(), String> {
|
||||
match outcome {
|
||||
ContentOutcome::Done { text } => repo::set_content_done(tx, file_id, text, text_zstd),
|
||||
ContentOutcome::Done { text, properties } => {
|
||||
repo::set_content_done(tx, file_id, name, text, properties, text_zstd)
|
||||
}
|
||||
ContentOutcome::NotApplicable => repo::set_content_na(tx, file_id),
|
||||
ContentOutcome::Failed(reason) => repo::set_content_failed(tx, file_id, reason),
|
||||
}
|
||||
|
|
@ -384,7 +402,7 @@ pub fn store_content_outcome(
|
|||
/// to its [`repo::DocEncoder`] ahead of the lock.
|
||||
pub fn outcome_body(outcome: &ContentOutcome) -> Option<&str> {
|
||||
match outcome {
|
||||
ContentOutcome::Done { text } => Some(text),
|
||||
ContentOutcome::Done { text, .. } => Some(text),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ use crate::db::repo;
|
|||
use crate::extract::Registry;
|
||||
use crate::file_handling::{
|
||||
db_key_for_missing_path, extract_and_store, filtered_walk, prepare_file_record_from_path,
|
||||
ExtractCursor, UnreadableDirs,
|
||||
store_inline_text, ExtractCursor, UnreadableDirs,
|
||||
};
|
||||
use crate::platform::path_has_hidden_component_under;
|
||||
use crate::watcher::FsEvent;
|
||||
|
|
@ -138,11 +138,12 @@ fn upsert_file(
|
|||
} else if let Some(text) = rec.inline_text.as_deref() {
|
||||
// `prepare_file_record_from_path` already read the whole file.
|
||||
let zstd = repo::encode_one(text, config.processing.store_text_for_snippets)?;
|
||||
repo::set_content_done(&tx, file_id, text, zstd.as_deref())?;
|
||||
store_inline_text(&tx, file_id, &rec, text, zstd.as_deref())?;
|
||||
} else {
|
||||
extract_and_store(
|
||||
&tx,
|
||||
file_id,
|
||||
&rec.name,
|
||||
&rec.path,
|
||||
rec.mime.as_deref(),
|
||||
registry,
|
||||
|
|
@ -158,6 +159,25 @@ fn remove_path(conn: &mut Connection, path: &Path) -> Result<(), String> {
|
|||
remove_paths(conn, std::slice::from_ref(&path.to_path_buf()), 1)
|
||||
}
|
||||
|
||||
/// Drop removals that a removal of one of their ancestors already covers.
|
||||
///
|
||||
/// `rm -rf dir/` reports `dir` *and* every file beneath it; removing `dir`
|
||||
/// sweeps its whole path range, so each descendant event is duplicate work.
|
||||
/// For callers holding a raw removal set — the coordinator collapses on
|
||||
/// arrival instead (`collapse_pending_removals`). Containment is
|
||||
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
|
||||
pub fn collapse_removal_roots(paths: Vec<std::path::PathBuf>) -> Vec<std::path::PathBuf> {
|
||||
if paths.len() < 2 {
|
||||
return paths;
|
||||
}
|
||||
let all: std::collections::HashSet<&Path> = paths.iter().map(|p| p.as_path()).collect();
|
||||
paths
|
||||
.iter()
|
||||
.filter(|p| !p.ancestors().skip(1).any(|a| all.contains(a)))
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Delete `paths` and everything indexed beneath them, in transactions of at
|
||||
/// most `chunk` paths.
|
||||
///
|
||||
|
|
@ -294,6 +314,47 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
fn collapse(paths: &[&str]) -> Vec<String> {
|
||||
let mut out: Vec<String> =
|
||||
collapse_removal_roots(paths.iter().map(std::path::PathBuf::from).collect())
|
||||
.iter()
|
||||
.map(|p| p.to_string_lossy().into_owned())
|
||||
.collect();
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn removal_roots_collapse_to_the_shallowest_ancestor() {
|
||||
assert_eq!(
|
||||
collapse(&["/dir", "/dir/a.txt", "/dir/b/c.txt", "/dir/b"]),
|
||||
vec!["/dir"]
|
||||
);
|
||||
|
||||
// Component-wise, so a name-prefix sibling is not swallowed.
|
||||
assert_eq!(
|
||||
collapse(&["/a/b", "/a/bc"]),
|
||||
vec!["/a/b", "/a/bc"],
|
||||
"/a/bc does not live under /a/b"
|
||||
);
|
||||
assert_eq!(
|
||||
collapse(&["/a/b", "/a/b.txt"]),
|
||||
vec!["/a/b", "/a/b.txt"],
|
||||
"a sibling file sorting between a dir and its children survives"
|
||||
);
|
||||
|
||||
// Unrelated removals all survive; order of input does not matter.
|
||||
assert_eq!(
|
||||
collapse(&["/x/deep/f", "/y", "/x"]),
|
||||
vec!["/x", "/y"],
|
||||
"/x/deep/f is covered by /x, /y is independent"
|
||||
);
|
||||
|
||||
// Degenerate inputs.
|
||||
assert!(collapse(&[]).is_empty());
|
||||
assert_eq!(collapse(&["/only"]), vec!["/only"]);
|
||||
}
|
||||
|
||||
/// The collapse must not change what ends up deleted — only how much work
|
||||
/// it takes to get there.
|
||||
#[test]
|
||||
|
|
@ -317,10 +378,9 @@ mod tests {
|
|||
format!("{}/a.txt", canonical_tree).into(),
|
||||
format!("{}/deep/b.txt", canonical_tree).into(),
|
||||
];
|
||||
// Collapsed to its root the way the coordinator collapses an
|
||||
// arriving queue (`collapse_pending_removals`): one range covers the
|
||||
// whole tree, which is what makes `remove_paths` cheap.
|
||||
let roots = vec![reported[0].clone()];
|
||||
let roots = collapse_removal_roots(reported);
|
||||
assert_eq!(roots.len(), 1, "one range covers the whole tree");
|
||||
|
||||
remove_paths(&mut f.conn, &roots, 200).unwrap();
|
||||
assert_eq!(f.counts(), (1, 1, 1), "only tree2 survives");
|
||||
let survivor = f.canonical(&f.dir.join("tree2").join("keep.txt"));
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ use std::collections::HashMap;
|
|||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{mpsc, Arc, Mutex};
|
||||
use std::thread;
|
||||
use std::time::Instant;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::db;
|
||||
|
|
@ -84,6 +84,7 @@ pub struct IndexingService {
|
|||
status: Arc<Mutex<IndexingStatus>>,
|
||||
command_tx: mpsc::Sender<IndexingCommand>,
|
||||
db_connection: Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
|
||||
suspend_flag: Arc<AtomicBool>,
|
||||
/// The long single statement a run is inside, if any: the prologue's
|
||||
/// reconcile scan or the epilogue's VACUUM — never both, so one slot
|
||||
/// serves and [`IndexingService::cancel_db_work`] reaches either.
|
||||
|
|
@ -91,6 +92,9 @@ pub struct IndexingService {
|
|||
_handle: thread::JoinHandle<()>,
|
||||
}
|
||||
|
||||
/// Polling interval for `should_abort` while suspended.
|
||||
const SUSPEND_POLL_MS: u64 = 100;
|
||||
|
||||
/// `indexing.root_workers` rekeyed from the spellings the user typed to the
|
||||
/// canonical roots the indexer walks, so an override survives a `~`, a
|
||||
/// trailing slash, a relative path or a symlinked root. Entries naming a
|
||||
|
|
@ -108,21 +112,39 @@ fn resolved_root_workers(config: &Config) -> HashMap<String, usize> {
|
|||
.collect()
|
||||
}
|
||||
|
||||
/// Combined stop/suspend check used by worker loops; `true` iff the caller
|
||||
/// should abort. While suspended (and not stopped) it parks the thread in
|
||||
/// short sleeps until `resume()`.
|
||||
pub(crate) fn should_abort(stop: &Arc<AtomicBool>, suspend: &Arc<AtomicBool>) -> bool {
|
||||
loop {
|
||||
if stop.load(Ordering::Relaxed) {
|
||||
return true;
|
||||
}
|
||||
if !suspend.load(Ordering::Relaxed) {
|
||||
return false;
|
||||
}
|
||||
thread::sleep(Duration::from_millis(SUSPEND_POLL_MS));
|
||||
}
|
||||
}
|
||||
|
||||
impl IndexingService {
|
||||
pub fn new() -> Self {
|
||||
let status = Arc::new(Mutex::new(IndexingStatus::Idle));
|
||||
let (command_tx, command_rx) = mpsc::channel();
|
||||
let db_connection = Arc::new(Mutex::new(None));
|
||||
let suspend_flag = Arc::new(AtomicBool::new(false));
|
||||
let interrupt: Arc<db::InterruptSlot> = Arc::new(db::InterruptSlot::default());
|
||||
|
||||
let status_clone = status.clone();
|
||||
let db_connection_clone = db_connection.clone();
|
||||
let suspend_clone = suspend_flag.clone();
|
||||
let interrupt_clone = interrupt.clone();
|
||||
let handle = thread::spawn(move || {
|
||||
Self::indexing_thread(
|
||||
status_clone,
|
||||
command_rx,
|
||||
db_connection_clone,
|
||||
suspend_clone,
|
||||
interrupt_clone,
|
||||
);
|
||||
});
|
||||
|
|
@ -131,6 +153,7 @@ impl IndexingService {
|
|||
status,
|
||||
command_tx,
|
||||
db_connection,
|
||||
suspend_flag,
|
||||
interrupt,
|
||||
_handle: handle,
|
||||
}
|
||||
|
|
@ -148,6 +171,22 @@ impl IndexingService {
|
|||
db::interrupt(&self.interrupt)
|
||||
}
|
||||
|
||||
/// Pause the indexer: worker loops calling [`should_abort`] block until
|
||||
/// [`resume`](Self::resume). Does not stop the worker.
|
||||
pub fn suspend(&self) {
|
||||
self.suspend_flag.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Resume indexing after [`suspend`](Self::suspend). No-op if not
|
||||
/// suspended.
|
||||
pub fn resume(&self) {
|
||||
self.suspend_flag.store(false, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn is_suspended(&self) -> bool {
|
||||
self.suspend_flag.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Start indexing one or more roots; all walk concurrently, funnelling
|
||||
/// into one writer thread. Duplicate roots collapse to one walk, and a
|
||||
/// file reachable from more than one is written once. Returns `Err` if a
|
||||
|
|
@ -262,6 +301,11 @@ impl IndexingService {
|
|||
}
|
||||
}
|
||||
|
||||
/// Force graceful shutdown - used for signal handling
|
||||
pub fn graceful_shutdown(&self) -> Result<(), String> {
|
||||
self.stop_indexing()
|
||||
}
|
||||
|
||||
/// Check if configuration changes require index recreation. A pure
|
||||
/// *read* check that never wipes; a missing or incompatible DB means
|
||||
/// there is nothing to validate.
|
||||
|
|
@ -323,6 +367,7 @@ impl IndexingService {
|
|||
status: Arc<Mutex<IndexingStatus>>,
|
||||
command_rx: mpsc::Receiver<IndexingCommand>,
|
||||
db_connection: Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
|
||||
suspend_flag: Arc<AtomicBool>,
|
||||
interrupt: Arc<db::InterruptSlot>,
|
||||
) {
|
||||
let stop_flag = Arc::new(AtomicBool::new(false));
|
||||
|
|
@ -351,6 +396,7 @@ impl IndexingService {
|
|||
let config_owned = config.clone();
|
||||
|
||||
let db_connection_clone = db_connection.clone();
|
||||
let suspend_clone = suspend_flag.clone();
|
||||
let interrupt_clone = interrupt.clone();
|
||||
indexing_handle = Some(thread::spawn(move || {
|
||||
// The writer thread: every DB write and every text
|
||||
|
|
@ -361,6 +407,7 @@ impl IndexingService {
|
|||
&paths_owned,
|
||||
&db_path_owned,
|
||||
&stop_flag_clone,
|
||||
&suspend_clone,
|
||||
&config_owned,
|
||||
&db_connection_clone,
|
||||
&interrupt_clone,
|
||||
|
|
|
|||
|
|
@ -13,9 +13,9 @@ use crate::db;
|
|||
use crate::db::repo;
|
||||
use crate::extract::Registry;
|
||||
use crate::file_handling::{
|
||||
cleanup_stale_index_entries, count_tree_entries_fast, fts_finalize_after_text_indexing,
|
||||
mark_oversize_pending_na, normalize_root_string, process_batch_inserts, process_batch_updates,
|
||||
store_extracted, ExtractCursor, ExtractScope, FileIndexAction, OwnedNewFile,
|
||||
cleanup_stale_index_entries, count_tree_entries_fast, extract_scope_prepare,
|
||||
fts_finalize_after_text_indexing, normalize_root_string, process_batch_inserts,
|
||||
process_batch_updates, store_extracted, ExtractCursor, FileIndexAction, OwnedNewFile,
|
||||
};
|
||||
use crate::walk::{thread_count_for, walk_indexable_files, ParallelWalk, TryNext, WalkEvent};
|
||||
|
||||
|
|
@ -84,11 +84,6 @@ impl Drop for CancelOnDrop {
|
|||
}
|
||||
}
|
||||
|
||||
/// Most extracted rows a root holds back between turns. Not `quantum`: a row
|
||||
/// carries up to `maximum_text_size` of text, and 500 of those would be
|
||||
/// 128 MiB per root. At 64 it is 16 MiB.
|
||||
const READY_TOPUP: usize = 64;
|
||||
|
||||
/// One root's in-flight indexing state, owned by the writer loop.
|
||||
pub(super) struct RootPipeline {
|
||||
pub(super) root: String,
|
||||
|
|
@ -105,14 +100,8 @@ pub(super) struct RootPipeline {
|
|||
pub(super) phase: RootPhase,
|
||||
/// The running content pass, once this root's walk has finished.
|
||||
pub(super) content: Option<crate::content::ContentPass>,
|
||||
/// Extracted rows pulled off the pass and not yet written. A turn writes
|
||||
/// for its slice, not for its batch, so it may leave some behind.
|
||||
pub(super) ready: Vec<crate::content::ExtractedRow>,
|
||||
/// Rows this run's content pass has written for this root.
|
||||
pub(super) written: usize,
|
||||
/// The pass's range counts, cached once known so a `Done` root still has
|
||||
/// them after its pass is gone.
|
||||
pub(super) totals: Option<ExtractScope>,
|
||||
pub(super) extract_total: usize,
|
||||
pub(super) extracted: usize,
|
||||
pub(super) current_file: Option<String>,
|
||||
/// When this root's current phase began, for the one line each phase logs
|
||||
/// when it ends.
|
||||
|
|
@ -184,16 +173,8 @@ impl RootPipeline {
|
|||
stats.map_or((0, 0), |s| (s.active(), s.total()))
|
||||
}
|
||||
|
||||
/// The pass's counts, from the cache or — until the cache is filled — from
|
||||
/// the pass itself.
|
||||
fn extract_totals(&self) -> Option<ExtractScope> {
|
||||
self.totals
|
||||
.or_else(|| self.content.as_ref().and_then(|p| p.totals()))
|
||||
}
|
||||
|
||||
pub(super) fn snapshot(&self) -> RootProgress {
|
||||
fn snapshot(&self) -> RootProgress {
|
||||
let (active_workers, total_workers) = self.worker_counts();
|
||||
let totals = self.extract_totals();
|
||||
RootProgress {
|
||||
root: self.root.clone(),
|
||||
phase: self.phase,
|
||||
|
|
@ -202,58 +183,30 @@ impl RootPipeline {
|
|||
0 => None,
|
||||
n => Some(n),
|
||||
},
|
||||
// Earlier runs' rows count once the pass has counted them; until
|
||||
// then only this run's, so the figure never goes backwards.
|
||||
extracted: totals.map_or(self.written, |t| t.already_done + self.written),
|
||||
extract_total: totals.map(|t| t.pending + t.already_done),
|
||||
extracted: self.extracted,
|
||||
extract_total: self.extract_total,
|
||||
current_file: self.current_file.clone(),
|
||||
active_workers,
|
||||
total_workers,
|
||||
}
|
||||
}
|
||||
|
||||
/// Drain walk events into the pending batches for up to one slice,
|
||||
/// Drain up to one quantum of walk events into the pending batches,
|
||||
/// finishing the walk if it ends. Returns whether anything happened.
|
||||
///
|
||||
/// Batches still land per quantum; the slice only decides how many of
|
||||
/// them one turn may write. A walk slower than the writer ends its turn at
|
||||
/// `Empty` well inside the slice; only a walk that has the writer
|
||||
/// saturated uses all of it.
|
||||
pub(super) fn service_walking(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
|
||||
let deadline = Instant::now() + cx.slice;
|
||||
fn service_walking(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
|
||||
let mut took = 0usize;
|
||||
let mut finished = false;
|
||||
while !finished {
|
||||
let quantum_end = took + cx.quantum;
|
||||
let more = self.walk_quantum(cx, &mut took, quantum_end, &mut finished)?;
|
||||
if !more || Instant::now() >= deadline {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(finished || took > 0)
|
||||
}
|
||||
|
||||
/// One quantum of [`RootPipeline::service_walking`]. Returns whether the
|
||||
/// channel still had events when the quantum ended — false on `Empty` or
|
||||
/// on the walk finishing.
|
||||
fn walk_quantum(
|
||||
&mut self,
|
||||
cx: &mut RunCx<'_>,
|
||||
took: &mut usize,
|
||||
quantum_end: usize,
|
||||
finished: &mut bool,
|
||||
) -> Result<bool, String> {
|
||||
while *took < quantum_end {
|
||||
while took < cx.quantum {
|
||||
match self.walk.try_next() {
|
||||
TryNext::Item(WalkEvent::Stale(paths)) => {
|
||||
*took += 1;
|
||||
took += 1;
|
||||
// Applied at the end of the run: deleting mid-walk would
|
||||
// break "a stopped run deletes nothing", and an aliased
|
||||
// sighting that exempts a path may still be ahead.
|
||||
cx.stale_candidates.extend(paths);
|
||||
}
|
||||
TryNext::Item(WalkEvent::File(file)) => {
|
||||
*took += 1;
|
||||
took += 1;
|
||||
self.walked += 1;
|
||||
if self.walked.is_multiple_of(64) {
|
||||
self.current_file = Some(file.path.clone());
|
||||
|
|
@ -294,15 +247,15 @@ impl RootPipeline {
|
|||
}
|
||||
}
|
||||
}
|
||||
TryNext::Empty => return Ok(false),
|
||||
TryNext::Empty => break,
|
||||
TryNext::Finished => {
|
||||
self.finish_walk(cx)?;
|
||||
*finished = true;
|
||||
return Ok(false);
|
||||
finished = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(true)
|
||||
Ok(finished || took > 0)
|
||||
}
|
||||
|
||||
/// The walk ended: land the buffered batches, then either hand the root
|
||||
|
|
@ -361,60 +314,42 @@ impl RootPipeline {
|
|||
}
|
||||
}
|
||||
let cursor = ExtractCursor::for_root(&self.root);
|
||||
// Only the sweep runs on the writer. Counting the range is the
|
||||
// pass's own job, on its own connection: on a large root it is
|
||||
// seconds, and here that was seconds of every other root's walk
|
||||
// standing still.
|
||||
{
|
||||
let conn = crate::lock_ok(&cx.conn_mutex);
|
||||
mark_oversize_pending_na(&conn, &cursor, cx.config)?;
|
||||
}
|
||||
self.totals = None;
|
||||
self.written = 0;
|
||||
self.ready.clear();
|
||||
// Starts only now: the rows have to exist before the feeder can
|
||||
// page over them. Started even when nothing may be pending — the
|
||||
// count that would say so is the pass's — and an empty range
|
||||
// finishes on its own next turn.
|
||||
let scope = extract_scope_prepare(&cx.conn_mutex, &cursor, cx.config)?;
|
||||
// Progress counts the root's whole searchable set: files extracted
|
||||
// in earlier runs start the counter, so an unchanged root shows
|
||||
// "X of X" rather than "0 of 0".
|
||||
self.extract_total = scope.pending + scope.already_done;
|
||||
self.extracted = scope.already_done;
|
||||
if scope.pending == 0 {
|
||||
self.phase = RootPhase::Done;
|
||||
} else {
|
||||
// Starts only now: the rows have to exist before the feeder
|
||||
// can page over them.
|
||||
self.content = Some(crate::content::extract_content(
|
||||
cx.db_path,
|
||||
&cursor,
|
||||
cx.registry.clone(),
|
||||
cx.config.clone(),
|
||||
cx.stop_flag.clone(),
|
||||
cx.suspend_flag.clone(),
|
||||
self.workers,
|
||||
));
|
||||
self.phase = RootPhase::Extracting;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Write finished extraction work for up to one slice; extraction itself
|
||||
/// runs on this root's own pool. Returns whether anything happened.
|
||||
///
|
||||
/// Rows the slice does not reach stay in `ready` for the next turn, and
|
||||
/// the pass is not declared done until they have all landed.
|
||||
pub(super) fn service_extracting(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
|
||||
let deadline = Instant::now() + cx.slice;
|
||||
/// Drain up to a quantum of finished extraction work, then write it;
|
||||
/// extraction runs on this root's own pool. Returns whether anything
|
||||
/// happened.
|
||||
fn service_extracting(&mut self, cx: &mut RunCx<'_>) -> Result<bool, String> {
|
||||
let pass = self.content.as_mut().expect("extracting root has a pass");
|
||||
let mut batch: Vec<crate::content::ExtractedRow> = Vec::new();
|
||||
let mut finished = false;
|
||||
let mut consumed = 0usize;
|
||||
// Disjoint borrows: the pass is held across the store.
|
||||
let Self {
|
||||
content,
|
||||
ready,
|
||||
written,
|
||||
totals,
|
||||
current_file,
|
||||
..
|
||||
} = self;
|
||||
let pass = content.as_mut().expect("extracting root has a pass");
|
||||
if totals.is_none() {
|
||||
*totals = pass.totals();
|
||||
}
|
||||
loop {
|
||||
while ready.len() < READY_TOPUP {
|
||||
while batch.len() < cx.quantum {
|
||||
match pass.try_next() {
|
||||
TryNext::Item(row) => ready.push(row),
|
||||
TryNext::Item(row) => batch.push(row),
|
||||
TryNext::Empty => break,
|
||||
TryNext::Finished => {
|
||||
finished = true;
|
||||
|
|
@ -422,28 +357,12 @@ impl RootPipeline {
|
|||
}
|
||||
}
|
||||
}
|
||||
if ready.is_empty() {
|
||||
break;
|
||||
}
|
||||
let stored = store_extracted(&cx.conn_mutex, ready, cx.stop_flag, cx.config, deadline)?;
|
||||
if stored.consumed > 0 {
|
||||
// The last row *written*, not the last fetched: with leftovers
|
||||
// the two can be a slice apart.
|
||||
*current_file = Some(ready[stored.consumed - 1].name.clone());
|
||||
}
|
||||
ready.drain(..stored.consumed);
|
||||
*written += stored.written;
|
||||
consumed += stored.consumed;
|
||||
// Stopped, out of time, or still holding rows the deadline cut
|
||||
// short — the next turn takes it from here.
|
||||
if stored.consumed == 0 || !ready.is_empty() || Instant::now() >= deadline {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if finished && ready.is_empty() {
|
||||
if totals.is_none() {
|
||||
*totals = pass.totals();
|
||||
if let Some(row) = batch.last() {
|
||||
self.current_file = Some(row.name.clone());
|
||||
}
|
||||
let took = batch.len();
|
||||
self.extracted += store_extracted(&cx.conn_mutex, &batch, cx.stop_flag, cx.config)?;
|
||||
if finished {
|
||||
// Join before deciding; see `ParallelWalk::finish`.
|
||||
if !pass.finish() {
|
||||
crate::log_warn!("a content worker for {} terminated abnormally", self.root);
|
||||
|
|
@ -451,69 +370,39 @@ impl RootPipeline {
|
|||
self.content = None;
|
||||
self.phase = RootPhase::Done;
|
||||
let extract_time = self.phase_elapsed();
|
||||
// Quiet for the pass that found nothing to do: every root passes
|
||||
// through here now, changed or not.
|
||||
if self.written > 0 {
|
||||
crate::log_info!(
|
||||
"{}: content done — {}",
|
||||
self.root,
|
||||
phase_summary(self.written, "files with text", extract_time)
|
||||
phase_summary(self.extracted, "files with text", extract_time)
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(finished || consumed > 0)
|
||||
Ok(finished || took > 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// One run's shared environment and cross-root state, threaded through the
|
||||
/// per-phase [`RootPipeline`] service methods.
|
||||
pub(super) struct RunCx<'a> {
|
||||
pub(super) conn_mutex: Arc<Mutex<Connection>>,
|
||||
pub(super) config: &'a Config,
|
||||
pub(super) db_path: &'a str,
|
||||
pub(super) stop_flag: &'a Arc<AtomicBool>,
|
||||
/// Shared with every root's walk workers, which use it to finish small
|
||||
/// text files without handing them to the content pass.
|
||||
pub(super) registry: Arc<Registry>,
|
||||
pub(super) quantum: usize,
|
||||
/// Writer time one root's turn may take before the round moves on; see
|
||||
/// [`crate::config::ProcessingConfig::writer_turn_slice_ms`], which is
|
||||
/// where the default and the reasoning live. Zero is one quantum a turn.
|
||||
pub(super) slice: Duration,
|
||||
/// 128-bit path digests, not paths: at millions of files, owning every
|
||||
/// path string again was the single largest allocation in a run. See
|
||||
/// `walk::path_digest`.
|
||||
pub(super) seen_paths: HashSet<u128>,
|
||||
/// Rows the per-directory reconciliation found no file behind, plus
|
||||
/// whatever the vanished-directory sweep adds once the walks end.
|
||||
pub(super) stale_candidates: Vec<String>,
|
||||
/// Paths reached by resolving a symlink, whose row lives under a parent
|
||||
/// that may be outside every root.
|
||||
pub(super) aliased_paths: HashSet<String>,
|
||||
pub(super) stale_cleanup_ok: bool,
|
||||
}
|
||||
|
||||
impl<'a> RunCx<'a> {
|
||||
pub(super) fn new(
|
||||
struct RunCx<'a> {
|
||||
conn_mutex: Arc<Mutex<Connection>>,
|
||||
config: &'a Config,
|
||||
db_path: &'a str,
|
||||
stop_flag: &'a Arc<AtomicBool>,
|
||||
) -> RunCx<'a> {
|
||||
RunCx {
|
||||
conn_mutex,
|
||||
config,
|
||||
db_path,
|
||||
stop_flag,
|
||||
registry: Arc::new(Registry::default_set()),
|
||||
quantum: config.processing.batch_size.max(1),
|
||||
slice: Duration::from_millis(config.processing.writer_turn_slice_ms),
|
||||
seen_paths: HashSet::new(),
|
||||
stale_candidates: Vec::new(),
|
||||
aliased_paths: HashSet::new(),
|
||||
stale_cleanup_ok: true,
|
||||
}
|
||||
}
|
||||
suspend_flag: &'a Arc<AtomicBool>,
|
||||
/// Shared with every root's walk workers, which use it to finish small
|
||||
/// text files without handing them to the content pass.
|
||||
registry: Arc<Registry>,
|
||||
quantum: usize,
|
||||
/// 128-bit path digests, not paths: at millions of files, owning every
|
||||
/// path string again was the single largest allocation in a run. See
|
||||
/// `walk::path_digest`.
|
||||
seen_paths: HashSet<u128>,
|
||||
/// Rows the per-directory reconciliation found no file behind, plus
|
||||
/// whatever the vanished-directory sweep adds once the walks end.
|
||||
stale_candidates: Vec<String>,
|
||||
/// Paths reached by resolving a symlink, whose row lives under a parent
|
||||
/// that may be outside every root.
|
||||
aliased_paths: HashSet<String>,
|
||||
stale_cleanup_ok: bool,
|
||||
}
|
||||
|
||||
/// Publish a status snapshot. Never clobbers Stopping — the command thread
|
||||
|
|
@ -558,6 +447,7 @@ fn build_pipeline(
|
|||
cx.config.clone(),
|
||||
cx.registry.clone(),
|
||||
cx.stop_flag.clone(),
|
||||
cx.suspend_flag.clone(),
|
||||
workers,
|
||||
);
|
||||
|
||||
|
|
@ -601,9 +491,8 @@ fn build_pipeline(
|
|||
phase: RootPhase::Walking,
|
||||
workers,
|
||||
content: None,
|
||||
ready: Vec::new(),
|
||||
written: 0,
|
||||
totals: None,
|
||||
extract_total: 0,
|
||||
extracted: 0,
|
||||
current_file: None,
|
||||
phase_started: Instant::now(),
|
||||
})
|
||||
|
|
@ -668,6 +557,7 @@ fn cleanup_stale(pipelines: &mut [RootPipeline], cx: &mut RunCx<'_>) -> Result<(
|
|||
&cx.conn_mutex,
|
||||
stale_paths.as_slice(),
|
||||
cx.stop_flag,
|
||||
cx.suspend_flag,
|
||||
cx.config,
|
||||
)?;
|
||||
crate::log_info!(
|
||||
|
|
@ -685,6 +575,7 @@ impl IndexingService {
|
|||
paths: &[String],
|
||||
db_path: &str,
|
||||
stop_flag: &Arc<AtomicBool>,
|
||||
suspend_flag: &Arc<AtomicBool>,
|
||||
config: &Config,
|
||||
db_connection: &Arc<Mutex<Option<Arc<Mutex<Connection>>>>>,
|
||||
interrupt: &db::InterruptSlot,
|
||||
|
|
@ -746,7 +637,19 @@ impl IndexingService {
|
|||
let count_cancel = Arc::new(AtomicBool::new(false));
|
||||
let _count_guard = CancelOnDrop(count_cancel.clone());
|
||||
|
||||
let mut cx = RunCx::new(conn_mutex, config, db_path, stop_flag);
|
||||
let mut cx = RunCx {
|
||||
conn_mutex,
|
||||
config,
|
||||
db_path,
|
||||
stop_flag,
|
||||
suspend_flag,
|
||||
registry: Arc::new(Registry::default_set()),
|
||||
quantum: config.processing.batch_size.max(1),
|
||||
seen_paths: HashSet::new(),
|
||||
stale_candidates: Vec::new(),
|
||||
aliased_paths: HashSet::new(),
|
||||
stale_cleanup_ok: true,
|
||||
};
|
||||
|
||||
// Read stored counts up front, under one lock, before the walks
|
||||
// compete for the connection.
|
||||
|
|
@ -786,19 +689,10 @@ impl IndexingService {
|
|||
};
|
||||
let mut checkpoint_at = wal_cap;
|
||||
|
||||
// Walks first, one slice each, then a single extraction slice.
|
||||
//
|
||||
// The walk is the disk-bound phase and the one whose stall shows: its
|
||||
// workers can only run as far ahead as their channel, so a writer that
|
||||
// does not come back to it soon enough parks a whole pool behind one
|
||||
// root's tokenizing. Serving every walking root before any extraction
|
||||
// caps a walk's wait at one slice per round; taking one extraction
|
||||
// slice per round, not one per root, keeps that cap independent of
|
||||
// how many roots are extracting — while still handing extraction a
|
||||
// slice every round, so it is never starved either. Any root's turn
|
||||
// ends early the moment it has nothing ready.
|
||||
// Round-robin with skipping: each round takes at most one quantum
|
||||
// from every root that has work ready.
|
||||
loop {
|
||||
if stop_flag.load(Ordering::Relaxed) {
|
||||
if should_abort(stop_flag, suspend_flag) {
|
||||
aborted = true;
|
||||
break;
|
||||
}
|
||||
|
|
@ -806,25 +700,11 @@ impl IndexingService {
|
|||
let n = pipelines.len();
|
||||
for k in 0..n {
|
||||
let p = &mut pipelines[(rr + k) % n];
|
||||
if p.phase == RootPhase::Walking {
|
||||
progressed |= p.service_walking(&mut cx)?;
|
||||
}
|
||||
}
|
||||
// Between the stages, not only at the end of the round: a root
|
||||
// enters `Extracting` in the walk stage above, and the stage
|
||||
// below can finish its pass in the same round. Published once a
|
||||
// round, the whole phase falls between two snapshots whenever a
|
||||
// root's content pass is short — a small root reads as
|
||||
// `Walking → Done`, having never reported the phase it spent its
|
||||
// extraction in.
|
||||
publish_status(status, run_start, &pipelines);
|
||||
|
||||
for k in 0..n {
|
||||
let p = &mut pipelines[(rr + k) % n];
|
||||
if p.phase == RootPhase::Extracting {
|
||||
progressed |= p.service_extracting(&mut cx)?;
|
||||
break;
|
||||
}
|
||||
progressed |= match p.phase {
|
||||
RootPhase::Walking => p.service_walking(&mut cx)?,
|
||||
RootPhase::Extracting => p.service_extracting(&mut cx)?,
|
||||
RootPhase::Done => false,
|
||||
};
|
||||
}
|
||||
rr = rr.wrapping_add(1);
|
||||
|
||||
|
|
@ -886,19 +766,15 @@ impl IndexingService {
|
|||
}
|
||||
|
||||
if aborted {
|
||||
// Nothing is landed on the way out, and there used to be a loop
|
||||
// here that looked as though it did: `aborted` implies the stop
|
||||
// flag is set, and both batch writers return on it before their
|
||||
// first chunk, so it wrote nothing. What a stop drops is each
|
||||
// root's part-filled insert/update batch (under `batch_size`
|
||||
// rows) and whatever extraction had ready — all of it still
|
||||
// `content_state = 0` or absent, so the next run finds it again.
|
||||
// That is what "a stopped run promises nothing" already means,
|
||||
// and it is cheaper than tokenizing a slice's worth of documents
|
||||
// while someone waits for the window to close.
|
||||
//
|
||||
// No stale cleanup either: a partial walk's seen set would delete
|
||||
// most of the index.
|
||||
// Buffered records are valid work — land them before leaving.
|
||||
for p in &mut pipelines {
|
||||
process_batch_updates(&cx.conn_mutex, &p.pending_updates, stop_flag, config)?;
|
||||
p.pending_updates.clear();
|
||||
process_batch_inserts(&cx.conn_mutex, &p.pending_inserts, stop_flag, config)?;
|
||||
p.pending_inserts.clear();
|
||||
}
|
||||
// No stale cleanup: a partial walk's seen set would delete most
|
||||
// of the index.
|
||||
report_run_warnings();
|
||||
crate::log_info!(
|
||||
"indexing stopped after {:.1}s",
|
||||
|
|
|
|||
|
|
@ -26,15 +26,12 @@ pub struct RootProgress {
|
|||
/// counts tree *entries* and so reads high. Read it through
|
||||
/// [`RootProgress::walk_denominator`].
|
||||
pub walk_total: Option<usize>,
|
||||
/// Rows with searchable text: this run's, plus earlier runs' once
|
||||
/// `extract_total` is known.
|
||||
/// Rows with searchable text: extracted in earlier runs plus this one.
|
||||
pub extracted: usize,
|
||||
/// The root's whole searchable set: pending + already-extracted rows when
|
||||
/// the walk finished — the count of files that have or will have text,
|
||||
/// not of files under the root. `None` until the root's content pass has
|
||||
/// counted its range: a scan that takes seconds on a large root, and one
|
||||
/// that used to run on the writer thread with every other root waiting.
|
||||
pub extract_total: Option<usize>,
|
||||
/// not of files under the root.
|
||||
pub extract_total: usize,
|
||||
pub current_file: Option<String>,
|
||||
/// Threads busy right now / pool size, for the pool this root's current
|
||||
/// phase is running. Both zero once the root is done: its threads are
|
||||
|
|
@ -83,22 +80,14 @@ impl OverallProgress {
|
|||
}
|
||||
|
||||
/// Aggregate every root's progress into the one pair the status bar shows.
|
||||
///
|
||||
/// A root contributes its extraction half only once `extract_total` is known
|
||||
/// — both to `processed` and to `total`, so the two stay in step. Until then
|
||||
/// (during the walk, and for the moments after it while the pass counts) it
|
||||
/// contributes its walk alone.
|
||||
/// `extract_total` is exact the moment a root's walk ends; before that a
|
||||
/// root contributes only its walk.
|
||||
pub fn overall_progress(roots: &[RootProgress]) -> OverallProgress {
|
||||
let processed = roots
|
||||
.iter()
|
||||
.map(|r| r.walked + r.extract_total.map_or(0, |_| r.extracted))
|
||||
.sum();
|
||||
let processed = roots.iter().map(|r| r.walked + r.extracted).sum();
|
||||
let mut total = Some(0usize);
|
||||
for r in roots {
|
||||
match (total, r.walk_denominator()) {
|
||||
(Some(acc), Some(walk)) => {
|
||||
total = Some(acc + walk + r.extract_total.unwrap_or(0));
|
||||
}
|
||||
(Some(acc), Some(walk)) => total = Some(acc + walk + r.extract_total),
|
||||
_ => {
|
||||
total = None;
|
||||
break;
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ use crate::extract::Registry;
|
|||
use crate::file_handling::ExtractCursor;
|
||||
use crate::walk::walk_indexable_files;
|
||||
use std::sync::atomic::AtomicUsize;
|
||||
use std::time::Duration;
|
||||
|
||||
fn tmp_dir(tag: &str) -> std::path::PathBuf {
|
||||
// Canonical: the temp dir itself may sit behind a symlink
|
||||
|
|
@ -87,6 +86,7 @@ fn worker_counts_follow_the_phase() {
|
|||
|
||||
let root = dir.to_string_lossy().into_owned();
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let suspend = Arc::new(AtomicBool::new(false));
|
||||
let walk = walk_indexable_files(
|
||||
std::slice::from_ref(&root),
|
||||
false,
|
||||
|
|
@ -96,6 +96,7 @@ fn worker_counts_follow_the_phase() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
stop.clone(),
|
||||
suspend.clone(),
|
||||
3,
|
||||
);
|
||||
// An empty range, so the pass ends immediately — but its pool size is
|
||||
|
|
@ -106,6 +107,7 @@ fn worker_counts_follow_the_phase() {
|
|||
Arc::new(Registry::default_set()),
|
||||
Config::default(),
|
||||
stop,
|
||||
suspend,
|
||||
2,
|
||||
);
|
||||
|
||||
|
|
@ -121,9 +123,8 @@ fn worker_counts_follow_the_phase() {
|
|||
phase: RootPhase::Walking,
|
||||
phase_started: Instant::now(),
|
||||
content: Some(content),
|
||||
ready: Vec::new(),
|
||||
written: 0,
|
||||
totals: None,
|
||||
extract_total: 0,
|
||||
extracted: 0,
|
||||
current_file: None,
|
||||
};
|
||||
|
||||
|
|
@ -137,154 +138,6 @@ fn worker_counts_follow_the_phase() {
|
|||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// The writer's extraction turn is bounded by its slice, not by what is
|
||||
/// ready: rows the slice does not reach are carried to the next turn, and the
|
||||
/// root is not `Done` until they have all landed. Pinned with a zero slice,
|
||||
/// under which every turn writes exactly one row.
|
||||
#[test]
|
||||
fn an_extracting_turn_lands_its_leftovers_one_slice_at_a_time() {
|
||||
use super::pipeline::RunCx;
|
||||
use crate::content::ExtractedRow;
|
||||
use crate::db::repo::{insert_file, NewFile};
|
||||
use crate::file_handling::ContentOutcome;
|
||||
use crate::mime::FileType;
|
||||
|
||||
let dir = tmp_dir("slice-leftovers");
|
||||
let db_path = dir.join("index.db").to_string_lossy().into_owned();
|
||||
let mut conn = db::open_or_recreate(&db_path, "trigram").unwrap();
|
||||
let tree = dir.join("tree");
|
||||
std::fs::create_dir_all(&tree).unwrap();
|
||||
// Five rows the walk would have written, whose extracted text is
|
||||
// hand-built below rather than read back — the pass is not the subject.
|
||||
let mut ready: Vec<ExtractedRow> = Vec::new();
|
||||
{
|
||||
let tx = conn.transaction().unwrap();
|
||||
for i in 0..5 {
|
||||
let path = tree.join(format!("f{}.txt", i));
|
||||
std::fs::write(&path, "sphinx of black quartz").unwrap();
|
||||
let file_id = insert_file(
|
||||
&tx,
|
||||
&NewFile {
|
||||
name: &format!("f{}.txt", i),
|
||||
path: &path.to_string_lossy(),
|
||||
parent: &tree.to_string_lossy(),
|
||||
size: 22,
|
||||
mtime: 1,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
needs_content: true,
|
||||
},
|
||||
)
|
||||
.unwrap()
|
||||
.expect("unique path");
|
||||
ready.push(ExtractedRow {
|
||||
file_id,
|
||||
name: format!("f{}.txt", i),
|
||||
outcome: ContentOutcome::Done {
|
||||
text: format!("sphinx of black quartz {}", i),
|
||||
},
|
||||
});
|
||||
}
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
let conn_mutex = Arc::new(Mutex::new(conn));
|
||||
|
||||
let root = dir.to_string_lossy().into_owned();
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let config = Config::default();
|
||||
let walk = walk_indexable_files(
|
||||
std::slice::from_ref(&root),
|
||||
false,
|
||||
false,
|
||||
crate::config::IgnoreSet::compile(&[]).unwrap(),
|
||||
&db_path,
|
||||
config.clone(),
|
||||
Arc::new(Registry::default_set()),
|
||||
stop.clone(),
|
||||
1,
|
||||
);
|
||||
// An empty range: the pass reports `Finished` on its own, and the turn
|
||||
// has to keep going past that until `ready` is empty.
|
||||
let content = crate::content::extract_content(
|
||||
&db_path,
|
||||
&ExtractCursor::for_root(&dir.join("nothing").to_string_lossy()),
|
||||
Arc::new(Registry::default_set()),
|
||||
config.clone(),
|
||||
stop.clone(),
|
||||
1,
|
||||
);
|
||||
let mut p = RootPipeline {
|
||||
root,
|
||||
walk,
|
||||
count_total: Arc::new(AtomicUsize::new(0)),
|
||||
workers: 1,
|
||||
pending_updates: Vec::new(),
|
||||
pending_inserts: Vec::new(),
|
||||
walked: 0,
|
||||
walk_clean: true,
|
||||
phase: RootPhase::Extracting,
|
||||
phase_started: Instant::now(),
|
||||
content: Some(content),
|
||||
ready,
|
||||
written: 0,
|
||||
totals: None,
|
||||
current_file: None,
|
||||
};
|
||||
let mut cx = RunCx::new(conn_mutex.clone(), &config, &db_path, &stop);
|
||||
cx.slice = Duration::ZERO;
|
||||
|
||||
let mut turns = 0;
|
||||
while p.phase == RootPhase::Extracting {
|
||||
turns += 1;
|
||||
assert!(
|
||||
turns < 200,
|
||||
"the root never finished: {} written",
|
||||
p.written
|
||||
);
|
||||
let before = p.written;
|
||||
let progressed = p.service_extracting(&mut cx).unwrap();
|
||||
assert!(
|
||||
p.written - before <= 1,
|
||||
"a zero slice wrote {} rows in one turn",
|
||||
p.written - before
|
||||
);
|
||||
assert!(
|
||||
p.phase != RootPhase::Done || p.ready.is_empty(),
|
||||
"Done with {} rows still to write",
|
||||
p.ready.len()
|
||||
);
|
||||
if !progressed {
|
||||
// The empty pass has not reported `Finished` yet.
|
||||
std::thread::sleep(Duration::from_millis(1));
|
||||
}
|
||||
}
|
||||
assert_eq!(p.written, 5);
|
||||
assert!(p.ready.is_empty());
|
||||
assert!(
|
||||
turns >= 5,
|
||||
"five rows cannot land in {} zero-slice turns",
|
||||
turns
|
||||
);
|
||||
let done: i64 = conn_mutex
|
||||
.lock()
|
||||
.unwrap()
|
||||
.query_row(
|
||||
"SELECT COUNT(*) FROM files WHERE content_state = 1",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(done, 5, "every row reached the index");
|
||||
// The empty pass counted its (empty) range, so the totals are known and
|
||||
// the snapshot reports this run's rows on top of the range's zero.
|
||||
assert_eq!(p.snapshot().extracted, 5);
|
||||
assert_eq!(p.snapshot().extract_total, Some(0));
|
||||
|
||||
drop(p);
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// One full run over `config`'s roots, driven directly so the caller owns
|
||||
/// the stop flag. Returns when the run does.
|
||||
fn run_with(config: &Config, db_path: &str, stop: &Arc<AtomicBool>) -> Result<(), String> {
|
||||
|
|
@ -293,6 +146,7 @@ fn run_with(config: &Config, db_path: &str, stop: &Arc<AtomicBool>) -> Result<()
|
|||
&config.paths.indexing_paths,
|
||||
db_path,
|
||||
stop,
|
||||
&Arc::new(AtomicBool::new(false)),
|
||||
config,
|
||||
&Arc::new(Mutex::new(None)),
|
||||
&db::InterruptSlot::default(),
|
||||
|
|
@ -520,7 +374,7 @@ fn progress(phase: RootPhase, walked: usize, walk_total: Option<usize>) -> RootP
|
|||
walked,
|
||||
walk_total,
|
||||
extracted: 0,
|
||||
extract_total: None,
|
||||
extract_total: 0,
|
||||
current_file: None,
|
||||
active_workers: 0,
|
||||
total_workers: 0,
|
||||
|
|
@ -572,7 +426,7 @@ fn overall_progress_sums_both_halves_of_every_root() {
|
|||
let mut walking = progress(RootPhase::Walking, 100, Some(1000));
|
||||
let mut extracting = progress(RootPhase::Extracting, 500, Some(9999));
|
||||
extracting.extracted = 200;
|
||||
extracting.extract_total = Some(400);
|
||||
extracting.extract_total = 400;
|
||||
walking.extracted = 0;
|
||||
|
||||
let o = overall_progress(&[walking, extracting]);
|
||||
|
|
@ -581,19 +435,6 @@ fn overall_progress_sums_both_halves_of_every_root() {
|
|||
assert_eq!(o.total, Some(1900));
|
||||
}
|
||||
|
||||
/// A root whose content pass has not counted its range yet contributes only
|
||||
/// its walk — to both halves, so processed and total stay in step and the
|
||||
/// bar cannot jump when the count lands.
|
||||
#[test]
|
||||
fn an_uncounted_extraction_contributes_only_its_walk() {
|
||||
let mut counting = progress(RootPhase::Extracting, 500, None);
|
||||
counting.extracted = 7;
|
||||
counting.extract_total = None;
|
||||
let o = overall_progress(&[counting]);
|
||||
assert_eq!(o.processed, 500);
|
||||
assert_eq!(o.total, Some(500));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_uncounted_walking_root_leaves_the_whole_total_unknown() {
|
||||
let known = progress(RootPhase::Done, 10, Some(10));
|
||||
|
|
@ -631,7 +472,7 @@ fn a_finished_run_reaches_exactly_one_hundred_percent() {
|
|||
.map(|&(walked, extracted)| {
|
||||
let mut p = progress(RootPhase::Done, walked, Some(walked * 2));
|
||||
p.extracted = extracted;
|
||||
p.extract_total = Some(extracted);
|
||||
p.extract_total = extracted;
|
||||
p
|
||||
})
|
||||
.collect();
|
||||
|
|
@ -655,9 +496,6 @@ fn a_run_with_nothing_to_do_has_no_fraction_to_show() {
|
|||
fn the_fraction_never_exceeds_one() {
|
||||
let mut p = progress(RootPhase::Done, 10, None);
|
||||
p.extracted = 100;
|
||||
// A counted scope the writes then overran; an uncounted one would be
|
||||
// left out of both halves and prove nothing here.
|
||||
p.extract_total = Some(0);
|
||||
let o = overall_progress(&[p]);
|
||||
assert_eq!(o.processed, 110);
|
||||
assert_eq!(o.total, Some(10));
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@ pub mod extract;
|
|||
pub mod file_handling;
|
||||
pub mod incremental;
|
||||
pub mod indexing;
|
||||
pub mod live;
|
||||
pub mod log;
|
||||
pub mod mime;
|
||||
pub mod platform;
|
||||
|
|
@ -20,7 +19,6 @@ pub mod snippet;
|
|||
#[doc(hidden)]
|
||||
pub mod testutil;
|
||||
pub mod textenc;
|
||||
pub mod verify;
|
||||
pub mod walk;
|
||||
pub mod watcher;
|
||||
|
||||
|
|
|
|||
|
|
@ -1,734 +0,0 @@
|
|||
//! Watching the search results a frontend is actually showing.
|
||||
//!
|
||||
//! What a row shows is read from the *file*, never from the index. That is
|
||||
//! what lets this work with indexing stopped, with a file outside every
|
||||
//! indexed root, or against a row the indexer has not caught up with yet —
|
||||
//! and it is the whole point of the feature: the list on screen describes the
|
||||
//! disk, not a snapshot of it.
|
||||
//!
|
||||
//! Nothing here writes to the index. Keeping it so is what lets this run
|
||||
//! alongside the indexer without a second writer; a frontend that wants the
|
||||
//! index brought back in line with what it just displayed hands the paths to
|
||||
//! [`crate::coordinator::IndexCoordinator::update_paths`], which does the
|
||||
//! write on its own thread.
|
||||
//!
|
||||
//! # Why directories, not files
|
||||
//!
|
||||
//! The obvious design is a watch per result file. It does not work. Editors
|
||||
//! save by writing a temporary file and renaming it over the target, so the
|
||||
//! event lands on the *directory* and the old inode — the one a file watch is
|
||||
//! attached to — is simply orphaned. A file watch also cannot report the new
|
||||
//! name of a rename. Watching the deduplicated set of parent directories
|
||||
//! `NonRecursive` sees both, on inotify and on `ReadDirectoryChangesW` alike.
|
||||
//!
|
||||
//! # Why this is not [`crate::watcher`]
|
||||
//!
|
||||
//! That module exists to cover *subtrees*: it walks each root registering
|
||||
//! every directory beneath it, adds directories that appear later, and backs a
|
||||
//! 128k budget with an all-or-nothing guarantee. Pointing it at a result's
|
||||
//! parent would register that parent's whole tree. This is a flat, fixed, tiny
|
||||
//! set with no growth and no budget, and its timings are a tenth of that one's
|
||||
//! — the indexer can afford to coalesce for thirty seconds, a cursor blinking
|
||||
//! next to a stale filename cannot.
|
||||
//!
|
||||
//! # What an event turns into
|
||||
//!
|
||||
//! A rename is applied from the event itself. A content change is answered by
|
||||
//! reading the file: `metadata` for size and modified time, and — for a row
|
||||
//! whose cell shows body text — the same MIME sniffing and extractors the
|
||||
//! indexer uses, re-cut through the same [`crate::search::cascade::text_snippet`]
|
||||
//! (or, for a fuzzy hit, [`crate::search::cascade::fuzzy_snippet`]) the
|
||||
//! search itself uses.
|
||||
//!
|
||||
//! Arming also sweeps every target once, comparing the file on disk against
|
||||
//! the size and modified time the row is *currently displaying*. Since a fresh
|
||||
//! result carries what the index said, that sweep is exactly a check of the
|
||||
//! index against the disk, and it is what makes a row corrected while it was
|
||||
//! scrolled out of view right itself the moment it comes back. It is also the
|
||||
//! only thing that works on a filesystem the platform reports no events for.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::mpsc;
|
||||
use std::sync::Arc;
|
||||
use std::thread::{self, JoinHandle};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use notify::{
|
||||
Config as NotifyConfig, Event as NotifyEvent, EventKind, RecommendedWatcher, RecursiveMode,
|
||||
Watcher as NotifyWatcher,
|
||||
};
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::extract::Registry;
|
||||
use crate::query::split::split_for_cascade;
|
||||
use crate::search::fuzzy::{edit_budget, Bitap};
|
||||
use crate::search::ContentTier;
|
||||
use crate::snippet::Snippet;
|
||||
|
||||
/// How long events for one path are pooled before being acted on, so the
|
||||
/// several notify emits behind a single save collapse into one update.
|
||||
const SETTLE: Duration = Duration::from_millis(150);
|
||||
|
||||
/// Floor on how often any one path may produce an update. A file being written
|
||||
/// in a loop — a log, a build artifact — cannot spin the UI.
|
||||
const MIN_INTERVAL: Duration = Duration::from_millis(750);
|
||||
|
||||
/// How often the loop wakes while it has work pending. It blocks outright when
|
||||
/// it has none, so an idle QuickSearch does not tick at all.
|
||||
const TICK: Duration = Duration::from_millis(50);
|
||||
|
||||
/// Ceiling on updates emitted per tick, so a directory-wide change (an
|
||||
/// unpack, a `chmod -R`) drains over several frames instead of one.
|
||||
const MAX_PER_TICK: usize = 4;
|
||||
|
||||
/// Most watches to register. Results cluster hard — a query's hits usually
|
||||
/// share a handful of directories — so this is generous for the visible rows
|
||||
/// while staying negligible against the indexer's 128k budget.
|
||||
const MAX_DIRS: usize = 64;
|
||||
|
||||
/// Most rows to track, whatever the frontend asks for.
|
||||
const MAX_TARGETS: usize = 256;
|
||||
|
||||
/// One row the frontend is showing and wants kept current.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Target {
|
||||
/// The row's path, spelled exactly as the search returned it. Event paths
|
||||
/// are compared against this byte for byte — see the note in [`watch`].
|
||||
///
|
||||
/// [`watch`]: LiveWatcher::watch
|
||||
pub path: String,
|
||||
/// `Some` when this row displays text from the file's *body*, and so
|
||||
/// needs its snippet re-cut when the file changes — and how, since an
|
||||
/// exact-tier window is cut around the literal term and a fuzzy one
|
||||
/// around a bitap match the literal is usually absent from. `None` for a
|
||||
/// filename or path match, which costs one `metadata` call per change and
|
||||
/// never opens the file.
|
||||
pub text: Option<ContentTier>,
|
||||
/// The size the row is displaying. The arm-time sweep compares the file
|
||||
/// against this, so on a fresh result — where it is whatever the index
|
||||
/// said — the sweep doubles as a check of the index against the disk.
|
||||
pub size: u64,
|
||||
/// The modified time the row is displaying; see [`Target::size`].
|
||||
pub mtime: i64,
|
||||
}
|
||||
|
||||
/// What a change did to a row's Content Match window.
|
||||
///
|
||||
/// Three states, not an `Option`, because "I did not look" and "I looked and
|
||||
/// it is not there any more" have to reach the frontend as different answers.
|
||||
/// Blanking a cell because the file was too large to re-read would lose a
|
||||
/// window the search legitimately found.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum WindowUpdate {
|
||||
/// Nothing to say: not a body-text row, or its body could not be re-read
|
||||
/// (too large, no extractor, unreadable). The cell keeps what it has.
|
||||
Unchanged,
|
||||
/// Re-cut from the file as it is now.
|
||||
Cut(Snippet),
|
||||
/// The body was read and the query is no longer in it. The cell has
|
||||
/// nothing to show and falls back to its dash.
|
||||
NoMatch,
|
||||
}
|
||||
|
||||
/// A ready-to-apply change to one row on screen.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum LiveUpdate {
|
||||
/// The file moved. `path` is the row's old path — the frontend's key.
|
||||
Renamed {
|
||||
path: String,
|
||||
to: String,
|
||||
name: String,
|
||||
},
|
||||
/// The file's contents changed, as read from the file itself.
|
||||
Changed {
|
||||
path: String,
|
||||
size: u64,
|
||||
mtime: i64,
|
||||
window: WindowUpdate,
|
||||
},
|
||||
/// The file is no longer there. Reversible: the directory watch stays, so
|
||||
/// a file recreated at the same path reports [`LiveUpdate::Changed`].
|
||||
Gone { path: String },
|
||||
}
|
||||
|
||||
impl LiveUpdate {
|
||||
/// The row this update is keyed by — the path the frontend knows it as.
|
||||
pub fn path(&self) -> &str {
|
||||
match self {
|
||||
LiveUpdate::Renamed { path, .. }
|
||||
| LiveUpdate::Changed { path, .. }
|
||||
| LiveUpdate::Gone { path } => path,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What one settled event window decided about a path, before the filesystem
|
||||
/// is consulted.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
enum Op {
|
||||
Changed,
|
||||
Renamed(PathBuf),
|
||||
/// Provisional: on Linux the `From` half of a rename arrives before the
|
||||
/// paired event that names the destination, so this may still be upgraded
|
||||
/// to [`Op::Renamed`] inside the same window.
|
||||
Gone,
|
||||
}
|
||||
|
||||
/// Commands and events share one channel so the loop can block on `recv()`
|
||||
/// whenever nothing is pending.
|
||||
enum Msg {
|
||||
Event(NotifyEvent),
|
||||
Watch {
|
||||
query: String,
|
||||
targets: Vec<Target>,
|
||||
/// Boxed: this is by far the largest variant, and a `Watch` is rare
|
||||
/// next to the events sharing the channel with it.
|
||||
config: Box<Config>,
|
||||
},
|
||||
Clear,
|
||||
Stop,
|
||||
}
|
||||
|
||||
/// Handle on the watcher thread. Dropping it stops the thread.
|
||||
pub struct LiveWatcher {
|
||||
tx: mpsc::Sender<Msg>,
|
||||
handle: Option<JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl LiveWatcher {
|
||||
/// Spawn the watcher. `notify` is called after every update is queued, so
|
||||
/// an egui frontend can `request_repaint`; pass a no-op for headless use.
|
||||
pub fn start(notify: Arc<dyn Fn() + Send + Sync>) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>) {
|
||||
let (tx, rx) = mpsc::channel::<Msg>();
|
||||
let (update_tx, update_rx) = mpsc::channel::<LiveUpdate>();
|
||||
let event_tx = tx.clone();
|
||||
let handle = thread::Builder::new()
|
||||
.name("qs-live".into())
|
||||
.spawn(move || {
|
||||
Loop {
|
||||
rx,
|
||||
event_tx,
|
||||
update_tx,
|
||||
notify,
|
||||
watcher: None,
|
||||
targets: HashMap::new(),
|
||||
pending: HashMap::new(),
|
||||
last_emit: HashMap::new(),
|
||||
orphan_to: Vec::new(),
|
||||
settle_at: None,
|
||||
query: None,
|
||||
fuzzy: None,
|
||||
config: None,
|
||||
registry: Registry::default_set(),
|
||||
}
|
||||
.run()
|
||||
})
|
||||
.expect("spawn live watcher");
|
||||
(
|
||||
LiveWatcher {
|
||||
tx,
|
||||
handle: Some(handle),
|
||||
},
|
||||
update_rx,
|
||||
)
|
||||
}
|
||||
|
||||
/// Replace the watched set wholesale.
|
||||
///
|
||||
/// `query` is the search these rows came from; it is what a re-cut snippet
|
||||
/// is marked against. `config` supplies the extraction limits and filters,
|
||||
/// so a snippet cut here is the text the indexer would have stored.
|
||||
/// Registration happens on the watcher's own thread, so this never blocks
|
||||
/// the caller on a spun-down disk or a stale mount.
|
||||
pub fn watch(&self, query: &str, targets: Vec<Target>, config: &Config) {
|
||||
let _ = self.tx.send(Msg::Watch {
|
||||
query: query.to_string(),
|
||||
targets,
|
||||
config: Box::new(config.clone()),
|
||||
});
|
||||
}
|
||||
|
||||
/// Drop every watch and forget every pending update.
|
||||
pub fn clear(&self) {
|
||||
let _ = self.tx.send(Msg::Clear);
|
||||
}
|
||||
|
||||
/// Stop the thread and join it. Idempotent; [`Drop`] calls it.
|
||||
pub fn stop(&mut self) {
|
||||
let _ = self.tx.send(Msg::Stop);
|
||||
if let Some(handle) = self.handle.take() {
|
||||
let _ = handle.join();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for LiveWatcher {
|
||||
fn drop(&mut self) {
|
||||
self.stop();
|
||||
}
|
||||
}
|
||||
|
||||
struct Loop {
|
||||
rx: mpsc::Receiver<Msg>,
|
||||
event_tx: mpsc::Sender<Msg>,
|
||||
update_tx: mpsc::Sender<LiveUpdate>,
|
||||
notify: Arc<dyn Fn() + Send + Sync>,
|
||||
watcher: Option<RecommendedWatcher>,
|
||||
targets: HashMap<String, Target>,
|
||||
pending: HashMap<String, Op>,
|
||||
last_emit: HashMap<String, Instant>,
|
||||
/// Rename destinations seen in this window whose source is not a target —
|
||||
/// the other half of a Windows rename, which carries no pairing cookie.
|
||||
orphan_to: Vec<PathBuf>,
|
||||
settle_at: Option<Instant>,
|
||||
query: Option<crate::query::split::CascadeQuery>,
|
||||
/// The fuzzy matcher for `query`, built at arm time — building it is the
|
||||
/// cost, running it is cheap — for re-cutting [`ContentTier::Fuzzy`]
|
||||
/// rows. `None` when the term does not fuzz (too short, wildcarded, or a
|
||||
/// zero edit budget), which is when the fuzzy pass would not have run.
|
||||
fuzzy: Option<Bitap>,
|
||||
config: Option<Box<Config>>,
|
||||
/// Built once and reused: the extractors are stateless, and the frontend
|
||||
/// re-arms often enough that rebuilding the table per arm would be waste.
|
||||
registry: Registry,
|
||||
}
|
||||
|
||||
impl Loop {
|
||||
fn run(mut self) {
|
||||
loop {
|
||||
// Block outright when there is nothing to time out on: an idle
|
||||
// window costs no wakeups at all.
|
||||
let msg = match self.settle_at {
|
||||
None => match self.rx.recv() {
|
||||
Ok(msg) => Some(msg),
|
||||
Err(_) => return,
|
||||
},
|
||||
Some(deadline) => {
|
||||
let wait = deadline.saturating_duration_since(Instant::now());
|
||||
match self.rx.recv_timeout(wait.min(TICK)) {
|
||||
Ok(msg) => Some(msg),
|
||||
Err(mpsc::RecvTimeoutError::Timeout) => None,
|
||||
Err(mpsc::RecvTimeoutError::Disconnected) => return,
|
||||
}
|
||||
}
|
||||
};
|
||||
match msg {
|
||||
Some(Msg::Stop) => return,
|
||||
Some(Msg::Clear) => self.reset(),
|
||||
Some(Msg::Watch {
|
||||
query,
|
||||
targets,
|
||||
config,
|
||||
}) => self.rearm(&query, targets, config),
|
||||
Some(Msg::Event(event)) => {
|
||||
classify(
|
||||
&event,
|
||||
&self.targets,
|
||||
&mut self.pending,
|
||||
&mut self.orphan_to,
|
||||
);
|
||||
if !self.pending.is_empty() && self.settle_at.is_none() {
|
||||
self.settle_at = Some(Instant::now() + SETTLE);
|
||||
}
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
self.flush_settled();
|
||||
}
|
||||
}
|
||||
|
||||
fn reset(&mut self) {
|
||||
self.watcher = None;
|
||||
self.targets.clear();
|
||||
self.pending.clear();
|
||||
self.last_emit.clear();
|
||||
self.orphan_to.clear();
|
||||
self.settle_at = None;
|
||||
self.query = None;
|
||||
self.fuzzy = None;
|
||||
self.config = None;
|
||||
}
|
||||
|
||||
/// Point the watcher at a new set of rows, dropping everything about the
|
||||
/// old one, then check each one against the disk. Registration failures
|
||||
/// are per-directory and silent beyond the log: this is a cosmetic
|
||||
/// feature, and a modal about a missing highlight would be worse than the
|
||||
/// missing highlight.
|
||||
fn rearm(&mut self, query: &str, targets: Vec<Target>, config: Box<Config>) {
|
||||
self.reset();
|
||||
if targets.is_empty() {
|
||||
return;
|
||||
}
|
||||
self.query = split_for_cascade(query).ok();
|
||||
// The same construction the fuzzy pass makes for a scan, so a fuzzy
|
||||
// row is re-cut against exactly what matched it.
|
||||
self.fuzzy = self.query.as_ref().and_then(|q| {
|
||||
if q.pattern.is_wildcard() {
|
||||
return None;
|
||||
}
|
||||
let folded = q.term.to_ascii_lowercase();
|
||||
let k = edit_budget(folded.len(), config.search.fuzzy_max_edits)?;
|
||||
Bitap::new(folded.as_bytes(), k)
|
||||
});
|
||||
self.config = Some(config);
|
||||
|
||||
let mut watcher = {
|
||||
let tx = self.event_tx.clone();
|
||||
let sink = move |res: notify::Result<NotifyEvent>| {
|
||||
if let Ok(event) = res {
|
||||
let _ = tx.send(Msg::Event(event));
|
||||
}
|
||||
};
|
||||
match RecommendedWatcher::new(sink, NotifyConfig::default()) {
|
||||
Ok(w) => w,
|
||||
Err(e) => {
|
||||
crate::log_warn!("live results: no watcher available: {}", e);
|
||||
return;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
let mut dirs: Vec<PathBuf> = Vec::new();
|
||||
for target in targets.into_iter().take(MAX_TARGETS) {
|
||||
// Derived from the row's own path, never canonicalized: notify
|
||||
// builds each event path as `watched_dir.join(name)`, so leaving
|
||||
// this spelled as the index spells it is what lets event paths be
|
||||
// compared to `Target::path` as plain strings.
|
||||
let Some(dir) = Path::new(&target.path).parent().map(Path::to_path_buf) else {
|
||||
continue;
|
||||
};
|
||||
if !dirs.contains(&dir) {
|
||||
if dirs.len() >= MAX_DIRS {
|
||||
continue;
|
||||
}
|
||||
if let Err(e) = watcher.watch(&dir, RecursiveMode::NonRecursive) {
|
||||
// Deliberately not the indexer's all-or-nothing: partial
|
||||
// coverage of a display nicety is fine. Rows under this
|
||||
// directory are dropped rather than swept — a row nothing
|
||||
// can follow is better left alone than corrected once and
|
||||
// then silently frozen.
|
||||
crate::log_warn!("live results: not watching {}: {}", dir.display(), e);
|
||||
continue;
|
||||
}
|
||||
dirs.push(dir);
|
||||
}
|
||||
self.targets.insert(target.path.clone(), target);
|
||||
}
|
||||
if self.targets.is_empty() {
|
||||
return;
|
||||
}
|
||||
self.watcher = Some(watcher);
|
||||
self.sweep();
|
||||
}
|
||||
|
||||
/// Compare every target against the disk once and emit what disagrees.
|
||||
///
|
||||
/// Nothing here waits for an event, which is the point: a row whose file
|
||||
/// changed while it was scrolled out of view — or one whose index row was
|
||||
/// simply out of date when the search returned it — is corrected the
|
||||
/// moment it is watched.
|
||||
fn sweep(&mut self) {
|
||||
let mut updates: Vec<LiveUpdate> = Vec::new();
|
||||
for target in self.targets.values() {
|
||||
match std::fs::metadata(&target.path) {
|
||||
Ok(meta) if meta.is_file() => {
|
||||
let (size, mtime) = (meta.len(), mtime_of(&meta));
|
||||
if size == target.size && mtime == target.mtime {
|
||||
continue;
|
||||
}
|
||||
updates.push(self.changed_update(&target.path, target.text, size, mtime));
|
||||
}
|
||||
// Unreadable counts as gone: the row cannot be shown as
|
||||
// current when we cannot see the file at all.
|
||||
_ => updates.push(LiveUpdate::Gone {
|
||||
path: target.path.clone(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
let now = Instant::now();
|
||||
for update in updates {
|
||||
// Recorded so an event arriving right behind the sweep — a save
|
||||
// still in flight when the row came on screen — does not repeat
|
||||
// the same answer a moment later.
|
||||
self.last_emit.insert(update.path().to_string(), now);
|
||||
self.note_emitted(&update);
|
||||
self.send(update);
|
||||
}
|
||||
}
|
||||
|
||||
/// Turn the settled window's decisions into updates.
|
||||
fn flush_settled(&mut self) {
|
||||
let Some(at) = self.settle_at else { return };
|
||||
if Instant::now() < at {
|
||||
return;
|
||||
}
|
||||
self.settle_at = None;
|
||||
|
||||
// Windows never reports the two halves of a rename as one event and
|
||||
// gives no cookie to pair them by. When exactly one target went away
|
||||
// and exactly one unclaimed destination appeared in the same window,
|
||||
// they are the same file; anything more ambiguous resolves to the
|
||||
// truthful "gone".
|
||||
let orphans = std::mem::take(&mut self.orphan_to);
|
||||
let gone: Vec<String> = self
|
||||
.pending
|
||||
.iter()
|
||||
.filter(|(_, op)| **op == Op::Gone)
|
||||
.map(|(path, _)| path.clone())
|
||||
.collect();
|
||||
if gone.len() == 1 && orphans.len() == 1 {
|
||||
self.pending
|
||||
.insert(gone[0].clone(), Op::Renamed(orphans[0].clone()));
|
||||
}
|
||||
|
||||
let now = Instant::now();
|
||||
let ready: Vec<(String, Op)> = self
|
||||
.pending
|
||||
.iter()
|
||||
.filter(|(path, _)| {
|
||||
self.last_emit
|
||||
.get(*path)
|
||||
.is_none_or(|t| now.duration_since(*t) >= MIN_INTERVAL)
|
||||
})
|
||||
.take(MAX_PER_TICK)
|
||||
.map(|(path, op)| (path.clone(), op.clone()))
|
||||
.collect();
|
||||
|
||||
for (path, op) in ready {
|
||||
self.pending.remove(&path);
|
||||
self.last_emit.insert(path.clone(), now);
|
||||
self.apply(path, op);
|
||||
}
|
||||
if !self.pending.is_empty() {
|
||||
self.settle_at = Some(now + SETTLE);
|
||||
}
|
||||
}
|
||||
|
||||
fn apply(&mut self, path: String, op: Op) {
|
||||
let update = match op {
|
||||
Op::Gone => LiveUpdate::Gone { path },
|
||||
Op::Renamed(to) => {
|
||||
let name = to
|
||||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())
|
||||
.unwrap_or_default();
|
||||
LiveUpdate::Renamed {
|
||||
path,
|
||||
to: to.to_string_lossy().into_owned(),
|
||||
name,
|
||||
}
|
||||
}
|
||||
Op::Changed => {
|
||||
// Read from the file, not from the index: this has to land
|
||||
// whatever the indexer is doing, or not doing.
|
||||
match std::fs::metadata(&path) {
|
||||
Ok(meta) if meta.is_file() => {
|
||||
let text = self.targets.get(&path).and_then(|t| t.text);
|
||||
self.changed_update(&path, text, meta.len(), mtime_of(&meta))
|
||||
}
|
||||
_ => LiveUpdate::Gone { path },
|
||||
}
|
||||
}
|
||||
};
|
||||
self.note_emitted(&update);
|
||||
self.send(update);
|
||||
}
|
||||
|
||||
/// What the file at `path` implies for the row showing it.
|
||||
fn changed_update(
|
||||
&self,
|
||||
path: &str,
|
||||
text: Option<ContentTier>,
|
||||
size: u64,
|
||||
mtime: i64,
|
||||
) -> LiveUpdate {
|
||||
LiveUpdate::Changed {
|
||||
path: path.to_string(),
|
||||
size,
|
||||
mtime,
|
||||
window: text.map_or(WindowUpdate::Unchanged, |tier| {
|
||||
self.window_from_disk(path, size, tier)
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Re-cut this row's Content Match window from the file on disk.
|
||||
///
|
||||
/// Deliberately the indexer's own path — [`crate::mime::guess_mime_from_head`]
|
||||
/// then [`crate::file_handling::decide_content`], which already applies
|
||||
/// `content_extensions` and the `maximum_text_size` truncation — so the
|
||||
/// text a window is cut from is the text the index would have stored, and
|
||||
/// a refreshed row cannot disagree with a re-run search about anything but
|
||||
/// timing. Then the tier's own matcher, for the same reason.
|
||||
fn window_from_disk(&self, path: &str, size: u64, tier: ContentTier) -> WindowUpdate {
|
||||
let (Some(config), Some(query)) = (self.config.as_deref(), self.query.as_ref()) else {
|
||||
return WindowUpdate::Unchanged;
|
||||
};
|
||||
// The indexer would not have stored text for a file this large, so
|
||||
// neither does the row — reading it would stall this thread over a
|
||||
// window nobody can see the whole of anyway.
|
||||
if size > config.processing.maximum_text_file_size {
|
||||
return WindowUpdate::Unchanged;
|
||||
}
|
||||
let file = Path::new(path);
|
||||
let Some(head) = read_head(file, config.processing.hash_length) else {
|
||||
return WindowUpdate::Unchanged;
|
||||
};
|
||||
let mime = crate::mime::guess_mime_from_head(file, &head);
|
||||
// Extractors run third-party parsers over whatever the file now
|
||||
// holds. One that panics must not take this thread — and with it every
|
||||
// live row for the rest of the session — down with it.
|
||||
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||
crate::file_handling::decide_content(path, mime.as_deref(), &self.registry, config)
|
||||
}));
|
||||
let outcome = match outcome {
|
||||
Ok(outcome) => outcome,
|
||||
Err(_) => {
|
||||
crate::log_warn!("live results: extracting {} panicked", path);
|
||||
return WindowUpdate::Unchanged;
|
||||
}
|
||||
};
|
||||
let Some(text) = crate::file_handling::outcome_body(&outcome) else {
|
||||
return WindowUpdate::Unchanged;
|
||||
};
|
||||
let folded = text.to_ascii_lowercase();
|
||||
let cut = match tier {
|
||||
ContentTier::Exact => {
|
||||
// A literal term always yields a window, marked or not,
|
||||
// because the passes only ever call this for a body FTS
|
||||
// already matched. Here the body may genuinely have stopped
|
||||
// matching, and an unmarked window is how that reads.
|
||||
crate::search::cascade::text_snippet(&query.pattern, text, &folded)
|
||||
.filter(|snip| !snip.ranges.is_empty())
|
||||
}
|
||||
ContentTier::Fuzzy => match &self.fuzzy {
|
||||
Some(bitap) => {
|
||||
crate::search::cascade::fuzzy_snippet(bitap, text, &folded).map(|(_, s)| s)
|
||||
}
|
||||
// The term does not fuzz, so a fuzzy row cannot be re-judged;
|
||||
// leaving it is the honest reading.
|
||||
None => return WindowUpdate::Unchanged,
|
||||
},
|
||||
};
|
||||
match cut {
|
||||
Some(snip) => WindowUpdate::Cut(snip),
|
||||
None => WindowUpdate::NoMatch,
|
||||
}
|
||||
}
|
||||
|
||||
/// Keep the target's baseline in step with what the frontend was just
|
||||
/// told, so a later sweep over the same arm does not repeat itself.
|
||||
fn note_emitted(&mut self, update: &LiveUpdate) {
|
||||
let LiveUpdate::Changed {
|
||||
path, size, mtime, ..
|
||||
} = update
|
||||
else {
|
||||
return;
|
||||
};
|
||||
if let Some(target) = self.targets.get_mut(path) {
|
||||
target.size = *size;
|
||||
target.mtime = *mtime;
|
||||
}
|
||||
}
|
||||
|
||||
fn send(&self, update: LiveUpdate) {
|
||||
if self.update_tx.send(update).is_ok() {
|
||||
(self.notify)();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The first `limit` bytes of a file, for MIME sniffing. A short read is the
|
||||
/// whole file and is not an error; an unreadable file simply has no MIME.
|
||||
fn read_head(path: &Path, limit: usize) -> Option<Vec<u8>> {
|
||||
let file = std::fs::File::open(path).ok()?;
|
||||
let mut head = Vec::new();
|
||||
file.take(limit as u64).read_to_end(&mut head).ok()?;
|
||||
Some(head)
|
||||
}
|
||||
|
||||
fn mtime_of(meta: &std::fs::Metadata) -> i64 {
|
||||
meta.modified()
|
||||
.ok()
|
||||
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
|
||||
.map(|d| d.as_secs() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Fold one notify event into the pending decisions for this window.
|
||||
///
|
||||
/// Pure and filesystem-free, which is what makes the platform differences
|
||||
/// testable: every shape below is a real emission from `notify` 6.1 on one
|
||||
/// platform or the other, and the coalescing window is what reconciles them.
|
||||
fn classify(
|
||||
event: &NotifyEvent,
|
||||
targets: &HashMap<String, Target>,
|
||||
pending: &mut HashMap<String, Op>,
|
||||
orphan_to: &mut Vec<PathBuf>,
|
||||
) {
|
||||
use notify::event::{ModifyKind, RenameMode};
|
||||
|
||||
let key = |p: &PathBuf| p.to_string_lossy().into_owned();
|
||||
let is_target = |p: &PathBuf| targets.contains_key(&key(p));
|
||||
|
||||
match event.kind {
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::Both)) => {
|
||||
// Linux pairs the halves and emits this *after* the From/To pair,
|
||||
// so it lands in the same window and overwrites the provisional
|
||||
// Gone recorded below.
|
||||
let (Some(from), Some(to)) = (event.paths.first(), event.paths.get(1)) else {
|
||||
return;
|
||||
};
|
||||
if is_target(from) {
|
||||
pending.insert(key(from), Op::Renamed(to.clone()));
|
||||
} else if is_target(to) {
|
||||
// The atomic-save shape: a temporary file renamed over a row
|
||||
// we are watching. The row did not move; its contents changed.
|
||||
pending.insert(key(to), Op::Changed);
|
||||
}
|
||||
}
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)) => {
|
||||
for path in &event.paths {
|
||||
if is_target(path) {
|
||||
pending.insert(key(path), Op::Changed);
|
||||
} else {
|
||||
orphan_to.push(path.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)) => {
|
||||
for path in &event.paths {
|
||||
if is_target(path) {
|
||||
// Provisional; a Both in this same window upgrades it.
|
||||
pending.entry(key(path)).or_insert(Op::Gone);
|
||||
}
|
||||
}
|
||||
}
|
||||
EventKind::Remove(_) => {
|
||||
for path in &event.paths {
|
||||
if is_target(path) {
|
||||
pending.insert(key(path), Op::Gone);
|
||||
}
|
||||
}
|
||||
}
|
||||
EventKind::Create(_) | EventKind::Modify(_) => {
|
||||
for path in &event.paths {
|
||||
if is_target(path) {
|
||||
// A Create at a watched path un-deletes the row.
|
||||
pending.insert(key(path), Op::Changed);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Access events, and anything for a path we are not showing. Live
|
||||
// results never *add* rows: we have no way to know an unrelated new
|
||||
// file matches the query, and guessing would be a second, unranked
|
||||
// search wearing the first one's clothes.
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "live_tests.rs"]
|
||||
mod tests;
|
||||
|
|
@ -1,655 +0,0 @@
|
|||
//! Tests for live result watching.
|
||||
//!
|
||||
//! The [`classify`] tests are pure: they feed the exact event shapes `notify`
|
||||
//! 6.1 emits on each platform and check what one settled window decides. They
|
||||
//! are the ones that pin the design — in particular that a watch on a
|
||||
//! *directory* sees an atomic save, which a watch on the file would not.
|
||||
//!
|
||||
//! The end-to-end tests drive a real filesystem through a real watcher.
|
||||
|
||||
use super::*;
|
||||
|
||||
use notify::event::{CreateKind, ModifyKind, RemoveKind, RenameMode};
|
||||
|
||||
fn target(path: &str) -> (String, Target) {
|
||||
(
|
||||
path.to_string(),
|
||||
Target {
|
||||
path: path.to_string(),
|
||||
text: Some(ContentTier::Exact),
|
||||
size: 0,
|
||||
mtime: 0,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn targets(paths: &[&str]) -> HashMap<String, Target> {
|
||||
paths.iter().map(|p| target(p)).collect()
|
||||
}
|
||||
|
||||
fn event(kind: EventKind, paths: &[&str]) -> NotifyEvent {
|
||||
NotifyEvent {
|
||||
kind,
|
||||
paths: paths.iter().map(PathBuf::from).collect(),
|
||||
attrs: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Feed a window of events and report what it decided, after the same
|
||||
/// orphan-pairing `flush_settled` applies.
|
||||
fn window(targets: &HashMap<String, Target>, events: Vec<NotifyEvent>) -> HashMap<String, Op> {
|
||||
let mut pending = HashMap::new();
|
||||
let mut orphan_to = Vec::new();
|
||||
for event in &events {
|
||||
classify(event, targets, &mut pending, &mut orphan_to);
|
||||
}
|
||||
let gone: Vec<String> = pending
|
||||
.iter()
|
||||
.filter(|(_, op)| **op == Op::Gone)
|
||||
.map(|(p, _)| p.clone())
|
||||
.collect();
|
||||
if gone.len() == 1 && orphan_to.len() == 1 {
|
||||
pending.insert(gone[0].clone(), Op::Renamed(orphan_to[0].clone()));
|
||||
}
|
||||
pending
|
||||
}
|
||||
|
||||
/// An editor saving a file writes a temporary and renames it over the target.
|
||||
/// The row did not move — its contents changed — and a watch on the file
|
||||
/// itself would have seen none of this, because the inode it was attached to
|
||||
/// is the one that got orphaned. This test is why the watches are on
|
||||
/// directories.
|
||||
#[test]
|
||||
fn an_atomic_save_reads_as_a_content_change() {
|
||||
let t = targets(&["/docs/report.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
|
||||
&["/docs/.report.txt.swp"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
|
||||
&["/docs/report.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::Both)),
|
||||
&["/docs/.report.txt.swp", "/docs/report.txt"],
|
||||
),
|
||||
],
|
||||
);
|
||||
assert_eq!(decided.get("/docs/report.txt"), Some(&Op::Changed));
|
||||
assert_eq!(decided.len(), 1, "nothing else was decided: {decided:?}");
|
||||
}
|
||||
|
||||
/// Linux emits From, To and then Both for one in-directory rename. The
|
||||
/// provisional Gone recorded for the From half must not escape the window.
|
||||
#[test]
|
||||
fn a_linux_rename_pairs_without_leaking_a_gone() {
|
||||
let t = targets(&["/docs/old.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
|
||||
&["/docs/old.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
|
||||
&["/docs/new.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::Both)),
|
||||
&["/docs/old.txt", "/docs/new.txt"],
|
||||
),
|
||||
],
|
||||
);
|
||||
assert_eq!(
|
||||
decided.get("/docs/old.txt"),
|
||||
Some(&Op::Renamed(PathBuf::from("/docs/new.txt")))
|
||||
);
|
||||
assert!(
|
||||
!decided.values().any(|op| *op == Op::Gone),
|
||||
"a provisional Gone escaped: {decided:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Windows never emits `Both` and gives no cookie to pair the halves by, so
|
||||
/// one unclaimed destination beside one departed target is paired by position.
|
||||
#[test]
|
||||
fn a_windows_rename_pairs_by_elimination() {
|
||||
let t = targets(&["/docs/old.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
|
||||
&["/docs/old.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
|
||||
&["/docs/new.txt"],
|
||||
),
|
||||
],
|
||||
);
|
||||
assert_eq!(
|
||||
decided.get("/docs/old.txt"),
|
||||
Some(&Op::Renamed(PathBuf::from("/docs/new.txt")))
|
||||
);
|
||||
}
|
||||
|
||||
/// Two departures and two arrivals in one window cannot be paired without
|
||||
/// guessing. Guessing wrong renames a row to someone else's file, so the
|
||||
/// ambiguous case resolves to the truthful answer instead.
|
||||
#[test]
|
||||
fn an_ambiguous_windows_window_reports_gone_rather_than_guessing() {
|
||||
let t = targets(&["/docs/a.txt", "/docs/b.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
|
||||
&["/docs/a.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::From)),
|
||||
&["/docs/b.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
|
||||
&["/docs/x.txt"],
|
||||
),
|
||||
event(
|
||||
EventKind::Modify(ModifyKind::Name(RenameMode::To)),
|
||||
&["/docs/y.txt"],
|
||||
),
|
||||
],
|
||||
);
|
||||
assert_eq!(decided.get("/docs/a.txt"), Some(&Op::Gone));
|
||||
assert_eq!(decided.get("/docs/b.txt"), Some(&Op::Gone));
|
||||
}
|
||||
|
||||
/// A watched directory is full of files that are not on screen. None of them
|
||||
/// may produce an update — live results never add rows.
|
||||
#[test]
|
||||
fn events_for_paths_that_are_not_shown_decide_nothing() {
|
||||
let t = targets(&["/docs/shown.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(EventKind::Create(CreateKind::File), &["/docs/other.txt"]),
|
||||
event(EventKind::Modify(ModifyKind::Any), &["/docs/another.txt"]),
|
||||
event(EventKind::Remove(RemoveKind::File), &["/docs/third.txt"]),
|
||||
],
|
||||
);
|
||||
assert!(decided.is_empty(), "{decided:?}");
|
||||
}
|
||||
|
||||
/// A delete marks the row; a file recreated at the same path un-marks it,
|
||||
/// which is what makes the mark reversible without re-registering anything.
|
||||
#[test]
|
||||
fn a_delete_marks_the_row_and_a_recreate_clears_it() {
|
||||
let t = targets(&["/docs/report.txt"]);
|
||||
let gone = window(
|
||||
&t,
|
||||
vec![event(
|
||||
EventKind::Remove(RemoveKind::File),
|
||||
&["/docs/report.txt"],
|
||||
)],
|
||||
);
|
||||
assert_eq!(gone.get("/docs/report.txt"), Some(&Op::Gone));
|
||||
|
||||
let back = window(
|
||||
&t,
|
||||
vec![event(
|
||||
EventKind::Create(CreateKind::File),
|
||||
&["/docs/report.txt"],
|
||||
)],
|
||||
);
|
||||
assert_eq!(back.get("/docs/report.txt"), Some(&Op::Changed));
|
||||
}
|
||||
|
||||
/// Several writes to one file inside a window are one decision, not several.
|
||||
#[test]
|
||||
fn repeated_writes_in_one_window_coalesce() {
|
||||
let t = targets(&["/docs/log.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![
|
||||
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
|
||||
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
|
||||
event(EventKind::Modify(ModifyKind::Any), &["/docs/log.txt"]),
|
||||
],
|
||||
);
|
||||
assert_eq!(decided.len(), 1);
|
||||
assert_eq!(decided.get("/docs/log.txt"), Some(&Op::Changed));
|
||||
}
|
||||
|
||||
// --- end to end ----------------------------------------------------------
|
||||
|
||||
use crate::testutil::scratch_dir;
|
||||
|
||||
/// Collect updates until `want` of them arrive or the timeout expires.
|
||||
fn collect(rx: &mpsc::Receiver<LiveUpdate>, want: usize, timeout: Duration) -> Vec<LiveUpdate> {
|
||||
let deadline = Instant::now() + timeout;
|
||||
let mut out = Vec::new();
|
||||
while out.len() < want {
|
||||
let left = deadline.saturating_duration_since(Instant::now());
|
||||
if left.is_zero() {
|
||||
break;
|
||||
}
|
||||
match rx.recv_timeout(left) {
|
||||
Ok(update) => out.push(update),
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// A target describing `path` exactly as it is on disk right now, so the
|
||||
/// arm-time sweep finds nothing to report and the test sees only what it
|
||||
/// provokes afterwards.
|
||||
fn current_target(path: &str, text: Option<ContentTier>) -> Target {
|
||||
let meta = std::fs::metadata(path).expect("target file exists");
|
||||
Target {
|
||||
path: path.to_string(),
|
||||
text,
|
||||
size: meta.len(),
|
||||
mtime: mtime_of(&meta),
|
||||
}
|
||||
}
|
||||
|
||||
fn watch_one(dir: &Path, name: &str) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>, String) {
|
||||
watch_one_matching(dir, name, "hello world", None)
|
||||
}
|
||||
|
||||
/// Write `body` at `dir/name` and watch it for the query `hello`.
|
||||
fn watch_one_matching(
|
||||
dir: &Path,
|
||||
name: &str,
|
||||
body: &str,
|
||||
text: Option<ContentTier>,
|
||||
) -> (LiveWatcher, mpsc::Receiver<LiveUpdate>, String) {
|
||||
let path = dir.join(name).to_string_lossy().into_owned();
|
||||
std::fs::write(&path, body).unwrap();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![current_target(&path, text)],
|
||||
&Config::default(),
|
||||
);
|
||||
// Registration happens on the watcher thread.
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
(watcher, rx, path)
|
||||
}
|
||||
|
||||
/// Stop the watcher, then take the tree with it.
|
||||
///
|
||||
/// The stop is not optional: the watcher holds an inotify registration on the
|
||||
/// directory, and pulling the directory out from under a live one is a race
|
||||
/// worth not having. Unlike most of this crate's tests these do clean up on
|
||||
/// the way out — the trees are a file or two apiece, generated identically
|
||||
/// every run, so they hold no evidence the assertion message does not already
|
||||
/// carry.
|
||||
fn stop_and_clean(mut watcher: LiveWatcher, dir: &Path) {
|
||||
watcher.stop();
|
||||
std::fs::remove_dir_all(dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn e2e_a_rename_surfaces_with_the_new_name() {
|
||||
let dir = scratch_dir("live-rename");
|
||||
let (watcher, rx, path) = watch_one(&dir, "before.txt");
|
||||
let renamed = dir.join("after.txt");
|
||||
std::fs::rename(&path, &renamed).unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let Some(LiveUpdate::Renamed {
|
||||
path: from, name, ..
|
||||
}) = updates.first()
|
||||
else {
|
||||
panic!("expected a rename, got {updates:?}");
|
||||
};
|
||||
assert_eq!(from, &path);
|
||||
assert_eq!(name, "after.txt");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn e2e_a_delete_surfaces_as_gone() {
|
||||
let dir = scratch_dir("live-delete");
|
||||
let (watcher, rx, path) = watch_one(&dir, "doomed.txt");
|
||||
std::fs::remove_file(&path).unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
assert!(
|
||||
updates
|
||||
.iter()
|
||||
.any(|u| matches!(u, LiveUpdate::Gone { path: p } if *p == path)),
|
||||
"expected a Gone for {path}, got {updates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The design test, on a real filesystem: write a temporary and rename it over
|
||||
/// the target, the way an editor saves. It must read as a change to the row,
|
||||
/// not as the row disappearing — which is exactly what a watch on the file
|
||||
/// itself would have reported.
|
||||
#[test]
|
||||
fn e2e_an_atomic_save_does_not_read_as_a_delete() {
|
||||
let dir = scratch_dir("live-atomic");
|
||||
let (watcher, rx, path) = watch_one(&dir, "report.txt");
|
||||
let tmp = dir.join("report.txt.tmp");
|
||||
std::fs::write(&tmp, "hello, replaced").unwrap();
|
||||
std::fs::rename(&tmp, &path).unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
assert!(
|
||||
!updates.iter().any(|u| matches!(u, LiveUpdate::Gone { .. })),
|
||||
"an atomic save was reported as a deletion: {updates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// A row whose directory does not exist must not panic, and must not stop the
|
||||
/// watcher from covering the rows whose directories do.
|
||||
#[test]
|
||||
fn a_missing_directory_does_not_stop_the_others() {
|
||||
let dir = scratch_dir("live-missing-dir");
|
||||
let good = dir.join("present.txt");
|
||||
std::fs::write(&good, "hello world").unwrap();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![
|
||||
Target {
|
||||
path: dir
|
||||
.join("nowhere")
|
||||
.join("ghost.txt")
|
||||
.to_string_lossy()
|
||||
.into_owned(),
|
||||
text: None,
|
||||
size: 0,
|
||||
mtime: 0,
|
||||
},
|
||||
current_target(&good.to_string_lossy(), None),
|
||||
],
|
||||
&Config::default(),
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
std::fs::remove_file(&good).unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
assert!(
|
||||
updates.iter().any(|u| matches!(u, LiveUpdate::Gone { .. })),
|
||||
"the reachable row stopped working: {updates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
// --- content, read from the file rather than from the index ---------------
|
||||
//
|
||||
// None of these open a database. That is the assertion they all share: a row
|
||||
// on screen tracks the disk with no indexer involved, which is what the
|
||||
// feature is for.
|
||||
|
||||
/// Pull the one `Changed` out of a batch, failing loudly on anything else.
|
||||
fn one_change(updates: &[LiveUpdate], path: &str) -> (u64, i64, WindowUpdate) {
|
||||
let found = updates.iter().find_map(|u| match u {
|
||||
LiveUpdate::Changed {
|
||||
path: p,
|
||||
size,
|
||||
mtime,
|
||||
window,
|
||||
} if p == path => Some((*size, *mtime, window.clone())),
|
||||
_ => None,
|
||||
});
|
||||
found.unwrap_or_else(|| panic!("expected a Changed for {path}, got {updates:?}"))
|
||||
}
|
||||
|
||||
/// The test the old index-backed design could not have: edit a watched file
|
||||
/// with no index anywhere, and the row's size, modified time and Content Match
|
||||
/// window all follow it.
|
||||
#[test]
|
||||
fn e2e_a_content_change_re_cuts_the_snippet_with_no_index() {
|
||||
let dir = scratch_dir("live-content");
|
||||
let (watcher, rx, path) =
|
||||
watch_one_matching(&dir, "notes.txt", "hello world", Some(ContentTier::Exact));
|
||||
std::fs::write(&path, "hello there, a much longer world").unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (size, mtime, window) = one_change(&updates, &path);
|
||||
assert_eq!(size, "hello there, a much longer world".len() as u64);
|
||||
assert!(mtime > 0, "no modified time was read");
|
||||
let WindowUpdate::Cut(snippet) = window else {
|
||||
panic!("the body still matches, so it should carry a window: {window:?}");
|
||||
};
|
||||
assert!(
|
||||
snippet.window.contains("much longer"),
|
||||
"the window is stale: {:?}",
|
||||
snippet.window
|
||||
);
|
||||
assert!(!snippet.ranges.is_empty(), "the match was not marked");
|
||||
}
|
||||
|
||||
/// Edited until it no longer matches, the row keeps its place and its metadata
|
||||
/// but loses its window — the Content Match cell falls back to its dash rather
|
||||
/// than showing text that is no longer a hit.
|
||||
#[test]
|
||||
fn e2e_an_edit_that_removes_the_match_clears_the_window() {
|
||||
let dir = scratch_dir("live-unmatch");
|
||||
let (watcher, rx, path) =
|
||||
watch_one_matching(&dir, "notes.txt", "hello world", Some(ContentTier::Exact));
|
||||
std::fs::write(&path, "nothing of interest here").unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (size, _, window) = one_change(&updates, &path);
|
||||
assert_eq!(size, "nothing of interest here".len() as u64);
|
||||
assert_eq!(
|
||||
window,
|
||||
WindowUpdate::NoMatch,
|
||||
"a window survived the match it no longer has"
|
||||
);
|
||||
}
|
||||
|
||||
/// A fuzzy content hit is re-cut with the fuzzy matcher, not the literal one.
|
||||
/// The literal is absent from the body by construction — that is what made
|
||||
/// it a fuzzy hit — so re-cutting it as an exact row would read as "no longer
|
||||
/// matches" and blank a cell that still has a hit in it.
|
||||
#[test]
|
||||
fn e2e_a_fuzzy_content_hit_is_re_cut_with_the_fuzzy_matcher() {
|
||||
let dir = scratch_dir("live-fuzzy");
|
||||
let path = dir.join("notes.txt").to_string_lossy().into_owned();
|
||||
// "quartz" within one edit of "quarts": a stage-8 hit for that query.
|
||||
std::fs::write(&path, "sphinx of black quarts judge my vow").unwrap();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
watcher.watch(
|
||||
"quartz",
|
||||
vec![current_target(&path, Some(ContentTier::Fuzzy))],
|
||||
&Config::default(),
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
std::fs::write(&path, "the sphinx of black quarts judged my vow again").unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (_, _, window) = one_change(&updates, &path);
|
||||
let WindowUpdate::Cut(snippet) = window else {
|
||||
panic!("a fuzzy hit was re-judged as exact: {window:?}");
|
||||
};
|
||||
assert!(
|
||||
snippet.window.contains("quarts judged"),
|
||||
"the window was not re-cut from the new body: {:?}",
|
||||
snippet.window
|
||||
);
|
||||
assert!(!snippet.ranges.is_empty(), "the fuzzy match was not marked");
|
||||
}
|
||||
|
||||
/// A file no extractor claims still reports what `metadata` knows. Size and
|
||||
/// Modified are not the text columns' to withhold.
|
||||
#[test]
|
||||
fn e2e_a_file_with_no_extractable_text_still_reports_its_metadata() {
|
||||
let dir = scratch_dir("live-binary");
|
||||
let path = dir.join("blob.bin").to_string_lossy().into_owned();
|
||||
std::fs::write(&path, [0x00u8, 0x01, 0x02, 0xFF]).unwrap();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![current_target(&path, Some(ContentTier::Exact))],
|
||||
&Config::default(),
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
std::fs::write(&path, [0x00u8, 0x01, 0x02, 0xFF, 0xFE, 0xFD]).unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (size, _, window) = one_change(&updates, &path);
|
||||
assert_eq!(size, 6);
|
||||
// `Unchanged`, not `NoMatch`: nothing readable came back, so there is no
|
||||
// evidence the row's window is wrong — only that it is unverifiable.
|
||||
assert_eq!(window, WindowUpdate::Unchanged);
|
||||
}
|
||||
|
||||
/// Past `maximum_text_file_size` the indexer stores no text, so neither does
|
||||
/// the row — but it still says how big the file got.
|
||||
#[test]
|
||||
fn e2e_a_file_over_the_text_size_limit_reports_size_but_no_window() {
|
||||
let dir = scratch_dir("live-oversize");
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
let path = dir.join("huge.txt").to_string_lossy().into_owned();
|
||||
std::fs::write(&path, "hello world").unwrap();
|
||||
let mut config = Config::default();
|
||||
config.processing.maximum_text_file_size = 16;
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![current_target(&path, Some(ContentTier::Exact))],
|
||||
&config,
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
std::fs::write(&path, "hello world, and rather more of it besides").unwrap();
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (size, _, window) = one_change(&updates, &path);
|
||||
assert_eq!(
|
||||
size,
|
||||
"hello world, and rather more of it besides".len() as u64
|
||||
);
|
||||
// Not read, so not disproved: the window the search found stands.
|
||||
assert_eq!(window, WindowUpdate::Unchanged);
|
||||
}
|
||||
|
||||
// --- the arm-time sweep ---------------------------------------------------
|
||||
|
||||
/// A row armed with what the *index* said about a file that has since moved on
|
||||
/// is corrected the moment it is watched. This is the check of the index
|
||||
/// against the disk, and it is also the only thing that reports anything at
|
||||
/// all on a filesystem the platform sends no events for.
|
||||
#[test]
|
||||
fn arming_corrects_a_row_that_went_stale_while_it_was_not_watched() {
|
||||
let dir = scratch_dir("live-sweep-stale");
|
||||
let path = dir.join("drifted.txt").to_string_lossy().into_owned();
|
||||
std::fs::write(&path, "hello, a body the index never saw").unwrap();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
// What a stale index row would have claimed.
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![Target {
|
||||
path: path.clone(),
|
||||
text: Some(ContentTier::Exact),
|
||||
size: 5,
|
||||
mtime: 1,
|
||||
}],
|
||||
&Config::default(),
|
||||
);
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
let (size, mtime, window) = one_change(&updates, &path);
|
||||
assert_eq!(size, "hello, a body the index never saw".len() as u64);
|
||||
assert!(mtime > 1, "the stale modified time survived");
|
||||
assert!(
|
||||
matches!(window, WindowUpdate::Cut(_)),
|
||||
"the window was not re-cut: {window:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Same sweep, for a row whose file is simply not there any more.
|
||||
#[test]
|
||||
fn arming_reports_a_row_whose_file_vanished_while_it_was_not_watched() {
|
||||
let dir = scratch_dir("live-sweep-gone");
|
||||
// A sibling keeps the directory watchable, so the ghost is dropped for
|
||||
// being missing rather than for its directory being missing.
|
||||
let sibling = dir.join("present.txt");
|
||||
std::fs::write(&sibling, "hello world").unwrap();
|
||||
let ghost = dir.join("vanished.txt").to_string_lossy().into_owned();
|
||||
let (watcher, rx) = LiveWatcher::start(Arc::new(|| {}));
|
||||
watcher.watch(
|
||||
"hello",
|
||||
vec![Target {
|
||||
path: ghost.clone(),
|
||||
text: None,
|
||||
size: 11,
|
||||
mtime: 1,
|
||||
}],
|
||||
&Config::default(),
|
||||
);
|
||||
|
||||
let updates = collect(&rx, 1, Duration::from_secs(5));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
assert!(
|
||||
updates
|
||||
.iter()
|
||||
.any(|u| matches!(u, LiveUpdate::Gone { path } if *path == ghost)),
|
||||
"expected a Gone for {ghost}, got {updates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The other half of the sweep, and the one that keeps it quiet: a row that
|
||||
/// already agrees with the disk is not touched. Without this the watcher would
|
||||
/// repaint every visible row on every scroll.
|
||||
#[test]
|
||||
fn arming_says_nothing_about_a_row_that_already_agrees_with_the_disk() {
|
||||
let dir = scratch_dir("live-sweep-quiet");
|
||||
let (watcher, rx, _path) =
|
||||
watch_one_matching(&dir, "steady.txt", "hello world", Some(ContentTier::Exact));
|
||||
|
||||
// `watch_one_matching` already waited out registration; anything the
|
||||
// sweep decided has been sent by now.
|
||||
let updates = collect(&rx, 1, Duration::from_millis(500));
|
||||
stop_and_clean(watcher, &dir);
|
||||
|
||||
assert!(
|
||||
updates.is_empty(),
|
||||
"the sweep invented an update: {updates:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Re-arming replaces the set wholesale; an event for a path that is no longer
|
||||
/// shown decides nothing.
|
||||
#[test]
|
||||
fn re_arming_drops_the_previous_targets() {
|
||||
let t = targets(&["/docs/new.txt"]);
|
||||
let decided = window(
|
||||
&t,
|
||||
vec![event(
|
||||
EventKind::Modify(ModifyKind::Any),
|
||||
&["/docs/old.txt"],
|
||||
)],
|
||||
);
|
||||
assert!(decided.is_empty(), "{decided:?}");
|
||||
}
|
||||
|
|
@ -490,6 +490,26 @@ pub fn release_free_heap() {
|
|||
// spans to the kernel on free.
|
||||
}
|
||||
|
||||
/// Live and free-but-retained heap bytes, as `(in_use, free)`: `free` is
|
||||
/// memory already given back to the allocator that glibc still charges the
|
||||
/// process for. `None` where the platform has no way to answer.
|
||||
pub fn heap_stats() -> Option<(u64, u64)> {
|
||||
#[cfg(all(target_os = "linux", target_env = "gnu"))]
|
||||
{
|
||||
// `mallinfo2`, not `mallinfo`: the older struct is `int`-typed and
|
||||
// silently wraps past 2 GiB, which is exactly the size where the
|
||||
// answer starts to matter.
|
||||
//
|
||||
// SAFETY: no arguments, returns a plain struct by value.
|
||||
let info = unsafe { libc::mallinfo2() };
|
||||
Some((info.uordblks as u64, info.fordblks as u64))
|
||||
}
|
||||
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
|
||||
{
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// How long to keep retrying a delete that fails because something else holds
|
||||
/// the file open.
|
||||
#[cfg(windows)]
|
||||
|
|
|
|||
|
|
@ -11,8 +11,8 @@
|
|||
//! stopped being followed. The rows to drop are picked out by a predicate
|
||||
//! no SQL range can express, so [`Scope::covers`] re-runs the walker's own
|
||||
//! filtering rules against each stored path.
|
||||
//! * The content filter moved. The rows stay; only their extracted text
|
||||
//! and FTS entry are re-decided.
|
||||
//! * The content filter moved. The rows stay; only their extracted text,
|
||||
//! properties and FTS entry are re-decided.
|
||||
//!
|
||||
//! Only settings that make stored data unreadable or incomparable — the FTS
|
||||
//! tokenizer, the hash length, the encryption key — still force a wipe. See
|
||||
|
|
@ -89,6 +89,15 @@ impl Scope {
|
|||
})
|
||||
}
|
||||
|
||||
/// The configured root `path` lives under, if any. Containment is
|
||||
/// component-wise, per [`crate::file_handling::UnreadableDirs::covers`].
|
||||
pub fn owning_root(&self, path: &Path) -> Option<&Path> {
|
||||
self.roots
|
||||
.iter()
|
||||
.map(|r| r.path.as_path())
|
||||
.find(|root| path.starts_with(root) && path != *root)
|
||||
}
|
||||
|
||||
/// Whether the walker would still emit `path` while walking `root`.
|
||||
///
|
||||
/// Mirrors `read_directory`'s three `continue`s. Full-path ignore
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ fn walked(config: &Config, db: &Path) -> HashSet<PathBuf> {
|
|||
config.clone(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
2,
|
||||
)
|
||||
.filter_map(|e| match e {
|
||||
|
|
@ -54,6 +55,8 @@ fn seed(conn: &mut Connection, paths: &[PathBuf]) {
|
|||
parent,
|
||||
size: 1,
|
||||
mtime: 1,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: crate::mime::FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -152,6 +155,28 @@ fn a_root_is_never_filtered_but_its_children_still_are() {
|
|||
std::fs::remove_dir_all(&base).ok();
|
||||
}
|
||||
|
||||
/// Root ownership compares whole components, so a sibling whose name
|
||||
/// merely starts with a root's is not inside it — a prune that got this
|
||||
/// wrong would delete a neighbouring folder's entire index.
|
||||
#[test]
|
||||
fn owning_root_does_not_match_name_prefixes() {
|
||||
let base = tmp_tree("prefix");
|
||||
let root = base.join("data");
|
||||
let sibling = base.join("database");
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
std::fs::create_dir_all(&sibling).unwrap();
|
||||
|
||||
let mut config = Config::default();
|
||||
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
|
||||
let scope = Scope::from_config(&config).unwrap();
|
||||
|
||||
assert_eq!(scope.owning_root(&root.join("f.txt")), Some(root.as_path()));
|
||||
assert_eq!(scope.owning_root(&sibling.join("f.txt")), None);
|
||||
// The root itself is a directory, never a row, and owns nothing.
|
||||
assert_eq!(scope.owning_root(&root), None);
|
||||
std::fs::remove_dir_all(&base).ok();
|
||||
}
|
||||
|
||||
/// The counters the status display reads: a scan that reports nothing is
|
||||
/// indistinguishable from a hang.
|
||||
#[test]
|
||||
|
|
@ -298,3 +323,21 @@ fn cancelling_stops_the_scan_without_finishing_it() {
|
|||
std::fs::remove_dir_all(&root).ok();
|
||||
std::fs::remove_dir_all(&db_dir).ok();
|
||||
}
|
||||
|
||||
/// A path under no configured root has no rules to apply — a followed
|
||||
/// symlink's target is the real case; `owning_root` returning `None` is
|
||||
/// what keeps it alive.
|
||||
#[test]
|
||||
fn a_path_outside_every_root_has_no_owner() {
|
||||
let base = tmp_tree("outside");
|
||||
let root = base.join("indexed");
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
|
||||
let mut config = Config::default();
|
||||
config.paths.indexing_paths = vec![root.to_string_lossy().into_owned()];
|
||||
config.indexing.ignore_patterns = vec!["*".into()];
|
||||
let scope = Scope::from_config(&config).unwrap();
|
||||
|
||||
assert_eq!(scope.owning_root(Path::new("/elsewhere/target.txt")), None);
|
||||
std::fs::remove_dir_all(&base).ok();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -80,62 +80,6 @@ const CANCEL_CHECK_ROWS: usize = 256;
|
|||
/// around the match, and the mouseover shows the rest as extended context.
|
||||
const SNIPPET_WINDOW_CHARS: usize = 600;
|
||||
|
||||
/// The Content Match snippet for one document body, cut exactly as the
|
||||
/// full-text passes cut it.
|
||||
///
|
||||
/// `folded` must be `text` ASCII-lowercased. That fold is byte-length
|
||||
/// preserving, which is the whole reason offsets found in it can slice `text`;
|
||||
/// the passes hold one reusable fold buffer per scan and hand it in here
|
||||
/// rather than paying for a second copy.
|
||||
///
|
||||
/// Shared so that [`crate::live`], re-cutting a snippet for a file that
|
||||
/// changed under a result already on screen, produces the same window the
|
||||
/// search itself would — otherwise a row would visibly re-frame its own match
|
||||
/// the moment the file was touched.
|
||||
pub fn text_snippet(
|
||||
pattern: &crate::query::pattern::TermPattern,
|
||||
text: &str,
|
||||
folded: &str,
|
||||
) -> Option<snippet::Snippet> {
|
||||
let opts = snippet::Options {
|
||||
approx_chars: SNIPPET_WINDOW_CHARS,
|
||||
};
|
||||
match pattern.literal() {
|
||||
// Literal terms keep the richer multi-occurrence extract; a wildcard
|
||||
// match marks its own first range.
|
||||
Some(term) => Some(snippet::extract_folded(text, folded, &[term], &opts)),
|
||||
None => pattern.find_first_folded(folded).map(|r| {
|
||||
// A greedy pattern can match megabytes; clamp before the window.
|
||||
let r = clamp_match_range(text, r, SNIPPET_WINDOW_CHARS);
|
||||
snippet::window_around(text, (r.start, r.end), &opts)
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// The fuzzy full-text match in one document body: how many times the term
|
||||
/// occurs within the edit budget, and the Content Match window cut around
|
||||
/// the first occurrence at the cascade's own width. `None` when it does not
|
||||
/// occur at all.
|
||||
///
|
||||
/// Shared with [`crate::live`] for the same reason as [`text_snippet`]: a
|
||||
/// fuzzy row whose file changes has to be re-cut the way it was cut, and
|
||||
/// bitap's range is what it was cut around. `bitap` is built once by the
|
||||
/// caller — per scan in the pass, per arm in the live watcher — since
|
||||
/// building it is the cost, and `folded` must be `text` ASCII-lowercased.
|
||||
pub fn fuzzy_snippet(
|
||||
bitap: &crate::search::fuzzy::Bitap,
|
||||
text: &str,
|
||||
folded: &str,
|
||||
) -> Option<(usize, snippet::Snippet)> {
|
||||
let opts = snippet::Options {
|
||||
approx_chars: SNIPPET_WINDOW_CHARS,
|
||||
};
|
||||
// `first` is `Some` exactly when `count` is non-zero: it *is* the first
|
||||
// of them.
|
||||
let (count, first) = bitap.count_and_first(folded.as_bytes());
|
||||
first.map(|range| (count, snippet::window_around(text, range, &opts)))
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct Outcome {
|
||||
pub total: usize,
|
||||
|
|
|
|||
|
|
@ -10,6 +10,82 @@ enum RowHit {
|
|||
Defer(SearchHit),
|
||||
}
|
||||
|
||||
/// Reusable decode buffer and decompression context for the passes that read
|
||||
/// document text.
|
||||
///
|
||||
/// `zstd::decode_all` builds and tears down a `ZSTD_DCtx` *and* allocates a
|
||||
/// fresh output `Vec` on every call, and it is called once per candidate row.
|
||||
/// One context and one buffer, reused across a whole scan, make that a
|
||||
/// per-scan cost instead of a per-row one.
|
||||
struct DocDecoder {
|
||||
dctx: zstd::bulk::Decompressor<'static>,
|
||||
buf: Vec<u8>,
|
||||
}
|
||||
|
||||
/// Where [`DocDecoder::decode`]'s buffer starts before it has seen a document.
|
||||
/// Most extracted text is well under this, so the doubling below rarely runs.
|
||||
const INITIAL_DOC_CAPACITY: usize = 64 * 1024;
|
||||
|
||||
/// Where the doubling stops. Stored text is capped at
|
||||
/// `processing.maximum_text_size` (256 KiB by default), so this is far above
|
||||
/// any legitimate document even if that setting is raised — past it, a failure
|
||||
/// is a corrupt frame rather than a buffer that is too small.
|
||||
const MAX_DOC_CAPACITY: usize = 64 * 1024 * 1024;
|
||||
|
||||
impl DocDecoder {
|
||||
fn new() -> Result<Self, String> {
|
||||
Ok(DocDecoder {
|
||||
dctx: zstd::bulk::Decompressor::new().map_err(|e| e.to_string())?,
|
||||
buf: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Decompress `blob` and borrow the result as text.
|
||||
///
|
||||
/// Returns `None` for a corrupt frame or non-UTF-8 content. Nothing is
|
||||
/// copied: the indexer stores UTF-8, so the bytes are borrowed in place
|
||||
/// rather than run through `String::from_utf8_lossy(..).into_owned()`,
|
||||
/// which duplicated the whole document even when it was already valid.
|
||||
fn decode(&mut self, blob: &[u8]) -> Option<&str> {
|
||||
self.buf.clear();
|
||||
// `decompress_to_buffer` writes into spare capacity and fails rather
|
||||
// than growing, so the room has to be there first.
|
||||
//
|
||||
// The frame header would say how much is needed, but the indexer
|
||||
// writes with `zstd::encode_all`, which is *stream*-based and so
|
||||
// records no content size — `get_frame_content_size` says `None` for
|
||||
// every row this ever sees. Falling back to `zstd::decode_all` there
|
||||
// looked harmless and was not: it builds a streaming decoder per call,
|
||||
// which measured as one ~2.4 MiB allocation per document and 27 of the
|
||||
// 30 GiB a fuzzy search moved through the allocator.
|
||||
//
|
||||
// So grow this buffer instead and keep reusing it. It settles at the
|
||||
// largest document in the scan within the first few rows, after which
|
||||
// decoding a row allocates nothing at all.
|
||||
if let Ok(Some(size)) = zstd::zstd_safe::get_frame_content_size(blob) {
|
||||
self.buf.reserve(usize::try_from(size).ok()?);
|
||||
}
|
||||
loop {
|
||||
if self.buf.capacity() == 0 {
|
||||
self.buf.reserve(INITIAL_DOC_CAPACITY);
|
||||
}
|
||||
match self.dctx.decompress_to_buffer(blob, &mut self.buf) {
|
||||
Ok(_) => break,
|
||||
// Too small, or corrupt — the bulk API cannot tell us which.
|
||||
// Growing is only worth trying while the buffer is still
|
||||
// smaller than any document could legitimately be.
|
||||
Err(_) if self.buf.capacity() < MAX_DOC_CAPACITY => {
|
||||
let bigger = self.buf.capacity().saturating_mul(2);
|
||||
self.buf.clear();
|
||||
self.buf.reserve(bigger);
|
||||
}
|
||||
Err(_) => return None,
|
||||
}
|
||||
}
|
||||
std::str::from_utf8(&self.buf).ok()
|
||||
}
|
||||
}
|
||||
|
||||
/// Fold `text` into `dst` in place, reusing its allocation.
|
||||
///
|
||||
/// The ASCII fold is byte-length preserving, which is what lets the cascade
|
||||
|
|
@ -166,10 +242,16 @@ impl<'a> Cx<'a> {
|
|||
let is_path_tier = rank >= 9.0;
|
||||
// The "snippet" of a name or path hit is that field itself
|
||||
// with the matched span marked.
|
||||
let snip = snippet::whole_field(
|
||||
if is_path_tier { path } else { name.as_str() },
|
||||
match_range,
|
||||
);
|
||||
let snip = snippet::Snippet {
|
||||
ranges: vec![match_range],
|
||||
window: if is_path_tier {
|
||||
path.to_string()
|
||||
} else {
|
||||
name.clone()
|
||||
},
|
||||
truncated_start: false,
|
||||
truncated_end: false,
|
||||
};
|
||||
let (size, mtime) = size_and_mtime(row)?;
|
||||
let hit = SearchHit {
|
||||
file_id,
|
||||
|
|
@ -245,9 +327,12 @@ impl<'a> Cx<'a> {
|
|||
self.params_with_filters(Vec::new()),
|
||||
),
|
||||
};
|
||||
let snippet_opts = snippet::Options {
|
||||
approx_chars: SNIPPET_WINDOW_CHARS,
|
||||
};
|
||||
// One decoder and one fold buffer for the whole scan; both are reused
|
||||
// per row rather than reallocated.
|
||||
let mut doc = crate::db::repo::DocDecoder::new()?;
|
||||
let mut doc = DocDecoder::new()?;
|
||||
let mut lower = String::new();
|
||||
// Decompression dominates: check cancellation every row.
|
||||
self.scan_pass(&sql, params, 1, None, |cx, row, file_id, path| {
|
||||
|
|
@ -284,7 +369,20 @@ impl<'a> Cx<'a> {
|
|||
if !folded {
|
||||
fold_into(&mut lower, text);
|
||||
}
|
||||
let snip = super::text_snippet(pattern, text, &lower);
|
||||
// Literal terms keep the richer multi-occurrence
|
||||
// extract; a wildcard match marks its own first range.
|
||||
let snip = match pattern.literal() {
|
||||
Some(term) => Some(snippet::extract_folded(
|
||||
text,
|
||||
&lower,
|
||||
&[term],
|
||||
&snippet_opts,
|
||||
)),
|
||||
None => pattern.find_first_folded(&lower).map(|r| {
|
||||
let r = clamp_match_range(text, r, SNIPPET_WINDOW_CHARS);
|
||||
snippet::window_around(text, (r.start, r.end), &snippet_opts)
|
||||
}),
|
||||
};
|
||||
(stage as f64 + count_frac(count), stage as u8, snip)
|
||||
}
|
||||
// No stored text: can't case-verify or count. On the
|
||||
|
|
@ -367,14 +465,15 @@ impl<'a> Cx<'a> {
|
|||
if !cx.regex_accepts(file_id, path, None)? {
|
||||
return Ok(RowHit::Skip);
|
||||
}
|
||||
// The matched field itself with the fuzzy span marked — the
|
||||
// same shape pass A emits, and what `SearchHit::snippet`
|
||||
// documents for the name and path tiers. Windowing it here
|
||||
// used to hand back a *suffix* whenever the match sat past
|
||||
// two thirds of the way through, which broke that contract
|
||||
// and left a frontend unable to line the ranges up against
|
||||
// the field it paints.
|
||||
let snip = Some(snippet::whole_field(field, range));
|
||||
// Mark the approximate matched span in the matched field;
|
||||
// window_around clamps and aligns.
|
||||
let snip = Some(snippet::window_around(
|
||||
field,
|
||||
range,
|
||||
&snippet::Options {
|
||||
approx_chars: field.len().saturating_mul(2).max(8),
|
||||
},
|
||||
));
|
||||
let is_path_tier = rank >= 11.0;
|
||||
let (size, mtime) = size_and_mtime(row)?;
|
||||
let hit = SearchHit {
|
||||
|
|
@ -384,14 +483,7 @@ impl<'a> Cx<'a> {
|
|||
size,
|
||||
mtime,
|
||||
rank,
|
||||
// Stamped, not truncated from `rank`: this is the one pass
|
||||
// whose ranks carry a fraction large enough to reach the
|
||||
// next integer. `edit_budget` is only warned about above
|
||||
// 3, so a distance of 10 makes rank 8.0 — and truncating
|
||||
// that would file a *filename* hit under stage 8, the
|
||||
// fuzzy full-text tier, telling every frontend to render
|
||||
// it as a content match.
|
||||
stage: if is_path_tier { 11 } else { 7 },
|
||||
stage: rank as u8,
|
||||
snippet: snip,
|
||||
};
|
||||
Ok(if is_path_tier {
|
||||
|
|
@ -426,8 +518,11 @@ impl<'a> Cx<'a> {
|
|||
HIT_COLUMNS, self.query.filter_sql
|
||||
);
|
||||
let params = self.params_with_filters(Vec::new());
|
||||
let snippet_opts = snippet::Options {
|
||||
approx_chars: SNIPPET_WINDOW_CHARS,
|
||||
};
|
||||
// One decoder and one fold buffer for the whole scan, reused per row.
|
||||
let mut doc = crate::db::repo::DocDecoder::new()?;
|
||||
let mut doc = DocDecoder::new()?;
|
||||
let mut folded = String::new();
|
||||
// Decompression dominates: check cancellation every row.
|
||||
self.scan_pass(&sql, params, 1, None, |cx, row, file_id, path| {
|
||||
|
|
@ -442,12 +537,14 @@ impl<'a> Cx<'a> {
|
|||
// ASCII folding is byte-length preserving, so ranges found in
|
||||
// the folded buffer are valid in the original.
|
||||
fold_into(&mut folded, text);
|
||||
let Some((count, snip)) = super::fuzzy_snippet(&bitap, text, &folded) else {
|
||||
let (count, first) = bitap.count_and_first(folded.as_bytes());
|
||||
if count == 0 {
|
||||
return Ok(RowHit::Skip);
|
||||
};
|
||||
}
|
||||
if !cx.regex_accepts(file_id, path, Some(text))? {
|
||||
return Ok(RowHit::Skip);
|
||||
}
|
||||
let snip = first.map(|range| snippet::window_around(text, range, &snippet_opts));
|
||||
let (size, mtime) = size_and_mtime(row)?;
|
||||
Ok(RowHit::Emit(SearchHit {
|
||||
file_id,
|
||||
|
|
@ -457,7 +554,7 @@ impl<'a> Cx<'a> {
|
|||
mtime,
|
||||
rank: 8.0 + count_frac(count),
|
||||
stage: 8,
|
||||
snippet: Some(snip),
|
||||
snippet: snip,
|
||||
}))
|
||||
})
|
||||
}
|
||||
|
|
@ -490,10 +587,16 @@ impl<'a> Cx<'a> {
|
|||
None => return Ok(RowHit::Skip),
|
||||
},
|
||||
};
|
||||
let snip = snippet::whole_field(
|
||||
if is_path_tier { path } else { name.as_str() },
|
||||
match_range,
|
||||
);
|
||||
let snip = snippet::Snippet {
|
||||
ranges: vec![match_range],
|
||||
window: if is_path_tier {
|
||||
path.to_string()
|
||||
} else {
|
||||
name.clone()
|
||||
},
|
||||
truncated_start: false,
|
||||
truncated_end: false,
|
||||
};
|
||||
let (size, mtime) = size_and_mtime(row)?;
|
||||
let hit = SearchHit {
|
||||
file_id,
|
||||
|
|
@ -528,7 +631,7 @@ impl<'a> Cx<'a> {
|
|||
approx_chars: SNIPPET_WINDOW_CHARS,
|
||||
};
|
||||
// One decoder for the whole scan, reused per row.
|
||||
let mut doc = crate::db::repo::DocDecoder::new()?;
|
||||
let mut doc = DocDecoder::new()?;
|
||||
// Decompression dominates: check cancellation every row.
|
||||
self.scan_pass(&sql, params, 1, None, |_cx, row, file_id, path| {
|
||||
let blob: Option<&[u8]> = row
|
||||
|
|
|
|||
|
|
@ -100,6 +100,8 @@ mod tests {
|
|||
parent: "/d",
|
||||
size,
|
||||
mtime: 1_700_000_000,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: None,
|
||||
ftype: FileType::TEXT,
|
||||
hash,
|
||||
|
|
|
|||
|
|
@ -21,10 +21,6 @@ const MAX_REGISTERS: usize = 22;
|
|||
pub struct Bitap {
|
||||
/// `masks[c]` has bit `i` set iff `pattern[i] == c`.
|
||||
masks: [u64; 256],
|
||||
/// The same table for the *reversed* pattern, which is what lets
|
||||
/// [`Bitap::match_start`] find where a match began by scanning backwards
|
||||
/// from where it ended.
|
||||
rev_masks: [u64; 256],
|
||||
/// Pattern length in bytes (1..=64).
|
||||
len: usize,
|
||||
/// Maximum edit distance.
|
||||
|
|
@ -40,14 +36,11 @@ impl Bitap {
|
|||
return None;
|
||||
}
|
||||
let mut masks = [0u64; 256];
|
||||
let mut rev_masks = [0u64; 256];
|
||||
for (i, &b) in pattern.iter().enumerate() {
|
||||
masks[b as usize] |= 1u64 << i;
|
||||
rev_masks[b as usize] |= 1u64 << (pattern.len() - 1 - i);
|
||||
}
|
||||
Some(Bitap {
|
||||
masks,
|
||||
rev_masks,
|
||||
len: pattern.len(),
|
||||
k,
|
||||
})
|
||||
|
|
@ -65,14 +58,9 @@ impl Bitap {
|
|||
|
||||
/// Advance all registers by one haystack byte. Returns the smallest
|
||||
/// error count d for which the full pattern just matched, if any.
|
||||
///
|
||||
/// `masks` selects the direction: [`Bitap::masks`] to scan forwards,
|
||||
/// [`Bitap::rev_masks`] to scan backwards. Everything else — `len`, `k`,
|
||||
/// the `done` bit, `reset` — is the same either way, since a reversed
|
||||
/// pattern is still a pattern of the same length.
|
||||
#[inline]
|
||||
fn step(&self, masks: &[u64; 256], r: &mut [u64], byte: u8) -> Option<usize> {
|
||||
let mask = masks[byte as usize];
|
||||
fn step(&self, r: &mut [u64], byte: u8) -> Option<usize> {
|
||||
let mask = self.masks[byte as usize];
|
||||
let done = 1u64 << (self.len - 1);
|
||||
let mut hit = None;
|
||||
let mut prev_old = r[0]; // R_old[d-1] for the d-th iteration
|
||||
|
|
@ -95,127 +83,52 @@ impl Bitap {
|
|||
hit
|
||||
}
|
||||
|
||||
/// Where the match that ended at `end` with `errors` edits began.
|
||||
///
|
||||
/// The forward scan knows an occurrence's *end* exactly — that is the bit
|
||||
/// it tests — but not its start, and with an insertion or a deletion the
|
||||
/// match is not `len` bytes long, so `end - len` is simply the wrong
|
||||
/// offset. Highlighting it put the marks a byte or two off the match and
|
||||
/// over whatever preceded it: `repot` against `1Reporter` marked `1Repo`.
|
||||
///
|
||||
/// So the same automaton runs over the *reversed* pattern, backwards from
|
||||
/// `end`. The first position it accepts **within `errors` edits** is the
|
||||
/// start. That bound is what makes this correct rather than merely
|
||||
/// plausible: the reversed pattern will also accept far shorter spans by
|
||||
/// spending its whole budget on deletions — against `xabc` with a 2-edit
|
||||
/// budget it accepts `c` alone on the very first byte — and taking that
|
||||
/// would mark one letter of an exact three-letter match. The true
|
||||
/// alignment costs the same read either way, so requiring `≤ errors`
|
||||
/// rejects the cheap wrong answers and is still guaranteed to fire at or
|
||||
/// before the real start.
|
||||
///
|
||||
/// Which occurrence gets marked is settled elsewhere — see
|
||||
/// [`Bitap::refine_end`]. The rule both producers land on is *the earliest
|
||||
/// alignment at the smallest edit distance*, so a mark is never longer
|
||||
/// than the term and never shorter by more than the budget.
|
||||
fn match_start(&self, hay: &[u8], end: usize, errors: usize) -> usize {
|
||||
// A ≤k-edit alignment of a len-byte pattern is at most len+k long,
|
||||
// so nothing before this can be the start.
|
||||
let floor = end.saturating_sub(self.len + self.k);
|
||||
let mut r = [0u64; MAX_REGISTERS];
|
||||
self.reset(&mut r);
|
||||
for (back, &b) in hay[floor..end].iter().rev().enumerate() {
|
||||
if self
|
||||
.step(&self.rev_masks, &mut r, b)
|
||||
.is_some_and(|d| d <= errors)
|
||||
{
|
||||
return end - (back + 1);
|
||||
}
|
||||
}
|
||||
// Unreachable: the forward scan proved an alignment ends here, and
|
||||
// reversed it costs the same. Falling back to the floor keeps a
|
||||
// hypothetical miss inside the haystack.
|
||||
floor
|
||||
/// Minimum edit distance (≤ k) of any occurrence of the pattern in
|
||||
/// `hay`, or `None` if nothing matches within k edits.
|
||||
pub fn best_distance(&self, hay: &[u8]) -> Option<usize> {
|
||||
self.best_distance_and_first(hay).map(|(d, _)| d)
|
||||
}
|
||||
|
||||
/// Improve on the *earliest* accepting end by looking a little past it.
|
||||
///
|
||||
/// The automaton accepts as soon as a leading part of the pattern has
|
||||
/// matched, paying for the rest with trailing deletions — so the first
|
||||
/// end it reports is systematically short. Searching `abcdef` over
|
||||
/// `zzabcdefzz` accepts after `abcd`, two deletions, with the whole word
|
||||
/// sitting right there.
|
||||
///
|
||||
/// Each further byte can turn one of those deletions into a match, so a
|
||||
/// better alignment ends at most `errors` bytes later and never more.
|
||||
/// Stepping a *copy* of the registers that far finds it without
|
||||
/// disturbing the caller's scan, or its count.
|
||||
fn refine_end(
|
||||
&self,
|
||||
hay: &[u8],
|
||||
r: &[u64; MAX_REGISTERS],
|
||||
end: usize,
|
||||
errors: usize,
|
||||
) -> (usize, usize) {
|
||||
let mut best = (errors, end);
|
||||
let mut probe = *r;
|
||||
for (ahead, &b) in hay[end..].iter().take(errors).enumerate() {
|
||||
if let Some(d) = self.step(&self.masks, &mut probe, b) {
|
||||
if d < best.0 {
|
||||
best = (d, end + ahead + 1);
|
||||
/// [`best_distance`](Self::best_distance) plus the first match's
|
||||
/// approximate byte range, from one sweep. The range carries the same
|
||||
/// caveat as [`count_and_first`](Self::count_and_first): it assumes a
|
||||
/// pattern-length match, so edits can shift the true start by up to `k`.
|
||||
pub fn best_distance_and_first(&self, hay: &[u8]) -> Option<(usize, (usize, usize))> {
|
||||
let mut r = [0u64; MAX_REGISTERS];
|
||||
self.reset(&mut r);
|
||||
let mut best: Option<(usize, (usize, usize))> = None;
|
||||
for (i, &b) in hay.iter().enumerate() {
|
||||
if let Some(d) = self.step(&mut r, b) {
|
||||
let end = i + 1;
|
||||
let range = (end.saturating_sub(self.len), end);
|
||||
if d == 0 {
|
||||
return Some((0, range));
|
||||
}
|
||||
if best.is_none_or(|(cur, _)| d < cur) {
|
||||
best = Some((d, range));
|
||||
}
|
||||
}
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
/// The smallest edit distance (≤ k) at which the pattern occurs in `hay`,
|
||||
/// and that occurrence's byte range — the span a frontend marks.
|
||||
///
|
||||
/// This one sweeps the whole haystack, so it finds the best alignment
|
||||
/// without help; [`Bitap::count_and_first`] resets after every hit and
|
||||
/// needs [`Bitap::refine_end`] instead.
|
||||
pub fn best_distance_and_first(&self, hay: &[u8]) -> Option<(usize, (usize, usize))> {
|
||||
let mut r = [0u64; MAX_REGISTERS];
|
||||
self.reset(&mut r);
|
||||
// (errors, end). The start is resolved once, at the end, rather than
|
||||
// per improvement — `match_start` is a second scan, however short.
|
||||
let mut best: Option<(usize, usize)> = None;
|
||||
for (i, &b) in hay.iter().enumerate() {
|
||||
if let Some(d) = self.step(&self.masks, &mut r, b) {
|
||||
let end = i + 1;
|
||||
if d == 0 {
|
||||
best = Some((0, end));
|
||||
break;
|
||||
}
|
||||
if best.is_none_or(|(cur, _)| d < cur) {
|
||||
best = Some((d, end));
|
||||
}
|
||||
}
|
||||
}
|
||||
best.map(|(d, end)| (d, (self.match_start(hay, end, d), end)))
|
||||
}
|
||||
|
||||
/// Count non-overlapping occurrences (at ≤ k edits) and report the first
|
||||
/// one's byte range in `hay`. After each hit the automaton resets, so an
|
||||
/// exact match followed by trailing bytes counts once, and overlapping
|
||||
/// suffix matches don't inflate counts.
|
||||
///
|
||||
/// The range is the occurrence itself: [`Bitap::refine_end`] settles which
|
||||
/// end, then [`Bitap::match_start`] finds where it began. Both run for the
|
||||
/// first hit only, and both are bounded by the edit budget, so the cost is
|
||||
/// O(len + k) per row rather than per byte.
|
||||
/// Count non-overlapping occurrences (at ≤ k edits) and report the
|
||||
/// first match's approximate byte range in `hay`. After each hit the
|
||||
/// automaton resets, so an exact match followed by trailing bytes
|
||||
/// counts once, and overlapping suffix matches don't inflate counts.
|
||||
/// The reported range assumes pattern-length matches — edits can shift
|
||||
/// the true start by up to k bytes, which is fine for snippet windows.
|
||||
pub fn count_and_first(&self, hay: &[u8]) -> (usize, Option<(usize, usize)>) {
|
||||
let mut r = [0u64; MAX_REGISTERS];
|
||||
self.reset(&mut r);
|
||||
let mut count = 0usize;
|
||||
let mut first: Option<(usize, usize)> = None;
|
||||
for (i, &b) in hay.iter().enumerate() {
|
||||
if let Some(d) = self.step(&self.masks, &mut r, b) {
|
||||
if self.step(&mut r, b).is_some() {
|
||||
count += 1;
|
||||
if first.is_none() {
|
||||
let (errors, end) = self.refine_end(hay, &r, i + 1, d);
|
||||
first = Some((self.match_start(hay, end, errors), end));
|
||||
let end = i + 1;
|
||||
first = Some((end.saturating_sub(self.len), end));
|
||||
}
|
||||
self.reset(&mut r);
|
||||
}
|
||||
|
|
@ -245,28 +158,7 @@ mod tests {
|
|||
fn best(pattern: &str, hay: &str, k: usize) -> Option<usize> {
|
||||
Bitap::new(pattern.as_bytes(), k)
|
||||
.unwrap()
|
||||
.best_distance_and_first(hay.as_bytes())
|
||||
.map(|(d, _)| d)
|
||||
}
|
||||
|
||||
/// The slice of `hay` that a pattern's first occurrence marks — what a
|
||||
/// frontend highlights.
|
||||
fn marked<'h>(pattern: &str, hay: &'h str, k: usize) -> &'h str {
|
||||
let (_, first) = Bitap::new(pattern.as_bytes(), k)
|
||||
.unwrap()
|
||||
.count_and_first(hay.as_bytes());
|
||||
let (s, e) = first.expect("the pattern occurs");
|
||||
&hay[s..e]
|
||||
}
|
||||
|
||||
/// [`marked`] through the other range producer, which shares
|
||||
/// `match_start` but reaches it by a different route.
|
||||
fn marked_best<'h>(pattern: &str, hay: &'h str, k: usize) -> &'h str {
|
||||
let (_, (s, e)) = Bitap::new(pattern.as_bytes(), k)
|
||||
.unwrap()
|
||||
.best_distance_and_first(hay.as_bytes())
|
||||
.expect("the pattern occurs");
|
||||
&hay[s..e]
|
||||
.best_distance(hay.as_bytes())
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -331,137 +223,20 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn count_fuzzy_and_range_is_the_occurrence_itself() {
|
||||
fn count_fuzzy_and_range_sane() {
|
||||
let b = Bitap::new(b"hello", 1).unwrap();
|
||||
let hay = b"say helo and hxllo again";
|
||||
let (count, first) = b.count_and_first(hay);
|
||||
assert_eq!(count, 2);
|
||||
// The occurrence, not a five-byte window ending where it ends: that
|
||||
// reached back over the space and marked " helo".
|
||||
assert_eq!(first, Some((4, 8)));
|
||||
assert_eq!(&hay[4..8], b"helo");
|
||||
}
|
||||
|
||||
/// The reported bug, exactly: `repot` marked `1Repo` in `1Reporter`.
|
||||
///
|
||||
/// The match is `repo` — one deletion, dropping the `t` — so it is four
|
||||
/// bytes where the term is five, and a range assumed to be term-length
|
||||
/// reached one byte too far left, over the `1`. Both producers, since
|
||||
/// they share `match_start`.
|
||||
#[test]
|
||||
fn a_match_shorter_than_the_term_is_still_marked_exactly() {
|
||||
assert_eq!(marked("repot", "1reporter", 1), "repo");
|
||||
assert_eq!(marked_best("repot", "1reporter", 1), "repo");
|
||||
|
||||
// Substitution keeps the length, which is the case that always
|
||||
// worked — worth holding, since it is the one the old arithmetic got
|
||||
// right by accident.
|
||||
assert_eq!(marked("hello", "xx hxllo xx", 1), "hxllo");
|
||||
assert_eq!(marked_best("hello", "xx hxllo xx", 1), "hxllo");
|
||||
}
|
||||
|
||||
/// Text with a byte inserted into the term marks up to the insertion, not
|
||||
/// across it: `abxc` is a one-edit alignment of `abc`, but so is the `ab`
|
||||
/// that ends two bytes earlier, and the rule is the *earliest* alignment
|
||||
/// at the best distance. Nothing longer than the term can win — spanning
|
||||
/// an inserted byte costs an edit, and deleting instead costs the same and
|
||||
/// ends sooner.
|
||||
#[test]
|
||||
fn an_insertion_marks_up_to_it_rather_than_over_it() {
|
||||
assert_eq!(marked("abc", "abxcd", 1), "ab");
|
||||
assert_eq!(marked_best("abc", "zzabxczz", 1), "ab");
|
||||
}
|
||||
|
||||
/// The trap in resolving the start backwards: the reversed pattern will
|
||||
/// happily accept a much shorter span by spending its budget on
|
||||
/// deletions, so taking its *first* acceptance marks one letter of an
|
||||
/// exact match. `abc` occurs verbatim in `xabc`, and a 2-edit budget lets
|
||||
/// the reverse pass accept `c` alone one byte in.
|
||||
///
|
||||
/// Only the whole-haystack producer can reach the trap — it is the one
|
||||
/// that reports an exact match while the budget is still generous, so
|
||||
/// `errors` is 0 where `k` is 2.
|
||||
#[test]
|
||||
fn a_generous_budget_does_not_shrink_an_exact_match() {
|
||||
assert_eq!(marked_best("abc", "xabc", 2), "abc");
|
||||
assert_eq!(marked_best("abcdef", "zzabcdefzz", 2), "abcdef");
|
||||
assert_eq!(marked("hello", "say hello world", 2), "hello");
|
||||
}
|
||||
|
||||
/// The automaton accepts as soon as a leading part of the term has
|
||||
/// matched, spending the rest of the budget on trailing deletions — so
|
||||
/// the earliest end is short, and marking it highlighted `abcd` for a
|
||||
/// search for `abcdef` with the whole word right there. `refine_end`
|
||||
/// looks the budget's worth of bytes past the first acceptance.
|
||||
///
|
||||
/// Checked against a brute-force Levenshtein oracle over every span.
|
||||
#[test]
|
||||
fn the_mark_is_not_truncated_to_a_leading_part_of_the_term() {
|
||||
assert_eq!(marked("abcdef", "zzabcdefzz", 2), "abcdef");
|
||||
assert_eq!(marked("abc", "xabc", 1), "abc");
|
||||
assert_eq!(marked("reports", "the report went out", 2), "report");
|
||||
|
||||
// Not every term can be extended: `repot` against `1reporter` stops
|
||||
// at `repo` because the next byte (`r`) costs an edit of its own, so
|
||||
// one is the best it does either way.
|
||||
assert_eq!(marked("repot", "1reporter", 1), "repo");
|
||||
// And an alignment already at zero errors has nothing to improve.
|
||||
assert_eq!(marked("hello", "say hello world", 0), "hello");
|
||||
}
|
||||
|
||||
/// However the span is chosen it is a real alignment at the best distance,
|
||||
/// so it is never longer than the term and never shorter by more than the
|
||||
/// budget. That bound is what keeps a mark recognisable: at the production
|
||||
/// ladder of one edit per three characters, a mark is always at least two
|
||||
/// thirds of the term.
|
||||
#[test]
|
||||
fn the_marked_span_is_within_the_budget_of_the_terms_length() {
|
||||
for (term, hay) in [
|
||||
("repot", "1reporter"),
|
||||
("abcdef", "zzabcdefzz"),
|
||||
("quarterly", "the quartrly budget"),
|
||||
("hello", "say helo and hxllo again"),
|
||||
("reports", "the report went out"),
|
||||
] {
|
||||
let k = edit_budget(term.len(), 2).expect("a real budget");
|
||||
let span = marked(term, hay, k).len();
|
||||
let (s, e) = first.unwrap();
|
||||
assert!(s < e && e <= hay.len());
|
||||
let window = &hay[s..e];
|
||||
assert!(
|
||||
span <= term.len() && term.len() - span <= k,
|
||||
"{term:?} in {hay:?} (k={k}) marked {span} bytes"
|
||||
std::str::from_utf8(window).unwrap().contains("hel"),
|
||||
"first range should cover the first hit, got {:?}",
|
||||
std::str::from_utf8(window)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_zero_budget_marks_exactly_the_term() {
|
||||
assert_eq!(marked("hello", "say hello world", 0), "hello");
|
||||
assert_eq!(marked("ab", "ab ab", 0), "ab");
|
||||
}
|
||||
|
||||
/// A match at the very start, and one whose end is inside the term's own
|
||||
/// length, are where the offset arithmetic can underflow.
|
||||
#[test]
|
||||
fn a_match_at_the_start_of_the_haystack_stays_in_bounds() {
|
||||
assert_eq!(marked("repot", "reporter", 1), "repo");
|
||||
// The haystack is shorter than the term: "ab" matches "abc" with one
|
||||
// deletion, ending at 2.
|
||||
let (_, first) = Bitap::new(b"abc", 1).unwrap().count_and_first(b"ab");
|
||||
assert_eq!(first, Some((0, 2)));
|
||||
}
|
||||
|
||||
/// Bitap works on bytes over an ASCII-folded copy, so a range can land
|
||||
/// inside a multi-byte character. `snippet::aligned_range` is what widens
|
||||
/// it before anything slices; this only pins that the range stays inside
|
||||
/// the haystack so that alignment has something valid to work from.
|
||||
#[test]
|
||||
fn a_range_over_multibyte_text_stays_within_the_haystack() {
|
||||
let hay = "café notes — le rapport";
|
||||
let (_, first) = Bitap::new(b"raport", 1)
|
||||
.unwrap()
|
||||
.count_and_first(hay.as_bytes());
|
||||
let (s, e) = first.expect("one deletion from 'rapport'");
|
||||
assert!(s < e && e <= hay.len(), "({s}, {e}) outside {}", hay.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edit_budget_default_cap_is_the_historic_ladder() {
|
||||
|
|
@ -560,10 +335,7 @@ mod tests {
|
|||
let pattern: Vec<u8> = (0..plen).map(|_| alphabet[rng() % 4]).collect();
|
||||
let hay: Vec<u8> = (0..hlen).map(|_| alphabet[rng() % 4]).collect();
|
||||
for k in 0..=4 {
|
||||
let got = Bitap::new(&pattern, k)
|
||||
.unwrap()
|
||||
.best_distance_and_first(&hay)
|
||||
.map(|(d, _)| d);
|
||||
let got = Bitap::new(&pattern, k).unwrap().best_distance(&hay);
|
||||
let want = oracle(&pattern, &hay, k);
|
||||
assert_eq!(
|
||||
got,
|
||||
|
|
|
|||
|
|
@ -69,55 +69,6 @@ pub struct SearchHit {
|
|||
pub snippet: Option<Snippet>,
|
||||
}
|
||||
|
||||
/// Which field a hit's [`SearchHit::snippet`] excerpts, derived from the
|
||||
/// cascade stage. See the rank table at the top of [`crate::search::cascade`].
|
||||
///
|
||||
/// Frontends branch on this rather than on the raw stage number, so a new tier
|
||||
/// is classified in one place instead of in every renderer.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum MatchField {
|
||||
Name,
|
||||
Contents,
|
||||
Path,
|
||||
}
|
||||
|
||||
/// How a [`MatchField::Contents`] hit matched its body — what has to be
|
||||
/// re-run to cut its snippet again from the file as it now stands.
|
||||
///
|
||||
/// The two are not interchangeable: an exact tier's snippet is cut around the
|
||||
/// literal term, and a fuzzy tier's around a bitap match the literal is
|
||||
/// usually *absent* from. Re-cutting a fuzzy hit as if it were exact finds
|
||||
/// nothing and reads as "the file no longer matches".
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum ContentTier {
|
||||
/// Stages 5 and 6: the body contains the term as written.
|
||||
Exact,
|
||||
/// Stage 8: the body contains something within the fuzzy edit budget of
|
||||
/// the term.
|
||||
Fuzzy,
|
||||
}
|
||||
|
||||
impl SearchHit {
|
||||
pub fn match_field(&self) -> MatchField {
|
||||
match self.stage {
|
||||
1..=4 | 7 => MatchField::Name,
|
||||
5 | 6 | 8 => MatchField::Contents,
|
||||
// 9..=11, and whatever a later tier adds: the path is the safe
|
||||
// reading, since it is the one field every hit carries in full.
|
||||
_ => MatchField::Path,
|
||||
}
|
||||
}
|
||||
|
||||
/// `Some` for a hit whose snippet is a window on the file's body.
|
||||
pub fn content_tier(&self) -> Option<ContentTier> {
|
||||
match self.stage {
|
||||
5 | 6 => Some(ContentTier::Exact),
|
||||
8 => Some(ContentTier::Fuzzy),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum SearchUpdate {
|
||||
Started {
|
||||
|
|
|
|||
|
|
@ -52,6 +52,13 @@ impl Snippet {
|
|||
}
|
||||
}
|
||||
|
||||
/// Extract a snippet from `text` marking every occurrence of any term in
|
||||
/// `terms` (ASCII-case-insensitive). With no terms or no matches, returns
|
||||
/// the head of the text as the window with no ranges.
|
||||
pub fn extract(text: &str, terms: &[&str], opts: &Options) -> Snippet {
|
||||
extract_folded(text, &text.to_ascii_lowercase(), terms, opts)
|
||||
}
|
||||
|
||||
/// [`extract`] against a haystack the caller has already ASCII-folded.
|
||||
/// `folded` must be `text.to_ascii_lowercase()` — the fold is byte-length
|
||||
/// preserving, which is what lets offsets found in it slice the original.
|
||||
|
|
@ -132,48 +139,6 @@ pub fn extract_folded(text: &str, folded: &str, terms: &[&str], opts: &Options)
|
|||
}
|
||||
}
|
||||
|
||||
/// Clamp `range` into `text` and widen it to the nearest char boundaries.
|
||||
///
|
||||
/// Both callers below take ranges from matchers that work on bytes — bitap
|
||||
/// over an ASCII-folded copy — so an endpoint can land inside a multi-byte
|
||||
/// character. Slicing there panics, and `Snippet::ranges` promises boundaries.
|
||||
fn aligned_range(text: &str, range: (usize, usize)) -> (usize, usize) {
|
||||
let (mut start, mut end) = range;
|
||||
start = start.min(text.len());
|
||||
end = end.clamp(start, text.len());
|
||||
while start > 0 && !text.is_char_boundary(start) {
|
||||
start -= 1;
|
||||
}
|
||||
while end < text.len() && !text.is_char_boundary(end) {
|
||||
end += 1;
|
||||
}
|
||||
(start, end)
|
||||
}
|
||||
|
||||
/// The whole of `text` as the window, with `range` marked.
|
||||
///
|
||||
/// This is the shape [`crate::search::SearchHit::snippet`] documents for the
|
||||
/// name and path tiers, and what lets a frontend highlight the matched span
|
||||
/// inside its own Name or Path column: `window` is that field verbatim, so the
|
||||
/// ranges index the field the column is already painting. A filename or a path
|
||||
/// is short enough to carry whole, so there is nothing to gain by windowing it.
|
||||
pub fn whole_field(text: &str, range: (usize, usize)) -> Snippet {
|
||||
if text.is_empty() {
|
||||
return Snippet::empty();
|
||||
}
|
||||
let (start, end) = aligned_range(text, range);
|
||||
Snippet {
|
||||
window: text.to_string(),
|
||||
ranges: if end > start {
|
||||
vec![(start, end)]
|
||||
} else {
|
||||
Vec::new()
|
||||
},
|
||||
truncated_start: false,
|
||||
truncated_end: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a snippet window around one known match range in `text` (byte
|
||||
/// offsets into `text`). Used by fuzzy full-text search, where the match
|
||||
/// was located by the fuzzy matcher rather than exact term search. The
|
||||
|
|
@ -182,7 +147,15 @@ pub fn window_around(text: &str, range: (usize, usize), opts: &Options) -> Snipp
|
|||
if text.is_empty() {
|
||||
return Snippet::empty();
|
||||
}
|
||||
let (ms, me) = aligned_range(text, range);
|
||||
let (mut ms, mut me) = range;
|
||||
ms = ms.min(text.len());
|
||||
me = me.clamp(ms, text.len());
|
||||
while ms > 0 && !text.is_char_boundary(ms) {
|
||||
ms -= 1;
|
||||
}
|
||||
while me < text.len() && !text.is_char_boundary(me) {
|
||||
me += 1;
|
||||
}
|
||||
|
||||
let pre_pad = opts.approx_chars / 3;
|
||||
let mut win_start = ms.saturating_sub(pre_pad);
|
||||
|
|
@ -271,13 +244,6 @@ fn coalesce_overlapping(v: Vec<(usize, usize)>) -> Vec<(usize, usize)> {
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The tests were written against a since-removed `extract` wrapper.
|
||||
/// Production always holds a fold buffer already, so the wrapper earned
|
||||
/// nothing; folding here keeps its coverage of the window logic.
|
||||
fn extract(text: &str, terms: &[&str], opts: &Options) -> Snippet {
|
||||
extract_folded(text, &text.to_ascii_lowercase(), terms, opts)
|
||||
}
|
||||
|
||||
fn opts_small() -> Options {
|
||||
Options { approx_chars: 40 }
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,78 +21,12 @@ pub fn zstd_of(text: &str) -> Option<Vec<u8>> {
|
|||
crate::db::repo::encode_one(text, true).expect("zstd encode")
|
||||
}
|
||||
|
||||
/// How old a leftover scratch directory must be before [`sweep_stale`] takes
|
||||
/// it. Far longer than any test run, so a failure investigated the same day —
|
||||
/// or the next morning — still has its tree.
|
||||
const STALE_AFTER: std::time::Duration = std::time::Duration::from_secs(12 * 60 * 60);
|
||||
|
||||
/// Whether `name` is one of [`scratch_dir`]'s own directories.
|
||||
///
|
||||
/// Matched on the *shape* — `quicksearch-{tag}-{pid}-{seq}`, so the last two
|
||||
/// dash-separated components must be numbers — rather than on the
|
||||
/// `quicksearch-` prefix alone. `packaging/capture.sh` keeps its output in
|
||||
/// `quicksearch-capture` in the same directory, and a prefix match would eat a
|
||||
/// capture run's screenshots along with the litter.
|
||||
fn is_scratch_name(name: &str) -> bool {
|
||||
let Some(rest) = name.strip_prefix("quicksearch-") else {
|
||||
return false;
|
||||
};
|
||||
let numeric = |part: Option<&str>| {
|
||||
part.is_some_and(|p| !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit()))
|
||||
};
|
||||
let mut tail = rest.rsplitn(3, '-');
|
||||
// seq, then pid, and a tag must remain in front of them.
|
||||
numeric(tail.next()) && numeric(tail.next()) && tail.next().is_some_and(|tag| !tag.is_empty())
|
||||
}
|
||||
|
||||
/// Remove scratch directories left by runs that are long over.
|
||||
///
|
||||
/// Nothing here cleans up on the way *out*: a failed test's tree is most of
|
||||
/// the evidence, which is why [`scratch_dir`] deliberately leaves it. But
|
||||
/// passing tests leave theirs too, and most never remove it — so the temp
|
||||
/// directory grew by roughly three hundred directories per full run and had
|
||||
/// accumulated some nine thousand of them. Where `/tmp` is a tmpfs that is
|
||||
/// gigabytes of RAM, which slows the whole suite and pushes the
|
||||
/// timing-sensitive tests toward their budgets.
|
||||
///
|
||||
/// Sweeping on the way *in* keeps both halves: this run's evidence survives,
|
||||
/// and so does yesterday's, while nothing accumulates without bound. Only
|
||||
/// [`scratch_dir`]'s own naming is touched.
|
||||
fn sweep_stale() {
|
||||
let Ok(entries) = std::fs::read_dir(std::env::temp_dir()) else {
|
||||
return;
|
||||
};
|
||||
let now = std::time::SystemTime::now();
|
||||
for entry in entries.flatten() {
|
||||
let name = entry.file_name();
|
||||
if !name.to_str().is_some_and(is_scratch_name) {
|
||||
continue;
|
||||
}
|
||||
let stale = entry
|
||||
.metadata()
|
||||
.and_then(|m| m.modified())
|
||||
.ok()
|
||||
.and_then(|t| now.duration_since(t).ok())
|
||||
.is_some_and(|age| age >= STALE_AFTER);
|
||||
if stale {
|
||||
// Best effort throughout: two test binaries starting together race
|
||||
// on the same directory and one of them loses, which is fine.
|
||||
std::fs::remove_dir_all(entry.path()).ok();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A fresh, empty directory under the system temp dir, named for `tag`.
|
||||
///
|
||||
/// Not cleaned up on drop: when a test fails, the tree it built is most of
|
||||
/// the evidence. Long-dead runs' trees are swept once per process instead —
|
||||
/// see [`sweep_stale`]. Panics — a test that cannot create a directory has
|
||||
/// nothing left to assert.
|
||||
/// the evidence. Panics — a test that cannot create a directory has nothing
|
||||
/// left to assert.
|
||||
#[doc(hidden)]
|
||||
pub fn scratch_dir(tag: &str) -> PathBuf {
|
||||
static SWEPT: std::sync::Once = std::sync::Once::new();
|
||||
SWEPT.call_once(sweep_stale);
|
||||
|
||||
let mut p = std::env::temp_dir();
|
||||
p.push(format!(
|
||||
"quicksearch-{}-{}-{}",
|
||||
|
|
@ -161,75 +95,4 @@ mod tests {
|
|||
touch(&deep, b"hi");
|
||||
assert_eq!(std::fs::read(&deep).unwrap(), b"hi");
|
||||
}
|
||||
|
||||
/// The sweep runs against a shared temp directory, so what it matches is
|
||||
/// the whole safety argument. `quicksearch-capture` is the one that would
|
||||
/// hurt: `packaging/capture.sh` puts a run's screenshots and screencasts
|
||||
/// there, and a prefix match would delete them mid-capture.
|
||||
#[test]
|
||||
fn only_scratch_directories_are_swept() {
|
||||
for ours in [
|
||||
"quicksearch-coord-1234-0",
|
||||
"quicksearch-stall-heavy-1001402-7",
|
||||
"quicksearch-a-0-0",
|
||||
// Tags contain dashes of their own; only the last two components
|
||||
// are read as numbers.
|
||||
"quicksearch-sniff-binary-db-2621744-1",
|
||||
] {
|
||||
assert!(is_scratch_name(ours), "{ours} should be swept");
|
||||
}
|
||||
|
||||
for theirs in [
|
||||
// The capture output directory, the reason this is a shape match.
|
||||
"quicksearch-capture",
|
||||
"quicksearch",
|
||||
"quicksearch-",
|
||||
// A tag but no pid/seq pair.
|
||||
"quicksearch-coord",
|
||||
"quicksearch-coord-1234",
|
||||
// Numbers, but nothing in front of them to be a tag.
|
||||
"quicksearch-1234-0",
|
||||
// Not ours at all.
|
||||
"cargo-install-abc-1-2",
|
||||
"tmp-quicksearch-coord-1-2",
|
||||
] {
|
||||
assert!(!is_scratch_name(theirs), "{theirs} must not be swept");
|
||||
}
|
||||
}
|
||||
|
||||
/// Fresh directories survive; only long-dead runs are collected. Uses a
|
||||
/// hand-built name rather than `scratch_dir` so the assertion is about the
|
||||
/// age gate and not about whatever else the suite has left lying around.
|
||||
#[test]
|
||||
fn the_sweep_keeps_recent_trees_and_takes_old_ones() {
|
||||
let fresh = scratch_dir("sweep-fresh");
|
||||
touch(&fresh.join("evidence.txt"), b"kept");
|
||||
|
||||
// Same shape, but back-dated past the threshold. `set_times` is the
|
||||
// only way to age a directory without waiting twelve hours for it.
|
||||
let old = std::env::temp_dir().join(format!(
|
||||
"quicksearch-sweep-old-{}-{}",
|
||||
std::process::id(),
|
||||
NEXT.fetch_add(1, Ordering::Relaxed)
|
||||
));
|
||||
std::fs::create_dir_all(&old).expect("create the aged directory");
|
||||
let long_ago =
|
||||
std::time::SystemTime::now() - STALE_AFTER - std::time::Duration::from_secs(60);
|
||||
std::fs::File::open(&old)
|
||||
.and_then(|d| {
|
||||
d.set_times(
|
||||
std::fs::FileTimes::new()
|
||||
.set_accessed(long_ago)
|
||||
.set_modified(long_ago),
|
||||
)
|
||||
})
|
||||
.expect("back-date the aged directory");
|
||||
|
||||
sweep_stale();
|
||||
|
||||
assert!(fresh.exists(), "a fresh scratch tree was swept away");
|
||||
assert!(!old.exists(), "a long-dead scratch tree survived the sweep");
|
||||
|
||||
std::fs::remove_dir_all(&fresh).ok();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,299 +0,0 @@
|
|||
//! Byte-for-byte verification that a set of files really is identical.
|
||||
//!
|
||||
//! The index groups duplicates by `sha256(size ‖ first hash_length bytes)`
|
||||
//! (see [`crate::file_handling`]), which reads a file's head and nothing else
|
||||
//! — a deliberate trade, since hashing every byte on a disk is most of the
|
||||
//! cost of indexing it. Files of the same size whose heads agree are therefore
|
||||
//! listed as duplicates whether or not they are: a fixed-size VHD keeps what
|
||||
//! makes it unique in a footer, and a freshly pre-allocated disk image is
|
||||
//! zeroes as far as the head can see. This turns that advisory grouping into
|
||||
//! an answer, for the moment before someone deletes something.
|
||||
//!
|
||||
//! No hashing here, by policy. A digest per file would be shorter code and the
|
||||
//! same answer nearly always — but "nearly always" is what the head hash
|
||||
//! already offers, and the whole point of asking a second time is that this
|
||||
//! time the bytes are compared.
|
||||
|
||||
use std::fs::File;
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// Total read-buffer memory, split across the files being compared. A group is
|
||||
/// usually two files and can be thousands — a hardlink farm, which is exactly
|
||||
/// what `[indexing] ignore_patterns` warns about — so a per-file buffer of any
|
||||
/// fixed size would become the largest allocation the process ever makes.
|
||||
const CHUNK_BUDGET: usize = 8 * 1024 * 1024;
|
||||
const MIN_CHUNK: usize = 16 * 1024;
|
||||
const MAX_CHUNK: usize = 256 * 1024;
|
||||
|
||||
/// How often progress is emitted. Each one repaints the UI, and a chunk off a
|
||||
/// warm page cache takes microseconds.
|
||||
const PROGRESS_INTERVAL: Duration = Duration::from_millis(100);
|
||||
|
||||
/// What one member turned out to be, against the reference.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum MemberVerdict {
|
||||
/// Same length, every byte agreed. The reference itself reads this.
|
||||
Identical,
|
||||
/// Offset of the first byte that disagreed.
|
||||
DiffersAt(u64),
|
||||
/// Lengths disagree, so nothing was read. Within a duplicate group this
|
||||
/// can only mean a stale index — the hash covers the size.
|
||||
LengthDiffers { len: u64, reference_len: u64 },
|
||||
/// Could not be opened, or stopped being readable part way through.
|
||||
Unreadable(String),
|
||||
}
|
||||
|
||||
impl MemberVerdict {
|
||||
pub fn is_identical(&self) -> bool {
|
||||
matches!(self, MemberVerdict::Identical)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct VerifyReport {
|
||||
/// Index into the input paths of the file everything else was compared
|
||||
/// against: the first one that opened. `None` when none of them did.
|
||||
pub reference: Option<usize>,
|
||||
/// One verdict per input path, in the input order.
|
||||
pub verdicts: Vec<MemberVerdict>,
|
||||
/// Bytes actually read from disk, across every file.
|
||||
pub bytes_read: u64,
|
||||
}
|
||||
|
||||
impl VerifyReport {
|
||||
/// Whether every member was read and matched. An empty or single-file set
|
||||
/// is vacuously identical.
|
||||
pub fn all_identical(&self) -> bool {
|
||||
self.verdicts.iter().all(MemberVerdict::is_identical)
|
||||
}
|
||||
|
||||
pub fn differing(&self) -> usize {
|
||||
self.verdicts.iter().filter(|v| !v.is_identical()).count()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum VerifyUpdate {
|
||||
Progress { bytes_read: u64, bytes_total: u64 },
|
||||
Done(VerifyReport),
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
/// A file still in the running, with its own read buffer.
|
||||
struct Live {
|
||||
index: usize,
|
||||
file: File,
|
||||
buf: Vec<u8>,
|
||||
}
|
||||
|
||||
/// Compare every path against the first one that opens, byte for byte, and
|
||||
/// report what each turned out to be.
|
||||
///
|
||||
/// Emits `Progress` while it works and exactly one terminal update — `Done`,
|
||||
/// or `Cancelled` if `cancel` went up before the comparison finished.
|
||||
pub fn verify_identical(paths: &[PathBuf], cancel: &AtomicBool, on: &mut dyn FnMut(VerifyUpdate)) {
|
||||
// Checked before the files are even opened, so a run cancelled before it
|
||||
// starts reports the cancellation rather than a verdict nobody waited for.
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
on(VerifyUpdate::Cancelled);
|
||||
return;
|
||||
}
|
||||
let mut verdicts = vec![MemberVerdict::Identical; paths.len()];
|
||||
|
||||
// The reference is the first path that both opens *and* stats, not simply
|
||||
// the first path: one unreadable member must not cost the answer about all
|
||||
// the others.
|
||||
let mut reference: Option<(usize, File, u64)> = None;
|
||||
let mut rest: Vec<(usize, File)> = Vec::new();
|
||||
for (i, path) in paths.iter().enumerate() {
|
||||
let file = match File::open(path) {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
verdicts[i] = MemberVerdict::Unreadable(describe(path, &e));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if reference.is_some() {
|
||||
rest.push((i, file));
|
||||
continue;
|
||||
}
|
||||
match file.metadata() {
|
||||
Ok(m) => reference = Some((i, file, m.len())),
|
||||
Err(e) => verdicts[i] = MemberVerdict::Unreadable(describe(path, &e)),
|
||||
}
|
||||
}
|
||||
|
||||
let Some((reference, mut reference_file, reference_len)) = reference else {
|
||||
on(VerifyUpdate::Done(VerifyReport {
|
||||
reference: None,
|
||||
verdicts,
|
||||
bytes_read: 0,
|
||||
}));
|
||||
return;
|
||||
};
|
||||
|
||||
// A length mismatch is decided from the handles, before a byte is read.
|
||||
let mut live: Vec<Live> = Vec::with_capacity(rest.len());
|
||||
for (i, file) in rest {
|
||||
match file.metadata() {
|
||||
Ok(m) if m.len() != reference_len => {
|
||||
verdicts[i] = MemberVerdict::LengthDiffers {
|
||||
len: m.len(),
|
||||
reference_len,
|
||||
};
|
||||
}
|
||||
Ok(_) => live.push(Live {
|
||||
index: i,
|
||||
file,
|
||||
buf: Vec::new(),
|
||||
}),
|
||||
Err(e) => verdicts[i] = MemberVerdict::Unreadable(describe(&paths[i], &e)),
|
||||
}
|
||||
}
|
||||
|
||||
let chunk = (CHUNK_BUDGET / (live.len() + 1)).clamp(MIN_CHUNK, MAX_CHUNK);
|
||||
let mut reference_buf = vec![0u8; chunk];
|
||||
for l in live.iter_mut() {
|
||||
l.buf = vec![0u8; chunk];
|
||||
}
|
||||
|
||||
let bytes_total = match live.len() {
|
||||
0 => 0,
|
||||
n => reference_len.saturating_mul(n as u64 + 1),
|
||||
};
|
||||
let mut bytes_read = 0u64;
|
||||
let mut offset = 0u64;
|
||||
// Backdated so the first chunk reports: a progress bar that only appears
|
||||
// after the first interval reads as a frozen window on a slow disk, which
|
||||
// is the case this is for.
|
||||
let mut last_progress = Instant::now()
|
||||
.checked_sub(PROGRESS_INTERVAL)
|
||||
.unwrap_or_else(Instant::now);
|
||||
|
||||
while !live.is_empty() {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
on(VerifyUpdate::Cancelled);
|
||||
return;
|
||||
}
|
||||
|
||||
// Termination is driven by what the reference actually reads rather
|
||||
// than by the length it claimed, so a file truncated underneath us
|
||||
// degrades to a short comparison instead of a hang or a false match.
|
||||
let n = match read_chunk(&mut reference_file, &mut reference_buf) {
|
||||
Ok(0) => break, // EOF: everything still live matched all the way.
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
verdicts[reference] = MemberVerdict::Unreadable(describe(&paths[reference], &e));
|
||||
// Survivors agreed up to here but cannot be finished. Saying
|
||||
// so is the only honest answer; "identical" would not be.
|
||||
for l in live.iter() {
|
||||
verdicts[l.index] = MemberVerdict::Unreadable(format!(
|
||||
"compared only to byte {offset}: {} could not be read to the end",
|
||||
paths[reference].display()
|
||||
));
|
||||
}
|
||||
break;
|
||||
}
|
||||
};
|
||||
bytes_read += n as u64;
|
||||
|
||||
let mut i = 0;
|
||||
while i < live.len() {
|
||||
let (got, verdict) = {
|
||||
let l = &mut live[i];
|
||||
match read_chunk(&mut l.file, &mut l.buf[..n]) {
|
||||
Ok(m) => {
|
||||
let common = n.min(m);
|
||||
if let Some(k) =
|
||||
first_difference(&reference_buf[..common], &l.buf[..common])
|
||||
{
|
||||
(m, Some(MemberVerdict::DiffersAt(offset + k as u64)))
|
||||
} else if m < n {
|
||||
// Same length a moment ago, shorter now.
|
||||
(
|
||||
m,
|
||||
Some(MemberVerdict::Unreadable(format!(
|
||||
"{}: ended at byte {} while the file it was compared \
|
||||
against had more",
|
||||
paths[l.index].display(),
|
||||
offset + m as u64
|
||||
))),
|
||||
)
|
||||
} else {
|
||||
(m, None)
|
||||
}
|
||||
}
|
||||
Err(e) => (
|
||||
0,
|
||||
Some(MemberVerdict::Unreadable(describe(&paths[l.index], &e))),
|
||||
),
|
||||
}
|
||||
};
|
||||
bytes_read += got as u64;
|
||||
match verdict {
|
||||
Some(v) => {
|
||||
verdicts[live[i].index] = v;
|
||||
live.swap_remove(i);
|
||||
}
|
||||
None => i += 1,
|
||||
}
|
||||
}
|
||||
offset += n as u64;
|
||||
|
||||
if last_progress.elapsed() >= PROGRESS_INTERVAL {
|
||||
last_progress = Instant::now();
|
||||
on(VerifyUpdate::Progress {
|
||||
bytes_read,
|
||||
bytes_total,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
on(VerifyUpdate::Done(VerifyReport {
|
||||
reference: Some(reference),
|
||||
verdicts,
|
||||
bytes_read,
|
||||
}));
|
||||
}
|
||||
|
||||
/// Fill `buf` as far as the file allows, returning how much. Short reads are
|
||||
/// resumed and `Interrupted` retried, the way `extract::plaintext` does, so a
|
||||
/// short return really does mean end of file.
|
||||
fn read_chunk(f: &mut File, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
let mut filled = 0;
|
||||
while filled < buf.len() {
|
||||
match f.read(&mut buf[filled..]) {
|
||||
Ok(0) => break,
|
||||
Ok(n) => filled += n,
|
||||
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
Ok(filled)
|
||||
}
|
||||
|
||||
/// Offset of the first byte that differs. The equality test comes first
|
||||
/// because it is a `memcmp`; the byte walk only ever runs on the one chunk
|
||||
/// that turned out to differ.
|
||||
fn first_difference(a: &[u8], b: &[u8]) -> Option<usize> {
|
||||
if a == b {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
a.iter()
|
||||
.zip(b.iter())
|
||||
.position(|(x, y)| x != y)
|
||||
.unwrap_or(a.len().min(b.len())),
|
||||
)
|
||||
}
|
||||
|
||||
fn describe(path: &Path, e: &std::io::Error) -> String {
|
||||
format!("{}: {}", path.display(), e)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "verify_tests.rs"]
|
||||
mod tests;
|
||||
|
|
@ -1,281 +0,0 @@
|
|||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use super::*;
|
||||
use crate::testutil::{scratch_dir, touch};
|
||||
|
||||
/// Run to completion with cancellation switched off, returning every update.
|
||||
fn run(paths: &[PathBuf]) -> Vec<VerifyUpdate> {
|
||||
let cancel = AtomicBool::new(false);
|
||||
let mut seen = Vec::new();
|
||||
verify_identical(paths, &cancel, &mut |u| seen.push(u));
|
||||
seen
|
||||
}
|
||||
|
||||
/// The report from a run, asserting it produced exactly one terminal update
|
||||
/// and that the update was `Done`.
|
||||
fn report(paths: &[PathBuf]) -> VerifyReport {
|
||||
let seen = run(paths);
|
||||
let terminal: Vec<&VerifyUpdate> = seen
|
||||
.iter()
|
||||
.filter(|u| !matches!(u, VerifyUpdate::Progress { .. }))
|
||||
.collect();
|
||||
assert_eq!(terminal.len(), 1, "expected one terminal update: {seen:?}");
|
||||
match terminal[0] {
|
||||
VerifyUpdate::Done(r) => r.clone(),
|
||||
other => panic!("expected Done, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// `n` files in a fresh directory, each with the body it is given.
|
||||
fn files(tag: &str, bodies: &[&[u8]]) -> Vec<PathBuf> {
|
||||
let dir = scratch_dir(tag);
|
||||
bodies
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, body)| {
|
||||
let p = dir.join(format!("copy{i}.bin"));
|
||||
touch(&p, body);
|
||||
p
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identical_files_all_report_identical() {
|
||||
for count in [2, 3] {
|
||||
let bodies = vec![&b"the same bytes in every copy"[..]; count];
|
||||
let paths = files("verify-same", &bodies);
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.reference, Some(0));
|
||||
assert!(r.all_identical(), "{r:?}");
|
||||
assert_eq!(r.differing(), 0);
|
||||
// Every file was read through: the head hash alone would not do.
|
||||
assert_eq!(r.bytes_read, 28 * count as u64);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_difference_is_reported_at_its_offset() {
|
||||
// First byte, mid-file, and the very last byte: the last is the one a
|
||||
// head-only hash can never see, and the reason this module exists.
|
||||
for (label, a, b, at) in [
|
||||
("first", &b"Xbcdefgh"[..], &b"abcdefgh"[..], 0),
|
||||
("middle", &b"abcdefgh"[..], &b"abcXefgh"[..], 3),
|
||||
("last", &b"abcdefgh"[..], &b"abcdefgX"[..], 7),
|
||||
] {
|
||||
let paths = files("verify-diff", &[a, b]);
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.verdicts[0], MemberVerdict::Identical, "{label}");
|
||||
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(at), "{label}");
|
||||
assert!(!r.all_identical(), "{label}");
|
||||
assert_eq!(r.differing(), 1, "{label}");
|
||||
}
|
||||
}
|
||||
|
||||
/// The realistic false positive: same size, same head, different tail — a
|
||||
/// pre-allocated disk image, which is what `[processing] hash_length`
|
||||
/// documents as the known limitation.
|
||||
#[test]
|
||||
fn a_shared_head_with_a_different_tail_is_caught() {
|
||||
let head = vec![0u8; 64 * 1024];
|
||||
let mut a = head.clone();
|
||||
let mut b = head;
|
||||
a.extend_from_slice(b"footer-a");
|
||||
b.extend_from_slice(b"footer-b");
|
||||
let paths = files("verify-tail", &[a.as_slice(), b.as_slice()]);
|
||||
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(64 * 1024 + 7));
|
||||
}
|
||||
|
||||
/// Bigger than one chunk, so the multi-chunk path and the running offset are
|
||||
/// both exercised rather than assumed.
|
||||
#[test]
|
||||
fn a_difference_past_the_first_chunk_is_found() {
|
||||
// Two files, so the chunk is the 256 KiB ceiling and the difference sits
|
||||
// in the second one.
|
||||
let size = MAX_CHUNK * 2 + 1234;
|
||||
let a = vec![7u8; size];
|
||||
let mut b = a.clone();
|
||||
let at = MAX_CHUNK + 500;
|
||||
b[at] = 8;
|
||||
let paths = files("verify-chunks", &[a.as_slice(), b.as_slice()]);
|
||||
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.verdicts[1], MemberVerdict::DiffersAt(at as u64));
|
||||
// Two chunks out of each file and then it stops: with nothing left to
|
||||
// compare against, reading the remainder of the reference would be work
|
||||
// that cannot change the answer.
|
||||
assert_eq!(
|
||||
r.bytes_read,
|
||||
4 * MAX_CHUNK as u64,
|
||||
"did not stop once the last live file dropped out"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn different_lengths_are_decided_without_reading() {
|
||||
let paths = files("verify-len", &[b"abcdefgh", b"abcdefghij"]);
|
||||
let r = report(&paths);
|
||||
assert_eq!(
|
||||
r.verdicts[1],
|
||||
MemberVerdict::LengthDiffers {
|
||||
len: 10,
|
||||
reference_len: 8
|
||||
}
|
||||
);
|
||||
assert_eq!(r.bytes_read, 0, "a length mismatch reads nothing");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_files_are_identical() {
|
||||
let paths = files("verify-empty", &[b"", b""]);
|
||||
let r = report(&paths);
|
||||
assert!(r.all_identical(), "{r:?}");
|
||||
assert_eq!(r.bytes_read, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_member_is_unreadable_and_the_rest_still_compare() {
|
||||
let mut paths = files("verify-missing", &[b"same", b"same"]);
|
||||
paths.insert(1, PathBuf::from("/nonexistent/quicksearch-verify-missing"));
|
||||
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.reference, Some(0));
|
||||
assert!(matches!(r.verdicts[1], MemberVerdict::Unreadable(_)));
|
||||
assert_eq!(
|
||||
r.verdicts[2],
|
||||
MemberVerdict::Identical,
|
||||
"an unreadable member stopped the others being compared"
|
||||
);
|
||||
assert_eq!(r.differing(), 1);
|
||||
}
|
||||
|
||||
/// The first path is the obvious reference, but not a required one: an
|
||||
/// unreadable first member must not sink the whole run.
|
||||
#[test]
|
||||
fn the_reference_falls_through_to_the_first_readable_member() {
|
||||
let mut paths = files("verify-refmissing", &[b"same", b"same"]);
|
||||
paths.insert(0, PathBuf::from("/nonexistent/quicksearch-verify-ref"));
|
||||
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.reference, Some(1));
|
||||
assert!(matches!(r.verdicts[0], MemberVerdict::Unreadable(_)));
|
||||
assert_eq!(r.verdicts[1], MemberVerdict::Identical);
|
||||
assert_eq!(r.verdicts[2], MemberVerdict::Identical);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nothing_readable_reports_no_reference() {
|
||||
let paths = vec![
|
||||
PathBuf::from("/nonexistent/quicksearch-verify-a"),
|
||||
PathBuf::from("/nonexistent/quicksearch-verify-b"),
|
||||
];
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.reference, None);
|
||||
assert_eq!(r.verdicts.len(), 2);
|
||||
assert!(r.verdicts.iter().all(|v| !v.is_identical()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_single_file_and_an_empty_set_are_vacuously_identical() {
|
||||
let paths = files("verify-one", &[b"alone"]);
|
||||
let r = report(&paths);
|
||||
assert_eq!(r.reference, Some(0));
|
||||
assert!(r.all_identical());
|
||||
assert_eq!(r.bytes_read, 0, "nothing to compare it against");
|
||||
|
||||
let r = report(&[]);
|
||||
assert_eq!(r.reference, None);
|
||||
assert!(r.verdicts.is_empty());
|
||||
assert!(r.all_identical());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_run_cancelled_before_it_starts_reports_only_that() {
|
||||
let paths = files("verify-cancel", &[b"same", b"same"]);
|
||||
let cancel = AtomicBool::new(true);
|
||||
let mut seen = Vec::new();
|
||||
verify_identical(&paths, &cancel, &mut |u| seen.push(u));
|
||||
assert_eq!(seen, vec![VerifyUpdate::Cancelled]);
|
||||
}
|
||||
|
||||
/// Cancelling part way through ends the run there, with no `Done` claiming a
|
||||
/// verdict it never reached. The first chunk always reports progress, so the
|
||||
/// flag goes up between two chunks rather than at a time the test has to race
|
||||
/// for.
|
||||
#[test]
|
||||
fn cancelling_mid_run_ends_it_without_a_verdict() {
|
||||
let body = vec![3u8; MAX_CHUNK * 4];
|
||||
let paths = files("verify-cancel-mid", &[body.as_slice(), body.as_slice()]);
|
||||
|
||||
let cancel = AtomicBool::new(false);
|
||||
let mut seen = Vec::new();
|
||||
verify_identical(&paths, &cancel, &mut |u| {
|
||||
cancel.store(true, Ordering::Relaxed);
|
||||
seen.push(u);
|
||||
});
|
||||
assert!(
|
||||
matches!(seen.first(), Some(VerifyUpdate::Progress { .. })),
|
||||
"the first chunk did not report progress: {seen:?}"
|
||||
);
|
||||
assert_eq!(seen.last(), Some(&VerifyUpdate::Cancelled), "{seen:?}");
|
||||
assert!(
|
||||
!seen.iter().any(|u| matches!(u, VerifyUpdate::Done(_))),
|
||||
"a cancelled run still reported a verdict: {seen:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Whatever progress reports, it is a fraction that makes sense: monotonic,
|
||||
/// and never past its own denominator.
|
||||
#[test]
|
||||
fn progress_climbs_and_never_overruns_its_denominator() {
|
||||
let body = vec![9u8; MAX_CHUNK * 6];
|
||||
let paths = files(
|
||||
"verify-progress",
|
||||
&[body.as_slice(), body.as_slice(), body.as_slice()],
|
||||
);
|
||||
|
||||
let mut last = 0;
|
||||
let mut count = 0;
|
||||
for update in run(&paths) {
|
||||
if let VerifyUpdate::Progress {
|
||||
bytes_read,
|
||||
bytes_total,
|
||||
} = update
|
||||
{
|
||||
assert!(
|
||||
bytes_read <= bytes_total,
|
||||
"{bytes_read} read of {bytes_total}"
|
||||
);
|
||||
assert!(bytes_read >= last, "progress went backwards");
|
||||
last = bytes_read;
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
assert!(count > 0, "a six-chunk comparison reported no progress");
|
||||
}
|
||||
|
||||
/// A directory is not a file this can compare, however the platform refuses
|
||||
/// it — `File::open` fails outright on Windows, while on Linux it opens and
|
||||
/// then refuses to be read. Either way it is that member's problem, not the
|
||||
/// run's.
|
||||
#[test]
|
||||
fn an_unreadable_member_does_not_stop_the_run() {
|
||||
let dir = scratch_dir("verify-dir");
|
||||
let a = dir.join("a.bin");
|
||||
let b = dir.join("b.bin");
|
||||
touch(&a, b"identical bytes");
|
||||
touch(&b, b"identical bytes");
|
||||
let sub = dir.join("subdir");
|
||||
std::fs::create_dir_all(&sub).unwrap();
|
||||
|
||||
let r = report(&[a, sub, b]);
|
||||
assert_eq!(r.reference, Some(0));
|
||||
assert!(
|
||||
!r.verdicts[1].is_identical(),
|
||||
"a directory was called an identical file"
|
||||
);
|
||||
assert_eq!(r.verdicts[2], MemberVerdict::Identical);
|
||||
}
|
||||
|
|
@ -29,6 +29,7 @@ use crate::file_handling::{
|
|||
classify_by_mtime, classify_for_indexing, path_to_db_string, prepare_file_record,
|
||||
warn_if_unrepresentable, DirRows, FileIndexAction, OwnedNewFile, UnreadableDirs,
|
||||
};
|
||||
use crate::indexing::should_abort;
|
||||
|
||||
mod pool;
|
||||
#[cfg(test)]
|
||||
|
|
@ -169,6 +170,7 @@ struct Ctx {
|
|||
registry: Arc<Registry>,
|
||||
unreadable: UnreadableDirs,
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
suspend_flag: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
/// Individual unreadable-directory warnings allowed per run before only the
|
||||
|
|
@ -420,7 +422,7 @@ fn prepare(file: PendingFile, known: Known<'_>, ctx: &Ctx) -> WalkedFile {
|
|||
fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
|
||||
while let Some((job, slot)) = shared.take() {
|
||||
let _busy = shared.stats.enter();
|
||||
if ctx.stop_flag.load(Ordering::Relaxed) {
|
||||
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
|
||||
shared.shutdown();
|
||||
return;
|
||||
}
|
||||
|
|
@ -457,7 +459,7 @@ fn worker(shared: &Shared, ctx: &Ctx, tx: &mpsc::SyncSender<WalkEvent>) {
|
|||
}
|
||||
|
||||
for file in files {
|
||||
if ctx.stop_flag.load(Ordering::Relaxed) {
|
||||
if should_abort(&ctx.stop_flag, &ctx.suspend_flag) {
|
||||
shared.shutdown();
|
||||
return;
|
||||
}
|
||||
|
|
@ -701,6 +703,7 @@ pub fn walk_indexable_files(
|
|||
config: Config,
|
||||
registry: Arc<Registry>,
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
suspend_flag: Arc<AtomicBool>,
|
||||
workers: usize,
|
||||
) -> ParallelWalk {
|
||||
let mut queue = Queue::default();
|
||||
|
|
@ -743,6 +746,7 @@ pub fn walk_indexable_files(
|
|||
registry,
|
||||
unreadable: UnreadableDirs::default(),
|
||||
stop_flag,
|
||||
suspend_flag,
|
||||
});
|
||||
|
||||
for root in unresolvable {
|
||||
|
|
|
|||
|
|
@ -16,8 +16,9 @@ fn db_with(tag: &str, rows: &[(String, u64)]) -> PathBuf {
|
|||
for (path, mtime) in rows {
|
||||
let as_path = Path::new(path);
|
||||
conn.execute(
|
||||
"INSERT INTO files (name, path, parent, size, mtime, type, content_state)
|
||||
VALUES (?1, ?2, ?3, 0, ?4, 0, 3)",
|
||||
"INSERT INTO files (name, path, parent, size, mtime, type, \
|
||||
basic_state, content_state)
|
||||
VALUES (?1, ?2, ?3, 0, ?4, 0, 1, 3)",
|
||||
rusqlite::params![
|
||||
as_path.file_name().unwrap().to_string_lossy(),
|
||||
path,
|
||||
|
|
@ -53,6 +54,7 @@ fn walk_with(
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
))
|
||||
}
|
||||
|
|
@ -302,6 +304,7 @@ fn unreadable_directory_is_recorded_not_silently_empty() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
let files: Vec<WalkedFile> = w
|
||||
|
|
@ -459,6 +462,7 @@ fn hidden_and_ignored_entries_are_pruned() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
));
|
||||
assert_eq!(names(&files), vec!["keep.txt", "keep2.txt"]);
|
||||
|
|
@ -504,6 +508,7 @@ fn pruned_entries_are_counted_by_reason() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
let files: Vec<WalkedFile> = (&mut walk)
|
||||
|
|
@ -552,6 +557,7 @@ fn a_tree_with_nothing_pruned_reports_no_summary() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
let files: Vec<WalkedFile> = (&mut walk)
|
||||
|
|
@ -597,6 +603,7 @@ fn a_directory_reports_rows_with_no_file_behind_them() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
));
|
||||
stale.sort();
|
||||
|
|
@ -636,6 +643,7 @@ fn an_unreadable_directory_reports_nothing_stale() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
));
|
||||
fs::set_permissions(&locked, fs::Permissions::from_mode(0o755)).ok();
|
||||
|
|
@ -676,6 +684,7 @@ fn stop_flag_ends_the_walk_without_hanging() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
stop,
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
));
|
||||
|
||||
|
|
@ -704,6 +713,7 @@ fn dropping_the_walk_early_does_not_hang() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
assert!(w.next().is_some());
|
||||
|
|
@ -729,6 +739,7 @@ fn overlapping_roots_yield_each_file_once() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
));
|
||||
|
||||
|
|
@ -776,6 +787,7 @@ fn finish_reports_a_clean_walk_and_is_idempotent() {
|
|||
Config::default(),
|
||||
Arc::new(Registry::default_set()),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
Arc::new(AtomicBool::new(false)),
|
||||
4,
|
||||
);
|
||||
let files: Vec<WalkedFile> = w
|
||||
|
|
|
|||
|
|
@ -9,9 +9,7 @@ use quicksearch_core::db::open_or_recreate;
|
|||
use quicksearch_core::db::repo::{insert_file, set_content_done, NewFile};
|
||||
use quicksearch_core::mime::FileType;
|
||||
use quicksearch_core::query::split::split_for_cascade;
|
||||
use quicksearch_core::search::{
|
||||
cascade, MatchField, SearchHit, SearchOptions, SearchService, SearchUpdate,
|
||||
};
|
||||
use quicksearch_core::search::{cascade, SearchHit, SearchOptions, SearchService, SearchUpdate};
|
||||
use quicksearch_core::testutil::zstd_of;
|
||||
|
||||
mod common;
|
||||
|
|
@ -42,6 +40,8 @@ impl Seeder {
|
|||
parent: dir,
|
||||
size: 42,
|
||||
mtime,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -52,7 +52,7 @@ impl Seeder {
|
|||
.expect("unique path");
|
||||
if let Some(text) = text {
|
||||
let zstd = self.store_text.then(|| zstd_of(text)).flatten();
|
||||
set_content_done(&tx, id, text, zstd.as_deref()).unwrap();
|
||||
set_content_done(&tx, id, name, text, &[], zstd.as_deref()).unwrap();
|
||||
}
|
||||
tx.commit().unwrap();
|
||||
id
|
||||
|
|
@ -355,39 +355,6 @@ fn fuzzy_max_edits_widens_and_narrows_the_budget() {
|
|||
std::fs::remove_file(&p).ok();
|
||||
}
|
||||
|
||||
/// Regression: the fuzzy filename tier stamps stage 7 rather than truncating
|
||||
/// its own rank. `7.0 + 0.1 * distance` reaches 8.0 at ten edits — the fuzzy
|
||||
/// *full-text* tier — and every frontend reads `match_field()`, so a filename
|
||||
/// hit would have been rendered as a match on the file's contents.
|
||||
#[test]
|
||||
fn a_distant_fuzzy_filename_hit_stays_a_name_hit() {
|
||||
let p = tmp_db("fuzzystage");
|
||||
let mut s = Seeder::new(&p, true);
|
||||
// Ten substitutions against a 30-character term, whose budget is ten.
|
||||
let far = s.add("abcdefghijklmnopqrst##########", "/d", 1, None);
|
||||
let conn = s.done();
|
||||
|
||||
let (hits, _) = run_collect(
|
||||
&conn,
|
||||
"abcdefghijklmnopqrstuvwxyz0123",
|
||||
&fuzzy_options_with_edits(10),
|
||||
);
|
||||
assert_eq!(
|
||||
hits.iter().map(|h| h.file_id).collect::<Vec<_>>(),
|
||||
vec![far]
|
||||
);
|
||||
assert!(
|
||||
(hits[0].rank - 8.0).abs() < 1e-9,
|
||||
"rank {} is not the 8.0 that used to truncate into the next stage",
|
||||
hits[0].rank
|
||||
);
|
||||
assert_eq!(hits[0].stage, 7, "the name tier is stage 7 at any distance");
|
||||
assert_eq!(hits[0].match_field(), MatchField::Name);
|
||||
|
||||
drop(conn);
|
||||
std::fs::remove_file(&p).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dedup_keeps_best_rank() {
|
||||
let p = tmp_db("dedup");
|
||||
|
|
@ -1433,138 +1400,3 @@ fn streaming_does_not_change_the_result_set() {
|
|||
assert_eq!(outcome.total, 3, "batch size {}", batch);
|
||||
}
|
||||
}
|
||||
|
||||
/// `SearchHit::snippet` is documented as "the filename for name stages, the
|
||||
/// full path for path stages", and a frontend relies on it to highlight the
|
||||
/// match inside the Name or Path column it is already painting: the ranges
|
||||
/// index that field, so they only line up if the window *is* that field.
|
||||
///
|
||||
/// The fuzzy tiers used to window it instead, which silently broke the
|
||||
/// contract whenever the match landed past two thirds of the way through a
|
||||
/// long name — the ranges then indexed a suffix, and a column that trusted
|
||||
/// them would mark the wrong glyphs.
|
||||
#[test]
|
||||
fn fuzzy_name_and_path_snippets_carry_the_whole_field() {
|
||||
let p = tmp_db("fuzzy-whole-field");
|
||||
let mut s = Seeder::new(&p, true);
|
||||
// The match sits in the last third of the name, which is what used to
|
||||
// push the window's start off zero.
|
||||
s.add(
|
||||
"a_long_and_deliberately_padded_out_quarterly_repot.txt",
|
||||
"/home/me/documents/archive",
|
||||
1,
|
||||
None,
|
||||
);
|
||||
let conn = s.done();
|
||||
|
||||
let (hits, _) = run_collect(&conn, "report", &fuzzy_options());
|
||||
let hit = hits
|
||||
.iter()
|
||||
.find(|h| h.stage == 7)
|
||||
.expect("a fuzzy filename hit");
|
||||
|
||||
let snip = hit.snippet.as_ref().expect("a name hit carries a snippet");
|
||||
assert_eq!(snip.window, hit.name, "the window is not the whole name");
|
||||
assert!(!snip.truncated_start, "the name was windowed");
|
||||
assert!(!snip.truncated_end, "the name was windowed");
|
||||
for &(a, b) in &snip.ranges {
|
||||
assert!(b <= hit.name.len(), "range {a}..{b} runs past the name");
|
||||
assert!(
|
||||
hit.name.is_char_boundary(a) && hit.name.is_char_boundary(b),
|
||||
"range {a}..{b} is not on char boundaries"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A fuzzy tier's marks have to cover the *matched* text and nothing else.
|
||||
///
|
||||
/// The bug this pins: bitap reports where a match ends, and the range took
|
||||
/// its start to be `end - term.len()`, which is only right when the match
|
||||
/// happens to be as long as the term. Searching `repot` marked `1Repo` inside
|
||||
/// `1Reporter` — one byte too far left, over a character that matched nothing.
|
||||
///
|
||||
/// Asserted on the *text* rather than on offsets, so it reads as the symptom
|
||||
/// and survives the fixture being reworded.
|
||||
#[test]
|
||||
fn a_fuzzy_mark_covers_the_matched_text_and_nothing_else() {
|
||||
let p = tmp_db("fuzzy-mark-span");
|
||||
let mut s = Seeder::new(&p, true);
|
||||
// The leading digit is the point: it is what the old range reached back
|
||||
// over. In the body too, for the full-text tier.
|
||||
s.add("1Reporter.txt", "/home/me/docs", 1, None);
|
||||
s.add(
|
||||
"body.txt",
|
||||
"/home/me/docs",
|
||||
2,
|
||||
Some("filed under 1Reporter last week"),
|
||||
);
|
||||
let conn = s.done();
|
||||
|
||||
let (hits, _) = run_collect(&conn, "repot", &fuzzy_options());
|
||||
|
||||
// Stage 7 — fuzzy filename, marked inside the whole name.
|
||||
let name_hit = hits
|
||||
.iter()
|
||||
.find(|h| h.stage == 7)
|
||||
.expect("a fuzzy filename hit");
|
||||
let snip = name_hit.snippet.as_ref().expect("name tiers carry one");
|
||||
let (a, b) = snip.ranges[0];
|
||||
assert_eq!(
|
||||
&snip.window[a..b],
|
||||
"Repo",
|
||||
"the mark is {:?}; it must cover the match and not the leading digit",
|
||||
&snip.window[a..b]
|
||||
);
|
||||
|
||||
// Stage 8 — fuzzy full text, marked inside the snippet window.
|
||||
let body_hit = hits
|
||||
.iter()
|
||||
.find(|h| h.stage == 8)
|
||||
.expect("a fuzzy full-text hit");
|
||||
let snip = body_hit.snippet.as_ref().expect("content tiers carry one");
|
||||
let (a, b) = snip.ranges[0];
|
||||
assert_eq!(
|
||||
&snip.window[a..b],
|
||||
"Repo",
|
||||
"the mark is {:?}; it must cover the match and not the leading digit",
|
||||
&snip.window[a..b]
|
||||
);
|
||||
}
|
||||
|
||||
/// The other half of the same defect: the automaton accepts the moment a
|
||||
/// leading part of the term has matched, paying for the term's tail with
|
||||
/// deletions — so the mark stopped short of the text that actually matched.
|
||||
///
|
||||
/// `quarterly` against a body holding `quartrly` accepts after `quartrl`,
|
||||
/// spending both edits on the missing `y` and the dropped `e`. One byte
|
||||
/// further is a *better* alignment (one edit) covering the whole word, which
|
||||
/// is what a reader expects to see lit up.
|
||||
#[test]
|
||||
fn a_fuzzy_mark_is_not_truncated_to_a_leading_part_of_the_term() {
|
||||
let p = tmp_db("fuzzy-mark-full");
|
||||
let mut s = Seeder::new(&p, true);
|
||||
// The name must not match at all: a row the filename tier claims never
|
||||
// reaches the full-text tier. And the body must hold a fuzzy *variant* —
|
||||
// the term verbatim would be an exact content match, stage 5 or 6.
|
||||
s.add(
|
||||
"notes.txt",
|
||||
"/home/me/docs",
|
||||
1,
|
||||
Some("the quartrly budget was revised"),
|
||||
);
|
||||
let conn = s.done();
|
||||
|
||||
let (hits, _) = run_collect(&conn, "quarterly", &fuzzy_options());
|
||||
let hit = hits
|
||||
.iter()
|
||||
.find(|h| h.stage == 8)
|
||||
.expect("a fuzzy full-text hit");
|
||||
let snip = hit.snippet.as_ref().expect("content tiers carry one");
|
||||
let (a, b) = snip.ranges[0];
|
||||
assert_eq!(
|
||||
&snip.window[a..b],
|
||||
"quartrly",
|
||||
"the mark is {:?}, a leading part of what matched",
|
||||
&snip.window[a..b]
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -91,11 +91,11 @@ fn encrypted_index_lifecycle() {
|
|||
{
|
||||
let conn = db::open_existing(&db_path.to_string_lossy(), true).unwrap();
|
||||
conn.execute_batch(
|
||||
"INSERT INTO files (name, path, parent, size, mtime, type, content_state)
|
||||
"INSERT INTO files (name, path, parent, size, mtime, type, basic_state, content_state)
|
||||
WITH RECURSIVE n(i) AS (
|
||||
SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 20000
|
||||
)
|
||||
SELECT 'p' || i, '/pad/' || i, '/pad', 0, 0, 0, 3 FROM n;
|
||||
SELECT 'p' || i, '/pad/' || i, '/pad', 0, 0, 0, 1, 3 FROM n;
|
||||
DELETE FROM files WHERE parent = '/pad';",
|
||||
)
|
||||
.unwrap();
|
||||
|
|
|
|||
|
|
@ -11,26 +11,12 @@ use std::sync::{Arc, Mutex};
|
|||
use std::time::{Duration, Instant, SystemTime};
|
||||
|
||||
use quicksearch_core::config::Config;
|
||||
use quicksearch_core::file_handling::{
|
||||
count_extract_scope, mark_oversize_pending_na, ExtractCursor, ExtractScope,
|
||||
};
|
||||
use quicksearch_core::file_handling::{extract_scope_prepare, ExtractCursor};
|
||||
use quicksearch_core::indexing::{IndexingService, IndexingStatus, RootPhase};
|
||||
|
||||
mod common;
|
||||
use common::{scratch_dir as tmp_dir, touch};
|
||||
|
||||
/// The removed `extract_scope_prepare`: the oversize sweep the writer still
|
||||
/// does, then the count the content pass now does on its own connection.
|
||||
fn extract_scope_prepare(
|
||||
conn_mutex: &Arc<Mutex<rusqlite::Connection>>,
|
||||
cursor: &ExtractCursor,
|
||||
config: &Config,
|
||||
) -> Result<ExtractScope, String> {
|
||||
let conn = conn_mutex.lock().unwrap();
|
||||
mark_oversize_pending_na(&conn, cursor, config).unwrap();
|
||||
count_extract_scope(&conn, cursor, config)
|
||||
}
|
||||
|
||||
/// Run one full index over `root` and wait for it to finish.
|
||||
fn index_once(root: &Path, db: &Path, config: &Config) {
|
||||
common::IndexOnce {
|
||||
|
|
@ -749,22 +735,26 @@ fn a_directory_that_becomes_unreadable_deletes_nothing() {
|
|||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Everything about a file's indexed content that a user can observe: its
|
||||
/// state, its failure reason, and the compressed size of its stored body.
|
||||
type ContentRow = (String, i64, Option<String>, Option<i64>);
|
||||
/// state, the stored snippet body, and its property rows.
|
||||
type ContentRow = (String, i64, Option<String>, Option<i64>, String);
|
||||
|
||||
fn content_rows(db: &Path) -> Vec<ContentRow> {
|
||||
let conn = rusqlite::Connection::open(db).unwrap();
|
||||
let mut stmt = conn
|
||||
.prepare(
|
||||
"SELECT f.path, f.content_state, ff.reason, LENGTH(d.text_zstd)
|
||||
"SELECT f.path, f.content_state, f.failure_msg, d.text_len,
|
||||
COALESCE(GROUP_CONCAT(p.key || '=' || p.value, ','), '')
|
||||
FROM files f
|
||||
LEFT JOIN documents_text d ON d.file_id = f.id
|
||||
LEFT JOIN failed_files ff ON ff.file_id = f.id
|
||||
LEFT JOIN properties p ON p.file_id = f.id
|
||||
GROUP BY f.id
|
||||
ORDER BY f.path",
|
||||
)
|
||||
.unwrap();
|
||||
let out = stmt
|
||||
.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)))
|
||||
.query_map([], |r| {
|
||||
Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?, r.get(4)?))
|
||||
})
|
||||
.unwrap()
|
||||
.map(|r| r.unwrap())
|
||||
.collect();
|
||||
|
|
@ -898,9 +888,7 @@ fn undecodable_small_files_are_reported_as_failures_not_silently_skipped() {
|
|||
let conn = rusqlite::Connection::open(&db).unwrap();
|
||||
let (state, msg): (i64, Option<String>) = conn
|
||||
.query_row(
|
||||
"SELECT f.content_state, ff.reason FROM files f \
|
||||
LEFT JOIN failed_files ff ON ff.file_id = f.id \
|
||||
WHERE f.path LIKE '%bad.txt'",
|
||||
"SELECT content_state, failure_msg FROM files WHERE path LIKE '%bad.txt'",
|
||||
[],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
|
|
@ -941,9 +929,7 @@ fn an_unreadable_legacy_office_file_fails_with_a_reason() {
|
|||
let conn = rusqlite::Connection::open(&db).unwrap();
|
||||
let (state, msg): (i64, Option<String>) = conn
|
||||
.query_row(
|
||||
"SELECT f.content_state, ff.reason FROM files f \
|
||||
LEFT JOIN failed_files ff ON ff.file_id = f.id \
|
||||
WHERE f.path LIKE '%broken.doc'",
|
||||
"SELECT content_state, failure_msg FROM files WHERE path LIKE '%broken.doc'",
|
||||
[],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
|
|
@ -1244,154 +1230,6 @@ fn contentless_mode_still_indexes_inlined_files_without_storing_bodies() {
|
|||
std::fs::remove_dir_all(&db_dir).ok();
|
||||
}
|
||||
|
||||
/// What one watch of a heavy/light overlap saw; see [`observe_overlap`].
|
||||
struct Overlap {
|
||||
/// Light files drained and heavy rows stored, across the window in which
|
||||
/// the heavy root extracted while the light root walked.
|
||||
light_drained: usize,
|
||||
heavy_stored: usize,
|
||||
/// The heavy root's `extract_total` and pool size, for the fixture guards.
|
||||
heavy_pending: usize,
|
||||
heavy_pool: usize,
|
||||
/// Publications in which the heavy root's row count moved — how many
|
||||
/// separate writer rounds its rows arrived over, counted by the counter
|
||||
/// changing rather than by polls (the watcher polls far faster than the
|
||||
/// writer publishes, so poll count would say nothing).
|
||||
///
|
||||
/// Reported, never asserted on. It reads as granularity but it is really
|
||||
/// `min(writer rounds, watcher polls)`: at a 1 ms slice the writer
|
||||
/// published faster than the 500 µs poll could see and twelve rows read
|
||||
/// as two steps. Coverage (`heavy_stored`) and the ratio are the verdict;
|
||||
/// this is here to make a surprising run legible.
|
||||
heavy_steps: usize,
|
||||
/// Longest this watcher itself went between polls. A short window has two
|
||||
/// very different causes — a writer that gulped the pass in one turn, or a
|
||||
/// watcher that was descheduled past it — and on a loaded two-core box the
|
||||
/// second is real. Without this the two are indistinguishable in a failure.
|
||||
worst_gap: Duration,
|
||||
}
|
||||
|
||||
/// Watch a two-root run until the heavy root has finished extracting and
|
||||
/// report how the two counters moved while both were in flight, then stop
|
||||
/// the run. Removing the fixture is the caller's.
|
||||
///
|
||||
/// Deltas across the overlap, never durations. Sparse samples cost only the
|
||||
/// window's edges, and they trim both counters together. Panics if the window
|
||||
/// never opened — a fixture that does not exercise the case proves nothing.
|
||||
///
|
||||
/// The window **opens** on the first published snapshot holding both roots in
|
||||
/// flight, and **closes when the heavy root leaves `Extracting`** — not when
|
||||
/// the light root finishes walking. Closing it with the light walk is what CI
|
||||
/// caught: it makes the measurement depend on a race between the light root's
|
||||
/// per-file rate and the heavy root's bandwidth, two things that keep no fixed
|
||||
/// ratio across hosts. A starved runner ran the light root's whole 16,000-file
|
||||
/// walk while one heavy row landed, and the pass the deltas were supposed to
|
||||
/// describe was 1/12th sampled. Ending with the heavy root's own pass makes
|
||||
/// the measured interval one unit of work — that pass, all of it, whatever the
|
||||
/// light root does meanwhile. If the light walk ends early its counter simply
|
||||
/// stops, which understates the interleaving and can never overstate it.
|
||||
///
|
||||
/// What can only be seen here is what the writer *published*, once a round
|
||||
/// (`publish_status` in `indexing/pipeline.rs`). A caller whose heavy root
|
||||
/// finishes its content pass inside one round leaves no snapshot holding both
|
||||
/// phases at once and lands on the panic below however healthy the run was —
|
||||
/// which is why the callers set `writer_turn_slice_ms` small enough that a
|
||||
/// round is far shorter than the pass, and why the panic prints what the
|
||||
/// phases actually did.
|
||||
fn observe_overlap(service: &IndexingService, heavy_tag: &str, light_tag: &str) -> Overlap {
|
||||
let mut opened: Option<(usize, usize)> = None; // (light.walked, heavy.extracted)
|
||||
let mut last = (0usize, 0usize);
|
||||
let mut heavy_pending = 0usize;
|
||||
let mut heavy_pool = 0usize;
|
||||
let mut heavy_steps = 0usize;
|
||||
let mut stepped_at = 0usize;
|
||||
let mut worst_gap = Duration::ZERO;
|
||||
let mut polled_at = Instant::now();
|
||||
// Every (heavy, light) phase pair published, in order and without repeats.
|
||||
// Only the diagnosis uses it: both phases are monotone, so this is at most
|
||||
// a handful of entries and it says exactly which phase went missing.
|
||||
let mut phases: Vec<(RootPhase, RootPhase)> = Vec::new();
|
||||
let deadline = Instant::now() + Duration::from_secs(120);
|
||||
while Instant::now() < deadline {
|
||||
let mut in_window = false;
|
||||
worst_gap = worst_gap.max(polled_at.elapsed());
|
||||
polled_at = Instant::now();
|
||||
match service.get_status() {
|
||||
IndexingStatus::Running { roots, .. } => {
|
||||
let heavy_p = roots.iter().find(|r| r.root.contains(heavy_tag));
|
||||
let light_p = roots.iter().find(|r| r.root.contains(light_tag));
|
||||
if let (Some(h), Some(l)) = (heavy_p, light_p) {
|
||||
if phases.last() != Some(&(h.phase, l.phase)) {
|
||||
phases.push((h.phase, l.phase));
|
||||
}
|
||||
// Opening takes both roots in flight; staying open takes
|
||||
// only the heavy root's pass, which is the work the deltas
|
||||
// describe. See this function's docs for why the light
|
||||
// root's walk is not allowed to end the measurement.
|
||||
in_window = if opened.is_none() {
|
||||
h.phase == RootPhase::Extracting && l.phase == RootPhase::Walking
|
||||
} else {
|
||||
h.phase == RootPhase::Extracting
|
||||
};
|
||||
if in_window {
|
||||
last = (l.walked, h.extracted);
|
||||
if opened.is_none() {
|
||||
opened = Some(last);
|
||||
stepped_at = h.extracted;
|
||||
}
|
||||
// A fresh publication, not a fresh poll: the row count
|
||||
// only moves when the writer has finished a round with
|
||||
// rows in it.
|
||||
if h.extracted > stepped_at {
|
||||
stepped_at = h.extracted;
|
||||
heavy_steps += 1;
|
||||
}
|
||||
if let Some(total) = h.extract_total {
|
||||
heavy_pending = total;
|
||||
}
|
||||
heavy_pool = h.total_workers;
|
||||
}
|
||||
}
|
||||
}
|
||||
// The run is claimed but has not reached its walk yet; there is
|
||||
// nothing to sample, and breaking here would end the watch before
|
||||
// the run it is watching had started.
|
||||
IndexingStatus::Preparing { .. } => {}
|
||||
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
||||
_ => break,
|
||||
}
|
||||
// Both phases are monotone, so a closed window will not reopen.
|
||||
if opened.is_some() && !in_window {
|
||||
break;
|
||||
}
|
||||
// Finer than a writer round, or the window's edges are set by this
|
||||
// loop instead of by the phase it is watching. One mutex and a small
|
||||
// clone per poll, so 2000/s costs the run nothing measurable.
|
||||
std::thread::sleep(Duration::from_micros(500));
|
||||
}
|
||||
service.stop_indexing().unwrap();
|
||||
|
||||
let Some((light_open, heavy_open)) = opened else {
|
||||
panic!(
|
||||
"never observed the heavy root extracting while the light root walked. \
|
||||
Published (heavy, light) phases: {:?}. An empty list means neither \
|
||||
root matched the tags {:?}/{:?}; a list with no heavy Extracting in \
|
||||
it means the heavy root's content pass began and ended between two \
|
||||
status publications, so lower `writer_turn_slice_ms` until a writer \
|
||||
round is shorter than that pass.",
|
||||
phases, heavy_tag, light_tag
|
||||
);
|
||||
};
|
||||
Overlap {
|
||||
light_drained: last.0 - light_open,
|
||||
heavy_stored: last.1 - heavy_open,
|
||||
heavy_pending,
|
||||
heavy_pool,
|
||||
heavy_steps,
|
||||
worst_gap,
|
||||
}
|
||||
}
|
||||
|
||||
/// A slow root must not stall the others.
|
||||
///
|
||||
/// This is the complaint stated directly: one root doing heavy extraction used
|
||||
|
|
@ -1403,329 +1241,120 @@ fn observe_overlap(service: &IndexingService, heavy_tag: &str, light_tag: &str)
|
|||
/// wrong measure: writing is serial by construction (one SQLite connection),
|
||||
/// so on a local disk the writer, not extraction, is the bottleneck and a
|
||||
/// wall-clock comparison would mostly measure the machine.
|
||||
///
|
||||
/// A stall is therefore counted in *work*, not in milliseconds: while the heavy
|
||||
/// root extracts, how many files the light root's walk was drained of, against
|
||||
/// how many rows the heavy root's extraction landed. Both counters are advanced
|
||||
/// by the same writer loop, each root's turn bounded by one slice
|
||||
/// (`service_walking`, `service_extracting` in `indexing/pipeline.rs`), so their
|
||||
/// ratio *is* the interleaving.
|
||||
///
|
||||
/// - Serialised — the regression — the writer reads the heavy batch itself and
|
||||
/// drains nobody meanwhile. Whatever shape that takes it obeys
|
||||
/// `light < heavy + quantum`: one quantum of each per round is the most a
|
||||
/// single thread taking turns can manage. Its own time budget says the same
|
||||
/// from the other side, since time spent reading is time not spent inserting.
|
||||
/// - As built, extraction is off on the root's own pool and the writer's turn
|
||||
/// for the heavy root is a store and nothing more, so the light root is
|
||||
/// drained at the writer's full rate throughout — on this fixture several
|
||||
/// times the bound.
|
||||
///
|
||||
/// Counting rather than timing is what makes the verdict the same on a loaded
|
||||
/// CI runner and an idle workstation. Every way a host can be slow — a
|
||||
/// preempted writer, a checkpoint, a long round — freezes *both* counters, and
|
||||
/// cancels. The wall-clock figure this replaced did not cancel: the same
|
||||
/// correct behaviour measured ~20 ms here and 188 ms on the CI runner, which is
|
||||
/// *more* than the 130 ms the broken design measured here. At that point CI was
|
||||
/// overriding the budget six-fold and the check had stopped telling the two
|
||||
/// designs apart. A bound that has to be calibrated per host is not an
|
||||
/// assertion.
|
||||
#[test]
|
||||
fn a_heavy_root_does_not_stall_a_light_one() {
|
||||
// The writer's round-robin quantum. Set here rather than inherited from the
|
||||
// default 500 because the bound below is arithmetic in it, and because a
|
||||
// 500-file round is a coarse enough publish interval to look like a stall
|
||||
// on a slow host all by itself.
|
||||
const QUANTUM: usize = 16;
|
||||
// HEAVY: few files, each big enough that reading one is real work, with a
|
||||
// HEAVY: few files, each big enough that reading it is real work, with a
|
||||
// small `maximum_text_size` so the cost lands in extraction rather than in
|
||||
// the writer's tokenising. Few and large rather than many and small: the
|
||||
// bound the light root must beat, `3 × (rows + quantum)`, grows with the
|
||||
// row count, while what it drains does not.
|
||||
//
|
||||
// Total bytes are the runtime, and this is one starved thread reading them
|
||||
// — a loaded two-core runner has measured under 2 MB/s for exactly this
|
||||
// work. So the fixture is sized for the *guards*, not for margin: 24 MB is
|
||||
// enough that the rows arrive over separate writer rounds (`heavy_steps`
|
||||
// below) and cheap enough that a bad runner still finishes in seconds. The
|
||||
// margin is a rate ratio and needs no help — CI has measured it in the
|
||||
// hundreds.
|
||||
const HEAVY_FILES: usize = 12;
|
||||
// LIGHT: a wide tree of tiny files, so its counter moves finely. Each is
|
||||
// inlined by its walk worker, so this root has no extraction phase of its
|
||||
// own to confuse the window with.
|
||||
//
|
||||
// It no longer has to outlast the heavy root's pass — the window ends with
|
||||
// that pass, and a light walk that finishes first just stops contributing.
|
||||
// What it does have to do is still be walking when the pass *starts*, and
|
||||
// supply more than `3 × (rows + quantum)` files before it ends.
|
||||
const LIGHT_FILES: usize = 16_000;
|
||||
// Light files drained per (heavy row + quantum). Three times a bound the
|
||||
// serialised design provably cannot reach: with turns bounded by rows
|
||||
// rather than time it managed one quantum of each per round, or 1x. The
|
||||
// built one floors at `QUANTUM`:1 and measures well above that.
|
||||
const MIN_INTERLEAVE: usize = 3;
|
||||
// Fewest heavy rows a window has to contain for the ratio to be evidence.
|
||||
// From the bound itself: a window of `n` rows drains `QUANTUM × n` light
|
||||
// files at the floor and must beat `MIN_INTERLEAVE × (n + QUANTUM)`, so
|
||||
// `n ≥ MIN_INTERLEAVE × QUANTUM / (QUANTUM - MIN_INTERLEAVE)` — under four
|
||||
// rows the constant term decides the comparison instead of the design.
|
||||
const MIN_ROWS_SAMPLED: usize = 4;
|
||||
|
||||
// the writer's tokenising.
|
||||
let heavy = tmp_dir("stall-heavy");
|
||||
// 36 bytes a repeat, so just under 2 MiB: twelve of them is 24 MB of
|
||||
// fixture against the 92 MB this used to build, and a quarter of the
|
||||
// reading for the runner to get through.
|
||||
let body: Vec<u8> = "sphinx of black quartz judge my vow "
|
||||
.repeat(58_000)
|
||||
.repeat(40_000)
|
||||
.into_bytes();
|
||||
for i in 0..HEAVY_FILES {
|
||||
touch(&heavy.join(format!("d{}/big{:04}.txt", i % 4, i)), &body);
|
||||
for i in 0..200 {
|
||||
touch(&heavy.join(format!("d{}/big{:04}.txt", i % 8, i)), &body);
|
||||
}
|
||||
// LIGHT: a wide tree of tiny files, so its walk runs long enough to sample
|
||||
// and its progress counter moves finely.
|
||||
let light = tmp_dir("stall-light");
|
||||
for i in 0..LIGHT_FILES {
|
||||
for i in 0..6000 {
|
||||
touch(&light.join(format!("d{}/f{:05}.txt", i % 60, i)), b"x");
|
||||
}
|
||||
|
||||
let db_dir = tmp_dir("stall-db");
|
||||
let db = db_dir.join("index.sqlite");
|
||||
let roots = vec![
|
||||
heavy.to_string_lossy().into_owned(),
|
||||
light.to_string_lossy().into_owned(),
|
||||
];
|
||||
|
||||
let mut config = test_config();
|
||||
config.processing.maximum_text_size = 1024;
|
||||
// Above the heavy files, or `mark_oversize_pending_na` writes them off as
|
||||
// N/A before the pass starts and there is no extraction phase at all.
|
||||
config.processing.maximum_text_file_size = 4 * 1024 * 1024;
|
||||
config.processing.batch_size = QUANTUM;
|
||||
// Zero, which is what makes this test's verdict arithmetic rather than a
|
||||
// measurement of the host. It is the whole answer to two CI failures that
|
||||
// were both really the same thing: a bound in files-per-second compared
|
||||
// against one in bytes-per-second, on a container that slows the first and
|
||||
// not the second.
|
||||
//
|
||||
// With no time in a turn, a writer round is exactly one bounded piece of
|
||||
// work per root. `service_walking` runs one `batch_size` quantum and then
|
||||
// meets its already-expired deadline; `store_extracted` consumes exactly
|
||||
// one row ("the deadline is checked after every row... at least one row is
|
||||
// always consumed"). So the round, not the second, is the unit, and the
|
||||
// interleave floor is `quantum : 1` — 16:1 here — by construction on any
|
||||
// host. Load can only raise it: a slow reader means rounds where the heavy
|
||||
// root has nothing ready and the light root drains anyway.
|
||||
//
|
||||
// It also makes the ratio *uniform across the pass*, which is what lets the
|
||||
// sample below be a partial one. This watcher is one thread among the
|
||||
// suite's on a two-core runner and can be descheduled through a chunk of a
|
||||
// 40 ms pass; when every round contributes the same ratio, the part it does
|
||||
// see answers the same question as the whole.
|
||||
config.processing.writer_turn_slice_ms = 0;
|
||||
// One extraction thread for the heavy root, so its pass costs about what
|
||||
// the broken design's inline read would and the two differ only in *which*
|
||||
// thread pays for it. `root_workers` is keyed by the `indexing_paths`
|
||||
// spelling; both sides canonicalize before matching.
|
||||
config.paths.indexing_paths = roots.clone();
|
||||
config.indexing.root_workers.insert(roots[0].clone(), 1);
|
||||
// The default WAL cap is far above anything this run writes, so no forced
|
||||
// checkpoint lands inside the window. That stops being true if the fixture
|
||||
// ever grows by an order of magnitude.
|
||||
config.processing.maximum_text_file_size = 8 * 1024 * 1024;
|
||||
|
||||
let service = IndexingService::new();
|
||||
service
|
||||
.start_indexing(roots, db.to_string_lossy().into_owned(), config.clone())
|
||||
.unwrap();
|
||||
let seen = observe_overlap(&service, "stall-heavy", "stall-light");
|
||||
drop(service);
|
||||
|
||||
// Before the assertions, unlike the rest of this file: those tests keep
|
||||
// their trees because a failing test's tree is the evidence, but this
|
||||
// fixture is generated and identical every run, and its evidence is the two
|
||||
// counters printed below. Leaving 92 MB of it in a RAM-backed /tmp behind a
|
||||
// failure is itself a reason for the next run to fail.
|
||||
std::fs::remove_dir_all(&heavy).ok();
|
||||
std::fs::remove_dir_all(&light).ok();
|
||||
std::fs::remove_dir_all(&db_dir).ok();
|
||||
|
||||
// The fixture is as configured. Each of these silently costs a factor of
|
||||
// the margin below if it stops holding, so they are checked before the
|
||||
// ratio is read as a verdict on the design.
|
||||
assert_eq!(
|
||||
seen.heavy_pool, 1,
|
||||
"the heavy root must extract on the single worker root_workers asked for; \
|
||||
with the default four its pass is four times shorter and so is the margin"
|
||||
);
|
||||
assert_eq!(
|
||||
seen.heavy_pending, HEAVY_FILES,
|
||||
"every heavy file must reach the content pass; one inlined by its walk \
|
||||
worker never produces an extraction phase to overlap with"
|
||||
);
|
||||
// Four rows, not half of them. With a zero slice every round contributes
|
||||
// the same `quantum : 1`, so the window is allowed to be a sub-sample of
|
||||
// the pass — it answers the same question either way, and both counters
|
||||
// are trimmed by the same edge. What it cannot be is degenerate: below
|
||||
// four rows the bound's `+ QUANTUM` term dominates and a passing ratio
|
||||
// would be arithmetic rather than evidence.
|
||||
assert!(
|
||||
seen.heavy_stored >= MIN_ROWS_SAMPLED,
|
||||
"only {} of {} heavy rows landed inside the observed window, fewer than \
|
||||
the {} a verdict needs (worst watcher gap {:?}, rows seen over {} \
|
||||
rounds) — a gap near the pass's own length means this watcher was \
|
||||
descheduled past it, not that the writer gulped it",
|
||||
seen.heavy_stored,
|
||||
HEAVY_FILES,
|
||||
MIN_ROWS_SAMPLED,
|
||||
seen.worst_gap,
|
||||
seen.heavy_steps
|
||||
);
|
||||
|
||||
eprintln!(
|
||||
"light files drained while the heavy root extracted: {} against {} heavy \
|
||||
rows (quantum {}) landing over {} rounds, worst watcher gap {:?} — \
|
||||
{}x the {}x required; the \
|
||||
serialised design cannot exceed 1x",
|
||||
seen.light_drained,
|
||||
seen.heavy_stored,
|
||||
QUANTUM,
|
||||
seen.heavy_steps,
|
||||
seen.worst_gap,
|
||||
seen.light_drained / (seen.heavy_stored + QUANTUM),
|
||||
MIN_INTERLEAVE
|
||||
);
|
||||
assert!(
|
||||
seen.light_drained >= MIN_INTERLEAVE * (seen.heavy_stored + QUANTUM),
|
||||
"the light root was drained of only {} files while the heavy root landed \
|
||||
{} rows; one quantum of each per round is all a writer that extracts \
|
||||
inline can manage, so anything near {} means the extraction is back on \
|
||||
the writer thread",
|
||||
seen.light_drained,
|
||||
seen.heavy_stored,
|
||||
seen.heavy_stored + QUANTUM
|
||||
);
|
||||
}
|
||||
|
||||
/// The sibling of [`a_heavy_root_does_not_stall_a_light_one`] for the cost that
|
||||
/// test deliberately keeps small: the writer's own tokenising.
|
||||
///
|
||||
/// There the heavy files are expensive to *read* and cheap to *write*
|
||||
/// (`maximum_text_size = 1024`), so it never exercised the writer. Here each
|
||||
/// heavy row carries the default 256 KiB of text and its FTS5 trigram insert is
|
||||
/// the expensive step — and it runs on the writer thread, inside the
|
||||
/// transaction, where nothing can take it off. Four workers keep the ready
|
||||
/// channel full, so what one turn finds waiting is a whole channel of them.
|
||||
///
|
||||
/// Before turns had a slice, an extraction turn wrote everything it found —
|
||||
/// half a second to two seconds of tokenising — and the light root's walk got
|
||||
/// one quantum in between: the ratio below came in under one. With turns
|
||||
/// bounded by `writer_turn_slice_ms` and walks served first, the light root
|
||||
/// drains at
|
||||
/// its own rate while the heavy root lands a row or two per round.
|
||||
#[test]
|
||||
fn a_heavy_root_does_not_stall_a_light_one_at_the_writer() {
|
||||
const QUANTUM: usize = 16;
|
||||
// Over the walk's inline threshold, and enough that the stored text is the
|
||||
// full `maximum_text_size` (256 KiB) — the tokenising is what is measured.
|
||||
const HEAVY_FILES: usize = 32;
|
||||
// Wider than the sibling's: with the walk no longer waiting on the writer
|
||||
// it drains so fast that 6000 files were gone before half the heavy rows
|
||||
// had landed, and the window closed on a sample too short to trust.
|
||||
const LIGHT_FILES: usize = 16_000;
|
||||
// As in the sibling: three times a bound the unsliced writer cannot reach.
|
||||
const MIN_INTERLEAVE: usize = 3;
|
||||
// Fewest heavy rows a window has to contain for the ratio to be evidence.
|
||||
// From the bound itself: a window of `n` rows drains `QUANTUM × n` light
|
||||
// files at the floor and must beat `MIN_INTERLEAVE × (n + QUANTUM)`, so
|
||||
// `n ≥ MIN_INTERLEAVE × QUANTUM / (QUANTUM - MIN_INTERLEAVE)` — under four
|
||||
// rows the constant term decides the comparison instead of the design.
|
||||
const MIN_ROWS_SAMPLED: usize = 4;
|
||||
|
||||
let heavy = tmp_dir("stall-writer-heavy");
|
||||
let body: Vec<u8> = "sphinx of black quartz judge my vow "
|
||||
.repeat(9_000)
|
||||
.into_bytes();
|
||||
for i in 0..HEAVY_FILES {
|
||||
touch(&heavy.join(format!("d{}/big{:04}.txt", i % 8, i)), &body);
|
||||
}
|
||||
let light = tmp_dir("stall-writer-light");
|
||||
for i in 0..LIGHT_FILES {
|
||||
touch(&light.join(format!("d{}/f{:05}.txt", i % 60, i)), b"x");
|
||||
}
|
||||
|
||||
let db_dir = tmp_dir("stall-writer-db");
|
||||
let db = db_dir.join("index.sqlite");
|
||||
let roots = vec![
|
||||
.start_indexing(
|
||||
vec![
|
||||
heavy.to_string_lossy().into_owned(),
|
||||
light.to_string_lossy().into_owned(),
|
||||
];
|
||||
|
||||
let mut config = test_config();
|
||||
config.processing.batch_size = QUANTUM;
|
||||
// As in the sibling, and for the same reason: at zero the round is the
|
||||
// unit of measurement and the interleave floor is `quantum : 1` whatever
|
||||
// the host does. This one's pass is long on its own account — the writer
|
||||
// tokenises 256 KiB a row — but nothing in the fixture guarantees that on
|
||||
// a host whose FTS5 is quicker than this one's.
|
||||
config.processing.writer_turn_slice_ms = 0;
|
||||
config.paths.indexing_paths = roots.clone();
|
||||
// Four readers, so the heavy rows reach the writer faster than it can
|
||||
// tokenise them and the ready channel is full when its turn comes.
|
||||
config.indexing.root_workers.insert(roots[0].clone(), 4);
|
||||
|
||||
let service = IndexingService::new();
|
||||
service
|
||||
.start_indexing(roots, db.to_string_lossy().into_owned(), config.clone())
|
||||
],
|
||||
db.to_string_lossy().into_owned(),
|
||||
config.clone(),
|
||||
)
|
||||
.unwrap();
|
||||
let seen = observe_overlap(&service, "stall-writer-heavy", "stall-writer-light");
|
||||
|
||||
// Sample the light root's progress while the heavy one is extracting, and
|
||||
// keep the longest interval over which it did not move.
|
||||
let mut worst = Duration::ZERO;
|
||||
let mut last_change = Instant::now();
|
||||
let mut last_seen = 0usize;
|
||||
let mut sampled_together = false;
|
||||
let deadline = Instant::now() + Duration::from_secs(120);
|
||||
while Instant::now() < deadline {
|
||||
match service.get_status() {
|
||||
IndexingStatus::Running { roots, .. } => {
|
||||
let heavy_p = roots.iter().find(|r| r.root.contains("stall-heavy"));
|
||||
let light_p = roots.iter().find(|r| r.root.contains("stall-light"));
|
||||
if let (Some(h), Some(l)) = (heavy_p, light_p) {
|
||||
let light_busy = l.phase != RootPhase::Done;
|
||||
if h.phase == RootPhase::Extracting && light_busy {
|
||||
sampled_together = true;
|
||||
let now = l.walked + l.extracted;
|
||||
if now != last_seen {
|
||||
last_seen = now;
|
||||
last_change = Instant::now();
|
||||
} else {
|
||||
worst = worst.max(last_change.elapsed());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// The run is claimed but has not reached its walk yet; there is
|
||||
// nothing to sample, and breaking here would end the watch before
|
||||
// the run it is watching had started.
|
||||
IndexingStatus::Preparing { .. } => {}
|
||||
IndexingStatus::Error(e) => panic!("indexing failed: {}", e),
|
||||
_ => break,
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
service.stop_indexing().unwrap();
|
||||
drop(service);
|
||||
|
||||
assert!(
|
||||
sampled_together,
|
||||
"never observed the two roots overlapping; the fixture is not exercising the case"
|
||||
);
|
||||
// Measured on this fixture: ~20 ms with the extraction pools, ~130 ms when
|
||||
// the file reading is forced back onto the writer thread (and unbounded in
|
||||
// the real failure, where the heavy root is on a network share). The bound
|
||||
// sits between, with several times the observed headroom.
|
||||
//
|
||||
// The fixed design's stall does not grow with the heavy root's cost — it is
|
||||
// one round-robin pass plus one commit — so making that root heavier only
|
||||
// widens the margin.
|
||||
// The figures above are wall-clock, so they scale with the host: a small CI
|
||||
// VM running the other tests in this binary alongside this one measures
|
||||
// several times the developer-machine number without the design having
|
||||
// changed at all. QSB_STALL_BUDGET_MS lets that environment say so out loud
|
||||
// instead of the bound being quietly loosened for everyone. Keep any override
|
||||
// well under the broken design's figure scaled by the same factor, or the
|
||||
// test stops discriminating between the two.
|
||||
let budget = Duration::from_millis(
|
||||
std::env::var("QSB_STALL_BUDGET_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(100),
|
||||
);
|
||||
eprintln!(
|
||||
"longest light-root stall while heavy extracted: {:?} (budget {:?})",
|
||||
worst, budget
|
||||
);
|
||||
assert!(
|
||||
worst < budget,
|
||||
"the light root stalled for {:?} while the heavy root extracted (budget {:?})",
|
||||
worst,
|
||||
budget
|
||||
);
|
||||
|
||||
std::fs::remove_dir_all(&heavy).ok();
|
||||
std::fs::remove_dir_all(&light).ok();
|
||||
std::fs::remove_dir_all(&db_dir).ok();
|
||||
|
||||
assert_eq!(
|
||||
seen.heavy_pool, 4,
|
||||
"the heavy root must extract on four workers"
|
||||
);
|
||||
assert_eq!(
|
||||
seen.heavy_pending, HEAVY_FILES,
|
||||
"every heavy file must reach the content pass"
|
||||
);
|
||||
// As in the sibling: a partial window answers the same question when every
|
||||
// round contributes the same ratio, so this asks only that it was not
|
||||
// degenerate.
|
||||
assert!(
|
||||
seen.heavy_stored >= MIN_ROWS_SAMPLED,
|
||||
"only {} of {} heavy rows landed inside the observed window, fewer than \
|
||||
the {} a verdict needs (worst watcher gap {:?}, rows seen over {} \
|
||||
rounds) — a gap near the pass's own length means this watcher was \
|
||||
descheduled past it, not that the writer gulped it",
|
||||
seen.heavy_stored,
|
||||
HEAVY_FILES,
|
||||
MIN_ROWS_SAMPLED,
|
||||
seen.worst_gap,
|
||||
seen.heavy_steps
|
||||
);
|
||||
|
||||
eprintln!(
|
||||
"light files drained while the heavy root tokenised: {} against {} heavy \
|
||||
rows (quantum {}) landing over {} rounds, worst watcher gap {:?} — \
|
||||
{}x the {}x required",
|
||||
seen.light_drained,
|
||||
seen.heavy_stored,
|
||||
QUANTUM,
|
||||
seen.heavy_steps,
|
||||
seen.worst_gap,
|
||||
seen.light_drained / (seen.heavy_stored + QUANTUM),
|
||||
MIN_INTERLEAVE
|
||||
);
|
||||
assert!(
|
||||
seen.light_drained >= MIN_INTERLEAVE * (seen.heavy_stored + QUANTUM),
|
||||
"the light root was drained of only {} files while the heavy root landed \
|
||||
{} rows; an extraction turn is writing to the end of its batch again \
|
||||
instead of yielding at its slice",
|
||||
seen.light_drained,
|
||||
seen.heavy_stored
|
||||
);
|
||||
}
|
||||
|
||||
/// The write-ahead log must not grow for the length of a run.
|
||||
|
|
|
|||
|
|
@ -110,6 +110,7 @@ fn orphans(db: &Path) -> i64 {
|
|||
[
|
||||
("searchabletext", "rowid"),
|
||||
("documents_text", "file_id"),
|
||||
("properties", "file_id"),
|
||||
("failed_files", "file_id"),
|
||||
]
|
||||
.iter()
|
||||
|
|
|
|||
|
|
@ -124,6 +124,8 @@ fn seed(path: &std::path::Path) {
|
|||
parent: &dir,
|
||||
size: 4096,
|
||||
mtime: 1_700_000_000 + i as u64,
|
||||
inode: None,
|
||||
device_id: None,
|
||||
mime: Some("text/plain"),
|
||||
ftype: FileType::TEXT,
|
||||
hash: None,
|
||||
|
|
@ -137,7 +139,7 @@ fn seed(path: &std::path::Path) {
|
|||
.map(|_| WORDS[(rng.next() as usize) % WORDS.len()])
|
||||
.collect();
|
||||
let body = body.join(" ");
|
||||
set_content_done(&tx, id, &body, zstd_of(&body).as_deref()).unwrap();
|
||||
set_content_done(&tx, id, &name, &body, &[], zstd_of(&body).as_deref()).unwrap();
|
||||
}
|
||||
}
|
||||
tx.commit().unwrap();
|
||||
|
|
|
|||
|
|
@ -157,14 +157,15 @@ fn snippet_paths_perf_comparison() {
|
|||
// measured against the same DB layout production runs against.
|
||||
conn.execute_batch(
|
||||
"CREATE VIRTUAL TABLE st_contentless USING fts5(
|
||||
text,
|
||||
name, text,
|
||||
tokenize='trigram remove_diacritics 1',
|
||||
content='',
|
||||
contentless_delete=1
|
||||
);
|
||||
CREATE TABLE documents_text (
|
||||
file_id INTEGER PRIMARY KEY,
|
||||
text_zstd BLOB NOT NULL
|
||||
text_zstd BLOB NOT NULL,
|
||||
text_len INTEGER NOT NULL
|
||||
);",
|
||||
)
|
||||
.unwrap();
|
||||
|
|
@ -179,19 +180,23 @@ fn snippet_paths_perf_comparison() {
|
|||
.prepare("INSERT INTO st_regular(rowid, name, text) VALUES (?1, ?2, ?3)")
|
||||
.unwrap();
|
||||
let mut ins_con = tx
|
||||
.prepare("INSERT INTO st_contentless(rowid, text) VALUES (?1, ?2)")
|
||||
.prepare("INSERT INTO st_contentless(rowid, name, text) VALUES (?1, ?2, ?3)")
|
||||
.unwrap();
|
||||
let mut ins_blob = tx
|
||||
.prepare("INSERT INTO documents_text(file_id, text_zstd) VALUES (?1, ?2)")
|
||||
.prepare(
|
||||
"INSERT INTO documents_text(file_id, text_zstd, text_len) VALUES (?1, ?2, ?3)",
|
||||
)
|
||||
.unwrap();
|
||||
for i in 1..=NUM_DOCS {
|
||||
let target = 50 + ((rng as usize) % 400);
|
||||
let text = seed_text(&mut rng, target);
|
||||
let name = format!("doc_{:05}.txt", i);
|
||||
ins_reg.execute(params![i as i64, &name, &text]).unwrap();
|
||||
ins_con.execute(params![i as i64, &text]).unwrap();
|
||||
ins_con.execute(params![i as i64, &name, &text]).unwrap();
|
||||
let compressed = zstd::encode_all(text.as_bytes(), 3).unwrap();
|
||||
ins_blob.execute(params![i as i64, &compressed]).unwrap();
|
||||
ins_blob
|
||||
.execute(params![i as i64, &compressed, text.len() as i64])
|
||||
.unwrap();
|
||||
}
|
||||
}
|
||||
tx.commit().expect("seed commit");
|
||||
|
|
@ -284,8 +289,7 @@ fn snippet_paths_perf_comparison() {
|
|||
}
|
||||
None => String::new(),
|
||||
};
|
||||
let folded = text.to_ascii_lowercase();
|
||||
let _snip = snippet::extract_folded(&text, &folded, &[q], &opts);
|
||||
let _snip = snippet::extract(&text, &[q], &opts);
|
||||
rows_b_total += 1;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -43,19 +43,8 @@ eframe = { version = "0.32", default-features = false, features = [
|
|||
"glow",
|
||||
"persistence",
|
||||
] }
|
||||
# `default-features = false` for one feature, `default_fonts`, which embeds
|
||||
# four TTFs totalling 1,407,752 bytes - 38% of `.rodata`. The
|
||||
# `default-features = false` on eframe above does not cover it: feature
|
||||
# resolution unions the two declarations, so naming egui plainly here switched
|
||||
# it back on for the whole graph. `src/fonts.rs` installs the two faces the app
|
||||
# actually paints with; the two emoji faces, 736,668 bytes of it, are gone.
|
||||
# `persistence`/`serde` still reach egui through eframe, so nothing else moves.
|
||||
egui = { version = "0.32", default-features = false }
|
||||
# Only `Column`/`TableBuilder` are used. The lone default feature is
|
||||
# `dep:mime_guess2`, for an image loader that is never registered - the linker
|
||||
# already dropped every byte of it, so this compiles one crate less and ships
|
||||
# the same bytes.
|
||||
egui_extras = { version = "0.32", default-features = false }
|
||||
egui = "0.32"
|
||||
egui_extras = "0.32"
|
||||
rfd = "0.15"
|
||||
open = "5"
|
||||
chrono = { version = "0.4", default-features = false, features = ["clock"] }
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -1,31 +0,0 @@
|
|||
The work in the Hack project is Copyright 2018 Source Foundry Authors and licensed under the MIT License
|
||||
|
||||
The work in the DejaVu project was committed to the public domain.
|
||||
|
||||
Bitstream Vera Sans Mono Copyright 2003 Bitstream Inc. and licensed under the Bitstream Vera License with Reserved Font Names "Bitstream" and "Vera"
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2018 Source Foundry Authors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
BITSTREAM VERA LICENSE
|
||||
|
||||
Copyright (c) 2003 by Bitstream, Inc. All Rights Reserved. Bitstream Vera is a trademark of Bitstream, Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of the fonts accompanying this license ("Fonts") and associated documentation files (the "Font Software"), to reproduce and distribute the Font Software, including without limitation the rights to use, copy, merge, publish, distribute, and/or sell copies of the Font Software, and to permit persons to whom the Font Software is furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright and trademark notices and this permission notice shall be included in all copies of one or more of the Font Software typefaces.
|
||||
|
||||
The Font Software may be modified, altered, or added to, and in particular the designs of glyphs or characters in the Fonts may be modified and additional glyphs or characters may be added to the Fonts, only if the fonts are renamed to names not containing either the words "Bitstream" or the word "Vera".
|
||||
|
||||
This License becomes null and void to the extent applicable to Fonts or Font Software that has been modified and is distributed under the "Bitstream Vera" names.
|
||||
|
||||
The Font Software may be sold as part of a larger software package but no copy of one or more of the Font Software typefaces may be sold by itself.
|
||||
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL BITSTREAM OR THE GNOME FOUNDATION BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
|
||||
Except as contained in this notice, the names of Gnome, the Gnome Foundation, and Bitstream Inc., shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Font Software without prior written authorization from the Gnome Foundation or Bitstream Inc., respectively. For further information, contact: fonts at gnome dot org.
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
-------------------------------
|
||||
UBUNTU FONT LICENCE Version 1.0
|
||||
-------------------------------
|
||||
|
||||
PREAMBLE
|
||||
This licence allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely. The fonts, including any derivative works, can be
|
||||
bundled, embedded, and redistributed provided the terms of this licence
|
||||
are met. The fonts and derivatives, however, cannot be released under
|
||||
any other licence. The requirement for fonts to remain under this
|
||||
licence does not require any document created using the fonts or their
|
||||
derivatives to be published under this licence, as long as the primary
|
||||
purpose of the document is not to be a vehicle for the distribution of
|
||||
the fonts.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this licence and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Original Version" refers to the collection of Font Software components
|
||||
as received under this licence.
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to
|
||||
a new environment.
|
||||
|
||||
"Copyright Holder(s)" refers to all individuals and companies who have a
|
||||
copyright ownership of the Font Software.
|
||||
|
||||
"Substantially Changed" refers to Modified Versions which can be easily
|
||||
identified as dissimilar to the Font Software by users of the Font
|
||||
Software comparing the Original Version with the Modified Version.
|
||||
|
||||
To "Propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification and with or without charging
|
||||
a redistribution fee), making available to the public, and in some
|
||||
countries other activities as well.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
This licence does not grant any rights under trademark law and all such
|
||||
rights are reserved.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a
|
||||
copy of the Font Software, to propagate the Font Software, subject to
|
||||
the below conditions:
|
||||
|
||||
1) Each copy of the Font Software must contain the above copyright
|
||||
notice and this licence. These can be included either as stand-alone
|
||||
text files, human-readable headers or in the appropriate machine-
|
||||
readable metadata fields within text or binary files as long as those
|
||||
fields can be easily viewed by the user.
|
||||
|
||||
2) The font name complies with the following:
|
||||
(a) The Original Version must retain its name, unmodified.
|
||||
(b) Modified Versions which are Substantially Changed must be renamed to
|
||||
avoid use of the name of the Original Version or similar names entirely.
|
||||
(c) Modified Versions which are not Substantially Changed must be
|
||||
renamed to both (i) retain the name of the Original Version and (ii) add
|
||||
additional naming elements to distinguish the Modified Version from the
|
||||
Original Version. The name of such Modified Versions must be the name of
|
||||
the Original Version, with "derivative X" where X represents the name of
|
||||
the new work, appended to that name.
|
||||
|
||||
3) The name(s) of the Copyright Holder(s) and any contributor to the
|
||||
Font Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except (i) as required by this licence, (ii) to
|
||||
acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with
|
||||
their explicit written permission.
|
||||
|
||||
4) The Font Software, modified or unmodified, in part or in whole, must
|
||||
be distributed entirely under this licence, and must not be distributed
|
||||
under any other licence. The requirement for fonts to remain under this
|
||||
licence does not affect any document created using the Font Software,
|
||||
except any version of the Font Software extracted from a document
|
||||
created using the Font Software may only be distributed under this
|
||||
licence.
|
||||
|
||||
TERMINATION
|
||||
This licence becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
|
||||
COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER
|
||||
DEALINGS IN THE FONT SOFTWARE.
|
||||
Binary file not shown.
|
|
@ -1,7 +1,6 @@
|
|||
//! Application shell: tab strip, per-frame event drains, debounce,
|
||||
//! status bar, and config-change routing.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::mpsc;
|
||||
use std::time::Duration;
|
||||
|
||||
|
|
@ -23,8 +22,8 @@ use crate::format::{fmt_interval, group_thousands};
|
|||
use crate::keychain;
|
||||
use crate::logs_tab::LogsTab;
|
||||
use crate::manage_tab::ManageTab;
|
||||
use crate::options::{OptionsWindow, SecurityAction};
|
||||
use crate::search_tab::SearchTab;
|
||||
use crate::settings_tab::{SecurityAction, SettingsTab};
|
||||
use crate::unlock::KeySource;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
|
|
@ -34,17 +33,6 @@ pub(crate) enum Tab {
|
|||
Duplicates,
|
||||
Logs,
|
||||
Help,
|
||||
Settings,
|
||||
}
|
||||
|
||||
/// The editor a tab holds, if it stages its edits on a draft rather than
|
||||
/// saving them the moment they change.
|
||||
fn tab_editor(tab: Tab) -> Option<UnsavedSource> {
|
||||
match tab {
|
||||
Tab::Manage => Some(UnsavedSource::Manage),
|
||||
Tab::Settings => Some(UnsavedSource::Settings),
|
||||
Tab::Search | Tab::Duplicates | Tab::Logs | Tab::Help => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// A navigation the unsaved-changes guard put on hold; once nothing relevant
|
||||
|
|
@ -52,6 +40,7 @@ fn tab_editor(tab: Tab) -> Option<UnsavedSource> {
|
|||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum NavIntent {
|
||||
SwitchTab(Tab),
|
||||
CloseOptions,
|
||||
Quit,
|
||||
}
|
||||
|
||||
|
|
@ -59,30 +48,21 @@ enum NavIntent {
|
|||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum UnsavedSource {
|
||||
Manage,
|
||||
Settings,
|
||||
Options,
|
||||
}
|
||||
|
||||
/// The editor `from` holds, if it has one and it is holding unapplied edits.
|
||||
fn dirty_editor(from: Tab, manage_dirty: bool, settings_dirty: bool) -> Option<UnsavedSource> {
|
||||
match tab_editor(from)? {
|
||||
UnsavedSource::Manage => manage_dirty.then_some(UnsavedSource::Manage),
|
||||
UnsavedSource::Settings => settings_dirty.then_some(UnsavedSource::Settings),
|
||||
}
|
||||
}
|
||||
|
||||
/// Which editor the guard must ask about for `intent` while sitting on
|
||||
/// `from`, if any. A tab switch asks only about the tab being left; Quit asks
|
||||
/// about Settings before Manage, one prompt at a time, because each draft is
|
||||
/// a full `Config` snapshot and applying both at once would revert the first.
|
||||
/// Which editor the guard must ask about for `intent`, if any. Quit asks
|
||||
/// about Options before Manage, one prompt at a time: each draft is a full
|
||||
/// `Config` snapshot, so applying both at once would revert the first.
|
||||
fn guard_source(
|
||||
intent: NavIntent,
|
||||
from: Tab,
|
||||
manage_dirty: bool,
|
||||
settings_dirty: bool,
|
||||
options_dirty: bool,
|
||||
) -> Option<UnsavedSource> {
|
||||
match intent {
|
||||
NavIntent::SwitchTab(_) => dirty_editor(from, manage_dirty, settings_dirty),
|
||||
NavIntent::Quit if settings_dirty => Some(UnsavedSource::Settings),
|
||||
NavIntent::SwitchTab(_) => manage_dirty.then_some(UnsavedSource::Manage),
|
||||
NavIntent::CloseOptions => options_dirty.then_some(UnsavedSource::Options),
|
||||
NavIntent::Quit if options_dirty => Some(UnsavedSource::Options),
|
||||
NavIntent::Quit if manage_dirty => Some(UnsavedSource::Manage),
|
||||
NavIntent::Quit => None,
|
||||
}
|
||||
|
|
@ -98,13 +78,8 @@ fn quit_needs_reconcile_warning(intent: NavIntent, reconciling: bool) -> bool {
|
|||
/// Whether leaving the current tab has to go through the unsaved-changes
|
||||
/// guard. A navigation already on hold wins: a second intent would replace
|
||||
/// the answer the guard is waiting for.
|
||||
fn switch_needs_guard(
|
||||
from: Tab,
|
||||
manage_dirty: bool,
|
||||
settings_dirty: bool,
|
||||
nav_pending: bool,
|
||||
) -> bool {
|
||||
!nav_pending && dirty_editor(from, manage_dirty, settings_dirty).is_some()
|
||||
fn switch_needs_guard(from: Tab, manage_dirty: bool, nav_pending: bool) -> bool {
|
||||
from == Tab::Manage && manage_dirty && !nav_pending
|
||||
}
|
||||
|
||||
pub struct QuickSearchApp {
|
||||
|
|
@ -115,12 +90,9 @@ pub struct QuickSearchApp {
|
|||
manage: ManageTab,
|
||||
dups: DuplicatesTab,
|
||||
logs: LogsTab,
|
||||
settings: SettingsTab,
|
||||
options: OptionsWindow,
|
||||
/// Set when applying a config that invalidates the stored index.
|
||||
rebuild_prompt: Option<Vec<ConfigChange>>,
|
||||
/// The first-start tour, while it is open. Only ever `Some` for a config
|
||||
/// file this version created — see [`crate::tutorial`].
|
||||
tutorial: Option<crate::tutorial::Tutorial>,
|
||||
/// Set while the "delete the index?" confirmation is open.
|
||||
clear_prompt: bool,
|
||||
/// Nested roots found in the loaded config; shown as a modal over the
|
||||
|
|
@ -140,12 +112,8 @@ pub struct QuickSearchApp {
|
|||
/// Set when the watcher gave up on the directory budget and live
|
||||
/// updates are off.
|
||||
watch_cap_prompt: Option<WatchError>,
|
||||
/// The byte-for-byte check of one duplicate group, while its modal is up.
|
||||
verify: Option<VerifyModal>,
|
||||
/// In-flight security flow (enable/disable/change password).
|
||||
security_prompt: Option<SecurityPrompt>,
|
||||
/// In-flight show-key flow (confirm password, then reveal).
|
||||
key_prompt: Option<KeyPrompt>,
|
||||
/// A navigation held by the unsaved-changes guard; see [`NavIntent`].
|
||||
pending_nav: Option<NavIntent>,
|
||||
/// The guard resolved a Quit: let the next close request through.
|
||||
|
|
@ -161,10 +129,8 @@ mod security;
|
|||
mod status_bar;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
mod verify;
|
||||
|
||||
use security::{KeyPrompt, SecurityPrompt};
|
||||
use verify::VerifyModal;
|
||||
use security::SecurityPrompt;
|
||||
|
||||
impl QuickSearchApp {
|
||||
/// `initial_query` pre-fills the search box and fires a search on the
|
||||
|
|
@ -212,12 +178,7 @@ impl QuickSearchApp {
|
|||
} else {
|
||||
(Tab::Manage, Some(nested))
|
||||
};
|
||||
// `Some(false)` means a config file *this version wrote*, which is
|
||||
// the only thing that counts as a first start. A key that is absent
|
||||
// (`None`) belongs to an installation that upgraded into this version
|
||||
// and has already found its way around.
|
||||
let tutorial = (cfg.ui.tutorial_seen == Some(false)).then(crate::tutorial::Tutorial::new);
|
||||
let mut search = SearchTab::new(fuzzy, cfg.search.columns.clone(), cfg.search.live_results);
|
||||
let mut search = SearchTab::new(fuzzy);
|
||||
if let Some(query) = initial_query {
|
||||
search.seed(query);
|
||||
}
|
||||
|
|
@ -229,9 +190,8 @@ impl QuickSearchApp {
|
|||
manage: ManageTab::new(),
|
||||
dups: DuplicatesTab::new(),
|
||||
logs: LogsTab::new(),
|
||||
settings: SettingsTab::new(),
|
||||
options: OptionsWindow::new(),
|
||||
rebuild_prompt: None,
|
||||
tutorial,
|
||||
clear_prompt: false,
|
||||
nested_prompt,
|
||||
key_source,
|
||||
|
|
@ -239,9 +199,7 @@ impl QuickSearchApp {
|
|||
reconcile_owed,
|
||||
reconcile_owed_since,
|
||||
watch_cap_prompt: None,
|
||||
verify: None,
|
||||
security_prompt: None,
|
||||
key_prompt: None,
|
||||
pending_nav: None,
|
||||
quit_confirmed: false,
|
||||
config_error,
|
||||
|
|
@ -264,10 +222,6 @@ impl QuickSearchApp {
|
|||
let Some(search) = self.backend.search() else {
|
||||
return;
|
||||
};
|
||||
// The single funnel every search goes through — the debounce, `seed`,
|
||||
// and every `actions.rerun` producer — so it is the one place the old
|
||||
// results' watches have to be dropped.
|
||||
self.backend.clear_live();
|
||||
let generation = search.search(&self.search.query, self.search_options());
|
||||
self.search.on_search_started(generation);
|
||||
}
|
||||
|
|
@ -278,32 +232,6 @@ impl QuickSearchApp {
|
|||
self.backend.start_duplicates(&cfg, ctx.clone());
|
||||
}
|
||||
|
||||
/// Move to another tab, running what leaving one tab and arriving at the
|
||||
/// other owe. Every switch goes through here — including the ones the
|
||||
/// unsaved-changes guard completes a frame later, which is why this is a
|
||||
/// funnel rather than a comparison against the previous frame's tab.
|
||||
fn switch_tab(&mut self, ctx: &egui::Context, to: Tab) {
|
||||
if self.tab == to {
|
||||
return;
|
||||
}
|
||||
match self.tab {
|
||||
// Watching rows nobody is looking at costs descriptors for
|
||||
// nothing.
|
||||
Tab::Search => {
|
||||
self.backend.clear_live();
|
||||
self.search.reset_live();
|
||||
}
|
||||
// A draft kept while the config is edited elsewhere would go
|
||||
// stale, and applying it later would revert those edits.
|
||||
Tab::Settings => self.settings.discard(),
|
||||
_ => {}
|
||||
}
|
||||
self.tab = to;
|
||||
if to == Tab::Duplicates {
|
||||
self.start_duplicates_scan(ctx);
|
||||
}
|
||||
}
|
||||
|
||||
/// Save + route an edited config to the running services. Reports
|
||||
/// whether the config was accepted — a `false` means nothing was saved
|
||||
/// and the caller must keep any staged edits alive.
|
||||
|
|
@ -362,16 +290,6 @@ impl QuickSearchApp {
|
|||
self.rebuild_prompt = Some(changes);
|
||||
}
|
||||
}
|
||||
self.search.live_enabled = new.search.live_results;
|
||||
if new.search.live_results {
|
||||
// The watcher holds a copy of the config for its extraction
|
||||
// limits and filters, so a config edit has to re-arm; dropping
|
||||
// the tab-side state is what makes the next frame do it.
|
||||
self.search.reset_live();
|
||||
} else {
|
||||
self.backend.clear_live();
|
||||
self.search.reset_live();
|
||||
}
|
||||
self.cfg = new;
|
||||
true
|
||||
}
|
||||
|
|
@ -379,23 +297,18 @@ impl QuickSearchApp {
|
|||
/// What the system-wide search shortcut does once the window is up:
|
||||
/// show the Search tab with the caret in the query box and any existing
|
||||
/// text selected.
|
||||
pub(crate) fn activate_search(&mut self, ctx: &egui::Context) {
|
||||
if switch_needs_guard(
|
||||
self.tab,
|
||||
self.manage.is_dirty(),
|
||||
self.settings.is_dirty(&self.cfg),
|
||||
self.pending_nav.is_some(),
|
||||
) {
|
||||
pub(crate) fn activate_search(&mut self) {
|
||||
if switch_needs_guard(self.tab, self.manage.is_dirty(), self.pending_nav.is_some()) {
|
||||
self.pending_nav = Some(NavIntent::SwitchTab(Tab::Search));
|
||||
} else {
|
||||
self.switch_tab(ctx, Tab::Search);
|
||||
self.tab = Tab::Search;
|
||||
}
|
||||
self.search.request_focus();
|
||||
}
|
||||
|
||||
/// Whether the Settings tab is currently reading a key press to bind.
|
||||
/// Whether the Options window is currently reading a key press to bind.
|
||||
pub(crate) fn capturing_hotkey(&self) -> bool {
|
||||
self.tab == Tab::Settings && self.settings.capturing_hotkey()
|
||||
self.options.capturing_hotkey()
|
||||
}
|
||||
|
||||
/// Switch the indexing mode and write it to the config immediately: a
|
||||
|
|
@ -421,21 +334,6 @@ impl QuickSearchApp {
|
|||
self.search
|
||||
.apply_update(update, self.cfg.search.display_limit);
|
||||
}
|
||||
// Every live update is something the watcher read off the disk that
|
||||
// the index has not been told about. Handing the paths back keeps the
|
||||
// index from drifting away from the rows on screen — and it is the
|
||||
// only thing that does so while indexing is stopped.
|
||||
let mut touched: Vec<PathBuf> = Vec::new();
|
||||
while let Ok(update) = self.backend.live_rx.try_recv() {
|
||||
touched.push(PathBuf::from(update.path()));
|
||||
// A rename has two sides: the old path leaves the index and the
|
||||
// new one enters it.
|
||||
if let quicksearch_core::live::LiveUpdate::Renamed { to, .. } = &update {
|
||||
touched.push(PathBuf::from(to));
|
||||
}
|
||||
self.search.apply_live(update);
|
||||
}
|
||||
self.backend.reindex_live_paths(touched);
|
||||
// Duplicates worker.
|
||||
if let Some(rx) = &self.backend.dup_job {
|
||||
use std::sync::mpsc::TryRecvError;
|
||||
|
|
@ -452,7 +350,6 @@ impl QuickSearchApp {
|
|||
self.backend.dup_job = None;
|
||||
}
|
||||
}
|
||||
self.drain_verify();
|
||||
}
|
||||
|
||||
fn tick_debounce(&mut self, ctx: &egui::Context) {
|
||||
|
|
@ -492,7 +389,7 @@ impl QuickSearchApp {
|
|||
}
|
||||
|
||||
pub(crate) fn capture_search_settled(&self) -> bool {
|
||||
self.search.settled()
|
||||
self.search.capture_settled()
|
||||
}
|
||||
|
||||
pub(crate) fn capture_dups_done(&self) -> bool {
|
||||
|
|
@ -522,12 +419,6 @@ impl QuickSearchApp {
|
|||
pub(crate) fn pin_live_fields(new: &mut Config, live: &Config) {
|
||||
new.security = live.security.clone();
|
||||
new.indexing.auto_index = live.indexing.auto_index;
|
||||
// The column picker writes straight to the live config the moment a
|
||||
// checkbox moves — from the table header *or* from the Settings tab,
|
||||
// which is why the Settings controls for it are not draft-backed. Pinning
|
||||
// here is what stops a draft taken before a header-menu change from
|
||||
// undoing it on Apply.
|
||||
new.search.columns = live.search.columns.clone();
|
||||
}
|
||||
|
||||
/// Keep the configured UI scale within sane, recoverable bounds.
|
||||
|
|
@ -557,8 +448,7 @@ pub(crate) fn apply_theme(ctx: &egui::Context, setting: &str) {
|
|||
|
||||
impl eframe::App for QuickSearchApp {
|
||||
fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
|
||||
// First, so a scripted navigation is held before the tab strip reads
|
||||
// this frame's state.
|
||||
// First, so `previous_tab` below sees pre-navigation state.
|
||||
#[cfg(feature = "capture")]
|
||||
self.capture_tick(ctx);
|
||||
|
||||
|
|
@ -571,7 +461,7 @@ impl eframe::App for QuickSearchApp {
|
|||
if ctx.input(|i| i.viewport().close_requested())
|
||||
&& !self.quit_confirmed
|
||||
&& (self.manage.is_dirty()
|
||||
|| self.settings.is_dirty(&self.cfg)
|
||||
|| self.options.is_dirty(&self.cfg)
|
||||
|| self.backend.coordinator.reconciling())
|
||||
{
|
||||
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
|
||||
|
|
@ -581,6 +471,7 @@ impl eframe::App for QuickSearchApp {
|
|||
|
||||
self.status_bar(ctx);
|
||||
|
||||
let previous_tab = self.tab;
|
||||
// Tab clicks land on a local first so the unsaved-changes guard can
|
||||
// hold them.
|
||||
let mut requested = self.tab;
|
||||
|
|
@ -591,21 +482,31 @@ impl eframe::App for QuickSearchApp {
|
|||
ui.selectable_value(&mut requested, Tab::Duplicates, "Duplicates");
|
||||
ui.selectable_value(&mut requested, Tab::Logs, "Logs");
|
||||
ui.selectable_value(&mut requested, Tab::Help, "Help");
|
||||
ui.selectable_value(&mut requested, Tab::Settings, "Settings");
|
||||
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
||||
if ui.button("⚙").on_hover_text("Options").clicked() {
|
||||
if !self.options.open {
|
||||
self.options.open_with(&self.cfg);
|
||||
} else if self.options.is_dirty(&self.cfg) {
|
||||
if self.pending_nav.is_none() {
|
||||
self.pending_nav = Some(NavIntent::CloseOptions);
|
||||
}
|
||||
} else {
|
||||
self.options.close_discard();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
if requested != self.tab {
|
||||
if switch_needs_guard(
|
||||
self.tab,
|
||||
self.manage.is_dirty(),
|
||||
self.settings.is_dirty(&self.cfg),
|
||||
self.pending_nav.is_some(),
|
||||
) {
|
||||
if switch_needs_guard(self.tab, self.manage.is_dirty(), self.pending_nav.is_some()) {
|
||||
self.pending_nav = Some(NavIntent::SwitchTab(requested));
|
||||
} else {
|
||||
self.switch_tab(ctx, requested);
|
||||
self.tab = requested;
|
||||
}
|
||||
}
|
||||
if self.tab == Tab::Duplicates && previous_tab != Tab::Duplicates {
|
||||
self.start_duplicates_scan(ctx);
|
||||
}
|
||||
|
||||
if let Some(err) = &self.config_error {
|
||||
let err = err.clone();
|
||||
|
|
@ -632,17 +533,6 @@ impl eframe::App for QuickSearchApp {
|
|||
self.config_error = Some(e);
|
||||
}
|
||||
}
|
||||
// Live state, saved the moment it changes — like the fuzzy
|
||||
// default above, and unlike anything edited through the
|
||||
// Settings draft. The Settings tab's own column controls take
|
||||
// this same path, so the two editors cannot disagree and a
|
||||
// stale draft cannot revert either of them.
|
||||
if let Some(columns) = actions.save_columns {
|
||||
self.cfg.search.columns = columns;
|
||||
if let Err(e) = self.cfg.save() {
|
||||
self.config_error = Some(e);
|
||||
}
|
||||
}
|
||||
if let Some(pattern) = actions.persist_ignore {
|
||||
let mut new_cfg = self.cfg.clone();
|
||||
if !new_cfg.indexing.ignore_patterns.contains(&pattern) {
|
||||
|
|
@ -650,10 +540,6 @@ impl eframe::App for QuickSearchApp {
|
|||
self.apply_new_config(ctx, new_cfg);
|
||||
}
|
||||
}
|
||||
if let Some(targets) = actions.live_targets {
|
||||
self.backend
|
||||
.watch_live(&self.search.query, targets, &self.cfg);
|
||||
}
|
||||
if actions.rerun {
|
||||
self.start_search();
|
||||
}
|
||||
|
|
@ -685,54 +571,32 @@ impl eframe::App for QuickSearchApp {
|
|||
}
|
||||
}
|
||||
Tab::Duplicates => {
|
||||
let actions = self.dups.ui(ui, self.verify.is_some());
|
||||
let actions = self.dups.ui(ui);
|
||||
if actions.refresh {
|
||||
self.start_duplicates_scan(ctx);
|
||||
}
|
||||
if let Some(paths) = actions.verify {
|
||||
let paths: Vec<std::path::PathBuf> =
|
||||
paths.into_iter().map(std::path::PathBuf::from).collect();
|
||||
self.backend.start_verify(paths.clone(), ctx.clone());
|
||||
self.verify = Some(VerifyModal::new(paths));
|
||||
}
|
||||
}
|
||||
Tab::Logs => self.logs.ui(ui),
|
||||
Tab::Help => {
|
||||
if crate::help_tab::ui(ui) {
|
||||
self.show_tutorial();
|
||||
}
|
||||
}
|
||||
Tab::Settings => {
|
||||
let out = self.settings.ui(ui, &self.cfg);
|
||||
if let Some(new_cfg) = out.applied {
|
||||
self.apply_new_config(ctx, new_cfg);
|
||||
}
|
||||
if let Some(action) = out.security {
|
||||
self.handle_security_action(action);
|
||||
}
|
||||
// Same live path the table header's picker takes, so the two
|
||||
// controls stay in step and neither needs an Apply.
|
||||
if let Some(columns) = out.columns {
|
||||
self.cfg.search.columns = columns.clone();
|
||||
self.search.columns = columns;
|
||||
self.search.mark_sort_dirty();
|
||||
if let Err(e) = self.cfg.save() {
|
||||
self.config_error = Some(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
Tab::Help => crate::help_tab::ui(ui),
|
||||
});
|
||||
|
||||
let options_out = self.options.ui(ctx, &self.cfg);
|
||||
if let Some(new_cfg) = options_out.applied {
|
||||
self.apply_new_config(ctx, new_cfg);
|
||||
}
|
||||
if let Some(action) = options_out.security {
|
||||
self.handle_security_action(action);
|
||||
}
|
||||
if options_out.close_requested && self.pending_nav.is_none() {
|
||||
self.pending_nav = Some(NavIntent::CloseOptions);
|
||||
}
|
||||
self.rebuild_prompt_ui(ctx);
|
||||
self.security_prompt_ui(ctx);
|
||||
self.key_prompt_ui(ctx);
|
||||
self.clear_prompt_ui(ctx);
|
||||
self.nested_prompt_ui(ctx);
|
||||
// Ahead of the watch-cap warning: on a fresh upgrade both can be true.
|
||||
self.stale_index_prompt_ui(ctx);
|
||||
self.watch_cap_prompt_ui(ctx);
|
||||
self.verify_modal_ui(ctx);
|
||||
self.tutorial_ui(ctx);
|
||||
// Last: the guard must sit above everything else on screen.
|
||||
self.unsaved_prompt_ui(ctx);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ impl QuickSearchApp {
|
|||
});
|
||||
if close == Some(true) {
|
||||
self.nested_prompt = None;
|
||||
self.switch_tab(ctx, Tab::Manage);
|
||||
self.tab = Tab::Manage;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -187,29 +187,6 @@ impl QuickSearchApp {
|
|||
}
|
||||
}
|
||||
|
||||
/// The first-start tour. Dismissal is written straight to the config, the
|
||||
/// way the fuzzy default is — not through the Settings draft, which this
|
||||
/// has nothing to do with.
|
||||
pub(super) fn tutorial_ui(&mut self, ctx: &egui::Context) {
|
||||
let Some(tour) = &mut self.tutorial else {
|
||||
return;
|
||||
};
|
||||
let roots = self.cfg.paths.indexing_paths.clone();
|
||||
if tour.ui(ctx, &roots) {
|
||||
self.tutorial = None;
|
||||
self.cfg.ui.tutorial_seen = Some(true);
|
||||
if let Err(e) = self.cfg.save() {
|
||||
self.config_error = Some(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Re-open the tour from the Help tab. Nothing is written until it is
|
||||
/// dismissed again, so a re-read costs the config nothing.
|
||||
pub(crate) fn show_tutorial(&mut self) {
|
||||
self.tutorial = Some(crate::tutorial::Tutorial::new());
|
||||
}
|
||||
|
||||
pub(super) fn clear_prompt_ui(&mut self, ctx: &egui::Context) {
|
||||
if !self.clear_prompt {
|
||||
return;
|
||||
|
|
@ -248,10 +225,8 @@ impl QuickSearchApp {
|
|||
let Some(intent) = self.pending_nav else {
|
||||
return;
|
||||
};
|
||||
let dirty = (self.manage.is_dirty(), self.settings.is_dirty(&self.cfg));
|
||||
// `self.tab` is the tab being left: the switch itself is what the
|
||||
// intent is holding back.
|
||||
let Some(source) = guard_source(intent, self.tab, dirty.0, dirty.1) else {
|
||||
let dirty = (self.manage.is_dirty(), self.options.is_dirty(&self.cfg));
|
||||
let Some(source) = guard_source(intent, dirty.0, dirty.1) else {
|
||||
// Inside the guard: the Discard-then-quit path sets
|
||||
// `quit_confirmed` and never returns to the close-request check,
|
||||
// so a warning living only there would be skipped.
|
||||
|
|
@ -274,7 +249,7 @@ impl QuickSearchApp {
|
|||
Some(UnsavedChoice::Cancel) => self.pending_nav = None,
|
||||
Some(UnsavedChoice::Discard) => match source {
|
||||
UnsavedSource::Manage => self.manage.discard(),
|
||||
UnsavedSource::Settings => self.settings.discard(),
|
||||
UnsavedSource::Options => self.options.close_discard(),
|
||||
},
|
||||
Some(UnsavedChoice::Apply) => {
|
||||
let ok = match source {
|
||||
|
|
@ -288,11 +263,11 @@ impl QuickSearchApp {
|
|||
}
|
||||
None => true,
|
||||
},
|
||||
UnsavedSource::Settings => match self.settings.draft_config() {
|
||||
UnsavedSource::Options => match self.options.draft_config() {
|
||||
Some(cfg) => {
|
||||
let ok = self.apply_new_config(ctx, cfg);
|
||||
if ok {
|
||||
self.settings.discard();
|
||||
self.options.close_discard();
|
||||
}
|
||||
ok
|
||||
}
|
||||
|
|
@ -312,7 +287,14 @@ impl QuickSearchApp {
|
|||
pub(super) fn complete_nav(&mut self, ctx: &egui::Context, intent: NavIntent) {
|
||||
self.pending_nav = None;
|
||||
match intent {
|
||||
NavIntent::SwitchTab(tab) => self.switch_tab(ctx, tab),
|
||||
NavIntent::SwitchTab(tab) => {
|
||||
let was = self.tab;
|
||||
self.tab = tab;
|
||||
if tab == Tab::Duplicates && was != Tab::Duplicates {
|
||||
self.start_duplicates_scan(ctx);
|
||||
}
|
||||
}
|
||||
NavIntent::CloseOptions => self.options.close_discard(),
|
||||
NavIntent::Quit => {
|
||||
self.quit_confirmed = true;
|
||||
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
|
||||
|
|
@ -326,7 +308,7 @@ impl QuickSearchApp {
|
|||
///
|
||||
/// Unlike the centered `egui::Window` the other prompts use, `egui::Modal`'s
|
||||
/// backdrop blocks input to everything behind it — a click landing on the
|
||||
/// tab strip would re-trigger or bypass the guard.
|
||||
/// tab strip or the Options ✕ would re-trigger or bypass the guard.
|
||||
fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<UnsavedChoice> {
|
||||
let mut choice = None;
|
||||
let modal = egui::Modal::new(egui::Id::new("unsaved-guard")).show(ctx, |ui| {
|
||||
|
|
@ -334,7 +316,7 @@ fn unsaved_changes_modal(ctx: &egui::Context, source: UnsavedSource) -> Option<U
|
|||
ui.heading("Unsaved changes");
|
||||
ui.label(match source {
|
||||
UnsavedSource::Manage => "The Manage Index tab has edits that have not been applied.",
|
||||
UnsavedSource::Settings => "The Settings tab has edits that have not been applied.",
|
||||
UnsavedSource::Options => "The Options window has edits that have not been applied.",
|
||||
});
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal(|ui| {
|
||||
|
|
@ -378,7 +360,7 @@ fn reconcile_owed_banner(ctx: &egui::Context) -> Option<ReconcileOwedChoice> {
|
|||
ui.horizontal(|ui| {
|
||||
ui.colored_label(
|
||||
ui.visuals().warn_fg_color,
|
||||
"Your indexing settings have not been applied to the index yet.",
|
||||
"⚠ Your indexing settings have not been applied to the index yet.",
|
||||
);
|
||||
if ui.small_button("Start indexing now").clicked() {
|
||||
choice = Some(ReconcileOwedChoice::StartIndexing);
|
||||
|
|
@ -505,7 +487,7 @@ mod tests {
|
|||
KeySource::Prompt,
|
||||
KeySource::Keychain,
|
||||
] {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
assert!(
|
||||
!frame(&ctx, source, Vec::new()),
|
||||
"an untouched frame must not request a rebuild"
|
||||
|
|
@ -545,8 +527,8 @@ mod tests {
|
|||
/// buttons fire, Esc cancels, and an untouched frame decides nothing.
|
||||
#[test]
|
||||
fn the_unsaved_modal_reports_each_choice() {
|
||||
for source in [UnsavedSource::Manage, UnsavedSource::Settings] {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
for source in [UnsavedSource::Manage, UnsavedSource::Options] {
|
||||
let ctx = egui::Context::default();
|
||||
assert_eq!(
|
||||
modal_frame(&ctx, source, Vec::new()),
|
||||
None,
|
||||
|
|
@ -598,7 +580,7 @@ mod tests {
|
|||
/// Both ways out of the quit warning work, and neither is the default.
|
||||
#[test]
|
||||
fn the_quit_warning_reports_both_answers() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
assert_eq!(
|
||||
reconcile_modal_frame(&ctx, Vec::new()),
|
||||
None,
|
||||
|
|
@ -641,7 +623,7 @@ mod tests {
|
|||
/// Both banner buttons report their clicks.
|
||||
#[test]
|
||||
fn the_reconcile_banner_reports_both_buttons() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
assert_eq!(banner_frame(&ctx, Vec::new()), None);
|
||||
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
|
|
|
|||
|
|
@ -3,9 +3,7 @@
|
|||
|
||||
use super::*;
|
||||
|
||||
use quicksearch_core::security::SALT_LEN;
|
||||
|
||||
use crate::ui_util::{centered_modal, hint};
|
||||
use crate::ui_util::centered_modal;
|
||||
|
||||
/// The two-step security flow: collect a password (enable/change), derive
|
||||
/// its key off the UI thread, then confirm the mandatory index rebuild.
|
||||
|
|
@ -18,10 +16,7 @@ pub(super) enum SecurityPrompt {
|
|||
change: bool,
|
||||
},
|
||||
Deriving {
|
||||
rx: mpsc::Receiver<IndexKey>,
|
||||
/// Built with the salt the pending key is being derived from, so the
|
||||
/// two always describe each other.
|
||||
new_security: SecurityConfig,
|
||||
rx: mpsc::Receiver<(SecurityConfig, IndexKey)>,
|
||||
},
|
||||
ConfirmRebuild {
|
||||
new_security: SecurityConfig,
|
||||
|
|
@ -38,69 +33,8 @@ impl Drop for SecurityPrompt {
|
|||
}
|
||||
}
|
||||
|
||||
/// The show-key flow: confirm the password, re-derive from it, then reveal
|
||||
/// the installed key. Nothing here can change the key or the config.
|
||||
pub(super) enum KeyPrompt {
|
||||
Confirm {
|
||||
pw: String,
|
||||
wrong: bool,
|
||||
},
|
||||
Deriving {
|
||||
rx: mpsc::Receiver<IndexKey>,
|
||||
},
|
||||
/// The key as displayed: `0x` followed by 64 hex digits.
|
||||
Reveal {
|
||||
display: String,
|
||||
},
|
||||
}
|
||||
|
||||
impl Drop for KeyPrompt {
|
||||
fn drop(&mut self) {
|
||||
match self {
|
||||
KeyPrompt::Confirm { pw, .. } => pw.zeroize(),
|
||||
KeyPrompt::Reveal { display } => display.zeroize(),
|
||||
KeyPrompt::Deriving { .. } => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Derive a key off the UI thread. The password is consumed and dropped
|
||||
/// there, so it never outlives the derivation.
|
||||
fn spawn_derive(
|
||||
ctx: &egui::Context,
|
||||
password: Zeroizing<String>,
|
||||
salt: [u8; SALT_LEN],
|
||||
) -> mpsc::Receiver<IndexKey> {
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let repaint = ctx.clone();
|
||||
std::thread::spawn(move || {
|
||||
let key = derive_key(&password, &salt);
|
||||
drop(password);
|
||||
let _ = tx.send(key);
|
||||
repaint.request_repaint();
|
||||
});
|
||||
rx
|
||||
}
|
||||
|
||||
/// Paint the spinner shown while a derivation runs. Not `centered_modal`:
|
||||
/// this one hides its title bar.
|
||||
fn deriving_window(ctx: &egui::Context) {
|
||||
egui::Window::new("Deriving key")
|
||||
.collapsible(false)
|
||||
.resizable(false)
|
||||
.title_bar(false)
|
||||
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
|
||||
.show(ctx, |ui| {
|
||||
ui.horizontal(|ui| {
|
||||
ui.spinner();
|
||||
ui.label("Deriving key…");
|
||||
});
|
||||
});
|
||||
ctx.request_repaint_after(Duration::from_millis(100));
|
||||
}
|
||||
|
||||
impl QuickSearchApp {
|
||||
/// Route a click in the Settings tab's Security block. Keychain
|
||||
/// Route a click in the Options window's Security block. Keychain
|
||||
/// toggles act immediately; everything else opens the two-step flow.
|
||||
pub(super) fn handle_security_action(&mut self, action: SecurityAction) {
|
||||
match action {
|
||||
|
|
@ -118,12 +52,6 @@ impl QuickSearchApp {
|
|||
new_key: None,
|
||||
});
|
||||
}
|
||||
SecurityAction::ShowKey => {
|
||||
self.key_prompt = Some(KeyPrompt::Confirm {
|
||||
pw: String::new(),
|
||||
wrong: false,
|
||||
});
|
||||
}
|
||||
SecurityAction::SetKeychain(remember) => {
|
||||
let db_path = self.cfg.resolved_database_path();
|
||||
if remember {
|
||||
|
|
@ -211,25 +139,47 @@ impl QuickSearchApp {
|
|||
} else if submit {
|
||||
let password = Zeroizing::new(std::mem::take(pw1));
|
||||
pw2.zeroize();
|
||||
let remember = *remember;
|
||||
purge_security_field_state(ctx);
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let repaint = ctx.clone();
|
||||
std::thread::spawn(move || {
|
||||
let salt = generate_salt();
|
||||
let key = derive_key(&password, &salt);
|
||||
drop(password);
|
||||
let new_security = SecurityConfig {
|
||||
password_protected: true,
|
||||
salt: Some(salt_to_hex(&salt)),
|
||||
use_keychain: *remember,
|
||||
use_keychain: remember,
|
||||
};
|
||||
purge_security_field_state(ctx);
|
||||
let rx = spawn_derive(ctx, password, salt);
|
||||
self.security_prompt = Some(SecurityPrompt::Deriving { rx, new_security });
|
||||
let _ = tx.send((new_security, key));
|
||||
repaint.request_repaint();
|
||||
});
|
||||
self.security_prompt = Some(SecurityPrompt::Deriving { rx });
|
||||
}
|
||||
}
|
||||
SecurityPrompt::Deriving { rx, new_security } => match rx.try_recv() {
|
||||
Ok(key) => {
|
||||
SecurityPrompt::Deriving { rx } => match rx.try_recv() {
|
||||
Ok((new_security, key)) => {
|
||||
self.security_prompt = Some(SecurityPrompt::ConfirmRebuild {
|
||||
new_security: new_security.clone(),
|
||||
new_security,
|
||||
new_key: Some(key),
|
||||
});
|
||||
}
|
||||
Err(mpsc::TryRecvError::Empty) => deriving_window(ctx),
|
||||
Err(mpsc::TryRecvError::Empty) => {
|
||||
// Not `centered_modal`: this one hides its title bar.
|
||||
egui::Window::new("Deriving key")
|
||||
.collapsible(false)
|
||||
.resizable(false)
|
||||
.title_bar(false)
|
||||
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
|
||||
.show(ctx, |ui| {
|
||||
ui.horizontal(|ui| {
|
||||
ui.spinner();
|
||||
ui.label("Deriving key…");
|
||||
});
|
||||
});
|
||||
ctx.request_repaint_after(Duration::from_millis(100));
|
||||
}
|
||||
Err(mpsc::TryRecvError::Disconnected) => {
|
||||
self.config_error = Some("key derivation thread died".to_string());
|
||||
self.security_prompt = None;
|
||||
|
|
@ -272,74 +222,6 @@ impl QuickSearchApp {
|
|||
}
|
||||
}
|
||||
|
||||
/// Render the show-key flow (drawn with the other modals). Only ever
|
||||
/// open while protection is on, so a salt and a process key both exist.
|
||||
pub(super) fn key_prompt_ui(&mut self, ctx: &egui::Context) {
|
||||
let Some(prompt) = &mut self.key_prompt else {
|
||||
return;
|
||||
};
|
||||
match prompt {
|
||||
KeyPrompt::Confirm { pw, wrong } => {
|
||||
let (submit, cancel) = confirm_key_modal(ctx, pw, *wrong);
|
||||
if cancel {
|
||||
self.key_prompt = None; // Drop impl zeroizes
|
||||
purge_security_field_state(ctx);
|
||||
} else if submit {
|
||||
let password = Zeroizing::new(std::mem::take(pw));
|
||||
purge_security_field_state(ctx);
|
||||
match self.cfg.security.salt_bytes() {
|
||||
Ok(salt) => {
|
||||
let rx = spawn_derive(ctx, password, salt);
|
||||
self.key_prompt = Some(KeyPrompt::Deriving { rx });
|
||||
}
|
||||
Err(e) => {
|
||||
self.config_error = Some(e);
|
||||
self.key_prompt = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
KeyPrompt::Deriving { rx } => match rx.try_recv() {
|
||||
Ok(key) => match db::process_key_hex() {
|
||||
// What is shown is the installed key, not the derived
|
||||
// one: it is the key that actually opens the index.
|
||||
Some(installed) => match reveal_display(&installed, &key.to_hex()) {
|
||||
Some(display) => {
|
||||
self.key_prompt = Some(KeyPrompt::Reveal { display });
|
||||
}
|
||||
None => {
|
||||
self.key_prompt = Some(KeyPrompt::Confirm {
|
||||
pw: String::new(),
|
||||
wrong: true,
|
||||
});
|
||||
}
|
||||
},
|
||||
None => {
|
||||
// Unreachable while protected — the gate always
|
||||
// installs a key before the app starts.
|
||||
self.config_error =
|
||||
Some("no key installed; restart and unlock first".to_string());
|
||||
self.key_prompt = None;
|
||||
}
|
||||
},
|
||||
Err(mpsc::TryRecvError::Empty) => deriving_window(ctx),
|
||||
Err(mpsc::TryRecvError::Disconnected) => {
|
||||
self.config_error = Some("key derivation thread died".to_string());
|
||||
self.key_prompt = None;
|
||||
}
|
||||
},
|
||||
KeyPrompt::Reveal { display } => {
|
||||
let (copy, close) = reveal_key_modal(ctx, display);
|
||||
if copy {
|
||||
ctx.copy_text(display.clone());
|
||||
}
|
||||
if close {
|
||||
self.key_prompt = None; // Drop impl zeroizes
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Commit a confirmed security change: config, keychain, process key —
|
||||
/// in that order, before the rebuild so the fresh index is created
|
||||
/// under the new key (or none).
|
||||
|
|
@ -373,92 +255,11 @@ impl QuickSearchApp {
|
|||
}
|
||||
}
|
||||
|
||||
/// Id of the show-key confirmation field, shared by the widget and the
|
||||
/// purge below.
|
||||
const SHOW_KEY_FIELD: &str = "show-key-pw";
|
||||
|
||||
/// Drop egui's retained text-field state (buffer + undo history) for the
|
||||
/// password dialog fields.
|
||||
fn purge_security_field_state(ctx: &egui::Context) {
|
||||
ctx.data_mut(|d| {
|
||||
d.remove::<egui::text_edit::TextEditState>(egui::Id::new("security-pw1"));
|
||||
d.remove::<egui::text_edit::TextEditState>(egui::Id::new("security-pw2"));
|
||||
d.remove::<egui::text_edit::TextEditState>(egui::Id::new(SHOW_KEY_FIELD));
|
||||
});
|
||||
}
|
||||
|
||||
/// The display form of the installed key, or `None` when the password the
|
||||
/// user typed does not derive it. Both arguments come from
|
||||
/// [`IndexKey::to_hex`], which is always lowercase, so a plain comparison is
|
||||
/// exact; nothing secret is learned from its timing, since the caller
|
||||
/// already holds the guess.
|
||||
fn reveal_display(installed_hex: &str, derived_hex: &str) -> Option<String> {
|
||||
(installed_hex == derived_hex).then(|| format!("0x{}", installed_hex))
|
||||
}
|
||||
|
||||
/// Paint the password confirmation; `(submit, cancel)` from its buttons.
|
||||
/// Free, like the reveal below, so both halves of the flow can be rendered
|
||||
/// against a bare context.
|
||||
fn confirm_key_modal(ctx: &egui::Context, pw: &mut String, wrong: bool) -> (bool, bool) {
|
||||
centered_modal(ctx, "Show database key", |ui| {
|
||||
ui.set_max_width(360.0);
|
||||
ui.label(
|
||||
"Confirm your password to show the raw key the index is \
|
||||
encrypted with.",
|
||||
);
|
||||
let field = ui.add(
|
||||
egui::TextEdit::singleline(pw)
|
||||
.id(egui::Id::new(SHOW_KEY_FIELD))
|
||||
.password(true)
|
||||
.hint_text("Password")
|
||||
.desired_width(240.0),
|
||||
);
|
||||
// On open, and again after a wrong attempt. Never steals focus from
|
||||
// something the user moved to themselves.
|
||||
if ui.memory(|m| m.focused().is_none()) {
|
||||
field.request_focus();
|
||||
}
|
||||
if wrong {
|
||||
ui.colored_label(ui.visuals().error_fg_color, "That password is not correct.");
|
||||
}
|
||||
ui.horizontal(|ui| {
|
||||
let ok = !pw.is_empty();
|
||||
// Enter in the field submits, like the unlock screen.
|
||||
let entered = field.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter));
|
||||
let submit =
|
||||
ui.add_enabled(ok, egui::Button::new("Show key")).clicked() || (ok && entered);
|
||||
(submit, ui.button("Cancel").clicked())
|
||||
})
|
||||
.inner
|
||||
})
|
||||
.unwrap_or((false, false))
|
||||
}
|
||||
|
||||
/// Paint the revealed key; `(copy, close)` from its buttons. A free function
|
||||
/// rather than a method so it can be rendered against a bare context.
|
||||
fn reveal_key_modal(ctx: &egui::Context, display: &str) -> (bool, bool) {
|
||||
centered_modal(ctx, "Database key", |ui| {
|
||||
ui.set_max_width(420.0);
|
||||
ui.label(
|
||||
"This is the SQLCipher raw key for the index. Anyone holding it can \
|
||||
read the index without the password.",
|
||||
);
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.label(egui::RichText::new(display).monospace());
|
||||
});
|
||||
ui.add_space(6.0);
|
||||
ui.label(hint(
|
||||
"Other SQLCipher tools take the key in this form. A copy stays on the \
|
||||
clipboard until something else replaces it.",
|
||||
));
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal(|ui| (ui.button("Copy").clicked(), ui.button("Close").clicked()))
|
||||
.inner
|
||||
})
|
||||
.unwrap_or((false, false))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "security_tests.rs"]
|
||||
mod tests;
|
||||
|
|
|
|||
|
|
@ -1,169 +0,0 @@
|
|||
use super::*;
|
||||
|
||||
use crate::test_ui::{click_at, painted_text, painted_text_center, raw_input};
|
||||
|
||||
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
|
||||
|
||||
/// Two 64-digit keys that differ, in the lowercase form [`IndexKey::to_hex`]
|
||||
/// produces.
|
||||
const KEY: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
|
||||
const OTHER: &str = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210";
|
||||
|
||||
/// Two passes: an `egui::Window` is measured on its first frame and placed on
|
||||
/// the next, so a single pass paints nothing to read back. Same shape as the
|
||||
/// verify modal's test frame.
|
||||
fn frame(
|
||||
ctx: &egui::Context,
|
||||
display: &str,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (egui::FullOutput, (bool, bool)) {
|
||||
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
reveal_key_modal(ctx, display);
|
||||
});
|
||||
let mut buttons = (false, false);
|
||||
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
|
||||
buttons = reveal_key_modal(ctx, display);
|
||||
});
|
||||
(out, buttons)
|
||||
}
|
||||
|
||||
/// The confirmation half, in the same two passes.
|
||||
fn confirm_frame(
|
||||
ctx: &egui::Context,
|
||||
pw: &mut String,
|
||||
wrong: bool,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (egui::FullOutput, (bool, bool)) {
|
||||
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
confirm_key_modal(ctx, pw, wrong);
|
||||
});
|
||||
let mut buttons = (false, false);
|
||||
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
|
||||
buttons = confirm_key_modal(ctx, pw, wrong);
|
||||
});
|
||||
(out, buttons)
|
||||
}
|
||||
|
||||
/// An empty field cannot submit: there is nothing to derive from, and a
|
||||
/// dead button says so more clearly than a rejected attempt would.
|
||||
#[test]
|
||||
fn the_confirmation_will_not_submit_an_empty_password() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut pw = String::new();
|
||||
let (out, buttons) = confirm_frame(&ctx, &mut pw, false, Vec::new());
|
||||
assert_eq!(buttons, (false, false));
|
||||
|
||||
let pos = painted_text_center(&out, "Show key").expect("no submit button painted");
|
||||
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(pos));
|
||||
assert!(!buttons.0, "an empty password was submitted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_confirmation_submits_a_typed_password_and_cancels_on_request() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut pw = "hunter2".to_string();
|
||||
let (out, _) = confirm_frame(&ctx, &mut pw, false, Vec::new());
|
||||
assert!(
|
||||
!painted_text(&out).contains(&pw),
|
||||
"the password was painted in the clear: {:?}",
|
||||
painted_text(&out)
|
||||
);
|
||||
|
||||
let submit = painted_text_center(&out, "Show key").expect("no submit button painted");
|
||||
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(submit));
|
||||
assert_eq!(buttons, (true, false));
|
||||
|
||||
let cancel = painted_text_center(&out, "Cancel").expect("no cancel button painted");
|
||||
let (_, buttons) = confirm_frame(&ctx, &mut pw, false, click_at(cancel));
|
||||
assert_eq!(buttons, (false, true));
|
||||
}
|
||||
|
||||
/// A retry has to say why it is asking again, or it reads as the dialog
|
||||
/// having ignored the first attempt.
|
||||
#[test]
|
||||
fn a_retry_says_the_password_was_wrong() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut pw = String::new();
|
||||
let (quiet, _) = confirm_frame(&ctx, &mut pw, false, Vec::new());
|
||||
assert!(
|
||||
!painted_text(&quiet)
|
||||
.iter()
|
||||
.any(|t| t.contains("not correct")),
|
||||
"the first attempt was called wrong before it was made"
|
||||
);
|
||||
|
||||
let (out, _) = confirm_frame(&ctx, &mut pw, true, Vec::new());
|
||||
assert!(
|
||||
painted_text(&out)
|
||||
.iter()
|
||||
.any(|t| t.contains("That password is not correct")),
|
||||
"{:?}",
|
||||
painted_text(&out)
|
||||
);
|
||||
}
|
||||
|
||||
/// The right password derives the installed key, and the key is shown in the
|
||||
/// `0x` form other SQLCipher tools take.
|
||||
#[test]
|
||||
fn the_matching_password_reveals_the_installed_key() {
|
||||
assert_eq!(reveal_display(KEY, KEY), Some(format!("0x{KEY}")));
|
||||
}
|
||||
|
||||
/// A wrong password derives some other key. Nothing about the real one may
|
||||
/// leak from the attempt, so the caller gets no display string at all.
|
||||
#[test]
|
||||
fn a_password_that_derives_another_key_reveals_nothing() {
|
||||
assert_eq!(reveal_display(KEY, OTHER), None);
|
||||
assert_eq!(reveal_display(KEY, ""), None);
|
||||
// A prefix must not pass: the whole key is compared, not the start of it.
|
||||
assert_eq!(reveal_display(KEY, &KEY[..62]), None);
|
||||
// Both sides come from `to_hex`, which is always lowercase, so an
|
||||
// uppercase spelling is a mismatch rather than a value to normalise.
|
||||
assert_eq!(reveal_display(KEY, &KEY.to_uppercase()), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_reveal_shows_the_key_and_what_holding_it_means() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let display = format!("0x{KEY}");
|
||||
let painted = painted_text(&frame(&ctx, &display, Vec::new()).0);
|
||||
|
||||
assert!(
|
||||
painted.contains(&display),
|
||||
"the key itself is not on screen: {painted:?}"
|
||||
);
|
||||
assert!(
|
||||
painted
|
||||
.iter()
|
||||
.any(|t| t.contains("read the index without the password")),
|
||||
"no warning about what the key is: {painted:?}"
|
||||
);
|
||||
assert!(painted.contains(&"Copy".to_string()), "{painted:?}");
|
||||
assert!(painted.contains(&"Close".to_string()), "{painted:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn both_of_the_reveal_buttons_report_their_click() {
|
||||
let display = format!("0x{KEY}");
|
||||
for (label, expected) in [("Copy", (true, false)), ("Close", (false, true))] {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let (out, _) = frame(&ctx, &display, Vec::new());
|
||||
let pos =
|
||||
painted_text_center(&out, label).unwrap_or_else(|| panic!("no {label} button painted"));
|
||||
let (_, buttons) = frame(&ctx, &display, click_at(pos));
|
||||
assert_eq!(
|
||||
buttons, expected,
|
||||
"clicking {label} reported the wrong pair"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The displayed string is the whole key and nothing else: a truncated or
|
||||
/// annotated form would be pasted into other tools and fail there.
|
||||
#[test]
|
||||
fn the_display_form_is_the_prefix_and_the_whole_key() {
|
||||
let display = reveal_display(KEY, KEY).expect("a match reveals");
|
||||
assert_eq!(display.len(), 66);
|
||||
assert!(display.starts_with("0x"));
|
||||
assert!(display[2..].bytes().all(|b| b.is_ascii_hexdigit()));
|
||||
}
|
||||
|
|
@ -261,7 +261,7 @@ mod tests {
|
|||
walked,
|
||||
walk_total,
|
||||
extracted: 0,
|
||||
extract_total: None,
|
||||
extract_total: 0,
|
||||
current_file: None,
|
||||
active_workers: 2,
|
||||
total_workers: 4,
|
||||
|
|
@ -299,11 +299,11 @@ mod tests {
|
|||
|
||||
let mut extracting = root(RootPhase::Extracting, 1_000, None);
|
||||
extracting.extracted = 200;
|
||||
extracting.extract_total = Some(800);
|
||||
extracting.extract_total = 800;
|
||||
extracting.active_workers = 3;
|
||||
let mut done = root(RootPhase::Done, 500, None);
|
||||
done.extracted = 500;
|
||||
done.extract_total = Some(500);
|
||||
done.extract_total = 500;
|
||||
done.active_workers = 0;
|
||||
done.total_workers = 0;
|
||||
assert_eq!(
|
||||
|
|
|
|||
|
|
@ -30,96 +30,61 @@ fn a_stale_draft_cannot_revert_the_indexing_mode_or_security() {
|
|||
}
|
||||
|
||||
/// The guard decision table for leaving a tab, however it is asked for.
|
||||
/// Both draft-backed tabs guard their own departure, and neither answers for
|
||||
/// the other.
|
||||
#[test]
|
||||
fn leaving_a_dirty_editor_tab_is_guarded_however_it_is_asked_for() {
|
||||
// (the tab, whether *its* editor is the dirty one in the pair below)
|
||||
for (tab, manage_dirty, settings_dirty) in
|
||||
[(Tab::Manage, true, false), (Tab::Settings, false, true)]
|
||||
{
|
||||
fn leaving_a_dirty_manage_tab_is_guarded_however_it_is_asked_for() {
|
||||
assert!(switch_needs_guard(Tab::Manage, true, false));
|
||||
assert!(
|
||||
switch_needs_guard(tab, manage_dirty, settings_dirty, false),
|
||||
"{tab:?} must guard its own unapplied edits"
|
||||
!switch_needs_guard(Tab::Manage, false, false),
|
||||
"a clean editor has nothing to ask about"
|
||||
);
|
||||
assert!(
|
||||
!switch_needs_guard(tab, false, false, false),
|
||||
"{tab:?}: a clean editor has nothing to ask about"
|
||||
!switch_needs_guard(Tab::Manage, true, true),
|
||||
"one held navigation at a time"
|
||||
);
|
||||
assert!(
|
||||
!switch_needs_guard(tab, manage_dirty, settings_dirty, true),
|
||||
"{tab:?}: one held navigation at a time"
|
||||
);
|
||||
assert!(
|
||||
!switch_needs_guard(tab, !manage_dirty, !settings_dirty, false),
|
||||
"{tab:?} must not answer for the other editor"
|
||||
);
|
||||
}
|
||||
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
|
||||
assert!(
|
||||
!switch_needs_guard(tab, true, true, false),
|
||||
!switch_needs_guard(tab, true, false),
|
||||
"{tab:?} holds no unapplied edits of its own"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn guard_source_orders_quit_prompts_settings_first() {
|
||||
fn guard_source_orders_quit_prompts_options_first() {
|
||||
use super::NavIntent::*;
|
||||
let leave = SwitchTab(Tab::Search);
|
||||
let tab = SwitchTab(Tab::Search);
|
||||
|
||||
// A switch asks about the tab being left, and only about that one.
|
||||
assert_eq!(guard_source(tab, true, true), Some(UnsavedSource::Manage));
|
||||
assert_eq!(guard_source(tab, true, false), Some(UnsavedSource::Manage));
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Manage, true, true),
|
||||
Some(UnsavedSource::Manage)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Manage, true, false),
|
||||
Some(UnsavedSource::Manage)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Manage, false, true),
|
||||
guard_source(tab, false, true),
|
||||
None,
|
||||
"the Settings draft is not what leaving Manage disturbs"
|
||||
"options guard its own close"
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Settings, true, true),
|
||||
Some(UnsavedSource::Settings)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Settings, false, true),
|
||||
Some(UnsavedSource::Settings)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(leave, Tab::Settings, true, false),
|
||||
None,
|
||||
"the Manage draft is not what leaving Settings disturbs"
|
||||
);
|
||||
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
|
||||
assert_eq!(
|
||||
guard_source(leave, tab, true, true),
|
||||
None,
|
||||
"{tab:?} stages nothing, so leaving it asks nothing"
|
||||
);
|
||||
}
|
||||
assert_eq!(guard_source(tab, false, false), None);
|
||||
|
||||
// Quit asks about both, Settings first.
|
||||
for from in [Tab::Search, Tab::Manage, Tab::Settings] {
|
||||
assert_eq!(
|
||||
guard_source(Quit, from, true, true),
|
||||
Some(UnsavedSource::Settings),
|
||||
"{from:?}: quitting asks about Settings before Manage"
|
||||
guard_source(CloseOptions, true, true),
|
||||
Some(UnsavedSource::Options)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(Quit, from, false, true),
|
||||
Some(UnsavedSource::Settings)
|
||||
guard_source(CloseOptions, false, true),
|
||||
Some(UnsavedSource::Options)
|
||||
);
|
||||
assert_eq!(
|
||||
guard_source(Quit, from, true, false),
|
||||
Some(UnsavedSource::Manage)
|
||||
guard_source(CloseOptions, true, false),
|
||||
None,
|
||||
"manage guards tab switches"
|
||||
);
|
||||
assert_eq!(guard_source(Quit, from, false, false), None);
|
||||
}
|
||||
assert_eq!(guard_source(CloseOptions, false, false), None);
|
||||
|
||||
assert_eq!(guard_source(Quit, true, true), Some(UnsavedSource::Options));
|
||||
assert_eq!(
|
||||
guard_source(Quit, false, true),
|
||||
Some(UnsavedSource::Options)
|
||||
);
|
||||
assert_eq!(guard_source(Quit, true, false), Some(UnsavedSource::Manage));
|
||||
assert_eq!(guard_source(Quit, false, false), None);
|
||||
}
|
||||
|
||||
/// Only a Quit during a running reconcile warns; a tab switch does not end
|
||||
|
|
@ -130,25 +95,10 @@ fn only_quitting_during_a_reconcile_warns() {
|
|||
assert!(quit_needs_reconcile_warning(Quit, true));
|
||||
assert!(!quit_needs_reconcile_warning(Quit, false));
|
||||
assert!(!quit_needs_reconcile_warning(SwitchTab(Tab::Search), true));
|
||||
assert!(!quit_needs_reconcile_warning(
|
||||
SwitchTab(Tab::Settings),
|
||||
true
|
||||
));
|
||||
assert!(!quit_needs_reconcile_warning(CloseOptions, true));
|
||||
}
|
||||
|
||||
/// Every tab is placed on exactly one side of the guard, so a tab added
|
||||
/// later cannot quietly inherit "stages nothing".
|
||||
#[test]
|
||||
fn only_the_two_draft_backed_tabs_have_an_editor() {
|
||||
assert_eq!(tab_editor(Tab::Manage), Some(UnsavedSource::Manage));
|
||||
assert_eq!(tab_editor(Tab::Settings), Some(UnsavedSource::Settings));
|
||||
for tab in [Tab::Search, Tab::Duplicates, Tab::Logs, Tab::Help] {
|
||||
assert_eq!(tab_editor(tab), None, "{tab:?} saves as it goes");
|
||||
}
|
||||
}
|
||||
|
||||
/// The two values the Settings tab's color-scheme box writes, plus
|
||||
/// hand-edited variants.
|
||||
/// The two values the Options window writes, plus hand-edited variants.
|
||||
#[test]
|
||||
fn only_light_is_light() {
|
||||
assert_eq!(theme_for("light"), egui::Theme::Light);
|
||||
|
|
|
|||
|
|
@ -1,239 +0,0 @@
|
|||
//! Byte-for-byte verification of one duplicate group, and the modal that
|
||||
//! reports it.
|
||||
//!
|
||||
//! The Duplicates tab groups by a hash of each file's size and head, which is
|
||||
//! all the indexer ever reads (see [`quicksearch_core::verify`]). This is the
|
||||
//! second opinion, asked for one group at a time, and it exists because the
|
||||
//! action it precedes is usually deletion.
|
||||
|
||||
use super::*;
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use quicksearch_core::verify::{MemberVerdict, VerifyReport, VerifyUpdate};
|
||||
|
||||
use crate::format::{group_thousands, human_size};
|
||||
use crate::ui_util::{centered_modal, hint, progress_bar};
|
||||
|
||||
const MODAL_WIDTH: f32 = 560.0;
|
||||
|
||||
pub(crate) enum VerifyState {
|
||||
Running {
|
||||
bytes_read: u64,
|
||||
/// Zero until the worker's first progress update lands, which is what
|
||||
/// puts the bar in its indeterminate state to begin with.
|
||||
bytes_total: u64,
|
||||
},
|
||||
Done(Box<VerifyReport>),
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
pub(crate) struct VerifyModal {
|
||||
pub paths: Vec<PathBuf>,
|
||||
pub state: VerifyState,
|
||||
}
|
||||
|
||||
impl VerifyModal {
|
||||
pub(crate) fn new(paths: Vec<PathBuf>) -> VerifyModal {
|
||||
VerifyModal {
|
||||
paths,
|
||||
state: VerifyState::Running {
|
||||
bytes_read: 0,
|
||||
bytes_total: 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One line of the report: what happened to `path`, in the words the modal
|
||||
/// paints. Split out from the rendering so the wording is testable without a
|
||||
/// frame.
|
||||
pub(crate) fn verdict_line(verdict: &MemberVerdict, reference: bool) -> String {
|
||||
match verdict {
|
||||
MemberVerdict::Identical if reference => "compared against".to_string(),
|
||||
MemberVerdict::Identical => "identical".to_string(),
|
||||
MemberVerdict::DiffersAt(offset) => {
|
||||
format!("differs at byte {}", group_thousands(*offset))
|
||||
}
|
||||
MemberVerdict::LengthDiffers { len, reference_len } => format!(
|
||||
"size differs: {} against {}",
|
||||
human_size(*len),
|
||||
human_size(*reference_len)
|
||||
),
|
||||
MemberVerdict::Unreadable(e) => format!("could not be read — {e}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The headline the report earns.
|
||||
pub(crate) fn summary_line(report: &VerifyReport) -> String {
|
||||
let total = report.verdicts.len();
|
||||
if report.reference.is_none() {
|
||||
return "None of these files could be read.".to_string();
|
||||
}
|
||||
let differing = report.differing();
|
||||
if differing == 0 {
|
||||
return match total {
|
||||
0 | 1 => "Nothing to compare: the group holds one file.".to_string(),
|
||||
n => format!("All {n} files are byte-for-byte identical."),
|
||||
};
|
||||
}
|
||||
format!(
|
||||
"{} of {} files {} not identical.",
|
||||
differing,
|
||||
total,
|
||||
if differing == 1 { "is" } else { "are" }
|
||||
)
|
||||
}
|
||||
|
||||
impl QuickSearchApp {
|
||||
/// Drain the worker and fold its updates into the modal.
|
||||
pub(super) fn drain_verify(&mut self) {
|
||||
use std::sync::mpsc::TryRecvError;
|
||||
let Some(job) = &self.backend.verify_job else {
|
||||
return;
|
||||
};
|
||||
let mut finished = false;
|
||||
loop {
|
||||
match job.rx.try_recv() {
|
||||
Ok(VerifyUpdate::Progress {
|
||||
bytes_read: read,
|
||||
bytes_total: total,
|
||||
}) => {
|
||||
if let Some(modal) = &mut self.verify {
|
||||
modal.state = VerifyState::Running {
|
||||
bytes_read: read,
|
||||
bytes_total: total,
|
||||
};
|
||||
}
|
||||
}
|
||||
Ok(VerifyUpdate::Done(report)) => {
|
||||
if let Some(modal) = &mut self.verify {
|
||||
modal.state = VerifyState::Done(Box::new(report));
|
||||
}
|
||||
finished = true;
|
||||
break;
|
||||
}
|
||||
Ok(VerifyUpdate::Cancelled) => {
|
||||
if let Some(modal) = &mut self.verify {
|
||||
modal.state = VerifyState::Cancelled;
|
||||
}
|
||||
finished = true;
|
||||
break;
|
||||
}
|
||||
Err(TryRecvError::Empty) => break,
|
||||
// The worker died without a terminal update. Nothing else can
|
||||
// arrive, so say so rather than spinning on an empty channel.
|
||||
Err(TryRecvError::Disconnected) => {
|
||||
if let Some(modal) = &mut self.verify {
|
||||
if matches!(modal.state, VerifyState::Running { .. }) {
|
||||
modal.state = VerifyState::Cancelled;
|
||||
}
|
||||
}
|
||||
finished = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if finished {
|
||||
self.backend.verify_job = None;
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn verify_modal_ui(&mut self, ctx: &egui::Context) {
|
||||
let Some(modal) = &self.verify else {
|
||||
return;
|
||||
};
|
||||
if verify_modal(ctx, modal) {
|
||||
self.backend.cancel_verify();
|
||||
self.verify = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Paint the modal; `true` when its dismiss button was clicked. A free
|
||||
/// function rather than a method so it can be rendered against a bare
|
||||
/// context, without an app and the index behind it.
|
||||
pub(crate) fn verify_modal(ctx: &egui::Context, modal: &VerifyModal) -> bool {
|
||||
centered_modal(ctx, "Verify duplicates", |ui| {
|
||||
ui.set_max_width(MODAL_WIDTH);
|
||||
match &modal.state {
|
||||
VerifyState::Running {
|
||||
bytes_read,
|
||||
bytes_total,
|
||||
} => {
|
||||
ui.label(format!(
|
||||
"Comparing {} files byte for byte…",
|
||||
modal.paths.len()
|
||||
));
|
||||
// No denominator until the first update lands, which is what
|
||||
// the indeterminate bar is for.
|
||||
let fraction =
|
||||
(*bytes_total > 0).then(|| (*bytes_read as f64 / *bytes_total as f64) as f32);
|
||||
progress_bar(ui, fraction, MODAL_WIDTH);
|
||||
ui.label(hint(match bytes_total {
|
||||
0 => format!("{} read", human_size(*bytes_read)),
|
||||
total => format!("{} of {}", human_size(*bytes_read), human_size(*total)),
|
||||
}));
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal(|ui| ui.button("Cancel").clicked()).inner
|
||||
}
|
||||
VerifyState::Cancelled => {
|
||||
ui.label("Verification cancelled.");
|
||||
ui.add_space(6.0);
|
||||
ui.horizontal(|ui| ui.button("Close").clicked()).inner
|
||||
}
|
||||
VerifyState::Done(report) => {
|
||||
let p = crate::color::palette(ui.visuals().dark_mode);
|
||||
let identical = report.all_identical() && report.reference.is_some();
|
||||
let color = if identical {
|
||||
p.green
|
||||
} else {
|
||||
ui.visuals().error_fg_color
|
||||
};
|
||||
ui.colored_label(color, summary_line(report));
|
||||
if !identical {
|
||||
ui.label(hint(
|
||||
"Files are grouped by size and how they begin, which is all \
|
||||
indexing reads. This compared every byte.",
|
||||
));
|
||||
}
|
||||
ui.add_space(6.0);
|
||||
// Listed even when everything matched: it is the record of
|
||||
// what was actually read.
|
||||
egui::ScrollArea::vertical()
|
||||
.max_height(260.0)
|
||||
.auto_shrink([false, true])
|
||||
.show(ui, |ui| {
|
||||
for (i, path) in modal.paths.iter().enumerate() {
|
||||
let Some(verdict) = report.verdicts.get(i) else {
|
||||
continue;
|
||||
};
|
||||
let is_reference = report.reference == Some(i);
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.label(
|
||||
egui::RichText::new(path.display().to_string()).monospace(),
|
||||
);
|
||||
let line = verdict_line(verdict, is_reference);
|
||||
if verdict.is_identical() {
|
||||
ui.label(hint(line));
|
||||
} else {
|
||||
ui.colored_label(
|
||||
ui.visuals().error_fg_color,
|
||||
egui::RichText::new(line).small(),
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
ui.add_space(6.0);
|
||||
ui.label(hint(format!("{} read", human_size(report.bytes_read))));
|
||||
ui.horizontal(|ui| ui.button("Close").clicked()).inner
|
||||
}
|
||||
}
|
||||
})
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "verify_tests.rs"]
|
||||
mod tests;
|
||||
|
|
@ -1,216 +0,0 @@
|
|||
use super::*;
|
||||
|
||||
use quicksearch_core::verify::MemberVerdict::{
|
||||
DiffersAt, Identical, LengthDiffers, Unreadable as CannotRead,
|
||||
};
|
||||
|
||||
use crate::test_ui::{click_at, painted_text, painted_text_center, raw_input};
|
||||
|
||||
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
|
||||
|
||||
fn modal(state: VerifyState, n: usize) -> VerifyModal {
|
||||
VerifyModal {
|
||||
paths: (0..n)
|
||||
.map(|i| PathBuf::from(format!("/d/copy{i}.bin")))
|
||||
.collect(),
|
||||
state,
|
||||
}
|
||||
}
|
||||
|
||||
fn report(verdicts: Vec<MemberVerdict>, bytes_read: u64) -> VerifyState {
|
||||
VerifyState::Done(Box::new(VerifyReport {
|
||||
reference: Some(0),
|
||||
verdicts,
|
||||
bytes_read,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Two passes: an `egui::Window` is measured on its first frame and placed on
|
||||
/// the next, so a single pass paints nothing to read back.
|
||||
fn frame(
|
||||
ctx: &egui::Context,
|
||||
m: &VerifyModal,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (egui::FullOutput, bool) {
|
||||
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
verify_modal(ctx, m);
|
||||
});
|
||||
let mut closed = false;
|
||||
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
|
||||
closed = verify_modal(ctx, m);
|
||||
});
|
||||
(out, closed)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_run_in_progress_says_what_it_is_doing_and_offers_a_way_out() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(
|
||||
VerifyState::Running {
|
||||
bytes_read: 5 * 1024 * 1024,
|
||||
bytes_total: 20 * 1024 * 1024,
|
||||
},
|
||||
3,
|
||||
);
|
||||
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
|
||||
assert!(
|
||||
painted.contains(&"Comparing 3 files byte for byte…".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(
|
||||
painted.iter().any(|t| t.contains("5.2 MB of 21.0 MB")),
|
||||
"no byte counter: {painted:?}"
|
||||
);
|
||||
assert!(painted.contains(&"Cancel".to_string()), "{painted:?}");
|
||||
}
|
||||
|
||||
/// Before the worker's first update there is no denominator, so the modal
|
||||
/// reports what it has rather than dividing by zero.
|
||||
#[test]
|
||||
fn a_run_with_no_denominator_yet_still_reports() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(
|
||||
VerifyState::Running {
|
||||
bytes_read: 0,
|
||||
bytes_total: 0,
|
||||
},
|
||||
2,
|
||||
);
|
||||
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
|
||||
assert!(
|
||||
painted.iter().any(|t| t.contains("0 B read")),
|
||||
"{painted:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_clean_result_says_so_and_lists_what_was_read() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(report(vec![Identical, Identical, Identical], 300), 3);
|
||||
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
|
||||
assert!(
|
||||
painted.contains(&"All 3 files are byte-for-byte identical.".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(painted.contains(&"/d/copy2.bin".to_string()), "{painted:?}");
|
||||
assert!(
|
||||
painted.contains(&"compared against".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(painted.contains(&"Close".to_string()), "{painted:?}");
|
||||
}
|
||||
|
||||
/// The case the feature exists for: same size, same head, different bytes.
|
||||
#[test]
|
||||
fn a_mismatch_names_the_file_and_the_offset() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(report(vec![Identical, DiffersAt(1_234_567)], 2), 2);
|
||||
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
|
||||
assert!(
|
||||
painted.contains(&"1 of 2 files is not identical.".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(
|
||||
painted.contains(&"differs at byte 1,234,567".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(painted.contains(&"/d/copy1.bin".to_string()), "{painted:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_cancelled_run_says_so_rather_than_showing_a_verdict() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(VerifyState::Cancelled, 2);
|
||||
let painted = painted_text(&frame(&ctx, &m, Vec::new()).0);
|
||||
assert!(
|
||||
painted.contains(&"Verification cancelled.".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
assert!(
|
||||
!painted.iter().any(|t| t.contains("identical")),
|
||||
"a cancelled run claimed a verdict: {painted:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Every state's dismiss button reports the dismissal, whatever it is called.
|
||||
#[test]
|
||||
fn both_dismiss_buttons_report_the_dismissal() {
|
||||
for (label, state) in [
|
||||
(
|
||||
"Cancel",
|
||||
VerifyState::Running {
|
||||
bytes_read: 1,
|
||||
bytes_total: 2,
|
||||
},
|
||||
),
|
||||
("Close", VerifyState::Cancelled),
|
||||
("Close", report(vec![Identical, Identical], 8)),
|
||||
] {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let m = modal(state, 2);
|
||||
let (out, _) = frame(&ctx, &m, Vec::new());
|
||||
let pos =
|
||||
painted_text_center(&out, label).unwrap_or_else(|| panic!("no {label} button painted"));
|
||||
let (_, closed) = frame(&ctx, &m, click_at(pos));
|
||||
assert!(closed, "clicking {label} did not dismiss the modal");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_verdict_reads_as_a_sentence_about_the_file() {
|
||||
assert_eq!(verdict_line(&Identical, false), "identical");
|
||||
assert_eq!(verdict_line(&Identical, true), "compared against");
|
||||
assert_eq!(verdict_line(&DiffersAt(0), false), "differs at byte 0");
|
||||
assert_eq!(
|
||||
verdict_line(&DiffersAt(1_048_576), false),
|
||||
"differs at byte 1,048,576"
|
||||
);
|
||||
assert_eq!(
|
||||
verdict_line(
|
||||
&LengthDiffers {
|
||||
len: 2048,
|
||||
reference_len: 1024
|
||||
},
|
||||
false
|
||||
),
|
||||
"size differs: 2.0 KB against 1.0 KB"
|
||||
);
|
||||
assert!(verdict_line(&CannotRead("/d/x: denied".into()), false)
|
||||
.contains("could not be read — /d/x: denied"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_summary_counts_what_it_found() {
|
||||
let of = |verdicts: Vec<MemberVerdict>, reference| {
|
||||
summary_line(&VerifyReport {
|
||||
reference,
|
||||
verdicts,
|
||||
bytes_read: 0,
|
||||
})
|
||||
};
|
||||
assert_eq!(
|
||||
of(vec![Identical, Identical], Some(0)),
|
||||
"All 2 files are byte-for-byte identical."
|
||||
);
|
||||
assert_eq!(
|
||||
of(vec![Identical, DiffersAt(4)], Some(0)),
|
||||
"1 of 2 files is not identical."
|
||||
);
|
||||
assert_eq!(
|
||||
of(vec![Identical, DiffersAt(4), DiffersAt(9)], Some(0)),
|
||||
"2 of 3 files are not identical."
|
||||
);
|
||||
// A group of one cannot disagree with itself, and saying "all 1 files are
|
||||
// identical" would read as an answer to a question nobody asked.
|
||||
assert_eq!(
|
||||
of(vec![Identical], Some(0)),
|
||||
"Nothing to compare: the group holds one file."
|
||||
);
|
||||
assert_eq!(
|
||||
of(
|
||||
vec![CannotRead("gone".into()), CannotRead("gone".into())],
|
||||
None
|
||||
),
|
||||
"None of these files could be read."
|
||||
);
|
||||
}
|
||||
|
|
@ -6,48 +6,22 @@
|
|||
//! Every core thread wakes the UI through `ctx.request_repaint()`, which is
|
||||
//! what makes polling enough.
|
||||
//!
|
||||
//! The duplicates scan and the byte-for-byte verification of one of its
|
||||
//! groups are the throwaway threads, and both fire on a user action rather
|
||||
//! than a timer: a thread per refresh opens its own connection — a page cache
|
||||
//! and an allocator arena glibc never gives back. The verification opens no
|
||||
//! connection at all, but it can hold a large group's worth of file handles,
|
||||
//! so it carries a cancel flag and shutdown raises it.
|
||||
//! The duplicates scan is the only throwaway thread, and it fires on a user
|
||||
//! action, not a timer: a thread per refresh opens its own connection — a
|
||||
//! page cache and an allocator arena glibc never gives back.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{mpsc, Arc};
|
||||
|
||||
use quicksearch_core::config::Config;
|
||||
use quicksearch_core::coordinator::IndexCoordinator;
|
||||
use quicksearch_core::live::{LiveUpdate, LiveWatcher};
|
||||
use quicksearch_core::search::{DuplicateGroup, SearchService, SearchUpdate};
|
||||
use quicksearch_core::shutdown;
|
||||
use quicksearch_core::verify::{verify_identical, VerifyUpdate};
|
||||
|
||||
/// A duplicate group being read through. The thread is detached and owns
|
||||
/// nothing the app needs back, so cancelling is just raising the flag: the
|
||||
/// worker notices between chunks and drops the receiver's other end.
|
||||
pub struct VerifyJob {
|
||||
pub rx: mpsc::Receiver<VerifyUpdate>,
|
||||
cancel: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
impl VerifyJob {
|
||||
pub fn cancel(&self) {
|
||||
self.cancel.store(true, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
pub struct Backend {
|
||||
pub coordinator: Arc<IndexCoordinator>,
|
||||
pub search: Option<SearchService>,
|
||||
pub search_rx: mpsc::Receiver<SearchUpdate>,
|
||||
pub dup_job: Option<mpsc::Receiver<Result<Vec<DuplicateGroup>, String>>>,
|
||||
pub verify_job: Option<VerifyJob>,
|
||||
/// Watches the results currently on screen; see [`quicksearch_core::live`].
|
||||
/// `None` only after [`Backend::shutdown`].
|
||||
pub live: Option<LiveWatcher>,
|
||||
pub live_rx: mpsc::Receiver<LiveUpdate>,
|
||||
}
|
||||
|
||||
impl Backend {
|
||||
|
|
@ -70,49 +44,14 @@ impl Backend {
|
|||
Arc::new(move || repaint_ctx.request_repaint()),
|
||||
);
|
||||
|
||||
let live_ctx = ctx.clone();
|
||||
let (live, live_rx) = LiveWatcher::start(Arc::new(move || live_ctx.request_repaint()));
|
||||
|
||||
Ok(Backend {
|
||||
coordinator,
|
||||
search: Some(search),
|
||||
search_rx,
|
||||
dup_job: None,
|
||||
verify_job: None,
|
||||
live: Some(live),
|
||||
live_rx,
|
||||
})
|
||||
}
|
||||
|
||||
/// Point the live watcher at the rows currently on screen, or clear it
|
||||
/// with an empty `targets`.
|
||||
pub fn watch_live(
|
||||
&self,
|
||||
query: &str,
|
||||
targets: Vec<quicksearch_core::live::Target>,
|
||||
config: &Config,
|
||||
) {
|
||||
let Some(live) = &self.live else { return };
|
||||
if targets.is_empty() {
|
||||
live.clear();
|
||||
} else {
|
||||
live.watch(query, targets, config);
|
||||
}
|
||||
}
|
||||
|
||||
/// Ask the coordinator to bring the index in line with these paths — the
|
||||
/// files the live watcher has just read from disk on the frontend's
|
||||
/// behalf, so the index does not drift from what is on screen.
|
||||
pub fn reindex_live_paths(&self, paths: Vec<PathBuf>) {
|
||||
self.coordinator.update_paths(paths);
|
||||
}
|
||||
|
||||
pub fn clear_live(&self) {
|
||||
if let Some(live) = &self.live {
|
||||
live.clear();
|
||||
}
|
||||
}
|
||||
|
||||
/// `None` only after [`Backend::shutdown`], i.e. during teardown frames.
|
||||
pub fn search(&self) -> Option<&SearchService> {
|
||||
self.search.as_ref()
|
||||
|
|
@ -131,46 +70,12 @@ impl Backend {
|
|||
self.dup_job = Some(rx);
|
||||
}
|
||||
|
||||
/// Read a duplicate group through on a worker thread, comparing every
|
||||
/// member against the first byte for byte. Replaces any run already going.
|
||||
pub fn start_verify(&mut self, paths: Vec<PathBuf>, ctx: egui::Context) {
|
||||
if let Some(job) = &self.verify_job {
|
||||
job.cancel();
|
||||
}
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
let worker_cancel = cancel.clone();
|
||||
std::thread::spawn(move || {
|
||||
verify_identical(&paths, &worker_cancel, &mut |update| {
|
||||
// A closed receiver means the app moved on; the cancel flag
|
||||
// is what stops the work, so there is nothing to do here.
|
||||
let _ = tx.send(update);
|
||||
ctx.request_repaint();
|
||||
});
|
||||
});
|
||||
self.verify_job = Some(VerifyJob { rx, cancel });
|
||||
}
|
||||
|
||||
/// Stop a verification and forget it. The worker sees the flag between
|
||||
/// chunks and exits on its own.
|
||||
pub fn cancel_verify(&mut self) {
|
||||
if let Some(job) = self.verify_job.take() {
|
||||
job.cancel();
|
||||
}
|
||||
}
|
||||
|
||||
/// Join the search worker and stop the coordinator. Called once from
|
||||
/// `on_exit`.
|
||||
pub fn shutdown(&mut self) {
|
||||
// Detached and holding open file handles: the flag is what makes a
|
||||
// verification of a slow, large group let go on the way out.
|
||||
self.cancel_verify();
|
||||
if let Some(search) = self.search.take() {
|
||||
search.shutdown();
|
||||
}
|
||||
if let Some(mut live) = self.live.take() {
|
||||
live.stop();
|
||||
}
|
||||
self.coordinator.shutdown();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -82,7 +82,7 @@ pub(crate) struct CaptureDriver {
|
|||
/// Screenshot in flight: requested, PNG not yet written.
|
||||
shot: Option<PathBuf>,
|
||||
rec: Option<Recorder>,
|
||||
/// Content Match cell row the pointer is pinned to (`hover_match`), and the
|
||||
/// Match-cell row the pointer is pinned to (`hover_match`), and the
|
||||
/// on-screen position it resolved to on the last rendered frame.
|
||||
hover: Option<usize>,
|
||||
hover_pos: Option<egui::Pos2>,
|
||||
|
|
|
|||
|
|
@ -11,11 +11,9 @@ use crate::app::Tab;
|
|||
/// window INT INT # resize to width x height, in the same
|
||||
/// # logical points as the startup size
|
||||
/// hover_match INT # pin the pointer over the Nth visible
|
||||
/// # Content Match cell (0-based) until
|
||||
/// # hover_off. Counts every visible row,
|
||||
/// # including those showing a dash.
|
||||
/// # Match cell (0-based) until hover_off
|
||||
/// hover_off # release the pinned pointer
|
||||
/// tab (search|manage|duplicates|logs|help|settings)
|
||||
/// tab (search|manage|duplicates|logs|help)
|
||||
/// wait_index_running [max INT] # caps in ms; a capped wait cannot fail
|
||||
/// wait_index_idle [max INT]
|
||||
/// wait_search_done [max INT]
|
||||
|
|
@ -189,11 +187,9 @@ fn parse_line(tokens: &[Token], line_no: usize) -> Result<Option<Cmd>, ParseErro
|
|||
"duplicates" => Tab::Duplicates,
|
||||
"logs" => Tab::Logs,
|
||||
"help" => Tab::Help,
|
||||
"settings" => Tab::Settings,
|
||||
other => {
|
||||
return Err(err(format!(
|
||||
"unknown tab {other:?}: expected search, manage, duplicates, \
|
||||
logs, help or settings"
|
||||
"unknown tab {other:?}: expected search, manage, duplicates, logs or help"
|
||||
)));
|
||||
}
|
||||
}),
|
||||
|
|
@ -306,7 +302,6 @@ mod tests {
|
|||
tab duplicates
|
||||
tab logs
|
||||
tab help
|
||||
tab settings
|
||||
wait_index_running
|
||||
wait_index_running max 15000
|
||||
wait_index_idle max 13000
|
||||
|
|
@ -340,7 +335,6 @@ mod tests {
|
|||
Cmd::Tab(Tab::Duplicates),
|
||||
Cmd::Tab(Tab::Logs),
|
||||
Cmd::Tab(Tab::Help),
|
||||
Cmd::Tab(Tab::Settings),
|
||||
Cmd::WaitIndexRunning { max_ms: None },
|
||||
Cmd::WaitIndexRunning {
|
||||
max_ms: Some(15000)
|
||||
|
|
@ -453,7 +447,7 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn unknown_tab_and_bad_cps_are_rejected() {
|
||||
assert!(parse_err("tab preferences").msg.contains("unknown tab"));
|
||||
assert!(parse_err("tab settings").msg.contains("unknown tab"));
|
||||
assert!(parse_err(r#"type "x" cps 0"#).msg.contains("positive"));
|
||||
assert!(parse_err(r#"type "x" cps -3"#).msg.contains("positive"));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,18 +21,8 @@ pub struct DuplicatesTab {
|
|||
#[derive(Default)]
|
||||
pub struct DuplicatesActions {
|
||||
pub refresh: bool,
|
||||
/// Every member of one group, to be read through and compared byte for
|
||||
/// byte. Group-scoped whichever row it was asked for from: the question
|
||||
/// "is this row really a duplicate" is a question about the group.
|
||||
pub verify: Option<Vec<String>>,
|
||||
}
|
||||
|
||||
/// The entry both context menus carry. Named for what it settles, since the
|
||||
/// grouping itself never claimed more than a shared size and head.
|
||||
const VERIFY_LABEL: &str = "Verify copies are identical…";
|
||||
const VERIFY_TIP: &str = "Reads every file in the group in full and compares them byte for \
|
||||
byte. Grouping only reads each file's size and how it begins.";
|
||||
|
||||
impl DuplicatesTab {
|
||||
pub fn new() -> DuplicatesTab {
|
||||
DuplicatesTab {
|
||||
|
|
@ -40,10 +30,7 @@ impl DuplicatesTab {
|
|||
}
|
||||
}
|
||||
|
||||
/// `verify_open` is the verification window being up — running or showing
|
||||
/// a result. There is one of it, so the entry greys out rather than
|
||||
/// replacing what someone is reading.
|
||||
pub fn ui(&mut self, ui: &mut egui::Ui, verify_open: bool) -> DuplicatesActions {
|
||||
pub fn ui(&mut self, ui: &mut egui::Ui) -> DuplicatesActions {
|
||||
let mut actions = DuplicatesActions::default();
|
||||
|
||||
ui.horizontal(|ui| {
|
||||
|
|
@ -97,7 +84,6 @@ impl DuplicatesTab {
|
|||
human_size(group.redundant_size.max(0) as u64),
|
||||
human_size(group.total_size.max(0) as u64),
|
||||
);
|
||||
let header =
|
||||
egui::CollapsingHeader::new(title)
|
||||
.id_salt(i)
|
||||
.show(ui, |ui| {
|
||||
|
|
@ -114,31 +100,18 @@ impl DuplicatesTab {
|
|||
platform::open_file(path);
|
||||
}
|
||||
response.context_menu(|ui| {
|
||||
if ui.button("Open File").clicked() {
|
||||
if ui.button("Open").clicked() {
|
||||
platform::open_file(path);
|
||||
ui.close();
|
||||
}
|
||||
if ui.button("Open containing folder").clicked()
|
||||
{
|
||||
if ui.button("Open containing folder").clicked() {
|
||||
platform::reveal_in_folder(path);
|
||||
ui.close();
|
||||
}
|
||||
ui.separator();
|
||||
if verify_entry(ui, verify_open) {
|
||||
actions.verify = Some(member_paths(group));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
});
|
||||
// Also on the group's own row: the question is
|
||||
// about the group, and the rows it is about are
|
||||
// behind a collapsed header until they are not.
|
||||
header.header_response.context_menu(|ui| {
|
||||
if verify_entry(ui, verify_open) {
|
||||
actions.verify = Some(member_paths(group));
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
crate::ui_util::more_below_hint(ui, &scroll);
|
||||
|
|
@ -147,24 +120,3 @@ impl DuplicatesTab {
|
|||
actions
|
||||
}
|
||||
}
|
||||
|
||||
/// The shared menu entry. Returns whether it was clicked, and closes the menu
|
||||
/// when it was.
|
||||
fn verify_entry(ui: &mut egui::Ui, open: bool) -> bool {
|
||||
let clicked = ui
|
||||
.add_enabled(!open, egui::Button::new(VERIFY_LABEL))
|
||||
.on_hover_text(VERIFY_TIP)
|
||||
.on_disabled_hover_text("Close the verification window first.")
|
||||
.clicked();
|
||||
if clicked {
|
||||
ui.close();
|
||||
}
|
||||
clicked
|
||||
}
|
||||
|
||||
fn member_paths(group: &DuplicateGroup) -> Vec<String> {
|
||||
group.members.iter().map(|m| m.2.clone()).collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
|
|
|||
|
|
@ -1,187 +0,0 @@
|
|||
use super::*;
|
||||
|
||||
use crate::test_ui::{painted_text, painted_text_center, raw_input};
|
||||
|
||||
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
|
||||
|
||||
fn group(paths: &[&str]) -> DuplicateGroup {
|
||||
DuplicateGroup {
|
||||
hash: vec![0xab; 32],
|
||||
count: paths.len() as i64,
|
||||
total_size: 100 * paths.len() as i64,
|
||||
redundant_size: 100 * (paths.len() as i64 - 1),
|
||||
members: paths
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, p)| {
|
||||
let name = p.rsplit('/').next().unwrap_or(p).to_string();
|
||||
(i as i64, name, p.to_string(), 100u64, 1_700_000_000i64)
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn loaded(paths: &[&str]) -> DuplicatesTab {
|
||||
DuplicatesTab {
|
||||
state: DupState::Loaded(vec![group(paths)]),
|
||||
}
|
||||
}
|
||||
|
||||
fn frame(
|
||||
ctx: &egui::Context,
|
||||
tab: &mut DuplicatesTab,
|
||||
busy: bool,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (egui::FullOutput, DuplicatesActions) {
|
||||
let mut actions = DuplicatesActions::default();
|
||||
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
actions = tab.ui(ui, busy);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &out);
|
||||
(out, actions)
|
||||
}
|
||||
|
||||
fn click(pos: egui::Pos2, button: egui::PointerButton) -> Vec<egui::Event> {
|
||||
let mut events = vec![egui::Event::PointerMoved(pos)];
|
||||
events.extend(
|
||||
[true, false]
|
||||
.into_iter()
|
||||
.map(|pressed| egui::Event::PointerButton {
|
||||
pos,
|
||||
button,
|
||||
pressed,
|
||||
modifiers: egui::Modifiers::default(),
|
||||
}),
|
||||
);
|
||||
events
|
||||
}
|
||||
|
||||
/// Right-click `needle` and return what the menu it opened painted, plus the
|
||||
/// actions from that frame.
|
||||
fn context_menu_on(
|
||||
ctx: &egui::Context,
|
||||
tab: &mut DuplicatesTab,
|
||||
busy: bool,
|
||||
needle: &str,
|
||||
) -> (Vec<String>, egui::Pos2) {
|
||||
let (out, _) = frame(ctx, tab, busy, Vec::new());
|
||||
let target = painted_text_center(&out, needle)
|
||||
.unwrap_or_else(|| panic!("nothing painted for {needle:?}"));
|
||||
frame(
|
||||
ctx,
|
||||
tab,
|
||||
busy,
|
||||
click(target, egui::PointerButton::Secondary),
|
||||
);
|
||||
// The menu is its own area, painted on the frame after the click.
|
||||
let (out, _) = frame(ctx, tab, busy, Vec::new());
|
||||
(painted_text(&out), target)
|
||||
}
|
||||
|
||||
/// The title line carries the group; find it without rebuilding its wording.
|
||||
fn header_of(ctx: &egui::Context, tab: &mut DuplicatesTab) -> String {
|
||||
let (out, _) = frame(ctx, tab, false, Vec::new());
|
||||
painted_text(&out)
|
||||
.into_iter()
|
||||
.find(|t| t.contains("reclaimable"))
|
||||
.expect("no group header painted")
|
||||
}
|
||||
|
||||
const PATHS: [&str; 3] = ["/a/img.raw", "/b/img.raw", "/c/img.raw"];
|
||||
|
||||
#[test]
|
||||
fn a_group_header_offers_the_verification() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tab = loaded(&PATHS);
|
||||
let header = header_of(&ctx, &mut tab);
|
||||
let (menu, _) = context_menu_on(&ctx, &mut tab, false, &header);
|
||||
assert!(
|
||||
menu.contains(&VERIFY_LABEL.to_string()),
|
||||
"the group's own row does not offer it: {menu:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Clicking it asks for the whole group, not the one row it was asked from.
|
||||
#[test]
|
||||
fn verifying_asks_for_every_member_of_the_group() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tab = loaded(&PATHS);
|
||||
let header = header_of(&ctx, &mut tab);
|
||||
context_menu_on(&ctx, &mut tab, false, &header);
|
||||
|
||||
let (out, _) = frame(&ctx, &mut tab, false, Vec::new());
|
||||
let entry = painted_text_center(&out, VERIFY_LABEL).expect("no verify entry painted");
|
||||
let (_, actions) = frame(
|
||||
&ctx,
|
||||
&mut tab,
|
||||
false,
|
||||
click(entry, egui::PointerButton::Primary),
|
||||
);
|
||||
assert_eq!(
|
||||
actions.verify,
|
||||
Some(PATHS.iter().map(|p| p.to_string()).collect::<Vec<_>>())
|
||||
);
|
||||
}
|
||||
|
||||
/// There is one verification window, so a second run is refused where it is
|
||||
/// asked for rather than replacing what someone is reading.
|
||||
#[test]
|
||||
fn a_second_verification_is_refused_while_the_window_is_open() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tab = loaded(&PATHS);
|
||||
let header = header_of(&ctx, &mut tab);
|
||||
context_menu_on(&ctx, &mut tab, true, &header);
|
||||
|
||||
let (out, _) = frame(&ctx, &mut tab, true, Vec::new());
|
||||
let entry = painted_text_center(&out, VERIFY_LABEL).expect("the entry should still be listed");
|
||||
let (_, actions) = frame(
|
||||
&ctx,
|
||||
&mut tab,
|
||||
true,
|
||||
click(entry, egui::PointerButton::Primary),
|
||||
);
|
||||
assert_eq!(actions.verify, None, "a disabled entry still fired");
|
||||
}
|
||||
|
||||
/// Expanding a group and right-clicking one of its files offers the same
|
||||
/// thing: the rows are what someone is looking at when the question occurs.
|
||||
#[test]
|
||||
fn a_member_row_offers_the_verification_too() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tab = loaded(&PATHS);
|
||||
let header = header_of(&ctx, &mut tab);
|
||||
|
||||
let (out, _) = frame(&ctx, &mut tab, false, Vec::new());
|
||||
let pos = painted_text_center(&out, &header).expect("no header painted");
|
||||
frame(
|
||||
&ctx,
|
||||
&mut tab,
|
||||
false,
|
||||
click(pos, egui::PointerButton::Primary),
|
||||
);
|
||||
|
||||
let (menu, _) = context_menu_on(&ctx, &mut tab, false, PATHS[1]);
|
||||
assert!(
|
||||
menu.contains(&VERIFY_LABEL.to_string()),
|
||||
"an expanded member row does not offer it: {menu:?}"
|
||||
);
|
||||
assert!(
|
||||
menu.contains(&"Open File".to_string()),
|
||||
"the existing entries went missing: {menu:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_empty_result_says_so_rather_than_showing_an_empty_list() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tab = DuplicatesTab {
|
||||
state: DupState::Loaded(Vec::new()),
|
||||
};
|
||||
let painted = painted_text(&frame(&ctx, &mut tab, false, Vec::new()).0);
|
||||
assert!(
|
||||
painted.contains(&"No duplicate files found.".to_string()),
|
||||
"{painted:?}"
|
||||
);
|
||||
}
|
||||
|
|
@ -1,119 +0,0 @@
|
|||
//! The app's own font set.
|
||||
//!
|
||||
//! `egui` is declared with `default-features = false`, so it embeds no fonts
|
||||
//! at all and `FontDefinitions::default()` is `empty()`. Everything the app
|
||||
//! paints comes from the two faces below.
|
||||
//!
|
||||
//! They are the same two files egui would have bundled, copied verbatim and
|
||||
//! carrying no `FontTweak` — exactly as epaint registers them — so metrics and
|
||||
//! rendering are unchanged. What is gone is egui's two *emoji* faces,
|
||||
//! `NotoEmoji-Regular` (418,804) and `emoji-icon-font` (317,864): 736,668 bytes
|
||||
//! of `.rodata`, a fifth of the section, for four glyphs the UI used and none
|
||||
//! it needed. Those four are now `↻`, `×` and a colour (see `search_tab` and
|
||||
//! `manage_tab`).
|
||||
//!
|
||||
//! The faces have no CJK, Hebrew, Arabic, Devanagari or Hangul and never did,
|
||||
//! so filenames in those scripts render as `◻` here just as they always have.
|
||||
//! Emoji in filenames now join them; that is the one real regression.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Proportional body text. Ubuntu Font Licence 1.0, unmodified —
|
||||
/// `assets/fonts/UFL.txt`.
|
||||
const UBUNTU_LIGHT: &[u8] = include_bytes!("../assets/fonts/Ubuntu-Light.ttf");
|
||||
|
||||
/// Monospace: paths, snippets, keys. MIT over public-domain DejaVu over the
|
||||
/// Bitstream Vera licence — `assets/fonts/Hack-Regular.txt`.
|
||||
const HACK_REGULAR: &[u8] = include_bytes!("../assets/fonts/Hack-Regular.ttf");
|
||||
|
||||
/// Install the two faces on `ctx`.
|
||||
///
|
||||
/// `Context::set_fonts` only queues the definitions; they are applied in
|
||||
/// `begin_pass`, before any user code of that frame runs. So this takes effect
|
||||
/// on frame 1 wherever it is called, as long as it is called before the first
|
||||
/// `Context::run`.
|
||||
pub fn install(ctx: &egui::Context) {
|
||||
let mut fonts = egui::FontDefinitions::empty();
|
||||
|
||||
fonts.font_data.insert(
|
||||
"Ubuntu-Light".to_owned(),
|
||||
Arc::new(egui::FontData::from_static(UBUNTU_LIGHT)),
|
||||
);
|
||||
fonts.font_data.insert(
|
||||
"Hack".to_owned(),
|
||||
Arc::new(egui::FontData::from_static(HACK_REGULAR)),
|
||||
);
|
||||
|
||||
// Hack trails Ubuntu-Light in *both* families, which egui's own defaults do
|
||||
// not do — there the proportional fallbacks are the two emoji faces. The
|
||||
// bytes are linked either way, and it buys two things. It is the only
|
||||
// remaining source of `◻`, the replacement glyph epaint reaches for when
|
||||
// nothing has the character; without it proportional text falls back to a
|
||||
// bare `?`. And it covers Greek, Cyrillic, Armenian, Georgian, arrows and
|
||||
// box drawing that Ubuntu-Light lacks, all of which turn up in a filename.
|
||||
// Armenian and Georgian names in fact render here for the first time.
|
||||
fonts.families.insert(
|
||||
egui::FontFamily::Proportional,
|
||||
vec!["Ubuntu-Light".to_owned(), "Hack".to_owned()],
|
||||
);
|
||||
// Ubuntu-Light second, as epaint has it: "fallback for √ etc".
|
||||
fonts.families.insert(
|
||||
egui::FontFamily::Monospace,
|
||||
vec!["Hack".to_owned(), "Ubuntu-Light".to_owned()],
|
||||
);
|
||||
|
||||
ctx.set_fonts(fonts);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
/// A tripwire on the Cargo feature, not on this module.
|
||||
///
|
||||
/// `builtin_font_names` is `&[]` exactly when `epaint/default_fonts` is
|
||||
/// off, so this fails the moment some dependency edge unions the feature
|
||||
/// back on and relinks all 1,407,752 bytes of bundled TTFs. Feature
|
||||
/// unification is silent and is precisely how they got in.
|
||||
/// The coverage contract the UI now depends on, and the proof that
|
||||
/// `assert_no_tofu` can fail: the glyphs the tabs paint resolve, and the
|
||||
/// emoji that used to come from the two dropped faces do not.
|
||||
///
|
||||
/// `↻` and `×` are the two characters the Search tab was moved onto; the
|
||||
/// rest are what the status lines and buttons paint. `↻` is the load-bearing
|
||||
/// one for *Proportional*: U+21BB is in Hack and not in Ubuntu-Light, so it
|
||||
/// passing here is the proof that Hack really is in the proportional
|
||||
/// fallback chain — which is what keeps `◻` available and stops epaint
|
||||
/// degrading to a bare `?`.
|
||||
///
|
||||
/// `◻` itself cannot be asserted: `has_glyph` is
|
||||
/// `glyph_info(c) != replacement_glyph`, so the replacement glyph always
|
||||
/// reports as missing. That is epaint's own documented quirk, not ours.
|
||||
#[test]
|
||||
fn the_installed_faces_cover_what_the_ui_paints_and_no_more() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
// `Context::fonts` panics until the first pass has run.
|
||||
let _ = ctx.run(egui::RawInput::default(), |_| {});
|
||||
|
||||
for family in [egui::FontFamily::Proportional, egui::FontFamily::Monospace] {
|
||||
let id = egui::FontId::new(14.0, family.clone());
|
||||
ctx.fonts(|fonts| {
|
||||
for c in ['↻', '×', '·', '…', '•', '−', '°', 'é', 'Ω', 'д'] {
|
||||
assert!(fonts.has_glyph(&id, c), "{family:?} lost {c:?}");
|
||||
}
|
||||
// Dropped with NotoEmoji and emoji-icon-font. If these start
|
||||
// resolving, the 736,668 bytes are back.
|
||||
for c in ['⟳', '🗙', '⚠', '🔥', '📋'] {
|
||||
assert!(!fonts.has_glyph(&id, c), "{family:?} still has {c:?}");
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn egui_bundles_no_fonts_of_its_own() {
|
||||
assert!(
|
||||
egui::FontDefinitions::builtin_font_names().is_empty(),
|
||||
"egui's `default_fonts` feature is back on: {:?}",
|
||||
egui::FontDefinitions::builtin_font_names()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,9 +1,7 @@
|
|||
//! The Help tab: a quickstart guide for first-time users. The complete
|
||||
//! technical reference stays in README.md.
|
||||
|
||||
/// Returns true when the "Show the introduction again" button was clicked.
|
||||
pub fn ui(ui: &mut egui::Ui) -> bool {
|
||||
let mut replay = false;
|
||||
pub fn ui(ui: &mut egui::Ui) {
|
||||
let scroll = egui::ScrollArea::vertical()
|
||||
.auto_shrink([false; 2])
|
||||
.show(ui, |ui| {
|
||||
|
|
@ -18,11 +16,6 @@ pub fn ui(ui: &mut egui::Ui) -> bool {
|
|||
files by name and by what is inside them, as you type.",
|
||||
);
|
||||
|
||||
ui.add_space(6.0);
|
||||
if ui.button("Show the introduction again").clicked() {
|
||||
replay = true;
|
||||
}
|
||||
|
||||
ui.add_space(12.0);
|
||||
ui.heading(egui::RichText::new("Getting started").strong());
|
||||
ui.add_space(4.0);
|
||||
|
|
@ -51,26 +44,23 @@ pub fn ui(ui: &mut egui::Ui) -> bool {
|
|||
words:",
|
||||
);
|
||||
ui.monospace("type:Document modified:>=2024-01-01 report");
|
||||
ui.label("The ? button left of the search box shows the full query syntax.");
|
||||
ui.label("The ? button next to the search box shows the full query syntax.");
|
||||
ui.add_space(6.0);
|
||||
ui.label(
|
||||
"• Tick Fuzzy to also find matches with typos in them, at some \
|
||||
cost in speed.",
|
||||
);
|
||||
ui.label(
|
||||
"• Click a column header to sort the results; click it again to \
|
||||
reverse the order. Right-click any header to choose which \
|
||||
columns are shown — size and modified date start hidden.",
|
||||
"• Click a column header — Name, Path, Size, Modified, Rank — to \
|
||||
sort the results; click it again to reverse the order.",
|
||||
);
|
||||
ui.label(
|
||||
"• Right-click a result to open it, open its containing folder, \
|
||||
or hide files like it from the results.",
|
||||
);
|
||||
ui.label(
|
||||
"• A match in a file's name or path is highlighted in that \
|
||||
column; a match in its contents shows a snippet of the \
|
||||
surrounding text in the Content Match column, with the rest on \
|
||||
hover.",
|
||||
"• Matches inside a file's contents show a snippet of the \
|
||||
surrounding text under the file name.",
|
||||
);
|
||||
|
||||
ui.add_space(12.0);
|
||||
|
|
@ -102,12 +92,7 @@ pub fn ui(ui: &mut egui::Ui) -> bool {
|
|||
"warnings from indexing and folder watching that a \
|
||||
terminal would have shown",
|
||||
);
|
||||
row(
|
||||
ui,
|
||||
"Settings",
|
||||
"everything QuickSearch can be told to do, in one \
|
||||
place; hover any control for what it means",
|
||||
);
|
||||
row(ui, "⚙ (top right)", "application options");
|
||||
});
|
||||
|
||||
ui.add_space(12.0);
|
||||
|
|
@ -146,7 +131,6 @@ pub fn ui(ui: &mut egui::Ui) -> bool {
|
|||
});
|
||||
});
|
||||
crate::ui_util::more_below_hint(ui, &scroll);
|
||||
replay
|
||||
}
|
||||
|
||||
/// Where this build left the README: under the install prefix's `share/doc`
|
||||
|
|
@ -164,26 +148,3 @@ fn readme_path() -> Option<std::path::PathBuf> {
|
|||
.chain(dir.ancestors().take(4).map(|d| d.join("README.md")))
|
||||
.find(|p| p.is_file())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
/// The Help tab is the largest block of prose in the app and the only tab
|
||||
/// with no other test, which makes it the widest net for the one thing
|
||||
/// dropping egui's emoji faces could break: a character with no glyph,
|
||||
/// painted as `◻`.
|
||||
#[test]
|
||||
fn the_help_tab_paints_no_missing_glyphs() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), vec![]);
|
||||
let out = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
super::ui(ui);
|
||||
});
|
||||
});
|
||||
assert!(
|
||||
!crate::test_ui::painted_text(&out).is_empty(),
|
||||
"the tab painted nothing, so the glyph check proves nothing"
|
||||
);
|
||||
crate::test_ui::assert_no_tofu(&ctx, &out);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,10 +2,10 @@
|
|||
//! produce.
|
||||
//!
|
||||
//! A shortcut is written in three different vocabularies before it reaches an
|
||||
//! operating system: the text in `config.toml` and on the Settings tab's
|
||||
//! button, the token `global-hotkey` parses for `RegisterHotKey`/`XGrabKey`,
|
||||
//! and the xkbcommon keysym name the XDG *shortcuts* specification wants for
|
||||
//! the Wayland portal. All three come out of [`KEYS`], so a key cannot be
|
||||
//! operating system: the text in `config.toml` and on the Options button, the
|
||||
//! token `global-hotkey` parses for `RegisterHotKey`/`XGrabKey`, and the
|
||||
//! xkbcommon keysym name the XDG *shortcuts* specification wants for the
|
||||
//! Wayland portal. All three come out of [`KEYS`], so a key cannot be
|
||||
//! spelled correctly for one backend and wrongly for the other.
|
||||
//!
|
||||
//! The config text and the `global-hotkey` token are the same string: every
|
||||
|
|
@ -103,7 +103,7 @@ const KEYS: &[(Key, &str, &str)] = &[
|
|||
(Key::CloseBracket, "BracketRight", "bracketright"),
|
||||
];
|
||||
|
||||
/// Escape is reserved: it cancels the Settings tab's capture, and a
|
||||
/// Escape is reserved: it cancels the Options window's capture, and a
|
||||
/// system-wide Escape would be unusable anyway.
|
||||
const RESERVED: &[Key] = &[Key::Escape];
|
||||
|
||||
|
|
@ -111,7 +111,7 @@ const RESERVED: &[Key] = &[Key::Escape];
|
|||
///
|
||||
/// Super/Meta is absent because `egui::Modifiers` has no field for it — egui
|
||||
/// reports alt, ctrl, shift and the Mac command key only — so a Super combo
|
||||
/// could never be captured in the Settings tab even if a backend could
|
||||
/// could never be captured in the Options window even if a backend could
|
||||
/// register it.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct Binding {
|
||||
|
|
@ -122,7 +122,7 @@ pub struct Binding {
|
|||
}
|
||||
|
||||
/// Why a string or a key press is not a usable shortcut; the wording is
|
||||
/// shown in the Settings tab.
|
||||
/// shown in the Options window.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum BindingError {
|
||||
Empty,
|
||||
|
|
@ -149,7 +149,7 @@ impl fmt::Display for BindingError {
|
|||
}
|
||||
|
||||
impl Binding {
|
||||
/// Build from a key press egui reported, for the Settings tab's capture
|
||||
/// Build from a key press egui reported, for the Options window's capture
|
||||
/// widget. `None` for a press that cannot be a shortcut: a key with no
|
||||
/// row in [`KEYS`], a reserved key, or a bare key with no modifier held.
|
||||
///
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ thread_local! {
|
|||
static REGISTRY: RefCell<Option<Registry>> = const { RefCell::new(None) };
|
||||
}
|
||||
|
||||
/// What the Settings tab says about the shortcut.
|
||||
/// What the Options window says about the shortcut.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Status {
|
||||
/// The setting is empty: no shortcut, by choice.
|
||||
|
|
@ -256,7 +256,7 @@ mod tests {
|
|||
use super::*;
|
||||
|
||||
/// Nothing may touch an OS registration before `init`, so that the
|
||||
/// headless UI tests can render the Settings row.
|
||||
/// headless UI tests can render the Options row.
|
||||
#[test]
|
||||
fn an_uninitialised_registry_is_inert() {
|
||||
apply("Ctrl+Shift+F");
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
//! `preferred_trigger`, and the compositor is free to bind something else, to
|
||||
//! ask the user first, or to let them change it later in its own settings.
|
||||
//! What it actually bound comes back as a human-readable
|
||||
//! `trigger_description`, which is what the Settings tab shows.
|
||||
//! `trigger_description`, which is what the Options window shows.
|
||||
//!
|
||||
//! All of this lives on its own thread. The portal is D-Bus, so every call
|
||||
//! is a round trip that could block for as long as a dialog stays on screen,
|
||||
|
|
@ -56,7 +56,7 @@ impl Portal {
|
|||
.spawn(move || pollster::block_on(run(ctx, status, rx)))
|
||||
{
|
||||
// Not worth taking the app down for — but the status must say
|
||||
// so, or the Settings tab shows "Waiting for your desktop…" forever.
|
||||
// so, or Options shows "Waiting for your desktop…" forever.
|
||||
quicksearch_core::log_warn!("global shortcut portal thread: {}", e);
|
||||
*lock_ok(&portal.status) =
|
||||
Status::Error(format!("the shortcut thread could not be started: {}", e));
|
||||
|
|
@ -194,7 +194,7 @@ fn unavailable(e: &ashpd::Error) -> String {
|
|||
|
||||
fn set(ctx: &egui::Context, status: &Mutex<Status>, next: Status) {
|
||||
*lock_ok(status) = next;
|
||||
// The Settings tab may be on screen and waiting for this.
|
||||
// The Options window may be open and waiting for this.
|
||||
ctx.request_repaint();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -16,22 +16,20 @@ mod capture;
|
|||
mod cli;
|
||||
mod color;
|
||||
mod duplicates_tab;
|
||||
mod fonts;
|
||||
mod format;
|
||||
mod help_tab;
|
||||
mod hotkey;
|
||||
mod keychain;
|
||||
mod logs_tab;
|
||||
mod manage_tab;
|
||||
mod options;
|
||||
mod platform;
|
||||
mod query_highlight;
|
||||
mod search_tab;
|
||||
mod settings_tab;
|
||||
#[cfg(test)]
|
||||
mod test_ui;
|
||||
mod tips;
|
||||
mod tracker;
|
||||
mod tutorial;
|
||||
mod ui_util;
|
||||
mod unlock;
|
||||
mod version;
|
||||
|
|
@ -107,11 +105,6 @@ fn main() {
|
|||
"QuickSearch",
|
||||
native_options,
|
||||
Box::new(move |cc| {
|
||||
// First: egui is built without its bundled fonts, so a context
|
||||
// starts with no faces at all and lays every string out at zero
|
||||
// height. `set_fonts` is applied in the next `begin_pass`, and
|
||||
// this closure is the last place that is still ahead of frame 1.
|
||||
fonts::install(&cc.egui_ctx);
|
||||
// On Windows the registration owns a hidden window whose messages
|
||||
// the event loop must dispatch, so it must be made on that loop's
|
||||
// thread with the loop running — this closure is the first place
|
||||
|
|
|
|||
|
|
@ -434,7 +434,7 @@ impl ManageTab {
|
|||
|
||||
ui.label(hint(
|
||||
"Reindex interval, symlinks, hidden files, tokenizer, and size \
|
||||
limits are on the Settings tab.",
|
||||
limits are in Options (⚙ in the toolbar).",
|
||||
));
|
||||
ui.add_space(8.0);
|
||||
|
||||
|
|
@ -542,10 +542,10 @@ fn db_size_tooltip(ui: &mut egui::Ui) {
|
|||
"Remove indexed folders you do not need, in Indexed folders above.",
|
||||
"Narrow the full-text extension whitelist, so text is only extracted \
|
||||
from the file types you actually search.",
|
||||
"Turn off \"Store text for snippets\" on the Settings tab: full-text search \
|
||||
keeps working, but without previews, occurrence ranking or fuzzy matching \
|
||||
"Turn off \"Store text for snippets\" in Options: full-text search keeps \
|
||||
working, but without previews, occurrence ranking or fuzzy matching \
|
||||
inside file contents.",
|
||||
"Lower \"Max text file size\" and \"Max stored text\", both on the Settings tab.",
|
||||
"Lower \"Max text file size\" and \"Max stored text\" in Options.",
|
||||
] {
|
||||
ui.label(format!("• {}", lever));
|
||||
}
|
||||
|
|
@ -631,7 +631,7 @@ fn watch_contents(ui: &mut egui::Ui, state: &IndexerState, config: &Config) {
|
|||
ui.colored_label(
|
||||
ui.visuals().warn_fg_color,
|
||||
format!(
|
||||
"Live updates off; reindexing every {}",
|
||||
"⚠ Live updates off; reindexing every {}",
|
||||
fmt_interval(config.indexing.reindex_interval_minutes)
|
||||
),
|
||||
)
|
||||
|
|
@ -809,35 +809,21 @@ fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
|
|||
RootPhase::Extracting => {
|
||||
ui.label(egui::RichText::new("extracting text").color(phase.green));
|
||||
divider(ui);
|
||||
let workers = format!("{}/{} workers", r.active_workers, r.total_workers);
|
||||
match r.extract_total {
|
||||
Some(total) => {
|
||||
let frac = if total > 0 {
|
||||
(r.extracted as f32 / total as f32).clamp(0.0, 1.0)
|
||||
let frac = if r.extract_total > 0 {
|
||||
(r.extracted as f32 / r.extract_total as f32).clamp(0.0, 1.0)
|
||||
} else {
|
||||
1.0
|
||||
};
|
||||
ui.label(format!(
|
||||
"{} / {} ({:.0}%) · {}",
|
||||
"{} / {} ({:.0}%) · {}/{} workers",
|
||||
group_thousands(r.extracted as u64),
|
||||
group_thousands(total as u64),
|
||||
group_thousands(r.extract_total as u64),
|
||||
frac * 100.0,
|
||||
workers
|
||||
r.active_workers,
|
||||
r.total_workers
|
||||
));
|
||||
crate::ui_util::progress_bar(ui, Some(frac), 160.0);
|
||||
}
|
||||
// The pass is still counting its range — the same shape
|
||||
// as a walk without a denominator yet.
|
||||
None => {
|
||||
ui.label(format!(
|
||||
"{} files · {}",
|
||||
group_thousands(r.extracted as u64),
|
||||
workers
|
||||
));
|
||||
crate::ui_util::progress_bar(ui, None, 160.0);
|
||||
}
|
||||
}
|
||||
}
|
||||
RootPhase::Done => {
|
||||
// Whole-root totals: `walked` counts every file the walk saw
|
||||
// and `extracted` all rows with searchable text, not just
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ fn running_state(roots: &[&str], current_file: Option<&str>) -> IndexerState {
|
|||
walked: 100,
|
||||
walk_total: Some(1000),
|
||||
extracted: 0,
|
||||
extract_total: None,
|
||||
extract_total: 0,
|
||||
current_file: current_file.map(str::to_string),
|
||||
active_workers: 4,
|
||||
total_workers: 4,
|
||||
|
|
@ -112,12 +112,11 @@ fn frame(
|
|||
) -> ManageActions {
|
||||
WIDGETS.with(|w| w.borrow_mut().clear());
|
||||
let mut actions = ManageActions::default();
|
||||
let out = ctx.run(raw_input(events), |ctx| {
|
||||
let _ = ctx.run(raw_input(events), |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
actions = tab.ui(ui, state, cfg);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &out);
|
||||
actions
|
||||
}
|
||||
|
||||
|
|
@ -172,7 +171,7 @@ fn staged_workers(tab: &ManageTab) -> Option<usize> {
|
|||
/// that id, so a field that is renamed mid-run silently drops the edit.
|
||||
#[test]
|
||||
fn the_worker_field_keeps_its_identity_as_the_status_changes() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let cfg = cfg_with_root();
|
||||
|
||||
|
|
@ -214,7 +213,7 @@ fn the_worker_field_keeps_its_identity_as_the_status_changes() {
|
|||
/// reporting progress the whole time.
|
||||
#[test]
|
||||
fn a_typed_worker_count_reaches_the_applied_config() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let cfg = cfg_with_root();
|
||||
|
||||
|
|
@ -260,7 +259,7 @@ fn a_typed_worker_count_reaches_the_applied_config() {
|
|||
/// The other way to set the field: drag it.
|
||||
#[test]
|
||||
fn a_dragged_worker_count_is_staged() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let cfg = cfg_with_root();
|
||||
let busy = running_state(&["/data"], None);
|
||||
|
|
@ -305,7 +304,7 @@ fn root_progress(phase: RootPhase, walked: usize, walk_total: Option<usize>) ->
|
|||
walked,
|
||||
walk_total,
|
||||
extracted: 0,
|
||||
extract_total: None,
|
||||
extract_total: 0,
|
||||
current_file: None,
|
||||
active_workers: 4,
|
||||
total_workers: 4,
|
||||
|
|
@ -316,7 +315,7 @@ fn root_progress(phase: RootPhase, walked: usize, walk_total: Option<usize>) ->
|
|||
/// the files a walk emits; a finished root must show the exact count.
|
||||
#[test]
|
||||
fn a_finished_root_reports_its_exact_count_not_the_estimate() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let mut done = root_progress(RootPhase::Done, 261_088, Some(6_677_062));
|
||||
done.extracted = 238_929;
|
||||
|
|
@ -340,7 +339,7 @@ fn a_finished_root_reports_its_exact_count_not_the_estimate() {
|
|||
/// holds survives the run that counted it.
|
||||
#[test]
|
||||
fn a_configured_root_shows_what_the_last_run_counted() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
|
||||
let text = frame_text(&ctx, &mut tab, &counted_state(1_234_567, 456_789)).join(" | ");
|
||||
|
|
@ -355,7 +354,7 @@ fn a_configured_root_shows_what_the_last_run_counted() {
|
|||
/// folder is empty — where the truth is that nothing has counted it yet.
|
||||
#[test]
|
||||
fn a_root_the_index_has_never_counted_says_so() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
|
||||
let text = frame_text(&ctx, &mut tab, &idle_state()).join(" | ");
|
||||
|
|
@ -372,7 +371,7 @@ fn a_root_the_index_has_never_counted_says_so() {
|
|||
/// rather than borrowing another root's numbers.
|
||||
#[test]
|
||||
fn figures_belong_to_the_root_they_were_counted_for() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let state = IndexerState {
|
||||
root_counts: Arc::new(vec![RootCount {
|
||||
|
|
@ -395,7 +394,7 @@ fn figures_belong_to_the_root_they_were_counted_for() {
|
|||
/// already been walked, or the row would read as a hang at 100%.
|
||||
#[test]
|
||||
fn a_walking_root_shows_the_estimate_raised_to_what_it_has_walked() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
|
||||
let honest = frame_text(
|
||||
|
|
@ -441,7 +440,7 @@ fn frame_spans(
|
|||
#[test]
|
||||
fn every_phase_word_is_painted_in_its_hint_color() {
|
||||
for theme in [egui::Theme::Dark, egui::Theme::Light] {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
ctx.set_theme(theme);
|
||||
let mut tab = ManageTab::new();
|
||||
let colors = crate::color::palette(theme == egui::Theme::Dark);
|
||||
|
|
@ -462,7 +461,7 @@ fn every_phase_word_is_painted_in_its_hint_color() {
|
|||
/// No count has landed yet: an indeterminate row, not a fabricated one.
|
||||
#[test]
|
||||
fn a_walking_root_without_a_count_shows_no_denominator() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let text = frame_text(
|
||||
&ctx,
|
||||
|
|
@ -477,7 +476,7 @@ fn a_walking_root_without_a_count_shows_no_denominator() {
|
|||
/// Every step of the prologue names itself and carries a clock.
|
||||
#[test]
|
||||
fn each_prologue_step_says_what_it_is_waiting_on() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
|
||||
for (step, expected) in [
|
||||
|
|
@ -495,7 +494,7 @@ fn each_prologue_step_says_what_it_is_waiting_on() {
|
|||
/// something to count. It reports its position in the scan.
|
||||
#[test]
|
||||
fn a_reconcile_reports_how_far_through_the_index_it_is() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let text = frame_text(
|
||||
&ctx,
|
||||
|
|
@ -522,7 +521,7 @@ fn a_reconcile_reports_how_far_through_the_index_it_is() {
|
|||
/// with nothing to divide by; it must not invent a denominator.
|
||||
#[test]
|
||||
fn a_reconcile_without_a_row_count_shows_no_denominator() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let text = frame_text(
|
||||
&ctx,
|
||||
|
|
@ -538,7 +537,7 @@ fn a_reconcile_without_a_row_count_shows_no_denominator() {
|
|||
/// `Idle` while the thread scans every row.
|
||||
#[test]
|
||||
fn a_prune_between_runs_is_reported_instead_of_idle() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let state = IndexerState {
|
||||
reconcile: Some(ReconcileState::Running(ReconcileProgress {
|
||||
|
|
@ -568,7 +567,7 @@ fn a_prune_between_runs_is_reported_instead_of_idle() {
|
|||
/// is the only evidence it happened.
|
||||
#[test]
|
||||
fn a_finished_prune_reports_what_it_did() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let state = IndexerState {
|
||||
reconcile: Some(ReconcileState::Finished(ReconcileProgress {
|
||||
|
|
@ -598,7 +597,7 @@ fn a_finished_prune_reports_what_it_did() {
|
|||
/// The static "Starting…" placeholder must not reappear.
|
||||
#[test]
|
||||
fn the_starting_placeholder_is_gone() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
for state in [
|
||||
preparing_state(PrepStep::PreviousRun),
|
||||
|
|
@ -684,7 +683,7 @@ fn the_probe_caches_until_the_refresh_interval_is_up() {
|
|||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
/// A database path edited on the Settings tab must not keep showing the old
|
||||
/// A database path edited in Options must not keep showing the old
|
||||
/// database's size for the rest of the interval.
|
||||
#[test]
|
||||
fn the_probe_follows_a_changed_database_path() {
|
||||
|
|
@ -715,7 +714,7 @@ fn the_status_row_shows_the_total_index_size() {
|
|||
write_bytes(&dir.join("index.sqlite-wal"), 200_000);
|
||||
write_bytes(&dir.join("index.sqlite-shm"), 32_768);
|
||||
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let text = frame_text_with(&ctx, &mut tab, &cfg_with_db(&db), &idle_state()).join(" | ");
|
||||
assert!(
|
||||
|
|
@ -734,7 +733,7 @@ fn hovering_the_size_explains_how_to_shrink_the_index() {
|
|||
let db = dir.join("index.sqlite");
|
||||
write_bytes(&db, 2048);
|
||||
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
// egui holds tooltips back for a third of a second, and frames here
|
||||
// are only 1/60 s of simulated time apart.
|
||||
ctx.style_mut(|s| s.interaction.tooltip_delay = 0.0);
|
||||
|
|
@ -763,7 +762,7 @@ fn hovering_the_size_explains_how_to_shrink_the_index() {
|
|||
"Indexed folders",
|
||||
"whitelist",
|
||||
"Store text for snippets",
|
||||
"Settings tab",
|
||||
"Options",
|
||||
] {
|
||||
assert!(text.contains(lever), "tooltip never mentions {}", lever);
|
||||
}
|
||||
|
|
@ -994,7 +993,7 @@ fn a_rejected_apply_keeps_the_draft() {
|
|||
/// which egui hangs interaction state off.
|
||||
#[test]
|
||||
fn the_unsaved_label_appears_without_renaming_the_apply_button() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let cfg = cfg_with_root();
|
||||
|
||||
|
|
@ -1025,7 +1024,7 @@ fn the_unsaved_label_appears_without_renaming_the_apply_button() {
|
|||
/// changes mid-edit loses its buffer.
|
||||
#[test]
|
||||
fn the_prune_rows_come_and_go_without_renaming_anything_below() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut tab = ManageTab::new();
|
||||
let cfg = cfg_with_root();
|
||||
let progress = ReconcileProgress {
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
//! The Settings tab: every configuration control the GUI offers, grouped
|
||||
//! into sections. Edits happen on a draft; Apply validates, saves, and hands
|
||||
//! the new config to the app.
|
||||
//! The Options window and the shared config editor used by both the
|
||||
//! window and the Manage Index tab. Edits happen on a draft; Apply
|
||||
//! validates, saves, and hands the new config to the app.
|
||||
|
||||
use crate::keychain;
|
||||
use crate::tips::{self, tip_row, Tipped};
|
||||
use crate::ui_util::hint;
|
||||
use quicksearch_core::config::{ColumnsConfig, Config};
|
||||
use quicksearch_core::config::Config;
|
||||
|
||||
/// A [`tip_row`] holding one numeric [`egui::DragValue`] — the shape of most
|
||||
/// rows in the config editor.
|
||||
|
|
@ -22,7 +22,7 @@ fn drag_row<N: egui::emath::Numeric>(
|
|||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Section {
|
||||
pub enum Section {
|
||||
Indexing,
|
||||
Processing,
|
||||
Search,
|
||||
|
|
@ -36,26 +36,24 @@ pub enum SecurityAction {
|
|||
Disable,
|
||||
ChangePassword,
|
||||
SetKeychain(bool),
|
||||
ShowKey,
|
||||
}
|
||||
|
||||
/// What one frame of the Settings tab produced.
|
||||
/// What one frame of the Options window produced.
|
||||
#[derive(Default)]
|
||||
pub struct SettingsOutput {
|
||||
pub struct OptionsOutput {
|
||||
/// "Apply & Save" was clicked with this draft.
|
||||
pub applied: Option<Config>,
|
||||
/// A Security block action was clicked.
|
||||
pub security: Option<SecurityAction>,
|
||||
/// The Columns block changed. Like Security, it edits the live config
|
||||
/// rather than the draft, so it takes effect without Apply.
|
||||
pub columns: Option<ColumnsConfig>,
|
||||
/// The title-bar close was clicked while the draft holds unapplied
|
||||
/// edits; the window is held open and the app raises the guard.
|
||||
pub close_requested: bool,
|
||||
}
|
||||
|
||||
pub struct SettingsTab {
|
||||
/// The staged config, built from the live one the first frame the tab is
|
||||
/// shown and dropped again when it is left.
|
||||
pub struct OptionsWindow {
|
||||
pub open: bool,
|
||||
draft: Option<Config>,
|
||||
/// Cached answer from [`SettingsTab::keychain_active`], with the
|
||||
/// Cached answer from [`OptionsWindow::keychain_active`], with the
|
||||
/// `use_keychain` preference it was probed under.
|
||||
keychain_probed_for: Option<bool>,
|
||||
keychain_active: bool,
|
||||
|
|
@ -63,9 +61,10 @@ pub struct SettingsTab {
|
|||
capturing_hotkey: bool,
|
||||
}
|
||||
|
||||
impl SettingsTab {
|
||||
pub fn new() -> SettingsTab {
|
||||
SettingsTab {
|
||||
impl OptionsWindow {
|
||||
pub fn new() -> OptionsWindow {
|
||||
OptionsWindow {
|
||||
open: false,
|
||||
draft: None,
|
||||
keychain_probed_for: None,
|
||||
keychain_active: false,
|
||||
|
|
@ -74,16 +73,21 @@ impl SettingsTab {
|
|||
}
|
||||
|
||||
/// Whether the shortcut button is reading a key press right now, so the
|
||||
/// app can hold the shortcut it is about to replace. The app gates this
|
||||
/// on the tab being the one on screen. See
|
||||
/// app can hold the shortcut it is about to replace. See
|
||||
/// [`crate::unlock::Gate::handle_hotkey`].
|
||||
pub fn capturing_hotkey(&self) -> bool {
|
||||
self.capturing_hotkey
|
||||
self.open && self.capturing_hotkey
|
||||
}
|
||||
|
||||
pub fn open_with(&mut self, current: &Config) {
|
||||
self.open = true;
|
||||
self.draft = Some(current.clone());
|
||||
self.keychain_probed_for = None;
|
||||
}
|
||||
|
||||
/// Whether the draft differs from the live config. The fields the app
|
||||
/// pins on apply are neutralized first, so the Security block never
|
||||
/// makes the tab read as dirty.
|
||||
/// makes the window read as dirty.
|
||||
pub fn is_dirty(&self, current: &Config) -> bool {
|
||||
let Some(draft) = &self.draft else {
|
||||
return false;
|
||||
|
|
@ -98,27 +102,34 @@ impl SettingsTab {
|
|||
self.draft.clone()
|
||||
}
|
||||
|
||||
/// Drop the draft: Discard, or leaving the tab. The next frame that
|
||||
/// shows the tab stages a fresh copy of the live config, which is what
|
||||
/// keeps a draft from going stale against edits made on Manage Index.
|
||||
pub fn discard(&mut self) {
|
||||
/// Close and drop the draft (Discard, or a clean close).
|
||||
pub fn close_discard(&mut self) {
|
||||
self.open = false;
|
||||
self.draft = None;
|
||||
self.capturing_hotkey = false;
|
||||
self.keychain_probed_for = None;
|
||||
}
|
||||
|
||||
/// Take a draft if there is none: the first frame the tab is shown, and
|
||||
/// the first frame after it was left.
|
||||
fn stage(&mut self, current: &Config) {
|
||||
if self.draft.is_none() {
|
||||
self.draft = Some(current.clone());
|
||||
/// Adopt the window's open flag for this frame. A dirty close is
|
||||
/// intercepted: the window is held open and the caller is told to raise
|
||||
/// the unsaved-changes guard instead.
|
||||
fn intercept_close(&mut self, still_open: bool, current: &Config) -> bool {
|
||||
self.open = still_open;
|
||||
if self.open {
|
||||
return false;
|
||||
}
|
||||
if self.is_dirty(current) {
|
||||
self.open = true;
|
||||
true
|
||||
} else {
|
||||
self.draft = None;
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// True when this index's key really is in the OS keychain: the
|
||||
/// preference is on *and* the keychain answers with an entry (a dead
|
||||
/// daemon, a locked keyring or a denied prompt all read as "no").
|
||||
/// Probed when the tab is entered and when the preference changes — a
|
||||
/// Probed when the window opens and when the preference changes — a
|
||||
/// keychain read is an IPC round trip.
|
||||
fn keychain_active(&mut self, current: &Config) -> bool {
|
||||
if self.keychain_probed_for != Some(current.security.use_keychain) {
|
||||
|
|
@ -131,21 +142,29 @@ impl SettingsTab {
|
|||
}
|
||||
|
||||
/// Render; reports an applied draft config and/or a security action.
|
||||
pub fn ui(&mut self, ui: &mut egui::Ui, current: &Config) -> SettingsOutput {
|
||||
self.stage(current);
|
||||
let mut out = SettingsOutput::default();
|
||||
pub fn ui(&mut self, ctx: &egui::Context, current: &Config) -> OptionsOutput {
|
||||
if !self.open {
|
||||
self.draft = None;
|
||||
return OptionsOutput::default();
|
||||
}
|
||||
if self.draft.is_none() {
|
||||
self.draft = Some(current.clone());
|
||||
}
|
||||
let mut out = OptionsOutput::default();
|
||||
let mut open = self.open;
|
||||
let keychain_active = self.keychain_active(current);
|
||||
let dirty = self.is_dirty(current);
|
||||
let capturing = &mut self.capturing_hotkey;
|
||||
let draft = self.draft.as_mut().unwrap();
|
||||
|
||||
egui::Window::new("Options")
|
||||
.open(&mut open)
|
||||
.resizable(false)
|
||||
.default_width(420.0)
|
||||
.show(ctx, |ui| {
|
||||
let scroll = egui::ScrollArea::vertical()
|
||||
.auto_shrink([false; 2])
|
||||
.max_height(480.0)
|
||||
.show(ui, |ui| {
|
||||
// Cap the column like a document page: a maximized window
|
||||
// would otherwise stretch every hint into one long line.
|
||||
ui.set_max_width(620.0);
|
||||
|
||||
ui.heading(egui::RichText::new("Paths").strong());
|
||||
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
|
||||
tip_row(ui, "Database file", &tips::DATABASE_PATH, |ui| {
|
||||
|
|
@ -172,9 +191,6 @@ impl SettingsTab {
|
|||
|
||||
ui.heading(egui::RichText::new("Search").strong());
|
||||
config_editor_ui(ui, draft, Section::Search);
|
||||
ui.add_space(6.0);
|
||||
// Live, not drafted — see `columns_ui`.
|
||||
out.columns = columns_ui(ui, ¤t.search.columns);
|
||||
ui.separator();
|
||||
|
||||
ui.heading(egui::RichText::new("Interface").strong());
|
||||
|
|
@ -196,14 +212,15 @@ impl SettingsTab {
|
|||
hotkey_note(ui, &draft.ui.search_hotkey, ¤t.ui.search_hotkey);
|
||||
ui.separator();
|
||||
|
||||
// Security acts on the live config, not the draft. The KDF
|
||||
// salt is never shown here or anywhere else in the GUI.
|
||||
// Security acts on the live config, not the draft.
|
||||
// The KDF salt is never shown here or anywhere else
|
||||
// in the GUI.
|
||||
ui.heading(egui::RichText::new("Security").strong());
|
||||
out.security = security_ui(ui, current, keychain_active);
|
||||
ui.separator();
|
||||
});
|
||||
crate::ui_util::more_below_hint(ui, &scroll);
|
||||
|
||||
// Last in the scroll, where the Manage Index tab also puts
|
||||
// it, so the two draft-backed editors read the same way.
|
||||
ui.separator();
|
||||
let p = crate::color::palette(ui.visuals().dark_mode);
|
||||
ui.horizontal(|ui| {
|
||||
let apply = ui
|
||||
|
|
@ -233,8 +250,8 @@ impl SettingsTab {
|
|||
and hash length require a full rebuild.",
|
||||
));
|
||||
});
|
||||
crate::ui_util::more_below_hint(ui, &scroll);
|
||||
|
||||
out.close_requested = self.intercept_close(open, current);
|
||||
out
|
||||
}
|
||||
}
|
||||
|
|
@ -381,39 +398,6 @@ fn hotkey_note(ui: &mut egui::Ui, draft: &str, live: &str) {
|
|||
});
|
||||
}
|
||||
|
||||
/// The Search-tab column picker, mirroring the right-click menu on the table
|
||||
/// headers. Returns the new set when a checkbox moved.
|
||||
///
|
||||
/// Acts on the **live** config, not the draft, for the same reason the
|
||||
/// Security block does: the header menu writes columns the instant they
|
||||
/// change, and a draft-backed copy here would silently revert that on the next
|
||||
/// Apply. `app::pin_live_fields` keeps the draft out of this field entirely.
|
||||
fn columns_ui(ui: &mut egui::Ui, current: &ColumnsConfig) -> Option<ColumnsConfig> {
|
||||
let mut next = current.clone();
|
||||
ui.label("Search columns").on_hover_text(tips::COLUMNS.body);
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.checkbox(&mut next.name, "Name").tip(&tips::COLUMNS);
|
||||
// Checked and greyed rather than absent: an omitted entry reads as an
|
||||
// oversight, a disabled one answers the question.
|
||||
ui.add_enabled(false, egui::Checkbox::new(&mut true, "Path"))
|
||||
.on_disabled_hover_text(
|
||||
"The path is always shown — it is the only column that \
|
||||
identifies a result on its own.",
|
||||
);
|
||||
ui.checkbox(&mut next.content_match, "Content Match")
|
||||
.tip(&tips::COLUMNS);
|
||||
ui.checkbox(&mut next.size, "Size").tip(&tips::COLUMNS);
|
||||
ui.checkbox(&mut next.modified, "Modified")
|
||||
.tip(&tips::COLUMNS);
|
||||
ui.checkbox(&mut next.rank, "Rank").tip(&tips::COLUMNS);
|
||||
});
|
||||
ui.label(hint(
|
||||
"Also on the Search tab: right-click any column header. Applied and \
|
||||
saved immediately.",
|
||||
));
|
||||
(next != *current).then_some(next)
|
||||
}
|
||||
|
||||
/// The Security block: status plus action buttons. Never renders the salt.
|
||||
fn security_ui(
|
||||
ui: &mut egui::Ui,
|
||||
|
|
@ -446,14 +430,6 @@ fn security_ui(
|
|||
action = Some(SecurityAction::Disable);
|
||||
}
|
||||
});
|
||||
// Its own row: three buttons do not fit the window's width.
|
||||
if ui
|
||||
.button("Show database key…")
|
||||
.tip(&tips::SHOW_KEY)
|
||||
.clicked()
|
||||
{
|
||||
action = Some(SecurityAction::ShowKey);
|
||||
}
|
||||
let mut remember = current.security.use_keychain;
|
||||
if ui
|
||||
.checkbox(&mut remember, "Remember on this device")
|
||||
|
|
@ -479,10 +455,10 @@ fn security_ui(
|
|||
action
|
||||
}
|
||||
|
||||
/// The per-section config controls of the Settings tab. Every row goes
|
||||
/// The per-section config controls of the Options window. Every row goes
|
||||
/// through [`crate::tips::tip_row`], so a setting cannot arrive here
|
||||
/// without a tooltip.
|
||||
fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
|
||||
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
|
||||
match section {
|
||||
Section::Indexing => {
|
||||
egui::Grid::new("cfg-indexing")
|
||||
|
|
@ -622,10 +598,6 @@ fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
|
|||
&mut config.search.debounce_ms,
|
||||
0..=2000,
|
||||
);
|
||||
|
||||
tip_row(ui, "Live results", &tips::LIVE_RESULTS, |ui| {
|
||||
ui.checkbox(&mut config.search.live_results, "")
|
||||
});
|
||||
});
|
||||
// The warning comes and goes as the value is edited; keep it off
|
||||
// the ids of what follows (`ui_util::stable_section`).
|
||||
|
|
@ -5,74 +5,62 @@ use super::*;
|
|||
|
||||
#[test]
|
||||
fn a_fresh_draft_is_not_dirty() {
|
||||
let mut w = SettingsTab::new();
|
||||
let mut w = OptionsWindow::new();
|
||||
let cfg = Config::default();
|
||||
assert!(!w.is_dirty(&cfg), "no draft at all");
|
||||
w.stage(&cfg);
|
||||
w.open_with(&cfg);
|
||||
assert!(!w.is_dirty(&cfg));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_edited_draft_is_dirty_until_discarded() {
|
||||
let mut w = SettingsTab::new();
|
||||
let mut w = OptionsWindow::new();
|
||||
let cfg = Config::default();
|
||||
w.stage(&cfg);
|
||||
w.open_with(&cfg);
|
||||
w.draft.as_mut().unwrap().search.debounce_ms += 100;
|
||||
assert!(w.is_dirty(&cfg));
|
||||
w.discard();
|
||||
assert!(w.draft.is_none());
|
||||
w.close_discard();
|
||||
assert!(!w.open);
|
||||
assert!(!w.is_dirty(&cfg), "the draft is gone");
|
||||
}
|
||||
|
||||
/// The Security block and the mode buttons act on the live config while
|
||||
/// the tab is on screen; the stale copies in the draft are not edits.
|
||||
/// the window sits open; the stale copies in the draft are not edits.
|
||||
#[test]
|
||||
fn live_security_and_mode_changes_are_not_dirty() {
|
||||
let mut w = SettingsTab::new();
|
||||
let mut w = OptionsWindow::new();
|
||||
let mut cfg = Config::default();
|
||||
w.stage(&cfg);
|
||||
w.open_with(&cfg);
|
||||
cfg.security.use_keychain = !cfg.security.use_keychain;
|
||||
cfg.indexing.auto_index = !cfg.indexing.auto_index;
|
||||
assert!(!w.is_dirty(&cfg));
|
||||
}
|
||||
|
||||
/// Leaving the tab drops the draft, so the next visit stages the config as
|
||||
/// it stands *then*. Without this an edit made on the Manage Index tab in
|
||||
/// between would be reverted by a later Apply: `pin_live_fields` protects
|
||||
/// the fields saved live, but not the indexed folders or the filters.
|
||||
#[test]
|
||||
fn a_draft_is_restaged_from_the_live_config_after_leaving() {
|
||||
let mut w = SettingsTab::new();
|
||||
let mut cfg = Config::default();
|
||||
w.stage(&cfg);
|
||||
w.discard();
|
||||
|
||||
cfg.indexing.ignore_patterns.push("*.tmp".to_string());
|
||||
w.stage(&cfg);
|
||||
assert!(!w.is_dirty(&cfg), "the fresh draft matches the live config");
|
||||
assert_eq!(
|
||||
w.draft_config().unwrap().indexing.ignore_patterns,
|
||||
cfg.indexing.ignore_patterns,
|
||||
"the filter added while the tab was away survives"
|
||||
);
|
||||
}
|
||||
|
||||
/// A key capture in progress cannot outlive the tab: the app stops asking
|
||||
/// [`SettingsTab::capturing_hotkey`] once another tab is up, and the button
|
||||
/// must not be waiting when the tab comes back either.
|
||||
#[test]
|
||||
fn leaving_the_tab_ends_a_shortcut_capture() {
|
||||
let mut w = SettingsTab::new();
|
||||
fn a_dirty_close_is_held_and_a_clean_one_drops_the_draft() {
|
||||
let mut w = OptionsWindow::new();
|
||||
let cfg = Config::default();
|
||||
w.stage(&cfg);
|
||||
w.capturing_hotkey = true;
|
||||
w.discard();
|
||||
assert!(!w.capturing_hotkey());
|
||||
w.open_with(&cfg);
|
||||
w.draft.as_mut().unwrap().search.debounce_ms += 100;
|
||||
|
||||
assert!(
|
||||
w.intercept_close(false, &cfg),
|
||||
"dirty close raises the guard"
|
||||
);
|
||||
assert!(w.open, "the window is held open until the user decides");
|
||||
assert!(w.draft.is_some(), "the draft survives");
|
||||
|
||||
assert!(!w.intercept_close(true, &cfg), "still open: nothing to do");
|
||||
|
||||
w.draft = Some(cfg.clone());
|
||||
assert!(!w.intercept_close(false, &cfg), "a clean close just closes");
|
||||
assert!(!w.open);
|
||||
assert!(w.draft.is_none());
|
||||
}
|
||||
|
||||
use crate::test_ui::{click_at, painted_text, painted_text_center};
|
||||
|
||||
/// One frame of the shortcut control on its own, outside the tab's
|
||||
/// One frame of the shortcut control on its own, outside the window's
|
||||
/// scroll area so it is never below the fold.
|
||||
fn run_hotkey_edit(
|
||||
ctx: &egui::Context,
|
||||
|
|
@ -81,13 +69,11 @@ fn run_hotkey_edit(
|
|||
events: Vec<egui::Event>,
|
||||
) -> egui::FullOutput {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(600.0, 200.0), events);
|
||||
let out = ctx.run(input, |ctx| {
|
||||
ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
hotkey_edit(ui, setting, capturing);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &out);
|
||||
out
|
||||
})
|
||||
}
|
||||
|
||||
/// One frame of the color scheme control on its own, for the same reason
|
||||
|
|
@ -98,20 +84,18 @@ fn run_color_scheme_edit(
|
|||
events: Vec<egui::Event>,
|
||||
) -> egui::FullOutput {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(600.0, 200.0), events);
|
||||
let out = ctx.run(input, |ctx| {
|
||||
ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
color_scheme_edit(ui, setting);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &out);
|
||||
out
|
||||
})
|
||||
}
|
||||
|
||||
/// The dropdown says which scheme is in force and writes the one that is
|
||||
/// picked; what it shows and what it stores are not the same string.
|
||||
#[test]
|
||||
fn the_color_scheme_box_shows_and_sets_the_scheme() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut setting = "dark".to_string();
|
||||
|
||||
let closed = run_color_scheme_edit(&ctx, &mut setting, vec![]);
|
||||
|
|
@ -167,7 +151,7 @@ const CTRL_ALT: egui::Modifiers = egui::Modifiers {
|
|||
/// pressed.
|
||||
#[test]
|
||||
fn the_shortcut_button_binds_what_was_pressed() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut setting = "Ctrl+Shift+F".to_string();
|
||||
let mut capturing = false;
|
||||
|
||||
|
|
@ -199,7 +183,7 @@ fn the_shortcut_button_binds_what_was_pressed() {
|
|||
/// through rather than treated as one.
|
||||
#[test]
|
||||
fn capture_ignores_what_it_cannot_bind_and_escape_cancels() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut setting = "Ctrl+Shift+F".to_string();
|
||||
let mut capturing = true;
|
||||
|
||||
|
|
@ -225,7 +209,7 @@ fn capture_ignores_what_it_cannot_bind_and_escape_cancels() {
|
|||
|
||||
#[test]
|
||||
fn clear_switches_the_shortcut_off() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut setting = "Ctrl+Shift+F".to_string();
|
||||
let mut capturing = false;
|
||||
|
||||
|
|
@ -244,7 +228,7 @@ fn clear_switches_the_shortcut_off() {
|
|||
/// app is actually holding, and until Apply they can disagree.
|
||||
#[test]
|
||||
fn an_unapplied_shortcut_says_it_is_not_in_force_yet() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let run = |draft: &str, live: &str| {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(600.0, 200.0), vec![]);
|
||||
let out = ctx.run(input, |ctx| {
|
||||
|
|
@ -311,15 +295,14 @@ const ROWS: &[(Section, &str, &tips::Tip)] = &[
|
|||
&tips::RESULTS_PER_PAGE,
|
||||
),
|
||||
(Section::Search, "Debounce (ms)", &tips::DEBOUNCE),
|
||||
(Section::Search, "Live results", &tips::LIVE_RESULTS),
|
||||
];
|
||||
|
||||
/// Hovering a row's name paints that row's own explanation. Rendered
|
||||
/// without the tab's scroll area so nothing sits below the fold.
|
||||
/// without the window's scroll area so nothing sits below the fold.
|
||||
#[test]
|
||||
fn every_row_shows_its_own_tip() {
|
||||
for (section, label, tip) in ROWS {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
ctx.style_mut(|s| {
|
||||
s.interaction.tooltip_delay = 0.0;
|
||||
s.interaction.show_tooltips_only_when_still = false;
|
||||
|
|
@ -359,26 +342,33 @@ fn every_row_shows_its_own_tip() {
|
|||
/// wiring is under test, so tooltip timing is turned off.
|
||||
#[test]
|
||||
fn hovering_a_setting_label_explains_it() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
ctx.style_mut(|s| {
|
||||
s.interaction.tooltip_delay = 0.0;
|
||||
s.interaction.show_tooltips_only_when_still = false;
|
||||
});
|
||||
let cfg = Config::default();
|
||||
let mut w = SettingsTab::new();
|
||||
let mut w = OptionsWindow::new();
|
||||
w.open_with(&cfg);
|
||||
|
||||
let run = |w: &mut SettingsTab, events: Vec<egui::Event>| {
|
||||
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
|
||||
ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
w.ui(ui, &cfg);
|
||||
});
|
||||
w.ui(ctx, &cfg);
|
||||
})
|
||||
};
|
||||
|
||||
// The window spends its first frames sizing itself and painting
|
||||
// nothing; run until the label is on screen.
|
||||
let mut target = None;
|
||||
for _ in 0..5 {
|
||||
let full = run(&mut w, vec![]);
|
||||
let target =
|
||||
painted_text_center(&full, "Tokenizer").expect("the Tokenizer label was not painted");
|
||||
target = painted_text_center(&full, "Tokenizer");
|
||||
if target.is_some() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let target = target.expect("the Tokenizer label was not painted");
|
||||
|
||||
// The tooltip is an area of its own, so it can land a frame late.
|
||||
let mut out = run(&mut w, vec![egui::Event::PointerMoved(target)]);
|
||||
|
|
@ -394,30 +384,37 @@ fn hovering_a_setting_label_explains_it() {
|
|||
panic!("no tooltip appeared over the Tokenizer label");
|
||||
}
|
||||
|
||||
/// One real frame of the tab in a headless context: it renders, and the
|
||||
/// Apply & Save click comes back out as `applied`.
|
||||
/// One real frame of the window in a headless context: it renders, and
|
||||
/// the Apply & Save click comes back out as `applied`.
|
||||
#[test]
|
||||
fn the_tab_renders_and_apply_reports_the_draft() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
fn the_window_renders_and_apply_reports_the_draft() {
|
||||
let ctx = egui::Context::default();
|
||||
let cfg = Config::default();
|
||||
let mut w = SettingsTab::new();
|
||||
w.stage(&cfg);
|
||||
let mut w = OptionsWindow::new();
|
||||
w.open_with(&cfg);
|
||||
w.draft.as_mut().unwrap().search.debounce_ms += 100;
|
||||
|
||||
let run = |w: &mut SettingsTab, events: Vec<egui::Event>| {
|
||||
let run = |w: &mut OptionsWindow, events: Vec<egui::Event>| {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(1000.0, 900.0), events);
|
||||
let mut out = SettingsOutput::default();
|
||||
let full = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| out = w.ui(ui, &cfg));
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(&ctx, &full);
|
||||
let mut out = OptionsOutput::default();
|
||||
let full = ctx.run(input, |ctx| out = w.ui(ctx, &cfg));
|
||||
(out, full)
|
||||
};
|
||||
|
||||
// A new egui window spends its first frames in sizing passes that
|
||||
// suppress painting; run untouched frames until the settled button
|
||||
// is actually on screen.
|
||||
let mut target = None;
|
||||
for _ in 0..5 {
|
||||
let (untouched, full) = run(&mut w, vec![]);
|
||||
assert!(untouched.applied.is_none());
|
||||
let target = painted_text_center(&full, "Apply & Save")
|
||||
.expect("the Apply & Save button was not painted");
|
||||
assert!(!untouched.close_requested);
|
||||
target = painted_text_center(&full, "Apply & Save");
|
||||
if target.is_some() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let target = target.expect("the Apply & Save button was not painted");
|
||||
let clicks = [true, false]
|
||||
.into_iter()
|
||||
.map(|pressed| egui::Event::PointerButton {
|
||||
|
|
@ -435,158 +432,3 @@ fn the_tab_renders_and_apply_reports_the_draft() {
|
|||
"the click reported the edited draft"
|
||||
);
|
||||
}
|
||||
|
||||
/// One frame of the column picker on its own, outside the tab's scroll
|
||||
/// area so it is never below the fold — the same shape as [`run_hotkey_edit`].
|
||||
fn run_columns(
|
||||
ctx: &egui::Context,
|
||||
current: &ColumnsConfig,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (Option<ColumnsConfig>, egui::FullOutput) {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(700.0, 200.0), events);
|
||||
let mut picked = None;
|
||||
let full = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
picked = columns_ui(ui, current);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &full);
|
||||
(picked, full)
|
||||
}
|
||||
|
||||
/// The Settings copy of the column picker acts on the *live* config, not the
|
||||
/// draft — the same arrangement the Security block uses, and the reason it and
|
||||
/// the table header's menu cannot end up disagreeing. So it reports a change
|
||||
/// the moment a box moves, with no Apply.
|
||||
#[test]
|
||||
fn the_columns_block_reports_a_change_immediately() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let current = ColumnsConfig::default();
|
||||
assert!(!current.size, "the fixture assumes Size ships off");
|
||||
|
||||
let (quiet, full) = run_columns(&ctx, ¤t, vec![]);
|
||||
assert!(quiet.is_none(), "reported a change nobody made");
|
||||
let target = painted_text_center(&full, "Size").expect("no Size checkbox");
|
||||
|
||||
let (picked, _) = run_columns(&ctx, ¤t, click_at(target));
|
||||
let picked = picked.expect("the click reported nothing");
|
||||
assert!(picked.size, "clicking Size did not switch it on");
|
||||
// Only that one moved.
|
||||
assert_eq!(
|
||||
picked,
|
||||
ColumnsConfig {
|
||||
size: true,
|
||||
..current
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/// The path is not offered: it is the one column that identifies a result on
|
||||
/// its own. It is shown checked and greyed rather than left out, so the
|
||||
/// question "why can I not remove it?" has an answer on screen.
|
||||
#[test]
|
||||
fn the_columns_block_offers_every_column_but_the_path() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let (_, full) = run_columns(&ctx, &ColumnsConfig::default(), vec![]);
|
||||
let painted = painted_text(&full);
|
||||
for label in ["Name", "Path", "Content Match", "Size", "Modified", "Rank"] {
|
||||
assert!(
|
||||
painted.iter().any(|t| t == label),
|
||||
"{label} missing: {painted:?}"
|
||||
);
|
||||
}
|
||||
|
||||
// Clicking it does nothing, because it is disabled.
|
||||
let target = painted_text_center(&full, "Path").expect("no Path entry");
|
||||
let (picked, _) = run_columns(&ctx, &ColumnsConfig::default(), click_at(target));
|
||||
assert!(picked.is_none(), "the path column was switched off");
|
||||
}
|
||||
|
||||
/// One frame of the Security block on its own, in the shape of
|
||||
/// [`run_columns`]. `keychain_active` is passed straight through, so nothing
|
||||
/// here touches the OS keychain.
|
||||
fn run_security(
|
||||
ctx: &egui::Context,
|
||||
current: &Config,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (Option<SecurityAction>, egui::FullOutput) {
|
||||
let input = crate::test_ui::raw_input(egui::vec2(700.0, 300.0), events);
|
||||
let mut action = None;
|
||||
let full = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
action = security_ui(ui, current, false);
|
||||
});
|
||||
});
|
||||
crate::test_ui::assert_no_tofu(ctx, &full);
|
||||
(action, full)
|
||||
}
|
||||
|
||||
/// An unprotected index has no key at all, so there is nothing the button
|
||||
/// could show and it is left out rather than shown dead.
|
||||
#[test]
|
||||
fn the_key_button_appears_only_while_the_index_is_encrypted() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut cfg = Config::default();
|
||||
assert!(
|
||||
!cfg.security.password_protected,
|
||||
"the fixture assumes protection ships off"
|
||||
);
|
||||
|
||||
let (_, full) = run_security(&ctx, &cfg, vec![]);
|
||||
assert!(
|
||||
painted_text_center(&full, "Show database key…").is_none(),
|
||||
"offered the key of an unencrypted index: {:?}",
|
||||
painted_text(&full)
|
||||
);
|
||||
|
||||
cfg.security.password_protected = true;
|
||||
let (_, full) = run_security(&ctx, &cfg, vec![]);
|
||||
assert!(
|
||||
painted_text_center(&full, "Show database key…").is_some(),
|
||||
"no key button while encrypted: {:?}",
|
||||
painted_text(&full)
|
||||
);
|
||||
}
|
||||
|
||||
/// The click only asks for the flow; the password confirmation and the reveal
|
||||
/// both live in the app, so nothing about the key is decided here.
|
||||
#[test]
|
||||
fn clicking_the_key_button_reports_show_key() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let cfg = Config {
|
||||
security: quicksearch_core::config::SecurityConfig {
|
||||
password_protected: true,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (quiet, full) = run_security(&ctx, &cfg, vec![]);
|
||||
assert!(quiet.is_none(), "reported an action nobody clicked");
|
||||
let target = painted_text_center(&full, "Show database key…").expect("no key button");
|
||||
|
||||
let (action, _) = run_security(&ctx, &cfg, click_at(target));
|
||||
assert_eq!(action, Some(SecurityAction::ShowKey));
|
||||
}
|
||||
|
||||
/// Columns are live state, so a draft taken before one changed must not carry
|
||||
/// the old set back on Apply — `app::pin_live_fields` is what prevents that,
|
||||
/// and this is the assertion that it covers this field.
|
||||
#[test]
|
||||
fn a_stale_draft_cannot_revert_the_columns() {
|
||||
let mut w = SettingsTab::new();
|
||||
let mut cfg = Config::default();
|
||||
w.stage(&cfg);
|
||||
|
||||
// The header menu switches a column on while the tab is on screen.
|
||||
cfg.search.columns.size = true;
|
||||
assert!(!w.is_dirty(&cfg), "a live column change read as an edit");
|
||||
|
||||
let draft = w.draft_config().expect("a draft");
|
||||
let mut applied = draft;
|
||||
crate::app::pin_live_fields(&mut applied, &cfg);
|
||||
assert!(
|
||||
applied.search.columns.size,
|
||||
"applying the stale draft reverted the column"
|
||||
);
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -37,11 +37,6 @@ const SNIPPET_LEAD: &str = "… ";
|
|||
|
||||
/// Append `window[range]` to `job`, highlighting whatever parts of `ranges`
|
||||
/// (byte offsets into `window`) fall inside it.
|
||||
///
|
||||
/// Ranges are clipped to the slice, so a caller rendering a string in pieces
|
||||
/// can hand each piece the *whole* set: a range inside this one survives, one
|
||||
/// straddling an edge survives as the part that is here, and one wholly
|
||||
/// outside disappears.
|
||||
fn append_marked(
|
||||
job: &mut LayoutJob,
|
||||
fmt: &SnippetFormats,
|
||||
|
|
@ -66,18 +61,6 @@ fn append_marked(
|
|||
}
|
||||
}
|
||||
|
||||
/// A whole field — a filename — with its matched spans marked.
|
||||
///
|
||||
/// Wrapping is left at the job's defaults on purpose: `egui::Label` overwrites
|
||||
/// only `wrap.max_width`, so this is laid out exactly like the plain string it
|
||||
/// replaces, and the cell keeps the height and clipping it had before.
|
||||
pub(super) fn marked_field_job(ui: &egui::Ui, text: &str, ranges: &[(usize, usize)]) -> LayoutJob {
|
||||
let fmt = snippet_formats(ui);
|
||||
let mut job = LayoutJob::default();
|
||||
append_marked(&mut job, &fmt, text, ranges, 0..text.len());
|
||||
job
|
||||
}
|
||||
|
||||
/// The byte offset in `snip.window` that rendering has to start at for the
|
||||
/// first match to land on a row that survives `max_rows`; `0` when it
|
||||
/// already does. epaint stops at `wrap.max_rows` and *every* `\n` costs a
|
||||
|
|
@ -176,14 +159,16 @@ pub(super) fn snippet_job(ui: &egui::Ui, snip: &Snippet, max_rows: usize) -> Lay
|
|||
job
|
||||
}
|
||||
|
||||
/// The Content Match column cell: one line with the (first) matched span
|
||||
/// centered and an equal amount of context on both sides, trimmed to what fits
|
||||
/// the column width.
|
||||
///
|
||||
/// Only ever called with a content snippet. Name and path matches are
|
||||
/// highlighted in their own columns and leave a dash here, so the bracketed
|
||||
/// `[whole field]` rendering this used to carry is gone.
|
||||
pub(super) fn centered_match_job(ui: &egui::Ui, snip: &Snippet, width_px: f32) -> LayoutJob {
|
||||
/// The Match column cell: one line with the (first) matched span centered
|
||||
/// and an equal amount of context on both sides, trimmed to what fits the
|
||||
/// column width. Matches on a whole field — a filename or a path — are
|
||||
/// wrapped in brackets: `[name]`.
|
||||
pub(super) fn centered_match_job(
|
||||
ui: &egui::Ui,
|
||||
snip: &Snippet,
|
||||
width_px: f32,
|
||||
whole_field: bool,
|
||||
) -> LayoutJob {
|
||||
let fmt = snippet_formats(ui);
|
||||
|
||||
// Newlines force line breaks even in a one-row LayoutJob; flatten them
|
||||
|
|
@ -204,6 +189,11 @@ pub(super) fn centered_match_job(ui: &egui::Ui, snip: &Snippet, width_px: f32) -
|
|||
let font_id = &fmt.normal.font_id;
|
||||
let width_of = |c: char| f.glyph_width(font_id, c);
|
||||
let ellipsis = width_of('…');
|
||||
let brackets = if whole_field {
|
||||
width_of('[') + width_of(']')
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
let mut marks = 0.0;
|
||||
if snip.truncated_start {
|
||||
marks += ellipsis;
|
||||
|
|
@ -211,13 +201,13 @@ pub(super) fn centered_match_job(ui: &egui::Ui, snip: &Snippet, width_px: f32) -
|
|||
if snip.truncated_end {
|
||||
marks += ellipsis;
|
||||
}
|
||||
if fits_within(window, width_px - marks, width_of) {
|
||||
if fits_within(window, width_px - brackets - marks, width_of) {
|
||||
return (0, window.len(), true);
|
||||
}
|
||||
|
||||
// Something has to go, so either end may gain a mark; reserve for
|
||||
// both so a cut never overflows the column.
|
||||
let budget = width_px - 2.0 * ellipsis;
|
||||
let budget = width_px - brackets - 2.0 * ellipsis;
|
||||
let Some(&(a, b)) = snip.ranges.first() else {
|
||||
// No ranges (shouldn't happen for match cells) — head trim.
|
||||
return (0, take_forward(window, 0, budget.max(0.0), width_of), true);
|
||||
|
|
@ -271,52 +261,22 @@ pub(super) fn centered_match_job(ui: &egui::Ui, snip: &Snippet, width_px: f32) -
|
|||
let mut job = LayoutJob::default();
|
||||
job.wrap.max_rows = 1;
|
||||
job.wrap.break_anywhere = true;
|
||||
if whole_field && decorate {
|
||||
job.append("[", 0.0, fmt.weak.clone());
|
||||
}
|
||||
if decorate && (start > 0 || snip.truncated_start) {
|
||||
job.append("…", 0.0, fmt.weak.clone());
|
||||
}
|
||||
append_marked(&mut job, &fmt, window, &snip.ranges, start..end);
|
||||
if decorate && (end < window.len() || snip.truncated_end) {
|
||||
job.append("…", 0.0, fmt.weak);
|
||||
job.append("…", 0.0, fmt.weak.clone());
|
||||
}
|
||||
if whole_field && decorate {
|
||||
job.append("]", 0.0, fmt.weak);
|
||||
}
|
||||
job
|
||||
}
|
||||
|
||||
/// The Path column cell: middle-elided to `width_px`, with whatever of a
|
||||
/// path-tier match survives the cut highlighted.
|
||||
///
|
||||
/// The path reads at full strength — it is the one column that identifies a
|
||||
/// result on its own. Only the elision mark is weak, since it is punctuation
|
||||
/// this renderer added rather than anything the file is named.
|
||||
///
|
||||
/// Returns the job and whether anything was actually elided — the caller's
|
||||
/// trigger for a full-path tooltip, since egui offers one only when *it* did
|
||||
/// the eliding and it is handed an already-shortened string.
|
||||
pub(super) fn path_cell_job(
|
||||
ui: &egui::Ui,
|
||||
path: &str,
|
||||
ranges: &[(usize, usize)],
|
||||
width_px: f32,
|
||||
font_id: &egui::FontId,
|
||||
) -> (LayoutJob, bool) {
|
||||
let fmt = snippet_formats(ui);
|
||||
let mut job = LayoutJob::default();
|
||||
match crate::ui_util::middle_elide_cut(ui, path, width_px, font_id) {
|
||||
// It fits: the whole path, marked — which is exactly
|
||||
// [`marked_field_job`].
|
||||
None => (marked_field_job(ui, path, ranges), false),
|
||||
// The two surviving ends are appended straight from `path` at their
|
||||
// original offsets: `append_marked` clips the ranges to each end, so a
|
||||
// match that fell in the dropped middle drops with it rather than
|
||||
// landing on whatever glyphs moved into those offsets.
|
||||
Some((head, tail)) => {
|
||||
append_marked(&mut job, &fmt, path, ranges, 0..head);
|
||||
job.append("…", 0.0, fmt.weak.clone());
|
||||
append_marked(&mut job, &fmt, path, ranges, tail..path.len());
|
||||
(job, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the whole of `text` fits in `budget` pixels; stops at the first
|
||||
/// character that does not.
|
||||
fn fits_within(text: &str, budget: f32, width_of: impl Fn(char) -> f32) -> bool {
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -27,57 +27,10 @@ pub fn click_at(pos: egui::Pos2) -> Vec<egui::Event> {
|
|||
vec![egui::Event::PointerMoved(pos), button(true), button(false)]
|
||||
}
|
||||
|
||||
/// A context carrying the fonts the app installs.
|
||||
///
|
||||
/// Not `egui::Context::default()`, which every test here used to call: egui is
|
||||
/// built without `default_fonts`, so a default context has *no* faces. That
|
||||
/// does not fail loudly — epaint hands back a font of `row_height` 0.0 and
|
||||
/// zero-advance glyphs — so every measurement, every wrap and every click
|
||||
/// target below would quietly stop meaning anything.
|
||||
pub fn ctx() -> egui::Context {
|
||||
let ctx = egui::Context::default();
|
||||
crate::fonts::install(&ctx);
|
||||
ctx
|
||||
}
|
||||
|
||||
/// Assert that every character painted this frame has a real glyph in the
|
||||
/// installed fonts — that nothing on screen is a `◻`.
|
||||
///
|
||||
/// The `FontId` comes from each layout section, so a monospace run is checked
|
||||
/// against the monospace family and a proportional run against the
|
||||
/// proportional one, exactly as epaint resolved them. Whitespace and controls
|
||||
/// are skipped: epaint maps those to space-derived or invisible glyphs on
|
||||
/// purpose, and `\n` is documented to report as the replacement.
|
||||
///
|
||||
/// Only sees what this frame actually painted, so its reach is the reach of
|
||||
/// the test that calls it.
|
||||
pub fn assert_no_tofu(ctx: &egui::Context, out: &egui::FullOutput) {
|
||||
let mut missing: Vec<(char, String, egui::FontId)> = Vec::new();
|
||||
ctx.fonts(|fonts| {
|
||||
for (galley, _) in painted_galleys(out) {
|
||||
for section in &galley.job.sections {
|
||||
for c in galley.job.text[section.byte_range.clone()].chars() {
|
||||
if c.is_whitespace() || c.is_control() {
|
||||
continue;
|
||||
}
|
||||
if !fonts.has_glyph(§ion.format.font_id, c) {
|
||||
missing.push((
|
||||
c,
|
||||
galley.text().to_string(),
|
||||
section.format.font_id.clone(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
assert!(missing.is_empty(), "no glyph for: {missing:#?}");
|
||||
}
|
||||
|
||||
/// A `Ui` from a real (headless) egui pass, so measuring helpers see the
|
||||
/// same fonts the app paints with.
|
||||
pub fn with_ui<R>(f: impl FnOnce(&mut egui::Ui) -> R) -> R {
|
||||
let ctx = ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut f = Some(f);
|
||||
let mut out = None;
|
||||
let _ = ctx.run(egui::RawInput::default(), |ctx| {
|
||||
|
|
@ -148,31 +101,6 @@ pub fn painted_spans(out: &egui::FullOutput) -> Vec<(String, egui::Color32)> {
|
|||
.collect()
|
||||
}
|
||||
|
||||
/// Every styled run painted this frame that has a background behind it, with
|
||||
/// that background, in paint order.
|
||||
///
|
||||
/// The distinguishing mark of a highlighted match: the column headers and the
|
||||
/// strong parts of a snippet are painted in the same *text* color, so
|
||||
/// [`painted_spans`] alone cannot tell a match from a header.
|
||||
pub fn painted_backgrounds(out: &egui::FullOutput) -> Vec<(String, egui::Color32)> {
|
||||
painted_galleys(out)
|
||||
.into_iter()
|
||||
.flat_map(|(g, _)| {
|
||||
g.job
|
||||
.sections
|
||||
.iter()
|
||||
.filter(|s| s.format.background != egui::Color32::TRANSPARENT)
|
||||
.map(|s| {
|
||||
(
|
||||
g.job.text[s.byte_range.clone()].to_string(),
|
||||
s.format.background,
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Every *visible* row of every galley painted this frame, in paint order.
|
||||
/// Not the same as [`painted_text`]: a galley's `text()` is the job it was
|
||||
/// laid out from, including the rows epaint dropped at `wrap.max_rows` —
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
//! Plain-language tooltips for the configuration controls: every setting in
|
||||
//! the Settings tab, and every configuration control on the Manage Index
|
||||
//! the Options window, and every configuration control on the Manage Index
|
||||
//! tab, explains itself on hover.
|
||||
|
||||
/// How wide a tooltip may get; matches `manage_tab::db_size_tooltip`.
|
||||
|
|
@ -73,7 +73,7 @@ pub fn tip_row(
|
|||
ui.end_row();
|
||||
}
|
||||
|
||||
// --- Settings: Paths ------------------------------------------------------
|
||||
// --- Options: Paths ------------------------------------------------------
|
||||
|
||||
pub static DATABASE_PATH: Tip = Tip {
|
||||
title: "Database file",
|
||||
|
|
@ -90,7 +90,7 @@ pub static DATABASE_PATH: Tip = Tip {
|
|||
caution: None,
|
||||
};
|
||||
|
||||
// --- Settings: Indexing ---------------------------------------------------
|
||||
// --- Options: Indexing ---------------------------------------------------
|
||||
|
||||
pub static REINDEX_INTERVAL: Tip = Tip {
|
||||
title: "Full reindex every",
|
||||
|
|
@ -139,7 +139,7 @@ pub static INCLUDE_HIDDEN: Tip = Tip {
|
|||
caution: None,
|
||||
};
|
||||
|
||||
// --- Settings: Processing -------------------------------------------------
|
||||
// --- Options: Processing -------------------------------------------------
|
||||
|
||||
pub static TOKENIZER: Tip = Tip {
|
||||
title: "Tokenizer",
|
||||
|
|
@ -257,7 +257,7 @@ pub static STORE_TEXT: Tip = Tip {
|
|||
caution: None,
|
||||
};
|
||||
|
||||
// --- Settings: Search -----------------------------------------------------
|
||||
// --- Options: Search -----------------------------------------------------
|
||||
|
||||
pub static FUZZY_DEFAULT: Tip = Tip {
|
||||
title: "Fuzzy search ON by default",
|
||||
|
|
@ -329,43 +329,7 @@ pub static DEBOUNCE: Tip = Tip {
|
|||
caution: None,
|
||||
};
|
||||
|
||||
pub static LIVE_RESULTS: Tip = Tip {
|
||||
title: "Live results",
|
||||
body: "Watches the results currently on screen and updates them as the \
|
||||
files change, so a file you rename or edit in another window does \
|
||||
not sit there showing its old name or its old text.\n\n\
|
||||
Only the rows you can actually see are watched, and every one of \
|
||||
them is dropped the moment you change the search. Nothing is ever \
|
||||
added, removed or re-sorted while you read — a file that stops \
|
||||
matching stays where it is until you search again.",
|
||||
examples: &[
|
||||
"Renames, deletions and edited contents all show up within a second, \
|
||||
whatever the indexer is doing — what a row says is read from the \
|
||||
file, not from the index.",
|
||||
"A row is also checked against the disk as it comes on screen, so one \
|
||||
the index was out of date about puts itself right. That check is all \
|
||||
you get over a network share, where the system does not report other \
|
||||
machines' writes.",
|
||||
],
|
||||
caution: Some(
|
||||
"The files behind the rows you are looking at are kept up to date in \
|
||||
the index too, even while indexing is stopped. Turn this off if a \
|
||||
stopped index must mean nothing is written at all.",
|
||||
),
|
||||
};
|
||||
|
||||
pub static COLUMNS: Tip = Tip {
|
||||
title: "Search columns",
|
||||
body: "Which columns the results table shows. The path is always there — \
|
||||
it is the only column that identifies a result on its own.\n\n\
|
||||
Size and modified date start switched off: the width they take is \
|
||||
usually better spent on the path and the matched text. Turning a \
|
||||
column on also makes it available to sort by.",
|
||||
examples: &["Right-clicking any column header on the Search tab does the same thing."],
|
||||
caution: None,
|
||||
};
|
||||
|
||||
// --- Settings: Interface --------------------------------------------------
|
||||
// --- Options: Interface --------------------------------------------------
|
||||
|
||||
pub static UI_SCALE: Tip = Tip {
|
||||
title: "UI scale",
|
||||
|
|
@ -416,7 +380,7 @@ pub static COLOR_SCHEME: Tip = Tip {
|
|||
caution: None,
|
||||
};
|
||||
|
||||
// --- Settings: Security ---------------------------------------------------
|
||||
// --- Options: Security ---------------------------------------------------
|
||||
|
||||
pub static ENABLE_PASSWORD: Tip = Tip {
|
||||
title: "Enable password protection",
|
||||
|
|
@ -455,21 +419,6 @@ pub static DISABLE_PASSWORD: Tip = Tip {
|
|||
),
|
||||
};
|
||||
|
||||
pub static SHOW_KEY: Tip = Tip {
|
||||
title: "Show database key",
|
||||
body: "Reveals the raw SQLCipher key the index is encrypted with, once you \
|
||||
have confirmed your password. Tools such as DB Browser for SQLCipher \
|
||||
accept it in the 0x form shown and can then open the index file \
|
||||
directly.\n\n\
|
||||
The key is worked out from your password and the salt in the config \
|
||||
file, so it stays the same until the password changes.",
|
||||
examples: &[],
|
||||
caution: Some(
|
||||
"Anyone holding this key can read the index without the password. Treat a copy \
|
||||
of it as carefully as the password itself.",
|
||||
),
|
||||
};
|
||||
|
||||
pub static REMEMBER_KEYCHAIN: Tip = Tip {
|
||||
title: "Remember on this device",
|
||||
body: "Hands the key to the password store your system already has, such \
|
||||
|
|
@ -667,7 +616,6 @@ mod tests {
|
|||
&ENABLE_PASSWORD,
|
||||
&CHANGE_PASSWORD,
|
||||
&DISABLE_PASSWORD,
|
||||
&SHOW_KEY,
|
||||
&REMEMBER_KEYCHAIN,
|
||||
&START_NOW,
|
||||
&STOP_INDEXING,
|
||||
|
|
@ -777,7 +725,7 @@ mod tests {
|
|||
/// caution. Written against the tip with all four.
|
||||
#[test]
|
||||
fn show_paints_every_part() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
|
||||
let full = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| TOKENIZER.show(ui));
|
||||
|
|
@ -795,7 +743,7 @@ mod tests {
|
|||
/// A lone example reads as a sentence rather than a one-item list.
|
||||
#[test]
|
||||
fn a_single_example_is_prefixed_with_example() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let input = crate::test_ui::raw_input(egui::vec2(800.0, 600.0), vec![]);
|
||||
let full = ctx.run(input, |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| INCLUDE_HIDDEN.show(ui));
|
||||
|
|
@ -811,7 +759,7 @@ mod tests {
|
|||
/// disabled widget unless the *disabled* tooltip is set too.
|
||||
#[test]
|
||||
fn a_disabled_control_still_explains_itself() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
ctx.style_mut(|s| {
|
||||
s.interaction.tooltip_delay = 0.0;
|
||||
s.interaction.show_tooltips_only_when_still = false;
|
||||
|
|
|
|||
|
|
@ -1,441 +0,0 @@
|
|||
//! The first-start tour: a few pages explaining what QuickSearch indexes,
|
||||
//! how results are ranked, and what the parts of the Search tab do.
|
||||
//!
|
||||
//! Shown once, to an installation that has never run before — `[ui]
|
||||
//! tutorial_seen` is `Some(false)` only in a config file this version created,
|
||||
//! so upgrading into this version does not summon it. The Help tab can bring
|
||||
//! it back afterwards, which is also what keeps this from being write-only.
|
||||
|
||||
use crate::ui_util::{centered_modal, hint};
|
||||
|
||||
/// One page of the tour. Static text, so the pages are a table rather than a
|
||||
/// match arm each.
|
||||
struct Page {
|
||||
title: &'static str,
|
||||
/// Paragraphs. Rendered in order with a little space between them.
|
||||
body: &'static [&'static str],
|
||||
/// Rendered small and de-emphasised under the body — where to go, rather
|
||||
/// than what the thing is.
|
||||
pointer: Option<&'static str>,
|
||||
}
|
||||
|
||||
const PAGES: &[Page] = &[
|
||||
Page {
|
||||
title: "Welcome to QuickSearch",
|
||||
body: &[
|
||||
"QuickSearch keeps an index of the folders you choose, and searches \
|
||||
as you type.",
|
||||
"By default only your user folder is indexed and searchable.",
|
||||
"Because the answers come from the index rather than from reading \
|
||||
your disk, results appear as fast as you can type, even across \
|
||||
hundreds of thousands of files.",
|
||||
],
|
||||
pointer: Some("A quick tutorial for new users! Hit Skip to Exit."),
|
||||
},
|
||||
Page {
|
||||
title: "What is indexed",
|
||||
body: &[
|
||||
"Indexing is QuickSearch reading through your folders once and \
|
||||
remembering what it found, so that searching later is instant. It \
|
||||
runs on its own in the background and keeps up with changes as you \
|
||||
make them.",
|
||||
"QuickSearch never connects to the internet, and always respects your privacy. \
|
||||
QuickSearch can encrypt your index to make this remembered data more secure.",
|
||||
"These are the folders being indexed right now:",
|
||||
],
|
||||
pointer: Some(
|
||||
"To index more locations, open the Manage Index tab and add a folder. \
|
||||
To set an index password, look near the bottom of the Settings tab.",
|
||||
),
|
||||
},
|
||||
Page {
|
||||
title: "How results are ranked",
|
||||
body: &[
|
||||
"The best search matches come first (have the lowest rank). \
|
||||
Exact file-name matchs are best, then names that contain what you typed; then \
|
||||
files whose contents contain the search terms, the ones mentioning it most often \
|
||||
first; and last, files matched only by their full folder path.",
|
||||
"The coloured number in the Rank column is which of those tiers a \
|
||||
result came from: blue is a great match, red is a distant one. \
|
||||
Clicking a column heading sorts by something else instead.",
|
||||
],
|
||||
pointer: Some("Right-click any column heading to choose which columns are shown."),
|
||||
},
|
||||
Page {
|
||||
title: "The status bar",
|
||||
body: &[
|
||||
"The line along the bottom of the window is what QuickSearch is \
|
||||
doing. While it is indexing it shows the phase, how far through it \
|
||||
is, and how fast; when it has nothing to do it shows how many files \
|
||||
are indexed.",
|
||||
"Searching works the whole time, including during that first indexing run, \
|
||||
but some files might not be shown in the results until the scan completes.",
|
||||
],
|
||||
pointer: None,
|
||||
},
|
||||
Page {
|
||||
title: "Typos, and what a result can do",
|
||||
body: &[
|
||||
"Tick \"Fuzzy\"beside the search box to also match words with typos \
|
||||
in them — \"repot\" will find \"report\". It searches more \
|
||||
thoroughly, so it is a little slower; leave it off until you need \
|
||||
it.",
|
||||
"Right-click any result for more: open it, open the folder holding \
|
||||
it, copy its path, or build a filter that hides files like it from \
|
||||
future searches.",
|
||||
],
|
||||
pointer: Some(
|
||||
"The ? button left of the search box lists the filters you can type \
|
||||
into a query, like type:Document or modified:>=2024-01-01.",
|
||||
),
|
||||
},
|
||||
Page {
|
||||
title: "Duplicates",
|
||||
body: &[
|
||||
"The Duplicates tab looks for files across all indexed folders for identical copies. \
|
||||
They are shown grouped together, with the largest wasted space first.",
|
||||
"It is a quick way to find the same download sitting in three \
|
||||
places. QuickSearch only shows you the groups; deleting anything is \
|
||||
left to you.",
|
||||
],
|
||||
pointer: Some(
|
||||
"For speed, files are compared by size and by how they begin (first 8KB). \
|
||||
This is not a guarantee of an exact match. You can right click a result to verify before you delete anything.",
|
||||
),
|
||||
},
|
||||
Page {
|
||||
title: "Settings",
|
||||
body: &[
|
||||
"The Settings tab, at the right-hand end of the tab strip, is where \
|
||||
you can tweak and tune the software. Mouse over any of \
|
||||
the settings for a brief description of what they do.",
|
||||
"Most changes wait for the Apply & Save button at the bottom.",
|
||||
"QuickSearch is completely free for anyone to use. If you love it, please let your friends know about us!",
|
||||
],
|
||||
pointer: None,
|
||||
},
|
||||
];
|
||||
|
||||
/// The open tour.
|
||||
pub struct Tutorial {
|
||||
page: usize,
|
||||
}
|
||||
|
||||
impl Tutorial {
|
||||
pub fn new() -> Tutorial {
|
||||
Tutorial { page: 0 }
|
||||
}
|
||||
|
||||
/// Render. `roots` is the live indexed-folder list, named on the page
|
||||
/// about indexing so the tour describes this installation rather than a
|
||||
/// generic one.
|
||||
/// Returns true once the tour is finished with — skipped or read to the
|
||||
/// end — which is the caller's cue to remember that and drop it.
|
||||
pub fn ui(&mut self, ctx: &egui::Context, roots: &[String]) -> bool {
|
||||
let page = &PAGES[self.page.min(PAGES.len() - 1)];
|
||||
let (first, last) = (self.page == 0, self.page + 1 == PAGES.len());
|
||||
let mut dismissed = false;
|
||||
let mut step: i64 = 0;
|
||||
|
||||
centered_modal(ctx, page.title, |ui| {
|
||||
ui.set_max_width(520.0);
|
||||
for paragraph in page.body {
|
||||
ui.label(*paragraph);
|
||||
ui.add_space(6.0);
|
||||
}
|
||||
// The one page that shows live state rather than static text.
|
||||
if self.page == 1 {
|
||||
if roots.is_empty() {
|
||||
ui.label(hint("No folders are indexed yet."));
|
||||
} else {
|
||||
for root in roots {
|
||||
ui.monospace(root);
|
||||
}
|
||||
}
|
||||
ui.add_space(6.0);
|
||||
}
|
||||
if let Some(pointer) = page.pointer {
|
||||
ui.label(hint(pointer));
|
||||
}
|
||||
|
||||
ui.add_space(10.0);
|
||||
ui.separator();
|
||||
// Three equal thirds rather than one row: it is the only layout
|
||||
// that puts Skip in the middle without measuring the buttons
|
||||
// either side of it, whose widths change with the page ("Next"
|
||||
// becoming "Finish") and with the counter's digits.
|
||||
ui.columns(3, |cols| {
|
||||
// A column lays its contents out *justified*, so a button put
|
||||
// straight into one is stretched to the full third. The other
|
||||
// two escape that by nesting their own layout; this one has to
|
||||
// say so.
|
||||
cols[0].with_layout(egui::Layout::left_to_right(egui::Align::Min), |ui| {
|
||||
if ui.add_enabled(!first, egui::Button::new("Back")).clicked() {
|
||||
step = -1;
|
||||
}
|
||||
});
|
||||
cols[1].vertical_centered(|ui| {
|
||||
if ui.button("Skip").clicked() {
|
||||
dismissed = true;
|
||||
}
|
||||
});
|
||||
// `Align::Min`, not `Center`: a column is as tall as the rest
|
||||
// of the window, so centring in it drops the button a hundred
|
||||
// points below the two beside it.
|
||||
cols[2].with_layout(egui::Layout::right_to_left(egui::Align::Min), |ui| {
|
||||
let p = crate::color::palette(ui.visuals().dark_mode);
|
||||
let next = if last { "Finish" } else { "Next" };
|
||||
if ui
|
||||
.add(crate::ui_util::bordered_button(next, p.blue))
|
||||
.clicked()
|
||||
{
|
||||
if last {
|
||||
dismissed = true;
|
||||
} else {
|
||||
step = 1;
|
||||
}
|
||||
}
|
||||
ui.label(hint(format!("{} of {}", self.page + 1, PAGES.len())));
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// Applied after the closure so the page a frame rendered stays the page
|
||||
// its buttons were laid out for.
|
||||
if step != 0 {
|
||||
let next = self.page as i64 + step;
|
||||
self.page = next.clamp(0, PAGES.len() as i64 - 1) as usize;
|
||||
}
|
||||
dismissed
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::test_ui::{click_at, painted_text, raw_input};
|
||||
|
||||
const SCREEN: egui::Vec2 = egui::vec2(1000.0, 700.0);
|
||||
|
||||
/// Two passes: an `egui::Window` is measured on its first frame and only
|
||||
/// placed on the next, so a single pass paints nothing to read back and
|
||||
/// has nothing at a known position to click.
|
||||
fn frame(
|
||||
ctx: &egui::Context,
|
||||
tour: &mut Tutorial,
|
||||
events: Vec<egui::Event>,
|
||||
) -> (egui::FullOutput, bool) {
|
||||
let roots = ["/home/me".to_string()];
|
||||
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
tour.ui(ctx, &roots);
|
||||
});
|
||||
let mut dismissed = false;
|
||||
let out = ctx.run(raw_input(SCREEN, events), |ctx| {
|
||||
dismissed = tour.ui(ctx, &roots);
|
||||
});
|
||||
(out, dismissed)
|
||||
}
|
||||
|
||||
/// Every page has something to say, and says it.
|
||||
#[test]
|
||||
fn every_page_paints_its_own_title_and_body() {
|
||||
for (page, spec) in PAGES.iter().enumerate() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tour = Tutorial { page };
|
||||
let (out, _) = frame(&ctx, &mut tour, Vec::new());
|
||||
let painted = painted_text(&out);
|
||||
assert!(
|
||||
painted.iter().any(|t| t == spec.title),
|
||||
"page {page} painted no title: {painted:?}"
|
||||
);
|
||||
assert!(!spec.body.is_empty(), "page {page} has an empty body");
|
||||
assert!(
|
||||
painted
|
||||
.iter()
|
||||
.any(|t| t == &format!("{} of {}", page + 1, PAGES.len())),
|
||||
"page {page} did not say where it is: {painted:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The page about indexing names *this* installation's folders, not a
|
||||
/// generic example.
|
||||
#[test]
|
||||
fn the_indexing_page_lists_the_configured_folders() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tour = Tutorial { page: 1 };
|
||||
let roots = ["/srv/projects".to_string()];
|
||||
let _ = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
tour.ui(ctx, &roots);
|
||||
});
|
||||
let out = ctx.run(raw_input(SCREEN, Vec::new()), |ctx| {
|
||||
tour.ui(ctx, &roots);
|
||||
});
|
||||
let painted = painted_text(&out);
|
||||
assert!(painted.iter().any(|t| t == "/srv/projects"), "{painted:?}");
|
||||
}
|
||||
|
||||
/// The y of the footer row on `page`, found by walking down the middle
|
||||
/// column, which only Skip occupies. Wrapped text height moves the row
|
||||
/// from page to page, so it is probed rather than guessed.
|
||||
fn footer_y(ctx: &egui::Context, page: usize) -> f32 {
|
||||
for y in (150..600).step_by(2) {
|
||||
let mut t = Tutorial { page };
|
||||
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(500.0, y as f32)));
|
||||
if dismissed {
|
||||
return y as f32;
|
||||
}
|
||||
}
|
||||
panic!("no Skip button down the middle of page {page}");
|
||||
}
|
||||
|
||||
/// The stretch of x along the footer row that one button answers a click
|
||||
/// on — where it is, and how wide.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct Span {
|
||||
lo: f32,
|
||||
hi: f32,
|
||||
}
|
||||
|
||||
impl Span {
|
||||
fn width(&self) -> f32 {
|
||||
self.hi - self.lo
|
||||
}
|
||||
}
|
||||
|
||||
/// The three footer buttons, found by what clicking each one does: Back
|
||||
/// steps a page back, Next steps one forward, Skip dismisses.
|
||||
///
|
||||
/// Must be run on a middle page — on the last page Finish and Skip both
|
||||
/// dismiss without moving, and on the first Back is disabled.
|
||||
fn footer_spans(ctx: &egui::Context, page: usize) -> [Option<Span>; 3] {
|
||||
assert!(page > 0 && page + 1 < PAGES.len(), "probe a middle page");
|
||||
let y = footer_y(ctx, page);
|
||||
let mut spans: [Option<Span>; 3] = [None; 3];
|
||||
for x in 150..850 {
|
||||
let mut t = Tutorial { page };
|
||||
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(x as f32, y)));
|
||||
let x = x as f32;
|
||||
let which = if dismissed {
|
||||
1 // Skip
|
||||
} else if t.page + 1 == page {
|
||||
0 // Back
|
||||
} else if t.page == page + 1 {
|
||||
2 // Next
|
||||
} else {
|
||||
continue;
|
||||
};
|
||||
match &mut spans[which] {
|
||||
Some(span) => span.hi = x,
|
||||
slot => *slot = Some(Span { lo: x, hi: x }),
|
||||
}
|
||||
}
|
||||
spans
|
||||
}
|
||||
|
||||
/// The footer reads Back, then Skip, then Next — and each does what its
|
||||
/// label says. Positions are probed rather than asserted against numbers:
|
||||
/// the window auto-sizes to the page's text, so the thirds move.
|
||||
///
|
||||
/// The widths are the other half of it. `Ui::columns` lays a column out
|
||||
/// justified, so a button dropped straight into one comes out as wide as
|
||||
/// the whole third — which is what Back was until it was given a layout of
|
||||
/// its own. Two buttons with four-letter labels either side of the row
|
||||
/// have to come out the same size.
|
||||
#[test]
|
||||
fn the_footer_runs_back_then_skip_then_next_at_the_same_size() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let [back, skip, next] = footer_spans(&ctx, 1);
|
||||
let back = back.expect("no Back button in the footer");
|
||||
let skip = skip.expect("no Skip button in the footer");
|
||||
let next = next.expect("no Next button in the footer");
|
||||
assert!(
|
||||
back.lo < skip.lo,
|
||||
"Back ({back:?}) is not left of Skip ({skip:?})"
|
||||
);
|
||||
assert!(
|
||||
skip.lo < next.lo,
|
||||
"Skip ({skip:?}) is not left of Next ({next:?})"
|
||||
);
|
||||
|
||||
assert!(
|
||||
(back.width() - next.width()).abs() <= 2.0,
|
||||
"Back is {} wide against Next's {}",
|
||||
back.width(),
|
||||
next.width()
|
||||
);
|
||||
// Belt and braces on the shape of the bug: a stretched button fills
|
||||
// its third of a 520-point modal, which no four-letter label does.
|
||||
assert!(
|
||||
back.width() < 80.0,
|
||||
"Back is stretched to {} points",
|
||||
back.width()
|
||||
);
|
||||
}
|
||||
|
||||
/// Back is disabled on the first page, so nothing in the footer can walk
|
||||
/// the tour off the front.
|
||||
#[test]
|
||||
fn the_first_page_cannot_go_back() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let y = footer_y(&ctx, 0);
|
||||
for x in (150..850).step_by(4) {
|
||||
let mut t = Tutorial { page: 0 };
|
||||
let _ = frame(&ctx, &mut t, click_at(egui::pos2(x as f32, y)));
|
||||
assert!(
|
||||
t.page == 0 || t.page == 1,
|
||||
"clicking x={x} left page {}",
|
||||
t.page
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Clicking anywhere in the button row, on any page; collects what fired.
|
||||
fn sweep(ctx: &egui::Context, tour: &mut Tutorial) -> Vec<usize> {
|
||||
let mut seen = Vec::new();
|
||||
for y in (200..500).step_by(4) {
|
||||
for x in (240..760).step_by(8) {
|
||||
let mut t = Tutorial { page: tour.page };
|
||||
let (_, dismissed) = frame(ctx, &mut t, click_at(egui::pos2(x as f32, y as f32)));
|
||||
if dismissed {
|
||||
seen.push(t.page);
|
||||
}
|
||||
}
|
||||
}
|
||||
seen
|
||||
}
|
||||
|
||||
/// Both ways out of the tour report the dismissal, so the flag gets set
|
||||
/// whichever the user takes. Positions depend on wrapped text height, so
|
||||
/// the button row is swept rather than guessed at — the same approach the
|
||||
/// confirmation modals' tests take.
|
||||
#[test]
|
||||
fn skip_and_finish_both_dismiss() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
|
||||
// Skip is on every page.
|
||||
let mut tour = Tutorial { page: 0 };
|
||||
assert!(
|
||||
!sweep(&ctx, &mut tour).is_empty(),
|
||||
"Skip never fired on the first page"
|
||||
);
|
||||
|
||||
// Finish only on the last, where it replaces Next.
|
||||
let mut tour = Tutorial {
|
||||
page: PAGES.len() - 1,
|
||||
};
|
||||
assert!(
|
||||
!sweep(&ctx, &mut tour).is_empty(),
|
||||
"Finish never fired on the last page"
|
||||
);
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let mut tour = Tutorial {
|
||||
page: PAGES.len() - 1,
|
||||
};
|
||||
let (out, _) = frame(&ctx, &mut tour, Vec::new());
|
||||
assert!(
|
||||
painted_text(&out).contains(&"Finish".to_string()),
|
||||
"the last page still offers Next"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -122,10 +122,8 @@ pub fn pattern_edit(
|
|||
(response, valid)
|
||||
}
|
||||
|
||||
/// Where a middle-elide cuts `text` to fit `max_width` pixels in `font_id`:
|
||||
/// the byte offset the head keeps up to, and the one the tail resumes from,
|
||||
/// with a single `…` standing for everything between. `None` when the whole
|
||||
/// string fits and nothing is dropped.
|
||||
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
|
||||
/// `font_id`, returning it borrowed and untouched when it already fits.
|
||||
///
|
||||
/// The budget is in pixels, summed from the font's own glyph advances (the
|
||||
/// same numbers egui's layout adds up), not a character count scaled by one
|
||||
|
|
@ -133,20 +131,18 @@ pub fn pattern_edit(
|
|||
/// directions: overshoot and egui elides the result a *second* time,
|
||||
/// painting two ellipses; undershoot and the column sits visibly short.
|
||||
///
|
||||
/// Split out from [`middle_elide`] because a caller that also has *ranges* to
|
||||
/// highlight needs the cut itself, not just the shortened string: it renders
|
||||
/// the two surviving ends separately so its marks keep the offsets they had
|
||||
/// (see `snippet_render::path_cell_job`).
|
||||
pub fn middle_elide_cut(
|
||||
/// The borrowed/owned distinction is the caller's signal that something was
|
||||
/// dropped, which is what a "full text on hover" tooltip keys off.
|
||||
pub fn middle_elide<'a>(
|
||||
ui: &egui::Ui,
|
||||
text: &str,
|
||||
text: &'a str,
|
||||
max_width: f32,
|
||||
font_id: &egui::FontId,
|
||||
) -> Option<(usize, usize)> {
|
||||
) -> Cow<'a, str> {
|
||||
ui.fonts(|f| {
|
||||
let width_of = |c: char| f.glyph_width(font_id, c);
|
||||
if text.chars().map(width_of).sum::<f32>() <= max_width {
|
||||
return None;
|
||||
return Cow::Borrowed(text);
|
||||
}
|
||||
let budget = max_width - width_of('…');
|
||||
|
||||
|
|
@ -182,33 +178,15 @@ pub fn middle_elide_cut(
|
|||
}
|
||||
if head >= tail {
|
||||
// The two halves met without dropping anything.
|
||||
return None;
|
||||
}
|
||||
Some((head, tail))
|
||||
})
|
||||
return Cow::Borrowed(text);
|
||||
}
|
||||
|
||||
/// Middle-elide `text` so it fits `max_width` pixels when laid out in
|
||||
/// `font_id`, returning it borrowed and untouched when it already fits.
|
||||
///
|
||||
/// The borrowed/owned distinction is the caller's signal that something was
|
||||
/// dropped, which is what a "full text on hover" tooltip keys off.
|
||||
pub fn middle_elide<'a>(
|
||||
ui: &egui::Ui,
|
||||
text: &'a str,
|
||||
max_width: f32,
|
||||
font_id: &egui::FontId,
|
||||
) -> Cow<'a, str> {
|
||||
match middle_elide_cut(ui, text, max_width, font_id) {
|
||||
None => Cow::Borrowed(text),
|
||||
Some((head, tail)) => {
|
||||
let mut out = String::with_capacity(head + '…'.len_utf8() + (text.len() - tail));
|
||||
out.push_str(&text[..head]);
|
||||
out.push('…');
|
||||
out.push_str(&text[tail..]);
|
||||
Cow::Owned(out)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Paint a semitransparent down-arrow near the bottom edge of a scroll
|
||||
|
|
|
|||
|
|
@ -72,12 +72,12 @@ impl Gate {
|
|||
return;
|
||||
}
|
||||
if let Gate::Running(app) = self {
|
||||
// The Settings tab is waiting for a key press to bind; the
|
||||
// shortcut must not reshuffle the window underneath it.
|
||||
// The Options window is waiting for a key press to bind; the
|
||||
// shortcut must not reshuffle the window underneath that dialog.
|
||||
if app.capturing_hotkey() {
|
||||
return;
|
||||
}
|
||||
app.activate_search(ctx);
|
||||
app.activate_search();
|
||||
}
|
||||
crate::hotkey::raise(ctx, frame);
|
||||
}
|
||||
|
|
@ -534,7 +534,7 @@ mod tests {
|
|||
/// leave it.
|
||||
#[test]
|
||||
fn the_password_field_takes_focus_once_and_then_releases_it() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut screen = UnlockScreen::new(locked_config(), None, None);
|
||||
|
||||
frame(&ctx, &mut screen);
|
||||
|
|
@ -560,7 +560,7 @@ mod tests {
|
|||
/// was cleared on submit.
|
||||
#[test]
|
||||
fn a_failed_attempt_puts_the_caret_back() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut screen = UnlockScreen::new(locked_config(), None, None);
|
||||
frame(&ctx, &mut screen);
|
||||
ctx.memory_mut(|m| m.surrender_focus(pw_field_id()));
|
||||
|
|
@ -576,7 +576,7 @@ mod tests {
|
|||
/// the central one would compile and show nothing.
|
||||
#[test]
|
||||
fn the_lock_screen_shows_the_build_id() {
|
||||
let ctx = crate::test_ui::ctx();
|
||||
let ctx = egui::Context::default();
|
||||
let mut screen = UnlockScreen::new(locked_config(), None, None);
|
||||
let input = crate::test_ui::raw_input(SCREEN, Vec::new());
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
# "capture"). Edit this file to re-choreograph the captures; timings are in
|
||||
# milliseconds and `max` caps bound clip length without failing the run.
|
||||
|
||||
# The demo config sets [ui] scale = 1.25, so every window size here is 1.25x
|
||||
# The demo config sets [ui] scale = 1.5, so every window size here is 1.5x
|
||||
# the layout it shows: high-resolution captures of an unchanged layout.
|
||||
|
||||
# --- 1. manage-indexing.webm: the fresh auto-index in progress -------------
|
||||
|
|
@ -26,12 +26,8 @@ focus_search # tab switches drop egui focus; re-arm it
|
|||
clear_query
|
||||
window 1200 600 # compact clip: the smallest layout at
|
||||
# which every results column still fits
|
||||
# (any narrower clips the Content Match
|
||||
# column away, defeating the demo), at
|
||||
# 1.25x. Sized for the four columns the
|
||||
# demo config pins — Name, Path, Content
|
||||
# Match, Rank — so turning another one on
|
||||
# there means revisiting this width.
|
||||
# (any narrower clips the Match column
|
||||
# away, defeating the demo), at 1.5x
|
||||
wait_ms 800 # the resize lands asynchronously
|
||||
record_start search
|
||||
# The query is typed in quick bursts with human hesitations; each pause
|
||||
|
|
@ -46,8 +42,8 @@ wait_ms 300
|
|||
type "ition" cps 10
|
||||
wait_search_done max 8000
|
||||
wait_ms 1000
|
||||
hover_match 2 # pin the pointer on the 3rd result's Content
|
||||
# Match cell: the tooltip expands the snippet
|
||||
hover_match 2 # pin the pointer on the 3rd result's Match
|
||||
# cell: the tooltip expands the snippet
|
||||
# with surrounding file content
|
||||
wait_ms 2500 # tooltip delay, then linger on it
|
||||
record_stop # end the clip with the tooltip on screen
|
||||
|
|
|
|||
|
|
@ -68,34 +68,8 @@ auto_index = true
|
|||
include_hidden = true
|
||||
ignore_patterns = []
|
||||
|
||||
[search]
|
||||
# Off for the run: on, the watcher arms over the visible rows mid-recording,
|
||||
# re-reads them from disk and hands the paths back for reindexing — index
|
||||
# writes inside a clip that is supposed to be reproducible.
|
||||
live_results = false
|
||||
|
||||
[search.columns]
|
||||
# Exactly the four the captures are choreographed around: Name, Path, Content
|
||||
# Match, Rank. Every one is pinned rather than defaulted — the scenario's
|
||||
# window widths are chosen for this column set, and `hover_match` addresses
|
||||
# Content Match cells by index, so a shipped default that moved would silently
|
||||
# re-frame or break the clips.
|
||||
#
|
||||
# Path is absent because it is not optional: it is always drawn (see
|
||||
# `ColumnsConfig`, which deliberately does not represent it).
|
||||
name = true
|
||||
content_match = true
|
||||
rank = true
|
||||
size = false
|
||||
modified = false
|
||||
|
||||
# 1.25x zoom + proportionally larger windows (set in the scenario) render the
|
||||
# same layout at ~1.25x the pixel density, for crisper website assets.
|
||||
#
|
||||
# No `tutorial_seen` key here, deliberately: absent it deserializes to `None`,
|
||||
# which reads as "an installation that upgraded into this version". Writing
|
||||
# `false` would open the first-run tour over the first capture and wedge the
|
||||
# run.
|
||||
# 1.5x zoom + proportionally larger windows (set in the scenario) render the
|
||||
# same layout at ~1.5x the pixel density, for crisper website assets.
|
||||
[ui]
|
||||
scale = 1.25
|
||||
EOF
|
||||
|
|
|
|||
|
|
@ -11,25 +11,6 @@ Files: crates/quicksearch-gui/assets/icons/*
|
|||
Copyright: 2026 Jeremy <jeremy@karsttech.com>
|
||||
License: GPL-3.0-or-later
|
||||
|
||||
Files: crates/quicksearch-gui/assets/fonts/Ubuntu-Light.ttf
|
||||
Copyright: 2010-2011 Canonical Ltd.
|
||||
License: Ubuntu-font-1.0
|
||||
The full text is shipped beside the font as
|
||||
crates/quicksearch-gui/assets/fonts/UFL.txt. The file is redistributed
|
||||
unmodified, so the licence's naming conditions on Modified Versions do not
|
||||
arise.
|
||||
|
||||
Files: crates/quicksearch-gui/assets/fonts/Hack-Regular.ttf
|
||||
Copyright: 2018 Source Foundry Authors
|
||||
2003 Bitstream, Inc.
|
||||
License: MIT and Bitstream-Vera
|
||||
Hack is MIT-licensed work over the public-domain DejaVu project, which is in
|
||||
turn derived from Bitstream Vera Sans Mono under the Bitstream Vera License
|
||||
with Reserved Font Names "Bitstream" and "Vera". The full text of both is
|
||||
shipped beside the font as
|
||||
crates/quicksearch-gui/assets/fonts/Hack-Regular.txt. The file is
|
||||
redistributed unmodified, so the Bitstream renaming condition does not arise.
|
||||
|
||||
License: GPL-3.0-or-later
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
|
|
@ -51,7 +32,7 @@ Comment:
|
|||
/usr/bin/quicksearch and /usr/bin/quicksearch-cli are statically linked Rust
|
||||
binaries built from the same crates. Their third-party components are not
|
||||
shipped as separate files, so they are summarised here
|
||||
rather than given individual stanzas. Every one of the 525 crates in
|
||||
rather than given individual stanzas. Every one of the 512 crates in
|
||||
Cargo.lock is under a permissive licence, all of which are compatible with
|
||||
GPL-3.0-or-later:
|
||||
.
|
||||
|
|
@ -69,14 +50,12 @@ Comment:
|
|||
rusqlite), Zstandard (BSD-3-Clause arm of its BSD/GPL-2.0 dual licence,
|
||||
via zstd-sys) and bzip2 (BSD-style, via bzip2-sys).
|
||||
.
|
||||
The binary also embeds the two font files under
|
||||
crates/quicksearch-gui/assets/fonts/, which have their own stanzas above.
|
||||
egui is built with default-features = false so that it bundles no fonts of
|
||||
its own; the OFL-1.1 emoji faces it would otherwise embed are not shipped,
|
||||
and OFL-1.1 accordingly no longer applies to this package. The FSF regards
|
||||
the Ubuntu Font Licence as free but GPL-incompatible; it is included here as
|
||||
font data rather than as linked program code, which is the same basis on
|
||||
which Debian ships these fonts and other egui-based applications.
|
||||
The binary also embeds egui's default fonts (crate epaint_default_fonts,
|
||||
"(MIT OR Apache-2.0) AND OFL-1.1 AND Ubuntu-font-1.0"). The FSF regards
|
||||
OFL-1.1 and the Ubuntu Font Licence as free but GPL-incompatible; they are
|
||||
included here as font data rather than as linked program code, which is
|
||||
the same basis on which Debian ships these fonts and other egui-based
|
||||
applications.
|
||||
.
|
||||
Run `cargo metadata --all-features` against the source tree to reproduce
|
||||
the per-crate licence list.
|
||||
|
|
|
|||
|
|
@ -136,7 +136,7 @@ text stays part of the search phrase.
|
|||
.BR AND ", " OR
|
||||
and parentheses are treated as plain words.
|
||||
.SH PASSWORD PROTECTION
|
||||
The index can be encrypted with a password (the Settings tab, Security).
|
||||
The index can be encrypted with a password (application Options, Security).
|
||||
A protected index must be unlocked every time either binary starts. The
|
||||
application shows an unlock screen; terminal mode resolves the key from, in
|
||||
order: the OS keychain (when \(lqRemember on this device\(rq is enabled),
|
||||
|
|
|
|||
|
|
@ -61,18 +61,6 @@ VIAddVersionKey "FileDescription" "${APP} ${VERSION} installer"
|
|||
!insertmacro MUI_PAGE_COMPONENTS
|
||||
!insertmacro MUI_PAGE_DIRECTORY
|
||||
!insertmacro MUI_PAGE_INSTFILES
|
||||
; The payload is two binaries and three documents, so this install finishes in
|
||||
; about a second - fast enough that people reported it as a failure. The finish
|
||||
; page is the only place that can say otherwise, so it lists what was installed
|
||||
; and where instead of the stock "Setup was completed successfully."
|
||||
; Kept to six lines: the finish page's text field is a fixed height, and MUI
|
||||
; clips rather than scrolls what does not fit.
|
||||
!define MUI_FINISHPAGE_TITLE "${APP} ${VERSION} is installed"
|
||||
!define MUI_FINISHPAGE_TEXT "Installed into $INSTDIR:$\r$\n\
|
||||
quicksearch.exe (the app), quicksearch-cli.exe (terminal search),$\r$\n\
|
||||
README.md, LICENSE.txt and config_example.toml.$\r$\n$\r$\n\
|
||||
Your settings and search index are created on first run, under your \
|
||||
own account. Upgrading and uninstalling leave both alone."
|
||||
!define MUI_FINISHPAGE_RUN
|
||||
!define MUI_FINISHPAGE_RUN_TEXT "Run ${APP}"
|
||||
!define MUI_FINISHPAGE_RUN_FUNCTION LaunchAsUser
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue