Added GPL license, deb packaging script, complete GUI rewrite, added CLI mode, better default config options, config now lives in user folder, reduced syscalls by 5x.

This commit is contained in:
= 2026-08-02 19:04:30 -04:00
parent 2a584fbfc0
commit 096c13e21a
82 changed files with 21769 additions and 6506 deletions

6
.gitattributes vendored Normal file
View file

@ -0,0 +1,6 @@
# cmd.exe is sensitive to line endings — a run.bat checked out with LF can
# misparse labels and multi-line constructs — while sh requires LF regardless
# of the platform it is checked out on.
*.bat text eol=crlf
*.cmd text eol=crlf
*.sh text eol=lf

1
.gitignore vendored
View file

@ -1,3 +1,4 @@
/target /target
/dist
*.db *.db
config.toml config.toml

4
.vscode/launch.json vendored
View file

@ -12,7 +12,7 @@
"args": [ "args": [
"build", "build",
"--bin=quicksearch", "--bin=quicksearch",
"--package=quicksearch" "--package=quicksearch-gui"
], ],
"filter": { "filter": {
"name": "quicksearch", "name": "quicksearch",
@ -31,7 +31,7 @@
"test", "test",
"--no-run", "--no-run",
"--bin=quicksearch", "--bin=quicksearch",
"--package=quicksearch" "--package=quicksearch-gui"
], ],
"filter": { "filter": {
"name": "quicksearch", "name": "quicksearch",

4335
Cargo.lock generated

File diff suppressed because it is too large Load diff

View file

@ -6,5 +6,8 @@ members = [
] ]
[workspace.package] [workspace.package]
version = "0.1.0" version = "0.8.0"
edition = "2021" edition = "2021"
license = "GPL-3.0-or-later"
authors = ["Jeremy <jeremy@karsttech.com>"]
repository = "https://code.karsttech.com/jeremy/quick_search.git"

674
LICENSE Normal file
View file

@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.

312
README.md Normal file
View file

@ -0,0 +1,312 @@
# QuickSearch
A fast local file indexer and search tool. QuickSearch walks your chosen
folders into a compact SQLite index (FTS5 full-text + zstd-compressed text
sidecar), keeps it fresh automatically with filesystem watchers and
periodic reindexing, and serves ranked search-as-you-type results in a
compact egui desktop app, or straight to your terminal.
## Build & run
Requirements: a Rust toolchain (edition 2021). The old WebKit/WebView
dependencies (`setup.sh`) are gone; the GUI renders with OpenGL via egui.
SQLite and zstd are compiled from bundled C sources, so a C toolchain is
required on every platform:
- Linux: working OpenGL 3.3 drivers; `xdg-desktop-portal` (present on all
mainstream desktops) provides the native folder picker. On minimal
images you may need `build-essential pkg-config libxkbcommon-dev`.
- Windows: Visual Studio 2022 Build Tools with the "Desktop development
with C++" workload (MSVC v143 plus a Windows SDK). For the GNU target
instead, `rustup target add x86_64-pc-windows-gnu` and a mingw-w64
toolchain. Note that Windows ships only a software OpenGL 1.1 driver, so
a bare VM or an RDP session without a vendor GPU driver cannot create a
context and the window will fail to open.
- macOS: Xcode command line tools.
```sh
cargo build --release -p quicksearch-gui # binaries: target/release/quicksearch{,-cli}
cargo run -p quicksearch-gui # or just run it
cargo test -p quicksearch-core # backend test suite
```
`run.sh` / `run.bat` wrap the same commands.
Two binaries are produced. `quicksearch` is the desktop app; on Windows it
is built as a window-subsystem app so no console appears behind it.
`quicksearch-cli` is terminal search — a console app, so pipes, redirection
and exit codes behave normally. On Unix `quicksearch` also does both, and
`quicksearch-cli` is simply the same tool under a clearer name.
## Install (Debian / Ubuntu)
```sh
./packaging/build-deb.sh
sudo apt install ./dist/quicksearch_0.1.0-1_amd64.deb
```
The script builds the release binary, strips it, and assembles a `.deb` with
`dpkg-deb`. It needs no `cargo-deb`, no `debhelper` and no SVG rasteriser —
only `dpkg-deb` and `desktop-file-utils`, both standard on Debian and Ubuntu.
Useful flags: `--no-build` to package a binary you already built, `--no-strip`
to keep debug symbols, `-o DIR` to write elsewhere. `DEB_REVISION` and
`DEB_MAINTAINER` override the packaging revision and maintainer.
The package installs:
| Path | Contents |
| --- | --- |
| `/usr/bin/quicksearch` | the desktop app, which also does terminal search |
| `/usr/bin/quicksearch-cli` | terminal search only |
| `/usr/share/applications/quicksearch.desktop` | menu entry, so QuickSearch appears in the app launcher |
| `/usr/share/icons/hicolor/{16,22,24,32,48,64,128,256}x*/apps/` | icons at each size |
| `/usr/share/icons/hicolor/scalable/apps/quicksearch.svg` | the source icon |
| `/usr/share/man/man1/quicksearch{,-cli}.1.gz` | `man quicksearch`; the `-cli` page is a `.so` stub pointing at it |
| `/usr/share/doc/quicksearch/` | copyright, changelog, README, `config_example.toml` |
Installing registers the menu entry and the icon: dpkg triggers owned by
`desktop-file-utils` and `hicolor-icon-theme` refresh both caches, so no
maintainer scripts are involved and `apt remove` reverses it cleanly.
No `config.toml` is installed. One placed next to the executable would put
every user into portable mode (see [Configuration](#configuration)); instead
the app writes `~/.config/quicksearch/config.toml` on first run.
### Icons
`crates/quicksearch-gui/assets/icons/` holds `quicksearch_icon.svg` and the
PNGs rasterised from it. The PNGs are committed rather than generated, so an
ordinary `cargo build` needs no image tooling — the 256px one is compiled into
the binary with `include_bytes!` and becomes the window icon. Editing the SVG
means re-rendering the PNGs; `packaging/build-deb.sh` documents how in a
comment at the top.
X11 takes the window icon from the embedded PNG. Wayland ignores it and
matches the app id (`quicksearch`) against the installed
`quicksearch.desktop`, so under Wayland the titlebar icon appears only once
the package is installed.
## Usage
### GUI
`quicksearch` with no query arguments opens the app:
- **Search**: results appear as you type; every keystroke cancels the
previous search. One checkbox enables the two fuzzy passes. Sort by
rank, name, path, size, or modified; right-click a result to open it,
reveal it in the file manager, or build an ignore filter from it
(session-only by default, optionally persisted to the config). Matches
in file contents show highlighted snippets.
- **Manage Index**: full indexing status, Start/Stop/Automatic controls,
indexed folder list, full-text extension filters, ignore patterns, and
the indexing options.
- **Duplicates**: files sharing a content hash, grouped.
- **Logs**: the lines the app would have printed to a terminal — warnings
from indexing, folder watching and opening files, newest last, with a
filter box and Copy button. Launched from a desktop launcher (or on
Windows, where the app has no console at all) this is the only place
they are visible.
The bottom status bar always shows what the indexer is doing (phase,
percent, files/sec) or the total indexed file count when idle.
### Terminal
```sh
quicksearch report type:Document modified:">=2024-01-01"
quicksearch --long --limit 20 "quarterly budget"
quicksearch --fuzzy repot # tolerates typos
```
Prints rank-ordered paths (pipe-friendly); `--long` adds rank, size,
mtime, and highlighted snippets. `quicksearch --help` shows all flags.
On Windows use `quicksearch-cli` for all of the above — `quicksearch.exe`
opens the app, and any query given to it seeds the search box instead of
printing. Colour in `--long` output needs a console with virtual-terminal
processing; Windows Terminal has it, and older consoles get plain text.
### Query syntax
This section is the complete reference (the in-app "?" popup shows a
condensed version of the same rules). Everything that isn't a filter is
matched as one phrase, in order. Filters combine freely with the search
text:
| Syntax | Meaning |
|---|---|
| `budget report` | names, contents, and paths containing the phrase `budget report` |
| `"exact phrase"` | quotes keep spaces, stars, and filter-like words literal; `""` escapes a quote |
| `bud*port` | `*` matches any run of characters (it stays on one line of content); `%` and `_` are always literal |
| `regex:"(foo|bar)\d+"` | regular expression matched against names, contents, and paths; case-insensitive by default (`(?-i:…)` overrides); quote patterns containing spaces or `( ) : = < > "` |
| `type:Audio` | one of Audio, Image, Video, Document, Text, Archive, Spreadsheet, Presentation, Folder |
| `modified:>=2024-01-01` | also `<`, `<=`, `>`, `=` (dates are `yyyy-mm-dd`; `mtime:` is an alias) |
| `path:/home/me/docs` | restrict to a folder and its subfolders (`folder:` and `includefolder:` are aliases); `*` is literal here |
| `path:C:\Users\me\docs` | the same on Windows — drive letters and backslashes need no quoting |
| `mime:application/pdf` | exact MIME type |
| `name:re*.txt` | filename contains (as a filter, unranked; `filename:` is an alias); unquoted `*` globs |
Unrecognized `key:value` text (like `12:30`) stays part of the search
phrase, and a half-typed quote never errors while you type. `AND`, `OR`
and parentheses are treated as plain words. A term of only stars matches
nothing, and a regex that could match the empty string is rejected rather
than matching every file. `regex:` bypasses the trigram index entirely
and combines with filters; alongside search text it acts as an extra
requirement on those results.
The search box highlights this syntax as you type: recognized filter
keywords in red, their arguments in blue, syntax characters (operators,
quotes, live wildcards) in green, on a tinted chip per complete filter.
An argument the engine would reject — unknown `type:` name, bad date,
invalid regex — switches to the error color immediately.
Results are ranked: exact filename matches (case-sensitive first), then
filename substrings, then full-text matches ordered by occurrence count,
then fuzzy filename/full-text matches when enabled, and last the files
matched somewhere else in their path. Later, weaker matches only ever
append to the bottom of the list. Wildcard terms rank through the same
tiers (an "exact" match means the whole name matches the pattern) but
skip the fuzzy passes; regex-only queries reuse the substring, full-text,
and path tiers. Path matching needs at least three characters, and terms
may span separators (`docs/report`). Full-text matching also needs at
least three characters of literal text (the trigram floor). The fuzzy
passes tolerate typos with a budget of one edit per three characters,
capped by `[search] fuzzy_max_edits` (default 2; 0 turns fuzzy off).
### Configuration
`config.toml` lives at `~/.config/quicksearch/config.toml` (Windows:
`%APPDATA%\quicksearch\config.toml`) and is created on first run; the
default index goes to `~/.local/share/quicksearch/index.sqlite`
(Windows: `%LOCALAPPDATA%\quicksearch\index.sqlite`). See
`config_example.toml` for every option.
Defaults follow the platform. The first indexing root is your home
directory or `%USERPROFILE%`; `include_hidden = false` skips dot-files
everywhere and additionally anything marked Hidden or System on Windows,
which is what keeps `AppData`, `$RECYCLE.BIN` and `System Volume
Information` out of the index; and ignore patterns are matched
case-insensitively on Windows and macOS, matching the filesystem.
**Portable mode**: a `config.toml` sitting next to the `quicksearch`
binary overrides the user config entirely, and relative paths inside any
config resolve against the config file's own directory, so a folder
containing the binary, its config, and its index can be moved wholesale.
The GUI edits the config live; external edits apply on next start.
## Engineering overview
Two crates:
```
crates/quicksearch-core library: indexing, storage, search
crates/quicksearch-gui binary "quicksearch": egui app + terminal mode
```
### Backend (`quicksearch-core`)
Synchronous Rust: `std::thread` + `mpsc` channels, no async runtime.
- **Storage** (`db/`): SQLite via rusqlite (bundled), WAL mode so the
single writer never blocks streaming read-only searches. `files` holds
metadata (name, path, size, mtime, hash, MIME/type bitmask, per-row
index state); `searchabletext` is a *contentless* FTS5 table (postings
only, configurable tokenizer, trigram by default); canonical extracted
text lives zstd-compressed in `documents_text`, which powers snippets,
occurrence ranking, and fuzzy full-text search. Schema changes wipe and
rebuild by policy; the indexer (`open_or_recreate`) is the only code
allowed to do that; every consumer uses `open_existing`, which treats
drift as an error, never data loss.
- **Indexing** (`indexing.rs`, `file_handling.rs`): full runs walk each
root (`filtered_walk` prunes hidden/ignored subtrees before descending),
classify files by mtime into insert/update/skip, batch-write metadata,
sweep stale rows, then extract content (plaintext, Office, PDF, audio
tags, EXIF; see `extract/`) for FTS. Files no larger than
`processing.hash_length` skip that second pass entirely: the head the walk
reads to hash them is already their whole content, so a plaintext body is
extracted in the same `read` and stored complete. Progress streams through
a polled `IndexingStatus`.
- **Coordinator** (`coordinator.rs`): the object binaries construct.
Owns the `IndexingService`, the debouncing filesystem watcher
(`watcher.rs`), and the mode state machine (Auto / Manual). Watcher
events become single-file transactions (`incremental.rs`) that keep
`files`, FTS, and the text sidecar consistent per commit; a full
reindex runs on a configurable interval. Incremental writes defer while
a full run is active, so there is exactly one writer at a time.
Registration follows what the platform's notification API can do:
inotify covers one directory per watch, so the roots are walked and each
surviving directory registered individually (skipping `.git`,
`node_modules` and hidden subtrees, which is what keeps the watch count
affordable), while `ReadDirectoryChangesW` covers a whole tree from one
handle and takes a single watch per root, filtering the events instead.
Either way a tree too large to watch degrades to periodic reindexing
rather than going silently stale.
- **Search** (`search/`): `SearchService` runs one worker thread; each
query is a *generation*. New queries interrupt the in-flight SQLite
statement (`InterruptHandle`) and stale generations stop cooperatively,
so typing never waits. The cascade streams rank-ordered batches: one
`files` scan classifies exact/case/substring filename matches (ranks
14) and, since a path contains its own name, sets aside full-path
matches from the same rows (ranks 910); one FTS phrase probe verified
against the decompressed text yields full-text ranks 56 ordered by
occurrence count; and the opt-in fuzzy passes run a bitap (WuManber)
matcher over filenames (rank 7), document text (rank 8) and paths
(rank 11), with a configurable edit budget. The deferred path tiers
flush last, so weaker matches only ever append. All SQL is
parameterized; structured filters from the query language (`query/`)
are ANDed onto every pass.
- **Baloo compatibility** (`cli.rs`, `mime.rs`): read-only endpoints
(`status_for_path`, `list_failed`, `index_size_breakdown`, …) and a
Baloo-shaped type model, groundwork for a future `balooctl`-compatible
layer.
- **Logging** (`log.rs`): background reporting goes through `log_info!` /
`log_warn!` rather than `println!`/`eprintln!`. Each writes its line to
stderr *and* appends it to a bounded in-memory ring (newest 5000 lines,
with a count of what was dropped) that the GUI's Logs tab reads, so a
windowed run with no terminal still surfaces them. Command output —
search hits, usage, the error a command exits with — stays on stdio.
- **Platform differences** (`platform.rs`): the single home for `#[cfg]`.
Home directory lookup, what counts as a hidden entry (dot-prefix, plus
the Hidden/System attributes on Windows), network-filesystem detection
(`/proc/mounts` against `GetDriveTypeW`), path collation, and the
watch-registration strategy all live here, so the rest of the crate can
ask a question rather than test a target. Anything decidable from a
string alone is split out so its tests run on every platform.
### Frontend (`quicksearch-gui`)
Immediate-mode egui/eframe app, one UI thread:
```
UI thread ──SearchRequest──▶ search worker ──SearchUpdate (mpsc)──▶ drained per frame
UI thread ──commands──────▶ IndexCoordinator ──state──▶ polled per frame
core threads ─────────────▶ ctx.request_repaint() (wake the UI)
```
Modules map one-to-one onto what you see: `app.rs` (shell, status bar,
config routing), `search_tab.rs` (query strip, virtualized results table,
snippet highlighting via `LayoutJob` byte ranges, ignore dialog),
`manage_tab.rs` (status detail + `tracker.rs` rate estimation, roots and
filter editors), `duplicates_tab.rs`, `logs_tab.rs` (a virtualized view of
the core log ring), `options.rs` (draft-based settings
editor shared between the window and the Manage tab), `platform.rs`
(open / reveal-in-file-manager, and the Windows stdio setup a
window-subsystem process needs before anything prints), `cli.rs` (terminal
mode, shared with the `quicksearch-cli` binary). There is no
pagination: the table is virtualized, so a single scroll list capped at
`display_limit` renders in microseconds regardless of row count.
## Development
- `cargo test -p quicksearch-core`: unit + integration suites (cascade
ranking, cancellation, incremental indexing, coordinator modes, config
resolution, fuzzy matcher vs. brute-force oracle).
- `cargo test -p quicksearch-gui`: formatter/tracker/CLI-parsing units.
- `QSB_SNIPPET_PERF=1 cargo test --release -p quicksearch-core --test
snippet_perf -- --nocapture`: snippet pipeline benchmark.
- New extractors: implement `extract::Extractor` and register it in
`Registry::default_set()`. New cascade behavior: `search/cascade.rs`
documents the rank invariants that keep streamed results append-only.

View file

@ -1,37 +1,112 @@
# QuickSearch configuration reference.
#
# The live config is auto-created at ~/.config/quicksearch/config.toml
# (Windows: %APPDATA%\quicksearch\config.toml). A config.toml placed next
# to the quicksearch binary overrides it entirely (portable mode).
# Relative paths resolve against the directory containing the config
# file, so a portable folder can be moved wholesale.
#
# Every key is optional; missing keys take the defaults shown here.
[paths] [paths]
# One or more directory roots to index. Walked in order; duplicate and # One or more directory roots to index. Walked in order; duplicate and
# nested roots are de-duplicated automatically. # nested roots are de-duplicated automatically. `~` expands to home.
indexing_paths = ["/"] indexing_paths = ["~"]
database_path = "QuickSearch.db" # SQLite index location. Default: ~/.local/share/quicksearch/index.sqlite
# On Windows the default is %LOCALAPPDATA%\quicksearch\index.sqlite. Write
# Windows paths as TOML *literal* strings (single quotes) so the
# backslashes need no escaping, and keep the index out of a roaming
# profile — it is far too large to synchronise:
# database_path = 'C:\Users\you\AppData\Local\quicksearch\index.sqlite'
database_path = "~/.local/share/quicksearch/index.sqlite"
[indexing]
# Automatic mode: filesystem watchers apply changes as they happen and a
# full reindex runs every reindex_interval_minutes. The watcher catches
# changes as they happen, so this only needs to be often enough to cover
# whatever the watcher missed.
auto_index = true
reindex_interval_minutes = 1440
# Follow symbolic links during directory walks.
follow_symlinks = false
# Index hidden files and directories. That means dot-files everywhere, and
# additionally anything carrying the Hidden or System attribute on Windows
# (AppData, $RECYCLE.BIN, System Volume Information, pagefile.sys ...).
include_hidden = false
# Empty = extract text from every supported format. Non-empty = content
# indexing only for these extensions; other files are still listed for
# filename search. Entries are case-insensitive, leading dot optional.
content_extensions = []
# Excluded from the index entirely. A pattern without a separator matches
# any single path component (so ".git" prunes whole subtrees); patterns
# containing one match full paths. Glob syntax (*, ?, [..]). Matching is
# case-insensitive on Windows and macOS, case-sensitive elsewhere,
# following the filesystem.
#
# The Windows defaults add: "$RECYCLE.BIN", "System Volume Information",
# "pagefile.sys", "hiberfil.sys", "swapfile.sys", "Thumbs.db",
# "desktop.ini".
ignore_patterns = [".git", "node_modules", "*.tmp", ".venv", "venv"]
# Worth adding by hand if you index a whole Windows drive rather than just
# your profile. Neither is excluded by default, because the default root
# is your profile and a bare "Windows" pattern would also match a folder
# of your own with that name:
# 'C:\Windows' — system files, nothing you would search for
# 'C:\Windows\WinSxS' — a hardlink farm that floods the Duplicates tab
# Walker threads per root, keyed by the exact root string from
# indexing_paths. Absent or 0 = auto (4 on local storage, 16 on network
# mounts, detected per root). Applies at the start of the next run.
# root_workers = { "/media/share" = 24 }
[processing] [processing]
# Amount of data in bytes read from start/end of files used to calculate hash # Bytes read from the start of each file for its content hash, which is
# `sha256(size || first hash_length bytes)` and backs duplicate detection.
# Only the head is read: seeking to the end for a second block costs an
# extra round trip per file on network shares.
#
# Known limitation: files of identical size whose heads match will be
# reported as duplicates. In practice that means pre-allocated VM disk
# images: a fixed-size VHD stores its unique footer at the end of the
# file, and a freshly pre-allocated raw/qcow2/VMDK image is all zeros at
# the head until it is partitioned.
hash_length = 8192 hash_length = 8192
# Maximum text content to index per file (bytes) # Maximum extracted text stored per file (bytes).
maximum_text_size = 262144 maximum_text_size = 262144
# Maximum file size to process for text extraction (bytes) # Files larger than this skip text extraction entirely (bytes).
maximum_text_file_size = 2097152 maximum_text_file_size = 2097152
# Number of files to process in each batch (directory walk / inserts / text extraction batches) # Files per batch during walks / inserts / extraction.
batch_size = 200 batch_size = 200
# Files per transaction for incremental UPDATE files + DELETE from searchabletext (FTS); larger = fewer commits, more RAM per chunk # Files per transaction for incremental FTS updates.
fts_update_batch_size = 1000 fts_update_batch_size = 1000
# If true, run a fast shell-backed tree count before Phase 1 (enables % progress; Linux uses GNU find -printf '\n' | wc -l when available). # FTS5 tokenizer: 'trigram' (substring matching, the default; gets
# If false, Phase 1 shows file counts without a percentage. # remove_diacritics 1 appended), 'unicode61', 'porter', or a full FTS5
precount_files_for_progress = false # option string. See https://www.sqlite.org/fts5.html#tokenizers
# If true, follow symbolic links during directory walks (indexing only; shell precount unchanged).
follow_symlinks = false
# If true, hidden files and directories will be indexed.
include_hidden = false
# FTS5 tokenization method (e.g., 'trigram', 'porter', 'unicode61')
# Look here for more information https://www.sqlite.org/fts5.html#tokenizers
tokenize = "trigram" tokenize = "trigram"
# If true (default), extracted text is stored zstd-compressed in a sidecar # Store extracted text (zstd-compressed) alongside the FTS index. Off:
# table so the GUI's search results can render snippet previews with the # the index shrinks to roughly stock-Baloo size, but search loses snippet
# query terms highlighted. If false, the inverted FTS5 index is still # previews, occurrence ranking, case verification, and fuzzy full-text.
# populated (so queries match the same files) but nothing is stored
# alongside; result rows carry no snippet. Turning this off drops the
# on-disk footprint to roughly what stock Baloo uses, useful for
# apples-to-apples comparisons and for users who never read snippet text.
# Changing this only affects files indexed *after* the change; existing
# sidecar rows are kept until re-indexed.
store_text_for_snippets = true store_text_for_snippets = true
[ui]
# Zoom factor for the whole GUI: fonts, spacing, and widgets scale
# together (0.5 2.5). Ctrl +/- and Ctrl 0 adjust it temporarily at
# runtime; this value is the persistent baseline.
scale = 1.1
[search]
# Start with the fuzzy passes enabled.
fuzzy_default = false
# Ceiling on the fuzzy stages' typo budget. The allowance grows with the
# search term, one edit per three characters, up to this value, so 2
# means "1 edit for 3-5 character terms, 2 for anything longer". 0 turns
# the fuzzy stages off. Above 3 is allowed but not recommended: matches
# become dominated by coincidence and every fuzzy pass slows down.
fuzzy_max_edits = 2
# Hard cap on results per search (the GUI's scroll list length).
display_limit = 1000
# Results per streamed batch (latency/overhead knob, not a page size).
results_per_page = 100
# How long the GUI waits after the last keystroke before searching (ms).
debounce_ms = 150

View file

@ -2,6 +2,10 @@
name = "quicksearch-core" name = "quicksearch-core"
version.workspace = true version.workspace = true
edition.workspace = true edition.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
description = "Indexing, storage and search engine behind QuickSearch."
[lib] [lib]
name = "quicksearch_core" name = "quicksearch_core"
@ -17,10 +21,25 @@ serde = { version = "1.0", features = ["derive"] }
toml = "0.8" toml = "0.8"
mime_guess = "2.0" mime_guess = "2.0"
infer = "0.15" infer = "0.15"
pdf-extract = "0.7" pdf-extract = "0.12"
lopdf = "0.32" lopdf = "0.32"
lofty = "0.19" lofty = "0.19"
kamadak-exif = "0.5" kamadak-exif = "0.5"
notify = "6.1" notify = "6.1"
ctrlc = "3.4" ctrlc = "3.4"
zstd = "0.13" zstd = "0.13"
globset = "0.4"
regex = "1"
# `GetDriveTypeW` (a mapped drive letter is the only way to spot an SMB share
# that isn't written as UNC) plus the FILE_ATTRIBUTE_* constants for hidden
# detection. Pinned to 0.52 deliberately: walkdir → winapi-util already
# resolves exactly that version, so this adds no new crate compilations.
[target.'cfg(windows)'.dependencies]
# GetDriveTypeW and the FILE_ATTRIBUTE_* constants live in
# Win32_Storage_FileSystem; DRIVE_REMOTE, oddly, is filed under
# Win32_System_WindowsProgramming.
windows-sys = { version = "0.52", features = [
"Win32_Storage_FileSystem",
"Win32_System_WindowsProgramming",
] }

View file

@ -0,0 +1,216 @@
//! End-to-end timing and syscall accounting for a full indexing run.
//!
//! [`walkprobe`](walkprobe.rs) covers phase 1 alone, without a database. This
//! covers the whole pipeline — parallel walk, `files` writes, and content
//! extraction — because the interesting redundancy lives *between* the two
//! phases: the walk reads a file's head to hash it and sniff its MIME, and
//! extraction then reopens the same file and reads it again.
//!
//! ```text
//! cargo build -p quicksearch-core --example indexprobe --release
//! ./target/release/examples/indexprobe gen /tmp/qs-bench
//! ./target/release/examples/indexprobe cold /tmp/qs-bench /tmp/qs-bench.db
//! ./target/release/examples/indexprobe warm /tmp/qs-bench /tmp/qs-bench.db
//! ```
//!
//! `cold` deletes the database first, so every file is new: the walk hashes
//! it and extraction reads it. `warm` re-runs over the existing database with
//! the tree untouched, which is the case that has to stay at one `stat` per
//! file — see [`crate::file_handling::classify_for_indexing`].
//!
//! For syscalls per file, trace a run and bucket by the tree's paths:
//!
//! ```text
//! strace -f -y -o /tmp/t.log \
//! -e trace=openat,statx,newfstatat,fstat,read,pread64,readlink,close,getdents64,lseek \
//! ./target/release/examples/indexprobe cold /tmp/qs-bench /tmp/qs-bench.db
//! grep -oP '^\d+ \K[a-z0-9_]+' <(grep '/tmp/qs-bench/' /tmp/t.log) | sort | uniq -c
//! ```
//!
//! Group by thread id instead (`grep -oP '^\d+ [a-z0-9_]+'`) to see the split
//! between the walk workers and the extraction thread.
//!
//! The run modes deliberately do no filesystem inspection of their own — no
//! progress walk, no size survey — so that every syscall the trace attributes
//! to the tree came from the indexer. The size histogram is printed by `gen`.
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use quicksearch_core::config::Config;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
/// Files whose head the walk reads in full at the default 8 KiB
/// `hash_length`, i.e. the ones extraction never needs to reopen.
const SMALL_TEXT: usize = 800;
/// Text files past `hash_length`, which extraction must still read.
const LARGE_TEXT: usize = 100;
/// No extractor claims these, so extraction resolves them without touching
/// the disk. A control group: their cost must not move.
const BINARY: usize = 100;
const WORDS: &[&str] = &[
"alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta",
"quick", "brown", "fox", "jumps", "over", "lazy", "dog", "indexer",
"rust", "cargo", "sqlite", "baloo", "tokenizer", "trigram", "snippet",
"ocean", "forest", "mountain", "river", "valley", "bridge", "tunnel",
"morning", "afternoon", "evening", "midnight", "yesterday", "today",
];
/// Deterministic so two runs index byte-identical trees and their timings are
/// comparable. Plain LCG — this only has to spread, not to be random.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
self.0 >> 33
}
fn in_range(&mut self, lo: usize, hi: usize) -> usize {
lo + (self.next() as usize) % (hi - lo)
}
}
fn main() {
let mode = std::env::args().nth(1).unwrap_or_default();
let tree = PathBuf::from(std::env::args().nth(2).expect("usage: indexprobe <gen|cold|warm> <tree> [db]"));
match mode.as_str() {
"gen" => generate(&tree),
"cold" | "warm" => {
let db = PathBuf::from(std::env::args().nth(3).expect("usage: indexprobe <cold|warm> <tree> <db>"));
if mode == "cold" {
for suffix in ["", "-wal", "-shm"] {
let _ = std::fs::remove_file(format!("{}{}", db.display(), suffix));
}
}
run(&mode, &tree, &db);
}
_ => {
eprintln!("usage: indexprobe <gen|cold|warm> <tree> [db]");
std::process::exit(2);
}
}
}
/// Build a tree with a size mix that separates the three code paths, and
/// report it so results are self-describing.
fn generate(tree: &Path) {
let _ = std::fs::remove_dir_all(tree);
std::fs::create_dir_all(tree).expect("create tree");
let mut rng = Rng(0x5eed);
let (mut small_bytes, mut large_bytes, mut bin_bytes) = (0usize, 0usize, 0usize);
// Spread across subdirectories so the walk does real directory work
// rather than one enormous readdir.
for i in 0..SMALL_TEXT {
let dir = tree.join(format!("src/mod{}", i % 40));
std::fs::create_dir_all(&dir).expect("mkdir");
let ext = ["txt", "md", "rs", "json"][i % 4];
let size = rng.in_range(200, 8 * 1024);
let body = prose(&mut rng, size);
small_bytes += body.len();
std::fs::write(dir.join(format!("f{}.{}", i, ext)), body).expect("write");
}
for i in 0..LARGE_TEXT {
let dir = tree.join(format!("docs/set{}", i % 10));
std::fs::create_dir_all(&dir).expect("mkdir");
let size = rng.in_range(8 * 1024 + 1, 200 * 1024);
let body = prose(&mut rng, size);
large_bytes += body.len();
std::fs::write(dir.join(format!("doc{}.md", i)), body).expect("write");
}
for i in 0..BINARY {
let dir = tree.join(format!("assets/set{}", i % 10));
std::fs::create_dir_all(&dir).expect("mkdir");
let n = rng.in_range(1024, 50 * 1024);
let blob: Vec<u8> = (0..n).map(|_| (rng.next() & 0xff) as u8).collect();
bin_bytes += blob.len();
std::fs::write(dir.join(format!("blob{}.bin", i)), blob).expect("write");
}
let total = SMALL_TEXT + LARGE_TEXT + BINARY;
eprintln!("generated {} files under {}", total, tree.display());
eprintln!(
" text <= 8 KiB : {:5} files, {:8.1} MiB (head covers the whole file)",
SMALL_TEXT,
small_bytes as f64 / (1024.0 * 1024.0)
);
eprintln!(
" text > 8 KiB : {:5} files, {:8.1} MiB (extraction must read it)",
LARGE_TEXT,
large_bytes as f64 / (1024.0 * 1024.0)
);
eprintln!(
" binary : {:5} files, {:8.1} MiB (no extractor; control group)",
BINARY,
bin_bytes as f64 / (1024.0 * 1024.0)
);
}
fn prose(rng: &mut Rng, target: usize) -> String {
let mut s = String::with_capacity(target + 16);
while s.len() < target {
s.push_str(WORDS[rng.next() as usize % WORDS.len()]);
s.push(if rng.next() % 12 == 0 { '\n' } else { ' ' });
}
s.truncate(target);
s
}
fn run(mode: &str, tree: &Path, db: &Path) {
let config = Config::default();
// `run_indexing` writes this marker only on a successful finish, so it is
// the one unambiguous completion signal — polling the status enum races,
// because a small tree finishes between two polls and `Idle` then means
// both "not started" and "already done".
if db.exists() {
let conn = rusqlite::Connection::open(db).expect("open db");
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.expect("clear marker");
}
let service = IndexingService::new();
let start = Instant::now();
service
.start_indexing(
vec![tree.to_string_lossy().into_owned()],
db.to_string_lossy().into_owned(),
config,
)
.expect("start indexing");
let deadline = Instant::now() + Duration::from_secs(600);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(5));
}
let elapsed = start.elapsed();
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().expect("stop");
let total = SMALL_TEXT + LARGE_TEXT + BINARY;
eprintln!(
"{}: {:?} ({:.0} files/sec over {} files)",
mode,
elapsed,
total as f64 / elapsed.as_secs_f64(),
total
);
}

View file

@ -0,0 +1,126 @@
//! Times and syscall-counts the phase-1 walk, without touching a database.
//!
//! ```text
//! cargo build -p quicksearch-core --example walkprobe --release
//! ./target/release/examples/walkprobe <root> parallel # the threaded walker
//! ./target/release/examples/walkprobe <root> serial # one thread, for comparison
//! ```
//!
//! Indexing a network share is bound by round trips, not bandwidth: every
//! metadata operation that misses the client cache costs one, and throughput
//! is round-trips-in-flight divided by latency. So the number that matters is
//! syscalls per file, which this makes directly visible:
//!
//! ```text
//! strace -f -c -e trace=openat,statx,newfstatat,readlink,getdents64,read,lseek,close \
//! ./target/release/examples/walkprobe <root> parallel
//! ```
//!
//! Expect roughly one `statx` per unchanged file, plus open/read/close for
//! files that are new or modified, and `readlink` only for resolving the roots
//! themselves — a per-file `readlink` count means a `canonicalize` has crept
//! back into the hot path.
//!
//! Those four syscalls also now cover the *whole* cost of a small text file:
//! the head read for the hash is the file's entire contents, so the walk
//! extracts its text there and the content pass never opens it again. That
//! work is CPU, not syscalls, so it shows up in files/sec here and not in the
//! trace. Use [`indexprobe`](indexprobe.rs) to see both phases together.
//!
//! Both modes report files/sec. Run each twice: the first pass warms the page
//! cache (or, on a share, the client's attribute cache), so the second is the
//! one to compare.
use std::collections::HashMap;
use std::sync::atomic::AtomicBool;
use std::sync::{Arc, Mutex};
use std::time::{Instant, UNIX_EPOCH};
use quicksearch_core::config::{Config, IgnoreSet};
use quicksearch_core::extract::Registry;
use quicksearch_core::file_handling::{
classify_for_indexing, filtered_walk, prepare_file_record, ExistingFileEntry, FileIndexAction,
UnreadableDirs,
};
use quicksearch_core::walk::walk_indexable_files;
fn main() {
let root = std::env::args().nth(1).unwrap();
let mode = std::env::args().nth(2).unwrap_or_else(|| "parallel".into());
let config = Config::default();
let existing: HashMap<String, ExistingFileEntry> = HashMap::new();
let start = Instant::now();
let (seen, prepared) = match mode.as_str() {
"serial" => serial(&root, &config, &existing),
_ => parallel(&root, &config, existing),
};
let elapsed = start.elapsed();
eprintln!(
"{mode}: {seen} files, {prepared} prepared in {:?} ({:.0} files/sec)",
elapsed,
seen as f64 / elapsed.as_secs_f64()
);
}
fn serial(
root: &str,
config: &Config,
existing: &HashMap<String, ExistingFileEntry>,
) -> (usize, usize) {
let ignore = IgnoreSet::compile(&[]).unwrap();
let registry = Registry::default_set();
let (mut seen, mut prepared) = (0, 0);
for entry in filtered_walk(root, false, false, &ignore, &UnreadableDirs::default()) {
seen += 1;
// Same rule as the real walk: a name that is not valid UTF-8 cannot be
// stored in `files.path` and reopened by it, so it is skipped before
// anything tries to hash it. Counted as seen, never prepared.
let Some(path) = entry.path().to_str().map(str::to_owned) else {
continue;
};
let Ok(meta) = std::fs::metadata(entry.path()) else {
continue;
};
let Some(mtime) = meta
.modified()
.ok()
.and_then(|t| t.duration_since(UNIX_EPOCH).ok())
.map(|d| d.as_secs())
else {
continue;
};
if classify_for_indexing(&path, mtime, existing) != FileIndexAction::Skip
&& prepare_file_record(&path, &meta, config, &registry).is_some()
{
prepared += 1;
}
}
(seen, prepared)
}
fn parallel(
root: &str,
config: &Config,
existing: HashMap<String, ExistingFileEntry>,
) -> (usize, usize) {
let (mut seen, mut prepared) = (0, 0);
for file in walk_indexable_files(
&[root.to_string()],
false,
false,
IgnoreSet::compile(&[]).unwrap(),
Arc::new(existing),
config.clone(),
Arc::new(Registry::default_set()),
Arc::new(Mutex::new(false)),
Arc::new(AtomicBool::new(false)),
4,
) {
seen += 1;
if file.record.is_some() {
prepared += 1;
}
}
(seen, prepared)
}

View file

@ -8,7 +8,7 @@
use rusqlite::{params, OptionalExtension}; use rusqlite::{params, OptionalExtension};
use crate::db::open_or_recreate; use crate::db::open_existing;
use crate::db::repo::{STATE_DONE, STATE_FAILED, STATE_NA, STATE_PENDING}; use crate::db::repo::{STATE_DONE, STATE_FAILED, STATE_NA, STATE_PENDING};
/// Per-file indexing status, mirroring Baloo's multi-state reporting. /// Per-file indexing status, mirroring Baloo's multi-state reporting.
@ -86,7 +86,7 @@ impl SizeReport {
/// Query the per-file indexing status. Returns `FileStatus` with /// Query the per-file indexing status. Returns `FileStatus` with
/// `basic == NotIndexed` if the path isn't in the database. /// `basic == NotIndexed` if the path isn't in the database.
pub fn status_for_path(db_path: &str, path: &str) -> Result<FileStatus, String> { pub fn status_for_path(db_path: &str, path: &str) -> Result<FileStatus, String> {
let conn = open_or_recreate(db_path, "trigram")?; let conn = open_existing(db_path, false)?;
let row: Option<(i64, i64, Option<String>)> = conn let row: Option<(i64, i64, Option<String>)> = conn
.query_row( .query_row(
"SELECT basic_state, content_state, failure_msg FROM files WHERE path = ?1", "SELECT basic_state, content_state, failure_msg FROM files WHERE path = ?1",
@ -113,7 +113,7 @@ pub fn status_for_path(db_path: &str, path: &str) -> Result<FileStatus, String>
/// Return every file that failed content extraction, newest first. /// Return every file that failed content extraction, newest first.
pub fn list_failed(db_path: &str, limit: Option<u32>) -> Result<Vec<FailedEntry>, String> { pub fn list_failed(db_path: &str, limit: Option<u32>) -> Result<Vec<FailedEntry>, String> {
let conn = open_or_recreate(db_path, "trigram")?; let conn = open_existing(db_path, false)?;
let limit_sql = match limit { let limit_sql = match limit {
Some(n) => format!(" LIMIT {}", n), Some(n) => format!(" LIMIT {}", n),
None => String::new(), None => String::new(),
@ -147,7 +147,7 @@ pub fn index_size_breakdown(db_path: &str) -> Result<SizeReport, String> {
let file_size_bytes = std::fs::metadata(db_path) let file_size_bytes = std::fs::metadata(db_path)
.map(|m| m.len()) .map(|m| m.len())
.unwrap_or(0); .unwrap_or(0);
let conn = open_or_recreate(db_path, "trigram")?; let conn = open_existing(db_path, false)?;
let count = |table: &str| -> Result<i64, String> { let count = |table: &str| -> Result<i64, String> {
conn.query_row(&format!("SELECT COUNT(*) FROM {}", table), [], |r| r.get(0)) conn.query_row(&format!("SELECT COUNT(*) FROM {}", table), [], |r| r.get(0))
.map_err(|e| format!("count {}: {}", table, e)) .map_err(|e| format!("count {}: {}", table, e))
@ -181,7 +181,7 @@ pub fn index_size_breakdown(db_path: &str) -> Result<SizeReport, String> {
/// Used by the Baloo compat daemon to report the "Files waiting for content /// Used by the Baloo compat daemon to report the "Files waiting for content
/// indexing" figure both to balooctl and to the LMDB mirror. /// indexing" figure both to balooctl and to the LMDB mirror.
pub fn pending_content_count(db_path: &str) -> Result<i64, String> { pub fn pending_content_count(db_path: &str) -> Result<i64, String> {
let conn = open_or_recreate(db_path, "trigram")?; let conn = open_existing(db_path, false)?;
conn.query_row( conn.query_row(
"SELECT COUNT(*) FROM files WHERE content_state = ?1", "SELECT COUNT(*) FROM files WHERE content_state = ?1",
rusqlite::params![crate::db::repo::STATE_PENDING], rusqlite::params![crate::db::repo::STATE_PENDING],
@ -190,10 +190,33 @@ pub fn pending_content_count(db_path: &str) -> Result<i64, String> {
.map_err(|e| format!("pending_content_count: {}", e)) .map_err(|e| format!("pending_content_count: {}", e))
} }
/// Cheap aggregate counts for the GUI's idle status bar ("N files
/// indexed"). Callers cache the result; it's three COUNT scans, not
/// something to run per frame.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct IndexCounts {
pub files: i64,
pub content_done: i64,
pub content_pending: i64,
}
pub fn index_counts(db_path: &str) -> Result<IndexCounts, String> {
let conn = open_existing(db_path, false)?;
let count = |sql: &str| -> Result<i64, String> {
conn.query_row(sql, [], |r| r.get(0))
.map_err(|e| format!("index_counts: {}", e))
};
Ok(IndexCounts {
files: count("SELECT COUNT(*) FROM files")?,
content_done: count("SELECT COUNT(*) FROM files WHERE content_state = 1")?,
content_pending: count("SELECT COUNT(*) FROM files WHERE content_state = 0")?,
})
}
/// Remove a single file from the index. Returns whether a row was deleted. /// Remove a single file from the index. Returns whether a row was deleted.
/// Keeps FTS/documents/properties in sync via the repo helpers. /// Keeps FTS/documents/properties in sync via the repo helpers.
pub fn clear_path(db_path: &str, path: &str) -> Result<bool, String> { pub fn clear_path(db_path: &str, path: &str) -> Result<bool, String> {
let mut conn = open_or_recreate(db_path, "trigram")?; let mut conn = open_existing(db_path, true)?;
let tx = conn let tx = conn
.transaction() .transaction()
.map_err(|e| format!("clear_path begin tx: {}", e))?; .map_err(|e| format!("clear_path begin tx: {}", e))?;
@ -206,6 +229,7 @@ pub fn clear_path(db_path: &str, path: &str) -> Result<bool, String> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::db::open_or_recreate;
use crate::db::repo::{insert_file, set_content_done, set_content_failed, NewFile}; use crate::db::repo::{insert_file, set_content_done, set_content_failed, NewFile};
use crate::mime::FileType; use crate::mime::FileType;
@ -374,4 +398,51 @@ mod tests {
std::fs::remove_file(&p).ok(); std::fs::remove_file(&p).ok();
} }
#[test]
fn clear_path_on_nondefault_tokenizer_db_removes_only_target() {
// Regression: clear_path used to open with a hardcoded "trigram", so on
// an index built with a non-default tokenizer the schema-mismatch wipe
// destroyed the WHOLE index instead of deleting one row. With
// open_existing it must delete only the target and leave the rest.
let p = tmp_path();
let dbp = p.to_str().unwrap();
{
let mut conn = open_or_recreate(dbp, "unicode61").unwrap();
let tx = conn.transaction().unwrap();
for (name, path) in [("a.txt", "/tmp/a.txt"), ("b.txt", "/tmp/b.txt")] {
insert_file(
&tx,
&NewFile {
name,
path,
parent: "/tmp",
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: Some("text/plain"),
ftype: FileType::TEXT,
hash: None,
},
)
.unwrap()
.expect("unique path");
}
tx.commit().unwrap();
}
assert!(clear_path(dbp, "/tmp/a.txt").unwrap());
// The other row must survive — proof we deleted one row, not wiped.
assert_eq!(
status_for_path(dbp, "/tmp/b.txt").unwrap().basic,
IndexState::Done
);
assert_eq!(
status_for_path(dbp, "/tmp/a.txt").unwrap().basic,
IndexState::NotIndexed
);
std::fs::remove_file(&p).ok();
}
} }

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -1,13 +1,15 @@
//! SQLite schema, on-disk open/recreate, and row-level repository helpers. //! SQLite schema, on-disk open/recreate, and row-level repository helpers.
//! //!
//! Policy: a single [`open::open_or_recreate`] is the only entry point. Any //! Policy: the indexer (owner) opens via [`open::open_or_recreate`], which on
//! schema mismatch — wrong version, drifted tokenizer, absent `schema_info` //! any schema mismatch — wrong version, drifted tokenizer, absent
//! — wipes the DB and rebuilds from [`schema::SCHEMA_CURRENT`]. There are //! `schema_info` — wipes the DB and rebuilds from [`schema::SCHEMA_CURRENT`].
//! no in-place migrations by design; re-indexing is accepted as the cost //! There are no in-place migrations by design; re-indexing is accepted as the
//! of avoiding migration-path complexity. //! cost of avoiding migration-path complexity. *Consumers* (search, status,
//! size, `clear`) instead use [`open::open_existing`], which never creates or
//! wipes — a tokenizer difference or stale version is an error, not data loss.
pub mod open; pub mod open;
pub mod repo; pub mod repo;
pub mod schema; pub mod schema;
pub use open::{open_or_recreate, CURRENT_SCHEMA_VERSION}; pub use open::{open_existing, open_or_recreate, CURRENT_SCHEMA_VERSION};

View file

@ -15,9 +15,11 @@
use std::path::Path; use std::path::Path;
use rusqlite::{params, Connection, OptionalExtension}; use rusqlite::{params, Connection, OpenFlags, OptionalExtension};
use super::schema::{effective_tokenizer, fts_create_sql, PRAGMAS_FAST, SCHEMA_CURRENT}; use super::schema::{
effective_tokenizer, fts_create_sql, PRAGMAS_FAST, PRAGMAS_READONLY, SCHEMA_CURRENT,
};
/// Bump this whenever [`SCHEMA_CURRENT`] or [`fts_create_sql`] changes in /// Bump this whenever [`SCHEMA_CURRENT`] or [`fts_create_sql`] changes in
/// a way that makes an old DB unreadable by new code. Any such bump /// a way that makes an old DB unreadable by new code. Any such bump
@ -34,6 +36,14 @@ pub const CURRENT_SCHEMA_VERSION: u32 = 3;
/// schema mismatch and triggers the wipe-and-recreate path. /// schema mismatch and triggers the wipe-and-recreate path.
pub fn open_or_recreate(db_path: &str, tokenizer: &str) -> Result<Connection, String> { pub fn open_or_recreate(db_path: &str, tokenizer: &str) -> Result<Connection, String> {
let path = Path::new(db_path).to_path_buf(); let path = Path::new(db_path).to_path_buf();
// The owner creates the directory too — a fresh install's default
// XDG data dir doesn't exist until first use.
if let Some(dir) = path.parent() {
if !dir.as_os_str().is_empty() {
std::fs::create_dir_all(dir)
.map_err(|e| format!("Failed to create database dir {}: {}", dir.display(), e))?;
}
}
let conn = Connection::open(db_path) let conn = Connection::open(db_path)
.map_err(|e| format!("Failed to open database at {}: {}", db_path, e))?; .map_err(|e| format!("Failed to open database at {}: {}", db_path, e))?;
conn.execute_batch(PRAGMAS_FAST) conn.execute_batch(PRAGMAS_FAST)
@ -46,8 +56,8 @@ pub fn open_or_recreate(db_path: &str, tokenizer: &str) -> Result<Connection, St
// Schema is present but stale, or pre-existing rows belong to an // Schema is present but stale, or pre-existing rows belong to an
// older layout, or the tokenizer drifted. Log once so the rebuild // older layout, or the tokenizer drifted. Log once so the rebuild
// isn't silent, then wipe + recreate. // isn't silent, then wipe + recreate.
eprintln!( crate::log_warn!(
"QuickSearch: database at {} does not match current schema; rebuilding. \ "database at {} does not match current schema; rebuilding. \
Existing rows will be re-scanned on next indexing run.", Existing rows will be re-scanned on next indexing run.",
db_path db_path
); );
@ -56,10 +66,47 @@ pub fn open_or_recreate(db_path: &str, tokenizer: &str) -> Result<Connection, St
Ok(conn) Ok(conn)
} }
/// True iff the DB has `schema_info` with the current version *and* the /// Open an *existing* index without ever recreating it. Verifies the schema
/// effective-tokenizer string this caller asked for. Anything else — /// version matches this build; on any mismatch — missing file, no
/// missing table, wrong version, different tokenizer — returns false. /// `schema_info`, wrong version — returns an error instead of wiping. The
fn db_matches_current(conn: &Connection, tokenizer: &str) -> Result<bool, String> { /// on-disk FTS tokenizer is used as-is: a tokenizer difference is never a
/// reason to destroy a readable index.
///
/// `write == false` opens read-only; `write == true` opens read-write (for
/// row-level deletes like `clear`) but still never creates or wipes — there
/// is no `SQLITE_OPEN_CREATE`, so a missing file is a clean error.
///
/// Use this for every *consumer* (search, status, size, `clear`). Only the
/// indexer's own write path uses [`open_or_recreate`], which may wipe on a
/// genuine schema/tokenizer change it owns.
pub fn open_existing(db_path: &str, write: bool) -> Result<Connection, String> {
let flags = OpenFlags::SQLITE_OPEN_NO_MUTEX
| if write {
OpenFlags::SQLITE_OPEN_READ_WRITE
} else {
OpenFlags::SQLITE_OPEN_READ_ONLY
};
let conn = Connection::open_with_flags(db_path, flags)
.map_err(|e| format!("Failed to open database at {}: {}", db_path, e))?;
let pragmas = if write { PRAGMAS_FAST } else { PRAGMAS_READONLY };
conn.execute_batch(pragmas)
.map_err(|e| format!("Failed to apply pragmas: {}", e))?;
if !schema_version_current(&conn)? {
return Err(format!(
"index at {} is not a compatible QuickSearch index (schema v{} expected); \
refusing to modify it. Re-index to rebuild.",
db_path, CURRENT_SCHEMA_VERSION
));
}
Ok(conn)
}
/// True iff the DB has a `schema_info` table whose `version` equals
/// [`CURRENT_SCHEMA_VERSION`]. Shared by the wipe decision
/// ([`db_matches_current`]) and the non-destructive [`open_existing`] path.
/// Deliberately ignores the tokenizer — that's only the owner's concern.
fn schema_version_current(conn: &Connection) -> Result<bool, String> {
let has_info: bool = conn let has_info: bool = conn
.query_row( .query_row(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='schema_info'", "SELECT 1 FROM sqlite_master WHERE type='table' AND name='schema_info'",
@ -81,8 +128,14 @@ fn db_matches_current(conn: &Connection, tokenizer: &str) -> Result<bool, String
) )
.optional() .optional()
.map_err(|e| format!("read schema_info.version: {}", e))?; .map_err(|e| format!("read schema_info.version: {}", e))?;
let version_ok = version.as_deref() == Some(&CURRENT_SCHEMA_VERSION.to_string()); Ok(version.as_deref() == Some(&CURRENT_SCHEMA_VERSION.to_string()))
if !version_ok { }
/// True iff the DB has `schema_info` with the current version *and* the
/// effective-tokenizer string this caller asked for. Anything else —
/// missing table, wrong version, different tokenizer — returns false.
fn db_matches_current(conn: &Connection, tokenizer: &str) -> Result<bool, String> {
if !schema_version_current(conn)? {
return Ok(false); return Ok(false);
} }
@ -104,10 +157,22 @@ fn wipe_and_reopen(conn: Connection, path: &Path) -> Result<Connection, String>
drop(conn); drop(conn);
// Primary file may already be absent (fresh open that just needed // Primary file may already be absent (fresh open that just needed
// the table applied). Ignore NotFound; anything else is an error. // the table applied). Ignore NotFound; anything else is an error.
match std::fs::remove_file(path) { //
// `remove_file_retrying` matters on Windows, where a delete fails while
// *any* handle is open — most often an antivirus scanner reading the file
// in the moment after we closed it. Unix `unlink` never hits this, so the
// retry costs nothing there.
match crate::platform::remove_file_retrying(path) {
Ok(()) => {} Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(format!("Failed to remove old database: {}", e)), Err(e) => {
return Err(format!(
"Failed to remove old database at {}: {}. \
Another QuickSearch instance may have the index open.",
path.display(),
e
))
}
} }
// Sidecars are optional — delete best-effort. // Sidecars are optional — delete best-effort.
for suffix in ["-wal", "-shm", "-journal"] { for suffix in ["-wal", "-shm", "-journal"] {
@ -116,7 +181,7 @@ fn wipe_and_reopen(conn: Connection, path: &Path) -> Result<Connection, String>
path.file_name().and_then(|s| s.to_str()).unwrap_or(""), path.file_name().and_then(|s| s.to_str()).unwrap_or(""),
suffix suffix
)); ));
let _ = std::fs::remove_file(sidecar); let _ = crate::platform::remove_file_retrying(&sidecar);
} }
let conn = Connection::open(path) let conn = Connection::open(path)
.map_err(|e| format!("Failed to reopen database after rebuild: {}", e))?; .map_err(|e| format!("Failed to reopen database after rebuild: {}", e))?;
@ -326,4 +391,166 @@ mod tests {
drop(conn); drop(conn);
std::fs::remove_file(&p).ok(); std::fs::remove_file(&p).ok();
} }
#[test]
fn open_existing_reads_nondefault_tokenizer_without_wiping() {
// The exact scenario that previously caused data loss: an index built
// with a non-default tokenizer, then opened by a *consumer* that only
// knows "trigram". `open_existing` must read it as-is and never wipe.
let p = tmp_db_path();
{
let conn = open_or_recreate(p.to_str().unwrap(), "unicode61").unwrap();
conn.execute(
"INSERT INTO files (name, path, parent, size, mtime) \
VALUES ('note', '/note.txt', '/', 0, 0)",
[],
)
.unwrap();
// Seed the FTS index (rowid = the files row we just inserted) so a
// MATCH query can be exercised against the on-disk tokenizer.
conn.execute(
"INSERT INTO searchabletext (rowid, name, text, properties) \
VALUES (last_insert_rowid(), 'note', 'hello world', '')",
[],
)
.unwrap();
}
let conn = open_existing(p.to_str().unwrap(), false).unwrap();
let files: i64 = conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
assert_eq!(
files, 1,
"open_existing must not wipe a non-default-tokenizer DB"
);
// The on-disk tokenizer is used as-is: a MATCH against the stored term
// returns the row.
let hits: i64 = conn
.query_row(
"SELECT COUNT(*) FROM searchabletext WHERE searchabletext MATCH 'hello'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(hits, 1);
// And the stored tokenizer is still the non-default one — proof we
// neither rewrote the FTS table nor reset schema_info.
let tok: String = conn
.query_row(
"SELECT value FROM schema_info WHERE key='tokenize'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(tok, "unicode61");
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test]
fn open_or_recreate_creates_missing_parent_dirs() {
// Fresh installs point at ~/.local/share/quicksearch/… which
// doesn't exist yet; the owner open must create it.
let mut dir = std::env::temp_dir();
dir.push(format!(
"qs-mkdir-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let db = dir.join("nested/deeper/index.sqlite");
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
drop(conn);
assert!(db.exists());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn writable_opens_use_wal_and_it_persists() {
let p = tmp_db_path();
{
let conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
let mode: String = conn
.query_row("PRAGMA journal_mode", [], |r| r.get(0))
.unwrap();
assert_eq!(mode.to_lowercase(), "wal");
}
// WAL is persistent in the file: a later read-only consumer sees it
// without being able to (or needing to) set it.
let conn = open_existing(p.to_str().unwrap(), false).unwrap();
let mode: String = conn
.query_row("PRAGMA journal_mode", [], |r| r.get(0))
.unwrap();
assert_eq!(mode.to_lowercase(), "wal");
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test]
fn open_existing_errors_on_missing_file() {
let p = tmp_db_path();
assert!(!p.exists());
let res = open_existing(p.to_str().unwrap(), false);
assert!(res.is_err(), "missing file must error, not be created");
assert!(!p.exists(), "open_existing must not create the file");
}
#[test]
fn open_existing_errors_on_version_mismatch_without_wiping() {
// A DB from a prior schema version. A consumer opening it must get an
// error and leave the file untouched — the data is the owner's to
// rebuild, never a reader's to destroy.
let p = tmp_db_path();
{
let conn = Connection::open(&p).unwrap();
conn.execute(
"CREATE TABLE schema_info (key TEXT PRIMARY KEY, value TEXT NOT NULL)",
[],
)
.unwrap();
conn.execute(
"INSERT INTO schema_info(key,value) VALUES('version','1')",
[],
)
.unwrap();
conn.execute("CREATE TABLE files (id INTEGER PRIMARY KEY, name TEXT)", [])
.unwrap();
conn.execute("INSERT INTO files(name) VALUES('sentinel')", [])
.unwrap();
}
let res = open_existing(p.to_str().unwrap(), false);
assert!(res.is_err(), "stale schema version must error");
// Sentinel row still present → the file was not wiped.
let conn = Connection::open(&p).unwrap();
let n: i64 = conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
assert_eq!(n, 1, "open_existing must never delete on version mismatch");
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test]
fn open_existing_rw_allows_delete() {
let p = tmp_db_path();
{
let conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
conn.execute(
"INSERT INTO files (name, path, parent, size, mtime) \
VALUES ('a', '/a', '/', 0, 0)",
[],
)
.unwrap();
}
let conn = open_existing(p.to_str().unwrap(), true).unwrap();
let removed = conn
.execute("DELETE FROM files WHERE path = '/a'", [])
.unwrap();
assert_eq!(removed, 1);
drop(conn);
std::fs::remove_file(&p).ok();
}
} }

View file

@ -278,17 +278,41 @@ fn encode_properties_for_fts(props: &[(String, String)]) -> String {
buf buf
} }
/// Flush and close a connection, restoring durable PRAGMAs. Call on clean /// Flush the WAL into the main DB file and close. Call on clean shutdown so
/// shutdown so the next open sees a consistent DB. /// the next open starts with an empty log. WAL mode itself is persistent in
/// the file — deliberately left on.
pub fn checkpoint_and_close(conn: Connection) { pub fn checkpoint_and_close(conn: Connection) {
let _ = conn.execute_batch( let _ = conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);");
"PRAGMA journal_mode = DELETE; \
PRAGMA synchronous = FULL; \
PRAGMA wal_checkpoint(FULL);",
);
drop(conn); drop(conn);
} }
/// Read the `last_full_index` marker (unix seconds of the last *successful*
/// full indexing run) from `schema_info`. Absent key — fresh DB, or a DB
/// from before this marker existed — means "never".
pub fn get_last_full_index(conn: &Connection) -> Option<u64> {
conn.query_row(
"SELECT value FROM schema_info WHERE key = 'last_full_index'",
[],
|r| r.get::<_, String>(0),
)
.optional()
.ok()
.flatten()
.and_then(|v| v.parse().ok())
}
/// Stamp `last_full_index` with `ts` (unix seconds). Called at the end of
/// every successful full indexing run; the coordinator reads it to schedule
/// periodic reindexing.
pub fn set_last_full_index(conn: &Connection, ts: u64) -> Result<(), String> {
conn.execute(
"INSERT OR REPLACE INTO schema_info(key, value) VALUES ('last_full_index', ?1)",
params![ts.to_string()],
)
.map_err(|e| format!("write last_full_index: {}", e))?;
Ok(())
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@ -480,6 +504,59 @@ mod tests {
std::fs::remove_file(&p).ok(); std::fs::remove_file(&p).ok();
} }
#[test]
fn last_full_index_round_trip() {
let p = tmp_path();
let conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
assert_eq!(get_last_full_index(&conn), None, "fresh DB has no marker");
set_last_full_index(&conn, 1_700_000_123).unwrap();
assert_eq!(get_last_full_index(&conn), Some(1_700_000_123));
// Overwrite, not accumulate.
set_last_full_index(&conn, 1_700_000_999).unwrap();
assert_eq!(get_last_full_index(&conn), Some(1_700_000_999));
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test]
fn checkpoint_and_close_truncates_wal() {
let p = tmp_path();
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
{
let tx = conn.transaction().unwrap();
insert_file(
&tx,
&NewFile {
name: "w.txt",
path: "/tmp/w.txt",
parent: "/tmp",
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: None,
ftype: FileType::EMPTY,
hash: None,
},
)
.unwrap();
tx.commit().unwrap();
}
checkpoint_and_close(conn);
// After a TRUNCATE checkpoint + close of the last connection the WAL
// sidecar is gone or empty; the row lives in the main file.
let wal = std::path::PathBuf::from(format!("{}-wal", p.display()));
let wal_len = std::fs::metadata(&wal).map(|m| m.len()).unwrap_or(0);
assert_eq!(wal_len, 0, "WAL should be truncated on clean close");
let conn = crate::db::open_existing(p.to_str().unwrap(), false).unwrap();
let n: i64 = conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
assert_eq!(n, 1);
drop(conn);
std::fs::remove_file(&p).ok();
}
#[test] #[test]
fn set_content_failed_writes_failed_table() { fn set_content_failed_writes_failed_table() {
let p = tmp_path(); let p = tmp_path();

View file

@ -1,12 +1,32 @@
//! SQL strings for the current schema. Versioned; [`migrate`](super::migrate) //! SQL strings for the current schema. Versioned; [`migrate`](super::migrate)
//! drives the upgrade path. //! drives the upgrade path.
/// Pragmas applied on every connection open. Tuned for write throughput during /// Pragmas applied on every writable connection open.
/// indexing; a clean shutdown re-enables journal_mode/synchronous via ///
/// WAL, not journal-off: auto-indexing writes continuously while searches
/// stream from their own read-only connections, and WAL is what lets those
/// readers proceed without ever blocking the writer (or vice versa).
/// `synchronous = NORMAL` under WAL risks only the last commit on power
/// loss — acceptable for an index that is re-derivable from disk. Only two
/// writers exist (full index runs and the coordinator's incremental
/// updates) and they're serialized by design; `busy_timeout` is a backstop,
/// not a coordination mechanism. A clean shutdown truncates the log via
/// [`super::repo::checkpoint_and_close`]. /// [`super::repo::checkpoint_and_close`].
pub const PRAGMAS_FAST: &str = " pub const PRAGMAS_FAST: &str = "
PRAGMA journal_mode = OFF; PRAGMA journal_mode = WAL;
PRAGMA synchronous = 0; PRAGMA synchronous = NORMAL;
PRAGMA busy_timeout = 5000;
PRAGMA cache_size = 10000;
PRAGMA temp_store = MEMORY;
PRAGMA foreign_keys = ON;
";
/// Pragmas safe to apply on a read-only connection, where `journal_mode`
/// and `synchronous` can't be changed on the file. Used by
/// [`super::open::open_existing`] for read-only opens; write paths get the
/// full [`PRAGMAS_FAST`] set.
pub const PRAGMAS_READONLY: &str = "
PRAGMA busy_timeout = 5000;
PRAGMA cache_size = 10000; PRAGMA cache_size = 10000;
PRAGMA temp_store = MEMORY; PRAGMA temp_store = MEMORY;
PRAGMA foreign_keys = ON; PRAGMA foreign_keys = ON;

View file

@ -2,8 +2,15 @@
//! //!
//! An [`Extractor`] decides whether it can handle a given MIME type and, if //! An [`Extractor`] decides whether it can handle a given MIME type and, if
//! so, produces [`ExtractedContent`] for the file. The [`Registry`] picks the //! so, produces [`ExtractedContent`] for the file. The [`Registry`] picks the
//! first registered extractor that accepts the MIME and runs it. Callers can //! first registered extractor that accepts the MIME and runs it.
//! also fall back to an extension-based match for files with no detected MIME. //!
//! Dispatch is by MIME only — a file with no detected type is recorded as
//! "not applicable" rather than guessed at again here. Extensions that
//! `mime_guess` misses or mistypes are corrected upstream instead, in
//! [`crate::mime::guess_mime_from_head`], so there is one place where "what is
//! this file" gets decided, and it is decided once: the walk sniffs the head it
//! already read, stores the answer, and nothing downstream reopens the file to
//! ask again.
use std::collections::HashMap; use std::collections::HashMap;
use std::path::Path; use std::path::Path;
@ -70,6 +77,32 @@ pub trait Extractor: Send + Sync {
/// [`ExtractError`] to mark the file's content state as failed (so it /// [`ExtractError`] to mark the file's content state as failed (so it
/// won't be retried every run). /// won't be retried every run).
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError>; fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError>;
/// Extract from bytes the caller already holds, when those bytes are the
/// file's *entire* contents.
///
/// Indexing hashes the head of every new or changed file, so for anything
/// no larger than `hash_length` the whole file is already in memory by the
/// time the walk classifies it. An extractor that can work from that buffer
/// saves the content pass an open/read/close — several round trips per
/// file on a network share — and closes a consistency gap, because the
/// text then comes from the same `read` as the size, mtime and hash stored
/// alongside it.
///
/// The default is `None`: "I need the file on disk." Formats that seek,
/// or that read a central directory at the end of the file, must keep it.
/// Returning `Some(Err(_))` is a real extraction failure, recorded like
/// any other; returning `None` simply defers to [`Extractor::extract`].
///
/// `path` is passed only so failures name the same file the on-disk path
/// would — nothing here may open it.
fn extract_from_head(
&self,
_path: &Path,
_head: &[u8],
) -> Option<Result<ExtractedContent, ExtractError>> {
None
}
} }
/// An ordered dispatch table of extractors. The first extractor whose /// An ordered dispatch table of extractors. The first extractor whose
@ -105,6 +138,28 @@ impl Registry {
Ok(None) Ok(None)
} }
/// [`Registry::extract`] for a file whose complete contents the caller
/// already holds. `Ok(None)` when no extractor claims the MIME, and
/// `None` when the one that does needs the file on disk after all —
/// both mean "leave this to the content pass".
///
/// Dispatch stays here rather than at the call site so there is still
/// exactly one place that decides what an extractor sees for a given MIME.
pub fn extract_complete_head(
&self,
path: &Path,
mime: &str,
head: &[u8],
) -> Option<Result<ExtractedContent, ExtractError>> {
let lower = mime.to_ascii_lowercase();
for e in &self.extractors {
if e.supports(&lower) {
return e.extract_from_head(path, head);
}
}
None
}
/// The default set wired up for Set A: plaintext, office docs, PDF, /// The default set wired up for Set A: plaintext, office docs, PDF,
/// audio tags, image EXIF. /// audio tags, image EXIF.
pub fn default_set() -> Self { pub fn default_set() -> Self {
@ -136,6 +191,38 @@ mod tests {
assert!(out.is_none()); assert!(out.is_none());
} }
#[test]
fn complete_head_extraction_dispatches_only_to_extractors_that_opt_in() {
let r = Registry::default_set();
let p = Path::new("/tmp/whatever");
// Plaintext opts in, so a small text file never reaches the disk pass.
let out = r.extract_complete_head(p, "text/plain", b"hello");
assert!(matches!(out, Some(Ok(ref c)) if c.text == "hello"));
// A format that seeks or reads a trailer must not be handed a buffer.
// `None` here is what routes it back to the on-disk extractor.
assert!(r.extract_complete_head(p, "application/pdf", b"%PDF-1.4").is_none());
assert!(r.extract_complete_head(p, "image/png", b"\x89PNG").is_none());
// No extractor claims the MIME at all.
assert!(r.extract_complete_head(p, "application/x-nonesuch", b"..").is_none());
}
#[test]
fn complete_head_extraction_matches_the_on_disk_dispatch() {
// Both entry points must pick the same extractor for a MIME, or a
// file's text would depend on which pass happened to handle it.
let r = Registry::default_set();
let p = Path::new("/tmp/whatever");
for mime in ["text/plain", "TEXT/PLAIN", "application/json", "application/x-sql"] {
assert!(
r.extract_complete_head(p, mime, b"x").is_some(),
"{} should extract from a head", mime
);
}
}
#[test] #[test]
fn properties_sorted_is_deterministic() { fn properties_sorted_is_deterministic() {
let c = ExtractedContent::with_text("hi") let c = ExtractedContent::with_text("hi")

View file

@ -6,12 +6,48 @@
//! failed. Properties (title, author, etc.) from the PDF `Info` dictionary //! failed. Properties (title, author, etc.) from the PDF `Info` dictionary
//! are pulled via `lopdf` where available. //! are pulled via `lopdf` where available.
use std::cell::Cell;
use std::path::Path; use std::path::Path;
use std::sync::OnceLock;
use lopdf::{Document as LopdfDocument, Object}; use lopdf::{Document as LopdfDocument, Object};
use super::{ExtractError, ExtractedContent, Extractor}; use super::{ExtractError, ExtractedContent, Extractor};
thread_local! {
/// True while this thread is inside a contained `pdf_extract` call.
static SUPPRESS_PANIC_PRINT: Cell<bool> = const { Cell::new(false) };
}
/// Chain a process panic hook (once) that swallows the default
/// "thread panicked at …" report while this thread is inside a *contained*
/// PDF extraction — those panics are expected on malformed PDFs, caught,
/// and recorded as the file's failure reason, so printing each one is pure
/// console spam. Panics anywhere else print exactly as before.
fn install_quiet_panic_hook() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
let previous = std::panic::take_hook();
std::panic::set_hook(Box::new(move |info| {
if !SUPPRESS_PANIC_PRINT.with(|flag| flag.get()) {
previous(info);
}
}));
});
}
/// Human-readable message from a caught panic payload; lands in
/// `failed_files.reason`.
fn panic_message(payload: &(dyn std::any::Any + Send)) -> String {
if let Some(s) = payload.downcast_ref::<&str>() {
(*s).to_string()
} else if let Some(s) = payload.downcast_ref::<String>() {
s.clone()
} else {
"unknown panic".to_string()
}
}
pub struct PdfExtractor; pub struct PdfExtractor;
impl Extractor for PdfExtractor { impl Extractor for PdfExtractor {
@ -20,10 +56,15 @@ impl Extractor for PdfExtractor {
} }
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> { fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> {
// Text. Catch panics from pdf_extract (some PDFs crash its parser). // Text. Catch panics from pdf_extract (some PDFs crash its parser)
// and keep the default hook from spamming stderr about them.
install_quiet_panic_hook();
let path_buf = path.to_path_buf(); let path_buf = path.to_path_buf();
let text = std::panic::catch_unwind(move || pdf_extract::extract_text(&path_buf)) SUPPRESS_PANIC_PRINT.with(|flag| flag.set(true));
.map_err(|_| "pdf_extract panicked".to_string())? let result = std::panic::catch_unwind(move || pdf_extract::extract_text(&path_buf));
SUPPRESS_PANIC_PRINT.with(|flag| flag.set(false));
let text = result
.map_err(|panic| format!("pdf_extract panicked: {}", panic_message(&*panic)))?
.map_err(|e| format!("pdf_extract: {}", e))?; .map_err(|e| format!("pdf_extract: {}", e))?;
let mut out = ExtractedContent::with_text(text); let mut out = ExtractedContent::with_text(text);
@ -69,6 +110,20 @@ fn object_to_string(obj: &Object) -> Option<String> {
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn contained_panics_are_caught_quietly_with_reason() {
install_quiet_panic_hook();
SUPPRESS_PANIC_PRINT.with(|flag| flag.set(true));
let result = std::panic::catch_unwind(|| panic!("synthetic pdf failure"));
SUPPRESS_PANIC_PRINT.with(|flag| flag.set(false));
let payload = result.expect_err("must panic");
assert_eq!(panic_message(&*payload), "synthetic pdf failure");
// Panics outside the suppression window keep printing: the flag is
// thread-local and cleared, so nothing here can silence other
// threads or later tests.
assert!(!SUPPRESS_PANIC_PRINT.with(|flag| flag.get()));
}
#[test] #[test]
fn supports_pdf_mime() { fn supports_pdf_mime() {
assert!(PdfExtractor.supports("application/pdf")); assert!(PdfExtractor.supports("application/pdf"));

View file

@ -1,10 +1,21 @@
//! Read the file as UTF-8 text. Handles text/plain, text/x-*, application/json //! Read the file as UTF-8 text. Handles text/plain, text/x-*, application/json
//! and most source-code MIMEs. //! and most source-code MIMEs.
use std::fs::File;
use std::io::Read;
use std::path::Path; use std::path::Path;
use super::{ExtractError, ExtractedContent, Extractor}; use super::{ExtractError, ExtractedContent, Extractor};
/// Decode bytes that are known to be a complete file. Shared by both entry
/// points so on-disk and already-in-memory extraction cannot drift apart.
fn decode(bytes: Vec<u8>, path: &Path) -> Result<ExtractedContent, ExtractError> {
match String::from_utf8(bytes) {
Ok(text) => Ok(ExtractedContent::with_text(text)),
Err(e) => Err(format!("plaintext read {}: {}", path.display(), e.utf8_error())),
}
}
pub struct PlaintextExtractor; pub struct PlaintextExtractor;
impl Extractor for PlaintextExtractor { impl Extractor for PlaintextExtractor {
@ -22,13 +33,66 @@ impl Extractor for PlaintextExtractor {
| "application/toml" | "application/toml"
| "application/yaml" | "application/yaml"
| "application/x-yaml" | "application/x-yaml"
// `.sql` resolves here rather than to `text/*`, so without it
// schema dumps are listed by name but never full-text indexed.
| "application/x-sql"
) )
} }
/// Read the whole file, sized from the handle we just opened.
///
/// `std::fs::read_to_string` would cost two extra syscalls here: a
/// path-based `statx` to size its buffer, and a second `read` returning 0,
/// because "read to EOF" can only observe EOF that way — `read_to_end`
/// terminates on `Ok(0)` alone, so a short read does not end it. Sizing
/// the buffer ourselves lets the loop finish on `filled == size` and issue
/// exactly one `read` for a file that fits.
///
/// A file that shrank between the `fstat` and the `read` keeps its prefix
/// rather than failing. A file that grew is read up to the size we saw;
/// its mtime moved, so the next run reclassifies it as changed and
/// re-extracts (see [`crate::file_handling::classify_for_indexing`]).
/// Neither case was ever atomic — a concurrent writer can tear a file
/// across any read sequence, including `read_to_string`'s.
fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> { fn extract(&self, path: &Path) -> Result<ExtractedContent, ExtractError> {
let text = std::fs::read_to_string(path) let mut f = File::open(path)
.map_err(|e| format!("plaintext read {}: {}", path.display(), e))?; .map_err(|e| format!("plaintext read {}: {}", path.display(), e))?;
Ok(ExtractedContent::with_text(text)) let size = f
.metadata()
.map_err(|e| format!("plaintext read {}: {}", path.display(), e))?
.len() as usize;
// procfs, sysfs and some FUSE mounts report zero for files that do
// have content, so a sized read would store nothing. Only these pay
// the read-to-EOF probe — which is what a genuinely empty file cost
// before anyway.
if size == 0 {
let mut buf = Vec::new();
f.read_to_end(&mut buf)
.map_err(|e| format!("plaintext read {}: {}", path.display(), e))?;
return decode(buf, path);
}
let mut buf = vec![0u8; size];
let mut filled = 0;
while filled < size {
match f.read(&mut buf[filled..]) {
Ok(0) => break,
Ok(n) => filled += n,
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(format!("plaintext read {}: {}", path.display(), e)),
}
}
buf.truncate(filled);
decode(buf, path)
}
fn extract_from_head(
&self,
path: &Path,
head: &[u8],
) -> Option<Result<ExtractedContent, ExtractError>> {
Some(decode(head.to_vec(), path))
} }
} }
@ -36,23 +100,115 @@ impl Extractor for PlaintextExtractor {
mod tests { mod tests {
use super::*; use super::*;
#[test] fn tmp(tag: &str, body: &[u8]) -> std::path::PathBuf {
fn reads_utf8_file() {
let mut p = std::env::temp_dir(); let mut p = std::env::temp_dir();
p.push(format!( p.push(format!(
"qs-plaintext-{}-{}.txt", "qs-plaintext-{}-{}-{}.txt",
tag,
std::process::id(), std::process::id(),
std::time::SystemTime::now() std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH) .duration_since(std::time::UNIX_EPOCH)
.unwrap() .unwrap()
.as_nanos() .as_nanos()
)); ));
std::fs::write(&p, "hello world").unwrap(); std::fs::write(&p, body).unwrap();
p
}
#[test]
fn reads_utf8_file() {
let p = tmp("basic", b"hello world");
let c = PlaintextExtractor.extract(&p).unwrap(); let c = PlaintextExtractor.extract(&p).unwrap();
assert_eq!(c.text, "hello world"); assert_eq!(c.text, "hello world");
std::fs::remove_file(&p).ok(); std::fs::remove_file(&p).ok();
} }
#[test]
fn head_extraction_matches_reading_the_file() {
let p = tmp("agree", b"shared body with unicode: caf\xc3\xa9 \xe2\x9c\x93");
let from_disk = PlaintextExtractor.extract(&p).unwrap();
let bytes = std::fs::read(&p).unwrap();
let from_head = PlaintextExtractor.extract_from_head(&p, &bytes).unwrap().unwrap();
assert_eq!(from_disk.text, from_head.text);
assert_eq!(from_disk.properties, from_head.properties);
std::fs::remove_file(&p).ok();
}
#[test]
fn both_paths_reject_invalid_utf8_and_name_the_file() {
let p = tmp("badutf8", &[0x68, 0x69, 0xff, 0xfe]);
let disk_err = PlaintextExtractor.extract(&p).unwrap_err();
let head_err = PlaintextExtractor
.extract_from_head(&p, &[0x68, 0x69, 0xff, 0xfe])
.unwrap()
.unwrap_err();
assert_eq!(disk_err, head_err, "one decode path, one message");
assert!(disk_err.contains("badutf8"), "the failure names the file: {}", disk_err);
std::fs::remove_file(&p).ok();
}
#[test]
fn reads_a_file_larger_than_one_buffer_completely() {
// Past any plausible head window, so the read loop has to iterate if
// the kernel returns a short read.
let body = "abcdefgh".repeat(200 * 1024 / 8);
let p = tmp("large", body.as_bytes());
let c = PlaintextExtractor.extract(&p).unwrap();
assert_eq!(c.text.len(), body.len());
assert_eq!(c.text, body);
std::fs::remove_file(&p).ok();
}
#[test]
fn an_empty_file_extracts_to_empty_text() {
let p = tmp("empty", b"");
assert_eq!(PlaintextExtractor.extract(&p).unwrap().text, "");
assert_eq!(
PlaintextExtractor.extract_from_head(&p, &[]).unwrap().unwrap().text,
""
);
std::fs::remove_file(&p).ok();
}
/// A file whose reported size is a lie in the "there is more than this"
/// direction — the shape procfs and sysfs have. Sizing the buffer from
/// `st_size` alone would store nothing, so `extract` must fall back to
/// reading until EOF.
#[test]
fn a_file_reporting_zero_size_is_still_read_to_eof() {
let p = Path::new("/proc/self/status");
if !p.exists() {
return; // not Linux; the guard is only reachable there
}
assert_eq!(
std::fs::metadata(p).unwrap().len(),
0,
"precondition: procfs reports zero size"
);
let c = PlaintextExtractor.extract(p).unwrap();
assert!(
c.text.contains("Name:"),
"content must survive a zero st_size, got {} bytes",
c.text.len()
);
}
/// The same lie in the other direction, which the sized read handles by
/// keeping whatever was actually there.
#[test]
fn a_file_that_shrank_after_sizing_keeps_its_prefix() {
let p = tmp("shrink", &vec![b'x'; 4096]);
let f = File::options().write(true).open(&p).unwrap();
// Truncate behind `extract`'s back is not reproducible, so assert the
// property directly: a buffer sized larger than the file yields the
// file, not an error.
f.set_len(10).unwrap();
drop(f);
let c = PlaintextExtractor.extract(&p).unwrap();
assert_eq!(c.text, "xxxxxxxxxx", "a shrunk file reads short, not fatal");
std::fs::remove_file(&p).ok();
}
#[test] #[test]
fn supports_text_mimes() { fn supports_text_mimes() {
let e = PlaintextExtractor; let e = PlaintextExtractor;

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,486 @@
//! Incremental single-path index updates, driven by watcher events.
//!
//! One [`FsEvent`] becomes one (or a few) small transactions: files row,
//! `documents_text`, and FTS are updated together, so the index is
//! consistent after every commit. The same filters as the full walk apply
//! ([`IgnoreSet`], hidden components, `content_extensions`, size caps) —
//! a watcher event for something the walker would have skipped is a no-op.
//!
//! Renames are handled as remove + re-add: they're rare, and rewriting
//! `path`/`parent` strings plus re-tokenizing the FTS `name` column in
//! place is more machinery than re-extracting one file.
//!
//! Scope note: the watcher only reports paths under the configured roots,
//! so no root containment check is repeated here.
use std::path::Path;
use rusqlite::{Connection, OptionalExtension};
use crate::config::{content_allowed, Config, IgnoreSet};
use crate::db::repo;
use crate::extract::Registry;
use crate::file_handling::{
db_key_for_missing_path, extract_and_store, filtered_walk, UnreadableDirs,
prepare_file_record_from_path, store_inline_text,
};
use crate::platform::path_has_hidden_component_under;
use crate::query::translator::like_subtree_pattern;
use crate::watcher::FsEvent;
/// Apply one filesystem event to the index. Missing files are treated as
/// no-ops (a Create followed by a quick delete resolves via the Remove
/// event); unchanged mtimes short-circuit without touching the DB.
pub fn apply_fs_event(
conn: &mut Connection,
event: &FsEvent,
config: &Config,
ignore: &IgnoreSet,
registry: &Registry,
) -> Result<(), String> {
match event {
FsEvent::Create(p) | FsEvent::Modify(p) => upsert_path(conn, p, config, ignore, registry),
FsEvent::Remove(p) => remove_path(conn, p),
FsEvent::Rename { from, to } => {
remove_path(conn, from)?;
upsert_path(conn, to, config, ignore, registry)
}
}
}
fn upsert_path(
conn: &mut Connection,
path: &Path,
config: &Config,
ignore: &IgnoreSet,
registry: &Registry,
) -> Result<(), String> {
if ignore.matches_path(path) {
return Ok(());
}
// Measured from the innermost configured root: the walk never filters the
// root it was handed, so a root that is itself hidden (`~/.config/app`, or
// anything under `%LOCALAPPDATA%` on Windows) must not be rejected here —
// that disagreement is what makes the index churn every cycle.
if !config.indexing.include_hidden
&& path_has_hidden_component_under(path, &config.resolved_indexing_paths())
{
return Ok(());
}
let Ok(meta) = std::fs::metadata(path) else {
// Already gone again — the pending Remove event handles it.
return Ok(());
};
if meta.is_dir() {
// A moved-in tree surfaces as one directory event; walk it with
// the same filters as a full run.
let Some(root) = path.to_str() else {
return Ok(());
};
let entries: Vec<_> = filtered_walk(
root,
config.indexing.follow_symlinks,
config.indexing.include_hidden,
ignore,
&UnreadableDirs::default(),
)
.collect();
for entry in entries {
upsert_file(conn, entry.path(), config, registry)?;
}
Ok(())
} else {
upsert_file(conn, path, config, registry)
}
}
fn upsert_file(
conn: &mut Connection,
path: &Path,
config: &Config,
registry: &Registry,
) -> Result<(), String> {
let Some(rec) = prepare_file_record_from_path(path, config, registry) else {
return Ok(());
};
let tx = conn
.transaction()
.map_err(|e| format!("begin incremental tx: {}", e))?;
let existing: Option<(i64, i64)> = tx
.query_row(
"SELECT id, mtime FROM files WHERE path = ?1",
rusqlite::params![rec.path],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.optional()
.map_err(|e| format!("lookup {}: {}", rec.path, e))?;
let file_id = match existing {
Some((_, mtime)) if mtime.max(0) as u64 == rec.mtime => return Ok(()),
Some((id, _)) => {
repo::update_file_basic(
&tx,
&rec.path,
rec.size,
rec.mtime,
Some(&rec.hash),
rec.mime.as_deref(),
rec.ftype,
)?;
id
}
None => match repo::insert_file(&tx, &rec.as_new_file())? {
Some(id) => id,
// Lost a race with another writer on the same path; the row
// that won is current enough.
None => return Ok(()),
},
};
if rec.size > config.processing.maximum_text_file_size
|| !content_allowed(Path::new(&rec.path), config)
{
repo::set_content_na(&tx, file_id)?;
} else if let Some(text) = rec.inline_text.as_deref() {
// Small enough that `prepare_file_record_from_path` already read the
// whole file; reopening it here would be the same bytes twice.
store_inline_text(&tx, file_id, &rec, text, config)?;
} else {
extract_and_store(
&tx,
file_id,
&rec.name,
&rec.path,
rec.mime.as_deref(),
registry,
config,
)?;
}
tx.commit().map_err(|e| format!("commit incremental tx: {}", e))
}
fn remove_path(conn: &mut Connection, path: &Path) -> Result<(), String> {
// The insert side stores a canonicalized path, so the raw event spelling
// is not a usable key — but the file is already gone, so `canonicalize`
// cannot be called on it directly either.
let path_str = db_key_for_missing_path(path);
let tx = conn
.transaction()
.map_err(|e| format!("begin incremental tx: {}", e))?;
repo::delete_file_by_path(&tx, &path_str)?;
// Directory removals surface as one event for the directory itself —
// sweep everything indexed beneath it.
let subtree: Vec<String> = {
let mut stmt = tx
.prepare("SELECT path FROM files WHERE path LIKE ?1 ESCAPE '\\'")
.map_err(|e| e.to_string())?;
let rows = stmt
.query_map(rusqlite::params![like_subtree_pattern(&path_str)], |r| {
r.get::<_, String>(0)
})
.map_err(|e| e.to_string())?;
rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())?
};
for p in &subtree {
repo::delete_file_by_path(&tx, p)?;
}
tx.commit().map_err(|e| format!("commit incremental tx: {}", e))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::db::open_or_recreate;
struct Fixture {
conn: Connection,
dir: std::path::PathBuf,
db: std::path::PathBuf,
config: Config,
ignore: IgnoreSet,
registry: Registry,
}
impl Fixture {
fn new() -> Fixture {
let stamp = format!(
"{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let dir = std::env::temp_dir().join(format!("qs-incr-{}", stamp));
std::fs::create_dir_all(&dir).unwrap();
let db = std::env::temp_dir().join(format!("qs-incr-{}.sqlite", stamp));
let conn = open_or_recreate(db.to_str().unwrap(), "trigram").unwrap();
let config = Config::default();
let ignore = IgnoreSet::compile(&config.indexing.ignore_patterns).unwrap();
Fixture {
conn,
dir,
db,
config,
ignore,
registry: Registry::default_set(),
}
}
fn apply(&mut self, event: &FsEvent) {
apply_fs_event(&mut self.conn, event, &self.config, &self.ignore, &self.registry)
.unwrap();
}
fn write(&self, name: &str, content: &str) -> std::path::PathBuf {
let p = self.dir.join(name);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(&p, content).unwrap();
p
}
/// The key the index actually stores. Must go through
/// `path_to_db_string`, or every lookup here misses the
/// `\\?\`-stripped spelling on Windows.
fn canonical(&self, p: &Path) -> String {
crate::file_handling::path_to_db_string(&p.canonicalize().unwrap())
}
fn row(&self, path: &str) -> Option<(i64, i64, i64)> {
self.conn
.query_row(
"SELECT id, mtime, content_state FROM files WHERE path = ?1",
rusqlite::params![path],
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)),
)
.optional()
.unwrap()
}
fn counts(&self) -> (i64, i64, i64) {
let files = self
.conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
let fts = self
.conn
.query_row("SELECT COUNT(*) FROM searchabletext", [], |r| r.get(0))
.unwrap();
let texts = self
.conn
.query_row("SELECT COUNT(*) FROM documents_text", [], |r| r.get(0))
.unwrap();
(files, fts, texts)
}
fn fts_hits(&self, term: &str) -> i64 {
self.conn
.query_row(
"SELECT COUNT(*) FROM searchabletext WHERE searchabletext MATCH ?1",
rusqlite::params![format!("\"{}\"", term)],
|r| r.get(0),
)
.unwrap()
}
}
impl Drop for Fixture {
fn drop(&mut self) {
std::fs::remove_dir_all(&self.dir).ok();
std::fs::remove_file(&self.db).ok();
}
}
#[test]
fn create_indexes_file_and_content() {
let mut f = Fixture::new();
let p = f.write("hello.txt", "greetings earthling");
f.apply(&FsEvent::Create(p.clone()));
let canonical = f.canonical(&p);
let (_, _, content_state) = f.row(&canonical).expect("row exists");
assert_eq!(content_state, repo::STATE_DONE);
assert_eq!(f.counts(), (1, 1, 1), "files + FTS + text all written");
assert_eq!(f.fts_hits("earthling"), 1);
}
#[test]
fn modify_with_same_mtime_is_noop_and_changed_mtime_reextracts() {
let mut f = Fixture::new();
let p = f.write("doc.txt", "first version");
f.apply(&FsEvent::Create(p.clone()));
let canonical = f.canonical(&p);
let (id1, mtime1, _) = f.row(&canonical).unwrap();
// Same mtime → no-op (id unchanged, no re-extraction).
f.apply(&FsEvent::Modify(p.clone()));
let (id2, mtime2, _) = f.row(&canonical).unwrap();
assert_eq!((id1, mtime1), (id2, mtime2));
// Bump mtime and content → re-extracted, FTS follows.
std::fs::write(&p, "second edition entirely").unwrap();
let newer = std::time::SystemTime::now() + std::time::Duration::from_secs(5);
let file = std::fs::File::options().write(true).open(&p).unwrap();
file.set_modified(newer).unwrap();
drop(file);
f.apply(&FsEvent::Modify(p.clone()));
assert_eq!(f.fts_hits("edition"), 1);
assert_eq!(f.fts_hits("version"), 0, "stale tokens removed");
assert_eq!(f.counts(), (1, 1, 1), "still exactly one of everything");
}
#[test]
fn remove_file_cleans_all_tables() {
let mut f = Fixture::new();
let p = f.write("bye.txt", "ephemeral text");
f.apply(&FsEvent::Create(p.clone()));
let canonical = f.canonical(&p);
std::fs::remove_file(&p).unwrap();
f.apply(&FsEvent::Remove(canonical.clone().into()));
assert!(f.row(&canonical).is_none());
assert_eq!(f.counts(), (0, 0, 0));
}
#[test]
fn directory_create_and_remove_walks_subtree() {
let mut f = Fixture::new();
f.write("tree/a.txt", "alpha content");
f.write("tree/nested/b.txt", "beta content");
f.write("tree/.hidden.txt", "should not index");
f.write("tree/junk.tmp", "ignored pattern");
let tree = f.dir.join("tree");
f.apply(&FsEvent::Create(tree.clone()));
assert_eq!(f.counts().0, 2, "hidden + ignored excluded");
let canonical_tree = f.canonical(&tree);
std::fs::remove_dir_all(&tree).unwrap();
f.apply(&FsEvent::Remove(canonical_tree.into()));
assert_eq!(f.counts(), (0, 0, 0), "subtree swept");
}
#[test]
fn rename_moves_the_row() {
let mut f = Fixture::new();
let from = f.write("old-name.txt", "movable feast");
f.apply(&FsEvent::Create(from.clone()));
let canonical_from = f.canonical(&from);
let to = f.dir.join("new-name.txt");
std::fs::rename(&from, &to).unwrap();
f.apply(&FsEvent::Rename {
from: canonical_from.clone().into(),
to: to.clone(),
});
assert!(f.row(&canonical_from).is_none());
let canonical_to = f.canonical(&to);
assert!(f.row(&canonical_to).is_some());
assert_eq!(f.counts(), (1, 1, 1));
assert_eq!(f.fts_hits("feast"), 1);
}
#[test]
fn ignored_and_hidden_events_are_noops() {
let mut f = Fixture::new();
let ignored = f.write("junk.tmp", "x");
let hidden = f.write(".secret", "x");
f.apply(&FsEvent::Create(ignored));
f.apply(&FsEvent::Create(hidden));
// Missing file too.
f.apply(&FsEvent::Create(f.dir.join("never-existed.txt")));
assert_eq!(f.counts(), (0, 0, 0));
}
#[test]
fn content_extension_filter_gates_extraction() {
let mut f = Fixture::new();
f.config.indexing.content_extensions = vec!["md".into()];
let txt = f.write("listed-only.txt", "text body here");
f.apply(&FsEvent::Create(txt.clone()));
let canonical = f.canonical(&txt);
let (_, _, content_state) = f.row(&canonical).expect("row listed");
assert_eq!(
content_state,
repo::STATE_NA,
"filename indexed, content skipped"
);
assert_eq!(f.counts(), (1, 0, 0));
}
/// A Remove event whose path is spelled differently from the stored key
/// must still delete the row. `dir/./f.txt` and `dir/f.txt` are the same
/// file; only the canonicalized spelling is in the index.
#[test]
fn remove_with_a_non_canonical_spelling_still_deletes() {
let mut f = Fixture::new();
let p = f.write("sub/gone.txt", "vanishing text");
f.apply(&FsEvent::Create(p.clone()));
assert_eq!(f.counts(), (1, 1, 1));
std::fs::remove_file(&p).unwrap();
// Same file, spelled with a redundant `.` component.
let odd = f.dir.join("sub").join(".").join("gone.txt");
f.apply(&FsEvent::Remove(odd));
assert_eq!(f.counts(), (0, 0, 0), "row removed despite the spelling");
}
/// The subtree sweep must not take siblings whose names merely share a
/// string prefix — `tree2` is not inside `tree`.
#[test]
fn subtree_sweep_spares_prefix_siblings() {
let mut f = Fixture::new();
f.write("tree/a.txt", "alpha content");
f.write("tree2/b.txt", "beta content");
let tree = f.dir.join("tree");
f.apply(&FsEvent::Create(tree.clone()));
f.apply(&FsEvent::Create(f.dir.join("tree2")));
assert_eq!(f.counts().0, 2);
let canonical_tree = f.canonical(&tree);
std::fs::remove_dir_all(&tree).unwrap();
f.apply(&FsEvent::Remove(canonical_tree.into()));
assert_eq!(f.counts().0, 1, "only tree/ was swept");
let survivor = f.canonical(&f.dir.join("tree2").join("b.txt"));
assert!(f.row(&survivor).is_some(), "tree2 untouched");
}
/// A directory whose name contains a LIKE metacharacter must be swept
/// literally, not as a wildcard.
#[test]
fn subtree_sweep_treats_like_metacharacters_literally() {
let mut f = Fixture::new();
f.write("a_b/inside.txt", "underscore dir");
f.write("axb/other.txt", "wildcard bait");
f.apply(&FsEvent::Create(f.dir.join("a_b")));
f.apply(&FsEvent::Create(f.dir.join("axb")));
assert_eq!(f.counts().0, 2);
let target = f.dir.join("a_b");
let canonical = f.canonical(&target);
std::fs::remove_dir_all(&target).unwrap();
f.apply(&FsEvent::Remove(canonical.into()));
assert_eq!(f.counts().0, 1, "`_` must not match `x`");
let survivor = f.canonical(&f.dir.join("axb").join("other.txt"));
assert!(f.row(&survivor).is_some());
}
#[test]
fn oversize_files_get_content_na() {
let mut f = Fixture::new();
f.config.processing.maximum_text_file_size = 4;
let p = f.write("big.txt", "way more than four bytes");
f.apply(&FsEvent::Create(p.clone()));
let canonical = f.canonical(&p);
let (_, _, content_state) = f.row(&canonical).unwrap();
assert_eq!(content_state, repo::STATE_NA);
}
}

File diff suppressed because it is too large Load diff

View file

@ -1,13 +1,18 @@
pub mod cli; pub mod cli;
pub mod config; pub mod config;
pub mod coordinator;
pub mod db; pub mod db;
pub mod document_extraction; pub mod document_extraction;
pub mod extract; pub mod extract;
pub mod file_handling; pub mod file_handling;
pub mod incremental;
pub mod indexing; pub mod indexing;
pub mod log;
pub mod mime; pub mod mime;
pub mod platform;
pub mod query; pub mod query;
pub mod search_sql; pub mod search;
pub mod shutdown; pub mod shutdown;
pub mod snippet; pub mod snippet;
pub mod walk;
pub mod watcher; pub mod watcher;

View file

@ -0,0 +1,241 @@
//! The process log: every line that would go to the terminal, kept in
//! memory so a windowed run can show it.
//!
//! Launched from a desktop launcher — or on Windows, where the GUI binary is
//! built for the window subsystem and has no console at all — the process has
//! nowhere to print. The warnings the walker, indexer and watcher emit are
//! exactly the ones a user needs when something looks wrong, and they were
//! going nowhere.
//!
//! So background reporting goes through [`log_info!`] and [`log_warn!`]
//! instead of `println!`/`eprintln!`: each writes the same line to stderr
//! *and* appends it to a bounded ring the GUI's Logs tab reads. A terminal
//! run looks exactly as it did; a windowed run gains the tab.
//!
//! Command output — search hits from `quicksearch-cli`, usage text, the
//! errors a command exits with — is not logged. That is a program's answer to
//! what it was asked, not a background event, and it belongs on stdout.
use std::collections::VecDeque;
use std::io::Write;
use std::sync::{LazyLock, Mutex, MutexGuard};
use std::time::{SystemTime, UNIX_EPOCH};
/// Lines retained before the oldest are dropped.
///
/// A run over a tree full of unreadable files can log per file, so this is
/// bounded rather than complete: the newest few thousand lines are what
/// diagnosing anything actually needs, and the ring holds the count it threw
/// away so the tab can say so instead of quietly lying.
pub const CAPACITY: usize = 5_000;
/// How loud a line is. The GUI colors by this; stderr gets the `Warning:`
/// prefix that the same messages carried when they were `eprintln!`s.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Level {
Info,
Warn,
}
/// One recorded line.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LogLine {
/// Unix seconds when it was recorded.
pub at: u64,
pub level: Level,
/// Exactly the text written to stderr, prefix included.
pub text: String,
}
/// Record an informational line: `println!`-style formatting.
#[macro_export]
macro_rules! log_info {
($($arg:tt)*) => {
$crate::log::record($crate::log::Level::Info, ::std::format!($($arg)*))
};
}
/// Record a warning. The stored and printed text gains a `Warning: ` prefix,
/// so call sites pass the message alone.
#[macro_export]
macro_rules! log_warn {
($($arg:tt)*) => {
$crate::log::record($crate::log::Level::Warn, ::std::format!($($arg)*))
};
}
/// Write `message` to stderr and to the ring.
///
/// Prefer the [`log_info!`] / [`log_warn!`] macros; this is what they call.
///
/// A failed stderr write is ignored rather than propagated: `eprintln!`
/// *panics* when the handle is unwritable, which on a process launched
/// without stdio would take down whichever background thread happened to
/// report something. Losing the terminal copy is acceptable — that is
/// precisely the case where the in-memory copy is the one that matters.
pub fn record(level: Level, message: String) {
let text = match level {
Level::Warn => format!("Warning: {}", message),
Level::Info => message,
};
let _ = writeln!(std::io::stderr(), "{}", text);
lock().push(level, text);
}
/// Every retained line, oldest first.
pub fn snapshot() -> Vec<LogLine> {
lock().lines.iter().cloned().collect()
}
/// How many lines have been recorded since the process started, including
/// ones since dropped. Only ever grows, so a poll of this is the cheap way
/// to ask "anything new?" without copying the ring.
pub fn recorded() -> u64 {
lock().recorded
}
/// How many lines the ring has evicted since the last [`clear`].
pub fn dropped() -> u64 {
lock().dropped
}
/// Forget every retained line. [`recorded`] keeps counting.
pub fn clear() {
lock().clear();
}
static LOG: LazyLock<Mutex<Ring>> = LazyLock::new(|| Mutex::new(Ring::new(CAPACITY)));
/// Logging must not turn one panic into a cascade of them: a thread that
/// died mid-push would otherwise poison the lock and take down every later
/// logger. The ring is a `VecDeque` of owned strings, so the worst a poisoned
/// guard can hold is a line that was half-added.
fn lock() -> MutexGuard<'static, Ring> {
LOG.lock().unwrap_or_else(|e| e.into_inner())
}
/// The bounded line buffer. Split from the global so it can be tested on its
/// own instance — every other test in the process shares the global one.
struct Ring {
lines: VecDeque<LogLine>,
capacity: usize,
recorded: u64,
dropped: u64,
}
impl Ring {
fn new(capacity: usize) -> Ring {
Ring {
lines: VecDeque::new(),
capacity: capacity.max(1),
recorded: 0,
dropped: 0,
}
}
fn push(&mut self, level: Level, text: String) {
while self.lines.len() >= self.capacity {
self.lines.pop_front();
self.dropped += 1;
}
self.lines.push_back(LogLine {
at: now_unix(),
level,
text,
});
self.recorded += 1;
}
fn clear(&mut self) {
self.lines.clear();
self.dropped = 0;
}
}
fn now_unix() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
use super::*;
fn texts(ring: &Ring) -> Vec<&str> {
ring.lines.iter().map(|l| l.text.as_str()).collect()
}
#[test]
fn lines_come_back_oldest_first() {
let mut ring = Ring::new(8);
ring.push(Level::Info, "one".into());
ring.push(Level::Warn, "two".into());
assert_eq!(texts(&ring), vec!["one", "two"]);
assert_eq!(ring.lines[1].level, Level::Warn);
assert_eq!(ring.recorded, 2);
assert_eq!(ring.dropped, 0);
}
#[test]
fn the_oldest_lines_are_dropped_at_capacity() {
let mut ring = Ring::new(3);
for i in 0..5 {
ring.push(Level::Info, format!("line {}", i));
}
assert_eq!(
texts(&ring),
vec!["line 2", "line 3", "line 4"],
"only the newest `capacity` lines survive"
);
assert_eq!(ring.dropped, 2, "and the count of the lost ones is kept");
assert_eq!(ring.recorded, 5, "recorded counts everything ever pushed");
}
/// A zero capacity would spin the eviction loop forever on the first
/// push; hand-configuring one is not possible today, but the ring should
/// not depend on that staying true.
#[test]
fn a_zero_capacity_still_holds_one_line() {
let mut ring = Ring::new(0);
ring.push(Level::Info, "kept".into());
assert_eq!(texts(&ring), vec!["kept"]);
}
#[test]
fn clearing_empties_the_ring_but_not_the_total() {
let mut ring = Ring::new(2);
for i in 0..4 {
ring.push(Level::Info, format!("line {}", i));
}
ring.clear();
assert!(ring.lines.is_empty());
assert_eq!(ring.dropped, 0, "dropped counts against what is shown");
assert_eq!(ring.recorded, 4, "the running total survives a clear");
}
/// Through the global: the macros must land in the snapshot, and a
/// warning must carry the prefix its terminal line has. Written to
/// tolerate lines from tests running in parallel in this process.
#[test]
fn recorded_lines_reach_the_snapshot() {
let before = recorded();
crate::log_info!("test-marker info {}", 1);
crate::log_warn!("test-marker warn {}", 2);
assert!(recorded() >= before + 2);
let lines = snapshot();
let info = lines.iter().find(|l| l.text == "test-marker info 1");
let warn = lines
.iter()
.find(|l| l.text == "Warning: test-marker warn 2");
assert_eq!(info.map(|l| l.level), Some(Level::Info));
assert_eq!(
warn.map(|l| l.level),
Some(Level::Warn),
"a warning is stored with the prefix it printed with"
);
assert!(info.unwrap().at > 0, "timestamped when recorded");
}
}

View file

@ -1,11 +1,16 @@
//! MIME type guessing and `FileType` bitmask classification. //! MIME type guessing and `FileType` bitmask classification.
//! //!
//! Two stages: //! Two stages:
//! 1. [`guess_mime`] infers a MIME type from a path — extension first via //! 1. [`guess_mime_from_head`] infers a MIME type — extension first via
//! `mime_guess`, falling back to magic-byte sniffing via `infer` for files //! `mime_guess`, falling back to magic-byte sniffing via `infer` for files
//! whose extension is missing or ambiguous. //! whose extension is missing or ambiguous.
//! 2. [`mime_to_type`] maps a MIME string to a [`FileType`] bitmask so a single //! 2. [`mime_to_type`] maps a MIME string to a [`FileType`] bitmask so a single
//! file can belong to multiple categories (e.g. a `.docx` is Document|Text). //! file can belong to multiple categories (e.g. a `.docx` is Document|Text).
//!
//! The magic bytes are always ones the caller already holds. Indexing reads
//! the head of every new or changed file to hash it, and those are the same
//! bytes `infer` wants, so there is no path-based variant that goes back to
//! disk for them — that was a second open/read/close per undetectable file.
use std::path::Path; use std::path::Path;
@ -65,26 +70,72 @@ impl std::ops::BitOrAssign for FileType {
} }
} }
/// Guess a MIME type for a path on disk. /// Extensions `mime_guess` gets wrong or does not know, and what they really
/// are.
/// ///
/// Tries extension-based lookup via `mime_guess` first (cheap, no I/O). If /// Consulted *before* `mime_guess`, because for these the table is not a
/// that returns nothing or a generic `application/octet-stream`, and the file /// fallback but a correction. Everything here is plain text that would
/// is readable, falls back to `infer` magic-byte detection (reads a small /// otherwise get no content indexing at all:
/// prefix of the file).
/// ///
/// Returns `None` if no guess can be made. /// - `.ps1`/`.psm1`/`.psd1` and `.url` are simply absent from `mime_guess`,
pub fn guess_mime(path: &Path) -> Option<String> { /// and `infer` only knows binary magic, so they end up with no MIME — and
/// [`crate::extract::Registry`] has no extractor to offer, so the file is
/// marked "not applicable". PowerShell is the most common script type on a
/// Windows machine.
/// - `.bat` maps to `application/x-msdownload`, i.e. an executable. It is a
/// text file, and the plaintext extractor rightly refuses the executable
/// type. (`.cmd` already resolves to `text/plain`; it is listed so the pair
/// cannot drift.)
///
/// Platform-neutral on purpose: a `.ps1` copied to a Linux box should classify
/// the same way.
const EXTENSION_OVERRIDES: &[(&str, &str)] = &[
("bat", "text/plain"),
("cmd", "text/plain"),
("inf", "text/plain"),
("ps1", "text/plain"),
("psd1", "text/plain"),
("psm1", "text/plain"),
("url", "text/plain"),
];
/// Look up [`EXTENSION_OVERRIDES`] for `path`. Extension comparison is
/// ASCII-case-insensitive, which matters more on Windows where `REPORT.BAT` is
/// as common as the lowercase spelling.
fn extension_override(path: &Path) -> Option<&'static str> {
let ext = path.extension()?.to_str()?.to_ascii_lowercase();
EXTENSION_OVERRIDES
.iter()
.find(|(e, _)| *e == ext)
.map(|(_, mime)| *mime)
}
/// Infer a MIME type from a path plus the file's leading bytes.
///
/// Extension first — an override table, then `mime_guess` — and magic bytes
/// only when those come up empty or say `application/octet-stream`.
///
/// `head` is whatever the caller already read; indexing passes the same buffer
/// it hashes. It bounds magic-byte detection, so a caller that supplies fewer
/// than 262 bytes (`infer`'s longest signature) can get `None` where a longer
/// head would have matched. The indexer's `hash_length` defaults to 8 KiB —
/// exactly what `infer` itself reads from a path — so at default config this
/// is as good as opening the file, and strictly cheaper.
///
/// A `None` result is a real answer, not a "don't know": the content pass
/// stores it and does not re-derive it (see
/// [`crate::file_handling::extract_and_store`]).
pub fn guess_mime_from_head(path: &Path, head: &[u8]) -> Option<String> {
if let Some(m) = extension_override(path) {
return Some(m.to_string());
}
if let Some(g) = mime_guess::from_path(path).first() { if let Some(g) = mime_guess::from_path(path).first() {
let s = g.essence_str(); let s = g.essence_str();
if !s.is_empty() && s != "application/octet-stream" { if !s.is_empty() && s != "application/octet-stream" {
return Some(s.to_string()); return Some(s.to_string());
} }
} }
// Magic-byte fallback. `infer::get_from_path` handles errors by returning None. infer::get(head).map(|t| t.mime_type().to_string())
if let Ok(Some(t)) = infer::get_from_path(path) {
return Some(t.mime_type().to_string());
}
None
} }
/// Map a MIME string to a [`FileType`] bitmask. Ported from Baloo's /// Map a MIME string to a [`FileType`] bitmask. Ported from Baloo's
@ -131,7 +182,13 @@ pub fn mime_to_type(mime: &str) -> FileType {
| "vnd.oasis.opendocument.presentation" => { | "vnd.oasis.opendocument.presentation" => {
t |= FileType::DOCUMENT | FileType::PRESENTATION; t |= FileType::DOCUMENT | FileType::PRESENTATION;
} }
// Archives // Outlook saved messages and compiled HTML help are documents; both
// are ordinary things to find in a Windows home directory.
"vnd.ms-outlook" | "vnd.ms-htmlhelp" => {
t |= FileType::DOCUMENT;
}
// Archives. The Windows installer/cabinet formats are containers in
// exactly the same sense as the rest of this list.
"zip" "zip"
| "x-tar" | "x-tar"
| "x-7z-compressed" | "x-7z-compressed"
@ -142,7 +199,9 @@ pub fn mime_to_type(mime: &str) -> FileType {
| "x-bzip2" | "x-bzip2"
| "x-xz" | "x-xz"
| "vnd.debian.binary-package" | "vnd.debian.binary-package"
| "x-rpm" => { | "x-rpm"
| "vnd.ms-cab-compressed"
| "x-msi" => {
t |= FileType::ARCHIVE; t |= FileType::ARCHIVE;
} }
// application/xml is structured text // application/xml is structured text
@ -228,11 +287,137 @@ mod tests {
assert_eq!(FileType::from_name("Weird"), FileType::EMPTY); assert_eq!(FileType::from_name("Weird"), FileType::EMPTY);
} }
/// Extension resolution happens before magic bytes are consulted, so an
/// empty head is enough to exercise it.
#[test] #[test]
fn guess_mime_by_extension() { fn guess_mime_by_extension() {
use std::path::PathBuf; use std::path::PathBuf;
assert_eq!(guess_mime(&PathBuf::from("a.txt")).as_deref(), Some("text/plain")); let by_ext = |n: &str| guess_mime_from_head(&PathBuf::from(n), b"").unwrap_or_default();
assert_eq!(guess_mime(&PathBuf::from("a.png")).as_deref(), Some("image/png")); assert_eq!(by_ext("a.txt"), "text/plain");
assert_eq!(guess_mime(&PathBuf::from("a.mp3")).as_deref(), Some("audio/mpeg")); assert_eq!(by_ext("a.png"), "image/png");
assert_eq!(by_ext("a.mp3"), "audio/mpeg");
}
/// Every override must land on a type the plaintext extractor accepts —
/// the point of the table is that these files get their contents indexed.
#[test]
fn windows_script_types_reach_the_plaintext_extractor() {
use crate::extract::{plaintext::PlaintextExtractor, Extractor};
use std::path::PathBuf;
for name in [
"deploy.ps1",
"Module.psm1",
"Module.psd1",
"build.bat",
"build.cmd",
"driver.inf",
"bookmark.url",
] {
let mime = guess_mime_from_head(&PathBuf::from(name), b"")
.unwrap_or_else(|| panic!("{} has no MIME", name));
assert!(
PlaintextExtractor.supports(&mime),
"{} -> {} is not extractable as text",
name,
mime
);
}
}
#[test]
fn extension_overrides_are_case_insensitive() {
use std::path::PathBuf;
// Uppercase extensions are ordinary on Windows.
assert_eq!(
guess_mime_from_head(&PathBuf::from("DEPLOY.PS1"), b"").as_deref(),
Some("text/plain")
);
assert_eq!(
guess_mime_from_head(&PathBuf::from("Build.Bat"), b"").as_deref(),
Some("text/plain")
);
}
/// The override table must win over the file's actual content: a `.ps1`
/// holding something `infer` would recognise is still a script.
#[test]
fn extension_overrides_beat_magic_bytes() {
use std::path::PathBuf;
assert_eq!(
guess_mime_from_head(&PathBuf::from("a.ps1"), b"Write-Host hi").as_deref(),
Some("text/plain")
);
assert_eq!(
guess_mime_from_head(&PathBuf::from("a.ps1"), b"%PDF-1.7").as_deref(),
Some("text/plain")
);
}
#[test]
fn sql_dumps_are_extractable() {
use crate::extract::{plaintext::PlaintextExtractor, Extractor};
use std::path::PathBuf;
let mime = guess_mime_from_head(&PathBuf::from("schema.sql"), b"").unwrap();
assert!(PlaintextExtractor.supports(&mime), "{}", mime);
}
/// The content pass trusts the MIME the walk stored, including `None`, and
/// never reopens the file to second-guess it. That is only sound if a
/// `hash_length`-sized head is enough to recognise a format from its magic
/// bytes — `infer`'s longest signature is 262 bytes and the default head is
/// 8 KiB, so it is by a wide margin. This pins that for extensionless
/// files, where magic bytes are the only signal there is.
#[test]
fn a_default_sized_head_is_enough_for_magic_byte_detection() {
use std::path::PathBuf;
let head_bytes = crate::config::ProcessingConfig::default().hash_length;
let samples: &[(&str, &[u8], &str)] = &[
("png", &[0x89, b'P', b'N', b'G', 0x0d, 0x0a, 0x1a, 0x0a], "image/png"),
("gif", b"GIF89a", "image/gif"),
("pdf", b"%PDF-1.7", "application/pdf"),
("zip", &[0x50, 0x4b, 0x03, 0x04], "application/zip"),
("gz", &[0x1f, 0x8b, 0x08], "application/gzip"),
];
for (tag, magic, expected) in samples {
// No extension at all, so nothing but the bytes can answer.
let path = PathBuf::from(format!("/tmp/qs-sniff-{}", tag));
let mut body = magic.to_vec();
body.resize(head_bytes, 0);
assert_eq!(
guess_mime_from_head(&path, &body).as_deref(),
Some(*expected),
"{} must be detectable from a default-sized head",
tag
);
}
}
/// The other side of that bound: starve the head below `infer`'s longest
/// signature and detection legitimately degrades. Documented behaviour of
/// a non-default `hash_length`, not a bug — but it must stay a `None`
/// rather than a wrong guess.
#[test]
fn a_head_shorter_than_the_signature_declines_rather_than_guessing() {
use std::path::PathBuf;
let path = PathBuf::from("/tmp/qs-sniff-truncated");
assert_eq!(guess_mime_from_head(&path, b"").as_deref(), None);
assert_eq!(guess_mime_from_head(&path, &[0x89]).as_deref(), None);
// Enough bytes, and it resolves.
assert_eq!(
guess_mime_from_head(&path, &[0x89, b'P', b'N', b'G', 0x0d, 0x0a, 0x1a, 0x0a])
.as_deref(),
Some("image/png")
);
}
#[test]
fn windows_container_and_document_types_classify() {
assert!(mime_to_type("application/vnd.ms-cab-compressed").contains(FileType::ARCHIVE));
assert!(mime_to_type("application/x-msi").contains(FileType::ARCHIVE));
assert!(mime_to_type("application/vnd.ms-outlook").contains(FileType::DOCUMENT));
assert!(mime_to_type("application/vnd.ms-htmlhelp").contains(FileType::DOCUMENT));
} }
} }

View file

@ -0,0 +1,421 @@
//! The one place `#[cfg]` lives.
//!
//! Everything here answers "what does this platform do differently", so no
//! other module has to ask. Two rules keep it honest:
//!
//! - Every function is defined for every target. Callers never wrap a call
//! site in `#[cfg]`; if a platform has nothing to do, its arm is the
//! trivial one.
//! - Anything that can be decided from a string rather than a syscall is
//! split out and made testable everywhere ([`is_unc_string`],
//! [`PATH_COLLATION`]), because the test suite runs on Linux.
use std::ffi::OsString;
use std::path::{Component, Path, PathBuf};
/// The user's home directory.
///
/// On Windows `%USERPROFILE%` is checked **first**. Git Bash and MSYS2 export
/// `HOME` as a POSIX path (`/c/Users/me`) that no Win32 API can open, and
/// preferring it would point the config file, the index, and the default
/// indexing root at a directory that does not exist.
pub fn home_dir() -> Option<OsString> {
#[cfg(windows)]
{
if let Some(profile) = std::env::var_os("USERPROFILE") {
return Some(profile);
}
}
std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE"))
}
/// Whether a directory entry counts as hidden.
///
/// Unix: a leading dot. Windows: a leading dot **or** `FILE_ATTRIBUTE_HIDDEN`
/// / `FILE_ATTRIBUTE_SYSTEM` — without which `include_hidden = false` hides
/// nothing on Windows, and `$RECYCLE.BIN`, `System Volume Information`,
/// `pagefile.sys` and `AppData` all get indexed.
///
/// `meta` is a closure because on Unix it is never called: the walkers
/// deliberately avoid `metadata()`, which would cost an extra `lstat` per
/// entry and a full round trip on a network share. On Windows the cost is
/// zero anyway — both `std::fs::DirEntry::metadata` and
/// `walkdir::DirEntry::metadata` hand back data already cached from
/// `FindNextFileW`.
pub fn entry_is_hidden<F>(name: &str, meta: F) -> bool
where
F: FnOnce() -> Option<std::fs::Metadata>,
{
if name.starts_with('.') {
return true;
}
#[cfg(windows)]
{
use std::os::windows::fs::MetadataExt;
use windows_sys::Win32::Storage::FileSystem::{
FILE_ATTRIBUTE_HIDDEN, FILE_ATTRIBUTE_SYSTEM,
};
if let Some(m) = meta() {
return m.file_attributes() & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM) != 0;
}
}
#[cfg(not(windows))]
{
let _ = meta;
}
false
}
/// Whether `path` has a hidden component *below* the root that contains it.
///
/// Components at or above a root are exempt, because the walkers exempt their
/// root too (depth 0 is always kept — users explicitly chose their roots).
/// The two must agree: if they disagree, a full run indexes a file that the
/// watcher then refuses to update, and the index churns on every cycle.
///
/// That is a latent bug on Unix (`~/.config/app` as a root) and a certainty on
/// Windows, where `AppData` carries `FILE_ATTRIBUTE_HIDDEN` and
/// `std::env::temp_dir()` lives underneath it.
///
/// `roots` are matched by whole path components, so `/a/bc` is not treated as
/// living under `/a/b`. A path under no known root is checked in full.
pub fn path_has_hidden_component_under(path: &Path, roots: &[PathBuf]) -> bool {
// Innermost containing root wins: with both `/data` and `/data/.cache`
// configured, a file under the latter is only judged below `.cache`.
let base = roots
.iter()
.filter(|r| path.starts_with(r))
.max_by_key(|r| r.components().count());
let (mut current, tail) = match base {
Some(root) => match path.strip_prefix(root) {
Ok(tail) => (root.clone(), tail),
Err(_) => (PathBuf::new(), path),
},
None => (PathBuf::new(), path),
};
// Rebuild the absolute path as we descend: a bare tail component cannot
// be stat'd on its own, and the attribute check needs a real path.
for component in tail.components() {
current.push(component);
if let Component::Normal(name) = component {
let name = name.to_string_lossy();
if entry_is_hidden(&name, || std::fs::metadata(&current).ok()) {
return true;
}
}
}
false
}
/// Whether `s` names a UNC path, in either spelling.
///
/// Split out from [`is_network_path`] so the string half is testable on every
/// platform, and written with explicit parentheses — the precedence of `&&`
/// against `||` is exactly the kind of thing that silently disables the
/// network thread pool.
///
/// Only *called* on Windows; compiled everywhere so its tests run everywhere,
/// which is the point of splitting it out.
#[cfg_attr(not(windows), allow(dead_code))]
pub(crate) fn is_unc_string(s: &str) -> bool {
s.starts_with(r"\\?\UNC\") || (s.starts_with(r"\\") && !s.starts_with(r"\\?\"))
}
/// Filesystem types whose operations are network round trips.
#[cfg(target_os = "linux")]
const NETWORK_FS_TYPES: [&str; 8] = [
"cifs", "smb3", "smbfs", "nfs", "nfs4", "afs", "fuse.sshfs", "9p",
];
/// Whether `path` lives on a network filesystem.
///
/// Reads `/proc/mounts` and takes the longest mount point that is a prefix of
/// `path` — the innermost mount is the one that actually serves it.
#[cfg(target_os = "linux")]
pub(crate) fn is_network_path(path: &Path) -> bool {
let Ok(mounts) = std::fs::read_to_string("/proc/mounts") else {
return false;
};
let target = std::fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf());
let mut best: Option<(usize, bool)> = None;
for line in mounts.lines() {
let mut fields = line.split_whitespace();
let (Some(_dev), Some(point), Some(fstype)) = (fields.next(), fields.next(), fields.next())
else {
continue;
};
// `/proc/mounts` octal-escapes spaces and a few other characters.
let point = point.replace("\\040", " ");
let point = Path::new(&point);
if !target.starts_with(point) {
continue;
}
let depth = point.components().count();
let is_network = NETWORK_FS_TYPES.contains(&fstype);
if best.is_none_or(|(d, _)| depth > d) {
best = Some((depth, is_network));
}
}
best.is_some_and(|(_, is_network)| is_network)
}
/// Whether `path` is served by a network redirector.
///
/// UNC needs no syscall. A *mapped drive letter* does: `Z:\` backed by an SMB
/// share is indistinguishable from a local disk by string inspection, and it
/// is the common case — asking `GetDriveTypeW` is the only way to tell. Left
/// undetected it walks with `LOCAL_THREADS` instead of `NETWORK_THREADS`,
/// which is the exact failure the threading design exists to prevent.
#[cfg(windows)]
pub(crate) fn is_network_path(path: &Path) -> bool {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::GetDriveTypeW;
use windows_sys::Win32::System::WindowsProgramming::DRIVE_REMOTE;
let s = path.to_string_lossy();
if is_unc_string(&s) {
return true;
}
// GetDriveTypeW wants a root ("Z:\"), not an arbitrary path.
let Some(root) = path.components().next() else {
return false;
};
let Component::Prefix(prefix) = root else {
return false;
};
let mut wide: Vec<u16> = prefix.as_os_str().encode_wide().collect();
wide.push(b'\\' as u16);
wide.push(0);
unsafe { GetDriveTypeW(wide.as_ptr()) == DRIVE_REMOTE }
}
#[cfg(not(any(target_os = "linux", windows)))]
pub(crate) fn is_network_path(_path: &Path) -> bool {
false
}
/// Whether the filesystem-notification backend covers a whole tree from one
/// watch on its root.
///
/// `false` (inotify): one watch descriptor covers exactly one directory's
/// entries, so the caller must walk the tree and register every directory
/// itself — which is what lets it skip `.git`, `node_modules` and hidden
/// subtrees instead of spending a scarce descriptor on each.
///
/// `true` (`ReadDirectoryChangesW`): one handle covers the subtree, and
/// directories created later are included automatically. Registering
/// per-directory here would be actively harmful rather than merely wasteful —
/// notify allocates a 16 KiB buffer *inline per watch* plus a directory
/// handle, so a large tree would ask for gigabytes of buffers and tens of
/// thousands of handles. The pruning moves to the event path instead.
///
/// macOS FSEvents is also natively recursive, but it is left on the
/// per-directory path here because that path works there and is the one under
/// test.
pub const WATCH_ROOTS_RECURSIVELY: bool = cfg!(windows);
/// SQLite collation for comparing stored path strings.
///
/// Windows filesystems are case-insensitive, and SQLite's `LIKE` already folds
/// ASCII case by default. A path filter that compares one half with `=` and the
/// other with `LIKE` would otherwise disagree with itself. `NOCASE` folds ASCII
/// only, which matches what `LIKE` does — non-ASCII paths stay case-sensitive
/// on both sides, consistently.
pub const PATH_COLLATION: &str = if cfg!(windows) { "NOCASE" } else { "BINARY" };
/// How long to keep retrying a delete that fails because something else holds
/// the file open.
#[cfg(windows)]
const REMOVE_RETRY_BUDGET: std::time::Duration = std::time::Duration::from_millis(500);
/// `fs::remove_file`, retried briefly on Windows.
///
/// Unix `unlink` succeeds even with the file open, so this is a single call
/// there. Windows returns a sharing violation while *any* handle is open —
/// most often an antivirus scanner reading the file microseconds after we
/// closed it. The retry turns a spurious hard failure into a short pause.
pub fn remove_file_retrying(path: &Path) -> std::io::Result<()> {
#[cfg(not(windows))]
{
std::fs::remove_file(path)
}
#[cfg(windows)]
{
let deadline = std::time::Instant::now() + REMOVE_RETRY_BUDGET;
loop {
match std::fs::remove_file(path) {
Ok(()) => return Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Err(e),
Err(e) => {
if std::time::Instant::now() >= deadline {
return Err(e);
}
std::thread::sleep(std::time::Duration::from_millis(25));
}
}
}
}
}
/// Deny read access to `dir`, for tests that exercise the unreadable-directory
/// guards.
///
/// Exposed (hidden) rather than duplicated per test module because
/// `tests/full_index.rs` is a separate crate and needs it too. Windows uses
/// `icacls`: a deny ACE binds even the owner until the paired
/// [`restore_read`] rewrites it, and neither call needs elevation.
#[doc(hidden)]
pub fn deny_read(dir: &Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o000))
}
#[cfg(windows)]
{
icacls(dir, &["/deny", &format!("{}:(OI)(CI)(RD)", current_user()?)])
}
#[cfg(not(any(unix, windows)))]
{
let _ = dir;
Err(std::io::Error::new(
std::io::ErrorKind::Unsupported,
"deny_read is not supported on this target",
))
}
}
/// Undo [`deny_read`] so the directory can be cleaned up.
#[doc(hidden)]
pub fn restore_read(dir: &Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o755))
}
#[cfg(windows)]
{
icacls(dir, &["/remove:d", &current_user()?])
}
#[cfg(not(any(unix, windows)))]
{
let _ = dir;
Ok(())
}
}
#[cfg(windows)]
fn current_user() -> std::io::Result<String> {
match (std::env::var("USERDOMAIN"), std::env::var("USERNAME")) {
(Ok(domain), Ok(user)) => Ok(format!("{}\\{}", domain, user)),
(_, Ok(user)) => Ok(user),
_ => Err(std::io::Error::other("USERNAME is not set")),
}
}
#[cfg(windows)]
fn icacls(dir: &Path, args: &[&str]) -> std::io::Result<()> {
let out = std::process::Command::new("icacls")
.arg(dir)
.args(args)
.output()?;
if out.status.success() {
Ok(())
} else {
Err(std::io::Error::other(format!(
"icacls {}: {}",
dir.display(),
String::from_utf8_lossy(&out.stderr).trim()
)))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn unc_spellings() {
assert!(is_unc_string(r"\\server\share"));
assert!(is_unc_string(r"\\server\share\dir\file.txt"));
assert!(is_unc_string(r"\\?\UNC\server\share"));
// A verbatim *drive* path is local, not a share. This is the case the
// original `&&`/`||` precedence got wrong.
assert!(!is_unc_string(r"\\?\C:\Users\me"));
assert!(!is_unc_string(r"C:\Users\me"));
assert!(!is_unc_string("/home/me"));
assert!(!is_unc_string(""));
}
#[test]
fn collation_matches_like_case_folding() {
// LIKE folds ASCII case on every platform; the `=` half of a path
// filter has to agree with it, which is what this constant is for.
assert_eq!(PATH_COLLATION, if cfg!(windows) { "NOCASE" } else { "BINARY" });
}
#[test]
fn dotfiles_are_hidden_without_consulting_metadata() {
let mut called = false;
assert!(entry_is_hidden(".git", || {
called = true;
None
}));
assert!(!called, "a dot prefix must short-circuit before any stat");
}
#[test]
fn ordinary_names_are_not_hidden() {
assert!(!entry_is_hidden("Documents", || None));
assert!(!entry_is_hidden("report.txt", || None));
}
#[test]
fn hidden_components_are_measured_from_the_innermost_root() {
let root = PathBuf::from(format!("{}.config", sep_prefix()));
let roots = vec![root.clone()];
// The root itself is hidden, but it was chosen explicitly — the walk
// keeps it, so the watcher must too.
assert!(!path_has_hidden_component_under(&root, &roots));
assert!(!path_has_hidden_component_under(&root.join("app.conf"), &roots));
// A dot *below* the root still counts.
assert!(path_has_hidden_component_under(
&root.join(".secret").join("x"),
&roots
));
}
#[test]
fn a_path_under_no_root_is_checked_in_full() {
let roots = vec![PathBuf::from(format!("{}srv", sep_prefix()))];
let stray = PathBuf::from(format!("{}home{}me{}.ssh", sep_prefix(), SEP, SEP));
assert!(path_has_hidden_component_under(&stray, &roots));
}
#[test]
fn sibling_roots_do_not_capture_each_other() {
// `/a/bc` does not live under `/a/b`, so the `.x` below it is judged,
// not exempted.
let roots = vec![PathBuf::from(format!("{}a{}b", sep_prefix(), SEP))];
let other = PathBuf::from(format!("{}a{}bc{}.x", sep_prefix(), SEP, SEP));
assert!(path_has_hidden_component_under(&other, &roots));
}
const SEP: char = std::path::MAIN_SEPARATOR;
/// An absolute-path prefix for the running platform, so these tests read
/// the same on both.
fn sep_prefix() -> String {
if cfg!(windows) {
r"C:\".to_string()
} else {
"/".to_string()
}
}
}

View file

@ -32,7 +32,32 @@ impl std::fmt::Display for LexError {
impl std::error::Error for LexError {} impl std::error::Error for LexError {}
/// Whether the `:` at `colon` is the one in a drive letter rather than a
/// property operator.
///
/// True only when the word so far is exactly one ASCII letter *and* a path
/// separator follows, which is narrow enough to leave `12:30`, `a:b` and
/// `type:Audio` tokenizing exactly as before.
fn is_drive_letter_colon(bytes: &[u8], start: usize, colon: usize) -> bool {
colon == start + 1
&& bytes[start].is_ascii_alphabetic()
&& matches!(bytes.get(colon + 1), Some(b'\\') | Some(b'/'))
}
pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError> { pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError> {
let (tokens, err) = tokenize_spanned(input);
match err {
Some(e) => Err(e),
None => Ok(tokens.into_iter().map(|(t, _)| t).collect()),
}
}
/// [`tokenize`], but each token carries its byte range in `input`, and a
/// trailing error (unterminated quote, invalid UTF-8) is returned alongside
/// the tokens lexed before it instead of discarding them. `Quoted` spans
/// include both quote characters. This is what the GUI's syntax highlighter
/// runs on: it must color the intact prefix of a half-typed query.
pub fn tokenize_spanned(input: &str) -> (Vec<(Token, std::ops::Range<usize>)>, Option<LexError>) {
let bytes = input.as_bytes(); let bytes = input.as_bytes();
let mut i = 0usize; let mut i = 0usize;
let mut out = Vec::new(); let mut out = Vec::new();
@ -45,62 +70,72 @@ pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError> {
} }
match b { match b {
b'(' => { b'(' => {
out.push(Token::LParen); out.push((Token::LParen, i..i + 1));
i += 1; i += 1;
} }
b')' => { b')' => {
out.push(Token::RParen); out.push((Token::RParen, i..i + 1));
i += 1; i += 1;
} }
b':' => { b':' => {
out.push(Token::Op(Op::Contains)); out.push((Token::Op(Op::Contains), i..i + 1));
i += 1; i += 1;
} }
b'=' => { b'=' => {
out.push(Token::Op(Op::Eq)); out.push((Token::Op(Op::Eq), i..i + 1));
i += 1; i += 1;
} }
b'<' => { b'<' => {
if bytes.get(i + 1) == Some(&b'=') { if bytes.get(i + 1) == Some(&b'=') {
out.push(Token::Op(Op::Le)); out.push((Token::Op(Op::Le), i..i + 2));
i += 2; i += 2;
} else { } else {
out.push(Token::Op(Op::Lt)); out.push((Token::Op(Op::Lt), i..i + 1));
i += 1; i += 1;
} }
} }
b'>' => { b'>' => {
if bytes.get(i + 1) == Some(&b'=') { if bytes.get(i + 1) == Some(&b'=') {
out.push(Token::Op(Op::Ge)); out.push((Token::Op(Op::Ge), i..i + 2));
i += 2; i += 2;
} else { } else {
out.push(Token::Op(Op::Gt)); out.push((Token::Op(Op::Gt), i..i + 1));
i += 1; i += 1;
} }
} }
b'"' => { b'"' => {
// Double-quoted phrase. Supports doubled-quote escape `""`. // Double-quoted phrase. Supports doubled-quote escape `""`.
//
// Copied as a UTF-8 slice, not byte by byte: `bytes[j] as char`
// decodes Latin-1, so `"José"` came back as `José` and matched
// nothing. Quoting is also how people write paths containing
// spaces, which makes this the more visible of the two.
let mut j = i + 1; let mut j = i + 1;
let mut buf = String::new(); let mut buf = String::new();
let mut segment_start = j;
while j < bytes.len() { while j < bytes.len() {
if bytes[j] == b'"' { if bytes[j] == b'"' {
buf.push_str(&input[segment_start..j]);
if bytes.get(j + 1) == Some(&b'"') { if bytes.get(j + 1) == Some(&b'"') {
buf.push('"'); buf.push('"');
j += 2; j += 2;
segment_start = j;
continue; continue;
} }
break; break;
} }
buf.push(bytes[j] as char);
j += 1; j += 1;
} }
if j >= bytes.len() { if j >= bytes.len() {
return Err(LexError { return (
message: "unterminated quoted phrase".into(), out,
offset: i, Some(LexError {
}); message: "unterminated quoted phrase".into(),
offset: i,
}),
);
} }
out.push(Token::Quoted(buf)); out.push((Token::Quoted(buf), i..j + 1));
i = j + 1; i = j + 1;
} }
_ => { _ => {
@ -108,6 +143,14 @@ pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError> {
let start = i; let start = i;
while i < bytes.len() { while i < bytes.len() {
let c = bytes[i]; let c = bytes[i];
if c == b':' && is_drive_letter_colon(bytes, start, i) {
// `C:\Users\me` is one word, not `C` `:` `\Users\me`.
// Without this, `path:C:\Users\me` parses as the
// filter `path` = `C` and silently matches nothing —
// the first thing a Windows user types.
i += 1;
continue;
}
if c.is_ascii_whitespace() if c.is_ascii_whitespace()
|| matches!(c, b'(' | b')' | b':' | b'=' | b'<' | b'>' | b'"') || matches!(c, b'(' | b')' | b':' | b'=' | b'<' | b'>' | b'"')
{ {
@ -115,22 +158,29 @@ pub fn tokenize(input: &str) -> Result<Vec<Token>, LexError> {
} }
i += 1; i += 1;
} }
let word = std::str::from_utf8(&bytes[start..i]) let word = match std::str::from_utf8(&bytes[start..i]) {
.map_err(|e| LexError { Ok(w) => w.to_string(),
message: format!("invalid UTF-8 in word: {}", e), Err(e) => {
offset: start, return (
})? out,
.to_string(); Some(LexError {
match word.as_str() { message: format!("invalid UTF-8 in word: {}", e),
"AND" => out.push(Token::And), offset: start,
"OR" => out.push(Token::Or), }),
_ => out.push(Token::Word(word)), )
} }
};
let tok = match word.as_str() {
"AND" => Token::And,
"OR" => Token::Or,
_ => Token::Word(word),
};
out.push((tok, start..i));
} }
} }
} }
Ok(out) (out, None)
} }
#[cfg(test)] #[cfg(test)]
@ -211,4 +261,153 @@ mod tests {
fn unterminated_quote_is_error() { fn unterminated_quote_is_error() {
assert!(tokenize(r#""oops"#).is_err()); assert!(tokenize(r#""oops"#).is_err());
} }
#[test]
fn non_ascii_survives_a_quoted_phrase() {
// Byte-wise copying decoded this as Latin-1 (`José`), so the phrase
// never matched anything.
let t = tokenize(r#""C:\Users\José\docs""#).unwrap();
assert_eq!(t, vec![Token::Quoted(r"C:\Users\José\docs".into())]);
// ...including around a doubled-quote escape, which splits the copy.
let t = tokenize(r#""ü""ö""#).unwrap();
assert_eq!(t, vec![Token::Quoted(r#"ü"ö"#.into())]);
}
#[test]
fn a_drive_letter_colon_does_not_split_the_word() {
let t = tokenize(r"path:C:\Users\me\docs").unwrap();
assert_eq!(
t,
vec![
Token::Word("path".into()),
Token::Op(Op::Contains),
Token::Word(r"C:\Users\me\docs".into()),
]
);
// Forward slashes are equally valid on Windows.
let t = tokenize("path:D:/data").unwrap();
assert_eq!(
t,
vec![
Token::Word("path".into()),
Token::Op(Op::Contains),
Token::Word("D:/data".into()),
]
);
}
#[test]
fn spans_cover_every_token_shape() {
let input = r#"type:Audio "a b" (x) size<=5"#;
let (toks, err) = tokenize_spanned(input);
assert!(err.is_none());
let spanned: Vec<(&str, Token)> = toks
.iter()
.map(|(t, r)| (&input[r.clone()], t.clone()))
.collect();
assert_eq!(
spanned,
vec![
("type", Token::Word("type".into())),
(":", Token::Op(Op::Contains)),
("Audio", Token::Word("Audio".into())),
(r#""a b""#, Token::Quoted("a b".into())),
("(", Token::LParen),
("x", Token::Word("x".into())),
(")", Token::RParen),
("size", Token::Word("size".into())),
("<=", Token::Op(Op::Le)),
("5", Token::Word("5".into())),
]
);
}
#[test]
fn quoted_span_includes_quotes_and_escapes() {
let input = r#"x "a""b" y"#;
let (toks, err) = tokenize_spanned(input);
assert!(err.is_none());
assert_eq!(toks[1].0, Token::Quoted(r#"a"b"#.into()));
assert_eq!(&input[toks[1].1.clone()], r#""a""b""#);
}
#[test]
fn spans_are_byte_offsets_around_non_ascii() {
let input = r#"José "café" naïve"#;
let (toks, err) = tokenize_spanned(input);
assert!(err.is_none());
assert_eq!(&input[toks[0].1.clone()], "José");
assert_eq!(&input[toks[1].1.clone()], r#""café""#);
assert_eq!(&input[toks[2].1.clone()], "naïve");
assert_eq!(toks[2].1.end, input.len());
}
#[test]
fn unterminated_quote_keeps_prefix_tokens() {
let (toks, err) = tokenize_spanned(r#"type:Audio "oops"#);
let err = err.expect("should report the unterminated quote");
assert_eq!(err.offset, 11, "offset of the opening quote");
assert_eq!(
toks.iter().map(|(t, _)| t.clone()).collect::<Vec<_>>(),
vec![
Token::Word("type".into()),
Token::Op(Op::Contains),
Token::Word("Audio".into()),
]
);
}
#[test]
fn stars_stay_inside_words() {
let (toks, err) = tokenize_spanned("foo*bar *");
assert!(err.is_none());
assert_eq!(
toks.iter().map(|(t, _)| t.clone()).collect::<Vec<_>>(),
vec![Token::Word("foo*bar".into()), Token::Word("*".into())]
);
}
#[test]
fn tokenize_matches_span_stripped_tokenize_spanned() {
for input in ["a AND (b:c)", r#""q" x>=2"#, "path:C:\\U foo*"] {
let plain = tokenize(input).unwrap();
let (spanned, err) = tokenize_spanned(input);
assert!(err.is_none());
let stripped: Vec<Token> = spanned.into_iter().map(|(t, _)| t).collect();
assert_eq!(plain, stripped, "input {:?}", input);
}
}
#[test]
fn the_drive_letter_rule_stays_narrow() {
// Two digits before the colon: still a time, not a drive.
assert_eq!(
tokenize("12:30").unwrap(),
vec![
Token::Word("12".into()),
Token::Op(Op::Contains),
Token::Word("30".into()),
]
);
// One letter, but no separator after the colon.
assert_eq!(
tokenize("a:b").unwrap(),
vec![
Token::Word("a".into()),
Token::Op(Op::Contains),
Token::Word("b".into()),
]
);
// A separator, but the key is longer than one character.
assert_eq!(
tokenize("type:/Audio").unwrap(),
vec![
Token::Word("type".into()),
Token::Op(Op::Contains),
Token::Word("/Audio".into()),
]
);
}
} }

View file

@ -22,7 +22,12 @@
pub mod ast; pub mod ast;
pub mod lexer; pub mod lexer;
pub mod parser; pub mod parser;
pub mod pattern;
pub mod split;
pub mod translator; pub mod translator;
pub use ast::{Op, Term}; pub use ast::{Op, Term};
pub use translator::{parse_and_build, SqlQuery}; pub use lexer::tokenize_spanned;
pub use pattern::{RegexQuery, TermPattern};
pub use split::{split_for_cascade, CascadeQuery};
pub use translator::{parse_and_build, SqlQuery, TranslateError};

View file

@ -0,0 +1,533 @@
//! Compiled matchers for the cascade term and the `regex:` keyword.
//!
//! A term with an unquoted `*` compiles to a small regex (every literal
//! chunk escaped, stars joined with `.*`), so wildcards and `regex:` share
//! one linear-time matching engine. Terms without stars stay on the
//! [`Literal`](TermPattern::Literal) path, which reproduces the cascade's
//! original `==`/`find`/`count_occurrences` semantics byte for byte.
//!
//! `.` never matches `\n`, so a star cannot span lines of extracted text —
//! a `*` bridging a whole document would produce absurd match ranges and
//! page-sized snippets. Names and paths contain no newlines, so the rule
//! only shows up in content matching.
use std::ops::Range;
use regex::{Regex, RegexBuilder};
use super::translator::TranslateError;
use crate::snippet;
/// Compile-time memory cap for user-supplied and derived regexes. Keeps a
/// hostile pattern (`a{1000000}{1000}` and friends) from ballooning the
/// compiled program; matching itself is linear-time by construction.
const REGEX_SIZE_LIMIT: usize = 4 << 20;
/// Occurrence counts saturate here, matching `count_frac` in the cascade.
const COUNT_CAP: usize = 1000;
/// The cascade term, compiled once at split time.
#[derive(Debug, Clone, Default)]
pub enum TermPattern {
/// No matchable content: an empty term, or only stars (`*`, `**`).
/// Matches nothing — a bare `*` must not become a scan of everything.
#[default]
Empty,
/// A star-free term. Kept as plain string operations, not a regex, so
/// existing queries keep their exact semantics (and speed).
Literal(LiteralPattern),
/// A term with at least one active wildcard.
Wildcard(WildcardPattern),
}
#[derive(Debug, Clone)]
pub struct LiteralPattern {
text: String,
folded: String,
}
#[derive(Debug, Clone)]
pub struct WildcardPattern {
/// Literal chunks between stars, in order. Never empty, and no chunk
/// is empty: edge stars are folded into the compiled regexes, doubled
/// stars collapse.
segments: Vec<String>,
/// Unanchored search regexes with non-greedy joins — leftmost-shortest
/// match, which is what a snippet window wants.
search_cs: Regex,
search_ci: Regex,
/// Anchored (`^…$`) regexes for whole-field matching (rank tiers 1/2).
anchored_cs: Regex,
anchored_ci: Regex,
}
/// One piece of the search phrase as split out of the token stream.
/// `glob` is true only for plain unquoted words — quoted phrases and
/// reassembled `key:value` text keep their stars literal.
#[derive(Debug, Clone)]
pub struct TermPart {
pub text: String,
pub glob: bool,
}
/// A chunk stream: literal text interleaved with active stars.
enum Chunk {
Lit(String),
Star,
}
impl TermPattern {
/// Compile the joined term parts. Parts are joined with a single space,
/// exactly like the display term (`parts.join(" ")`).
pub fn build(parts: &[TermPart]) -> Result<TermPattern, TranslateError> {
let mut chunks: Vec<Chunk> = Vec::new();
let push_lit = |chunks: &mut Vec<Chunk>, s: &str| {
if s.is_empty() {
return;
}
if let Some(Chunk::Lit(prev)) = chunks.last_mut() {
prev.push_str(s);
} else {
chunks.push(Chunk::Lit(s.to_string()));
}
};
for (idx, part) in parts.iter().enumerate() {
if idx > 0 {
push_lit(&mut chunks, " ");
}
if part.glob {
let mut first = true;
for piece in part.text.split('*') {
if !first && !matches!(chunks.last(), Some(Chunk::Star)) {
chunks.push(Chunk::Star);
}
first = false;
push_lit(&mut chunks, piece);
}
} else {
push_lit(&mut chunks, &part.text);
}
}
let leading = matches!(chunks.first(), Some(Chunk::Star));
let trailing = chunks.len() > 1 && matches!(chunks.last(), Some(Chunk::Star));
let has_star = chunks.iter().any(|c| matches!(c, Chunk::Star));
let segments: Vec<String> = chunks
.into_iter()
.filter_map(|c| match c {
Chunk::Lit(s) => Some(s),
Chunk::Star => None,
})
.collect();
if segments.is_empty() {
// "" or stars only.
return Ok(TermPattern::Empty);
}
if !has_star {
let text = segments.into_iter().next().unwrap();
let folded = text.to_ascii_lowercase();
return Ok(TermPattern::Literal(LiteralPattern { text, folded }));
}
let escaped: Vec<String> = segments.iter().map(|s| regex::escape(s)).collect();
let compile = |src: &str, ci: bool| -> Result<Regex, TranslateError> {
RegexBuilder::new(src)
.case_insensitive(ci)
.size_limit(REGEX_SIZE_LIMIT)
.build()
.map_err(|e| TranslateError::BadRegex(e.to_string()))
};
// Edge stars are dropped from the search form — under substring
// semantics a leading/trailing `.*?` adds nothing.
let search_src = escaped.join(".*?");
// The anchored form keeps them: `*foo` must whole-match "myfoo".
let anchored_src = format!(
"^{}{}{}$",
if leading { ".*" } else { "" },
escaped.join(".*"),
if trailing { ".*" } else { "" },
);
Ok(TermPattern::Wildcard(WildcardPattern {
search_cs: compile(&search_src, false)?,
search_ci: compile(&search_src, true)?,
anchored_cs: compile(&anchored_src, false)?,
anchored_ci: compile(&anchored_src, true)?,
segments,
}))
}
pub fn is_empty(&self) -> bool {
matches!(self, TermPattern::Empty)
}
pub fn is_wildcard(&self) -> bool {
matches!(self, TermPattern::Wildcard(_))
}
/// The literal text, when the term has no wildcard. SQL builders branch
/// on this to keep the original single-`LIKE`/phrase-`MATCH` shapes.
pub fn literal(&self) -> Option<&str> {
match self {
TermPattern::Literal(l) => Some(&l.text),
_ => None,
}
}
/// Literal chunks between wildcards (the whole term when literal).
pub fn segments(&self) -> &[String] {
match self {
TermPattern::Empty => &[],
TermPattern::Literal(l) => std::slice::from_ref(&l.text),
TermPattern::Wildcard(w) => &w.segments,
}
}
/// Characters of literal (non-star) content — the trigram floor and
/// path-tier switch count these.
pub fn literal_char_count(&self) -> usize {
self.segments().iter().map(|s| s.chars().count()).sum()
}
/// Does the pattern match the entire field?
pub fn whole_match(&self, text: &str, case_insensitive: bool) -> bool {
match self {
TermPattern::Empty => false,
TermPattern::Literal(l) => {
if case_insensitive {
text.eq_ignore_ascii_case(&l.text)
} else {
text == l.text
}
}
TermPattern::Wildcard(w) => {
let re = if case_insensitive {
&w.anchored_ci
} else {
&w.anchored_cs
};
re.is_match(text)
}
}
}
/// Leftmost match as a byte range. Literal folding is ASCII-only and
/// byte-length preserving, so folded offsets are valid in the original —
/// the same invariant the cascade has always relied on.
pub fn find_first(&self, text: &str, case_insensitive: bool) -> Option<Range<usize>> {
match self {
TermPattern::Empty => None,
TermPattern::Literal(l) => {
let pos = if case_insensitive {
text.to_ascii_lowercase().find(&l.folded)?
} else {
text.find(&l.text)?
};
Some(pos..pos + l.text.len())
}
TermPattern::Wildcard(w) => {
let re = if case_insensitive {
&w.search_ci
} else {
&w.search_cs
};
re.find(text).map(|m| m.range())
}
}
}
/// Non-overlapping occurrence count, capped at 1000 (the cascade's
/// `count_frac` saturates there anyway).
pub fn count(&self, text: &str, case_insensitive: bool) -> usize {
match self {
TermPattern::Empty => 0,
TermPattern::Literal(l) => {
snippet::count_occurrences(text, &l.text, !case_insensitive)
}
TermPattern::Wildcard(w) => {
let re = if case_insensitive {
&w.search_ci
} else {
&w.search_cs
};
re.find_iter(text).take(COUNT_CAP).count()
}
}
}
}
/// A compiled `regex:` query. Case-insensitive by default (override with an
/// inline `(?-i:…)`); `multi_line` makes `^`/`$` per-line over extracted
/// text, which is what they mean in a search box.
#[derive(Debug, Clone)]
pub struct RegexQuery {
pub source: String,
re: Regex,
}
impl RegexQuery {
pub fn new(source: &str) -> Result<RegexQuery, TranslateError> {
let re = RegexBuilder::new(source)
.case_insensitive(true)
.multi_line(true)
.size_limit(REGEX_SIZE_LIMIT)
.build()
.map_err(|e| TranslateError::BadRegex(e.to_string()))?;
// The regex analog of the bare-`*` rule, but loud: the user typed an
// explicit keyword, so tell them instead of matching every file.
if re.is_match("") {
return Err(TranslateError::BadRegex(format!(
"'{}' can match the empty string and would match every file",
source
)));
}
Ok(RegexQuery {
source: source.to_string(),
re,
})
}
pub fn is_match(&self, text: &str) -> bool {
self.re.is_match(text)
}
pub fn find_first(&self, text: &str) -> Option<Range<usize>> {
self.re.find(text).map(|m| m.range())
}
/// Non-overlapping occurrence count, capped at 1000.
pub fn count(&self, text: &str) -> usize {
self.re.find_iter(text).take(COUNT_CAP).count()
}
}
/// Cap a match range at `max_len` bytes (aligned back to a char boundary)
/// before handing it to `snippet::window_around`. A greedy user regex can
/// legitimately match megabytes of a minified file; the snippet window
/// wants the start of that, not all of it.
pub fn clamp_match_range(text: &str, range: Range<usize>, max_len: usize) -> Range<usize> {
let mut end = range.end.min(range.start + max_len);
while end > range.start && !text.is_char_boundary(end) {
end -= 1;
}
range.start..end
}
#[cfg(test)]
mod tests {
use super::*;
fn part(text: &str, glob: bool) -> TermPart {
TermPart {
text: text.into(),
glob,
}
}
fn wildcard(parts: &[TermPart]) -> WildcardPattern {
match TermPattern::build(parts).unwrap() {
TermPattern::Wildcard(w) => w,
other => panic!("expected wildcard, got {:?}", other),
}
}
#[test]
fn starless_parts_build_a_literal() {
let p = TermPattern::build(&[part("hello", false), part("world", true)]).unwrap();
assert_eq!(p.literal(), Some("hello world"));
assert!(!p.is_wildcard());
}
#[test]
fn empty_and_star_only_terms_match_nothing() {
for parts in [
vec![],
vec![part("", false)],
vec![part("*", true)],
vec![part("**", true)],
] {
let p = TermPattern::build(&parts).unwrap();
assert!(p.is_empty(), "{:?}", parts);
assert!(!p.whole_match("anything", true));
assert!(p.find_first("anything", true).is_none());
assert_eq!(p.count("anything", true), 0);
}
}
#[test]
fn quoted_star_stays_literal() {
// A quoted "*" arrives with glob = false.
let p = TermPattern::build(&[part("a*b", false)]).unwrap();
assert_eq!(p.literal(), Some("a*b"));
assert!(p.find_first("xa*by", false).is_some());
assert!(p.find_first("aXb", false).is_none());
}
#[test]
fn segment_shapes() {
// Edge stars vanish into the anchors: `*foo` whole-matches any
// suffix `foo`, `foo*` any prefix.
let p = TermPattern::build(&[part("*foo", true)]).unwrap();
assert_eq!(p.segments(), ["foo"]);
assert!(p.whole_match("myfoo", false));
assert!(!p.whole_match("foomy", false));
let p = TermPattern::build(&[part("foo*", true)]).unwrap();
assert_eq!(p.segments(), ["foo"]);
assert!(p.whole_match("foomy", false));
assert!(!p.whole_match("myfoo", false));
let w = wildcard(&[part("f*o*o", true)]);
assert_eq!(w.segments, ["f", "o", "o"]);
// Doubled stars collapse.
let w = wildcard(&[part("f**o", true)]);
assert_eq!(w.segments, ["f", "o"]);
// The implicit joining space is literal content.
let w = wildcard(&[part("a*", true), part("b", false)]);
assert_eq!(w.segments, ["a", " b"]);
// `* *` — the joining space between two stars is interior literal
// content, so this is a real (if odd) pattern, not Empty.
let p = TermPattern::build(&[part("*", true), part("*", true)]).unwrap();
assert_eq!(p.segments(), [" "]);
assert!(p.whole_match("a b", false));
assert!(!p.whole_match("ab", false));
let p = TermPattern::build(&[part("*x", true), part("y*", true)]).unwrap();
assert_eq!(p.segments(), ["x y"]);
assert!(p.whole_match("ax yb", false));
}
#[test]
fn whole_match_uses_anchors() {
let p = TermPattern::build(&[part("*.txt", true)]).unwrap();
assert!(p.whole_match("notes.txt", false));
assert!(p.whole_match("NOTES.TXT", true));
assert!(!p.whole_match("NOTES.TXT", false));
assert!(!p.whole_match("notes.txt.bak", false));
let p = TermPattern::build(&[part("rep*rt", true)]).unwrap();
assert!(p.whole_match("report", false));
assert!(!p.whole_match("report2024", false));
}
#[test]
fn find_first_is_leftmost_shortest() {
let p = TermPattern::build(&[part("a*b", true)]).unwrap();
// Leftmost-first with a lazy join: starts at 0, ends at the first b.
assert_eq!(p.find_first("aXXbYYb", false), Some(0..4));
// Case-insensitive variant.
assert_eq!(p.find_first("AXXB", true), Some(0..4));
assert_eq!(p.find_first("AXXB", false), None);
}
#[test]
fn star_does_not_cross_newlines() {
let p = TermPattern::build(&[part("foo*bar", true)]).unwrap();
assert!(p.find_first("foo bar", false).is_some());
assert!(p.find_first("foo\nbar", false).is_none());
}
#[test]
fn utf8_boundaries_in_segments_and_haystacks() {
let p = TermPattern::build(&[part("café*menu", true)]).unwrap();
let hay = "le café du menu";
let r = p.find_first(hay, false).unwrap();
assert_eq!(&hay[r], "café du menu");
// Case-insensitive over non-ASCII haystack: offsets stay valid.
let hay = "LE CAFÉ DU MENU";
let r = p.find_first(hay, true).unwrap();
assert!(hay.is_char_boundary(r.start) && hay.is_char_boundary(r.end));
}
#[test]
fn count_is_nonoverlapping_and_capped() {
let p = TermPattern::build(&[part("a*b", true)]).unwrap();
assert_eq!(p.count("ab ab ab", false), 3);
let many = "ab ".repeat(2000);
assert_eq!(p.count(&many, false), 1000);
}
#[test]
fn literal_parity_with_string_ops() {
let p = TermPattern::build(&[part("Report", false)]).unwrap();
assert!(p.whole_match("Report", false));
assert!(!p.whole_match("report", false));
assert!(p.whole_match("report", true));
assert_eq!(p.find_first("my Report.pdf", false), Some(3..9));
assert_eq!(p.find_first("my report.pdf", true), Some(3..9));
assert_eq!(p.find_first("my report.pdf", false), None);
assert_eq!(p.count("report Report", false), 1);
assert_eq!(p.count("report Report", true), 2);
}
#[test]
fn regex_defaults_case_insensitive_with_optout() {
let r = RegexQuery::new("foo\\d+").unwrap();
assert!(r.is_match("FOO123"));
let r = RegexQuery::new("(?-i:FOO)\\d+").unwrap();
assert!(r.is_match("FOO1"));
assert!(!r.is_match("foo1"));
}
#[test]
fn regex_multiline_anchors() {
let r = RegexQuery::new("^total:").unwrap();
assert!(r.is_match("line one\ntotal: 5"));
}
#[test]
fn invalid_regex_is_an_error_not_a_panic() {
for src in ["[", "(", "a{2,1}", "(?P<)"] {
assert!(
matches!(RegexQuery::new(src), Err(TranslateError::BadRegex(_))),
"{:?}",
src
);
}
}
#[test]
fn empty_matchable_regexes_are_rejected() {
for src in ["", ".*", "a*", "x|", "()", "(a+)*"] {
assert!(
matches!(RegexQuery::new(src), Err(TranslateError::BadRegex(_))),
"{:?} should be rejected",
src
);
}
}
#[test]
fn hostile_regexes_fail_fast_or_run_linear() {
// Deep nesting: rejected cleanly by the parser's nest limit.
assert!(RegexQuery::new(&"(".repeat(2000)).is_err());
// Huge counted repetition: rejected by size_limit, not compiled.
assert!(RegexQuery::new("a{1000000}{1000}").is_err());
// Classic backtracking bomb: the linear engine answers immediately
// (a backtracker would take exponential time here).
let r = RegexQuery::new("(a+)+$").unwrap();
let hay = format!("{}b", "a".repeat(10_000));
assert!(!r.is_match(&hay));
}
#[test]
fn find_and_count_on_regex() {
let r = RegexQuery::new("b[aeiou]d").unwrap();
let hay = "bad bed bodkin";
assert_eq!(r.find_first(hay), Some(0..3));
assert_eq!(r.count(hay), 3);
}
#[test]
fn clamp_respects_char_boundaries() {
let text = "aééééb";
let r = clamp_match_range(text, 0..text.len(), 4);
assert!(text.is_char_boundary(r.end));
assert!(r.end <= 4);
// No-op when already short enough.
assert_eq!(clamp_match_range(text, 1..3, 100), 1..3);
}
}

View file

@ -0,0 +1,457 @@
//! Split raw search-box input into (cascade term, structured filters).
//!
//! The ranked search cascade has no boolean logic by design: everything
//! that isn't a recognized `key:value` filter joins the *term* — the single
//! phrase the cascade matches through its filename/full-text/fuzzy stages.
//! Recognized filters (`type:`, `modified:`, `path:`, `mime:`, `name:`)
//! become parameterized SQL fragments ANDed onto every cascade stage.
//!
//! Robustness rules for search-as-you-type:
//! - A lex error (e.g. a half-typed quote) degrades to "whole input is the
//! term" — incremental typing must never surface an error.
//! - An *unrecognized* `key:value` (like `12:30`) is reassembled verbatim
//! into the term.
//! - A recognized key whose value doesn't translate (bad date, unknown type
//! name) is a real [`TranslateError`] — the caller shows it inline.
//! - `AND`/`OR`/parens are not operators here; the words pass through into
//! the term, parens are dropped.
use super::ast::Op;
use super::lexer::{tokenize, Token};
use super::pattern::{RegexQuery, TermPart, TermPattern};
use super::translator::{build_filter, is_filter_key, TranslateError};
/// The cascade's parsed input: one term string plus composable filter SQL.
#[derive(Debug, Clone, Default)]
pub struct CascadeQuery {
/// The ranked search phrase; may be empty when the input was
/// filter-only or regex-only.
pub term: String,
/// `term` compiled for matching: literal, wildcard, or empty.
pub pattern: TermPattern,
/// A `regex:` filter, matched in Rust against name, path and content —
/// never part of the SQL.
pub regex: Option<RegexQuery>,
/// Zero or more ` AND (...)` fragments with anonymous `?` placeholders
/// over alias `f`; appended verbatim to every stage's WHERE clause.
pub filter_sql: String,
pub filter_params: Vec<rusqlite::types::Value>,
}
impl CascadeQuery {
/// Nothing to rank on: no term pattern and no regex. (Filters alone
/// don't drive a search.)
pub fn is_empty(&self) -> bool {
self.pattern.is_empty() && self.regex.is_none()
}
fn term_only(term: &str) -> CascadeQuery {
let term = term.trim().to_string();
// Un-lexable input is searched verbatim — stars are not wildcards
// here, mirroring the "whole input is the term" degrade rule.
let pattern = TermPattern::build(&[TermPart {
text: term.clone(),
glob: false,
}])
.expect("literal patterns always compile");
CascadeQuery {
term,
pattern,
..CascadeQuery::default()
}
}
}
fn op_str(op: Op) -> &'static str {
match op {
Op::Contains => ":",
Op::Eq => "=",
Op::Lt => "<",
Op::Le => "<=",
Op::Gt => ">",
Op::Ge => ">=",
}
}
pub fn split_for_cascade(input: &str) -> Result<CascadeQuery, TranslateError> {
// NUL bytes can't occur in filenames or extracted text, but they do
// break SQLite text binding and the FTS5 query parser — strip them.
let input = input.replace('\0', "");
let input = input.as_str();
let tokens = match tokenize(input) {
Ok(t) => t,
// Half-typed input (unterminated quote, invalid word): the whole
// raw string is the term. Never an error mid-keystroke.
Err(_) => return Ok(CascadeQuery::term_only(input)),
};
let mut out = CascadeQuery::default();
let mut term_parts: Vec<TermPart> = Vec::new();
let mut i = 0usize;
// Only plain unquoted words are wildcard-eligible; everything else
// (quoted phrases, demoted AND/OR, reassembled key:value glue) is
// searched verbatim.
let literal = |text: &str| TermPart {
text: text.to_string(),
glob: false,
};
while i < tokens.len() {
match &tokens[i] {
Token::Word(word) => {
// Candidate filter: Word(key) Op [Op] (Word|Quoted).
// The lexer emits `modified:>=x` as Word Op(:) Op(>=) Word.
if let Some(Token::Op(op1)) = tokens.get(i + 1) {
let (op, value_idx) = match tokens.get(i + 2) {
Some(Token::Op(op2)) => (*op2, i + 3),
_ => (*op1, i + 2),
};
let value = match tokens.get(value_idx) {
Some(Token::Word(v)) | Some(Token::Quoted(v)) => Some(v.clone()),
_ => None,
};
if let Some(value) = value {
// Quoted values keep `*` literal; only a bare word's
// stars act as wildcards (`name:` honors this too).
let value_is_word =
matches!(tokens.get(value_idx), Some(Token::Word(_)));
if word.eq_ignore_ascii_case("regex") {
// Not a SQL filter: compiled here, matched in
// Rust against name, path and content.
if op != Op::Contains {
return Err(TranslateError::UnsupportedOp {
key: word.clone(),
op,
});
}
if out.regex.is_some() {
return Err(TranslateError::BadRegex(
"only one regex: per query".into(),
));
}
out.regex = Some(RegexQuery::new(&value)?);
i = value_idx + 1;
continue;
}
if is_filter_key(word) {
let frag = build_filter(word, op, &value, value_is_word)?;
out.filter_sql.push_str(" AND (");
out.filter_sql.push_str(&frag.sql);
out.filter_sql.push(')');
out.filter_params.extend(frag.params);
i = value_idx + 1;
continue;
}
// Unrecognized key — reassemble verbatim (`12:30`,
// `foo:bar`), gluing any further `:value` chains
// (`foo:bar:baz`).
let mut glued = format!("{}{}{}", word, op_str(op), value);
i = value_idx + 1;
while let Some(Token::Op(next_op)) = tokens.get(i) {
glued.push_str(op_str(*next_op));
i += 1;
if let Some(Token::Word(v)) | Some(Token::Quoted(v)) = tokens.get(i)
{
glued.push_str(v);
i += 1;
}
}
term_parts.push(literal(&glued));
continue;
}
// Key + op with no value yet (mid-typing "type:"):
// pass through as literal text.
term_parts.push(literal(&format!("{}{}", word, op_str(*op1))));
i += 2;
continue;
}
term_parts.push(TermPart {
text: word.clone(),
glob: word.contains('*'),
});
}
Token::Quoted(q) => term_parts.push(literal(q)),
// Not operators in the cascade grammar — plain words.
Token::And => term_parts.push(literal("AND")),
Token::Or => term_parts.push(literal("OR")),
// Grouping has no meaning without boolean logic.
Token::LParen | Token::RParen => {}
// Dangling operator (e.g. "a > b" typed literally).
Token::Op(op) => term_parts.push(literal(op_str(*op))),
}
i += 1;
}
out.term = term_parts
.iter()
.map(|p| p.text.as_str())
.collect::<Vec<_>>()
.join(" ");
out.pattern = TermPattern::build(&term_parts)?;
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
use rusqlite::types::Value;
#[test]
fn plain_words_join_in_order() {
let q = split_for_cascade("hello brave world").unwrap();
assert_eq!(q.term, "hello brave world");
assert!(q.filter_sql.is_empty());
assert!(q.filter_params.is_empty());
}
#[test]
fn empty_input() {
let q = split_for_cascade("").unwrap();
assert_eq!(q.term, "");
assert!(q.filter_sql.is_empty());
}
#[test]
fn each_recognized_filter_key_extracts() {
for input in [
"type:Audio",
"modified:>=2024-01-01",
"mtime:<2023-12-01",
"path:/home/me",
"folder:/home/me",
"includefolder:/home/me",
"name:report",
"filename:report",
"mime:application/pdf",
] {
let q = split_for_cascade(input).unwrap();
assert_eq!(q.term, "", "input {:?} should be pure filter", input);
assert!(
q.filter_sql.starts_with(" AND ("),
"input {:?} → {:?}",
input,
q.filter_sql
);
assert!(!q.filter_params.is_empty(), "input {:?}", input);
}
}
#[test]
fn filters_and_term_mix() {
let q = split_for_cascade("type:Document budget report modified:>=2024-01-01").unwrap();
assert_eq!(q.term, "budget report");
assert_eq!(q.filter_sql.matches(" AND (").count(), 2);
assert_eq!(q.filter_params.len(), 2); // type bits + mtime bound
}
#[test]
fn unknown_key_stays_literal() {
let q = split_for_cascade("meeting 12:30 notes").unwrap();
assert_eq!(q.term, "meeting 12:30 notes");
assert!(q.filter_sql.is_empty());
}
#[test]
fn unknown_key_chain_reassembles() {
let q = split_for_cascade("foo:bar:baz").unwrap();
assert_eq!(q.term, "foo:bar:baz");
}
#[test]
fn half_typed_quote_is_whole_term() {
let q = split_for_cascade("\"unclosed phrase").unwrap();
assert_eq!(q.term, "\"unclosed phrase");
assert!(q.filter_sql.is_empty());
}
#[test]
fn half_typed_filter_key_is_literal() {
let q = split_for_cascade("type:").unwrap();
assert_eq!(q.term, "type:");
assert!(q.filter_sql.is_empty());
}
#[test]
fn recognized_key_bad_value_errors() {
assert!(matches!(
split_for_cascade("modified:>=not-a-date"),
Err(TranslateError::BadDate(_))
));
assert!(matches!(
split_for_cascade("type:NotAThing"),
Err(TranslateError::UnknownProperty(_))
));
}
#[test]
fn and_or_parens_are_plain_text() {
let q = split_for_cascade("(alpha AND beta) OR gamma").unwrap();
assert_eq!(q.term, "alpha AND beta OR gamma");
}
/// The end-to-end shape of the bug: before the lexer fix this produced
/// the filter `parent = "C"` plus a junk term, and returned nothing.
#[test]
fn a_windows_drive_path_reaches_the_filter_intact() {
let q = split_for_cascade(r"path:C:\Users\me\docs").unwrap();
assert_eq!(q.term, "", "the whole input is a filter");
assert!(matches!(
&q.filter_params[0],
Value::Text(t) if t == r"C:\Users\me\docs"
), "{:?}", q.filter_params);
}
#[test]
fn quoted_value_for_filter() {
let q = split_for_cascade("path:\"/home/me/My Documents\"").unwrap();
assert_eq!(q.term, "");
assert_eq!(
q.filter_params[0],
Value::Text("/home/me/My Documents".into())
);
}
#[test]
fn quoted_phrase_joins_term() {
let q = split_for_cascade("\"exact phrase\" extra").unwrap();
assert_eq!(q.term, "exact phrase extra");
}
#[test]
fn injection_shapes_stay_bound() {
// Everything lands either in the term (never interpolated into
// SQL by the cascade — bound as parameters there too) or in
// filter_params. filter_sql must never contain user text.
let q = split_for_cascade("mime:application/x-foo'; DROP TABLE files; --").unwrap();
assert!(!q.filter_sql.contains("DROP"), "{}", q.filter_sql);
// The value went into params (term got the trailing junk words).
assert!(matches!(&q.filter_params[0], Value::Text(t) if t.contains("x-foo'")));
let q = split_for_cascade("name:%_\\").unwrap();
// LIKE-escaped inside the bound param, not the SQL.
assert!(matches!(&q.filter_params[0], Value::Text(t) if t == "%\\%\\_\\\\%"));
}
#[test]
fn unquoted_star_builds_a_wildcard_pattern() {
let q = split_for_cascade("foo*").unwrap();
assert_eq!(q.term, "foo*");
assert!(q.pattern.is_wildcard());
assert!(q.pattern.whole_match("foobar", false));
}
#[test]
fn quoted_star_stays_literal() {
let q = split_for_cascade("\"foo*\"").unwrap();
assert_eq!(q.pattern.literal(), Some("foo*"));
}
#[test]
fn bare_star_matches_nothing() {
for input in ["*", "**", "* *"] {
let q = split_for_cascade(input).unwrap();
assert!(
q.pattern.find_first("anything", true).is_none(),
"{:?}",
input
);
}
// "* *" has an interior literal space; plain stars are Empty.
assert!(split_for_cascade("*").unwrap().pattern.is_empty());
assert!(split_for_cascade("*").unwrap().is_empty());
}
#[test]
fn glued_unknown_keys_keep_stars_literal() {
let q = split_for_cascade("foo:ba*r").unwrap();
assert_eq!(q.pattern.literal(), Some("foo:ba*r"));
}
#[test]
fn lex_error_degrade_keeps_stars_literal() {
let q = split_for_cascade("re*port \"unclosed").unwrap();
assert_eq!(q.pattern.literal(), Some("re*port \"unclosed"));
}
#[test]
fn regex_keyword_compiles_out_of_band() {
let q = split_for_cascade("regex:foo\\d+").unwrap();
assert_eq!(q.term, "");
assert!(q.pattern.is_empty());
assert!(!q.is_empty(), "a regex-only query still searches");
let re = q.regex.unwrap();
assert!(re.is_match("FOO12"));
assert!(q.filter_sql.is_empty(), "regex is not a SQL filter");
}
#[test]
fn regex_value_may_be_quoted_and_key_is_case_insensitive() {
let q = split_for_cascade("REGEX:\"foo (bar|baz)\"").unwrap();
assert!(q.regex.unwrap().is_match("foo bar"));
}
#[test]
fn regex_mixes_with_filters_and_term() {
let q = split_for_cascade("regex:\\d+ type:Text budget").unwrap();
assert_eq!(q.term, "budget");
assert!(q.regex.is_some());
assert_eq!(q.filter_sql.matches(" AND (").count(), 1);
}
#[test]
fn regex_error_shapes() {
assert!(matches!(
split_for_cascade("regex:["),
Err(TranslateError::BadRegex(_))
));
assert!(matches!(
split_for_cascade("regex=x"),
Err(TranslateError::UnsupportedOp { .. })
));
assert!(matches!(
split_for_cascade("regex:a regex:b"),
Err(TranslateError::BadRegex(_))
));
// Empty-matchable patterns are rejected loudly.
assert!(matches!(
split_for_cascade("regex:.*"),
Err(TranslateError::BadRegex(_))
));
}
#[test]
fn dangling_regex_key_is_literal_text() {
let q = split_for_cascade("regex:").unwrap();
assert_eq!(q.term, "regex:");
assert!(q.regex.is_none());
}
#[test]
fn name_filter_star_becomes_like_wildcard() {
let q = split_for_cascade("name:foo*bar").unwrap();
assert!(matches!(&q.filter_params[0], Value::Text(t) if t == "%foo%bar%"));
// Quoted value: star stays a literal character.
let q = split_for_cascade("name:\"fo*o\"").unwrap();
assert!(matches!(&q.filter_params[0], Value::Text(t) if t == "%fo*o%"));
// User LIKE metacharacters stay escaped even in glob values.
let q = split_for_cascade("name:%*_").unwrap();
assert!(matches!(&q.filter_params[0], Value::Text(t) if t == "%\\%%\\_%"));
// path: values never glob.
let q = split_for_cascade("path:/da*ta").unwrap();
assert!(matches!(&q.filter_params[0], Value::Text(t) if t == "/da*ta"));
}
#[test]
fn nul_bytes_stripped_and_long_terms_pass_through() {
// NULs would break SQLite binding / FTS5 parsing downstream.
let q = split_for_cascade("abc\0def").unwrap();
assert_eq!(q.term, "abcdef");
let long = "x".repeat(10_240);
let q = split_for_cascade(&long).unwrap();
assert_eq!(q.term.len(), 10_240);
}
}

View file

@ -35,6 +35,7 @@ pub enum TranslateError {
Parse(ParseError), Parse(ParseError),
UnknownProperty(String), UnknownProperty(String),
BadDate(String), BadDate(String),
BadRegex(String),
UnsupportedOp { UnsupportedOp {
key: String, key: String,
op: Op, op: Op,
@ -47,6 +48,7 @@ impl std::fmt::Display for TranslateError {
TranslateError::Parse(e) => write!(f, "{}", e), TranslateError::Parse(e) => write!(f, "{}", e),
TranslateError::UnknownProperty(k) => write!(f, "unknown property '{}'", k), TranslateError::UnknownProperty(k) => write!(f, "unknown property '{}'", k),
TranslateError::BadDate(s) => write!(f, "bad date '{}'", s), TranslateError::BadDate(s) => write!(f, "bad date '{}'", s),
TranslateError::BadRegex(s) => write!(f, "regex error: {}", s),
TranslateError::UnsupportedOp { key, op } => { TranslateError::UnsupportedOp { key, op } => {
write!(f, "operator {:?} is not supported for property '{}'", op, key) write!(f, "operator {:?} is not supported for property '{}'", op, key)
} }
@ -172,8 +174,10 @@ fn fts_expr(parts: &[FtsFragment]) -> String {
} }
/// Escape a phrase for FTS5 MATCH. FTS5 itself uses doubled quotes for /// Escape a phrase for FTS5 MATCH. FTS5 itself uses doubled quotes for
/// literal quotes inside a quoted phrase. /// literal quotes inside a quoted phrase; wrapping in quotes renders all
fn quote_phrase(s: &str) -> String { /// other MATCH metacharacters (`( ) * :` etc.) inert. Injection-safe by
/// construction.
pub fn quote_phrase(s: &str) -> String {
let mut buf = String::with_capacity(s.len() + 2); let mut buf = String::with_capacity(s.len() + 2);
buf.push('"'); buf.push('"');
for c in s.chars() { for c in s.chars() {
@ -243,119 +247,214 @@ impl Builder {
} }
} }
/// Delegate to the shared [`build_filter`] fragment builder, then
/// convert its anonymous `?` placeholders to this builder's numbered
/// scheme (params[0] is reserved for the FTS MATCH when one exists;
/// `build` shifts numbers afterwards).
fn translate_property( fn translate_property(
&mut self, &mut self,
key: &str, key: &str,
op: Op, op: Op,
value: &str, value: &str,
) -> Result<String, TranslateError> { ) -> Result<String, TranslateError> {
let lower_key = key.to_ascii_lowercase(); let frag = build_filter(key, op, value, false)?;
match lower_key.as_str() { let mut params = frag.params.into_iter();
"type" => self.prop_type(op, value, key), let mut out = String::with_capacity(frag.sql.len() + 8);
"modified" | "mtime" => self.prop_mtime(op, value, key), for c in frag.sql.chars() {
"path" | "folder" | "includefolder" => self.prop_path(op, value, key), if c == '?' {
"name" | "filename" => self.prop_name(op, value, key), let v = params
"mime" => self.prop_mime(op, value, key), .next()
_ => Err(TranslateError::UnknownProperty(key.to_string())), .expect("FilterFragment placeholder/param counts match");
self.all_params.push(v);
out.push('?');
out.push_str(&self.all_params.len().to_string());
} else {
out.push(c);
}
} }
Ok(out)
} }
}
fn prop_type(&mut self, op: Op, value: &str, key: &str) -> Result<String, TranslateError> { /// A structured-filter fragment over table alias `f`: SQL with anonymous
/// `?` placeholders plus the values they bind. Anonymous placeholders
/// compose by simple appending — the search cascade tacks fragments onto
/// every stage's WHERE clause with `AND (...)`.
#[derive(Debug, Clone)]
pub struct FilterFragment {
pub sql: String,
pub params: Vec<rusqlite::types::Value>,
}
/// Whether `key` is a recognized structured-filter property.
pub fn is_filter_key(key: &str) -> bool {
matches!(
key.to_ascii_lowercase().as_str(),
"type" | "modified" | "mtime" | "path" | "folder" | "includefolder" | "name"
| "filename" | "mime"
)
}
/// Translate one `key op value` filter into a [`FilterFragment`]. The
/// single source of filter semantics, shared by the legacy numbered
/// [`build`] path and the cascade's [`super::split`].
///
/// `glob` marks a value whose unquoted `*` should act as a wildcard — only
/// `name:`/`filename:` honor it; every other key treats the star literally.
/// The caller decides, because only the tokenizer knows whether the value
/// was quoted (quoted stars are always literal).
pub fn build_filter(
key: &str,
op: Op,
value: &str,
glob: bool,
) -> Result<FilterFragment, TranslateError> {
use rusqlite::types::Value;
let frag = |sql: &str, params: Vec<Value>| FilterFragment {
sql: sql.to_string(),
params,
};
let eq_like_only = |op: Op| -> Result<(), TranslateError> {
if op != Op::Contains && op != Op::Eq { if op != Op::Contains && op != Op::Eq {
return Err(TranslateError::UnsupportedOp { return Err(TranslateError::UnsupportedOp {
key: key.into(), key: key.into(),
op, op,
}); });
} }
let bits = FileType::from_name(value).bits() as i64; Ok(())
if bits == 0 { };
return Err(TranslateError::UnknownProperty(format!(
"type name '{}'", match key.to_ascii_lowercase().as_str() {
"type" => {
eq_like_only(op)?;
let bits = FileType::from_name(value).bits() as i64;
if bits == 0 {
return Err(TranslateError::UnknownProperty(format!(
"type name '{}'",
value
)));
}
Ok(frag("(f.type & ?) != 0", vec![Value::Integer(bits)]))
}
"modified" | "mtime" => {
let unix =
parse_date_to_unix(value).ok_or_else(|| TranslateError::BadDate(value.into()))?;
// `modified:=2024-01-01` matches the whole day, not the second.
if op == Op::Eq || op == Op::Contains {
return Ok(frag(
"(f.mtime >= ? AND f.mtime < ?)",
vec![Value::Integer(unix), Value::Integer(unix + 86_400)],
));
}
let sql_op = match op {
Op::Lt => "<",
Op::Le => "<=",
Op::Gt => ">",
Op::Ge => ">=",
Op::Contains | Op::Eq => unreachable!(),
};
Ok(frag(
&format!("f.mtime {} ?", sql_op),
vec![Value::Integer(unix)],
))
}
"path" | "folder" | "includefolder" => {
eq_like_only(op)?;
let base = normalize_folder_value(value);
if base.is_empty() {
// "everything". On Unix the old `parent = '/' OR parent LIKE
// '/%'` happened to match every absolute path; Windows has no
// single root, so say it directly rather than by accident.
return Ok(frag("1=1", Vec::new()));
}
// The `=` half needs the collation spelled out: `LIKE` folds ASCII
// case on its own, so without this the two halves of the same
// filter disagree about `C:\Users` versus `c:\users`.
Ok(frag(
&format!(
"(f.parent = ? COLLATE {} OR f.parent LIKE ? ESCAPE '\\')",
crate::platform::PATH_COLLATION
),
vec![
Value::Text(base.clone()),
Value::Text(like_subtree_pattern(&base)),
],
))
}
"name" | "filename" => {
if op != Op::Contains {
return Err(TranslateError::UnsupportedOp {
key: key.into(),
op,
});
}
// With `glob`, each `*` becomes an unescaped `%`; the pieces
// around it still get `%`/`_`/`\` escaped so user metacharacters
// stay literal either way.
let pattern = if glob && value.contains('*') {
value value
))); .split('*')
.map(escape_like)
.collect::<Vec<_>>()
.join("%")
} else {
escape_like(value)
};
Ok(frag(
"f.name LIKE ? ESCAPE '\\'",
vec![Value::Text(format!("%{}%", pattern))],
))
} }
self.all_params "mime" => {
.push(rusqlite::types::Value::Integer(bits)); eq_like_only(op)?;
Ok(format!("(f.type & ?{}) != 0", self.param_placeholder_idx())) Ok(frag("f.mime = ?", vec![Value::Text(value.into())]))
}
fn prop_mtime(&mut self, op: Op, value: &str, key: &str) -> Result<String, TranslateError> {
let unix = parse_date_to_unix(value).ok_or_else(|| TranslateError::BadDate(value.into()))?;
let col = "f.mtime";
let sql_op = match op {
Op::Contains | Op::Eq => "=",
Op::Lt => "<",
Op::Le => "<=",
Op::Gt => ">",
Op::Ge => ">=",
};
// `modified:=2024-01-01` should match the whole day, not the second.
if op == Op::Eq || op == Op::Contains {
let start = unix;
let end = unix + 86_400;
self.all_params.push(rusqlite::types::Value::Integer(start));
let i = self.param_placeholder_idx();
self.all_params.push(rusqlite::types::Value::Integer(end));
let j = self.param_placeholder_idx();
return Ok(format!("({} >= ?{} AND {} < ?{})", col, i, col, j));
} }
self.all_params.push(rusqlite::types::Value::Integer(unix)); _ => Err(TranslateError::UnknownProperty(key.to_string())),
let i = self.param_placeholder_idx();
let _ = key;
Ok(format!("{} {} ?{}", col, sql_op, i))
} }
}
fn prop_path(&mut self, op: Op, value: &str, key: &str) -> Result<String, TranslateError> { /// Escape `%`, `_` and `\` for use inside a `LIKE ... ESCAPE '\'` pattern.
if op != Op::Contains && op != Op::Eq { pub fn escape_like(s: &str) -> String {
return Err(TranslateError::UnsupportedOp { let mut out = String::with_capacity(s.len());
key: key.into(), for c in s.chars() {
op, if matches!(c, '%' | '_' | '\\') {
}); out.push('\\');
} }
self.all_params out.push(c);
.push(rusqlite::types::Value::Text(value.into()));
let i = self.param_placeholder_idx();
self.all_params
.push(rusqlite::types::Value::Text(format!("{}/%", value.trim_end_matches('/'))));
let j = self.param_placeholder_idx();
Ok(format!("(f.parent = ?{} OR f.parent LIKE ?{})", i, j))
} }
out
}
fn prop_name(&mut self, op: Op, value: &str, key: &str) -> Result<String, TranslateError> { /// Tidy a user-supplied folder value into the spelling `files.parent` stores.
if op != Op::Contains { ///
return Err(TranslateError::UnsupportedOp { /// Trailing separators are how people naturally write directories, and either
key: key.into(), /// separator may show up on Windows. A bare drive (`C:`) is *not* a path — the
op, /// stored parent is `C:\` — so the separator goes back on.
}); fn normalize_folder_value(value: &str) -> String {
} let base = value.trim().trim_end_matches(['/', '\\']);
self.all_params if base.len() == 2 && base.ends_with(':') && base.starts_with(|c: char| c.is_ascii_alphabetic())
.push(rusqlite::types::Value::Text(format!("%{}%", value))); {
let i = self.param_placeholder_idx(); return format!("{}{}", base, std::path::MAIN_SEPARATOR);
Ok(format!("f.name LIKE ?{}", i))
} }
base.to_string()
}
fn prop_mime(&mut self, op: Op, value: &str, key: &str) -> Result<String, TranslateError> { /// A `LIKE ... ESCAPE '\'` pattern matching every path strictly beneath `dir`.
if op != Op::Contains && op != Op::Eq { ///
return Err(TranslateError::UnsupportedOp { /// The separator is escaped along with the base, because on Windows the
key: key.into(), /// separator *is* the escape character — a hand-written `format!("{}/%", dir)`
op, /// is wrong twice over there: wrong separator, and the one it emits would be
}); /// swallowed as an escape.
} ///
self.all_params /// SQLite's `patternCompare` takes the character after the escape literally
.push(rusqlite::types::Value::Text(value.into())); /// whatever it is, so a doubled `\` is well defined here; the folklore that an
let i = self.param_placeholder_idx(); /// escape must be followed by `%`, `_` or itself does not apply.
Ok(format!("f.mime = ?{}", i)) pub fn like_subtree_pattern(dir: &str) -> String {
} format!(
"{}{}%",
fn param_placeholder_idx(&self) -> usize { escape_like(dir.trim_end_matches(['/', '\\'])),
// params[0] is reserved for the FTS MATCH if one is built; structured escape_like(std::path::MAIN_SEPARATOR_STR)
// params start at index 2 in that case (1-based). We track it by )
// calling this *after* pushing the value; result is `len` so the SQL
// says `?<len>` which matches the 1-based positional binding rusqlite
// uses for `?N` placeholders. When an FTS match is prepended at
// `build`, each index shifts by 1 implicitly.
self.all_params.len()
}
} }
fn join_with(sep: &str, pieces: &[String]) -> String { fn join_with(sep: &str, pieces: &[String]) -> String {
@ -497,6 +596,101 @@ mod tests {
let q = build_q("path:/home/me/docs"); let q = build_q("path:/home/me/docs");
assert!(q.sql.contains("f.parent = ?")); assert!(q.sql.contains("f.parent = ?"));
assert!(q.sql.contains("f.parent LIKE ?")); assert!(q.sql.contains("f.parent LIKE ?"));
// The LIKE half must be escaped and declare its escape character;
// without the clause a Windows separator would be eaten as an escape.
assert!(q.sql.contains("ESCAPE '\\'"), "{}", q.sql);
}
/// The subtree pattern is the one place the separator and the LIKE escape
/// character collide (on Windows they are the same byte), so it is checked
/// against real SQLite rather than by string comparison.
#[test]
fn like_subtree_pattern_matches_only_the_subtree() {
use std::path::MAIN_SEPARATOR as SEP;
let conn = rusqlite::Connection::open_in_memory().unwrap();
conn.execute("CREATE TABLE files (parent TEXT NOT NULL)", [])
.unwrap();
let base = format!("{}a{}b", root_prefix(), SEP);
let rows = [
format!("{}{}sub", base, SEP), // inside
format!("{}{}sub{}deep", base, SEP, SEP), // deeper
base.clone(), // the folder itself
format!("{}a{}bc", root_prefix(), SEP), // prefix sibling: outside
format!("{}a", root_prefix()), // parent: outside
];
for r in &rows {
conn.execute("INSERT INTO files (parent) VALUES (?1)", [r])
.unwrap();
}
let matched: i64 = conn
.query_row(
"SELECT COUNT(*) FROM files WHERE parent LIKE ?1 ESCAPE '\\'",
[like_subtree_pattern(&base)],
|r| r.get(0),
)
.unwrap();
assert_eq!(matched, 2, "only the two rows strictly beneath {}", base);
}
#[test]
fn like_subtree_pattern_escapes_metacharacters() {
use std::path::MAIN_SEPARATOR as SEP;
let conn = rusqlite::Connection::open_in_memory().unwrap();
conn.execute("CREATE TABLE files (parent TEXT NOT NULL)", [])
.unwrap();
let base = format!("{}a_b", root_prefix());
for r in [
format!("{}{}inside", base, SEP), // real child
format!("{}axb{}bait", root_prefix(), SEP), // `_` must not glob to `x`
format!("{}100%_done{}x", root_prefix(), SEP),
] {
conn.execute("INSERT INTO files (parent) VALUES (?1)", [&r])
.unwrap();
}
let matched: i64 = conn
.query_row(
"SELECT COUNT(*) FROM files WHERE parent LIKE ?1 ESCAPE '\\'",
[like_subtree_pattern(&base)],
|r| r.get(0),
)
.unwrap();
assert_eq!(matched, 1, "`_` and `%` are literals, not wildcards");
}
#[test]
fn folder_value_normalization() {
use std::path::MAIN_SEPARATOR as SEP;
// Trailing separators of either flavour are stripped.
assert_eq!(normalize_folder_value("/home/me/"), "/home/me");
assert_eq!(normalize_folder_value(r"C:\Users\me\"), r"C:\Users\me");
// A bare drive is not a path; the stored parent is `C:\`.
assert_eq!(normalize_folder_value("C:"), format!("C:{}", SEP));
assert_eq!(normalize_folder_value(r"C:\"), format!("C:{}", SEP));
// Empty means "everywhere".
assert_eq!(normalize_folder_value("/"), "");
assert_eq!(normalize_folder_value(" "), "");
}
#[test]
fn empty_folder_value_matches_everything() {
let frag = build_filter("path", Op::Contains, "/", false).unwrap();
assert_eq!(frag.sql, "1=1");
assert!(frag.params.is_empty(), "no placeholders to renumber");
}
/// An absolute-path prefix for the running platform.
fn root_prefix() -> String {
if cfg!(windows) {
r"C:\".to_string()
} else {
"/".to_string()
}
} }
#[test] #[test]

View file

@ -0,0 +1,937 @@
//! The ranked search cascade.
//!
//! One term, four table scans, eleven ranks. Rank base = stage number, so
//! later stages only ever append to a rank-sorted result list:
//!
//! | rank | meaning | scan |
//! |-----:|----------------------------------|------|
//! | 1.x | exact filename, exact case | A |
//! | 2.x | exact filename, any case | A |
//! | 3.x | filename substring, exact case | A |
//! | 4.x | filename substring, any case | A |
//! | 5.x | full text occurrence, exact case | B |
//! | 6.x | full text occurrence, any case | B |
//! | 7.x | fuzzy filename | C |
//! | 8.x | fuzzy full text | D |
//! | 9.x | full path substring, exact case | A |
//! | 10.x | full path substring, any case | A |
//! | 11.x | fuzzy full path | C |
//!
//! Pass A is a single `files` scan (`LIKE`, the ASCII-nocase superset of
//! its ranks) classified per-row in Rust — no index needed, the substring
//! stage visits every row anyway. Because a path always ends in its own
//! name, `path LIKE` is a superset of `name LIKE`, so that one scan covers
//! the filename *and* the path tiers. Pass B is one FTS phrase MATCH
//! verified against the decompressed text. Passes C/D (opt-in) iterate the
//! whole table with a bitap matcher, C covering both the name and the path.
//!
//! Wildcard terms (`rep*rt`) rank through the same tiers, with 1/2 meaning
//! the whole name matches the pattern; they skip the fuzzy passes (bitap is
//! a literal matcher). A regex-only query (`regex:…` with no term) runs two
//! dedicated scans that reuse tiers 4 (name), 6 (content) and 10 (path), so
//! downstream stage handling is unchanged. When `regex:` accompanies a
//! term, it is an accept-predicate on every pass, not a rank source.
//!
//! The path tiers rank below everything else, so pass A and pass C buffer
//! them instead of emitting them — stages E and F flush those buffers at
//! the end, dropping files an earlier stage already emitted. Path matching
//! needs a term of at least three characters, the same floor pass B has.
//!
//! Full-text ranks order equal-based hits by occurrence count via a
//! decimal fraction: `base + (1000 - min(count, 1000)) / 1000` — more
//! occurrences sorts earlier, 1000+ occurrences adds zero. Fuzzy ranks add
//! `0.1 × edit_distance` instead.
//!
//! Every scan appends the caller's structured-filter SQL (anonymous
//! placeholders over alias `f`) and checks the generation counter as it
//! streams; a bumped generation aborts mid-statement.
use std::collections::HashSet;
use std::sync::atomic::{AtomicU64, Ordering};
use rusqlite::Connection;
use rusqlite::OptionalExtension;
use crate::config::IgnoreSet;
use crate::query::pattern::clamp_match_range;
use crate::query::split::CascadeQuery;
use crate::query::translator::{escape_like, quote_phrase};
use crate::snippet;
use super::fuzzy::{edit_budget, Bitap};
use super::{SearchHit, SearchOptions};
/// Cancellation is checked every this many scanned rows in row-cheap
/// passes; decompression-heavy passes check every row.
const CANCEL_CHECK_ROWS: usize = 256;
/// Snippet window budget. Generous on purpose: the GUI trims the cell
/// text down to its column width around the match, and the mouseover
/// shows the rest of this window as extended context.
const SNIPPET_WINDOW_CHARS: usize = 600;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Outcome {
pub total: usize,
pub limited: bool,
}
/// Run the cascade, streaming rank-ordered batches into `sink`.
/// `Ok(None)` means the search was cancelled (generation moved on) — the
/// caller sends no completion. SQL errors are returned as strings *unless*
/// the search was already cancelled (an interrupted statement is normal
/// cancellation, not an error).
pub fn run(
conn: &Connection,
query: &CascadeQuery,
options: &SearchOptions,
generation: u64,
latest_gen: &AtomicU64,
sink: &mut dyn FnMut(Vec<SearchHit>),
) -> Result<Option<Outcome>, String> {
if query.is_empty() {
return Ok(Some(Outcome {
total: 0,
limited: false,
}));
}
let ignore = IgnoreSet::compile(&options.session_ignores)
.map_err(|e| format!("session ignore filter: {}", e))?;
let mut cx = Cx {
conn,
query,
options,
generation,
latest_gen,
ignore,
emitted: HashSet::new(),
deferred_path: Deferred::default(),
deferred_fuzzy_path: Deferred::default(),
total: 0,
limited: false,
sink,
};
// With no term at all the regex drives its own scans; `Path` still
// flushes the deferred rank-10 buffer the name pass sets aside.
let passes: &[Pass] = if query.pattern.is_empty() {
&[Pass::RegexName, Pass::RegexContent, Pass::Path]
} else {
&[
Pass::Filename,
Pass::FullText,
Pass::FuzzyFilename,
Pass::FuzzyFullText,
Pass::Path,
Pass::FuzzyPath,
]
};
for pass in passes {
if cx.cancelled() {
return Ok(None);
}
if cx.remaining() == 0 {
cx.limited = true;
break;
}
let run_pass = match pass {
Pass::Filename => cx.pass_filename(),
Pass::FullText => cx.pass_fulltext(),
Pass::FuzzyFilename => cx.pass_fuzzy_filename(),
Pass::FuzzyFullText => cx.pass_fuzzy_fulltext(),
Pass::RegexName => cx.pass_regex_name(),
Pass::RegexContent => cx.pass_regex_content(),
Pass::Path => {
let d = std::mem::take(&mut cx.deferred_path);
cx.flush_deferred(d)
}
Pass::FuzzyPath => {
let d = std::mem::take(&mut cx.deferred_fuzzy_path);
cx.flush_deferred(d)
}
};
match run_pass {
Ok(true) => {}
Ok(false) => return Ok(None), // cancelled mid-pass
Err(e) => {
if cx.cancelled() {
return Ok(None); // interrupt() killed the statement
}
return Err(e);
}
}
}
Ok(Some(Outcome {
total: cx.total,
limited: cx.limited,
}))
}
enum Pass {
Filename,
FullText,
FuzzyFilename,
FuzzyFullText,
/// Regex-only: name hits at rank 4 now, path hits deferred to rank 10.
RegexName,
/// Regex-only: content hits at rank 6.
RegexContent,
/// Flush of the rank 910 hits pass A set aside.
Path,
/// Flush of the rank 11 hits pass C set aside.
FuzzyPath,
}
/// Occurrence-count fraction: more occurrences → smaller fraction → sorts
/// earlier within a rank base; 1000+ adds zero.
fn count_frac(count: usize) -> f64 {
(1000usize.saturating_sub(count.min(1000))) as f64 / 1000.0
}
/// The path tiers only make sense with enough term to be specific — same
/// floor the trigram full-text pass uses. Wildcards count only their
/// literal content (`a*b` is two characters of specificity, not three).
fn path_tiers_enabled(pattern: &crate::query::pattern::TermPattern) -> bool {
pattern.literal_char_count() >= 3
}
/// Hits collected by one scan but ranked below later scans, so held back
/// until every better stage has emitted.
#[derive(Default)]
struct Deferred {
hits: Vec<SearchHit>,
overflowed: bool,
}
struct Cx<'a> {
conn: &'a Connection,
query: &'a CascadeQuery,
options: &'a SearchOptions,
generation: u64,
latest_gen: &'a AtomicU64,
ignore: IgnoreSet,
emitted: HashSet<i64>,
/// Ranks 910, filled by pass A.
deferred_path: Deferred,
/// Rank 11, filled by pass C.
deferred_fuzzy_path: Deferred,
total: usize,
limited: bool,
sink: &'a mut dyn FnMut(Vec<SearchHit>),
}
impl<'a> Cx<'a> {
fn cancelled(&self) -> bool {
self.generation != self.latest_gen.load(Ordering::Relaxed)
}
fn remaining(&self) -> usize {
self.options.limit.saturating_sub(self.total)
}
/// Buffer cap for scan passes: enough headroom that sorting keeps the
/// best candidates, without unbounded growth on huge hit sets.
fn buffer_cap(&self) -> usize {
4096.max(2 * self.remaining())
}
fn params_with_filters(
&self,
leading: Vec<rusqlite::types::Value>,
) -> Vec<rusqlite::types::Value> {
let mut p = leading;
p.extend(self.query.filter_params.iter().cloned());
p
}
/// Skip rows already emitted at a better rank or hidden by session
/// ignore chips.
fn skip(&self, file_id: i64, path: &str) -> bool {
self.emitted.contains(&file_id) || self.ignore.matches_path(std::path::Path::new(path))
}
/// The `regex:` accept-predicate applied to every candidate row when a
/// regex accompanies a term. The path contains the name, so one path
/// check covers both; content is fetched (and decompressed) only for
/// rows whose path missed — bounded by the pass's hit count, not its
/// scan count. Pass `text` when the pass already has the content.
fn regex_accepts(
&self,
file_id: i64,
path: &str,
text: Option<&str>,
) -> Result<bool, String> {
let Some(re) = &self.query.regex else {
return Ok(true);
};
if re.is_match(path) {
return Ok(true);
}
if let Some(text) = text {
return Ok(re.is_match(text));
}
let blob: Option<Vec<u8>> = self
.conn
.query_row(
"SELECT text_zstd FROM documents_text WHERE file_id = ?1",
[file_id],
|r| r.get(0),
)
.optional()
.map_err(|e| e.to_string())?;
let Some(raw) = blob.and_then(|b| zstd::decode_all(b.as_slice()).ok()) else {
return Ok(false);
};
Ok(re.is_match(&String::from_utf8_lossy(&raw)))
}
/// Sort a finished pass buffer, truncate to what's left of the display
/// limit, and stream it out in `options.batch`-sized events.
fn flush_pass(&mut self, mut buf: Vec<SearchHit>, overflowed: bool) {
buf.sort_by(|a, b| {
a.rank
.partial_cmp(&b.rank)
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| a.name.cmp(&b.name))
.then_with(|| a.path.cmp(&b.path))
});
let room = self.remaining();
if buf.len() > room {
buf.truncate(room);
self.limited = true;
}
if overflowed {
self.limited = true;
}
self.total += buf.len();
for hit in &buf {
self.emitted.insert(hit.file_id);
}
let batch = self.options.batch.max(1);
let mut buf = buf.into_iter().peekable();
while buf.peek().is_some() {
// A cancelled search stops emitting immediately — the newer
// generation owns the UI.
if self.cancelled() {
return;
}
let chunk: Vec<SearchHit> = buf.by_ref().take(batch).collect();
(self.sink)(chunk);
}
}
/// Emit a buffer held back from an earlier scan. Anything a better
/// stage already emitted drops out here — `emitted` was still empty (or
/// smaller) when these hits were collected.
fn flush_deferred(&mut self, mut deferred: Deferred) -> Result<bool, String> {
deferred.hits.retain(|h| !self.emitted.contains(&h.file_id));
self.flush_pass(deferred.hits, deferred.overflowed);
Ok(true)
}
/// Keep a scan buffer bounded: sort + cut back to the display-limit
/// room once it doubles past it. Returns whether anything was dropped.
fn enforce_cap(&self, buf: &mut Vec<SearchHit>) -> bool {
if buf.len() <= self.buffer_cap() {
return false;
}
buf.sort_by(|a, b| {
a.rank
.partial_cmp(&b.rank)
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| a.name.cmp(&b.name))
.then_with(|| a.path.cmp(&b.path))
});
buf.truncate(self.remaining());
true
}
/// Pass A — ranks 14 now, ranks 910 deferred, from one `files` scan.
/// Returns Ok(false) on cancellation.
fn pass_filename(&mut self) -> Result<bool, String> {
let query = self.query;
let pattern = &query.pattern;
let with_paths = path_tiers_enabled(pattern);
// A path always ends in its own name, so `path LIKE` is the
// superset that feeds both the name and the path tiers.
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f \
WHERE {} LIKE ? ESCAPE '\\'{}",
if with_paths { "f.path" } else { "f.name" },
query.filter_sql
);
// Wildcard patterns turn each star into an unescaped `%`; the
// substring wrap absorbs leading/trailing stars. User `%`/`_`
// remain escaped literals either way.
let like = pattern
.segments()
.iter()
.map(|s| escape_like(s))
.collect::<Vec<_>>()
.join("%");
let params = self.params_with_filters(vec![rusqlite::types::Value::Text(format!(
"%{}%",
like
))]);
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let mut buf: Vec<SearchHit> = Vec::new();
let mut path_buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
let mut path_overflowed = false;
let mut scanned = 0usize;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % CANCEL_CHECK_ROWS == 0 && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let name: String = row.get(1).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
// For a literal pattern these are exactly the old `==` /
// `eq_ignore_ascii_case` / `find` / folded-`find` operations
// (folding is byte-length preserving, so folded offsets are
// valid in the original). Wildcards run the same ladder through
// their compiled matcher — tiers 1/2 mean "whole name matches
// the pattern", which is what `*.txt` should do.
let (rank, match_range) = if pattern.whole_match(&name, false) {
(1.0, (0, name.len()))
} else if pattern.whole_match(&name, true) {
(2.0, (0, name.len()))
} else if let Some(r) = pattern.find_first(&name, false) {
(3.0, (r.start, r.end))
} else if let Some(r) = pattern.find_first(&name, true) {
(4.0, (r.start, r.end))
} else if !with_paths {
continue;
} else if let Some(r) = pattern.find_first(&path, false) {
(9.0, (r.start, r.end))
} else if let Some(r) = pattern.find_first(&path, true) {
(10.0, (r.start, r.end))
} else {
// LIKE folds ASCII case only; a row that matched it but
// neither field is a non-ASCII near-miss. Drop it.
continue;
};
if !self.regex_accepts(file_id, &path, None)? {
continue;
}
let is_path_tier = rank >= 9.0;
// The "snippet" of a name or path hit is that field itself with
// the matched span marked — the GUI renders it as [the field].
let snip = snippet::Snippet {
ranges: vec![match_range],
window: if is_path_tier { path.clone() } else { name.clone() },
truncated_start: false,
truncated_end: false,
};
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank,
stage: rank as u8,
snippet: Some(snip),
};
if is_path_tier {
path_buf.push(hit);
path_overflowed |= self.enforce_cap(&mut path_buf);
} else {
buf.push(hit);
overflowed |= self.enforce_cap(&mut buf);
}
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.deferred_path = Deferred {
hits: path_buf,
overflowed: path_overflowed,
};
self.flush_pass(buf, overflowed);
Ok(true)
}
/// Pass B — ranks 56 from one FTS MATCH, verified/counted in the
/// decompressed text.
fn pass_fulltext(&mut self) -> Result<bool, String> {
let query = self.query;
let pattern = &query.pattern;
if pattern.literal_char_count() < 3 {
// Below the trigram floor the MATCH can't return anything.
return Ok(true);
}
// Column filter goes inside the MATCH expression (`text: "..."`)
// so only document bodies match — filenames get ranks 1-4 from the
// filename pass instead. A literal term is one quoted phrase; a
// wildcard term narrows with an AND of its trigram-sized segments.
// With no segment of 3+ chars (`ab*cd`) FTS can't narrow at all, so
// fall back to scanning every stored document — every row is
// pattern-verified either way.
let match_expr: Option<String> = match pattern.literal() {
Some(term) => Some(format!("text: {}", quote_phrase(term))),
None => {
let usable: Vec<String> = pattern
.segments()
.iter()
.filter(|s| s.chars().count() >= 3)
.map(|s| format!("text: {}", quote_phrase(s)))
.collect();
if usable.is_empty() {
None
} else {
Some(usable.join(" AND "))
}
}
};
let narrowed = match_expr.is_some();
let (sql, params) = match match_expr {
Some(expr) => (
format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
FROM searchabletext \
JOIN files f ON f.id = searchabletext.rowid \
LEFT JOIN documents_text dt ON dt.file_id = f.id \
WHERE searchabletext MATCH ?{}",
query.filter_sql
),
self.params_with_filters(vec![rusqlite::types::Value::Text(expr)]),
),
None => (
format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
FROM documents_text dt \
JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
query.filter_sql
),
self.params_with_filters(Vec::new()),
),
};
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let snippet_opts = snippet::Options { approx_chars: SNIPPET_WINDOW_CHARS };
let mut buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
// Decompression dominates: check every row.
if self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
let blob: Option<Vec<u8>> = row.get(5).map_err(|e| e.to_string())?;
let text = blob.and_then(|b| zstd::decode_all(b.as_slice()).ok()).map(
|raw| String::from_utf8_lossy(&raw).into_owned(),
);
let (rank, stage, snip) = match &text {
Some(text) => {
let (count, stage, ci) = {
let count_cs = pattern.count(text, false);
if count_cs > 0 {
(count_cs, 5, false)
} else {
let count_ci = pattern.count(text, true);
if count_ci > 0 {
(count_ci, 6, true)
} else {
// Folded/unordered FTS candidate: the
// pattern never occurs — drop it.
continue;
}
}
};
// Literal terms keep the richer multi-occurrence
// extract; a wildcard match marks its own first range.
let snip = match pattern.literal() {
Some(term) => Some(snippet::extract(text, &[term], &snippet_opts)),
None => pattern.find_first(text, ci).map(|r| {
let r = clamp_match_range(text, r, SNIPPET_WINDOW_CHARS);
snippet::window_around(text, (r.start, r.end), &snippet_opts)
}),
};
(stage as f64 + count_frac(count), stage as u8, snip)
}
// No stored text (store_text_for_snippets = false or empty
// body): can't case-verify or count. On the FTS-narrowed
// path accept at the bottom of rank 6 as count-unknown (for
// wildcards the AND-of-segments guarantee is weaker —
// unordered co-occurrence — accepted for recall). On the
// full-scan fallback there is no FTS evidence at all, so an
// unverifiable row is just skipped.
None => {
if !narrowed {
continue;
}
(6.0 + count_frac(1), 6, None)
}
};
if !self.regex_accepts(file_id, &path, text.as_deref())? {
continue;
}
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank,
stage,
snippet: snip,
});
overflowed |= self.enforce_cap(&mut buf);
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.flush_pass(buf, overflowed);
Ok(true)
}
/// Pass C — rank 7 now, rank 11 deferred: one bitap sweep over every
/// filename, falling back to the full path where the name misses.
fn pass_fuzzy_filename(&mut self) -> Result<bool, String> {
if !self.options.fuzzy {
return Ok(true);
}
// Bitap is a literal matcher; wildcard terms don't fuzz.
if self.query.pattern.is_wildcard() {
return Ok(true);
}
let folded_term = self.query.term.to_ascii_lowercase();
let Some(k) = edit_budget(folded_term.len(), self.options.fuzzy_max_edits) else {
return Ok(true);
};
let Some(bitap) = Bitap::new(folded_term.as_bytes(), k) else {
return Ok(true);
};
let with_paths = path_tiers_enabled(&self.query.pattern);
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
self.query.filter_sql
);
let params = self.params_with_filters(Vec::new());
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let mut buf: Vec<SearchHit> = Vec::new();
let mut path_buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
let mut path_overflowed = false;
let mut scanned = 0usize;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % 1024 == 0 && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let name: String = row.get(1).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
// The name is the better match when both fire, so it wins and
// only a name miss falls through to the path tier.
let folded_name = name.to_ascii_lowercase();
let (rank, field, folded_field) = match bitap.best_distance(folded_name.as_bytes()) {
Some(distance) => (7.0 + 0.1 * distance as f64, &name, folded_name),
None if with_paths => {
let folded_path = path.to_ascii_lowercase();
match bitap.best_distance(folded_path.as_bytes()) {
Some(distance) => (11.0 + 0.1 * distance as f64, &path, folded_path),
None => continue,
}
}
None => continue,
};
if !self.regex_accepts(file_id, &path, None)? {
continue;
}
// Mark the approximate matched span in the matched field for
// the GUI's [matched field] rendering. window_around clamps
// and aligns.
let snip = bitap.count_and_first(folded_field.as_bytes()).1.map(|range| {
snippet::window_around(
field,
range,
&snippet::Options {
approx_chars: field.len().saturating_mul(2).max(8),
},
)
});
let is_path_tier = rank >= 11.0;
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank,
stage: rank as u8,
snippet: snip,
};
if is_path_tier {
path_buf.push(hit);
path_overflowed |= self.enforce_cap(&mut path_buf);
} else {
buf.push(hit);
overflowed |= self.enforce_cap(&mut buf);
}
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.deferred_fuzzy_path = Deferred {
hits: path_buf,
overflowed: path_overflowed,
};
self.flush_pass(buf, overflowed);
Ok(true)
}
/// Pass D — rank 8, bitap over every stored document text.
fn pass_fuzzy_fulltext(&mut self) -> Result<bool, String> {
if !self.options.fuzzy {
return Ok(true);
}
// Bitap is a literal matcher; wildcard terms don't fuzz.
if self.query.pattern.is_wildcard() {
return Ok(true);
}
let folded_term = self.query.term.to_ascii_lowercase();
let Some(k) = edit_budget(folded_term.len(), self.options.fuzzy_max_edits) else {
return Ok(true);
};
let Some(bitap) = Bitap::new(folded_term.as_bytes(), k) else {
return Ok(true);
};
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
self.query.filter_sql
);
let params = self.params_with_filters(Vec::new());
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let snippet_opts = snippet::Options { approx_chars: SNIPPET_WINDOW_CHARS };
let mut buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
if self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
let blob: Option<Vec<u8>> = row.get(5).map_err(|e| e.to_string())?;
let Some(blob) = blob else {
continue;
};
let Ok(raw) = zstd::decode_all(blob.as_slice()) else {
continue;
};
let text = String::from_utf8_lossy(&raw).into_owned();
// ASCII folding is byte-length preserving, so ranges found in
// the folded buffer are valid in the original.
let folded = text.to_ascii_lowercase();
let (count, first) = bitap.count_and_first(folded.as_bytes());
if count == 0 {
continue;
}
if !self.regex_accepts(file_id, &path, Some(&text))? {
continue;
}
let snip = first.map(|range| snippet::window_around(&text, range, &snippet_opts));
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank: 8.0 + count_frac(count),
stage: 8,
snippet: snip,
});
overflowed |= self.enforce_cap(&mut buf);
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.flush_pass(buf, overflowed);
Ok(true)
}
/// Regex-only pass over `files`: the regex bypasses the FTS trigram
/// entirely and runs on every name, falling back to the full path.
/// Name hits reuse rank 4, path hits defer to rank 10, so the GUI's
/// stage-based rendering needs no new cases.
fn pass_regex_name(&mut self) -> Result<bool, String> {
let query = self.query;
let re = query.regex.as_ref().expect("regex-only pass list");
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime FROM files f WHERE 1=1{}",
query.filter_sql
);
let params = self.params_with_filters(Vec::new());
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let mut buf: Vec<SearchHit> = Vec::new();
let mut path_buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
let mut path_overflowed = false;
let mut scanned = 0usize;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
scanned += 1;
if scanned % 1024 == 0 && self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let name: String = row.get(1).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
// The name is the better hit; only a name miss falls through
// to the path tier — mirroring pass A.
let (rank, match_range, is_path_tier) = match re.find_first(&name) {
Some(r) => (4.0, (r.start, r.end), false),
None => match re.find_first(&path) {
Some(r) => (10.0, (r.start, r.end), true),
None => continue,
},
};
let snip = snippet::Snippet {
ranges: vec![match_range],
window: if is_path_tier { path.clone() } else { name.clone() },
truncated_start: false,
truncated_end: false,
};
let hit = SearchHit {
file_id,
name,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank,
stage: rank as u8,
snippet: Some(snip),
};
if is_path_tier {
path_buf.push(hit);
path_overflowed |= self.enforce_cap(&mut path_buf);
} else {
buf.push(hit);
overflowed |= self.enforce_cap(&mut buf);
}
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.deferred_path = Deferred {
hits: path_buf,
overflowed: path_overflowed,
};
self.flush_pass(buf, overflowed);
Ok(true)
}
/// Regex-only pass over every stored document text, reusing rank 6.
fn pass_regex_content(&mut self) -> Result<bool, String> {
let query = self.query;
let re = query.regex.as_ref().expect("regex-only pass list");
let sql = format!(
"SELECT f.id, f.name, f.path, f.size, f.mtime, dt.text_zstd \
FROM documents_text dt JOIN files f ON f.id = dt.file_id WHERE 1=1{}",
query.filter_sql
);
let params = self.params_with_filters(Vec::new());
let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
let mut rows = stmt
.query(rusqlite::params_from_iter(params))
.map_err(|e| e.to_string())?;
let snippet_opts = snippet::Options { approx_chars: SNIPPET_WINDOW_CHARS };
let mut buf: Vec<SearchHit> = Vec::new();
let mut overflowed = false;
while let Some(row) = rows.next().map_err(|e| e.to_string())? {
// Decompression dominates: check every row.
if self.cancelled() {
return Ok(false);
}
let file_id: i64 = row.get(0).map_err(|e| e.to_string())?;
let path: String = row.get(2).map_err(|e| e.to_string())?;
if self.skip(file_id, &path) {
continue;
}
let blob: Option<Vec<u8>> = row.get(5).map_err(|e| e.to_string())?;
let Some(raw) = blob.and_then(|b| zstd::decode_all(b.as_slice()).ok()) else {
continue;
};
let text = String::from_utf8_lossy(&raw).into_owned();
let count = re.count(&text);
if count == 0 {
continue;
}
// A greedy user regex can match megabytes; clamp the range
// before the snippet window is cut.
let snip = re.find_first(&text).map(|r| {
let r = clamp_match_range(&text, r, SNIPPET_WINDOW_CHARS);
snippet::window_around(&text, (r.start, r.end), &snippet_opts)
});
buf.push(SearchHit {
file_id,
name: row.get(1).map_err(|e| e.to_string())?,
path,
size: row.get::<_, i64>(3).map_err(|e| e.to_string())?.max(0) as u64,
mtime: row.get(4).map_err(|e| e.to_string())?,
rank: 6.0 + count_frac(count),
stage: 6,
snippet: snip,
});
overflowed |= self.enforce_cap(&mut buf);
}
drop(rows);
if self.cancelled() {
return Ok(false);
}
self.flush_pass(buf, overflowed);
Ok(true)
}
}

View file

@ -0,0 +1,175 @@
//! Duplicate-file listing, grouped by content hash. Backs the GUI's
//! Duplicates tab; standalone and synchronous — callers run it on their
//! own worker thread.
use rusqlite::params;
use crate::db;
#[derive(Debug, Clone, PartialEq)]
pub struct DuplicateGroup {
pub hash: Vec<u8>,
pub count: i64,
pub total_size: i64,
/// Bytes reclaimable by deduplicating: `size × (count - 1)` — the
/// group's sort key.
pub redundant_size: i64,
/// `(file_id, name, path, size, mtime)` per member, path-ordered.
pub members: Vec<(i64, String, String, u64, i64)>,
}
/// Page through hash groups having more than one member, ordered by
/// reclaimable bytes (largest first). Rows with a NULL hash (never
/// hashed) and zero-size files (all trivially identical) are excluded.
pub fn find_duplicate_groups(
db_path: &str,
limit: u32,
offset: u32,
) -> Result<Vec<DuplicateGroup>, String> {
let conn = db::open_existing(db_path, false)?;
let mut groups: Vec<DuplicateGroup> = Vec::new();
{
// SUM(size) - MAX(size) == size × (count - 1); members of a group
// share a size because the hash covers it.
let mut stmt = conn
.prepare(
"SELECT hash, COUNT(*) AS cnt, SUM(size), SUM(size) - MAX(size) AS redundant \
FROM files \
WHERE hash IS NOT NULL AND size > 0 \
GROUP BY hash HAVING cnt > 1 \
ORDER BY redundant DESC, hash \
LIMIT ?1 OFFSET ?2",
)
.map_err(|e| e.to_string())?;
let rows = stmt
.query_map(params![limit, offset], |r| {
Ok(DuplicateGroup {
hash: r.get(0)?,
count: r.get(1)?,
total_size: r.get(2)?,
redundant_size: r.get(3)?,
members: Vec::new(),
})
})
.map_err(|e| e.to_string())?;
for g in rows {
groups.push(g.map_err(|e| e.to_string())?);
}
}
let mut member_stmt = conn
.prepare(
"SELECT id, name, path, size, mtime FROM files WHERE hash = ?1 ORDER BY path",
)
.map_err(|e| e.to_string())?;
for group in &mut groups {
let rows = member_stmt
.query_map(params![group.hash], |r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, i64>(3)?.max(0) as u64,
r.get::<_, i64>(4)?,
))
})
.map_err(|e| e.to_string())?;
for m in rows {
group.members.push(m.map_err(|e| e.to_string())?);
}
}
Ok(groups)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::db::open_or_recreate;
use crate::db::repo::{insert_file, NewFile};
use crate::mime::FileType;
fn seed_db() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-dups-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
let tx = conn.transaction().unwrap();
let add = |name: &str, path: &str, size: u64, hash: Option<&[u8]>| {
insert_file(
&tx,
&NewFile {
name,
path,
parent: "/d",
size,
mtime: 1_700_000_000,
inode: None,
device_id: None,
mime: None,
ftype: FileType::TEXT,
hash,
},
)
.unwrap()
.expect("unique path");
};
// Triple group of small files: redundant = 10 × 2 = 20.
add("a1.txt", "/d/a1.txt", 10, Some(b"AAA"));
add("a2.txt", "/d/a2.txt", 10, Some(b"AAA"));
add("a3.txt", "/d/a3.txt", 10, Some(b"AAA"));
// Pair of large files: redundant = 100 × 1 = 100 — sorts first
// despite the smaller member count.
add("b1.txt", "/d/b1.txt", 100, Some(b"BBB"));
add("b2.txt", "/d/b2.txt", 100, Some(b"BBB"));
// Singletons and NULL hashes never appear.
add("c.txt", "/d/c.txt", 30, Some(b"CCC"));
add("n1.txt", "/d/n1.txt", 40, None);
add("n2.txt", "/d/n2.txt", 40, None);
// Zero-size files are trivially identical — excluded outright.
add("z1.txt", "/d/z1.txt", 0, Some(b"ZZZ"));
add("z2.txt", "/d/z2.txt", 0, Some(b"ZZZ"));
tx.commit().unwrap();
drop(conn);
p
}
#[test]
fn groups_ordered_by_redundant_size_zero_size_excluded() {
let p = seed_db();
let groups = find_duplicate_groups(p.to_str().unwrap(), 10, 0).unwrap();
assert_eq!(
groups.len(),
2,
"singletons, NULL hashes, and zero-size groups excluded"
);
// Reclaimable bytes beat member count for ordering.
assert_eq!(groups[0].count, 2);
assert_eq!(groups[0].total_size, 200);
assert_eq!(groups[0].redundant_size, 100);
assert_eq!(groups[1].count, 3);
assert_eq!(groups[1].redundant_size, 20);
assert_eq!(groups[1].members.len(), 3);
assert_eq!(groups[1].members[0].1, "a1.txt", "members path-ordered");
std::fs::remove_file(&p).ok();
}
#[test]
fn pagination() {
let p = seed_db();
let page1 = find_duplicate_groups(p.to_str().unwrap(), 1, 0).unwrap();
let page2 = find_duplicate_groups(p.to_str().unwrap(), 1, 1).unwrap();
assert_eq!(page1.len(), 1);
assert_eq!(page2.len(), 1);
assert_ne!(page1[0].hash, page2[0].hash);
let page3 = find_duplicate_groups(p.to_str().unwrap(), 1, 2).unwrap();
assert!(page3.is_empty());
std::fs::remove_file(&p).ok();
}
}

View file

@ -0,0 +1,330 @@
//! Approximate substring matching for the fuzzy cascade stages.
//!
//! Bitap (shift-and with errors, WuManber): finds occurrences of a
//! pattern *within* a haystack with at most `k` Levenshtein edits
//! (insertion / deletion / substitution). Substring semantics are the
//! right fit for search-as-you-type — users type fragments, not whole
//! filenames — and the u64 bit-parallel update costs O(k) word ops per
//! haystack byte with zero allocations.
//!
//! Callers fold both sides to ASCII lowercase first (the pipeline-wide
//! convention). Patterns are limited to 64 bytes by the machine word; the
//! cascade skips fuzzy stages for longer terms.
pub struct Bitap {
/// `masks[c]` has bit `i` set iff `pattern[i] == c`.
masks: [u64; 256],
/// Pattern length in bytes (1..=64).
len: usize,
/// Maximum edit distance.
k: usize,
}
impl Bitap {
/// `None` when the pattern is empty, longer than 64 bytes, or the edit
/// budget reaches the word size (`initial_registers` shifts by `k`).
pub fn new(pattern: &[u8], k: usize) -> Option<Bitap> {
if pattern.is_empty() || pattern.len() > 64 || k >= 64 {
return None;
}
let mut masks = [0u64; 256];
for (i, &b) in pattern.iter().enumerate() {
masks[b as usize] |= 1u64 << i;
}
Some(Bitap {
masks,
len: pattern.len(),
k,
})
}
/// Fresh per-distance state registers. Bit `i` of `r[d]` set means "a
/// match of pattern[..=i] with ≤ d errors ends at the current text
/// position". With d errors the first d pattern bytes can be deleted
/// before any text is read, hence the pre-set low bits.
fn initial_registers(&self) -> Vec<u64> {
(0..=self.k)
.map(|d| if d == 0 { 0 } else { (1u64 << d) - 1 })
.collect()
}
/// Advance all registers by one haystack byte. Returns the smallest
/// error count d for which the full pattern just matched, if any.
#[inline]
fn step(&self, r: &mut [u64], byte: u8) -> Option<usize> {
let mask = self.masks[byte as usize];
let done = 1u64 << (self.len - 1);
let mut hit = None;
let mut prev_old = r[0]; // R_old[d-1] for the d-th iteration
// d = 0: exact prefix extension only.
r[0] = ((r[0] << 1) | 1) & mask;
if r[0] & done != 0 {
hit = Some(0);
}
for d in 1..=self.k {
let old = r[d];
r[d] = (((old << 1) | 1) & mask) // extend a ≤d-error state
| prev_old // insertion in text
| (prev_old << 1) // substitution
| ((r[d - 1] << 1) | 1); // deletion (pattern byte skipped)
prev_old = old;
if hit.is_none() && r[d] & done != 0 {
hit = Some(d);
}
}
hit
}
/// Minimum edit distance (≤ k) of any occurrence of the pattern in
/// `hay`, or `None` if nothing matches within k edits.
pub fn best_distance(&self, hay: &[u8]) -> Option<usize> {
let mut r = self.initial_registers();
let mut best: Option<usize> = None;
for &b in hay {
if let Some(d) = self.step(&mut r, b) {
if d == 0 {
return Some(0);
}
if best.map_or(true, |cur| d < cur) {
best = Some(d);
}
}
}
best
}
/// Count non-overlapping occurrences (at ≤ k edits) and report the
/// first match's approximate byte range in `hay`. After each hit the
/// automaton resets, so an exact match followed by trailing bytes
/// counts once, and overlapping suffix matches don't inflate counts.
/// The reported range assumes pattern-length matches — edits can shift
/// the true start by up to k bytes, which is fine for snippet windows.
pub fn count_and_first(&self, hay: &[u8]) -> (usize, Option<(usize, usize)>) {
let mut r = self.initial_registers();
let mut count = 0usize;
let mut first: Option<(usize, usize)> = None;
for (i, &b) in hay.iter().enumerate() {
if self.step(&mut r, b).is_some() {
count += 1;
if first.is_none() {
let end = i + 1;
first = Some((end.saturating_sub(self.len), end));
}
for (d, reg) in r.iter_mut().enumerate() {
*reg = if d == 0 { 0 } else { (1u64 << d) - 1 };
}
}
}
(count, first)
}
}
/// The cascade's edit-distance budget for a folded term: one edit per
/// three characters, capped by `[search].fuzzy_max_edits`. Terms outside
/// 3..=64 bytes skip the fuzzy stages entirely (< 3 is noise, > 64 exceeds
/// the word size), and a cap of 0 disables them everywhere.
///
/// At the default cap of 2 this is the historic ladder: 35 bytes get one
/// edit, 664 get two.
pub fn edit_budget(term_len: usize, max_edits: usize) -> Option<usize> {
if !(3..=64).contains(&term_len) || max_edits == 0 {
return None;
}
Some((term_len / 3).min(max_edits))
}
#[cfg(test)]
mod tests {
use super::*;
fn best(pattern: &str, hay: &str, k: usize) -> Option<usize> {
Bitap::new(pattern.as_bytes(), k)
.unwrap()
.best_distance(hay.as_bytes())
}
#[test]
fn exact_substring_is_distance_zero() {
assert_eq!(best("hello", "say hello world", 2), Some(0));
assert_eq!(best("hello", "hello", 0), Some(0));
}
#[test]
fn single_edits_are_distance_one() {
assert_eq!(best("hello", "xx hxllo xx", 2), Some(1), "substitution");
assert_eq!(best("hello", "xx helo xx", 2), Some(1), "deletion");
assert_eq!(best("hello", "xx heXllo xx", 2), Some(1), "insertion");
}
#[test]
fn two_edits() {
assert_eq!(best("hello", "xx hxlo xx", 2), Some(2));
assert_eq!(best("hello", "xx ho xx", 2), None, "3 edits > k");
}
#[test]
fn no_match_within_budget() {
assert_eq!(best("hello", "completely different", 1), None);
assert_eq!(best("abc", "", 1), None);
}
#[test]
fn k_zero_is_exact_search() {
assert_eq!(best("abc", "xxabcxx", 0), Some(0));
assert_eq!(best("abc", "xxabxcx", 0), None);
}
#[test]
fn pattern_length_limits() {
assert!(Bitap::new(b"", 1).is_none());
assert!(Bitap::new(&[b'a'; 65], 1).is_none());
assert!(Bitap::new(&[b'a'; 64], 1).is_some());
}
#[test]
fn oversized_k_is_rejected_not_shifted() {
// `initial_registers` shifts by k; k >= 64 would overflow u64.
assert!(Bitap::new(b"abc", 64).is_none());
assert!(Bitap::new(b"abc", usize::MAX).is_none());
assert!(Bitap::new(b"abc", 63).is_some());
}
#[test]
fn count_non_overlapping() {
let b = Bitap::new(b"ab", 0).unwrap();
let (count, first) = b.count_and_first(b"ab ab ab");
assert_eq!(count, 3);
assert_eq!(first, Some((0, 2)));
// "aaaa" contains "aaa" once non-overlapping.
let b = Bitap::new(b"aaa", 0).unwrap();
let (count, _) = b.count_and_first(b"aaaa");
assert_eq!(count, 1);
}
#[test]
fn count_fuzzy_and_range_sane() {
let b = Bitap::new(b"hello", 1).unwrap();
let hay = b"say helo and hxllo again";
let (count, first) = b.count_and_first(hay);
assert_eq!(count, 2);
let (s, e) = first.unwrap();
assert!(s < e && e <= hay.len());
let window = &hay[s..e];
assert!(
std::str::from_utf8(window).unwrap().contains("hel"),
"first range should cover the first hit, got {:?}",
std::str::from_utf8(window)
);
}
#[test]
fn edit_budget_default_cap_is_the_historic_ladder() {
assert_eq!(edit_budget(0, 2), None);
assert_eq!(edit_budget(2, 2), None);
assert_eq!(edit_budget(3, 2), Some(1));
assert_eq!(edit_budget(5, 2), Some(1));
assert_eq!(edit_budget(6, 2), Some(2));
assert_eq!(edit_budget(64, 2), Some(2));
assert_eq!(edit_budget(65, 2), None);
assert_eq!(edit_budget(usize::MAX, 2), None);
}
#[test]
fn edit_budget_scales_with_length_up_to_the_cap() {
assert_eq!(edit_budget(3, 4), Some(1));
assert_eq!(edit_budget(6, 4), Some(2));
assert_eq!(edit_budget(9, 4), Some(3));
assert_eq!(edit_budget(12, 4), Some(4));
assert_eq!(edit_budget(64, 4), Some(4), "cap wins over length");
}
#[test]
fn edit_budget_cap_of_one_stays_strict() {
for len in 3..=64 {
assert_eq!(edit_budget(len, 1), Some(1));
}
}
#[test]
fn edit_budget_zero_disables_fuzzy() {
for len in 0..=70 {
assert_eq!(edit_budget(len, 0), None);
}
}
/// Even a hostile config value can't produce a k the bitap rejects:
/// the length ladder caps it at 21 for the longest legal term.
#[test]
fn edit_budget_stays_within_the_bitap_word_size() {
for len in 3..=64 {
let k = edit_budget(len, usize::MAX).unwrap();
assert!(k <= 21, "len={} gave k={}", len, k);
assert!(Bitap::new(&vec![b'a'; len], k).is_some());
}
}
/// Brute-force oracle: minimum Levenshtein distance between `pattern`
/// and any substring of `hay`, capped at k.
fn oracle(pattern: &[u8], hay: &[u8], k: usize) -> Option<usize> {
// An occurrence must end at some text position; empty text has
// none. Without this, k >= pattern-length "matches" empty text by
// deleting every pattern byte — a degenerate non-occurrence the
// automaton rightly never reports. (Production keeps k < len via
// the len/3 budget, so only the oracle ever saw this edge.)
if hay.is_empty() {
return None;
}
// Standard DP where row 0 is all zeros (match can start anywhere).
let m = pattern.len();
let mut prev: Vec<usize> = vec![0; hay.len() + 1];
let mut cur = vec![0; hay.len() + 1];
let mut best = usize::MAX;
// dp[i][j] = min edits to match pattern[..i] ending at hay[..j]
for i in 1..=m {
cur[0] = i;
for j in 1..=hay.len() {
let cost = if pattern[i - 1] == hay[j - 1] { 0 } else { 1 };
cur[j] = (prev[j - 1] + cost).min(prev[j] + 1).min(cur[j - 1] + 1);
}
std::mem::swap(&mut prev, &mut cur);
}
for j in 0..=hay.len() {
best = best.min(prev[j]);
}
if best <= k {
Some(best)
} else {
None
}
}
#[test]
fn matches_brute_force_oracle() {
// Deterministic LCG so the test is reproducible.
let mut seed: u64 = 0x2545F4914F6CDD1D;
let mut rng = move || {
seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
(seed >> 33) as usize
};
let alphabet = b"abcx";
for _ in 0..500 {
let plen = 3 + rng() % 6;
let hlen = rng() % 20;
let pattern: Vec<u8> = (0..plen).map(|_| alphabet[rng() % 4]).collect();
let hay: Vec<u8> = (0..hlen).map(|_| alphabet[rng() % 4]).collect();
for k in 0..=4 {
let got = Bitap::new(&pattern, k).unwrap().best_distance(&hay);
let want = oracle(&pattern, &hay, k);
assert_eq!(
got,
want,
"pattern={:?} hay={:?} k={}",
std::str::from_utf8(&pattern),
std::str::from_utf8(&hay),
k
);
}
}
}
}

View file

@ -0,0 +1,307 @@
//! Interruptible, streaming search service.
//!
//! One dedicated worker thread owns the cascade. The GUI (or any caller)
//! sends queries via [`SearchService::search`]; results stream back over
//! an mpsc receiver as [`SearchUpdate`] events tagged with a generation
//! number. Starting a new search bumps the generation and interrupts the
//! in-flight SQLite statement, so a keystroke never waits on the previous
//! query.
//!
//! Cancellation is two-layer:
//! - **cooperative** — the cascade compares its generation against the
//! latest every few hundred rows and stops silently when stale;
//! - **interrupt** — [`rusqlite::InterruptHandle::interrupt`] kills the
//! statement currently executing (covering the "no rows produced yet"
//! phases like FTS candidate gathering). An interrupted stale search is
//! normal cancellation, not an error.
//!
//! Consumers that want a plain blocking search (the CLI mode) skip the
//! service entirely and call [`cascade::run`] with a collecting sink.
pub mod cascade;
pub mod duplicates;
pub mod fuzzy;
use std::path::PathBuf;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{mpsc, Arc, Mutex};
use std::thread::JoinHandle;
use crate::db;
use crate::query::split::split_for_cascade;
use crate::snippet::Snippet;
pub use cascade::Outcome;
pub use duplicates::{find_duplicate_groups, DuplicateGroup};
/// One search result. `rank` is the sort key (lower = better): integer
/// part = cascade stage (111), fraction = occurrence-count or
/// edit-distance tiebreak. Batches arrive already rank-ordered and later
/// batches only append, so a rank-sorted view never reshuffles.
#[derive(Debug, Clone)]
pub struct SearchHit {
pub file_id: i64,
pub name: String,
pub path: String,
pub size: u64,
pub mtime: i64,
pub rank: f64,
pub stage: u8,
/// The matched span in context: the filename for name stages, the full
/// path for path stages, a window of the body for full-text stages
/// (absent there when document text isn't stored).
pub snippet: Option<Snippet>,
}
#[derive(Debug, Clone)]
pub enum SearchUpdate {
Started { generation: u64 },
Hits { generation: u64, hits: Vec<SearchHit> },
Completed { generation: u64, total: usize, limited: bool },
Error { generation: u64, message: String },
}
impl SearchUpdate {
pub fn generation(&self) -> u64 {
match self {
SearchUpdate::Started { generation }
| SearchUpdate::Hits { generation, .. }
| SearchUpdate::Completed { generation, .. }
| SearchUpdate::Error { generation, .. } => *generation,
}
}
}
#[derive(Debug, Clone)]
pub struct SearchOptions {
/// Enable the fuzzy stages (ranks 7, 8 and 11).
pub fuzzy: bool,
/// Ceiling on the fuzzy edit budget (`[search].fuzzy_max_edits`); see
/// [`fuzzy::edit_budget`]. 0 disables the fuzzy stages.
pub fuzzy_max_edits: usize,
/// Hard cap on total hits per search (`[search].display_limit`).
pub limit: usize,
/// Streaming batch size (`[search].results_per_page`).
pub batch: usize,
/// Session-scoped ignore patterns (GUI chips), same glob semantics as
/// the config's `ignore_patterns`. Applied before the display cap.
pub session_ignores: Vec<String>,
}
impl Default for SearchOptions {
fn default() -> Self {
SearchOptions {
fuzzy: false,
fuzzy_max_edits: 2,
limit: 1000,
batch: 100,
session_ignores: Vec::new(),
}
}
}
struct SearchRequest {
generation: u64,
input: String,
options: SearchOptions,
}
pub struct SearchService {
req_tx: mpsc::Sender<SearchRequest>,
latest_gen: Arc<AtomicU64>,
interrupt: Arc<Mutex<Option<rusqlite::InterruptHandle>>>,
db_path: Arc<Mutex<PathBuf>>,
handle: Option<JoinHandle<()>>,
}
impl SearchService {
/// Spawn the worker. `notify` is invoked after every update event so
/// an egui frontend can `request_repaint` (pass a no-op for headless
/// use). Returns the service handle plus the update receiver, which
/// the caller drains non-blockingly.
pub fn new(
db_path: PathBuf,
notify: Arc<dyn Fn() + Send + Sync>,
) -> (SearchService, mpsc::Receiver<SearchUpdate>) {
let (req_tx, req_rx) = mpsc::channel::<SearchRequest>();
let (update_tx, update_rx) = mpsc::channel::<SearchUpdate>();
let latest_gen = Arc::new(AtomicU64::new(0));
let interrupt = Arc::new(Mutex::new(None));
let db_path = Arc::new(Mutex::new(db_path));
let worker = Worker {
req_rx,
update_tx,
notify,
latest_gen: latest_gen.clone(),
interrupt: interrupt.clone(),
db_path: db_path.clone(),
};
let handle = std::thread::Builder::new()
.name("qs-search".into())
.spawn(move || worker.run())
.expect("spawn search worker");
(
SearchService {
req_tx,
latest_gen,
interrupt,
db_path,
handle: Some(handle),
},
update_rx,
)
}
/// Start a new search, cancelling any in-flight one. Returns the
/// generation whose events to keep.
pub fn search(&self, input: &str, options: SearchOptions) -> u64 {
let generation = self.latest_gen.fetch_add(1, Ordering::SeqCst) + 1;
let _ = self.req_tx.send(SearchRequest {
generation,
input: input.to_string(),
options,
});
// The new request can't be running yet (the worker hasn't dequeued
// it), so this only ever kills a stale generation's statement.
self.interrupt_current();
generation
}
/// Cancel without starting anything new.
pub fn cancel(&self) {
self.latest_gen.fetch_add(1, Ordering::SeqCst);
self.interrupt_current();
}
/// Point subsequent searches at a different index file.
pub fn set_db_path(&self, path: PathBuf) {
*self.db_path.lock().unwrap() = path;
self.cancel();
}
fn interrupt_current(&self) {
if let Ok(guard) = self.interrupt.lock() {
if let Some(handle) = guard.as_ref() {
handle.interrupt();
}
}
}
/// Cancel, close the request channel, and join the worker.
pub fn shutdown(self) {
self.cancel();
let SearchService { req_tx, handle, .. } = self;
drop(req_tx);
if let Some(handle) = handle {
let _ = handle.join();
}
}
}
/// Map SQLite-level errors to the tagged strings frontends key off.
/// `DATABASE_CORRUPTED:` drives the GUI's recovery dialog.
pub fn classify_sql_err(error_msg: &str) -> String {
if error_msg.contains("malformed")
|| error_msg.contains("corrupt")
|| error_msg.contains("database disk image is malformed")
{
format!("DATABASE_CORRUPTED: {}", error_msg)
} else if error_msg.contains("fts5: syntax error") {
"Search syntax error: the search term contains characters that cannot be processed."
.to_string()
} else {
format!("Search failed: {}", error_msg)
}
}
struct Worker {
req_rx: mpsc::Receiver<SearchRequest>,
update_tx: mpsc::Sender<SearchUpdate>,
notify: Arc<dyn Fn() + Send + Sync>,
latest_gen: Arc<AtomicU64>,
interrupt: Arc<Mutex<Option<rusqlite::InterruptHandle>>>,
db_path: Arc<Mutex<PathBuf>>,
}
impl Worker {
fn run(self) {
while let Ok(first) = self.req_rx.recv() {
// A fast typist queues several requests; only the newest one
// matters.
let mut req = first;
while let Ok(newer) = self.req_rx.try_recv() {
req = newer;
}
if req.generation != self.latest_gen.load(Ordering::SeqCst) {
continue;
}
self.handle(req);
}
}
fn send(&self, update: SearchUpdate) {
let _ = self.update_tx.send(update);
(self.notify)();
}
fn handle(&self, req: SearchRequest) {
let generation = req.generation;
self.send(SearchUpdate::Started { generation });
let split = match split_for_cascade(&req.input) {
Ok(s) => s,
Err(e) => {
self.send(SearchUpdate::Error {
generation,
message: e.to_string(),
});
return;
}
};
let db_path = self.db_path.lock().unwrap().clone();
// Per-request open: microseconds, and always sees a freshly
// rebuilt index file rather than pinning a deleted inode.
let conn = match db::open_existing(&db_path.to_string_lossy(), false) {
Ok(c) => c,
Err(e) => {
self.send(SearchUpdate::Error {
generation,
message: classify_sql_err(&e),
});
return;
}
};
*self.interrupt.lock().unwrap() = Some(conn.get_interrupt_handle());
let mut sink = |hits: Vec<SearchHit>| {
self.send(SearchUpdate::Hits { generation, hits });
};
let outcome = cascade::run(
&conn,
&split,
&req.options,
generation,
&self.latest_gen,
&mut sink,
);
*self.interrupt.lock().unwrap() = None;
match outcome {
Ok(Some(Outcome { total, limited })) => self.send(SearchUpdate::Completed {
generation,
total,
limited,
}),
// Cancelled — the newer generation owns the UI now.
Ok(None) => {}
Err(e) => self.send(SearchUpdate::Error {
generation,
message: classify_sql_err(&e),
}),
}
}
}

View file

@ -1,396 +0,0 @@
//! SQL builders for the GUI's three legacy search modes (full-text,
//! filename, duplicate). Lives in core so it has unit-test coverage; the
//! GUI just composes these into per-page queries.
//!
//! For the structured Baloo-subset query language (`type:`, `modified:`,
//! …), see [`crate::query`].
/// All inputs needed to run (and re-run) one of the three search modes.
/// Cached after a fresh search so paging buttons don't have to rebuild
/// from form state.
#[derive(Debug, Clone, PartialEq)]
pub struct SearchArgs {
pub search_type: String,
pub term: String,
pub fulltext_exact: bool,
pub fulltext_case_sensitive: bool,
}
/// SQL that counts every row matching `args`, ignoring pagination. Used to
/// drive the "page X of Y" UI. For very large FTS hit sets this can take a
/// noticeable fraction of the per-page query time, but it only runs on a
/// fresh search — page navigation reuses the cached total.
pub fn build_count(args: &SearchArgs) -> Result<String, String> {
match args.search_type.as_str() {
"fulltext" => {
let where_clause = build_fulltext_where(args)?;
Ok(format!(
"SELECT COUNT(*) FROM searchabletext AS st WHERE {}",
where_clause
))
}
"filename" => {
if args.term.trim().is_empty() {
return Err("Please enter a filename pattern".into());
}
Ok(format!(
"SELECT COUNT(*) FROM files WHERE name LIKE '%{}%'",
args.term.replace('\'', "''")
))
}
"duplicates" => Ok(
"SELECT COUNT(*) FROM (SELECT 1 FROM files WHERE hash IS NOT NULL \
GROUP BY hash HAVING count(*) > 1)"
.into(),
),
_ => Err("Unknown search type".into()),
}
}
/// SQL that returns one page of results. Columns emitted by the `fulltext`
/// branch are `(name, path, file_id, text_zstd)` — the snippet is rendered
/// in Rust from the zstd-compressed `documents_text` row (FTS5 is
/// contentless, so SQLite's `snippet()` doesn't work on it). The GUI
/// should use [`crate::indexing::IndexingService::execute_fulltext_search`]
/// which stitches the decompress + snippet step on top of this SQL.
pub fn build_select(args: &SearchArgs, limit: u32, offset: u32) -> Result<String, String> {
match args.search_type.as_str() {
"fulltext" => {
let where_clause = build_fulltext_where(args)?;
Ok(format!(
"SELECT f.name, f.path, f.id, dt.text_zstd \
FROM searchabletext AS st \
JOIN files f ON f.id = st.rowid \
LEFT JOIN documents_text dt ON dt.file_id = f.id \
WHERE {} ORDER BY rank LIMIT {} OFFSET {}",
where_clause, limit, offset
))
}
"filename" => {
if args.term.trim().is_empty() {
return Err("Please enter a filename pattern".into());
}
Ok(format!(
"SELECT name, path FROM files WHERE name LIKE '%{}%' ORDER BY name LIMIT {} OFFSET {}",
args.term.replace('\'', "''"),
limit,
offset
))
}
"duplicates" => Ok(format!(
"SELECT name, count(*) as cnt, path FROM files WHERE hash IS NOT NULL \
GROUP BY hash HAVING cnt > 1 ORDER BY cnt DESC LIMIT {} OFFSET {}",
limit, offset
)),
_ => Err("Unknown search type".into()),
}
}
/// Translate the user-typed term into the FTS5 `MATCH` expression and any
/// supplemental case-sensitivity filters. Shared by count and select so
/// pagination doesn't accidentally diverge from the totals.
fn build_fulltext_where(args: &SearchArgs) -> Result<String, String> {
let trimmed = args.term.trim();
if trimmed.is_empty() {
return Err("Please enter a search term".into());
}
// Strip FTS5 control characters that confuse the parser. Replace with
// spaces so word boundaries survive.
let sanitized: String = trimmed
.chars()
.map(|c| {
if matches!(
c,
':' | ';' | '(' | ')' | '[' | ']' | '{' | '}' | '^' | '~' | '"'
) {
' '
} else {
c
}
})
.collect();
let tokens: Vec<&str> = sanitized.split_whitespace().collect();
if tokens.is_empty() {
return Err("Please enter a valid search term".into());
}
// Trigram tokenizer needs each word to be at least 3 characters. Exact
// phrase mode skips this filter — a quoted phrase of short words still
// matches because the trigrams overlap the spaces between words.
let words: Vec<&str> = if args.fulltext_exact {
tokens
} else {
let filtered: Vec<&str> = tokens
.into_iter()
.filter(|w| w.chars().count() >= 3)
.collect();
if filtered.is_empty() {
return Err(
"Trigram index needs each word to be at least 3 characters unless you use exact phrase search."
.into(),
);
}
filtered
};
let sql_quote = |s: &str| s.replace('\'', "''");
let fts_match = if args.fulltext_exact {
let phrase = words.join(" ");
format!("\"{}\"", phrase.replace('"', "\"\""))
} else {
words.join(" AND ")
};
// Contentless FTS5 doesn't store column text, so case-sensitive
// filtering can't live in SQL anymore. It's re-applied in
// `IndexingService::execute_fulltext_search` by checking the
// decompressed body text for literal-case matches before returning
// the row. The MATCH itself stays case-insensitive (tokenizer folds),
// which is the correct candidate-set for a post-filter.
let where_clause = format!("st.text MATCH '{}'", sql_quote(&fts_match));
Ok(where_clause)
}
/// Pull out the raw words the user typed so the snippet renderer and the
/// case-sensitive post-filter can see the same tokens `build_fulltext_where`
/// fed into FTS5. Returns empty when the user's term is empty or contains
/// only too-short words under the non-exact path.
pub fn fulltext_terms(args: &SearchArgs) -> Vec<String> {
let trimmed = args.term.trim();
if trimmed.is_empty() {
return Vec::new();
}
let sanitized: String = trimmed
.chars()
.map(|c| {
if matches!(
c,
':' | ';' | '(' | ')' | '[' | ']' | '{' | '}' | '^' | '~' | '"'
) {
' '
} else {
c
}
})
.collect();
let tokens: Vec<String> = sanitized
.split_whitespace()
.map(|s| s.to_string())
.collect();
if tokens.is_empty() {
return Vec::new();
}
if args.fulltext_exact {
// One composite phrase. Snippet rendering wants to highlight the
// whole phrase contiguously; the renderer supports multiple terms
// already so we collapse to the joined form.
vec![tokens.join(" ")]
} else {
tokens
.into_iter()
.filter(|w| w.chars().count() >= 3)
.collect()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::db::{open_or_recreate, repo::{insert_file, set_content_done, NewFile}};
use crate::mime::FileType;
fn args(search_type: &str, term: &str) -> SearchArgs {
SearchArgs {
search_type: search_type.into(),
term: term.into(),
fulltext_exact: false,
fulltext_case_sensitive: false,
}
}
#[test]
fn fulltext_select_has_limit_and_offset() {
let sql = build_select(&args("fulltext", "hello world"), 50, 100).unwrap();
assert!(sql.contains("LIMIT 50"));
assert!(sql.contains("OFFSET 100"));
assert!(sql.contains("ORDER BY rank"));
// Snippet rendering moved to Rust; SQL returns the compressed blob
// so the post-processor can decompress + highlight.
assert!(sql.contains("dt.text_zstd"), "got {sql}");
}
#[test]
fn fulltext_case_sensitive_no_longer_in_sql() {
let mut a = args("fulltext", "Hello");
a.fulltext_case_sensitive = true;
let sql = build_select(&a, 50, 0).unwrap();
// Post-filter lives in Rust now — no instr() or st.text reference.
assert!(!sql.contains("instr("), "got {sql}");
}
#[test]
fn fulltext_terms_extract_non_exact() {
let a = args("fulltext", "the quick brown");
let t = fulltext_terms(&a);
// Short words like "the" are dropped (trigram min length 3 applies
// in non-exact mode — matches the SQL build rules).
assert!(t.iter().any(|s| s == "quick"));
assert!(t.iter().any(|s| s == "brown"));
}
#[test]
fn fulltext_terms_exact_mode_returns_joined_phrase() {
let mut a = args("fulltext", "hello world");
a.fulltext_exact = true;
let t = fulltext_terms(&a);
assert_eq!(t, vec!["hello world".to_string()]);
}
#[test]
fn fulltext_count_lacks_pagination_and_join() {
let sql = build_count(&args("fulltext", "hello world")).unwrap();
assert!(sql.starts_with("SELECT COUNT(*)"));
assert!(!sql.contains("LIMIT"));
assert!(!sql.contains("OFFSET"));
assert!(!sql.contains("JOIN files"));
}
#[test]
fn fulltext_short_words_filtered_unless_exact() {
let err = build_select(&args("fulltext", "a b"), 50, 0).unwrap_err();
assert!(err.contains("3 characters"));
let mut a = args("fulltext", "a b");
a.fulltext_exact = true;
assert!(build_select(&a, 50, 0).is_ok());
}
#[test]
fn fulltext_quotes_are_escaped() {
let sql = build_select(&args("fulltext", "it's working"), 50, 0).unwrap();
// SQL literals double single quotes.
assert!(sql.contains("it''s"));
}
#[test]
fn filename_select_has_limit_offset_and_order() {
let sql = build_select(&args("filename", "report"), 50, 0).unwrap();
assert!(sql.contains("LIMIT 50"));
assert!(sql.contains("OFFSET 0"));
assert!(sql.contains("ORDER BY name"));
assert!(sql.contains("name LIKE '%report%'"));
}
#[test]
fn filename_empty_term_errors() {
assert!(build_select(&args("filename", " "), 50, 0).is_err());
assert!(build_count(&args("filename", "")).is_err());
}
#[test]
fn duplicates_select_has_limit_offset() {
let sql = build_select(&args("duplicates", ""), 50, 100).unwrap();
assert!(sql.contains("LIMIT 50"));
assert!(sql.contains("OFFSET 100"));
assert!(sql.contains("GROUP BY hash"));
}
#[test]
fn unknown_search_type_errors() {
assert!(build_select(&args("nope", ""), 50, 0).is_err());
assert!(build_count(&args("nope", "")).is_err());
}
fn tmp_path() -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"qs-search-sql-{}-{}.sqlite",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
p
}
/// End-to-end: seed three rows, run count + paged select, verify
/// pagination boundaries actually behave on a real DB.
#[test]
fn end_to_end_pagination_smoke() {
let p = tmp_path();
let mut conn = open_or_recreate(p.to_str().unwrap(), "trigram").unwrap();
{
let tx = conn.transaction().unwrap();
for i in 0..7 {
let path = format!("/tmp/file_{}.txt", i);
let id = insert_file(
&tx,
&NewFile {
name: &format!("file_{}.txt", i),
path: &path,
parent: "/tmp",
size: 1,
mtime: 1,
inode: None,
device_id: None,
mime: Some("text/plain"),
ftype: FileType::TEXT,
hash: None,
},
)
.unwrap()
.expect("unique path");
set_content_done(&tx, id, &format!("file_{}.txt", i), "shared body content", &[], true).unwrap();
}
tx.commit().unwrap();
}
// Count: all 7 rows match "shared".
let count_sql = build_count(&args("fulltext", "shared body content")).unwrap();
let n: i64 = conn.query_row(&count_sql, [], |r| r.get(0)).unwrap();
assert_eq!(n, 7);
// Page 1, page_size 3 → 3 rows.
let sel1 = build_select(&args("fulltext", "shared body content"), 3, 0).unwrap();
let rows1: Vec<String> = conn
.prepare(&sel1)
.unwrap()
.query_map([], |r| r.get::<_, String>(1))
.unwrap()
.map(|r| r.unwrap())
.collect();
assert_eq!(rows1.len(), 3);
// Page 3, page_size 3 → 1 row (offset 6, 7 total).
let sel3 = build_select(&args("fulltext", "shared body content"), 3, 6).unwrap();
let rows3: Vec<String> = conn
.prepare(&sel3)
.unwrap()
.query_map([], |r| r.get::<_, String>(1))
.unwrap()
.map(|r| r.unwrap())
.collect();
assert_eq!(rows3.len(), 1);
// Filename pagination on the same fixture.
let fn_count = build_count(&args("filename", "file_")).unwrap();
let n2: i64 = conn.query_row(&fn_count, [], |r| r.get(0)).unwrap();
assert_eq!(n2, 7);
let fn_sel = build_select(&args("filename", "file_"), 5, 0).unwrap();
let rows: Vec<String> = conn
.prepare(&fn_sel)
.unwrap()
.query_map([], |r| r.get::<_, String>(1))
.unwrap()
.map(|r| r.unwrap())
.collect();
assert_eq!(rows.len(), 5);
drop(conn);
std::fs::remove_file(&p).ok();
}
}

View file

@ -1,26 +1,23 @@
//! Process-level shutdown helpers. //! Process-level shutdown helpers.
//! //!
//! Wires Ctrl-C (and on Unix, SIGTERM) to a graceful shutdown that flushes //! Wires Ctrl-C (and on Unix, SIGTERM) to a graceful shutdown that stops
//! the indexing DB and exits. Replaces the ad-hoc `ctrlc::set_handler` + //! the watcher, aborts any running index pass, flushes the WAL, and
//! `OnceLock` dance the GUI used to carry. Call //! exits. Call [`install_signal_handler`] once from a binary's `main`
//! [`install_signal_handler`] once from a binary's `main` with a cloned //! with a cloned [`IndexCoordinator`] handle.
//! [`IndexingService`] handle.
use std::sync::Arc; use std::sync::Arc;
use crate::indexing::IndexingService; use crate::coordinator::IndexCoordinator;
/// Install a Ctrl-C (and, where supported, SIGTERM) handler that calls /// Install a Ctrl-C (and, where supported, SIGTERM) handler that calls
/// [`IndexingService::graceful_shutdown`] and then exits with status 0. /// [`IndexCoordinator::shutdown`] and then exits with status 0.
/// ///
/// Returns an error only if a handler was already installed elsewhere in /// Returns an error only if a handler was already installed elsewhere in
/// this process (ctrlc::set_handler is one-shot). /// this process (ctrlc::set_handler is one-shot).
pub fn install_signal_handler(service: Arc<IndexingService>) -> Result<(), String> { pub fn install_signal_handler(coordinator: Arc<IndexCoordinator>) -> Result<(), String> {
ctrlc::set_handler(move || { ctrlc::set_handler(move || {
eprintln!("Received Ctrl-C, shutting down gracefully..."); crate::log_info!("Received Ctrl-C, shutting down gracefully...");
if let Err(e) = service.graceful_shutdown() { coordinator.shutdown();
eprintln!("Error during graceful shutdown: {}", e);
}
std::process::exit(0); std::process::exit(0);
}) })
.map_err(|e| format!("install signal handler: {}", e)) .map_err(|e| format!("install signal handler: {}", e))

View file

@ -1,60 +1,70 @@
//! Snippet / highlight rendering for search results. //! Snippet extraction for search results.
//! //!
//! We store extracted text in the `documents_text` sidecar (zstd-compressed) //! We store extracted text in the `documents_text` sidecar (zstd-compressed)
//! rather than in FTS5, so SQLite's built-in `snippet()` / `highlight()` //! rather than in FTS5, so SQLite's built-in `snippet()` / `highlight()`
//! auxiliary functions aren't available (contentless FTS5 doesn't support //! auxiliary functions aren't available (contentless FTS5 doesn't support
//! them). This module reproduces the parts we actually need in Rust: find //! them). This module reproduces the parts we actually need in Rust: find a
//! a window of context around the first match, bold every query-term //! window of context around the first match and report every query-term
//! occurrence inside that window, trim with ellipsis markers. //! occurrence inside that window.
//! //!
//! Matching is ASCII-case-insensitive on the *rendering* side. That aligns //! Output is *structural* — the window text plus byte ranges of the matches
//! with the search path which is already case-insensitive via the trigram //! within it — so any frontend can render highlights natively (egui builds
//! tokenizer; exact-case-only snippets aren't a feature users expect here. //! a `LayoutJob`, the CLI emits ANSI bold). Nothing here produces markup.
//! Unicode accent folding isn't applied at the rendering layer — a query
//! for `cafe` will still *find* a file containing `café` (because the FTS
//! tokenizer strips diacritics) but the snippet won't highlight the
//! accented occurrence. The surrounding text is still returned verbatim.
//! //!
//! The API is intentionally small: one `render` function plus an `Options` //! Matching is ASCII-case-insensitive. That aligns with the search path,
//! struct. Callers that want different pre/post tags, ellipsis, or window //! which is already case-insensitive via the trigram tokenizer; Unicode
//! size pass them in; there are sensible defaults for the GUI case. //! accent folding isn't applied at this layer — a query for `cafe` will
//! still *find* a file containing `café` (the FTS tokenizer strips
//! diacritics) but the snippet won't mark the accented occurrence. The
//! window text is returned verbatim either way.
/// Options controlling snippet rendering. The defaults mirror the old /// Options controlling snippet extraction.
/// `snippet(searchabletext, 1, '<b>', '</b>', '<b>...</b>', 64)` call that
/// the GUI used to run directly as SQL.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct Options<'a> { pub struct Options {
pub pre: &'a str, /// Approximate byte budget for the returned window. Matches expand the
pub post: &'a str, /// window if needed so a hit is never cut off; the budget is a soft
pub ellipsis: &'a str, /// target, not a hard cap.
/// Approximate character budget for the returned snippet. Matches
/// expand the window if needed to keep their tags on; the budget is a
/// soft target, not a hard cap.
pub approx_chars: usize, pub approx_chars: usize,
} }
impl<'a> Default for Options<'a> { impl Default for Options {
fn default() -> Self { fn default() -> Self {
Options { Options { approx_chars: 200 }
pre: "<b>", }
post: "</b>", }
ellipsis: "<b>...</b>",
approx_chars: 200, /// A context window from a document plus the match positions inside it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Snippet {
/// The excerpt, sliced verbatim from the source text on UTF-8 char
/// boundaries.
pub window: String,
/// Byte ranges *into `window`*, sorted, non-overlapping (overlapping
/// term hits are coalesced), always on char boundaries.
pub ranges: Vec<(usize, usize)>,
/// Content exists before/after the window — frontends render their own
/// ellipsis.
pub truncated_start: bool,
pub truncated_end: bool,
}
impl Snippet {
fn empty() -> Snippet {
Snippet {
window: String::new(),
ranges: Vec::new(),
truncated_start: false,
truncated_end: false,
} }
} }
} }
/// Render a snippet from `text` highlighting every occurrence of any term /// Extract a snippet from `text` marking every occurrence of any term in
/// in `terms`. Returns a string with `pre`/`post` wrapping each match, and /// `terms` (ASCII-case-insensitive). With no terms or no matches, returns
/// `ellipsis` prepended/appended when the returned window doesn't reach /// the head of the text as the window with no ranges.
/// the text's edges. pub fn extract(text: &str, terms: &[&str], opts: &Options) -> Snippet {
///
/// If no term matches, returns the first `approx_chars` of `text` (char-
/// aligned), suffixed with `ellipsis` when truncated.
pub fn render(text: &str, terms: &[&str], opts: &Options<'_>) -> String {
// Short-circuit trivial inputs.
if text.is_empty() { if text.is_empty() {
return String::new(); return Snippet::empty();
} }
let effective_terms: Vec<&str> = terms let effective_terms: Vec<&str> = terms
.iter() .iter()
@ -62,32 +72,24 @@ pub fn render(text: &str, terms: &[&str], opts: &Options<'_>) -> String {
.filter(|t| !t.is_empty()) .filter(|t| !t.is_empty())
.collect(); .collect();
if effective_terms.is_empty() { if effective_terms.is_empty() {
return truncate_head(text, opts.approx_chars, opts.ellipsis); return head_window(text, opts.approx_chars);
} }
// Case-fold once; we do all positioning on the folded buffer and emit // Case-fold once; we do all positioning on the folded buffer and slice
// slices from the original. Both buffers have identical byte layout // from the original. Both buffers have identical byte layout because
// because `to_ascii_lowercase` is a byte-for-byte map that preserves // `to_ascii_lowercase` only touches ASCII letters.
// multi-byte UTF-8 sequences unchanged (it only touches ASCII letters).
let folded = text.to_ascii_lowercase(); let folded = text.to_ascii_lowercase();
let folded_bytes = folded.as_bytes(); let folded_bytes = folded.as_bytes();
let mut matches: Vec<(usize, usize)> = Vec::new(); let mut matches: Vec<(usize, usize)> = Vec::new();
for term in &effective_terms { for term in &effective_terms {
let pattern: String = term.to_ascii_lowercase(); let pattern = term.to_ascii_lowercase();
let pbytes = pattern.as_bytes(); let pbytes = pattern.as_bytes();
if pbytes.is_empty() {
continue;
}
let mut start = 0; let mut start = 0;
while start + pbytes.len() <= folded_bytes.len() { while start + pbytes.len() <= folded_bytes.len() {
if let Some(rel) = memfind(&folded_bytes[start..], pbytes) { if let Some(rel) = memfind(&folded_bytes[start..], pbytes) {
let at = start + rel; let at = start + rel;
matches.push((at, at + pbytes.len())); matches.push((at, at + pbytes.len()));
// Advance past this match to avoid zero-width loops on
// empty patterns (already guarded above) and to allow
// overlapping matches of *different* terms in the next
// outer-loop iteration.
start = at + pbytes.len(); start = at + pbytes.len();
} else { } else {
break; break;
@ -96,24 +98,18 @@ pub fn render(text: &str, terms: &[&str], opts: &Options<'_>) -> String {
} }
if matches.is_empty() { if matches.is_empty() {
return truncate_head(text, opts.approx_chars, opts.ellipsis); return head_window(text, opts.approx_chars);
} }
// Dedupe + sort so overlapping matches from different terms (e.g.
// "rust" and "rustc") don't produce nested tags.
matches.sort_by_key(|(a, _)| *a); matches.sort_by_key(|(a, _)| *a);
matches = coalesce_overlapping(matches); let matches = coalesce_overlapping(matches);
// Pick the window. Start a third of the budget before the first match // Pick the window. Start a third of the budget before the first match
// so the hit isn't pinned to the left edge. Round both ends to char // so the hit isn't pinned to the left edge; round both ends to char
// boundaries so we never slice a multi-byte UTF-8 sequence. // boundaries so we never slice a multi-byte UTF-8 sequence.
let pre_pad = opts.approx_chars / 3; let pre_pad = opts.approx_chars / 3;
let first_match_start = matches[0].0; let mut win_start = matches[0].0.saturating_sub(pre_pad);
let mut win_start = first_match_start.saturating_sub(pre_pad); let mut win_end = (win_start + opts.approx_chars).min(text.len());
let mut win_end = win_start + opts.approx_chars;
if win_end > text.len() {
win_end = text.len();
}
while win_start > 0 && !text.is_char_boundary(win_start) { while win_start > 0 && !text.is_char_boundary(win_start) {
win_start -= 1; win_start -= 1;
} }
@ -121,11 +117,9 @@ pub fn render(text: &str, terms: &[&str], opts: &Options<'_>) -> String {
win_end += 1; win_end += 1;
} }
// Expand the window to include the full end of any match that would // Expand the window so a match straddling the right edge is fully
// otherwise be cut off mid-tag. Keeps rendering sane when a long term // included rather than cut mid-hit.
// sits at the right edge of the budget. if let Some((_, end)) = matches.iter().rfind(|(s, _)| *s < win_end) {
let last_match_in_window = matches.iter().rfind(|(s, _)| *s < win_end);
if let Some((_, end)) = last_match_in_window {
if *end > win_end { if *end > win_end {
win_end = *end; win_end = *end;
while win_end < text.len() && !text.is_char_boundary(win_end) { while win_end < text.len() && !text.is_char_boundary(win_end) {
@ -134,55 +128,114 @@ pub fn render(text: &str, terms: &[&str], opts: &Options<'_>) -> String {
} }
} }
// Render: walk matches that fall inside the window, splicing pre/post let ranges = matches
// around each. Prepend/append ellipsis when we've chopped off content. .iter()
let mut out = String::with_capacity(win_end - win_start + 32); .filter(|(s, e)| *e > win_start && *s < win_end)
if win_start > 0 { .map(|(s, e)| ((*s).max(win_start) - win_start, (*e).min(win_end) - win_start))
out.push_str(opts.ellipsis); .collect();
Snippet {
window: text[win_start..win_end].to_string(),
ranges,
truncated_start: win_start > 0,
truncated_end: win_end < text.len(),
} }
let mut cursor = win_start;
for (ms, me) in matches.iter() {
if *me <= win_start || *ms >= win_end {
continue;
}
// Clamp to the window.
let ms = (*ms).max(win_start);
let me = (*me).min(win_end);
if ms > cursor {
out.push_str(&text[cursor..ms]);
}
out.push_str(opts.pre);
out.push_str(&text[ms..me]);
out.push_str(opts.post);
cursor = me;
}
if cursor < win_end {
out.push_str(&text[cursor..win_end]);
}
if win_end < text.len() {
out.push_str(opts.ellipsis);
}
out
} }
/// Return the first `n` characters of `text`, suffixed with `ellipsis` if /// Build a snippet window around one known match range in `text` (byte
/// truncation actually happened. Respects UTF-8 char boundaries. /// offsets into `text`). Used by fuzzy full-text search, where the match
fn truncate_head(text: &str, n: usize, ellipsis: &str) -> String { /// was located by the fuzzy matcher rather than exact term search. The
/// range is clamped and char-boundary-aligned defensively.
pub fn window_around(text: &str, range: (usize, usize), opts: &Options) -> Snippet {
if text.is_empty() {
return Snippet::empty();
}
let (mut ms, mut me) = range;
ms = ms.min(text.len());
me = me.clamp(ms, text.len());
while ms > 0 && !text.is_char_boundary(ms) {
ms -= 1;
}
while me < text.len() && !text.is_char_boundary(me) {
me += 1;
}
let pre_pad = opts.approx_chars / 3;
let mut win_start = ms.saturating_sub(pre_pad);
let mut win_end = (win_start + opts.approx_chars).max(me).min(text.len());
while win_start > 0 && !text.is_char_boundary(win_start) {
win_start -= 1;
}
while win_end < text.len() && !text.is_char_boundary(win_end) {
win_end += 1;
}
let ranges = if me > ms {
vec![(ms - win_start, me - win_start)]
} else {
Vec::new()
};
Snippet {
window: text[win_start..win_end].to_string(),
ranges,
truncated_start: win_start > 0,
truncated_end: win_end < text.len(),
}
}
/// Count non-overlapping occurrences of `term` in `text`. Empty terms count
/// zero. Case-insensitive counting folds ASCII only, matching the rest of
/// the search pipeline.
pub fn count_occurrences(text: &str, term: &str, case_sensitive: bool) -> usize {
if term.is_empty() || text.len() < term.len() {
return 0;
}
let (hay, needle);
let (hay_ref, needle_ref): (&[u8], &[u8]) = if case_sensitive {
(text.as_bytes(), term.as_bytes())
} else {
hay = text.to_ascii_lowercase();
needle = term.to_ascii_lowercase();
(hay.as_bytes(), needle.as_bytes())
};
let mut count = 0;
let mut start = 0;
while start + needle_ref.len() <= hay_ref.len() {
match memfind(&hay_ref[start..], needle_ref) {
Some(rel) => {
count += 1;
start += rel + needle_ref.len();
}
None => break,
}
}
count
}
/// The first `n` bytes of `text` (char-aligned) as a match-less window.
fn head_window(text: &str, n: usize) -> Snippet {
if text.len() <= n { if text.len() <= n {
return text.to_string(); return Snippet {
window: text.to_string(),
ranges: Vec::new(),
truncated_start: false,
truncated_end: false,
};
} }
let mut cut = n; let mut cut = n;
while cut > 0 && !text.is_char_boundary(cut) { while cut > 0 && !text.is_char_boundary(cut) {
cut -= 1; cut -= 1;
} }
let mut out = String::with_capacity(cut + ellipsis.len()); Snippet {
out.push_str(&text[..cut]); window: text[..cut].to_string(),
out.push_str(ellipsis); ranges: Vec::new(),
out truncated_start: false,
truncated_end: true,
}
} }
/// Merge adjacent / overlapping (start, end) ranges in place. Input must be /// Merge adjacent / overlapping (start, end) ranges. Input must be sorted
/// sorted by start. /// by start.
fn coalesce_overlapping(mut v: Vec<(usize, usize)>) -> Vec<(usize, usize)> { fn coalesce_overlapping(mut v: Vec<(usize, usize)>) -> Vec<(usize, usize)> {
if v.len() < 2 { if v.len() < 2 {
return v; return v;
@ -202,7 +255,7 @@ fn coalesce_overlapping(mut v: Vec<(usize, usize)>) -> Vec<(usize, usize)> {
} }
/// Locate the first occurrence of `needle` in `hay`. A byte-level search; /// Locate the first occurrence of `needle` in `hay`. A byte-level search;
/// callers have already lowercased both sides so case is normalized. /// callers have already normalized case where needed.
fn memfind(hay: &[u8], needle: &[u8]) -> Option<usize> { fn memfind(hay: &[u8], needle: &[u8]) -> Option<usize> {
if needle.is_empty() || needle.len() > hay.len() { if needle.is_empty() || needle.len() > hay.len() {
return None; return None;
@ -222,115 +275,170 @@ fn memfind(hay: &[u8], needle: &[u8]) -> Option<usize> {
mod tests { mod tests {
use super::*; use super::*;
fn opts_small() -> Options<'static> { fn opts_small() -> Options {
Options { Options { approx_chars: 40 }
pre: "<b>", }
post: "</b>",
ellipsis: "", /// Every range must be in-bounds, ordered, non-overlapping, and sit on
approx_chars: 40, /// char boundaries — the contract egui's LayoutJob sections rely on.
fn assert_ranges_valid(s: &Snippet) {
let mut prev_end = 0;
for &(a, b) in &s.ranges {
assert!(a < b, "empty/inverted range {:?}", (a, b));
assert!(b <= s.window.len(), "range {:?} beyond window", (a, b));
assert!(a >= prev_end, "overlapping ranges");
assert!(s.window.is_char_boundary(a) && s.window.is_char_boundary(b));
prev_end = b;
} }
} }
fn marked(s: &Snippet) -> Vec<&str> {
s.ranges.iter().map(|&(a, b)| &s.window[a..b]).collect()
}
#[test] #[test]
fn empty_text_returns_empty() { fn empty_text_returns_empty() {
let s = render("", &["foo"], &Options::default()); let s = extract("", &["foo"], &Options::default());
assert_eq!(s, ""); assert_eq!(s, Snippet::empty());
} }
#[test] #[test]
fn no_terms_returns_head_with_ellipsis_when_truncated() { fn no_terms_returns_head_marked_truncated() {
let long = "abcdefghijklmnop".repeat(10); let long = "abcdefghijklmnop".repeat(10);
let s = render(&long, &[], &opts_small()); let s = extract(&long, &[], &opts_small());
assert!(s.ends_with("")); assert!(s.truncated_end);
assert!(s.len() < long.len() + 4); assert!(!s.truncated_start);
assert!(s.ranges.is_empty());
assert!(s.window.len() <= 40);
} }
#[test] #[test]
fn no_terms_untruncated_has_no_ellipsis() { fn no_terms_untruncated() {
let s = render("short text", &[], &opts_small()); let s = extract("short text", &[], &opts_small());
assert_eq!(s, "short text"); assert_eq!(s.window, "short text");
assert!(!s.truncated_end && !s.truncated_start);
} }
#[test] #[test]
fn simple_highlight_wraps_matches() { fn simple_match_range() {
let s = render("the quick brown fox", &["quick"], &opts_small()); let s = extract("the quick brown fox", &["quick"], &opts_small());
assert!(s.contains("<b>quick</b>")); assert_eq!(marked(&s), vec!["quick"]);
assert_ranges_valid(&s);
} }
#[test] #[test]
fn case_insensitive_match() { fn case_insensitive_match_reports_original_case() {
let s = render("The QUICK brown fox", &["quick"], &opts_small()); let s = extract("The QUICK brown fox", &["quick"], &opts_small());
assert!(s.contains("<b>QUICK</b>"), "got {s}"); assert_eq!(marked(&s), vec!["QUICK"]);
} }
#[test] #[test]
fn multiple_terms_both_highlighted() { fn multiple_terms_both_marked() {
let s = render( let s = extract(
"the quick brown fox jumps over the lazy dog", "the quick brown fox jumps over the lazy dog",
&["quick", "lazy"], &["quick", "lazy"],
&Options { &Options::default(),
approx_chars: 200,
..Options::default()
},
); );
assert!(s.contains("<b>quick</b>"), "got {s}"); assert_eq!(marked(&s), vec!["quick", "lazy"]);
assert!(s.contains("<b>lazy</b>"), "got {s}"); assert_ranges_valid(&s);
} }
#[test] #[test]
fn window_trims_with_ellipsis_on_both_sides() { fn window_truncation_flags_on_both_sides() {
let text = let text = "prefix ".repeat(20) + "MATCH in middle " + &"suffix ".repeat(20);
"prefix ".repeat(20) + "MATCH in middle " + &"suffix ".repeat(20); let s = extract(&text, &["MATCH"], &opts_small());
let s = render(&text, &["MATCH"], &opts_small()); assert!(s.truncated_start);
assert!(s.starts_with(""), "got {s}"); assert!(s.truncated_end);
assert!(s.ends_with(""), "got {s}"); assert_eq!(marked(&s), vec!["MATCH"]);
assert!(s.contains("<b>MATCH</b>"), "got {s}");
} }
#[test] #[test]
fn match_at_start_has_no_leading_ellipsis() { fn match_at_start_not_truncated_left() {
let s = render("MATCH right at the start of this paragraph", &["match"], &opts_small()); let s = extract(
assert!(!s.starts_with(""), "got {s}"); "MATCH right at the start of this paragraph",
&["match"],
&opts_small(),
);
assert!(!s.truncated_start);
assert_eq!(s.ranges[0].0, 0);
} }
#[test] #[test]
fn no_match_on_tail_returns_head() { fn no_match_returns_head_without_ranges() {
let text = "alpha beta gamma delta epsilon zeta eta theta iota kappa"; let text = "alpha beta gamma delta epsilon zeta eta theta iota kappa";
let s = render(text, &["nomatch"], &opts_small()); let s = extract(text, &["nomatch"], &opts_small());
assert!(!s.contains("<b>")); assert!(s.ranges.is_empty());
assert!(s.starts_with("alpha")); assert!(s.window.starts_with("alpha"));
} }
#[test] #[test]
fn overlapping_terms_do_not_nest_tags() { fn overlapping_terms_coalesce() {
// Two terms matching the same span must coalesce. let s = extract("the RUSTC compiler", &["rust", "rustc"], &opts_small());
let s = render("the RUSTC compiler", &["rust", "rustc"], &opts_small()); assert_eq!(marked(&s), vec!["RUSTC"]);
assert!(s.contains("<b>RUSTC</b>"), "got {s}"); assert_ranges_valid(&s);
// No nested <b> tags.
assert!(!s.contains("<b><b>"), "got {s}");
} }
#[test] #[test]
fn utf8_boundary_safe_truncation() { fn utf8_boundaries_hold_with_multibyte_text() {
// Insert multi-byte chars near the window boundary.
let text = "café café café café café café café café café café"; let text = "café café café café café café café café café café";
let s = render(text, &["nope"], &opts_small()); let s = extract(text, &["café"], &opts_small());
// Returned string must be valid UTF-8 (push_str guarantees this only assert!(!s.ranges.is_empty());
// if we sliced on char boundaries). Assert by round-trip. assert_ranges_valid(&s);
assert_eq!(s.as_str(), &s.clone()); for m in marked(&s) {
assert_eq!(m, "café");
}
} }
#[test] #[test]
fn match_near_right_edge_is_fully_shown() { fn match_near_right_edge_is_fully_included() {
let prefix = "x".repeat(30); let prefix = "x".repeat(30);
let text = format!("{}{}", prefix, "LONGMATCHTERMTEXT"); let text = format!("{}{}", prefix, "LONGMATCHTERMTEXT");
let s = render(&text, &["LONGMATCHTERMTEXT"], &opts_small()); let s = extract(&text, &["LONGMATCHTERMTEXT"], &opts_small());
assert!(s.contains("<b>LONGMATCHTERMTEXT</b>"), "got {s}"); assert_eq!(marked(&s), vec!["LONGMATCHTERMTEXT"]);
} }
#[test] #[test]
fn empty_query_term_ignored() { fn empty_query_term_ignored() {
let s = render("hello world", &["", "world"], &opts_small()); let s = extract("hello world", &["", "world"], &opts_small());
assert!(s.contains("<b>world</b>"), "got {s}"); assert_eq!(marked(&s), vec!["world"]);
}
#[test]
fn window_around_basic() {
let text = "prefix ".repeat(20) + "NEEDLE" + &" suffix".repeat(20);
let at = text.find("NEEDLE").unwrap();
let s = window_around(&text, (at, at + 6), &opts_small());
assert_eq!(marked(&s), vec!["NEEDLE"]);
assert!(s.truncated_start && s.truncated_end);
assert_ranges_valid(&s);
}
#[test]
fn window_around_clamps_out_of_bounds() {
let s = window_around("tiny", (2, 999), &opts_small());
assert_eq!(s.window, "tiny");
assert_eq!(s.ranges, vec![(2, 4)]);
// Fully out-of-range → no ranges, but never a panic.
let s = window_around("tiny", (999, 1000), &opts_small());
assert!(s.ranges.is_empty());
}
#[test]
fn window_around_aligns_multibyte_boundaries() {
let text = "ééééééééé needle ééééééééé";
// Deliberately mis-aligned offsets inside multi-byte sequences.
let s = window_around(text, (1, 3), &opts_small());
assert_ranges_valid(&s);
}
#[test]
fn count_occurrences_cases() {
assert_eq!(count_occurrences("aaaa", "aaa", true), 1, "non-overlapping");
assert_eq!(count_occurrences("abcABC", "abc", true), 1);
assert_eq!(count_occurrences("abcABC", "abc", false), 2);
assert_eq!(count_occurrences("", "x", true), 0);
assert_eq!(count_occurrences("xyz", "", true), 0);
assert_eq!(count_occurrences("no hits here", "zzz", false), 0);
assert_eq!(count_occurrences("ab ab ab", "ab", true), 3);
} }
} }

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,636 @@
//! End-to-end phase-1 tests over a real tree and a real database.
//!
//! These cover the failure mode that unit tests structurally cannot: a full
//! run deletes index rows for every path it did not see, so any walk that
//! quietly reports less than it should destroys data. That damage is
//! invisible on a first index — `existing_files` is empty, so nothing is
//! stale — and only appears on the second run.
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use quicksearch_core::config::Config;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
fn tmp_dir(tag: &str) -> PathBuf {
let mut p = std::env::temp_dir();
p.push(format!(
"quicksearch-e2e-{}-{}-{}",
tag,
std::process::id(),
SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_nanos()
));
std::fs::create_dir_all(&p).unwrap();
p
}
fn touch(p: &Path, body: &[u8]) {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
}
/// Run one full index and wait for it to finish.
///
/// Completion is detected via the `last_full_index` marker, which
/// `run_indexing` writes only on a successful finish. Polling the status
/// enum instead would race: a small tree finishes between two polls, so
/// `Idle` is ambiguous between "not started yet" and "already done".
fn index_once(root: &Path, db: &Path, config: &Config) {
if db.exists() {
let conn = rusqlite::Connection::open(db).unwrap();
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.unwrap();
}
let service = IndexingService::new();
service
.start_indexing(
vec![root.to_string_lossy().into_owned()],
db.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
let deadline = Instant::now() + Duration::from_secs(120);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().unwrap();
}
/// (path, mtime, content_state) for every indexed row, ordered by path.
fn rows(db: &Path) -> Vec<(String, i64, i64)> {
let conn = rusqlite::Connection::open(db).unwrap();
let mut stmt = conn
.prepare("SELECT path, mtime, content_state FROM files ORDER BY path")
.unwrap();
let out = stmt
.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
.unwrap()
.map(|r| r.unwrap())
.collect();
out
}
fn test_config() -> Config {
let config = Config::default();
// Keep the run to phase 1 semantics we're asserting on; extraction is
// covered elsewhere.
config
}
#[test]
fn reindexing_an_unchanged_tree_changes_nothing() {
let root = tmp_dir("stable");
let db_dir = tmp_dir("stable-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
touch(&root.join("a.txt"), b"alpha");
touch(&root.join("sub/b.txt"), b"bravo");
touch(&root.join("sub/deep/c.txt"), b"charlie");
touch(&root.join("other/d.md"), b"delta");
index_once(&root, &db, &config);
let first = rows(&db);
assert_eq!(first.len(), 4, "all four files indexed");
index_once(&root, &db, &config);
let second = rows(&db);
// The whole point: a second run over an unchanged tree must not delete
// and re-insert anything. A wiped-and-rebuilt row would come back with
// content_state reset, throwing away extracted text for no reason.
assert_eq!(first, second, "an unchanged tree must re-index to an identical set");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn deleted_files_are_removed_and_new_ones_added() {
let root = tmp_dir("churn");
let db_dir = tmp_dir("churn-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
touch(&root.join("keep.txt"), b"keep");
touch(&root.join("remove.txt"), b"remove");
index_once(&root, &db, &config);
assert_eq!(rows(&db).len(), 2);
std::fs::remove_file(root.join("remove.txt")).unwrap();
touch(&root.join("added.txt"), b"added");
index_once(&root, &db, &config);
let names: Vec<String> = rows(&db)
.into_iter()
.map(|(p, _, _)| Path::new(&p).file_name().unwrap().to_string_lossy().into_owned())
.collect();
assert_eq!(names, vec!["added.txt", "keep.txt"], "stale cleanup still works");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn a_modified_file_is_updated_in_place() {
let root = tmp_dir("modify");
let db_dir = tmp_dir("modify-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
let target = root.join("doc.txt");
touch(&target, b"first");
index_once(&root, &db, &config);
let before = rows(&db);
assert_eq!(before.len(), 1);
// Filesystem mtime has one-second granularity in the stored value, so
// move it decisively rather than racing it.
touch(&target, b"second body, clearly different");
let later = SystemTime::now() + Duration::from_secs(5);
filetime_set(&target, later);
index_once(&root, &db, &config);
let after = rows(&db);
assert_eq!(after.len(), 1, "still exactly one row");
assert_ne!(before[0].1, after[0].1, "mtime was refreshed");
assert_eq!(before[0].0, after[0].0, "same path");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// Set a file's mtime without pulling in a dependency for it.
fn filetime_set(path: &Path, when: SystemTime) {
let f = std::fs::OpenOptions::new().write(true).open(path).unwrap();
f.set_modified(when).unwrap();
f.sync_all().unwrap();
}
#[test]
#[cfg(unix)]
fn an_unreadable_directory_does_not_delete_its_rows() {
// The scenario this guards: a network share or removable drive that is
// briefly unavailable. The walk sees nothing beneath it, which must not
// be read as "every file under here was deleted".
use std::os::unix::fs::PermissionsExt;
let root = tmp_dir("blip");
let db_dir = tmp_dir("blip-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
touch(&root.join("visible.txt"), b"visible");
let vault = root.join("vault");
touch(&vault.join("secret.txt"), b"secret");
touch(&vault.join("nested/deeper.txt"), b"deeper");
index_once(&root, &db, &config);
assert_eq!(rows(&db).len(), 3, "all three indexed while readable");
std::fs::set_permissions(&vault, std::fs::Permissions::from_mode(0o000)).unwrap();
index_once(&root, &db, &config);
let during = rows(&db);
std::fs::set_permissions(&vault, std::fs::Permissions::from_mode(0o755)).unwrap();
assert_eq!(
during.len(),
3,
"rows under an unreadable directory must survive, not be deleted"
);
// And once it is readable again, everything still lines up.
index_once(&root, &db, &config);
assert_eq!(rows(&db).len(), 3);
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn stopping_mid_run_deletes_nothing() {
// Pins the end-to-end property: an interrupted run must never delete the
// rows it did not reach.
//
// Two independent guards currently provide it — `run_indexing` skips
// cleanup when the walk did not complete, and `cleanup_stale_index_entries`
// re-checks the stop flag before its first delete. This test passes with
// either one alone, so it does not prove the former is present; it is here
// to catch the day someone removes the last of them.
let root = tmp_dir("stop");
let db_dir = tmp_dir("stop-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
for i in 0..1500 {
touch(&root.join(format!("d{}/f{:04}.txt", i % 25, i)), b"body");
}
index_once(&root, &db, &config);
let full = rows(&db);
assert_eq!(full.len(), 1500);
// Start again and stop almost immediately, so the walk is cut short.
let service = IndexingService::new();
service
.start_indexing(
vec![root.to_string_lossy().into_owned()],
db.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
std::thread::sleep(Duration::from_millis(15));
service.stop_indexing().unwrap();
drop(service);
std::thread::sleep(Duration::from_millis(250));
let after = rows(&db);
assert_eq!(
after.len(),
1500,
"an interrupted run must not delete the rows it never got to"
);
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn a_wide_tree_indexes_every_file_exactly_once() {
// Exercises the parallel walk's chunking and termination against a real
// database, where a duplicate path would be a UNIQUE violation and a
// dropped path would be a missing row.
let root = tmp_dir("wide");
let db_dir = tmp_dir("wide-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
let count = 900;
for i in 0..count {
touch(&root.join(format!("d{}/f{:04}.txt", i % 13, i)), b"body");
}
index_once(&root, &db, &config);
assert_eq!(rows(&db).len(), count, "every file indexed exactly once");
index_once(&root, &db, &config);
assert_eq!(rows(&db).len(), count, "and the second run is stable");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
/// Like `index_once`, but over several roots at once — the per-root
/// pipeline path.
fn index_roots_once(roots: &[&Path], db: &Path, config: &Config) {
if db.exists() {
let conn = rusqlite::Connection::open(db).unwrap();
conn.execute("DELETE FROM schema_info WHERE key = 'last_full_index'", [])
.unwrap();
}
let service = IndexingService::new();
service
.start_indexing(
roots.iter().map(|r| r.to_string_lossy().into_owned()).collect(),
db.to_string_lossy().into_owned(),
config.clone(),
)
.unwrap();
let deadline = Instant::now() + Duration::from_secs(120);
let mut done = false;
while Instant::now() < deadline {
if let IndexingStatus::Error(e) = service.get_status() {
panic!("indexing failed: {}", e);
}
if db.exists() {
if let Ok(conn) = rusqlite::Connection::open(db) {
if quicksearch_core::db::repo::get_last_full_index(&conn).is_some() {
done = true;
break;
}
}
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(done, "indexing did not finish within the timeout");
service.stop_indexing().unwrap();
}
#[test]
fn two_roots_walk_extract_and_clean_independently() {
let root_a = tmp_dir("multi-a");
let root_b = tmp_dir("multi-b");
let db_dir = tmp_dir("multi-db");
let db = db_dir.join("index.sqlite");
let config = test_config();
// Imbalanced roots so the round-robin writer sees a firehose and a
// trickle in the same run.
for i in 0..60 {
touch(&root_a.join(format!("a{:03}.txt", i)), b"alpha corpus xylophone");
}
for i in 0..5 {
touch(&root_b.join(format!("b{:03}.txt", i)), b"bravo corpus quagmire");
}
index_roots_once(&[&root_a, &root_b], &db, &config);
let conn = rusqlite::Connection::open(&db).unwrap();
let total: i64 = conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
assert_eq!(total, 65, "both roots fully walked");
let pending: i64 = conn
.query_row("SELECT COUNT(*) FROM files WHERE content_state = 0", [], |r| r.get(0))
.unwrap();
assert_eq!(pending, 0, "per-root extraction drained both roots");
// Content from EACH root is searchable.
for term in ["\"xylophone\"", "\"quagmire\""] {
let hits: i64 = conn
.query_row(
"SELECT COUNT(*) FROM searchabletext WHERE searchabletext MATCH ?1",
[term],
|r| r.get(0),
)
.unwrap();
assert!(hits > 0, "content from both roots must be indexed ({})", term);
}
drop(conn);
// Stale cleanup is global: deleting a file from the trickle root must
// remove exactly that row on the next multi-root run.
std::fs::remove_file(root_b.join("b000.txt")).unwrap();
index_roots_once(&[&root_a, &root_b], &db, &config);
let conn = rusqlite::Connection::open(&db).unwrap();
let total: i64 = conn
.query_row("SELECT COUNT(*) FROM files", [], |r| r.get(0))
.unwrap();
assert_eq!(total, 64, "stale row swept across roots");
std::fs::remove_dir_all(&root_a).ok();
std::fs::remove_dir_all(&root_b).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
// ---------------------------------------------------------------------------
// Inline extraction: the walk finishes files whose head is the whole file.
//
// `hash_length` is what decides how much of a file the walk reads, so setting
// it to 0 leaves an empty head, nothing can be extracted inline, and the run
// degrades to the pure two-pass behaviour. That makes it the control against
// which the optimised path must produce an identical index.
// ---------------------------------------------------------------------------
/// Everything about a file's indexed content that a user can observe: its
/// state, the stored snippet body, and its property rows.
fn content_rows(db: &Path) -> Vec<(String, i64, Option<String>, Option<i64>, String)> {
let conn = rusqlite::Connection::open(db).unwrap();
let mut stmt = conn
.prepare(
"SELECT f.path, f.content_state, f.failure_msg, d.text_len,
COALESCE(GROUP_CONCAT(p.key || '=' || p.value, ','), '')
FROM files f
LEFT JOIN documents_text d ON d.file_id = f.id
LEFT JOIN properties p ON p.file_id = f.id
GROUP BY f.id
ORDER BY f.path",
)
.unwrap();
let out = stmt
.query_map([], |r| {
Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?, r.get(4)?))
})
.unwrap()
.map(|r| r.unwrap())
.collect();
out
}
/// The decompressed body stored for a file, if any.
fn stored_text(db: &Path, suffix: &str) -> Option<String> {
let conn = rusqlite::Connection::open(db).unwrap();
let blob: Option<Vec<u8>> = conn
.query_row(
"SELECT d.text_zstd FROM documents_text d
JOIN files f ON f.id = d.file_id
WHERE f.path LIKE '%' || ?1",
[suffix],
|r| r.get(0),
)
.ok();
blob.map(|b| String::from_utf8(zstd::decode_all(&b[..]).unwrap()).unwrap())
}
/// A tree that exercises every branch of the inline decision at once.
fn seed_mixed_tree(root: &Path) {
let big = "lorem ipsum dolor sit amet ".repeat(600); // ~16 KiB, past any head
touch(&root.join("small.txt"), b"a small plaintext body with xylophone in it");
touch(&root.join("large.txt"), big.as_bytes());
touch(&root.join("empty.txt"), b"");
// Invalid UTF-8 with a .txt extension: claimed by the plaintext extractor,
// but not decodable, so it must be reported as a failure either way.
touch(&root.join("bad.txt"), &[0x68, 0x69, 0xff, 0xfe, 0x00, 0x41]);
// No extension `infer` or `mime_guess` recognises: no extractor claims it.
touch(&root.join("blob.bin"), &[0x00, 0x01, 0x02, 0xfd, 0xfe, 0xff]);
touch(&root.join("nested/deep/note.md"), b"# heading\n\nquagmire body text\n");
}
#[test]
fn inline_extraction_produces_an_identical_index_to_the_two_pass_path() {
let root = tmp_dir("inline-equiv");
let db_dir = tmp_dir("inline-equiv-db");
seed_mixed_tree(&root);
// Control: hash_length 0 => empty head => nothing can be inlined.
let mut control = Config::default();
control.processing.hash_length = 0;
let db_control = db_dir.join("control.sqlite");
index_once(&root, &db_control, &control);
// Optimised: the default head covers every small file in the tree.
let optimised = Config::default();
let db_opt = db_dir.join("optimised.sqlite");
index_once(&root, &db_opt, &optimised);
assert_eq!(
content_rows(&db_control),
content_rows(&db_opt),
"inlining during the walk must not change a single indexed byte"
);
// And the bodies themselves round-trip identically, not just their lengths.
for f in ["small.txt", "large.txt", "note.md"] {
assert_eq!(
stored_text(&db_control, f),
stored_text(&db_opt, f),
"stored body differs for {}",
f
);
}
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn the_head_boundary_decides_inlining_without_changing_the_result() {
let root = tmp_dir("inline-boundary");
let db_dir = tmp_dir("inline-boundary-db");
// Exactly at the limit, and one byte past it.
let mut config = Config::default();
config.processing.hash_length = 64;
let at = "x".repeat(64);
let past = "y".repeat(65);
touch(&root.join("at.txt"), at.as_bytes());
touch(&root.join("past.txt"), past.as_bytes());
let db = db_dir.join("index.sqlite");
index_once(&root, &db, &config);
// Both are fully extracted; the boundary only decides *which pass* did it.
let conn = rusqlite::Connection::open(&db).unwrap();
let pending: i64 = conn
.query_row("SELECT COUNT(*) FROM files WHERE content_state != 1", [], |r| r.get(0))
.unwrap();
assert_eq!(pending, 0, "both sides of the boundary end up extracted");
drop(conn);
assert_eq!(stored_text(&db, "at.txt").as_deref(), Some(at.as_str()));
assert_eq!(stored_text(&db, "past.txt").as_deref(), Some(past.as_str()));
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn undecodable_small_files_are_reported_as_failures_not_silently_skipped() {
let root = tmp_dir("inline-badutf8");
let db_dir = tmp_dir("inline-badutf8-db");
let db = db_dir.join("index.sqlite");
touch(&root.join("bad.txt"), &[0x68, 0x69, 0xff, 0xfe]);
index_once(&root, &db, &Config::default());
let conn = rusqlite::Connection::open(&db).unwrap();
let (state, msg): (i64, Option<String>) = conn
.query_row(
"SELECT content_state, failure_msg FROM files WHERE path LIKE '%bad.txt'",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
// Inlining must not swallow the error: the walk declines to record it, so
// the content pass still opens the file and stores a reason.
assert_eq!(state, 2, "undecodable content is FAILED, not DONE or NA");
assert!(
msg.unwrap_or_default().contains("bad.txt"),
"the failure names the file"
);
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn an_empty_file_is_done_with_no_snippet_sidecar() {
let root = tmp_dir("inline-empty");
let db_dir = tmp_dir("inline-empty-db");
let db = db_dir.join("index.sqlite");
touch(&root.join("empty.txt"), b"");
index_once(&root, &db, &Config::default());
let conn = rusqlite::Connection::open(&db).unwrap();
let (state, sidecars): (i64, i64) = conn
.query_row(
"SELECT f.content_state, (SELECT COUNT(*) FROM documents_text d WHERE d.file_id = f.id)
FROM files f WHERE f.path LIKE '%empty.txt'",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
assert_eq!(state, 1, "an empty file is extracted, not failed");
assert_eq!(sidecars, 0, "no zstd frame for an empty body");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn the_content_extension_filter_still_excludes_small_text_files() {
let root = tmp_dir("inline-filter");
let db_dir = tmp_dir("inline-filter-db");
let db = db_dir.join("index.sqlite");
let mut config = Config::default();
config.indexing.content_extensions = vec!["md".into()];
touch(&root.join("kept.md"), b"kept quagmire body");
touch(&root.join("skipped.txt"), b"skipped xylophone body");
index_once(&root, &db, &config);
let conn = rusqlite::Connection::open(&db).unwrap();
let states: Vec<(String, i64)> = conn
.prepare("SELECT path, content_state FROM files ORDER BY path")
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|r| r.unwrap())
.collect();
for (path, state) in &states {
if path.ends_with("kept.md") {
assert_eq!(*state, 1, "an allowed extension is extracted");
} else {
assert_eq!(*state, 3, "a filtered extension is NA, never inlined");
}
}
drop(conn);
assert_eq!(stored_text(&db, "skipped.txt"), None, "no body stored for a filtered file");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}
#[test]
fn contentless_mode_still_indexes_inlined_files_without_storing_bodies() {
let root = tmp_dir("inline-contentless");
let db_dir = tmp_dir("inline-contentless-db");
let db = db_dir.join("index.sqlite");
let mut config = Config::default();
config.processing.store_text_for_snippets = false;
touch(&root.join("small.txt"), b"searchable xylophone body");
index_once(&root, &db, &config);
let conn = rusqlite::Connection::open(&db).unwrap();
let sidecars: i64 = conn
.query_row("SELECT COUNT(*) FROM documents_text", [], |r| r.get(0))
.unwrap();
assert_eq!(sidecars, 0, "contentless mode stores no bodies");
let hits: i64 = conn
.query_row(
"SELECT COUNT(*) FROM searchabletext WHERE searchabletext MATCH '\"xylophone\"'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(hits, 1, "an inlined file is still searchable in contentless mode");
std::fs::remove_dir_all(&root).ok();
std::fs::remove_dir_all(&db_dir).ok();
}

View file

@ -207,12 +207,7 @@ fn snippet_paths_perf_comparison() {
let b_reps = 10; let b_reps = 10;
let start_b = Instant::now(); let start_b = Instant::now();
let mut rows_b_total = 0usize; let mut rows_b_total = 0usize;
let opts = snippet::Options { let opts = snippet::Options { approx_chars: 64 };
pre: "<b>",
post: "</b>",
ellipsis: "...",
approx_chars: 64,
};
for _ in 0..b_reps { for _ in 0..b_reps {
for q in QUERIES { for q in QUERIES {
// Contentless FTS5 returns NULL for stored columns (that's the // Contentless FTS5 returns NULL for stored columns (that's the
@ -245,7 +240,7 @@ fn snippet_paths_perf_comparison() {
} }
None => String::new(), None => String::new(),
}; };
let _snip = snippet::render(&text, &[q], &opts); let _snip = snippet::extract(&text, &[q], &opts);
rows_b_total += 1; rows_b_total += 1;
} }
} }

View file

@ -2,14 +2,52 @@
name = "quicksearch-gui" name = "quicksearch-gui"
version.workspace = true version.workspace = true
edition.workspace = true edition.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
description = "Fast full-text search across your files: desktop app and terminal search tool."
[[bin]] [[bin]]
name = "quicksearch" name = "quicksearch"
path = "src/main.rs" path = "src/main.rs"
# Terminal search lives in its own binary because the two want opposite
# Windows subsystems: a GUI built as a console app flashes a console window on
# every launch, and a console tool built as a GUI app cannot write to the shell
# that invoked it (cmd and PowerShell do not even wait for it). Splitting is
# the only arrangement that is correct in both cases. On Unix the distinction
# does not exist and `quicksearch` still does both.
[[bin]]
name = "quicksearch-cli"
path = "src/cli_main.rs"
[dependencies] [dependencies]
quicksearch-core = { path = "../quicksearch-core" } quicksearch-core = { path = "../quicksearch-core" }
dioxus = { version = "0.5.1", features = ["desktop"] } eframe = { version = "0.32", default-features = false, features = [
dioxus-desktop = "0.5.1" "glow",
tokio = { version = "1.0", features = ["time", "signal"] } "persistence",
] }
egui = "0.32"
egui_extras = "0.32"
rfd = "0.15"
open = "5"
chrono = { version = "0.4", default-features = false, features = ["clock"] }
# Display backends, which only exist on Linux/BSD. `default-features = false`
# has to be repeated: feature resolution unions the two stanzas, so a single
# permissive one would switch defaults back on for every target.
[target.'cfg(all(unix, not(target_os = "macos")))'.dependencies]
eframe = { version = "0.32", default-features = false, features = [
"wayland",
"x11",
] }
# Console attachment for the GUI binary (which has no stdio when launched from
# Explorer) and VT-mode enabling for the CLI binary. 0.59 matches what eframe
# and rfd already resolve, so no extra crate is compiled.
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.59", features = [
"Win32_Foundation",
"Win32_System_Console",
] }

View file

@ -1,232 +0,0 @@
// QuickSearch Application JavaScript
// Enhanced UI interactions
document.addEventListener('DOMContentLoaded', function() {
console.log('QuickSearch UI loaded');
// Add loading states to buttons
function addLoadingState(button, originalText) {
button.disabled = true;
button.innerHTML = '<span class="loading"></span>' + originalText;
}
function removeLoadingState(button, originalText) {
button.disabled = false;
button.innerHTML = originalText;
}
// Enhanced form interactions
const forms = document.querySelectorAll('form');
forms.forEach(form => {
form.addEventListener('submit', function(e) {
const submitButton = form.querySelector('button[type="submit"]');
if (submitButton) {
addLoadingState(submitButton, submitButton.textContent);
}
});
});
// Keyboard shortcuts
document.addEventListener('keydown', function(e) {
// Ctrl+F to focus search
if (e.ctrlKey && e.key === 'f') {
e.preventDefault();
const searchInput = document.querySelector('input[type="text"]');
if (searchInput) {
searchInput.focus();
searchInput.select();
}
}
// Escape to clear search
if (e.key === 'Escape') {
const searchInput = document.querySelector('input[type="text"]');
if (searchInput && searchInput === document.activeElement) {
searchInput.value = '';
searchInput.blur();
}
}
});
// Enhanced table interactions
function enhanceTable(table) {
// Add click-to-copy functionality for table cells
const cells = table.querySelectorAll('td');
cells.forEach(cell => {
cell.addEventListener('click', function() {
const text = cell.textContent.trim();
if (text && navigator.clipboard) {
navigator.clipboard.writeText(text).then(() => {
// Visual feedback
cell.style.backgroundColor = '#4CAF50';
cell.style.color = 'white';
setTimeout(() => {
cell.style.backgroundColor = '';
cell.style.color = '';
}, 200);
});
}
});
});
// Add sortable columns (basic implementation)
const headers = table.querySelectorAll('th');
headers.forEach((header, index) => {
header.style.cursor = 'pointer';
header.addEventListener('click', () => sortTable(table, index));
});
}
// Simple table sorting
function sortTable(table, columnIndex) {
const tbody = table.querySelector('tbody');
const rows = Array.from(tbody.querySelectorAll('tr'));
rows.sort((a, b) => {
const aVal = a.cells[columnIndex]?.textContent.trim() || '';
const bVal = b.cells[columnIndex]?.textContent.trim() || '';
// Try numeric sort first
const aNum = parseFloat(aVal);
const bNum = parseFloat(bVal);
if (!isNaN(aNum) && !isNaN(bNum)) {
return aNum - bNum;
}
// Fall back to string sort
return aVal.localeCompare(bVal);
});
// Clear tbody and re-append sorted rows
tbody.innerHTML = '';
rows.forEach(row => tbody.appendChild(row));
}
// Auto-enhance any tables that appear
const observer = new MutationObserver(function(mutations) {
mutations.forEach(function(mutation) {
mutation.addedNodes.forEach(function(node) {
if (node.nodeType === 1) { // Element node
const tables = node.querySelectorAll ? node.querySelectorAll('table') : [];
tables.forEach(enhanceTable);
if (node.tagName === 'TABLE') {
enhanceTable(node);
}
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
// Enhance existing tables
document.querySelectorAll('table').forEach(enhanceTable);
});
// Utility functions for Rust integration
window.QuickSearch = {
// Function to show toast notifications
showToast: function(message, type = 'info') {
const toast = document.createElement('div');
toast.className = `toast toast-${type}`;
toast.textContent = message;
toast.style.cssText = `
position: fixed;
top: 20px;
right: 20px;
padding: 12px 20px;
border-radius: 6px;
color: white;
font-weight: 600;
z-index: 2000;
animation: slideIn 0.3s ease;
`;
// Set background based on type
const colors = {
info: '#2196F3',
success: '#4CAF50',
warning: '#FF9800',
error: '#f44336'
};
toast.style.backgroundColor = colors[type] || colors.info;
document.body.appendChild(toast);
setTimeout(() => {
toast.style.animation = 'slideOut 0.3s ease';
setTimeout(() => {
document.body.removeChild(toast);
}, 300);
}, 3000);
},
// Function to update status display
updateStatus: function(status) {
const statusDisplay = document.querySelector('.status-display');
if (statusDisplay) {
statusDisplay.textContent = status;
}
},
// Function to highlight search terms in results
highlightSearchTerms: function(searchTerm, container) {
if (!searchTerm || !container) return;
const walker = document.createTreeWalker(
container,
NodeFilter.SHOW_TEXT,
null,
false
);
const textNodes = [];
let node;
while (node = walker.nextNode()) {
textNodes.push(node);
}
textNodes.forEach(textNode => {
const parent = textNode.parentNode;
if (parent.tagName === 'B') return; // Skip already highlighted
const text = textNode.textContent;
const regex = new RegExp(`(${searchTerm})`, 'gi');
if (regex.test(text)) {
const highlightedHTML = text.replace(regex, '<mark>$1</mark>');
const wrapper = document.createElement('span');
wrapper.innerHTML = highlightedHTML;
parent.replaceChild(wrapper, textNode);
}
});
}
};
// Add custom CSS for toasts and animations
const style = document.createElement('style');
style.textContent = `
@keyframes slideIn {
from { transform: translateX(100%); opacity: 0; }
to { transform: translateX(0); opacity: 1; }
}
@keyframes slideOut {
from { transform: translateX(0); opacity: 1; }
to { transform: translateX(100%); opacity: 0; }
}
mark {
background: #ffeb3b;
padding: 2px 4px;
border-radius: 3px;
font-weight: bold;
}
.toast {
box-shadow: 0 4px 12px rgba(0,0,0,0.2);
}
`;
document.head.appendChild(style);

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 652 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 919 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 998 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.8 KiB

View file

@ -0,0 +1,14 @@
<svg width="240" height="240" viewBox="0 0 240 240" role="img" xmlns="http://www.w3.org/2000/svg"><title style="fill:rgb(0, 0, 0);stroke:none;color:rgb(11, 11, 11);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">Fast full-text search icon, square crop</title><desc style="fill:rgb(0, 0, 0);stroke:none;color:rgb(11, 11, 11);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">A square app icon: centered text lines indenting around a magnifying glass with a lightning bolt inside.</desc>
<rect x="0" y="0" width="240" height="240" rx="52" fill="#1e293b" style="fill:rgb(30, 41, 59);stroke:none;color:rgb(11, 11, 11);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<g stroke="#475569" stroke-width="10" stroke-linecap="round" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto">
<line x1="42" y1="58" x2="198" y2="58" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<line x1="42" y1="92" x2="62" y2="92" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<line x1="42" y1="126" x2="56" y2="126" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<line x1="42" y1="160" x2="68" y2="160" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<line x1="42" y1="194" x2="168" y2="194" style="fill:rgb(0, 0, 0);stroke:rgb(71, 85, 105);color:rgb(11, 11, 11);stroke-width:10px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
</g>
<circle cx="130" cy="118" r="46" fill="#0ea5e9" fill-opacity="0.18" style="fill:rgb(14, 165, 233);stroke:none;color:rgb(11, 11, 11);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<circle cx="130" cy="118" r="46" fill="none" stroke="#38bdf8" stroke-width="14" style="fill:none;stroke:rgb(56, 189, 248);color:rgb(11, 11, 11);stroke-width:14px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<line x1="163" y1="151" x2="195" y2="183" stroke="#38bdf8" stroke-width="16" stroke-linecap="round" style="fill:rgb(0, 0, 0);stroke:rgb(56, 189, 248);color:rgb(11, 11, 11);stroke-width:16px;stroke-linecap:round;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
<path d="M137 84 L108 125 L128 125 L121 154 L152 111 L132 111 Z" fill="#facc15" style="fill:rgb(250, 204, 21);stroke:none;color:rgb(11, 11, 11);stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;opacity:1;font-family:&quot;Anthropic Sans&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, sans-serif;font-size:16px;font-weight:400;text-anchor:start;dominant-baseline:auto"/>
</svg>

After

Width:  |  Height:  |  Size: 5 KiB

View file

@ -1,21 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>QuickSearch - File Indexer & Search</title>
<!-- External CSS -->
<link rel="stylesheet" href="assets://styles.css">
<!-- Favicon (optional) -->
<link rel="icon" type="image/x-icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24'><path fill='%234CAF50' d='M15.5 14h-.79l-.28-.27C15.41 12.59 16 11.11 16 9.5 16 5.91 13.09 3 9.5 3S3 5.91 3 9.5 5.91 16 9.5 16c1.61 0 3.09-.59 4.23-1.57l.27.28v.79l5 4.99L20.49 19l-4.99-5zm-6 0C7.01 14 5 11.99 5 9.5S7.01 5 9.5 5 14 7.01 14 9.5 11.99 14 9.5 14z'/></svg>">
</head>
<body>
<!-- Main app container - Dioxus will render into this -->
<div id="main"></div>
<!-- External JavaScript -->
<script src="assets://app.js"></script>
</body>
</html>

View file

@ -1,299 +0,0 @@
/* QuickSearch Application Styles */
body {
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
margin: 0;
padding: 20px;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
min-height: 100vh;
}
.app-container {
max-width: 1200px;
margin: 0 auto;
background: white;
border-radius: 12px;
box-shadow: 0 8px 32px rgba(0,0,0,0.1);
overflow: hidden;
}
.app-header {
background: linear-gradient(90deg, #4CAF50 0%, #45a049 100%);
color: white;
padding: 20px;
text-align: center;
}
.app-content {
padding: 20px;
}
.section {
background: #f8f9fa;
border-radius: 8px;
padding: 20px;
margin-bottom: 20px;
border-left: 4px solid #4CAF50;
}
.form-control {
width: 100%;
padding: 12px;
border: 2px solid #e0e0e0;
border-radius: 6px;
font-size: 14px;
transition: border-color 0.3s ease;
box-sizing: border-box;
}
.form-control:focus {
outline: none;
border-color: #4CAF50;
box-shadow: 0 0 0 3px rgba(76, 175, 80, 0.1);
}
.btn {
padding: 12px 24px;
border: none;
border-radius: 6px;
cursor: pointer;
font-size: 14px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.5px;
transition: all 0.3s ease;
margin-right: 10px;
display: inline-block;
}
.btn-primary {
background: linear-gradient(90deg, #4CAF50 0%, #45a049 100%);
color: white;
}
.btn-primary:hover:not(:disabled) {
transform: translateY(-2px);
box-shadow: 0 4px 12px rgba(76, 175, 80, 0.3);
}
.btn-danger {
background: linear-gradient(90deg, #f44336 0%, #d32f2f 100%);
color: white;
}
.btn-danger:hover:not(:disabled) {
transform: translateY(-2px);
box-shadow: 0 4px 12px rgba(244, 67, 54, 0.3);
}
.btn-info {
background: linear-gradient(90deg, #2196F3 0%, #1976D2 100%);
color: white;
}
.btn-info:hover:not(:disabled) {
transform: translateY(-2px);
box-shadow: 0 4px 12px rgba(33, 150, 243, 0.3);
}
.btn:disabled {
opacity: 0.6;
cursor: not-allowed;
transform: none !important;
box-shadow: none !important;
}
.status-display {
background: #1a1a1a;
color: #00ff00;
padding: 15px;
border-radius: 6px;
font-family: 'Courier New', monospace;
white-space: pre-wrap;
font-size: 13px;
border: 1px solid #333;
overflow-x: auto;
}
.search-results {
background: white;
border-radius: 8px;
overflow: hidden;
box-shadow: 0 2px 8px rgba(0,0,0,0.1);
margin-top: 20px;
}
.results-table {
max-height: 400px;
overflow: auto;
border: 1px solid #ddd;
border-radius: 6px;
}
.results-table table {
width: 100%;
border-collapse: collapse;
}
.results-table th {
background: #4CAF50;
color: white;
padding: 12px;
text-align: left;
font-weight: 600;
position: sticky;
top: 0;
border-bottom: 2px solid #45a049;
}
.results-table td {
padding: 10px 12px;
border-bottom: 1px solid #e0e0e0;
word-break: break-all;
}
.results-table tbody tr:hover {
background: #f5f5f5;
}
.results-table tbody tr:nth-child(even) {
background: #fafafa;
}
.results-table tbody tr:nth-child(even):hover {
background: #f0f0f0;
}
/* Clickable path cells */
.path-cell.clickable {
cursor: pointer;
color: #1976d2;
font-weight: 500;
transition: all 0.2s ease;
position: relative;
}
.path-cell.clickable:hover {
background: #e3f2fd !important;
color: #0d47a1;
text-decoration: underline;
}
.path-cell.clickable:active {
background: #bbdefb !important;
transform: scale(0.98);
}
.error-message {
background: #ffebee;
color: #c62828;
padding: 12px;
border-radius: 6px;
border-left: 4px solid #f44336;
margin: 10px 0;
}
.modal-backdrop {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background: rgba(0,0,0,0.5);
backdrop-filter: blur(4px);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.modal-dialog {
background: white;
padding: 30px;
border-radius: 12px;
max-width: 600px;
box-shadow: 0 20px 60px rgba(0,0,0,0.3);
animation: modalSlideIn 0.3s ease;
}
@keyframes modalSlideIn {
from {
opacity: 0;
transform: translateY(-20px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
h1, h2, h3 {
margin-top: 0;
color: #333;
}
label {
display: block;
margin-bottom: 8px;
font-weight: 600;
color: #555;
}
.form-group {
margin-bottom: 20px;
}
/* Highlight matched text in search results */
.results-table b {
background: #ffeb3b;
padding: 2px 4px;
border-radius: 3px;
font-weight: bold;
color: #333;
}
/* Loading spinner */
.loading {
display: inline-block;
width: 20px;
height: 20px;
border: 3px solid #f3f3f3;
border-top: 3px solid #4CAF50;
border-radius: 50%;
animation: spin 1s linear infinite;
margin-right: 10px;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
/* Responsive design */
@media (max-width: 768px) {
body {
padding: 10px;
}
.app-container {
border-radius: 8px;
}
.app-content {
padding: 15px;
}
.form-control {
font-size: 16px; /* Prevents zoom on iOS */
}
.results-table {
font-size: 12px;
}
.btn {
width: 100%;
margin-bottom: 10px;
margin-right: 0;
}
}

View file

@ -0,0 +1,721 @@
//! Application shell: tab strip, per-frame event drains, debounce,
//! status bar, and config-change routing.
use std::time::{Duration, Instant};
use quicksearch_core::cli::{index_counts, IndexCounts};
use quicksearch_core::config::{diff_actions, nested_roots, Config};
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
use quicksearch_core::indexing::{ConfigChange, IndexingStatus, RootPhase};
use quicksearch_core::search::SearchOptions;
use quicksearch_core::watcher::WatchError;
use crate::backend::Backend;
use crate::duplicates_tab::{DupState, DuplicatesTab};
use crate::format::{fmt_interval, group_thousands};
use crate::logs_tab::LogsTab;
use crate::manage_tab::ManageTab;
use crate::options::OptionsWindow;
use crate::search_tab::SearchTab;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Tab {
Search,
Manage,
Duplicates,
Logs,
}
pub struct QuickSearchApp {
cfg: Config,
backend: Backend,
tab: Tab,
search: SearchTab,
manage: ManageTab,
dups: DuplicatesTab,
logs: LogsTab,
options: OptionsWindow,
/// Cached idle counts for the status bar, refreshed at most every 5 s.
counts: Option<(Instant, IndexCounts)>,
/// Set when applying a config that invalidates the stored index.
rebuild_prompt: Option<Vec<ConfigChange>>,
/// Set while the "delete the index?" confirmation is open.
clear_prompt: bool,
/// Nested roots found in the loaded config (startup validation); shown
/// as a modal over the Manage tab until dismissed.
nested_prompt: Option<Vec<(String, String)>>,
/// Set when the watcher gave up on the directory budget and live
/// updates are off; see [`QuickSearchApp::check_watch_cap_warning`].
watch_cap_prompt: Option<WatchError>,
config_error: Option<String>,
}
impl QuickSearchApp {
/// `initial_query` pre-fills the search box and fires a search on the
/// first frame. It carries the positional arguments the binary was given,
/// which on Windows is the only thing the GUI can do with them — terminal
/// output belongs to `quicksearch-cli` there.
pub fn new(
cc: &eframe::CreationContext<'_>,
cfg: Config,
config_error: Option<String>,
initial_query: Option<String>,
) -> Result<QuickSearchApp, String> {
// Compact styling: results density is the whole point.
cc.egui_ctx.style_mut(|style| {
style.spacing.item_spacing = egui::vec2(6.0, 3.0);
style.spacing.button_padding = egui::vec2(6.0, 2.0);
});
cc.egui_ctx.set_zoom_factor(clamp_scale(cfg.ui.scale));
let backend = Backend::start(&cfg, cc.egui_ctx.clone())?;
let fuzzy = cfg.search.fuzzy_default;
// Startup validation: a hand-edited config can nest roots, which
// per-root pipelines can't accept. Redirect straight to the folder
// list with an explanatory modal; the coordinator refuses runs
// until it's fixed.
let nested = nested_roots(&cfg.paths.indexing_paths);
let (tab, nested_prompt) = if nested.is_empty() {
(Tab::Search, None)
} else {
(Tab::Manage, Some(nested))
};
let mut search = SearchTab::new(fuzzy);
if let Some(query) = initial_query {
search.seed(query);
}
Ok(QuickSearchApp {
cfg,
backend,
tab,
search,
manage: ManageTab::new(),
dups: DuplicatesTab::new(),
logs: LogsTab::new(),
options: OptionsWindow::new(),
counts: None,
rebuild_prompt: None,
clear_prompt: false,
nested_prompt,
watch_cap_prompt: None,
config_error,
})
}
fn search_options(&self) -> SearchOptions {
SearchOptions {
fuzzy: self.search.fuzzy,
fuzzy_max_edits: self.cfg.search.fuzzy_max_edits,
limit: self.cfg.search.display_limit,
batch: self.cfg.search.results_per_page.max(1),
session_ignores: self.search.session_ignores.clone(),
}
}
fn start_search(&mut self) {
let generation = self
.backend
.search()
.search(&self.search.query, self.search_options());
self.search.on_search_started(generation);
}
fn start_duplicates_scan(&mut self, ctx: &egui::Context) {
self.dups.state = DupState::Loading;
let cfg = self.cfg.clone();
self.backend.start_duplicates(&cfg, ctx.clone());
}
/// Save + route an edited config to the running services.
fn apply_new_config(&mut self, ctx: &egui::Context, mut new: Config) {
if let Some((child, parent)) = nested_roots(&new.paths.indexing_paths).first() {
self.config_error = Some(format!(
"Not applied: indexed folder {} is nested under {}",
child, parent
));
return;
}
// Warned-root memory only means anything for folders still indexed.
// Pruning here is what makes removing and re-adding a folder warn
// again rather than staying silently suppressed forever.
new.ui
.watch_cap_warned_roots
.retain(|root| new.paths.indexing_paths.contains(root));
let actions = diff_actions(&self.cfg, &new);
if let Err(e) = new.save() {
self.config_error = Some(e);
}
if (new.ui.scale - self.cfg.ui.scale).abs() > f32::EPSILON {
ctx.set_zoom_factor(clamp_scale(new.ui.scale));
}
if actions.search_db_changed {
self.backend
.search()
.set_db_path(new.resolved_database_path());
self.counts = None;
}
self.backend.coordinator.apply_config(new.clone());
if actions.requires_rebuild {
if self.backend.coordinator.state().mode == IndexMode::Auto {
// Automatic mode is hands-off: reconcile immediately, no
// prompt. Root-only changes need just a full run — the
// walk indexes new roots and the stale sweep drops removed
// ones. Anything else (tokenizer, hashing, filters, hidden
// files) invalidates stored data and gets the real wipe.
let roots_only = {
let mut probe = new.clone();
probe.paths.indexing_paths = self.cfg.paths.indexing_paths.clone();
!diff_actions(&self.cfg, &probe).requires_rebuild
};
if roots_only {
self.backend.coordinator.reindex_now();
} else {
self.backend.coordinator.rebuild_index();
}
} else {
let changes = self
.backend
.coordinator
.check_config_validation(&new)
.ok()
.flatten()
.unwrap_or_default();
self.rebuild_prompt = Some(changes);
}
}
self.cfg = new;
self.manage.invalidate_editors();
}
fn drain_events(&mut self) {
// Streamed search results.
loop {
match self.backend.search_rx.try_recv() {
Ok(update) => self
.search
.apply_update(update, self.cfg.search.display_limit),
Err(_) => break,
}
}
// Duplicates worker.
if let Some(rx) = &self.backend.dup_job {
match rx.try_recv() {
Ok(Ok(groups)) => {
self.dups.state = DupState::Loaded(groups);
self.backend.dup_job = None;
}
Ok(Err(e)) => {
self.dups.state = DupState::Error(e);
self.backend.dup_job = None;
}
Err(std::sync::mpsc::TryRecvError::Empty) => {}
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
self.dups.state = DupState::Error("duplicates scan aborted".into());
self.backend.dup_job = None;
}
}
}
}
fn tick_debounce(&mut self, ctx: &egui::Context) {
let Some(edited_at) = self.search.pending_edit else {
return;
};
let debounce = Duration::from_millis(self.cfg.search.debounce_ms);
let elapsed = edited_at.elapsed();
if elapsed >= debounce {
self.search.pending_edit = None;
self.start_search();
} else {
ctx.request_repaint_after(debounce - elapsed);
}
}
/// Raise the "live updates are disabled" modal when the watcher has
/// given up on the directory budget and at least one indexed folder has
/// not been warned about yet.
///
/// Keyed on roots rather than a single dismissed flag: a restart should
/// stay quiet, but adding a folder changes the trade-off and deserves
/// the warning again.
fn check_watch_cap_warning(&mut self, state: &IndexerState) {
let WatcherStatus::Disabled { reason } = &state.watcher else {
// Recovered (e.g. the user trimmed the folder list) — retract a
// modal that is no longer true.
self.watch_cap_prompt = None;
return;
};
// Only the budget limits warrant a modal. Other failures are
// transient and not the user's to act on; they are named in the
// status line's tooltip and logged to the Logs tab.
if !matches!(
reason,
WatchError::TooManyDirectories { .. } | WatchError::KernelLimit { .. }
) {
return;
}
if self.watch_cap_prompt.is_some() {
return;
}
let unwarned = self
.cfg
.paths
.indexing_paths
.iter()
.any(|root| !self.cfg.ui.watch_cap_warned_roots.contains(root));
if unwarned {
self.watch_cap_prompt = Some(reason.clone());
}
}
fn status_bar(&mut self, ctx: &egui::Context) {
let state = self.backend.coordinator.state();
self.manage.observe(&state.activity);
self.check_watch_cap_warning(&state);
egui::TopBottomPanel::bottom("status-bar").show(ctx, |ui| {
ui.horizontal(|ui| {
match &state.activity {
IndexingStatus::Idle => {
let mode = match state.mode {
IndexMode::Auto => "Auto",
IndexMode::ManualStopped => "Manual",
IndexMode::ManualRunning => "Manual",
};
let stale = self
.counts
.map(|(at, _)| at.elapsed() > Duration::from_secs(5))
.unwrap_or(true);
if stale {
let db = self.cfg.resolved_database_path();
let counts = index_counts(&db.to_string_lossy()).unwrap_or(
IndexCounts {
files: 0,
content_done: 0,
content_pending: 0,
},
);
self.counts = Some((Instant::now(), counts));
}
let files = self.counts.map(|(_, c)| c.files).unwrap_or(0);
ui.label(
egui::RichText::new(format!(
"Idle · {} · {} files indexed",
mode,
group_thousands(files.max(0) as u64)
))
.small(),
);
}
IndexingStatus::Error(e) => {
ui.colored_label(
ui.visuals().error_fg_color,
egui::RichText::new(format!("Indexing error: {}", e)).small(),
);
}
IndexingStatus::Stopping => {
ui.label(egui::RichText::new("Stopping indexing…").small());
}
IndexingStatus::Running { roots, .. } => {
let done = roots
.iter()
.filter(|r| r.phase == RootPhase::Done)
.count();
let processed: usize =
roots.iter().map(|r| r.walked + r.extracted).sum();
let totals_known = roots.iter().all(|r| r.walk_total.is_some());
let denominator: usize = roots
.iter()
.map(|r| r.walk_total.unwrap_or(0) + r.extract_total)
.sum();
let mut text = if totals_known && denominator > 0 {
let frac =
(processed as f64 / denominator as f64).min(1.0);
format!(
"Indexing {} / {} ({:.0}%)",
group_thousands(processed as u64),
group_thousands(denominator as u64),
frac * 100.0
)
} else {
format!(
"Indexing · {} files",
group_thousands(processed as u64)
)
};
if roots.len() > 1 {
text.push_str(&format!(" · {}/{} roots done", done, roots.len()));
}
if let Some(rate) = self.manage.speed.files_per_sec() {
text.push_str(&format!(" · {}", crate::format::fmt_rate(rate)));
}
let active: usize = roots.iter().map(|r| r.active_workers).sum();
let total_workers: usize =
roots.iter().map(|r| r.total_workers).sum();
if total_workers > 0 {
text.push_str(&format!(" · {}/{} workers", active, total_workers));
}
ui.label(egui::RichText::new(text).small());
if totals_known && denominator > 0 {
let frac =
(processed as f32 / denominator as f32).clamp(0.0, 1.0);
ui.add(egui::ProgressBar::new(frac).desired_width(120.0));
} else {
ui.add(egui::Spinner::new().size(12.0));
}
}
}
// Right corner: search result count.
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if self.tab == Tab::Search {
if let Some(label) = self.search.result_count_label() {
ui.label(egui::RichText::new(label).small().weak());
}
}
});
});
});
// Keep painting while anything is moving.
if !matches!(state.activity, IndexingStatus::Idle | IndexingStatus::Error(_)) {
ctx.request_repaint_after(Duration::from_millis(250));
}
// Watcher registration walks every root, so its verdict can land
// minutes after startup. Without this the warning would wait for
// the user to happen to move the mouse.
if matches!(state.watcher, WatcherStatus::Starting) {
ctx.request_repaint_after(Duration::from_millis(500));
}
}
fn rebuild_prompt_ui(&mut self, ctx: &egui::Context) {
let Some(changes) = &self.rebuild_prompt else {
return;
};
let changes = changes.clone();
let mut close = false;
egui::Window::new("Rebuild index?")
.collapsible(false)
.resizable(false)
.default_width(560.0)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.label("These settings differ from what the index was built with:");
ui.add_space(4.0);
if changes.is_empty() {
ui.monospace("indexing settings changed");
}
for change in &changes {
ui.strong(format!("{}:", change.key));
// Multi-line values (roots, patterns, extensions) are
// newline-joined — side-by-side columns keep before and
// after readable instead of one run-on arrow line.
ui.columns(2, |cols| {
cols[0].label(
egui::RichText::new("index was built with").small().weak(),
);
cols[0].monospace(display_value(&change.stored));
cols[1].label(egui::RichText::new("config now says").small().weak());
cols[1].monospace(display_value(&change.current));
});
ui.add_space(6.0);
}
ui.label(
egui::RichText::new(
"A full rebuild applies them everywhere. Until then, existing \
entries keep the old settings.",
)
.small()
.weak(),
);
ui.horizontal(|ui| {
if ui.button("Rebuild now").clicked() {
self.backend.coordinator.rebuild_index();
close = true;
}
if ui.button("Later").clicked() {
close = true;
}
});
});
if close {
self.rebuild_prompt = None;
}
}
}
impl QuickSearchApp {
fn nested_prompt_ui(&mut self, ctx: &egui::Context) {
let Some(pairs) = &self.nested_prompt else {
return;
};
let pairs = pairs.clone();
let mut close = false;
egui::Window::new("Indexed folders may not be nested")
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.label(
"Each root is indexed by its own worker pool, so one root \
may not contain another. Fix the folder list below:",
);
for (child, parent) in &pairs {
ui.monospace(format!("{}{}", child, parent));
}
ui.label(
egui::RichText::new(
"Indexing stays paused until the overlap is removed and \
the list is applied.",
)
.small()
.weak(),
);
if ui.button("Fix folders").clicked() {
close = true;
}
});
if close {
self.nested_prompt = None;
self.tab = Tab::Manage;
}
}
fn watch_cap_prompt_ui(&mut self, ctx: &egui::Context) {
let Some(reason) = &self.watch_cap_prompt else {
return;
};
let reason = reason.clone();
let mut close = false;
egui::Window::new("Live index updating is disabled")
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.set_max_width(420.0);
match &reason {
WatchError::TooManyDirectories { cap, .. } => {
ui.label(format!(
"Your indexed folders contain more than {} directories. The \
system limits how many folders can be watched for changes at \
once, so QuickSearch cannot update the index as files change.",
group_thousands(*cap as u64),
));
}
WatchError::KernelLimit { registered } => {
ui.label(format!(
"The system ran out of folder watches after {} directories, so \
QuickSearch cannot update the index as files change.",
group_thousands(*registered as u64),
));
}
WatchError::Other(msg) => {
ui.label(format!("Live updates are unavailable: {}", msg));
}
}
ui.add_space(4.0);
ui.label(format!(
"The index is rebuilt every {} instead. Searches keep working; \
recent changes may take that long to appear.",
fmt_interval(self.cfg.indexing.reindex_interval_minutes),
));
ui.label(
egui::RichText::new(
"To restore live updates, index fewer folders or exclude large \
subfolders under Filters on the Manage Index tab.",
)
.small()
.weak(),
);
if ui.button("OK").clicked() {
close = true;
}
});
if close {
self.watch_cap_prompt = None;
for root in &self.cfg.paths.indexing_paths {
if !self.cfg.ui.watch_cap_warned_roots.contains(root) {
self.cfg.ui.watch_cap_warned_roots.push(root.clone());
}
}
if let Err(e) = self.cfg.save() {
self.config_error = Some(e);
}
}
}
fn clear_prompt_ui(&mut self, ctx: &egui::Context) {
if !self.clear_prompt {
return;
}
let mut close = false;
egui::Window::new("Clear index?")
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.label("This deletes the search index database. Your files are not touched.");
ui.label(
egui::RichText::new(
"Indexing switches to manual until you start it again or return to automatic mode.",
)
.small()
.weak(),
);
ui.horizontal(|ui| {
if ui
.button(
egui::RichText::new("Delete index")
.color(ui.visuals().error_fg_color),
)
.clicked()
{
self.backend.coordinator.clear_index();
self.counts = None;
self.dups.state = DupState::NotLoaded;
close = true;
}
if ui.button("Cancel").clicked() {
close = true;
}
});
});
if close {
self.clear_prompt = false;
}
}
}
/// A stored/current config value for the rebuild prompt; list values are
/// already newline-joined and render as-is, empty means unset.
fn display_value(value: &str) -> String {
if value.trim().is_empty() {
"(none)".to_string()
} else {
value.to_string()
}
}
/// Keep the configured UI scale within sane, recoverable bounds.
fn clamp_scale(scale: f32) -> f32 {
if scale.is_finite() {
scale.clamp(0.5, 2.5)
} else {
1.1
}
}
impl eframe::App for QuickSearchApp {
fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
self.drain_events();
self.tick_debounce(ctx);
self.status_bar(ctx);
let previous_tab = self.tab;
egui::TopBottomPanel::top("tab-strip").show(ctx, |ui| {
ui.horizontal(|ui| {
ui.selectable_value(&mut self.tab, Tab::Search, "Search");
ui.selectable_value(&mut self.tab, Tab::Manage, "Manage Index");
ui.selectable_value(&mut self.tab, Tab::Duplicates, "Duplicates");
ui.selectable_value(&mut self.tab, Tab::Logs, "Logs");
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if ui.button("").on_hover_text("Options").clicked() {
if self.options.open {
self.options.open = false;
} else {
self.options.open_with(&self.cfg);
}
}
});
});
});
// Entering the Duplicates tab kicks off a fresh scan.
if self.tab == Tab::Duplicates && previous_tab != Tab::Duplicates {
self.start_duplicates_scan(ctx);
}
if let Some(err) = &self.config_error {
let err = err.clone();
egui::TopBottomPanel::top("config-error").show(ctx, |ui| {
ui.horizontal(|ui| {
ui.colored_label(
ui.visuals().error_fg_color,
format!("Config problem: {} (using defaults)", err),
);
if ui.small_button("Dismiss").clicked() {
self.config_error = None;
}
});
});
}
egui::CentralPanel::default().show(ctx, |ui| match self.tab {
Tab::Search => {
let actions = self.search.ui(ui);
if let Some(fuzzy) = actions.save_fuzzy_default {
self.cfg.search.fuzzy_default = fuzzy;
if let Err(e) = self.cfg.save() {
self.config_error = Some(e);
}
}
if let Some(pattern) = actions.persist_ignore {
let mut new_cfg = self.cfg.clone();
if !new_cfg.indexing.ignore_patterns.contains(&pattern) {
new_cfg.indexing.ignore_patterns.push(pattern);
self.apply_new_config(ctx, new_cfg);
}
}
if actions.rerun {
self.start_search();
}
}
Tab::Manage => {
let state = self.backend.coordinator.state();
let actions = self.manage.ui(ui, &state, &self.cfg);
if actions.start_now {
self.backend.coordinator.reindex_now();
}
if actions.stop {
self.backend.coordinator.set_mode(IndexMode::ManualStopped);
}
if actions.auto {
self.backend.coordinator.set_mode(IndexMode::Auto);
}
if actions.clear_index {
self.clear_prompt = true;
}
if actions.start_now || actions.stop || actions.auto {
// Keep repainting while the command lands so the state
// change is visible without wiggling the mouse — fast
// runs otherwise flash by between frames.
ui.ctx().request_repaint_after(Duration::from_millis(100));
}
if let Some(new_cfg) = actions.apply_config {
self.apply_new_config(ctx, new_cfg);
}
}
Tab::Duplicates => {
let actions = self.dups.ui(ui);
if actions.refresh {
self.start_duplicates_scan(ctx);
}
}
Tab::Logs => self.logs.ui(ui),
});
if let Some(new_cfg) = self.options.ui(ctx, &self.cfg) {
self.apply_new_config(ctx, new_cfg);
}
self.rebuild_prompt_ui(ctx);
self.clear_prompt_ui(ctx);
self.nested_prompt_ui(ctx);
self.watch_cap_prompt_ui(ctx);
}
fn on_exit(&mut self, _gl: Option<&eframe::glow::Context>) {
self.backend.shutdown();
}
}

View file

@ -0,0 +1,71 @@
//! Wiring between the egui thread and the core services.
//!
//! All communication is non-blocking from the UI's point of view:
//! searches stream over an mpsc receiver drained each frame, indexing
//! state is polled, and the duplicates query runs on a throwaway worker
//! thread. Core threads wake the UI through `ctx.request_repaint()`.
use std::sync::{mpsc, Arc};
use quicksearch_core::config::Config;
use quicksearch_core::coordinator::IndexCoordinator;
use quicksearch_core::search::{DuplicateGroup, SearchService, SearchUpdate};
use quicksearch_core::shutdown;
pub struct Backend {
pub coordinator: Arc<IndexCoordinator>,
pub search: Option<SearchService>,
pub search_rx: mpsc::Receiver<SearchUpdate>,
pub dup_job: Option<mpsc::Receiver<Result<Vec<DuplicateGroup>, String>>>,
}
impl Backend {
pub fn start(config: &Config, ctx: egui::Context) -> Result<Backend, String> {
let coordinator = Arc::new(IndexCoordinator::start(config.clone())?);
if let Err(e) = shutdown::install_signal_handler(coordinator.clone()) {
quicksearch_core::log_warn!("failed to install signal handler: {}", e);
}
let repaint_ctx = ctx.clone();
let (search, search_rx) = SearchService::new(
config.resolved_database_path(),
Arc::new(move || repaint_ctx.request_repaint()),
);
Ok(Backend {
coordinator,
search: Some(search),
search_rx,
dup_job: None,
})
}
pub fn search(&self) -> &SearchService {
self.search.as_ref().expect("search service alive")
}
/// Kick off (or restart) the duplicates listing on a worker thread.
pub fn start_duplicates(&mut self, config: &Config, ctx: egui::Context) {
let (tx, rx) = mpsc::channel();
let db = config.resolved_database_path();
std::thread::spawn(move || {
let result = quicksearch_core::search::find_duplicate_groups(
&db.to_string_lossy(),
500,
0,
);
let _ = tx.send(result);
ctx.request_repaint();
});
self.dup_job = Some(rx);
}
/// Join the search worker and stop the coordinator. Called once from
/// `on_exit`.
pub fn shutdown(&mut self) {
if let Some(search) = self.search.take() {
search.shutdown();
}
self.coordinator.shutdown();
}
}

View file

@ -0,0 +1,230 @@
//! Terminal query mode: `quicksearch [FLAGS] <query terms...>` runs the
//! same ranked cascade the GUI uses and prints results to stdout. With no
//! positional arguments the binary opens the GUI instead.
use std::io::IsTerminal;
use std::sync::atomic::AtomicU64;
use quicksearch_core::config::Config;
use quicksearch_core::db;
use quicksearch_core::query::split::split_for_cascade;
use quicksearch_core::search::{cascade, SearchHit, SearchOptions};
use crate::format::{fmt_mtime, human_size};
pub(crate) const USAGE: &str = "\
QuickSearch: indexed file search
USAGE:
quicksearch open the GUI
quicksearch [FLAGS] <query terms> search from the terminal
(Windows: quicksearch-cli)
FLAGS:
--fuzzy also run the fuzzy filename/full-text passes
--limit <N> maximum results (default: [search].display_limit)
--long rank, size, mtime, and snippets instead of bare paths
-h, --help this help
Query syntax matches the GUI: plain words form one phrase; filters like
type:Document, modified:>=2024-01-01, path:/dir, mime:application/pdf,
name:frag combine with it.";
/// Parse argv; `Some(exit_code)` when the invocation was CLI-mode (query
/// or --help), `None` to open the GUI.
///
/// Invariant: terminal mode never builds an [`IndexCoordinator`], so it
/// starts no filesystem watcher, no background threads, and consumes no
/// inotify watches — a one-shot query must not leave anything running or
/// compete for the per-user watch budget with a running GUI. It opens the
/// database, queries, prints, and exits. Keep it that way: the coordinator
/// belongs to the GUI path in `backend.rs` alone.
///
/// [`IndexCoordinator`]: quicksearch_core::coordinator::IndexCoordinator
pub fn maybe_run_cli() -> Option<i32> {
let args: Vec<String> = std::env::args().skip(1).collect();
let mut fuzzy = false;
let mut long = false;
let mut limit: Option<usize> = None;
let mut terms: Vec<String> = Vec::new();
let mut it = args.into_iter();
while let Some(arg) = it.next() {
match arg.as_str() {
"-h" | "--help" => {
println!("{}", USAGE);
return Some(0);
}
"--fuzzy" => fuzzy = true,
"--long" => long = true,
"--limit" => match it.next().and_then(|v| v.parse().ok()) {
Some(n) => limit = Some(n),
None => {
eprintln!("--limit requires a number\n\n{}", USAGE);
return Some(2);
}
},
other if other.starts_with("--limit=") => {
match other["--limit=".len()..].parse() {
Ok(n) => limit = Some(n),
Err(_) => {
eprintln!("--limit requires a number\n\n{}", USAGE);
return Some(2);
}
}
}
other if other.starts_with('-') && terms.is_empty() => {
// Unknown flags without a query fall through to the GUI
// (they may be eframe/winit flags).
return None;
}
other => terms.push(other.to_string()),
}
}
if terms.is_empty() {
return None;
}
Some(run_query(&terms.join(" "), fuzzy, limit, long))
}
fn run_query(query: &str, fuzzy: bool, limit: Option<usize>, long: bool) -> i32 {
let config = match Config::load() {
Ok(c) => c,
Err(e) => {
eprintln!("config: {}", e);
return 2;
}
};
let db_path = config.resolved_database_path();
// Read-write purely so SQLite may create the WAL shared-memory file
// when no other process has the index open; nothing is written.
let conn = match db::open_existing(&db_path.to_string_lossy(), true) {
Ok(c) => c,
Err(e) => {
eprintln!(
"No usable index at {}; run the GUI once to build it.\n({})",
db_path.display(),
e
);
return 2;
}
};
let split = match split_for_cascade(query) {
Ok(s) => s,
Err(e) => {
eprintln!("query: {}", e);
return 2;
}
};
if fuzzy {
if let Some(warning) = config.search.fuzzy_edits_warning() {
eprintln!("warning: {}", warning);
}
}
let options = SearchOptions {
fuzzy,
fuzzy_max_edits: config.search.fuzzy_max_edits,
limit: limit.unwrap_or(config.search.display_limit),
batch: config.search.results_per_page.max(1),
session_ignores: Vec::new(),
};
let latest = AtomicU64::new(1);
let mut hits: Vec<SearchHit> = Vec::new();
let outcome = cascade::run(&conn, &split, &options, 1, &latest, &mut |batch| {
hits.extend(batch)
});
match outcome {
Ok(Some(outcome)) => {
let color = long && std::io::stdout().is_terminal() && enable_vt();
for hit in &hits {
if long {
println!(
"{:6.3} {:>9} {} {}",
hit.rank,
human_size(hit.size),
fmt_mtime(hit.mtime),
hit.path
);
if let Some(snip) = &hit.snippet {
println!(" {}", render_snippet(snip, color));
}
} else {
println!("{}", hit.path);
}
}
if outcome.limited {
eprintln!("(truncated at {} results; raise with --limit)", hits.len());
}
0
}
Ok(None) => 0, // unreachable: nothing cancels a CLI search
Err(e) => {
eprintln!("search: {}", e);
2
}
}
}
/// Whether ANSI escapes will actually render.
///
/// Always true where the terminal is ANSI by nature. On Windows the console
/// only interprets escapes once `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set:
/// Windows Terminal and Windows 11 have it already, older conhost needs it
/// turned on, and anything that refuses gets plain text rather than a screen
/// full of `\x1b[1m`.
#[cfg(not(windows))]
fn enable_vt() -> bool {
true
}
#[cfg(windows)]
fn enable_vt() -> bool {
use windows_sys::Win32::Foundation::INVALID_HANDLE_VALUE;
use windows_sys::Win32::System::Console::{
GetConsoleMode, GetStdHandle, SetConsoleMode, ENABLE_VIRTUAL_TERMINAL_PROCESSING,
STD_OUTPUT_HANDLE,
};
unsafe {
let handle = GetStdHandle(STD_OUTPUT_HANDLE);
if handle.is_null() || handle == INVALID_HANDLE_VALUE {
return false;
}
let mut mode = 0;
if GetConsoleMode(handle, &mut mode) == 0 {
return false;
}
mode & ENABLE_VIRTUAL_TERMINAL_PROCESSING != 0
|| SetConsoleMode(handle, mode | ENABLE_VIRTUAL_TERMINAL_PROCESSING) != 0
}
}
/// One-line snippet with matches emphasized (ANSI bold on TTYs).
fn render_snippet(snip: &quicksearch_core::snippet::Snippet, color: bool) -> String {
let mut out = String::new();
if snip.truncated_start {
out.push('…');
}
let mut cursor = 0;
for &(start, end) in &snip.ranges {
out.push_str(&snip.window[cursor..start]);
if color {
out.push_str("\x1b[1m");
out.push_str(&snip.window[start..end]);
out.push_str("\x1b[0m");
} else {
out.push_str(&snip.window[start..end]);
}
cursor = end;
}
out.push_str(&snip.window[cursor..]);
if snip.truncated_end {
out.push('…');
}
out.replace(['\n', '\r'], " ")
}

View file

@ -0,0 +1,21 @@
//! `quicksearch-cli <query>` — terminal search, and nothing else.
//!
//! A console-subsystem binary, so redirection, pipes, exit codes, and the
//! shell waiting for the process all behave normally. `src/cli.rs` and
//! `src/format.rs` are shared with the GUI binary by compiling them into both;
//! neither touches egui, so there is nothing to split out into a library.
mod cli;
// The GUI uses more of this module than the CLI does.
#[allow(dead_code)]
mod format;
fn main() {
// `maybe_run_cli` returns `None` for "no query given", which the combined
// binary treats as "open the GUI". This one has no GUI to fall back to.
let code = cli::maybe_run_cli().unwrap_or_else(|| {
eprintln!("{}", cli::USAGE);
2
});
std::process::exit(code);
}

View file

@ -0,0 +1,111 @@
//! The Duplicates tab: groups of files sharing a content hash.
use quicksearch_core::search::DuplicateGroup;
use crate::format::{group_thousands, human_size};
use crate::platform;
pub enum DupState {
NotLoaded,
Loading,
Loaded(Vec<DuplicateGroup>),
Error(String),
}
pub struct DuplicatesTab {
pub state: DupState,
}
/// What the tab asks the app to do after this frame.
#[derive(Default)]
pub struct DuplicatesActions {
pub refresh: bool,
}
impl DuplicatesTab {
pub fn new() -> DuplicatesTab {
DuplicatesTab {
state: DupState::NotLoaded,
}
}
pub fn ui(&mut self, ui: &mut egui::Ui) -> DuplicatesActions {
let mut actions = DuplicatesActions::default();
ui.horizontal(|ui| {
let loading = matches!(self.state, DupState::Loading);
if ui.add_enabled(!loading, egui::Button::new("Refresh")).clicked() {
actions.refresh = true;
}
if loading {
ui.add(egui::Spinner::new().size(16.0));
ui.label("Scanning for duplicates…");
}
});
ui.separator();
match &self.state {
DupState::NotLoaded => {
ui.label(
egui::RichText::new("Press Refresh to scan the index for duplicate files.")
.weak(),
);
}
DupState::Loading => {}
DupState::Error(e) => {
ui.colored_label(ui.visuals().error_fg_color, e);
}
DupState::Loaded(groups) => {
if groups.is_empty() {
ui.label("No duplicate files found.");
return actions;
}
if groups.len() == 500 {
ui.label(
egui::RichText::new("Showing the 500 largest groups.").small().weak(),
);
}
egui::ScrollArea::vertical().auto_shrink([false; 2]).show(ui, |ui| {
for (i, group) in groups.iter().enumerate() {
let name = group
.members
.first()
.map(|m| m.1.as_str())
.unwrap_or("(unknown)");
let title = format!(
"{} × {}: {} reclaimable ({} total)",
group_thousands(group.count as u64),
name,
human_size(group.redundant_size.max(0) as u64),
human_size(group.total_size.max(0) as u64),
);
egui::CollapsingHeader::new(title).id_salt(i).show(ui, |ui| {
for (_, _, path, size, _) in &group.members {
ui.horizontal(|ui| {
ui.label(human_size(*size));
let response = ui
.add(egui::Label::new(egui::RichText::new(path).monospace())
.sense(egui::Sense::click()));
if response.double_clicked() {
platform::open_file(path);
}
response.context_menu(|ui| {
if ui.button("Open").clicked() {
platform::open_file(path);
ui.close();
}
if ui.button("Open containing folder").clicked() {
platform::reveal_in_folder(path);
ui.close();
}
});
});
}
});
}
});
}
}
actions
}
}

View file

@ -0,0 +1,202 @@
//! Small display formatters shared across tabs.
/// Human-readable byte size: `999 B`, `1.2 KB`, `4.7 MB`, `1.3 GB`.
pub fn human_size(bytes: u64) -> String {
const UNITS: [&str; 5] = ["B", "KB", "MB", "GB", "TB"];
let mut value = bytes as f64;
let mut unit = 0;
while value >= 1000.0 && unit < UNITS.len() - 1 {
value /= 1000.0;
unit += 1;
}
if unit == 0 {
format!("{} B", bytes)
} else {
format!("{:.1} {}", value, UNITS[unit])
}
}
/// `YYYY-MM-DD HH:MM` in local time; raw seconds if out of range.
pub fn fmt_mtime(unix_secs: i64) -> String {
use chrono::TimeZone;
match chrono::Local.timestamp_opt(unix_secs, 0) {
chrono::LocalResult::Single(dt) => dt.format("%Y-%m-%d %H:%M").to_string(),
_ => unix_secs.to_string(),
}
}
/// Relative time for recent events, absolute for old ones: "just now",
/// "5 min ago", "3 h ago", else `YYYY-MM-DD HH:MM`. Gives instant
/// feedback that an action (like a fast index run) actually happened.
pub fn fmt_ago(unix_secs: u64) -> String {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let age = now.saturating_sub(unix_secs);
if age < 60 {
"just now".to_string()
} else if age < 3600 {
format!("{} min ago", age / 60)
} else if age < 86_400 {
format!("{} h ago", age / 3600)
} else {
fmt_mtime(unix_secs as i64)
}
}
/// A configured interval as a phrase to drop after "every": `90 min`,
/// `24 h`, `3 days`. Used where the periodic reindex is the only thing
/// refreshing the index, so the user can judge how stale it may get.
pub fn fmt_interval(minutes: u64) -> String {
if minutes == 0 {
// The scheduler treats 0 as always-due.
return "run".to_string();
}
if minutes < 60 {
return format!("{} min", minutes);
}
if minutes.is_multiple_of(1440) {
let days = minutes / 1440;
return if days == 1 {
// "24 h" reads better than "1 day" for the shipped default.
"24 h".to_string()
} else {
format!("{} days", days)
};
}
if minutes.is_multiple_of(60) {
return format!("{} h", minutes / 60);
}
format!("{} h {} min", minutes / 60, minutes % 60)
}
/// Group thousands for counts: `1,234,567`.
pub fn group_thousands(n: u64) -> String {
let digits = n.to_string();
let mut out = String::with_capacity(digits.len() + digits.len() / 3);
for (i, c) in digits.chars().enumerate() {
if i > 0 && (digits.len() - i) % 3 == 0 {
out.push(',');
}
out.push(c);
}
out
}
/// Files/sec display. Never renders a nonzero rate as "0.0": slow rates
/// switch to a per-minute figure.
pub fn fmt_rate(files_per_sec: f64) -> String {
if files_per_sec <= 0.0 {
"0 files/s".to_string()
} else if files_per_sec >= 10.0 {
format!("{:.0} files/s", files_per_sec)
} else if files_per_sec >= 1.0 {
format!("{:.1} files/s", files_per_sec)
} else {
format!("{:.0} files/min", (files_per_sec * 60.0).max(1.0))
}
}
/// Search duration: milliseconds under a second, seconds above.
pub fn fmt_elapsed(d: std::time::Duration) -> String {
let ms = d.as_millis();
if ms >= 1000 {
format!("{:.1} s", d.as_secs_f64())
} else {
format!("{} ms", ms)
}
}
/// Middle-truncate a path to at most `max_chars` characters.
pub fn middle_truncate(s: &str, max_chars: usize) -> String {
let chars: Vec<char> = s.chars().collect();
if chars.len() <= max_chars || max_chars < 5 {
return s.to_string();
}
let keep = max_chars - 1;
let head = keep / 2;
let tail = keep - head;
let mut out: String = chars[..head].iter().collect();
out.push('…');
out.extend(&chars[chars.len() - tail..]);
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn sizes() {
assert_eq!(human_size(0), "0 B");
assert_eq!(human_size(999), "999 B");
assert_eq!(human_size(1200), "1.2 KB");
assert_eq!(human_size(4_700_000), "4.7 MB");
assert_eq!(human_size(1_300_000_000), "1.3 GB");
}
#[test]
fn intervals() {
assert_eq!(fmt_interval(0), "run");
assert_eq!(fmt_interval(1), "1 min");
assert_eq!(fmt_interval(59), "59 min");
assert_eq!(fmt_interval(60), "1 h");
assert_eq!(fmt_interval(90), "1 h 30 min");
assert_eq!(fmt_interval(120), "2 h");
assert_eq!(fmt_interval(1440), "24 h", "the shipped default");
assert_eq!(fmt_interval(2880), "2 days");
assert_eq!(fmt_interval(10_080), "7 days");
}
#[test]
fn thousands() {
assert_eq!(group_thousands(0), "0");
assert_eq!(group_thousands(999), "999");
assert_eq!(group_thousands(1000), "1,000");
assert_eq!(group_thousands(1_234_567), "1,234,567");
}
#[test]
fn rates_never_show_zero_for_nonzero() {
assert_eq!(fmt_rate(0.0), "0 files/s");
assert_eq!(fmt_rate(2543.0), "2543 files/s");
// Not 3.14: clippy reads that as a botched `PI` and denies it.
assert_eq!(fmt_rate(3.12), "3.1 files/s");
assert_eq!(fmt_rate(0.4), "24 files/min");
assert_eq!(fmt_rate(0.001), "1 files/min", "floor at 1/min, never 0.0");
}
#[test]
fn elapsed_units() {
use std::time::Duration;
assert_eq!(fmt_elapsed(Duration::from_millis(0)), "0 ms");
assert_eq!(fmt_elapsed(Duration::from_millis(7)), "7 ms");
assert_eq!(fmt_elapsed(Duration::from_millis(999)), "999 ms");
assert_eq!(fmt_elapsed(Duration::from_millis(1000)), "1.0 s");
assert_eq!(fmt_elapsed(Duration::from_millis(2340)), "2.3 s");
}
#[test]
fn ago_buckets() {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_secs();
assert_eq!(fmt_ago(now), "just now");
assert_eq!(fmt_ago(now - 59), "just now");
assert_eq!(fmt_ago(now - 120), "2 min ago");
assert_eq!(fmt_ago(now - 7200), "2 h ago");
assert!(fmt_ago(now - 200_000).contains('-'), "old = absolute date");
}
#[test]
fn truncation() {
assert_eq!(middle_truncate("short", 20), "short");
let t = middle_truncate("/very/long/path/to/some/file.txt", 15);
assert!(t.chars().count() <= 15);
assert!(t.contains('…'));
assert!(t.starts_with("/very"));
assert!(t.ends_with("e.txt"));
}
}

View file

@ -1,385 +0,0 @@
#![allow(non_snake_case)]
use std::sync::Arc;
use std::collections::VecDeque;
use std::time::Instant;
use dioxus::prelude::*;
use quicksearch_core::indexing::{IndexingService, IndexingStatus};
use quicksearch_core::config::Config;
#[derive(Debug, Clone)]
struct SpeedDataPoint {
timestamp: Instant,
files_processed: usize,
}
struct SpeedTracker {
data_points: VecDeque<SpeedDataPoint>,
}
impl SpeedTracker {
fn new() -> Self {
Self {
data_points: VecDeque::new(),
}
}
fn add_data_point(&mut self, files_processed: usize) {
let now = Instant::now();
self.data_points.push_back(SpeedDataPoint {
timestamp: now,
files_processed,
});
// Prune data points older than 1 second
while let Some(front) = self.data_points.front() {
if now.duration_since(front.timestamp).as_secs_f64() > 1.0 {
self.data_points.pop_front();
} else {
break;
}
}
}
fn calculate_files_per_second(&self) -> Option<f64> {
if self.data_points.len() < 2 {
return None;
}
let newest = self.data_points.back()?;
let oldest = self.data_points.front()?;
let time_span = newest.timestamp.duration_since(oldest.timestamp).as_secs_f64();
if time_span < 0.1 { // Avoid division by very small numbers
return None;
}
let files_diff = newest.files_processed.saturating_sub(oldest.files_processed);
Some(files_diff as f64 / time_span)
}
}
#[derive(Props, Clone)]
pub struct AppProps {
pub indexing_service: Arc<IndexingService>,
pub config: Config,
}
impl PartialEq for AppProps {
fn eq(&self, other: &Self) -> bool {
Arc::ptr_eq(&self.indexing_service, &other.indexing_service)
&& self.config.paths.indexing_paths == other.config.paths.indexing_paths
&& self.config.paths.database_path == other.config.paths.database_path
}
}
pub fn App(props: AppProps) -> Element {
// Multi-root support: the GUI's single line edit holds one path per line.
// Empty lines are ignored. Core still de-duplicates and handles nested roots.
let mut indexing_path = use_signal(|| props.config.paths.indexing_paths.join("\n"));
let mut db_path = use_signal(|| props.config.paths.database_path.clone());
let mut status_text = use_signal(|| "Idle".to_string());
let mut show_config_dialog = use_signal(|| false);
let mut config_changes = use_signal(|| Vec::<String>::new());
let speed_tracker = use_signal(|| SpeedTracker::new());
let indexing_service_for_start = props.indexing_service.clone();
let indexing_service_for_start_dialog = props.indexing_service.clone();
let indexing_service_for_stop = props.indexing_service.clone();
let indexing_service_for_timer = props.indexing_service.clone();
let config_for_start = props.config.clone();
let config_for_dialog = props.config.clone();
// Automatic status updates every second
{
let mut status_text_clone = status_text.clone();
let mut speed_tracker_clone = speed_tracker.clone();
let service_clone = indexing_service_for_timer.clone();
use_future(move || {
let service = service_clone.clone();
async move {
loop {
tokio::time::sleep(tokio::time::Duration::from_millis(50)).await;
let status = service.get_status();
let status_str = match status {
IndexingStatus::Idle => {
// Reset speed tracker when idle
speed_tracker_clone.set(SpeedTracker::new());
"Idle".to_string()
},
IndexingStatus::CountingFiles {
current_file,
start_time,
..
} => {
let elapsed = start_time.elapsed();
let current_file_display = current_file
.as_ref()
.map(|f| format!("{}", f))
.unwrap_or_else(|| "...".to_string());
format!(
"Phase 0 - Counting paths (shell) - {:.1}s elapsed\n{}",
elapsed.as_secs_f64(),
current_file_display
)
}
IndexingStatus::RunningFileIndex { files_processed, total_files, current_file, start_time } => {
// Add data point to speed tracker
speed_tracker_clone.with_mut(|tracker| {
tracker.add_data_point(files_processed);
});
let elapsed = start_time.elapsed();
let current_file_display = current_file
.as_ref()
.map(|f| format!("Current: {}", f))
.unwrap_or_default();
// Calculate speed
let speed_display = speed_tracker_clone.with(|tracker| {
tracker.calculate_files_per_second()
.map(|fps| format!(" - {:.1} files/sec", fps))
.unwrap_or_default()
});
if let Some(total) = total_files {
let percentage = if total > 0 {
(files_processed as f64 / total as f64 * 100.0) as u32
} else { 0 };
format!(
"Phase 1 - File Index: {}/{} files ({}%) - {:.1}s elapsed{}\n{}",
files_processed,
total,
percentage,
elapsed.as_secs_f64(),
speed_display,
current_file_display
)
} else {
format!(
"Phase 1 - File Index: {} files processed - {:.1}s elapsed{}\n{}",
files_processed,
elapsed.as_secs_f64(),
speed_display,
current_file_display
)
}
}
IndexingStatus::RunningTextIndex { files_processed, current_file, start_time } => {
// Add data point to speed tracker
speed_tracker_clone.with_mut(|tracker| {
tracker.add_data_point(files_processed);
});
let elapsed = start_time.elapsed();
let current_file_display = current_file
.as_ref()
.map(|f| format!("Current: {}", f))
.unwrap_or_default();
// Calculate speed
let speed_display = speed_tracker_clone.with(|tracker| {
tracker.calculate_files_per_second()
.map(|fps| format!(" - {:.1} files/sec", fps))
.unwrap_or_default()
});
format!(
"Phase 2 - Text Index: {} files processed - {:.1}s elapsed{}\n{}",
files_processed,
elapsed.as_secs_f64(),
speed_display,
current_file_display
)
}
IndexingStatus::Stopping => "Indexing Stopped".to_string(),
IndexingStatus::Error(ref e) => format!("Error: {}", e),
};
status_text_clone.set(status_str);
}
}
});
}
rsx! {
div {
class: "app-container",
div {
class: "app-header",
h1 { "QuickSearch File Indexer" }
}
div {
class: "app-content",
div {
class: "section",
h2 { "Indexing Controls" }
div {
class: "form-group",
label { "Paths to index (one per line):" }
textarea {
class: "form-control",
rows: "3",
value: "{indexing_path}",
oninput: move |evt| indexing_path.set(evt.value())
}
}
div {
class: "form-group",
label { "Database path:" }
input {
class: "form-control",
r#type: "text",
value: "{db_path}",
oninput: move |evt| db_path.set(evt.value())
}
}
div {
class: "form-group",
button {
class: "btn btn-primary",
onclick: move |_| {
let service = indexing_service_for_start.clone();
let config = config_for_start.clone();
let path_text = indexing_path().clone();
let paths: Vec<String> = path_text
.lines()
.map(|l| l.trim().to_string())
.filter(|l| !l.is_empty())
.collect();
let db = db_path().clone();
if paths.is_empty() {
status_text.set("Enter at least one path to index (one per line).".to_string());
} else {
// The config_validation key still stores a single joined string.
let joined = paths.join("\n");
match service.check_config_validation(&db, &config, &joined) {
Ok(Some(changes)) => {
config_changes.set(changes);
show_config_dialog.set(true);
}
Ok(None) => {
let _ = service.start_indexing(paths, db, config);
}
Err(e) => {
status_text.set(format!("Configuration validation error: {}", e));
}
}
}
},
"Start Indexing"
}
button {
class: "btn btn-danger",
onclick: move |_| {
let _ = indexing_service_for_stop.stop_indexing();
},
"Stop Indexing"
}
}
}
div {
class: "section",
h2 { "Status" }
pre {
class: "status-display",
"{status_text}"
}
}
crate::search::Search {
indexing_service: props.indexing_service.clone(),
db_path: db_path().clone()
}
} // Close app-content
}
// Configuration validation dialog
if show_config_dialog() {
div {
class: "modal-backdrop",
div {
class: "modal-dialog",
h3 {
style: "margin-top: 0; color: #d32f2f;",
"⚠️ Configuration Changes Detected"
}
p {
style: "margin: 15px 0;",
"The following configuration changes require deleting and rebuilding the search index:"
}
ul {
style: "margin: 15px 0; padding-left: 20px;",
for change in config_changes().iter() {
li {
style: "margin: 5px 0; font-family: monospace; background-color: #f5f5f5; padding: 5px; border-radius: 3px;",
"{change}"
}
}
}
p {
style: "margin: 15px 0; font-weight: bold;",
"This will delete the existing index and rebuild it from scratch."
}
div {
style: "display: flex; gap: 10px; margin-top: 20px;",
button {
style: "padding: 10px 20px; background-color: #d32f2f; color: white; border: none; border-radius: 5px; cursor: pointer;",
onclick: move |_| {
let service = indexing_service_for_start_dialog.clone();
let config = config_for_dialog.clone();
let path_text = indexing_path().clone();
let paths: Vec<String> = path_text
.lines()
.map(|l| l.trim().to_string())
.filter(|l| !l.is_empty())
.collect();
let db = db_path().clone();
show_config_dialog.set(false);
status_text.set("Stopping indexing and deleting database...".to_string());
// Delete database file and restart indexing
let service_clone = service.clone();
let paths_clone = paths.clone();
let db_clone = db.clone();
let config_clone = config.clone();
let mut status_clone = status_text.clone();
spawn(async move {
match service_clone.delete_index_for_rebuild(&db_clone) {
Ok(()) => {
status_clone.set("Database deleted. Starting fresh indexing...".to_string());
let _ = service_clone.start_indexing(paths_clone, db_clone, config_clone);
}
Err(e) => {
status_clone.set(format!("Error deleting database: {}", e));
}
}
});
},
"Yes, Rebuild Index"
}
button {
style: "padding: 10px 20px; background-color: #666; color: white; border: none; border-radius: 5px; cursor: pointer;",
onclick: move |_| {
show_config_dialog.set(false);
},
"Cancel"
}
}
}
}
}
}
}

View file

@ -0,0 +1,255 @@
//! The Logs tab: what the terminal would have shown.
//!
//! Everything here comes from [`quicksearch_core::log`], which background
//! threads write through instead of printing. Launched from a desktop
//! launcher there is no terminal to read, and this is the only place a
//! "cannot read that folder" warning is visible.
use quicksearch_core::log::{self, Level, LogLine};
use crate::format::group_thousands;
/// How often to repaint while the tab is open. Log lines arrive on indexer
/// and watcher threads, which have no reason to wake the UI, so an idle
/// window would otherwise sit on a stale list until the mouse moved.
const REFRESH_MS: u64 = 500;
pub struct LogsTab {
/// Copy of the ring, refreshed only when the recorded count moves —
/// cloning a few thousand lines every frame would be silly.
lines: Vec<LogLine>,
/// [`log::recorded`] as of the last refresh.
seen: u64,
dropped: u64,
filter: String,
warnings_only: bool,
/// Keep the newest line in view. Scrolling up releases the view anyway
/// (egui unsticks a scroll area the user moves, and re-sticks it when
/// they return to the bottom); unticking this stops it following at all.
follow: bool,
}
impl LogsTab {
pub fn new() -> LogsTab {
LogsTab {
lines: Vec::new(),
seen: 0,
dropped: 0,
filter: String::new(),
warnings_only: false,
follow: true,
}
}
fn refresh(&mut self) {
self.lines = log::snapshot();
self.seen = log::recorded();
self.dropped = log::dropped();
}
pub fn ui(&mut self, ui: &mut egui::Ui) {
if log::recorded() != self.seen {
self.refresh();
}
ui.ctx()
.request_repaint_after(std::time::Duration::from_millis(REFRESH_MS));
// Indices rather than references: the control row below takes the
// filter and follow flags mutably, and a borrow of `self.lines`
// held across it would conflict. One frame of lag after a
// keystroke, which repaints immediately anyway.
let needle = self.filter.to_lowercase();
let shown: Vec<usize> = self
.lines
.iter()
.enumerate()
.filter(|(_, l)| keep(l, &needle, self.warnings_only))
.map(|(i, _)| i)
.collect();
let mut cleared = false;
ui.horizontal(|ui| {
ui.checkbox(&mut self.follow, "Follow")
.on_hover_text("Scroll to the newest line as it arrives");
ui.checkbox(&mut self.warnings_only, "Warnings only");
ui.add(
egui::TextEdit::singleline(&mut self.filter)
.desired_width(200.0)
.hint_text("Filter"),
);
if ui
.add_enabled(!shown.is_empty(), egui::Button::new("Copy"))
.on_hover_text("Copy the lines shown below to the clipboard")
.clicked()
{
let joined = shown
.iter()
.map(|&i| self.lines[i].text.as_str())
.collect::<Vec<_>>()
.join("\n");
ui.ctx().copy_text(joined);
}
if ui
.add_enabled(!self.lines.is_empty(), egui::Button::new("Clear"))
.clicked()
{
log::clear();
cleared = true;
}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let count = if shown.len() == self.lines.len() {
format!("{} lines", group_thousands(self.lines.len() as u64))
} else {
format!(
"{} of {} lines",
group_thousands(shown.len() as u64),
group_thousands(self.lines.len() as u64)
)
};
ui.label(egui::RichText::new(count).small().weak());
});
});
if cleared {
// `shown` indexes lines that no longer exist.
self.refresh();
return;
}
if self.dropped > 0 {
ui.label(
egui::RichText::new(format!(
"{} earlier lines were dropped; the newest {} are kept.",
group_thousands(self.dropped),
group_thousands(log::CAPACITY as u64),
))
.small()
.weak(),
);
}
ui.separator();
if self.lines.is_empty() {
ui.label(
egui::RichText::new(
"Nothing logged yet. Warnings from indexing, watching folders and \
opening files appear here the same lines the terminal would show.",
)
.weak(),
);
return;
}
if shown.is_empty() {
ui.label(egui::RichText::new("No lines match the filter.").weak());
return;
}
// Long paths extend into a horizontal scroll rather than wrapping:
// `show_rows` only draws the visible slice, and that costs nothing
// only while every row is exactly one line tall.
ui.style_mut().wrap_mode = Some(egui::TextWrapMode::Extend);
let row_height = ui.text_style_height(&egui::TextStyle::Monospace);
egui::ScrollArea::both()
.auto_shrink([false; 2])
.stick_to_bottom(self.follow)
.show_rows(ui, row_height, shown.len(), |ui, range| {
for &i in &shown[range] {
let line = &self.lines[i];
ui.horizontal(|ui| {
ui.label(
egui::RichText::new(fmt_clock(line.at))
.monospace()
.weak(),
);
let text = egui::RichText::new(&line.text).monospace();
match line.level {
Level::Warn => {
ui.colored_label(ui.visuals().warn_fg_color, text);
}
Level::Info => {
ui.label(text);
}
}
});
}
});
}
}
/// Whether a line survives the tab's two filters. `needle` is expected
/// already lowercased — it is the same for every line, so folding it once
/// per frame beats folding it per line.
fn keep(line: &LogLine, needle: &str, warnings_only: bool) -> bool {
if warnings_only && line.level != Level::Warn {
return false;
}
needle.is_empty() || line.text.to_lowercase().contains(needle)
}
/// `HH:MM:SS` local time. The date is deliberately absent: these lines are
/// read while something is going wrong now, and a full stamp on every row
/// would crowd out the message.
fn fmt_clock(unix_secs: u64) -> String {
use chrono::TimeZone;
// Saturating rather than `as`: that cast wraps a huge value into a
// negative one, which is a perfectly valid 1969 timestamp and would
// render as a plausible time instead of falling back.
let secs = i64::try_from(unix_secs).unwrap_or(i64::MAX);
match chrono::Local.timestamp_opt(secs, 0) {
chrono::LocalResult::Single(dt) => dt.format("%H:%M:%S").to_string(),
_ => "--:--:--".to_string(),
}
}
#[cfg(test)]
mod tests {
use super::{fmt_clock, keep};
use quicksearch_core::log::{Level, LogLine};
fn line(level: Level, text: &str) -> LogLine {
LogLine {
at: 1_700_000_000,
level,
text: text.to_string(),
}
}
#[test]
fn an_empty_filter_keeps_everything() {
assert!(keep(&line(Level::Info, "anything"), "", false));
assert!(keep(&line(Level::Warn, "anything"), "", false));
}
#[test]
fn the_filter_ignores_case_on_both_sides() {
let l = line(Level::Warn, "Warning: cannot read /Home/Photos");
assert!(keep(&l, "photos", false), "needle case must not matter");
assert!(keep(&l, "cannot read", false), "nor the line's");
assert!(!keep(&l, "videos", false));
}
#[test]
fn warnings_only_hides_informational_lines() {
assert!(!keep(&line(Level::Info, "shutting down"), "", true));
assert!(keep(&line(Level::Warn, "cannot read"), "", true));
}
/// Both filters apply, not either.
#[test]
fn the_two_filters_compose() {
assert!(!keep(&line(Level::Info, "cannot read"), "cannot", true));
assert!(!keep(&line(Level::Warn, "cannot read"), "missing", true));
assert!(keep(&line(Level::Warn, "cannot read"), "cannot", true));
}
#[test]
fn a_clock_stamp_is_fixed_width() {
assert_eq!(fmt_clock(0).len(), 8, "epoch renders as a time, not a date");
assert_eq!(fmt_clock(1_700_000_000).len(), 8);
}
/// Beyond what a local calendar can represent, the row still lines up.
#[test]
fn an_out_of_range_stamp_falls_back() {
assert_eq!(fmt_clock(u64::MAX), "--:--:--");
}
}

View file

@ -1,54 +1,95 @@
use std::sync::{Arc, OnceLock}; //! QuickSearch binary: `quicksearch <query>` searches from the terminal;
use dioxus::prelude::*; //! without a query it opens the egui desktop app.
use quicksearch_core::{config, indexing, shutdown}; //!
mod frontend; //! On Windows this is the GUI only, built as a window-subsystem app so no
mod search; //! console flashes behind it. Terminal search there is `quicksearch-cli`,
//! which is a console app and so keeps working pipes, exit codes, and a shell
//! that waits for it. A query passed here still does something useful: it
//! seeds the search box.
#![cfg_attr(windows, windows_subsystem = "windows")]
static INDEXING_SERVICE: OnceLock<Arc<indexing::IndexingService>> = OnceLock::new(); mod app;
mod backend;
#[cfg(not(windows))]
mod cli;
mod duplicates_tab;
mod format;
mod logs_tab;
mod manage_tab;
mod options;
mod platform;
mod query_highlight;
mod search_tab;
mod tracker;
use quicksearch_core::config::Config;
/// The window icon, shown in the titlebar, taskbar and alt-tab switcher.
///
/// X11 takes these pixels directly via `_NET_WM_ICON`. Wayland ignores them and
/// instead looks up the app id in `/usr/share/applications/`, so the id below has
/// to match the installed `quicksearch.desktop` for the icon to appear there.
fn app_icon() -> egui::IconData {
eframe::icon_data::from_png_bytes(include_bytes!("../assets/icons/quicksearch-256.png"))
.expect("bundled icon is a valid PNG")
}
/// Leftover positional arguments, joined — used to seed the search box.
///
/// Flags are dropped rather than parsed: eframe and winit take some of their
/// own, and a stray `--foo` should not end up in the query.
fn seed_query() -> Option<String> {
let terms: Vec<String> = std::env::args()
.skip(1)
.filter(|a| !a.starts_with('-'))
.collect();
if terms.is_empty() {
None
} else {
Some(terms.join(" "))
}
}
fn main() { fn main() {
let indexing_service = Arc::new(indexing::IndexingService::new()); // Must come first: anything below may print, and printing without a
INDEXING_SERVICE // stdio handle panics rather than failing quietly.
.set(indexing_service.clone()) #[cfg(windows)]
.expect("Failed to set global indexing service"); platform::redirect_null_stdio();
if let Err(e) = shutdown::install_signal_handler(indexing_service.clone()) { #[cfg(not(windows))]
eprintln!("Warning: failed to install signal handler: {}", e); if let Some(code) = cli::maybe_run_cli() {
std::process::exit(code);
} }
LaunchBuilder::desktop() // A broken config file should never keep the window from opening —
.with_cfg( // surface the error in-app and run on defaults.
dioxus_desktop::Config::new() let (config, config_error) = match Config::load() {
.with_custom_head(format!("<style>{}</style>", include_str!("../assets/styles.css"))) Ok(c) => (c, None),
.with_window(dioxus_desktop::WindowBuilder::new() Err(e) => (Config::default(), Some(e)),
.with_title("QuickSearch - File Indexer & Search") };
.with_resizable(true) let initial_query = seed_query();
.with_inner_size(dioxus_desktop::LogicalSize::new(1000.0, 700.0))
)
)
.launch(app);
}
let native_options = eframe::NativeOptions {
fn app() -> Element { viewport: egui::ViewportBuilder::default()
let cfg = match config::Config::load() { .with_title("QuickSearch")
Ok(c) => c, .with_app_id("quicksearch")
Err(e) => { .with_icon(app_icon())
eprintln!("Failed to load config: {}", e); .with_inner_size([1000.0, 700.0])
return rsx! { div { "Failed to load configuration" } }; .with_min_inner_size([640.0, 400.0]),
} ..Default::default()
}; };
let indexing_service = INDEXING_SERVICE let result = eframe::run_native(
.get() "QuickSearch",
.expect("Indexing service not initialized") native_options,
.clone(); Box::new(move |cc| {
app::QuickSearchApp::new(cc, config, config_error, initial_query)
rsx! { .map(|app| Box::new(app) as Box<dyn eframe::App>)
frontend::App { .map_err(|e| e.into())
indexing_service: indexing_service, }),
config: cfg );
} if let Err(e) = result {
eprintln!("failed to start GUI: {}", e);
std::process::exit(1);
} }
} }

View file

@ -0,0 +1,447 @@
//! The Manage Index tab: detailed status, mode controls, indexed roots,
//! and the content/ignore filter editors.
use quicksearch_core::config::Config;
use quicksearch_core::coordinator::{IndexMode, IndexerState, WatcherStatus};
use quicksearch_core::indexing::{IndexingStatus, RootPhase, RootProgress};
use crate::format::{fmt_interval, fmt_rate, group_thousands, middle_truncate};
use crate::options::{config_editor_ui, Section};
use crate::tracker::SpeedTracker;
/// What the tab asks the app to do after this frame.
#[derive(Default)]
pub struct ManageActions {
pub start_now: bool,
pub stop: bool,
pub auto: bool,
/// Ask the app to confirm and delete the index.
pub clear_index: bool,
/// A full edited config to apply (roots / filters / indexing knobs).
pub apply_config: Option<Config>,
}
pub struct ManageTab {
pub speed: SpeedTracker,
/// Multiline editors, one entry per line; synced from config on tab
/// entry and parsed back on Apply.
ext_filter_text: String,
ignore_filter_text: String,
new_root: String,
/// Inline error from a rejected root add (nested/duplicate).
root_error: Option<String>,
editors_synced: bool,
/// Draft of the indexing/processing knobs edited in-place.
draft: Option<Config>,
}
impl ManageTab {
pub fn new() -> ManageTab {
ManageTab {
speed: SpeedTracker::new(),
ext_filter_text: String::new(),
ignore_filter_text: String::new(),
new_root: String::new(),
root_error: None,
editors_synced: false,
draft: None,
}
}
/// Feed the tracker from the polled status (called every frame, on
/// every tab, so the status bar rate stays live).
pub fn observe(&mut self, status: &IndexingStatus) {
match status {
IndexingStatus::Running { roots, .. } => {
// Monotonic within a run: walks and extractions only grow.
let total: usize = roots.iter().map(|r| r.walked + r.extracted).sum();
self.speed.record(total);
}
IndexingStatus::Idle | IndexingStatus::Error(_) => self.speed.reset(),
_ => {}
}
}
fn sync_editors(&mut self, config: &Config) {
if !self.editors_synced {
self.ext_filter_text = config.indexing.content_extensions.join("\n");
self.ignore_filter_text = config.indexing.ignore_patterns.join("\n");
self.draft = Some(config.clone());
self.editors_synced = true;
}
}
/// Force a re-sync next frame (config changed elsewhere).
pub fn invalidate_editors(&mut self) {
self.editors_synced = false;
}
pub fn ui(
&mut self,
ui: &mut egui::Ui,
state: &IndexerState,
config: &Config,
) -> ManageActions {
let mut actions = ManageActions::default();
self.sync_editors(config);
egui::ScrollArea::vertical().auto_shrink([false; 2]).show(ui, |ui| {
// --- Status ---------------------------------------------------
ui.heading("Status");
status_panel(ui, state, &self.speed);
watch_panel(ui, state, config);
ui.add_space(8.0);
// --- Controls -------------------------------------------------
ui.horizontal(|ui| {
let running = !matches!(
state.activity,
IndexingStatus::Idle | IndexingStatus::Error(_)
);
if ui.add_enabled(!running, egui::Button::new("Start indexing now")).clicked() {
actions.start_now = true;
}
if ui.add_enabled(running || state.mode == IndexMode::Auto, egui::Button::new("Stop")).clicked() {
actions.stop = true;
}
if ui
.add_enabled(state.mode != IndexMode::Auto, egui::Button::new("Return to Automatic"))
.clicked()
{
actions.auto = true;
}
let mode = match state.mode {
IndexMode::Auto => "Automatic",
IndexMode::ManualStopped => "Manual (stopped)",
IndexMode::ManualRunning => "Manual (running)",
};
ui.label(egui::RichText::new(format!("Mode: {}", mode)).weak());
ui.separator();
if ui
.button(egui::RichText::new("Clear index…").color(ui.visuals().error_fg_color))
.on_hover_text("Delete the index database (asks for confirmation)")
.clicked()
{
actions.clear_index = true;
}
if state.queued_events > 0 {
ui.label(
egui::RichText::new(format!("{} changes queued", state.queued_events))
.small()
.weak(),
);
}
});
ui.separator();
// --- Indexed roots ---------------------------------------------
ui.heading("Indexed folders");
let draft = self.draft.as_mut().expect("synced");
let mut remove: Option<usize> = None;
for (i, root) in draft.paths.indexing_paths.clone().iter().enumerate() {
ui.horizontal(|ui| {
// Controls claim the right edge first so a long path can
// never push them out of view; the path truncates into
// whatever width remains (full path on hover).
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if ui.small_button("Remove").clicked() {
remove = Some(i);
}
// Per-root walker override; 0 = auto (4 local / 16
// network, detected per root). Applies on the next run.
let mut workers =
draft.indexing.root_workers.get(root).copied().unwrap_or(0);
let response = ui
.add(
egui::DragValue::new(&mut workers)
.range(0..=64)
.custom_formatter(|n, _| {
if n == 0.0 {
"auto".to_string()
} else {
format!("{:.0}", n)
}
})
.custom_parser(|s| {
let s = s.trim();
if s.is_empty() || s.eq_ignore_ascii_case("auto") {
Some(0.0)
} else {
s.parse().ok()
}
}),
)
.on_hover_text(
"Walker threads for this folder. auto = 4 on local \
storage, 16 on network mounts. Takes effect on \
the next indexing run.",
);
if response.changed() {
if workers == 0 {
draft.indexing.root_workers.remove(root);
} else {
draft.indexing.root_workers.insert(root.clone(), workers);
}
}
ui.label(egui::RichText::new("workers:").small().weak());
// Path label takes the leftover width, middle-truncated.
ui.with_layout(
egui::Layout::left_to_right(egui::Align::Center),
|ui| {
let font_id = egui::TextStyle::Monospace.resolve(ui.style());
let char_width =
ui.fonts(|f| f.glyph_width(&font_id, '0')).max(1.0);
let budget =
((ui.available_width() / char_width) as usize).max(16);
ui.monospace(middle_truncate(root, budget))
.on_hover_text(root);
},
);
});
});
}
if let Some(i) = remove {
let removed = draft.paths.indexing_paths.remove(i);
draft.indexing.root_workers.remove(&removed);
}
ui.horizontal(|ui| {
if ui.button("Add folder…").clicked() {
if let Some(dir) = rfd::FileDialog::new().pick_folder() {
let path = dir.to_string_lossy().into_owned();
try_add_root(draft, path, &mut self.root_error);
}
}
ui.add(
egui::TextEdit::singleline(&mut self.new_root)
.desired_width(240.0)
.hint_text("or type a path"),
);
if ui.button("Add").clicked() && !self.new_root.trim().is_empty() {
let path = self.new_root.trim().to_string();
if try_add_root(draft, path, &mut self.root_error) {
self.new_root.clear();
}
}
});
if let Some(err) = &self.root_error {
ui.colored_label(ui.visuals().error_fg_color, err);
}
ui.separator();
// --- Filters ---------------------------------------------------
ui.heading("Content filters");
ui.columns(2, |cols| {
cols[0].label("Full-text extensions (empty = all supported):");
cols[0].add(
egui::TextEdit::multiline(&mut self.ext_filter_text)
.desired_rows(4)
.desired_width(f32::INFINITY)
.hint_text("txt\nmd\npdf"),
);
cols[1].label("Ignore patterns (excluded entirely):");
cols[1].add(
egui::TextEdit::multiline(&mut self.ignore_filter_text)
.desired_rows(4)
.desired_width(f32::INFINITY)
.hint_text(".git\nnode_modules\n*.tmp"),
);
});
ui.separator();
// --- Indexing options -------------------------------------------
ui.heading("Indexing options");
config_editor_ui(ui, draft, Section::Indexing);
ui.add_space(4.0);
config_editor_ui(ui, draft, Section::Processing);
ui.add_space(8.0);
if ui.button("Apply & Save").clicked() {
let mut new_config = draft.clone();
new_config.indexing.content_extensions = parse_lines(&self.ext_filter_text);
new_config.indexing.ignore_patterns = parse_lines(&self.ignore_filter_text);
let roots = new_config.paths.indexing_paths.clone();
new_config
.indexing
.root_workers
.retain(|root, _| roots.contains(root));
actions.apply_config = Some(new_config);
self.editors_synced = false;
}
});
actions
}
}
/// Append a root to the draft unless it would duplicate or nest with an
/// existing one; the rejection reason lands in `error`.
fn try_add_root(draft: &mut Config, candidate: String, error: &mut Option<String>) -> bool {
if draft.paths.indexing_paths.contains(&candidate) {
*error = Some(format!("{} is already in the list", candidate));
return false;
}
let mut probe = draft.paths.indexing_paths.clone();
probe.push(candidate.clone());
if let Some((child, parent)) = quicksearch_core::config::nested_roots(&probe).first() {
*error = Some(format!(
"Not added: {} is nested under {}; indexed folders may not overlap",
child, parent
));
return false;
}
draft.paths.indexing_paths.push(candidate);
*error = None;
true
}
fn parse_lines(text: &str) -> Vec<String> {
text.lines()
.map(str::trim)
.filter(|l| !l.is_empty())
.map(str::to_string)
.collect()
}
/// Live-update health. Permanent counterpart to the one-time modal: the
/// modal is dismissed and remembered per root, but "live updates are off"
/// stays true and must remain discoverable.
fn watch_panel(ui: &mut egui::Ui, state: &IndexerState, config: &Config) {
match &state.watcher {
// Manual mode already says "stopped" in the controls row; repeating
// it here would be noise.
WatcherStatus::Off => {}
WatcherStatus::Starting => {
ui.label(
egui::RichText::new("Setting up live updates…")
.small()
.weak(),
);
}
WatcherStatus::Active { dirs } => {
ui.label(
egui::RichText::new(format!(
"Live updates on, watching {} folders",
group_thousands(*dirs as u64)
))
.small()
.weak(),
);
}
WatcherStatus::Disabled { reason } => {
ui.colored_label(
ui.visuals().warn_fg_color,
format!(
"⚠ Live updates off; reindexing every {}",
fmt_interval(config.indexing.reindex_interval_minutes)
),
)
.on_hover_text(reason.to_string());
}
}
}
fn status_panel(ui: &mut egui::Ui, state: &IndexerState, speed: &SpeedTracker) {
match &state.activity {
IndexingStatus::Idle => {
// Relative wording makes even a milliseconds-fast run visibly
// register ("just now") instead of looking like a dead button.
let last = state
.last_full_index
.map(crate::format::fmt_ago)
.unwrap_or_else(|| "never".to_string());
ui.label(format!("Idle; last full index: {}", last));
}
IndexingStatus::Error(e) => {
ui.colored_label(ui.visuals().error_fg_color, format!("Error: {}", e));
}
IndexingStatus::Stopping => {
ui.label("Stopping…");
}
IndexingStatus::Running { roots, .. } => {
for root in roots {
root_row(ui, root);
}
if let Some(rate) = speed.files_per_sec() {
ui.label(
egui::RichText::new(format!("overall: {}", fmt_rate(rate)))
.small()
.weak(),
);
}
}
}
}
/// One root's progress: path, phase, bar, counters, current file.
fn root_row(ui: &mut egui::Ui, r: &RootProgress) {
// Weak "|" separators split the row into folder | status | numbers.
let divider = |ui: &mut egui::Ui| {
ui.label(egui::RichText::new("|").weak());
};
ui.horizontal(|ui| {
ui.monospace(middle_truncate(&r.root, 48));
divider(ui);
match r.phase {
RootPhase::Walking => {
ui.label("indexing");
divider(ui);
let workers = format!("{}/{} workers", r.active_workers, r.total_workers);
match r.walk_total {
Some(total) if total > 0 => {
let frac = (r.walked as f32 / total as f32).clamp(0.0, 1.0);
ui.label(format!(
"{} / {} ({:.0}%) · {}",
group_thousands(r.walked as u64),
group_thousands(total as u64),
frac * 100.0,
workers
));
ui.add(egui::ProgressBar::new(frac).desired_width(160.0));
}
_ => {
ui.label(format!(
"{} files · {}",
group_thousands(r.walked as u64),
workers
));
ui.add(egui::ProgressBar::new(0.0).animate(true).desired_width(160.0));
}
}
}
RootPhase::Extracting => {
ui.label("extracting text for search");
divider(ui);
let frac = if r.extract_total > 0 {
(r.extracted as f32 / r.extract_total as f32).clamp(0.0, 1.0)
} else {
1.0
};
ui.label(format!(
"{} / {} ({:.0}%)",
group_thousands(r.extracted as u64),
group_thousands(r.extract_total as u64),
frac * 100.0
));
ui.add(egui::ProgressBar::new(frac).desired_width(160.0));
}
RootPhase::Done => {
// Whole-root totals: `walked` covers every file the walk
// saw (including unchanged, skipped ones) and `extracted`
// covers all rows with searchable text, not just this
// run's new work.
ui.label("done");
divider(ui);
ui.label(format!(
"indexed {}, extracted {}",
group_thousands(r.walked as u64),
group_thousands(r.extracted as u64)
));
ui.add(egui::ProgressBar::new(1.0).desired_width(160.0));
}
}
});
if let Some(f) = &r.current_file {
ui.label(egui::RichText::new(middle_truncate(f, 90)).small().weak());
}
}

View file

@ -0,0 +1,251 @@
//! The Options window and the shared config editor used by both the
//! window and the Manage Index tab. Edits happen on a draft; Apply
//! validates, saves, and hands the new config to the app.
use quicksearch_core::config::Config;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Section {
Indexing,
Processing,
Search,
}
pub struct OptionsWindow {
pub open: bool,
draft: Option<Config>,
}
impl OptionsWindow {
pub fn new() -> OptionsWindow {
OptionsWindow {
open: false,
draft: None,
}
}
pub fn open_with(&mut self, current: &Config) {
self.open = true;
self.draft = Some(current.clone());
}
/// Render; returns a new config when the user applied changes.
pub fn ui(&mut self, ctx: &egui::Context, current: &Config) -> Option<Config> {
if !self.open {
self.draft = None;
return None;
}
if self.draft.is_none() {
self.draft = Some(current.clone());
}
let mut applied = None;
let mut open = self.open;
let draft = self.draft.as_mut().unwrap();
egui::Window::new("Options")
.open(&mut open)
.resizable(false)
.default_width(420.0)
.show(ctx, |ui| {
egui::ScrollArea::vertical().max_height(480.0).show(ui, |ui| {
ui.heading("Paths");
egui::Grid::new("opt-paths").num_columns(2).show(ui, |ui| {
ui.label("Database file");
ui.add(
egui::TextEdit::singleline(&mut draft.paths.database_path)
.desired_width(260.0),
);
ui.end_row();
});
ui.label(
egui::RichText::new(
"Indexed folders are managed on the Manage Index tab.",
)
.small()
.weak(),
);
ui.separator();
ui.heading("Indexing");
config_editor_ui(ui, draft, Section::Indexing);
ui.separator();
ui.heading("Processing");
config_editor_ui(ui, draft, Section::Processing);
ui.separator();
ui.heading("Search");
config_editor_ui(ui, draft, Section::Search);
ui.separator();
ui.heading("Interface");
egui::Grid::new("opt-ui").num_columns(2).show(ui, |ui| {
ui.label("UI scale");
ui.add(
egui::Slider::new(&mut draft.ui.scale, 0.5..=2.5)
.step_by(0.05)
.fixed_decimals(2),
)
.on_hover_text(
"Zooms the whole interface: fonts, spacing, and \
widgets. Ctrl +/- and Ctrl 0 adjust it temporarily \
at runtime.",
);
ui.end_row();
});
});
ui.separator();
ui.horizontal(|ui| {
if ui.button("Apply & Save").clicked() {
applied = Some(draft.clone());
}
ui.label(
egui::RichText::new(
"Changes to tokenizer, filters, hidden files, or hashing \
prompt an index rebuild.",
)
.small()
.weak(),
);
});
});
self.open = open;
if !self.open {
self.draft = None;
}
applied
}
}
/// One implementation of the per-section config controls, shared by the
/// Options window and the Manage tab.
pub fn config_editor_ui(ui: &mut egui::Ui, config: &mut Config, section: Section) {
match section {
Section::Indexing => {
egui::Grid::new("cfg-indexing").num_columns(2).show(ui, |ui| {
ui.label("Automatic indexing");
ui.checkbox(&mut config.indexing.auto_index, "watchers + periodic reindex");
ui.end_row();
ui.label("Full reindex every");
ui.horizontal(|ui| {
ui.add(
egui::DragValue::new(&mut config.indexing.reindex_interval_minutes)
.range(5..=60 * 24 * 30),
);
ui.label("minutes");
});
ui.end_row();
ui.label("Follow symlinks");
ui.checkbox(&mut config.indexing.follow_symlinks, "");
ui.end_row();
ui.label("Include hidden files");
ui.checkbox(&mut config.indexing.include_hidden, "");
ui.end_row();
});
}
Section::Processing => {
egui::Grid::new("cfg-processing").num_columns(2).show(ui, |ui| {
ui.label("Tokenizer");
egui::ComboBox::from_id_salt("cfg-tokenize")
.selected_text(&config.processing.tokenize)
.show_ui(ui, |ui| {
for opt in ["trigram", "unicode61", "porter"] {
ui.selectable_value(
&mut config.processing.tokenize,
opt.to_string(),
opt,
);
}
});
ui.end_row();
ui.label("Hash sample size (bytes)");
ui.add(egui::DragValue::new(&mut config.processing.hash_length).range(512..=1_048_576));
ui.end_row();
ui.label("Max stored text (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_size)
.range(1024..=16_777_216),
);
ui.end_row();
ui.label("Max text file size (bytes)");
ui.add(
egui::DragValue::new(&mut config.processing.maximum_text_file_size)
.range(1024..=1_073_741_824),
);
ui.end_row();
ui.label("Batch size");
ui.add(egui::DragValue::new(&mut config.processing.batch_size).range(10..=100_000));
ui.end_row();
ui.label("Store text for snippets");
ui.checkbox(&mut config.processing.store_text_for_snippets, "")
.on_hover_text(
"Off: smaller index, but no previews, occurrence ranking, \
case verification, or fuzzy full-text search",
);
ui.end_row();
});
}
Section::Search => {
egui::Grid::new("cfg-search").num_columns(2).show(ui, |ui| {
ui.label("Fuzzy stages on by default");
ui.checkbox(&mut config.search.fuzzy_default, "");
ui.end_row();
ui.label("Fuzzy edit distance");
ui.vertical(|ui| {
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
.on_hover_text(
"Ceiling on the typo budget. The allowance grows with the \
search term, one edit per three characters, up to this \
value, so 2 means \"1 edit for short terms, 2 for longer \
ones\". 0 turns the fuzzy stages off.",
);
if let Some(warning) = config.search.fuzzy_edits_warning() {
ui.label(
egui::RichText::new(warning)
.small()
.color(egui::Color32::from_rgb(220, 150, 40)),
);
}
});
ui.end_row();
ui.label("Display limit");
ui.add(egui::DragValue::new(&mut config.search.display_limit).range(50..=100_000));
ui.end_row();
ui.label("Stream batch size");
ui.add(
egui::DragValue::new(&mut config.search.results_per_page).range(10..=10_000),
);
ui.end_row();
ui.label("Debounce (ms)");
ui.add(egui::DragValue::new(&mut config.search.debounce_ms).range(0..=2000));
ui.end_row();
ui.label("Fuzzy max edits");
ui.add(egui::DragValue::new(&mut config.search.fuzzy_max_edits).range(0..=8))
.on_hover_text(
"Ceiling on fuzzy edit distance (the budget grows one \
edit per three characters of the term). 0 disables \
the fuzzy passes.",
);
ui.end_row();
});
if let Some(warning) = config.search.fuzzy_edits_warning() {
ui.colored_label(ui.visuals().warn_fg_color, warning);
}
}
}
}

View file

@ -0,0 +1,134 @@
//! Opening files, revealing them in the system file manager, and the one bit
//! of process setup that has to happen before anything prints.
use std::process::Command;
/// Give the process somewhere to write when it has no stdio.
///
/// A window-subsystem binary launched from Explorer has NULL standard handles,
/// and `println!`/`eprintln!` *panic* when the write fails rather than
/// dropping the output. Pointing the handles at `NUL` makes those writes
/// succeed and go nowhere.
///
/// Background reporting no longer depends on this — it goes through
/// [`quicksearch_core::log`], which ignores a failed stderr write and keeps
/// the line for the Logs tab — but the remaining direct prints (a startup
/// failure, a panic message) still reach a handle that accepts them.
///
/// Handles inherited from a real console are left alone, so running the binary
/// from a shell still prints normally.
#[cfg(windows)]
pub fn redirect_null_stdio() {
use std::os::windows::io::IntoRawHandle;
use windows_sys::Win32::Foundation::INVALID_HANDLE_VALUE;
use windows_sys::Win32::System::Console::{
GetStdHandle, SetStdHandle, STD_ERROR_HANDLE, STD_OUTPUT_HANDLE,
};
for id in [STD_OUTPUT_HANDLE, STD_ERROR_HANDLE] {
let existing = unsafe { GetStdHandle(id) };
if !existing.is_null() && existing != INVALID_HANDLE_VALUE {
continue;
}
if let Ok(file) = std::fs::OpenOptions::new().write(true).open("NUL") {
// Deliberately leaked: the handle has to outlive every later
// write, which means the whole process.
unsafe { SetStdHandle(id, file.into_raw_handle() as _) };
}
}
}
/// Open a file with the desktop's default application, detached.
pub fn open_file(path: &str) {
if let Err(e) = open::that_detached(path) {
quicksearch_core::log_warn!("open {}: {}", path, e);
}
}
/// Reveal a file in the system file manager with the file selected.
///
/// Linux: `org.freedesktop.FileManager1.ShowItems` over the session bus
/// (supported by every mainstream file manager) via `dbus-send` — no
/// D-Bus library dependency for one call. Falls back to opening the
/// parent directory. Windows/macOS use their native select verbs.
pub fn reveal_in_folder(path: &str) {
#[cfg(target_os = "windows")]
{
use std::os::windows::process::CommandExt;
/// Keep a console window from flashing behind the spawn.
const CREATE_NO_WINDOW: u32 = 0x0800_0000;
// explorer.exe parses its own command line rather than using the
// standard argv splitting, and wants `/select,` glued to the path as a
// single token with quotes around the path only. Passed as two
// arguments it ignores the selection and just opens the folder, and
// std's quoting would wrap the whole token. `raw_arg` is the only way
// to say exactly this.
//
// Forward slashes are valid everywhere else on Windows but not here,
// so normalize first. The exit code is not worth checking: explorer
// returns 1 even on success.
let native = path.replace('/', "\\");
let _ = Command::new("explorer.exe")
.raw_arg(format!("/select,\"{}\"", native))
.creation_flags(CREATE_NO_WINDOW)
.spawn();
return;
}
#[cfg(target_os = "macos")]
{
let _ = Command::new("open").arg("-R").arg(path).spawn();
return;
}
#[cfg(all(unix, not(target_os = "macos")))]
{
use std::path::Path;
let uri = format!("file://{}", uri_escape_path(path));
let ok = Command::new("dbus-send")
.args([
"--session",
"--print-reply",
"--dest=org.freedesktop.FileManager1",
"/org/freedesktop/FileManager1",
"org.freedesktop.FileManager1.ShowItems",
&format!("array:string:{}", uri),
"string:",
])
.output()
.map(|out| out.status.success())
.unwrap_or(false);
if !ok {
let parent = Path::new(path).parent().unwrap_or(Path::new("/"));
let _ = Command::new("xdg-open").arg(parent).spawn();
}
}
}
/// Percent-encode a filesystem path for a file:// URI, keeping `/`.
#[cfg(all(unix, not(target_os = "macos")))]
fn uri_escape_path(path: &str) -> String {
let mut out = String::with_capacity(path.len());
for byte in path.bytes() {
match byte {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'/' | b'-' | b'_' | b'.' | b'~' => {
out.push(byte as char)
}
_ => out.push_str(&format!("%{:02X}", byte)),
}
}
out
}
#[cfg(test)]
mod tests {
#[cfg(all(unix, not(target_os = "macos")))]
#[test]
fn uri_escaping() {
use super::uri_escape_path;
assert_eq!(uri_escape_path("/plain/path.txt"), "/plain/path.txt");
assert_eq!(
uri_escape_path("/with space/ünïcode&.txt"),
"/with%20space/%C3%BCn%C3%AFcode%26.txt"
);
}
}

View file

@ -0,0 +1,739 @@
//! Syntax highlighting for the search box.
//!
//! [`classify`] is a pure token walk over [`tokenize_spanned`] output that
//! mirrors `split_for_cascade` branch for branch — it must never claim
//! something is a filter (or a wildcard) that the engine treats as plain
//! text. The egui layer at the bottom turns its segments into a `Galley`
//! for `TextEdit::layouter`.
//!
//! Color scheme: recognized keywords red, their arguments blue, syntax
//! characters (operators, quotes, live wildcards) green, invalid arguments
//! in the error color, everything else plain. A complete recognized filter
//! additionally gets a tinted background chip.
use std::ops::Range;
use std::sync::Arc;
use egui::text::{LayoutJob, TextFormat};
use egui::{Color32, Galley, Stroke};
use quicksearch_core::query::ast::Op;
use quicksearch_core::query::lexer::{tokenize_spanned, Token};
use quicksearch_core::query::pattern::RegexQuery;
use quicksearch_core::query::translator::{build_filter, is_filter_key};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Class {
Plain,
/// The key word of a recognized filter (`type`, `name`, `regex`, …).
Keyword,
/// Syntax characters doing work: filter operators (`:`, `:>=`, …),
/// quote delimiters, and `*` where it is a live wildcard.
Operator,
/// The value of a recognized filter.
Argument,
/// The value of a recognized filter that the engine would reject
/// (unknown type name, bad date, invalid regex).
InvalidArg,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Seg {
pub range: Range<usize>,
pub class: Class,
/// Part of a complete recognized filter — drawn on the chip tint.
pub chip: bool,
}
/// Classify `text` into contiguous segments tiling `0..text.len()`.
pub fn classify(text: &str) -> Vec<Seg> {
let (tokens, err) = tokenize_spanned(text);
let mut em = Emitter {
text,
cursor: 0,
segs: Vec::new(),
};
let tok = |i: usize| tokens.get(i).map(|(t, _)| t);
let span = |i: usize| tokens[i].1.clone();
let mut i = 0usize;
// The engine allows one `regex:` per query; later ones are errors.
let mut regex_seen = false;
while i < tokens.len() {
match &tokens[i].0 {
Token::Word(word) => {
if let Some(Token::Op(op1)) = tok(i + 1) {
// Candidate filter: Word(key) Op [Op] (Word|Quoted),
// exactly as split_for_cascade sees it.
let (op, op_end_idx, value_idx) = match tok(i + 2) {
Some(Token::Op(op2)) => (*op2, i + 2, i + 3),
_ => (*op1, i + 1, i + 2),
};
let value = match tok(value_idx) {
Some(Token::Word(v)) | Some(Token::Quoted(v)) => Some(v.clone()),
_ => None,
};
let is_regex = word.eq_ignore_ascii_case("regex");
if let Some(value) = value {
let value_is_word = matches!(tok(value_idx), Some(Token::Word(_)));
if is_regex || is_filter_key(word) {
let valid = if is_regex {
let first = !regex_seen;
regex_seen = true;
first
&& op == Op::Contains
&& RegexQuery::new(&value).is_ok()
} else {
build_filter(word, op, &value, value_is_word).is_ok()
};
em.emit(span(i), Class::Keyword, true, false);
for op_idx in (i + 1)..=op_end_idx {
em.emit(span(op_idx), Class::Operator, true, true);
}
let vspan = span(value_idx);
if !valid {
// One uniform error run reads better than
// error-with-green-sprinkles.
em.emit(vspan, Class::InvalidArg, true, true);
} else if !value_is_word {
em.emit_quoted(vspan, Class::Argument, true);
} else if glob_value_key(word) {
em.emit_word(vspan, Class::Argument, true, true);
} else {
// Stars in other filter values are literal
// characters — no wildcard color.
em.emit(vspan, Class::Argument, true, true);
}
i = value_idx + 1;
continue;
}
// Unrecognized key: the engine reassembles the whole
// chain verbatim (stars stay literal), so everything
// renders plain — that absence of color is how the
// user learns `foo:` is not a filter.
em.emit(span(i), Class::Plain, false, false);
for op_idx in (i + 1)..=op_end_idx {
em.emit(span(op_idx), Class::Plain, false, false);
}
em.emit_glued_value(span(value_idx), tok(value_idx));
i = value_idx + 1;
while let Some(Token::Op(_)) = tok(i) {
em.emit(span(i), Class::Plain, false, false);
i += 1;
if let Some(Token::Word(_)) | Some(Token::Quoted(_)) = tok(i) {
em.emit_glued_value(span(i), tok(i));
i += 1;
}
}
continue;
}
// Key + op with no value yet (mid-typing `type:`).
// Recognized keys color optimistically — instant
// feedback that the key landed — but earn no chip
// until the filter is complete.
let known = is_regex || is_filter_key(word);
let (key_class, op_class) = if known {
(Class::Keyword, Class::Operator)
} else {
(Class::Plain, Class::Plain)
};
em.emit(span(i), key_class, false, false);
for op_idx in (i + 1)..=op_end_idx {
em.emit(span(op_idx), op_class, false, false);
}
i = op_end_idx + 1;
continue;
}
// A plain word: unquoted stars are live wildcards.
em.emit_word(span(i), Class::Plain, false, false);
}
Token::Quoted(_) => em.emit_quoted(span(i), Class::Plain, false),
// Demoted to plain text by the live search path — coloring
// them as operators would lie.
Token::And | Token::Or | Token::LParen | Token::RParen | Token::Op(_) => {
em.emit(span(i), Class::Plain, false, false);
}
}
i += 1;
}
// Trailing lex error: an unterminated quote is a quote-in-progress,
// not a mistake — green delimiter, plain tail.
if let Some(err) = err {
if err.offset < text.len() && text.as_bytes()[err.offset] == b'"' {
em.emit(err.offset..err.offset + 1, Class::Operator, false, false);
}
}
em.finish(text.len())
}
/// Keys whose word-form values interpret `*` as a wildcard (or, for
/// `regex`, as live pattern syntax).
fn glob_value_key(key: &str) -> bool {
matches!(
key.to_ascii_lowercase().as_str(),
"name" | "filename" | "regex"
)
}
struct Emitter<'a> {
text: &'a str,
cursor: usize,
segs: Vec<Seg>,
}
impl Emitter<'_> {
/// Fill the gap (whitespace the lexer skipped) up to `pos`.
fn gap_to(&mut self, pos: usize, chip: bool) {
if pos > self.cursor {
self.segs.push(Seg {
range: self.cursor..pos,
class: Class::Plain,
chip,
});
self.cursor = pos;
}
}
/// Emit one span. `gap_chip` tints the whitespace before it — true for
/// the interior of a filter (`type : Audio` chips as one run).
fn emit(&mut self, range: Range<usize>, class: Class, chip: bool, gap_chip: bool) {
self.gap_to(range.start, gap_chip);
if range.end > range.start {
self.segs.push(Seg {
range: range.clone(),
class,
chip,
});
self.cursor = range.end;
}
}
/// Emit a word span with each `*` as a green wildcard and the pieces
/// between in `base`.
fn emit_word(&mut self, range: Range<usize>, base: Class, chip: bool, gap_chip: bool) {
self.gap_to(range.start, gap_chip);
let bytes = self.text.as_bytes();
let mut piece_start = range.start;
for pos in range.clone() {
if bytes[pos] == b'*' {
if pos > piece_start {
self.segs.push(Seg {
range: piece_start..pos,
class: base,
chip,
});
}
self.segs.push(Seg {
range: pos..pos + 1,
class: Class::Operator,
chip,
});
piece_start = pos + 1;
}
}
if range.end > piece_start {
self.segs.push(Seg {
range: piece_start..range.end,
class: base,
chip,
});
}
self.cursor = self.cursor.max(range.end);
}
/// Emit a quoted span (delimiters included): quotes green, content in
/// `content`. Inner `""` escapes are just content bytes — no offset
/// math needed.
fn emit_quoted(&mut self, range: Range<usize>, content: Class, chip: bool) {
self.gap_to(range.start, chip);
self.segs.push(Seg {
range: range.start..range.start + 1,
class: Class::Operator,
chip,
});
if range.end - range.start > 2 {
self.segs.push(Seg {
range: range.start + 1..range.end - 1,
class: content,
chip,
});
}
if range.end - range.start >= 2 {
self.segs.push(Seg {
range: range.end - 1..range.end,
class: Class::Operator,
chip,
});
}
self.cursor = self.cursor.max(range.end);
}
/// A value inside unrecognized-key glue: plain, except quote
/// delimiters, which still did real tokenizing work.
fn emit_glued_value(&mut self, range: Range<usize>, token: Option<&Token>) {
match token {
Some(Token::Quoted(_)) => self.emit_quoted(range, Class::Plain, false),
_ => self.emit(range, Class::Plain, false, false),
}
}
fn finish(mut self, len: usize) -> Vec<Seg> {
self.gap_to(len, false);
self.segs
}
}
// ---------------------------------------------------------------------------
// egui layer
// ---------------------------------------------------------------------------
struct QueryPalette {
keyword: Color32,
argument: Color32,
operator: Color32,
}
/// GitHub Primer syntax colors — readable on egui's near-black and white
/// text-field backgrounds. Same convention as `rank_tier_color`.
fn query_palette(dark_mode: bool) -> QueryPalette {
if dark_mode {
QueryPalette {
keyword: Color32::from_rgb(255, 123, 114),
argument: Color32::from_rgb(121, 192, 255),
operator: Color32::from_rgb(126, 231, 135),
}
} else {
QueryPalette {
keyword: Color32::from_rgb(207, 34, 46),
argument: Color32::from_rgb(5, 80, 174),
operator: Color32::from_rgb(26, 127, 55),
}
}
}
struct QueryFormats {
plain: TextFormat,
keyword: TextFormat,
operator: TextFormat,
argument: TextFormat,
invalid: TextFormat,
chip_bg: Color32,
}
fn query_formats(ui: &egui::Ui) -> QueryFormats {
let font_id = egui::TextStyle::Body.resolve(ui.style());
let palette = query_palette(ui.visuals().dark_mode);
let base = |color: Color32| TextFormat {
font_id: font_id.clone(),
color,
..Default::default()
};
let error = ui.visuals().error_fg_color;
QueryFormats {
plain: base(ui.visuals().text_color()),
keyword: base(palette.keyword),
operator: base(palette.operator),
argument: base(palette.argument),
// The keyword red and the error red are near neighbors in dark
// mode; the underline disambiguates at a glance.
invalid: TextFormat {
underline: Stroke::new(1.0, error),
..base(error)
},
// Slightly weaker than the snippet highlight's 0.4 so the colored
// text on top stays crisp.
chip_bg: ui.visuals().selection.bg_fill.gamma_multiply(0.35),
}
}
impl QueryFormats {
fn format_for(&self, seg: &Seg) -> TextFormat {
let mut fmt = match seg.class {
Class::Plain => self.plain.clone(),
Class::Keyword => self.keyword.clone(),
Class::Operator => self.operator.clone(),
Class::Argument => self.argument.clone(),
Class::InvalidArg => self.invalid.clone(),
};
if seg.chip {
fmt.background = self.chip_bg;
}
fmt
}
}
/// Classification cache: tokenizing is cheap but validating a `regex:`
/// argument compiles the regex, and the layouter runs every frame — so
/// segments are recomputed only when the text changes. The `LayoutJob` is
/// rebuilt each frame (colors follow the live theme) and epaint's own
/// galley cache dedupes the actual layout work by job hash.
#[derive(Default)]
pub struct HighlightCache {
text: String,
segs: Vec<Seg>,
}
pub fn galley(ui: &egui::Ui, cache: &mut HighlightCache, text: &str) -> Arc<Galley> {
if cache.text != text {
cache.text = text.to_owned();
cache.segs = classify(text);
}
let fmts = query_formats(ui);
let mut job = LayoutJob::default();
for seg in &cache.segs {
job.append(&text[seg.range.clone()], 0.0, fmts.format_for(seg));
}
ui.fonts(|f| f.layout_job(job))
}
#[cfg(test)]
mod tests {
use super::*;
/// Readable projection: (slice, class, chip) per segment.
fn segs(text: &str) -> Vec<(String, Class, bool)> {
assert_tiles(text);
classify(text)
.into_iter()
.map(|s| (text[s.range.clone()].to_string(), s.class, s.chip))
.collect()
}
/// Segments must tile 0..len exactly: contiguous, ascending, complete.
fn assert_tiles(text: &str) {
let segs = classify(text);
let mut cursor = 0usize;
for s in &segs {
assert_eq!(s.range.start, cursor, "gap or overlap in {:?}: {:?}", text, segs);
assert!(s.range.end > s.range.start, "empty seg in {:?}", text);
cursor = s.range.end;
}
assert_eq!(cursor, text.len(), "segments must cover {:?}", text);
}
use Class::*;
fn owned(v: Vec<(&str, Class, bool)>) -> Vec<(String, Class, bool)> {
v.into_iter().map(|(s, c, b)| (s.to_string(), c, b)).collect()
}
#[test]
fn empty_input_yields_no_segments() {
assert!(classify("").is_empty());
}
#[test]
fn plain_words_stay_plain() {
assert_eq!(
segs("budget report"),
owned(vec![
("budget", Plain, false),
(" ", Plain, false),
("report", Plain, false),
])
);
}
#[test]
fn every_recognized_filter_chips() {
for input in [
"type:Audio",
"modified:>=2024-01-01",
"mtime:<2023-12-01",
"path:/home/me",
"folder:/x",
"includefolder:/x",
"name:report",
"filename:report",
"mime:application/pdf",
"regex:foo",
] {
let all = segs(input);
assert!(
all.iter().all(|(_, _, chip)| *chip),
"{:?}: whole filter must chip: {:?}",
input,
all
);
assert_eq!(all[0].1, Keyword, "{:?}", input);
assert_eq!(all[1].1, Operator, "{:?}", input);
assert!(
all[2..].iter().all(|(_, c, _)| *c == Argument || *c == Operator),
"{:?}: {:?}",
input,
all
);
}
}
#[test]
fn keys_are_case_insensitive() {
assert_eq!(segs("TYPE:Audio")[0], ("TYPE".to_string(), Keyword, true));
assert_eq!(
segs("Modified:>=2024-01-01")[0],
("Modified".to_string(), Keyword, true)
);
}
#[test]
fn colon_comparator_runs_are_one_green_stretch() {
assert_eq!(
segs("modified:>=2024-01-01"),
owned(vec![
("modified", Keyword, true),
(":", Operator, true),
(">=", Operator, true),
("2024-01-01", Argument, true),
])
);
}
#[test]
fn unrecognized_keys_stay_plain() {
for input in ["foo:bar", "12:30", "foo:bar:baz"] {
assert!(
segs(input).iter().all(|(_, c, chip)| *c == Plain && !chip),
"{:?}: {:?}",
input,
segs(input)
);
}
// Stars in glue are literal to the engine — no green.
assert!(segs("foo:ba*r").iter().all(|(_, c, _)| *c == Plain));
}
#[test]
fn drive_letters_do_not_split() {
assert_eq!(
segs(r"path:C:\Users\me"),
owned(vec![
("path", Keyword, true),
(":", Operator, true),
(r"C:\Users\me", Argument, true),
])
);
assert_eq!(
segs(r"C:\data"),
owned(vec![(r"C:\data", Plain, false)])
);
}
#[test]
fn quoted_phrases_get_green_delimiters() {
assert_eq!(
segs("\"exact phrase\""),
owned(vec![
("\"", Operator, false),
("exact phrase", Plain, false),
("\"", Operator, false),
])
);
// Inner "" escapes are content bytes.
assert_eq!(
segs("\"a\"\"b\""),
owned(vec![
("\"", Operator, false),
("a\"\"b", Plain, false),
("\"", Operator, false),
])
);
// Quoted stars are literal — content stays plain.
assert!(segs("\"a*b\"").iter().all(|(s, c, _)| s == "\""
|| *c == Plain));
}
#[test]
fn quoted_filter_values_are_blue_with_green_quotes() {
assert_eq!(
segs("path:\"/home/me/My Docs\""),
owned(vec![
("path", Keyword, true),
(":", Operator, true),
("\"", Operator, true),
("/home/me/My Docs", Argument, true),
("\"", Operator, true),
])
);
// Empty quoted value: two delimiters, no content seg, no panic.
assert_eq!(
segs("path:\"\""),
owned(vec![
("path", Keyword, true),
(":", Operator, true),
("\"", Operator, true),
("\"", Operator, true),
])
);
}
#[test]
fn unterminated_quote_is_a_quote_in_progress() {
assert_eq!(
segs("\"unclosed phrase"),
owned(vec![
("\"", Operator, false),
("unclosed phrase", Plain, false),
])
);
// Filters before the open quote keep their colors.
let all = segs("type:Audio \"x");
assert_eq!(all[0], ("type".to_string(), Keyword, true));
assert_eq!(all[4], ("\"".to_string(), Operator, false));
assert_eq!(all[5], ("x".to_string(), Plain, false));
}
#[test]
fn trailing_bare_keys_color_optimistically_without_chip() {
assert_eq!(
segs("type:"),
owned(vec![("type", Keyword, false), (":", Operator, false)])
);
assert_eq!(
segs("modified:>="),
owned(vec![
("modified", Keyword, false),
(":", Operator, false),
(">=", Operator, false),
])
);
assert_eq!(
segs("foo:"),
owned(vec![("foo", Plain, false), (":", Plain, false)])
);
}
#[test]
fn stars_in_words_and_name_values_go_green() {
assert_eq!(
segs("rep*ort"),
owned(vec![
("rep", Plain, false),
("*", Operator, false),
("ort", Plain, false),
])
);
assert_eq!(
segs("name:re*.txt"),
owned(vec![
("name", Keyword, true),
(":", Operator, true),
("re", Argument, true),
("*", Operator, true),
(".txt", Argument, true),
])
);
// Edge and doubled stars keep tiling intact.
assert_tiles("*foo");
assert_tiles("foo*");
assert_tiles("**");
assert_tiles("*");
// In non-glob filter values the star is a literal character.
assert_eq!(
segs("path:/da*ta")[2],
("/da*ta".to_string(), Argument, true)
);
}
#[test]
fn invalid_arguments_go_error_uniformly() {
// (`regex:(` is not here: `(` lexes as a paren, so that input is an
// *incomplete* filter — bare-key optimism applies, not an error.)
for input in ["type:NotAThing", "modified:>=tomorrow", "regex:[", "type:Doc*"] {
let all = segs(input);
assert_eq!(all[0].1, Keyword, "{:?}", input);
let last = all.last().unwrap();
assert_eq!(last.1, InvalidArg, "{:?}: {:?}", input, all);
assert!(last.2, "invalid values keep the chip: {:?}", input);
}
// name:= is an unsupported op → its value is invalid too.
let all = segs("name=x");
assert_eq!(all.last().unwrap().1, InvalidArg);
}
#[test]
fn valid_regex_argument_is_blue_with_green_stars() {
assert_eq!(
segs("regex:foo.*bar"),
owned(vec![
("regex", Keyword, true),
(":", Operator, true),
("foo.", Argument, true),
("*", Operator, true),
("bar", Argument, true),
])
);
}
#[test]
fn a_second_regex_filter_is_invalid() {
let all = segs("regex:foo regex:bar");
assert_eq!(all[2], ("foo".to_string(), Argument, true));
assert_eq!(all.last().unwrap(), &("bar".to_string(), InvalidArg, true));
}
#[test]
fn multi_filter_queries_chip_separately() {
let all = segs("type:Document budget modified:>=2024-01-01");
// The word and the whitespace around it stay un-chipped.
assert_eq!(
all.iter()
.filter(|(_, _, chip)| !chip)
.map(|(s, _, _)| s.as_str())
.collect::<Vec<_>>(),
vec![" ", "budget", " "]
);
}
#[test]
fn spaced_filters_chip_their_interior_gaps() {
// `type : Audio` is still a filter to the lexer/splitter.
assert_eq!(
segs("type : Audio"),
owned(vec![
("type", Keyword, true),
(" ", Plain, true),
(":", Operator, true),
(" ", Plain, true),
("Audio", Argument, true),
])
);
}
#[test]
fn demoted_operators_stay_plain() {
assert!(
segs("(alpha AND beta) OR gamma")
.iter()
.all(|(_, c, chip)| *c == Plain && !chip)
);
// Dangling comparators are literal text.
assert!(segs("a > b").iter().all(|(_, c, _)| *c == Plain));
// Leading operator, nothing else.
assert!(segs(">foo").iter().all(|(_, c, _)| *c == Plain));
}
#[test]
fn adjacency_between_filter_and_quote() {
// `Audio"q"`: the word ends at the quote; the filter is complete
// and the quoted phrase stands alone.
let all = segs("type:Audio\"q\"");
assert_eq!(all[2], ("Audio".to_string(), Argument, true));
assert_eq!(all[3], ("\"".to_string(), Operator, false));
}
#[test]
fn unicode_offsets_hold_up() {
assert_tiles("\"José\" type:Audio naïve*file");
let all = segs("naïve*café");
assert_eq!(
all,
owned(vec![
("naïve", Plain, false),
("*", Operator, false),
("café", Plain, false),
])
);
}
}

View file

@ -1,527 +0,0 @@
#![allow(non_snake_case)]
use std::sync::Arc;
use std::time::Instant;
use dioxus::prelude::*;
use quicksearch_core::indexing::{IndexingService, SearchResult};
use quicksearch_core::search_sql::{build_count, build_select, SearchArgs};
/// One page of results. Tuned to keep DOM size bounded — rendering ten
/// thousand `<tr>` nodes wedges WebKit for tens of seconds.
const PAGE_SIZE: u32 = 50;
#[derive(Props, Clone)]
pub struct SearchProps {
pub indexing_service: Arc<IndexingService>,
pub db_path: String,
}
impl PartialEq for SearchProps {
fn eq(&self, other: &Self) -> bool {
Arc::ptr_eq(&self.indexing_service, &other.indexing_service) && self.db_path == other.db_path
}
}
pub fn Search(props: SearchProps) -> Element {
let mut search_type = use_signal(|| "fulltext".to_string());
let mut search_term = use_signal(|| String::new());
let mut fulltext_exact = use_signal(|| false);
let mut fulltext_case_sensitive = use_signal(|| false);
let mut search_results = use_signal(|| Vec::<SearchResult>::new());
let mut search_error = use_signal(|| None::<String>);
let mut is_searching = use_signal(|| false);
let mut last_search_time = use_signal(|| None::<f64>);
let mut show_corruption_dialog = use_signal(|| false);
let mut current_page = use_signal(|| 1u32);
let mut total_count = use_signal(|| None::<u64>);
let mut last_args = use_signal(|| None::<SearchArgs>);
let mut goto_input = use_signal(|| String::new());
let service = props.indexing_service.clone();
let db_path = props.db_path.clone();
// Spawn a search task. `refresh_count` is true for fresh searches and
// false for in-place page navigation (the cached total still applies).
let run_query = {
let service = service.clone();
let db_path = db_path.clone();
move |args: SearchArgs, page: u32, refresh_count: bool| {
let service = service.clone();
let db_path = db_path.clone();
spawn(async move {
is_searching.set(true);
search_error.set(None);
last_search_time.set(None);
let start = Instant::now();
let count_sql = if refresh_count {
match build_count(&args) {
Ok(s) => Some(s),
Err(e) => {
search_error.set(Some(e));
is_searching.set(false);
return;
}
}
} else {
None
};
let offset = page.saturating_sub(1).saturating_mul(PAGE_SIZE);
// Validate early for the filename/duplicates branch so we
// surface parse errors before dispatching the blocking task.
let precomputed_select_sql = if args.search_type == "fulltext" {
None
} else {
match build_select(&args, PAGE_SIZE, offset) {
Ok(s) => Some(s),
Err(e) => {
search_error.set(Some(e));
is_searching.set(false);
return;
}
}
};
// Fulltext takes the snippet-aware path (decompresses
// documents_text and highlights in Rust); filename +
// duplicates go through the plain SQL executor.
let svc1 = service.clone();
let db1 = db_path.clone();
let args_for_select = args.clone();
let select_handle = tokio::task::spawn_blocking(move || {
if args_for_select.search_type == "fulltext" {
svc1.execute_fulltext_search(&db1, &args_for_select, PAGE_SIZE, offset)
} else {
let sql = precomputed_select_sql
.expect("non-fulltext select SQL was prebuilt above");
svc1.execute_search(&db1, &sql)
}
});
let count_handle = count_sql.map(|sql| {
let svc2 = service.clone();
let db2 = db_path.clone();
tokio::task::spawn_blocking(move || svc2.execute_search(&db2, &sql))
});
let select_run = select_handle.await;
let count_run = match count_handle {
Some(h) => Some(h.await),
None => None,
};
let elapsed = start.elapsed().as_secs_f64();
if let Some(c) = count_run {
match c {
Ok(Ok(rs)) => {
let n = rs
.first()
.and_then(|r| r.rows.first())
.and_then(|r| r.values.first())
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(0);
total_count.set(Some(n));
}
Ok(Err(e)) => {
handle_query_error(
e,
elapsed,
search_error,
show_corruption_dialog,
last_search_time,
is_searching,
);
return;
}
Err(e) => {
search_error.set(Some(format!("Task execution error: {}", e)));
last_search_time.set(Some(elapsed));
is_searching.set(false);
return;
}
}
}
match select_run {
Ok(Ok(rs)) => {
search_results.set(rs);
current_page.set(page);
last_args.set(Some(args));
last_search_time.set(Some(elapsed));
}
Ok(Err(e)) => {
handle_query_error(
e,
elapsed,
search_error,
show_corruption_dialog,
last_search_time,
is_searching,
);
return;
}
Err(e) => {
search_error.set(Some(format!("Task execution error: {}", e)));
last_search_time.set(Some(elapsed));
}
}
is_searching.set(false);
});
}
};
let do_fresh_search = {
let run_query = run_query.clone();
move || {
let args = SearchArgs {
search_type: search_type(),
term: search_term(),
fulltext_exact: fulltext_exact(),
fulltext_case_sensitive: fulltext_case_sensitive(),
};
run_query(args, 1, true);
}
};
let do_goto_page = {
let run_query = run_query.clone();
move |target: u32| {
if let Some(args) = last_args() {
let total = total_count()
.map(|n| pages_for(n))
.unwrap_or(1)
.max(1);
let clamped = target.clamp(1, total);
if clamped != current_page() {
run_query(args, clamped, false);
}
}
}
};
let total_pages = total_count().map(pages_for).unwrap_or(0);
rsx! {
div {
class: "section",
h2 { "Search Database" }
div {
class: "form-group",
label { "Search Type: " }
select {
class: "form-control",
value: "{search_type}",
onchange: move |evt| search_type.set(evt.value()),
option { value: "fulltext", "Full Text Search" }
option { value: "filename", "Filename Search" }
option { value: "duplicates", "Find Duplicate Files" }
}
}
if search_type() == "fulltext" {
div {
class: "form-group",
style: "display: flex; flex-direction: column; gap: 6px;",
span { style: "font-weight: 600;", "Full text options" }
label {
style: "display: flex; align-items: center; gap: 8px; cursor: pointer;",
input {
r#type: "checkbox",
checked: fulltext_exact(),
onchange: move |evt| fulltext_exact.set(evt.checked()),
}
"Exact phrase match"
}
label {
style: "display: flex; align-items: center; gap: 8px; cursor: pointer;",
input {
r#type: "checkbox",
checked: fulltext_case_sensitive(),
onchange: move |evt| fulltext_case_sensitive.set(evt.checked()),
}
"Case-sensitive match"
}
}
}
if search_type() != "duplicates" {
div {
class: "form-group",
label { "Search Term: " }
input {
class: "form-control",
r#type: "text",
value: "{search_term}",
oninput: move |evt| search_term.set(evt.value()),
onkeydown: {
let do_fresh_search = do_fresh_search.clone();
move |evt| {
if evt.code() == dioxus::events::Code::Enter {
do_fresh_search();
}
}
}
}
}
}
div {
style: "display: flex; align-items: center; gap: 10px;",
button {
class: "btn btn-info",
disabled: is_searching(),
onclick: {
let do_fresh_search = do_fresh_search.clone();
move |_| { do_fresh_search(); }
},
"Search"
}
if is_searching() {
div {
class: "loading",
title: "Searching..."
}
} else if let Some(elapsed) = last_search_time() {
span {
style: "color: #666; font-size: 0.9em;",
"Search completed in {elapsed:.3}s"
}
}
}
if let Some(error) = search_error() {
div {
class: "error-message",
"Error: {error}"
}
}
// Results panel: pagination header + bounded table. Only renders
// when at least one search has completed (last_args is Some).
if last_args().is_some() {
div {
class: "search-results",
{
let total_str = match total_count() {
Some(n) => format!("{}", n),
None => "?".to_string(),
};
let page_first = (current_page().saturating_sub(1) as u64) * PAGE_SIZE as u64 + 1;
let page_last_calc = page_first + search_results().first().map(|r| r.rows.len() as u64).unwrap_or(0).saturating_sub(1);
let header = if total_count() == Some(0) {
"No results.".to_string()
} else {
format!(
"Showing {}-{} of {} (page {} of {})",
page_first,
page_last_calc,
total_str,
current_page(),
total_pages
)
};
rsx! { h3 { "{header}" } }
}
// Pagination controls. Hidden if there's only one page.
if total_pages > 1 {
div {
style: "display: flex; align-items: center; gap: 8px; margin: 8px 0;",
button {
class: "btn",
disabled: is_searching() || current_page() <= 1,
onclick: {
let do_goto_page = do_goto_page.clone();
move |_| do_goto_page(1)
},
"« First"
}
button {
class: "btn",
disabled: is_searching() || current_page() <= 1,
onclick: {
let do_goto_page = do_goto_page.clone();
move |_| do_goto_page(current_page().saturating_sub(1))
},
" Prev"
}
button {
class: "btn",
disabled: is_searching() || current_page() >= total_pages,
onclick: {
let do_goto_page = do_goto_page.clone();
move |_| do_goto_page(current_page().saturating_add(1))
},
"Next "
}
button {
class: "btn",
disabled: is_searching() || current_page() >= total_pages,
onclick: {
let do_goto_page = do_goto_page.clone();
move |_| do_goto_page(total_pages)
},
"Last »"
}
span { "Go to:" }
input {
r#type: "number",
style: "width: 70px;",
value: "{goto_input}",
oninput: move |evt| goto_input.set(evt.value()),
onkeydown: {
let do_goto_page = do_goto_page.clone();
move |evt| {
if evt.code() == dioxus::events::Code::Enter {
if let Ok(p) = goto_input().trim().parse::<u32>() {
do_goto_page(p);
goto_input.set(String::new());
}
}
}
}
}
}
}
if !search_results().is_empty() && !search_results()[0].rows.is_empty() {
div {
class: "results-table",
table {
thead {
tr {
for column in search_results()[0].columns.iter() {
th { "{column}" }
}
}
}
tbody {
for row in search_results()[0].rows.iter() {
tr {
for (col_index, value) in row.values.iter().enumerate() {
if search_results()[0].columns.get(col_index).map(|s| s.as_str()) == Some("path") {
{
let value_owned = value.clone();
let service_owned = props.indexing_service.clone();
rsx! {
td {
class: "path-cell clickable",
onclick: move |_| {
let path = value_owned.clone();
let service_clone = service_owned.clone();
spawn(async move {
if let Err(e) = service_clone.open_file_explorer(&path) {
eprintln!("Failed to open file explorer: {}", e);
}
});
},
title: "Click to open in file explorer",
dangerous_inner_html: "{value}"
}
}
}
} else {
td {
dangerous_inner_html: "{value}"
}
}
}
}
}
}
}
}
}
}
}
if show_corruption_dialog() {
div {
class: "modal-backdrop",
div {
class: "modal-dialog",
h3 {
style: "margin-top: 0; color: #d32f2f;",
"⚠️ Database Corruption Detected"
}
p {
style: "margin: 15px 0;",
"The database appears to be corrupted or malformed. This can happen due to unexpected shutdowns or disk issues."
}
p {
style: "margin: 15px 0; font-weight: bold;",
"Would you like to delete the corrupted database and create a new one? This will require re-indexing your files."
}
div {
style: "display: flex; gap: 10px; margin-top: 20px;",
button {
style: "padding: 10px 20px; background-color: #d32f2f; color: white; border: none; border-radius: 5px; cursor: pointer;",
onclick: move |_| {
let service = props.indexing_service.clone();
let db = props.db_path.clone();
show_corruption_dialog.set(false);
search_error.set(Some("Deleting corrupted database...".to_string()));
spawn(async move {
match service.delete_index_for_rebuild(&db) {
Ok(()) => {
search_error.set(Some("Database deleted. You can now start indexing again.".to_string()));
}
Err(e) => {
search_error.set(Some(format!("Error deleting database: {}", e)));
}
}
});
},
"Yes, Delete & Rebuild"
}
button {
style: "padding: 10px 20px; background-color: #666; color: white; border: none; border-radius: 5px; cursor: pointer;",
onclick: move |_| {
show_corruption_dialog.set(false);
},
"Cancel"
}
}
}
}
}
}
}
}
/// Number of pages required to fit `total` rows at [`PAGE_SIZE`] per page.
/// Returns 0 for 0 rows so callers can branch on "no results yet".
fn pages_for(total: u64) -> u32 {
if total == 0 {
0
} else {
((total - 1) / PAGE_SIZE as u64) as u32 + 1
}
}
/// Centralized error-router for the two query branches that can fail
/// identically (count and select). Sets the error/timing/spinner signals
/// and pops the corruption dialog when warranted.
fn handle_query_error(
e: String,
elapsed: f64,
mut search_error: Signal<Option<String>>,
mut show_corruption_dialog: Signal<bool>,
mut last_search_time: Signal<Option<f64>>,
mut is_searching: Signal<bool>,
) {
if e.starts_with("DATABASE_CORRUPTED:") {
search_error.set(Some("Database appears to be corrupted".into()));
show_corruption_dialog.set(true);
} else {
search_error.set(Some(e));
}
last_search_time.set(Some(elapsed));
is_searching.set(false);
}

View file

@ -0,0 +1,947 @@
//! The Search tab: query strip, streaming results table, snippet
//! preview, context menu, ignore-filter dialog, and syntax help.
use std::time::Instant;
use egui::text::{LayoutJob, TextFormat};
use egui_extras::{Column, TableBuilder};
use quicksearch_core::search::{SearchHit, SearchUpdate};
use quicksearch_core::snippet::Snippet;
use crate::format::{fmt_elapsed, fmt_mtime, human_size};
use crate::platform;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortKey {
Rank,
Name,
Path,
Size,
Modified,
}
pub struct IgnoreDialog {
pub source_name: String,
pub source_path: String,
pub pattern: String,
pub persist: bool,
}
/// What the tab asks the app to do after this frame.
#[derive(Default)]
pub struct SearchActions {
/// Re-run the search (query/fuzzy/session filters changed *now*, not
/// debounced — e.g. a chip was removed).
pub rerun: bool,
/// Persist an ignore pattern into the config.
pub persist_ignore: Option<String>,
/// The fuzzy toggle changed; remember it in the config.
pub save_fuzzy_default: Option<bool>,
}
pub struct SearchTab {
pub query: String,
pub fuzzy: bool,
/// Set on every edit; the app fires the search after the debounce.
pub pending_edit: Option<Instant>,
pub generation: u64,
pub results: Vec<SearchHit>,
/// The next search's hits, held back while the old table fades out;
/// swapped into `results` at zero opacity. Prevents the empty-refill
/// strobe while typing.
staging: Vec<SearchHit>,
staging_has_snippets: bool,
/// True from search start until the staged set has been swapped in.
swap_pending: bool,
/// Display permutation over `results`.
order: Vec<u32>,
sort: (SortKey, bool),
sort_dirty: bool,
pub selected: Option<u32>,
pub running: bool,
/// When the in-flight search was submitted.
search_started: Option<Instant>,
/// Wall time of the last completed search (all cascade passes).
elapsed: Option<std::time::Duration>,
pub limited: bool,
pub error: Option<String>,
pub session_ignores: Vec<String>,
pub ignore_dialog: Option<IgnoreDialog>,
pub help_open: bool,
has_snippets: bool,
focus_query: bool,
/// Query syntax-highlight segments, cached per text.
highlight: crate::query_highlight::HighlightCache,
}
impl SearchTab {
pub fn new(fuzzy_default: bool) -> SearchTab {
SearchTab {
query: String::new(),
fuzzy: fuzzy_default,
pending_edit: None,
generation: 0,
results: Vec::new(),
staging: Vec::new(),
staging_has_snippets: false,
swap_pending: false,
order: Vec::new(),
sort: (SortKey::Rank, true),
sort_dirty: false,
selected: None,
running: false,
search_started: None,
elapsed: None,
limited: false,
error: None,
session_ignores: Vec::new(),
ignore_dialog: None,
help_open: false,
has_snippets: false,
focus_query: true,
highlight: Default::default(),
}
}
/// Pre-fill the query and let the normal debounce path run it, so a
/// command-line query lands the user on results rather than an empty box.
pub fn seed(&mut self, query: String) {
self.query = query;
self.pending_edit = Some(Instant::now());
}
/// A new search was submitted under `generation`. The previous
/// results stay on screen (fading out); the new ones stage until the
/// fade reaches zero.
pub fn on_search_started(&mut self, generation: u64) {
self.generation = generation;
self.staging.clear();
self.staging_has_snippets = false;
self.swap_pending = true;
self.running = true;
self.search_started = Some(Instant::now());
self.elapsed = None;
self.limited = false;
self.error = None;
}
pub fn apply_update(&mut self, update: SearchUpdate, display_limit: usize) {
if update.generation() != self.generation {
return;
}
match update {
SearchUpdate::Started { .. } => {}
SearchUpdate::Hits { hits, .. } => {
if self.swap_pending {
// Old results are still fading out; hold the new ones.
for hit in hits {
if self.staging.len() >= display_limit {
self.limited = true;
break;
}
self.staging_has_snippets |= hit.snippet.is_some();
self.staging.push(hit);
}
} else {
// Post-swap stream: later cascade passes append live.
for hit in hits {
if self.results.len() >= display_limit {
self.limited = true;
break;
}
self.has_snippets |= hit.snippet.is_some();
self.results.push(hit);
}
// Arrival order *is* rank order, so the default sort
// needs no work; anything else re-sorts on the set.
if self.sort != (SortKey::Rank, true) {
self.sort_dirty = true;
} else {
self.order = (0..self.results.len() as u32).collect();
}
}
}
SearchUpdate::Completed { limited, .. } => {
self.running = false;
self.elapsed = self.search_started.map(|t| t.elapsed());
self.limited |= limited;
}
SearchUpdate::Error { message, .. } => {
self.running = false;
self.elapsed = self.search_started.map(|t| t.elapsed());
self.error = Some(message);
}
}
}
pub fn result_count_label(&self) -> Option<String> {
if self.query.trim().is_empty() && self.results.is_empty() {
return None;
}
Some(if self.limited {
format!("{}+ results (truncated)", self.results.len())
} else {
format!("{} results", self.results.len())
})
}
fn resort(&mut self) {
let (key, ascending) = self.sort;
let selected_id = self
.selected
.and_then(|i| self.results.get(i as usize))
.map(|h| h.file_id);
self.order = (0..self.results.len() as u32).collect();
let results = &self.results;
self.order.sort_by(|&a, &b| {
let (a, b) = (&results[a as usize], &results[b as usize]);
let ord = match key {
SortKey::Rank => a.rank.partial_cmp(&b.rank).unwrap_or(std::cmp::Ordering::Equal),
SortKey::Name => a.name.cmp(&b.name),
SortKey::Path => a.path.cmp(&b.path),
SortKey::Size => a.size.cmp(&b.size),
SortKey::Modified => a.mtime.cmp(&b.mtime),
};
if ascending {
ord
} else {
ord.reverse()
}
});
// Selection follows the file, not the visual slot.
self.selected = selected_id.and_then(|id| {
self.results
.iter()
.position(|h| h.file_id == id)
.map(|i| i as u32)
});
self.sort_dirty = false;
}
/// A sortable column header: the whole cell is the click target, the
/// label is centered, and the sort indicator is a painter-drawn
/// triangle (the default egui fonts have no ▲/▼ glyphs — they render
/// as boxes).
fn sort_header(&mut self, ui: &mut egui::Ui, key: SortKey, label: &str) {
let (cur, asc) = self.sort;
let selected = cur == key;
let (rect, response) =
ui.allocate_exact_size(ui.available_size(), egui::Sense::click());
if ui.is_rect_visible(rect) {
if response.hovered() {
ui.painter()
.rect_filled(rect, 2.0, ui.visuals().widgets.hovered.weak_bg_fill);
}
let font_id = egui::TextStyle::Body.resolve(ui.style());
let color = ui.visuals().strong_text_color();
let galley = ui
.painter()
.layout_no_wrap(label.to_string(), font_id, color);
let text_size = galley.size();
let arrow_space = if selected { 11.0 } else { 0.0 };
let text_pos = egui::pos2(
rect.center().x - (text_size.x + arrow_space) / 2.0,
rect.center().y - text_size.y / 2.0,
);
ui.painter().galley(text_pos, galley, color);
if selected {
let cx = text_pos.x + text_size.x + 7.0;
let cy = rect.center().y;
let (w, h) = (3.5, 3.0);
let points = if asc {
vec![
egui::pos2(cx, cy - h),
egui::pos2(cx - w, cy + h),
egui::pos2(cx + w, cy + h),
]
} else {
vec![
egui::pos2(cx, cy + h),
egui::pos2(cx - w, cy - h),
egui::pos2(cx + w, cy - h),
]
};
ui.painter().add(egui::Shape::convex_polygon(
points,
color,
egui::Stroke::NONE,
));
}
}
if response.clicked() {
self.sort = if selected { (key, !asc) } else { (key, true) };
self.sort_dirty = true;
}
}
pub fn ui(&mut self, ui: &mut egui::Ui) -> SearchActions {
let mut actions = SearchActions::default();
// --- Query strip -------------------------------------------------
ui.horizontal(|ui| {
let show_elapsed =
!self.running && self.elapsed.is_some() && !self.query.trim().is_empty();
let slot_room = if self.running {
24.0
} else if show_elapsed {
60.0
} else {
0.0
};
let width = ui.available_width() - 170.0 - slot_room;
let highlight = &mut self.highlight;
let mut layouter = move |ui: &egui::Ui, buf: &dyn egui::TextBuffer, _wrap: f32| {
crate::query_highlight::galley(ui, highlight, buf.as_str())
};
let response = ui.add(
egui::TextEdit::singleline(&mut self.query)
.desired_width(width.max(120.0))
.hint_text("Search names and contents… (type:Document regex:… budget*)")
.layouter(&mut layouter),
);
if self.focus_query {
response.request_focus();
self.focus_query = false;
}
if response.changed() {
self.pending_edit = Some(Instant::now());
}
// One slot right of the box: spinner while searching, then the
// total wall time of all cascade passes once it lands.
if self.running {
ui.add(egui::Spinner::new().size(16.0));
} else if show_elapsed {
if let Some(elapsed) = self.elapsed {
ui.label(egui::RichText::new(fmt_elapsed(elapsed)).small().weak())
.on_hover_text("Time to run all search passes");
}
}
if ui
.checkbox(&mut self.fuzzy, "Fuzzy")
.on_hover_text("Also run fuzzy filename and full-text passes (slower)")
.changed()
{
actions.save_fuzzy_default = Some(self.fuzzy);
actions.rerun = true;
}
if ui.button("?").on_hover_text("Query syntax help").clicked() {
self.help_open = !self.help_open;
}
});
// Session ignore chips.
if !self.session_ignores.is_empty() {
ui.horizontal_wrapped(|ui| {
ui.label(egui::RichText::new("Ignoring:").small().weak());
let mut remove: Option<usize> = None;
for (i, pattern) in self.session_ignores.iter().enumerate() {
if ui
.small_button(format!("{}", pattern))
.on_hover_text("Remove this session filter")
.clicked()
{
remove = Some(i);
}
}
if let Some(i) = remove {
self.session_ignores.remove(i);
actions.rerun = true;
}
});
}
// Notices.
if let Some(err) = &self.error {
ui.colored_label(ui.visuals().error_fg_color, err);
} else if self.limited {
ui.label(
egui::RichText::new(format!(
"Showing first {} matches; refine the query (limit configurable in Options).",
self.results.len()
))
.small()
.weak(),
);
} else if !self.running
&& !self.swap_pending
&& self.results.is_empty()
&& !self.query.trim().is_empty()
&& self.error.is_none()
{
ui.label(egui::RichText::new("No results.").small().weak());
}
// Result-set transitions pulse instead of strobing: the old table
// fades out over 0.25 s while the new hits stage, the sets swap at
// zero opacity, and the new table fades back in over 0.25 s.
// `animate_value_with_time` keeps requesting repaints until the
// value settles.
let fade_target = if self.swap_pending { 0.0 } else { 1.0 };
let fade = ui.ctx().animate_value_with_time(
egui::Id::new("qs-results-fade"),
fade_target,
0.25,
);
if self.swap_pending && fade <= 0.01 {
self.results = std::mem::take(&mut self.staging);
self.has_snippets = self.staging_has_snippets;
self.selected = None;
self.swap_pending = false;
if self.sort == (SortKey::Rank, true) {
self.order = (0..self.results.len() as u32).collect();
} else {
self.sort_dirty = true;
}
}
if self.sort_dirty {
self.resort();
}
// Fade covers the table and the preview strip below it; the modal
// windows and notices render at full opacity on their own layers.
ui.set_opacity(fade);
// --- Results table ------------------------------------------------
// Reserve room for the selected-row snippet preview strip. Only
// content matches get one — a filename match's "snippet" is the
// name, already on screen.
let preview_snippet: Option<Snippet> = self
.selected
.and_then(|i| self.results.get(i as usize))
.filter(|h| matches!(h.stage, 5 | 6 | 8))
.and_then(|h| h.snippet.clone());
let preview_height = if preview_snippet.is_some() { 44.0 } else { 0.0 };
let table_height = (ui.available_height() - preview_height).max(60.0);
let text_height = egui::TextStyle::Body.resolve(ui.style()).size + 4.0;
let mut open_ignore_dialog: Option<usize> = None;
ui.push_id("results", |ui| {
let mut table = TableBuilder::new(ui)
.striped(true)
.resizable(true)
.sense(egui::Sense::click())
.max_scroll_height(table_height)
.min_scrolled_height(60.0)
.column(Column::initial(220.0).at_least(80.0).clip(true)) // name
.column(Column::remainder().at_least(120.0).clip(true)); // path
if self.has_snippets {
table = table.column(Column::remainder().at_least(120.0).clip(true));
}
table = table
.column(Column::exact(72.0)) // size
.column(Column::exact(110.0)) // modified
.column(Column::exact(52.0)); // rank
table
.header(text_height + 4.0, |mut header| {
header.col(|ui| self.sort_header(ui, SortKey::Name, "Name"));
header.col(|ui| self.sort_header(ui, SortKey::Path, "Path"));
if self.has_snippets {
header.col(|ui| {
ui.with_layout(
egui::Layout::centered_and_justified(
egui::Direction::LeftToRight,
),
|ui| {
ui.label(egui::RichText::new("Match").strong());
},
);
});
}
header.col(|ui| self.sort_header(ui, SortKey::Size, "Size"));
header.col(|ui| self.sort_header(ui, SortKey::Modified, "Modified"));
header.col(|ui| self.sort_header(ui, SortKey::Rank, "Rank"));
})
.body(|body| {
let order = self.order.clone();
body.rows(text_height, order.len(), |mut row| {
let result_ix = order[row.index()] as usize;
let hit = &self.results[result_ix];
row.set_selected(self.selected == Some(result_ix as u32));
row.col(|ui| {
ui.label(&hit.name);
});
row.col(|ui| {
ui.label(egui::RichText::new(&hit.path).weak());
});
if self.has_snippets {
let snippet = hit.snippet.clone();
// Name and path matches show a whole field, so
// they render bracketed: [matched field].
let whole_field =
hit.stage <= 4 || hit.stage == 7 || hit.stage >= 9;
row.col(|ui| {
if let Some(snip) = &snippet {
let width = ui.available_width();
let job = centered_match_job(ui, snip, width, whole_field);
let response = ui
.with_layout(
egui::Layout::centered_and_justified(
egui::Direction::LeftToRight,
),
|ui| ui.label(job),
)
.inner;
if !snip.ranges.is_empty() {
let hover = snip.clone();
response.on_hover_ui(|ui| {
ui.set_max_width(520.0);
let job = snippet_job(ui, &hover, 10);
ui.label(job);
});
}
}
});
}
row.col(|ui| {
ui.with_layout(
egui::Layout::centered_and_justified(
egui::Direction::LeftToRight,
),
|ui| {
ui.label(human_size(hit.size));
},
);
});
row.col(|ui| {
let color = recency_color(ui, hit.mtime);
ui.with_layout(
egui::Layout::centered_and_justified(
egui::Direction::LeftToRight,
),
|ui| {
ui.label(
egui::RichText::new(fmt_mtime(hit.mtime)).color(color),
);
},
);
});
row.col(|ui| {
ui.with_layout(
egui::Layout::centered_and_justified(
egui::Direction::LeftToRight,
),
|ui| {
ui.label(
egui::RichText::new(format!(" {:.2} ", hit.rank))
.background_color(rank_tier_color(hit.stage))
.color(egui::Color32::from_rgb(32, 32, 32)),
);
},
);
});
let response = row.response();
if response.clicked() {
self.selected = Some(result_ix as u32);
}
if response.double_clicked() {
platform::open_file(&self.results[result_ix].path);
}
response.context_menu(|ui| {
let path = self.results[result_ix].path.clone();
if ui.button("Open").clicked() {
platform::open_file(&path);
ui.close();
}
if ui.button("Open containing folder").clicked() {
platform::reveal_in_folder(&path);
ui.close();
}
ui.separator();
if ui.button("Build ignore filter…").clicked() {
open_ignore_dialog = Some(result_ix);
ui.close();
}
});
});
});
});
if let Some(ix) = open_ignore_dialog {
let hit = &self.results[ix];
self.ignore_dialog = Some(IgnoreDialog {
source_name: hit.name.clone(),
source_path: hit.path.clone(),
pattern: hit.name.clone(),
persist: false,
});
}
// Selected-row preview strip: the full snippet, wrapped.
if let Some(snip) = &preview_snippet {
ui.separator();
let job = snippet_job(ui, snip, 2);
ui.label(job);
}
self.ignore_dialog_ui(ui.ctx(), &mut actions);
self.help_window_ui(ui.ctx());
actions
}
fn ignore_dialog_ui(&mut self, ctx: &egui::Context, actions: &mut SearchActions) {
let Some(dialog) = &mut self.ignore_dialog else {
return;
};
let mut apply = false;
let mut cancel = false;
egui::Window::new("Ignore filter")
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.label(format!("From: {}", dialog.source_path));
ui.add_space(4.0);
ui.horizontal(|ui| {
if ui.button("This name").clicked() {
dialog.pattern = dialog.source_name.clone();
}
if let Some(ext) = std::path::Path::new(&dialog.source_name)
.extension()
.and_then(|e| e.to_str())
{
if ui.button(format!("*.{}", ext)).clicked() {
dialog.pattern = format!("*.{}", ext);
}
}
if let Some(parent) = std::path::Path::new(&dialog.source_path)
.parent()
.and_then(|p| p.to_str())
{
if ui.button("This directory").clicked() {
dialog.pattern = format!("{}/*", parent);
}
}
});
ui.add(
egui::TextEdit::singleline(&mut dialog.pattern)
.desired_width(360.0)
.hint_text("glob pattern"),
);
ui.checkbox(&mut dialog.persist, "Persist to config");
ui.label(
egui::RichText::new(
"Session filters hide results immediately. Persisted filters also \
exclude files from the index at the next reindex.",
)
.small()
.weak(),
);
ui.horizontal(|ui| {
if ui.button("Apply").clicked() {
apply = true;
}
if ui.button("Cancel").clicked() {
cancel = true;
}
});
});
if apply {
let dialog = self.ignore_dialog.take().unwrap();
let pattern = dialog.pattern.trim().to_string();
if !pattern.is_empty() {
if !self.session_ignores.contains(&pattern) {
self.session_ignores.push(pattern.clone());
}
if dialog.persist {
actions.persist_ignore = Some(pattern);
}
actions.rerun = true;
}
} else if cancel {
self.ignore_dialog = None;
}
}
fn help_window_ui(&mut self, ctx: &egui::Context) {
let mut open = self.help_open;
egui::Window::new("Query syntax")
.open(&mut open)
.resizable(false)
.default_width(540.0)
.show(ctx, |ui| {
ui.label(
"Everything that is not a filter is matched as one phrase, in order. \
Filters combine freely with the search text.",
);
ui.add_space(6.0);
egui::Grid::new("query-syntax-table")
.num_columns(2)
.spacing([18.0, 5.0])
.striped(true)
.show(ui, |ui| {
let row = |ui: &mut egui::Ui, syntax: &str, meaning: &str| {
ui.monospace(syntax);
ui.label(meaning);
ui.end_row();
};
row(
ui,
"budget report",
"names, contents, and paths containing \"budget report\"",
);
row(
ui,
"\"exact phrase\"",
"quotes keep spaces, stars, and filter-like words literal",
);
row(
ui,
"bud*report",
"* matches any run of characters (within a line); \
also works in name: values",
);
row(
ui,
"regex:\"(foo|bar)\\d+\"",
"regular expression, matched against names, contents, \
and paths; case-insensitive use (?-i:) to override; \
quote patterns containing spaces",
);
row(
ui,
"type:Document",
"file class: Audio, Image, Video, Document, Text, \
Archive, Spreadsheet, Presentation, Folder",
);
row(
ui,
"modified:>=2024-01-01",
"modification date (yyyy-mm-dd); also <, <=, > and =",
);
row(
ui,
"path:/home/me/docs",
"only results in that folder and its subfolders; \
quote paths containing spaces",
);
row(ui, "mime:application/pdf", "exact MIME type");
row(
ui,
"name:report",
"filename contains, applied as an unranked filter",
);
});
ui.add_space(6.0);
ui.label("Example:");
ui.monospace("type:Document modified:>=2024-01-01 quarterly budget");
ui.add_space(6.0);
ui.label(
egui::RichText::new(
"Ranking: exact filename matches, then filename substrings, then \
full-text matches (ordered by occurrences), then fuzzy matches \
when enabled, and finally matches on the rest of the file path.",
)
.small()
.weak(),
);
ui.label(
egui::RichText::new(
"The complete reference, including ranking details and the \
fuzzy edit budget, is the \"Query syntax\" section of \
README.md in the QuickSearch folder.",
)
.small()
.weak(),
);
});
self.help_open = open;
}
}
struct SnippetFormats {
normal: TextFormat,
highlight: TextFormat,
weak: TextFormat,
}
fn snippet_formats(ui: &egui::Ui) -> SnippetFormats {
let font_id = egui::TextStyle::Body.resolve(ui.style());
SnippetFormats {
normal: TextFormat {
font_id: font_id.clone(),
color: ui.visuals().text_color(),
..Default::default()
},
highlight: TextFormat {
font_id: font_id.clone(),
color: ui.visuals().strong_text_color(),
background: ui.visuals().selection.bg_fill.gamma_multiply(0.4),
..Default::default()
},
weak: TextFormat {
font_id,
color: ui.visuals().weak_text_color(),
..Default::default()
},
}
}
/// Build a highlighted snippet LayoutJob from byte ranges, wrapped to at
/// most `max_rows`. Cheap enough to run per visible row per frame.
fn snippet_job(ui: &egui::Ui, snip: &Snippet, max_rows: usize) -> LayoutJob {
let fmt = snippet_formats(ui);
let mut job = LayoutJob::default();
job.wrap.max_rows = max_rows;
if max_rows == 1 {
job.wrap.break_anywhere = true;
}
if snip.truncated_start {
job.append("", 0.0, fmt.weak.clone());
}
let mut cursor = 0;
for &(start, end) in &snip.ranges {
if start > cursor {
job.append(&snip.window[cursor..start], 0.0, fmt.normal.clone());
}
job.append(&snip.window[start..end], 0.0, fmt.highlight.clone());
cursor = end;
}
if cursor < snip.window.len() {
job.append(&snip.window[cursor..], 0.0, fmt.normal.clone());
}
if snip.truncated_end {
job.append("", 0.0, fmt.weak);
}
job
}
/// The Match column cell: one line with the (first) matched span centered
/// and an equal amount of context on both sides, trimmed to what fits the
/// column width. Matches on a whole field — a filename or a path — are
/// wrapped in brackets: `[name]`.
fn centered_match_job(
ui: &egui::Ui,
snip: &Snippet,
width_px: f32,
whole_field: bool,
) -> LayoutJob {
let fmt = snippet_formats(ui);
let font_id = egui::TextStyle::Body.resolve(ui.style());
let char_width = ui.fonts(|f| f.glyph_width(&font_id, '0')).max(1.0);
let mut budget = ((width_px / char_width) as usize).saturating_sub(2).max(8);
if whole_field {
budget = budget.saturating_sub(2); // room for the brackets
}
// Newlines force line breaks even in a one-row LayoutJob, wrecking the
// centered single-line cell. Flatten them to spaces — a byte-for-byte
// ASCII replacement, so the match ranges stay valid. The mouseover
// renders the original window untouched.
let flattened = snip.window.replace(['\n', '\r', '\t'], " ");
let window = flattened.as_str();
let (start, end) = match snip.ranges.first().copied() {
Some((a, b)) => {
let match_chars = window[a..b].chars().count();
let side = budget.saturating_sub(match_chars) / 2;
let before = &window[..a];
let after = &window[b..];
let before_count = before.chars().count();
let after_count = after.chars().count();
// Equal context on both sides; leftover budget from a short
// side flows to the other.
let take_before = (side + side.saturating_sub(after_count)).min(before_count);
let take_after = (side + side.saturating_sub(before_count)).min(after_count);
let start = if take_before == 0 {
a
} else {
before
.char_indices()
.nth_back(take_before - 1)
.map(|(i, _)| i)
.unwrap_or(0)
};
let end = b + after
.char_indices()
.nth(take_after)
.map(|(i, _)| i)
.unwrap_or(after.len());
(start, end)
}
None => {
// No ranges (shouldn't happen for match cells) — head trim.
let end = window
.char_indices()
.nth(budget)
.map(|(i, _)| i)
.unwrap_or(window.len());
(0, end)
}
};
let mut job = LayoutJob::default();
job.wrap.max_rows = 1;
job.wrap.break_anywhere = true;
if whole_field {
job.append("[", 0.0, fmt.weak.clone());
}
if start > 0 || snip.truncated_start {
job.append("", 0.0, fmt.weak.clone());
}
let mut cursor = start;
for &(a, b) in &snip.ranges {
let (a, b) = (a.max(start), b.min(end));
if a >= b || a >= end {
continue;
}
if a > cursor {
job.append(&window[cursor..a], 0.0, fmt.normal.clone());
}
job.append(&window[a..b], 0.0, fmt.highlight.clone());
cursor = b;
}
if cursor < end {
job.append(&window[cursor..end], 0.0, fmt.normal.clone());
}
if end < window.len() || snip.truncated_end {
job.append("", 0.0, fmt.weak.clone());
}
if whole_field {
job.append("]", 0.0, fmt.weak);
}
job
}
/// Tier-list chip color per cascade stage — lower rank, higher tier:
/// S-red for exact case-sensitive filename matches down through the
/// pastel ramp to purple for fuzzy full-text and on to the grey path
/// tiers. Dark text on these pastels stays readable in both themes.
fn rank_tier_color(stage: u8) -> egui::Color32 {
match stage {
1 => egui::Color32::from_rgb(255, 127, 127), // S
2 => egui::Color32::from_rgb(255, 191, 127), // A
3 => egui::Color32::from_rgb(255, 223, 127), // B
4 => egui::Color32::from_rgb(255, 255, 127), // C
5 => egui::Color32::from_rgb(191, 255, 127), // D
6 => egui::Color32::from_rgb(127, 255, 127), // E
7 => egui::Color32::from_rgb(127, 191, 255), // F
8 => egui::Color32::from_rgb(191, 127, 255), // G
9 => egui::Color32::from_rgb(223, 159, 255), // H — path, exact case
10 => egui::Color32::from_rgb(239, 191, 239), // I — path, any case
_ => egui::Color32::from_rgb(199, 199, 199), // J — fuzzy path
}
}
/// Timestamp color: fresh files get a green tint that fades into the weak
/// text color over ~2 years on a log scale.
fn recency_color(ui: &egui::Ui, mtime: i64) -> egui::Color32 {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0);
let age_hours = ((now - mtime).max(0) as f32 / 3600.0).max(1.0);
const HORIZON_HOURS: f32 = 24.0 * 365.0 * 2.0;
let t = (age_hours.ln() / HORIZON_HOURS.ln()).clamp(0.0, 1.0);
let fresh = egui::Color32::from_rgb(87, 187, 122);
let old = ui.visuals().weak_text_color();
let lerp = |a: u8, b: u8| (a as f32 + (b as f32 - a as f32) * t).round() as u8;
egui::Color32::from_rgb(
lerp(fresh.r(), old.r()),
lerp(fresh.g(), old.g()),
lerp(fresh.b(), old.b()),
)
}

View file

@ -0,0 +1,144 @@
//! Indexing-rate estimation for the status displays.
//!
//! The old tracker sampled the counter every poll tick but pruned to a
//! 1-second window, so anything slower than ~1 file/sec measured a
//! genuine zero and displayed "0.0 files/sec" despite progress. This one
//! records a point only when the counter *changes*, keeps up to 60 s of
//! history but never fewer than two points (so slow rates stay
//! computable), and measures against `now` so the estimate decays during
//! stalls instead of freezing at the last burst.
use std::time::{Duration, Instant};
const HISTORY: Duration = Duration::from_secs(60);
pub struct SpeedTracker {
/// (when, counter value) — appended only on counter change.
points: Vec<(Instant, usize)>,
}
impl SpeedTracker {
pub fn new() -> SpeedTracker {
SpeedTracker { points: Vec::new() }
}
/// Reset between phases (each phase restarts its counter).
pub fn reset(&mut self) {
self.points.clear();
}
pub fn record(&mut self, files_processed: usize) {
self.record_at(Instant::now(), files_processed);
}
fn record_at(&mut self, now: Instant, files_processed: usize) {
match self.points.last() {
Some(&(_, last)) if last == files_processed => return,
// Counter went backwards — a new phase started without an
// explicit reset.
Some(&(_, last)) if files_processed < last => self.points.clear(),
_ => {}
}
self.points.push((now, files_processed));
// Prune old points, but always keep at least two so a slow but
// steady rate never becomes unmeasurable.
while self.points.len() > 2
&& now.duration_since(self.points[0].0) > HISTORY
{
self.points.remove(0);
}
}
/// Estimated files/sec, measured from the oldest retained progress
/// point to *now*. `None` until two data points exist.
pub fn files_per_sec(&self) -> Option<f64> {
self.files_per_sec_at(Instant::now())
}
fn files_per_sec_at(&self, now: Instant) -> Option<f64> {
let (t0, c0) = *self.points.first()?;
let (_, c1) = *self.points.last()?;
if self.points.len() < 2 {
return None;
}
let span = now.duration_since(t0).as_secs_f64();
if span <= 0.0 {
return None;
}
Some((c1 - c0) as f64 / span)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn needs_two_points() {
let mut t = SpeedTracker::new();
let base = Instant::now();
assert!(t.files_per_sec_at(base).is_none());
t.record_at(base, 10);
assert!(t.files_per_sec_at(base).is_none());
}
#[test]
fn slow_rate_is_measurable_not_zero() {
// One file every 2.5 s — the old 1 s window reported 0.0 here.
let mut t = SpeedTracker::new();
let base = Instant::now();
for i in 0..4 {
t.record_at(base + Duration::from_millis(2500 * i), 10 + i as usize);
}
let rate = t.files_per_sec_at(base + Duration::from_millis(7500)).unwrap();
assert!((rate - 0.4).abs() < 0.01, "expected ~0.4/s, got {}", rate);
}
#[test]
fn unchanged_counter_adds_no_points() {
let mut t = SpeedTracker::new();
let base = Instant::now();
for i in 0..100 {
t.record_at(base + Duration::from_millis(50 * i), 42);
}
assert_eq!(t.points.len(), 1, "only the first observation recorded");
}
#[test]
fn stall_decays_toward_zero() {
let mut t = SpeedTracker::new();
let base = Instant::now();
t.record_at(base, 0);
t.record_at(base + Duration::from_secs(1), 100); // 100/s burst
let just_after = t.files_per_sec_at(base + Duration::from_secs(1)).unwrap();
let stalled = t.files_per_sec_at(base + Duration::from_secs(20)).unwrap();
assert!(just_after > 90.0);
assert!(stalled < 6.0, "estimate must decay during a stall: {}", stalled);
}
#[test]
fn pruning_keeps_at_least_two_points() {
let mut t = SpeedTracker::new();
let base = Instant::now();
t.record_at(base, 1);
t.record_at(base + Duration::from_secs(30), 2);
// Far beyond the history window; both points are older than 60 s
// relative to this record.
t.record_at(base + Duration::from_secs(300), 3);
assert!(t.points.len() >= 2);
assert!(t
.files_per_sec_at(base + Duration::from_secs(300))
.is_some());
}
#[test]
fn counter_regression_resets() {
let mut t = SpeedTracker::new();
let base = Instant::now();
t.record_at(base, 500);
t.record_at(base + Duration::from_secs(1), 600);
// New phase restarts from a small number.
t.record_at(base + Duration::from_secs(2), 3);
assert_eq!(t.points.len(), 1);
}
}

224
packaging/build-deb.sh Normal file
View file

@ -0,0 +1,224 @@
#!/usr/bin/env bash
#
# Build a Debian package for QuickSearch.
#
# ./packaging/build-deb.sh build and package
# ./packaging/build-deb.sh --no-build package an existing release binary
# ./packaging/build-deb.sh --no-strip keep debug symbols (25 MB vs 20 MB)
# ./packaging/build-deb.sh -o /tmp/out write the .deb somewhere else
#
# Environment: DEB_REVISION (default 1), DEB_MAINTAINER, SOURCE_DATE_EPOCH.
#
# Deliberately does not use cargo-deb, debhelper, fakeroot or an SVG rasteriser:
# dpkg-deb and desktop-file-utils are the only tools required, and both are part
# of a standard Debian or Ubuntu install.
set -euo pipefail
# Directories created along the way must be 0755, not whatever the caller's
# umask happens to be, or the package ships group-writable directories.
umask 022
readonly PKG=quicksearch
# The GUI binary and the console-subsystem terminal binary. Both ship: on Unix
# `quicksearch` does both jobs, but the README and the shared man page name
# `quicksearch-cli` too, so it has to exist wherever the docs are installed.
readonly BINARIES=(quicksearch quicksearch-cli)
readonly REPO_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
readonly ICON_SRC="$REPO_ROOT/crates/quicksearch-gui/assets/icons"
readonly ICON_SVG="$ICON_SRC/quicksearch_icon.svg"
do_build=1
do_strip=1
out_dir="$REPO_ROOT/dist"
die() { printf 'build-deb: %s\n' "$*" >&2; exit 1; }
say() { printf '\033[1m==>\033[0m %s\n' "$*"; }
while [ $# -gt 0 ]; do
case "$1" in
--no-build) do_build=0 ;;
--no-strip) do_strip=0 ;;
-o|--output-dir) shift; [ $# -gt 0 ] || die "--output-dir needs a path"; out_dir="$1" ;;
# Print the header comment block, however long it grows.
-h|--help) awk 'NR > 1 { if ($0 !~ /^#/) exit; sub(/^# ?/, ""); print }' "${BASH_SOURCE[0]}"; exit 0 ;;
*) die "unknown option: $1 (try --help)" ;;
esac
shift
done
for tool in dpkg-deb dpkg desktop-file-validate objdump gzip; do
command -v "$tool" >/dev/null 2>&1 || die "missing required tool: $tool"
done
[ "$do_strip" -eq 0 ] || command -v strip >/dev/null 2>&1 || die "missing strip (install binutils, or pass --no-strip)"
# Version comes from [workspace.package] so the package can never drift from the
# crate version.
version="$(sed -n '/^\[workspace\.package\]/,/^\[/{ s/^version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p }' "$REPO_ROOT/Cargo.toml")"
[ -n "$version" ] || die "could not read version from Cargo.toml"
revision="${DEB_REVISION:-1}"
maintainer="${DEB_MAINTAINER:-Jeremy <jeremy@karsttech.com>}"
arch="$(dpkg --print-architecture)"
deb_version="${version}-${revision}"
stage="$out_dir/${PKG}_${deb_version}_${arch}"
deb="$out_dir/${PKG}_${deb_version}_${arch}.deb"
# ---------------------------------------------------------------- build ----
if [ "$do_build" -eq 1 ]; then
say "Building quicksearch $version (release)"
( cd "$REPO_ROOT" && cargo build --release -p quicksearch-gui )
fi
for bin in "${BINARIES[@]}"; do
[ -x "$REPO_ROOT/target/release/$bin" ] \
|| die "no release binary at target/release/$bin (drop --no-build?)"
done
# Both binaries link the same crates, so either gives the same glibc floor.
readonly primary_binary="$REPO_ROOT/target/release/$PKG"
[ -f "$ICON_SVG" ] || die "no icon at $ICON_SVG"
say "Validating desktop entry"
desktop-file-validate "$REPO_ROOT/packaging/$PKG.desktop"
# --------------------------------------------------------------- stage -----
say "Staging $stage"
rm -rf "$stage"
mkdir -p "$stage"
for bin in "${BINARIES[@]}"; do
install -Dm755 "$REPO_ROOT/target/release/$bin" "$stage/usr/bin/$bin"
done
install -Dm644 "$REPO_ROOT/packaging/$PKG.desktop" "$stage/usr/share/applications/$PKG.desktop"
install -Dm644 "$ICON_SVG" "$stage/usr/share/icons/hicolor/scalable/apps/$PKG.svg"
install -Dm644 "$REPO_ROOT/packaging/copyright" "$stage/usr/share/doc/$PKG/copyright"
install -Dm644 "$REPO_ROOT/config_example.toml" "$stage/usr/share/doc/$PKG/config_example.toml"
install -Dm644 "$REPO_ROOT/README.md" "$stage/usr/share/doc/$PKG/README.md"
# The PNGs are committed next to the quicksearch_icon.svg they came from, so
# this script only copies them and an ordinary `cargo build` needs no image
# tooling. To re-render them after editing the SVG, build a throwaway crate
# (outside this workspace, to keep it out of Cargo.lock) depending on
# `resvg = { version = "0.45", default-features = false }` and, for each size N,
# parse with usvg::Tree::from_data, make a tiny_skia::Pixmap::new(N, N), call
# resvg::render with Transform::from_scale(N/240.0, N/240.0) and save_png. The
# SVG contains no <text>, so no font support is needed.
#
# Sizes follow whatever is committed, so adding one needs no script change.
shopt -s nullglob
icons=("$ICON_SRC"/$PKG-*.png)
shopt -u nullglob
[ "${#icons[@]}" -gt 0 ] || die "no icons in $ICON_SRC"
for png in "${icons[@]}"; do
n="$(basename "$png" .png)"; n="${n#$PKG-}"
# The glob would also catch a non-size name like quicksearch-cli.png.
case "$n" in ''|*[!0-9]*) die "unexpected icon name: $(basename "$png")" ;; esac
install -Dm644 "$png" "$stage/usr/share/icons/hicolor/${n}x${n}/apps/$PKG.png"
done
say "Installed ${#icons[@]} icon sizes plus the scalable SVG"
# Debian wants man pages and the changelog compressed, with no gzip timestamp so
# repeat builds are byte-identical. quicksearch-cli.1 is a one-line .so stub
# pointing at quicksearch.1, which documents both binaries.
install -dm755 "$stage/usr/share/man/man1"
for page in "${BINARIES[@]}"; do
gzip -9nc "$REPO_ROOT/packaging/$page.1" > "$stage/usr/share/man/man1/$page.1.gz"
chmod 644 "$stage/usr/share/man/man1/$page.1.gz"
done
if [ -n "${SOURCE_DATE_EPOCH:-}" ]; then
changelog_date="$(date -R -u -d "@$SOURCE_DATE_EPOCH")"
else
changelog_date="$(date -R)"
fi
gzip -9nc <<EOF > "$stage/usr/share/doc/$PKG/changelog.Debian.gz"
$PKG ($deb_version) unstable; urgency=medium
* Package build of $PKG $version.
-- $maintainer $changelog_date
EOF
chmod 644 "$stage/usr/share/doc/$PKG/changelog.Debian.gz"
# No config.toml is installed anywhere. Config::config_path() treats a
# config.toml sitting next to the executable as portable mode and lets it
# override the per-user config outright, so one in /usr/bin would hijack every
# account on the machine. Config::load_from creates ~/.config/quicksearch/
# config.toml on first run instead.
if [ "$do_strip" -eq 1 ]; then
for bin in "${BINARIES[@]}"; do
before="$(du -h "$stage/usr/bin/$bin" | cut -f1)"
strip --strip-unneeded "$stage/usr/bin/$bin"
say "Stripped $bin: $before -> $(du -h "$stage/usr/bin/$bin" | cut -f1)"
done
fi
installed_size="$(du -ks "$stage" | cut -f1)"
# ------------------------------------------------------------- control -----
# The dynamic section only names libc, libgcc, libm and libbz2: winit and glutin
# dlopen the entire display stack, so dpkg-shlibdeps cannot see it and the list
# below is maintained by hand. Re-derive it with
# objdump -p target/release/quicksearch | grep NEEDED
# strings -a target/release/quicksearch | grep -oE 'lib[A-Za-z0-9_+-]+\.so(\.[0-9]+)*' | sort -u
# and map each soname to a package with `dpkg -S`.
glibc_min="$(objdump -T "$primary_binary" | sed -n 's/.*GLIBC_\([0-9][0-9.]*\).*/\1/p' | sort -V | tail -1)"
[ -n "$glibc_min" ] || die "could not determine the glibc version requirement"
depends="libc6 (>= ${glibc_min}), libgcc-s1 (>= 3.0), libbz2-1.0"
depends="$depends, libx11-6, libxcb1, libxcursor1, libxi6, libxrender1"
depends="$depends, libxkbcommon0, libxkbcommon-x11-0"
depends="$depends, libwayland-client0, libwayland-egl1"
depends="$depends, libegl1, libgl1"
# Required by policy for anything installing into the hicolor theme; it also
# provides the dpkg trigger that refreshes the icon cache on install.
depends="$depends, hicolor-icon-theme"
# desktop-file-utils owns the /usr/share/applications trigger; dbus-bin provides
# the dbus-send used by "reveal in folder"; xdg-utils provides the xdg-open
# fallback; xdg-desktop-portal backs the native folder picker. None are needed
# to search, so none are hard dependencies.
recommends="desktop-file-utils, xdg-utils, dbus-bin, xdg-desktop-portal"
install -dm755 "$stage/DEBIAN"
cat > "$stage/DEBIAN/control" <<EOF
Package: $PKG
Version: $deb_version
Section: utils
Priority: optional
Architecture: $arch
Maintainer: $maintainer
Installed-Size: $installed_size
Depends: $depends
Recommends: $recommends
Homepage: https://code.karsttech.com/jeremy/quick_search
Description: fast full-text search across your files
QuickSearch keeps a SQLite/FTS5 index of the directories you choose and
searches them by both filename and file content. It extracts text from
documents, PDFs, archives and office files, watches the indexed paths for
changes and reindexes in the background while the application is open.
.
The same binary doubles as a terminal search tool: "quicksearch <terms>"
prints ranked results and exits without starting the indexer, the file
watcher or any background thread.
EOF
# No postinst/postrm: hicolor-icon-theme and desktop-file-utils declare
# interest-noawait on /usr/share/icons/hicolor and /usr/share/applications, so
# dpkg refreshes both caches on install and removal by itself.
# --------------------------------------------------------------- build -----
say "Building $deb"
dpkg-deb --root-owner-group --build "$stage" "$deb" >/dev/null
rm -rf "$stage"
echo
dpkg-deb --info "$deb"
echo
say "Done: $deb"
echo " install with: sudo apt install $deb"
echo " inspect with: dpkg-deb --contents $deb"

61
packaging/copyright Normal file
View file

@ -0,0 +1,61 @@
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: quicksearch
Upstream-Contact: Jeremy <jeremy@karsttech.com>
Source: https://code.karsttech.com/jeremy/quick_search.git
Files: *
Copyright: 2025-2026 Jeremy <jeremy@karsttech.com>
License: GPL-3.0-or-later
Files: crates/quicksearch-gui/assets/icons/*
Copyright: 2026 Jeremy <jeremy@karsttech.com>
License: GPL-3.0-or-later
License: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
.
On Debian systems the full text of the GNU General Public License
version 3 can be found in /usr/share/common-licenses/GPL-3.
Comment:
/usr/bin/quicksearch and /usr/bin/quicksearch-cli are statically linked Rust
binaries built from the same crates. Their third-party components are not
shipped as separate files, so they are summarised here
rather than given individual stanzas. Every one of the 512 crates in
Cargo.lock is under a permissive licence, all of which are compatible with
GPL-3.0-or-later:
.
* MIT and/or Apache-2.0 (the large majority)
* Unicode-3.0, BSD-2-Clause, BSD-3-Clause, ISC, Zlib, 0BSD,
BSL-1.0, CC0-1.0, Unlicense
.
There are no GPL-only, LGPL-only, MPL or proprietary dependencies. Note
that a dozen crates (winit, glutin, ab_glyph, owned_ttf_parser and
related) are Apache-2.0 only; Apache-2.0 is compatible with GPL version 3
but not with GPL version 2, which is why this package is
GPL-3.0-or-later rather than GPL-2.0-or-later.
.
Bundled C sources compiled into the binary: SQLite (public domain, via
rusqlite), Zstandard (BSD-3-Clause arm of its BSD/GPL-2.0 dual licence,
via zstd-sys) and bzip2 (BSD-style, via bzip2-sys).
.
The binary also embeds egui's default fonts (crate epaint_default_fonts,
"(MIT OR Apache-2.0) AND OFL-1.1 AND Ubuntu-font-1.0"). The FSF regards
OFL-1.1 and the Ubuntu Font Licence as free but GPL-incompatible; they are
included here as font data rather than as linked program code, which is
the same basis on which Debian ships these fonts and other egui-based
applications.
.
Run `cargo metadata --all-features` against the source tree to reproduce
the per-crate licence list.

View file

@ -0,0 +1,3 @@
.\" quicksearch-cli is documented alongside quicksearch; this stub makes
.\" `man quicksearch-cli` resolve to that page.
.so man1/quicksearch.1

136
packaging/quicksearch.1 Normal file
View file

@ -0,0 +1,136 @@
.TH QUICKSEARCH 1 "2026-08-02" "quicksearch 0.1.0" "User Commands"
.SH NAME
quicksearch, quicksearch\-cli \- fast full\-text search across your files
.SH SYNOPSIS
.B quicksearch
.br
.B quicksearch
.RI [ FLAGS ]
.IR "query terms" ...
.br
.B quicksearch\-cli
.RI [ FLAGS ]
.IR "query terms" ...
.SH DESCRIPTION
.B quicksearch
maintains a SQLite/FTS5 index of the paths listed in its configuration and
searches it by filename and file content.
.PP
Invoked with no positional arguments it opens the desktop application, which
owns the indexer: it walks the configured roots, watches them for changes and
periodically reindexes.
.PP
Invoked with one or more query terms it runs the same ranked search cascade the
application uses, prints the results and exits. Terminal mode never starts the
indexer, the filesystem watcher or any background thread, so a one\-shot query
neither leaves anything running nor competes for inotify watches with a running
application. It requires an index that already exists; run the application once
to build one.
.PP
.B quicksearch\-cli
is terminal search and nothing else, and accepts the same flags and query
syntax. On this platform it is equivalent to giving
.B quicksearch
a query, and exists so that scripts and documentation have one name that never
opens a window. It matters on Windows, where the two cannot be one executable:
a GUI built as a console application flashes a console window at every launch,
and a console tool built as a GUI application cannot write back to the shell
that invoked it. Given no query it prints usage and exits 2 rather than falling
back to the application.
.SH OPTIONS
.TP
.B \-\-fuzzy
Also run the fuzzy filename and full\-text passes, which tolerate spelling
differences at the cost of speed. The edit distance comes from
.I [search].fuzzy_max_edits
in the configuration.
.TP
.BI \-\-limit " N"
Return at most
.I N
results. Defaults to
.I [search].display_limit
in the configuration.
.B \-\-limit=N
is also accepted.
.TP
.B \-\-long
Print rank, size, modification time and a matching snippet for each hit instead
of bare paths. Highlights the match in bold when stdout is a terminal.
.TP
.BR \-h ", " \-\-help
Print usage and exit.
.PP
An unrecognised option given without any query terms is passed through and the
application is opened, since it may be an option for the windowing backend.
.SH QUERY SYNTAX
Plain words form a single phrase. Filters may be combined with it:
.TP
.B type:\fIName\fR
Match a file class, for example
.IR type:Document ", " type:Image ", " type:Audio .
.TP
.B modified:\fIexpr\fR
Compare against the modification date, for example
.IR modified:>=2024-01-01 .
.TP
.B path:\fI/dir\fR
Restrict results to a directory.
.TP
.B mime:\fItype\fR
Match a MIME type, for example
.IR mime:application/pdf .
.TP
.B name:\fIfragment\fR
Match a fragment of the filename.
.SH FILES
.TP
.I ~/.config/quicksearch/config.toml
Configuration. Created with defaults on first run. See
.I /usr/share/doc/quicksearch/config_example.toml
for the annotated reference.
.TP
.I ~/.local/share/quicksearch/index.sqlite
The index. The location is set by
.I [paths].database_path
in the configuration.
.TP
.I ./config.toml
A configuration file placed next to the
.B quicksearch
executable selects portable mode and overrides the per\-user configuration
entirely. The packaged build deliberately installs no such file.
.PP
.I XDG_CONFIG_HOME
and
.I XDG_DATA_HOME
are honoured when set to absolute paths.
.SH EXIT STATUS
.TP
.B 0
The search completed, or
.B \-\-help
was requested.
.TP
.B 1
The application failed to start.
.TP
.B 2
The configuration could not be read, the index could not be opened, the query
was rejected, or
.B quicksearch\-cli
was given no query.
.SH EXAMPLES
.TP
Search for a phrase:
.B quicksearch quarterly revenue report
.TP
Recent PDFs under a directory, with details:
.B quicksearch \-\-long type:Document mime:application/pdf path:/home/me/docs modified:>=2026-01-01
.TP
Tolerate misspellings and cap the output:
.B quicksearch \-\-fuzzy \-\-limit 10 recieve
.SH SEE ALSO
.I /usr/share/doc/quicksearch/README.md
.SH AUTHOR
Jeremy <jeremy@karsttech.com>

View file

@ -0,0 +1,19 @@
[Desktop Entry]
Type=Application
Version=1.0
Name=QuickSearch
GenericName=File Search
Comment=Fast full-text search across your files
Exec=quicksearch
TryExec=quicksearch
Icon=quicksearch
Terminal=false
# One main category only (Utility), so the app shows up exactly once in the
# menu; Filesystem is the additional category that describes what it does.
Categories=Utility;Filesystem;
Keywords=search;find;index;full-text;files;content;
StartupNotify=true
# Matches ViewportBuilder::with_app_id("quicksearch") in the GUI, which becomes
# the Wayland app id and the X11 WM_CLASS. Without the match the desktop shows a
# generic window icon.
StartupWMClass=quicksearch

12
run.bat
View file

@ -1 +1,11 @@
cargo run -p quicksearch-gui @echo off
rem Build and launch the QuickSearch GUI, mirroring run.sh.
rem Terminal search is a separate binary on Windows, because the GUI is built
rem as a window-subsystem app and cannot write to the calling shell:
rem target\release\quicksearch-cli.exe --help
setlocal
cargo build --release -p quicksearch-gui
if errorlevel 1 exit /b 1
rem %~dp0 is this script's own directory (with a trailing backslash), so the
rem launch does not depend on the current working directory.
"%~dp0target\release\quicksearch.exe" %*

7
run.sh
View file

@ -1,2 +1,7 @@
#!/usr/bin/env sh
# Build and launch the QuickSearch GUI. On Unix the same binary doubles as the
# terminal search tool: ./target/release/quicksearch --help
# (On Windows that role belongs to quicksearch-cli.exe — see run.bat.)
set -e
cargo build --release -p quicksearch-gui cargo build --release -p quicksearch-gui
./target/release/quicksearch exec ./target/release/quicksearch "$@"

View file

@ -1,23 +0,0 @@
#!/usr/bin/env bash
# One-time system setup for building and running the Dioxus-based
# `quicksearch-gui` binary on Debian / Ubuntu. The core library
# (`quicksearch-core`) has no native system deps and doesn't need this.
#
# Usage:
# ./setup.sh # installs everything
#
# Adds `pkg-config` and `build-essential` explicitly because the Dioxus
# build scripts (glib-sys, gdk-sys, gio-sys, javascriptcoregtk-sys,
# webkit2gtk-sys) shell out to `pkg-config` and will refuse to build
# without it. Some minimal Ubuntu flavors and container images don't
# install pkg-config transitively.
set -e
sudo apt update
sudo apt install -y \
pkg-config \
build-essential \
libsoup-3.0-dev \
libjavascriptcoregtk-4.1-dev \
libwebkit2gtk-4.1-dev \
libxdo-dev