Re: Can GNU-find command search by xattrs?

On 1/7/26 01:24, raf wrote:
> On Sat, Jan 03, 2026 at 10:05:16PM +0100, "Peter B." <[email protected]> wrote:
>
>> Hi!
>>
>> I'm super happy to see that I'm not the only one anymore interested in
>> increased xattr, and  therefore possible key/value usage and query
>> functionality!
>>
>> 😀️
>>
>> **I'm writing this with personal interest**
>> Yes, we ALL have data to deal with, and I like to tag and find my kids'
>> photos and files - regardless which "app" or system-
>>
>> **as well as my professional interest**
>> Working with small-to-medium-to-very-very-large digital heritage
>> collections.
>>
>> (And therefore loads of meta+data wrangling clever-hacks-and-stunts for a
>> living 😉️)
>>
>>
>> So, my hopefully useful "few-cents" to this thread are here:
>>
>> On 1/2/26 13:11, raf wrote:
>>> On Wed, Dec 31, 2025 at 04:21:14PM +0100, Bernhard Voelker <[email protected]> wrote:
>>>
>>>> On 12/30/25 17:45, Peter B. wrote:
>>>>> I'll also checkout Morgan's find-patch:
>>>>> I'd really love to see xattr-support in the basic "most likely to be
>>>>> present-on-any-box" tools.
>>>> The technical coding under the hood for reading xattr from the file system
>>>> is there in Morgan's patch for quite a while, indeed.
>>>>
>>>> But before adding, I feel we need to discuss once again the interface
>>>> to the find(1) user:
>>>>
>>>> - Do we only want to provide an option to search for files having xattrs?
>>>>       find -xattr
>>>>
>>>> - Do we want a test option to search for a file having an xattr key matching
>>>>     a certain string (or eventually pattern)?
>>>>       find -xattr 'mykey'        # xattr key equals string
>>>>       find -xattr '*mykey*'      # xattr key matches pattern
>>>>     Or better explicitly mention in the option that we match for the xattr keys?
>>>>       find -xattr-key 'mykey'
>>>>       find -xattr-key '*mykey*'
>>>>
>>>> - Do we want a test option to search for a file having an xattr value matching
>>>>     a certain string (or eventually pattern)?
>>>>       find -xattr-value 'myval'   # files with xattr value equals string
>>>>       find -xattr-value '*myval*' # ... or pattern
>>>>
>>>> - Or search for files with xattr having a certain mixture of 'key=val'?
>>>>       find -xattr-match 'mykey=myval'      # search by key+val as strings
>>>>       find -xattr-match 'mykey=*someval*'  # search for key matching a val pattern
>>> There are versions of find that (I think) use -xattr to
>>> just identify the existence of EAs. I don't think
>>> that's enough, but it might make sense to have -xattr
>>> do that
>> +1
>> absolutely agree.
>> To both: useful and not-enough.
>>
>>
>>> (for compatibility with those other versions of
>>> find), but to also have -xattr-key and -xattr-value (or
>>> -xattr-match for both).
>> +1 again.
>> It indeed makes perfect sense (and has use-cases!) where one wants to query
>> (key OR value) AND (key AND value).
>>
>> A lot useful if that provides some RegEx (for matching wildcards/patterns).
>>
>>
>>> The thing to watch out for if both the key and value
>>> are combined, is that if you format it like "key=value"
>>> you need to consider the case where the key includes
>>> "=". There might be malicious EAs trying to be tricky.
>> True.
>> There will always be potential for malicious attempts, yet I would suggest
>> to apply basic "query-escaping", until more strict antitrust seems
>> necessary?
>>
>>
>>> My rawhide program matches EAs formatted like "key:
>>> value" but non-ascii bytes are encoded (like \x1b or \n
>>> or \t etc.) and any ": " in the key itself is encoded
>>> as "\x3a " to disappoint the creators of malicious EAs.
>>> I've never seen an EA whose key contained ": " so I
>>> don't think it'll bother anyone.
>> I think we're on the same page here :)
>> Yet, one may say "xattrs are not-yet-popular enough, but you'll see, once
>> they do..." - like with all formats and data-exchange protocols, I guess?
> EAs/xattrs are used for various things by various
> systems. (e.g. selinux on Linux, quarantine on macOS)
> but user/app-specific usage probably isn't popular yet.
> Although it can be. Some systems allow 64KiB of EA data.

Thanks for info! I can confirm that.
btw, app-specific usage: yt-dlp supports `--xattrs` 😉️

I've put some FAQ infos together regarding xattrs (in 2024/25):
https://github.com/ArkThis/AHAlodeck/blob/main/doc/xattr_faq.md

The FAQ also contains xattr size-constraints per filesystem I was able 
to find (or test it).
ZFS can do 64kiB, indeed: I must admit I still want to test if PER "key 
and/or value" - or in total.
I've done tests, attaching a ~45kB uuencoded JPG image as xattr on ZFS 
and back: MD5 match.

I love the fact that EAs are used for security purposes:
I can assume it means relying on them is enterprise-production-stable 
(at least regarding the kernel/filesystem side)?


> > What if:
> > - I search for "some term", but don't care if it's a key or value string?

> Having -xattr-key and -attr-value wouldn't help you in
> that case. If you didn't care if it's a key or a value,
> then you'd need to use both predicates in your search
> (-xattr-key ... -o -xattr-value ...).
>
> But the problem I was thinking of was that, if these
> two predicates existed, then a user might reasonably
> expect them to be connected to each such that using
> both of them would refer to the same extended
> attribute, i.e. that -xattr-key ... -a -xaatr-value ...
> would match a file with an EA whose key matched the
> -xattr-key and whose value matched the -xattr-value.
> But it would almost certainly match a file with an EA
> with the matching key and an EA with the matching
> value, but they wouldn't have to be the same EA. They
> could be different EAs. So I think it would be
> difficult for a user to guess the correct behaviour.

I was thinking of defaulting to "linked" (=like `keep aspect ratio` by 
default),
and options to declare if my intention is linked/unlinked?

>> I have that use-case a lot: plain fulltext search in collections.
> That use case is best served (I think) by a single
> predicate that can match both the key and value encoded
> in some way as I described earlier (either by
> "key=val\n" or "key: val\n"). I chose ": " because
> that's the format output by xattr on macOS (and
> probably elsewhere).

Hm.
Does that mean you're matching against the output of `xattr` for 
searching, or just that you'd like to stay pattern-compatible with it?

> That way, if you don't care whether your search
> criteria applies to the key or the value, you can just
> do -xattr-match 'something' and if you want to match a
> key only you can do -regexxattr-match '^somekey: ' and
> if you want to match a value only you can do
> -regexxattr-match '^[^\n]+: [^\n]*someval', and to
> match both the key and the value, you can do something
> like -regexxattr-match '^somekey: [^\n]*someval'.

Roger that.
Is there any overhead due to requiring regex for this, compared to 
non-pattern matching here?


> The above assumes that all EAs are encoded as a single
> piece of text with one line per EA that is matched
> against. The choice of encoding and whether it's
> encoded like this at all would affect the examples.
> This sort of matching probably isn't possible with
> globbing unless the ksh extensions are available.
>
>>> I'd vote for just -xattr and -regexxattr (and maybe
>>> -ixattr and -iregexxattr) and have it match text that
>>> looks like "key1=val1\nkey2=val2\nkey3=val3\n" or
>>> "key1: val1\nkey2: val2\nkey3: val3\n".
>> I like args names that explain what they are, but just for "style"
>> suggestion:
>>
>> [quick bike-shedding]:
>> What if, you'd call it "--xattr" and "--xxattr" or "--xattRX/xattrx"? (like
>> the "RX" at the end is "RegEx" :P)
>>
>> Just an idea...
> I suggested names that I thought were like the existing predicate names.
> I don't know what would be best.

Sorry, didn't know there were existing ones:
Then of course it makes sense to keep their names.

>
>> And I assume "ixattr" is "case Insensitive"?
> That's right.
>
>>>> - Do we need support for -printf formats to print xattr keys and/or values?
>>>>     How? The % 1-character directives are almost all used, maybe begin with
>>>>     the reserved ones? (%{ %[ %(
>>>>     How to output several xattr keys or values?
>>>>     How to select which xattr value to print?
>>>>       find -xattr -printf '%p %{xattr-keys=hello*} %{xattr-valuekey=hello}
>>> There are plenty of conversion letters available!
>>>
>>> See https://savannah.gnu.org/bugs/?64100
>> Having quickly read over that thread, I totally agree "IF those features be
>> added, IF possible, it'd be great if rawhide and find could stay
>> syntax-compatible".
>>
>> Also, I think (If I understood correctly) that some common-library-printf
>> formatting syntax may save A LOT of re-occurring bash-or-python-style
>> "formatting-caller-works-on-my-machine" scripts, I guess?
>> But I'm far out on a limb here; I've not really read the thread, and I've
>> never used find-output formatting (as I didn't know it had any).
> Only GNU find has it.
>
>>>> There's a lot of possibilities, and I don't want to introduce something
>>>> which contradicts the typical use cases, or is not extensible or not
>>>> maintainable.
>>>> The complexity - especially when it comes to -printf - is that files can have
>>>> several xattr while all other attributes (name, timestamps, permissions, size,
>>>> etc.) only exists once.
>>> My rawhide program also has %j for JSON output. The
>>> current version outputs EAs like %x but the next
>>> version outputs them as a JSON object with key/value
>>> pairs matching the EA names and values. It's still
>>> encoded because EA values can be text or binary and
>>> JSON can't represent binary without the user choosing
>>> some encoding. So that's something to think about.
>> If any output data is available in a machine readable format (like JSON),
>> it'd definitely make the re-usability and interoperability between tools
>> (and scripts) written around find/rawhide, IMO (and experience).
>>
>>>> I think I asked this kind of questions already some years ago, but there was
>>>> no input yet.
>>>> I personally don't have a use case to search for xattrs, so the usual pattern
>>>> with '-exec getfattr ...' works for me.
>>>> Anyone?
>> ME! Meee! :)
>> As I said, I'm doing devops for digital-and-physical GLAM-collections - and
>> a lot of metadata layout, storage and retrieval/usage.
>>
>> For example:
>> After de-embedding (=copying as-is) all exiftool-readable metadata into
>> xattrs, being able to do some queries on those key/value pairs is AMAZING!
>>
>> Combining that with RDF URIs for keys, combined with Wikibase-and-Wikidata
>> engine and data, I have many great use cases for xattrs (and find).
> That sounds cool. A much more interesting use for EAs than
> just selinux or quarantining. :-)

:)

I really like the fact that once I can rely more on xattrs use,
**any file-format ever is basically irrelevant from now on, for "all 
things metadata"**.
And this feels really good!

Like: Why must I choose another file-format, if I simply want to give 
"my file/data" annotation information?
(BMP vs PNG vs TIFF vs ... ".txt" regarding storing "their title"?)

Friend of mine has written a small "updatedb/locate" version with 
xattr-only support a few days ago for testing:
He ran it on a collection of mixed-format, real-world annotated music 
collection (~14.000 files) - which have all exiftool metadata as xattrs: 
13917 files took 1.416 seconds to index.

The index.db entries is a simple CSV, with: `"filename", "xattr-key", 
"xattr-value"`

Using xattrs already feel more streamlined, consistent and stable than 
any MAM/DAM metadata handling paradigm I've seen so far... ;)

I've rediscovered xattrs btw, when evaluating object storage 
implementations (FOSS) for large-scale data handling:

When I found out that S3 has very limited size for tags (127/256 chars) 
and other limitations (<=4kB) for metadata, or ASCII-only in popular 
implementations. See: [Minio issue #19576: "non ASCII characters in 
metadata"](https://github.com/minio/minio/discussions/19576)

...and pre-S3 Object Storages use the underlying filesystem's xattrs for 
their object-metadata! ;)


>>>> P.S. Finally, I wouldn't want to introduce too much and complex code for
>>>> 0.25% use cases.  Eventually, the simple test to search for files having
>>>> xattrs is enough, and the application logic then extracts them with another
>>>> tool.
>> Is this about the code to handle xattrs (in `gnu-find`)?
>> There are libs for plain access to xattr key/value, but is there any
>> xattr-libs for higher-level functions (like queries, regex, etc)?
> I think the actual searching would be done by what find is already doing
> (glob, regex).

Roger.

>> Like what HaikuOS (probably?) does in their BeFS filesystem libraries?
>> A friend of mine (author of "https://sen-labs.org/": a semantic desktop
>> engine for Haiku) told me that BeFS has some database-like query functions
>> built-into their filesystem "xattrs".
>>
>> I know any additional library-dependence is another "whole thing" to
>> maintain.
>> However, IMO being able to query key/value data "on the spot" is something
>> that will eventually replace the legacy "files-in-folders open/save dialogs
>> - and wildcards" as-we-know-it...?
>>
>> Might be an incentive to invest in a common, and commonly supported
>> "attribute-handling" lib.
>> Just 2¢ of mine.
>>
>> I just like interoperable and open tech-systems :)
>>
>>
>>> I don't think it's necessarily too complex. The code in
>>> rawhide to obtain EAs is 492 lines (335 excluding
>>> blanks line, 278 excluding comment lines as well) for
>>> Linux, macOS, Cygwin, Solaris, and FreeBSD (OpenBSD and
>>> NetBSD don't have EAs). But of course, that's only the
>>> start.
>> Including so many OSs already in your consideration(s), wouldn't that
>> already suffice for a common lib?
>> Ignore and correct me, if I'm totally on the wrong track here?
>> Thanks :)
> rawhide is GPLv3+ so the code is available for a
> library but if potential clients don't like my choices
> for encoding the EA data it might not be suitable for
> their needs. Encoding is always tricky to get right,
> and maybe I didn't. And it's probably unwise to use a
> third-party library until it has become popular enough
> to be packaged on lots of systems. And GNU find
> supports many more systems than rawhide does, and I
> don't know which other systems support EAs. I only
> support systems that I have real or virtual machines
> for.

Thanks for those interesting insights!
Indeed, good point: no lib-dependency until "popular enough".

Kind regards, Peter