Re: HSTS cache cap allows eviction of security entries

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--===============8519937636978710781==
Content-Language: en-US
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature";
 boundary="------------L1T088J7BCaysNXv9LugOMdh"

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--------------L1T088J7BCaysNXv9LugOMdh
Content-Type: multipart/mixed; boundary="------------3A90H8p6KqHKaktT4xnMrAFb";
 protected-headers="v1"
From: Timothe Litt <[email protected]>
To: Daniel Stenberg <[email protected]>,
 Timothe Litt via curl-library <[email protected]>
Message-ID: <[email protected]>
Subject: Re: HSTS cache cap allows eviction of security entries
References: <[email protected]>
 <[email protected]>
 <[email protected]>
In-Reply-To: <[email protected]>

--------------3A90H8p6KqHKaktT4xnMrAFb
Content-Type: multipart/alternative;
 boundary="------------UGFSo50UBaKlVD90cL6bbt4M"

--------------UGFSo50UBaKlVD90cL6bbt4M
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64

WW91IHdvdWxkIGFsd2F5cyBzdG9yZSAobmV3LCB1cGRhdGVkKSByZWNvcmRzIG9uIGRpc2sg
LSB3aGljaCBpcyANCmVzc2VudGlhbGx5IHVubGltaXRlZC7CoCBbSWYgYW4gYWR2ZXJzYXJ5
IGNhbiBtYW5hZ2UgdG8gZmlsbCB5b3VyIGRpc2sgDQp3aXRoIEhTVFMgZGF0YSwgcGVyaGFw
cyBoZSBkZXNlcnZlcyB0byB3aW4uwqAgWW91IGhhdmUgYmlnZ2VyIHByb2JsZW1zLl0NCg0K
SW5pdGlhbGx5LCB5b3UgcmVhZCBmcm9tIGRpc2sgdG8gZ2V0IHRoZSBIU1RTIHJlY29yZChz
KSBlYWNoIHRpbWUgeW91IA0KZW5jb3VudGVyIGEgc2l0ZS4NCg0KWW91IGRvIG5vdCBsb2Fk
IHRoZSBlbnRpcmUgZGlzayBmaWxlIGludG8gbWVtb3J5Lg0KDQpOb3cgc2l0ZXMgd2l0aG91
dCBIU1RTIGluY3VyIGEgZGlzayByZWFkLCBidXQgdGhlcmUncyBubyByZWNvcmQuDQoNCkNh
Y2hlIHRoZSByZXN1bHQgKGluIG1lbW9yeSnCoCBlaXRoZXIgd2F5LsKgIFRoZSBuZXh0IHJl
ZmVyZW5jZSB0byBzdWNoIGEgDQpzaXRlIHdvbid0IGRvIGEgcG9pbnRsZXNzIHJlYWQuDQoN
Ck1vc3Qgc2l0ZXMgZG9uJ3QgaGF2ZSBIU1RTIHJlY29yZHMsIHNvIGEgbmVnYXRpdmUgcmVz
dWx0IGlzIHRoZSBjb21tb24gDQpjYXNlLi4NCg0KTWFrZSB0aGUgKm1lbW9yeSogY2FjaGUg
TFJVICYgbGltaXQgaXRzIHNpemUuwqAgRnJlcXVlbnRseSB1c2VkIHNpdGVzIA0Kd2lsbCBo
aXQgaW4gbWVtb3J5IGFzIGJlZm9yZS7CoCBGaXJzdCBhY2Nlc3Mgd2lsbCBpbmN1ciB0aGUg
ZGlzayBwZW5hbHR5LCANCndoaWNoIHlvdSBjYW4gb3B0aW9uYWxseSByZWR1Y2UgYnkgbG9h
ZGluZy9zYXZpbmcgdGhlICptZW1vcnkqIGNhY2hlIGluIA0KYW5vdGhlciBmaWxlIC0gaWYg
YWNjZXNzIHBhdHRlcm5zIG1ha2UgaXQgd29ydGh3aGlsZS4NCg0KVGhpcyBtYWtlcyB0aGUg
SFNUUyBkYXRhIHN0b3JlIHVubGltaXRlZCwgYW5kIGJvdW5kcyB0aGUgbWVtb3J5IA0KZm9v
dHByaW50IG5vIG1hdHRlciBob3cgbWFueSBzaXRlcyBhbiBpbnN0YW5jZSBvZiBsaWJjdXJs
IHZpc2l0cy4gVGhlcmUgDQppcyBhIHBlcmZvcm1hbmNlIHRyYWRlb2ZmIC0gZGVwZW5kaW5n
IG9uIGhvdyBlZmZlY3RpdmUgeW91ciBjYWNoZSBpcy4NCg0KSWYgeW91IHRoaW5rIHlvdSds
bCBoYXZlICJ6aWxsaW9ucyIgb2Ygc2l0ZXMsIHlvdSBjYW4gaW5kZXgsIHNoYXJkLCBvciAN
CnNvcnQgdGhlIGZpbGUsIG9yIG1vdmUgZnJlcXVlbnQgcmVjb3JkcyB0byB0aGUgdG9wIGF0
IHJ1bmRvd24uDQoNCkJ1dCBhIHNpbXBsZSBsaW5lYXIgc2VhcmNoIG9mIHRoZSBkaXNrIGZp
bGUgcHJvYmFibHkgaGFuZGxlcyBhbGwgYnV0IA0KZXh0cmVtZSB1c2UgY2FzZXMuwqAgwqBB
IGRhdGFiYXNlIHNjYWxlcyB0byAxMDBzIG9mIG1pbGxpb25zIG9mIHNpdGVzIGZvciANCnN1
Y2ggY2FzZXMgLSBpZiB5b3UgdGhpbmsgdGhhdCBhIGEgbG9uZy1saXZlZCBjcmF3bGVyIGlz
IGdvaW5nIHRvIGFjY2VzcyANCnRoYXQgbWFueSBIU1RTIHNpdGVzLsKgIEknbSBub3QgaW5j
bGluZWQgdG8gd29ycnkgYWJvdXQgdGhhdCAtIGJ1dCB5b3UgDQphbHdheXMgZ28geW91ciBv
d24gd2F5Lg0KDQoNClRpbW90aGUgTGl0dA0KQUNNIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIN
Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpUaGlzIGNvbW11bmljYXRpb24gbWF5IG5v
dCByZXByZXNlbnQgdGhlIEFDTSBvciBteSBlbXBsb3llcidzIHZpZXdzLA0KaWYgYW55LCBv
biB0aGUgbWF0dGVycyBkaXNjdXNzZWQuDQoNCk9uIDAxLUFwci0yNiAxNzoxNCwgRGFuaWVs
IFN0ZW5iZXJnIHdyb3RlOg0KPiBPbiBXZWQsIDEgQXByIDIwMjYsIFRpbW90aGUgTGl0dCB2
aWEgY3VybC1saWJyYXJ5IHdyb3RlOg0KPg0KPj4gU3RvcmUgdGhlIEhTVFMgbGlzdCBvbiBk
aXNrIChwZXJzaXN0aW5nIGl0IGlzIGdvb2QpOw0KPg0KPiBUaGF0J3Mgd2h5IGxpYmN1cmwg
b2ZmZXJzIHRoYXQgaW4gaXRzIEFQSS4gSXQgc3RpbGwgbmVlZHMgdG8gYmUgYWJsZSANCj4g
dG8gZnVuY3Rpb24gd2l0aCB0aGUgY2FjaGUgaW4gbWVtb3J5Lg0KPg0KPj4geW91IGNhbiB1
c2UgYSBtZW1vcnkgY2FjaGUgb2YgYm90aCBwb3NpdGl2ZSBbc2l0ZSAtPiBIU1RTIHJlY29y
ZHNdwqAgDQo+PiBhbmQgbmVnYXRpdmUgW3NpdGUgLT4gJ2hhcyBubyBIU1RTJ13CoCBlbnRy
aWVzIC0gYW5kIGxpbWl0IGl0cyBzaXplLg0KPg0KPiBBIHR5cGljYWwgdXNlciBzY2VuYXJp
byBoYXMgcGVyaGFwcyBhIGhhbmRmdWwgb2YgaG9zdG5hbWVzIGluIHRoZSBsaXN0IA0KPiB0
aGF0IHNob3VsZCBiZSBidW1wZWQgdG8gSFRUUFMuIEkgZG9uJ3Qgc2VlIGhvdyBhZGRpbmcg
bmVnYXRpdmUgaW5mbyANCj4gdG8gdGhpcyBtYWtlcyB0aGUgZGF0YSBzbWFsbGVyLg0KPg0K
Pj4gT3IgbGV0IGEgZGF0YWJhc2UgKGUuZy4gU1FMaXRlKSBtYW5hZ2UgdGhlIGxpc3QuwqAg
WW91IGRvbid0IGhhdmUgdG8gDQo+PiBpbnZlbnQgeW91ciBvd24uDQo+DQo+IFRoaXMgaXMg
b3Zlci1lbmdpbmVlcmluZyB0ZXJyb3JpdHkuIFNRTGl0ZSBpcyBpdHNlbGYgbGFyZ2VyIHRo
YW4gdGhlIA0KPiB3aG9sZSBvZiBsaWJjdXJsLiBIU1RTIGlzIGEgcmF0aGVyIHRpbnkgZWRn
ZSBmZWF0dXJlLiBObyBvbmUgd2FudHMgDQo+IGxpYmN1cmwgdG8gZXhwbG9kZSBpbiBzaXpl
IGFuZCBjb21wbGV4aXR5IGp1c3QgdG8gc3VwcG9ydCB0aGlzLg0KPg0K
--------------UGFSo50UBaKlVD90cL6bbt4M
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF=
-8">
  </head>
  <body>
    <p>You would always store (new, updated) records on disk - which is
      essentially unlimited.=C2=A0 [If an adversary can manage to fill yo=
ur
      disk with HSTS data, perhaps he deserves to win.=C2=A0 You have big=
ger
      problems.]</p>
    <p>Initially, you read from disk to get the HSTS record(s) each time
      you encounter a site.</p>
    <p>You do not load the entire disk file into memory.=C2=A0=C2=A0</p>
    <p></p>
    <p>Now sites without HSTS incur a disk read, but there's no record.</=
p>
    <p>Cache the result (in memory)=C2=A0 either way.=C2=A0 The next refe=
rence to
      such a site won't do a pointless read.</p>
    <p>Most sites don't have HSTS records, so a negative result is the
      common case..</p>
    <p>Make the *memory* cache LRU &amp; limit its size.=C2=A0 Frequently=

      used sites will hit in memory as before.=C2=A0 First access will in=
cur
      the disk penalty, which you can optionally reduce by
      loading/saving the *memory* cache in another file - if access
      patterns make it worthwhile.</p>
    <p>This makes the HSTS data store unlimited, and bounds the memory
      footprint no matter how many sites an instance of libcurl visits.=C2=
=A0
      There is a performance tradeoff - depending on how effective your
      cache is.</p>
    <p>If you think you'll have "zillions" of sites, you can index,
      shard, or sort the file, or move frequent records to the top at
      rundown.=C2=A0=C2=A0</p>
    <p>But a simple linear search of the disk file probably handles all
      but extreme use cases.=C2=A0 =C2=A0A database scales to 100s of mil=
lions of
      sites for such cases - if you think that a a long-lived crawler is
      going to access that many HSTS sites.=C2=A0 I'm not inclined to wor=
ry
      about that - but you always go your own way.</p>
    <p><br>
    </p>
    <pre class=3D"moz-signature" cols=3D"72">Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.=20
</pre>
    <div class=3D"moz-cite-prefix">On 01-Apr-26 17:14, Daniel Stenberg
      wrote:<br>
    </div>
    <blockquote type=3D"cite"
      cite=3D"mid:[email protected]">On Wed, 1=

      Apr 2026, Timothe Litt via curl-library wrote:
      <br>
      <br>
      <blockquote type=3D"cite">Store the HSTS list on disk (persisting i=
t
        is good);
        <br>
      </blockquote>
      <br>
      That's why libcurl offers that in its API. It still needs to be
      able to function with the cache in memory.
      <br>
      <br>
      <blockquote type=3D"cite">you can use a memory cache of both
        positive [site -&gt; HSTS records]=C2=A0 and negative [site -&gt;=

        'has no HSTS']=C2=A0 entries - and limit its size.
        <br>
      </blockquote>
      <br>
      A typical user scenario has perhaps a handful of hostnames in the
      list that should be bumped to HTTPS. I don't see how adding
      negative info to this makes the data smaller.
      <br>
      <br>
      <blockquote type=3D"cite">Or let a database (e.g. SQLite) manage th=
e
        list.=C2=A0 You don't have to invent your own.
        <br>
      </blockquote>
      <br>
      This is over-engineering terrority. SQLite is itself larger than
      the whole of libcurl. HSTS is a rather tiny edge feature. No one
      wants libcurl to explode in size and complexity just to support
      this.
      <br>
      <br>
    </blockquote>
    <div id=3D"grammalecte_menu_main_button_shadow_host"
      style=3D"width: 0px; height: 0px;"></div>
  </body>
</html>

--------------UGFSo50UBaKlVD90cL6bbt4M--

--------------3A90H8p6KqHKaktT4xnMrAFb--

--------------L1T088J7BCaysNXv9LugOMdh
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature.asc"

-----BEGIN PGP SIGNATURE-----

wsB5BAABCAAjFiEE0UvvF0GpbrNhifE5DTaRiR4XoSQFAmnNlZMFAwAAAAAACgkQDTaRiR4XoSRd
AAf/TSB+h3zaAV5gt5rIZB8ieBMdtYQISOHbiDCkQSWInMYkKvj006rGcZcPcUVKxNSpI/dHAeib
zP3H6FCo3mz1hyJicP38Kk7zkzcBL5cfvFd3HpzqFg0zZ5tNAhZix0B44IbfZu1cwdHduVbUSOYU
IDGVnxDRkTo2Rhz44hnglptEnqKvj+8tl7ZTz3JGEc+aj7FAht9kxVLYOshn/N2OXsQY1j7Yf2zb
VXxxvBB0fA3yXfIcuYy8YVDcW7A59bYe1VbvSqz8+uo5AFPN1Fm6Lnn/gCXW5z7V1LqgxS6WVamp
sDXy/osM2oVy/wSr73qmKEONBUpB7Nh22rAy1uBNOQ==
=WMHh
-----END PGP SIGNATURE-----

--------------L1T088J7BCaysNXv9LugOMdh--

--===============8519937636978710781==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

--===============8519937636978710781==--