User-agent enforcement for downloads from dumps.wikimedia.org

--===============0369022642685341944==
Content-Type: multipart/alternative; boundary="0000000000002e123906544f5dc9"

--0000000000002e123906544f5dc9
Content-Type: text/plain; charset="UTF-8"

Howdy dumps users,

Starting by the end of June, we will begin enforcing our user-agent policy
<https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy>
on dumps published to dumps.wikimedia.org. This means that a valid
user-agent will be required on all automated requests to download dumps
files from the public facing dumps website. Requests that are missing or
provide a generic user agent (such as a basic library name) will be
blocked.



Why will valid user-agents be required?

In August 2025, we began enforcing
<https://phabricator.wikimedia.org/T400119>our long-standing user-agent
policy on other forms of programmatic access, including APIs and automated
crawlers. We are now following up with dumps
<https://phabricator.wikimedia.org/T427836> to ensure consistent policy
enforcement.

Today, more than 70% of database_backup download traffic originates from
automated processes, yet only 29% of that traffic is compliant with our
user-agent policy. Enforcing the existing user-agent policy ensures
access continues
trending more towards responsible reuse.


WMCS Access

Users who access dumps data through the Wikimedia Cloud Services fileshare
<https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Dumps> will be
unaffected. Additionally, requests to the Dumps website that originate
through WMCS services (Toolforge, CloudVPS) will be allowed access
regardless of whether or not the user-agent is compliant.


Please feel free to reach out directly through email or leave a comment in
phabricator <https://phabricator.wikimedia.org/T427836> if you have any
questions, comments, or concerns.

Cheers,

Halley

*Halley Coplin* (she/her)
Sr. Product Manager, MediaWiki Interfaces
Wikimedia Foundation <https://wikimediafoundation.org/>

--0000000000002e123906544f5dc9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><span id=3D"gmail-docs-internal-guid-2648679c-7fff-40=
68-6252-61e9b54467fb"><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0p=
t;margin-bottom:0pt"><span style=3D"color:rgb(0,0,0);background-color:trans=
parent;font-variant:normal;vertical-align:baseline;white-space:pre-wrap"><f=
ont face=3D"verdana, sans-serif" style=3D"">Howdy dumps users,</font></span=
></p><font face=3D"verdana, sans-serif"><br></font><p dir=3D"ltr" style=3D"=
line-height:1.5;margin-top:0pt;margin-bottom:0pt"><font face=3D"verdana, sa=
ns-serif"><span style=3D"color:rgb(0,0,0);background-color:transparent;font=
-variant:normal;vertical-align:baseline;white-space:pre-wrap">Starting by t=
he end of June, we will begin enforcing our </span><a href=3D"https://found=
ation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy" sty=
le=3D"text-decoration:none"><span style=3D"background-color:transparent;fon=
t-variant:normal;text-decoration:underline;vertical-align:baseline;white-sp=
ace:pre-wrap">user-agent policy</span></a><span style=3D"color:rgb(0,0,0);b=
ackground-color:transparent;font-variant:normal;vertical-align:baseline;whi=
te-space:pre-wrap"> on dumps published to </span><a href=3D"http://dumps.wi=
kimedia.org" style=3D"text-decoration:none"><span style=3D"background-color=
:transparent;font-variant:normal;text-decoration:underline;vertical-align:b=
aseline;white-space:pre-wrap">dumps.wikimedia.org</span></a><span style=3D"=
color:rgb(0,0,0);background-color:transparent;font-variant:normal;vertical-=
align:baseline;white-space:pre-wrap">. This means that a valid user-agent w=
ill be required on all automated requests to download dumps files from the =
public facing dumps website. Requests that are missing or provide a generic=
 user agent (such as a basic library name) will be blocked.=C2=A0</span></f=
ont></p><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0pt;margin-botto=
m:0pt"><span style=3D"color:rgb(0,0,0);background-color:transparent;font-va=
riant:normal;vertical-align:baseline;white-space:pre-wrap"><font face=3D"ve=
rdana, sans-serif"><br></font></span></p><font face=3D"verdana, sans-serif"=
><br></font><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0pt;margin-b=
ottom:0pt"><span style=3D"color:rgb(0,0,0);background-color:transparent;fon=
t-weight:700;font-variant:normal;vertical-align:baseline;white-space:pre-wr=
ap"><font face=3D"verdana, sans-serif">Why will valid user-agents be requir=
ed?</font></span></p><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0pt=
;margin-bottom:0pt"><font face=3D"verdana, sans-serif"><a href=3D"https://p=
habricator.wikimedia.org/T400119" style=3D"text-decoration:none"><span styl=
e=3D"background-color:transparent;font-variant:normal;text-decoration:under=
line;vertical-align:baseline;white-space:pre-wrap">In August 2025, we began=
 enforcing </span></a><span style=3D"color:rgb(0,0,0);background-color:tran=
sparent;font-variant:normal;vertical-align:baseline;white-space:pre-wrap">o=
ur long-standing user-agent policy on other forms of programmatic access, i=
ncluding APIs and automated crawlers. We are now </span><a href=3D"https://=
phabricator.wikimedia.org/T427836" style=3D"text-decoration:none"><span sty=
le=3D"background-color:transparent;font-variant:normal;text-decoration:unde=
rline;vertical-align:baseline;white-space:pre-wrap">following up with dumps=
</span></a><span style=3D"color:rgb(0,0,0);background-color:transparent;fon=
t-variant:normal;vertical-align:baseline;white-space:pre-wrap"> to ensure c=
onsistent policy enforcement.=C2=A0</span></font></p><font face=3D"verdana,=
 sans-serif"><br></font><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:=
0pt;margin-bottom:0pt"><span style=3D"color:rgb(0,0,0);background-color:tra=
nsparent;font-variant:normal;vertical-align:baseline;white-space:pre-wrap">=
<font face=3D"verdana, sans-serif">Today, more than 70% of database_backup =
download traffic originates from automated processes, yet only 29% of that =
traffic is compliant with our user-agent policy. Enforcing the existing use=
r-agent policy ensures access <span class=3D"gmail_default" style=3D"font-f=
amily:verdana,sans-serif"></span>c<span class=3D"gmail_default" style=3D"fo=
nt-family:verdana,sans-serif">ontinues trending</span> more towards respons=
ible reuse.</font></span></p><font face=3D"verdana, sans-serif"><br><br></f=
ont><p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0pt;margin-bottom:0p=
t"><span style=3D"color:rgb(0,0,0);background-color:transparent;font-weight=
:700;font-variant:normal;vertical-align:baseline;white-space:pre-wrap"><fon=
t face=3D"verdana, sans-serif">WMCS Access</font></span></p><p dir=3D"ltr" =
style=3D"line-height:1.5;margin-top:0pt;margin-bottom:0pt"><font face=3D"ve=
rdana, sans-serif"><span style=3D"color:rgb(0,0,0);background-color:transpa=
rent;font-variant:normal;vertical-align:baseline;white-space:pre-wrap">User=
s who access dumps data through the Wikimedia Cloud Services </span><a href=
=3D"https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Dumps" style=3D=
"text-decoration:none"><span style=3D"background-color:transparent;font-var=
iant:normal;text-decoration:underline;vertical-align:baseline;white-space:p=
re-wrap">fileshare</span></a><span style=3D"color:rgb(0,0,0);background-col=
or:transparent;font-variant:normal;vertical-align:baseline;white-space:pre-=
wrap"> will be unaffected. Additionally, requests to the Dumps website that=
 originate through WMCS services (Toolforge, CloudVPS) will be allowed acce=
ss regardless of whether or not the user-agent is compliant.</span></font><=
/p><font face=3D"verdana, sans-serif"><br><br></font><p dir=3D"ltr" style=
=3D"line-height:1.5;margin-top:0pt;margin-bottom:0pt"><font face=3D"verdana=
, sans-serif"><span style=3D"color:rgb(0,0,0);background-color:transparent;=
font-variant:normal;vertical-align:baseline;white-space:pre-wrap">Please fe=
el free to reach out directly through email or leave a </span><a href=3D"ht=
tps://phabricator.wikimedia.org/T427836" style=3D"text-decoration:none"><sp=
an style=3D"background-color:transparent;font-variant:normal;text-decoratio=
n:underline;vertical-align:baseline;white-space:pre-wrap">comment in phabri=
cator</span></a><span style=3D"color:rgb(0,0,0);background-color:transparen=
t;font-variant:normal;vertical-align:baseline;white-space:pre-wrap"> if you=
 have any questions, comments, or concerns.=C2=A0</span></font></p><font fa=
ce=3D"verdana, sans-serif"><br></font><p dir=3D"ltr" style=3D"line-height:1=
.5;margin-top:0pt;margin-bottom:0pt"><span style=3D"color:rgb(0,0,0);backgr=
ound-color:transparent;font-variant:normal;vertical-align:baseline;white-sp=
ace:pre-wrap"><font face=3D"verdana, sans-serif">Cheers,</font></span></p><=
p dir=3D"ltr" style=3D"line-height:1.5;margin-top:0pt;margin-bottom:0pt"><s=
pan style=3D"color:rgb(0,0,0);background-color:transparent;font-variant:nor=
mal;vertical-align:baseline;white-space:pre-wrap"><font face=3D"verdana, sa=
ns-serif" style=3D"">Halley</font></span></p></span><br class=3D"gmail-Appl=
e-interchange-newline"></div><div><div dir=3D"ltr" class=3D"gmail_signature=
" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><table style=3D"font-=
size:14px;color:rgb(32,33,34);font-family:sans-serif;border:medium"><tbody>=
<tr><td style=3D"padding:5pt"><img src=3D"https://ci3.googleusercontent.com=
/mail-sig/AIorK4xedD1OqxuSsdYWxbOzn9kzD_wQty0km-o42tJNRtaflZWAiETpTUphgPhJy=
23nCCpFqIn69adb4Scv"><img alt=3D""><img alt=3D""></td><td style=3D"vertical=
-align:top;padding:5pt;line-height:1.38"><font face=3D"Verdana"><b style=3D=
"font-size:9pt">Halley Coplin</b><span style=3D"font-size:9pt">=C2=A0(she/h=
er)</span><br><font size=3D"1">Sr. Product Manager, MediaWiki Interfaces</f=
ont><br><a href=3D"https://wikimediafoundation.org/" title=3D"foundationsit=
e:" style=3D"font-size:9pt;color:rgb(102,51,102);background:none" target=3D=
"_blank"><span style=3D"color:rgb(0,0,0);font-size:8pt">Wikimedia Foundatio=
n</span></a></font></td></tr></tbody></table></div></div></div></div>

--0000000000002e123906544f5dc9--

--===============0369022642685341944==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Wikitech-l mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
--===============0369022642685341944==--