Re: Question about "Page abstract for Yahoo" dumps

--===============0141597869823948653==
Content-Type: multipart/alternative; boundary="000000000000de493a06518bb3cb"

--000000000000de493a06518bb3cb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

(For context to others reading this, the page abstract dumps were removed
on T382069 Undeploy and archive ActiveAbstract
<https://phabricator.wikimedia.org/T382069> due to cost and low usage).

Bernd:

I think you could re-implement your use case by leveraging Wikimedia
Enterprise's Snapshot API:

curl -L '
https://api.enterprise.wikimedia.com/v2/snapshots/enwikibooks_namespace_0/d=
ownload'
\
  -H 'Authorization: Bearer ACCESS_TOKEN' \
  --output enwikibooks.tar.gz

That will give you compressed NDJSON, and you can further parse it like so:

tar -xzf enwikibooks.tar.gz
# Each line is one article; extract name, url, abstract:
cat enwikibooks.ndjson | jq '{name: .name, url: .url, abstract: .abstract}'

More info at https://enterprise.wikimedia.com/project-data/wikibooks-api/

Note Enterprise is free for non-profit use cases. You do need to have an
account with them to get the authorization token.

Hope this helps,
-xabriel

On Mon, May 11, 2026 at 8:44=E2=80=AFAM Bernd Fehling via Xmldatadumps-l <
[email protected]> wrote:

> Dear list,
>
> we have included parts of wikibooks (languages with most content) in our
> academic search engine.
> The pages are generated by getting title, url and abstract from wikibooks
> abstract.xml
> This is combined with pages-arcticles.xml to build metadata pages for
> indexing.
> Unfortunately, the building of wikibooks-latest-abstract.xml.gz has been
> discontinued last year.
> Currently we have 389.429 records indexed with 24 languages but they are
> now over a year old.
>
> Is there any replacement for "Page abstract for Yahoo" dumps?
>
> If not, is there any easy solution to fetch or generate these data someho=
w?
>
> Kind regards
> Bernd
>
> --
> *************************************************************
> Bernd Fehling                    Bielefeld University Library
> Dipl.-Inform. (FH)                LibTec - Library Technology
> Universit=C3=A4tsstr. 25                  and Knowledge Management
> 33615 Bielefeld
> Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
>            https://www.ub.uni-bielefeld.de/~befehl/
>
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *************************************************************
>
> _______________________________________________
> Xmldatadumps-l mailing list -- [email protected]
> To unsubscribe send an email to xmldatadumps-l-leave-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org
>


--=20
Xabriel J. Collazo Mojica (he/him, pronunciation
<https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronuncia=
tion.ogg>
)
Staff Data Engineer
Wikimedia Foundation

--000000000000de493a06518bb3cb
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">(Fo=
r context to others reading this, the page abstract dumps were removed on=
=C2=A0<a href=3D"https://phabricator.wikimedia.org/T382069">T382069 Undeplo=
y and archive ActiveAbstract</a>=C2=A0due to cost and low usage).</div><div=
 class=3D"gmail_default" style=3D"font-size:small"><br></div><div class=3D"=
gmail_default" style=3D"font-size:small">Bernd:</div><div class=3D"gmail_de=
fault" style=3D"font-size:small"><br></div><div class=3D"gmail_default" sty=
le=3D"font-size:small">I think you could re-implement your use case by leve=
raging Wikimedia Enterprise&#39;s=C2=A0Snapshot API:</div><div class=3D"gma=
il_default" style=3D"font-size:small"><br></div><div class=3D"gmail_default=
" style=3D"font-size:small"><font face=3D"monospace">curl -L &#39;<a href=
=3D"https://api.enterprise.wikimedia.com/v2/snapshots/enwikibooks_namespace=
_0/download">https://api.enterprise.wikimedia.com/v2/snapshots/enwikibooks_=
namespace_0/download</a>&#39; \<br>=C2=A0 -H &#39;Authorization: Bearer ACC=
ESS_TOKEN&#39; \<br>=C2=A0 --output enwikibooks.tar.gz</font></div><div cla=
ss=3D"gmail_default" style=3D"font-size:small"><br></div><div class=3D"gmai=
l_default" style=3D"font-size:small">That will give you compressed NDJSON, =
and you can further parse it like so:</div><div class=3D"gmail_default" sty=
le=3D"font-size:small"><br></div><div class=3D"gmail_default" style=3D"font=
-size:small"><font face=3D"monospace">tar -xzf enwikibooks.tar.gz<br># Each=
 line is one article; extract name, url, abstract:<br>cat enwikibooks.ndjso=
n | jq &#39;{name: .name, url: .url, abstract: .abstract}&#39;</font></div>=
<div class=3D"gmail_default" style=3D"font-size:small"><br></div><div class=
=3D"gmail_default" style=3D"font-size:small">More info at=C2=A0<a href=3D"h=
ttps://enterprise.wikimedia.com/project-data/wikibooks-api/">https://enterp=
rise.wikimedia.com/project-data/wikibooks-api/</a></div><div class=3D"gmail=
_default" style=3D"font-size:small"><br></div><div class=3D"gmail_default" =
style=3D"font-size:small">Note Enterprise=C2=A0is free for non-profit use c=
ases. You do need to have an account with them to get the authorization tok=
en.</div><div class=3D"gmail_default" style=3D"font-size:small"><br></div><=
div class=3D"gmail_default" style=3D"font-size:small">Hope this helps,</div=
><div class=3D"gmail_default" style=3D"font-size:small">-xabriel</div></div=
><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" clas=
s=3D"gmail_attr">On Mon, May 11, 2026 at 8:44=E2=80=AFAM Bernd Fehling via =
Xmldatadumps-l &lt;<a href=3D"mailto:[email protected]">xm=
[email protected]</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">Dear list,<br>
<br>
we have included parts of wikibooks (languages with most content) in our ac=
ademic search engine.<br>
The pages are generated by getting title, url and abstract from wikibooks a=
bstract.xml<br>
This is combined with pages-arcticles.xml to build metadata pages for index=
ing.<br>
Unfortunately, the building of wikibooks-latest-abstract.xml.gz has been di=
scontinued last year.<br>
Currently we have 389.429 records indexed with 24 languages but they are no=
w over a year old.<br>
<br>
Is there any replacement for &quot;Page abstract for Yahoo&quot; dumps?<br>
<br>
If not, is there any easy solution to fetch or generate these data somehow?=
<br>
<br>
Kind regards<br>
Bernd<br>
<br>
-- <br>
*************************************************************<br>
Bernd Fehling=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 Bielefeld University Library<br>
Dipl.-Inform. (FH)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 L=
ibTec - Library Technology<br>
Universit=C3=A4tsstr. 25=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 and Knowledge Management<br>
33615 Bielefeld<br>
Tel. +49 521 106-4060=C2=A0 =C2=A0 =C2=A0 =C2=A0bernd.fehling(at)<a href=3D=
"http://uni-bielefeld.de" rel=3D"noreferrer" target=3D"_blank">uni-bielefel=
d.de</a><br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"https://www.ub.uni-biel=
efeld.de/~befehl/" rel=3D"noreferrer" target=3D"_blank">https://www.ub.uni-=
bielefeld.de/~befehl/</a><br>
<br>
BASE - Bielefeld Academic Search Engine - <a href=3D"http://www.base-search=
.net" rel=3D"noreferrer" target=3D"_blank">www.base-search.net</a><br>
*************************************************************<br>
<br>
_______________________________________________<br>
Xmldatadumps-l mailing list -- <a href=3D"mailto:[email protected]=
edia.org" target=3D"_blank">[email protected]</a><br>
To unsubscribe send an email to <a href=3D"mailto:xmldatadumps-l-leave@list=
s.wikimedia.org" target=3D"_blank">xmldatadumps-l-leave-RusutVdil2icGmH+5r0DM0B+6BGkLq7r@public.gmane.org=
</a><br>
</blockquote></div><div><br clear=3D"all"></div><div><br></div><span class=
=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_s=
ignature"><div dir=3D"ltr">Xabriel J. Collazo Mojica (he/him,=C2=A0<a href=
=3D"https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronun=
ciation.ogg" target=3D"_blank">pronunciation</a>)<div>Staff Data Engineer</=
div><div>Wikimedia Foundation</div></div></div>

--000000000000de493a06518bb3cb--

--===============0141597869823948653==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline