Re: [Xmldatadumps-l] Availability of Wikidata JSON dumps after Feb, 2019

--===============5231606385447024076==
Content-Type: multipart/alternative; boundary="0000000000001eaa0d05b4f03f1a"

--0000000000001eaa0d05b4f03f1a
Content-Type: text/plain; charset="UTF-8"

Gerhard,

I'm curious what you mean by "processing" and "comb through".
Can you describe how your processing and what system or database the output
gets loaded into?
Perhaps you have your scripts publicly available on something like GitHub?

It would be nice to know a bit more on what you also are doing.  Thanks in
advance!

Thad
https://www.linkedin.com/in/thadguidry/


On Wed, Nov 25, 2020 at 9:14 AM Gerhard Gonter <[email protected]> wrote:

> On Wed, Nov 25, 2020 at 1:22 PM Daniel Garijo <[email protected]> wrote:
> >
> > Hello,
> >
> > I am writing this message because I am analyzing the Wikidata JSON dumps
> > available in the Internet Archive and I have found there are no dumps
> > available after Feb 8th, 2019 (see
> >
> https://archive.org/details/wikimediadownloads?and%5B%5D=%22Wikidata%20entity%20dumps%22
> ).
> > I know the latest dumps are available at
> > https://dumps.wikimedia.org/wikidatawiki/entities/, but unfortunately
> > they only cover the last few months.
>
> Which dump files are exactly looking for?  Dumps like
>
>
> https://dumps.wikimedia.org/wikidatawiki/entities/20201116/wikidata-20201116-all.json.gz
>
> which can also be found on https://dumps.wikimedia.org/other/wikidata/
> as 20201116.json.gz ?
>
> > [...]
> > Does anyone on this list know where some of these missing Wikidata dumps
> > may be found? If anyone has pointers to a server where they can be
> > downloaded, I would highly appreciate it.
>
> If you are looking for these dumps, I have about 8 TB stored on
> external disks.  Transferring these over the network might be
> difficult, however.  Please contact me off-list, if this you need any
> of these dumps, maybe we can arrange something.
>
> I'm curious, what are you trying to do with all of these files?
> Processing all of them must take months.  My processor usually picks
> up the dump on Wednesday and takes 80 hours to comb through it.  But
> my processor is written in Perl, something in C or Rust might be a lot
> faster...
>
> regards, Gerhard Gonter
>
> _______________________________________________
> Wikidata mailing list
> [email protected]
> https://lists.wikimedia.org/mailman/listinfo/wikidata
>

--0000000000001eaa0d05b4f03f1a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:trebuche=
t ms,sans-serif;font-size:small">Gerhard,</div><div class=3D"gmail_default"=
 style=3D"font-family:trebuchet ms,sans-serif;font-size:small"><br></div><d=
iv class=3D"gmail_default" style=3D"font-family:trebuchet ms,sans-serif;fon=
t-size:small">I&#39;m curious what you mean by &quot;processing&quot; and &=
quot;comb through&quot;.</div><div class=3D"gmail_default" style=3D"font-fa=
mily:trebuchet ms,sans-serif;font-size:small">Can you describe how your pro=
cessing and what system or database the output gets loaded into?<br></div><=
div class=3D"gmail_default" style=3D"font-family:trebuchet ms,sans-serif;fo=
nt-size:small">Perhaps you have your scripts publicly available on somethin=
g like GitHub?</div><div class=3D"gmail_default" style=3D"font-family:trebu=
chet ms,sans-serif;font-size:small"><br></div><div class=3D"gmail_default" =
style=3D"font-family:trebuchet ms,sans-serif;font-size:small">It would be n=
ice to know a bit more on what you also are doing.=C2=A0 Thanks in advance!=
<br><br clear=3D"all"></div><div><div dir=3D"ltr" class=3D"gmail_signature"=
 data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr">=
<div><div dir=3D"ltr">Thad<div><a href=3D"https://www.linkedin.com/in/thadg=
uidry/" target=3D"_blank">https://www.linkedin.com/in/thadguidry/</a><br></=
div></div></div></div></div></div></div></div><br></div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Nov 25, 2020 at 9=
:14 AM Gerhard Gonter &lt;<a href=3D"mailto:[email protected]">ggonter@gmai=
l.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:=
1ex">On Wed, Nov 25, 2020 at 1:22 PM Daniel Garijo &lt;<a href=3D"mailto:dg=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br>
&gt;<br>
&gt; Hello,<br>
&gt;<br>
&gt; I am writing this message because I am analyzing the Wikidata JSON dum=
ps<br>
&gt; available in the Internet Archive and I have found there are no dumps<=
br>
&gt; available after Feb 8th, 2019 (see<br>
&gt; <a href=3D"https://archive.org/details/wikimediadownloads?and%5B%5D=3D=
%22Wikidata%20entity%20dumps%22" rel=3D"noreferrer" target=3D"_blank">https=
://archive.org/details/wikimediadownloads?and%5B%5D=3D%22Wikidata%20entity%=
20dumps%22</a>).<br>
&gt; I know the latest dumps are available at<br>
&gt; <a href=3D"https://dumps.wikimedia.org/wikidatawiki/entities/" rel=3D"=
noreferrer" target=3D"_blank">https://dumps.wikimedia.org/wikidatawiki/enti=
ties/</a>, but unfortunately<br>
&gt; they only cover the last few months.<br>
<br>
Which dump files are exactly looking for?=C2=A0 Dumps like<br>
<br>
<a href=3D"https://dumps.wikimedia.org/wikidatawiki/entities/20201116/wikid=
ata-20201116-all.json.gz" rel=3D"noreferrer" target=3D"_blank">https://dump=
s.wikimedia.org/wikidatawiki/entities/20201116/wikidata-20201116-all.json.g=
z</a><br>
<br>
which can also be found on <a href=3D"https://dumps.wikimedia.org/other/wik=
idata/" rel=3D"noreferrer" target=3D"_blank">https://dumps.wikimedia.org/ot=
her/wikidata/</a><br>
as 20201116.json.gz ?<br>
<br>
&gt; [...]<br>
&gt; Does anyone on this list know where some of these missing Wikidata dum=
ps<br>
&gt; may be found? If anyone has pointers to a server where they can be<br>
&gt; downloaded, I would highly appreciate it.<br>
<br>
If you are looking for these dumps, I have about 8 TB stored on<br>
external disks.=C2=A0 Transferring these over the network might be<br>
difficult, however.=C2=A0 Please contact me off-list, if this you need any<=
br>
of these dumps, maybe we can arrange something.<br>
<br>
I&#39;m curious, what are you trying to do with all of these files?<br>
Processing all of them must take months.=C2=A0 My processor usually picks<b=
r>
up the dump on Wednesday and takes 80 hours to comb through it.=C2=A0 But<b=
r>
my processor is written in Perl, something in C or Rust might be a lot<br>
faster...<br>
<br>
regards, Gerhard Gonter<br>
<br>
_______________________________________________<br>
Wikidata mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Wikidata@=
lists.wikimedia.org</a><br>
<a href=3D"https://lists.wikimedia.org/mailman/listinfo/wikidata" rel=3D"no=
referrer" target=3D"_blank">https://lists.wikimedia.org/mailman/listinfo/wi=
kidata</a><br>
</blockquote></div>

--0000000000001eaa0d05b4f03f1a--


--===============5231606385447024076==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KV2lraWRhdGEg
bWFpbGluZyBsaXN0Cldpa2lkYXRhQGxpc3RzLndpa2ltZWRpYS5vcmcKaHR0cHM6Ly9saXN0cy53
aWtpbWVkaWEub3JnL21haWxtYW4vbGlzdGluZm8vd2lraWRhdGEK

--===============5231606385447024076==--