Re: Starting to use Bioperl

--===============8552225141048308374==
Content-Type: multipart/alternative; boundary="00000000000042b520056c1e2808"

--00000000000042b520056c1e2808
Content-Type: text/plain; charset="UTF-8"

On Sun, May 13, 2018 at 12:26 AM, Gordon Haverland <
[email protected]> wrote:

> On Fri, 11 May 2018 10:12:04 +0100
> Peter Cock <[email protected]> wrote:
>
> > This year the NCBI started offering this data in a slightly newer
> > format:
> >
> > https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/
> >
> > Most of these files are plain text tables using the rather
> > unusual field separator of "\t|\t" (tab, pipe, tab), but the
> > README files are very comprehensive.
>
> I found this, and got the tarball version.  I thought the README said
> it was \t|\n?  Doesn't matter, it's an unusual separator.
>

From memory, yes, the record separator is tab pipe newline,
but the field separator is tab pipe tab.


> There are Perl scripts in the tarball.  I think I read there, that if
> the NCBI dump files are older than 180 days, it downloads newer
> versions?  Or maybe I was reading something else.
>
> In any event, the BioSQL site at Github doesn't see much updating.  It
> looks to me like all the activity is in biopython, so I downloaded that
> for my Devuan machine.
>

As a mature database schema, we'd not expect much change.
The only substantial change in BioSQL in recent years was
extending the schema to work on SQLite.



> > This is in Python, but my most recent occasion to process
> > this data was to make a cut-down version of the NCBI
> > taxonomy as part of constructing a small test dataset:
> >
> > https://github.com/abaizan/kodoja/blob/master/test/
> taxonomy/filter_taxonomy.py
>
> I seen this at Google, you labelled something a bug.
>

Possibly you meant this recent work - something I had been
meaning to fix, but this conversation promoted me to do it:

https://github.com/abaizan/kodoja/pull/24



> In looking for the new_taxdump thing (via Google), another Perl script
> about findingSpeciesFromGenus (or something like that) popped up.  So,
> I have a few things of source to look through.
>
> Thanks.
>
> Gord
>
>
Yes, the NCBI taxonomy has existing in this format for over
a decade I think - there should be lots of scripts out there
for use/guidance.

Peter

--00000000000042b520056c1e2808
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On Sun, May 13, 2018 at 12:26 AM, Gordon Haverland <span dir=3D"ltr">&l=
t;<a href=3D"mailto:[email protected]" target=3D"_blank">ghaver=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204=
,204);padding-left:1ex"><span class=3D"gmail-">On Fri, 11 May 2018 10:12:04=
 +0100<br>
Peter Cock &lt;<a href=3D"mailto:[email protected]">p.j.a.cock@goog=
lemail.com</a>&gt; wrote:<br>
<br>
&gt; This year the NCBI started offering this data in a slightly newer<br>
&gt; format:<br>
&gt; <br>
&gt; <a href=3D"https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/" rel=
=3D"noreferrer" target=3D"_blank">https://ftp.ncbi.nlm.nih.gov/<wbr>pub/tax=
onomy/new_taxdump/</a><br>
&gt; <br>
&gt; Most of these files are plain text tables using the rather<br>
&gt; unusual field separator of &quot;\t|\t&quot; (tab, pipe, tab), but the=
<br>
&gt; README files are very comprehensive.<br>
<br>
</span>I found this, and got the tarball version.=C2=A0 I thought the READM=
E said<br>
it was \t|\n?=C2=A0 Doesn&#39;t matter, it&#39;s an unusual separator.<br><=
/blockquote><div><br></div><div>From memory, yes, the record separator is t=
ab pipe newline,</div><div>but the field separator is tab pipe tab.</div><d=
iv>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p=
x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
There are Perl scripts in the tarball.=C2=A0 I think I read there, that if<=
br>
the NCBI dump files are older than 180 days, it downloads newer<br>
versions?=C2=A0 Or maybe I was reading something else.<br>
<br>
In any event, the BioSQL site at Github doesn&#39;t see much updating.=C2=
=A0 It<br>
looks to me like all the activity is in biopython, so I downloaded that<br>
for my Devuan machine.<br></blockquote><div><br></div><div>As a mature data=
base schema, we&#39;d not expect much change.</div><div>The only substantia=
l change in BioSQL in recent years was</div><div>extending the schema to wo=
rk on SQLite.</div><div><br></div><div>=C2=A0</div><blockquote class=3D"gma=
il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2=
04,204);padding-left:1ex"><span class=3D"gmail-">
&gt; This is in Python, but my most recent occasion to process<br>
&gt; this data was to make a cut-down version of the NCBI<br>
&gt; taxonomy as part of constructing a small test dataset:<br>
&gt; <br>
&gt; <a href=3D"https://github.com/abaizan/kodoja/blob/master/test/taxonomy=
/filter_taxonomy.py" rel=3D"noreferrer" target=3D"_blank">https://github.co=
m/abaizan/<wbr>kodoja/blob/master/test/<wbr>taxonomy/filter_taxonomy.py</a>=
<br>
<br>
</span>I seen this at Google, you labelled something a bug.<br></blockquote=
><div><br></div><div>Possibly you meant this recent work - something I had =
been</div><div>meaning to fix, but this conversation promoted me to do it:<=
/div><div><br></div><div><a href=3D"https://github.com/abaizan/kodoja/pull/=
24">https://github.com/abaizan/kodoja/pull/24</a><br></div><div><br></div><=
div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
In looking for the new_taxdump thing (via Google), another Perl script<br>
about findingSpeciesFromGenus (or something like that) popped up.=C2=A0 So,=
<br>
I have a few things of source to look through.<br>
<br>
Thanks.<br>
<div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><br>
Gord</div><div class=3D"gmail-h5"><br></div></div></blockquote><div><br></d=
iv><div>Yes, the NCBI taxonomy has existing in this format for over</div><d=
iv>a decade I think - there should be lots of scripts out there</div><div>f=
or use/guidance.</div><div><br></div><div>Peter=C2=A0</div></div><br></div>=
</div>

--00000000000042b520056c1e2808--

--===============8552225141048308374==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Bioperl-l mailing list
[email protected]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
--===============8552225141048308374==--