Re: Starting to use Bioperl

--===============2520326717509473789==
Content-Type: multipart/alternative; boundary="000000000000ab048f056bea87e9"

--000000000000ab048f056bea87e9
Content-Type: text/plain; charset="UTF-8"

Hi Gordon,

A couple of bits of background reading for you.

First, there is a database schema called BioSQL which might be
of interest in that it includes taxon tables - based primarily on the
NCBI taxonomy tree but it could be used for another taxonomy.
There is an SQLite version of this (in use by Biopython) but that
has not as far as a I know been integrated into BioPerl yet.

http://biosql.org
https://github.com/biosql/biosql

I think given your taxonomy focus, you can ignore BioSQL which
is more suited to working with NCBI/EMBL annotated sequences.

Now, I mentioned the NCBI taxonomy, which is a de facto world
standard but will not always reflect the latest expert opinion in
all branches of life. Nevertheless, I would start there.

You can query the NCBI taxonomy via Entrez (and by hand on
the website), see how to walk the tree, ignore the boring ranks,
until you reach the root of the tree.

Or, you can download the NCBI taxonomy as a set of text files,
for which you should have no trouble finding examples scripts
to load and work with:

https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/

This year the NCBI started offering this data in a slightly newer
format:

https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/

Most of these files are plain text tables using the rather
unusual field separator of "\t|\t" (tab, pipe, tab), but the
README files are very comprehensive.

This is in Python, but my most recent occasion to process
this data was to make a cut-down version of the NCBI
taxonomy as part of constructing a small test dataset:

https://github.com/abaizan/kodoja/blob/master/test/taxonomy/filter_taxonomy.py

Peter


On Fri, May 11, 2018 at 1:50 AM, Gordon Haverland <
[email protected]> wrote:

> On Wed, 9 May 2018 09:54:18 -0700
> Gordon Haverland <[email protected]> wrote:
>
> >           ... I am researching a deer problem.
>
> There are BioPerl and Bio-LITE routines which can work with taxonomy
> information.  Finding something which can write a SQLite3 dbase took a
> little digging, but something does exist.
>
> I've never played with BioPerl before, and I am still trying to clean
> and expand my deer plant data, so I ran my latest effort with a call to
> BioPerl to look up a taxonid and then a taxon.  It just happened the
> first element in my list was a hybrid species (Abelia x grandiflora).
> Anyway, following some BioPerl documentation I connected to -entrez
> (excuse any spelling mistakes) and it came up with a hit.  A species
> hit, which is what I was hoping for.
>
> From that returned object, I can get an ancestor object (which is a
> genus), and from that I can get an ancestor object which is a family,
> and from that I can get an ancestor object which is an order and then
> further iterations on ancestor get non_ranked clade stuff which I am
> not sure how to handle.  I haven't tried iterating to the limit, I was
> hoping that at some point an attempt to return an ancestor would return
> under.  But I really don't know what to do with this non_rank clade
> stuff.
>
> I suspect, I need to iterate this ancestor stuff until I get to kingdom
> plantae?  This gives me a "root".  I now have a species (usually) with
> N ancestors up to a common root (kingdom plantae).  That constitutes a
> tree as I understand things, but it is all one sided.
>
> If I go to the next entry in my deer resistant plants data, I may have
> M ancestors up to kingdom plantae.   And do this for 1000 or so other
> entries.
>
> For each set of ancestor lookups, I need to make a tree.
>
> All of these trees have the same root (kingdom plantae).  So I should
> be able to add all these trees together.  And then I think I found the
> utilities to save this mess as SQLite.
>
> As I understand things, I probably want to be working with NCBI ID
> numbers on the species entered?  And what you call annotation, I would
> save in one or more separate SQLite3 dbases keyed on the NCBI ID number?
>
> Let's assume one of the fields of annotation is the USDA growing zone.
> A person thinks they want to do a query on USDA Zone 3, so the program
> changes this to a query for USDA Zones 2-4, which picks off all the
> NCBI ID numbers, and then a person can use BioPerl to make a picture of
> all the deer resistant taxonomy known.
>
> One of the sources of data into this, has colour of the flowers.  So
> someone could conceivably be looking for pink flowered, deer resistant
> plants.  That's why I suggested there might be more than 1 SQLite dbase
> of annotation to go with this stuff.
>
> I'll stop writing, and go back to reading code.  I downloaded the
> Bio-LITE modules (not at Debian/Devuan), and I think there were
> suggestions of other code to download.  And read.
>
> Have a great day!
> Gord
>
>
> _______________________________________________
> Bioperl-l mailing list
> [email protected]
> http://mailman.open-bio.org/mailman/listinfo/bioperl-l
>

--000000000000ab048f056bea87e9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Gordon,<div><br></div><div>A couple of bits of backgrou=
nd reading for you.</div><div><br></div><div>First, there is a database sch=
ema called BioSQL which might be</div><div>of interest in that it includes =
taxon tables - based primarily on the</div><div>NCBI taxonomy tree but it c=
ould be used for another taxonomy.</div><div>There is an SQLite version of =
this (in use by Biopython) but that</div><div>has not as far as a I know be=
en integrated into BioPerl yet.</div><div><br></div><div><a href=3D"http://=
biosql.org">http://biosql.org</a></div><div><a href=3D"https://github.com/b=
iosql/biosql">https://github.com/biosql/biosql</a><br></div><div><br></div>=
<div>I think given your taxonomy focus, you can ignore BioSQL which</div><d=
iv>is more suited to working with NCBI/EMBL annotated sequences.</div><div>=
<br></div><div>Now, I mentioned the NCBI taxonomy, which is a de facto worl=
d</div><div>standard but will not always reflect the latest expert opinion =
in</div><div>all branches of life. Nevertheless, I would start there.</div>=
<div><br></div><div>You can query the NCBI taxonomy via Entrez (and by hand=
 on</div><div>the website), see how to walk the tree, ignore the boring ran=
ks,</div><div>until you reach the root of the tree.</div><div><br></div><di=
v>Or, you can download the NCBI taxonomy as a set of text files,</div><div>=
for which you should have no trouble finding examples scripts</div><div>to =
load and work with:</div><div><br></div><div><a href=3D"https://ftp.ncbi.nl=
m.nih.gov/pub/taxonomy/">https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/</a><br>=
</div><div><br></div><div>This year the NCBI started offering this data in =
a slightly newer</div><div>format:</div><div><br></div><div><a href=3D"http=
s://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/">https://ftp.ncbi.nlm.ni=
h.gov/pub/taxonomy/new_taxdump/</a><br></div><div><br></div><div>Most of th=
ese files are plain text tables using the rather</div><div>unusual field se=
parator of &quot;\t|\t&quot; (tab, pipe, tab), but the</div><div>README fil=
es are very comprehensive.</div><div><br></div><div>This is in Python, but =
my most recent occasion to process</div><div>this data was to make a cut-do=
wn version of the NCBI</div><div>taxonomy as part of constructing a small t=
est dataset:</div><div><br></div><div><a href=3D"https://github.com/abaizan=
/kodoja/blob/master/test/taxonomy/filter_taxonomy.py">https://github.com/ab=
aizan/kodoja/blob/master/test/taxonomy/filter_taxonomy.py</a><br></div><div=
><br></div><div>Peter</div><div><br></div></div><div class=3D"gmail_extra">=
<br><div class=3D"gmail_quote">On Fri, May 11, 2018 at 1:50 AM, Gordon Have=
rland <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a>&gt;</span> wrote:<br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><span class=3D"">On Wed, 9 May 2018 09:54:18=
 -0700<br>
Gordon Haverland &lt;<a href=3D"mailto:[email protected]">ghave=
[email protected]</a><wbr>&gt; wrote:<br>
<br>
</span>&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... I am researching a =
deer problem.<br>
<br>
There are BioPerl and Bio-LITE routines which can work with taxonomy<br>
information.=C2=A0 Finding something which can write a SQLite3 dbase took a=
<br>
little digging, but something does exist.<br>
<br>
I&#39;ve never played with BioPerl before, and I am still trying to clean<b=
r>
and expand my deer plant data, so I ran my latest effort with a call to<br>
BioPerl to look up a taxonid and then a taxon.=C2=A0 It just happened the<b=
r>
first element in my list was a hybrid species (Abelia x grandiflora).<br>
Anyway, following some BioPerl documentation I connected to -entrez<br>
(excuse any spelling mistakes) and it came up with a hit.=C2=A0 A species<b=
r>
hit, which is what I was hoping for.<br>
<br>
From that returned object, I can get an ancestor object (which is a<br>
genus), and from that I can get an ancestor object which is a family,<br>
and from that I can get an ancestor object which is an order and then<br>
further iterations on ancestor get non_ranked clade stuff which I am<br>
not sure how to handle.=C2=A0 I haven&#39;t tried iterating to the limit, I=
 was<br>
hoping that at some point an attempt to return an ancestor would return<br>
under.=C2=A0 But I really don&#39;t know what to do with this non_rank clad=
e<br>
stuff.<br>
<br>
I suspect, I need to iterate this ancestor stuff until I get to kingdom<br>
plantae?=C2=A0 This gives me a &quot;root&quot;.=C2=A0 I now have a species=
 (usually) with<br>
N ancestors up to a common root (kingdom plantae).=C2=A0 That constitutes a=
<br>
tree as I understand things, but it is all one sided.<br>
<br>
If I go to the next entry in my deer resistant plants data, I may have<br>
M ancestors up to kingdom plantae.=C2=A0 =C2=A0And do this for 1000 or so o=
ther<br>
entries.<br>
<br>
For each set of ancestor lookups, I need to make a tree.<br>
<br>
All of these trees have the same root (kingdom plantae).=C2=A0 So I should<=
br>
be able to add all these trees together.=C2=A0 And then I think I found the=
<br>
utilities to save this mess as SQLite.<br>
<br>
As I understand things, I probably want to be working with NCBI ID<br>
numbers on the species entered?=C2=A0 And what you call annotation, I would=
<br>
save in one or more separate SQLite3 dbases keyed on the NCBI ID number?<br=
>
<br>
Let&#39;s assume one of the fields of annotation is the USDA growing zone.<=
br>
A person thinks they want to do a query on USDA Zone 3, so the program<br>
changes this to a query for USDA Zones 2-4, which picks off all the<br>
NCBI ID numbers, and then a person can use BioPerl to make a picture of<br>
all the deer resistant taxonomy known.<br>
<br>
One of the sources of data into this, has colour of the flowers.=C2=A0 So<b=
r>
someone could conceivably be looking for pink flowered, deer resistant<br>
plants.=C2=A0 That&#39;s why I suggested there might be more than 1 SQLite =
dbase<br>
of annotation to go with this stuff.<br>
<br>
I&#39;ll stop writing, and go back to reading code.=C2=A0 I downloaded the<=
br>
Bio-LITE modules (not at Debian/Devuan), and I think there were<br>
suggestions of other code to download.=C2=A0 And read.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
Have a great day!<br>
Gord<br>
<br>
<br>
______________________________<wbr>_________________<br>
Bioperl-l mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]=
o.org</a><br>
<a href=3D"http://mailman.open-bio.org/mailman/listinfo/bioperl-l" rel=3D"n=
oreferrer" target=3D"_blank">http://mailman.open-bio.org/<wbr>mailman/listi=
nfo/bioperl-l</a><br>
</div></div></blockquote></div><br></div>

--000000000000ab048f056bea87e9--

--===============2520326717509473789==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Bioperl-l mailing list
[email protected]
http://mailman.open-bio.org/mailman/listinfo/bioperl-l
--===============2520326717509473789==--