[CAnet - news] Free storage in the cloud for scientific datasets
"Bill St. Arnaud" <[email protected]> Wed, 22 Apr 2009 17:49:55 -0400
| Newsgroups | gmane.culture.publications.news |
|---|---|
| Organization | CANARIE Inc |
| Message-ID | <[email protected]> |
This is a multipart message in MIME format.
--===============0329173182==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0170_01C9C372.BFE3DF80"
Content-Language: en-us
This is a multipart message in MIME format.
------=_NextPart_000_0170_01C9C372.BFE3DF80
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: 7bit
For more information on this item please visit my blog at
http://green-broadband.blogspot.com/ or http://billstarnaud.blogspot.com
-------------------------------------------
[A number of cloud organizations are offering free hosting for open
scientific datasets. If this trend continues to develop it will create a
further impetus for "open" data which is a result of government funded
research as well as the architecture of research networks and the need for
large physical scientific computational facilities. One of the challenges
facing researchers in the use of clouds or virtualization is the high cost
of storage and the time it takes to ship large datasets over the network.
Free storage and collocating large datasets with the computational cloud
eliminates both challenges. Here are some pointers to initiatives in this
area. Thanks to Richard Ackerman's blog for this pointer - BSA]
Google provides free storage for datasets
http://www.dailybits.com/google-provides-free-storage-for-scientific-data/
Under Project Palimpsest, Google will be providing free storage and public
access to large scientific data sets in what could be a major data
organization challenge.
The storage would fill a major need for scientists who want to openly share
their data, and would allow citizen scientists access to an unprecedented
amount of data to explore. For example, two planned datasets are all 120
terabytes of Hubble Space Telescope data and the images from the Archimedes
Palimpsest, the 10th century manuscript that inspired the Google dataset
storage project.
The challenge would be in the ways in which Google is able to represent the
data to the public. Also, the
<http://www.searchenginejournal.com/google-to-offer-terabytes-of-free-databa
se-storage-to-scientists/6273/> Trendanalyzer acquisition would come really
handy here. The data source is open to the public which means that additions
can be made to it as well.
There is also a presentation
<http://www.searchenginejournal.com/google-to-offer-terabytes-of-free-databa
se-storage-to-scientists/6273/> available at SearchEngineJournal on the
talk delivered last May. And if you have huge datasets that just won't get
uploaded, Google is providing 3TB disk arrays for shipping them whole file
system for the dataset.
-----
Talis Connected Commons
http://scilib.typepad.com/science_library_pad/
Talis <http://blogs.talis.com/n2/cc> Connected Commons is about fostering
the Linked Data <http://en.wikipedia.org/wiki/Linked_data> community, by
providing a rich hosting service:
For qualifying data sets, Talis will provide, through the Talis Platform
<http://www.talis.com/platform> :
* Free hosting of up to 50 million RDF
<http://www.w3.org/TR/REC-rdf-syntax/> triples and 10Gb of content
* Access to data access services
<http://n2.talis.com/wiki/Platform_API> that operate on that data,
including data retrieval and text search
* Free access to a public SPARQL
<http://www.w3.org/TR/rdf-sparql-query/> endpoint for each dataset.
I asked Leigh how this fits with the Talis Project Xiphos
<http://www.talis.com/xiphos/> initiative, and he explained that Xiphos is
a more focussed initiative around "data in the education, library and
publishing sectors", whereas Connected Commons is about any kind of data.
Talis, like Amazon, understands that a modern business is about fostering an
ecosystem, a combination of shared data and services that can be used as a
platform for software development and business development.
------
Amazon Public Datasets service
http://aws.amazon.com/publicdatasets/
Public Data Sets on AWS provides a centralized repository of public data
sets that can be seamlessly integrated into AWS cloud-based applications.
AWS is hosting the public data sets at no charge for the community, and like
all AWS services, users pay only for the compute and storage they use for
their own applications.
Previously, large data sets such as the mapping of the Human Genome and the
US Census data required hours or days to locate, download, customize, and
analyze. Now, anyone can access these data sets from their Amazon Elastic
Compute Cloud (Amazon EC2) instances and start computing on the data within
minutes. Users can also leverage the entire AWS ecosystem and easily
collaborate with other AWS users. For example, users can produce or use
prebuilt server images with tools and applications to analyze the data sets.
Users can also discuss best practices and solutions in the dedicated Public
<http://developer.amazonwebservices.com/connect/forum.jspa?forumID=55> Data
Sets forum.
By hosting this important and useful data with cost-efficient services such
as Amazon EC2, AWS hopes to provide researchers across a variety of
disciplines and industries with tools to enable more innovation, more
quickly.
-------------------------------------
To subscribe or unsubscribe send e-mail to [email protected]
These news items and comments are mine alone and do not necessarily reflect
those of the CANARIE board or management.
------
[email protected]
[email protected]
http://billstarnaud.blogspot.com/
------=_NextPart_000_0170_01C9C372.BFE3DF80
Content-Type: text/html;
charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:x=3D"urn:schemas-microsoft-com:office:excel" =
xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" =
xmlns:a=3D"urn:schemas-microsoft-com:office:access" =
xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" =
xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" =
xmlns:rs=3D"urn:schemas-microsoft-com:rowset" xmlns:z=3D"#RowsetSchema" =
xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" =
xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" =
xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" =
xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" =
xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" =
xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:D=3D"DAV:" =
xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/" =
xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml" =
xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" =
xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/" =
xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#" =
xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp" =
xmlns:udc=3D"http://schemas.microsoft.com/data/udc" =
xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema" =
xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/"=
xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#" =
xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" =
xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/" =
xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance" =
xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap" =
xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" =
xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart" =
xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/" =
xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup" =
xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig" =
xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-sig=
nature" =
xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/2006=
" xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationshi=
ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages" =
xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types"=
=
xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messag=
es" =
xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/=
" =
xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pub=
lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" =
xmlns:st=3D"" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.caps
{mso-style-name:caps;}
span.padlt25
{mso-style-name:padlt25;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1335495523;
mso-list-template-ids:249324470;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3DEN-US link=3Dblue vlink=3Dpurple>
<div class=3DSection1>
<p class=3DMsoPlainText>For more information on this item please visit =
my blog at
<o:p></o:p></p>
<p class=3DMsoPlainText><a =
href=3D"http://green-broadband.blogspot.com/">http://green-broadband.blog=
spot.com/</a>
or <a =
href=3D"http://billstarnaud.blogspot.com">http://billstarnaud.blogspot.co=
m</a><o:p></o:p></p>
<p =
class=3DMsoPlainText>-------------------------------------------<o:p></o:=
p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'> [A
number of cloud organizations are offering free hosting for open =
scientific
datasets. If this trend continues to develop it will create a further =
impetus
for “open” data which is a result of government funded =
research as
well as the architecture of research networks and the need for large =
physical scientific
computational facilities. One of the challenges facing researchers in =
the use
of clouds or virtualization is the high cost of storage and the =
time it takes
to ship large datasets over the network. Free storage and =
collocating
large datasets with the computational cloud eliminates both =
challenges.
Here are some pointers to initiatives in this area. Thanks to =
Richard
Ackerman’s blog for this pointer – =
BSA]<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>Google
provides free storage for datasets<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><a
href=3D"http://www.dailybits.com/google-provides-free-storage-for-scienti=
fic-data/">http://www.dailybits.com/google-provides-free-storage-for-scie=
ntific-data/</a><o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>Under Project Palimpsest, Google =
will be
providing free storage and public access to large scientific data sets =
in what
could be a major data organization challenge.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>The storage would fill a major need =
for
scientists who want to openly share their data, and would allow citizen
scientists access to an unprecedented amount of data to explore. For =
example,
two planned datasets are all 120 terabytes of Hubble Space Telescope =
data and
the images from the Archimedes Palimpsest, the 10th century manuscript =
that
inspired the Google dataset storage project.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>The challenge would be in the ways =
in which
Google is able to represent the data to the public. Also, <a
href=3D"http://www.searchenginejournal.com/google-to-offer-terabytes-of-f=
ree-database-storage-to-scientists/6273/"
title=3D"Google to Offer Terabytes of Free Database Storage to =
Scientists">the
Trendanalyzer acquisition would come really handy here</a>. The data =
source is
open to the public which means that additions can be made to it as =
well.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>There is also a <a
href=3D"http://www.searchenginejournal.com/google-to-offer-terabytes-of-f=
ree-database-storage-to-scientists/6273/"
title=3D"Google to Offer Terabytes of Free Database Storage to =
Scientists">presentation
available at SearchEngineJournal</a> on the talk delivered last May. And =
if you
have huge datasets that just won’t get uploaded, Google is =
providing 3TB
disk arrays for shipping them whole file system for the =
dataset.<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>-----<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>Talis
Connected Commons<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>http://scilib.typepad.com/science_library_pad/<o:p></o:p>=
</span></p>
<p><span style=3D'font-size:11.0pt'><a =
href=3D"http://blogs.talis.com/n2/cc">Talis
Connected Commons</a> is about fostering the <a
href=3D"http://en.wikipedia.org/wiki/Linked_data">Linked Data</a> =
community, by
providing a rich hosting service: <o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>For qualifying data sets, Talis will =
provide,
through the <a href=3D"http://www.talis.com/platform">Talis =
Platform</a>:<o:p></o:p></span></p>
<ul type=3Ddisc>
<li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo1'><span style=3D'font-family:"Times New =
Roman","serif"'>Free
hosting of up to 50 million <a =
href=3D"http://www.w3.org/TR/REC-rdf-syntax/">RDF</a>
triples and 10Gb of content<o:p></o:p></span></li>
<li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo1'><span style=3D'font-family:"Times New =
Roman","serif"'>Access
to <a href=3D"http://n2.talis.com/wiki/Platform_API">data access =
services</a>
that operate on that data, including data retrieval and text =
search<o:p></o:p></span></li>
<li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo1'><span style=3D'font-family:"Times New =
Roman","serif"'>Free
access to a public <a =
href=3D"http://www.w3.org/TR/rdf-sparql-query/">SPARQL</a>
endpoint for each dataset.<o:p></o:p></span></li>
</ul>
<p><span style=3D'font-size:11.0pt'>I asked Leigh how this fits with the =
Talis <a
href=3D"http://www.talis.com/xiphos/">Project Xiphos</a> initiative, and =
he
explained that Xiphos is a more focussed initiative around "data in =
the
education, library and publishing sectors", whereas Connected =
Commons is
about any kind of data.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>Talis, like Amazon, understands that =
a modern
business is about fostering an ecosystem, a combination of shared data =
and
services that can be used as a platform for software development and =
business
development.<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>------<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>Amazon
Public Datasets service<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><a
href=3D"http://aws.amazon.com/publicdatasets/">http://aws.amazon.com/publ=
icdatasets/</a><o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>Public Data Sets on <span =
class=3Dcaps>AWS</span>
provides a centralized repository of public data sets that can be =
seamlessly
integrated into <span class=3Dcaps>AWS</span> cloud-based applications. =
<span
class=3Dcaps>AWS</span> is hosting the public data sets at no charge for =
the
community, and like all <span class=3Dcaps>AWS</span> services, users =
pay only
for the compute and storage they use for their own =
applications.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>Previously, large data sets such as =
the
mapping of the Human Genome and the US Census data required hours or =
days to
locate, download, customize, and analyze. Now, anyone can access these =
data
sets from their Amazon Elastic Compute Cloud (Amazon <span =
class=3Dcaps>EC2</span>)
instances and start computing on the data within minutes. Users can also
leverage the entire <span class=3Dcaps>AWS</span> ecosystem and easily
collaborate with other <span class=3Dcaps>AWS</span> users. For example, =
users
can produce or use prebuilt server images with tools and applications to
analyze the data sets. Users can also discuss best practices and =
solutions in
the dedicated <a
href=3D"http://developer.amazonwebservices.com/connect/forum.jspa?forumID=
=3D55">Public
Data Sets forum</a>.<o:p></o:p></span></p>
<p><span style=3D'font-size:11.0pt'>By hosting this important and useful =
data
with cost-efficient services such as Amazon <span =
class=3Dcaps>EC2</span>, AWS
hopes to provide researchers across a variety of disciplines and =
industries
with tools to enable more innovation, more =
quickly.<o:p></o:p></span></p>
<p class=3DMsoPlainText> =
<o:p></o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p =
class=3DMsoPlainText>-------------------------------------<o:p></o:p></p>=
<p class=3DMsoPlainText>To subscribe or unsubscribe send e-mail to <a
href=3D"mailto:[email protected]">[email protected]</a><o=
:p></o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoPlainText>These news items and comments are mine alone and =
do not
necessarily reflect those of the CANARIE board or =
management.<o:p></o:p></p>
<p class=3DMsoPlainText><o:p> </o:p></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>------<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>[email protected]<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>[email protected]<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'>http://billstarnaud.blogspot.com/<o:p></o:p></span></p>
<p class=3DMsoNormal><span style=3D'font-family:"Times New =
Roman","serif"'><o:p> </o:p></span></p>
</div>
</body>
</html>
------=_NextPart_000_0170_01C9C372.BFE3DF80--
--===============0329173182==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
news mailing list
[email protected]
http://lists.canarie.ca/mailman/listinfo/news
--===============0329173182==--