Re: When we cannot waste 1MB per datum.
Omer Zak <w1-W6cp89mEpD1mR6Xm/[email protected]> Mon, 25 Jun 2012 22:21:34 +0300
| Newsgroups | gmane.culture.hackers.israel |
|---|---|
| Message-ID | <1340652094.4818.31.camel@c4> |
--avL5AuxvnWyTJNL5nzSwQPecaCy-JhvEQeewTmO
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Nice to remind the world that I existed also at 2002. :-)
Obviously, representing a single FC position is not something that
should, need to or even could consume a full 1MB RAM.
What I had in mind was AI stuff.
--- Omer
On Mon, 2012-06-25 at 17:14 +0300, Shlomi Fish wrote:
> Hi all,
>=20
> Omer Zak (CCed to this message) wrote this old (2002) Hackers-IL message:
>=20
> http://tech.groups.yahoo.com/group/hackers-il/message/1894
>=20
> Titled "How to exploit 1MB data structures?" this message reads:
>=20
> <<<<
> The future computers will have terabytes of memory (mostly in disks, but
> some in RAM). They will be able to manipulate megabytes at the time
> needed by yesterday's computers to manipulate single bytes.
>=20
> I wonder how can all those extra bytes be harnassed. (AI anyone?)
>=20
> Let's try to use 1MB to represent the string "The beep has booped".
> The data structure will contain:
> >>>>
>=20
> Now I'd like to give a view from the other side of the fence: when have
> milliards (10^9s) of data items and we need to fit them all in RAM (first=
-
> 128 GB - now I also got access to a machine with 512 GB), because storing=
and
> retrieving them on disk is far too slow. This is the case of Freecell Sol=
ver's
> dbm_fc_solver where I've spent a lot of time reducing its RAM consumption=
:
>=20
> * http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1114
>=20
> And there's another optimisation coming up:
>=20
> * http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1115
>=20
> If every position in the game=E2=80=99s graph consumed 1MB of RAM, then I=
could fit
> at most 512*1024 of them in 512 GB of RAM, which is all I have at the mom=
ent.
> And I can reach much more than 512K of positions on my underpowered home
> computers. Furthermore, manipulating these 1MB data will likely be much m=
ore
> time-consuming than the compact ones I'm using at the moment. Some of my =
scans
> reached 3 milliard stored positions, and such an amount would have consum=
ed
> roughly 3 peta bytes (3 * 10^15), which I suspect are beyond any single H=
PC
> machine at the moment (to say nothing of consumer computers).
>=20
> I think that if we have large amounts of RAM resources at our disposal, w=
e
> should not carelessly waste them like Omer suggests (unless of course we =
ran
> into a use-case that necessitates that).
>=20
> Regards,
>=20
> Shlomi Fish=20
--=20
PHP - the language of the Vogons.
My own blog is at http://www.zak.co.il/tddpirate/
My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
--avL5AuxvnWyTJNL5nzSwQPecaCy-JhvEQeewTmO
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm=
l4/strict.dtd">
<html>
<head>
</head>
<body style=3D"background-color: #fff;">
<span style=3D"display:none"> </span>
<!--~-|**|PrettyHtmlStartT|**|-~-->
<div id=3D"ygrp-mlmsg" style=3D"position:relative;">
<div id=3D"ygrp-msg" style=3D"z-index: 1;">
<!--~-|**|PrettyHtmlEndT|**|-~-->
<div id=3D"ygrp-text" >
=20=20=20=20=20=20
=20=20=20=20=20=20
<p>Nice to remind the world that I existed also at 2002. :-)<br>
Obviously, representing a single FC position is not something that<br>
should, need to or even could consume a full 1MB RAM.<br>
<br>
What I had in mind was AI stuff.<br>
<br>
--- Omer<br>
<br>
On Mon, 2012-06-25 at 17:14 +0300, Shlomi Fish wrote:<br>
> Hi all,<br>
> <br>
> Omer Zak (CCed to this message) wrote this old (2002) Hackers-IL messa=
ge:<br>
> <br>
> <a href=3D"http://tech.groups.yahoo.com/group/hackers-il/message/1894"=
>http://tech.groups.yahoo.com/group/hackers-il/message/1894</a><br>
> <br>
> Titled "How to exploit 1MB data structures?" this message re=
ads:<br>
> <br>
> <<<<<br>
> The future computers will have terabytes of memory (mostly in disks, b=
ut<br>
> some in RAM). They will be able to manipulate megabytes at the time<br=
>
> needed by yesterday's computers to manipulate single bytes.<br>
> <br>
> I wonder how can all those extra bytes be harnassed. (AI anyone?)<br>
> <br>
> Let's try to use 1MB to represent the string "The beep has booped=
".<br>
> The data structure will contain:<br>
> >>>><br>
> <br>
> Now I'd like to give a view from the other side of the fence: when hav=
e<br>
> milliards (10^9s) of data items and we need to fit them all in RAM (fi=
rst -<br>
> 128 GB - now I also got access to a machine with 512 GB), because stor=
ing and<br>
> retrieving them on disk is far too slow. This is the case of Freecell =
Solver's<br>
> dbm_fc_solver where I've spent a lot of time reducing its RAM consumpt=
ion:<br>
> <br>
> * <a href=3D"http://tech.groups.yahoo.com/group/fc-solve-discuss/messa=
ge/1114">http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1114</=
a><br>
> <br>
> And there's another optimisation coming up:<br>
> <br>
> * <a href=3D"http://tech.groups.yahoo.com/group/fc-solve-discuss/messa=
ge/1115">http://tech.groups.yahoo.com/group/fc-solve-discuss/message/1115</=
a><br>
> <br>
> If every position in the game=E2=80=99s graph consumed 1MB of RAM, the=
n I could fit<br>
> at most 512*1024 of them in 512 GB of RAM, which is all I have at the =
moment.<br>
> And I can reach much more than 512K of positions on my underpowered ho=
me<br>
> computers. Furthermore, manipulating these 1MB data will likely be muc=
h more<br>
> time-consuming than the compact ones I'm using at the moment. Some of =
my scans<br>
> reached 3 milliard stored positions, and such an amount would have con=
sumed<br>
> roughly 3 peta bytes (3 * 10^15), which I suspect are beyond any singl=
e HPC<br>
> machine at the moment (to say nothing of consumer computers).<br>
> <br>
> I think that if we have large amounts of RAM resources at our disposal=
, we<br>
> should not carelessly waste them like Omer suggests (unless of course =
we ran<br>
> into a use-case that necessitates that).<br>
> <br>
> Regards,<br>
> <br>
> Shlomi Fish <br>
<br>
-- <br>
PHP - the language of the Vogons.<br>
My own blog is at <a href=3D"http://www.zak.co.il/tddpirate/">http://www.za=
k.co.il/tddpirate/</a><br>
<br>
My opinions, as expressed in this E-mail message, are mine alone.<br>
They do not represent the official policy of any organization with which<br=
>
I may be affiliated in any way.<br>
WARNING TO SPAMMERS: at <a href=3D"http://www.zak.co.il/spamwarning.html">=
http://www.zak.co.il/spamwarning.html</a><br>
<br>
</p>
</div>
=20=20=20=20=20
<!--~-|**|PrettyHtmlStart|**|-~-->
<div style=3D"color: #fff; height: 0;">__._,_.___</div>
=20=20=20=20=20=20=20=20
=20=20
=20=20=20
<div id=3D"ygrp-actbar" style=3D"clear: both; margin-bottom: 10px; whit=
e-space: nowrap; color: #666; padding-top: 15px;">
<div>
<a href=3D"mailto:w1-W6cp89mEpD1mR6Xm/[email protected]?subject=3DRe%3A%20When%20we%20cannot=
%20waste%201MB%20per%20datum%2E" style=3D"margin-right: 0; padding-right: 0=
;">
Reply to <span style=3D"font-weight: 700;">sender</span></a> |
<a href=3D"mailto:[email protected]?subject=3DRe%3A%20When=
%20we%20cannot%20waste%201MB%20per%20datum%2E">
Reply to <span style=3D"font-weight: 700;">group</span></a> |
<a href=3D"http://groups.yahoo.com/group/hackers-il/post;_ylc=3D=
X3oDMTJwaGt0YXN2BF9TAzk3MzU5NzE0BGdycElkAzE4NTczMzIEZ3Jwc3BJZAMxNzA1MDA2NzY=
0BG1zZ0lkAzUxODQEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxMzQwNjUyMTA3?act=3Dreply&m=
essageNum=3D5184">Reply <span style=3D"font-weight: 700;">via web post</spa=
n></a> |
<a href=3D"http://groups.yahoo.com/group/hackers-il/post;_ylc=
=3DX3oDMTJlcThybThkBF9TAzk3MzU5NzE0BGdycElkAzE4NTczMzIEZ3Jwc3BJZAMxNzA1MDA2=
NzY0BHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTM0MDY1MjEwNw--" style=3D"font-weight:=
700;">Start a New Topic</a>
</div>
<a href=3D"http://groups.yahoo.com/group/hackers-il/message=
/5184;_ylc=3DX3oDMTM0cGs1Y3Z1BF9TAzk3MzU5NzE0BGdycElkAzE4NTczMzIEZ3Jwc3BJZA=
MxNzA1MDA2NzY0BG1zZ0lkAzUxODQEc2VjA2Z0cgRzbGsDdnRwYwRzdGltZQMxMzQwNjUyMTA3B=
HRwY0lkAzUxODQ-">Messages in this topic</a>
(<span style=3D"font-weight: 700;">1</span>)
</div>=20
<!------- Start Nav Bar ------>
<!-- |**|begin egp html banner|**| -->
<div id=3D"ygrp-vital" style=3D"background-color: #e0ecee; font-family: Ver=
dana; font-size: 10px; margin-bottom: 10px; padding: 10px;">
<span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-tran=
sform: uppercase; ">Recent Activity:</span>
<ul style=3D"list-style-type: none; margin: 0; padding: 0; display: inl=
ine;">
</ul>
=20=20=20=20
<div style=3D"clear: both; padding-top: 2px; color: #1e66ae;">
<a href=3D"http://groups.yahoo.com/group/hackers-il;_ylc=3DX3oDMTJlOWFp=
NmtsBF9TAzk3MzU5NzE0BGdycElkAzE4NTczMzIEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwN2dGw=
Ec2xrA3ZnaHAEc3RpbWUDMTM0MDY1MjEwNw--" style=3D"text-decoration: none;">Vis=
it Your Group</a>
</div>
</div>
=20=20
<div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p=
x; padding: 0 2px 0 0; clear: both;">
<a href=3D"http://groups.yahoo.com/;_ylc=3DX3oDMTJkdXFoZW1uBF9TAzk3MzU5Nz=
E0BGdycElkAzE4NTczMzIEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA2dmcARzdGltZ=
QMxMzQwNjUyMTA3" style=3D"float: left;"><img src=3D"http://l.yimg.com/a/i/u=
s/yg/logo/us.gif" height=3D"15" width=3D"137" alt=3D"Yahoo! Groups" style=
=3D"border: 0;"/></a>
<div style=3D"color: #747575; float: right;">Switch to: <a href=3D"mailto=
:[email protected]?subject=3DChange Delivery Format: T=
raditional" style=3D"text-decoration: none;">Text-Only</a>, <a href=3D"mail=
to:[email protected]?subject=3DEmail Delivery: Digest" clas=
s=3D"margin-rt" style=3D"text-decoration: none;">Daily Digest</a> • <a=
href=3D"mailto:[email protected]?subject=3DUnsubscrib=
e" style=3D"text-decoration: none;">Unsubscribe</a> • <a href=3D"http:=
//docs.yahoo.com/info/terms/" style=3D"text-decoration: none;">Terms of Use=
</a></div>
</div>
<!-- |**|end egp html banner|**| -->
</div> <!-- ygrp-msg -->
<!-- Sponsor -->
<!-- |**|begin egp html banner|**| -->
<div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m=
argin:0 0 25px 0; background: #fff;">
<!-- Start Recommendations -->
<div id=3D"ygrp-reco">
</div>
<!-- End Recommendations -->
</div> <!-- |**|end egp html banner|**| -->
<div style=3D"clear:both; color: #FFF; font-size:1px;">.</div>
</div>
<img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D1857332/grpspI=
d=3D1705006764/msgId=3D5184/stime=3D1340652107/nc1=3D3848642/nc2=3D4507179/=
nc3=3D5898817" width=3D"1" height=3D"1"> <br>
<div style=3D"color: #fff; height: 0;">__,_._,___</div>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</body>
<!--~-|**|PrettyHtmlStart|**|-~-->
<head>
<style type=3D"text/css">
<!--
#ygrp-mkp {
border: 1px solid #d8d8d8;
font-family: Arial;
margin: 10px 0;
padding: 0 10px;
}
#ygrp-mkp hr {
border: 1px solid #d8d8d8;
}
#ygrp-mkp #hd {
color: #628c2a;
font-size: 85%;
font-weight: 700;
line-height: 122%;
margin: 10px 0;
}
#ygrp-mkp #ads {
margin-bottom: 10px;
}
#ygrp-mkp .ad {
padding: 0 0;
}
#ygrp-mkp .ad p {
margin: 0;
}
#ygrp-mkp .ad a {
color: #0000ff;
text-decoration: none;
}
#ygrp-sponsor #ygrp-lc {
font-family: Arial;
}
#ygrp-sponsor #ygrp-lc #hd {
margin: 10px 0px;
font-weight: 700;
font-size: 78%;
line-height: 122%;
}
#ygrp-sponsor #ygrp-lc .ad {
margin-bottom: 10px;
padding: 0 0;
}
a {
color: #1e66ae;
}
#actions {
font-family: Verdana;
font-size: 11px;
padding: 10px 0;
}
#activity {
background-color: #e0ecee;
float: left;
font-family: Verdana;
font-size: 10px;
padding: 10px;
}
#activity span {
font-weight: 700;
}
#activity span:first-child {
text-transform: uppercase;
}
#activity span a {
color: #5085b6;
text-decoration: none;
}
#activity span span {
color: #ff7900;
}
#activity span .underline {
text-decoration: underline;
}
.attach {
clear: both;
display: table;
font-family: Arial;
font-size: 12px;
padding: 10px 0;
width: 400px;
}
.attach div a {
text-decoration: none;
}
.attach img {
border: none;
padding-right: 5px;
}
.attach label {
display: block;
margin-bottom: 5px;
}
.attach label a {
text-decoration: none;
}
=20=20
blockquote {
margin: 0 0 0 4px;
}
.bold {
font-family: Arial;
font-size: 13px;
font-weight: 700;
}
.bold a {
text-decoration: none;
}
dd.last p a {
font-family: Verdana;
font-weight: 700;
}
dd.last p span {
margin-right: 10px;
font-family: Verdana;
font-weight: 700;
}
dd.last p span.yshortcuts {
margin-right: 0;
}
div.attach-table div div a {
text-decoration: none;
}
div.attach-table {
width: 400px;
}
div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi=
le-title a:visited {
text-decoration: none;
}
div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div=
.photo-title a:visited {
text-decoration: none;
}
div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts {
font-family: Verdana;
font-size: 10px;
font-weight: normal;
}
.green {
color: #628c2a;
}
.MsoNormal {
margin: 0 0 0 0;
}
o {
font-size: 0;
}
#photos div {
float: left;
width: 72px;
}
#photos div div {
border: 1px solid #666666;
height: 62px;
overflow: hidden;
width: 62px;
}
#photos div label {
color: #666666;
font-size: 10px;
overflow: hidden;
text-align: center;
white-space: nowrap;
width: 64px;
}
#reco-category {
font-size: 77%;
}
#reco-desc {
font-size: 77%;
}
.replbq {
margin: 4px;
}
#ygrp-actbar div a:first-child {
/* border-right: 0px solid #000;*/
margin-right: 2px;
padding-right: 5px;
}
#ygrp-mlmsg {
font-size: 13px;
font-family: Arial, helvetica,clean, sans-serif;
*font-size: small;
*font: x-small;
}
#ygrp-mlmsg table {
font-size: inherit;
font: 100%;
}
#ygrp-mlmsg select, input, textarea {
font: 99% Arial, Helvetica, clean, sans-serif;
}
#ygrp-mlmsg pre, code {
font:115% monospace;
*font-size:100%;
}
#ygrp-mlmsg * {
line-height: 1.22em;
}
#ygrp-mlmsg #logo {
padding-bottom: 10px;
}
#ygrp-mlmsg a {
color: #1E66AE;
}
#ygrp-msg p a {
font-family: Verdana;
}
#ygrp-msg p#attach-count span {
color: #1E66AE;
font-weight: 700;
}
#ygrp-reco #reco-head {
color: #ff7900;
font-weight: 700;
}
#ygrp-reco {
margin-bottom: 20px;
padding: 0px;
}
#ygrp-sponsor #ov li a {
font-size: 130%;
text-decoration: none;
}
#ygrp-sponsor #ov li {
font-size: 77%;
list-style-type: square;
padding: 6px 0;
}=20
#ygrp-sponsor #ov ul {
margin: 0;
padding: 0 0 0 8px;
}
#ygrp-text {
font-family: Georgia;
}
#ygrp-text p {
margin: 0 0 1em 0;
}
#ygrp-text tt {
font-size: 120%;
}
#ygrp-vital ul li:last-child {
border-right: none !important;=20
}=20
-->
</style>
</head>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</html>
<!-- end group email -->
--avL5AuxvnWyTJNL5nzSwQPecaCy-JhvEQeewTmO--