Re: Re:Still somebody using ubh?
Doug McLaren <[email protected]> Mon, 5 Nov 2007 14:19:26 -0600
| Newsgroups | gmane.network.ubh |
|---|---|
| Message-ID | <[email protected]> |
--5vBfeM-qd5ainXev2zwKF4rVjzbgNrZjsYWNlfY
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
On Mon, Nov 05, 2007 at 07:30:21PM -0000, Danny wrote:
| I had trouble configuring the SQL based UBH 3, so I've given up, and
| just keep using the old UBH 2 for now. Sometimes errors occur when the
| files are too big, or when there are too many headers, and I have to
| delete my install directory and start over again. It's a pain in the
| ass but, well worth it to keep ubh running.
| I would really welcome an upgrade, and would try to help too if people
| wanted to try to put together a new package.
I'm still using ubh2 as well.
I tried ubh3, but it seemed to just offload the work from the ubh
process to the database and it didn't seem to actually perform better.
For me, ubh's performance degraded to the point where it could enter a
group and spend 15 hours just deciding which articles to download. I
tracked the problem down to Set::IntSpan, as used by News::Newsrc --
there were tens of thousands of spans and it had to go through on
average half of them to check each article to see if it's had been
read or not -- and it had to do this for each article.
I started to fix Set::IntSpan to use a binary search, but gave up and
just added code to ubh to make a mirror copy of the Set::IntSpan with
Set::IntRange and do it's lookups using that. This turned a 15 hour
process into a few minute process in the biggest groups.
Yes, it used more memory, but not that much more memory --
Set::IntRange uses Bit::Vector which is very efficient about memory.
Even a range of ten million articles only works out to a little more
than a megabyte of memory.
In any event, here's the fix --
In sub _merge_subjects, this goes after
Ubh::Cache::refresh($server, $group, $cachefile);
--- cut here ---
# dougmc - build cache for massive speed improvement
print " Building read-only in-memory cache of $group newsrc entries ...\n" ;
my $intspan = Set::IntSpan->new($server->newsrc->get_articles($group)) ;
my $is_max = $intspan->max ;
my ($is_min, $is_set) ;
{
my @is_spans = $intspan->spans() ;
print " ... " . scalar @is_spans . " spans found.\n" ;
$is_min = ${$is_spans[0]}[1] ;
# sane values if a new group
$is_min = 1 if ($is_min < 1) ;
$is_max = 1 if ($is_max < 1) ;
$is_set = Set::IntRange->new($is_min,$is_max);
foreach my $span (@is_spans) {
my $min = $$span[0] ;
my $max = $$span[1] ;
$min = $max if ($min == 1) ;
$is_set->Interval_Fill($min, $max) ;
}
}
and this code --
next if (!$opt->{'a'} && !$opt->{'s'} &&
$server->newsrc->marked($group, $num));
is replaced with this code --
next if ($num < $is_min) ; # automatically marked.
if ($num <= $is_max) { # if this fails, automatically unmarked
next if $is_set->bit_test($num) ;
}
and add these linkes --
use Set::IntSpan ;
use Set::IntRange ;
right after the
use News::Newsrc;
line. I'd give you diffs, but my version is quite different from the
stock version now.
Really, I ought to finish fixing Set::IntSpan -- the changes would be
more universally usable, though I suspect it still wouldn't be faster
than what I did here.
--
Doug McLaren, [email protected]
Cable. It's more wonderful than I dared hope. --Homer Simpson
--5vBfeM-qd5ainXev2zwKF4rVjzbgNrZjsYWNlfY
Content-Type: text/html; charset=US-ASCII
Content-Transfer-Encoding: 7bit
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
</head>
<body style="background-color: #ffffff;">
<!--~-|**|PrettyHtmlStartT|**|-~-->
<div id="ygrp-mlmsg" style="width:655px; position:relative;">
<div id="ygrp-msg" style="width: 490px; padding: 0 15px 0 0; float:left; z-index:1;">
<!--~-|**|PrettyHtmlEndT|**|-~-->
<div id="ygrp-text">
<p>On Mon, Nov 05, 2007 at 07:30:21PM -0000, Danny wrote:<br>
<br>
| I had trouble configuring the SQL based UBH 3, so I've given up, and<br>
| just keep using the old UBH 2 for now. Sometimes errors occur when the<br>
| files are too big, or when there are too many headers, and I have to<br>
| delete my install directory and start over again. It's a pain in the<br>
| ass but, well worth it to keep ubh running.<br>
| I would really welcome an upgrade, and would try to help too if people<br>
| wanted to try to put together a new package.<br>
<br>
I'm still using ubh2 as well.<br>
<br>
I tried ubh3, but it seemed to just offload the work from the ubh<br>
process to the database and it didn't seem to actually perform better.<br>
<br>
For me, ubh's performance degraded to the point where it could enter a<br>
group and spend 15 hours just deciding which articles to download. I<br>
tracked the problem down to Set::IntSpan, as used by News::Newsrc --<br>
there were tens of thousands of spans and it had to go through on<br>
average half of them to check each article to see if it's had been<br>
read or not -- and it had to do this for each article.<br>
<br>
I started to fix Set::IntSpan to use a binary search, but gave up and<br>
just added code to ubh to make a mirror copy of the Set::IntSpan with<br>
Set::IntRange and do it's lookups using that. This turned a 15 hour<br>
process into a few minute process in the biggest groups.<br>
<br>
Yes, it used more memory, but not that much more memory --<br>
Set::IntRange uses Bit::Vector which is very efficient about memory.<br>
Even a range of ten million articles only works out to a little more<br>
than a megabyte of memory.<br>
<br>
In any event, here's the fix --<br>
<br>
In sub _merge_subjects, this goes after<br>
<br>
Ubh::Cache::<wbr>refresh($<wbr>server, $group, $cachefile); <br>
<br>
--- cut here ---<br>
<br>
# dougmc - build cache for massive speed improvement<br>
print " Building read-only in-memory cache of $group newsrc entries ...\n" ;<br>
my $intspan = Set::IntSpan-<wbr>>new($server-<wbr>>newsrc-><wbr>get_articles(<wbr>$group)) ;<br>
my $is_max = $intspan->max ;<br>
my ($is_min, $is_set) ;<br>
{ <br>
my @is_spans = $intspan->spans(<wbr>) ;<br>
print " ... " . scalar @is_spans . " spans found.\n" ;<br>
$is_min = ${$is_spans[<wbr>0]}[1] ;<br>
<br>
# sane values if a new group<br>
$is_min = 1 if ($is_min < 1) ;<br>
$is_max = 1 if ($is_max < 1) ;<br>
<br>
$is_set = Set::IntRange-<wbr>>new($is_<wbr>min,$is_max)<wbr>;<br>
foreach my $span (@is_spans) {<br>
my $min = $$span[0] ;<br>
my $max = $$span[1] ;<br>
$min = $max if ($min == 1) ;<br>
$is_set->Interval_<wbr>Fill($min, $max) ;<br>
}<br>
}<br>
<br>
and this code --<br>
<br>
next if (!$opt->{'a'<wbr>} && !$opt->{'s'} &&<br>
$server->newsrc-<wbr>>marked($<wbr>group, $num));<br>
<br>
is replaced with this code --<br>
<br>
next if ($num < $is_min) ; # automatically marked.<br>
if ($num <= $is_max) { # if this fails, automatically unmarked<br>
next if $is_set->bit_<wbr>test($num) ;<br>
}<br>
<br>
and add these linkes --<br>
<br>
use Set::IntSpan ;<br>
use Set::IntRange ;<br>
<br>
right after the<br>
<br>
use News::Newsrc;<br>
<br>
line. I'd give you diffs, but my version is quite different from the<br>
stock version now.<br>
<br>
Really, I ought to finish fixing Set::IntSpan -- the changes would be<br>
more universally usable, though I suspect it still wouldn't be faster<br>
than what I did here.<br>
<br>
--<br>
Doug McLaren, <a href="mailto:dougmc%40frenzied.us">dougmc@frenzied.<wbr>us</a><br>
Cable. It's more wonderful than I dared hope. --Homer Simpson<br>
</p>
</div>
<!--~-|**|PrettyHtmlStart|**|-~-->
<span width="1" style="color: white;">__._,_.___</span>
<!-- Start the section with Message In topic -->
<div id="ygrp-actbar">
<span class="left">
<a href="http://groups.yahoo.com/group/ubh/message/792;_ylc=X3oDMTMyMzdvaDJlBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBG1zZ0lkAzc5NARzZWMDZnRyBHNsawN2dHBjBHN0aW1lAzExOTQyOTU4MjMEdHBjSWQDNzky">
Messages in this topic </a> (<span class="bld">0</span>)
</span>
<a href="http://groups.yahoo.com/group/ubh/post;_ylc=X3oDMTJva2xvcnM4BF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBG1zZ0lkAzc5NARzZWMDZnRyBHNsawNycGx5BHN0aW1lAzExOTQyOTU4MjM-?act=reply&messageNum=794">
<span class="bld">
Reply </span> (via web post)
</a> |
<a href="http://groups.yahoo.com/group/ubh/post;_ylc=X3oDMTJlZDdpYTI2BF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTE5NDI5NTgyMw--" class="bld">
Start a new topic </a>
</div>
<!------- Start Nav Bar ------>
<!-- |**|begin egp html banner|**| -->
<div id="ygrp-vitnav">
<a href="http://groups.yahoo.com/group/ubh/messages;_ylc=X3oDMTJlaW8zb3BiBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA21zZ3MEc3RpbWUDMTE5NDI5NTgyMw--">Messages</a>
| <a href="http://groups.yahoo.com/group/ubh/files;_ylc=X3oDMTJmMjBrYWEzBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2ZpbGVzBHN0aW1lAzExOTQyOTU4MjM-">Files</a>
| <a href="http://groups.yahoo.com/group/ubh/photos;_ylc=X3oDMTJlZmxkbTNhBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA3Bob3QEc3RpbWUDMTE5NDI5NTgyMw--">Photos</a>
| <a href="http://groups.yahoo.com/group/ubh/links;_ylc=X3oDMTJmZTAzc29hBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2xpbmtzBHN0aW1lAzExOTQyOTU4MjM-">Links</a>
| <a href="http://groups.yahoo.com/group/ubh/database;_ylc=X3oDMTJjZ2IyaXFpBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2RiBHN0aW1lAzExOTQyOTU4MjM-">Database</a>
| <a href="http://groups.yahoo.com/group/ubh/polls;_ylc=X3oDMTJmNHN0aWg0BF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA3BvbGxzBHN0aW1lAzExOTQyOTU4MjM-">Polls</a>
| <a href="http://groups.yahoo.com/group/ubh/members;_ylc=X3oDMTJlNWw2ZXI0BF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA21icnMEc3RpbWUDMTE5NDI5NTgyMw--">Members</a>
| <a href="http://groups.yahoo.com/group/ubh/calendar;_ylc=X3oDMTJkNXQ4ODMwBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2NhbARzdGltZQMxMTk0Mjk1ODIz">Calendar</a>
</div>
<!-- |**|end egp html banner|**| -->
<div id="ygrp-grft">
</div>
<!-- yahoo logo -->
<!-- |**|begin egp html banner|**| -->
<div id="ygrp-ft">
<a href="http://groups.yahoo.com/;_ylc=X3oDMTJkaHMwbHE0BF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2dmcARzdGltZQMxMTk0Mjk1ODIz">
<img src="http://us.i1.yimg.com/us.yimg.com/i/yg/img/logo/ma_grp_160.gif" height="15" width="106" border="0" alt="Yahoo! Groups"></a> <br>
<a href="http://groups.yahoo.com/group/ubh/join;_ylc=X3oDMTJma2I0Ym5yBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA3N0bmdzBHN0aW1lAzExOTQyOTU4MjM-">Change settings via the Web</a> (Yahoo! ID required) <br>
Change settings via email: <a href="mailto:[email protected]?subject=Email Delivery: Digest">Switch delivery to Daily Digest</a> | <a href = "mailto:[email protected]?subject=Change Delivery Format: Traditional">Switch format to Traditional</a> <br>
<a href="http://groups.yahoo.com/group/ubh;_ylc=X3oDMTJkZjFzbHZyBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwNmdHIEc2xrA2hwZgRzdGltZQMxMTk0Mjk1ODIz">
Visit Your Group
</a> |
<a href="http://docs.yahoo.com/info/terms/">
Yahoo! Groups Terms of Use </a> |
<a href="mailto:[email protected]?subject=">
Unsubscribe </a>
</div> <!-- |**|end egp html banner|**| -->
</div> <!-- ygrp-msg -->
<!-- Sponsor -->
<!-- |**|begin egp html banner|**| -->
<div id="ygrp-sponsor" style="width:140px;float: left; clear: none; margin-left: 5px; background:white; margin-bottom:25px ;position:absolute; top:0; right: 0;">
<!-- Network content -->
<!-- Start vitality -->
<div id="ygrp-vital">
<a href="http://groups.yahoo.com/group/ubh;_ylc=X3oDMTJldHE1MjVjBF9TAzk3MzU5NzE0BGdycElkAzI5MTcwNTQEZ3Jwc3BJZAMxNzA1MDA1NTEyBHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTE5NDI5NTgyMw--">
Visit Your Group </a>
</div>
<!-- Network content -->
<div id="nc">
<div class="ad">
<div id="hd1">Cat Zone</div>
<p><a href="http://us.ard.yahoo.com/SIG=12j8jahu0/M=493064.11292562.11840176.8674578/D=groups/S=1705005512:NC/Y=YAHOO/EXP=1194303023/A=4836039/R=0/SIG=11olbte0b/*http://advision.webevents.yahoo.com/catzone/index.html">on Yahoo! Groups</a></p>
<p>Join a Group</p>
<p>all about cats.</p> </div>
<div class="ad">
<div id="hd1">Yahoo! Groups</div>
<p><a href="http://us.ard.yahoo.com/SIG=12j4enjln/M=493064.11036139.11614791.8674578/D=groups/S=1705005512:NC/Y=YAHOO/EXP=1194303023/A=4725796/R=0/SIG=1192rfjiu/*http://groups.yahoo.com/group/realfood/">Real Food Group</a></p>
<p>Share recipes</p>
<p>and favorite meals.</p> </div>
<div class="ad">
<div id="hd1">Women of Curves</div>
<p><a href="http://us.ard.yahoo.com/SIG=12kl6pjtn/M=493064.11674772.12153082.11322765/D=groups/S=1705005512:NC/Y=YAHOO/EXP=1194303023/A=4990222/R=0/SIG=11odsb6gn/*http://new.groups.yahoo.com/Women_Of_Curves_Everywhere">on Yahoo! Groups</a></p>
<p>see how women are</p>
<p>changing their lives.</p> </div>
</div>
</div> <!-- |**|end egp html banner|**| -->
<div style="clear:both; color: #FFF; font-size:1px;">.</div>
</div> <img src="http://geo.yahoo.com/serv?s=97359714/grpId=2917054/grpspId=1705005512/msgId=794/stime=1194295823/nc1=4836039/nc2=4725796/nc3=4990222" width="1" height="1"> <br>
<span style="color: white;">__,_._,___</span>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</body>
<!--~-|**|PrettyHtmlStart|**|-~-->
<head>
<style type="text/css">
<!--
#ygrp-mkp{
border: 1px solid #d8d8d8;
font-family: Arial;
margin: 14px 0px;
padding: 0px 14px;
}
#ygrp-mkp hr{
border: 1px solid #d8d8d8;
}
#ygrp-mkp #hd{
color: #628c2a;
font-size: 85%;
font-weight: bold;
line-height: 122%;
margin: 10px 0px;
}
#ygrp-mkp #ads{
margin-bottom: 10px;
}
#ygrp-mkp .ad{
padding: 0 0;
}
#ygrp-mkp .ad a{
color: #0000ff;
text-decoration: none;
}
-->
</style>
</head>
<head>
<style type="text/css">
<!--
#ygrp-sponsor #ygrp-lc{
font-family: Arial;
}
#ygrp-sponsor #ygrp-lc #hd{
margin: 10px 0px;
font-weight: bold;
font-size: 78%;
line-height: 122%;
}
#ygrp-sponsor #ygrp-lc .ad{
margin-bottom: 10px;
padding: 0 0;
}
-->
</style>
</head>
<head>
<style type="text/css">
<!--
#ygrp-mlmsg {font-size:13px; font-family: arial,helvetica,clean,sans-serif;*font-size:small;*font:x-small;}
#ygrp-mlmsg table {font-size:inherit;font:100%;}
#ygrp-mlmsg select, input, textarea {font:99% arial,helvetica,clean,sans-serif;}
#ygrp-mlmsg pre, code {font:115% monospace;*font-size:100%;}
#ygrp-mlmsg * {line-height:1.22em;}
#ygrp-text{
font-family: Georgia;
}
#ygrp-text p{
margin: 0 0 1em 0;
}
#ygrp-tpmsgs{
font-family: Arial;
clear: both;
}
#ygrp-vitnav{
padding-top: 10px;
font-family: Verdana;
font-size: 77%;
margin: 0;
}
#ygrp-vitnav a{
padding: 0 1px;
}
#ygrp-actbar{
clear: both;
margin: 25px 0;
white-space:nowrap;
color: #666;
text-align: right;
}
#ygrp-actbar .left{
float: left;
white-space:nowrap;
}
.bld{font-weight:bold;}
#ygrp-grft{
font-family: Verdana;
font-size: 77%;
padding: 15px 0;
}
#ygrp-ft{
font-family: verdana;
font-size: 77%;
border-top: 1px solid #666;
padding: 5px 0;
}
#ygrp-mlmsg #logo{
padding-bottom: 10px;
}
#ygrp-vital{
background-color: #e0ecee;
margin-bottom: 20px;
padding: 2px 0 8px 8px;
}
#ygrp-vital #vithd{
font-size: 77%;
font-family: Verdana;
font-weight: bold;
color: #333;
text-transform: uppercase;
}
#ygrp-vital ul{
padding: 0;
margin: 2px 0;
}
#ygrp-vital ul li{
list-style-type: none;
clear: both;
border: 1px solid #e0ecee;
}
#ygrp-vital ul li .ct{
font-weight: bold;
color: #ff7900;
float: right;
width: 2em;
text-align:right;
padding-right: .5em;
}
#ygrp-vital ul li .cat{
font-weight: bold;
}
#ygrp-vital a{
text-decoration: none;
}
#ygrp-vital a:hover{
text-decoration: underline;
}
#ygrp-sponsor #hd{
color: #999;
font-size: 77%;
}
#ygrp-sponsor #ov{
padding: 6px 13px;
background-color: #e0ecee;
margin-bottom: 20px;
}
#ygrp-sponsor #ov ul{
padding: 0 0 0 8px;
margin: 0;
}
#ygrp-sponsor #ov li{
list-style-type: square;
padding: 6px 0;
font-size: 77%;
}
#ygrp-sponsor #ov li a{
text-decoration: none;
font-size: 130%;
}
#ygrp-sponsor #nc{
background-color: #eee;
margin-bottom: 20px;
padding: 0 8px;
}
#ygrp-sponsor .ad{
padding: 8px 0;
}
#ygrp-sponsor .ad #hd1{
font-family: Arial;
font-weight: bold;
color: #628c2a;
font-size: 100%;
line-height: 122%;
}
#ygrp-sponsor .ad a{
text-decoration: none;
}
#ygrp-sponsor .ad a:hover{
text-decoration: underline;
}
#ygrp-sponsor .ad p{
margin: 0;
}
o{font-size: 0; }
.MsoNormal{
margin: 0 0 0 0;
}
#ygrp-text tt{
font-size: 120%;
}
blockquote{margin: 0 0 0 4px;}
.replbq{margin:4}
-->
</style>
</head>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</html><!--End group email -->
--5vBfeM-qd5ainXev2zwKF4rVjzbgNrZjsYWNlfY--