Re: [Pyds-users] Apostrophes in the aggregator

Bauer, Georg wrote:
> Hi!
> 
> 
>>Im bored.
> 
> 
> I checked with the RSS debugging features (go to your /aggregator/ page, go
> to the channel configuration, search the feed in question, click on the XML
> icon, select what format you want to see). It looks like the ' somehow
> just get's lost. The unquote method correctly translates ' to ', but it
> looks as if it actually doesn't make it through the parser ...

Per Georg's request in a later private email, I'm following up on this 
list on the complaint I made on the pyds-users list. If you are not 
subscribed to that list, I objected that in RSS feeds that have a ' 
in them, the apostrophe disappears before getting to the user's 
aggregator page and wondered if anyone else noticed.

I've fixed the bug but I haven't taken the time to track down *exactly* 
what happens; there's still a fuzzy spot in my understanding. But here's 
what I know:

* The aggregator parses the RSS feed with the RSSFeed class using the 
SGML library. This works correctly.
* In the updateFeed method, it processes the RSS for safety before 
storing it anywhere or displaying it. Among the things it does is call 
Stripogram.html2safehtml on the description (text) string. 
Stripogram.html2safehtml does the following:

 >>> Stripogram.html2safehtml("'")
'&apos'

This is in an effort to defuse potentially dangerous entities. This 
occurs in the handle_entityref method of Stripogram.HTML2SafeHTML, which 
is the following:

         if self.entitydefs.has_key(name):
             x = ';'
         else:
             # this breaks unstandard entities that end with ';'
             x = ''
         self.result = "%s&%s%s" % (self.result, name, x)

At the beginning of that class is a

     from htmlentitydefs import entitydefs

And here we get to the core of the problem. htmlentitydefs does not 
contain "apos", which is extremely surprising to me. Anyhow, adding

import htmlentitydefs
htmlentitydefs.entitydefs['apos'] = "'"

to the top of Stripogram makes the apostrophes come back.

I'm a little fuzzy on how "'" => "&apos" => "", but admit to not 
caring much. ;-)