Re: latex bibliographies with multiple databases

Hi David,

David Nebauer writes:
 > Yes, in my view the storage format is Unicode without markup:
 > 
 > 
 > BibTeX -------               ---------> DocBook
 >              |               |
 >              |               |
 > RIS ---------+--> STORAGE ----
 >              |   (Unicode)   |
 >              |               |
 > RISX ---------               ---------> LaTeX
 > 
 > 
 > 

I've done a little source code reading and testing in order to find
out how RefDB mangles these kinds of input and output data. My results
are as follows:

1) BibTeX input
bib2ris appears to work ok with UTF-8 encoded bibtex data. You can
import the resulting RIS data as long as the input encoding is set to
UTF-8 (the current default is ISO-8859-1, but it certainly makes sense
to change that). If your bibtex data is plain ASCII with foreign and
special characters encoded as LaTeX commands, the bib2ris output
should be sent through the new refdb_latex2utf8txt script. I don't
know whether it really has 100% coverage of the character-related
LaTeX command, but it is easy to extend if need arises. With this in
mind we can import bibtex data as plain Unicode.

2) RIS input
We'd have to educate users to author their RIS datasets in UTF-8, and
to run RIS data from web sources (like Pubmed) through iconv before
adding them to RefDB. All it takes is to set the default input
encoding of refdbd for RIS data to UTF-8 (see above). Currently there
are no provisions to translate entities or LaTeX commands, but if used
correctly there should be no need to use such hacks. The result is, as
above, plain Unicode.

3) risx input
I've rediscovered a nice feature of expat (which refdbd
uses to parse all incoming XML data). The output data of expat are
always UTF-8, with all entities expanded to their Unicode
equivalents. Thus no extra conversion step is required to get rid of
entities and to store plain Unicode.

4) SGML/XML output (bibliographies, db31/tei/html backends)
"<>&" are replaced with their corresponding entities. In addition, the
current code contains replacements for &mdash; &lsquo; and &rsquo;. I
know that I was asked to add these, but I can't remember the
context. I wonder whether it would make more sense to keep these
characters as Unicode.

5) LaTeX output
There are currently no attempts to escape LaTeX command
characters. I'm about to add this code.

6) other output (RIS, screen)
No replacements. If you retrieve data as UTF-8, you'll get what you
want.


As always, I might have missed some bordercases. If you experience a
different behaviour, please let me know.

One thing that should be discussed is how easy it is for RefDB users
to author UTF-8 data, be it RIS, bibtex, or XML. You can always insert
the numeric form into XML data (e.g. &#0x00B1) but I'm afraid this
won't work for the other data formats. As an Emacs user I've got Norm
Walsh's xmlunicode.el (http://nwalsh.com/emacs/xmlchars/) which allows
to select characters from a pop-up list or from the minibuffer with
entity-name completion, and which also defines an input mode which
offers on-the-fly replacement of entities. Is there similar support
available for other editors (vim, jedit) which should be mentioned in
the manual?

regards,
Markus

-- 
Markus Hoenicka
[email protected]
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV