On the current situation regarding ID3 tagging under Linux

Hi,

because I need a working solution for working ID3 tagging related tags which
contain non-ASCII characters under Linux for creating a small ripped library
from CDs with titles which contain lots of latin1 characters, I was forced to
do some research of the current situation (from this viewpoint).

As grip is probably the "most advanced combined ripper+editor"? for GNU/Linux,
(please tell me if you know other compareable or better programs) and the
research which I have done concerns grip and is also related to two bug entires
for grip, I post this to this list (in absence of a grip-devel list).

If you have more/different information, please speak up!

I think that most of this also applies to all UNIX-like operating systems,
since most programs in this are apprear to be widely cross-platform.

Currently, most traditional ID3-supporting programs seem to use
the id3lib which is apparently unmaintained at the moment.

http://id3lib.sourceforge.net/

It was written in C++ and the last release, which has been done in March 2003,
lacked Unicode support in it's C wrappers. The functions exist but do not work
as they should. This has been reported, even patches have been proposed, but
there has been no new release of the library.

10 days ago, Jérôme announced a new version of easytag on the id3lib list:
http://sourceforge.net/mailarchive/forum.php?thread_id=8882640&forum_id=5794

he wrote:
| I'm pleased to announce a new version of EasyTAG which fully
| supports Unicode ID3v2 tags with id3lib. This feature was sadly lacking
| in the previous versions...
| Indeed, as EasyTAG uses C wrappers for handling tags with id3lib, and as
| the required C wrappers to manage Unicode were missing, this problem
| wasn't solved into the program up to... ...the contribution of Alexey
| Illarionov.
| 
| After some weeks of adjustments, now it works perfectly ;-)

So there may be now a solution to make Unicode work, but apparently it's
not simple. However it may be visible when looking at the changes the the
new version.

Other than this announcement, I have not seen any information of a specific
id3lib program supporting unicoded ID3 tags, maybe some C++ programs may work
if they implemented Unicode support, but I don't know.

Personally, I do not need Unicode for ID3 tags at the moment, but 
people with Music from outside of the latin1 sphere may need it.

But e.g. the popular xmms which also uses id3lib does not support
Unicode tags, at least not if encoded as UTF-8. It even shows some
strange pattern of ASCII characters in it's track display when playing
an mp3 file which has UTF-8 tags.

Luckily, easytag can now deal with it and it can be used to convert
such tags to use ISO-8859-1 instead (where conversion or transliteration
is possible), at least manually by forcing it to write the tags.

Besides other Operating Systems being a source of non-Latin1 tags, the
taglib which is used by a number of KDE applications such as the
sophisticated amaroK mp3 collection player use it, and when editing
Tags with it's current SVN code, it writes the tags always in UTF-8
encoding, as far as I can see.

The bugs in id3lib regarding Unicode are:

Bug entries regarding grip:

UTF-8 not working in current grip versions:
http://sourceforge.net/tracker/index.php?func=detail&aid=1196919&group_id=3714&atid=103714

Explains more exactly what grip does currently and gives
some suggestions:
http://sourceforge.net/tracker/index.php?func=detail&aid=1200001&group_id=3714&atid=103714

Summarzing the above bugs from reading, testing and trying code:

grip currently has no proper support for writing Tags in Unicode, when it is told
to write ID3 tags as UTF-8, it does indeed so, but because the id3lib C wrapper
which would be needed to set the encoding bit in the tag to UTF-8 does apparently
nothing and grip has to use the C wrapper for ASCII instead, the tags are encoded
as UTF-8 but not tagged as such.

Other programs which use id3lib (at least xmms) seem to deal with that by way of
simplyy decoding the UTF-8, at least it seems to do so if it's started in an UTF-8
locale - xmms seems to behave different when reading tags, depending on the locale
charmap which the locale environment it has been started from uses.

But programs which use taglib (at least amarok) seem to correctly view the ID3
tag, causing the wrongly tagged non-ASCII characters in it to display as strange
characters, e.g. german Umlauts appear with an Ã, followed by some other character,
exactly as the UTF-8 bytes view, when shown in latin1 encodings.

The last bug above shows good reseach and very easy recommendations:

|  ID3v1 set as ISO_8859_1 but encoded as UTF-8
|
| I configured grip as suggested by the faq: id3v1 tags
| encoded as ISO_8859_1 and id3v2 encoded as UTF-8.
| but it looks like the id3v1 are being recorded as UTF-8.
| 
| Here is the id3v1 in the begining of the fhe file of
| the music "Botão de Rosa":
| Botão de Rosa
| and the id3v2 (in the end of file):
| Botão de Rosa
| 
| 
| Ops, I've just tried to encode it with "ISO-8859-1" and
| everything worked fine, the track appeared as "Botão de
| Rosa".
| 
| So this bug is for two things:
| 
| 1) Correct the documentation! Change the "ID3v1
| Character set encoding — The character encoding to be
| used for ID3v1 tags. ISO_8859_1 is recommended." to
| "ID3v1 Character set encoding — The character encoding
| to be used for ID3v1 tags. ISO-8859-1 is recommended."
| 
| 2) Print a warning if the id3 character encoding isn't
| supported (an unknown encoding, as ISO_8859_1)

I found this to be correct. The Documentation shows ISO_8859_1, but this
does not work. ISO-8859-1 is correct - the Documentation is in need of
correction.

I started doing the neccesary change to the Documentation (text and
screenshots, the latter by editing the screenshot image directly),
and would send these updates, if I'm asked by the maintainer of grip.

I would als update the grip web site with these changes if given the
permission.

Regarding the 2nd recommendation, I'm unsure, but at least the first
part is clear and in part of warnings, a warning for UTF-8 could be
added. If the solution used by easytag could be adopted by grip also,
then the default setting of easytag 1.99.9, which is to save as latin1
whenever possible and only save as Unicode when neccessay seems to be
the best default.

But I would like to have a word from the grip maintainer on that,
since I'm a burned kid from other projects where some maintainers
are simply not interested in some things and even refuse to take
tiny improvements on board, without even explaining anything and
I would like to know different before starting reseach and coding.

Actually, I do not need much for myself, but since the above bugs
bite me quite hard (after ripping lots of CDs with grip) I find
it quite unsatisfactory to know that other are also likely to be
bitten by the same bugs until the issues are addressed.

Other than this, I find the suggestion from Christoffer Gurell on
this list Multi-artist file names from 2nd Novenber quite useful
and I think his suggestion should be implemented. Would a patch
which implements this be accepted by the maintainer and a new
version released afterwards?

Besides that, I would like to make some information which is
available from freedb, such as CD comments and year information
on track, not only CD level, available to grip, at least for
storing this information (e.g. the track year) in id3v2 tags.

I have worked on projects under CVS in the past, so I see no
problems in contributing if we can create working relationship.

Bernhard

PS: id3lib bugs related to Unicode (all open and unfixed, apparently
worked-around in the new development version of easytags):

- reading Unicode broken, with test case and patch:
http://sourceforge.net/tracker/index.php?func=detail&aid=587475&group_id=979&atid=100979
- Problem with unicode when both ID3V1 and ID3V2 tags present:
http://sourceforge.net/tracker/index.php?func=detail&aid=980234&group_id=979&atid=100979 
- Unicode not working from C bindings:
http://sourceforge.net/tracker/index.php?func=detail&aid=918943&group_id=979&atid=100979