Re: embedded inline pictures

Luca Della Santina wrote:
> I think is a good assumption to think that USF, as well as any of the 
> other textual subtitle format, is not going to contain a large amount of 
> pictures inside itself, as well as no very big picture must be included 
> in a single subtitle, otherwise it is obvious that playing will be choppy.

> Cons: Since the picture is streamed along with the subtitle, the 
> processing time for that subtitle increases a lot, and this brings 
> higher probability of choppy playback.

This can only be seen by trying it. You are right that passing the 
pretty long string of such base64 encoded data can slow down some parts, 
but i hope it will be offset by the radically reduced composing 
complexity for the final bitmap. Yesterday i tried a sample where i 
muxed 4 inline pictures (1xPNG,1xJPEG,1xGIF,1xOPS) and testApp did not 
seem to lag visibly even tho their size was ranging from 30 to 130KB.
It is currently a major hassle to produce files with embedded images, 
but once i have a file with some more pictures in it, im goint to do 
timing tests. But of course this will only time pixifier, the additional 
lag produced in the splitter and pixishow probably has to be taken into 
consideration as well.

> Except when you want to use the same picture more than one time in the 
> script (like a logo or somethin else) in this case it is very useful the 
> old way (picture in <embedded> + link in the <image>).

Yes, inline pictures are clearly not for logos and other repeated 
pictures. Also i expect files with inline pictures not being produced 
manually, but by an application that takes a text based script and 
renders it into pictures and inlines those.
You can of course create inline pictures manually, for example if you 
have a few things that are not possible with USF styling and want them 
included in an otherwise text based script.


> I think "content-type" name for the attribute fits beter since "content" 
> seems that the image data are going to be contained in the attribute 
> itself.

Totally agreed. I will use content-type.

> Maybe the default value has to be "inline" and not "reference", since 
> the most of times you create a picture event the picture is meant for 
> that subtitle only, the case where you need to share the same picture 
> among more subtitles is not so common.

I'm unsure about this. From a logical point you are right. But it 
invalidates old scripts. Especially bad is that USFV100_sample.usf will 
become even more invalid than it already is :)
Maybe we should really use a new USF version number at some time...

> At this stage retrocompatibility has not to be a primary goal, since 
> there are not so many usf subtitles around and even less USF rasterizers.
It's a point. But XML would aready make it so easy to design extensible 
and still backward compatible file formats we should not waste it 
without always thinking about how we can reduce the impact on 
compatibility. Of course in our case, if the compatibility becomes a 
major drag it is better to break it and use a clean solution.

> Visualizng your proposal, a "reference" image event becomes:
> 
> <image content-type="reference">MyPicture.bmp</image>
> 
> while an "inline" image event becomes:
> 
> <image content-type="inline/bmp">MyBase64EncodedStuff</image>

Correct (except see below)

> There are some weak points:
> 1) Having a node content type that changes so much (a string to a b64 
> encoded string) is not always a good pratice (often in these cases you 
> use 2 different node types for that like <image> and <inline-image>, 
> which is obviously not our goal).

Again a good point. Still i opt for the one element for both content 
types. If people there are strong arguments against the "combined" type 
from the user side, we could still switch to a seperate <inline-image>. 
This is a proposal after all, and not like changing the base 
specification draft 0.16.
Now that you mention this, <image> with a reference is actually a bit 
unlucky, especially with the reference pointer in the pareable content 
(compared to an attribute). You could actually say, a reference is a 
reference no matter whether it points to a raster image or a SVG 
declaration or whatever else. And the use <image> for inline images only 
and <reference type="image" xref="this.bmp" /> to refer to an embedded 
or external image.

But this is kind of out of question IMO. A merge idea how it could also 
have to be done.
Please do not consider this as a proposal i back :)

> 2) Maybe is good to place in a separate attribute the MIME type / file 
  extension, this make "content-type" attribute parsing simpler, and
> because otherwise "inline" cannot be the default for <image>).

Agreed.

> 3) When you use an inlined picture you definitively loose the original 
> image filename (you only preserve its file extension/MIME type).

I have zero problem with losing the filename. I suspect you think about 
extracting the image again. But IMO embedded files are not meant to be 
extracted again. Of course you can (text editors and base64 decoders are 
cheap today ;)) but the name of an inlined image probably has no meaning 
anyway. If the application that embeds inline images specifies filenames 
such as "image_0001.png", "image_0002.png" you can as well not mention 
them and assign them a random filename at the time you extract them.

> In order to solve these issues please consider this modification to your 
> proposal:
> 
> - <image> node text content will become only a base64 encoded string.
> - A new attribute for the <image> node is declared: "filename"
> 
> in this way a "reference" image event becomes:
> 
> <image content-type="reference" filename="MyFileName.bmp"/>
> 
> while a "inline" image event becomes:
> 
> <image content-type="inline" 
> filename="MyFileName.bmp">ByBase64EncodedStuff</image>

MMM i should have read all of the mail before writing the other comment 
above =>
If you too think the reference does not belong into the content, it is 
probably really not that great.

I am a bit in doubt, but i think we should drop the possibility to 
specify the MIME-Type. We can give the extension in filename and this 
serves as a early-abort for unsupported types as well as a MIME-Type.

Now if you are going to change that much, i also tend to favor "inline" 
as default content type. Parially also because i came up with a error 
handling that should be "solid" for old images.

Also again, if the filename is no longer "content", the name 
"content-type" doesn't fit the meaning too well form the data structure 
point of view.
But "type" again is too general, and "content" has not become more 
fitting. So ok, let's stick with content-type, from the "user" point of 
view the image istill is sort of content of that element and there 
probably is no totally fitting name for that attribute.


*Revised* Proposal

the <image> element should be changed in the following way:

*introduce a new attribute "content-type"
This attribute is optional and defaults to the value "inline".
The possible values of this attribute are "reference"|"inline".

*introduce a new attribute "filename"
This attribute is mandatory.

*if the element is a reference the element must be empty. See below for 
a note on this. If the element is an inline image the content of the 
element is the base64 encoded file data.




Notes:

For reference elements, the value of the filename attribute is the name 
of the the ressource to be liked to. The ressource is either an embedded 
image with corresonding filename attribute or an external file with that 
filename.

For inline image elements, the value of the filename is merely 
informational. It is required to at least provide the correct extension 
of the inlined image, but it doesn not have to be a filename windows 
likes. And it does not have to be unique. It is perfectly legal to 
specify filename=".png" for all inlined images (if they are in fact PNGs 
, that is).

USF files using the old way for images are in error with this new 
specification. But the following error handling in this case is 
mandatory for all parsers compliant with this proposal:

If a <image> element does not have the attribute "filename" it is in 
error. If such an element is encountered, the parser must assume the 
<image> element follows the old standard and interpret the _content_ of 
the element as a _reference_. However the parser should notify the user 
of this, unless this inflicts with the uability of the application.
In other words, an editing application must inform the user of this 
error, but a playback application does not have to, if the notice would 
interrupt playback. If a playback application provides an informational 
output such as a "properties" or "info" panel or window, the error must 
be mentioned there, but it must not pop up an error dialog blocking the 
playback. A compliant software must not write the old type of image 
elements. Even if it is merely copying data, it must transform such 
elements into the new form.
(This will help getting rid of the old type since you only have to load 
an old script in a compliant software and immediately save it to 
transform into a compliant script).

If an image element "looks" like the new type (it does have the filename 
attribute), a parser must not interpret the content as reference, even 
if it is not valid base64 data. If the content in this case is not valid 
base64 data, the handling must be the same as if a reference were not 
found. If the decoded data is of a type the application does not 
understand, it may apply error handling for "bad file type". The 
handling for "bad file type" may be the same as for "file not found" or 
it may later be specified more precisely.


The comments of the old proposal about scope and internal handling still 
apply.

-- end of proposal --

PS: pixifier now can detect file types for PNG, BMP, JPEG, GIF and OPS 
from the data (without filename/extension), but i still think it's good 
to always provide at least the extension. It can help other appications 
, it allows for early out if the type is not supported and making the 
filename attribute mandatory is a good way to make a distinction between 
the new and the old type of image elements.

http://usf.corecodec.org