Re: Writing Deep tiled images: help !

 

It appears that support for buffering tiles to handle out-of-order
calls to writeTiles was present in the first version of OpenEXR that
supported tiles (OpenEXR-1.1.0) 

I realise my last reply about
scanlines was incorrect due to some poor editing skills on my part:


Scanline images have no buffering between calls to writePixels;
there's no way of delivering scanlines in any order since you don't pass
writePixels a y offset, merely a number of scanlines to write. Calling
writePixels(1) will always write currentScanline() 

The buffering that
happens is with tiled images in INCREASING_Y or DECREASING_Y, and then
only when you specify tiles out of order. 

Also worth pointing out that
OpenEXR buffers the compressed data, not the raw pixel data. If you are
creating tiles out of order and want INCREASING_Y or DECREASING_Y tile
order, it will be more memory efficient to pass the tiles immediately to
the library and have it manage buffering, rather than buffering them
yourself until you have them ready in the correct order. It may also be
appropriate to write a temporary file in RANDOM_Y order, and then read
it back in and write it out in INCREASING_Y order 

If the image is
small enough to fit in memory and you have multithreading enabled it is
best to write the entire image in one call to writeTiles or writePixels.


On 2014-04-12 06:35, Piotr Stanczyk wrote: 

> I am not sure about
that ... let me check 
> 
> Piotr 
> 
> On 11 April 2014 10:19, Larry
Gritz <[email protected]> wrote:
> 
>> Was it always like this, in 1.x
as well? Some of my code seems to bend over backwards to not allow
random order writes unless a specific request was made for random Y
order. I don't remember now why I wrote it that way, but the code is
structured as if at some time in the past, the library would not have
done the buffering without RANDOM_Y lineOrder. Was I just always wrong
about this?
>> 
>> -- lg
>> 
>> On Apr 11, 2014, at 10:13 AM, Larry
Gritz <[email protected]> wrote:
>> 
>> > OK, so let me restate it to
make sure I have it right:
>> >
>> > For scanline files, storage in the
file is either increasing or decreasing (determined by lineOrder);
libIlmImf will accept scanlines in any order, but if they are sent other
than with the line order you specified, the library will buffer
internally.
>> >
>> > For tiled files, libIlmImf will accept tiles in
any order, buffering if they are not in canonical order, UNLESS you set
lineOrder to RANDOM_Y, in which case it really will store it in random
order rather than buffering (but at the cost of reading that file in
canonical order having more seeks than if it had been written in
canonical tile order).
>> >
>> > Do I have that all correct?
>> >
>>
>
>> >
>> > On Apr 10, 2014, at 5:10 PM, Peter Hillman
<[email protected]> wrote:
>> >
>> >> RANDOM_Y is not supported for
scanline images - you either start at datawindow.min.y and work to
datawindow.max.y or vice versa. You can deliver scanlines out of order:
they'll be buffered before writing.
>> >>
>> >> With tiled images, you
can write tiles in any order, but you must set RANDOM_Y to allow it
write without buffering.
>> >>
>> >>
>> >>
>> >> On 11/04/14 12:02,
Larry Gritz wrote:
>> >>> Uh oh.
>> >>>
>> >>> I knew about RANDOM_Y for
scanline images. Are you saying that for tiled images, if we don't set
lineorder to RANDOM_Y, then there's internal buffering if we send the
tiles in anything but canonical order?
>> >>>
>> >>>
>> >>>
>> >>> On
Apr 10, 2014, at 4:54 PM, Peter Hillman <[email protected]> wrote:
>>
>>>
>> >>>> In general with OpenEXR, you cannot write to a file from
different threads unless they have a shared framebuffer. Calling
"setFrameBuffer" from one thread while another is writing may cause a
crash.
>> >>>>
>> >>>> Even if you do have a shared framebuffer, the
threads lock to serialise the writes: only one "writePixels" or
"writeTile" request goes through at once.
>> >>>> If you don't use the
IlmThread library to run your threads it may be wise to implement your
own lock mechanism around writePixel and writeTile calls.
>> >>>>
>>
>>>>
>> >>>> There are two ways of writing tiled data: if you use
xTileCoords/yTileCoords, then pixel(0,0) is the bottom left corner of
each tile, implying the image buffer pointed to by the slice is the same
size as the tile.
>> >>>> if you don't do that, then pixel(0,0) is the
bottom left corner of the displayWindow, implying the image buffer
pointed to be the slice is the size of the dataWindow, accounting for
offset data windows appropriately.
>> >>>>
>> >>>> In your case, it
would seem like you want the second option: preallocate your dataWindow
sized array of pointers-to-samples, and call setFrameBuffer before any
rendering begins.
>> >>>> Then each thread sets the pointers to the
samples for the tile it is going to write and fills in the corresponding
part of the samplecount array before calling writeTile.
>> >>>>
>> >>>>
The pointers need not be valid when setFrameBuffer is called. When you
call writeTile, only the pointers and sampleCount data for the pixels
being written need to be valid, and need not remain valid afterwards. So
yes, you can delete data for the actual sample data of a tile after
calling writeTile, though you'll need to keep the sampleCount array and
the pointers-to-samples array for each channel.
>> >>>>
>> >>>> Note
that if you are writing tiles in anything other than RANDOM_Y order,
then whenever your threads deliver tiles out of order OpenEXR will have
to buffer data so it can write the tiles sequentially.
>> >>>> I suspect
this may well be the case from multithreaded write applications. With
deep data this could take a significant amount of memory.
>> >>>> You
may wish to enforce RANDOM_Y order to allow the library to write tiles
to disk as soon as you call writeTile. If your read pattern is generally
also random, rather than in 'scanline order', there'll be little
performance hit.
>> >>>>
>> >>>> You might also consider having one
worker thread managing the writing of tile data and marshalling into the
format required for the DeepSlices, and have other threads compute data
and signal the worker thread when they have tiles ready for
processing.
>> >>>>
>> >>>> Yes, you can use an Array2D<float*> for your
data points (dataZ and dataA in the example). Each value is a pointer to
the first sample of the channel data for each pixel.
>> >>>> Subsequent
samples are found using the offset provided. That means you can store
pixel data in separate or interleaved channels.
>> >>>>
>> >>>>
>>
>>>>
>> >>>> On 11/04/14 02:16, Michel Lerenard wrote:
>> >>>>> Hi,
>>
>>>>>
>> >>>>> I'm trying to write deep data into an EXR file and am
having trouble understanding how the API works.
>> >>>>> I'd like to be
able to write tiles from different threads. The application I'm working
on (Isotropix Clarisse) runs several threads, each one render a part of
an image.
>> >>>>> I'd like to write deep data when the current tile is
completely computed.
>> >>>>>
>> >>>>> I'm using the "Reading And
Writing Image Files" documentation, and the section about Deep Tile File
writing on page 26 is confusing to me.
>> >>>>>
>> >>>>> From what I
understand, I need a DeepFrameBuffer to write an image. It seems it
should be the same for all threads, as a DeepTiledOutputFile can only
use one. Is it thread safe ? Can I use the same FrameBuffer from all
threads ?
>> >>>>> I was puzzled by the type of dataZ and data0, reading
the next example (reading a deep tiled file) I understand that the
arrays are only storing pointers to deep data for a particular pixel,
and that they could as well be Array2D<float*>. Is that correct ?
>>
>>>>>
>> >>>>> If it is, this means that each one of my threads should
set pointers in the framebuffer DeepSlice for each pixel of the tile
it's computing, then call writeTile. After that can I delete the data
and reset the pointers in the DeepSlice ?
>> >>>>>
>> >>>>>
>> >>>>>
Thanks,
>> >>>>>
>> >>>>> Michel
>> >>>>>
>> >>>>>
>> >>>>>
_______________________________________________
>> >>>>> Openexr-devel
mailing list
>> >>>>> [email protected]
>> >>>>>
https://lists.nongnu.org/mailman/listinfo/openexr-devel [1]
>> >>>>
>>
>>>> _______________________________________________
>> >>>>
Openexr-devel mailing list
>> >>>> [email protected]
>> >>>>
https://lists.nongnu.org/mailman/listinfo/openexr-devel [1]
>> >>> --
>>
>>> Larry Gritz
>> >>> [email protected]
>> >>>
>> >>>
>> >>>
>> >>
>>
>
>> > --
>> > Larry Gritz
>> > [email protected]
>> >
>> >
>> >
>> >
>>
> _______________________________________________
>> > Openexr-devel
mailing list
>> > [email protected]
>> >
https://lists.nongnu.org/mailman/listinfo/openexr-devel [1]
>> 
>> --
>>
Larry Gritz
>> [email protected]
>> 
>>
_______________________________________________
>> Openexr-devel mailing
list
>> [email protected]
>>
https://lists.nongnu.org/mailman/listinfo/openexr-devel [1]




Links:
------
[1]
https://lists.nongnu.org/mailman/listinfo/openexr-devel

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel