Re: Writing Deep tiled images: help !

Thanks for your very complete answer.

I think I'll go for the second option (one framebuffer), with a single 
thread writing data, and the RANDOM_Y flag set. (we start to render 
tiles for the center of the image, so they will be random)


Michel


On 04/11/2014 02:10 AM, Peter Hillman wrote:
> RANDOM_Y is not supported for scanline images - you either start at 
> datawindow.min.y and work to datawindow.max.y or vice versa. You can 
> deliver scanlines out of order: they'll be buffered before writing.
>
> With tiled images, you can write tiles in any order, but you must set 
> RANDOM_Y to allow it write without buffering.
>
>
>
> On 11/04/14 12:02, Larry Gritz wrote:
>> Uh oh.
>>
>> I knew about RANDOM_Y for scanline images. Are you saying that for 
>> tiled images, if we don't set lineorder to RANDOM_Y, then there's 
>> internal buffering if we send the tiles in anything but canonical order?
>>
>>
>>
>> On Apr 10, 2014, at 4:54 PM, Peter Hillman <[email protected]> wrote:
>>
>>> In general with OpenEXR, you cannot write to a file from different 
>>> threads unless they have a shared framebuffer. Calling 
>>> "setFrameBuffer" from one thread while another is writing may cause 
>>> a crash.
>>>
>>> Even if you do have a shared framebuffer, the threads lock to 
>>> serialise the writes: only one "writePixels" or "writeTile" request 
>>> goes through at once.
>>> If you don't use the IlmThread library to run your threads it may be 
>>> wise to implement your own lock mechanism around writePixel and 
>>> writeTile calls.
>>>
>>>
>>> There are two ways of writing tiled data: if you use 
>>> xTileCoords/yTileCoords, then pixel(0,0) is the bottom left corner 
>>> of each tile, implying the image buffer pointed to by the slice is 
>>> the same size as the tile.
>>> if you don't do that, then pixel(0,0) is the bottom left corner of 
>>> the displayWindow, implying the image buffer pointed to be the slice 
>>> is the size of the dataWindow, accounting for offset data windows 
>>> appropriately.
>>>
>>> In your case, it would seem like you want the second option: 
>>> preallocate your dataWindow sized array of pointers-to-samples, and 
>>> call setFrameBuffer before any rendering begins.
>>> Then each thread sets the pointers to the samples for the tile it is 
>>> going to write and fills in the corresponding part of the 
>>> samplecount array before calling writeTile.
>>>
>>> The pointers need not be valid when setFrameBuffer is called. When 
>>> you call writeTile, only the pointers and sampleCount data for the 
>>> pixels being written need to be valid, and need not remain valid 
>>> afterwards. So yes, you can delete data for the actual sample data 
>>> of a tile after calling writeTile, though you'll need to keep the 
>>> sampleCount array and the pointers-to-samples array for each channel.
>>>
>>> Note that if you are writing tiles in anything other than RANDOM_Y 
>>> order, then whenever your threads deliver tiles out of order OpenEXR 
>>> will have to buffer data so it can write the tiles sequentially.
>>> I suspect this may well be the case from multithreaded write 
>>> applications. With deep data this could take a significant amount of 
>>> memory.
>>> You may wish to enforce RANDOM_Y order to allow the library to write 
>>> tiles to disk as soon as you call writeTile. If your read pattern is 
>>> generally also random, rather than in 'scanline order', there'll be 
>>> little performance hit.
>>>
>>> You might also consider having one worker thread managing the 
>>> writing of tile data and marshalling into the format required for 
>>> the DeepSlices, and have other threads compute data and signal the 
>>> worker thread when they have tiles ready for processing.
>>>
>>> Yes, you can use an Array2D<float*> for your data points (dataZ and 
>>> dataA in the example). Each value is a pointer to the first sample 
>>> of the channel data for each pixel.
>>> Subsequent samples are found using the offset provided. That means 
>>> you can store pixel data in separate or interleaved channels.
>>>
>>>
>>>
>>> On 11/04/14 02:16, Michel Lerenard wrote:
>>>> Hi,
>>>>
>>>> I'm trying to write deep data into an EXR file and am having 
>>>> trouble understanding how the API works.
>>>> I'd like to be able to write tiles from different threads. The 
>>>> application I'm working on (Isotropix Clarisse) runs several 
>>>> threads, each one render a part of an image.
>>>> I'd like to write deep data when the current tile is completely 
>>>> computed.
>>>>
>>>> I'm using the "Reading And Writing Image Files" documentation, and 
>>>> the section about Deep Tile File writing on page 26 is confusing to 
>>>> me.
>>>>
>>>>  From what I understand, I need a DeepFrameBuffer to write an 
>>>> image. It seems it should be the same for all threads, as a 
>>>> DeepTiledOutputFile can only use one. Is it thread safe ? Can I use 
>>>> the same FrameBuffer from all threads ?
>>>> I was puzzled by the type of dataZ and data0, reading the next 
>>>> example (reading a deep tiled file) I understand that the arrays 
>>>> are only storing pointers to deep data for a particular pixel, and 
>>>> that they could as well be Array2D<float*>. Is that correct ?
>>>>
>>>> If it is, this means that each one of my threads should set 
>>>> pointers in the framebuffer DeepSlice for each pixel of the tile 
>>>> it's computing, then call writeTile. After that can I delete the 
>>>> data and reset the pointers in the DeepSlice ?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Michel
>>>>
>>>>
>>>> _______________________________________________
>>>> Openexr-devel mailing list
>>>> [email protected]
>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>
>>> _______________________________________________
>>> Openexr-devel mailing list
>>> [email protected]
>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>> -- 
>> Larry Gritz
>> [email protected]
>>
>>
>>
>
>
> _______________________________________________
> Openexr-devel mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>