Re: tile flushing with compression?

IO streams probably aren't flushed on process death:

http://www.gnu.org/software/libc/manual/html_node/Termination-Internals.html
"Note that streams are not flushed automatically when the process
terminates; see I/O on Streams
<http://www.gnu.org/software/libc/manual/html_node/I_002fO-on-Streams.html#I_002fO-on-Streams>
."

 If you replace Imf::StdOFStream with something that uses file descriptors
directly (read(), write(), etc), does that help?

Karl

On Wednesday, February 25, 2015, Larry Gritz <[email protected]> wrote:

> We're generating a whole bunch of exr's simultaneously (each may have ~3
> channels). At the point that the render is killed, the various files each
> are readable, but disagree on which buckets have been rendered.
>
> In fact, a mostly-black AOV may have NO buckets flushed at that point. So
> if we simply assume that we don't need to rerender any buckets that appear
> in all the output files, that often makes it look like we have to rerender
> the whole thing.
>
>
> On Feb 25, 2015, at 4:32 PM, Piotr Stanczyk <[email protected]
> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>
> Just so that I follow, you're generating a multi-channel single file exr
> for these and have no reliable way to checkpoint your renders?
>
> On 25 February 2015 at 16:27, Larry Gritz <[email protected]
> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>
>> Well, if a render job gets killed, we end up with a partial file. We can
>> then "resume" by starting a new render which takes an inventory of which
>> tiles ended up in the partially-written file, and only rendering the
>> buckets that didn't make it the first time.
>>
>> The problem is that we may be rendering several (and sometimes very,
>> very, way too many) outputs, i.e., lots of AOVs. So some output images may
>> be black in certain tiles where other images are not black in the same
>> tile. And so at the time when the render is killed, those highly compressed
>> empty tiles in one output image may not have been flushed, whereas they
>> were in other output images. The icky situation this leads to is that we
>> have a set of output files that don't seem to agree on which set of tiles
>> have been rendered and which have not. If we could ensure that all of the
>> outputs flushed after we send a bucket (as the "full" tiles already seem
>> to, but the "empty" tiles do not), it would make things a lot easier for us.
>>
>> -- lg
>>
>>
>> On Feb 25, 2015, at 4:18 PM, Piotr Stanczyk <[email protected]
>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>
>> Hmm - interesting.
>>
>> How exactly are you observing your current behaviour?
>>
>> On 25 February 2015 at 16:12, Larry Gritz <[email protected]
>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>
>>> When writing tiled OpenEXR files, we've noticed that if we use zip
>>> compression, the zip data is not fully "flushed" when each tile is sent
>>> (particularly noticeable with small post-zipped tiles, such as empty
>>> buckets in a render). With no compression, it seems, the tiles really do
>>> write to the file as we send them.
>>>
>>> I would be eager to hear any advice to how to ensure a full flush of the
>>> zip stream upon each tile. A call I've overlooked would be ideal, but I'd
>>> also be interested in suggestions for where to patch the IlmImf source code
>>> (or, alternately, any arguments for why it would not be wise to do so).
>>>
>>>         -- lg
>>>
>>> --
>>> Larry Gritz
>>> [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Openexr-devel mailing list
>>> [email protected]
>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>
>>
>>
>> --
>> Larry Gritz
>> [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>
>>
>>
>>
>
> --
> Larry Gritz
> [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>
>
>
>
>

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel