bug#39610: R6RS `flush-output-port` not playing along with `transcoded-port`

Ludovic Courtès writes:

> Hi,
>
> Andreas Rottmann <[email protected]> skribis:
>
>>> Andreas Rottmann <[email protected]> skribis:
>>>
>>>> Andreas Rottmann <[email protected]> writes:
>>>>
>>>>> [...] I isolated the cause; the following snippet hangs on Guile 2.2
>>>>> and 3.0, while it worked as expected on 2.0:
>>>>>
>>>>> [...]
>>>>>
>>>>> It seems the underlying port is no longer flushed to the OS, so the
>>>>> `get-u8` now hangs waiting for input, starting with Guile 2.2.
>>>>>
>>>> [...]
>>>
>>> Actually I think the code above behaves as expected.  ‘pipe’ returns
>>> buffered ports by default.  When flushing the transcoded port,
>>> ‘transcoded_port_write’ is called, but then bytes written to the pipe
>>> are buffered.
>>>
>>> The fix is to add:
>>>
>>>   (setvbuf (cdr p) 'none)
>>>
>>> Does that make sense?
>>>
>> It makes sense, and I can confirm that it makes the boiled-down example
>> I posted work.
>>
>> However, I'm not sure it is the expected behavior. Regardless of
>> buffering modes used, I would expect a `flush-output-port` in a "port
>> stack" (as produced by `transcoded-output-port`) to propagate all the
>> way to the OS. It seems that was the case in Guile 2.0, as I'm pretty
>> sure I observed the "breaking" behavior change with 8399e7af5 applied,
>> and not with the commit preceding it.
>
> Port types don’t have a “flush” operation, only “write”.  Thus, it’s
> impossible to define a port type that would propagate flushes.
> There are pros and cons I guess, but it seems like a reasonable choice
> to me.
>
> When implementing a “proxying” port type like ‘transcoded-port’ that
> does its own buffering, it’s probably OK to say that it’s the proxy’s
> responsibility to ensure there’s no double-buffering taking place.
>
In my understanding, the "proxy type" is the R6RS transcoded port in
this case, which has no control over the underlying port, but provides a
flush operation (`flush-output-port`), which R6RS specifies as:

  Flushes any buffered output from the buffer of output-port to the
  underlying file, device, or object. The flush-output-port procedure
  returns unspecified values.

So if I obtain a transcoded port from a pipe port, and call
`flush-output-port` on the transcoded port, I'd expect the bytes to end
up at _least_ at the pipe port. That probably happens currently;
however, the thing is that R6RS also says about `transcoded-port`:

   As a side effect, however, transcoded-port closes binary-port in a
   special way that allows the new textual port to continue to use the
   byte source or sink represented by binary-port, even though
   binary-port itself is closed and cannot be used by the input and
   output operations described in this chapter.

So, I conclude: when you use `transcoded-port` with any underlying Guile
port, and you care about buffering behavior, you _need_ to set the
underlying port's buffer mode to 'none`, _before_ constructing the
transcoded port. I have not tried if Guile enforces the "specially
closed mode", but I am thinking in the context of an abstraction over
`pipe` and `primitive-fork`, intended to provide a basis for "pure" R6RS
code [1], so the client code cannot just call Guile's `setvbuf`, without
losing portability.

[1] https://github.com/rotty/spells/blob/master/spells/process/compat.guile.sls

Do you agree with that conclusion?

  [ After writing the above, I realize that might be what you meant, after
    all: do you propose that, as a fix, `transcoded-port` sets the buffer
    mode of the underlying port to `none`? ]

I must admit that I am confused -- how can a port type have no flush
operation (which is evidently true from looking at scm_t_port_type), and
Guile's `force-output` procedure, generic over port types, still exist?

>> If the current behavior is indeed the intended one, we should make sure
>> the docs somehow reflect this caveat, as I imagine it may surprise
>> future Guile users which happen to use its R6RS support and pipes in
>> combination.
>
> Maybe we should not document this specific combination but rather the
> more general issue?  I’m not sure how to do that.  Thoughts?
>
I now realized I need to wrap my head around the changed port
implementation, and how it relates to R6RS in general, and
`transcoded-port` specifically, before starting to think about
documentation.

I think I have identified another related issue, this time for reading
from `trancoded-port`, but I'd rather make sure we have a shared
understanding of the expected flushing and buffering behavior, before
bringing that up.

Regards, Rotty