Implications of enlarging size of buffer in named pipe?

Bruno

I can enlarge the buffer used in the named pipe while using the "buffer"
command from the existing limit of 32M up to a maximum of
2048*internal_tape_block_size (64MB for 32K buffer, 128MB for 64K buffer),
or if I change to using "mbuffer" I can increase it to GBs.  A test run I
did with "buffer -b 2048 -p75 -B -s65536 -o /dev/st0l 2>>
/var/log/mondoarchive.log" streamed the tape quite nicely for a while, but
the tape was clearly emptying the buffer faster than I could fill it.

I have a concern however:  I don't quite understand how the code can recover
from a sig-pipe caused by end of tape when the buffer is full of data from
the last set of calls to fill it.

Consider:

DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2056: EXAT'g
set 0
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 0
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/0.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.0 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#555:
Too early to start deleting from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.0.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.0.gz' to tape (31 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '0.afio.' to tape (13824 KB)
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2056: EXAT'g
set 1
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 1
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/1.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.1 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.1.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.1.gz' to tape (26 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '1.afio.' to tape (14592 KB)
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2056: EXAT'g
set 2
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 2
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/2.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.2 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.2.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.2.gz' to tape (29 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '2.afio.' to tape (14336 KB)
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2056: EXAT'g
set 3
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 3
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/3.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.3 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.3.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.3.gz' to tape (16 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '3.afio.' to tape (15488 KB)
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2056: EXAT'g
set 4
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 4
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/4.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.4 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.4.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.4.gz' to tape (24 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '4.afio.' to tape (14976 KB)
 :
 :
DBG1: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2066:
Archiving set 8
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/8.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.8 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.8.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.8.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '8.afio.' to tape (57344 KB)
 :
 :
DBG4: [Main] libmondo-archive.c->archive_this_fileset#284: command = 'afio
-o -b 131072 -M 16m  /tmp/mondo.tmp.iEgZqv/tmpfs/11.afio. <
/tmp/mondo.tmp.iEgZqv/filelist.11 2>> /var/log/mondoarchive.log'
DBG2: [Main] libmondo-stream.c->maintain_collection_of_recent_archives#568:
Not far enough into tape to start deleting old archives from collection.
DBG2: [Main] libmondo-archive.c->make_afioballs_and_images_OLD#2077: Writing
EXAT files
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'xattr_list.11.gz' to tape (0 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file 'acl_list.11.gz' to tape (4 KB)
DBG2: [Main] libmondo-stream.c->write_file_to_stream_from_file#1681: Writing
file '11.afio.' to tape (20096 KB)

If the buffer is large enough that the tape only starts to be written during
" Writing file '4.afio.' to tape (14976 KB)".   We continue stuffing data
into the buffer, meanwhile the tape is filling and for the sake of argument
fills up part way through the data in the buffer that was put there by
"Writing file '8.afio.' to tape (57344 KB)".  However, we've been continuing
to stuff data into the buffer and are part way through " Writing file
'11.afio.' to tape (20096 KB)" when error cause by end of tape results in a
sigpipe.  How does the code manage to recover back to the correct point.

I may just have betrayed total ignorance of how fifos work!!!

Cheers
Dave




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel