Re: stalling

Thanks for your response. I ran truss on the two star processes that
got created from one star invocation:

/var/tmp# truss -p 21402
write(1, 0x0814304C, 5120)      (sleeping...)

/var/tmp# truss -p 21403
read(5, 0x08046CFB, 1)          (sleeping...)

Here is the pstack output from 21402:

 fedec317 write    (1, 814304c, 1400)
 fedcc794 _flsbuf  (20, 80b6398, 57, 35303030) + cc
 0808a14e filewrite (80b6398, 803c688, 68, 803c680) + 87
 08089f4b _bflush  (803c680, 803c680, 80991fc, 803c7c0) + 33
 0808a059 js_fprintf (80b6398, 80991fc, 808fa70, 803d690, 2ff90, 0) + 5b
 08066078 vprint   (803daa0, 803daa0) + 246
 08069c86 createi  (803d690, 803d690, 47, 803daa0, 803d680, 803daa0) + 645
 0806b69a put_dir  (803ea50, 803ea50, 3c, 803ee60, fe5e5c00, 803ea40) + 6bc
 08069868 createi  (803ea50, 803ea50, 3b, 803ee60, 803ea40, 803ee60) + 227
 0806b69a put_dir  (803fe10, 803fe10, 36, 8040220, feae9c00, 803fe00) + 6bc
 08069868 createi  (803fe10, 803fe10, 35, 8040220, 803fe00, 8040220) + 227
 0806b69a put_dir  (80411d0, 80411d0, 25, 80415e0, feae8600, 80411c0) + 6bc
 08069868 createi  (80411d0, 80411d0, 24, 80415e0, 80411c0, 80415e0) + 227
 0806b69a put_dir  (8042590, 8042590, 1d, 80429a0, fe896e00, 8042580) + 6bc
 08069868 createi  (8042590, 8042590, 1c, 80429a0, 8042580, 80429a0) + 227
 0806b69a put_dir  (8043950, 8043950, 14, 8043d60, fe86f200, 8043940) + 6bc
 08069868 createi  (8043950, 8043950, 13, 8043d60, 8043940, 8043d60) + 227
 0806b69a put_dir  (8044d10, 8044d10, 10, 8045120, fe86ec00, 8044d00) + 6bc
 08069868 createi  (8044d10, 8044d10, f, 8045120, 8044d00, 8045120) + 227
 0806b69a put_dir  (80460d2, 80460d2, 8, 80464e0, fe86e600, 80460c0) + 6bc
 08069868 createi  (80460d2, 80460d2, 7, 80464e0, 80460c0, 80464e0) + 227
 0806b69a put_dir  (80471da, 80471da, 2, 8046c50, fe441600, 0) + 6bc
 08069868 createi  (80471da, 80471da, 1, 8046c50, 0, 8046c50) + 227
 08069636 create   (80471da, 0, 1, 8091373) + dc
 0805ae4c star_create (1, 8046e60, 8046dbc, 805a76f) + 4e9
 0805a7a7 main     (20, 8046de4, 8046e68) + 48f
 08057a24 _start   (20, 8046fbc, 8046fdd, 8046fe0, 8046fe3, 804700f) + 80

and from 21403:

 fedebe37 read     (5, 8046cfb, 1)
 0807dcdd swait    (5, 0, 1, 8046d2c, 8046d40, 0) + 25
 0807df8c fifo_owait (40000, 1800, 19ff, 807dad7, 0, 8046da0) + 49
 0807e40e do_out   (0, 8046da0, 8046dbc, 805a60c, 1f, 8046de8) + 15
 0807db68 runfifo  (1f, 8046de8) + f2
 0805a60c main     (20, 8046de4, 8046e68) + 2f4
 08057a24 _start   (20, 8046fbc, 8046fdd, 8046fe0, 8046fe3, 804700f) + 80

I'll see if I can get the size of the total dumped data for you. I'm
also going to add some -v's to star in case its reliably
stalling/sleeping at the same point.

Ryan


On Tue, Apr 9, 2013 at 2:03 AM, Joerg Schilling
<[email protected]> wrote:
> Ryan Lovett <[email protected]> wrote:
>
>> With star 1.5.2 on the latest Solaris 10, we are seeing star appear to
>> stall in the middle of dumping a ZFS snapshot to tape. Attaching dtruss,
>> iosnoop, and opensnoop to the affected process shows no activity. We
>> invoke star with:
>>
>>   star -C /path/to/.zfs/snapshot/20130407 -c -S -acl artype=3Dexustar \
>>       -sparse -time -xdev \
>>       -multivol -wtardumps -new-volume-script=3D/usr/local/sbin/mtx-next=
 \
>>       errctl=3D/usr/local/etc/star-errctl.conf \
>>       -not pat=3D*/.Trash/* pat=3D*/.cache/* pat=3D*/.thumbnails/* pat=
=3D*/.nfs* \
>>       level=3D0 fs-name=3D/path/to blocks=3D512 file=3D/dev/rmt/1cn .
>
> First a warning: If you exclude parts of the filesystem from a dump, star=
 will
> not be able to "restore" files that have been brought back into the "visi=
ble"
> parts of the dump by renaming a a directory that was in the excluded part.
>
> This is because such an action does not set the "ctime" stamp on such fil=
es.
>
>> I don't believe it is a problem with the tape device since we can kill t=
he
>> hanging star and run another (on another filesystem) right after which
>> runs to completion.
>>
>> Does anything above suggest why the star process stalls or waits?
>>
>> How can one debug this further?
>
> Your current statements do not include any information that could help.
>
> The minimal information would be to run truss on both star processes in o=
rder
> to get the current state of the processes.
>
> I assume that star is not looping, so this should show a syscall...
>
> Also helpful would be a "pstack" outoput from both processes.
>
> How much data did star already dump before stalling?
>
> J=F6rg
>
> --
>  EMail:[email protected] (home) J=F6rg Schilling D-13353 =
Berlin
>        [email protected]                (uni)
>        [email protected] (work) Blog: http://schily.blo=
gspot.com/
>  URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily