Re: tgtadm/tgtd management issue: buffer size truncated sometimes

On 24 Jan 2014, at 14:21, FUJITA Tomonori <[email protected]> wrote:

> On Fri, 24 Jan 2014 12:51:13 +0400
> Artyom Pervukhin <[email protected]> wrote:
> 
>> Can you please specify what linux distribution and libraries version
>> are you using?
> 
> Debian squeeze amd64.
> 
>> I¢m able to reproduce this bug on current tgt codebase (17222f2b00d29368da214e2d59cbc9e00f53606b) built on  Debian wheezy amd64. Bug frequency increases with number of targets:
>> 
>> 
>> 1000 targets (all loaded, as `tgtadm -C 0 --op show --mode target --tid 1000` shows target settings):
>> 
>> root@sandbox:~ # for i in $(seq 100) ; do tgtadm -C 0 --op show --mode target | grep -c ^Target ; done | sort | uniq -c                                                     
>>    100 978
>> root@sandbox:~ # for i in $(seq 100) ; do tgtadm -C 0 --op show --mode target | wc -c ; done | sort | uniq -c                                                              
>>    100 1033140
> 
> I can reproduce the same problem with 1000.
> 
> root@rose:~# for i in $(seq 100); do tgtadm --op show --mode target|grep -c ^Target;done|sort|uniq -c
>     97 1000
>      1 972
>      1 995
>      1 997
> 
> Strangely, looks like I can get the correct result without 'grep':
> 
> root@rose:~# for i in $(seq 100); do tgtadm --op show --mode target|python -c "import sys; print len([line for line in sys.stdin if \"Target\" in line])";done|sort|uniq -c
>    100 1000
> 
> 

I've done some additional stracing on the tgtd's side this time, please see below:

Testing 300 targets:

command:

root@sandbox:~ # tgtadm -C 0 --op show --mode target

(command shows full output)

tgtd writes:

write(10, "\0\0\0\0\305\325\4\0", 8)    = 8
write(10, "Target 1: iqn.2014-01:d1\n    Sys"..., 316861) = 224000
write(10, "       State: ready\n    I_T nexu"..., 92861) = 92861

command:

root@sandbox:~ # tgtadm -C 0 --op show --mode target | wc -c                                                                                                                
316860

tgtd writes stays the same as above.



Testing 900 targets:

root@sandbox:~ # tgtadm -C 0 --op show --mode target

(command shows incomplete output)

tgtd writes:

write(10, "\0\0\0\0u\205\16\0", 8)      = 8
write(10, "Target 1: iqn.2014-01:d1\n    Sys"..., 951661) = 256000
write(10, "a: No\n            Prevent remova"..., 695661) = 176000
write(10, " Removable media: No\n           "..., 519661) = 176000
write(10, "T nexus information:\n    LUN inf"..., 343661) = 176000
write(10, "  Backing store path: None\n     "..., 167661) = 167661

command:

root@sandbox:~ # tgtadm -C 0 --op show --mode target | wc -c                                                                                                                
929621

tgtd writes:

write(10, "\0\0\0\0u\205\16\0", 8)      = 8
write(10, "Target 1: iqn.2014-01:d1\n    Sys"..., 951661) = 944000
write(10, "t removal: No\n            Readon"..., 7661) = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---



Seeing this, I assumed the bug is on tgtadm's side.

I've written a small program which sends exactly the same command as tgtadm does to tgtd's socket, then reads response back, dumping it to stdout (sorry it's in Go, my C ain't good: http://play.golang.org/p/EcFTboCncF).

Dumping output like this I have no problem receiving full targets list:

root@sandbox:~ # for i in $(seq 1000) ; do ./dump | wc -c ; done | sort | uniq -c                                                                                          
   1000 951660
root@sandbox:~ # for i in $(seq 1000) ; do ./dump | grep -c ^Target ; done | sort | uniq -c                                                                                
   1000 900


Whereas tgtadm shows incomplete output:

root@sandbox:~ # for i in $(seq 1000) ; do tgtadm -C 0 --op show --mode target | wc -c ; done | sort | uniq -c                                                              
   1000 929621
root@sandbox:~ # for i in $(seq 1000) ; do tgtadm -C 0 --op show --mode target | grep -c ^Target ; done | sort | uniq -c                                                    
   1000 880