dv file checker

I am looking for a dv diagnostic and maybe even repair app.

Every so often a .dv file gets screwed up in such a way that the playback is the 
same looking bunch of random colored blocks, and with some work looking at 
hexdump of the first few 100 bytes I have been able to repair them.

Now I have one that plays for 6min ok, then the colors start.  so thats 
somewhere around 10000 frames.  I don't really know how to zero in on where the 
corruption is.

I just wrote some code (below) that checks the first 80 bytes of every 12k, and 
it found a problem about where I expected.  from what I can tell, 672 bytes 
evaporated.

I am pretty fuzzy on how to fix it.  I was hoping I could just cut the file into 
2 parts where the 2nd part started on the first good frame boundary after the 
missing bytes, so I did this:
1440240000 - 672 = 1440239328

juser@dell30:~/sdb1/temp$ dd if=java.dv of=/home/juser/java-5b.dv bs=1 
skip=1440239328

so it kinda works...
mplayer java-5b.dv plays the video at about 1/2 speed, says "no audio."
ffmpeg2theora java-5b.dv -f dv -V 500 -A 48 -o java-5b.ogg says:
Input #0, dv, from 'java-5b.dv':
   Duration: 00:06:40.46, start: 0.000000, bitrate: 28771 kb/s
     Stream #0.0: Video: dvvideo, yuv411p, 720x480, 28771 kb/s, PAR 10:11 DAR 
15:11, 29.97 tbr, 29.97 tbn, 29.97 tbc
     Stream #0.1: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
   Resize: 720x480
       0:00:06.29 audio: 49kbps video: 557kbps, time remaining: 02:48:50

Then mplayer java-5b.ogg plays both audio and video at the right speed.

So it looks like this file is fixed, so I don't need anything right now, but it 
seems my tool chain has room for improvement, and I am hoping someone has 
already done it.   so... anything like this exist?


#!/usr/bin/python

# ckdv.py  Dump dv headers

HEAD_SIZE=3+80 # *3

def hex_dump(str):
     return ' '.join(['%02x'% ord(c) for c in str])

def binary(x):
     b=''
     for i in range(8):
         if x & 2**i: b='1'+b
         else: b='0'+b
     return b


f = open('java.dv','rb')


good_header="""1f 07 00 3f 68 78 78 78 ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff 3f 07 00 80 00 ff ff ff ff ff ff f0 01 ff ff ff ff ff ff f0 02 ff ff 
ff ff ff ff f0 03 ff ff ff ff ff ff f0 04 ff ff ff ff ff ff f0 05 ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff 3f 07 01 80 06 ff ff ff ff ff ff f0 07 ff ff ff ff ff ff f0 08 ff ff ff 
ff ff ff f0 09 ff ff ff ff ff ff f0 0a ff ff ff ff ff ff 80 0b ff ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff 56 07 00 70 c9 f6 ff 7f 7f ff ff 83 80 62 ff 21 04 09 63 ff 35 43 18 ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff ff 62 ff 21 04 09 63 ff 35 43 18 ff ff ff ff ff ff ff ff ff ff ff 
ff 56 07 01 ff ff ff ff ff ff ff ff ff ff 62 ff 21 04 09 63 ff 35 43 18 ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff 62 ff 21 04 09 63 ff 35 43 18 ff ff ff ff ff ff ff ff ff ff ff ff 
56 07 02 ff ff ff ff ff ff ff ff ff ff 62 ff 21 04 09 63 ff 35 43 18 ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 60 ff ff 00 ff 61 
0f c8 fc ff 62 ff 21 04 09 63 ff 35 43 18 ff ff ff ff ff ff ff ff ff ff ff ff 76 
07 00"""

gh=''.join([chr(int(h,16)) for h in good_header.split()])[:HEAD_SIZE]
head_size = len(gh)

frame_no=0
errs=0
while True:
     frame = f.read(12000)
     if not frame: break

     head = frame[:head_size]

     # construct what the current header should be
     ghx=gh

     # update top nibble of byte 1 and 81
     top_nibble = frame_no%10
     bottom_nibble = ord(gh[1])%16
     byte = top_nibble*16+bottom_nibble
     c=chr(byte)
     # stick it in bytes 1,81,161,241
     for p in [1,81,]:
         # ghx[p]=c
         ghx = ghx[:p]+c+ghx[p+1:]

     # byte 4: flip bit 4 every 10
     b=ord(gh[4])
     if frame_no/10%2 and frame_no<1370:
         b=b|(1 << 4)
     else:
	b=b & ~(1 << 4)
     c=chr(b)
     # i give up... fake it.
     c=head[4]
     ghx = ghx[:4]+c+ghx[4+1:]

     # set/clear top bit of a bunch of stuffs.
     ## this not working, so """commented out"""
     """
     for p in [83,91,99,107,115,123,163,171,179,187,195,203]:
         byte=ord(ghx[p])
         if frame_no/5%2:
             byte=byte&127 # clear bit 8
         else:
             byte=byte|128 # set bit 8
         c=chr(byte)
         ghx = ghx[:p]+c+ghx[p+1:]
     """

     # test current header against what it should be
     if head <> ghx:
         print "frame # %s, byte %s" % (frame_no, f.tell())
         print "expected:"	
         print hex_dump(ghx)
         print "found in file:"
         print hex_dump(head)

         z=zip(ghx,head)
         for i,t in enumerate(z):
             if t[0]!=t[1]:
                 print i,
                 print t, binary(ord(t[0])), binary(ord(t[1]))
         errs+=1
         print

     frame_no+=1
     if frame_no % 18000 == 0:
         print '%d frames done (about %d minutes)' % (frame_no, frame_no / 18000)
     if errs==1: break


juser@dell30:~/sdb1/temp$ python ckdv.py
18000 frames done (about 1 minutes)
36000 frames done (about 2 minutes)
54000 frames done (about 3 minutes)
72000 frames done (about 4 minutes)
90000 frames done (about 5 minutes)
108000 frames done (about 6 minutes)
frame # 120019, byte 1440240000
expected:
1f 97 00 3f 25 78 78 78 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 3f 
97 00
found in file:
4c 2f 43 d2 25 1c 1c af 77 06 87 e8 b6 1d 44 2a a0 6d cc b4 12 6d 00 90 1c f1 5a 
3b 01 91 c9 69 d7 d6 d7 7b f4 30 fc b3 81 8d ae 96 ce be be 0d 9b 07 02 03 ac 11 
87 89 e0 a3 0b de c9 b5 d5 e1 08 03 ac 9d 0f c9 80 c6 ae b9 a8 d6 6e 3e 8d 49 ac 
93 47
0 ('\x1f', 'L') 00011111 01001100
1 ('\x97', '/') 10010111 00101111
2 ('\x00', 'C') 00000000 01000011
3 ('?', '\xd2') 00111111 11010010
5 ('x', '\x1c') 01111000 00011100
6 ('x', '\x1c') 01111000 00011100
7 ('x', '\xaf') 01111000 10101111
8 ('\xff', 'w') 11111111 01110111
9 ('\xff', '\x06') 11111111 00000110
10 ('\xff', '\x87') 11111111 10000111
11 ('\xff', '\xe8') 11111111 11101000
12 ('\xff', '\xb6') 11111111 10110110
13 ('\xff', '\x1d') 11111111 00011101
14 ('\xff', 'D') 11111111 01000100
15 ('\xff', '*') 11111111 00101010
16 ('\xff', '\xa0') 11111111 10100000
17 ('\xff', 'm') 11111111 01101101
18 ('\xff', '\xcc') 11111111 11001100
19 ('\xff', '\xb4') 11111111 10110100
20 ('\xff', '\x12') 11111111 00010010
21 ('\xff', 'm') 11111111 01101101
22 ('\xff', '\x00') 11111111 00000000
23 ('\xff', '\x90') 11111111 10010000
24 ('\xff', '\x1c') 11111111 00011100
25 ('\xff', '\xf1') 11111111 11110001
26 ('\xff', 'Z') 11111111 01011010
27 ('\xff', ';') 11111111 00111011
28 ('\xff', '\x01') 11111111 00000001
29 ('\xff', '\x91') 11111111 10010001
30 ('\xff', '\xc9') 11111111 11001001
31 ('\xff', 'i') 11111111 01101001
32 ('\xff', '\xd7') 11111111 11010111
33 ('\xff', '\xd6') 11111111 11010110
34 ('\xff', '\xd7') 11111111 11010111
35 ('\xff', '{') 11111111 01111011
36 ('\xff', '\xf4') 11111111 11110100
37 ('\xff', '0') 11111111 00110000
38 ('\xff', '\xfc') 11111111 11111100
39 ('\xff', '\xb3') 11111111 10110011
40 ('\xff', '\x81') 11111111 10000001
41 ('\xff', '\x8d') 11111111 10001101
42 ('\xff', '\xae') 11111111 10101110
43 ('\xff', '\x96') 11111111 10010110
44 ('\xff', '\xce') 11111111 11001110
45 ('\xff', '\xbe') 11111111 10111110
46 ('\xff', '\xbe') 11111111 10111110
47 ('\xff', '\r') 11111111 00001101
48 ('\xff', '\x9b') 11111111 10011011
49 ('\xff', '\x07') 11111111 00000111
50 ('\xff', '\x02') 11111111 00000010
51 ('\xff', '\x03') 11111111 00000011
52 ('\xff', '\xac') 11111111 10101100
53 ('\xff', '\x11') 11111111 00010001
54 ('\xff', '\x87') 11111111 10000111
55 ('\xff', '\x89') 11111111 10001001
56 ('\xff', '\xe0') 11111111 11100000
57 ('\xff', '\xa3') 11111111 10100011
58 ('\xff', '\x0b') 11111111 00001011
59 ('\xff', '\xde') 11111111 11011110
60 ('\xff', '\xc9') 11111111 11001001
61 ('\xff', '\xb5') 11111111 10110101
62 ('\xff', '\xd5') 11111111 11010101
63 ('\xff', '\xe1') 11111111 11100001
64 ('\xff', '\x08') 11111111 00001000
65 ('\xff', '\x03') 11111111 00000011
66 ('\xff', '\xac') 11111111 10101100
67 ('\xff', '\x9d') 11111111 10011101
68 ('\xff', '\x0f') 11111111 00001111
69 ('\xff', '\xc9') 11111111 11001001
70 ('\xff', '\x80') 11111111 10000000
71 ('\xff', '\xc6') 11111111 11000110
72 ('\xff', '\xae') 11111111 10101110
73 ('\xff', '\xb9') 11111111 10111001
74 ('\xff', '\xa8') 11111111 10101000
75 ('\xff', '\xd6') 11111111 11010110
76 ('\xff', 'n') 11111111 01101110
77 ('\xff', '>') 11111111 00111110
78 ('\xff', '\x8d') 11111111 10001101
79 ('\xff', 'I') 11111111 01001001
80 ('?', '\xac') 00111111 10101100
81 ('\x97', '\x93') 10010111 10010011
82 ('\x00', 'G') 00000000 01000111



juser@dell30:~/sdb1/temp$ dd if=java.dv of=badframes.dv bs=12000 skip=120015 
count=20
20+0 records in
20+0 records out
240000 bytes (240 kB) copied, 0.00472596 s, 50.8 MB/s
juser@dell30:~/sdb1/temp$ hexdump badframes.dv |grep "7878 ffff"
0000000 571f 3f00 7878 7878 ffff ffff ffff ffff
0002ee0 671f 3f00 7878 7878 ffff ffff ffff ffff
0005dc0 771f 3f00 7878 7878 ffff ffff ffff ffff
0008ca0 871f 3f00 7878 7878 ffff ffff ffff ffff
000b8e0 071f 3f00 7808 7878 ffff ffff ffff ffff
000e7c0 171f 3f00 7808 7878 ffff ffff ffff ffff
00116a0 271f 3f00 7808 7878 ffff ffff ffff ffff
0014580 371f 3f00 7808 7878 ffff ffff ffff ffff
0017460 471f 3f00 7808 7878 ffff ffff ffff ffff
001a340 571f 3f00 7808 7878 ffff ffff ffff ffff
001d220 671f 3f00 7808 7878 ffff ffff ffff ffff
0020100 771f 3f00 7808 7878 ffff ffff ffff ffff
0022fe0 871f 3f00 7808 7878 ffff ffff ffff ffff
0025ec0 971f 3f00 7808 7878 ffff ffff ffff ffff
0028da0 071f 3f00 7808 7878 ffff ffff ffff ffff
002bc80 171f 3f00 7808 7878 ffff ffff ffff ffff
002eb60 271f 3f00 7808 7878 ffff ffff ffff ffff
0031a40 371f 3f00 7808 7878 ffff ffff ffff ffff
0034920 471f 3f00 7808 7878 ffff ffff ffff ffff
0037800 571f 3f00 7808 7878 ffff ffff ffff ffff
003a6e0 671f 3f00 7808 7878 ffff ffff ffff ffff

do some math, and you will see one of the frames is only 11328 bytes:

12000 - 0 = 12000
24000 - 12000 = 12000
36000 - 24000 = 12000
47328 - 36000 = 11328
59328 - 47328 = 12000
71328 - 59328 = 12000
83328 - 71328 = 12000
95328 - 83328 = 12000

Carl K

------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p