Follow up: Are metainfo (.torrent) files text files ? ;)

Well,

The short answer is:

HELL NO !

The long answer is done below:

After looking at .torrent file I noticed that it was in fact 
a "bencoded" string.

All the characters of the strings are probably in the range 0..255

I think this is quite dangerous since some language like delphi and C 
use char(0) to indicate the end of a string.

Fortunately delphi's native string type works with "length prefixes".

But it could also be typecasted to a Pchar which works like a null 
terminated character.

If pchar's are used to copy the string this could lead to problems 
since the copy routine will stop at the first null terminated 
character.

This code below illustrates the problem.

Fortunately for me... I code in delphi and I will prevent to usage of 
Pchars in my "soon to be" TmetaInfo class ;)

Ofcourse C doesn't even have a native string type... so C coders are 
used to not having any strings types at all... so I am sure C coders 
will manage ;)

Though I haven't yet tried to load these metainfo files as text files.

Maybe these binary chars could lead to other problems.

So I can see how another problem might pop-up...

Text files probably also use an "end of file" marker and maybe other 
kind of markers ?

Anyway I wrote another test function to test if delphi's text file 
routines can be used...

Test 2 indicates that this is not the case.

Apperently it goes wrong at byte value: 10

Occurding to this ascii table:

http://www.asciitable.com/

It's a line feed ;)

Ok now I can see what goes wrong :)

The read simply stops at a new line.

Then I tried changing it to readln

But now delphi simply filters out the (hexadecimal) #D character.
which is 13 decimal -> carriage return

Also the reading stops at hexadecimal #19 which is 25.

End of medium.

Well I decided to do one more test... since these text file routines 
are kinda "old" style of coding and I decided to use a "new" style of 
coding with filestreams etc...

Delphi's TStringList uses these filestreams.

The funny thing is I actually found a bug in Delphi's TstringList

It tries to turn the string that it read into multiple strings when 
#0, #10 or #13 is encountered

It starts with a while loop that checks for #0... if it encounters 
this character it simply stops... apperently the borland coder didn't 
expect the first character to be #0 and the rest to be text :)

So delphi's TstringList simply forgets to set the remaining text... 
nice bug.

Haaaahhh sigh... well it was a funny experience delving into this 
matter...

For now I give it a rest ;)

My final conclusion and advice is: don't try processing 
torrent/metainfo files as text files with Delphi ;) and possibly 
other languages... :D

Try to use binary instead and simply typecast all bytes to chars if 
needed...

Finally here is my test code ;)

// test 1 description: check if delphi's native string can copy "raw 
binary strings" completely
// test 1 result: successfull...
// test 1 notes: Don't use pchar's to copy the strings :)
procedure TForm1.Button1Click(Sender: TObject);
var
	a : array[0..255, 0..3] of byte;
	s1, s2 : string;
	b : array[0..255, 0..3] of byte;
	i,j : integer;
begin

	// let's fill the array 4 times with all possible byte values
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			a[i,j] := i;
		end;
	end;

	// now let's convert it to a "raw binary coded" string :D
	s1 := '';
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			s1 := s1 + char(a[i,j]);
		end;
	end;

	// hopefully delphi can also copy null terminated strings 
completely...
	// let's add one character to force a on-write.

	// this string copy works ok
{
	s2 := s1;
	s2 := s2 + 'end of string';
}

	// let's try a pchar string copy
	// as I expected this copy leads to an incomplete copy since 
it stops at the null terminated character
	// leading to an access violation in the code below.
	s2 := Pchar(s1);

	// now let's convert s2 back to the b array and then later 
test if all is ok :)
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			b[i,j] := byte(s2[1 + i + j*256]); // delphi 
strings start at index 1 ;)
		end;
	end;
	// check if everything went ok.
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			if a[i,j] <> b[i,j] then
			begin
				Memo1.Lines.Add(format('fault at i: %
d j: %d', [i,j] ));
			end;
		end;
	end;

	Memo1.Lines.Add('done');

end;

// test 2 description:
// Store the "raw code string" to a text file and then try to read 
the text file using
// text file routines
// test 2 result: very bad... read and readln stop at all kinds 
of "control characters"
procedure TForm1.Button2Click(Sender: TObject);
var
	a : array[0..255, 0..3] of byte;
	s1, s2 : string;
	b : array[0..255, 0..3] of byte;
	i,j : integer;
	f : textfile;
	DoBreak : boolean;

	input_s : string;

begin
	// let's fill the array 4 times with all possible byte values
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			a[i,j] := i;
		end;
	end;

	// now let's convert it to a "raw binary coded" string :D
	s1 := '';
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			s1 := s1 + char(a[i,j]);
		end;
	end;

	// store the string to a text file
	AssignFile( f, 'test.txt' );
	Rewrite( f );
	Write( f, s1 );
	CloseFile( f );

	// now read the string from a text file
	AssignFile( f, 'test.txt' );

	FileMode := fmOpenRead or fmShareDenyWrite;

	Reset( f );

//	Read( f, s2 ); // causes a problem: read stops at a newline 
character (byte value: 10)

// delphi documentation:
{
With a type string variable:

Read reads all characters up to, but not including, the next end-of-
line marker or
until Eof(F) becomes true; it does not skip to the next line after 
reading.

If the resulting string is longer than the maximum length of the 
string variable, it is truncated.

After the first Read, each subsequent Read sees the end-of-line 
marker and returns a zero-length string.
Use multiple Readln calls to read successive string values.
When the extended syntax is enabled, Read can read null-terminated 
strings into zero-based character arrays.
}

// ok so only using read would simply stop at the end of line marker 
I guess.
// so maybe readln can be used to read up to the newline character
// but then we will probably loose the newline character
// let see what happens... I also expect to stop at some kind of end 
of file marker :D

	s2 := '';
	repeat
// as I expected this still causes a problem at I:10... the newline 
character is excluded in the string
{
		ReadLn( f, input_s );
		s2 := s2 + input_s;
}
// no we fix the problem by adding #10 ourselfs to it ;)
// let's see what the next problem will be :)

// yup byte value 13
// for some reason we have a problem with byte value 13. carriage 
return
// apperently the readln procedure simply deletes it...
// and is replaced with byte value 14 the next read character

// let's temporarely fix that in the check routine to see what other 
problems we encounter
// unable to easily fix it

// the debugger shows that character #13 (#D) is simply "filtered 
out" in the string !
// further more the string stops reading at I think hexadecimal 19 
(debugger displays in hex
// which would be ascii 25.

// well I am done testing... MetaInfo files can't definetly not be 
called text files =D

{
		ReadLn( f, input_s );
		s2 := s2 + input_s + #10;
}

		ReadLn( f, input_s );
		s2 := s2 + input_s + #10;



	until EoF( f );

	CloseFile( f );

	// now let's convert s2 back to the b array and then later 
test if all is ok :)
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			b[i,j] := byte(s2[1 + i + j*256]); // delphi 
strings start at index 1 ;)
		end;
	end;

	b[13,0] := 13; // fix it temporarely to see what other 
problems we encounter
	b[13,1] := 13; // fix it temporarely to see what other 
problems we encounter
	b[13,2] := 13; // fix it temporarely to see what other 
problems we encounter
	b[13,3] := 13; // fix it temporarely to see what other 
problems we encounter

	// to bad all values are now off by 1.

	// check if everything went ok.
	DoBreak := false;
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			if a[i,j] <> b[i,j] then
			begin
				// something wrong at I:10 J:0 byte 
value: 10
				// the rest skipped.
				Memo1.Lines.Add(format('fault at i: %
d j: %d byte value: %d wrong value: %d', [i,j,a[i,j], b[i,j]] ));
				DoBreak := true;
			end;

			if DoBreak then break;
		end;
		if DoBreak then break;
	end;

	Memo1.Lines.Add('done');
end;

// for now the last test... let's see if Tstrings can read all the 
text :)
// test 3 description: use delphi's Tstrings class to read the text
// test 3 result: bad -> nothing loading stops at null terminated 
character !!!!!!
procedure TForm1.Button3Click(Sender: TObject);
var
	a : array[0..255, 0..3] of byte;
	s1, s2 : string;
	b : array[0..255, 0..3] of byte;
	i,j : integer;

	StringList1 : TstringList;
	StringList2 : TstringList;
begin

	// let's fill the array 4 times with all possible byte values
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			a[i,j] := i;
		end;
	end;

	// now let's convert it to a "raw binary coded" string :D
	s1 := '';
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			s1 := s1 + char(a[i,j]);
		end;
	end;

	// store the string to a file using the Tstrings
	StringList1 := TStringList.Create;

//	StringList1.Text := s1; // probably saves nothing
	// yup delphi checks for null terminated character
	// this results in saving nothing !

	StringList1.Add( s1 );  // this still works.

	StringList1.SaveToFile( 'test2.torrent' );

	StringList1.Destroy;

	// now load the string from the text file
	StringList2 := TStringList.Create;

	// for same reason this loads nothing.
	// I think this might be because of windows file routines 
thinking it is a text file
	// because of the filename.. test2.txt let's change it too 
something else

	// nope still loads nothing.
	// oh yeah I remember path bug heheh stupid delphi
	// must specify path ! :D
	// didn't work.
//	StringList2.LoadFromFile( IncludeTrailingPathDelimiter
(GetCurrentDir) + 'test2.torrent' );

	// still didn't work.
	// let's try a manualy created file
//	StringList2.LoadFromFile( 'C:\Delphi7Testen\bittorrent 
development\metainfo\version 0.01 read torrents\test2.torrent' );
// this does work:
//	StringList2.LoadFromFile( 'C:\Delphi7Testen\bittorrent 
development\metainfo\version 0.01 read torrents\fuck.txt' );

	// very weird... if first binary value is a 0 it just doesn't 
load !
	StringList2.LoadFromFile( 'C:\Delphi7Testen\bittorrent 
development\metainfo\version 0.01 read torrents\test2.torrent' );

	// this must be a general delphi FileStream bug or something 
since it just uses TFileStream
	// let s try that in a seperate test application.
	// nope that works ok...

	// must be a TStringList specific problem.

	// maybe stream.size is the problem :)
	// nope that works ok too... really weird.

	// problem found...
	// StringList contains a bug... it doesn't expect the first 
character to be #0
	// it simply ignores the rest ;) hehe pretty funny :D
	if StringList2.Count = 0 then
	begin
		ShowMessage( 'nothing loaded' );	// yup.
	end;

	s2 := StringList2.Text;

	StringList2.Destroy;

	// now let's convert s2 back to the b array and then later 
test if all is ok :)
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			try
				b[i,j] := byte(s2[1 + i + j*256]); // 
delphi strings start at index 1 ;)
			except
				on EAccessViolation do
				begin
					Memo1.Lines.Add(format
('access violation at i: %d j: %d', [i,j] ));

					exit;
				end;
			end;

		end;
	end;
	// check if everything went ok.
	for j:=0 to 3 do
	begin
		for i:=0 to 255 do
		begin
			if a[i,j] <> b[i,j] then
			begin
				Memo1.Lines.Add(format('fault at i: %
d j: %d', [i,j] ));
			end;
		end;
	end;

	Memo1.Lines.Add('done');
end;

Bye,
  Skybuck.





------------------------ Yahoo! Groups Sponsor --------------------~--> 
$9.95 domain names from Yahoo!. Register anything.
http://us.click.yahoo.com/J8kdrA/y20IAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~-> 

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/BitTorrent/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/