Re: parsing compressed files with flex & bison

Flex was not built to generate lexers for essentially binary files, =20
though you may try it. And one can easily pipe streams under UNIX-=20
like OS's.

   Hans Aberg


On 28 Dec 2006, at 14:27, A=EFmen Troudi wrote:

> I want to parse files in one pass for 2 reasons :
>
> 1- size of (5) files is important (>200MO).
> 2- it is more sophistical (execution time and I/O).
>
> The problem is that flex needs FILE* stream, and Zlib needs gzFile =20
> stream.
>
> I find these 2 solutions :
>
> 1) - find the type of compressed file (gzip or bunzip2 or Z).
>     - switch the type i execute FILE* yyin =3D popen(cmd,"r"), cmd : =20=

> "/usr/bin/bunzip2 -dfc file.bz2" or ...
>
> -> - this solution don't use zlib library, it use shell.
>     - easy to implement than 2).
>
> 2) - find the type of compressed file (gzip or bunzip2 or Z).
>     - open a pipe, fork.
>     - the father process use zlib library to read compressed files =20
> and put them in pipe,
>       gzFile stream =3D gzopen(); ... gzread(buffer); ... gzwrite(pipe=20=

> [0],buffer,..) ...
>     - the son process reads from pipe with FILE* yyin  =3D fdopen(pipe=20=

> [1] ..)
>
> -> this solution don't use shell. it use zlib library.
>
> I want to know if there is another solution to do this with a =20
> portable code to others platforms
>
> such as windoz. unpacking files is a solution to have portable code =20=

> but it is not efficient.
>
> *** find type of files ***
> to find type of files i have two methods :
>
> 1/ cmd : "/usr/bin/file file" | grep -i gzip /dev/null 2>&1" ... if =20=

> ( popen(cmd,"r" ) ) // gz file
> --> i dont like these method because i can do this without shell.
>
> 2/ open the file and read the heading and compare with heading of =20
> compressed files:
> for example heading of a gz file : unsigned char gzStamp[] =3D {0x1F,=20=

> 0x8B,0x08,0x08};
> --> these method is more efficient in time execution but :
>      I' m dont sure of heading files beacause i dont have this =20
> heading in the net but by testing many examples of files
>      and opning them in hexa mode.
>
> do u know heading of this files ?
>
> In attend to your suggestions,  i hope u a nice day,
>
>
> 2006/12/28, Hans Aberg <[email protected]>: On 26 Dec 2006, at =20
> 11:48, A=EFmen Troudi wrote:
>
> > I have to support compressed files in a project with flex & bison,
> >
> > actually, the project support only ascii files, my question is :
> >
> > How to manage compressed files (zip,gzip,bzip2,Z) with flex & =20
> bison ?
> >
> > Can I used zlib library with flex & bison ?
>
> What about unpacking them, before feeding them to the Flex generated
> lexer? The Bison generated parser does not care, as it only reads the
> tokens that the lexer feeds it.
>
>    Hans Aberg
>
>
>