RE: Cannot process zip file with Groovy

Not entirely sure that is what James is looking for…I THINK he’s more interested in reading than creating.

Commons compress has some example code at https://commons.apache.org/proper/commons-compress/examples.html:

===

InputStream fin = Files.newInputStream(Paths.get("some-file"));

BufferedInputStream in = new BufferedInputStream(fin);

OutputStream out = Files.newOutputStream(Paths.get("archive.tar"));

Deflate64CompressorInputStream defIn = new Deflate64CompressorInputStream(in);

final byte[] buffer = new byte[buffersize];

int n = 0;

while (-1 != (n = defIn.read(buffer))) {

    out.write(buffer, 0, n);

}

out.close();

defIn.close();
===

BOB

From: MG <[email protected]>
Sent: Saturday, February 17, 2024 10:37 AM
To: [email protected]; Bob Brown <[email protected]>
Subject: Re: Cannot process zip file with Groovy

I agree, would also recommend using Apache libs, we use e.g. the ZIP classes that come with the ant lib in the Groovy distribution (org.apache.tools.zip.*):

Here is a quickly sanitzed version of our code (disclaimer: Not compiled/tested; Zip64Mode.Always is important if you expect larger files):

InputStream zipInputStream(String compressedFilename) {
    final zipFile = new ZipFile(new File(compressedFilename))
    final zipEntry = (ZipEntry) zipFile.entries.nextElement()
    if(zipEntry === null) { throw new Exception("${zipFile.name} has no entries") }
    final zis = zipFile.getInputStream(zipEntry)
    return zis
}

OutputStream zipOutputStream(String filename, String compressedFileExtension = "zip") {
    final fos = new FileOutputStream(filename + '.' + compressedFileExtension)
    final zos = new ZipOutputStream(fos)
    zos.useZip64 = Zip64Mode.Always // To avoid org.apache.tools.zip.Zip64RequiredException: ... exceeds the limit of 4GByte.
    final zipFileName = org.apache.commons.io.FilenameUtils.getName(filename)
    final zipEntry = new ZipEntry(zipFileName)
    zos.putNextEntry(zipEntry)
    return zos
}

Cheers,
mg


On 17/02/2024 00:52, Bob Brown wrote:
MY first thought was “are you SURE it is a kosher Zip file?”

Sometimes one gets ‘odd’ gzip files masquerading as plain zip files.

Also, apparently “java.util.Zip does not support DEFLATE64 compression method.” : https://www.ibm.com/support/pages/zip-file-fails-route-invalid-compression-method-error

IF this is the case, you may need to use: https://commons.apache.org/proper/commons-compress/zip.html
(maybe worth looking at the “Known Interoperability Problems” section of the above doc)

May be helpful: https://stackoverflow.com/a/76321625

HTH

BOB

From: James McMahon <[email protected]><mailto:[email protected]>
Sent: Saturday, February 17, 2024 4:20 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: Cannot process zip file with Groovy

Hello Paul, and thanks again for taking a moment to look at this. I tried as you suggested:
- - - - - - - - - -
import java.util.zip.ZipInputStream

def ff = session.get()
if (!ff) return

try {
    ff = session.write(ff, { inputStream, outputStream ->
        def zipInputStream = new ZipInputStream(inputStream)
        def entry = zipInputStream.getNextEntry()
        while (entry != null) {
            entry = zipInputStream.getNextEntry()
        }
        outputStream = inputStream
    } as StreamCallback)

    session.transfer(ff, REL_SUCCESS)
} catch (Exception e) {
    log.error('Error occurred processing FlowFile', e)
    session.transfer(ff, REL_FAILURE)
}
- - - - - - - - - -

Once again it threw this error and failed:


ExecuteScript[id=ae3e5de5-018d-1000-ff81-b0c807b75086] Error occurred processing FlowFile: org.apache.nifi.processor.exception.ProcessException: IOException thrown from ExecuteScript[id=ae3e5de5-018d-1000-ff81-b0c807b75086]: java.util.zip.ZipException: invalid compression method

- Caused by: java.util.zip.ZipException: invalid compression method



It bears repeating: I am able to list and unzip the file at the linux command line, but cannot get it to work from the script.



What is interesting (and a little frustrating) is that the NiFi UnpackContent will successfully unzip the zip file. However, the reason I am trying to do it in Groovy is that UnpackContent exposes the file metadata for each file in a tar archive - lastModifiedDate, for example - but it does not do so for files extracted from zips. And I need that metadata. So here I be.



Can I explicitly set my (de)compression in the Groovy script? Where would I do that, and what values does one typically encounter for zip compression?



Jim

On Thu, Feb 15, 2024 at 9:26 PM Paul King <[email protected]<mailto:[email protected]>> wrote:
What you are doing to read the zip looks okay.

Just a guess, but it could be that because you haven't written to the
output stream, it is essentially a corrupt data stream as far as NiFi
processing is concerned. What happens if you set "outputStream =
inputStream" as the last line of your callback?

Paul.

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.www.avast.com<http://Virus-free.www.avast.com>
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Fri, Feb 16, 2024 at 8:48 AM James McMahon <[email protected]<mailto:[email protected]>> wrote:
>
> I am struggling to build a Groovy scri[t I can run from a NiFi ExecuteScript processor to extract from a zip file and stream to a tar archive.
>
> I tried to tackle it all at once and made little progress.
> I am now just trying to read the zip file, and am getting this error:
>
> ExecuteScript[id=ae3e5de5-018d-1000-ff81-b0c807b75086] Error occurred processing FlowFile: org.apache.nifi.processor.exception.ProcessException: IOException thrown from ExecuteScript[id=ae3e5de5-018d-1000-ff81-b0c807b75086]: java.util.zip.ZipException: invalid compression method
> - Caused by: java.util.zip.ZipException: invalid compression method
>
>
> This is my simplified code:
>
>
> import java.util.zip.ZipInputStream
>
> def ff = session.get()
> if (!ff) return
>
> try {
>     ff = session.write(ff, { inputStream, outputStream ->
>         def zipInputStream = new ZipInputStream(inputStream)
>         def entry = zipInputStream.getNextEntry()
>         while (entry != null) {
>             entry = zipInputStream.getNextEntry()
>         }
>     } as StreamCallback)
>
>     session.transfer(ff, REL_SUCCESS)
> } catch (Exception e) {
>     log.error('Error occurred processing FlowFile', e)
>     session.transfer(ff, REL_FAILURE)
> }
>
>
> I am able to list and unzip the file at the linux command line, but cannot get it to work from the script.
>
>
> Has anyone had success doing this? Can anyone help me get past this error?
>
>
> Thanks in advance.
>
> Jim
>
>