Re: random maddening bug - "corrupted" file upload contents

I made some experiments which might help to restrict the scope:

I applied my little "manual" scenario with a Pharo 5 image (
https://files.pharo.org/get-files/50/)
After 7 tries, the issue never show up.

With a Pharo 6.1 image, the issue shows up after maximum 3 / 4 tries.

My feeling: something changed between these 2 versions.
This idea would tend to be confirmed by the period we started to experiment
the issue: We recently migrated some production images from Pharo 4 to
Pharo 6.
Is it Zinc or some Pharo inside bricks used by Zinc ? I could not say.






2018-01-04 16:48 GMT+01:00 Cyrille Delaunay <[email protected]>:

> I tried several times and the test I wrote always passed.
>
> All I have been able to do, is to experiment some failures by playing with
> the client timeout.
> My test initially contains the following statement: *client timeout:
> 120000.*
> If I comment this statement, sometimes the server received a request with
> a nil entity.
> The test then get an unexpected response.
> But I am not sure it an unexpected behavior, as there is a lot of data
> being sent.
>
>
>
>
>
> 2018-01-02 20:28 GMT+01:00 Cyrille Delaunay <[email protected]>:
>
>>
>>
>> 2018-01-02 17:37 GMT+01:00 Sven Van Caekenberghe <[email protected]>:
>>
>>>
>>>
>>> > On 2 Jan 2018, at 12:53, Cyrille Delaunay <[email protected]>
>>> wrote:
>>> >
>>> > Hi Sven,
>>> >
>>> > I am back on this issue :)
>>> >
>>> > I have kind of a "clean" scenario that could help to reproduce.
>>> > It is still random, but at one point, it shows up for me.
>>> >
>>> > Here is how I am doing:
>>> > Download Pharo 6.1 on my Mac (Sierra 10.12.6):
>>> https://pharo.org/web/download
>>> > Then, iterate over this process till spotting the issue:
>>> >
>>> > => start the pharo image
>>> > => execute this piece of code in a playground
>>> >
>>> > ZnServer startDefaultOn: 1701.
>>> > ZnServer default maximumEntitySize: 80* 1024 * 1024.
>>> > '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |
>>> > out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0
>>> 0 0].
>>> > 18202065 timesRepeat: [ out nextPut: 0 ]
>>> > ].
>>> >
>>> > => Open a web browser page on: http://localhost:1701/form-test-3
>>> > => Upload the file zip file previously generated ('myzip.zip').
>>> > => If the web page displays: "contents=000000000a00..." (or anything
>>> unexpected), THIS IS THE ISSUE !
>>> > => If the web page displays: "contents=504b03040a00..", the upload
>>> worked fine. You can close the image (without saving)
>>> >
>>> > Mettre à jour Saisir temps Surveiller Copier Supprimer
>>> >
>>> >
>>> > Usually, after starting 3 / 4 times pharo image, I am getting the
>>> problem.
>>> > I have been able to reproduce with chrome and firefox.
>>> >
>>> >
>>> > Seing that I was able to reproduce within a fresh pharo image, I tried
>>> to automate this scenario.
>>> > I wrote the following test:
>>> >
>>> >
>>> > ZnServerTests >>testFormTest4
>>> >       | inputs client part fileReference fileContents entity |
>>> >       fileReference := 'test-zip.zip' asFileReference.
>>> >       fileReference
>>> >               writeStreamDo: [ :out |
>>> >                       out
>>> >                               binary;
>>> >                               nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125
>>> 83 67 73 0 0 0 0 0 0].
>>> >                       18202065 timesRepeat: [ out nextPut: 0 ] ].
>>> >       fileContents := fileReference binaryReadStreamDo: [ :in | in
>>> upToEnd ].
>>> >       15
>>> >               timesRepeat: [ self
>>> >                               withServerDo: [ :server |
>>> >                                       server maximumEntitySize: 80 *
>>> 1024 * 1024.
>>> >                                       server debugMode: true.
>>> >                                       (client := ZnClient new)
>>> >                                               url: server localUrl;
>>> >                                               addPathSegment:
>>> 'form-test-3'.
>>> >                                       client timeout: 120000.
>>> >                                       entity := (ZnByteArrayEntity
>>> type: (ZnMimeType main: 'application' sub: 'zip') length: fileContents
>>> size) bytes: fileContents.
>>> >                                       part := ZnMimePart fieldName:
>>> 'file' fileName: fileReference basename entity: entity.
>>> >                                       client
>>> >                                               resetEntity;
>>> >                                               addPart: part;
>>> >                                               post.
>>> >                                       self assert: client isSuccess.
>>> >                                       self assert: (client contents
>>> includesSubstring: '504b03040a') ] ]
>>> >
>>> >
>>> >
>>> >
>>> > In order to make this test run correctly I had to modify this method.
>>> > Without this change, I always got an empty html page as response
>>> >
>>> >
>>> >
>>> > ZnDefaultServerDelegate >> formTest3: request
>>> >       | contents filename contentType page |
>>> >       contents := filename := contentType := ''.
>>> >       (request hasEntity and: [ request contentType matches:
>>> ZnMimeType multiPartFormData ])
>>> >               ifTrue: [
>>> >                       (request entity partNamed: #file ifNone: [ nil ])
>>> >                               ifNotNil: [ :part |
>>> >                                       filename := part fileName.
>>> >                                       contents := part contents.
>>> >                                       contentType := part  contentType.
>>> >                                       contentType isBinary ifTrue: [
>>> contents := contents hex ] ] ].
>>> >       page := ZnHtmlOutputStream streamContents: [ :html |
>>> >               html page: 'Form Test 3' do: [
>>> >                       html
>>> >                               tag: #form
>>> >                               attributes: #(action 'form-test-3'
>>> 'accept-charset' 'utf-8'
>>> >
>>>  enctype 'multipart/form-data' method POST)
>>> >                               do: [
>>> >                                       html
>>> >                                               str: 'File'; space;
>>> >                                               tag: #input attributes:
>>> #(type file name file); space;
>>> >                                               tag: #input attributes:
>>> #(type submit) ];
>>> >                               tag: #p do: [ html str: 'filename = ';
>>> str: filename ];
>>> >                               tag: #p do: [ html str: 'content-type =
>>> '; str: contentType asString ];
>>> >                               tag: #p do: [ html str: 'contents = ';
>>> str: (contents copyFrom: 1 to: 20) asString ] ] ].
>>> >       ^ ZnResponse ok: (ZnEntity html: page)
>>> >
>>> >
>>> >
>>> > Unfortunately, the tests always passed till now !
>>>
>>> So you have to upload 80Mb ?
>>>
>>
>> Sometimes yes.
>> But here the file I build and use to reproduce is only 18 mb.
>> (the same size as the original file highlighting the issue on my machine).
>>
>>
>>> And if you script it in Pharo and repeat it 15 times it keeps on working
>>> ?
>>> Really ?
>>>
>>
>> Yes indeed.
>> But again, it is completely random.
>> I could not ensure that the problem would never show up with the script.
>> I will try to make more tries tomorrow.
>>
>>
>>>
>>> It seems that it only fails when a manually operated browser is involved.
>>>
>>
>> Yes, exactly
>>
>>
>>>
>>> Have you tried doing the upload with curl ?
>>>
>>
>> No.
>> How would I do that ?
>>
>>
>>>
>>> > I will try to go on refining the context parameters that could make
>>> the issue show up or not
>>> >
>>> >
>>> >
>>> > 2017-12-22 18:05 GMT+01:00 Sven Van Caekenberghe <[email protected]>:
>>> > Well, I can't repeat it, for now.
>>> >
>>> > Try to make a minimal self-contained example that fails (reliably),
>>> like how I did it, you can even try to do the upload in Pharo too (see
>>> #testFormTest3).
>>> >
>>> > > On 22 Dec 2017, at 18:00, Cyril Ferlicot <[email protected]>
>>> wrote:
>>> > >
>>> > >
>>> > > On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[email protected]>
>>> wrote:
>>> > > Cyrille,
>>> > >
>>> > > I wrote a file like this:
>>> > >
>>> > > FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
>>> > >   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0
>>> 0 0 0 0] ].
>>> > >
>>> > > Which I can read as follows:
>>> > >
>>> > > FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in
>>> upToEnd ].
>>> > >
>>> > >   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
>>> > >
>>> > > If I upload this file to http://localhost:1701/form-test-3 this
>>> works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome),
>>> first time and second time.
>>> > >
>>> > >
>>> > >
>>> > > Maybe you should try on a co-workers machine just to make sure it is
>>> not a local problem on your machine ?
>>> > >
>>> > > Hi Sven,
>>> > >
>>> > > It happen on Cyrille's machine, my machine, Yann's machine and our
>>> boss's machine.
>>> > >
>>> > > We first saw it on Windows and never on mac/Linux. But now we see it
>>> also on mac.
>>> > >
>>> > > I did not see it yet on Linux but we use it less for now.
>>> > >
>>> > > Also, when it happen, it tend to happen a lot afterward.
>>> > >
>>> > > Also, by random it means that we can have 1/2 week without problem,
>>> then 4 failure in 2h.
>>> > >
>>> > >
>>> > >
>>> > > Sven
>>> > >
>>> > > --
>>> > > Cyril Ferlicot
>>> > > https://ferlicot.fr
>>> > >
>>> > > http://www.synectique.eu
>>> > > 2 rue Jacques Prévert 01,
>>> > > 59650 Villeneuve d'ascq France
>>> > > <Screen Shot 2017-12-22 at 17.48.31.png>_________________
>>> ______________________________
>>> > > seaside mailing list
>>> > > [email protected]
>>> > > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>> >
>>> > _______________________________________________
>>> > seaside mailing list
>>> > [email protected]
>>> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>> >
>>> >
>>> >
>>> > --
>>> > Cyrille Delaunay
>>> > _______________________________________________
>>> > seaside mailing list
>>> > [email protected]
>>> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>>> _______________________________________________
>>> seaside mailing list
>>> [email protected]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>>
>>
>>
>> --
>> Cyrille Delaunay
>>
>
>
>
> --
> Cyrille Delaunay
>



-- 
Cyrille Delaunay

_______________________________________________
seaside mailing list
[email protected]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside