[ ijbswa-Bugs-3464439 ] Page truncation in Opera (platform-specific Privoxy bug)
SourceForge.net <[email protected]>
| Newsgroups | gmane.comp.web.privoxy.devel |
|---|---|
| Message-ID | <[email protected]> |
Bugs item #3464439, was opened at 2011-12-23 08:15
Message generated for change (Comment added) made by fabiankeil
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=111118&aid=3464439&group_id=11118
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
>Category: other
>Group: version 3.0.19
Status: Pending
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Kevin (photo2110)
Assigned to: Fabian Keil (fabiankeil)
>Summary: Page truncation in Opera (platform-specific Privoxy bug)
Initial Comment:
When working with Opera 11.60 and Privoxy, pages, images, etc will randomly get truncated. It just started with the Opera upgrade, so it might be a bug with Opera, but when I stopped using Privoxy, I couldn't get any pages to fail. Note: 'stopped using' means configuring Opera to not go through Privoxy, simply unchecking 'Enable' on the Privoxy menu doesn't do it.
I attempted to capture data with Wireshark, but the pages I caught were gzipped and I can't be certain that they were good. I did notice that they were http 1.1, if that makes a difference.
Only one other person has mentioned on the Opera news group that they are having an issue like this. They too are running Privoxy.
NYTimes.com seems especially bad to me, but it happens on all sites including SouceForge.
Privoxy 3.0.18.
Opera 11.60
Windows 7 and Windows XP (I have two machines, both have the trouble)
No other proxy software. Windows Firewall and MSE on both machines.
SF user: photo2110
----------------------------------------------------------------------
>Comment By: Fabian Keil (fabiankeil)
Date: 2012-10-17 03:25
Message:
You're welcome. Thanks again for providing the data necessary to properly
investigate and fix this.
Just to be clear: the drained data hasn't actually been read yet and is
expected not to be read due to Privoxy's limited pipelining support (this
will be improved in the next release, but things should "work" as-is as
well).
What wasn't expected was that the unread data affected the data that was
already written to the socket by Privoxy, but not yet transmitted by the
kernel.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-10-14 07:17
Message:
Well, I tried to reproduce the white page error after a reboot, but I could
not.
It seems that had to be some kind of Opera/OS error, not related to
Privoxy.
So, I turned on pipelining and have been browsing with it for a half an
hour and haven't noticed any truncated data. I will continue testing (aka
using the Internet :) ) on Linux; and as soon as something happens I'll
write.
Maybe it's worth to investigate why this issue appears on Linux and
Windows, why are data on the wire after it was read.
As far as I can tell, it works now, so I would like to say thank you for
your efforts for fixing this bug.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-10-13 10:27
Message:
Unfortunately so far and I haven't noticed any additional issues in the
pcap file. All responses I checked were complete.
Do you remember on which URL in it you got the "white page which loads
forever" problem?
Does Opera also have problems with http://config.privoxy.org now?
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-10-12 23:52
Message:
I tried the new version already.
As I saw in the logs, Privoxy always drains some kBytes before closing a
socket, so it was a good catch.
However, there's an other problem: Opera doesn't load any pages. Sometimes
I managed to get a page loaded, but in the most of the cases, I get only a
white page which loads forever.
Here are the logs of Privoxy and Wireshark:
http://home.sch.bme.hu/~szotsaki/privoxy7.tar.bz2
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-10-12 04:45
Message:
Thanks again for the logs.
I believe I finally understand the cause of the problem and suspect that
it's a platform-specific bug, which would explain why I haven't managed to
reproduce it on FreeBSD.
The pcap file shows that there's occasionally less data on the wire than
Privoxy wrote to the socket. My previous changes reduce the likelihood that
this happens and are useful in general, but do not actually address the
underlying problem. Hopefully I finally addressed it with today's commits,
though.
Please try to reproduce the problem with CVS sources from today after
running autoconf and configure before building.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-10-01 02:27
Message:
I uploaded a new file, you can reach it here:
http://home.sch.bme.hu/~szotsaki/privoxy6.tar.bz2
Unfortunately I could not spectacularly reproduce the issue, so the log
files are bigger than usual.
However, I attached a picture about a broken (half loaded) image. That is
from a site called "index.hu", and you can grep the logs for
"http://kep.index.hu/t/3/a/67856f36b336477c.jpg?r=737191699" for that
picture.
I disabled the debug levels 16, 512 and 32768.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-09-20 02:00
Message:
Yes, please provide new logs. You can disable the debug levels 16, 512 and
32768.
It would also help if you could note the time when the problem happens and
the URL shown in Opera's URL bar.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-09-19 22:38
Message:
I fear that I have to say, not much changed. I used Opera for a day with
the recompiled Privoxy and it seemed that Opera didn't load some style
sheets (or just loaded them only in half).
Do you need additional logs with the new patches?
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-09-18 11:38
Message:
Thanks for testing.
Apparently my recent changes were incomplete.
Please try again with sources from today.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-09-16 08:02
Message:
Hmm, interesting. I'm using the 3.0.20 version for some minutes and I don't
experience truncated pages. However, with one exception: hup.hu . I created
a full log file and a Wireshark capture file, you can find it here:
http://home.sch.bme.hu/~szotsaki/privoxy5.tar.bz2
Ps.: sorry for the delay, I had a full OS reinstall and that slowed down
some things.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-09-04 01:50
Message:
The patches have been committed to CVS.
Resetting the connection is fine as far as the standards are concerned. As
I don't know why Opera is doing it, I wouldn't call it a bug.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-08-31 11:48
Message:
Thank you very much! If I should download a snapshot or something like that
for testing, just say.
Can the connection resetting be considered a bug in Opera?
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-08-29 10:08
Message:
Thanks for the complete pcap file, unfortunately I didn't have time to look
into it until yesterday.
It shows that Opera is not only aggressively pipelining as expected, but
also resetting a couple of connections before receiving all the requested
responses. The reason for the resets isn't obvious to me, though.
I still have the suspicion that Opera has a problem with the way Privoxy
deals with pipelined requests and wrote a couple of patches that could
help. They still need some testing, but I'll probably commit them in the
next days.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-08-29 02:28
Message:
Could you figure out whether the bug is in Privoxy or in Opera?
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-08-16 13:27
Message:
I made a mistake in the previous capture with Wireshark (it was bound to
port 80 only).
Here is the updated files on all interfaces and on all ports:
http://home.sch.bme.hu/~szotsaki/privoxy4.tar.bz2
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-08-16 13:16
Message:
I'm here again, sorry for the delay.
On Linux I could successfully reproduce this error and Wireshark had an
option to capture on all interfaces.
I turned on all of the debug options in privoxy along with
prevent-compression.
The affected page was: index.hu.
I started Opera with a whole new "personal directory", so basically it was
a pure default installation without extensions or anything unrelated.
You can download the compressed log files here:
http://home.sch.bme.hu/~szotsaki/privoxy3.tar.bz2
----------------------------------------------------------------------
Comment By: Kevin (photos2110)
Date: 2012-07-24 12:12
Message:
IIRC, Wireshark can't capture localhost on Win 7. You'd have to install
privoxy on a different machine to capture the browser<->Privoxy traffic.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-07-24 11:48
Message:
Thanks for the additional information.
The pcap file still doesn't contain the traffic between Opera and Privoxy.
Is this intentional or is the tool you are using for the capture incapable
of capturing localhost traffic?
Anyway, I realized that I interpreted the the previous Privoxy log file
incorrectly. The client socket isn't actually kept open. I'll add another
log message to make this more obvious in the future.
Without seeing the traffic between Privoxy and Opera I can't be sure, but
my impression is that Opera is doing the page truncation itself even though
it gets the entire page from Privoxy.
Privoxy could be more aggressive about client socket reuse and I'll think
about that some more, but that's not a good reason for Opera to partially
discard already received data (assuming Opera is indeed doing that).
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-07-23 11:36
Message:
This time I could reproduce the problem with this very site.
I added the levels 16 and 32768 to the debug options and copied the
"{+prevent-compression} /" to the end of the Default action file.
And now I had quit Opera before I closed Wireshark.
You can find the uploaded files here:
http://home.sch.bme.hu/~szotsaki/privoxy2.tar.bz2
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-07-23 07:24
Message:
Thanks a lot for the log files, but unfortunately the redacted pcap files
do not show enough information.
PrivoxyNYTimes.pcapng shows that Privoxy got a complete response from the
server and privoxy-log-NYTimes.log indicates that Privoxy parsed and
decoded it as expected before forwarding it to the client.
The Privoxy log also indicates that the connection between Privoxy and
Opera is then kept open without Privoxy detecting an additional request.
This could be the result of a bug in Privoxy and would explain why
disabling pipelining in Opera works around the problem, but without seeing
the traffic between Privoxy and Opera it's impossible to tell if an
additional request has actually been sent.
The +prevent-compression action doesn't actually seem to be used.
Please additionally enable "debug 16" and "debug 32768" and reproduce the
log and pcap file, this time capturing the complete traffic between both
Privoxy and the server and Privoxy and Opera. Please quit Opera before
stopping the capture to make sure all connections between Privoxy and Opera
get closed.
If you can reproduce the problem with:
{+prevent-compression}
/
it would make analyzing the logs easier.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-07-23 03:04
Message:
Success :)
I could reproduce this issue on nytimes.com while both the logging and the
Wireshark was active.
I couldn't figure it out how can I add attachments here (…), so I
uploaded the compress files here:
http://home.sch.bme.hu/~szotsaki/privoxy.tar.bz2
The content:
- NYTimes.png: the truncated nytimes.com page. After the debugging I turned
off the pipelining in Opera and it loaded normally again
- PrivoxyNYTimes.pcapng: Wireshark pcap file with only one request to
nytimes.com
- privoxy-log-NYTimes.log: Privoxy log file with the recommended debug
values (1,2, 4, 8, 128, 256, 512, 1024, 4096, 8192, +prevent-compression)
which was made parallel with the above pcap file
- PrivoxyOperaDesktopTeam.pcapng: one request to my.opera.com/desktopteam,
ended up a "Network error" in Opera
I'll be offline for two weeks, but I hope I could help you a bit regarding
this issue.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-07-21 06:50
Message:
A log file would be appreciated, especially if it's created with
+prevent-compression and comes together with a pcap file showing the actual
packets.
----------------------------------------------------------------------
Comment By: szotsaki (szotsaki)
Date: 2012-07-20 04:05
Message:
I reported this bug to Opera bug tracker, but they forwarded me here.
Turning off opera:config#pipelining
(https://secure.wikimedia.org/wikipedia/en/wiki/HTTP_pipelining) really
solves the issues with Privoxy.
I can generate a log file, but if you start using Opera and set up Privoxy
there, in an hour you will encounter this bug.
Affected Hunarian sites:
* origo.hu
* index.hu
* hup.hu
Opera version: from 11.50 to 12.50 (now is alpha) all affected
Operating system: Windows XP, 7 and Linux also affected.
On Win7, with Opera 12.50 you can reproduce this bug _very_ often (I mean
on almost every page request) with slashdot.org.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-03-10 02:56
Message:
The fact that disabling pipelining or putting Polipo between Opera and
Privoxy helps could hint that Opera doesn't like the way Privoxy deals with
pipelined requests, but it's unclear if that's Opera's or Privoxy's fault.
The previously provided log indeed shows that Opera seems to be trying to
pipeline requests at first (for example in thread 0000025c), but when
Privoxy is done dealing with the first request it doesn't seem to get the
next one possibly because Opera grew impatient.
I can't reproduce this here (without Opera of course) so without a packet
trace I will not be able to analyze this any further.
----------------------------------------------------------------------
Comment By: Arcady Shpak ()
Date: 2012-03-09 19:32
Message:
opera:config -> Performance -> Enable Pipelining -> off
----------------------------------------------------------------------
Comment By: esmandil (esmandil1)
Date: 2012-02-25 01:58
Message:
I have the same problem.
One thing I have noticed is that when I install polipo proxy (
http://www.pps.jussieu.fr/~jch/software/polipo/ ) and put it between Opera
and Privoxy (i.e. pointing Opera to polipo, and setting up polipo to
forward to Privoxy), the problem seems to disappear.
If I put polipo between Privoxy and Internet, however, web pages in Opera
are still broken.
I don't know if this helps to diagnose the problem, but it's a workaround
at worst.
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2012-02-05 00:58
Message:
Yes, is from mashable.com
when i have some time i post a new log
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-01-23 10:33
Message:
I assume in the logfile from sendspace.com/ the truncated output is the
response for mashable.com?
I tried reproducing it by sending the same headers Opera used, but the
response appears to be complete.
Unfortunately Privoxy's log buffer is smaller than the receive buffer so
full receive buffers aren't logged completely in which case the received
data can't be reconstructed.
Can you still reproduce the problem if you enable +prevent-compression? A
log file without compressed content would probably be easier to analyze.
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2012-01-17 11:31
Message:
I never use turbo mode
----------------------------------------------------------------------
Comment By: Lee (ler762)
Date: 2012-01-15 12:59
Message:
Try disabling turbo mode in Opera.
I installed Opera and, with extremely limited testing, it worked fine until
I enabled "turbo mode". At which point it broke.
Changing the privoxy config to
keep-alive-timeout 0
helped some, but did NOT fix all of the problems.
I had wireshark capturing traffic during my testing. I thought I'd figured
out the problem but now I'm not so sure :( I need to look at the packet
traces again.
Lee
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2012-01-14 06:15
Message:
I cannot see "add a file" so i upload another log file (with same debugs
enabled) here:
http://www.sendspace.com/file/n4v8k2
Another 2 thing:
1. I have never page truncation on https
2. If i have page truncation on a site and the i use a cgi proxy to go to
the same page i have no page truncation
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2012-01-13 14:23
Message:
Downloading the file from zshare.net doesn't work for me.
The download server doesn't even bother to return HTTP headers.
In theory you should be able to upload a file through the "Add a file"
section on this page.
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2012-01-13 03:30
Message:
Page truncation on mashable.com with Opera 11.60 + Privoxy 3.0.19 and
socks4a + Tor
Log file with this debug enabled:
debug 1
debug 2
debug 4
debug 8
debug 64
debug 128
debug 256
debug 512
debug 1024
debug 4096
debug 8192
debug 32768
Log file is here:
http://www.zshare.net/download/986014474c9073b3/
How can i attach log file in this discussion?
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2012-01-01 10:09
Message:
Yes, i use Opera 11.60 and the problem is always on same site (i dont know
why)
After the upgrade to 3.0.19 and a quick surf seem that the problem is
socks5 + Tor (so Privoxy 3.0.18 for bug in socks4a) and NOT socks4a + Tor
(so Privoxy 3.0.19)
But i need more test (surf to more internet pages and try to use socks5 on
Privoxy 3.0.19)
If i have other page truncation i post a log
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2011-12-31 05:02
Message:
qqqqqw, are you using Opera as well?
Is the problem easy to reproduce?
Can you provide a logfile with the usual debug directives and additionally
"debug 16" and "debug 32768"?
It would be especially useful to additionally get a pcap file that shows
the packages Privoxy was receiving and sending.
I think it's unlikely that upgrading to 3.0.19 makes a difference. You
could try the recently uploaded privoxy-3.0.19-ipv4only.zip as it uses
different code paths, but it probably will not make a difference either.
----------------------------------------------------------------------
Comment By: Kevin (photo2110)
Date: 2011-12-29 11:58
Message:
Sorry I've not replied sooner. Xmas break took priority. I put in a section
to downgrade the NYTimes, and couldn't get any more issues with that site.
So, I put in a downgrade for all of .com, and haven't seen anymore short
pages on a .com site.
I'll have to try the new .19 version.
----------------------------------------------------------------------
Comment By: qqqqqw (qqqqqw)
Date: 2011-12-29 11:53
Message:
I have the same problem.
Initially i think that this problem is related with socks4a/5 bug, but
after read this i'm not so sure.
I have page truncation with Privoxy 3.0.18, not with 3.0.17.
Now i try with 3.0.19
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2011-12-23 11:21
Message:
Thanks.
I can't see any obvious problems in the log. Please additionally add "debug
16" so the data written to the network is included as well.
Does it make a difference if you add the following to one of your action
files?:
{+downgrade-http-version}
# Sticky Actions = +downgrade-http-version
# URL = http://www.nytimes.com/pages/travel/index.html
.nytimes.com/
----------------------------------------------------------------------
Comment By: Kevin (photo2110)
Date: 2011-12-23 10:06
Message:
New log file added with debug 32768 turned on. Truncated page is
http://www.nytimes.com/pages/travel/index.html which dies at:
<h6 class="kicker">Practical Traveler</h6>
<div class="thumbnail">
<a
href="http://travel.nytimes.com/2011/12/25/travel/apps-to-keep-children-happy.html?ref=travel"><img
src="http://graphics8.nytimes.com/images/2011/12/25/travel/25PRAC/25PRAC-thumbStandard-v2.jpg"
width="75" height="75" alt="" border="0" /></a>
</div>
<h3><a href="http://trav
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2011-12-23 09:41
Message:
Please additionally add "debug 32768" so we can see the data read from the
network.
----------------------------------------------------------------------
Comment By: Kevin (photo2110)
Date: 2011-12-23 09:23
Message:
I've attached a log file for a truncated page.
http://www.nytimes.com/interactive/2011/12/22/magazine/the-lives-they-lived.html?hp#view=dennis_ritchie
Opera's source view goes along until:
"100000001233615":{count:93, commentsEnabled:false},
"100000001233630":{count:57, commentsEnabled:false},
"100000001233632":{count:318, commentsEnabled:false},
"100000001233655":{count:441, commentsEnabled:fa
and we see it just stops in the middle of the word false.
I'm apparently not understanding the log file, it looks like it grabbed the
page twice, with the second grab bad.
----------------------------------------------------------------------
Comment By: Fabian Keil (fabiankeil)
Date: 2011-12-23 08:36
Message:
Thanks for the report.
Please reproduce the problem after enabling logging as described at
http://www.privoxy.org/user-manual/contact.html#CONTACT-BUGS and provide a
log file.
If you can make the pcap file created with Wireshark available, that may be
useful, too.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=111118&aid=3464439&group_id=11118
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Ijbswa-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ijbswa-developers