RE: Re: compressed data from an asynchronous socket !!

Becareful with data compressors: there may still exist some bytes left in
the current compressor state, which won't be encoded in its output if you
don't flush it; this may be caused by the fact that these final letters can
still have several alternate encodings depending on the input bytes after
them.

Flushing a data compressor (or closing it) is normally required if you want
to make sure that all the trailing bytes are sent. But it may impact the
compression ratio if you flush the output too often:

This depends on the compression algorithm used, the "deflate" compression
method normally does leave any pending output in its internal state in most
implementations, as Lempel-Ziv based methods are normally encoding
***RANGES*** of input bytes with a single code, something that is not
possible without keeping pending bytes in the current state; if it did not
do that, then each input byte would require outputting at least some bits in
the output, and the compression would be mostly without any effect.

In other words, if you intend to predict how much bytes a short Gnutella
message will take in the deflated stream, in order to see if it will fit
without a single datagram (and see if another shorter message could fit
there), chances are high that your output will be suboptimal (in addition,
it may change the expected output order of messages).

My opinion is that, in the case where you want to make sure that a
compressed data stream will fit in a single transmission datagram, to save a
duplicate copy of the current compression state just before trying to encode
an additional message, and flushing it. After flushing, if the resulting
output is too long, then revert back the compressor to its initial state,
flush it, send the output datagram, and then reencode the input message that
did not fit within the datagram you have just sent (and leave it there in
the compressor state, before proceeding with further messages):

You may want to do this strategy in order to facilitate the management of
data transmission and recoverability over a non reliable transmission
channel such as UDP, without depending on IP fragmentation whose
transmission is even less reliable, but you should know that it will reduce
the benefit of data compression. You won't ever need such strategy on
reliable channels such as TCP (or TCP emulation over UDP, like in the
protocol used in LimeWire to allow "firewall-to-firewall" transmission when
TCP does not work without a dedicated port mapping)

> -----Message d'origine-----
> De : [email protected] [mailto:[email protected]] De la part
> de mark W
> Envoyé : mercredi 18 juillet 2007 08:21
> À : [email protected]
> Objet : Re: [the_gdf] Re: compressed data from an asynchronous socket !!
> 
> 
> Yeah, it's a continuous stream of data and each received block is related
> to earlier received blocks.
> 
> One probable solution in mind is :
> Store the starting compressed chunk (beginning of stream) into a temp
> buffer. Whenever next compressed data comes, add this starting buffer to
> the received data and then pass it to inflater:
> 
> InflaterInputStream str = new InflaterInputStream (memStream) ;
> int i = str.Read(OutBuffer, 0, BytesInBuffer) ;
> 
> The buffer inflated successfully , extract inflated data. If decompressed
> data of starting chunk is processed , remove it and pass the remaining
> data to process.
> 
> Do you think differently or it should work?
> 
> Thanks in advance,
> *mark
> 
> 
> ----- Original Message ----
> From: "[email protected]" <[email protected]>
> To: [email protected]
> Sent: Wednesday, July 18, 2007 10:59:09 AM
> Subject: [the_gdf] Re: compressed data from an asynchronous socket !!
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>             Quoting mark W <aus_waugh@yahoo. com> from ml.gnutella. dev-
> forum:
> 
> :If I buffer all asynchronous data from the session and pass synchronously
> it to
> 
> :SharpZipLib inflater,e.g.
> 
> :
> 
> :InflaterInputStrea m str = new InflaterInputStream (memStream) ;
> 
> :int i = str.Read(OutBuffer, 0, BytesInBuffer) ;
> 
> :
> 
> :but due to the permanent connection, eventually I will eat up all my
> memory.
> 
> 
> 
> Of course.  The total amount of data sent during a "session" is
> potentially
> 
> infinite (say, very large, larger than any amount of RAM you can come up
> with).
> 
> 
> 
> What does this mean?  It means you're approaching the problem from the
> 
> wrong angle.
> 
> 
> 
> :In another way , if I pass asynchronous buffer to above inflater(whenever
> data
> 
> :arrived at socket), it will throw an exception of invalid header or
> checksum.
> 
> 
> 
> Hehe.  This means you still haven't figured how to do this, despite the
> 
> hint I gave you earlier (remember: it's a continuous stream of data).
> 
> 
> 
> You can't go forward until you figure this one out.  And note the fix is
> 
> trivial (I saw the problem immediately in the code snippet you posted) but
> 
> not being able to see it means you won't be able to continue your task.
> 
> 
> 
> Go back to the eariler problem and make it work first.  Then you'll see
> 
> that your current problem will be fixed as well and everything should be
> 
> crystal clear then.
> 
> 
> 
> Raphael
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> <!--
> 
> #ygrp-mlmsg {font-size:13px;font-family:arial, helvetica, clean, sans-
> serif;}
> #ygrp-mlmsg table {font-size:inherit;font:100%;}
> #ygrp-mlmsg select, input, textarea {font:99% arial, helvetica, clean,
> sans-serif;}
> #ygrp-mlmsg pre, code {font:115% monospace;}
> #ygrp-mlmsg * {line-height:1.22em;}
> #ygrp-text{
> font-family:Georgia;
> }
> #ygrp-text p{
> margin:0 0 1em 0;}
> #ygrp-tpmsgs{
> font-family:Arial;
> clear:both;}
> #ygrp-vitnav{
> padding-top:10px;font-family:Verdana;font-size:77%;margin:0;}
> #ygrp-vitnav a{
> padding:0 1px;}
> #ygrp-actbar{
> clear:both;margin:25px 0;white-space:nowrap;color:#666;text-align:right;}
> #ygrp-actbar .left{
> float:left;white-space:nowrap;}
> .bld{font-weight:bold;}
> #ygrp-grft{
> font-family:Verdana;font-size:77%;padding:15px 0;}
> #ygrp-ft{
> font-family:verdana;font-size:77%;border-top:1px solid #666;
> padding:5px 0;
> }
> #ygrp-mlmsg #logo{
> padding-bottom:10px;}
> 
> #ygrp-vital{
> background-color:#e0ecee;margin-bottom:20px;padding:2px 0 8px 8px;}
> #ygrp-vital #vithd{
> font-size:77%;font-family:Verdana;font-weight:bold;color:#333;text-
> transform:uppercase;}
> #ygrp-vital ul{
> padding:0;margin:2px 0;}
> #ygrp-vital ul li{
> list-style-type:none;clear:both;border:1px solid #e0ecee;
> }
> #ygrp-vital ul li .ct{
> font-weight:bold;color:#ff7900;float:right;width:2em;text-
> align:right;padding-right:.5em;}
> #ygrp-vital ul li .cat{
> font-weight:bold;}
> #ygrp-vital a {
> text-decoration:none;}
> 
> #ygrp-vital a:hover{
> text-decoration:underline;}
> 
> #ygrp-sponsor #hd{
> color:#999;font-size:77%;}
> #ygrp-sponsor #ov{
> padding:6px 13px;background-color:#e0ecee;margin-bottom:20px;}
> #ygrp-sponsor #ov ul{
> padding:0 0 0 8px;margin:0;}
> #ygrp-sponsor #ov li{
> list-style-type:square;padding:6px 0;font-size:77%;}
> #ygrp-sponsor #ov li a{
> text-decoration:none;font-size:130%;}
> #ygrp-sponsor #nc {
> background-color:#eee;margin-bottom:20px;padding:0 8px;}
> #ygrp-sponsor .ad{
> padding:8px 0;}
> #ygrp-sponsor .ad #hd1{
> font-family:Arial;font-weight:bold;color:#628c2a;font-size:100%;line-
> height:122%;}
> #ygrp-sponsor .ad a{
> text-decoration:none;}
> #ygrp-sponsor .ad a:hover{
> text-decoration:underline;}
> #ygrp-sponsor .ad p{
> margin:0;}
> o {font-size:0;}
> .MsoNormal {
> margin:0 0 0 0;}
> #ygrp-text tt{
> font-size:120%;}
> blockquote{margin:0 0 0 4px;}
> .replbq {margin:4;}
> -->
> 
> 
> 
> 
> 
> 
> 
> Send instant messages to your online friends http://uk.messenger.yahoo.com
> 
> [Non-text portions of this message have been removed]
> 
> 
> 
> 
> Yahoo! Groups Links
> 
> 
> 
>