Re: Fetching short text snippet for mails

On Wed, Dec 3, 2014, at 04:49 AM, Arnt Gulbrandsen wrote:
> Does it matter?
>
> As I see it, your proposal aims to solve the download problem. (Which
> is a prioblem. If it takes several roundtrips from when a mobile
> client has put up a notification until it's ready to ship a snippet,
> that's often >1s, painfully long on a mobile.) You don't solve the
> problem of extracting the right text, but rather presume that the
> client has logic to solve that.
>
> Instead of a dubious solution to two problems, you supply a good
> solution to one problem. Good, say I.

As someone who's done both ends of this - your solution isn't enough.
We now calculate an ANNOTATION on the server precisely because it's a
pain to fetch over IMAP.  I'm going to paste our "find the body part to
get preview from" code, so you can see how much complexity went into the
client to get efficient previews from the IMAP server.  And note, you
have to repeat this every time you lose your local cache.

There's plenty around this, but here's the preview and "has attachments"
finding bit.  Notice how it has to batch fetches by which bodystructure part
it wants to fetch the preview from:

    # Get bodystructure for messages we need bodystructure for {{{

    if (my @NeedBSUids = grep { $NeedMoreUids{$_}->{NeedBS} } keys %NeedMoreUids) {
      @NeedBSUids = sort { $a <=> $b } @NeedBSUids;

      # Hold UIDs that need each appropriate flag stored back
      my (@StoreHasAttUids, @StoreHasNoAttUids);

      my $IMAPBS = $Mailbox->fetch(\@NeedBSUids, "(bodystructure)")
        || die "IMAP fetch failed - $@";

      foreach my $Uid (@NeedBSUids) {

        # Get local references to data
        my $Row = $Uid2RowMap{$Uid};
        my $BS = ($IMAPBS->{$Uid} && $IMAPBS->{$Uid}->{bodystructure}) || next;

        # Store back 'hasatt' or 'hasnoatt' flag
        if ($NeedMoreUids{$Uid}->{NeedAttStore}) {

          # Quick heuristic to work out if has attachment
          my $HasAtt = $Self->_HasAttachment($BS);

          # Add id to appropriate list
          if ($HasAtt) {
            push @StoreHasAttUids, $Uid;
          } else {
            push @StoreHasNoAttUids, $Uid;
          }

          # Save new information back into row data
          $Row->{HasAtt} = $HasAtt;
        }

        # Fix 8-bit headers
        if ($NeedMoreUids{$Uid}->{NeedCharset}) {
          my $CharSet = lc($BS->{'Content-Type'}->{charset});

          # If no charset, or we're told it's us-ascii, (but we got here because there
          #  was 8-bit data in the header!), then assume users default charset
          $CharSet = $Self->{FallbackCharset} || 'iso-8859-1'
            if !$CharSet || $CharSet eq 'ascii' || $CharSet eq 'us-ascii';

          # For each 8-bit header, decode it with the charset we now think it's in
          my $Row = $Uid2RowMap{$Uid};
          for (grep { defined($_) && !is_utf8($_) && /[^\x20-\x7f]/ } @$Row{qw(To From Subject)}) {
            eval { $_ = decode($CharSet, $_); };
            # We decoded to utf-8, but there's still control chars? Strip them.
            s/[\x00-\x1f]//g if is_utf8($_) && /[\x00-\x1f]/;
          }
        }

        # Save copy of BS for body part below
        $Uid2BSMap{$Uid} = $BS;
      }

      # Save hasatt/hasnoatt flags back
      $Mailbox->store(\@StoreHasAttUids, '+flags.silent', "(hasatt)")
        if @StoreHasAttUids;
      $Mailbox->store(\@StoreHasNoAttUids, '+flags.silent', "(hasnoatt)")
        if @StoreHasNoAttUids;
    }

    # }}}

    # Get preview text for messages we need preview text for {{{

    if (my @NeedBodyUids = grep { $NeedMoreUids{$_}->{NeedPeek} } keys %NeedMoreUids) {
      @NeedBodyUids = sort { $a <=> $b } @NeedBodyUids;

      # Need to fetch by partnum. To fetch multiple at once, map partnum -> uids
      my (%PartNumToUidMap, %UidToPartInfoMap);
      foreach my $Uid (@NeedBodyUids) {

        # Use find_message to find the body part for this message
        #  (we stored the actual bodystructure for the mesage into
        #  %NeedBSUids above)
        my $Parts = find_message($Uid2BSMap{$Uid});
        my ($Part, $Type) = ($Parts->{text}, 'text');

        # If no text part, but an HTML part, use that
        if (!$Part && exists $Parts->{html}) {
          ($Part, $Type) = ($Parts->{html}, 'html');
        } elsif ($Part) {
          $Type = 'rtf' if lc $Part->{'MIME-Type'} eq 'text/enriched';
        }

        my $Partnum = $Part->{'IMAP-Partnum'} || '';

        push @{$PartNumToUidMap{$Partnum}}, $Uid;
        $UidToPartInfoMap{$Uid} = [ $Part, $Type ];
      }

      # Now iterate by partnums
      foreach my $Partnum (keys %PartNumToUidMap) {
        my @Uids = @{$PartNumToUidMap{$Partnum}};

        # Now get the data for all uids with body in this partnum
        my $BodyTexts = $Mailbox->fetch(\@Uids, "binary.peek[$Partnum]<0.2000>");

        # Bad transfer encodings cause error return, but there might be partial data
        if (!$BodyTexts || !ref($BodyTexts)) {
          $BodyTexts = delete $Mailbox->{PartialResp};
        }

        if (!$BodyTexts || !ref($BodyTexts)) {
          warn "fetch binary for part=$Partnum for uids=@Uids for $Self->{UserId} in $FolderDet->{FolderName} failed: $@";
        }

        # And process each of them
        foreach my $Uid (@Uids) {
          my ($Part, $Type) = @{$UidToPartInfoMap{$Uid}};
          my $BodyText = $BodyTexts->{$Uid} && $BodyTexts->{$Uid}->{binary};
          $BodyText = 'Content unavailable for preview. Probably invalid encoding'
            if !defined $BodyText;

          # Check if we need to decode
          # If any non-ascii chars, we have to decode in some way...
          if ($BodyText =~ /[^\x20-\x7f\r\n]/) {

            # Get and normalise charset
            my $Charset = lc($Part->{'Content-Type'}->{charset});
            $Charset = $Self->{FallbackCharset} || 'iso-8859-1'
              if !$Charset || $Charset eq 'ascii' || $Charset eq 'us-ascii';
            $Charset =~ s/^(iso)[\-_]?(\d+)[\-_](\d+)[\-_]?\w*/$1-$2-$3/i;

            eval { $BodyText = decode($Charset, $BodyText); };
          }

          my @Lines = CleanBodyText($BodyText, $Type);

          # Show the first 3 lines or so
          if ($PeekOutput == 1) {

            $BodyText = substr(join("\n", @Lines[0 .. (@Lines < 3 ? @Lines-1 : 2)]), 0, 1000) . "\n";

          } elsif ($PeekOutput == 2) {
            # Show only 1 line with <CR>'s replaced with \

            # Only first 8 lines and/or 200 chars
            $BodyText = substr(join($Self->{PeekSep} // " \\ ", @Lines[0 .. (@Lines < 8 ? @Lines-1 : 7)]), 0, 200) . "\n";
          } 

          # Store back into row data
          $Uid2RowMap{$Uid}->{BodyPeek} = $BodyText;
          $Uid2RowMap{$Uid}->{BodyType} = $Type;
        }
      }

    }


...

And here's the annotator that runs inside our server now, which calculates image sizes for attached images, a preview, and a bunch of things calculated from headers.

sub MakePreview {
    my ($message, $Parts) = @_;

    my ($Part, $Type) = ($Parts->{text}, 'text');

    # If no text part, but an HTML part, use that
    if (!$Part && exists $Parts->{html}) {
      ($Part, $Type) = ($Parts->{html}, 'html');
    } elsif ($Part) {
      $Type = 'rtf' if lc $Part->{'MIME-Type'} eq 'text/enriched';
    }

    my $Preview = "";
    if ($Part) {
      my $CharSet = lc($Part->{'Content-Type'}->{charset} // '');

      my $Content = $message->read_part_content($Part, MAX_CONTENT);
      $Content = MEDecodeBody($Content, $CharSet);

      my @Lines = clean_body_text($Content, $Type);

      # Want at least 3 lines or 160 characters of preview
      my ($Pos, $Length) = (0, 0);
      while ($Length < 160 && $Pos < @Lines) {
	$Length += length($Lines[$Pos++]) + 1;
      }
      $Pos = 3 if $Pos < 3;
      splice(@Lines, $Pos) if $Pos < @Lines;

      $Preview = join "\n", @Lines;
    }

    return Encode::encode_utf8($Preview);
}

And the client just does this:

    # Get previews for messages we need previews for {{{

    if (my @NeedPeekUids = grep { $NeedMoreUids{$_}->{NeedPeek} } @NeedMoreUids) {

      warn "getting preview for uids=" . join(" ", @NeedPeekUids) if DEBUG;

      my @WantItems = ("annotation", [ "/vendor/messagingengine.com/preview", "value.shared" ]);
      my $IMAPAnnot = $Mailbox->fetch(\@NeedPeekUids, \@WantItems)
        || die "IMAP fetch failed - $@ - items=" . join(" ", @WantItems) . ", uids=" . join(",", @NeedPeekUids);

      my $PeekSep = $Opts{PeekSep} // $Self->{PeekSep} // " \\ ";

      # Now extra data we need
      foreach my $Uid (@NeedPeekUids) {

        my $Row = $Uid2RowMap{$Uid};
	$Row->{BodyPeek} = '';

        # Message can always disappear, handle sanely
        my $MsgAnnot = $IMAPAnnot->{$Uid} // next;
	$MsgAnnot = $MsgAnnot->{annotation} // next;
	$MsgAnnot = $MsgAnnot->{"/vendor/messagingengine.com/preview"}->{"value.shared"} // next;

        # Data is utf-8 octets, \n separated lines
	my @Lines = split /\n/, decode_utf8($MsgAnnot);

        my $BodyText = '';

	# 3 lines
	if ($PeekOutput == 1) {

          splice(@Lines, 3) if @Lines > 3;
	  $BodyText = substr(join("\n", @Lines), 0, 1000) . "\n";

	# 1 line with <CR>'s replaced with \
	} elsif ($PeekOutput == 2) {

	  # Only first 8 lines and/or 130 chars
	  splice(@Lines, 8) if @Lines > 8;
	  $BodyText = substr(join($PeekSep, @Lines), 0, 130);
	}

	# Store back into row data
	$Row->{BodyPeek} = $BodyText;
      }
    }


Significantly simpler to implement on the client, and fewer round trips as well.  Best of all, you calculate it ONCE and store it on the server, rather than having to either cache every single one you ever see, or potentially have to re-calculate it every time you go back to a mailbox you haven't seen for a while.

Bron.

-- 
  Bron Gondwana
  [email protected]

_______________________________________________
imapext mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/imapext