Re: [viewvc-dev] Re: [viewvc-users] Don't display binary files

JJ <[email protected]> Thu, 20 Nov 2008 11:28:09 -0600
Newsgroups gmane.comp.version-control.cvs.viewcvs.devel
Message-ID <[email protected]>
On Thu, Nov 20, 2008 at 10:28 AM, C. Michael Pilato <[email protected]>wrote:

> JJ wrote:
> >     I like this, but I wouldn't add a new config file.  I'd just add a
> new
> >     configuration option that follows the format of our existing
> multi-value
> >     options.  And I'd let the values take wildcards (text/*,
> >     application/*-xml,
> >     etc.), too, so you wouldn't have to specify every single little
> thing.
> >
> > Patch attached.  I also need to clean up the default templates
> > to respect prefer_markup values.
>
> Nice patch, JJ.  It needs a couple of tweaks, though:
>
>   - the viewvc.conf.dist file should, if presenting an uncommented option,
>     present the same option value that is used as the default in
>     lib/config.py.  In this case, I think you got lib/config.py right
>     (in that we only want to assume that text/* is text by default),
>     but the viewvc.conf.dist file is wrong.  Just make the value in
>     that file "text/*", and maybe add a second, commented-out example
>     that shows some of the other types folks might want in there.
>
>   - I don't like the "convert to a valid regexp" logic.  If we're going
>     to use regexps, let's expose that directly to users.  If we really
>     only care about '*' as a wildcard, then maybe the 'fnmatch' module
>     is what you want to use for comparisons (since mime types are
>     slash-delimited like paths are).
>
> --
> C. Michael Pilato <[email protected]>
> CollabNet   <>   www.collab.net   <>   Distributed Development On Demand
>

How about this patch?  It includes the following changes.

* text_mime_types - as described before, but matching is now done with
fnmatch, and the config.py defaults and viewvc.conf match.

* text_file_extensions - List of file extensions to for text files.  Mime
types really only make sense in the browser world.  Not all files are meant
to view in a browser though, so making people add an entry into a mime.types
file for them doesn't make sense.  This also uses fnmatch, and by default it
is commented out, though it contains an example so people know how to format
it.

* prefer_markup_default - specifies what should be done if we can determine
if it is text file or viewable image based on mime type or file extension.
Default to 0 since that seems like the safest thing to do, though you could
change it if you prefer.

* Finished cleaning up templates to properly display links based on
prefer_markup value.  They weren't doing so before.

* Made changes.prefer_markup available in revisions.ezt, since it wasn't
before and should have been.

This is working nicely for me.  I hope you're okay with the changes.  :-)

JJ

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
patch.txt (text/plain, 3.8 KB)
Index: viewvc.conf.dist
===================================================================
--- viewvc.conf.dist    (revision 2047)
+++ viewvc.conf.dist    (working copy)
@@ -269,6 +269,13 @@
 # Possible values: "tar", "annotate", "co", "markup", "roots"
 allowed_views = markup, annotate, roots

+# Don't use the svn:mime-type property to determine how to display a
+# file in the markup view.  This is helpful when, for example, an image
+# has svn:mime-type set to application/octet-stream.  If ViewVC tries
+# to display the image using the mime type application/octet-stream,
+# the image will look unrecognizeable.
+ignore_svn_mimetype = 0
+
 # authorizer: The name of the ViewVC authorizer plugin to use when
 # authorizing access to repository contents.  This value must be the
 # name of a Python module addressable as vcauth.MODULENAME (most
Index: templates/file.ezt
===================================================================
--- templates/file.ezt  (revision 2047)
+++ templates/file.ezt  (working copy)
@@ -102,5 +102,12 @@
 [end]
 [end]

+[if-any is_viewable]
+[else]
+<h3>
+<em>Non-viewable file contents hidden</em>
+</h3>
+[end]
+
 [include "include/props.ezt"]
 [include "include/footer.ezt"]
Index: lib/viewvc.py
===================================================================
--- lib/viewvc.py       (revision 2047)
+++ lib/viewvc.py       (working copy)
@@ -972,6 +972,24 @@
 def is_text(mime_type):
   return not mime_type or mime_type[:5] == 'text/'

+text_characters = "".join(map(chr, range(32, 127)) + list("\n\r\t\b"))
+_null_trans = string.maketrans("", "")
+
+def istext(s):
+  if "\0" in s:
+    return 0
+
+  if not s:  # Empty files are considered text
+    return 1
+
+  # Get the non-text characters (maps a character to itself then
+  # use the 'remove' option to get rid of the text characters.)
+  t = s.translate(_null_trans, text_characters)
+
+  # If more than 30% non-text characters, then
+  # this is considered a binary file
+  return float(len(t))/len(s) <= 0.30
+
 def is_cvsroot_path(roottype, path_parts):
   return roottype == 'cvs' and path_parts and path_parts[0] == 'CVSROOT'

@@ -1465,7 +1483,8 @@
   mime_type = None
   if not path_parts:
     return None
-  if request.roottype == 'svn':
+  if request.roottype == 'svn' and \
+     not request.cfg.options.ignore_svn_mimetype:
     try:
       itemprops = request.repos.itemprops(path_parts, rev)
       mime_type = itemprops.get('svn:mime-type')
@@ -1481,6 +1500,7 @@
   lines = fp = image_src_href = None
   annotation = None
   revision = None
+  is_viewable = None
   mime_type = calculate_mime_type(request, path, rev)

   # Is this a viewable image type?
@@ -1493,6 +1513,7 @@
     annotation = 'binary'
     image_src_href = request.get_url(view_func=view_checkout,
                                      params={'revision': rev}, escape=1)
+    is_viewable = True

   # Not a viewable image.
   else:
@@ -1513,10 +1534,18 @@
     if check_freshness(request, None, revision, weak=1):
       fp.close()
       return
-    lines = markup_stream_pygments(request, cfg, blame_source, fp,
-                                   path[-1], mime_type)
-    fp.close()

+    # Make sure its a text file.
+    if istext(fp.read(512)):
+      is_viewable = True
+      fp.close()
+      fp, revision = request.repos.openfile(path, rev)
+      lines = markup_stream_pygments(request, cfg, blame_source, fp,
+                                     path[-1], mime_type)
+      fp.close()
+    else:
+      is_viewable = False
+
   data = common_template_data(request, revision)
   data.update({
     'mime_type' : mime_type,
@@ -1538,6 +1567,7 @@
     'lines' : lines,
     'properties' : get_itemprops(request, path, rev),
     'annotation' : annotation,
+    'is_viewable' : ezt.boolean(is_viewable),
     })

   if cfg.options.show_log_in_markup: