Re: Spam attack, observations, how to repair

"John P. Rouillard" <[email protected]> Wed, 23 Jul 2014 22:51:38 -0400
Newsgroups	gmane.comp.bug-tracking.roundup.user
Message-ID	<[email protected]>
Hello:

In message
<CAMEZBD+E_Q8YjnkoZbvwWzFkHPAQgHLLhShx4hadFT5Z9AVAAA@mail.gmail.com> ,
Kay Hayen writes:

>For the links, "rel=nofollow" already is a good think, and robots.txt
>ought to be efficient measure. I was thinking of ways how to
>discourage spammers even further.
>
>And I found
>https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?hl=de
>
>First: HTML attachment files if allowed and to be displayed could
>have <meta name="robots" context="noindex"> as that is probably very
>likely to prevent any benefit from making such attachments.

You can modify page.html and files.item.html to do this easily.  Try
putting:

   <tal:block metal:define-slot="metas" tal:replace="nothing"> </tal:block>

after the <head> element in the icing block of html/page.html in your
tracker directory.

Then add:

  <tal:span metal:fill-slot="metas" >
    <metaname="robots" content="noindex">
  </tal:span>

to file.item.html after </title>. This should add the meta header you
seek to the .../file3 , file4, fileN pages.

Now to get noindex to the download page is trickier. Since the
download page may not be html, we can't use a meta tag. 

>Second: As a server, roundup should provide "X-Robots-Tag: noindex" to any
>/filexxx and /userxxx URLs at least.

As you noted:

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

documents the X-Robots-Tag header that can be used with the values:

  noindex, nofollow, none (short way to say both noindex,nofollow)

It looks like the right place to put that is in roundup/cgi/client.py

   def _serve_file(self, lmt, mime_type, content=None, filename=None):
        """ guts of serve_file() and serve_static_file()
        """

        # spit out headers
        self.additional_headers['Content-Type'] = mime_type
        self.additional_headers['Last-Modified'] = rfc822.formatdate(lmt)
        self.additional_headers['X-Robots-Tag'] = "nofollow, noindex"

but hard coding it there seems wrong. I am not quite sure how to
control it. I can see the use case for allowing some objects to be
traversed/indexed and others to not be indexed.

At least initially I am thinking of a single tracker config variable
similar to allow_html_file. Not sure what to call it, maybe:

  allow_download_crawling = (yes|no)
or
  allow_download_indexing = (yes|no)
or
  allow_download_following = (yes|no)

I called it ...download..., since this would apply to anything that is
downloadable (I don't know of anything other than files that are
downloadable, but the comments in the function indicate it can be used
for other content.).

As you can tell, I really don't know what a good name for this would
be. Maybe in the future the config variable would be expanded to:

 allow_download_.... =  file:no resume:yes picture:yes

or for more fine grained control:

   allow_download_.... =  file:n resume:if picture:i

if you have data that contains random files (no indexing or
following), resumes (indexed and followed) and pictures (index only,
no follow).

I guess from a spam point of view, if any of them were
indexable/followable it would be an attack point, but if it were used
on an intranet that has a google or other search appliance, I can see
this being useful.

>Oh, and another aside, while at it, Python files attached to the database,
>are a text/plain, I would rather see mimetypes module employed:
>
>>>> import mimetypes
>>>> mimetypes.guess_type("test.py")
>('text/x-python', None)
>
>That could enable syntax highlighting in clients.

I think this is a job for a file creation auditor. The auditor can get
the name (in newvalue['name']) of the file and change mime types by
setting newvalue['type'].

(Since mimetypes.guess_type just uses the name, this should work. If
you use something that needs to look at the file contents to devine
the type, you ccan use the data in newvalues['content'].)

The function:

    def serve_file(self, designator, dre=re.compile(r'([^\d]+)(\d+)')):
        """ Serve the file from the content property of the designated item.

in client.py simply returns the type field for the designator passed
in. So if that's set right, it should be reported correctly.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds