Re: Privoxy-users Digest, Vol 41, Issue 7
Robert Klemme <[email protected]> Wed, 22 Jul 2020 22:55:51 -0500
| Newsgroups | gmane.comp.web.privoxy.user |
|---|---|
| Message-ID | <CABBjzH7JkGJuit__mQcdoS0m4NuwKt3y6SR6MtV0tmVaaCtOMg@mail.gmail.com> |
I don't know the status of this project but it seems like the privoxy-filter-test might be helpful in this case. If it is dormant, maybe we should resurrect development? https://www.fabiankeil.de/sourcecode/pft/ On Wed, Jul 22, 2020 at 10:15 PM <[email protected]> wrote: > Send Privoxy-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.privoxy.org/mailman/listinfo/privoxy-users > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Privoxy-users digest..." > > > Today's Topics: > > 1. external-filter example (U.Mutlu) > 2. Re: external-filter example (U.Mutlu) > 3. Re: external-filter example (Lee) > 4. Re: external-filter example (U.Mutlu) > 5. Re: external-filter example (U.Mutlu) > 6. Re: external-filter example (Lee) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 22 Jul 2020 19:03:04 +0200 > From: "U.Mutlu" <[email protected]> > To: [email protected] > Subject: [Privoxy-users] external-filter example > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Hi, > as a newbie to privoxy I've built the developer version (v3.0.29) with > ./configure --with-mbedtls and installed it. It works. > > I'm now trying to create an external-filter for experimenting. I did these > steps: > > config: > temporary-directory ./myTmp > (and even created that dir manually in the config-dir and set the > ownership > appropriately) > > user.action: > myfilters = +filter{html-annoyances} +filter{js-annoyances} > +filter{all-popups}\ > +filter{webbugs} +filter{banners-by-size}\ > +external-filter{myTee} > > user.filter: > EXTERNAL-FILTER: myTee MY TEE TEST > /usr/bin/tee /tmp/myTee_Test.txt > > But privoxy won't start, writes this error to log: > 2020-07-22 18:26:29.533 7f6ca8cfe700 Fatal error: can't load actions file > '/usr/local/etc/privoxy/user.action': invalid alias line (64): myfilters = > +filter{html-annoyances} +filter{js-annoyances} +filter{all-popups} > +filter{webbugs} +filter{banners-by-size} > +external-filter{myTee} > > What's the reason, and how to fix it? Thx. > > Btw, is the word "myfilters" in the config files a hardcoded special word > or > is it just a user-defined one? > > > > > ------------------------------ > > Message: 2 > Date: Wed, 22 Jul 2020 19:39:04 +0200 > From: "U.Mutlu" <[email protected]> > To: [email protected] > Subject: Re: [Privoxy-users] external-filter example > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Update: sorry, had forgotten to build with "--enable-external-filters". > After rebuilding and reinstalling it now starts ok, > but the external-filter does not get invoked :-( > How to fix that? > > > U.Mutlu wrote on 07/22/2020 07:03 PM: > > Hi, > > as a newbie to privoxy I've built the developer version (v3.0.29) with > > ./configure --with-mbedtls and installed it. It works. > > > > I'm now trying to create an external-filter for experimenting. I did > these steps: > > > > config: > > temporary-directory ./myTmp > > (and even created that dir manually in the config-dir and set the > ownership > > appropriately) > > > > user.action: > > myfilters = +filter{html-annoyances} +filter{js-annoyances} > > +filter{all-popups}\ > > +filter{webbugs} +filter{banners-by-size}\ > > +external-filter{myTee} > > > > user.filter: > > EXTERNAL-FILTER: myTee MY TEE TEST > > /usr/bin/tee /tmp/myTee_Test.txt > > > > But privoxy won't start, writes this error to log: > > 2020-07-22 18:26:29.533 7f6ca8cfe700 Fatal error: can't load actions file > > '/usr/local/etc/privoxy/user.action': invalid alias line (64): myfilters > = > > +filter{html-annoyances} +filter{js-annoyances} +filter{all-popups} > > +filter{webbugs} +filter{banners-by-size} > +external-filter{myTee} > > > > What's the reason, and how to fix it? Thx. > > > > Btw, is the word "myfilters" in the config files a hardcoded special > word or > > is it just a user-defined one? > > > > ------------------------------ > > Message: 3 > Date: Wed, 22 Jul 2020 15:29:07 -0400 > From: Lee <[email protected]> > To: "U.Mutlu" <[email protected]> > Cc: [email protected] > Subject: Re: [Privoxy-users] external-filter example > Message-ID: > < > CAD8GWsttbjoSRosRAWRkPNtCOU353y92z+eKJzOfXnptHbF-ZA@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On 7/22/20, U.Mutlu <[email protected]> wrote: > > Update: sorry, had forgotten to build with "--enable-external-filters". > > After rebuilding and reinstalling it now starts ok, > > but the external-filter does not get invoked :-( > > How to fix that? > > I've never tried using external filters, so I'm not going to be a > whole lot of help but > https://www.privoxy.org/user-manual/actions-file.html#FILTER > (Note: as of version 3.0.3 plain text documents are exempted from > filtering, because web servers often use the text/plain MIME type for > all files whose type they don't know.) > > Have you added > debug 64 # debug regular expression filters > to the config file to see if privoxy tries to run the filter? > > https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834 > > If the filter isn't being called, maybe you need > https://www.privoxy.org/user-manual/filter-file.html > Enabled content filters are applied to any content whose "Content > Type" header is recognised as a sign of text-based content, with the > exception of text/plain. Use the force-text-mode action to also filter > other content. > > > U.Mutlu wrote on 07/22/2020 07:03 PM: > >> Hi, > <.. snip ..> > >> I'm now trying to create an external-filter for experimenting. I did > these > >> steps: > <.. snip ..> > >> user.action: > >> myfilters = +filter{html-annoyances} +filter{js-annoyances} > >> +filter{all-popups}\ > >> +filter{webbugs} +filter{banners-by-size}\ > >> +external-filter{myTee} > <.. snip ..> > >> Btw, is the word "myfilters" in the config files a hardcoded special > word > >> or is it just a user-defined one? > > It's a user-defined alias > https://www.privoxy.org/user-manual/actions-file.html#ALIASES > > Lee > > > ------------------------------ > > Message: 4 > Date: Wed, 22 Jul 2020 23:06:46 +0200 > From: "U.Mutlu" <[email protected]> > To: [email protected] > Subject: Re: [Privoxy-users] external-filter example > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Lee wrote on 07/22/2020 09:29 PM: > > On 7/22/20, U.Mutlu <[email protected]> wrote: > >> Update: sorry, had forgotten to build with "--enable-external-filters". > >> After rebuilding and reinstalling it now starts ok, > >> but the external-filter does not get invoked :-( > >> How to fix that? > > > > Have you added > > debug 64 # debug regular expression filters > > to the config file to see if privoxy tries to run the filter? > > > https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834 > > Thx. Now tried with "debug 64" and it shows that it recognizes the said > external-filter, but IMHO the action for it is not executed: > > .... > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > "user-agent" > ("Tags the request with the complete User-Agent header.") > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job > 's@^User-Agent:.*@$0@i' to filter user-agent succeeded. > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > "referer" > ("Tags the request with the complete Referer header.") > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job > 's@^Referer:.*@$0@i' to filter referer succeeded. > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > "content-type" ("Tags the request with the content type declared by the > server.") > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > 's@^Content-Type:\s*([^;]+).*@$1@i' to filter content-type succeeded. > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control > headers.") > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded. > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > "privoxy-control" ("Removes X-Privoxy-Control headers.") > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded. > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control > headers.") > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded. > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > "privoxy-control" ("Removes X-Privoxy-Control headers.") > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded. > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter "myTee" > ("MY > TEE TEST") > ... > > There is no other log entry regarding this "myTee" filter. > Ie. it seems the action for it does not get executed. > > > If the filter isn't being called, maybe you need > > https://www.privoxy.org/user-manual/filter-file.html > > Enabled content filters are applied to any content whose "Content > > Type" header is recognised as a sign of text-based content, with the > > exception of text/plain. Use the force-text-mode action to also filter > > other content. > > But doing so seems to be dangerous according to the doc > https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE > It says: > "Warning - Think twice before activating this action. Filtering binary > data > with regular expressions can cause file damage." > > > I just want to get a copy of the whole HTML that privoxy gets from the > server, > either the unmodified raw data, or the filtered version after > privoxy applies the other defined filters. > > I think the HTML is always a text file, even if it contains binary data > like picture images etc. (they are somehow text encoded I think using > base64 encoding or so). Is this assumption wrong? > > If I read the doc right then the external-filter handler has to write > the received data either verbatim (ie. no filtering), or a filtered > version of the data, back to stdout, so privoxy can continue doing its job. > > A working example for such an external-filter would be helpful. > Unfortunately I couldn't find any on the web :-( > > > > > > ------------------------------ > > Message: 5 > Date: Thu, 23 Jul 2020 00:01:47 +0200 > From: "U.Mutlu" <[email protected]> > To: [email protected] > Subject: Re: [Privoxy-users] external-filter example > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > It's funny that nothing of default.action, user.action, and my.action > applies to the below URL [I'm using the default configs it has installed]. > (my.action and my.filter is just my experimental stuff with the said > "myTee" > external-filter (now moved from user.* to my.*)): > > http://config.privoxy.org/show-url-info?url=www.startpage.com > > Matches for http://www.startpage.com: > In file: match-all.action View > {+change-x-forwarded-for{block} > +client-header-tagger{css-requests} > +client-header-tagger{image-requests} > +client-header-tagger{range-requests} > +hide-from-header{block} > +set-image-blocker{pattern} } > / > In file: default.action View > (no matches in this file) > In file: user.action View > (no matches in this file) > In file: my.action View > (no matches in this file) > > > > U.Mutlu wrote on 07/22/2020 11:06 PM: > > Lee wrote on 07/22/2020 09:29 PM: > >> On 7/22/20, U.Mutlu <[email protected]> wrote: > >>> Update: sorry, had forgotten to build with "--enable-external-filters". > >>> After rebuilding and reinstalling it now starts ok, > >>> but the external-filter does not get invoked :-( > >>> How to fix that? > >> > >> Have you added > >> debug 64 # debug regular expression filters > >> to the config file to see if privoxy tries to run the filter? > >> > https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834 > >> > > > > Thx. Now tried with "debug 64" and it shows that it recognizes the said > > external-filter, but IMHO the action for it is not executed: > > > > .... > > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > "user-agent" > > ("Tags the request with the complete User-Agent header.") > > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job > > 's@^User-Agent:.*@$0@i' to filter user-agent succeeded. > > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > "referer" > > ("Tags the request with the complete Referer header.") > > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job > > 's@^Referer:.*@$0@i' to filter referer succeeded. > > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter > > "content-type" ("Tags the request with the content type declared by the > server.") > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > > 's@^Content-Type:\s*([^;]+).*@$1@i' to filter content-type succeeded. > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control > headers.") > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded. > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > > "privoxy-control" ("Removes X-Privoxy-Control headers.") > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded. > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control > headers.") > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded. > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > > "privoxy-control" ("Removes X-Privoxy-Control headers.") > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job > > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded. > > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter > "myTee" ("MY > > TEE TEST") > > ... > > > > There is no other log entry regarding this "myTee" filter. > > Ie. it seems the action for it does not get executed. > > > >> If the filter isn't being called, maybe you need > >> https://www.privoxy.org/user-manual/filter-file.html > >> Enabled content filters are applied to any content whose "Content > >> Type" header is recognised as a sign of text-based content, with the > >> exception of text/plain. Use the force-text-mode action to also filter > >> other content. > > > > But doing so seems to be dangerous according to the doc > > https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE > > It says: > > "Warning - Think twice before activating this action. Filtering binary > data > > with regular expressions can cause file damage." > > > > > > I just want to get a copy of the whole HTML that privoxy gets from the > server, > > either the unmodified raw data, or the filtered version after > > privoxy applies the other defined filters. > > > > I think the HTML is always a text file, even if it contains binary data > > like picture images etc. (they are somehow text encoded I think using > > base64 encoding or so). Is this assumption wrong? > > > > If I read the doc right then the external-filter handler has to write > > the received data either verbatim (ie. no filtering), or a filtered > > version of the data, back to stdout, so privoxy can continue doing its > job. > > > > A working example for such an external-filter would be helpful. > > Unfortunately I couldn't find any on the web :-( > > > > ------------------------------ > > Message: 6 > Date: Wed, 22 Jul 2020 23:15:18 -0400 > From: Lee <[email protected]> > To: "U.Mutlu" <[email protected]> > Cc: [email protected] > Subject: Re: [Privoxy-users] external-filter example > Message-ID: > < > CAD8GWsuRNCLPV8MAcKFi5Yrqgs3WSd5ye7XwNd1r+NWYZ6sbmQ@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On 7/22/20, U.Mutlu <[email protected]> wrote: > > Lee wrote on 07/22/2020 09:29 PM: > >> > >> Have you added > >> debug 64 # debug regular expression filters > >> to the config file to see if privoxy tries to run the filter? > > > > Thx. Now tried with "debug 64" and it shows that it recognizes the said > > external-filter, but IMHO the action for it is not executed: > <.. snip ..> > correct, a filter being applied looks like > 2020-07-22 21:38:06.979 000004a0 Re-Filter: filtering > 'Accept-Encoding: gzip, deflate, br' (size 34) with > 'no-brotli-accepted' ... > 2020-07-22 21:38:06.979 000004a0 Re-Filter: ... produced 1 hits (new size > 30). > > > There is no other log entry regarding this "myTee" filter. > > Ie. it seems the action for it does not get executed. > > Did you add the filter call to your user.action? for example: > A long time ago flamingtext made you wait for the pretty picture. I > got annoyed, so I added a filter to remove the wait. In user.filter: > # > # flamingtest: removes image wait timer > # > # flamingtest.com - removes the timeout obstruction so you don't have > to wait for your image > FILTER: flamingtext no waiting > > s@<META HTTP-EQUIV=Refresh CONTENT="15; URL=@<META HTTP-EQUIV=Refresh > CONTENT="1; URL=@ig > > Then activate the filter just for the flamingtext domain in user.actions: > ## don't wait for flamingtext.com > { +filter{flamingtext} } > .flamingtext.com/ > > and now, http://config.privoxy.org/show-url-info?url=flamingtext.com > gives me > In file: user.action > {+filter{flamingtext} } > .flamingtext.com/ > {+server-header-tagger{content-type} } > / > {-deanimate-gifs } > .flamingtext.com/ > > >> If the filter isn't being called, maybe you need > >> https://www.privoxy.org/user-manual/filter-file.html > >> Enabled content filters are applied to any content whose "Content > >> Type" header is recognised as a sign of text-based content, with the > >> exception of text/plain. Use the force-text-mode action to also filter > >> other content. > > > > But doing so seems to be dangerous according to the doc > > https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE > > It says: > > "Warning - Think twice before activating this action. Filtering binary > data > > with regular expressions can cause file damage." > > > > > > I just want to get a copy of the whole HTML that privoxy gets from the > > server, > > either the unmodified raw data, or the filtered version after > > privoxy applies the other defined filters. > > > > I think the HTML is always a text file, > > oops, no, not at all > > > even if it contains binary data > > like picture images etc. (they are somehow text encoded I think using > > base64 encoding or so). Is this assumption wrong? > > it is wrong > > > If I read the doc right then the external-filter handler has to write > > the received data either verbatim (ie. no filtering), or a filtered > > version of the data, back to stdout, so privoxy can continue doing its > job. > > > > A working example for such an external-filter would be helpful. > > Unfortunately I couldn't find any on the web :-( > > How about > > http://config.privoxy.org/user-manual/actions-file.html#SERVER-HEADER-TAGGER > > Add this to your user.action: > # Tag every request with the content type declared by the server > {+server-header-tagger{content-type}} > / > > # apply myTee filter to all text/html content > { +external-filter{myTee} +force-text-mode} > TAG:^text/html > > and the bit you have in user.filter should be something like > > EXTERNAL-FILTER: myTee make a copy of reveived text > /usr/bin/tee myCopiedText > > maybe that will work? > > Lee > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Privoxy-users mailing list > [email protected] > https://lists.privoxy.org/mailman/listinfo/privoxy-users > > > ------------------------------ > > End of Privoxy-users Digest, Vol 41, Issue 7 > ******************************************** > -- Robert Klemme Jr. President, Klemme Consulting LLC. [email protected]