Re: syslog-ng-3.38.1 causing type hint errors in patterndb - bug?

I feel that we are able to do "whatever we want" with the syntax of the 
syslog-ng configuration file and the add-contextual-data because those 
are 100% syslog-ng constructs. The patterndb is an XML file which in my 
opinion should attempt to stay within the spirit of the XML community as 
much as possible. To that end I feel that using XML attributes is more 
appropriate than placing type hints into the data values.

Using third party tools to view or even edit a patterndb file would be 
more intuitive if the reader does not have to continuously re-interpret 
values with type hints.

The other benefit of using XML attributes is that it can be documented 
in the XML schema which document objects which can have a type and which 
can not. Right now I'm really confused as to which objects can have type 
hints.

Evan.

On 2022-11-22 07:03, Balazs Scheidler wrote:
>
> Hi,
>
> You are right. This was a bug in 3.38 which was fixed in PR #4158, 
> which has already been merged to master, and I was testing with the 
> current master and not 3.38.1
>
> Even though I did that PR, I forgot we merged a fix for this already.
>
> With the current master the same is like this:
>
> $ bin/pdbtool test --validate sample.pdb
> WARNING: the template specified in value()/<value> options for your 
> grouping-by() or db-parser() configuration has been changed to support 
> typing from syslog-ng 4.0. You are using an older config version and 
> your template contains an unrecognized type-cast, probably a 
> parenthesis in the value field. This will be interpreted in the 
> `type(value)' format in future versions. Please add an explicit 
> string() cast as shown in the 'fixed-value' tag of this log message or 
> remove the parenthesis. The value will be processed as a 'string' 
> expression; config-version='3.38', name='AUTHPROGRAM', 
> value='su(pam_unix)', fixed-value='string(su(pam_unix))'
> $ echo $?
> 0
>
> The issue is that with the release of v4.0 (which we started to 
> prepare for), this would become an error in earnest, as pdbtool does 
> not allow you to specify the config version and uses the syslog-ng 
> version number by default. And I don't see too much value in extending 
> pdbtool in this direction, as having to add an explicit config version 
> number to pdbtool when using it is not very intuitive.
>
> The PR I mentioned in my previous email (#4222) improves this further 
> by checking the pdb version field (in the topmost <patterndb> tag) and 
> uses string based values if it's below v6.
>
> What I am thinking about is whether to revert to using an attribute to 
> specify the type, e.g.
>
>         <values>
>            <value name="AUTHPROGRAM" 
> type="whatevertype-syslog-ng-supports">su(pam_unix)</value>
>         </values>
>
> We would default to using a string. The only downside is that 
> add-contextual-data() uses the parenthesis based format in a CSV file 
> and I can't extend the CSV format as easily as XML.
>
> So by changing to the use of an attribute types would mean we have 
> different syntaxes for these use-cases where one can use typed templates:
>
> - config file anywhere: `type(value)`
> - add-contextual-data, 3rd column in the CSV file: `type(value)`
> - db-parser(): `<value name="foo" type="whatever"/>`
>
>
>
>
> On Tue, Nov 22, 2022 at 3:14 PM Evan Rempel <[email protected]> wrote:
>
>     I'm not sure if you still think this is only a warning.
>
>     [xerr@pangolin etc]$ rpm -q syslog-ng
>     syslog-ng-3.38.1-1.el8.x86_64
>     [xerr@pangolin etc]$ rpm -qi syslog-ng
>     Name        : syslog-ng
>     Version     : 3.38.1
>     Release     : 1.el8
>     Architecture: x86_64
>     Install Date: Thu 03 Nov 2022 04:08:02 AM PDT
>     Group       : System Environment/Daemons
>     Size        : 3651296
>     License     : GPLv2+
>     Signature   : RSA/SHA256, Mon 03 Oct 2022 03:00:51 AM PDT, Key ID
>     c96af856c87e88fe
>     Source RPM  : syslog-ng-3.38.1-1.el8.src.rpm
>     Build Date  : Mon 03 Oct 2022 03:00:39 AM PDT
>     Build Host  : copr-hv-x86-64-01-prod-02441780-20221003-095224
>     Relocations : (not relocatable)
>     Vendor      : Fedora Copr - user czanik
>     URL         : http://www.balabit.com/network-security/syslog-ng
>     Summary     : Next-generation syslog server
>
>     A very small patterndb
>
>     <patterndb version="4" pub_date="2009-09-01">
>       <ruleset name="su(pam_unix)"
>     id="RS-d2fae001-a7f6-4e39-ae62-5658906fe48c">
>         <pattern>su_pam_unix_</pattern>
>         <rules>
>           <rule id="unmatched-program-su(pam_unix)" class="unknown"
>     provider="UVic">
>             <patterns>
>                <pattern>@ANYSTRING@</pattern>
>             </patterns>
>             <values>
>                <value name="AUTHPROGRAM">su(pam_unix)</value>
>             </values>
>           </rule>
>         </rules>
>       </ruleset>
>     </patterndb>
>
>     [xerr@pangolin etc]$ /usr/bin/pdbtool test --validate test.xml
>     Error parsing pattern database file; filename='test.xml',
>     error='test.xml:10:51: Error compiling value template,
>     rule=unmatched-program-su(pam_unix), name=AUTHPROGRAM,
>     value=su(pam_unix), error=Unknown type specified in type hinting: su'
>
>     Evan.
>
>     On 2022-11-22 01:39, Balazs Scheidler wrote:
>>
>>     I only got warnings and not errors, when trying to use this in my
>>     XML file:
>>
>>               <value name="seq-matches">su(pam_unix)</value>
>>
>>     how did you execute syslog-ng so you got the error above? The
>>     only case where this is an error, if you are using "@version: 4.0"
>>
>>     This was the warning produced (both by syslog-ng and pdbtool):
>>
>>     WARNING: the template specified in value()/<value> options for
>>     your grouping-by() or db-parser() configuration has been changed
>>     to support typing from syslog-ng 4.0. You are using an older
>>     config version and your template contains an unrecognized
>>     type-cast, probably a parenthesis in the value field. This will
>>     be interpreted in the `type(value)' format in future versions.
>>     Please add an explicit string() cast as shown in the
>>     'fixed-value' tag of this log message or remove the parenthesis.
>>     The value will be processed as a 'string' expression;
>>     config-version='3.38', name='seq-matches', value='su(pam_unix)',
>>     fixed-value='string(su(pam_unix))'
>>
>>     I chose to embed the type-hint field in the body of the <value>
>>     tag, as this is the format that is used everywhere else. I might
>>     be able to bump the db-parser XML file version, in the file header:
>>
>>     <patterndb version='5'>
>>
>>     I could bump this up to version 6, in which case you'd only need
>>     to add the type-hint if you also bumped the version number. That
>>     I think is doable.
>>
>>     On the documentation front, there's an XML schema in the source
>>     tree under the doc/xsd/ directory for each version of patterndb,
>>     and I assume the documentation also has a chapter on the
>>     db-parser() format.
>>
>>     On Mon, Nov 21, 2022 at 5:22 PM Evan Rempel <[email protected]> wrote:
>>
>>         We have a patterndb file that contains both patterns and
>>         values with
>>         fixed text that include a strings of the form "xxxx (yyy)"
>>         which now can
>>         not be loaded by the patterndb.
>>
>>         Syslog-ng throws the error
>>
>>         Error parsing pattern database file; ... Error compiling
>>         value template,
>>         rule=FLARE-3543, name=AUTHPROGRAM, value=su(pam_unix),
>>         error=Unknown
>>         type specified in type hinting: su'
>>
>>         I have not enabled the version 4.0 testing of type hinting.
>>         The version
>>         configuration is
>>
>>         @version: 3.36
>>
>>         I am unable to find documentation for the full syntax of the
>>         patterndb
>>         file (seems to have been lost from the docs since perhaps 3.16?)
>>
>>         Is there a patterndb syntax specification document?
>>
>>         I would expect that even when type hinting is enabled in the
>>         patterndb
>>         file, given that it is an XML document, that the type hinting
>>         would be
>>         part of the XML tag metadata, rather than part of the XML
>>         static data.
>>
>>         -- 
>>         Evan
>>
>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq