xmltv/beta Augment.pm, NONE, 1.1 augment.conf, NONE, 1.1 augment.rules, NONE, 1.1 tv_augment, NONE, 1.1
Geoff <[email protected]>
| Newsgroups | gmane.comp.tv.xmltv.cvs |
|---|---|
| Message-ID | <[email protected]> |
Update of /cvsroot/xmltv/xmltv/beta
In directory sfp-cvs-1.v30.ch3.sourceforge.com:/tmp/cvs-serv32102
Added Files:
Augment.pm augment.conf augment.rules tv_augment
Log Message:
Augment XMLTV programme listings files via automatic and user-defined rules
--- NEW FILE: Augment.pm ---
#
# $Id: Augment.pm,v 1.1 2014/12/09 11:32:08 bilbo_uk Exp $
#
#
#
=pod
=head1 NAME
XMLTV::Augment - Augment XMLTV listings files with automatic and user-defined rules.
=head1 DESCRIPTION
Augment an XMLTV xml file by applying corrections ("fixups") to programmes
matching defined criteria ("rules").
Two types of rules are actioned: (i) automatic, (ii) user-defined.
[...3597 lines suppressed...]
}
###############################################
1; # keep eval happy ;-)
__END__
=pod
=head1 AUTHOR
Geoff Westcott, honir.at.gmail.dot.com, Dec. 2014.
This code is based on the "fixup" method/code defined in tv_grab_uk_rt grabber
and credit is given to that graber"s authors: Ed Avis and Nick Morrott.
=cut
--- NEW FILE: augment.conf ---
#
# Sample configuration file for tv_augment
#
# $Id: augment.conf,v 1.1 2014/12/09 11:32:08 bilbo_uk Exp $
#
# This file controls which augmentation rules are applied to the input XMLTV file
#
#
# It also controls what reporting is printed in the program's output log file.
#
# Set an option to 1 to enable it, 0 to disable.
#
# If specified, then this language code will be written to e.g. <credit> elements
language_code = en
# Set the following values to have XMLTV::Augment automatically fetch a file
# from the grabber's repository using XMLTV::Supplement
use_supplement = 1
supplement_grabber_name = tv_grab_uk_rt
supplement_grabber_file = prog_titles_to_process
###############################################################################
#
# RULES
# =====
#
# The option called 'enable_all_options' is a 'super-option' to quickly turn
# on or off ALL automatic and user rules. If this is set then ALL individual
# options are ignored.
#
enable_all_options = 1
# AUTOMATIC RULES
# ===============
#
# The rules are pre-determined and use no user-defined data.
#
# To enable a rule set it 1, to disable set it to 0.
#
# Note: if the 'enable_all_options' is set to 1 then the following lines
# will be ignored (i.e. they will be run as if they were set to 1).
#
# Rule #A1 - Remove "New $title :" from <sub-title>.
remove_duplicated_new_title_in_ep = 1
# Rule #A2 - Remove duplicated programme title *and* episode from <sub-title>.
remove_duplicated_title_and_ep_in_ep = 1
# Rule #A3 - Remove duplicated programme title from <sub-title>.
remove_duplicated_title_in_ep = 1
# Rule #A4 -
update_premiere_repeat_flags_from_desc = 1
# Rule #A5 - Check for potential series, episode and part numbering in the title, episode and description fields.
check_potential_numbering_in_text = 1
# Rule #A5.1 - Extract series/episode numbering found in <title>.
# (requires #A5 enabled)
extract_numbering_from_title = 1
# Rule #A5.2 - Extract series/episode numbering found in <sub-title>.
# (requires #A5 enabled)
extract_numbering_from_episode = 1
# Rule #A5.3 - Extract series/episode numbering found in <desc>.
# (requires #A5 enabled)
extract_numbering_from_desc = 1
# Rule #A6 - If no <sub-title> then make one from "part" numbers.
make_episode_from_part_numbers = 1
# USER RULES
# ==========
#
# These rules use data contained within the 'fixup' rules file to control their action.
#
# You could disable a rule simply by not having any entries in the rules file,
# but this provides a convenient way to switch on/off any particular rule.
#
# The corollary is, to action an particular rule type, you must have some
# relevant rules of the type in the rules file, *and* you must enable the
# option below (or enable the 'all' option above)
#
# Note: if the 'enable_all_options' is set to 1 then the following lines
# will be ignored (i.e. they will be run as if they were set to 1).
#
# Rule #user - Process programme against user-defined title fixups
# If you disable this option (by setting to 0) then none of the user rules
# will be actioned (irrespective of whether you have them set to 1 or 0)
#
process_title_fixups = 1
# Rule #1 - Remove specified non-title text from <title>.
process_non_title_info = 1
# Rule #2 - Extract sub-title from <title>.
process_mixed_title_subtitle = 1
# Rule #3 - Extract sub-title from <title>.
process_mixed_subtitle_title = 1
# Rule #4 - Reverse <title> and <sub-title>
process_reversed_title_subtitle = 1
# Rule #5 - Replace <title> with supplied text.
process_replacement_titles = 1
# Rule #6 - Replace <category> with supplied text.
process_replacement_genres = 1
# Rule #7 - Replace <sub-title> with supplied text.
process_replacement_episodes = 1
# Rule #8 - Replace specified <title> / <sub-title> with title/episode pair supplied.
process_replacement_titles_episodes = 1
# Rule #9 - Replace <sub-title> with supplied text when the <desc> matches that given.
process_replacement_ep_from_desc = 1
# Rule #10 - Replace specified <title> / <sub-title> with title/episode pair supplied using <desc>.
process_replacement_titles_desc = 1
# Rule #11 - Promote demoted title from <sub-title> to <title>.
process_demoted_titles = 1
# Rule #12 - Replace "Film"/"Films" <category> with supplied text.
process_replacement_film_genres = 1
# Rule #13 - Remove specified text from <sub-title> for a given <title>.
process_subtitle_remove_text = 1
# Rule #14 - Replace specified categories with another value
process_translate_genres = 1
--- NEW FILE: tv_augment ---
#!/usr/bin/perl -w
eval 'exec /usr/bin/perl -w -S $0 ${1+"$@"}'
if 0; # not running under some shell
=pod
=head1 NAME
tv_augment - Augment XMLTV listings files with automatic and user-defined rules.
=head1 SYNOPSIS
tv_augment [--rule <file>] [--config <file>]
[--input <file>] [--output <file>]
[--log <file>] [--nostats]
[--debug <level>]
tv_augment [-r <file>] [-c <file>]
[-i <file>] [-o <file>]
[-l <file>] [-n]
[-d <level>]
=head1 DESCRIPTION
Augment an XMLTV xml file by applying corrections ("fixups") to programmes
matching defined criteria ("rules").
Two types of rules are actioned: (i) automatic, (ii) user-defined.
Automatic rules use pre-programmed input and output to modify the input
programmes. E.g. removing a "title" where it is repeated in a "sub-title"
(e.g. "Horizon" / "Horizon: Star Wars"), or trying to identify and extract
series/episode numbers from the programme title, sub-title or description.
User-defined rules use the content of a "rules" file which allows programmes
matching certain user-defined criteria to be corrected/enhanced with the user
data suplied (e.g. adding/changing categories for all episodes of "Horizon",
or fixing misspellings in titles, etc.)
(see "perldoc -F Augment.pm" for more detail)
B<--input FILE> read from FILE rather than standard input.
B<--output FILE> write to FILE rather than standard output.
B<--rule FILE> file containing the user-defined rules.
B<--config FILE> configuration file containing a list of which rules
you want to run.
B<--nostats> do not print the summary log of actions performed, or list of
suggested fixups.
B<--log FILE> output the stats to this FILE (default = augment.log).
B<--debug LEVEL> print debug info to STDERR (debug level > 3 is not
likely to be of much use (it generates a lot of output))
=head1 SEE ALSO
L<xmltv(5)>
=head1 AUTHOR
Geoff Westcott, honir.at.gmail.dot.com, Dec. 2014.
=cut
use strict;
use XMLTV::Version '$Id: tv_augment,v 1.1 2014/12/09 11:32:08 bilbo_uk Exp $ ';
use Data::Dumper;
use Getopt::Long;
use XMLTV;
use XMLTV::Data::Recursive::Encode;
# simplify testing by also looking for package in current directory
eval 'use XMLTV::Augment';
if ($@ ne '') { eval 'use Augment'; }
use XMLTV::Usage <<END
$0: Augment programme listings with automatic and user-defined rules
$0 [--rule <file>] [--config <file>] [--input <file>] [--output <file>] [--nostats] [--log <file>] [--debug (1-10)]
$0 [-r <file>] [-c <file>] [-i <file>] [-o <file>] [-n] [-l <file>] [-d (1-10)]
END
;
my (
$opt_help,
$opt_input,
$opt_output,
$opt_rule,
$opt_config,
$opt_stats,
$opt_log,
$opt_debug,
$opt_do,
);
GetOptions(
'h|help' => \$opt_help,
'i|input=s' => \$opt_input,
'o|output=s' => \$opt_output,
'r|rule=s' => \$opt_rule,
'c|config|config-file=s' => \$opt_config,
'n|nostats:i' => \$opt_stats,
'l|log=s' => \$opt_log,
'd|debug:i' => \$opt_debug,
'do:i' => \$opt_do,
) or usage(0);
usage(1) if $opt_help;
#rule is now optional if using Supplement via config
# usage(0) if !$opt_rule;
$opt_input = '-' if ( !defined($opt_input) );
#$opt_output = 'STDOUT' if ( !defined($opt_output) );
$opt_debug = 0 if ( !defined($opt_debug) );
$opt_stats = 1 if ( !defined($opt_stats) );
# object construction & open log file
my $augment = new XMLTV::Augment(
'rule' => $opt_rule,
'config' => $opt_config,
'debug' => $opt_debug,
'stats' => $opt_stats,
'log' => $opt_log,
)
|| eval { print STDERR "Failed to create XMLTV::Augment object \n"; exit 1; };
my %w_args = ();
if (defined $opt_output) {
my $fh = new IO::File ">$opt_output";
die "cannot write to $opt_output\n" if not $fh;
%w_args = (OUTPUT => $fh);
}
# our XMLTV::Writer object
my $w;
# store the input file's encoding
my $encoding;
# count of input records
my $in_count = 0;
# parsefiles_callback needs an array
my @files = ( $opt_input );
XMLTV::parsefiles_callback(\&encoding_cb, \&credits_cb, \&channel_cb, \&programme_cb, @files);
# note: we only get a Writer if the encoding callback gets called
if ( $w ) {
$w->end();
}
# log the stats
$augment->printInfo();
# close the log file
$augment->end();
exit(0);
# callbacks used by parsefiles_callback
#
sub encoding_cb( $ ) {
die if defined $w;
$encoding = shift; # callback returns the file's encoding
$w = new XMLTV::Writer(%w_args, encoding => $encoding);
$augment->setEncoding($encoding);
}
#
sub credits_cb( $ ) {
$w->start(shift);
}
#
sub channel_cb( $ ) {
my $ch = shift;
# store the channel details
$augment->inputChannel( $ch );
# write the channel element to the output xml
$w->write_channel($ch);
}
#
sub programme_cb( $ ) {
my $prog = shift;
$in_count++;
# developer's option to only process a few records in input file and then stop
if ( defined($opt_do) && $in_count > $opt_do ) { return; }
# decode the incoming programme
$prog = XMLTV::Data::Recursive::Encode->decode($encoding, $prog);
# augmentProgramme will now do any requested processing of the input xml
$prog = $augment->augmentProgramme( $prog );
# re-code the modified programme back to original encoding
$prog = XMLTV::Data::Recursive::Encode->encode($encoding, $prog);
# output the augmented programme
$w->write_programme($prog);
}
#
--- NEW FILE: augment.rules ---
#
# Sample "rules" file for use with tv_augment
#
# $Id: augment.rules,v 1.1 2014/12/09 11:32:08 bilbo_uk Exp $
#
#
###############################################################################
#
# This file contains the rules used by XMLTV::Augment.
#
# The objective is to fix errors and inconsistencies in the incoming data from
# a grabber, and to enhance the programme xml where certain data are missing.
#
# For example:
# Some programme entries in the listings data may contain subtitle/episode
# information in the title field, others may contain the programme title
# and subtitle reversed, and yet more may contain 'episode' information that
# should be in the episode-num field.
#
# Rules are divided into a number of 'types' according to what they do.
# Individual types (rule sets) can be switched off in the augmentation config
# file.
#
# Matching is usually (but not exclusively) done by comparing the incoming
# <title> against the title specified in the rule below.
#
# A 'rule' definition consists of the rule 'type' separated rom the rule
# content by a pipe character ('|'). The rule content has a variable (but fixed
# for any given rule type) number of fields separated by tilde characters ('~')
#
# The action taken depends on the rule type:
#
# 1) non_title_info
# The content is text that is to be removed from any programme titles where
# this text occurs at the beginning of the <title> element followed by
# any of : ; or ,
# e.g.
# 1|Action Heroes Season
# "Action Heroes Season: Rambo" --> "Rambo"
#
# 2) mixed_title_subtitle
# The content is the desired title of a programme when the incoming title
# contains both the programme's title *and* episode separated by : ; or -
# ($title:$episode). We reassign the episode information to the <episode>
# element, leaving only the programme title in the <title> element.
# e.g.
# 2|Blackadder II
# "Blackadder II: Potato / " --> "Blackadder II / Potato"
#
# 3) mixed_subtitle_title
# The content is the desired title of a programme when the incoming title
# contains both the programme's episode *and* title separated by : ; or -
# ($episode:$title). We reassign the episode information to the <episode>
# element, leaving only the programme title in the <title> element.
# e.g.
# 3|Storyville
# "Kings of Pastry: Storyville / " --> "Storyville / Kings of Pastry"
#
# 4) reversed_title_subtitle
# The content is the desired title of a programme which is listed as the
# programme's episode (i.e. the title and episode details have been
# reversed). We therefore reverse the <title> and <episode> elements.
# e.g.
# 4|Storyville
# "Kings of Pastry / Storyville" --> "Storyville / Kings of Pastry"
#
# 5) replacement_titles
# The content contains two programme titles, separated by a
# tilde (~). The first title is replaced by the second in the listings
# output.
# This is useful to fix inconsistent naming (e.g. "Law and Order" vs.
# "Law & Order") or inconsistent grammar ("xxxx's" vs. "xxxxs'")
# e.g.
# 5|A Time Team Special~Time Team
# "A Time Team Special / Doncaster" --> "Time Team / Doncaster"
#
# 6) replacement_genres
# The content contains a programme title and a programme category(-ies)
# (genres), separated by tilde (~). Categories can be assigned to
# uncategorised programmes (which can be seen in the stats log).
# (Note that *all* categories are replaced for the title.)
# e.g.
# 6|Antiques Roadshow~Entertainment~Arts~Shopping
# "Antiques Roadshow / " category "Reality" -->
# "Antiques Roadshow / " category "Entertainment" + "Arts" + "Shopping"
#
# 7) replacement_episodes
# The content contains a programme title, an old episode value and a new
# episode value, separated by tildes (~).
# e.g.
# 7|Time Team~Time Team Special: Doncaster~Doncaster
# "Time Team / Time Team Special: Doncaster" --> "Time Team / Doncaster"
#
# 8) replacement_titles_episodes
# The content contains an old programme title, an old episode value, a new
# programme title and a new episode value. The old and new titles MUST be
# given, the episode fields can be left empty but the field itself must be
# present.
# e.g.
# 8|Top Gear USA Special~Detroit~Top Gear~USA Special
# "Top Gear USA Special / Detroit" --> "Top Gear / USA Special"
#
# 8|Top Gear USA Special~~Top Gear~USA Special
# "Top Gear USA Special / " --> "Top Gear / USA Special"
#
# 9) replacement_ep_from_desc
# The content contains a programme title, a new episode value to update,
# and a description (or at least the start of it) to match against. When
# title matches incoming data and the incoming description startswith the
# text given then the episode value will be replaced.
# e.g.
# 9|Heroes of Comedy~The Goons~The series celebrating great British
# comics pays tribute to the Goons.
# "Heroes of Comedy / " desc> = "The series celebrating great British
# comics pays tribute to the Goons."
# --> "Heroes of Comedy / The Goons"
# Should be used with care; e.g.
# "Heroes of Comedy / Spike Milligan" desc> = "The series celebrating
# great British comics pays tribute to the Goons."
# would *also* become
# "Heroes of Comedy / The Goons"
# this may not be what you want!
#
# 10) replacement_titles_desc
# The content contains an old programme title, an old episode value, a new
# programme title, a new episode value and the episode description (or at
# least the start of it) to match against.
# The old and new titles and description MUST be given, the episode fields
# can be left empty but the field itself must be present.
# This is useful to fix episodes where the series is unknown but can be
# pre-determined from the programme's description.
# e.g.
# 10|Which Doctor~~Gunsmoke~Which Doctor~Festus and Doc go fishing, but
# are captured by a family that is feuding with the Haggens.
# "Which Doctor / " desc> = "Festus and Doc go fishing, but are captured
# by a family that is feuding with the Haggens. [...]"
# --> "Gunsmoke / Which Doctor"
#
# 11) demoted_titles
# The content contains a programme 'brand' and a new title to be extracted
# from subtitle field and promoted to programme title, replacing the brand
# title.
# In other words, if title matches, and sub-title starts with text then
# remove the matching text from sub-title and move it into the title.
# Any text after 'separator' (any of .,:;-) in the sub-title is preserved.
# e.g.
# 11|Blackadder~Blackadder II
# "Blackadder / Blackadder II: Potato" --> "Blackadder II / Potato"
#
# 12) replacement_film_genres
# The content contains a film title and a category (genre) or categories,
# separated by a tilde (~).
# If title matches the rule's text and the prog has category "Film" or
# "Films", then use the replacement category(-ies) supplied.
# Use case: some film-related programmes are incorrectly flagged as films
# and should to be re-assigned to a more suitable category.
# (Note ALL categories are replaced, not just "Film")
# e.g.
# 12|The Hobbit Special~Entertainment~Interview
# "The Hobbit Special / " category "Film" + "Drama" -->
# "The Hobbit Special / " category "Entertainment" + "Interview"
#
# 13) subtitle_remove_text
# The content contains a programme title and arbitrary text to
# be removed from the start/end of the programme's subtitle. If the text to
# be removed precedes or follows a "separator" (any of .,:;-), the
# separator is removed also.
# e.g.
# 13|Time Team~A Time Team Special
# "Time Team / Doncaster : A Time Team Special" -->
# "Time Team / Doncaster"
#
# 14) process_replacement_genres
# The content contains a category (genre) value followed by replacement
# category(-ies) separated by a tilde (~).
# Use case: useful if your PVR doesn't understand some of the category
# values in the incoming data; you can translate them to another value.
# e.g.
# 14|Adventure/War~Action Adventure~War
# "The Guns of Navarone" category "Adventure/War" -->
# "The Guns of Navarone" category "Action Adventure" + "War"
#
#
###############################################################################
#
# Some sample rules follow; obviously you should delete these and replace with
# your own!
#
1|Action Heroes Season
2|Blackadder II
3|Storyville
4|Storyville
5|A Time Team Special~Time Team
6|Antiques Roadshow~Entertainment~Arts~Shopping
7|Time Team~Time Team Special: Doncaster~Doncaster
8|Top Gear USA Special~Detroit~Top Gear~USA Special
9|Heroes of Comedy~The Goons~The series celebrating great British comics pays tribute to the Goons.
10|Which Doctor~~Gunsmoke~Which Doctor~Festus and Doc go fishing, but are captured by a family that is feuding with the Haggens.
11|Blackadder~Blackadder II
12|The Hobbit Special~Entertainment~Interview
13|Time Team~A Time Team Special
#
###############################################################################
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk