Duplicated ratings entries with sd_json grabber upsetting mythfilldatabase

Ian Campbell <[email protected]> Sat, 29 Oct 2016 11:30:07 +0100
Newsgroups gmane.comp.tv.xmltv.devel
Message-ID <[email protected]> 
Hi all,

mythfilldatabase is complaining with about half a dozen of these (full
log attached):

    016-10-29 06:31:23.556459 E  DB Error (programrating insert):
    Query was:
    INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
    Bindings were:
    :CHANID=8173, :RATING="G", :START=2016-11-01T10:00:00Z,
    :SYS="Canadian Parental Rating"
    Driver error was [2/1062]:
    QMYSQL3: Unable to execute statement
    Database error was:
    Duplicate entry '8173-2016-11-01 10:00:00-Canadian-G' for key 'chanid'

I've seen this issue before and IIRC Robert (K) you fixed it at the
source, but it's happening again (has been for a while but I didn't get
round to writing this mail). The last time it seemed to be suggested
(in https://sourceforge.net/p/xmltv/mailman/message/35109896/) that
xmltv ought really to be dedupping (as well as SD upstream doing so?
although AIUI dedupping was already added to the server side last time
too) which is why I'm bringing this up here to start with rather than
the SD forums.

Looking at the corresponding xml for the above program I find:

  <programme start="20161101100000 +0000" stop="20161101110000 +0000" channel="24321">
    <title>Homes Under the Hammer</title>
    <desc>[...snip...]</desc>
    <credits>
        [...snip...]
    </credits>
    <category>House/garden</category>
    <category>Home improvement</category>
    <category>Series</category>
    <category>series</category>
    <episode-num system="xmltv_ns">19 . 46 . </episode-num>
    <episode-num system="dd_progid">EP012598221009</episode-num>
    <previously-shown start="20161101" />
    <new />
    <rating system="Canadian Parental Rating">
      <value>G</value>
    </rating>
    <rating system="Canadian Parental Rating">
      <value>G</value>
    </rating>
    <rating system="USA Parental Rating">
      <value>TVG</value>
    </rating>
    <rating system="USA Parental Rating">
      <value>TVG</value>
    </rating>
  </programme>

and in the original json:

    {
      "genres": [
        "House/garden",
        "House/garden",
        "Home improvement",
        "Home improvement"
      ],
      "showType": "Series",
      "resourceID": "402752",
      "hasImageArtwork": true,
      "entityType": "Episode",
      "contentRating": [
        {
          "body": "Canadian Parental Rating",
          "country": "CAN",
          "code": "G"
        },
        {
          "body": "Canadian Parental Rating",
          "country": "CAN",
          "code": "G"
        },
        {
          "body": "USA Parental Rating",
          "country": "USA",
          "code": "TVG"
        },
        {
          "body": "USA Parental Rating",
          "country": "USA",
          "code": "TVG"
        }
      ],
      "crew": [...],
      "originalAirDate": "2016-11-01",
      "cast": [...]    ,
  "programID": "EP012598221009",
[...]
  "md5": "RSXtIpywuLR7QFbI5Rd+vA",
  "metadata": [
    {
      "Gracenote": {
        "season": 20,
        "episode": 47
      }
    }
  ]
}

So it does seem to be coming from the source.

I guess I should start looking into dedup on the XMLTV side?

Ian.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive. 
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik

_______________________________________________
xmltv-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xmltv-devel
mythfilldatabase.log (text/x-log, 7 KB) 
2016-10-29 06:27:40.185035 C  mythfilldatabase version:  [v0.28] www.mythtv.org
2016-10-29 06:27:40.185056 C  Qt version: compile: 5.6.1, runtime: 5.6.1
2016-10-29 06:27:40.185059 N  Enabled verbose msgs:  general
2016-10-29 06:27:40.185070 N  Setting Log Level to LOG_INFO
2016-10-29 06:27:40.195582 I  Added logging to the console
2016-10-29 06:27:40.195997 I  Setup Interrupt handler
2016-10-29 06:27:40.196005 I  Setup Terminated handler
2016-10-29 06:27:40.196017 I  Setup Segmentation fault handler
2016-10-29 06:27:40.196024 I  Setup Aborted handler
2016-10-29 06:27:40.196035 I  Setup Bus error handler
2016-10-29 06:27:40.196043 I  Setup Floating point exception handler
2016-10-29 06:27:40.196052 I  Setup Illegal instruction handler
2016-10-29 06:27:40.196061 I  Setup Real-time signal 0 handler
2016-10-29 06:27:40.196071 I  Setup Hangup handler
2016-10-29 06:27:40.196101 N  Using runtime prefix = /usr
2016-10-29 06:27:40.196108 N  Using configuration directory = /var/lib/mythtv/.mythtv
2016-10-29 06:27:40.196166 I  Assumed character encoding: en_GB.UTF-8
2016-10-29 06:27:40.196390 I  Using localhost value of iranon
2016-10-29 06:27:40.209896 N  Setting QT default locale to en_GB
2016-10-29 06:27:40.209906 I  Current locale en_GB
2016-10-29 06:27:40.209936 N  Reading locale defaults from /usr/share/mythtv//locales/en_gb.xml
2016-10-29 06:27:40.218778 I  Loading en_gb translation for module mythfrontend
2016-10-29 06:27:40.220981 I  Current MythTV Schema Version (DBSchemaVer): 1344
2016-10-29 06:27:40.223349 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 192.168.1.4:6543 (try 1 of 1)
2016-10-29 06:27:40.224224 I  MythCoreContext::CheckProtoVersion(): Using protocol version 88 XmasGift
2016-10-29 06:27:40.224496 I  Opening blocking connection to master backend
2016-10-29 06:27:40.296355 I  Added syslogging
2016-10-29 06:27:40.876427 I  Updating source #4 (Freeview) with grabber tv_grab_sd_json_filtered
2016-10-29 06:27:40.877017 I  Found 103 channels for source 4 which use grabber
2016-10-29 06:27:40.877227 I  Starting process signal handler
2016-10-29 06:27:40.878516 I  Starting IO manager (read)
2016-10-29 06:27:40.878553 I  Starting process manager
2016-10-29 06:27:40.878666 I  Starting IO manager (write)
2016-10-29 06:27:41.578774 I  Grabber has capabilities: baseline manualconfig preferredmethod 
2016-10-29 06:27:42.128977 I  Grabber prefers method: allatonce
2016-10-29 06:27:42.129512 I  XMLTV config file is: /var/lib/mythtv/.mythtv/Freeview.xmltv
2016-10-29 06:31:23.556459 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=8173, :RATING="G", :START=2016-11-01T10:00:00Z,
:SYS="Canadian Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '8173-2016-11-01 10:00:00-Canadian-G' for key 'chanid'

2016-10-29 06:31:23.556922 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=8173, :RATING="TVG", :START=2016-11-01T10:00:00Z,
:SYS="USA Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '8173-2016-11-01 10:00:00-USA Pare-TVG' for key 'chanid'

2016-10-29 06:32:45.197344 I  Updated programs: 9555 Unchanged programs: 31946
2016-10-29 06:32:46.766628 I  Updating source #5 (FreeviewHD) with grabber tv_grab_sd_json_filtered
2016-10-29 06:32:46.767414 I  Found 135 channels for source 5 which use grabber
2016-10-29 06:32:47.330727 I  Grabber has capabilities: baseline manualconfig preferredmethod 
2016-10-29 06:32:48.031616 I  Grabber prefers method: allatonce
2016-10-29 06:32:48.032183 I  XMLTV config file is: /var/lib/mythtv/.mythtv/FreeviewHD.xmltv
2016-10-29 06:37:29.414051 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=5001, :RATING="G", :START=2016-11-01T10:00:00Z,
:SYS="Canadian Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '5001-2016-11-01 10:00:00-Canadian-G' for key 'chanid'

2016-10-29 06:37:29.414475 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=5001, :RATING="TVG", :START=2016-11-01T10:00:00Z,
:SYS="USA Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '5001-2016-11-01 10:00:00-USA Pare-TVG' for key 'chanid'

2016-10-29 06:38:40.786496 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=5101, :RATING="G", :START=2016-11-01T10:00:00Z,
:SYS="Canadian Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '5101-2016-11-01 10:00:00-Canadian-G' for key 'chanid'

2016-10-29 06:38:40.786925 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=5101, :RATING="TVG", :START=2016-11-01T10:00:00Z,
:SYS="USA Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '5101-2016-11-01 10:00:00-USA Pare-TVG' for key 'chanid'

2016-10-29 06:39:12.474964 I  Updated programs: 14001 Unchanged programs: 41988
2016-10-29 06:39:13.408820 N  Data fetching complete.
2016-10-29 06:39:13.408857 I  Adjusting program database end times.
2016-10-29 06:39:13.409339 I      0 replacements made
2016-10-29 06:39:13.409343 I  Marking generic episodes.
2016-10-29 06:39:14.917876 I      Found 12
2016-10-29 06:39:14.917883 I  Extending non-unique programids with multiple parts.
2016-10-29 06:39:15.534355 I      Found 0
2016-10-29 06:39:15.534362 I  Fixing missing original airdates.
2016-10-29 06:39:17.313334 I      Found 0 with programids
2016-10-29 06:39:17.349687 I      Found 0 without programids
2016-10-29 06:39:17.349693 I  Marking repeats.
2016-10-29 06:39:17.509402 I      Found 0
2016-10-29 06:39:17.509410 I  Unmarking new episode rebroadcast repeats.
2016-10-29 06:39:19.090045 I      Found 10052
2016-10-29 06:39:22.586262 I  Marking episode first showings.
2016-10-29 06:39:25.985648 I      Found 50890
2016-10-29 06:39:25.985657 I  Marking episode last showings.
2016-10-29 06:39:29.264223 I      Found 48177
2016-10-29 06:39:29.314817 I  
===============================================================
| Attempting to contact the master backend for rescheduling.  |
| If the master is not running, rescheduling will happen when |
| the master backend is restarted.                            |
===============================================================
2016-10-29 06:39:29.315416 N  mythfilldatabase run complete.
2016-10-29 06:39:29.315457 I  Waiting for threads to exit.
2016-10-29 06:39:29.412428 I  Removing syslogging