Re: [xmltv-users] UK Atlas load/usage warning email

On 12.05.2016 19:33, [email protected] wrote:
> On Thu, 12 May 2016 10:10:19 -0500, Robert Eden wrote:
>
>> The Schedules Direct DataDirect backend outputs normalized data. There's
>> a schedule section, program section, credits section, etc. Does Atlas
>> output XMLTV directly?
>
>> tv_grab_na_dd uses a *lot* of memory caching all the data so it can
>> generated non-normalized XMLTV format data.   If that approach was done
>> server-side, I can see it causing lots of memory issues.
>>
>> by breaking things down into less-than-a-day chunks,  Atlas hopes to
>> reduce memory usage in their server.
>
>
> Atlas output is normalized data dump in either JSON or XML format. The tv_grab_uk_atlas grabber converts it into XMLTV format.

I think there may be confusing terminology in these statements.

Schedules direct data is normalized as in
https://en.wikipedia.org/wiki/Database_normalization
with the data feed containing of lots of references.

But the Atlas API, with the given annotations, returns the actual data
instead of references, allowing for a simple streaming conversion
without record keeping.
https://en.wikipedia.org/wiki/Denormalization

> The API call (http) returns a simple dump of all the content (e.g. programmes, broadcasts, descriptions) requested. Since this is the sole purpose of the Atlas database/API one would expect it to be tuned for this purpose.

 From my understanding the Atlas API is designed to return all data
needed for one page of an interactive interface per call. Something
completely different then xmltv's batch interface with one call often
returning the whole database. So every adapter between Atlas and XMLTV
will have to find a translation for this general conceptual difference.

That is what I was trying to say on Tuesday. (But that was worded badly)

I see that _uk_atlas already grabs the whole time span but only one
channel per request.

Additionally grabbing data from x o'clock to x o'clock a day later
appears to be the easiest way to split the request as it aligns to the
xmltv interface allowing for a straightforward implementation.
Basically a second loop starting at day "offset" and ending at day
"offset+days" returning one day worth of programming per iteration.

Regards,
Karl

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j