EPG on 7025

[AJM]

Some files from the old forum.

If you've downloaded MView E2 plugin and have it working, and
if you've got an account for xmltv data,

then the attachments include a grabber, a converter (xml --> MView E2 format) and also the zlib support package that is used by the grabber.

Unfortunately, the grabber uses the old oztivo interface which will stop working after 31 Mar 2008. I'm hoping the author (Will Uther) might update tha grabber for us some time!

cheers

[poiuytrewq]

Nice work. I am hoping that the new PeterPan image will make all this much more seamless for us Aussies - once we work out how to convert everything to the new formats at each end.

[AJM]

Will Uther has provided a 'test' version of his grabber that uses the new API required by tvguide.org.au.

Will has not had time to implement file checking or caching yet. Accordingly, if you request channel/day data not yet filed then error messages result. This can be avoided by scheduling the grabber to run when the data is already available and restricting the number of days requested.

The Zoidberg's converter has also been updated by me to handle negative timezone offsets correctly. In the old API, the programme data was in UTC and the Australian timezones were always positive. With the new API, each channel/day data file seems to have local times. Accordingly, it may be necessary to use positive and/or negative timezone offsets when converting. These are specified in the config file and will probably have to be reviewed when daylight saving stops. (I would have preferred that they left everything in UTC).

These files will keep me in EPG business until the new PP E2 image is available.



cheers

[AJM]

The PeterPan EPG looks like it will be easy to adopt because the PP image author is going to provide a script to do the work required (see PMs from digsat.net below). Can't wait!

cheers,
AJM (aka Stick50jr)


-----Original Message-----
From: MMMMMM
Date: 14-03-2008 14:54
To: stick50jr
Subject: sqlite PP 7025

Hi,

As you may have seen we are releasing our PP image for 7025 soon.
All that is left is a Swedish translation and some testing with a Scandinavian NDS card.

You may have also seen the new EPG feature in our image, with the sqlite db feeding the EPG Viewer.
This means that basically all sources can be used with this epg solution. All you need is to create an import script to take the source file (eg. xmltv) and add the data into the db.

I have put together a few lines of code that you can use as a starting point for this script, that is if you need it at all
It is not set up to loop through several xml files, but you could easily adapt the code to do so.

BR,
MMMMMM

-----Reply:-----

Gidday MMMMMM,

The excitment is building. With you head start on the script, I reckon I'll be in business. At the moment my (python) grabber collects multiple xml files (each channel / day) and outputs a single xml consolidated. Then a separate (python) converter takes the consolidated xml and outputs it to 'mm.epg' for MView E2.

Sounds like I'll be able to run your script instead of my converter.

Thanks for keeping us in mind once again.

Looking forward to PP E2!

cheers
Stick50jr

[poiuytrewq]

Does look good. Any idea of the release date?

It is a bit of a shame about the timing because i am very impressed with the new LT image (particularly the media centre side of things) so i will be in two minds about which to use when this comes out.

p.

[AJM]

poiuytrewq,

You mentioned that you might soon have an xml to epg.dat converter. If this were available in python it could be easy to run within the DM7025. (I don't want to return to a scheme that is not automatically updated within the DM7025.) If it is only available in WINDOWS PC form then it would be interesting to see the source code and see whether it could be ported across to eliminate the PC.

In the meantime, I'll stick with MView E2 on other images or integrated sqlite under PP.

I don't know when PP will be released... soon but might be closer to 31 Mar than St Patrick's Day.

cheers

[poiuytrewq]

poiuytrewq,

You mentioned that you might soon have an xml to epg.dat converter. If this were available in python it could be easy to run within the DM7025. (I don't want to return to a scheme that is not automatically updated within the DM7025.) If it is only available in WINDOWS PC form then it would be interesting to see the source code and see whether it could be ported across to eliminate the PC.

In the meantime, I'll stick with MView E2 on other images or integrated sqlite under PP.

I don't know when PP will be released... soon but might be closer to 31 Mar than St Patrick's Day.

cheers

Yeah there are some italians that have developed an XMLTV to EPG.dat convertor for their services and i've convinced them to tweak it for here.

It has the advantage of not needing any special plugins or readers - it just writes straight to the dreambox EPG. On the downside it does need a PC - which is fine by me because mine is always on anyway. Personally it's a better mix - partially because i don't just want an EPG for f0xtel but for other services on other sats.

Anyhow, things are looking up either way at the moment.

p.

[AJM]

Gidday P,

If you're still around, I finally had a look at this stuff and got it going.

I've posted over with nfnovice's script:



Thanks for the tip to have a look at the Italians.

I got the PeterPan Sqlite version going with OzTivo also but it is only using the GraphicalEPG as far as I can see. The Ambrosa approach is more complete and the access to the EPG is more responsive.

cheers

[poiuytrewq]

Thanks AJM. Your work looks great. I will have a play with it this afternoon when i have a bit of time and let you know how i get on.

[poiuytrewq]

Hey AJM,

Tyring to figure out a few things here...

Firstly, how do i do the channel mapping manually in the OZTivo side of the things - the plugin seems good for that but doesn't work on the 7025. Right now it appears to be connecting fine but downloading blanks for every page.

Is it possible just to manually enter details from Lamedb or something? Or is there some other way of telling it which channels to get?

Right now it is downloading a blank epg... hmmm...

the other question is where is it best to install your files? There is no indication that i can find about where it is best to put them.

p.

[AJM]

Gidday P,

Sorry about the lack of instructions... I just got the two things going and I haven't documented what I did. I may do so sooner or later, but for now:

The oztivo_au.sh script works on my 7025 fine... I just commented out the last bit where it calls a program to load the e1 sqlite epg_db (not for us). This script is slow on my vSat internet connection but for experimenting you could restrict it to one day. This script is documented fairly well by nfnovice on this forum etc.

The e2_loadepg.py program is where I have done my adaptation... I have disabled the part where it downloads (because oztivo_au.sh has done that). The modifications take into account the different xml file format used (SKY Italy vs OzTivo) including the language marker ('ita' ==> 'eng'). So you need to change the e2_loadepg.py configuration so that it knows where to get the files that oztivo_au.sh has downloaded. This is documented in Italian on the dreamboxonline.com forum where this stuff was developed.

Info used on my setup...
/etc/tuxbox/config/dbepg/oztivo_au.dat edited to download only channels I want:
327:69:8=ABC-NT
259:69:6=ABC-WA
25a:69:6=GWN-WA
133:69:3=7CEN
262:69:6=IMP
25c:69:6=WIN-WA
25f:69:6=SBS-NT
12f:69:3=SBS-WA
135:69:3=SBS-Qld
324:69:8=ABC-Qld
321:69:8=ABC-SA
32a:69:8=ABC-Vic
263:69:6=NITV
7d0c:1000:20=ALJA
12e:69:3=SBS-Vic

/etc/tuxbox/config/dbepg/oztivo_au.txt edited to configure other oztivo_au stuff:
#=oztivo UserName and Password are case sensitive
user=poiuytrewqassword
#=user is needed Y or N
userneeded=Y
#=server name and directory
www=www.oztivo.net/xmltv/
#=name of file at oztivo
availfile=datalist.xml.gz
#=build the cache here
cachedir=/media/hdd/ausGuide/epgcache/
#=channel mapping file name
chnlfile=oztivo_au.dat
#=temporary availabilty file (no spaces)
tempfile=/var/tmp/chnltemp.xml
#=remove channel data file once loaded into epg - you might keep it if you are debugging
cleanup=N
#=where to put downloaded channel files while we work on them
datadir=/media/hdd/ausGuide/channelfiles/
#=server for other local boxes - will keep files in serverdir below
server=N

etc.

e2_loadepg.py I installed in /media/hdd/ausGuide/e2_loadepg/
This just a matter of putting e2_loadepg.py, e2_loadepg.conf, and channel_list.conf in this directory. The e2_loadepg.conf file contains (among other things) the loacation of the channel files downloaded by oztivo_au.sh:

XMLCACHEDIR=/media/hdd/ausGuide/channelfiles in my case.

To execute this program you can use TELNET or create a script:

cd /media/hdd/ausGuide/e2_loadepg/
python e2_loadepg.py

After this you need to edit enigma2.sh as per the Italian documentation in order to have the created file picked up and used by the inbuilt epg system.
If you follow the two programs documentation carefully and align them to use the same directories for storing the files then you should have success. (You don't have to use the directories identical to me).

Cheers

[poiuytrewq]

Thanks mate.


It seems to be downloading and caching the files (in the tmp directory) ok but it doesn't seem to be copying them to the hdd? I keep ending up with empty files there.

I'd made some headway on channel mappings and worked out how to extract the right info from Lamedb.

Back to playing with it

[nfnovice]

Thanks mate.


It seems to be downloading and caching the files (in the tmp directory) ok but it doesn't seem to be copying them to the hdd? I keep ending up with empty files there.

I'd made some headway on channel mappings and worked out how to extract the right info from Lamedb.

Back to playing with it

just for clarification my script has two purposes
1. create an empty file in epgcache directory - that just lets us know that we received data for a channel and when it was updated. Its the file name which is important here (channel, date and when updated) - no content in file

2. if localserver is turned on it creates a copy of the actual data in epgserver directory - these will be .xml.tar.gz files - actual xml content is here

so if you are seeing empty files in might be those in point 1

cheers

[poiuytrewq]

Yes, thanks for that NFNovice. It took me quite some time to figure that out but i got there shortly before you told me

Because there is no interface on the 7025 i was looking to those files for confirmation that it was downloading everything.

thanks for all the great work to both of you.

[poiuytrewq]

Ok, all working here finally but i have to say that i am blown away by how well it is working. This is awesome - i now have a 4 day EPG (which is what i've set it for) on pretty much every channel.

Some things i discovered:

I needed to CREATE ALL the directories manually. Check the names in the config files for the names but this included one for the channelfiles (called channelfiles for me) and one for the epgcache (epgcache for me). Initially i only created one of them and greatly confused myself trying to work out why it wasn't downloading.

I've also been mucking around with the channel mapping and these below should allow you to put most of the f0xSt@r channels in your channel_list.conf file:

[channel_list]

0133=1,7 Central,7CEN
80ea=1,ABC1,ABC-Vic
80f1=1,ABC2,ABC2
0263=1,NITV,NITV
0262=1,IMP TV,IMP
7d0c=1,Al Jazeera International,ALJA
4a3a=1,SBS,SBS-Vic
2ee3=1,NG Adventure,ADV1
0fa5=1,Animal Planet,ANIMAL
03ed=1,ARENA,ARNA
3a9d=1,ARENA +2,ARNA+2
1389=1,BBC World,BBC
0f86=1,Bio.,BIOG
0f14=1,Bloomberg,BLM
0bc2=1,Boomerang,BOOM
07db=1,Cartoon,CART
03ee=1,FOX Classics,CLAS
3a9e=1,Classics +2,CLAS+2
2ee6=1,Country Music,CMC
03eb=1,Comedy,CMDY
3a9b=1,Comedy +2,CMDY+2
271b=1,CNBC,CNBC
0fa9=1,CNN,CNNI
1390=1,CI,CRIME
07d7=1,Discovery,DISC
03f3=1,Disney,DISN
2ee8=1,E!,E!
0faa=1,ESPN,ESPN
2ee4=1,Eurosportnews,EUROSPORT
0fa7=1,Food,FOOD
03f0=1,FOX8,FOX8
3aa0=1,FOX8 +2,FOX8+2
0fa3=1,FOX News,FOXN
0bcb=1,Fox Sports 1,FS1
0bba=1,Fox Sports 2,FS2
07d5=1,Fox Sports 3,FS3
0bc1=1,FoxSportsNews,FSN
138f=1,FUEL TV,FUEL
07d6=1,Hallmark,HALL
2ee5=1,Disc Health,HEALTH
07d9=1,History,HIST
1993=1,How To,HOWTO
03f1=1,LifeStyle,LIFE
3aa1=1,LifeStyle +2,LIFE+2
03ef=1,Movie One,MOV1
3a9f=1,Movie Two,MOV1+2
07d3=1,Movie Greats,MOVG
07d2=1,Movie Extra,MOVX
1391=1,MTV,MTV
07d8=1,Nat Geo,NGEO
07d1=1,Nickelodeon,NICK
2eea=1,Nick Jr,NICKJR
791c=1,Channel 9,Nine-Mel
791a=1,Channel 9 Sydney,Nine-Syd
0bbe=1,Ovation,OVAT
426e=1,Playhouse,PHDISN
2ee9=1,Disc Science,SCIENCE
7d07=1,Setanta Sports,STAS
03ea=1,Showtime,SHOW
3a9a=1,Showtime +2,SHW2
0fab=1,Showcase,SHWC
07da=1,Show Greats,SHWGRTS
13fc=1,Sky News,SKYN
0bbf=1,Sky Business,SKYB
0bbd=1,Sky Racing,SKYR
138e=1,SCI FI,Sci-Fi
7d0d=1,Channel 7,Seven-Syd
138a=1,TCM,TCM
0fa6=1,Weather,TWC
2eeb=1,Disc Travel,TRAVEL
03e9=1,TV1,TV1
3a99=1,TV1 +2,TV1+2
791e=1,Channel 10,Ten-NSW
03ec=1,UKTV,UKTV
3a9c=1,UKTV +2,UKTV+2
138D=1,Channel [V],V
2ee1=1,Channel [V]2,V2
138b=1,Vh1,VH1
03f2=1,W,W
3aa2=1,W2,W+2
07d4=1,World Movies,WMOV
1392=1,MAX,max

I still need to get the whole cron setup working properly. It seems to not quite work for me.

Thanks to NFNnovice and AJM for all the great work. AJM: if you can get permission from Lastrico and NFNnovice for the pieces of their code that you have used (and credit them) it would be great to turn the whole thing into stand alone plugin package that can be installed easily for the dm7025 noobs.

thanks,

p.

[poiuytrewq]

Oh, and i noticed that GWN and WIN WA on aurora tend to automatically overwrite the 4 day EPG with their generic messages. Any way of stopping that for just those chans?

p.

[AJM]

If you find a way to stop the 'now' and 'next' overwrite problem let me know. I've posted a feature request on the main DM Forum but no reply as yet. I've also sent emails to SBS, WIN & GWN complaining. Only SBS has replied (to date).

SENT:
>>> AJM <xxxxxxx@xxxxxx.com.au> 6/26/2008 6:27 pm >>>

I live on a remote cattle station and our only source of TV is via Satellite (Optus Aurora on Optus C1 156°E).

SBS is one of the few channels on this service that have no electronic programme guide (epg) data. Instead, SBS fills the 'now' and 'next' epg data slots in the DVB stream with non programme specific info. For example the channel named 'SBS WA' shows 'SBS National Television' with a duration of 1440 minutes as the 'now' programme and the 'next' programme contains the same description!

Therefore we are unable to see the name of the current programme nor do we know what the next programme will be. (This information is available on the other channels such as ABC, 7 Central, Imparja, and even NITV!) Why can't SBS do as well as the other channels. or even better?

There are often good programmes on SBS but we easily miss them when checking out what the channels have to offer. Read the newspaper you say. well we only have irregular deliveries of mail and news.

Come on SBS catch up with the others.

Any advice on this issue would be appreciated.

regards
AJM

REPLY:

Dear AJM,

Thank you for your comments concerning the Electronic Program Guide (EPG).

SBS is currently reviewing its digital channel strategy. Due to the expense and technical complexity of running an EPG Channel SBS is not transmitting a full service at present.

However, we are hoping to roll out a full EPG guide in December.

Please accept our apologies for any inconvenience.

For your interest, if you have access to the internet, SBS provides 2-3 weeks of programming in advance on our website at:

Regards

Packaging of LoadEPG

I plan (not sure how long) to update Will Uther's tv_grab_au_reg.py to implement caching (it may be courteous for me to get his blessing at some stage). Likewise I will fiddle with e_loadEPG a bit more. Then I can make packages for the three alternatives: native epg.dat, MView-E2 and PP-SQLite . But this is not likely to get done overnight!

Incidently, MView-E2 can co-exists with the native epg.... so that's the only way of seeing the 'now' and 'next' area of WIN & GWN. And as for all those other channels you've got... how can you possibly watch them?

cheers

[AJM]

Oh, and i noticed that GWN and WIN WA on aurora tend to automatically overwrite the 4 day EPG with their generic messages. Any way of stopping that for just those chans?

p.

LT 5 has fixed this problem nicely .

They also provide a grabber written in python and therefore much quicker... however, it doesn't implement caching. (It has a minor time bug ie it ignores the minutes component of the source xmltv file timezone).

I'm going to use LT 5 now after having used Gemini for over a year!

cheers

[poiuytrewq]

Any luck getting it running AJK? I am having a play with it here and haven't yet been able to get it download the channel info? You had any luck with getting the XML file right?

[AJM]

Any luck getting it running AJK? I am having a play with it here and haven't yet been able to get it download the channel info? You had any luck with getting the XML file right?

Here is mine... works for me:

<?xml version="1.0" encoding="UTF-8"?>
<LTEpgGrabber>
<config>
<timezone>+00:00</timezone> <!-- Offset between source and box in this case -->
<days>5</days>
</config>
<channel>
<name>ABC TV WA</name>
<type>xmltv</type>
<service>259:6:69</service>
<url>http://xml.oztivo.net/xmltv/ABC-WA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>ABC TV SE</name>
<type>xmltv</type>
<service>32a:8:69</service>
<url>http://xml.oztivo.net/xmltv/ABC-Vic_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>ABC TV Q</name>
<type>xmltv</type>
<service>324:8:69</service>
<url>http://xml.oztivo.net/xmltv/ABC-Qld_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>GWN TV</name>
<type>xmltv</type>
<service>25a:6:69</service>
<url>http://xml.oztivo.net/xmltv/GWN-WA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>7 Central</name>
<type>xmltv</type>
<service>133:3:69</service>
<url>http://xml.oztivo.net/xmltv/7CEN_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>NITV</name>
<type>xmltv</type>
<service>263:6:69</service>
<url>http://xml.oztivo.net/xmltv/NITV_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>IMP TV</name>
<type>xmltv</type>
<service>262:6:69</service>
<url>http://xml.oztivo.net/xmltv/IMP_(yyyy)-(mm)-(dd).xml.gz</url>
</channel> -->
<channel>
<name>WIN</name>
<type>xmltv</type>
<service>25c:6:69</service>
<url>http://xml.oztivo.net/xmltv/WIN-WA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>Al Jazeera</name>
<type>xmltv</type>
<service>7d0c:1B:1000</service>
<url>http://xml.oztivo.net/xmltv/ALJA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>SBS WA</name>
<type>xmltv</type>
<service>12f:3:69</service>
<url>http://xml.oztivo.net/xmltv/SBS-WA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>SBS SE</name>
<type>xmltv</type>
<service>12e:3:69</service>
<url>http://xml.oztivo.net/xmltv/SBS-Vic_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>SBS QLD</name>
<type>xmltv</type>
<service>135:3:69</service>
<url>http://xml.oztivo.net/xmltv/SBS-Qld_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>WLK</name>
<type>xmltv</type>
<service>25b:6:69</service>
<url>http://xml.oztivo.net/xmltv/31-Per_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<config>
<timezone>-01:30</timezone> <!-- Offset between source and box in this case -->
<days>5</days>
</config>
<channel>
<name>ABC TV NT</name>
<type>xmltv</type>
<service>327:8:69</service>
<url>http://xml.oztivo.net/xmltv/ABC-NT_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>ABC TV SA</name>
<type>xmltv</type>
<service>321:8:69</service>
<url>http://xml.oztivo.net/xmltv/ABC-SA_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
<channel>
<name>SBS SA</name>
<type>xmltv</type>
<service>25f:6:69</service>
<url>http://xml.oztivo.net/xmltv/SBS-NT_(yyyy)-(mm)-(dd).xml.gz</url>
</channel>
</LTEpgGrabber>

each service is identified as
<service>ServiceID:TransponderNo:ProviderID</service>

so you'll need to get all that together for your payTV stuff.

cheers

PS. I've just edited the above xml to split the channels into two groups: the first with zero correction and the second with -00:30 timezone correction (Yes... using -01:30 gives -1800 seconds... another bug caused no doubt by using modulo arithmetic with negative numbers). This was necessary because the grabber doesn't read the input file minutes properly. May change again if LTTEAM correct various arithmetic bugs.

cheers again