Some thoughts on sectionsd

[FreddyFr0g]

I'd like to ask you guys a couple of things (@Seife or Rhabarber)

1. We noticed that some cable operators in the UK are putting long descriptions in the short description markup causing the secondsExtendedTextCache to be ignored. Is that something worth changing in the cvs or is that too specific to us in the UK! (those lines:

Code: Alles auswählen

		if ((evt.getExtendedText().length() > 0) &&
				(evt.times.begin()->startzeit < zeit + secondsExtendedTextCache))
			si->second->setExtendedText("OFF",evt.getExtendedText().c_str());
		if (evt.getText().length() > 0)
			si->second->setText("OFF",evt.getText().c_str());

we could have the && (evt.times.begin()->startzeit < zeit + secondsExtendedTextCache) restriction on the if (evt.getText().length() > 0) as well.

2. I'm working on getting 7 days epg from a different pid on our UK cable operator and I found a problem with the event_id marker... basically for the same event, the standard and the 7 days feed have a different event_id which causes duplicates when adding an event because the unique key is made of the channel freq + the event_id.... how about having that unique key made of the channel freg + the start_date/time?
I remember having the same kind of problems when I scripted my online EPG grabber (taking the data from a website an generating the xml files that sectionsd is then reading by using the sectionsdcontrol --repg) because I didn't have an event_id from the online feed!

Surely other grabbers have the same problems (TvInfo/Klack) and changing the unique key will get rid of the duplicates (events from the download and events from the stream)?

Any thoughts?

Thanks

[seife]

So you want to ignore short and long epg that's further in the future than secondsExtendedTextCache?

I'd at least combine it with a "if (time > xxx && epg.text.length() > 150)" condition or similar...

And yes, the duplicates based only on the event ID is a problem, sometimes here newer events also come with a newer ID and that causes small problems, too. I'm not sure what's the best way to solve this. I fear that only the time surely is also no solution... Somebody should check how vdr does it. It seems to work pretty well there.

[FreddyFr0g]

You mean, check first that it's long enough to be considered as long description? Good idea!

I had a quick look at vdr and I'm a bit confused! Looks like they are using 2 hashes (one with the event id and one with the startime) - so I'm probably not far off with the start time stuff .
But I also noticed those comments:

Code: Alles auswählen

                  // The segment overwrites all events from tables with higher ids, and
                  // within the same table id all events must have the same version.
                  // We can't delete the event right here because a timer might have
                  // a pointer to it, so let's set its id and start time to 0 to have it
                  // "phased out":

Which is from their cleanup function, so is the epg drop as soon as they have a new version number? So like newer events with newer ID (like you describe in your post) would be added but the 'old' ones would have been dropped because the version number is not current enough? Anyway, I know only very little about C and sectionsd so it was just a quick comment on a quick attempt to descrypt how vdr is doing it (having never seen vdr in action!)!

[seife]

Yes, some cleanup function like that is definitely needed. But it needs some more infrastructure and I have never reached the point of actually implementing it

[FreddyFr0g]

I understand!

In the meantime, I'm working towards a quick patch for this and replace the uniquekey() comparison with a customised function (that takes into account the start time) and the last insert checks that as well before inserting.
Basically everytime I had domething like

Code: Alles auswählen

.uniqueKey()    != e->uniqueKey()

or

Code: Alles auswählen

.uniqueKey()    == e->uniqueKey()

I replaced it with

Code: Alles auswählen

 compareChannelIdAndStartTime((*myNextEvent),e))

for example
and the last insert:

Code: Alles auswählen

if (!isAlreadyThere(evt)) 
		{
		  mySIeventsOrderUniqueKey.insert(std::make_pair(e->uniqueKey(), e));
...

With the following functions

Code: Alles auswählen

bool compareChannelIdAndStartTime(const SIevent &evt,SIeventPtr &e) 
{
  if ((evt.get_channel_id()==e->get_channel_id()) && 
	    (evt.times.begin()->startzeit==e->times.begin()->startzeit) ) {
    return true ;
  } else {
      return false ;
  }
}

bool isAlreadyThere(const SIevent &evt)
{
	MySIeventsOrderUniqueKey::iterator e = mySIeventsOrderUniqueKey.begin();
	bool found;
	found=false;
	while ((e != mySIeventsOrderUniqueKey.end()) && (!found) && (mySIeventsOrderUniqueKey.size()>0)) {
	  if ((evt.get_channel_id()==e->second->get_channel_id()) && 
	    (evt.times.begin()->startzeit==e->second->times.begin()->startzeit) ) found=true ;	    
	  e++;
	}
	return found;
}

That seems to work (i.e. getting rid of my duplicates) although the number of events captured has only dropped by about 3000. I would be extremely grateful though if you could cast a very quick eye on the function to see if they can be improved (just like the way they are declared with the static stuff not going into details!... as I know nothing about C programming!)

I guess some of those events not being there are thanks to your latest work on the Now&New Thread... I noticed that the N&N thread doesn't update all the channels in one go now but waits that you zap to one. I thought it was a bit slower to get it than before (i.e. you zap to a channel and you have to press ? again to see the epg) but that might just be because your latest work in combination with my crap patch !

Thanks in advance! And thanks for all the great work on sectionsd

[FreddyFr0g]

@Seife

Hi, you know when I was saying that the Now and Next Thread was a bit slow to get data? Well, I think that there might be a minor bug there.
There are 2 filters for that thread (one n&n data for current/ts, one for n&n data for other/ts *I think*) BUT I can't see the 2nd filter being changed! Something like:

Code: Alles auswählen

				if ( dmxCN.filter_index + 1 < (signed) dmxCN.filters.size() )
				{
					dprintf("[cnThread] skipping to next filter(%d) (> DMX_TIMEOUT_SKIPPING)\n", dmxCN.filter_index+1 );
					dmxCN.change(dmxCN.filter_index + 1);
				}
				else
					sendToSleepNow = true;

in the CNThread is needed!

EDIT:
Sorry, just ignore me... I added the other filter in my patch! So no bugs in the CVS... only in my customised code lol !