Over the years we have assembled a variety of tools for evaluting tuner performance. Most of these are internal home-grown command line utilities and scripts which perform some superset of comparable end-user applications, with a particular emphasis on automation and logging statistics that regular users typically wouldn’t need.
We use these tools for a variety of purposes:
- Comparison of various products to assess performance (to aid product selection for our commercial customers)
- Validation of drivers under development in terms of reliability and performance
- Detecting regressions during development (i.e. the ability to run an automated suite against the same driver regularly to identify cases where some code change introduced a bug or performance problem)
- Testing edge cases that are annoying and/or labor intensive to do by hand
These are tools we use every day against our own drivers to establish confidence that we’re delivering a high quality product. After all, how can you claim to have a high quality product if you’re not actually measuring in some objective way?
We have been running the tools againt our driver for the HVR-955q for months on a regular basis, at this point mostly just to catch regressions that might have snuck in during development for other tuners. However as a result of some unrelated work, I happened to install the latest open source media_build tree, which included the recently upstreamed open source driver for the HVR-955q. This provided an ideal opportunity to do a bit of comparison.
Why compare our driver against the open source driver? As an engineer, I want to be confident that our driver works well, and being able to compare it against somebody else’s driver for the same device allows me to ask several questions:
- Does the open source driver have better signal performance than ours in any specific cases? If so, why, and how do I fix it?
- Does the open source driver manage to get a lock on any channels that our driver missed? If so, why?
- Do we see comparable tuning time for typical use cases? If not, why?
In short, these are mostly sanity checks to ensure we didn’t do anything really dumb that we weren’t already catching.
Now for some pretty charts our scanner produced:
As you can see from the above, we’re locking on all the same channels (i.e. no channels missed), and the SNR is pretty much the same between the two (within 1dB). There might be a bit of opportunity for fine-tuning, but I would probably first need to make sure the means of computing the SNR are identical (i.e. the variation might just be a difference in the SNR algorithm).
Based on the above, the we’re seeing pretty much the same trend for signal power delivered to the tuner (+/- 1 dBmV). That said, there does appear to be something going on in the open source driver where it doesn’t perform as well below 375 MHz, where if the results were reversed I would want to better understand what was going on there. This might suggest a case where the driver wouldn’t perform as well in weak signal conditions.
All that said, one thing I immediately asked myself when doing this test:
Why on Earth is the scan taking so long when run against the open source driver?
My initial thought was that perhaps there was some bug in our internal tool which was exposed when running against the 955q open source driver (although we’ve run it against other open source drivers countless times). That prompted me to look at the time-to-lock for the tuner:
That looks nice and consistent (locking consistently within 50ms between the open source and Kernel Labs driver). Again, perhaps an opportunity for a bit more fine tuning, but nothing bad going on here.
Then I looked at the chart that shows lock time, including the time before the tuner gives up attempting to lock when a channel isn’t present. /Cue dramatic music here…
Hmmm, so with the Kernel Labs driver it consistently takes 3 seconds to give up on channel tuning, and with the open source driver it takes ***80*** seconds. The total scan time when run against the Kernel Labs driver is about 14 minutes, whereas with the open source driver it was 34 minutes.
Going back to my assumption that my tool was broken, I tried the same test with the stock Linux DVB /usr/bin/scan tool. Guess what? I got the same results.
These tests were run against a cable plant that was essentially full (about a dozen frequencies out of 135 which weren’t in use). When the same test is run against an empty plant (i.e. unplug the cable feed), the problem is exacerbated:
Scan time with Kernel Labs Driver: 8 minutes
Scan time with Open Source Driver: 152 minutes
Yup, it takes over two hours to do a QAM 256 scan of a cable plant where no channels are found. Ouch.
Why does this matter? Isn’t an empty cable plant a rare edge case? Unfortunately, it is not, especially for commercial customers. This is for a couple of reasons:
- Commercial customers often deploy against custom cable headends where they have only a few QAM modulators present, as opposed to a public cable headend provided by an MSO. In such a case you might get a signal lock on a dozen valid channels and 100+ empty channels.
- Commercial customers will often write their application to try all three scan types against a given cable feed (i.e. 8VSB, QAM64, QAM256), since they don’t trust the user to be able to specify whether their source is terrestrial or cable. Such a scan would take upwards of *six hours* with the open source driver, whereas it would take only around 20 minutes with the Kernel Labs driver.
The goal here isn’t to claim that the Kernel Lab driver is awesome and the open source driver is junk. Drivers sometimes have bugs, and it’s possible that the bug in the open source driver is shallow and easy to fix. However it does raise the question:
If stating a product is of high quality is the result of measurement, who among the people responsible for the open source driver is measuring its performance?
As I originally suggested in the previous blog post where I announced the availability of the Kernel Labs HVR-955q driver, the open source driver is probably “good enough” for consumers, but if you’re a business that depends on tuners working well, it probably makes sense to go with a product specifically intended for that audience.