Haven’t had much time to update the blog lately. All week I’ve been quietly grinding away at two basic issues: PAL VBI support for the em28xx chip, and identifying/addressing the HVR-1600 performance problem.
I often get questions from users along the lines of “how do you make boards work?”. Reverse engineering is a bit of a black art, but I figured I would offer some insight as to how the process works. If you don’t actually care how tuners are designed and how we figure out how to make them work, stop reading now. If this stuff interests you, read on, and feel free to put something in the comments if you want to see more posts of this nature. I’m still gauging whether there is enough interest in this topic for me to go through the work associated with writing about it.
So let’s take a step back for a moment: what am I trying to accomplish? In this case, there have been reports for quite a while that the Linux tuning performance of the HVR-1600 is worse than under Windows. In fact, people have pointed out that the SNR being reported is about 3dB lower under Linux. This results in visual artifacts being displayed, as well as not being able to hold a signal lock on weaker signals.
In this case, my suspicion is that either the mxl5005s silicon tuner or the cx24227 demodulator is not being configured properly. So the logical approach would be to see how it’s programmed under Windows, and compare that to how it’s programmed under Linux.
But then I hear you screaming, “But wait! You don’t have the source code to the Windows driver, so how do you know how the tuner and demod are configured?” Well, that depends. If it were a USB device, I would use a tool such as SniffUSB 2.0 to capture the USB traffic, and then decode the i2c traffic to see how the register is programmed. In this case, it’s a PCI card, so I cannot just look at the bus. Now what?
“When in doubt, use brute force.”
This next step needs you to think like a hardware developer. Hardware designs don’t work overnight. They aren’t magical, and like software they can have problems and need to be debugged. As a result, hardware engineers often add what are known as “test points” to a printed circuit board design. This makes it easier for him/her to poke around with an oscilloscope or attach debugging hardware. On debug boards, test points have pins attached. On production hardware, the test points are often still there but they simply don’t populate the board with the pins, since they are not needed and add cost to the bill of materials.
So, were the Hauppauge engineers who designed the HVR-1600 thinking about how they were going to be able to debug the design? Apparently so:
[lightbox title=”Em28xx PAL VBI” href=”../../blog/wp-content/uploads/2009/10/hvr1600tp-300×225.jpg”]
[/lightbox]
Oh look, two through-holes on the board marked “TP10” and “TP11”, right next to the EEPROM? Could those pins expose the HVR-1600’s i2c bus? Yup.
But don’t I need thousands of dollars worth of special hardware in order to read the i2c bus? Well, logic analyzers used to be really expensive, but now you can get a very capable logic analyzer for less than $200. In fact, I bought my Saleae Logic from SparkFun Electronics for $149.95. Break out your soldering iron, hook up a couple of pins, and start capturing i2c traffic. This approach lets you record the traffic using the same method regardless of whether you are under Windows or Linux. The logic analyzer even has the ability to decode the i2c traffic and export the data in CSV format to a text file:
Time [s],Event
"1.5155000",Start
"1.5155200",Setup write to 0x19. +ACK
"1.5156900",Wrote 0xF2 to 0x19. +ACK
"1.5158650",Wrote 0x00 to 0x19. +ACK
"1.5160350",Wrote 0x00 to 0x19. +ACK
"1.5162200",Stop
"1.5162250",Start
"1.5162450",Setup write to 0x19. +ACK
"1.5164200",Wrote 0xFA to 0x19. +ACK
"1.5165900",Wrote 0x00 to 0x19. +ACK
"1.5167650",Wrote 0x00 to 0x19. +ACK
"1.5169450",Stop
...
The raw data isn’t particularly easy on the eyes, but with a bit of Perl-Fu, you can turn it into something much more manageable:
0xF2 0x00 0x00
0xFA 0x00 0x00
...
A look at the existing s5h1409 driver shows that the first byte is the register address, followed by a 16-bit value. Since we are interested in the state the registers were left in when I stopped the capture, we need to weed out all the older register writes. Perl comes to the rescue again. Just spin over the file, and load the values in to a hash with the first byte as the key:
while (my $line = <FILE>) {
chomp $line;
my @vals = split / /, $line;
$final{$vals[0]} = \@vals;
}
Take the resulting captures for both Windows and Linux, pass it through sort and diff, and what do we get?
root@ubuntumb:~/1600traces# diff nbcwin1.txt.final nbclin1.txt.final
26,27c26
< 22 0x2F
< 25 0x36
---
> 22 0x3F
We went from a 1780 line analyzer capture, down to a file with 94 mxl5005s registers writes, down to a five line file showing the two mxl5005s registers that are programmed differently. Total code required: 64 lines of Perl.
The diff for the s5h1409 register was a little longer (eight registers), but it’s still much easier to use some regex magic to avoid having to hand inspect and compare over two hundred register writes.
As a test, we can now just hack a few register writes into the driver right after the tuning command and see if it works…. Yup. I’m now getting an extra 3dB compared to before I tweaked the driver. Now I just have to work backward and understand why the Linux driver programmed the chip with different values.
Total time spent from plugging in my soldering iron to verifying the 3dB improvement under Linux: most of Sunday afternoon.
I hope some of you have found this little tutorial informative/educational. If so, feel free to leave a comment saying so, as this will tell me whether anybody actually cares about this stuff and whether I should write about it in the future.
I would absolutely love more tutorials on reverse engineering hardware, creating drivers, and anything else along those lines. I like the way you did the write up for this one, but if you decide to do a series you might want to explain a few more things, like many people are probably not familiar with i2c etc.
Keep up the good work
Absolutely, Devin. In the military, I worked on electronic encryption systems, so I know about test points well. Because we had nearly unlimited taxpayer funds 😉 available, we used oscilloscopes to identify proper operation at the various test points on the hardware. With our accumulating debt, even the military will probably be using logic analyzers in the near future. 😉 Anyway, someone’s ALWAYS paying attention, Devin. 🙂
It’s funny you say that, since at my last job was at a company that built encryption hardware for government agencies. When done properly, pretty much nothing I could offer on this blog would be very effective on such devices (think FIPS 140-2 Level 4 protections and beyond)…
If you get some time, you might discuss the various types of test equipment further, e.g. MSO. Keep up the great work!
Thanks Devin,
This was fascinating. There’s nothing like the occassional “behind the scenes” post to stir up interest and admiration.
Very interesting. I always love hearing how things like this are accomplished.
This was a very interesting article- I kind of hope you do write some more like it.
Reading it makes me wonder if there’s any kind of “prep work” folks could do to make bringing up cards faster (sort of like what you did above there).
Devin, I found the article fascinating, thanks ! I don’t know that I could do this stuff, but its absorbing reading.
yes, please keep the tutorials coming.
My question to you about the test points: what indications did you have from the silk screen print that that these two test points were i2c compliant? Did the EEPROM give it away?
For this example what type of operations did you use to get your data to compare it Windows i.e. do driver init and watch t.v.?
NAN,
That’s a good question. I should write a post called “Finding the i2c bus when you don’t have the pinouts for the chips”. I’ve got a few different tricks in this area.
In this case, EEPROMs typically have a very low pincount, and there isn’t much interesting that someone might want to debug that close to the EEPROM other than the i2c. Of course, this trick doesn’t always work, in particular in cases where a bridge has multiple i2c busses (I’ve seen boards that have a two busses – one dedicated to the eeprom and the other for ‘everything else’).
Please write this post. I would love to see some tricks to finding the bus in the first place.
If there doesn’t seem to be an i2c bus or even a rs-232 to see some sort of output would you have to resort to a pci bus analyzer?
NAN,
Well, even if there is not an i2c bus exposed through test pins, there pretty much always is a bus. You just have to work a little harder to find it.
However, there are cases where you need to see things other than i2c traffic, and there are some ways to do that short of a PCI bus analyzer (which tend to be cost prohibitive). For example, dscaler bundled a tool for poking around at cx88 registers under Windows, which is pretty useful for getting a snapshot of the current register state.
Devin
sharing info about “white hacking” is very useful IMHO.
so i’d like to point you to a less expensive tool called “bus pirate”
http://dangerousprototypes.com/2009/10/04/prototype-bus-pirate-v3/
it’s completely hardware and software “open source”. of course it’s not free, BUT i ordered one for something like 20€.
it drives as a “host” a lot of peculiar low speed bus (SPI, I2C, JTAG and so on..)
AFAIK the i2c sniffing ability is something still in developo, but i’ll tell you more about it WHEN i’ll try my own.
hope it helps..
of course i’m not business related to those producing this stuff. just an info “AS IS”: YMMV.
regards
Despite coming from a security background and being familiar with the differences between “hacker” and “cracker” and the subtleties of “white hat” versus “black hat” and “grey hat”, I typically refrain from associating the term “hacking” with anything I do. It’s just too easy for people to misinterpret the intentions.
As for the bus pirate, it looks like a neat piece of hardware. There are indeed much cheaper solutions for i2c bus analysis (Google ‘warmcat’ and ‘cheap i2c’). However, I personally like the Saleae Logic because it’s a real logic analyzer that lets me see the actual waveform. The simpler dedicated i2c solutions are good in many cases, but I have seen more than one case where the problem was with the i2c bus implementation itself, and being able to visualize the actual waveform has proven itself very helpful for bugs like that (other solutions would show me the wrong data, but I wouldn’t be able to see *why* it is wrong).
Devin
Very nice, I’d like to see more.
If you make some video you might also want to post it to securitytube.net there’s a lack of videos about reverse engineering there and I think all the current ones are about software only, but it’d be nice with a little hardware too.
And of course hackaday.com would be a good place to post about it too, even if it’s not with video.
Hello Ebidk,
I’m just not sure the incremental value in having the content in video form is worth the effort of producing/editing it. Frankly, I actually think the content is *better* in text form (with photos where appropriate), where I can provide not just photos but screen dumps and output from commands that people can see or cut/paste. People don’t have to keep hitting pause/rewind just to see the output printed on my screen or trying to digest some of the more complex concepts. In textual form, the content is also more likely to be found by search engines.
I’m just not confident that many people would be willing to sit through a ten minute video just to watch and listen to me connecting leads to a logic analyzer. Don’t get me wrong, there are plenty of cases where I think instructional videos are great – I just don’t think this is one of them.
Feel free to post a link to the content from hackaday.com. The goal is to share the knowledge.
Devin
I’m mostly with you on this: producing video (especially good video) is very time consuming. Good text/picture write ups are much better. On the flip side, I think a poorly done video dump can be more informative than a poorly done text/photo writeup. This is because the video shows *everything*, so gets around one of the classic teacher’s dilemmas: getting into the “beginner’s mind”. What seems trivial to you (say, hooking up a logic analyzer correctly) may be just the thing a newbie needs.
Even so, stick with what your comfortable with: any info that’s at least as good as what you post is absolute gold for a lot of us. Thanks.
Awesome article – would love to see more like this.
Great post. I’ll definitely be reading follow-ups if you have the time to write them.
Nice work. I really like the linked logic analyzer. I have an old Tek ‘scope that I use to debug my designs. 2 channels + trigger aren’t quite enough to debug SPI (it’s possible, but extremely painful). More tutorials would be a treat!
In fairness, I use an eighteen year old dual-trace Tektronix scope for the bulk of my debugging. One of the key messages I’m trying to impart is that you don’t need thousands of dollars worth of hardware to do this stuff.
Devin
This is my first time on your site, and I can honestly say that you now have my attention. I’ve been getting into all things hardware recently (being a software guy for the most part), and this is exactly the type of stuff I am looking for.
Thanks!
Same as CB’s comment above, first time visit, honey pot, bookmarked. Thanks bro.
This was an excellent post. I think there are way to few blogs out there describing low-level programming and reverse engineering.
I would be very interested in reading how you have used your oscilloscope when writing/debugging drivers.
Thank you
You provide some excellent tips relative to the I2C bus. I agree with all the comments above but also add interest in investigating modifications for the HVR-1600. Did you notice any additional potential for the card?
Hello Yar,
The HVR-1600 is a pretty full-featured hardware design relative to what can be done with the underlying components. There don’t seem to be a whole lot you could do with the hardware that isn’t being done today. It’s not one of those cases where the PCB is used for different products and is “defeatured” for this particular product. The design does contain an additional connector which provides access to additional inputs, but these are well known in terms of how they can be used and if I recall Hauppauge actually sells the cable as an addon (it’s not any sort of “secret functionality”).
Devin
Great article. Thanks!
can you post the perl script please?
I can certainly do this, but I have to admit I would be pretty embarrassed to show it to anyone. The scripts clearly fall into the category of “get the job done” as opposed to the sort of code I usually show to others which is targeted to go into some sort of product and have a *much* higher level of quality.
Now if the above disclaimer is acceptable then sure, what the hell, I’ll throw the code up on the server. Keep an eye on the blog and I’ll see about doing it sometime in the next couple of days.
Devin
awesome work, I have the same tuner and the same problem under linux, could u post your diff so I can see if it works for me?
Hello Paul,
Please keep an eye on the blog and/or the linux-media mailing list. A tree will be posted and there will be a public call for testers.
Devin
Thanks! I enjoyed the article and look forward to testing out the changes. I was wondering why my 1600 was having pretty bad artifacts even with a new amplifier in place(the house has questionable cabling to start with).
Thanks for all y’all are doing and I hope that you can post more of this type of stuff.
One thing to watch out for is sometimes amplifiers can have a negative effect. They can overdrive the signal, making it harder for the tuner and demodulator to do its job.
You may wish to check your SNR both with and without the amplifier. If the SNR is reading as 40.0 dB (running azap or femon shows the SNR field as 0x0190), then you are probably overdriving the signal and may be better off without the amp.
Devin
I had originally thought of that.
Currently, with the amplifier and the command azap 110(or whatever channel I’ve got), it’s getting about 0138-013a.
The PCHD-5500 digital also gets a clear picture from the same amplifier, so I’m left to guess that it that card’s digital portion.
Again, thanks for all y’all are doing.
Matt,
Are you trying to do over-the-air ATSC or ClearQAM?
Devin
I’m using Coax cable with ClearQAM.
Matt,
Ah, ok. Yeah, you should definitely try out my tree when I release it. You should see about a 2.5dB improvement, and the difference between 31.2 and 33.7 can definitely be between “crappy/blocky picture” and “perfect picture”.
I would also be interested in hearing whether you are seeing BER and UNC errors showing up with azap (I suspect you do).
Devin
Thanks! Look forward to checking back and trying out the new fix.
You’re right in that I am getting BER & UNC errors. Every couple of seconds I’ll get a non zero in those columns.
Ok, well then you may very well see an improvement with the fixes. Keep an eye on the blog and there should be a call for testers in the next couple of days.
Devin
Love it. Very informative. Keep it coming -Read about you on Hackaday…
I too would love more articles like this one!
hi, nice article, although no single step here is difficult, and the use of ther seeprom as a point to hook into i2s an old trick, knowing what to do next take some real knowlege and understanding. i hope to see more articles from you in the future!
Additional: i also have a Saleae, they really are very very good for the money.
Rowan,
I’m certainly not claiming to have invented the idea of latching onto the eeprom to get the i2c traffic. The eeprom is a particularly desirable target even without test pins because of the size of the typical package and spacing on the pins. In fact, I was able to clip the analyzer probes directly onto the eeprom, although it would have been more difficult because the PCI card had to be installed in the machine.
That said, lots of people are unfamiliar with that technique, and one of the big goals is to share some of the “tricks” that you and I probably take for granted.
Cheers,
Devin
Thanks, please continue to write, people care!
Hi Devin,
I found the post very interesting.
I would like to know how to decode the i2c traffic captured with UsbSnoop to see how the register is programmed.
Is there any howto somewhere?
Thank you,
Xwang
Hi Xwang,
The format for i2c traffic sent over USB varies depending on the USB bridge. For example, for em28xx I typically use a tool called parser.pl, written my Markus Rechberger. He doesn’t make it available anymore (it was written for his “usbreplay” tool), but from what I understand there is a tool bundled somewhere in the v4l-dvb tree that performs a comparable function. For the au0828 bridge, I wrote a parser from scratch in Perl.
Devin
“As a test, we can now just hack a few register writes into the driver right after the tuning command and see if it works”.
Really enjoyed this.
Just wondering how you put the register writes into the driver and what format they were in and that aspect of things?
Thanks, Rob
Rob,
I just had to add a few s5h1409_writereg() calls to the end of s5h1409_set_frontend() in s5h1409.c. By doing that, I was able to confirm whether there was an improvement in SNR, and from there determine the appropriate heuristic for when to program those registers.
You can see the final result of what was ultimately required to fix the issue here:
http://kernellabs.com/hg/~dheitmueller/hvr-1600-perf-2/
Devin
Ah i see!
Very clever! (Thanks for the quick response too!).
Thanks, this is definitely good information for the linux community.