net6501 (Red Light Of Death) debugging

Hi!

The first of several net6501 units bought only 3 years ago are starting
to show the "Red Light of Death" symptom.  Being somebody who designs
embedded electronics and software for a living, I thought I'd invest a
(limited) amount of time into investigating this.

The obvious thought is of course to re-cap the board an as dried-up
electrolytic capacitors are a common aging problem of any electronics,
particularly switched voltage converters.

I ordered suitable replacement parts and replaced all aluminium caps,
both THT and SMT ones.  Testing the old ones however showed clearly
that there's nothing wrong with them:

position	nominal		actual
C406		820 uF		816 uF
C397		820 uF		819 uF
C30		150 uF		148 uF
C31		150 uF		155 uF
C380		220 uF		225 uF
C427		680 uF		748 uF
C418		680 uF		716 uF

So everything is well within normal expected tolerances.

The interesting part is also that this unit has not been used much.  It's been
sitting in storage unpowered for most of its life, hence any thermal aging
that might have dried up the electrolyte would have had to happen at room
temperature ;)

While looking at the board and probing around with a scope I found the
following noteworthy observations:

1) there's *very* intense chirping coming from the board.  Trying to locate
   it is a bit difficult, but using a "paper roll" between my ear and sections
   of the board it appears to come more or less directly from the CPU ?!

2) the chirping sound clearly correlates to the load of the point-of-load
   DC converter implemented using Q19.  It changes either by load of the
   consumer (I think it generates a 1V rail for the CPU?) or by the input
   voltage of the external DC power supply.  During successfull boot,
   the chirp is gone for the first 2s after reset, and then present for
   5s, after which it is gone again. Shortly later, the RED led is
   switched off.

3) changing the DC input voltage has a significant effect on the chirping.
   A peak of chirping is at 12.3V, and at least another one at 11.1V.

4) if the board is powered up, the current consumption very early on gives
   away if the unit is booting correctly or not.  Is the current consumption
   at ~ 470mA @ 15V input, it's not booting.  Is the current consumption about
   600mA @ 15V input, it is starting up correctly.

5) the chirping also gives away if the board is booting correctly or not

6) using just the reset button to power cycle, even many dozens of times
   will not recover, i.e. not trigger a good boot.  That seems only possible
   when actually cycling the DC input voltage

7) DC input voltage seems to have some kind of impact on the probability of
   a boot succeeding or not.  My best results were at 8.7V DC input.

8) The voltage at the aluminium cap charged by Q19 is very stable.  No
   significant switching noise is present, i.e. the filtering of the
   voltage rail works as it is supposed to.  Excess switching noise is
   thus unlikely causing this problem.

9) if  you look with a scope at the gate of Q19, you can observe that if
   there's regular / periodic switching at ~ 180 kHz when the board
   boots successfully.  When the board boots unsuccessfully, there's
   only irregular switching pulses visible, so the converter is probably
   in some mode where it stops/pauses at no/little load on the output

As a summary, I couldn't find anything wrong with the onboard DC/Dc
converters so far.  The bug is likely somewhere else, and the difference
in current consumption is just reflected by different operating points
of the DC/DC converters

That's it for now.  Maybe this is useful to share.  At this point I'm
not sure yet, if this will lead anywhere.

When I find another timeslot, I want to compare a not-yet-broken board
with one that exhibits the problem.

Regards,
	Harald
-- 
- Harald Welte <[email protected]>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)