Redundant clock switching

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
I'd like to implement a clock distribution system with a clock
source (24MHz) switchover in the case of failure. Not exactly
a trivial problem, especially when a smooth transition between
the clocks is required. Best to be delagated to a ready-made
chip, which... barely exists. The only two chips I was able to
find are the ICS581-2 (perfect match for my needs) and ICS580
(3 cycle stall, don't know yet if it's a problem or not). The
first one turns out to be unobtainium and Mouser labels it
as obsolete. So what is the modern way of doing this? I don't
think I have a sudden attack of dysgooglia, there seems not
to be much about the issue.

On a similar note: what oscillators should be used? Two exact
copies of the same part, or a combo of a crystal oscillator and
a MEMS? Is there any wisdom handed down over the ages about
the long-term reliability of MEMS oscillators?

    Best regards, Piotr

Re: Redundant clock switching
On Tuesday, 10 April 2018 09:36:14 UTC+1, Piotr Wyderski  wrote:
Quoted text here. Click to load it

Maybe I'm missing something here. But the use of an oscillator as the missing chip seems the obvious option. Your 2 other oscillators phase lock it, but if their input ceases to exist it oscillates on its own, preserving output.


NT

Re: Redundant clock switching
snipped-for-privacy@gmail.com wrote:

Quoted text here. Click to load it

The secondary stage can die and then you are left without the signal,  
despite the first oscillator still working correctly.

Doing this properly is a tricky stuff, so I wanted to pay somebody else
for doing it for me. But it turns out there are 2 chips that can do that  
(meaning, they have built-in loss of signal detectors), one of which is  
not available even as a sample. What's going on?

     Best regards, Piotr

Re: Redundant clock switching
On 04/10/18 07:32, Piotr Wyderski wrote:
Quoted text here. Click to load it


A few more words about what sort of failure you're guarding against  
would be useful.  With all the possible failure modes of a complex  
system, why pick on the poor oscillators?

Failure of an external clock, that I get.  But kitty's approach seems  
like it would cover that one.

Cheers

Phil Hobbs

--  
Dr Philip C D Hobbs
Principal Consultant
We've slightly trimmed the long signature. Click to see the full one.
Re: Redundant clock switching
On 04/10/2018 08:39 AM, Phil Hobbs wrote:
Quoted text here. Click to load it

Quoted text here. Click to load it

Quoted text here. Click to load it

Even the Apollo guidance computer didn't have a redundant clock source  
AFAIK...


Quoted text here. Click to load it

Another unusual but plausible situation I could think of is when there's  
not necessarily a permanent failure of the "master" clock but some kind  
of temporary out-of-bounds condition? Like an extreme temperature  
situation where the fast clock becomes unreliable somehow so you switch  
down to a slow clock that's designed to be better behaved at 130 C or  
-50 until hopefully the extreme condition passes and things return to normal

Quoted text here. Click to load it


Re: Redundant clock switching
bitrex wrote:

Quoted text here. Click to load it

But it was designed for just several days of continuous operation. In my  
case it is exactly the opposite: no high temperatures, no vibrations,
"infinite" endurance.

    Best regards, Piotr


Re: Redundant clock switching
On 04/10/2018 11:46 AM, Piotr Wyderski wrote:
Quoted text here. Click to load it

Quoted text here. Click to load it

"infinite" endurance is one of the hardest fault-tolerant design  
problems of all, I think. Guess you could say that the problem by nature  
is "underspecified."

I think trying to design something for "infinite" endurance that has at  
all graceful or stylish failure behavior, or even maintains stock  
performance after things have started going wrong, is going to be an  
exercise in frustration.

To make the problem tractable I think you have to accept that when shit  
inevitably breaks, performance is going to degrade in some proportion to  
how much has gone wrong, the challenge is to try to make it a somewhat  
linear degradation rather than a cliff.

As an analogy how does a badly damaged ship make it to port? Most ships  
don't have three backup boilers and watertight compartments designed  
such that they can take on 500,000 gallons of water and run at full  
speed, it's too expensive and just more stuff to go wrong!

But badly damaged ships do manage to struggle along for very long  
periods of time half full of water, badly listing to port at three knots  
with all but one boiler flooded or destroyed, toss all the dead weight  
overboard even the anchor, start ripping up the floorboards and toss the  
furniture in the burner, tear the roof off and knock the funnels out to  
get more airflow, whatever it takes, damn the torpedoes full steam  
ahead, whatever it takes!

That's how I think of "fault tolerance" I guess

Re: Redundant clock switching
bitrex wrote:

Quoted text here. Click to load it

And the most interesting. :-)

Quoted text here. Click to load it

It is. You discover how many parts and circuits you *cannot* use.

    Best regards, Piotr

Re: Redundant clock switching
On Tuesday, 10 April 2018 20:07:58 UTC+1, Piotr Wyderski  wrote:
Quoted text here. Click to load it
  
Quoted text here. Click to load it

None last forever. You can only figure out & pick the best you can come up  
with.

In my observations of hi-rel goods, most fail due to some weak spot that wa
sn't as reliable as the rest of the thing. The same pattern occurs in domes
tic goods, albeit at a much lower reliability & price point. Most of an ite
m might be built adequately but there's nearly always a weakspot somewhere.

And it can be surprising what turns out to be most reliable. Often it's old
 kit that was never built to great reliability standards, yet outlasts the  
modern stuff anyway.


NT

Re: Redundant clock switching
On 04/10/2018 03:16 PM, snipped-for-privacy@gmail.com wrote:
Quoted text here. Click to load it

The weak spot in men's leather belts is the holes. So do away with the  
holes:

<https://www.slidebelts.com

The weak spot in these belts is the tooth and lever mechanism that  
clamps the buckle on to one end of the belt. The teeth gripping into the  
leather gradually produces wear and eventually friction isn't enough to  
hold the buckle on and it falls off. Have to cut a bit of the end off to  
get to a fresh section of leather and re-attach, so the belt gradually  
becomes shorter with time. It helps belt longevity if you're on a diet  
and losing weight concurrently with this happening.

Re: Redundant clock switching
On 10/04/2018 16:46, Piotr Wyderski wrote:
Quoted text here. Click to load it

Thought they did have a secondary way to compute the critical burns in  
case the master system was unable to perform. ISTR whilst the LEM was in  
flight they had a fair few minor system glitches under load too.  
Thankfully none of them affected the trajectory or control systems  
sufficiently badly to abort the landing on Apollo 11.

http://www.klabs.org/history/apollo_11_alarms/eyles_2004/eyles_2004.htm

(this is a lot more detailed than the report I had intended to find)
Quoted text here. Click to load it

Even if the clock ends up being the only thing left alive? I have known  
systems that were supposed to be ultra reliable where due to a strange  
architectural quirk the only thing still working was the watchdog timer.

--  
Regards,
Martin Brown

Re: Redundant clock switching
Phil Hobbs wrote:

Quoted text here. Click to load it

Crystal aging and all the cumulative mechanical stress-induced failures.

Quoted text here. Click to load it

Many other problem sources can be TRMed and the entire modules
isolated from their buses with a bunch of low resistance analog
switches, but without a clock the entire system stops, hence
I think it'd be wise to have a failover hardware here.

Quoted text here. Click to load it

There will be no external clock.

    Best regards, Piotr

Re: Redundant clock switching
On Tuesday, April 10, 2018 at 11:43:51 AM UTC-4, Piotr Wyderski wrote:
Quoted text here. Click to load it

I think the usual way is to have your internal osc drive the clock....and when an external  clock is present it is used to discipline  the internal.

If the external clock goes away, the internal continues to run disciplined.


IDT probably makes what you need

https://www.idt.com/document/dst/82p33714-datasheet

mark

Re: Redundant clock switching
On 04/10/2018 05:25 AM, snipped-for-privacy@gmail.com wrote:
Quoted text here. Click to load it

idk how you have a "smooth transition" between two clocks one of which  
has failed entirely, you need some kind of watchdog or timeout circuit  
to detect the total clock loss in the first place and by the time you've  
detected it there's nothing to "transition" between now is there.

"E's not pinin'! 'E's passed on! This clock is no more! He has ceased to  
be! 'E's expired and gone to meet 'is maker! 'E's a stiff! Bereft of  
life, 'e rests in peace!"

Figure it must be some other kind of problem less than total failure if  
OP is talking about "smooth transitions"

Re: Redundant clock switching
On 04/10/2018 09:43 AM, bitrex wrote:
Quoted text here. Click to load it

Quoted text here. Click to load it

Quoted text here. Click to load it

At least not if both clocks are the same frequency, if the backup clock  
was significantly slower it could be triggering the watchdog to check on  
the fast clock during its own off time and if the fast clock has gone  
tits up during that period hopefully somehow figure out if there's  
enough time left in its own off period for the watchdog to set up a  
switchover without a glitch or if it needs to hold off a cycle.

Re: Redundant clock switching
On Tuesday, 10 April 2018 14:43:58 UTC+1, bitrex  wrote:
Quoted text here. Click to load it


Using an oscillator as I suggested would get a smooth transition. The 3rd osc is only locked by osc 2 when the phase of the 2 is close. Until that point, osc 3 runs on its own.
Osc1: feeds a bit into osc 3
Ocs2: feeds a lower level into osc 3
Now Osc 1 rules, until it dies when osc 2 rules, until that dies when osc 3 rules. And all switchovers are smooth


NT


Re: Redundant clock switching
On 04/10/2018 10:21 AM, snipped-for-privacy@gmail.com wrote:
Quoted text here. Click to load it

I'm having trouble thinking about this rig without a diagram but it  
sounds like that's still vulnerable to a single failure, what's the  
situation if it's oscillator 3 that fails first and not the other two?

Re: Redundant clock switching
On Tuesday, 10 April 2018 15:25:59 UTC+1, bitrex  wrote:
Quoted text here. Click to load it
 missing chip seems the obvious option. Your 2 other oscillators phase lock
 it, but if their input ceases to exist it oscillates on its own, preservin
g output.
Quoted text here. Click to load it
ve
to
f
rd osc is only locked by osc 2 when the phase of the 2 is close. Until that
 point, osc 3 runs on its own.
Quoted text here. Click to load it
sc 3 rules. And all switchovers are smooth
Quoted text here. Click to load it

It's proof against failure of osc 1&2. Of course not against osc 3, but tha
t one is inside the OP's equipment so can be made to whatever specs he choo
ses. I'm not aware of any option that is proof against failure of the final
 mix/choose/sync device, which is in this case just osc 3.


NT

Re: Redundant clock switching
On 04/10/2018 10:21 AM, snipped-for-privacy@gmail.com wrote:
Quoted text here. Click to load it

I think having three clocks is getting into  
"three-and-four-engine-aircraft-can-be-less-fault-tolerant-than-two"  
territory, my proposal would be to have two clocks and some kind of  
synchronizer where under "normal" operation they're each gating each  
other's half-cycles.

Set up the gating so if you lose either one everything continues on  
immediately as normal just at half-speed, without having to do any  
timeout or skipped-cycle detection stuff.

Re: Redundant clock switching
On Tuesday, 10 April 2018 16:04:04 UTC+1, bitrex  wrote:
Quoted text here. Click to load it

Osc 3 IS the synchroniser. It also has the benefit that if osc1 & osc 2 fail, osc 3 will take on the job instead.


NT

Site Timeline