Speak Embedded

CDR: Lift-off. The clock’s running.
LMP: Three seconds.
CDR: I got a yaw program -
LMP: Six seconds.
LMP: There’s 10 seconds.
CMP: Clear the tower.
CDR: Roger. Clear the tower. I got a pitch and a roll program, and this baby is really going.
CMP: Man, is it ever!
CC: Roger, Pete.
CMP: Twenty seconds.
CDR: That’s a lovely lift-off. That’s not bad at all.
CMP: Everything’s looking great. Sky’s getting lighter.
CDR: Okay.
LMP: Thirty seconds.
CDR: Looks good.
CDR: Roll’s complete.
I24P: This thing moves, doesn’t it?
CC: Roger, Pete.
CMP: What the hell was that?
CDR: Huh?
CMP: I lost a whole bunch of stuff; I don’t know - -

What the hell was that? That was lightning.

40 years ago last Sunday, November 14th, perhaps the second greatest engineering debug effort took place in NASA history. Second behind Apollo 13 was Apollo 12 and we hardly know about it. (Well maybe we could debate its place among Skylab and Solar Max Missions.) No great Ron Howard movie for this effort. Only a few months removed from the first lunar landing, Apollo 12 began its trip to the moon. As the rocket rose on a plume ionized exhaust, it created a big expensive lightning bolt. It only took 36 seconds into the mission before disaster came calling and then again 16 seconds later another bolt struck the rocket. The combination of the two strikes knocked out the astronaut’s attitude indicator, showing the orientation of the rocket. Telemetry systems that report the rockets status to the ground also began returning meaningless data and other systems began to quickly drop off-line as voltage issues ran through the rocket. Without telemetry data, ground control had no way of determining rocket status, and the mission director was slowly making a case in his head to abort the mission. In an abort, escape rockets would separate the small command module from the top of the whole rocket, and once it reached a safe distance the rest of the rocket would be exploded on purpose, thus bringing a quick end to man’s second attempt at the moon. Mission Director, Gerry Griffin was seconds away from issuing an abort command, without telemetry he had little choice but to save the lives of the three astronauts inside.

Then a surprising calm voice called out over the radio.

‘Try SCE to aux”

That voice was John Aaron. Aaron had seen the problem before and remembered a quick fix. He had seen a similar problem in the lab with the Signal Conditioning Equipment (SCE) system and by switching its power to auxiliary the problem was corrected.

So surprised by the un-scripted suggestion, the flight controllers asked him to repeat the request.

And a second time John said “Try SCE to aux”. Alan Bean inside the command module knew the switch and moved it the AUX position and telemetry was restored. NASA was able to verify that the rocket was on track and gave a “go” for the 1st staging while the astronauts began resetting the rocket’s power generation system. A few minutes later most systems were back to functional status and the astronauts joked about that being an interesting “simulation” they were given. In the highly scripted NASA world, such quick action and thinking is rarely needed. Thankfully 40 years ago this weekend, when it was needed - it was there.

As the recession continues, it amazes me how many companies begin to make advertising cuts to save money.  While I understand that less money in means less money to go out, I also realize that marketing your company and product helps to bring in that money in the first place.  Without getting your brand seen, you may very well get overlooked and be forced to make even more cuts in the future.

That being said, I do however think the recession is a good thing for marketing.  It makes us reanalyze things and focus our dollars on the items that are truly vital to lead generation and not just spend money hand over fist because its available in the budget….and we have to do something with it.

An example would be billboard advertising in left field of a baseball field.  Think about it.  Name five billboards on the outfield fence of a minor league game that you have attended in your life.  (That is assuming you like baseball)  Did any of those billboards lead you to purchase their product or service?  I doubt it.  Most of us don’t even recall what they were or know what product that company offered.  They are usually just simple designs with a logo, slogan, and simple image.  Thats not really something you pay attention to when you go to a game, but these placements can costs thousands over the course of a season.  To me its all about using the money you have where it can be most effective and not where everyone else is using it.  In regards to this aspect of marketing, I think the recession helps us to all bring in the reigns and gain a little more focus until we can all let loose again!

Joel Deer
Marketing Communications

I just finished Linked: How everything is connected to everything else. http://en.wikipedia.org/w/index.php?title=Special%3ABookSources&isbn=0452284392

The book is about scale free networks. It was extremely interesting especially since I had previously re-read Jeff Hawkin’s “On Intelligence.” I haltingly remember one little factoid for Linked ; that website with opposing views are rarely linked to together and that website with similar views tend to link together. This is seemingly a new introduction into our world’s forms of media and communication, the condensation of ideas into rarely interactive opposing networks. Since my job as CTO is to predict the future, I often wonder what this portends for us. The Tower of Babel story comes to mind… broken into opposing cultures and languages, the world’s inhabitants were lead into confusion and later this confusion lead to warring nations. Far from bringing the world together, the internet may indeed force us apart, into increasingly polarized volatile communities – that is a recipe for conflict. This conflict will happen on the virtual boundaries of the internet and not any current geo-political boundaries. If one looks for it you can see today at every level on the internet from large political discussions to minor sports team fans pages; an increase in polarization. I am beginning to fear the exploitation of that polarization even more the nuclear war with the USSR that I feared as child. Fortunately as a child, I had several people which shared my fear which help me understand there was a barrier to it happening – I don’t see the same barriers to prevent polarization and it happily goes on. The second law of thermodynamics is intact and the future is not a pretty as we think…

We recently had a very eye opening experience while helping a customer with a problem. They were having issues with our ATS1936 and TM1936-SFP+ in their engineering lab.  Their ATCA nodes and external nodes would link to the ATS1936, the ATS1936 reported the links as up, but they could not get traffic to pass between the nodes.  We checked spanning-tree settings.  We checked port physical settings.  We checked various other switch settings.  We checked SFP+ module numbers.  We checked fiber cable types.  We checked E-Keying settings.  We ran various tests and we could not find anything wrong.  It was getting frustrating.  We were close to issuing an RMA and then we remembered the Zeroth Law of Debugging.

Zeroth Law of Debugging: Never assume anything.

or

Zeroth Law of Debugging (rephrased): Verify all of your assumptions.

I call it the Zeroth Law of Debugging because it is just like the Zeroth Law of Thermodynamics.  It is the most fundamential rule of any list of ‘laws’ for debugging.   Debugging itself could be defined as making a list of assumptions and verifying them.

What is the most basic assumption of a layer 2 switch?  What is something you would rarely check? What is absolutely required for a layer 2 switch to connect two nodes?  Do you think you have figured it out?  Well if you answered unique MAC addresses you solved this problem.

Obviously, all nodes need unique MAC addresses and would have unique MAC addresses in the field, but this is testing in a engineering lab with prototype nodes.  The MAC addresses were the same on all the nodes.  The only way we figure it out was when we simplified the test down to just 2 nodes pinging each other.  The customer saw the particular behavior of ping working from node A to node B or node B to node A but not both at the same time.  That set off a red flag and we quickly figured it out from there once we verified our most basic assumption.

Every once in a while problems like this come up to remind you that not all issues are hard to fix.  As long as you keep the Zeroth Law of Debugging in mind, most (all?) product issues can be solved.

JP Landry
Network Division Manager

A Webinar lets you reach out to large groups of remote customers, prospects and employees at a fraction of the cost of meeting in person.  When you are a company based in Mississippi, this form of communicating can be key to the business’ success.  DTI uses our website, emailers and other forms of electronic communications to promote upcoming webinars to both potential and current customers.

Its a great way to educate industry engineers on our products and technical knowledge, plus it allows DTI to keep current customers up to date on what roadmap products are coming.  DTI is a strong believer in the webinar experience and we hope to have you attend one of our next sessions.  To request a webinar, please visit the following link and complete the short form.

http://www.dtims.com/support/webinar.php

Joel Deer
Marketing Communications

With the country going through a major recession, everyone is getting tight on their spending. Telecom companies are no exception.  All of the technology extras and whiz-bang features yet to be developed are being pushed out more because the once deep R&D pockets are barely deep enough now to hold water.  One technology that seems to keep being pushed back is AdvancedTCA.  The numbers projected were, and still today, are staggering for this COTS solution that was developed by only a small consortium of companies.  The goal was to open up the hardware footprint and allow for end-customers to pick the best solution for their needs while getting the costs down in the process.  You now have tons of options to pick from which will eliminate vendor lock-in issues.  The problem is, hardware isn’t cheap.  Manufacturing can be sent overseas to help lower that development cost but aren’t you then lowering the quality of the product which isn’t what the Telco NEBS environment needs for reliability.

I believe that this large “cost control” myth is one reason that AdvancedTCA hasn’t boomed as fast as some people had predicted.  (Note: those predicting this boom are the very ones getting paid to promote that future take off.)  Many of the smaller end-customers that begin looking in to AdvancedTCA hardware for their solutions soon realize that for the simple applications, this open blade system may be a costly overkill.  Does that mean there isn’t a good, solid market for AdvancedTCA?  No, that’s not what I’m saying.  My point is that many larger companies that jumped on the early AdvancedTCA standard to provide their board solutions have now gotten out because they realize this to be the case.  They simply can’t make the numbers add up to justify their participation with everything else they are having to battle.  That has helped to push back the timeline for the mass riches of AdvancedTCA and also lowered the numbers some too.  But this exodus is good for companies like Diversified Technology Inc., who with our 37 year embedded history are in tight for the long haul with PICMG (www.picmg.org) standards like AdvancedTCA.  So, much like the smaller banks who are now taking advantage of these tough economic times, DTI is doing the same.

Joel Deer
Marketing Communications

Although I’m not officially in the information technology field, all processes have information components.  In the world of manufacturing, the delivery of these information components is just as valuable as the delivery of the physical components to the line.  In fact, a good information delivery process is vital to success in today’s competitive environment, as detailed in this recent Harvard Business Review article.  I’ve seen many of these benefits first-hand, particularly with regards to propogating a process change immediately and efficiently, with full confidence that the change is fully implemented by front-line employees.

Paul Boykin
Process Engineering Manager

The “Production Database” may need a new name…Manufacturing Operations System (MOS), as described by Jason Spera. These capabilities are indeed very beneficial to the continuous improvement of the production process, particularly as Jason describes in the latter part of the article with regards to planning, controlling shop-floor execution, and data mining for analysis, reporting, monitoring, and traceability.

Jason mentions a “digital work package” that provides “every conceivable guidance, verification, and information needed to conduct a quality build efficiently.” That is exactly the intent of the “Production Database” as defined in goal #1 (from my earlier post). Our manufacturing and process engineers continually add to this store of information throughout the life of the product. However, there is one source of information that often isn’t utilized effectively: the front-line employees actually using the information. Their input tends to be filtered through the perspective (and the time constraints) of the attending engineers.

We’ve all experienced conversations with front-line employees where we “discovered” a good assembly technique for a product we’ve been building for quite some time, yet no one else knows about this technique. This good information is “trapped” in the employee’s experience and not shared due to lack of an easy, embedded system of communication. Maybe this technique was communicated at one time, but never became part of the official documentation. We hope to break that barrier through a “Tips” button that allows our operators to provide input directly–”wikipedia” style. These tips and comments will be clearly highlighted from the official documentation, of course. I’ll let you know how this experiment goes…

Paul Boykin
Process Engineering Manager