Speak Embedded

We recently had a very eye opening experience while helping a customer with a problem. They were having issues with our ATS1936 and TM1936-SFP+ in their engineering lab.  Their ATCA nodes and external nodes would link to the ATS1936, the ATS1936 reported the links as up, but they could not get traffic to pass between the nodes.  We checked spanning-tree settings.  We checked port physical settings.  We checked various other switch settings.  We checked SFP+ module numbers.  We checked fiber cable types.  We checked E-Keying settings.  We ran various tests and we could not find anything wrong.  It was getting frustrating.  We were close to issuing an RMA and then we remembered the Zeroth Law of Debugging.

Zeroth Law of Debugging: Never assume anything.

or

Zeroth Law of Debugging (rephrased): Verify all of your assumptions.

I call it the Zeroth Law of Debugging because it is just like the Zeroth Law of Thermodynamics.  It is the most fundamential rule of any list of ‘laws’ for debugging.   Debugging itself could be defined as making a list of assumptions and verifying them.

What is the most basic assumption of a layer 2 switch?  What is something you would rarely check? What is absolutely required for a layer 2 switch to connect two nodes?  Do you think you have figured it out?  Well if you answered unique MAC addresses you solved this problem.

Obviously, all nodes need unique MAC addresses and would have unique MAC addresses in the field, but this is testing in a engineering lab with prototype nodes.  The MAC addresses were the same on all the nodes.  The only way we figure it out was when we simplified the test down to just 2 nodes pinging each other.  The customer saw the particular behavior of ping working from node A to node B or node B to node A but not both at the same time.  That set off a red flag and we quickly figured it out from there once we verified our most basic assumption.

Every once in a while problems like this come up to remind you that not all issues are hard to fix.  As long as you keep the Zeroth Law of Debugging in mind, most (all?) product issues can be solved.

JP Landry
Network Division Manager

A Webinar lets you reach out to large groups of remote customers, prospects and employees at a fraction of the cost of meeting in person.  When you are a company based in Mississippi, this form of communicating can be key to the business’ success.  DTI uses our website, emailers and other forms of electronic communications to promote upcoming webinars to both potential and current customers.

Its a great way to educate industry engineers on our products and technical knowledge, plus it allows DTI to keep current customers up to date on what roadmap products are coming.  DTI is a strong believer in the webinar experience and we hope to have you attend one of our next sessions.  To request a webinar, please visit the following link and complete the short form.

http://www.dtims.com/support/webinar.php

Joel Deer
Marketing Communications

With the country going through a major recession, everyone is getting tight on their spending. Telecom companies are no exception.  All of the technology extras and whiz-bang features yet to be developed are being pushed out more because the once deep R&D pockets are barely deep enough now to hold water.  One technology that seems to keep being pushed back is AdvancedTCA.  The numbers projected were, and still today, are staggering for this COTS solution that was developed by only a small consortium of companies.  The goal was to open up the hardware footprint and allow for end-customers to pick the best solution for their needs while getting the costs down in the process.  You now have tons of options to pick from which will eliminate vendor lock-in issues.  The problem is, hardware isn’t cheap.  Manufacturing can be sent overseas to help lower that development cost but aren’t you then lowering the quality of the product which isn’t what the Telco NEBS environment needs for reliability.

I believe that this large “cost control” myth is one reason that AdvancedTCA hasn’t boomed as fast as some people had predicted.  (Note: those predicting this boom are the very ones getting paid to promote that future take off.)  Many of the smaller end-customers that begin looking in to AdvancedTCA hardware for their solutions soon realize that for the simple applications, this open blade system may be a costly overkill.  Does that mean there isn’t a good, solid market for AdvancedTCA?  No, that’s not what I’m saying.  My point is that many larger companies that jumped on the early AdvancedTCA standard to provide their board solutions have now gotten out because they realize this to be the case.  They simply can’t make the numbers add up to justify their participation with everything else they are having to battle.  That has helped to push back the timeline for the mass riches of AdvancedTCA and also lowered the numbers some too.  But this exodus is good for companies like Diversified Technology Inc., who with our 37 year embedded history are in tight for the long haul with PICMG (www.picmg.org) standards like AdvancedTCA.  So, much like the smaller banks who are now taking advantage of these tough economic times, DTI is doing the same.

Joel Deer
Marketing Communications

Although I’m not officially in the information technology field, all processes have information components.  In the world of manufacturing, the delivery of these information components is just as valuable as the delivery of the physical components to the line.  In fact, a good information delivery process is vital to success in today’s competitive environment, as detailed in this recent Harvard Business Review article.  I’ve seen many of these benefits first-hand, particularly with regards to propogating a process change immediately and efficiently, with full confidence that the change is fully implemented by front-line employees.

Paul Boykin
Process Engineering Manager

The “Production Database” may need a new name…Manufacturing Operations System (MOS), as described by Jason Spera. These capabilities are indeed very beneficial to the continuous improvement of the production process, particularly as Jason describes in the latter part of the article with regards to planning, controlling shop-floor execution, and data mining for analysis, reporting, monitoring, and traceability.

Jason mentions a “digital work package” that provides “every conceivable guidance, verification, and information needed to conduct a quality build efficiently.” That is exactly the intent of the “Production Database” as defined in goal #1 (from my earlier post). Our manufacturing and process engineers continually add to this store of information throughout the life of the product. However, there is one source of information that often isn’t utilized effectively: the front-line employees actually using the information. Their input tends to be filtered through the perspective (and the time constraints) of the attending engineers.

We’ve all experienced conversations with front-line employees where we “discovered” a good assembly technique for a product we’ve been building for quite some time, yet no one else knows about this technique. This good information is “trapped” in the employee’s experience and not shared due to lack of an easy, embedded system of communication. Maybe this technique was communicated at one time, but never became part of the official documentation. We hope to break that barrier through a “Tips” button that allows our operators to provide input directly–”wikipedia” style. These tips and comments will be clearly highlighted from the official documentation, of course. I’ll let you know how this experiment goes…

Paul Boykin
Process Engineering Manager

Found here…

Could a future organized DOS attack lead to a real war between two countries? it is an interesting thought experiment. What do you do about countries with weak internet rules that are harboring an almost privateering aspect to the web? I certainly don’t have answers but it is interesting to think about.. Something I first got on to 16 years ago is now so core to our existence that we have to think about saber rattling over someone else messing with it. Wonder what the next 16 years will bring?

Joe McDevitt
VP Technical Marketing

A key component in managing any production floor is the ability to direct the flow of materials and information. In many cases, this “just happens” because employees “know how” to build the product. A typical early step of Lean Manufacturing implementation is process mapping to move that “know how” from the heads of the employees to a documented form. My experience with documents, though, is that, except in the most bureacratic environments, they are referenced rarely and updated infrequently, rendering them useless for very long beyond their initial development. The production process is simply too dynamic for the static documents to keep up with the rate of change, because the flow of materials is separated from the flow of information. This situation is particularly acute in DTI’s environment of rapid prototyping and frequent introduction of new products.

The “Production Database” provides a solution by integrating the flow of information into the flow of materials. Each of DTI’s PCB assemblies is serialized (with a barcode) at the inception of the production process. Each serial number is tied to a specific “process routing” via its part number. Each operator workstation is equipped with a networked PC running the “Production Database” application and operators are required to scan each serial number as it progresses through their specific operation. The scan of each serial number not only collects production data but also provides critical build information to the operators. The order of operations in the “process routing” is enforced throughout the production process for each serial number. This effectively “locks in” the process map and closely links it to the physical flow of materials. If the physical flow needs to be changed, the “process routing” (effectively, the dynamic “process map”) must be changed to allow the physical change to happen on the production floor, eliminating the problem of infrequent updates allowing the documentation to get out of sync with the actual process.

In addition, production documentation is pushed to the operators for each serial number scanned. This information can take the form of critical build notes, attached documents (typically *.pdf files with abundant photos), quality alerts, pending Engineering Change Orders (ECOs), and “exceptions” for previous build steps. (I’ll discuss each of these items more in future posts.) The data presentation is very flexible and very targeted to present the specific information needed at the precise time it is required in the process, effectively enabling another critical Lean Manufacturing concept: standard work procedures.

Paul Boykin
Process Engineering Manager

The following is slanted toward holistic efforts, if you’ve have a surgical problem then instrumentation and knowledge alone should lead to an “a ha” moment and the fix. If you have not reached the fix with instrumentation for surgical problems add more instrumentation or knowledge. You can continue with hypothesis and test, because trying different things will be building your knowledge but it is generally a step that can be by-passed in surgical debugging.

For the holistic debug, this is really just the scientific process with a predefined goal of solving the problem. Using the information gathered in instrumentation make a hypothesis the root of the problem, apply a fix that corrects the hypothesis and test the results.

Read/Try

Frequently, you will know one or two components involved in the error. Print their datasheets (yes - print them) lay them out before you with all applications schematics and your schematics – look for differences. Read the datasheet again and again, finding areas of the datasheet that may pertain to your error. I love block diagrams of chip internals, study them, sometimes these pictures contain more information than written text.

Good Board/Bad Board

When a problem exists on one set of boards but not others – this is a gold mine. Add this as a data point BUT NOT THE FIX! Look at component date code, PCB revisions, build dates etc. Again be careful, See Phase IV Understand

The Change

Frequently in holistic debugging, you will hit on something that changes the problem. Often the change is small - a board that was failing 25% now fails 5% of the time. This is not the fix either but you are certainly on the right track. Look for the changes and understand them

Eliminate variables.

With every hypothesis and test, you get a result. Even if that result is “that change didn’t make a difference” this information should be tracked and documented to avoid needless retries. Eventually, you may arrive at “this is the only thing that could be wrong” by process of elimination. Can you eliminate a variable, circuit or component? Be careful, difficultly of a fix is not a reason for elimination… a missing pull-up on a un routed BGA is difficult to patch, but it is not a variable that can be eliminated – eventually you must go there. Many times I have helped fixed a problem by listening to what patch another engineer has said would be too difficult to try – this fact alone maybe the reason I identified the problem that started this blog.

Joe McDevitt
VP Technical Marketing