Speak Embedded

With the country going through a major recession, everyone is getting tight on their spending. Telecom companies are no exception.  All of the technology extras and whiz-bang features yet to be developed are being pushed out more because the once deep R&D pockets are barely deep enough now to hold water.  One technology that seems to keep being pushed back is AdvancedTCA.  The numbers projected were, and still today, are staggering for this COTS solution that was developed by only a small consortium of companies.  The goal was to open up the hardware footprint and allow for end-customers to pick the best solution for their needs while getting the costs down in the process.  You now have tons of options to pick from which will eliminate vendor lock-in issues.  The problem is, hardware isn’t cheap.  Manufacturing can be sent overseas to help lower that development cost but aren’t you then lowering the quality of the product which isn’t what the Telco NEBS environment needs for reliability.

I believe that this large “cost control” myth is one reason that AdvancedTCA hasn’t boomed as fast as some people had predicted.  (Note: those predicting this boom are the very ones getting paid to promote that future take off.)  Many of the smaller end-customers that begin looking in to AdvancedTCA hardware for their solutions soon realize that for the simple applications, this open blade system may be a costly overkill.  Does that mean there isn’t a good, solid market for AdvancedTCA?  No, that’s not what I’m saying.  My point is that many larger companies that jumped on the early AdvancedTCA standard to provide their board solutions have now gotten out because they realize this to be the case.  They simply can’t make the numbers add up to justify their participation with everything else they are having to battle.  That has helped to push back the timeline for the mass riches of AdvancedTCA and also lowered the numbers some too.  But this exodus is good for companies like Diversified Technology Inc., who with our 37 year embedded history are in tight for the long haul with PICMG (www.picmg.org) standards like AdvancedTCA.  So, much like the smaller banks who are now taking advantage of these tough economic times, DTI is doing the same.

Joel Deer
Marketing Communications

Although I’m not officially in the information technology field, all processes have information components.  In the world of manufacturing, the delivery of these information components is just as valuable as the delivery of the physical components to the line.  In fact, a good information delivery process is vital to success in today’s competitive environment, as detailed in this recent Harvard Business Review article.  I’ve seen many of these benefits first-hand, particularly with regards to propogating a process change immediately and efficiently, with full confidence that the change is fully implemented by front-line employees.

Paul Boykin
Process Engineering Manager

The “Production Database” may need a new name…Manufacturing Operations System (MOS), as described by Jason Spera. These capabilities are indeed very beneficial to the continuous improvement of the production process, particularly as Jason describes in the latter part of the article with regards to planning, controlling shop-floor execution, and data mining for analysis, reporting, monitoring, and traceability.

Jason mentions a “digital work package” that provides “every conceivable guidance, verification, and information needed to conduct a quality build efficiently.” That is exactly the intent of the “Production Database” as defined in goal #1 (from my earlier post). Our manufacturing and process engineers continually add to this store of information throughout the life of the product. However, there is one source of information that often isn’t utilized effectively: the front-line employees actually using the information. Their input tends to be filtered through the perspective (and the time constraints) of the attending engineers.

We’ve all experienced conversations with front-line employees where we “discovered” a good assembly technique for a product we’ve been building for quite some time, yet no one else knows about this technique. This good information is “trapped” in the employee’s experience and not shared due to lack of an easy, embedded system of communication. Maybe this technique was communicated at one time, but never became part of the official documentation. We hope to break that barrier through a “Tips” button that allows our operators to provide input directly–”wikipedia” style. These tips and comments will be clearly highlighted from the official documentation, of course. I’ll let you know how this experiment goes…

Paul Boykin
Process Engineering Manager

Found here…

Could a future organized DOS attack lead to a real war between two countries? it is an interesting thought experiment. What do you do about countries with weak internet rules that are harboring an almost privateering aspect to the web? I certainly don’t have answers but it is interesting to think about.. Something I first got on to 16 years ago is now so core to our existence that we have to think about saber rattling over someone else messing with it. Wonder what the next 16 years will bring?

Joe McDevitt
VP Technical Marketing

A key component in managing any production floor is the ability to direct the flow of materials and information. In many cases, this “just happens” because employees “know how” to build the product. A typical early step of Lean Manufacturing implementation is process mapping to move that “know how” from the heads of the employees to a documented form. My experience with documents, though, is that, except in the most bureacratic environments, they are referenced rarely and updated infrequently, rendering them useless for very long beyond their initial development. The production process is simply too dynamic for the static documents to keep up with the rate of change, because the flow of materials is separated from the flow of information. This situation is particularly acute in DTI’s environment of rapid prototyping and frequent introduction of new products.

The “Production Database” provides a solution by integrating the flow of information into the flow of materials. Each of DTI’s PCB assemblies is serialized (with a barcode) at the inception of the production process. Each serial number is tied to a specific “process routing” via its part number. Each operator workstation is equipped with a networked PC running the “Production Database” application and operators are required to scan each serial number as it progresses through their specific operation. The scan of each serial number not only collects production data but also provides critical build information to the operators. The order of operations in the “process routing” is enforced throughout the production process for each serial number. This effectively “locks in” the process map and closely links it to the physical flow of materials. If the physical flow needs to be changed, the “process routing” (effectively, the dynamic “process map”) must be changed to allow the physical change to happen on the production floor, eliminating the problem of infrequent updates allowing the documentation to get out of sync with the actual process.

In addition, production documentation is pushed to the operators for each serial number scanned. This information can take the form of critical build notes, attached documents (typically *.pdf files with abundant photos), quality alerts, pending Engineering Change Orders (ECOs), and “exceptions” for previous build steps. (I’ll discuss each of these items more in future posts.) The data presentation is very flexible and very targeted to present the specific information needed at the precise time it is required in the process, effectively enabling another critical Lean Manufacturing concept: standard work procedures.

Paul Boykin
Process Engineering Manager

The following is slanted toward holistic efforts, if you’ve have a surgical problem then instrumentation and knowledge alone should lead to an “a ha” moment and the fix. If you have not reached the fix with instrumentation for surgical problems add more instrumentation or knowledge. You can continue with hypothesis and test, because trying different things will be building your knowledge but it is generally a step that can be by-passed in surgical debugging.

For the holistic debug, this is really just the scientific process with a predefined goal of solving the problem. Using the information gathered in instrumentation make a hypothesis the root of the problem, apply a fix that corrects the hypothesis and test the results.

Read/Try

Frequently, you will know one or two components involved in the error. Print their datasheets (yes - print them) lay them out before you with all applications schematics and your schematics – look for differences. Read the datasheet again and again, finding areas of the datasheet that may pertain to your error. I love block diagrams of chip internals, study them, sometimes these pictures contain more information than written text.

Good Board/Bad Board

When a problem exists on one set of boards but not others – this is a gold mine. Add this as a data point BUT NOT THE FIX! Look at component date code, PCB revisions, build dates etc. Again be careful, See Phase IV Understand

The Change

Frequently in holistic debugging, you will hit on something that changes the problem. Often the change is small - a board that was failing 25% now fails 5% of the time. This is not the fix either but you are certainly on the right track. Look for the changes and understand them

Eliminate variables.

With every hypothesis and test, you get a result. Even if that result is “that change didn’t make a difference” this information should be tracked and documented to avoid needless retries. Eventually, you may arrive at “this is the only thing that could be wrong” by process of elimination. Can you eliminate a variable, circuit or component? Be careful, difficultly of a fix is not a reason for elimination… a missing pull-up on a un routed BGA is difficult to patch, but it is not a variable that can be eliminated – eventually you must go there. Many times I have helped fixed a problem by listening to what patch another engineer has said would be too difficult to try – this fact alone maybe the reason I identified the problem that started this blog.

Joe McDevitt
VP Technical Marketing

Results for the Regatta are posted…

Wind prediction was way off - white caps and almost a blow out on Sunday.

Had one good race where I was in the hunt… not sure what I did so I chalked it up to “blind squirrel finds acorn.” I think my letting out the bridle to reduce weather helm may have helped. One moment of inattention rounding the last mark and generally a bad last up wind leg (I was first around last downwind mark) cost me a 1st. I think lack of local knowledge killed me on Sunday.

Joe McDevitt
VP Technical Marketing

Upon entering the world of the blog, I’m reminded of a quote from leadership guru John Maxwell: “The person that thinks he’s leading, but has no one following him, is only taking a walk.” Only time will tell whether I’m leading or taking a walk in this endeavor. However, I’ll contribute observations and information primarily about DTI’s operations and manufacturing processes, as well as the electronics manufacturing industry as a whole.

On that note, I recently read an article from Circuits Assembly regarding DPMO Analysis Implementation. DTI has been tracking and analyzing defects per IPC-9261 and IPC-7912 since January of 2004 through the implementation of a production floor control system developed in-house and known here simply as the “Production Database” (very creative, I know). One of DTI’s strengths has always been the ability to rapidly transition a product from the “virtual” world of design software into a physical prototype, primarily due to our in-house production capabilities. However, increasing product complexity was straining our ability to manage this process in the early 2000s.

The “Production Database” grew out of this need to manage the manufacture of complex PCB assemblies in a rapidly changing New Product Introduction (NPI) mode. In early 2002, we embarked upon a mission to develop an in-house production floor control system that would meet four needs: (1) push information to operators to enable efficient, high quality production, (2) monitor and control critical process parameters, (3) collect and analyze data to enable continuous improvement, and (4) remain flexible to allow changing business process logic in response to customer needs. This system rolled out in its early form in late 2003 and became fully operational throughout our PCB manufacturing process by January 1, 2004. Development of this system is ongoing…truly a continuous improvement process.

The “Production Database” is a system that meets our needs in ways not possible for packaged, off-the-shelf software. It is a collection of Microsoft Access front-ends, customized for varying departmental needs, linked to a MySQL server back-end. There are also links into our MRP system, integrating materials data directly into the shop floor interface. The capabilities continue to be expanded upon as our (and our customer’s) needs change.

Many of my future posts will include lessons learned and improvements made as a result of this application development process. I’ll cover specific capabilities, technical development hurdles, the psychology of application development for a shop floor environment, and the benefits of having real data in solving problems, among other things learned along the path to continuous improvement.

Paul Boykin
Process Engineering Manager

Going to a regatta in Memphis this weekend. I sail on a lightning class sailboats. I am not very good. Unfortunately winds look light this week, I hate going out for a “bob” hopefully wind forecast will pick up.

Working on next installment of Debug Posts, sorry I have neglected this - coming soon.

Joe McDevitt
VP Technical Marketing