Speak Embedded

Both in my professional and personal life, I’m an advocate for using open source software. I have a coworker who is fond of saying “you get what you pay for” any time something goes wrong with OSS, but I could not disagree more.  The quality and quantity of OSS software has increased significantly over the years and there are some truly impressive OSS projects out there. When I actually sat down and considered all the open source software I use on at least a monthly basis it was eye opening.

Part 1: OSS At Work

You cannot talk about OSS without talking about the grand daddy of them all, the one that started it, GNU Linux.  To the ire of Richard Stallman, GNU Linux often gets shorted to just Linux, but Linux is a just a kernel, GNU is all the standard tools around the kernel that make an operating system. At DTI we use both CentOS and Ubuntu, the former being a copy of the defacto standard Red Hat Linux and the later a Debian based distribution that is being supported, even required, by more and more software development tools.

Most of DTI’s software, of my division’s at least, is written in C and is built for a Linux environment running on PowerPC CPU architecture. We have come to rely on two different toolchains for building our software, the ELDK and a OpenEmbbedded based toolchain. Both are based on the GNU tools with the GCC compiler at the center.

At DTI we have Windows PCs and Linux development servers.  Developers can write their software in either OS, and then compile the software on the Linux servers.  Personally, I do some of my development in Windows and some in Linux.  When I’m in Windows I use the excellent Notepad++ text editor and the TortoiseSVN Subversion client. When I’m in Linux if I’m feeling GUI (worst pun ever?) I use FreeNX or VNC to access the server and edit code in Geany.  When I’m in a console warrior mood SSH via PuTTY, screen (or Ubuntu’s improved version of screen byobu) and Vim (Emacs sucks!) are the tools of the day.

As for software development infrastructure my group moved from a commercial source control system to Subversion for version control a little over a year ago.  We recently passed the 1000 check-ins mark on our main repository which I feel is a respectable number for a team our size.  We use Trac for internal issue tracking and information sharing mainly because of its its excellent subversion integration.  We use the Trac clone, Redmine for our external systems because it provides a better a more complete feature set than Trac (and maybe we will migrate to Redmine internally one day).  All this infrastructure runs on the venerable Apache HTTP server.

When it comes to testing our products we mostly rely on commercial and propriety tools but there are a few OSS projects which have found a place in my test toolbox.  First, it is worth mentioning that many of DTI’s proprietary tools are developed in Python, Perl, or Tcl, all OSS languages with OSS interpreters.  I’m focused on developing DTI networking products, so I hate having to speed time installing OSes on nodes to be used for testing.  So I often used a custom Linux distribution based on TTYLinux.  TTYLinux is a excellent micro distribution that runs entirely out of a 4MB ramdisk and can boot from a flash drive.  It uses perhaps the most important OSS project for embedded systems, busybox which is a collection of micro versions of standard GNU tools.  I bundle some of my favorite test tools on this flash drive along with standards including iperf, nuttcp, ipmitool, stress, tcpdump, sysutils, ethtool, and or course everyone’s favorite, ping.  When I have a full blown Linux installation available or even Windows on my node, I use Wireshark for network traffic analysis.

For long term pattern analysis I rely on my mind… greatly aided by Cacti.  Cacti is a web application which graphs whatever sensors you can throw at it.  It is amazing what can be learned when data is visualized over time rather than just a viewing a single reading.  Cacti uses RRDTool as a backend to perform the data logging and graphing.  Most of the information we graph is available via SNMP, so we use net-snmp to gather than information.

I use OSS for everyday tasks not related to software development and product testing as well. Most readers are familiar with the Firefox web browser I’m sure, which I use a fair amount in both Windows and Linux.  I use Windows network shares via Samba to access my Linux servers’ file systems.  When I need to get or send something via FTP or SFTP Filezilla is my client of choice.  I use 7-Zip to deal with archived files and WinDirStat is an essential tool in dealing with my hording files habit.

That is more than 40 OOS projects and this is only Part 1! I’ve probably forgotten a few projects and I didn’t even try to touch all the OSS libraries which aren’t directly usable. Call it an open source initiative (the official term I believe), movement, revolution, or whatever you want, but their is no question that quantity of OSS projects is increasing and the quality of the software is growing.

Stay tuned for:

• Part 2: OSS At Home
• Part 3: Where Non-OSS is Still King

JP Landry
Network Division Manager

We recently had a very eye opening experience while helping a customer with a problem. They were having issues with our ATS1936 and TM1936-SFP+ in their engineering lab.  Their ATCA nodes and external nodes would link to the ATS1936, the ATS1936 reported the links as up, but they could not get traffic to pass between the nodes.  We checked spanning-tree settings.  We checked port physical settings.  We checked various other switch settings.  We checked SFP+ module numbers.  We checked fiber cable types.  We checked E-Keying settings.  We ran various tests and we could not find anything wrong.  It was getting frustrating.  We were close to issuing an RMA and then we remembered the Zeroth Law of Debugging.

Zeroth Law of Debugging: Never assume anything.

or

Zeroth Law of Debugging (rephrased): Verify all of your assumptions.

I call it the Zeroth Law of Debugging because it is just like the Zeroth Law of Thermodynamics.  It is the most fundamential rule of any list of ‘laws’ for debugging.   Debugging itself could be defined as making a list of assumptions and verifying them.

What is the most basic assumption of a layer 2 switch?  What is something you would rarely check? What is absolutely required for a layer 2 switch to connect two nodes?  Do you think you have figured it out?  Well if you answered unique MAC addresses you solved this problem.

Obviously, all nodes need unique MAC addresses and would have unique MAC addresses in the field, but this is testing in a engineering lab with prototype nodes.  The MAC addresses were the same on all the nodes.  The only way we figure it out was when we simplified the test down to just 2 nodes pinging each other.  The customer saw the particular behavior of ping working from node A to node B or node B to node A but not both at the same time.  That set off a red flag and we quickly figured it out from there once we verified our most basic assumption.

Every once in a while problems like this come up to remind you that not all issues are hard to fix.  As long as you keep the Zeroth Law of Debugging in mind, most (all?) product issues can be solved.

JP Landry
Network Division Manager

From Light Reading’s Valley Wonk blog:

Heavy Reading is predicting the ATCA market will reach $7.2 billion by 2012.
…
Stanley is counting application-ready ATCA systems — that is, the value of the type of system Alcatel-Lucent or Nortel Networks Ltd. would buy. Other studies, he says, have counted up the total end value of ATCA systems, measuring the products AlcaLu or Nortel would sell.

This is the same guy as my last post on this subject. $7.12 billion is a good bit less than $20 billion but is probably a more realistic number. $7.12 billion sounds like a strong market to me. Let’s hope Stanley’s analysis is accurate.

JP Landry
Network Division Manager

and fools seldom differ.

While Doug and I are colleagues, we do not interact much. We work at different locations in different departments on different things. I can probably count on one hand how many times I’ve actually talked to Doug and our interaction via email or other impersonal mediums is rare. That is why it intriguing that 1) Doug wrote about a book in general, which I planned to do today, and 2) he mentioned Gödel, Escher, and Bach which I started reading just yesterday. Maybe this coincidence even fits in with Doug’s argument somehow, but I’ll leave the philosophy up to Doug.

I can here to post about Dreaming in Code by Scott Rosenberg. While I’m an avid reader, I’ll somewhat ashamedly admit that I mostly read SciFi/Fantasy. When I was at the book store a week ago I decided to try something new and ended up with Dreaming in Code. I couldn’t be happier with my choice and maybe it will lead me into being nonfiction junky now. That is at least a little bit better than SciFi/Fantasy junky according to my wife. Like Doug I’m not here to write a book report, but rather to recommend Dreaming in Code. While at face value the subject matter of the book is a large software project that didn’t go anywhere, or at least hasn’t gone anywhere yet after many years of development, the underlying subject is about software development in general. Rosenberg does a wonderful job mixing the Chandler team’s struggles with discussions on why software development is hard and what people have tried to do about it.

An old software engineering buddy of mine used to love to point out the numerous articles that said “Software is hard”, or his favorite “Hardware is easy. Software is hard” because I was a hardware guy at the time. While I never doubted that software is hard, as I’ve become more and more involved in software development at DTI I have seen it first hand how difficult software development can be. I can empathize with some of the Chandler team’s difficulties and it is somehow comforting to hear that the entire industry has some of the same problems. Reading Dreaming in Code feels like having a beer with the guys after work. Sure discussing, theorizing, complaining, etc. about your projects usually doesn’t make them easier, but don’t you feel better afterwards?

JP Landry
Network Division Manager

Light reading has an ATCA analysis that is spot on. The ATCA market is not as big as initially (erroneously?) projected, but it is far from dead. Simon makes a good point that ATCA has something to prove in the next year and I believe that will happen. ATCA is a more attractive platform that ever with 10Gb Ethernet in the backplane. As requirements for 10Gb Ethernet increase, ATCA adoption rates will increase. Many companies will be looking for new platforms as their legacy platforms cannot handle 10Gb signaling rates.

JP Landry
Network Division Manager