Feedback to "Is Embedded Linux Worth the Effort"-
Murphy's Law, July 2003
Read the original artilce at Is
Linux Worth the Effort?
return to Murphy's Law
I enjoyed your article on embedded Linux and wondered if you
had any thoughts on using NetBSD as an alternative to Linux for
embedded systems. I say this because recently I compared the virtues
of Linux and NetBSD as potential systems for a SBC project I was
working on, and I wondered if you or your readers had any views
on the topic.
I found that the Linux kernel was pretty easy to port and install
on my SBC (an Intel XScale evaluation board) and getting a basic
system running with busybox only took me a few days. However,
trying to get anything in userland ported to XScale was a separate
battle for each application. For example, gcc would port easily,
but not ssh. I imagined such troubles would continue to bite with
each application I wanted to install, and the prospects of fighting
a new set of makefiles each day made me fairly glum.
NetBSD on the other hand was a great deal more trouble to get
running. The XScale support in the NetBSD kernel was stil under
development and there was much less documentation available in
the public domain than for Linux. It took me weeks of maddening
toil before I managed to get it working on my SBC. However, once
ported, NetBSD presented me with an entire working unix system
with all the usual userland applications included.
Unlike Linux, NetBSD is designed to be portable from the ground
up it. So once the machine dependant parts of the kernel are written,
the entire system is ported. As soon as I had configured the NetBSD
system on my SBC I could ssh onto it, send mail, download and
build other applciations, and even play wumpus (remember wumpus?).
This made NetBSD the clear winner for our project.
However, NetBSD was not all glorious. Since it has less market
acceptance than Linux, it tends to lag behind Linux in the number
of devices it supports. If you are building an embedded system,
chances are you are also using the latest hardware components,
so getting drivers is important. For instance, Linux had support
for a flash filesystem we wanted whereas NetBSD did not. Likewise,
there was no NetBSD driver for an ethernet chipset we wanted to
use, but Linux already had one. Linux also has better real-time
support than NetBSD, but that did not matter for our particular
project. Lastly, there is simply a lot more help available for
Linux in the public domain than there is for NetBSD.
Coming from a Linux background, my natural reaction would have
been to choose Linux without question and just get on with the
job. I really only considered NetBSD after a colleague impressed
me to do so, but after he did I am now impressed with NetBSD for
I wonder if others in the embedded unix community have had similar
experiences? If so, I can imagine the embedded Linux community
slowly drifting over to embedded NetBSD instead. After all, they
are blood brothers.
-- Stewart Heitmann
My name is Neil Horman, and I would like the opportunity to respond
to your recent article about the viability of Linux in the embedded
marketplace. I think you've got it quite wrong. You're commentary
suggests a fundamental lack of vision for the advantages that
Linux has to offer, as well as a less than fully developed understanding
of what those features are.
Niall> I think that I probably do see the vision that Linux
has to offer - but that vision takes work. In a desktop environment,
if I write some Linux code, I and the community get some payback
because the code can be shared. In an embedded environment external
hardware issues often mean that the code is useless to anyone
else, and so the price of the more elegant solution is not good
If I may, a point by point rebuttal:
> 1) "Linux is harder to port than other Operating Systems"
> Harder than what, and for whoom? The level of difficulty involved
in porting a software package as complicated as an operating system
can't is propaganda advanced by marketing groups. To say that
Linux is More difficult to port than Operating System X is completely
meaningless. I personally find it much easier to port linux to
a new system than various other operating systems. Clearly there
are those that feel otherwise. This is simply a function of the
individual preforming the port and their relative experince with
the product at hand. It should not be used as argument for or
against the product itself.
Niall> I never compared it to an "operating system" I compared
it to RTOS's typically used by embedded programmers (pSOS, RTXC
eCOS, etc.). This is by no means a fair comparison, but I use
it so the many embedded programmers who are used to RTOS's are
given some context.
> 2) "The Extra Power of Linux lies in its management of storage
> Yes, those are some advantages, but by no means all of the advantages.
Have you forgotten about the protected memory environment that
Linux has to offer, for instance. This alone allows the embedded
developer to construct their software in such a way that is far
reliable, debuggable, and portable than otherwise possible. >
> 3)"Some embedded applications require a general purpose machine..."
> Yes, but I think you underestimate the volume of embedded products
that use cpu complexes at least as powerful as those Linux was
origionally developed on. Of course, any number of counter-examples
can be made here, but as we move forward, embedded systems have
begun to look more and more like PC's. They have powerful processors
with all the requisite memory management hardware on board, mass
stoarge devices (and yes, Flash and its cousins can be made to
look like hard drives with freely available software), and a wide
array of commercial peripherals. Linux fits very well in this
environment. Niall> My reference to a general purpose machine
is a completely seperate issue to how much horsepower the processor
has. The question really is "Will a third party write an app that
can run on the box I sell", so size and power of processor is
> 4) "To permanently change the filesystem, you have to change
an image of the filesystem..."
> This is an example of the sort-sightedness of your article.
I suggest that you investigate the jffs2 file system, and the
mtd (memory technology device) component of the Linux kernel.
It allows you to partition FLASH memory on your device and use
it as a mountable read/write filesystem, eliminating the need
for you to re-flash your entire image when you want to change
one file. Of course this argument can be continued ad-nauseum
citing various different storage technologies, but I think you
will find that a little investigation will reveal most of you
storage woes have already been solved. And I understand that embedded
developer are used to re-flashing the system as one unit each
time an update is made, but did you ever consider that method
may be sub-optimal?
Niall> I am well aware of the other kinds of storage that
are available. I was merely pointing out what one particular vendor
does, and also making the point that while this solution is not
typical Linux, it is the sort of comproise which is often acceptable
in an embedded environment - not ideal, or recommended, just what
some people do to get the job done.
> 5) "But what happens when we try to communicate with custom
hardware?" > You write a device driver, same as you would _try_
to do on other operating systems. Regardless of the OS here, you
need code to control the device you want to work with. If you
are a writing a system on anything that is big enough to have
a significant operating system on board, then hopefully you will
try to write your device driver code in a modular way, if for
no other reason than to isolate hardware access to prevent unexpected
behavior in your software. The only difference between Linux and
other Os'es here is that Linux attempts to enforce a driver model
on the developer, while other system let you go "off the reservation"
so to speak, and access the hardware from anywhere in your system.
I have never, ever, seen a situation where accessing the hardware
from a non-centralized location has yielded a more stable system.
And if speed is your concern, Then you are probably developing
for a system which would really benefit from not having an operating
system at all.
Niall> I sorta' agree, but you seem to miss the point that
most of the embedded community regularly wirte code with no OS,
and that is the audience of the article. There are people who
need something more than no-OS and less than a full blown OS like
Linux, and those are the people who implement device drivers as
a set of centralised routines, but without any restrictions imposed
on them by the OS. My point was that those restrictions in Linux
impose a learning curve, where the pay-off is limited if there
are very small numbers of devlopers. More developers will give
more payoff, because Linux scales well - but not all developers
need scale, and can not afford to pay for it.
> 6) "The extra control and protection we get from a Linux device
driver buy us capabilities that are great in a multiuser system,
but don't have much upside on an embedded system. In a closed
system, designers can restrict the kind of activity that is allowed."
> You underestimate the advntages of the separation that a device
driver interface allows. With Linux, the use of a device driver
allows multiple processes to share access to a device in a well
controlled manner. Device driver don't universally restrict access
to one process, but rather police access so multiple processes
don't interfere with one another. This allows application software
to be segemented in a way that is both more modular, and easier
for a large group to develop. Also, don't forget about the abstraction
that a driver provides to an application. A properly written driver
can allow a CAN interface, for instance, to appear as a serial
line to an application. All of a sudden a developer can use kermit
or minicom to transfer data accross the CAN bus without the need
to write additional software.
Niall> Fair point, I may underestimate the advantages, but
you might be underestimating the investment in writing those drivers.
7)"Like files, devices are opened, closed, read, and written.
This makes sense for what Linux does best, which is manage storage
and network connections. It doesn't map as well to the control
of the A/D converters, general-purpose I/O ports, and custom hardware
found in many embedded systems. Directly accessing memory locations
that map to the registers of the chip being controlled provides
the kind of control an embedded developer wants. Doing it this
way also makes it easier to see a correlation between the values
written and the description in the data sheet." > You seem to
be very focused on the character device interface. Please recognized
this is one of many methods for communicating with device driver
code. Yes, file semantic access may not always be ideal for some
devices. But neither is low lever register access. In the case
of Custom hardware, yes, care needs to be taken for the proper
selection of interface semantics, but if focus is given to this
endeavor, rather than to the buldozer approach of low level register
access in the application, the net improvement in the system would
be large. I have personally writen a protocol interface to access
a switching engine and Network process in which I implemented
request, response and configuration packets. This allowed me to
use the BSD sockets interface to communicate with my device in
a way that made programming my application very easy (not to mention
more portable to future chips of this type) >
Niall > Fair point - my only device driver experience has
been with character files.
8)"Unfortunately the TRM/816 board does not provide drivers for
all of its peripherals. Drivers are provided for standard peripherals,
such as the serial port, but extras like the CAN controller, serial
EEPROM, and Compact Flash interface, come only with sample code,
not a complete driver." > You need to look harder. Some of these
devices are suported under diferent names in the kernel (the eeprom,
for example) is most likely a supported device under the I2C bus
driver. Compact Flash presents itself as an IDE device and so
should be supported with an additional configuration option passed
to the IDE subsystem. If you don't find what your looking for
in the kernel source, try asking the vendor for a code base. Finally,
failing that, remember all you have received for free here. Consider
writing an appropriate driver yourself. If its for commodity hardware,
and truly no driver exists, then consider giving it away. I'm
sure that there are others using this hardware out there that
would appreciate it greatly, and return to you in kind. > > I'm
sorry if this rebuttal has been overly harsh, but I feel very
strongly about this subject. Too often have I heard people discount
the usefullness of Linux in an embedded environment because the
traditional embedded developement models have been divergent from
those of a PC model, under which most people have a familiarity
with Linux. What they fail to realize again and again, is that
as technology advances, embedded system are adoping more and more
of the attributes of the PC. This is a fact to be taken advantage
of! by doing so, the benefits and features of an Operating System
designed under a PC can be leveraged to a phenominal degree. >
Thank you for your time.
Niall > Thank you for your feedback - while I critisized Linux
is spots, I do think it has a lot to offer the embedded community,
but only as long as they choose to use it as the solution to the
Dear Mr. Murphy,
I just finished reading your article "Is Linux worth the effort?"
in the current Embedded System Programming journal. I had a few
points to bring up you wrote that if "many developers are going
to use the resulting drivers" you should do one solution, but
if "the target user for the device driver is one developer", a
different solution should be followed. The flaw as I see it in
that logic that many items start with a limit scope of users,
but grow because of usefulness.
Another comment is about the following statement you made: "The
separation of application and device driver is an alien concept
to many embedded programmers, which makes it a significant obstacle
to using embedded Linux for custom hardware. " I've never thought
of myself as a Linux Purist. Working in the embedded industry
as well, I understand that most projects tend not to use drivers.
However, I would regard this not as a good thing. For each argument
why we should not use a device driver, I would say would be an
equally compelling reason why we should.
Niall> Peter makes a fair point about the elegance of drivers,
but unfortunatly there is a price in performance, and more importantly
in development effort to integrate a driver into an operating
system such as Linux. Obviously developers should restrict control
of hardware to a module or small set of functions, but turning
those routines into a device driver is not always a simple transformation.
Dave Hawkins describes
a project using a combination of CORBA, uC/OS and Linux,a nd makes
a few interesting points along the way...
I just read your article,
and also the comments on your web site.
I had a comment to
offer you on the aspect of controlling device registers directly.
If your device driver implements the mmap() function call, then
you can map a block of I/O memory directly into your own process
and you can 'tickle' the registers directly.
I think the application
of Linux in the embedded space is definitely application specific.
For my particular usage, I have Jean Labrosse's uCOS running on
DSPs for real-time control of hardware (1GHz digitizers and lots
of FPGAs). These boards perform signal processing on data received
from radio telescopes. A host computer running Linux controls
up to 20 boards plugged into two compact PCI crates (with the
crates linked by a PCI-to-PCI bridge). Control of such a large
number of devices is made simpler by the Linux device driver model.
Here's how things look;
Each device is split
into the following node:
- For access to PCI chipset registers
/dev/cobra_controlN - For access to board registers
- Terminal-like interface to the DSP
/dev/cobra_dataN - Read-only data interface
/dev/cobra_monitorN - Read-only monitor point interface N = 1,
2, 3, ...
The data processing
and monitoring are synchronous operations, data is ready on all
boards at 100ms, and monitor data is ready every 500ms. The select()
call under Linux allows the OS to monitor a large number of file
handles for their 'readiness', in this case, when the first device
is ready, a timeout is started for the others, each board's data
has to be received by the host within this timeout, otherwise
an error is asserted, and another process fixes the 'bad' board
up and brings it back online.
So, I believe some
of your assertions may be biased by experience with smaller embedded
systems that also have a real-time constraint as part of the overall
system. The application I have just mentioned also has a real-time
constraint at the DSP level, and a 100ms-timescale real-time constraint
at the host level, however, regular Linux (RH7.2 and 9.0) have
managed just fine.
The advantage of Linux
in this embedded system has been its flexibility as an overall
control system. Rolling my own drivers for uCOS running on the
host CPU to control 20 boards, deal with timeouts, implement a
network interface, etc etc, would not be a small project! Also
there is the fact that this data is then transported over the
network using CORBA (a memory/processor hungry implementation!).
The resource tradeoff here is that the embedded system is really
not that visible to the 'regular' programmers dealing with the
high-level system design. The host CPUs are 1GHz Intels with 1GB
RAM and they all PXE boot. The DSPs are from TI. The peripheral
boards are all custom.
I recently organized
one of Bill Gatliff's embedded Linux courses, and yes, porting
Linux to a new board can be difficult. However, that is a different
story that using Linux in an embedded situation.
Regards, and thanks
for the interesting articles. Keep 'em up!
Dave Hawkins, Caltech.