Hi Niall,

I enjoyed your article on embedded Linux and wondered if you had any thoughts on using NetBSD as an alternative to Linux for embedded systems. I say this because recently I compared the virtues of Linux and NetBSD as potential systems for a SBC project I was working on, and I wondered if you or your readers had any views on the topic.

I found that the Linux kernel was pretty easy to port and install on my SBC (an Intel XScale evaluation board) and getting a basic system running with busybox only took me a few days. However, trying to get anything in userland ported to XScale was a separate battle for each application. For example, gcc would port easily, but not ssh. I imagined such troubles would continue to bite with each application I wanted to install, and the prospects of fighting a new set of makefiles each day made me fairly glum.

NetBSD on the other hand was a great deal more trouble to get running. The XScale support in the NetBSD kernel was stil under development and there was much less documentation available in the public domain than for Linux. It took me weeks of maddening toil before I managed to get it working on my SBC. However, once ported, NetBSD presented me with an entire working unix system with all the usual userland applications included.

Unlike Linux, NetBSD is designed to be portable from the ground up it. So once the machine dependant parts of the kernel are written, the entire system is ported. As soon as I had configured the NetBSD system on my SBC I could ssh onto it, send mail, download and build other applciations, and even play wumpus (remember wumpus?). This made NetBSD the clear winner for our project.

However, NetBSD was not all glorious. Since it has less market acceptance than Linux, it tends to lag behind Linux in the number of devices it supports. If you are building an embedded system, chances are you are also using the latest hardware components, so getting drivers is important. For instance, Linux had support for a flash filesystem we wanted whereas NetBSD did not. Likewise, there was no NetBSD driver for an ethernet chipset we wanted to use, but Linux already had one. Linux also has better real-time support than NetBSD, but that did not matter for our particular project. Lastly, there is simply a lot more help available for Linux in the public domain than there is for NetBSD.

Coming from a Linux background, my natural reaction would have been to choose Linux without question and just get on with the job. I really only considered NetBSD after a colleague impressed me to do so, but after he did I am now impressed with NetBSD for its portability.

I wonder if others in the embedded unix community have had similar experiences? If so, I can imagine the embedded Linux community slowly drifting over to embedded NetBSD instead. After all, they are blood brothers.

Hello

My name is Neil Horman, and I would like the opportunity to respond to your recent article about the viability of Linux in the embedded marketplace. I think you've got it quite wrong. You're commentary suggests a fundamental lack of vision for the advantages that Linux has to offer, as well as a less than fully developed understanding of what those features are.

Niall> I think that I probably do see the vision that Linux has to offer - but that vision takes work. In a desktop environment, if I write some Linux code, I and the community get some payback because the code can be shared. In an embedded environment external hardware issues often mean that the code is useless to anyone else, and so the price of the more elegant solution is not good value-for-effort.

If I may, a point by point rebuttal:

> 1) "Linux is harder to port than other Operating Systems"
> Harder than what, and for whoom? The level of difficulty involved in porting a software package as complicated as an operating system can't is propaganda advanced by marketing groups. To say that Linux is More difficult to port than Operating System X is completely meaningless. I personally find it much easier to port linux to a new system than various other operating systems. Clearly there are those that feel otherwise. This is simply a function of the individual preforming the port and their relative experince with the product at hand. It should not be used as argument for or against the product itself.

Niall> I never compared it to an "operating system" I compared it to RTOS's typically used by embedded programmers (pSOS, RTXC eCOS, etc.). This is by no means a fair comparison, but I use it so the many embedded programmers who are used to RTOS's are given some context.

> 2) "The Extra Power of Linux lies in its management of storage and networking":
> Yes, those are some advantages, but by no means all of the advantages. Have you forgotten about the protected memory environment that Linux has to offer, for instance. This alone allows the embedded developer to construct their software in such a way that is far reliable, debuggable, and portable than otherwise possible. > > 3)"Some embedded applications require a general purpose machine..."
> Yes, but I think you underestimate the volume of embedded products that use cpu complexes at least as powerful as those Linux was origionally developed on. Of course, any number of counter-examples can be made here, but as we move forward, embedded systems have begun to look more and more like PC's. They have powerful processors with all the requisite memory management hardware on board, mass stoarge devices (and yes, Flash and its cousins can be made to look like hard drives with freely available software), and a wide array of commercial peripherals. Linux fits very well in this environment. Niall> My reference to a general purpose machine is a completely seperate issue to how much horsepower the processor has. The question really is "Will a third party write an app that can run on the box I sell", so size and power of processor is irrelevant.
> 4) "To permanently change the filesystem, you have to change an image of the filesystem..."
> This is an example of the sort-sightedness of your article. I suggest that you investigate the jffs2 file system, and the mtd (memory technology device) component of the Linux kernel. It allows you to partition FLASH memory on your device and use it as a mountable read/write filesystem, eliminating the need for you to re-flash your entire image when you want to change one file. Of course this argument can be continued ad-nauseum citing various different storage technologies, but I think you will find that a little investigation will reveal most of you storage woes have already been solved. And I understand that embedded developer are used to re-flashing the system as one unit each time an update is made, but did you ever consider that method may be sub-optimal?

Niall> I am well aware of the other kinds of storage that are available. I was merely pointing out what one particular vendor does, and also making the point that while this solution is not typical Linux, it is the sort of comproise which is often acceptable in an embedded environment - not ideal, or recommended, just what some people do to get the job done.

> 5) "But what happens when we try to communicate with custom hardware?" > You write a device driver, same as you would _try_ to do on other operating systems. Regardless of the OS here, you need code to control the device you want to work with. If you are a writing a system on anything that is big enough to have a significant operating system on board, then hopefully you will try to write your device driver code in a modular way, if for no other reason than to isolate hardware access to prevent unexpected behavior in your software. The only difference between Linux and other Os'es here is that Linux attempts to enforce a driver model on the developer, while other system let you go "off the reservation" so to speak, and access the hardware from anywhere in your system. I have never, ever, seen a situation where accessing the hardware from a non-centralized location has yielded a more stable system. And if speed is your concern, Then you are probably developing for a system which would really benefit from not having an operating system at all.

Niall> I sorta' agree, but you seem to miss the point that most of the embedded community regularly wirte code with no OS, and that is the audience of the article. There are people who need something more than no-OS and less than a full blown OS like Linux, and those are the people who implement device drivers as a set of centralised routines, but without any restrictions imposed on them by the OS. My point was that those restrictions in Linux impose a learning curve, where the pay-off is limited if there are very small numbers of devlopers. More developers will give more payoff, because Linux scales well - but not all developers need scale, and can not afford to pay for it.

> 6) "The extra control and protection we get from a Linux device driver buy us capabilities that are great in a multiuser system, but don't have much upside on an embedded system. In a closed system, designers can restrict the kind of activity that is allowed." > You underestimate the advntages of the separation that a device driver interface allows. With Linux, the use of a device driver allows multiple processes to share access to a device in a well controlled manner. Device driver don't universally restrict access to one process, but rather police access so multiple processes don't interfere with one another. This allows application software to be segemented in a way that is both more modular, and easier for a large group to develop. Also, don't forget about the abstraction that a driver provides to an application. A properly written driver can allow a CAN interface, for instance, to appear as a serial line to an application. All of a sudden a developer can use kermit or minicom to transfer data accross the CAN bus without the need to write additional software.

Niall> Fair point, I may underestimate the advantages, but you might be underestimating the investment in writing those drivers.

7)"Like files, devices are opened, closed, read, and written. This makes sense for what Linux does best, which is manage storage and network connections. It doesn't map as well to the control of the A/D converters, general-purpose I/O ports, and custom hardware found in many embedded systems. Directly accessing memory locations that map to the registers of the chip being controlled provides the kind of control an embedded developer wants. Doing it this way also makes it easier to see a correlation between the values written and the description in the data sheet." > You seem to be very focused on the character device interface. Please recognized this is one of many methods for communicating with device driver code. Yes, file semantic access may not always be ideal for some devices. But neither is low lever register access. In the case of Custom hardware, yes, care needs to be taken for the proper selection of interface semantics, but if focus is given to this endeavor, rather than to the buldozer approach of low level register access in the application, the net improvement in the system would be large. I have personally writen a protocol interface to access a switching engine and Network process in which I implemented request, response and configuration packets. This allowed me to use the BSD sockets interface to communicate with my device in a way that made programming my application very easy (not to mention more portable to future chips of this type) >

Niall > Fair point - my only device driver experience has been with character files.

8)"Unfortunately the TRM/816 board does not provide drivers for all of its peripherals. Drivers are provided for standard peripherals, such as the serial port, but extras like the CAN controller, serial EEPROM, and Compact Flash interface, come only with sample code, not a complete driver." > You need to look harder. Some of these devices are suported under diferent names in the kernel (the eeprom, for example) is most likely a supported device under the I2C bus driver. Compact Flash presents itself as an IDE device and so should be supported with an additional configuration option passed to the IDE subsystem. If you don't find what your looking for in the kernel source, try asking the vendor for a code base. Finally, failing that, remember all you have received for free here. Consider writing an appropriate driver yourself. If its for commodity hardware, and truly no driver exists, then consider giving it away. I'm sure that there are others using this hardware out there that would appreciate it greatly, and return to you in kind. > > I'm sorry if this rebuttal has been overly harsh, but I feel very strongly about this subject. Too often have I heard people discount the usefullness of Linux in an embedded environment because the traditional embedded developement models have been divergent from those of a PC model, under which most people have a familiarity with Linux. What they fail to realize again and again, is that as technology advances, embedded system are adoping more and more of the attributes of the PC. This is a fact to be taken advantage of! by doing so, the benefits and features of an Operating System designed under a PC can be leveraged to a phenominal degree. > >
Thank you for your time.
Neil Horman

Dear Mr. Murphy,

I just finished reading your article "Is Linux worth the effort?" in the current Embedded System Programming journal. I had a few points to bring up you wrote that if "many developers are going to use the resulting drivers" you should do one solution, but if "the target user for the device driver is one developer", a different solution should be followed. The flaw as I see it in that logic that many items start with a limit scope of users, but grow because of usefulness.

Another comment is about the following statement you made: "The separation of application and device driver is an alien concept to many embedded programmers, which makes it a significant obstacle to using embedded Linux for custom hardware. " I've never thought of myself as a Linux Purist. Working in the embedded industry as well, I understand that most projects tend not to use drivers. However, I would regard this not as a good thing. For each argument why we should not use a device driver, I would say would be an equally compelling reason why we should.

Peter Halliday