Blueprint for a 21st Century Operating System

Why Do We Need It?

The first generation computer systems in the 1940s and 1950s were constrained by the physical hardware: lights and switches and relays and mechanical printers and card or tape readers. The transition to the second generation began with magnetic tape drives and a primitive operating system which ran programs sequentially off the tape drives. Vestiges of this architecture remain today in DOS batch files and Unix shell scripts.

The second generation in full flower (in the 1960s and 1970s) is file oriented: everything is a file, everything looks the same to the software, whether it be a tape drive, disk file, keyboard or text on the display screen. Even mouse motion is converted into a sequence of data to be read like a file.

Macintosh opened up the third generation system software, which is event driven, in the 1980s. Unfortunately, Apple did not understand what they had, and after botching up the design, abandonned it and went back to a 2nd generation Unix system. Meanwhile Microsoft, also not understanding the underlying principles, continues to lurch forward into the 3rd generation. Their first attempt was, like modern Unix systems, a thin layer of event-driven paint covering an old file-oriented batch system. But Object-Oriented (OOPS) data structures are not inherently file-friendly, and by attempting to move their system towards OOPS, Microsoft has unknowingly made it more event-driven. Like the later Macintosh systems now gone, however, it is not well done. Microsoft has the resources but not the vision to do it right.

Unix proponents are proud of the "everything is a file" nature of their system -- much like the old Luddites were proud of their ability to swing a sledge hammer. That's nice, but it's inefficient and error-prone. The current plethora of viruses are largely made possible by the Unix file structures imposed on all Internet protocols, and by the pervasive use of the Unix programming language C in all systems software (not just Unix).

Data is no longer all text. Most modern application programs use (OOPS) structured data in proprietary formats. Unix programmers still think in terms of text files in and text files out, so the programs can be piped together, the output of one going in as the input to the next. This was a wonderful idea in 1970, but real software does not work that way any more. People don't type commands into a terminal line, they double-click icons or dragon-drop.

Ladies and gentlemen, it's time to bring our systems into the 21st century.
 

The Plan -- Personal

For the next few months I will be developing a prototype 21st century operating system -- let's call it OS21 -- which exhibits the key features that Macintosh had and Unix lacks. The system is being designed to be useful for doing real work, because I intend to use it as my primary platform. In this document I lay out these key features and identify the other (somewhat orthogonal) issues of modern system design which also need to be addressed in one way or the other. Eventually I hope to release the system and its development tools to the open source community for public evaluation and further development.

I am a programmer, so the software that this system will initially support will be software development tools. Eventually I expect to add document preparation tools and some image editing tools, simply because I need them. When I have the time to learn Internet protocols, I will also add a web browser and server, thus showing how this can be done without the need for a firewall and without the vulnerability inherent in all Unix and Unix-wannabe-based systems currently out there. There is a right way to do network design, and we have not yet seen it in commercially available products. I do not believe the problem is in the protocols, but in the operating systems which support them. With their latest upgrades Microsoft seems to agree, albeit somewhat clumsily.

If I had control over the hardware this system runs on, as Apple did in 1984, I might consider building it on that native hardware. The fact is, hardware evolves too fast and is controlled by forces independent of system design. Instead I have done the hardware control one better and constructed an idealized virtual computer (VC) for this first version. The machine language is simple and portable to any existing hardware. After the system is working, we can consider a revision that targets arbitrary real hardware. There is no assembler and no assembly language code, so that's not such a hard problem. Some parts of the system software necessarily relate to the VC hardware details -- and those parts need to be rewritten for any new target hardware -- but the rest should require nothing more than a recompile.

One advantage of the VC approach is that the host hardware can either be supported directly, or else indirectly through the host machine operating system. Thus OS21 can either ride piggy-back on the file system of the underlying OS, or implement its own file system, or both. It can either take over the screen and keyboard of the host system, or virtualize those functions into a hosted window.
 

A Business Plan

OS21 has the potential for a substantial market niche. It will be smaller and far more robust than any version of Windows, and it will be smaller and easier to use than any version of Unix. It's unclear at this time whether it can be made into a viable commercial product: for that to happen there must be a business model for profitability. The business model for Unix is that if it's hard enough to use, people will hire consultants to run and maintain it. The consultants all love Unix. OS21 wants to be easy to use, not hard. The business model for WinXP is lock-in; you can't get there from here. There is only room in any market for one monopoly, and Microsoft has it.

If we abandon profitability, if a large enough user base (possibly including a large corporation or government agency) recognizes that Total Cost of Ownership (TCO) includes the hassle of users running a clumsy system and consultants fixing a broken one, then they might fund it non-commercially. Right now that user base is size 1. But let's pursue that option a little...

Microsoft cannot grow its OS business, it already has monopoly status. The market is theirs to lose. 100,000,000 (more or less) computers running Windows is a big enough pie that even a small piece of it has room for several years of profitability at "New Economy" rates, and a lot longer at "Old School Business" rates. Let's not fool ourselves: if OS21 were to take off, Microsoft has the resources to keep their monopoly -- even if it means building a competitive OS. Would that be a bad thing? Probably not. How do we get there from here?

A robust commercial version of OS21, with all the features a marketable OS needs, is probably a million bytes of code, which translates into about $1 million in programmer efforts, if paid at market rates, perhaps less if they convert and build on existing open source code -- but that would probably require the result to be open-source also. That may not be the profit killer some people think, but it is a consideration. More importantly, you would also need additional programmers to convert existing open-source application suites like OpenOffice and Gimp to run in OS21. The converted code could remain open source -- in fact, we expect this system and its tools to be so robust that the conversion process would likely find a number of latent bugs in the existing code, which can be filtered back to the open source community as a way of marketing OS21.

Similar additional efforts need to be invested in Windows and Unix emulation packages...
 

Legacy Code

Every new system designer needs to facilitate the transition from existing systems onto the new, and OS21 no less so. When Apple moved from the 68000-based hardware to their current PowerPC iron, they provided an excellent upgrade path in a 68000 emulator in the system ROM. When they moved from the MacOS to Unix, they similarly provided a (somewhat crippled) "Classic" MacOS module to allow the continued operation of legacy customer code. Microsoft likewise has a DOS-prompt mode of operation in WinXP for customers stuck in the Unix-like 19th century. What do we do?

The easiest kind of full-system emulation we might be able to offer is to run a virtual Unix/Linux inside OS21 as a client program. Better than that would be to add in the Posix (Unix standard) library calls to the system, and enable the program loader to distinguish Unix binaries from OS21 programs, so to intermingle them the way Apple does with its legacy code. I do not at this time know whether the WinXP system calls are well enough defined to play the same game or not. It's all a simple matter of programming. I see now that there are Linux products that do just that for Windows code, so it's obviously feasible.

One important requirement of a minority system like OS21 or Linux is to support all the hardware that users might want to add to their system. Linux gets that by requiring vendors or third parties to write Linux-specific drivers for their hardware. This is of course the best solution from a system perspective, but many vendors won't bother, especially at first. The result is that Linux systems tend to lag about two years behind the hardware. Instead I propose building a "driver sandbox" that pretends to be Linux for the sake of the drivers, then passes the data through to OS21. This is necessarily less efficient than native drivers, but it works. Once enough people have adopted OS21, they will pressure the hardware vendors to do it right, but the penalty on users for vendors who refuse is minimal. To the extent that Windows driver specifications are well-defined and available, we can do the same for those drivers. I see that Linux already has one or more such products.
 

Distinctives

In this section I identify the qualities which distinguish OS21 from its two competitors, Windows and Unix. I consider any version of generic Unix, including Linux and OSX, to be essentially the same: a file-oriented 2nd-generation operating system, usually with some kind of graphical user interface (GUI) laid on top. Most Unix systems support Xwindows; Linux lays on top of that yet another layer of GUI, namely KDE or Gnome. Apple has their own Carbon/Cocoa GUI. The differences between these implementations are philosophically insignificant.

Error Prevention.  Any errors the computer can detect, it can also prevent from happening in the first place. It is not the computer's place to criticize the user for being stupid; instead the computer should enable the user to be smart. This means the user should never be allowed to type in erroneous commands or file names so the computer can then beep or otherwise ridicule the error. If there are too many choices for a popup menu or toolbar, it's acceptable to narrow the field by typing, but only if the computer actively refuses any wrong keystrokes, and only if the acceptable choices can be readily seen (if desired). Furthermore, usability testing should be applied to prevent different commands from being so similar that they are easily mistaken for each other.

Undo. If the computer cannot prevent an error, it should at least make it reversible. Forcing the user to confirm a hazardous but frequent operation is known to be ineffective: the user simply habituates the confirmation as part of the gesture, which then makes it even more dangerous than the simpler (and faster) gesture without confirmation. Anything the user can do and later regret should be Undoable if at all possible. Confirmation is still a reasonable step for rare and dangerous operations, like erasing the hard drive, but these should also be hard to invoke in the first place.

Events. The fundamental model for communication between programs and the system, and between each other, is events. We have files, of course, but they are a storage medium, not a communication medium. The system sends events to programs to let them know what the user has typed or moused, and the programs send events to each other if they wish to collaborate. Multiple threads in one program can communicate between themselves through the event model. It's rich and structured, and the computer science academics (experts) recognize it as the most robust synchronization model.

Key Events. The keyboard is a source of events, not a file.

Structured Display. The screen is a place to draw pictures (including text). It is not a file. Programmers should think about what information they need to present to the user, and then format the window to display that information. OOPS is all about structure.

No Console I/O. Programs with text-only console input and output are easy to write and hard to use. We want this computer to be easy to use. That will make it slightly (not a lot) harder to write for, but it's a small price to pay. Therefore we do not support console I/O at all. In Unix sandbox mode we can open a window for console I/O, but that is intentionally extra work, not less. Programs should put up their own window(s) and accept keyboard events. It's not hard to provide a "framework" to do that for you, and all programming environments already do that. It's time to discard the 19th century console.

No Command Line. Modern users start programs by double-clicking icons. This is the right way, and even file-oriented Unix lets you do that by default now. OS21 has no command line at all, so there is no place to put command-line arguments. This is good. The appropriate way to parameterize a program operation is by menus and preferences dialog(s). The Unix way to parameterize is by text files (which anybody can hack on, thereby causing the program to crash easily) and command-line options (requiring a good memory and agile fingers and a sturdy backspace key). Programmers often imagine themselves to have these qualities (however if you watch them, they spend much of their time on the backspace key), but real people work more efficiently in a GUI.

Fast. Windows is slow; Unix is slower. OS21 is small and quick: fast to boot up, fast to shut down, and generally quicker to launch programs. Menus at the top of the screen are measurably faster than menus on each window. Although it doesn't seem that way to the user, mousing is measurably faster than keyboarding.

Robust. OS21 is written in a new Java-like programming language Turk/2 which catches far more programming errors in the compiler -- before the program is deployed. Perhaps we will never allow C/C++ on it, but certainly not now. It's too dangerous and time-consuming. Using Turk/2 is one of the reasons the existing open-source code needs adaptation: the language is very similar to C++ and Java, so most programmers can learn it quickly, but there are significant differences. Those differences are intentional, resulting in much more reliable code; this means fewer crashes, fewer instances of strange behavior, and fewer security flaws (which are latent opportunities for virus attacks).

Stable. Microsoft can make money on their systems only by substantially changing the way it works from time to time, so you are forced to buy the upgrade in order to get the features -- or even just to run the new programs. Linux proliferates a zillion new and different versions, because the programmers are essentially anarchists, with the result that most software doesn't quite work the same (or maybe not at all) on many popular versions of the OS. The MacOS had published guidelines, which if you wrote software conforming to them, it continued to run, through multiple system revisions. That of course came to an end when Apple went to Unix; now they are making radical changes with each new release. We want upgrades to be optional. Sure, there is more money to be made from selling upgrades to each customer, but you won't get the customers in the first place if you force that. OS21 should be stable enough so that people stop thinking about their computer and just get their work done.
 

Keeping the Good

The Macintosh should have been designed from the beginning with true multitasking. They could have fixed the problem as early as the MacII, but by then the smart people were already gone or doing other things. Unix does multitasking, but badly (it's slow). The same goes for WinXP. WinCE and RTOS were designed to be a real-time systems, so their process management algorithms are better. OS21 does multi-tasking right, because context switches are frequent.

Unix and WinXP do virtual memory (VM), where the computer pretends to have infinite memory and swaps out to disk what won't fit in physical memory. This is good and should be done in OS21; however, implementing VM in my VC is slow and cumbersome, so it probably will depend on the host VM for now. Note that a well-done VM means you can turn the computer off and on almost instantly (everything is saved in the swap file); nobody does that -- yet, probably because their systems are not robust enough to survive it.

Direct Manipulation, also known as WIMP (Windows, Icons, Mouse, Pull-down menus). People work much more efficiently when they see what they are doing and directly move the parameters to effect it, such as by DragonDrop or pushing parts of the picture around to where they look good.

One of the compelling features of Unix that (I assume) keeps programmers glued to that obsolete system is shell scripts, the ability to write sequences of command lines to automate long and complex sequences of program operations. Windows still allows batch files, but they do not really support the automation of their GUI environment. The MacOS was designed to be able to record and play back sequences of events, and prior to MacOS/7 this was a powerful tool. Unfortunately, AppleScript lacks some of that power, and not all programs -- notably not even Apple's own system software -- support what it does do. OS21 does scripting right, the way the earlier MacOS did it, so all program operations can be automated.

Scalability. It is my opinion that OS21 scales nicely to larger and faster hardware and (with not very much additional effort) to multi-processor systems. Linux is only recently beginning to scale up like that; we can build it in from the get-go. That way when our customers need more computer power, they just buy a bigger box -- and all their software continues to run. Another way to provide the same upward migration path (with a comfort zone for the trepid) is to offer OS21 emulators to run on Linux and Windows. That way any custom software people acquire to run on this system can be ported to the other computers at only a loss in performance probably less than the speed-up they gain from the bigger box.

Educational Discounts. Apple achieved tremendous market penetration by offering their Macs at very steep discounts to educators and students; 30 years before that IBM unwittingly pulled off the same coup with mainframes. This is a loss leader, but very much worth the cost. If OS21 is in some way open source, we can do high-quality textbooks for standard college curricula around the code to get further buy-in.
 

Design Details

This section provides links to the particular design documents relevant to OS21.

Turkish Demitasse (Turk/2), the programming language that makes an ambitious project like this possible, because it significantly reduces the number of bugs that make it past the compiler. The compiler for this language currently generates code for the IttyBittyStackMachine, described below. Although specified to include most of the OOPS features of Java, these are not yet implemented.

Itty Bitty Stack Machine, a virtual computer designed for easily implementing new operating systems.
 

Not yet available:

Mini Operating System (MOS), a particular implementation of OS21. See MOS Progress Report.

Events in MOS, and how to use them.

MOS File System.

The MOS Scripting Language.

MOS Test Suite, a scripted collection of programs to validate every feature of Turk/2 and the operating system. This can serve initially to validate the system and its tools, and subsequently for regression testing.

Porting MOS to Other Platforms. Currently hosted on the MacOS, an important target platform is Windows, and later Linux and OSX.

The First Challenge
First Draft: 2004 July 31
Rev. 2005 March 11