It's recently been a year since I started working on my pet OS project, and I often end up looking backwards at what I have done, wondering what made things difficult in the beginning. One of my conclusions is that while there's a lot of documentation on OS development from a technical point of view, more should be written about the project management aspect of it. Namely, how to go from a blurry "I want to code an OS" vision to either a precise vision of what you want to achieve, or the decision to stop following this path before you hit a wall. This article series aims at putting those interested in hobby OS development on the right track, while keeping this aspect of things in mind.

The article which you're currently reading aims at helping you answering a rather simple, but very important question: are you really ready to get into OS development? Few hobbies are as demanding, time-consuming and late-rewarding at the same time. Most hobby OS projects end up reaching a dead-end and slowly being abandoned by their creators because they didn't understand well enough what they were getting into and wrote insufficiently maintainable and ill-designed code or were overwhelmed and depressed by the amount of time it takes to get a mere malloc()working.

Some bad reasons for trying hobby OS development

• "I'll bring the next revolution of computing": Simply put: no. Long answer: think of yourself as a small research team. You can make something very neat, sure, but only at a small scale. Without massive backing, your research won't make it to mass distribution. And no, you can't emulate an existing OS' software in order to appeal to a wider audience.

• "I want to become proficient with a programming language, and need some challenging exercises": Some people try to learn a new programming language through OS development, thinking that they need something difficult to do in order to really understand the language. That's not a good idea either.

  • First, because it makes things even harder than they already are for the rest of the OS-deving world, and you shouldn't overestimate your ability to withstand frustration and strain.

  • Second, because being new to the language, you'll make more mistakes and those mistakes are much harder to debug when done at a low level than they are at the user application level where you have neat tools like GDB and Valgrind to help you.

  • Third, because you won't actually learn the language itself, but only a subset of it. The whole standard library will be unavailable and except for a few languages designed for the job like C, most programming languages have features which require a runtime support you won't be able to provide for some time. A few examples: C++ exceptions and vtables, C# and Java's garbage collectors and real-time pointer checks, Pascal Object's dynamic arrays and nice strings... More details here andthere.

  In short, you'll have a hard time coding things. When you do, you'll write poor code and you won't actually learn the language. Don't do it.

• "Implementing a new kernel is horribly complicated, but there's a much easier path! I'll just take the Linux sources and tweak them till I get an operating system which fits my needs, it shouldn't be so hard": May I be the first to introduce you to what Linux's sources actually look like by showing you its main.c file. Your first mission will be to locate where the "actual" main function is. Your second mission will be to understand where each mentioned function is located and what it's supposed to do. Your third mission will be to estimate the time it'll take you to understand all of that.

  As you should have understood by now, modifying an existing, old codebase without a good knowledge of what you're getting into is really, really hard. Code reuse is a very noble goal and it has many advantages, but you must understand now that it won't make your life simpler but rather make it much worse because you'll not only have to learn kernel development as a whole but also all the specifics of the codebase you're looking at.

• "I don't like detail X in Linux/Windows/Mac OS X, these should totally be re-done": All modern operating systems have their quirks, especially the desktop ones, no question about that. On the other hand, if your gripes against them remain relatively minor, you really should consider trying to contribute patches which fix them to the relevant projects and only rewriting the incriminated component(s) if this approach fails.

• "I'm looking for an exciting job": cf. "I'll bring the next revolution of computing". Except if you're in a research lab which works on operating systems, you'll hardly get any money from your OS as long as it has no practical uses (and it can remain the case for a very long time).

Some good reasons for trying hobby OS development

• "I want to experiment with new OS design": You have your own idea of what an operating system should be like and you can't find it in the existing OS market ? Hobby OS development can be a way to try designing and implementing what you've envisioned and see if it actually works in practice.

• "I'm now quite experienced with a language and looking for a programming challenge, or want to know how my computer works at the lowest levels": OS development certainly provides both, and to a great extent. You'll learn how to debug software with hardly anything more than text output facilities, how to do without a standard library or any sort, why so many people complain about this x86 architecture which works so well on your computers, what bad documentation from the vendor truly is...

• "I've been working on an operating system for some time, know its codebase well, and have many gripes with it": If addressing these gripes would imply major changes, the kind of which makes software incompatible, you'll probably no choice but to fork, which while not described in this article is a totally valid path for OS development (and one which is certainly more rewarding than writing a new OS from scratch).

• "The set of changes I want to bring to an OS is too big for simply patching or forking part of it": There sure is a point where starting a new project becomes the best option. If you think you have reached it, then welcome to the club!

• "I'm looking for an exciting hobby": Well, depends a lot on what your definition of "exciting" is, of course, but OS development can be one of the most exciting hobbies on Earth if you're sufficiently prepared. It's somewhat akin to raising a child or growing plants: sure, it's slow, and sometimes it gets on your nerves and feels highly unrewarding. But this just makes success feel even better. Plus it's incredibly fun to be in full control of what's happening and have absolute power on your hardware.

So, are you ready?

Aside from the questioning phase, here are a few things which you must know or learn quickly in order to take this road successfully:

• General computer science: You must know well and be able to fluently manipulate binary and hexadecimal numbers, boolean logic, data structures (arrays, linked links, hash tables, and friends), and sorting algorithms.

• Computer architecture: You should know the global internal organisation of a desktop computer. The words and expressions ALU, interrupts, memory and PCI buses, and DMA should not sound mysterious to you. Knowledge of more automated mechanisms like the TLB, multilevel caching, or branch prediction, would be a very valuable asset in the long run, once you try to optimize code for performance.

• Compiler and executable files internals: You must roughly know what happens during the preprocessing, compilation, and linking steps of binary generation, and be able to find everything you need to control your toolchain in its manual. You should also learn about linker scripting and the internal structure of executable files.

• Assembly: You must know what Assembly is and how it works, and you must be able to code simple Assembly programs (playing with the stack and the registers, adding and subtracting numbers, implementing if-like branching...). You can write most of your operating system in other programming languages, but at some point you will need assembly snippets to do some tasks.

• C or C++: Even if you plan to use another programming language to write your OS in, you'll need to know C or C++ in order to understand the code you'll find on the web when looking at other operating systems' source or other OS-deving tutorials.

• Your system programming language: No matter whether you plan to write your OS in C#, Java, Assembly, BASIC, C, C++, Pascal, Erlang or REBOL, you should be very familiar with it. In fact, you should even have access to a description of its internals. As an example, you should know how to call a function written in your language of choice from Assembly code, what kind of runtime support your language requires and how it can be implemented.

(List based on this page of the OSdev wiki and tweaked based on personal experience.)

This ends the "Are you ready ?" part of this OS-deving tutorial. If you have reached this point, congratulations ! You should have the right mindset and sufficient knowledge to begin the hobby OS development adventure. The next article will explain to you how to set goals and expectations for your OS project, a truly vital step if you actually want it to get somewhere in the end.

So you have taken the test and you think you are ready to get started with OS development? At this point, many OS-deving hobbyists are tempted to go looking for a simple step-by-step tutorial which would guide them into making a binary boot, do some text I/O, and other "simple" stuff. The implicit plan is more or less as follow: any time they'll think about something which in their opinion would be cool to implement, they'll implement it. Gradually, feature after feature, their OS would supposedly build up, slowly getting superior to anything out there. This is, in my opinion, not the best way to get somewhere (if getting somewhere is your goal). In this article, I'll try to explain why, and what you should be doing at this stage instead in my opinion.

The importance of design

First, heavy reliance on tutorials is bad for creativity. More often than not we end up blindly following the thing without fully understanding the verbose paragraphs describing what we're doing, no matter which big red warning the author has put against this practice. What do we get this way? Many low-level tutorials on the web describe how to create a UNIX clone mainly written in C with a monolithic kernel and a simple console shell. Reminds you of something? That's normal, we already have tons of these around. It's quite sad to spend all this time in OS development only to get yet another UNIX-like kernel with more limited hardware support than the existing ones, in my opinion, though you might think otherwise.

Another reason for not following this approach is that having our head stuck in code makes us short-sighted. There are many pitfalls which can be avoided through some time spent designing our thing before coding. Simple examples : asynchronous I/O and multiprocessing. There's nothing intrinsically hard about supporting those, given sufficient doc on the subject, if you've made the decision to use them early enough and stick with it. It's just a different way of thinking. However, if you started without those and try to add them to your codebase later, the switching process will be quite painful.

Finally, the last reason why I think that design is important even for hobby projects is that their teams are small. Generally one or two developers. This is not necessarily a bad thing : in early stages of an OS project, you generally don't want endless arguments and heaps of code snippets written by different persons in different ways. The obvious drawback, however, is that unless you focus on a limited set of things and try to do these well, you're not going very far. This is why I think at this stage you should be not be writing code already but rather defining what the project's goals are, where you want to go.

Exploring motivations

To begin with, forget that you're targeting a specific computer, and even that you're writing code. You, as a team of 1-2, maybe more human beings, are designing something which will provide some kind of service to other human beings (which may simply be the members of the team). It all begins here. What is this service which you want to provide? When will you be able to say that your project is successful? When will you be able to say that you're not on the right track? To encounter success in a hobby OS project, like in many other kinds of personal projects, the first step is to define it clearly.

So go ahead, take a piece of paper or some other mean of text storage (in my case it was a blog), and explore your motivations. If you're writing an OS because you have some gripe with the existing ones you know of, spend some time finding out what you don't like in them. Try to understand why they are so, explore their evolution in time, read books on the subject if you can find them. Even if it turns out to be a random choice, you've found something : the importance of not doing this part randomly, and to pay particular attention to its design. If you're writing an OS in order to learn something, try to define what you want to learn in more details. Maybe you want to know where the process abstraction comes from? Through which process does one goes from a complex and dumb electrical appliance which does calculations to something which can actually be used by human beings without a high level of qualification?

In all the cases, take as much notes as possible without disturbing your thinking process. There are two reasons for this : first you can review said notes later, second translating thoughts in text forces you to be more rigorous than you would be if you could stick with ideas so vague that they can't even be directly expressed in English (or whatever language you may fluently speak). Try to be as precise as if these notes were to be published and read by other people who are not in your head.

Defining a target audience

After having defined why you're doing this, try to quickly define what your target audience is. An operating system is an interface between hardware, end users, and third-party software developers, so you have to define all three to some extent : what hardware is your operating system supposed to run on? Who is supposed to use it? Who is supposed to write software for it, and what kind of software? You don't have to be overly precise at this stage, but there are some things which you'd like to decide now.

On the hardware side: you should at least know how the users will interact with the hardware (Keyboard? Pointing devices? Which sort of each?), what range of screen areas you wish to cover, how powerful the oldest CPU which you want to target is and how much RAM is guaranteed to be there. You might also want to know right away if your OS requires an internet connection to operate properly, what sorts of mass storage devices you're going to use, and how third-party software (if there is some) will be installed.

Quickly check that the kind of hardware you want to code for is homebrew-friendly. Most desktops and laptops are (with maybe the upcoming ChromeOS notebooks as an exception), video games consoles are a mixed bag (Nintendo's ones often are, others generally aren't), products from Apple and Sony aren't until they are deprecated. A word of warning about cellphones: though they may sound like an attractive platform to code for, their hardware is generally-speaking extremely poorly documented, and you should keep in mind that even when you find one which does allow homebrew OSs to run and has sufficient documentation about it available on the web, the next generation is not guaranteed in any way to offer such comfort at all.

The end user is perhaps the one thing which is hardest to define precisely, except when the developers work only for themselves. You should probably keep around the description of a number of stereotypes representing the kind of people which you want to target. Some examples: "70-year old grandma who has been offered a computer by her grandchildren for Christmas. Never used one. Pay special attention to her sight problems.", "40-year old bearded sysadmin on a company's network. Won't use a computer if it can't be operated in CLI mode with a bash-like syntax. Wants to automate many tasks which are part of his job, think about powerful scripting engines. Needs multi-user support, and a special account to rule them all which only him may access.", "20-year old creative girl who draws things using a pen tablet and various free software. A computer is a tool like every single other for her, it shouldn't get in the way more than a block of paper."

Third-party developers; first thing to note is that if you want them to get interested in your project at some point, you'll first have to attract them as end users anyway. People don't write code for a platform which they wouldn't like to use, except when they are paid for it. On the other hand, devs have a special status in that, to state the obvious, they create software. One of your tasks will hence be to define which kind of software you let them create.

It can range from them writing vital parts of your operating system for you (frequently happens in the world of FOSS desktop operating systems) to every software being written in-house and third-party devs being non-existent (has grown out of fashion these days, except in some categories of embedded devices). Between these extreme situations, there's a wide range of possibilities. Here are a few...

• The design/management of OS parts is proprietary, but the spec is disclosed, so that third-party developers are free to implement or re-implement parts of it the way they want

• Most of the operating system itself is made in a proprietary and undisclosed fashion, but you're open to third-party drivers and low-level tools (Windows 9x, Mac OS)

• Same as above, but you ask that applications needing access to some low-level or otherwise "dangerous" functionality go through some approval or signing process (Recent releases of Windows, Symbian)

• You don't tolerate third-party low-level apps and will do everything you can to prevent their very existence, but you offer a native development kit for user applications with generous access to the system APIs

• Same as above, but native third-party user apps are considered dangerous and must go through some signing/approval process (iOS)

• No native third-party app, everything goes through managed apps which only have restricted access to system capabilities (Android, Windows Phone 7, most feature phones)

With all this information in mind, you should start to have a fairly good idea of where you're heading, and be ready for the next step.

Goals and frontiers

Now that the idea is there, try to put it on paper. Describe what your OS is supposed to achieve, for who, on which hardware, who will code what, and when you'll be able to tell that the project is successful according to its initial goals, that it has reached the 1.0 release, so to speak. By defining the project's goals you define criteria for objectively measuring its success and failure, which will prove to be a very valuable resource later, be it only as a mean to avoid feature bloat and other ways of wasting your very precious development resources.

In the end, look at this global view of the project you have now and ask yourself this simple question : "Do I think I can make it?". If you think you should maybe lower your expectations a little bit, now is the right time, because the later you do it the higher the damage done. Play with all this stuff you have gathered together, polish everything until you have a project you're satisfied with (or maybe drop this idea of writing an OS altogether if you don't feel like it any more), and then you'll be ready for the next step, namely designing your kernel.

Now that you have an idea of where your OS project is heading as a whole, it's time to go into specifics. The first component of your OS which you'll have to design, if you're building it from the ground up, is its kernel, so this article aims at being a quick guide to kernel design, describing the major areas which you'll have to think about and guiding you to places where you can find more information on the subject.

What is a kernel and what does it do?

I think it's best to put a definition there. The kernel is the central part of an operating system, whose role is to distribute hardware resources across other software in a controlled way. There are several reasons why such centralized hardware resource management is interesting, including reliability (the less power an application has, the less damage it can cause if it runs amok), security (for the very same reasons, but this time the app goes berserk intentionally), and several low-level system services which require a system-wide hardware management policy (pre-emptive multitasking, power management, memory allocation...).

Beyond these generic considerations, the central goal of most modern kernels is in practice to manage the process and thread abstractions. A process is an isolated software entity which can hold access to a limited amount of hardware resources, generally in an exclusive fashion, in order to avoid concurrency disasters. A thread is a task which may be run concurrently from other tasks. Both concepts are independent from each other, although it is common for each process to have at least one dedicated thread in modern multitasking OSs.

Hardware resources

So far, I've not given much depth to the "hardware resource" concept. When you read this expression, the first thing which you're thinking about is probably some pieces of real hardware which are actually fully independent from each other: mice, keyboards, storage devices, etc...

However, as you know, these peripherals are not directly connected to the CPU. They all are accessed via the same bus, through one single CPU port. So if you want to make sure that each process only has access to some peripherals, the kernel must be the one in control of the bus. Or you may also decide that the bus is the hardware resource which processes must request access to. Depending on how fine-grained your hardware resource model is, your position in the process isolation vs kernel complexity scale will vary.

To make things even more complex, modern OSs also manage some very useful hardware resources which do not actually exist from a pure hardware point of view. Consider, as an example, memory allocation. From a hardware point of view, there is only one RAM. You may have several RAM modules in your computer, but your CPU still sees them as one single, contiguous chunk of RAM. Yet you regularly want to allocate some part of it to one process, and another part of it to another process.

For this to work, the kernel has to take its finest cleaver and virtually slice the contiguous RAM in smaller chunks which can safely be allocated separately to various processes, based on each one's needs. There also has to be some mechanism for preventing different processes from peeking into each other's memory, which can be implemented in various ways but most frequently implies use of special hardware bundled in the CPU, the Memory Management Unit (MMU). This hardware allows the kernel to only give each process access to his limited region of memory, and to quickly switch between memory access permissions of various processes while the kernel is switching from one process to another.

Another typical example of abstract hardware resource is CPU time. I assume you have all noticed that desktop operating systems did not wait for multicore chips to appear before letting us run several applications at once. They all made sure that processes would share CPU time in some way, having the CPU frequently switching from one process to another, so that as a whole it looks like simultaneous execution under normal usage conditions.

Of course, this doesn't work by calling the CPU and telling it "Hi, fella, can you please run process A with priority 15 and process B with priority 8?". CPUs are fairly stupid, they just fetch an instruction of a binary, execute it, and then fetch the next one, unless some interrupt distracts them from their task. So in modern interactive operating systems, it's the kernel which will have to make sure that an interrupt occurs regularly, and that each time this interrupt occurs, a switch to another process occurs. This whole process is called pre-emptive multitasking.

Finally, it is common not to let processes access storage devices directly, but rather to give them access to some places in the file system. Of course, like allocated RAM, the file system is a purely virtual construct, which has no physical basis in HDDs/SSDs, and must be fully managed by the OS at some point.

In short, you'll have to define which hardware resources your kernel manages and gives processes access to. It is generally not a matter of just giving processes access to hardware x or not, there is often a certain amount of management work to be done in the kernel, compromises to be considered, and sometimes hardware resources must just be created by the kernel out of nowhere, as an example in the case of memory allocation, pre-emptive multitasking, and filesystem operation.

Inter-process communication and thread synchronization

Generally-speaking, the more isolated from each other processes are the better. As said earlier, malware can't do much in a tightly sandboxed environment, and reliability is greatly improved too. On the other hand, there are several occasions where it is convenient for processes to exchange information with each other.

Typical use case for this is a client-server architecture: somewhere in the depths of the system, there's a "server" process sleeping, waiting for orders. "Client" processes can wake it up and give it some work to do in a controlled way. At some point, "server" process is done and returns the result to "client" process. This way of doing things is especially common in the UNIX world. Another use case for inter-process communications is apps which are themselves made of several interacting processes.

There are several ways through which processes may communicate with each other. Here are a few:

• Signals: The dumbest form of inter-process communication, akin to an interrupt. Process A "rings a bell" in process B. Said bell, called a signal, has a number associated to it, but nothing more. Process B may be waiting for this signal to arrive at the time, or have defined a function attached to it which is called in a new thread by the kernel when the process receives it.

• Pipes and other data streams: Processes also frequently want to exchange data of various types. Most modern OSs provide a facility for doing this, although several ones only allow processes to exchange data on a byte-per-byte basis, for legacy reasons.

• Remote/Distant procedure calls: Once we are able to both send data from one process to another and to send signals to other processes so that one of their methods gets called, it's highly tempting to combine both and allow one process to call methods from another process (in a controlled way, obviously). This approach allows one to use processes like shared libraries, with the added advantage that contrary to shared libraries, processes may hold access to resources which the caller doesn't have access to, giving the caller access to these resources in a controlled way.

• Shared memory: Although in most cases processes are better isolated from each other, it may sometimes be practical for two processes to share a chunk of RAM and just do whatever they want in it without having the kernel going in the way. This approach is commonly used under the hood by kernels to speed up data transfers and to avoid loading shared libraries several times when several processes want to use them, but some kernels also make this functionality publicly available to processes which may have other uses for it.

Another issue, related to inter-process communication, is synchronization, that is situations where threads must act in a coordinated manner.

To get started to this problem, notice that in a multitasking environment, there are several occasions where you want to make sure that only a limited amount of threads (generally one) may access a given resource at a time. Imagine, as an example, the following scenario: two word processors are opened simultaneously, with different files inside of them. The user then brutally decides to print everything, and quickly clicks the "print" button of both windows.

Without a mechanism in place to avoid this, here's what would happen: both word processors start to feed data to the printer, which gets confused and prints garbled output, basically a mixture of both documents. Not a pretty sight. To avoid this, we must put somewhere in the printer driver a mechanism which ensures that only one thread may be printing a document at the same time. Or, if we have two printers available and if which one is used does not matter, we can have a mechanism which ensures that only two threads may be printing a document at the same time.

The usual mechanism for this is called a semaphore, and typically works as follows: on the inside, the semaphore has a counter which represents how much time a resource can still be accessed. Each time a thread tries to access a resource protected by a semaphore, this counter is checked. If its value is nonzero, it is decreased by one and the thread is permitted to access the resource. If its value is zero, the thread may not access the resource. Notice that to be perfectly bullet-proof, the mechanism in use must ensure that the value of the semaphore is checked and changed in a single processor instruction that can't be run on several CPU cores at once, which requires a bit of help from the hardware. It's not as simple as just checking and changing an integer value. But how exactly this is done is not our business at this design stage.

Apart from semaphores, another, less frequently used but still well-known synchronization mechanism, is the barrier. It allows N threads to wait until each one has finished its respective work before moving on. This is particularly useful in situations where a task is parallelized in several chunks that may not necessarily take the same time to be completed (think as an example about rendering a 3D picture by slicing it in a number of parts and having each part be computed by a separate thread).

So in short, having defined your process model, you'll have to define how they will communicate with each other, and how threads' actions will be synchronized.

Major concerns and compromises

You may have noticed that I've done my best to stick to generic concepts and put some care in showing that there are several ways to do a given thing. That's not just for fun. There are several compromises to play with when designing a kernel, and depending on what your OS project's goals are, you'll probably have to consider them in different ways.

Here's an overview of some major concerns which you should have in mind when designing a kernel (though the importance of each varies depending on what your OS project is):

• Performance: Both in terms of hardware resource use and in terms of performance perceived by the user. These are not totally the same. As an example, in a desktop operating system, prioritizing software which the user is directly interacting with over background services improves perceived performance without requiring much optimization work. In a real-time operating system, hardware resource usage does not matter as much as meeting deadlines. And so on...

• Maintainability: Kernels are pretty hard to code, therefore you generally want them to last a fairly long time. For this to happen, it must be easy to grab their code and tinker with it as soon as a flaw is found or a feature is to be added. The codebase must thus be kept as short as possible, well-commented, well-documented, well-organized, and leave room for tweaking things without breaking the API.

• Portability: In our world of quickly-evolving hardware, operating system kernels should be easily portable to a new architecture. This is especially true in the realm of embedded devices, where the hardware is much more complex and ever-changing than it is on the desktop.

• Scalability: The other side of the portability coin is that your kernel should adapt itself to future hardware evolutions in a given architecture. This noticeably implies optimizing for multi-core chips: you should strive to restrict the parts of the kernel which can only be accessed by a limited number of threads to a minimal amount of code, and aggressively optimize your kernel for multi-threaded use.

• Reliability: Processes should not crash. But when they do crash, the impact of their crash should be minimized, and recoverability should be maximized. This is where maximizing process isolation, reducing the amount of code in loosely-isolated processes, and investigating means of backing up process data and restarting even the most critical services without rebooting the OS really shines.

• Security: On platforms which allow untrusted third-party software to run, there should be some protection against malware. You must understand right away that things like open-source, antiviruses, firewalls, and having software validated by a bunch of testers, are simply neither enough nor very efficient. These should only be tools for paranoids and fallback methods when system security has failed, and system security should fail as infrequently as possible. Maximal isolation of processes is a way to reach that result, but you must also minimize the probability that system component can be exploited by low-privilege code.

• Modularity: You'll generally want to make kernel components as independent from each other as possible. Aside from improving maintainability, and even reliability if you reach a level of modularity where you can restart failing kernel components on the fly without having the live system take a big hit, it also permits you to make some kernel features optional, a very nice feature, especially when applied to hardware drivers in kernels which include them.

• Developer goodies: In the dark ages of DOS, it was considered okay to ask from developers that they literally code hardware drivers in their software, as the operating system would do nearly nothing for them. This is not the case anymore. For everything which you claim to support, you must provide nice and simple abstractions which hide the underlying complexity of hardware behind a friendly, universal interface.

• Cool factor: Who'll use a new kernel if it's just the same as others, but in a much superior way? Let's introduce power-efficient scheduling, rickrolling panic screens, colored and formatted log output, and other fun and unique stuff!

Now, let's see how they end up in conflict with each other...

• The quest for performance conflicts with everything but scalability when taken too far (writing everything in assembly, not using function calls, putting everything in the kernel for better speed, keeping the level of abstraction minimal...)

• Maintainability conflicts with scalability, along with anything else that makes the codebase more complex, especially if the extra complexity can't be put in separate modules.

• Portability is in conflict with everything that requires using or giving access to architecture-specific features, particularly when arch-specific code ends up being spread all over the place and not tightly packed in a known corner (as in some forms of performance optimizations).

• Scalability is in conflict with any feature or construct which can't be used on 65536 CPU cores at the same time. Aside from the obvious compromise with maintainability and reliability which are better without hard-to-code and hard-to-debug threads spread all over the place, there's also a balance with some developer goodies (an obvious example being the libc and its hundreds of blocking system calls).

• Reliability is the fiend of anything which adds complexity, as more code statistically means more bugs, especially when said code is hard to debug. The conflict with performance is particularly big, as many performance optimizations require to provide code with more access to hardware than it actually requires. It is also the sole design criteria in this list to have the awesome property of conflicting with itself, as some system features can improve reliability.

• Security is a cousin of reliability as far as code complexity is concerned, since bugs can be exploitable. It also doesn't like low-level code where every single action is not checked (pointers arithmetic, C-style arrays...), which is more prone to exploitable failures than the rest.

• Modularity doesn't like chunks of code which must be put at the same place of RAM. This means a serious conflict with performance, since code locality allows optimal use of CPU caches. The relationship between modularity and maintainability is ambiguous: separating system components from each other initially helps maintainability a lot, but extreme forms of modularity like putting the scheduler (part of the kernel which manages multitasking) in a process can make the code quite confusing.

• We've previously seen that developer goodies and other cool stuff conflict with a large part of the rest for a number of reasons. Notice also an extra side of the feature vs maintainability conflict: it's easy to add features, but hard to remove them, and you don't know in advance how useful they will be. If you're not careful, this results in the phenomenon of feature bloat where the number of useless features grows exponentially with time. A way to avoid this is to keep the feature set minimal in the first release, then examine user feedback to see what is actually lacking. But beware of the "second-system effect", where you just put everything you're asked for in the second release, resulting in even worse feature bloat than if you had put a more extensive feature set to start with.

Some examples of kernel design

There are many operating system kernels in existence, though not all meet the same level of success. Here are a few stereotypical designs which tend to be quite frequently encountered (this list is by no mean exhaustive):

Monolithic kernels

The way all OS kernels were written long ago, for performance reasons, and still the dominant kernel design as of today. The monolithic kernel model remains quite attractive due to the extreme simplicity of its design. Basically, the kernel is a single big process running with maximum privileges and tending to include everything but the kitchen sink. As an example, it is common for desktop monolithic kernels to include facilities for rendering GPU-accelerated graphics and managing every single filesystem in existence.

Monolithic kernels shine especially in areas where high performance is needed, as everything is part of the same process. They are also easier to design, since the hardware resource model can be made simpler (only the kernel has direct access to hardware, user-space processes only have access to kernel-crafted abstractions), and since user-space is not a major concern until late in the development process. On the other hand, this way of doing things highers the temptation of using bad coding practices, resulting in unmaintainable, non-portable, non modular code. Due to the large codebase and the full access to hardware, bugs in a monolithic kernel are also more frequent and have a larger impact than in more isolated kernel designs.

Examples of monolithic kernel include Linux and its Android fork, most BSDs' kernels, Windows NTand XNU (Yes, I know, the two latter call themselves hybrid kernels, but that's mostly marketing. If you put most services in the same address space, with full access to the hardware, and without any form of isolation between each other, the result is still a monolithic kernel, with the advantages and drawbacks of monolithic kernels).

Micro-kernels

This is the exact opposite of a monolithic kernel in terms of isolation. The part of the kernel which has full access to the hardware is kept minimal (a few thousands of lines of executable code in the case of MINIX 3, to be compared with the millions of lines of code of monolithic kernels like Linux or Windows NT), and most kernel services are moved in separate services whose access to hardware is fine-tuned for their specific purpose.

Microkernels are highly modular by their very nature, and the isolated design favors good coding practices. Process isolation and fine-tuned access to hardware resources also ensure optimal reliability and security. On the other hand, microkernels are harder to write as much as they are easier to maintain, and the need to constantly switch from one process to another makes the most straightforward implementation perform quite poorly: it takes more optimization work to have a microkernel reach high performance, especially on the IPC side (as IPC becomes a critical mechanism).

Examples of commercial-grade microkernels include QNX, µ-velOSity and PikeOS. On the research side, one can mention MINIX 3, GNU Hurd, the L4 family, and the EROS family (KeyKOS, EROS,Coyotos, CapROS).

VM-based kernels

A fairly recent approach, which at the time has not fully gotten out of research labs and proof-of-concept demos. Maybe you'll be the one implementing it successfully. The idea here is that since most bugs and exploits in software come from textbook mistakes with native code (buffer overflows, dangling pointers, memory leaks...), native code is evil and should be phased out. The challenge is thus to code a whole operating system, including its kernel, in a managed language like C# or Java.

Benefits of this approach include obviously a very high cool factor and increased reliability and security. It could also potentially reach better performance than microkernels while providing similar isolation in a distant future, by isolating processes through a purely software mechanism (since all pointers and accesses to the hardware are checked by the virtual machine, no process may access resources which it's not allowed to access). On the other hand, nothing is free in the world of kernel development, and VM-based kernels have several major drawbacks to compensate for these high promises.

• The kernel must include a full featured interpreter for the relevant language, which means that the codebase will be huge, hard to design, and that very nasty VM bugs are to be expected during implementation.

• Making a VM fast enough that it is suitable for running kernel-level code full of hardware access and pointer manipulation is another big programming challenge.

• A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system. At the same time, exploits will have even worse consequences. Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs, so clearly something must change in the way we write VMs before they are suitable for inclusion in operating system kernels.

Examples of active VM-based kernel projects include Singularity, JNode, PhantomOS and Cosmos. There are also some interesting projects that are not active anymore, like JX and SharpOS (whose developers are now working in the MOSA project).

Bibliography, links, and friends

Having defined what the general concepts which you'll have to care about are, I bet you want to get into more details. In fact you should. So here is some material for going deeper than this introductory article on the subject of kernel design:

• Modern Operating Systems (Andrew S. Tanenbaum): Should you only read one book on the subject, I strongly recommend this one. It is an enlightening, extensive description of the subject, covering a lot of aspects of kernel design and which you may also use in much more parts of your OS development work.

• You may also find a list of other books, along with some reviews, on the OSdev wiki.

• While you're on said wiki, you might also want to have a look at its main page, more precisely at the "Design Considerations" links in the left column (scroll down a bit). Globally, you should bookmark this website, because you'll have a vital need for it once you start working on implementation. It's, simply put, the best resource I know of on the subject.

• And when you have question, also consider asking them in their forum. They are being asked hundreds of "how do I?" and "I'm stuck, what should I do?" implementation questions per month, so a bit of theoretical discussions would really please them. But beware of stupid questions which are answered in the wiki, otherwise prepare to face Combuster's sharp tongue.

• Questions for an OS designer is also an interesting read, although it doesn't go too deeply into specifics. I should have linked to it in my previous article.

And that's all for now. Next time, we're going to go a bit more platform-specific, as I'm going to describe basic characteristics of the x86 architecture, which will be used for the rest of this tutorial (I'll noticeably explain why).