computer-science/coursepages/ostep/OSTEP.md

Operating Systems: Three Easy Pieces

Credit goes to palladian

Introduction

First, we should be frank: it's really hard to find a good self-contained online course on operating systems. OSTEP is the best course we've found so far, but it does have some issues.

This is the first course in the OSSU curriculum for which you'll need to learn some prerequisites on your own before starting it, in addition to the courses that come before it in the curriculum. You might also run into some issues running the scripts for homework demos and for testing your solutions to the projects (although we hope we've solved most of those by now).

What this means for you is that if you're under a significant time crunch, or you're just not all that interested in systems programming and OS development, there's no shame in skipping this course and coming back to it later. You could also do only a part of the course (e.g. you might choose to skip the homework and/or projects).

That said, if you're able to commit the time required for the prerequisites, we believe the reward is well worth the effort: this course is exciting, interesting, and quite useful for other fields of computer science and programming. One big attraction of this course is the opportunity to see a simplified but fully-functional Unix-like operating system in action and understand the concepts and design decisions that went into it as well as the low-level implementation details.

In order to satisfy OSSU's curricular guidelines, you should either watch all the lecture videos or read chapters 1 through 47 in the textbook (don't worry, the chapters are usually just a few pages long) as well as finish the projects listed below. We also strongly encourage you to do the homework exercises as they're assigned on the course website or in the book chapters; think of these like the "check-your-understanding" questions that pop up in the middle of lecture videos on sites like Coursera or edX.

Prerequisites

This class requires a lot of experience programming in C. You should finish one of the C books listed in the resources below before starting this course; if you try to learn C at the same time as the course material, you're likely to feel overwhelmed. If you haven't used C before, you should expect to spend a lot of time on this; it's hard to predict how long it might take for each person, but a rough estimate might be 8-10 hours per week for 3-5 weeks. You can always learn C alongside another OSSU course or even redo the exercises for other courses in C to gain practice with it.

You should also finish both parts of Nand2Tetris before starting this course. OSTEP focuses on the real-world x86 and x86_64 architectures, so you'll have to fill in some gaps in order to translate the concepts you learned in Nand2Tetris to a new architecture. You can do that with the x86 resources below, but note that they all assume you know C, so learn that first. This should take around 6-8 hours in total.

Course Links

• Course website
• Book
• Lecture videos
• Homework
• Homework repo
• Projects
• xv6

Roadmap

This course was originally taught as CS 537 at the University of Wisconsin by the author of the OSTEP textbook, so the projects are assigned in the course according to the best times to give UWisconsin students access to on-campus resources like recitation sections and office hours. That means they don't match up perfectly with the material being covered at that time in the lectures or textbook chapters. We recommend doing the course in the following order instead.

Reading order

• Read chapters 1 and 2 of the OSTEP textbook and watch the first half (the introduction) of lecture 1.
• Do the initial-utilities project; it's intended as a litmus test for you to make sure you're comfortable enough with C before taking this class. You can watch discussion 1 for help. If it takes you more than 2 hours to write the code (not counting the discussion time and any time spent debugging), you should consider spending more time learning C before moving on in the course. (If you want more practice, you can do initial-reverse too, but it's not required.)
• Watch lectures 1 through 5 and read chapters 3 through 24 of the OSTEP textbook. We recommend doing the homework assignments as they come up in the course calendar or book chapters.
• Watch discussion 3 and reread chapter 5, then do the processes-shell project.
• Read the annotated guide to xv6 linked in the resources section below, starting from the beginning and stopping after the System Calls: Processes section.
• Watch discussion 2, then do the initial-xv6 project.
• Watch discussion 5, then do the scheduling-xv6-lottery project.
• Watch discussion 7, then do the vm-xv6-intro project.
• Watch lectures 6 through 9 (and optionally, the review lecture) and read chapters 25 through 34; again, you're encouraged to do the homework.
• Watch discussion 10, then do the concurency-xv6-threads project.
• Watch discussions 11 and 12, then do the concurrency-mapreduce project.
• Watch lectures through 14 (and optionally, the second review lecture) and read chapters 35 through 47; remember to do the homework along with the lectures or chapters.
• Do the filesystems-checker project.

Running the Projects

This course was originally taught as CS 537 at the University of Wisconsin by the author of the OSTEP textbook, which means that the homework and projects were written with those students as a target audience and designed to be run on UWisconsin lab computers with specific software versions pre-installed for students. We hope this section fixes that so you can run them on other computers, but we haven't tested this on every computer, so if you run into other issues, let us know on the Discord channel and we'll try to help out.

In order to run the homework and projects on Linux or macOS, you'll need to have all of the following programs installed:

• gcc
• gas
• ld
• gdb
• make
• objcopy
• objdump
• dd
• python
• perl
• gawk
• expect
• git

You will also need to install qemu, but we recommend using the patched version provided by the xv6 authors; see this link for details.

On macOS, you'll need to install a cross-compiler gcc suite capable of producing x86 ELF binaries; see the link above for details as well.

On Windows, you can use a Linux virtual machine for the homework and projects. Some of these packages are not yet supported on Apple M1 computers, and virtual machine software has not yet been ported to the new processor architecture; some students have used a VPS to do the homework and projects instead.

Next, clone the ostep-homework and ostep-projects repositories:

git clone https://github.com/remzi-arpacidusseau/ostep-homework/
git clone https://github.com/remzi-arpacidusseau/ostep-projects/
cd ostep-projects

You'll have to clone the xv6-public repository into the directory for each xv6-related OSTEP project. You could use the same copy for all the projects, but we recommend using separate copies to avoid previous projects causing bugs for later ones. Run the following commands in each of the initial-xv6, scheduling-xv6-lottery, vm-xv6-intro, concurrency-xv6-threads, and filesystems-checker directories.

mkdir src
git clone https://github.com/mit-pdos/xv6-public src

Hints and tips for Projects

• initial-reverse: the error messages that are needed to pass the tests were wrong! The provided text said "error: ..." but the tests expected "reverse: ..." so make sure to match the tests' expectations in your code.
• processes-shell: I had to edit /tests/3.pre to use /bin/ls due to how it's set up on my system, in order to pass all the tests. Alternatively you can add export PATH="/bin:$PATH" to your .profile or .bashrc file.
• hints for Project 1B: initial-xv6
• hints for scheduling-xv6-lottery
• hints for vm-xv6-intro

Resources

C

Please don't try to learn C from sites like GeeksforGeeks, TutorialsPoint, or Hackr.io (we're not even gonna link to them here). Those are great resources for other languages, but C has way too many pitfalls, and C tutorials online are often filled with dangerous errors and bad coding practices. We looked at many C resources for the recommendations below and unfortunately found many bad or unsafe ones; we'll only include the best ones here, so look no further!

We recommend learning C by working through (the entirety of) Jens Gustedt's Modern C, which is freely available online. This book is relatively short and will bring you up to speed on the C language itself as well as modern coding practices for it. Make sure to do all the exercises in the footnotes!

While the book above is our default recommendation, we also recommend K.N. King's C Programming: A Modern Approach as a second, more beginner-friendly option. It has some disadvantages: it's much longer (almost 850 pages), it's not available for free (and copies can be hard to find), and it's not quite as recent as Modern C (but still relevant nonetheless). That said, it has more exercises if you want extra practice, and the Q&A sections at the end of each chapter are filled with pearls of C wisdom and answers to C FAQs. It also covers almost the entirety of the C language and standard library, so it doubles as a reference book.

CS 50 doesn't quite cover enough C for OSTEP, but if you've already taken CS 50, you can supplement it with the books above.

Additional (optional) resources include:

• CS 50 Manual Pages: a great reference for looking up C library functions; most functions include both the usual manual as well as a beginner-friendly "less comfortable" option (just note that the "less comfortable" version uses string as an alias for char *.)
• cdecl: a tool to translate C gibberish into English.
• C track on exercism.io: additional practice exercises.
• Secure Coding Practices in C and C++: if you want to understand why other C resources are so unsafe.
• The C Programming Language: the original book on C by its creators. Too outdated for OSTEP, but a good read if you manage to find a copy.

x86 Architecture and Assembly Language

Nand2Tetris has already introduced most of the concepts you'll need to understand systems and computer architectures, so now you just need to port that knowledge to the real-world (32-bit) x86 architecture.

The easiest way to do that is by watching a subset of the lectures from the Computer Systems: A Programmer's Perspective course (or reading the matching chapters in the textbook of the same name). The lectures you'll need are:

• Machine-Level Programming I: Basics
• Machine-Level Programming II: Control
• Machine-Level Programming III: Procedures
• Machine-Level Programming IV: Data
• Machine-Level Programming V: Advanced Topics
• Linking

Additional (optional) resources include:

• CPU Registers x86: good for looking up specific registers.
• PC Assembly Language: a short book on x86 assembly.
• GCC Inline Assembly HOWTO: a guide to writing assembly code inside a C program.
• Intel 80386 Programmer's Reference Manual: the official (and huge) resourcefrom Intel.

xv6

You don't need to read anything about xv6 until after you start OSTEP; in fact, we recommend holding off on the xv6-related projects until you've finished the entire section on virtualization. After that, you'll need a guide to walk you through the source code.

The xv6 authors provide a book that you can read alongside the source code. There's also a handy line-numbered PDF version of the code with an index to see exactly where each function or constant gets used.

However, that book glosses over a lot of the details in the code that you might find challenging, including the advanced C features used, the x86 architecture- specific instructions, and the concurrency aspects (if you haven't finished that section of OSTEP before starting the xv6 projects). To solve this problem, we provide an annotated guide to xv6 that goes over the entire xv6 code and analyzes it line-by-line with explanations of the C features, hardware specs, and x86 conventions used. That means it's longer than the official xv6 book, so you don't have to read all of it (and you can probably skip the optional sections unless you care about device drivers), but you can use it as a reference if you're scratching your head about some part of the code.

Also here is an excellent video series walking through much of the xv6 code.

Miscellaneous

You'll need a general sense of how Makefiles work in order to use the Makefile for xv6. This tutorial covers much more than you need; just read the "Getting Started" and "Targets" sections and come back to the rest later if you need to look something up (but you shouldn't have to).

Additional (optional) resources include:

• GCC Command Options: a guide to command-line flags for the GNU C compiler gcc.
• Linker Scripts: a guide to writing scripts for the GNU linker ld.
• OSDev Wiki: a great resource for all kinds of OS concepts and implementation details.
• Linux Kernel Development: if you want to apply your xv6 knowledge toward contributing to the Linux kernel, this is a great read after OSTEP.