OSDEV Series - Episode 2.2: Short recap

Theory: RAM memory

• contains: code, data
• indexed linearly, using hexadecimal numbers
• measured in bytes and multiples (kilobytes, megabytes, gigabytes, terabytes etc)

Frequently used units of data measurement:

• bit = 1 or 0
• nibble = 4 bits (hex digit)
• byte = 8 bits
• word = 2 bytes = 16 bits
• double word (dword) = 4 bytes = 32 bits
• quadruple word (qword) = 8 bytes = 64 bits

Memory map:

• 'low' memory (under 1 mb)
• 'upper' memory (above 1 mb)

The separation appeared in the 16-bit era. At the beginning, 16-bit indexed memory could only reach 64kb in size (0xFFFF is the maximum number in 16 bits). Because engineers didn't imagine that anything would need more than 640kb of memory, they came up with the Segment:Offset memory addressing scheme, which addresses memory using 32 bits.

In the Segment:Offset addressing scheme, the address is specified using 2 words. The first one is the segment; every segment has 64kb in size, and a new segment starts every 16 bytes. This means that the segments overlap each other.  The other word is self explanatory.

There are many ways to specify the same address using this memory scheme, for example:

• Linear address: 0xA0000 (this is where the video ram starts)
• Segment:offset address: A000:0000 or 9100:F000 or 9FF0:0100

There can be up to 4096 ways to specify the same address.

Calculating the linear address, you need to multiply the segment with 0x10 (16 in hex) and add the offset.


Not all memory is available for use. The restricted areas include:
Low memory (< 1MB):
0x0 - 0x3FF:  Interrupt vector table
0x400 - 0x4FF: Bios data area
0x7C00 - 0x7DFF: The boot sector of the kernel. You can overwrite this area if you don't need it any more.

0x9FC00 - 0x9FFFF: The extended bios data area (may start from 0x80000)
0xA0000 - 0xBFFFF: Video RAM. This is where you usually write graphics to the screen (in graphic modes), or text (in text mode). For the default text mode (0x03), the address to write to is 0xB8000.
0xC0000 - 0xFFFFF: Read only, mostly code area. Contains the Video BIOS, some mapped hardware, and the motherboard BIOS.

Upper memory (>1MB):
These areas, if exist, must be read from the multiboot information structure. More about it in the future. There is a standard memory hole between 15 and 16MB, but again, you must read the information from the multiboot information structure.

Practice: the screen

If you read carefully above, to write text to the screen you need to write to address 0xB8000. For each character, you output 2 bytes: the ASCII code, and the color.

For example, to write a colored 'A' in position (3, 10):

• write ASCII code to 2 * (10 * ConsoleWidth + 3)
• write color code to 2 * (10 * ConsoleWidth + 3) + 1 (high nibble is the background, low nibble is the foreground).

To clear the screen, write zeros to all the bytes. The screen size in the default text mode is 80 x 25 characters, so you need to write a total of 80 * 25 * 2 bytes. If you want to color the background when the screen is cleared, set the color property for every character to the desired color (high nibble for background, low for foreground).

All these routines are implemented and explained in the video tutorial. You can also download the project files here: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-2.2.zip

OSDEV Series - Episode 2.2 - The screen

Section 2: The basics
Episode 2: The screen

Project files: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-2.2.zip

Covered topics:
* Theory: we learn about memory, how it works and what it contains, and how to use it properly
* Practice: we write some very basic routines that display text to the screen.

OSDEV Series - Episode 1.4: Short recap

In this episode, we set up a proper environment for Windows. Unlike Linux, where things are much easier, on Windows we have to build ourselves the tools that we have to use.

Download list:

• Virtual Floppy Drive: http://sourceforge.net/projects/vfd/
• GRUB 0.97: http://ftp.kaist.ac.kr/gnu-alpha/grub/grub-0.97-i386-pc.tar.gz
• Binutils Source Code: ftp://ftp.gnu.org/gnu/binutils/
• GCC Source Code: ftp://ftp.gnu.org/gnu/gcc/

Creating GRUB floppy image
Same as under Linux (Episode 1.3), except that we use Virtual Floppy Drive (VFD) instead of mount.

Building binutils and GCC
Step 1: System requirements
Make sure you have enough disk space (my installation of Cygwin had 1.6 GB after building and installing everything), and enough RAM memory (at least 512 MB).


Step 2: Download necessary tools
There are some additional tools you need for the build. To install them, you need to run the Cygwin installation. If you deleted the file, you can download it here. Install the following packages:

• Devel > bison
• Devel > flex
• Devel > gcc
• Devel > gcc-core
• Devel > gcc-g++
• Devel > make
• Libs > libgmp-devel 
• Libs > libmpfr-devel 
• Libs > libmpc-devel

Step 3: Remove other GCC ports
If you have MinGW or DJGPP installed, you will need to rename the folder they are located in. For example, you can add an underscore at the end of the folder name. This way, GCC and binutils won't find them, and won't use them. If you don't, the build could fail.


Step 4: Download the source code
Use the links provided above to download binutils and GCC source code. Unpack them in the /usr/src folder, this is usually found in [cygwin_installation]\usr\src (default is C:\cygwin\usr\src). You should use 7zip, WinRar, or the tar command in cygwin to do this, otherwise the build could fail.



Step 5: Preparation
Launch Cygwin, and type the following commands:

export PREFIX=/usr/local/cross
export TARGET=i586-elf
cd /usr/src
mkdir build-binutils build-gcc

The first two commands will create two environment variables. PREFIX is where binutils and GCC will be installed, and TARGET is what platform they will create executables for.
Then we create two directories in /usr/src, which will be used to build these tools.


Step 6: Build and install binutils
Type the following commands:

cd /usr/src/build-binutils
../binutils-x.xx/configure --target=$TARGET --prefix=$PREFIX --disable-nls
make all
make install

Replace x.xx with the version number (you can press TAB to autocomplete in Cygwin).
So we configure binutils, build it, and then install it.


Step 7: Build and install GCC
Type the following commands:

cd /usr/src/build-gcc
export PATH=$PATH:$PREFIX/bin
../gcc-x.x.x/configure --target=$TARGET --prefix=$PREFIX --disable-nls --enable-languages=c,c++ --without-headers
make all-gcc
make install-gcc

Replace x.x.x with the version number (you can press TAB to autocomplete).
Now we append to the PATH environment variable (which already exists) the location where binutils was installed.
Then, configure gcc, build and install it.


Step 8: Build and install libgcc
As a useful addition, you may also like to build libgcc, the GCC low-level runtime library. Linking against libgcc provides integer, floating point, decimal, stack unwinding (useful for exception handling) and other support functions. Once you have built and installed the GCC Cross-Compiler, keep Cygwin opened, and type:

make all-target-libgcc
make install-target-libgcc

Step 9: Edit PATH environment variable
Right click on My Computer, and go to Properties. If you have Windows 7, look for where it says "Computer name". There should be a link there which says Change Settings.
Go to the Advanced tab, and click Environment Variables. Look for the PATH variable, and edit it. Put a semicolon at the end, and add the following path:

[cygwin installation]\usr\local\cross\bin 

(default is C:\cygwin\usr\local\cross\bin)


Step 10: Cleanup
If you renamed any folder at Step 3, you may restore the original name. You can now delete the source code for GCC and binutils, as well as the build-gcc and build-binutils folders.

Troubleshooting: You can visit this link for more information on how to build binutils and GCC, as well as troubleshooting information.


Building our operating system
Now we can go back to our operating system and build it.

Open Cygwin, and create two folders:

mkdir /mnt

mkdir /mnt/floppy

Now edit build.sh:
Add two variables:

compiler=i586-elf-gcc

linker=i586-elf-ld

Now replace gcc and ld, with $compiler, respectively $linker. This way, if somehow the compiler or linker may change, we only have to edit one single line instead of a thousand.

Also, replace all the code that copies kernel.bin to the floppy image, with these two lines:

mount A: /mnt/floppy

cp kernel.bin /mnt/floppy/

You can save and close the file.


The last step is to open the myos.img image, using VFD, and launching the build.
To launch the build, navigate to your work directory (/cygdrive/[drive]/[path]), and execute ./build.sh

OSDEV Series - Episode 1.4 - Windows trouble

Section 1: Introduction
Episode 4: Windows trouble

Project files: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-1.4.zip

Covered topics:
In this episode, we create a proper environment to create operating systems, on Windows.
* Using Virtual Floppy Drive (to replace mount)
* Creating a GRUB floppy image
* Building GCC and Binutils
* Building the kernel created in the previous episode, on a Windows computer.

Special credits:
http://wiki.osdev.org for the building GCC tutorial.

OSDEV Series - Episode 2.1: Short recap

Rings

Rings offer a protection layer for programs. They allow certain levels of resource access to processes, so it keeps bad programs from messing things up. There are a total of 4 rings in most common architectures. However, many architectures have only two rings (e.g. x86_64), corresponding to ring 0 and 3 in this description.

Ring 0:
This is kernel mode or supervisor mode. This level has the least protection, and the most access to resources. When starting up, the OS runs in this mode unless it switches out. Interrupt handlers run in this mode.

Rings 1 and 2:
These rings are mostly used for device drivers. They offer more protection, but not as much as ring 3.

Ring 3:
This is the ring that most OS's use for applications. This ring is also called Userland, or Userspace. It has the most protection and the least resource access.

Most operating systems use only Ring 0 and 3. This is because rings 1 and 2 are unneeded, as device drivers can run in either ring.
Sometimes applications need access to resources that their ring wont allow. If they try to access them, a General Protection Fault (int 13) will be triggered, and the application shutdown. The application must interface with the kernel somehow, and mostly this is done with System Calls.

Kernel designs

Microkernels:
A microkernel only implements the basic services needed for applications, such as memory management, or multitasking. Other services implemented in user space, also known as servers, offer additional functionality that the kernel would offer in other designs. Such services may include networking, file systems etc.

Pros: Lightweight kernel, easy to maintain.
Cons: A large number of system calls is required for servers, and this costs a lot of performance.


Monolithic kernels:
All the services are run in kernel space

Pros:
Rich and powerful hardware access
Easier to implement than other designs

Cons:
All the services must run smoothly, otherwise the entire system could crash
Harder to maintain, since the kernel is pretty large

Examples of kernels:
UNIX (and UNIX-like)
DOS


Other kernel designs:

• Hybrid kernels: A compromise between the two big designs. The most popular example of a hybrid kernel is Windows NT.
• Nanokernels: Very small microkernels.
• Exokernels: As little abstraction as possible. This gives very much power to applications. However, it is not a very deeply researched design, and implementation can be very difficult.

Abstraction

In computer science, abstraction is the process by which data and programs are defined with a representation similar to its meaning (semantics), while hiding away the implementation details. In other words, it is the method of hiding away ugly stuff behind really nice functions that programmers can use.

To exemplify, let's look at a typical file read operation. The application reads a file from the computer, by calling an fread() function from the system library. The fread() function itself, makes a system call, asking the kernel to read the file. The kernel goes through the virtual file system to find the file. First it finds the device, such as the hard disk or memory card, then reads the actual sectors, and then parses the file system. The sector read function may be calling the firmware present in the device.

All these steps are in fact abstraction layers. A read operation starts at top level, where the application is situated, and goes all the way down to hardware level.

OSDEV Series - Episode 2.1 - Rings and kernel designs

Section 2: The basics
Episode 1: Rings and kernel designs


Covered topics:

Theory: Rings, kernel designs, abstraction
Practice: Coding the entry point in assembly, creating linker and shell scripts for compiling
Project files: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-2.1.zip

OSDEV Series - Episode 1.3: Short recap

The boot process:
When you turn on your computer, several things happen:
1. All the hardware is powered
2. When power is good, the processor starts, executing firmware
3. The firmware (also known as the BIOS = Basic Input/Output System) runs system checks (POST = Power On Self Test)
4. The BIOS looks for bootable devices, such as diskettes, hard disks, or DVDs
5. When a bootable device is found, the first sector is loaded in memory, and executed.
Inside the first sector, there is a part of our operating system. Now the operating system has full control over the computer.


Components of an operating system:
1. Bootloader
Description: It is a small program, that is located in the boot sector.
Role: Does hardware detection, prepares the computer for operating mode, and executes the kernel. Some bootloaders also give the possibility of booting multiple kernels, and show a menu to the user.
Why it exists: The space in the boot sector is very limited. For example, on a floppy disk there are only 512 bytes.


2. Kernel
Description: The core of all operating systems.
Role: It does essential stuff, such as:
- Hardware input and output
- Memory management
- Makes executing programs possible

3. Programs
Special programs:
Drivers: have special privileges in comparison with normal programs.
Shell: comes with most operating systems. It is like an 'add-on' that allows user interaction.
Normal programs:
Examples: text editor, browser, videogames etc.



Setting up GRUB:
There are 2 floppy images needed:
1. myos.img - will contain the operating system
2. aux.img - auxiliary image, used to install GRUB on image #1.

The following commands will be executed in the Linux terminal, or Cygwin.


Step 1: Generating the images:
dd bs=512 count=2880 if=/dev/zero of=myos.img
dd bs=512 count=2880 if=/dev/zero of=aux.img

Step 2: Format and mount myos.img:
Attention: Linux users need administrator priviledges for the following instructions. This is done by placing sudo before the commands, in Debian based distributions.
Format the image: mkfs.msdos myos.img
Create a directory: mkdir /media/floppy1
Mount the image: mount -o loop myos.img /media/floppy1

Step 3: Create configuration file for GRUB
Create a new file called menu.cfg, and using your favorite text editor, write the following lines:

  default 0

  timeout 0

  hiddenmenu

  title MyOS

  root (fd0)

  kernel /kernel.bin

  boot

Step 4: Put all necessary files on the mounted floppy image
The floppy disk will contain the following file structure:
/boot/stage1
/boot/stage2
/boot/menu.cfg
kernel.bin


kernel.bin is the kernel of the operating system. We will create this file in the next episode.

Now we will copy all the files on the disk:
Create boot folder: mkdir /media/floppy1/boot
Copy stage files: cp stage? /media/floppy1/boot
Copy configuration file: cp menu.cfg /media/floppy1/boot

Step 5: Unmount the image
Unmount it: umount /media/floppy1
Delete folder: rm -r /media/floppy1

Step 6: Copy stage1 and stage2 directly inside the auxiliary image
stage1: dd bs=512 count=1 if=stage1 of=aux.img conv=notrunc
stage2: dd bs=512 seek=1 if=stage2 of=aux.img conv=notrunc

Step 7: Install GRUB on disk #1 using the auxiliary image
Using VirtualBox, we will start a virtual machine, with the auxiliary image inserted.
After the GRUB command line appears, insert myos.img, and type the following command:
install (fd0)/boot/stage1 (fd0) (fd0)/boot/stage2 (fd0)/boot/menu.cfg




Finally, I organized the files a little bit, and this is the new folder structure:
Boot/grub/aux.img
Boot/grub/menu.cfg

Boot/grub/stage1

Boot/grub/stage2

Boot/myos.img

OSDEV Series - Episode 1.3 - Babysteps

Section 1: Introduction
Episode 3: Babysteps



Covered topics in this episode:

Theory: the boot process, components of an operating system
Practice: setting up GRUB

In the next episode, we start writing our kernel, and we will learn more about kernel designs.

OSDEV Series - Episode 1.2: Software list

Here is a list with all the software used, and how to get it.

Name: Cygwin
Operating system: Windows
Download source: http://www.cygwin.com/
Description: Linux like shell that runs in windows, also includes compiler, linker and many other useful tools.

Name: GCC Toolchain
Operating system: Linux
Download source: Should be already installed on most Linux distributions. Can be installed using** sudo apt-get install gcc-core gcc-g++
Description: C and C++ compiler.

Name: NASM (NetWide Assembler)
Operating system: Windows*, Linux
Download source: ** sudo apt-get install nasm
Description: x86 assembler.

Name: Notepad2
Operating system: Windows
Download source: http://www.flos-freeware.ch/notepad2.html
Description: Simple text editor with syntax highlighting.

Name: Notepad++
Operating system: Windows
Download source: http://notepad-plus-plus.org/
Description: Advanced source code editor, has many more features than Notepad2.

Name: GEdit
Operating system: Linux
Download source: Already installed on Gnome distributions.
Description: Simple text editor with syntax highlighting.

Name: Kate
Operating system: Linux
Download source: Already installed on KDE distributions.
Description: Simple text editor with syntax highlighting.

Name: HxD
Operating system: Windows
Download source: http://mh-nexus.de/en/hxd/
Description: Powerful hex editor.

Name: GNOME Hexadecimal Editor
Operating system: Linux
Download source: ** sudo apt-get install ghex
Description: Hex editor for Gnome.

Name: VirtualBox
Operating system: Windows, Linux
Download source: http://www.virtualbox.org/wiki/Downloads
Description: Virtualization software.

Name: Qemu
Operating system: Linux
Download source: ** sudo apt-get install qemu
Description: Virtualization software for linux.

Name: Bochs
Operating system: Windows, Linux
Download source:
Linux:** sudo apt-get install bochs
Windows: http://bochs.sourceforge.net/
Description: Operating system debugger.

Notes:
* On Windows, Nasm should be installed using Cygwin, for full compatibility with the tutorial.
** The apt-get command is only available on Debian/Ubuntu based distributions. On other distributions, there may be Yum or other package managers installed.

OSDEV Series - Episode 1.2 - Setting up the environment

Section 1: Introduction
Episode 2: Setting up the environment



This is the second episode in the series, and this time we will set up our environment for operating system development.
Windows starts at 0:00, Linux at 5:00.

In the next episode, we will install the GRUB bootloader on a virtual floppy image, and we will cover some basic theory about operating systems.

OSDEV Series - Episode 1.1 - To begin with

Section 1: Introduction
Episode 1: To begin with




This episode is an introduction to the OSDEV Series, that will teach how to create an operating system. Topics covered:
* Beginner mistakes
* Checklist (needed tools)

In the next episodes, we will download, install and configure these tools both on Linux and Windows.