LUX Operating System build 0.1.0.361 (08/24/2011): back in business

The lux operating system is back in business. The old project, CTA OS was completely rewritten, and now has a new name: lux. This time, the bootloader used is Grub.

Here are some screenshots of the latest build:

The operating system now has a logging system, which shows what is going on. It can also be turned off, at compile time. It is planned to expand this feature, the log to be written to files and optionally to the console, once the file systems are implemented.

Keyboard driver is implemented, and also mouse driver... the white cursor you see is in fact the mouse cursor.

The console is colorful as always...  however, development on the console will not continue until it is moved in user space.

A new nice feature of the console, it displays a full memory map (red blocks are used, green are free).

Floppy driver is implemented, and it was tested and works on real hardware. However, I have only tested on 3.5" 1.44 MB disks.

OSDEV Series - Episode 1.4: Short recap

In this episode, we set up a proper environment for Windows. Unlike Linux, where things are much easier, on Windows we have to build ourselves the tools that we have to use.

Download list:

• Virtual Floppy Drive: http://sourceforge.net/projects/vfd/
• GRUB 0.97: http://ftp.kaist.ac.kr/gnu-alpha/grub/grub-0.97-i386-pc.tar.gz
• Binutils Source Code: ftp://ftp.gnu.org/gnu/binutils/
• GCC Source Code: ftp://ftp.gnu.org/gnu/gcc/

Creating GRUB floppy image
Same as under Linux (Episode 1.3), except that we use Virtual Floppy Drive (VFD) instead of mount.

Building binutils and GCC
Step 1: System requirements
Make sure you have enough disk space (my installation of Cygwin had 1.6 GB after building and installing everything), and enough RAM memory (at least 512 MB).


Step 2: Download necessary tools
There are some additional tools you need for the build. To install them, you need to run the Cygwin installation. If you deleted the file, you can download it here. Install the following packages:

• Devel > bison
• Devel > flex
• Devel > gcc
• Devel > gcc-core
• Devel > gcc-g++
• Devel > make
• Libs > libgmp-devel 
• Libs > libmpfr-devel 
• Libs > libmpc-devel

Step 3: Remove other GCC ports
If you have MinGW or DJGPP installed, you will need to rename the folder they are located in. For example, you can add an underscore at the end of the folder name. This way, GCC and binutils won't find them, and won't use them. If you don't, the build could fail.


Step 4: Download the source code
Use the links provided above to download binutils and GCC source code. Unpack them in the /usr/src folder, this is usually found in [cygwin_installation]\usr\src (default is C:\cygwin\usr\src). You should use 7zip, WinRar, or the tar command in cygwin to do this, otherwise the build could fail.



Step 5: Preparation
Launch Cygwin, and type the following commands:

export PREFIX=/usr/local/cross
export TARGET=i586-elf
cd /usr/src
mkdir build-binutils build-gcc

The first two commands will create two environment variables. PREFIX is where binutils and GCC will be installed, and TARGET is what platform they will create executables for.
Then we create two directories in /usr/src, which will be used to build these tools.


Step 6: Build and install binutils
Type the following commands:

cd /usr/src/build-binutils
../binutils-x.xx/configure --target=$TARGET --prefix=$PREFIX --disable-nls
make all
make install

Replace x.xx with the version number (you can press TAB to autocomplete in Cygwin).
So we configure binutils, build it, and then install it.


Step 7: Build and install GCC
Type the following commands:

cd /usr/src/build-gcc
export PATH=$PATH:$PREFIX/bin
../gcc-x.x.x/configure --target=$TARGET --prefix=$PREFIX --disable-nls --enable-languages=c,c++ --without-headers
make all-gcc
make install-gcc

Replace x.x.x with the version number (you can press TAB to autocomplete).
Now we append to the PATH environment variable (which already exists) the location where binutils was installed.
Then, configure gcc, build and install it.


Step 8: Build and install libgcc
As a useful addition, you may also like to build libgcc, the GCC low-level runtime library. Linking against libgcc provides integer, floating point, decimal, stack unwinding (useful for exception handling) and other support functions. Once you have built and installed the GCC Cross-Compiler, keep Cygwin opened, and type:

make all-target-libgcc
make install-target-libgcc

Step 9: Edit PATH environment variable
Right click on My Computer, and go to Properties. If you have Windows 7, look for where it says "Computer name". There should be a link there which says Change Settings.
Go to the Advanced tab, and click Environment Variables. Look for the PATH variable, and edit it. Put a semicolon at the end, and add the following path:

[cygwin installation]\usr\local\cross\bin 

(default is C:\cygwin\usr\local\cross\bin)


Step 10: Cleanup
If you renamed any folder at Step 3, you may restore the original name. You can now delete the source code for GCC and binutils, as well as the build-gcc and build-binutils folders.

Troubleshooting: You can visit this link for more information on how to build binutils and GCC, as well as troubleshooting information.


Building our operating system
Now we can go back to our operating system and build it.

Open Cygwin, and create two folders:

mkdir /mnt

mkdir /mnt/floppy

Now edit build.sh:
Add two variables:

compiler=i586-elf-gcc

linker=i586-elf-ld

Now replace gcc and ld, with $compiler, respectively $linker. This way, if somehow the compiler or linker may change, we only have to edit one single line instead of a thousand.

Also, replace all the code that copies kernel.bin to the floppy image, with these two lines:

mount A: /mnt/floppy

cp kernel.bin /mnt/floppy/

You can save and close the file.


The last step is to open the myos.img image, using VFD, and launching the build.
To launch the build, navigate to your work directory (/cygdrive/[drive]/[path]), and execute ./build.sh

OSDEV Series - Episode 1.4 - Windows trouble

Section 1: Introduction
Episode 4: Windows trouble

Project files: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-1.4.zip

Covered topics:
In this episode, we create a proper environment to create operating systems, on Windows.
* Using Virtual Floppy Drive (to replace mount)
* Creating a GRUB floppy image
* Building GCC and Binutils
* Building the kernel created in the previous episode, on a Windows computer.

Special credits:
http://wiki.osdev.org for the building GCC tutorial.

OSDEV Series - Episode 2.1: Short recap

Rings

Rings offer a protection layer for programs. They allow certain levels of resource access to processes, so it keeps bad programs from messing things up. There are a total of 4 rings in most common architectures. However, many architectures have only two rings (e.g. x86_64), corresponding to ring 0 and 3 in this description.

Ring 0:
This is kernel mode or supervisor mode. This level has the least protection, and the most access to resources. When starting up, the OS runs in this mode unless it switches out. Interrupt handlers run in this mode.

Rings 1 and 2:
These rings are mostly used for device drivers. They offer more protection, but not as much as ring 3.

Ring 3:
This is the ring that most OS's use for applications. This ring is also called Userland, or Userspace. It has the most protection and the least resource access.

Most operating systems use only Ring 0 and 3. This is because rings 1 and 2 are unneeded, as device drivers can run in either ring.
Sometimes applications need access to resources that their ring wont allow. If they try to access them, a General Protection Fault (int 13) will be triggered, and the application shutdown. The application must interface with the kernel somehow, and mostly this is done with System Calls.

Kernel designs

Microkernels:
A microkernel only implements the basic services needed for applications, such as memory management, or multitasking. Other services implemented in user space, also known as servers, offer additional functionality that the kernel would offer in other designs. Such services may include networking, file systems etc.

Pros: Lightweight kernel, easy to maintain.
Cons: A large number of system calls is required for servers, and this costs a lot of performance.


Monolithic kernels:
All the services are run in kernel space

Pros:
Rich and powerful hardware access
Easier to implement than other designs

Cons:
All the services must run smoothly, otherwise the entire system could crash
Harder to maintain, since the kernel is pretty large

Examples of kernels:
UNIX (and UNIX-like)
DOS


Other kernel designs:

• Hybrid kernels: A compromise between the two big designs. The most popular example of a hybrid kernel is Windows NT.
• Nanokernels: Very small microkernels.
• Exokernels: As little abstraction as possible. This gives very much power to applications. However, it is not a very deeply researched design, and implementation can be very difficult.

Abstraction

In computer science, abstraction is the process by which data and programs are defined with a representation similar to its meaning (semantics), while hiding away the implementation details. In other words, it is the method of hiding away ugly stuff behind really nice functions that programmers can use.

To exemplify, let's look at a typical file read operation. The application reads a file from the computer, by calling an fread() function from the system library. The fread() function itself, makes a system call, asking the kernel to read the file. The kernel goes through the virtual file system to find the file. First it finds the device, such as the hard disk or memory card, then reads the actual sectors, and then parses the file system. The sector read function may be calling the firmware present in the device.

All these steps are in fact abstraction layers. A read operation starts at top level, where the application is situated, and goes all the way down to hardware level.

OSDEV Series - Episode 2.1 - Rings and kernel designs

Section 2: The basics
Episode 1: Rings and kernel designs


Covered topics:

Theory: Rings, kernel designs, abstraction
Practice: Coding the entry point in assembly, creating linker and shell scripts for compiling
Project files: http://dl.dropbox.com/u/24832466/OSDEV/MyOS-2.1.zip

OSDEV Series - Episode 1.3: Short recap

The boot process:
When you turn on your computer, several things happen:
1. All the hardware is powered
2. When power is good, the processor starts, executing firmware
3. The firmware (also known as the BIOS = Basic Input/Output System) runs system checks (POST = Power On Self Test)
4. The BIOS looks for bootable devices, such as diskettes, hard disks, or DVDs
5. When a bootable device is found, the first sector is loaded in memory, and executed.
Inside the first sector, there is a part of our operating system. Now the operating system has full control over the computer.


Components of an operating system:
1. Bootloader
Description: It is a small program, that is located in the boot sector.
Role: Does hardware detection, prepares the computer for operating mode, and executes the kernel. Some bootloaders also give the possibility of booting multiple kernels, and show a menu to the user.
Why it exists: The space in the boot sector is very limited. For example, on a floppy disk there are only 512 bytes.


2. Kernel
Description: The core of all operating systems.
Role: It does essential stuff, such as:
- Hardware input and output
- Memory management
- Makes executing programs possible

3. Programs
Special programs:
Drivers: have special privileges in comparison with normal programs.
Shell: comes with most operating systems. It is like an 'add-on' that allows user interaction.
Normal programs:
Examples: text editor, browser, videogames etc.



Setting up GRUB:
There are 2 floppy images needed:
1. myos.img - will contain the operating system
2. aux.img - auxiliary image, used to install GRUB on image #1.

The following commands will be executed in the Linux terminal, or Cygwin.


Step 1: Generating the images:
dd bs=512 count=2880 if=/dev/zero of=myos.img
dd bs=512 count=2880 if=/dev/zero of=aux.img

Step 2: Format and mount myos.img:
Attention: Linux users need administrator priviledges for the following instructions. This is done by placing sudo before the commands, in Debian based distributions.
Format the image: mkfs.msdos myos.img
Create a directory: mkdir /media/floppy1
Mount the image: mount -o loop myos.img /media/floppy1

Step 3: Create configuration file for GRUB
Create a new file called menu.cfg, and using your favorite text editor, write the following lines:

  default 0

  timeout 0

  hiddenmenu

  title MyOS

  root (fd0)

  kernel /kernel.bin

  boot

Step 4: Put all necessary files on the mounted floppy image
The floppy disk will contain the following file structure:
/boot/stage1
/boot/stage2
/boot/menu.cfg
kernel.bin


kernel.bin is the kernel of the operating system. We will create this file in the next episode.

Now we will copy all the files on the disk:
Create boot folder: mkdir /media/floppy1/boot
Copy stage files: cp stage? /media/floppy1/boot
Copy configuration file: cp menu.cfg /media/floppy1/boot

Step 5: Unmount the image
Unmount it: umount /media/floppy1
Delete folder: rm -r /media/floppy1

Step 6: Copy stage1 and stage2 directly inside the auxiliary image
stage1: dd bs=512 count=1 if=stage1 of=aux.img conv=notrunc
stage2: dd bs=512 seek=1 if=stage2 of=aux.img conv=notrunc

Step 7: Install GRUB on disk #1 using the auxiliary image
Using VirtualBox, we will start a virtual machine, with the auxiliary image inserted.
After the GRUB command line appears, insert myos.img, and type the following command:
install (fd0)/boot/stage1 (fd0) (fd0)/boot/stage2 (fd0)/boot/menu.cfg




Finally, I organized the files a little bit, and this is the new folder structure:
Boot/grub/aux.img
Boot/grub/menu.cfg

Boot/grub/stage1

Boot/grub/stage2

Boot/myos.img

OSDEV Series - Episode 1.3 - Babysteps

Section 1: Introduction
Episode 3: Babysteps



Covered topics in this episode:

Theory: the boot process, components of an operating system
Practice: setting up GRUB

In the next episode, we start writing our kernel, and we will learn more about kernel designs.