LTSP+OpenMosix: A Mini How-To
Revision 3.3 - April 19, 2002
What is this?
This document will provide details (via an example set-up) of combining the OpenMosix clustering system with the K12LTSP 2.0 distribution, an innovative GNU/Linux distribution based on RedHat 7.2 & LTSP 3.0 (a set of Open Source software packages from the Linux Terminal Server Project that makes it all happen). OpenMosix (http://openmosix.sourceforge.net/) is a completely Open Source fork of the MOSIX clustering system.
OpenMosix, like the original MOSIX, is a clustering system that in the most basic of terms combines the cpu resources of the client systems into what could be compared to an SMP processing system. Processes get "moved" through load balancing to the the nodes (Linux Terminal Client computers) in the cluster that can be most effective in running them. Fully automated. No special programming required as in other clustering systems. The hope or theory is that since the K12LTSP client is simply executing XWindows and nothing more that OpenMosix can be implemented to utilize the idle processor cycles on these clients to maximize the efficiency of the entire system--relieving some of the stress that the server is under.
This document was derived as a result of Richard Camps very useful How-To, "Mosix Cluster with Diskless Nodes", which detailed adding MOSIX clustering functionality to an installation of the K12LTSP 2.0 distribution. (Without his article I would still be fighting to get this stuff working!). However, it should be noted that I alone am responsible for this particular document and all of the mistakes in it are mine! :-)
Mandatory Disclaimer: This article is provided as an example of how I got this working on my test setup. It is not the final word on the subject and it should be noted--use this information at your own risk. No warranty or guarantee is expressed or implied. I am not responsible for any damage resulting from your implementation of this information. Again, use this How-To at your own risk.
Having said that, let's get going!
Why OpenMosix instead of the original MOSIX?
- As I stated previously, Richard Camp posted an excellent How-To on installing the original MOSIX system with the K12LTSP 2.0.x distribution. It pointed me in the right direction and I'm truly grateful for his efforts. Without which, who knows if I'd have ever gotten anywhere with the concept of combining LTSP & MOSIX. Unfortunately in my case, the MOSIX install script that came from the MOSIX download always failed when the 2.4.17 kernel compile would throw errors during the modules_install phase. MOSIX simply would not patch & install automatically with the installation script. I began investigating a manual MOSIX kernel patch & userland tools installation and came across the OpenMosix project in the process. I found OpenMosix very simple to implement.
- MOSIX is either no longer Open Source or it's status as Open Source software is uncertain at this time. Without clarity on this issue, I would think that anyone implementing MOSIX should first obtain permission to do so from the copyright holder, Amnon Barak.
As of this writing, I don't have all of the details of MOSIX's current licensing status and I did not find any statement regarding the issue on the MOSIX site but you may wish to review the following for more information on the matter--
About OpenMosix Page:
http://openmosix.sourceforge.net/about.html
Interview with Moshe Bar (founder of OpenMosix & a former project manager for MOSIX):
http://foundries.sourceforge.net/clusters/index.pl?node_id=41457&lastnode_id=131
The OpenMosix FAQ:
http://openmosix.sourceforge.net/faq.html
The uncertainty surrounding MOSIX licensing and the ease of installation makes OpenMosix an ideal alternative for our purposes.
MOSIX Installation background Information, do not proceed without it!!!
Please! Do yourself a favor, obtain, study, and digest Richard Camps original MOSIX+LTSP How-To "Mosix Cluster with Diskless Nodes". Richard's How-To contains information that will not be covered here but may prove useful in getting your OpenMosix cluster running.
You can find his original article at http://www.k12os.org/print.php?sid=14
And, without further ado, away we go...
My Test Set-up used in this example:
Hardware:
Server:
Pentium II w/ 6GB IDE drive, 96MB RAM, Intel 10/100 NIC. K12LTSP 2.0 installed.
Clients:
Various systems, booting from floppy: 486, Pentium, Pentium MMX, Pentium II.
3com 10Mbit or Intel 10/100Mbit NICs. 16-64MB RAM
Network:
Simple 4-port 10baseT hub. (How's that for bandwidth? ;-)
Nothing spectacular here just an odd assortment of stuff, but it's been a useful test setup for me. And, even more importantly, it actually works! :-)
Software:
K12LTSP 2.0.x Distribution:
http://k12ltsp.org/download.html
LTSP Files (particularly "ltsp_initrd_kit-3.0.1-i386.tgz"):
http://sourceforge.net/project/showfiles.php?group_id=17723
OpenMosix Files:
http://sourceforge.net/project/showfiles.php?group_id=46729
Files needed in this example (Note: this example employs the latest openmosix software at this writing, openmosix5. Obtain the following OpenMosix files from SourceForge):
- The K12LTSP 2.0.x Distribution set
- ltsp_initrd_kit-3.0.1-i386.tgz (from the LTSP File List page. Currently available in the LTSP 3.0 ISO image under the directory /3.0/tgzs/ or here. The md5sum for this file from the CD is: bc5e68e1ee75a6ecf304c629ec7f0687.)
- openmosix-kernel-2.4.16-openmosix5.i686.rpm (Server Kernel)
- openmosix-tools-0.1.3-6.i386.rpm (Userland Tools for OpenMosix)
- openmosix-kernel-2.4.16-openmosix5.src.rpm (Source files for Client Kernel)
OR (instead of the openmosix-kernel-2.4.16-openmosix5.src.rpm)
- openMosix2.4.16-4.gz (Patch for Kernel 2.4.16 if not using the above source rpm)
- Kernel 2.4.16 from The Linux Kernel Archives (if not using the above source rpm)
mcedit is the text editor that will be used throughout this example. If you wish to utilize it, you can install it from the RPMS section on Disc 1 of the K12LTSP 2.0.x distribution via: rpm -ivh mc-4.5.51-36.i386.rpm, or substitute the editor of your choice.
The step-by-step:
(Note: items following the ">" are command line items you need to run.)
- Download the files listed above.
- Install the K12LTSP 2.0.x distribution. When prompted in the Package Group Selection portion of the installation add "Software Development" as one of the selections (helps to have these packages installed for adding OpenMosix).
- Copy the downloaded OpenMosix & LTSP files to your LTSP server.
- Navigate to the folder containing the downloads for the project: > cd /your-download-folder/
- Install the OpenMosix kernel to the server via the command: > rpm -ivh openmosix-kernel-2.4.16-openmosix5.i686.rpm. This adds the kernel to the grub bootloader as well, or a link "vmlinuz-openmosix" that can be used for lilo (just add it to lilo.conf).
- Install the Userland tools "openmosix-tools-0.1.3-6.i386.rpm" via the command: > rpm -ivh openmosix-tools-0.1.3-6.i386.rpm
- Install the OpenMosix kernel source "openmosix-kernel-2.4.16-openmosix5.src.rpm" via the command: > rpm -ivh openmosix-kernel-2.4.16-openmosix5.src.rpm
OR
- Download "openMosix2.4.16-4.gz" and a copy of linux-2.4.16.tar.bz2 from The Linux Kernel Archives
- > cp /your-download-folder/linux-2.4.16.tar.bz2 /usr/src
or, if you installed the openmosix-kernel-2.4.16-openmosix5.src.rpm then run:
> cp /usr/src/redhat/SOURCES/linux-2.4.16.tar.bz2 /usr/src/
- > cd /usr/src
- > tar xjvf linux-2.4.16.tar.bz2
- > mv /usr/src/linux /usr/src/linux-2.4.16-om5
- > cd /usr/src/linux-2.4.16-om5
- > mcedit /usr/src/linux-2.4.16-om5/Makefile
- Set "EXTRAVERSION = -om5" (without the quotation marks)
- F10 to save & quit.
- > cp/your-download-folder/openMosix2.4.16-4.gz /usr/src/linux-2.4.16-om5/
or, if you installed the openmosix-kernel-2.4.16-openmosix5.src.rpm then run:
> cp /usr/src/redhat/SOURCES/openMosix2.4.16-4.gz /usr/src/linux-2.4.16-om5/
- > gunzip /usr/src/linux-2.4.16-om5/openMosix2.4.16-4.gz
- > patch -Np1 < openMosix2.4.16-4
- > make mrproper
- > cp /opt/ltsp/kernel-configs/config.2.4.9-ltsp-5 /usr/src/linux-2.4.16-om5/.config
- > make oldconfig (or menuconfig if you prefer)
- Enable the OpenMosix options
a)Yes to process migration support, Kernel Diagnostics, Direct File-System Access, OpenMosix File-System.
b) No to Kernel Debugger.
c) Defaults for remaining items.
- > make dep
- > make bzImage
- > make modules
- > make modules_install
- > cp -R /lib/modules/2.4.16-om5 /opt/ltsp/i386/lib/modules/
- If you haven't already done so, download "ltsp_initrd_kit-3.0.1-i386.tgz" from the LTSP Project File List, if available or obtain it from the ltsp_cd.iso, or here. (ltsp_initrd_3.0.3-i386.tgz will not work for this example!)
- > cp /your-download-folder/ltsp_initrd_kit-3.0.1-i386.tgz /usr/src/
- > cd /usr/src
- > tar xzvf ltsp_initrd_kit-3.0.1-i386.tgz
- > cd /usr/src/ltsp_initrd_kit
- > mcedit buildk
- Scroll to the bottom and comment out the "prepare_kernel" lines with a preceeding "#" (without the quotation marks)
- Add the following line "prepare_kernel /usr/src/linux-2.4.16-om5 2.4.16-om5" (without quotation marks)
- F10 to save & quit
- > ./buildk
- > cd /tftpboot/lts/
- > ls
- Your new OpenMosix client kernel should appear as "vmlinuz-2.4.16-om5"
- If you wish, create a symbolic link to the OpenMosix kernel for easier upgrading in the future (you won't have to edit the client filename line in dhcpd.conf later)
- > ln -s vmlinuz-2.4.16-om5 vmlinuz-openmosix
- > cd /etc
- > mcedit dhcpd.conf
- Edit the "filename..." line using one of the two lines below depending on
whether or not you created the symbolic link:
a) filename "/lts/vmlinuz-openmosix";
b) filename "/lts/vmlinuz-2.4.16-om5";
- Add the following line (all on one line) in dhcpd.conf (above the trick from Peter comment):
option host-name = concat( "ws" , binary-to-ascii( 10, 8, "", substring( reverse( 1, leased-address), 0, 1)));
- F10 to save & exit
- > service dhcpd restart
- Copy & edit other files as indicated in Richard's How-To "Mosix Cluster with Diskless Nodes". Specifically, we are concerned with the following:
- Create & save the mosix.map file:
- > cd /etc
- > mcedit mosix.map
- Enter the following text for mosix.map:
1 192.168.0.254 1
2 192.168.0.1 253
- F10 to Save & Exit
- > cp /etc/mosix.map /opt/ltsp/i386/etc/ (Note: exact copies of mosix.map should be maintained in both locations.)
- Copy the userland tools & other files for the LTSP clients:
- > cp /sbin/setpe /opt/ltsp/i386/sbin/
- > cp /bin/mosrun /opt/ltsp/i386/bin/
- > cp /usr/bin/mosmon /opt/ltsp/i386/usr/bin/
- > cp /usr/bin/mosctl /opt/ltsp/i386/usr/bin/
- > cp /usr/bin/migrate /opt/ltsp/i386/usr/bin/
- ***NOTE***
See Appendix A below for a list of other Userland tools that you may wish to copy for Linux Terminal Client node use. Simply copy them from their existing locations to the corresponding /opt/ltsp/i386/ locations as you just did for the previous files. You may need to make some or all of them available to the Linux Terminal nodes to effectively utilize the cluster.
- > cp /bin/touch /opt/ltsp/i386/bin/
- > rm /opt/ltsp/i386/etc/hosts
- > cp /etc/hosts /opt/ltsp/i386/etc/
- > cp /etc/rc.d/init.d/openmosix /opt/ltsp/i386/etc/rc.openmosix
- > mkdir /opt/ltsp/i386/mfs
- > cd /opt/ltsp/i386/etc/
- > mcedit fstab
- Add the following line to fstab:
none /mfs mfs dfsa=1 0 0
- F10 to Save & Exit
- > mcedit /opt/ltsp/i386/etc/rc.local
- Add the following lines to the end of rc.local:
# OpenMosix startup section
# we don't want any terminal processes to migrate
echo 1 > /proc/mosix/admin/lstay
# start mosix
/etc/rc.openmosix start
# mount mfs filesystem. doesn't work when done earlier
mount /mfs
# End OpenMosix startup section
- F10 to Save and exit
- Once completed, and you verify that everything boots as it should, use the command > mosmon -t to check your work, hopefully you will see all of your OpenMosix nodes in action!
Some other quick notes once you have everything running properly and you're experienced with K12LTSP:
- If you don't need a GUI on your LT/OpenMosix nodes edit the dhcpd.conf & lts.conf files to set the system(s) to "RUNLEVEL = 3".
- The MOSIX site lists a couple of ways to test your cluster with awk that will work just as well with OpenMosix. See the following:
This document (Rev. 3.3) corrects errors found in the previous version, particularly the fact that you cannot use ltsp_initrd_kit-3.0.3-i386.tgz in the installation as described above, you need 3.0.1. (If you have success with 3.0.3 please let me know the details.) Hopefully, with these corrections, this How-To gives us a complete step-by-step process for easily implementing OpenMosix on a K12LTSP server. Richard Camp is in the process of producing more complete documentation, scripts & other items that he will make available at his website when ready.
Thanks to everyone out there that develops & supports Open Source software. Particular thanks to the folks at the Linux Terminal Server Project, the K-12Linux Project, and the openMosix Project for making this possible. And, finally special thanks goes out to Richard Camp, B. Smith, and T. Rhodes for their valuable feedback as this document has developed.
I hope with this revision that you will have success in implementing OpenMosix with LTSP. Amazing to me is the fact that while I can do this install "in my sleep" it has been somewhat difficult to get it right on "paper". My apologies to those who may have been become frustrated in previous attempts at implementing this due to my errors. This really does work! Don't give up!
Happy clustering!
James Jensen
jmsjnsn@nojunkmailplease.linuxfreemail.com
Rev. 3.3
Friday, April 19, 2002
Appendix A
OpenMosix Userland Tools Chart
Summary of each file is based on it's manpage entry.
Userland File |
Summary |
/bin/mosrun |
Run a command with particular node-allocation preferences. |
/bin/mps |
Report multicomputer process status. |
/etc/mosix.map |
OpenMosix node map file. |
/etc/rc.d/init.d/openmosix |
OpenMosix init script. |
/sbin/setpe |
MOSIX node configuration. |
/usr/bin/cpujob |
mosrun frontend script: Runs a command advising MOSIX that it is a computation intensive program. |
/usr/bin/fastdecay |
mosrun frontend script: runs a command advising MOSIX to apply fast statistics-decaying. |
/usr/bin/fixvfork.sh |
|
/usr/bin/iojob |
mosrun frontend script: runs a command, advising MOSIX that it is an I/O intensive program. |
/usr/bin/migrate |
Request migration of a particular process on MOSIX. |
/usr/bin/mosctl |
MOSIX system administrator's tools. |
/usr/bin/mosmon |
MOSIX load monitor |
/usr/bin/mtop |
Display multicomputer top CPU processes. |
/usr/bin/nodecay |
mosrun frontend script: runs a command, advising MOSIX to not decay the program's statistics. |
/usr/bin/nomig |
mosrun frontend script: runs a command with a node-lock. |
/usr/bin/runhome |
Runs a command locked at the home node. |
/usr/bin/runon |
mosrun frontend script: runs a command locked on a particular MOSIX node. |
/usr/bin/slowdecay |
mosrun frontend script: runs a command, advising MOSIX to apply slow statistics-decaying. |
/usr/include/libmosix.h |
|
/usr/include/mos.h |
|
/usr/lib/fixvfork.so |
|
/usr/lib/libmos.a |
|
/usr/lib/libmos.so |
|