Home page of this blog

Thursday, August 11, 2011

Custom kernel 3.0.1 ubuntu 64 bit kernel optimized for i3, i5 and i7 processors


For a nice pdf of document of the same click here
Download links are provided at the bottom of this article along with SHA256SUMS

Again thanks to Linux Mint 11, I am enjoying it
Introduction

I faced the following with earlier kernel builds

  1. Symlink errors while installing kernel debs in another system where the kernel build did not happen and lots of comments in the blog also reported the same
  2. DKMS modules did not work in different systems where the kernel build did not happen
  3. Cross compiling 32 bit kernels from my 64 bit PC was cumbersome

This kernel tries to address the above problems and give a kernel as similar to Ubuntu but with optimization. Towards the end of this document I have explained a step by step way of building the kernel Ubuntu way . Also I have made the same as a shell script which can be rerun again and again to build kernels Ubuntu way

This document is divided into two major sections, the first one is enough info about deb files and the second one is about how to build them

Info about the deb files

Requirements


This package is created in Linux Mint 11 and should install fine in all versions of Linux Mint 11 (as well Mint 10) and Ubuntu 11.04 (as well 10.10) (should as well in debian squeeze but not sure about dkms).

The build is done with march=core2 and mtune=native

This ensures the build will run on all core2 and above but will run with optimum speed on i3, i5 and i7 processors, so

The build is for 64bit i3,i5 and i7 processors

The following are the configuration settings


  1. Processor is selected as CORE2
  2. Makefile and Makefile_32.cpu modified to reflect march settings to core2 and mtune settings to native (my processor is i7 920)
  3. Configuration of oneiric kernel is taken as base, http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.0.1-oneiric/
  4. Kernel Timer Frequency is 300 HZ

How to install


To install

sudo dpkg -i linux-headers-3.0.1-030001_3.0.1-030001.201108060905_all.deb linux-headers-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb linux-image-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb


After this reboot into the newly installed kernel from grub (use update-initramfs in case of 10.04 or older if it is not building initramfs)

Kernel Source


I downloaded the kernel source tarball from kernel.org

Version: 3.0.1



Ubuntu Oneiric Patches




How to build yourself


This kernel follows Ubuntu way of building kernel and has all ubuntu oneiric patches

The following is a documentation of building it the Ubuntu way, I used my article about Ubuntu Maverick kernel compilation as a base for this, with some modifications. Here is a step by step of the same (and this explains the shell script commands which I created for my own reference)

Install dependencies for building Kernel




Here are the apt-get commands to install the dependencies required for building kernel the Ubuntu way
sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge libncurses5 binutils-dev libelf-dev libdw-dev libnewt-dev libncurses5-dev
sudo apt-get --no-install-recommends install asciidoc xmlto


Get the Kernel source and Ubuntu Patches



Create a directory for the building kernel source


#Enter into Home directory of the user
cd ~

#create a kernel directory if it is not there
mkdir -p kernel3.0.1

#enter into kernel directory
cd kernel3.0.1

The following wget commands is used to fetch the Kernel source and Ubuntu Patches


#get the kernel sources and ubuntu patches
#kernel 3.0.1

#Ubuntu Oneiric Patches from kernel ppa



Create a soft link of extracted kernel folder




#unpack linux kernel 3.0.1
tar xjf linux-3.0.1.tar.bz2



#create a soft link for the kernel directory
ln -s linux-3.0.1 linux



#enter into unpacked directory
cd linux



Install Ubuntu patches we downloaded earlier




#install the ubuntu/debian patches
patch -p1 < ../0001-base-packaging.patch
patch -p1 < ../0002-debian-changelog.patch
patch -p1 < ../0003-default-configs.patch



Create and configure a new kernel flavor, here I create it as i7




After the patches are applied, we come here to create a new flavor



Before this we need to give execute permissions to debian scripts and debian rules



Give execute permissions to debian/rules and debian/scripts




#make debian/rules as executable
chmod -Rv +x debian/rules
#make debian/scripts directory executable
chmod -Rv +x debian/scripts/



Copy existing generic flavor (or use existing flavor to create new flavor, here i7)




#create i7 flavor
cp debian.master/config/amd64/config.flavour.generic debian.master/config/amd64/config.flavour.i7



Configure the newly created flavor i7




#clean
fakeroot debian/rules clean
fakeroot debian/rules updateconfigs



When calling editconfigs, say n to avoid editing any other flavor except the created flavor, here i7. When a prompt asks us to edit i7 (or any other flavor you created) give y, this will open menuconfig of the kernel, feel free to change kernel configurations to hearts wish and dont forget to save while exiting the menuconfig screen. This is the place where I gave CORE2 as the processor type and changed kernel timer frequency to 300 HZ (or whichever you prefer)


#give y only to i7 flavor, leave the rest untouched
fakeroot debian/rules editconfigs



Edit debian related abi information for newly created flavor, here i7




#copy the abi entries for i7 flavor and add i7 entries in vars and amd64.mk
cp debian.master/abi/3.0.0-7.9/amd64/generic debian.master/abi/3.0.0-7.9/amd64/i7
cp debian.master/abi/3.0.0-7.9/amd64/generic.modules debian.master/abi/3.0.0-7.9/amd64/i7.modules
sed -i s/getall\ amd64\ generic\ server\ virtual/getall\ amd64\ generic\ server\ virtual\ i7/g debian.master/etc/getabis
sed -i s/\=\ generic\ server\ virtual/\=\ generic\ server\ virtual\ i7/g debian.master/rules.d/amd64.mk
cp debian.master/control.d/vars.generic debian.master/control.d/vars.i7
sed -i s/\"Generic\"/\"core\ i7\"/g debian.master/control.d/vars.i7



Edit architecture dependent Makefiles to finetune gcc flags for compilation




Architecture dependent Makefiles for kernel compilation will reside in arch//Makefile
For x86 architecture, the following are the two Makefiles
arch/x86/Makefile and arch/x86/Makefile_32.cpu



#edit arch/x86/Makefile and arch/x86/Makefile_32.cpu if you want to tweak the march and mtune params of gcc
vi arch/x86/Makefile
vi arch/x86/Makefile_32.cpu


After opening the Makefile search for CORE2 and change march=core2 and mtune=native



In Makefile the following will be the change



cflags-$(CONFIG_MCORE2) += \
                $(call cc-option,-march=core2,$(call cc-option,-mtune=native))


In Makefile_32.cpu the following will be the change

cflags-$(CONFIG_MCORE2) += -march=core2 $(call tune,native)



Build




Clean before building




#clean
fakeroot debian/rules clean



Build source and headers debian package which is independent of architecture or flavor, hence indep




#build headers/source deb files
skipabi=true skipmodule=true fakeroot debian/rules binary-indep



Set the CONCURRENCY_LEVEL for concurrent builds which will speed up the kernel compilation tremendously




#set the CONCURRENCY_LEVEL to the number of processor cores in system to do parallel compilation and thus speed up the kernel compilation
#`cat /proc/cpuinfo | grep "cpu cores" | head -1 | cut -d":" -f2 | cut -c2-` is used to get the number of cores
export CONCURRENCY_LEVEL=`cat /proc/cpuinfo | grep "cpu cores" | head -1 | cut -d":" -f2 | cut -c2-`



cat /proc/cpuinfo gives all info about cpu in the system
grepping for “cpu cores” gives multiple lines one fore each core
head -1 takes the first line of output
cut -d”:” -f2 uses : as delimiter and gives the second field, which is the number of cores
cut -c2- skips leading space


export makes this variable exported for the coming lines



To test the CONCURRENCY_LEVEL, use echo $CONCURRENCY_LEVEL



Build the flavor




It is nice to time a build, just prefix time command before any other command, here the kernel build command. While calling debian/rules, we are invoking the newly created flavor which is binary-i7 in this case



#build the actual kernel flavor i7 (and time it)
time skipabi=true skipmodule=true fakeroot debian/rules binary-i7



Wait for some time, have a coffee, relax and wait for your system to finish build. My system took 20 minutes to finish the build

Enjoy

Dropbox Download Links


http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/BuildKernel3.0.1.sh
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/config-3.0.1-030001-i7
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/linux-headers-3.0.1-030001_3.0.1-030001.201108060905_all.deb
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/linux-headers-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/linux-image-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/Readme.odt
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/Readme.pdf
http://dl.dropbox.com/u/22900905/MyKernelBuilds/3.0.1-030001-i7/SHA256SUMS

SHA256SUMS



1291a52bbcbb74f6b044171ae10d4efe399cff2a615ac03ff8b4ff2c64983e53  BuildKernel3.0.1.sh
202e4bef97a05f1df4c93fb6821036018f1a4ccd25745eb16f21aca02c8e6be8  config-3.0.1-030001-i7
7ef4a2f7e22335498535e980bfed3afe58999922120ff2ae735970994d1d1318  linux-headers-3.0.1-030001_3.0.1-030001.201108060905_all.deb
05ba8bc60980868ac299a473c832768e4d185b3764e00b9012527578449afc71  linux-headers-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb
48bb53f0e50b6da024f40189bf859d686daad6801b76464829737a69ac93583e  linux-image-3.0.1-030001-i7_3.0.1-030001.201108060905_amd64.deb
f0cd271d2d4413a29b08963a9b9d99ef966513ecf815def2fed2c63a3638eb8b  Readme.odt
e56ef8600f25f99ea6707d87b3a48f793327111d6579031779acc54a8e61ea7c  Readme.pdf

28 comments:

  1. Kernel 3.0.1 did not build dkms modules for 270 series nvidia drivers, I updated nvidia drivers to 280 series using the following x updates ppa

    sudo apt-add-repository ppa:ubuntu-x-swat/x-updates

    sudo apt-get update

    After this it built dkms modules for kernel 3 properly and I could boot with nvidia latest graphics drivers

    ReplyDelete
  2. After doing sudo apt-get update, install the updated nvidia drivers from update manager (here from mint update manager)

    ReplyDelete
  3. Hi,

    This kernel compilation does ott work for me. Only black screen appears and ALT - SYSRQ + B solved it ;-)

    vbox 4.1 did not compiled properly within linux-header.

    let me know how i can investigate why it is not starting properly?

    Martin

    ReplyDelete
  4. Lots of old sources or packages not ready for kernel 3.0 will fail. I faced a similar thing when broadcom did not work. Then I downloaded latest broadcom driver from packages.ubuntu.com in oneiric section (http://packages.ubuntu.com/oneiric/i386/bcmwl-kernel-source/download)

    Maybe you need to download virtualbox 4.1 from oneiric repository and try. Also regarding black screen is due to nvidia or proprietary graphics driver not able to build with 3.0

    Get the updated proprietary driver from oneiric before installing kernel 3.0

    ReplyDelete
  5. Old packages may expect kernel version to look like 2.6.x whereas new kernel is 3.0 or 3.0.1 and it may break their dependency check. The reason Fedora 15 opted to use 2.6.40 as its kernel version instead of 3.0 though there is no kernel 2.6.40 out there

    ReplyDelete
  6. Hi,

    My pre-previous post (last day) has been lost somewhere again :-)

    There is more information about installation of this kernel:

    I am running now on ubuntu kernel 3.0.0-8 (3 independent instances) (and one ubuntu 3.0.0-7) with catalyst 11.7 without any problem (one frozen with 'stress' test).
    Catalyst 11.7 has been properly compiled with your kernel and vbox 4.1 has been compiled properly partly. I mean with "linux-image..." compilation succeeded but NOT with "linux-headers...". So, the problem should not be the proprietary graphic driver and virtualbox does not mind me now even if I am strongly dependent on it.

    Overview within your compiled kernel:
    -------------------------------------
    proprietary catalyst 11.7 graphic driver - compiled OK !!!
    virtualbox 4.1 - "headers..." compilation failed
    virtualbox 4.1 - "image..." compilation ok

    but what is the main problem is that system is not going to start (after restarting), even if catalyst graphic driver has been compiled properly. So, it should not be problem within proprietary graphic driver - Please let me know if I am wrong and where?

    And I do not know, if there are some log/s files to investigate what went wrong?

    My intention is now to investigate and properly make compilation of kernel within steps described here. But if this compiled kernel is not working for me I would like to know how to compile the kernel to make it work

    I have no clue where to look to start investigate to properly compile kernel.

    If you have any suggestions , please I really much appreciate it to share knowledge.

    Thanks,
    Martin

    ReplyDelete
  7. Hi again,

    I forgot to ask. Should I install new gcc/g++ compilators in ubuntu 11.04 to follow these steps for compilation of this kernel?
    The original compilator is: gcc/g++ (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2

    Thanks for answer,
    Martin

    ReplyDelete
  8. Couple of minutes ago I sent comment - it had appeared and disappeared again ???

    ReplyDelete
  9. No buddy, it was in spam, I checked and ticked it as not spam.

    I used Linux Mint 11 and default gcc 4.5.2 (as my ubuntu went down due to no graphics and maybe btrfs)

    If you are not going to use your compiled kernel away from your box, use make-kpkg way to compile (which is not the ubuntu way)

    Can you remove catalyst and add it again. Sometimes nvidia drivers also does the same. Regarding dkms failures, you can see them in /var/lib/dkms/ folder

    You should see catalyst folder and inside it some log file of your make file will be there

    If you find errors, please share it, I can see if I missed something while building kernel

    ReplyDelete
  10. According to kernel ppa documentation here

    installing kernel is done by

    sudo dpkg -i *.deb

    Maybe the order in which the kernel packages installed is not correct, maybe, I donno just guessing

    Maybe if you try like above it may build vbox

    ReplyDelete
  11. Hi,

    I have big trouble to post something here. It is not going through :-(

    -- the order of installation is ok.
    -- I cannot put logs here - it is refusing to do it.
    -- Logs, for catalyst 11.7 and vbox 4.1 are ok. No errors.

    I start to believe that there has to be something important in config within compilation kernel to make it work.

    I have no clue where and what to investigate?

    If you have any idea what logs could be important to find what went wrong, please let me know, maybe some log when it is starting the kernel modules or something.

    Martin

    ReplyDelete
  12. Can you upload logs and refer me the links to the uploaded link (like dropbox public link)

    It is using the oneiric kernel with only 2 changes

    Processor type changed to CORE2 and HZ to 300

    I can try building with a configuration, say el6 configuration and upload for you to try

    You can use

    dmesg

    command to read the logs

    All logs will be stored in /var/log/messages

    which requires su permission to open and read

    Also, did you verify the SHA256SUM after downloading, maybe a corrupted download ?

    ReplyDelete
  13. Hi,

    log files are here:

    http://dl.dropbox.com/u/38489690/UBUEXT4_1/var/lib/dkms/fglrx/8.872/3.0.1-030001-i7/x86_64/log/make.log
    http://dl.dropbox.com/u/38489690/UBUEXT4_1/var/lib/dkms/vboxhost/4.1.0/3.0.1-030001-i7/x86_64/log/make.log

    (I installed this kernel on my old laptop with core 2 processor and nvidia proprietary driver and it works.)

    I also installed tree different versions of oneiric kernel and none of them work.
    Maybe it could be the oneiric patches which causes not to work.

    I have a looked into logs by this command:
    egrep -i '(3.0.1-030001-i7)' /var/log/*
    but I did not find anything wrong.
    and

    egrep -i '(fail|denied|segfault|segmentation|reject|oops|warn)' /var/log/*

    but I did not find anything relevant as far as I understand what informations meant.

    dmesg found errors but related to unsuccessfully mount my btrfs partition.

    I do not know what is el6 configuration but I am willing to do anything to reveal why oneiric kernel compilations does not work on my computer.

    I have checked SHA256SUM for all three kernel files (even after installation) and they are fine.

    thanks for any help,
    Martin

    ReplyDelete
  14. Hi,

    May last post has been lost in your spam folder probably :-)

    Martin

    ReplyDelete
  15. Yeah, checked it was in spam, ticked as not spam

    ReplyDelete
  16. Sankaran,

    Looking ar "cpu cores" gives the number of cores on the current processor, however most systems have more the one physical processor.

    You can call 'nproc' to get the number of cores on the system or via 'grep -c processor /proc/cpuinfo'

    Alex.

    ReplyDelete
  17. Very strange with procinfo! All i7 cpu's have 4 physical and 4 logical (hyperthreading) cores. Why then does cpuinfo and grepping, output only 4?

    I think that exporting concurrency for 4 cpu's really slows things down (it's still fast I think but it would be better by exporting 7). I would pick 7, leaving one left over for the system. Any thoughts on this? Appreciate any feedback. Thanks.

    ReplyDelete
  18. Core I7 microprocessors actually have 8 cores. 4 physical and 4 logical. Will what is suggested here for the kernel compile, actually use all 8? I would prefer 7, leaving one hyperthread left for the system, and the rest for kernel compilation. Any thoughts on this?
    Thanks in advance.

    BTW, this blog is probably THE only coherent, step by step guide to do such a thing! The "kernelcheck" utility comes close, but it's not nearly as thorough, with arch, mtune, flavors and the like. absolutely excellent. Thanks. Compiling as I type this :) BTW, this is a core i7-950 first-gen. Very, very fast!

    ReplyDelete
  19. Just wondering, where's the shell script you said you made? Can't find it here within these instructions?

    Thanks in advance.

    Marsh

    ReplyDelete
  20. All the scripts and the built deb are in the bottom of the article. Hope I did not miss any thing in haste

    ReplyDelete
  21. Someone commented about using the following to get the count of cpu, here it is, use it to get the count (usually i use 7 on my i7 like you bingo, one for my browsing habits)

    cat /proc/cpuinfo | grep -c "cpu cores"

    ReplyDelete
  22. Recent find from gentoo, use -pipe in compiling to speed up if you have lots of RAM, how great

    ReplyDelete
  23. I know about that and did that as you instructed but my machine still only reports a 4 cpu machine, when in fact, the i7-950 cpu is a hyperthreaded chip, making 4 physical and 4 logical cores. Interesting that cpuinfo would return 4, but the system monitor reports 8!!! Right?

    ReplyDelete
  24. Thanks for the script. I can tweak the hell out of my kernel, stripping it of all the junk, like wireless cards and Adaptec SCSI cards. Eggcellent. Thanks again Sankaran. BTW, where did you learn how to do all of this? I'd love to really master the Linux command shell and system, any hints at what books I should study? Thanks in advance.

    ReplyDelete
  25. Actually I am a newbie in kernel compiling and not mastered anything.

    There is no one book. Read as many as possible, as for me, I am a heavy surfer and tried various ways of building kernel in Fedora, Debian, Arch, Gentoo and now Ubuntu

    The best resource would be Arch Linux or Gentoo

    ReplyDelete
  26. Gentoo is meant for building everything from source, I have tried Gentoo four times and failed three times. It is fun if you can spend a lots of time. System will run extremely fast, but even to get firefox, it will start building, so I moved away from Gentoo to Ubuntu and Windows

    The only other linux which comes close or even better than Ubuntu is Archlinux. There everything is provided as binary, but building is just a step away and it is very easy. Here again the problem is internet speed in India is poor and I dont/cant wait get every source with the horrible broadband speed.

    I set up Fedora 8 or Fedora 7 with all the rebuilt kernels and one day when I was sleepy, I booted into live cd and was testing, I erased 500 GB fedora partition thinking it is my pendrive which was 4 GB. After that I lost fedora completely and did not get inspiration to go again with Fedora as setting Fedora is not as easy as Ubuntu. lol

    Debian is as good or better than Ubuntu, but I am impatient to wait for new software. Linux mint I love very much except the fact it brings me nausea of green color everywhere

    ReplyDelete
  27. you are awsome man i just did a newb install and it already runs great on ubuntu 11.04

    ReplyDelete