gentoo

I just upgraded my mythtv backend's gcc from 3.4.6 to 4.1.2. It's still compiling all of world now and will be have been about 24 hours when it's all done. Then I did an emerge --sync and now there is a gcc 4.3! Arghh.

I've always touted gentoo's advantage of allowed me to run a mixed system. For a long time I was running an old version of mysql, and old kernel, and old glibc, and finally an old gcc. I would pull the odd update of certain packages if I felt like it. But the shear magnitude of the output when I would run emerge -uva world got on my nerves. I eventually caved and decided to go back to keeping an updated system. I had originally stopped updating most packages because I wanted my system to be as stable as possible (because it my MythTV machine and it records TV shows for me). At that time I was using it as a backend and as a frontend, but now I'm just using it as a backend which is a bit simpler (less room for breakage) and I'm a bit more careful now when doing upgrades. If any important packages are being updated I make sure to update them then reboot to make sure my system still boots. I also try not to fiddle with anything else. Anyways, in the case of my MythTV box I'm not longer taking advantage of Gentoo's biggest advantage, running system with mixed packages.

So I am on a Kubuntu edgy laptop at work and I am trying to get WPA wireless networking working before the upcoming demo that we are putting on. Since I have no idea what kind of networking is set up at both demo locations I am making sure all options work. WPA is the last one to take care of.

I finally found a front-end for wpa_supplicant that actually works. It's called kwlan and it is currently at version 0.5.8. Ubuntu edgy only has version 0.4.7. I really want 0.5.8 because 0.4.7 has some glaring bugs. Feisty actually has 0.5.8. There is no way I am upgrading this entire box to feisty. Because of ubuntu's binary nature, in order to get kwlan-0.5.8 installed I might actually have to bump a lot of packages to feisty just to get it working. Even if I just want kwlan-0.5.8, how to I get it? I'll have to add some sources to /etc/apt/sources.list and possibly jump through a few more hoops to be able to install one package from feisty without apt-get upgrade wanting to upgrade everything to feisty. If kwlan-0.5.8 wasn't available in feisty, my only option would be to make my own deb package. Good luck with that (I maintained a debian package once, called camserv a long time go. Way harder than ebuilds.)

In gentoo it is as simple as coping the ebuild to my /usr/local/portage folder, renaming it to kwlan-0.5.8.ebuild, creating the digest and then emerging the 0.5.8 version. Assuming the upstream developer did not make massive changes to the build process, simply renaming the ebuild name from kwlan-0.4.7.ebuild to kwlan-0.5.8.ebuild should work. Like a lot of things, it's a hell of a lot easier in Gentoo.

Sure that's just one reason why Gentoo is better than Ubuntu. There are many others. Just look at Ubuntu Guide. A whole friggin' guide to install a lot of things that can be easily installed in Gentoo (mplayer, acrobat, java, flash, etc...). Not to mention that eix rocks compared to anything apt has, and paludis is awesome. Every time I think Ubuntu is better than gentoo, I end up taking that back soon after.

I just bought two new 320G hard drives last weekend and they are finally up and running with my same old Gentoo Linux OS. I wanted to put them in a RAID1 configuration and this was my first experience with software RAID.

The first time I did it I just made took one third of the new drive (about 120G) and made it a RAID1, then copied the old hard drive over. I purposefully chose not to use LVM because I have used it before and although it is extremely handy I have always been worried about the difficulty of recovering data from a bricked drive when the data is scattered all about. After copying data over to this one big partition I realized that it wasn't so easy to resize a RAID. I also read about how bad it was to have everything all on one partition (/home, /, /var, etc...). So I did a complete 180 and decided to use LVM.

Here is my partition layout:

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1         1       13      104422   fd  Linux raid autodetect (/boot, in /dev/md1)
/dev/sda2        14      622     4891792+  fd  Linux raid autodetect (empty, for Xen later, in /dev/md2)
/dev/sda3       623      866     1959930   fd  Linux raid autodetect (/, in /dev/md3)
/dev/sda4       867    38913   305612527+   5  Extended
/dev/sda5       867     5730    39070079+  fd  Linux raid autodetect (for LVM, in /dev/md5)
/dev/sda6      5731    10594    39070079+  fd  Linux raid autodetect (for LVM, in /dev/md6)
/dev/sda7     10595    15458    39070079+  fd  Linux raid autodetect (for LVM, in /dev/md7)
/dev/sda8     15459    20322    39070079+  fd  Linux raid autodetect (for LVM, in /dev/md8)
...
lots of free space
...
/dev/sda9     38670    38913     51657007   82 Linux swap / Solaris (in /dev/md9)

/dev/sdb looks exactly the same of course, which you can do very easily with:

sfdisk -d /dev/sda | sfdisk /dev/sdb

Basically /dev/sda5 onward are all 40G physical partitions. I'll RAID1 them with the corresponding partitions on /dev/sdb. I have also RAID'ed the /boot partition and the root partition, and the sdb2/sda3 partitions for future use. So far, I have 7 raids:

-(~:$)-> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md8 : active raid1 sdb8[1] sda8[0]
      39069952 blocks [2/2] [UU]

md7 : active raid1 sdb7[1] sda7[0]
      39069952 blocks [2/2] [UU]

md1 : active raid1 sdb1[1] sda1[0]
      104320 blocks [2/2] [UU]

md2 : active raid1 sdb2[1] sda2[0]
      4891712 blocks [2/2] [UU]

md3 : active raid1 sdb3[1] sda3[0]
      1959808 blocks [2/2] [UU]

md5 : active raid1 sdb5[1] sda5[0]
      39069952 blocks [2/2] [UU]

md6 : active raid1 sdb6[1] sda6[0]
      39069952 blocks [2/2] [UU]

unused devices: 

I then made md5, 6, 7, and 8 LVM physical volumes and then shoved into an LVM volume group using pvcreate, vgcreate, and vgextend:

# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/md5   vg   lvm2 a-   37.26G      0
  /dev/md6   vg   lvm2 a-   37.26G 528.00M
  /dev/md7   vg   lvm2 a-   37.26G      0
  /dev/md8   vg   lvm2 a-   37.26G  29.52G
# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg     4   5   0 wz--n- 149.03G 30.03Gsonata opt # vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg     4   5   0 wz--n- 149.03G 30.03G

Then I created some LVM logical volumes using lvcreate:

# lvs
  LV   VG   Attr   LSize  Origin Snap%  Move Log Copy%
  home vg   -wi-ao 80.00G
  opt  vg   -wi-ao  2.00G
  tmp  vg   -wi-ao  2.00G
  usr  vg   -wi-ao 20.00G
  var  vg   -wi-ao 15.00G

I keep /home, /opt, /tmp, /usr and /var separate. That's why my root (/) partition on /dev/md3 only needs to be 2G.

The reason I made a bunch of RAID's and put them in the LVM rather than just one big RAID is because it means I haven't tied my entire drive down to anything. I can always move data off of one partition later using a few LVM commands and install another OS if I have to. Few people do this, but I think it is a good idea. On my computer at work, I did something similar. I split my hard drive into chunks of 40G and then threw into an LVM. Splitting your hard drive into multiple physical partitions is very useful.

The other thing I did was put the swap drive at the end of the hard drive. It is easy to make it bigger that way. So many people make one of their primary partitions at the beginning of the hard drive into a swap partition, which I can never understand. I almost always end up adding more RAM to my computer and thus needing to make my swap partition bigger. When it is at the beginning of the drive, it means you have to add a new one at the end, where you have free space. Putting my swap at the end of the drive in the first place means I don't have to have 2 swaps later. Given the complexity of my drives as it is, this swap thing is a minor concern.

I also plan on putting Xen on /dev/md2 later. I hope to be able to boot into an OS on /dev/md2, then start up the Gentoo Linux on /dev/md3. If I ever need to restart my Linux on /dev/md3 I can reboot it without turning off the computer. It would also allow me to run Windows XP under Xen rather than under VMWare as I do know. From what I hear Xen has better performance.

The main thing left for me to do is to buy another 320G drive and set up an rsync backup from the 320G RAID to the backup drive (probably in a USB adapter case). As most people probably know, RAID does not replace backups. RAID just protects you against hardware failure. It does not protect you against "rm -rf /". These drives were only $110 CAD each, so getting another backup drive is no big deal.

One other thing that I did differently than usual is that I made all my filesystems ext3, rather than reiserfs (as I normally do). One of the reasons is that I had far too many cases where my reiserfs partitions got screwed up. In fact even my old hard drive had a screwed up directory that was not fixable, unless I ran the reiserfsck program with the --rebuild-tree option, which is not a very safe option. It turns out the directory was not an important one. Anyways, I have never had a problem with ext2 or ext3 ever, so I went with those instead.

I switched to Paludis yesterday. For those who don't know what Paludis is, it's an alternative Gentoo package manager, or a Portage/emerge replacement. It has not all been smooth. I had to fight with it for a while, re-installing a bunch of packages that thought they depended on an installed package that no longer existed. I think this was because Portage left some cruft around that made it think that. The problems were mainly to do with kdelibs-3.4* (which a lot of installed packages thought they had a block on) and with guile (which moved to a new category but some packages weren't aware of the switch). Re-installing worked in most cases.

The other wrinkle was that the portage2paludis.bash script that I used according to the migration instructions didn't bring all my repositories over, including my /usr/local/portage and a few others. The second annoyance was that a few repositories didn't have profiles/repo_name and profiles/categories files in them (which paludis requires) and this causes a bunch of warnings about how paludis is "faking it." I finally got it to the point where I could do "paludis -ip world" and I got no warnings and no errors.

The only remaining problem is that the gentoo java packages are mostly unusable with paludis because none of the hooks in the java gentoo eclasses get installed into paludis' hooks system (and paludis doesn't call the hooks in the java eclasses). Apparently the Gentoo Java team was "abusing the [portage] hooks in [their] eclasses." They should be installing the hooks into the paludis hooks directories. Paludis hooks are similar to subversion's hooks system. They allow you to do a lot of fancy things.

Overall I am impressed with Paludis. Running "paludis -ip world" is a lot faster than "emerge -uva world" or "emerge -uvaDNt world" but more important than speed is that it was designed from scratch, without all of portage's warts and messy hard-to-change code. In a way, paludis is to portage what svn is to cvs. You can read all about the differences between Paludis and portage here.

One more thing happened yesterday, that was a little bit scary. I tried to reboot to try out a new kernel and I noticed the "reboot" and "shutdown" commands were no longer there. I rebooted anyways but hitting the reset button but the system wouldn't load, it stopped right after the kernel was done starting up. It turned out Paludis had uninstalled sysvinit when I ran "paludis --uninstall-unused" and I didn't notice. Reinstalling it fixed everything but I got really scared that the reboot and shutdown binaries were missing due to dataloss.

I am excited to be a part of Planet Larry. I don't write about Gentoo too often, but once in a while I might have something interesting to say about it. For those that are interested, my past articles on Gentoo, and my past articles on Linux.

[img_assist|nid=161|title=|desc=|link=url,http://|align=left|width=100|height=100]Beryl is now in Gentoo's official ebuild repository, portage. There are instructions on installing it for Gentoo as well as how to set up your nvidia graphics card if you have one. The beautiful thing about having a nVidia graphics card is that with the latest nVidia drivers (build 9625) you no longer need to install XGL. You just add the AddARGBGLXVisuals to your xorg.conf file and install the Beryl window manager and you get snazzy effects right out of the box. The other great thing is that you don't need to edit any other files on your computer. Just launch KDE for example, as usual, then run "beryl-manager" from a konsole window. You can change your window manager to beryl from the taskbar icon that shows up, restart beryl if it crashes, or switch back to kwin. You can also use emerald (the window-decorator) and change your emerald theme from the taskbar icon.

What is Beryl for? Well a lot of it is just eye-candy, like the spinning cube desktop for example. But some of it is really useful. In my opinion the most useful feature is the thing that imitates the Mac OS X Expose feature, that shows every window on your desktop and allows you to switch to the one you click on. You can invoke it with the F9 or F8 keys. The reveal desktop feature is pretty nice as well (activated with the F6 key).

Why ssmtp should be removed from gentoo:

• ssmtp is no longer maintained upstream
• Even its successor, bssmtp has been abandoned. The writer of bssmtp and maintainer of ssmtp said himself "I have now switched back to using Postfix on my laptop."

According to this page by the maintainer, the two advantages of ssmtp are:

• it is light (doesn't matter if it's crap)
• it is easy to configure (it appears that way at first, but it isn't necessarily so, if you check out the gentoo forums)

He continues by saying that it has loads of problems:

• it doesn't fail in a nice way (this is huge, I was running into problem with ssmtp not obeying the rules in /etc/ssmtp/revaliases; the output in syslog looked the same as a succesful send, but the mail wasn't sent successfully and there is no debugging output or logging whatsoever. Compare that to postfix's excellent logging.)
• it doesn't do local delivery
• it doesn't queue messages.
• there is no nice support for several network environments.

Let me recount my recent experiences. I wanted a simple outgoing mail server. Something that I would allow me to run the following:

echo test |mail -s "test mail sent to external" <a href="mailto:me@gmail.com">me@gmail.com</a>
echo test |mail -s "test mail sent to david" david
echo test |mail -s "test mail sent to root" root

and have all 3 message sent to <a href="mailto:me@gmail.com">me@gmail.com</a>. With ssmtp I never got this working after many, many hours of trying. In the end, the last two commands never worked, /etc/ssmtp/revaliases was not working for some reason. Maybe I goofed, but so what. Getting postfix to work was so damn easy. All I had to do with postfix was change relayhost in /etc/postfix/main.cf like so:

relayhost = smtp.vc.shawcable.net

and add a .forward files in the home directory of the root and david users, like so:

echo "<a href="mailto:me@gmail.com">me@gmail.com</a>" > /root/.forward
echo "<a href="mailto:me@gmail.com">me@gmail.com</a>" > /home/david/.forward

then,

/etc/init.d/postfix start
rc-update add postfix default

Presto! It all works! I am sure that there is an easy way to make local mail to all other users on my system get forwarded to gmail as well. Additionally, if I wanted to use gmail's smtp servers instead of shaw's I'm sure that would also be easy.

Let's look at smtp's setup for a bit.

In ssmtp.conf there is a "root=" options. There you can set "who gets all mail for userids less than 1000." The "mailhub=" option is simple enough and it like the relayhost option in postfix. The "hostname=" is the strangest option. You would think that it should be set to the hostname. According to many ssmtp howtos out there, it should be set to an email address. In this case I set it to my email address at shaw (which I don't use), again, according to many howtos. Then there is the revaliases file. In here you can re-route mail to local accounts to to external email addresses. Too bad it doesn't work. Although it does work for some users apparently. The main thing I didn't like about ssmtp config files were the lack of documentation in them compared to postfix's. I definitely will never be returning to ssmtp now that everything works with postfix (and only took 2 minutes to set up).

MythTV 0.20 was released today. I can't wait to install it and check out the new features. I completely rebuilt my MythTV box last week and it is totally kick-ass now. It has a new 250GB hard drive which allows me to record 100 hours of TV. Some other improvments I made were, setting up XvMC which allows my video card's GPU to help out with some of the MPEG-2 decoding (bringing my CPU usage from 20% to 10% during video playback). Thanks to the Hauppauge PVR-350 card my CPU usage during recording is nil. I also got a new video card with composite out rather than s-video. That fixed a problem with the flaky s-video connector I had (whereby it would slip out and cause the screen to go all fuzzy) and also fixed a problem I had with 3 vertical lines on the TV during playback. We also bought and expensive composite cable to make sure that there are no problems. Finally stuck everything in an Antec Sonata II case for maximum silencing and cooling (not to mention how cool the black case looks).

Now that I have 100 hours of recording space (compared to 3 hours before) I set up a whole bunch of movies to record using the movie list view. It is amazing how many great movies are playing on TV at 3 am. I've also got the CBC National, Simpsons, Seinfeld, and the Colbert Report recorded daily (at any time on any channel, excluding already seen episodes), countless documentary programs and other miscellaneous things like Saturday Night Live. I love watching the TV shows I want, when I want, without commercials, and at faster speed if I want without the audio pitch getting screwed up. For example, I often watch the National at 1.4x speed without commercials. This means I can watch the entire hour-long news in about 32 minutes. Pretty sweet eh?

Crazy news tonight, that Daniel Robbins (gentoo founder) is returning to Gentoo. Gentoo rocks with or without him, but that would be awesome if he came back. No mention of it over on his blog yet, but I'll be sure to checking there over the next few days. Oops, I was wrong, there is an article on his blog. that apparently started it all. So looks like maybe he'll just be becoming a developer again, rather than resuming his former role of Chief Architect.

Here's all the info I have about the bug about it (grabbed from an IRC bot):

http://bugs.gentoo.org/145649 nor, P2, All, christel@gentoo.org->recruiters@gentoo.org, NEW, pending, New Developer: Daniel Robbins (drobbins)(kingtaco, due soon)

Also, Gentoo 2006.1 was released today. Doesn't really mean much to me as my systems are always updated on a daily basis. I kind of wished they had made gcc-4.1.1 stable on x86 prior to the 2006.1 release so that more users could have had a crack at it.

There is already one review written about it.